Merge pull request #8677 from Riduidel/fix/rabbitmq-message-not-serializable
[BEAM-7414] fix for message being not serializable due to LongString in headers
diff --git a/.gitattributes b/.gitattributes
index 58379be..6ac63eb 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -31,6 +31,5 @@
# git repository
.gitignore export-ignore
.gitattributes export-ignore
-gradlew export-ignore
-gradlew.bat export-ignore
+/gradlew* export-ignore
/gradle export-ignore
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 5b38bbd..1aa17d4 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -13,16 +13,17 @@
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
-Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
-Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
-Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
+Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
+Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
+Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
+XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
-Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
+Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
diff --git a/.gitignore b/.gitignore
index 900989d..f9cc388 100644
--- a/.gitignore
+++ b/.gitignore
@@ -43,7 +43,7 @@
sdks/python/README.md
sdks/python/apache_beam/portability/api/*pb2*.*
sdks/python/apache_beam/portability/api/*.yaml
-sdks/python/nosetests.xml
+sdks/python/nosetests*.xml
sdks/python/postcommit_requirements.txt
# Ignore IntelliJ files.
diff --git a/.test-infra/dataproc/create_flink_cluster.sh b/.test-infra/dataproc/create_flink_cluster.sh
deleted file mode 100755
index 4c3cd9a..0000000
--- a/.test-infra/dataproc/create_flink_cluster.sh
+++ /dev/null
@@ -1,138 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# Runs init actions for Docker, Portability framework (Beam) and Flink cluster
-# and opens an SSH tunnel to connect with Flink easily and run Beam jobs.
-#
-# Provide the following environment to run this script:
-#
-# GCLOUD_ZONE: Google cloud zone. Optional. Default: "us-central1-a"
-# DATAPROC_VERSION: Dataproc version. Optional. Default: 1.2
-# CLUSTER_NAME: Cluster name
-# GCS_BUCKET: GCS bucket url for Dataproc resources (init actions)
-# HARNESS_IMAGES_TO_PULL: Urls to SDK Harness' images to pull on dataproc workers (optional: 0, 1 or multiple urls for every harness image)
-# JOB_SERVER_IMAGE: Url to job server docker image to pull on dataproc master (optional)
-# ARTIFACTS_DIR: Url to bucket where artifacts will be stored for staging (optional)
-# FLINK_DOWNLOAD_URL: Url to Flink .tar archive to be installed on the cluster
-# FLINK_NUM_WORKERS: Number of Flink workers
-# DETACHED_MODE: Detached mode: should the SSH tunnel run in detached mode?
-#
-# Example usage:
-# CLUSTER_NAME=flink \
-# GCS_BUCKET=gs://<GCS_BUCKET>/flink \
-# HARNESS_IMAGES_TO_PULL='gcr.io/<IMAGE_REPOSITORY>/python:latest gcr.io/<IMAGE_REPOSITORY>/java:latest' \
-# JOB_SERVER_IMAGE=gcr.io/<IMAGE_REPOSITORY>/job-server-flink:latest \
-# ARTIFACTS_DIR=gs://<bucket-for-artifacts> \
-# FLINK_DOWNLOAD_URL=http://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.12.tgz \
-# FLINK_NUM_WORKERS=2 \
-# DETACHED_MODE=false \
-# ./create_flink_cluster.sh
-#
-set -Eeuxo pipefail
-
-# GCloud properties
-GCLOUD_ZONE="${GCLOUD_ZONE:=us-central1-a}"
-DATAPROC_VERSION="${DATAPROC_VERSION:=1.2}"
-
-MASTER_NAME="$CLUSTER_NAME-m"
-
-# GCS properties
-INIT_ACTIONS_FOLDER_NAME="init-actions"
-FLINK_INIT="$GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME/flink.sh"
-BEAM_INIT="$GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME/beam.sh"
-DOCKER_INIT="$GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME/docker.sh"
-
-# Flink properties
-FLINK_LOCAL_PORT=8081
-TASK_MANAGER_MEM=10240
-YARN_APPLICATION_MASTER=""
-
-function upload_init_actions() {
- echo "Uploading initialization actions to GCS bucket: $GCS_BUCKET"
- gsutil cp -r $INIT_ACTIONS_FOLDER_NAME/* $GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME
-}
-
-function get_leader() {
- local i=0
- local application_ids
- local application_masters
-
- echo "Yarn Applications"
- while read line; do
- echo $line
- application_ids[$i]=`echo $line | sed "s/ .*//"`
- application_masters[$i]=`echo $line | sed "s/.*$CLUSTER_NAME/$CLUSTER_NAME/" | sed "s/ .*//"`
- i=$((i+1))
- done <<< $(gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@$MASTER_NAME --command="yarn application -list" | grep "$CLUSTER_NAME")
-
- if [ $i != 1 ]; then
- echo "Multiple applications found. Make sure that only 1 application is running on the cluster."
- for app in ${application_ids[*]};
- do
- echo $app
- done
-
- echo "Execute 'gcloud compute ssh --zone=$GCLOUD_ZONE yarn@$MASTER_NAME --command=\"yarn application -kill <APP_NAME>\"' to kill the yarn application."
- exit 1
- fi
-
- YARN_APPLICATION_MASTER=${application_masters[0]}
- echo "Using Yarn Application master: $YARN_APPLICATION_MASTER"
-}
-
-function start_job_server() {
- gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@${MASTER_NAME} --command="sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud ${JOB_SERVER_IMAGE} --flink-master-url=${YARN_APPLICATION_MASTER} --artifacts-dir=${ARTIFACTS_DIR}"
-}
-
-function start_tunnel() {
- local job_server_config=`gcloud compute ssh --quiet --zone=$GCLOUD_ZONE yarn@$MASTER_NAME --command="curl -s \"http://$YARN_APPLICATION_MASTER/jobmanager/config\""`
- local key="jobmanager.rpc.port"
- local yarn_application_master_host=`echo $YARN_APPLICATION_MASTER | cut -d ":" -f1`
- local jobmanager_rpc_port=`echo $job_server_config | python -c "import sys, json; print [ e['value'] for e in json.load(sys.stdin) if e['key'] == u'$key'][0]"`
-
- local detached_mode_params=$([[ $DETACHED_MODE == "true" ]] && echo " -Nf >& /dev/null" || echo "")
-
- local job_server_ports_forwarding=$([[ -n "${JOB_SERVER_IMAGE:=}" ]] && echo "-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097" || echo "")
-
- local tunnel_command="gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@${MASTER_NAME} -- -L ${FLINK_LOCAL_PORT}:${YARN_APPLICATION_MASTER} -L ${jobmanager_rpc_port}:${yarn_application_master_host}:${jobmanager_rpc_port} ${job_server_ports_forwarding} -D 1080 ${detached_mode_params}"
-
- eval $tunnel_command
-}
-
-function create_cluster() {
- local metadata="flink-snapshot-url=${FLINK_DOWNLOAD_URL},"
- metadata+="flink-start-yarn-session=true"
-
- [[ -n "${HARNESS_IMAGES_TO_PULL:=}" ]] && metadata+=",beam-sdk-harness-images-to-pull=${HARNESS_IMAGES_TO_PULL}"
- [[ -n "${JOB_SERVER_IMAGE:=}" ]] && metadata+=",beam-job-server-image=${JOB_SERVER_IMAGE}"
-
- local image_version=$DATAPROC_VERSION
- echo "Starting dataproc cluster. Dataproc version: $image_version"
-
- # Docker init action restarts yarn so we need to start yarn session after this restart happens.
- # This is why flink init action is invoked last.
- gcloud dataproc clusters create $CLUSTER_NAME --num-workers=$FLINK_NUM_WORKERS --initialization-actions $DOCKER_INIT,$BEAM_INIT,$FLINK_INIT --metadata "${metadata}", --image-version=$image_version --zone=$GCLOUD_ZONE --quiet
-}
-
-function main() {
- upload_init_actions
- create_cluster
- get_leader
- [[ -n "${JOB_SERVER_IMAGE:=}" ]] && start_job_server
- start_tunnel
-}
-
-main "$@"
\ No newline at end of file
diff --git a/.test-infra/dataproc/flink_cluster.sh b/.test-infra/dataproc/flink_cluster.sh
new file mode 100755
index 0000000..86d9b23
--- /dev/null
+++ b/.test-infra/dataproc/flink_cluster.sh
@@ -0,0 +1,158 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Provide the following environment to run this script:
+#
+# GCLOUD_ZONE: Google cloud zone. Optional. Default: "us-central1-a"
+# DATAPROC_VERSION: Dataproc version. Optional. Default: 1.2
+# CLUSTER_NAME: Cluster name
+# GCS_BUCKET: GCS bucket url for Dataproc resources (init actions)
+# HARNESS_IMAGES_TO_PULL: Urls to SDK Harness' images to pull on dataproc workers (optional: 0, 1 or multiple urls for every harness image)
+# JOB_SERVER_IMAGE: Url to job server docker image to pull on dataproc master (optional)
+# ARTIFACTS_DIR: Url to bucket where artifacts will be stored for staging (optional)
+# FLINK_DOWNLOAD_URL: Url to Flink .tar archive to be installed on the cluster
+# FLINK_NUM_WORKERS: Number of Flink workers
+# FLINK_TASKMANAGER_SLOTS: Number of slots per Flink task manager
+# DETACHED_MODE: Detached mode: should the SSH tunnel run in detached mode?
+#
+# Example usage:
+# CLUSTER_NAME=flink \
+# GCS_BUCKET=gs://<GCS_BUCKET>/flink \
+# HARNESS_IMAGES_TO_PULL='gcr.io/<IMAGE_REPOSITORY>/python:latest gcr.io/<IMAGE_REPOSITORY>/java:latest' \
+# JOB_SERVER_IMAGE=gcr.io/<IMAGE_REPOSITORY>/job-server-flink:latest \
+# ARTIFACTS_DIR=gs://<bucket-for-artifacts> \
+# FLINK_DOWNLOAD_URL=http://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.12.tgz \
+# FLINK_NUM_WORKERS=2 \
+# FLINK_TASKMANAGER_SLOTS=1 \
+# DETACHED_MODE=false \
+# ./flink_cluster.sh create
+#
+set -Eeuxo pipefail
+
+# GCloud properties
+GCLOUD_ZONE="${GCLOUD_ZONE:=us-central1-a}"
+DATAPROC_VERSION="${DATAPROC_VERSION:=1.2}"
+
+MASTER_NAME="$CLUSTER_NAME-m"
+
+# GCS properties
+INIT_ACTIONS_FOLDER_NAME="init-actions"
+FLINK_INIT="$GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME/flink.sh"
+BEAM_INIT="$GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME/beam.sh"
+DOCKER_INIT="$GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME/docker.sh"
+
+# Flink properties
+FLINK_LOCAL_PORT=8081
+
+# By default each taskmanager has one slot - use that value to avoid sharing SDK Harness by multiple tasks.
+FLINK_TASKMANAGER_SLOTS="${FLINK_TASKMANAGER_SLOTS:=1}"
+
+TASK_MANAGER_MEM=10240
+YARN_APPLICATION_MASTER=""
+
+function upload_init_actions() {
+ echo "Uploading initialization actions to GCS bucket: $GCS_BUCKET"
+ gsutil cp -r $INIT_ACTIONS_FOLDER_NAME/* $GCS_BUCKET/$INIT_ACTIONS_FOLDER_NAME
+}
+
+function get_leader() {
+ local i=0
+ local application_ids
+ local application_masters
+
+ echo "Yarn Applications"
+ while read line; do
+ echo $line
+ application_ids[$i]=`echo $line | sed "s/ .*//"`
+ application_masters[$i]=`echo $line | sed "s/.*$CLUSTER_NAME/$CLUSTER_NAME/" | sed "s/ .*//"`
+ i=$((i+1))
+ done <<< $(gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@$MASTER_NAME --command="yarn application -list" | grep "$CLUSTER_NAME")
+
+ if [ $i != 1 ]; then
+ echo "Multiple applications found. Make sure that only 1 application is running on the cluster."
+ for app in ${application_ids[*]};
+ do
+ echo $app
+ done
+
+ echo "Execute 'gcloud compute ssh --zone=$GCLOUD_ZONE yarn@$MASTER_NAME --command=\"yarn application -kill <APP_NAME>\"' to kill the yarn application."
+ exit 1
+ fi
+
+ YARN_APPLICATION_MASTER=${application_masters[0]}
+ echo "Using Yarn Application master: $YARN_APPLICATION_MASTER"
+}
+
+function start_job_server() {
+ gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@${MASTER_NAME} --command="sudo --user yarn docker run --detach --publish 8099:8099 --publish 8098:8098 --publish 8097:8097 --volume ~/.config/gcloud:/root/.config/gcloud ${JOB_SERVER_IMAGE} --flink-master-url=${YARN_APPLICATION_MASTER} --artifacts-dir=${ARTIFACTS_DIR}"
+}
+
+function start_tunnel() {
+ local job_server_config=`gcloud compute ssh --quiet --zone=$GCLOUD_ZONE yarn@$MASTER_NAME --command="curl -s \"http://$YARN_APPLICATION_MASTER/jobmanager/config\""`
+ local key="jobmanager.rpc.port"
+ local yarn_application_master_host=`echo $YARN_APPLICATION_MASTER | cut -d ":" -f1`
+ local jobmanager_rpc_port=`echo $job_server_config | python -c "import sys, json; print [ e['value'] for e in json.load(sys.stdin) if e['key'] == u'$key'][0]"`
+
+ local detached_mode_params=$([[ $DETACHED_MODE == "true" ]] && echo " -Nf >& /dev/null" || echo "")
+
+ local job_server_ports_forwarding=$([[ -n "${JOB_SERVER_IMAGE:=}" ]] && echo "-L 8099:localhost:8099 -L 8098:localhost:8098 -L 8097:localhost:8097" || echo "")
+
+ local tunnel_command="gcloud compute ssh --zone=$GCLOUD_ZONE --quiet yarn@${MASTER_NAME} -- -L ${FLINK_LOCAL_PORT}:${YARN_APPLICATION_MASTER} -L ${jobmanager_rpc_port}:${yarn_application_master_host}:${jobmanager_rpc_port} ${job_server_ports_forwarding} -D 1080 ${detached_mode_params}"
+
+ eval $tunnel_command
+}
+
+function create_cluster() {
+ local metadata="flink-snapshot-url=${FLINK_DOWNLOAD_URL},"
+ metadata+="flink-start-yarn-session=true,"
+ metadata+="flink-taskmanager-slots=${FLINK_TASKMANAGER_SLOTS}"
+
+ [[ -n "${HARNESS_IMAGES_TO_PULL:=}" ]] && metadata+=",beam-sdk-harness-images-to-pull=${HARNESS_IMAGES_TO_PULL}"
+ [[ -n "${JOB_SERVER_IMAGE:=}" ]] && metadata+=",beam-job-server-image=${JOB_SERVER_IMAGE}"
+
+ local image_version=$DATAPROC_VERSION
+ echo "Starting dataproc cluster. Dataproc version: $image_version"
+
+ # Create one extra Dataproc VM for Flink's Job Manager
+ local num_dataproc_workers="$(($FLINK_NUM_WORKERS + 1))"
+
+ # Docker init action restarts yarn so we need to start yarn session after this restart happens.
+ # This is why flink init action is invoked last.
+ gcloud dataproc clusters create $CLUSTER_NAME --num-workers=$num_dataproc_workers --initialization-actions $DOCKER_INIT,$BEAM_INIT,$FLINK_INIT --metadata "${metadata}", --image-version=$image_version --zone=$GCLOUD_ZONE --quiet
+}
+
+# Runs init actions for Docker, Portability framework (Beam) and Flink cluster
+# and opens an SSH tunnel to connect with Flink easily and run Beam jobs.
+function create() {
+ upload_init_actions
+ create_cluster
+ get_leader
+ [[ -n "${JOB_SERVER_IMAGE:=}" ]] && start_job_server
+ start_tunnel
+}
+
+# Recreates a Flink cluster.
+function restart() {
+ delete
+ create
+}
+
+# Deletes a Flink cluster.
+function delete() {
+ gcloud dataproc clusters delete $CLUSTER_NAME --quiet
+}
+
+"$@"
diff --git a/.test-infra/dataproc/init-actions/flink.sh b/.test-infra/dataproc/init-actions/flink.sh
index 7dfcd36..1959872 100644
--- a/.test-infra/dataproc/init-actions/flink.sh
+++ b/.test-infra/dataproc/init-actions/flink.sh
@@ -56,6 +56,8 @@
# Set this to install flink from a snapshot URL instead of apt
readonly FLINK_SNAPSHOT_URL_METADATA_KEY='flink-snapshot-url'
+# Set this to define how many task slots are there per flink task manager
+readonly FLINK_TASKMANAGER_SLOTS_METADATA_KEY='flink-taskmanager-slots'
function err() {
@@ -111,17 +113,17 @@
# NB: This assumes > 1 worker node.
local num_taskmanagers="$(($num_workers - 1))"
- # Determine the number of task slots per worker.
- # TODO: Dataproc does not currently set the number of worker cores on the
- # master node. However, the spark configuration sets the number of executors
- # to be half the number of CPU cores per worker. We use this value to
- # determine the number of worker cores. Fix this hack when
- # yarn.nodemanager.resource.cpu-vcores is correctly populated.
- local spark_executor_cores=$(\
- grep 'spark\.executor\.cores' /etc/spark/conf/spark-defaults.conf \
- | tail -n1 \
- | cut -d'=' -f2)
- local flink_taskmanager_slots="$(($spark_executor_cores * 2))"
+ local num_cores="$(grep -c processor /proc/cpuinfo)"
+ local flink_taskmanager_slots_default=$num_cores
+
+ local slots="$(/usr/share/google/get_metadata_value \
+ "attributes/${START_FLINK_YARN_SESSION_METADATA_KEY}" \
+ || echo "${START_FLINK_YARN_SESSION_DEFAULT}")"
+
+ # if provided, use user defined number of slots.
+ local flink_taskmanager_slots="$(/usr/share/google/get_metadata_value \
+ "attributes/${FLINK_TASKMANAGER_SLOTS_METADATA_KEY}" \
+ || echo "${flink_taskmanager_slots_default}")"
# Determine the default parallelism.
local flink_parallelism=$(python -c \
diff --git a/.test-infra/dockerized-jenkins/README.md b/.test-infra/dockerized-jenkins/README.md
index 78874e2..fc1178e 100644
--- a/.test-infra/dockerized-jenkins/README.md
+++ b/.test-infra/dockerized-jenkins/README.md
@@ -144,6 +144,29 @@
1. Go to Jenkins -> New Item -> Freestyle project
1. Build step: Process Job DSLs
+## Additional Jenkins hints
+
+### Importing DSL jobs from a local git repository
+
+By default, Seed job imports DSL job definitions from the Apache Beam Github
+repository. But there is also a possibility to import these definitions from
+your local git repository, which makes testing much easier because you don't
+have to git push every time changes were made.
+
+1. Build Jenkins image using provided scripts.
+1. Provide an environment variable *BEAM_HOME* pointing to the beam root
+ directory, for example: `export BEAM_HOME=~/my/beam/directory`.
+1. Run image using the following command: `docker run -d -p 127.0.0.1:8080:8080
+ -v $BEAM_HOME:/var/jenkins_real_home/beam:ro beamjenkins`. The only difference is
+ the *-v* option which sets up a bind mount.
+1. Sign in to Jenkins.
+ 1. Go to the *sample_seed_job* and open its configuration. Scroll down to
+ the **Source Code Management** section.
+ 1. Fill the **Repository URL** field with *file:///var/jenkins_real_home/beam*.
+
+You can choose any branch from your local repo. Just remember that all changes
+must be committed. You don’t have to checkout the branch you chose.
+
## Additional docker hints
### Running image vs starting container
diff --git a/.test-infra/jenkins/CommonJobProperties.groovy b/.test-infra/jenkins/CommonJobProperties.groovy
index 2dddaad..653e9f6 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -58,7 +58,7 @@
}
branch('${sha1}')
extensions {
- cleanAfterCheckout()
+ wipeOutWorkspace()
relativeTargetDirectory(checkoutDir)
if (!allowRemotePoll) {
disableRemotePoll()
diff --git a/.test-infra/jenkins/CommonTestProperties.groovy b/.test-infra/jenkins/CommonTestProperties.groovy
index 2d8d596..203e398 100644
--- a/.test-infra/jenkins/CommonTestProperties.groovy
+++ b/.test-infra/jenkins/CommonTestProperties.groovy
@@ -29,17 +29,19 @@
SPARK("SparkRunner"),
FLINK("TestFlinkRunner"),
DIRECT("DirectRunner"),
+ PORTABLE("PortableRunner")
def RUNNER_DEPENDENCY_MAP = [
JAVA: [
- DATAFLOW: ":beam-runners-google-cloud-dataflow-java",
- SPARK: ":beam-runners-spark",
- FLINK: ":beam-runners-flink_2.11",
- DIRECT: ":beam-runners-direct-java"
+ DATAFLOW: ":runners:google-cloud-dataflow-java",
+ SPARK: ":runners:spark",
+ FLINK: ":runners:flink:1.8",
+ DIRECT: ":runners:direct-java"
],
PYTHON: [
DATAFLOW: "TestDataflowRunner",
DIRECT: "DirectRunner",
+ PORTABLE: "PortableRunner"
]
]
diff --git a/.test-infra/jenkins/Docker.groovy b/.test-infra/jenkins/Docker.groovy
new file mode 100644
index 0000000..561685c
--- /dev/null
+++ b/.test-infra/jenkins/Docker.groovy
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as common
+
+class Docker {
+ private def job
+ private String repositoryRoot
+
+ Docker(job, String repositoryRoot) {
+ this.job = job
+ this.repositoryRoot = repositoryRoot
+ }
+
+ /**
+ * Builds a Docker image from a gradle task and pushes it to the registry.
+ *
+ * @param gradleTask - name of a Gradle task
+ * @param imageName - name of a docker image
+ * @param imageTag - tag of a docker image
+ */
+ final void publish(String gradleTask, String imageName, String imageTag = 'latest') {
+ build(gradleTask, imageTag)
+ push(imageName, imageTag)
+ }
+
+ private void build(String gradleTask, String imageTag) {
+ job.steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ tasks(gradleTask)
+ switches("-Pdocker-repository-root=${repositoryRoot}")
+ switches("-Pdocker-tag=${imageTag}")
+ }
+ }
+ }
+
+ private void push(String imageName, String imageTag) {
+ String image = "${repositoryRoot}/${imageName}"
+ String targetImage = getFullImageName(imageName, imageTag)
+
+ job.steps {
+ shell("echo \"Tagging image\"...")
+ shell("docker tag ${image} ${targetImage}")
+ shell("echo \"Pushing image\"...")
+ shell("docker push ${targetImage}")
+ }
+ }
+
+ /**
+ * Returns the name of a docker image in the following format: <repositoryRoot>/<imageName>:<imageTag>
+ *
+ * @param imageName - name of a docker image
+ * @param imageTag - tag of a docker image
+ */
+ final String getFullImageName(String imageName, String imageTag = 'latest') {
+ String image = "${repositoryRoot}/${imageName}"
+ return "${image}:${imageTag}"
+ }
+}
diff --git a/.test-infra/jenkins/Flink.groovy b/.test-infra/jenkins/Flink.groovy
new file mode 100644
index 0000000..a986d64
--- /dev/null
+++ b/.test-infra/jenkins/Flink.groovy
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+class Flink {
+ private static final String flinkDownloadUrl = 'https://archive.apache.org/dist/flink/flink-1.7.0/flink-1.7.0-bin-hadoop28-scala_2.11.tgz'
+ private static final String FLINK_DIR = '"$WORKSPACE/src/.test-infra/dataproc"'
+ private static final String FLINK_SCRIPT = 'flink_cluster.sh'
+ private def job
+ private String jobName
+
+ Flink(job, String jobName) {
+ this.job = job
+ this.jobName = jobName
+ }
+
+ /**
+ * Creates Flink cluster and specifies cleanup steps.
+ *
+ * @param sdkHarnessImages - the list of published SDK Harness images tags
+ * @param workerCount - the initial number of worker nodes
+ * @param jobServerImage - the Flink job server image tag. If left empty, cluster will be set up without the job server.
+ * @param slotsPerTaskmanager - the number of slots per Flink task manager
+ */
+ void setUp(List<String> sdkHarnessImages, Integer workerCount, String jobServerImage = '', Integer slotsPerTaskmanager = 1) {
+ setupFlinkCluster(sdkHarnessImages, workerCount, jobServerImage, slotsPerTaskmanager)
+ addTeardownFlinkStep()
+ }
+
+ private void setupFlinkCluster(List<String> sdkHarnessImages, Integer workerCount, String jobServerImage, Integer slotsPerTaskmanager) {
+ String gcsBucket = 'gs://beam-flink-cluster'
+ String clusterName = getClusterName()
+ String artifactsDir = "${gcsBucket}/${clusterName}"
+ String imagesToPull = sdkHarnessImages.join(' ')
+
+ job.steps {
+ environmentVariables {
+ env("GCLOUD_ZONE", "us-central1-a")
+ env("CLUSTER_NAME", clusterName)
+ env("GCS_BUCKET", gcsBucket)
+ env("FLINK_DOWNLOAD_URL", flinkDownloadUrl)
+ env("FLINK_NUM_WORKERS", workerCount)
+ env("FLINK_TASKMANAGER_SLOTS", slotsPerTaskmanager)
+ env("DETACHED_MODE", 'true')
+
+ if(imagesToPull) {
+ env("HARNESS_IMAGES_TO_PULL", imagesToPull)
+ }
+
+ if(jobServerImage) {
+ env("JOB_SERVER_IMAGE", jobServerImage)
+ env("ARTIFACTS_DIR", artifactsDir)
+ }
+ }
+
+ shell('echo Setting up flink cluster')
+ shell("cd ${FLINK_DIR}; ./${FLINK_SCRIPT} create")
+ }
+ }
+
+ /**
+ * Updates the number of worker nodes in a cluster.
+ *
+ * @param workerCount - the new number of worker nodes in the cluster
+ */
+ void scaleCluster(Integer workerCount) {
+ job.steps {
+ shell("echo Changing number of workers to ${workerCount}")
+ environmentVariables {
+ env("FLINK_NUM_WORKERS", workerCount)
+ }
+ shell("cd ${FLINK_DIR}; ./${FLINK_SCRIPT} restart")
+ }
+ }
+
+ private GString getClusterName() {
+ return "${jobName.toLowerCase().replace("_", "-")}-\$BUILD_ID"
+ }
+
+ private void addTeardownFlinkStep() {
+ job.publishers {
+ postBuildScripts {
+ steps {
+ shell("cd ${FLINK_DIR}; ./${FLINK_SCRIPT} delete")
+ }
+ onlyIfBuildSucceeds(false)
+ onlyIfBuildFails(false)
+ }
+ }
+ }
+}
diff --git a/.test-infra/jenkins/Kubernetes.groovy b/.test-infra/jenkins/Kubernetes.groovy
new file mode 100644
index 0000000..6e50383
--- /dev/null
+++ b/.test-infra/jenkins/Kubernetes.groovy
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/** Facilitates creation of jenkins steps to setup and cleanup Kubernetes infrastructure. */
+class Kubernetes {
+
+ private static final String KUBERNETES_DIR = '"$WORKSPACE/src/.test-infra/kubernetes"'
+
+ private static final String KUBERNETES_SCRIPT = "${KUBERNETES_DIR}/kubernetes.sh"
+
+ private static def job
+
+ private static String kubeconfigLocation
+
+ private static String namespace
+
+ private Kubernetes(job, String kubeconfigLocation, String namespace) {
+ this.job = job
+ this.kubeconfigLocation = kubeconfigLocation
+ this.namespace = namespace
+ }
+
+ /**
+ * Creates separate kubeconfig, kubernetes namespace and specifies related cleanup steps.
+ *
+ * @param job - jenkins job
+ * @param kubeconfigLocation - place where kubeconfig will be created
+ * @param namepsace - kubernetes namespace
+ */
+ static Kubernetes create(job, String kubeconfigLocation, String namespace) {
+ Kubernetes kubernetes = new Kubernetes(job, kubeconfigLocation, namespace)
+ setupKubeconfig()
+ setupNamespace()
+ addCleanupSteps()
+ return kubernetes
+ }
+
+ private static void setupKubeconfig() {
+ job.steps {
+ shell("cp /home/jenkins/.kube/config ${kubeconfigLocation}")
+ environmentVariables {
+ env('KUBECONFIG', kubeconfigLocation)
+ }
+ }
+ }
+
+ private static void setupNamespace() {
+ job.steps {
+ shell("${KUBERNETES_SCRIPT} createNamespace ${namespace}")
+ environmentVariables {
+ env('KUBERNETES_NAMESPACE', namespace)
+ }
+ }
+ }
+
+ private static void addCleanupSteps() {
+ job.publishers {
+ postBuildScripts {
+ steps {
+ shell("${KUBERNETES_SCRIPT} deleteNamespace ${namespace}")
+ shell("rm ${kubeconfigLocation}")
+ }
+ onlyIfBuildSucceeds(false)
+ onlyIfBuildFails(false)
+ }
+ }
+ }
+
+ /**
+ * Specifies steps to run Kubernetes .yaml script.
+ */
+ void apply(String pathToScript) {
+ job.steps {
+ shell("${KUBERNETES_SCRIPT} apply ${pathToScript}")
+ }
+ }
+
+ /**
+ * Specifies steps that will save specified load balancer serivce address
+ * as an environment variable that can be used in later steps if needed.
+ *
+ * @param serviceName - name of the load balancer Kubernetes service
+ * @param referenceName - name of the environment variable
+ */
+ void loadBalancerIP(String serviceName, String referenceName) {
+ job.steps {
+ String command = "${KUBERNETES_SCRIPT} loadBalancerIP ${serviceName}"
+ shell("set -eo pipefail; eval ${command} | sed 's/^/${referenceName}=/' > job.properties")
+ environmentVariables {
+ propertiesFile('job.properties')
+ }
+ }
+ }
+}
diff --git a/.test-infra/jenkins/LoadTestsBuilder.groovy b/.test-infra/jenkins/LoadTestsBuilder.groovy
index 08bbb30..c259033 100644
--- a/.test-infra/jenkins/LoadTestsBuilder.groovy
+++ b/.test-infra/jenkins/LoadTestsBuilder.groovy
@@ -22,16 +22,21 @@
import CommonTestProperties.TriggeringContext
class LoadTestsBuilder {
- static void loadTest(context, String title, Runner runner, SDK sdk, Map<String, ?> options, String mainClass, TriggeringContext triggeringContext) {
+ final static String DOCKER_CONTAINER_REGISTRY = 'gcr.io/apache-beam-testing/beam_portability'
- options.put('runner', runner.option)
+ static void loadTests(scope, CommonTestProperties.SDK sdk, List testConfigurations, String test, String mode){
+ scope.description("Runs ${sdk.toString().toLowerCase().capitalize()} ${test} load tests in ${mode} mode")
- String datasetKey = 'bigQueryDataset'
- String datasetValue = options.get(datasetKey)
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
- if (datasetValue) {
- options.put(datasetKey, setContextualDatasetName(datasetValue, triggeringContext))
+ for (testConfiguration in testConfigurations) {
+ loadTest(scope, testConfiguration.title, testConfiguration.runner, sdk, testConfiguration.pipelineOptions, testConfiguration.test)
}
+ }
+
+
+ static void loadTest(context, String title, Runner runner, SDK sdk, Map<String, ?> options, String mainClass) {
+ options.put('runner', runner.option)
context.steps {
shell("echo *** ${title} ***")
@@ -46,11 +51,19 @@
}
}
+ static String getBigQueryDataset(String baseName, TriggeringContext triggeringContext) {
+ if (triggeringContext == TriggeringContext.PR) {
+ return baseName + '_PRs'
+ } else {
+ return baseName
+ }
+ }
+
private static String getGradleTaskName(SDK sdk) {
if (sdk == SDK.JAVA) {
- return ':beam-sdks-java-load-tests:run'
+ return ':sdks:java:testing:load-tests:run'
} else if (sdk == SDK.PYTHON) {
- return ':beam-sdks-python-load-tests:run'
+ return ':sdks:python:apache_beam:testing:load_tests:run'
} else {
throw new RuntimeException("No task name defined for SDK: $SDK")
}
@@ -61,14 +74,6 @@
"--${it.key}=$it.value".replace('\"', '\\\"').replace('\'', '\\\'')
}.join(' ')
}
-
- private static String setContextualDatasetName(String baseName, TriggeringContext triggeringContext) {
- if (triggeringContext == TriggeringContext.PR) {
- return baseName + '_PRs'
- } else {
- return baseName
- }
- }
}
diff --git a/.test-infra/jenkins/NexmarkBuilder.groovy b/.test-infra/jenkins/NexmarkBuilder.groovy
index f2b0c24..32a4e13 100644
--- a/.test-infra/jenkins/NexmarkBuilder.groovy
+++ b/.test-infra/jenkins/NexmarkBuilder.groovy
@@ -79,7 +79,7 @@
shell("echo *** RUN ${title} ***")
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
switches("-Pnexmark.runner=${runner.getDepenedencyBySDK(sdk)}")
switches("-Pnexmark.args=\"${parseOptions(options)}\"")
diff --git a/.test-infra/jenkins/README.md b/.test-infra/jenkins/README.md
index 22376bd..9643fb2 100644
--- a/.test-infra/jenkins/README.md
+++ b/.test-infra/jenkins/README.md
@@ -35,8 +35,9 @@
| beam_PreCommit_JavaPortabilityApi | [commit](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Phrase/) | `Run JavaPortabilityApi PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_JavaPortabilityApi_Cron) |
| beam_PreCommit_Java_Examples_Dataflow | [commit](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/) | `Run Java PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Cron) |
| beam_PreCommit_Portable_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/) | `Run Portable PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron) |
+| beam_PreCommit_PythonLint | [commit](https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_PythonLint_Phrase/) | `Run PythonLint PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron) |
| beam_PreCommit_Python | [commit](https://builds.apache.org/job/beam_PreCommit_Python_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron) |
-| beam_PreCommit_Python_PVR_Flink | [commit](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron) |
+| beam_PreCommit_Python2_PVR_Flink | [commit](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Phrase/) | `Run Python PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron) |
| beam_PreCommit_RAT | [commit](https://builds.apache.org/job/beam_PreCommit_RAT_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_RAT_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_RAT_Phrase/) | `Run RAT PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_RAT_Cron) |
| beam_PreCommit_Spotless | [commit](https://builds.apache.org/job/beam_PreCommit_Spotless_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Spotless_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Spotless_Phrase/) | `Run Spotless PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Spotless_Cron) |
| beam_PreCommit_Website | [commit](https://builds.apache.org/job/beam_PreCommit_Website_Commit/), [cron](https://builds.apache.org/job/beam_PreCommit_Website_Cron/), [phrase](https://builds.apache.org/job/beam_PreCommit_Website_Phrase/) | `Run Website PreCommit` | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron) |
@@ -47,6 +48,8 @@
| Name | Link | PR Trigger Phrase | Cron Status |
|------|------|-------------------|-------------|
| beam_PostCommit_Go | [cron](https://builds.apache.org/job/beam_PostCommit_Go/), [phrase](https://builds.apache.org/job/beam_PostCommit_Go_PR/) | `Run Go PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Go) |
+| beam_PostCommit_Go_VR_Flink | [cron](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink_PR/) | `Run Go Flink ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/) |
+| beam_PostCommit_Go_VR_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark_PR/) | `Run Go Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/) |
| beam_PostCommit_Java | [cron](https://builds.apache.org/job/beam_PostCommit_Java/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PR/) | `Run Java PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Java) |
| beam_PostCommit_Java_Nexmark_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_PR/) | `Dataflow Runner Nexmark Tests` | [](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow) |
| beam_PostCommit_Java_Nexmark_Direct | [cron](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Direct/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Direct_PR/) | `Direct Runner Nexmark Tests` | [](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Direct) |
@@ -54,6 +57,7 @@
| beam_PostCommit_Java_Nexmark_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark_PR/) | `Spark Runner Nexmark Tests` | [](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark) |
| beam_PostCommit_Java_PVR_Flink_Batch | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch_PR/) | `Run Java Flink PortableValidatesRunner Batch` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch) |
| beam_PostCommit_Java_PVR_Flink_Streaming | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming_PR/) | `Run Java Flink PortableValidatesRunner Streaming` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming) |
+| beam_PostCommit_Java_PVR_Spark_Batch | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch_PR/) | `Run Java Spark PortableValidatesRunner Batch` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch) |
| beam_PostCommit_Java_PortabilityApi | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi_PR/) | `Run Java PortabilityApi PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi) |
| beam_PostCommit_Java_ValidatesRunner_Apex | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_PR/) | `Run Apex ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex) |
| beam_PostCommit_Java_ValidatesRunner_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/) | `Run Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow) |
@@ -67,8 +71,12 @@
| beam_PostCommit_Java_ValidatesRunner_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_PR/) | `Run Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark) |
| beam_PostCommit_Py_VR_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_PR/) | `Run Python Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow) |
| beam_PostCommit_Py_ValCont | [cron](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/), [phrase](https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/) | `Run Python Dataflow ValidatesContainer` | [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont) |
-| beam_PostCommit_Python_Verify | [cron](https://builds.apache.org/job/beam_PostCommit_Python_Verify/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python_Verify_PR/) | `Run Python PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify) |
-| beam_PostCommit_Python3_Verify | [cron](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python3_Verify_PR/) | `Run Python PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python3_Verify) |
+| beam_PostCommit_Python35_VR_Flink | [cron](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/) | `Run Python 3.5 Flink ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink) |
+| beam_PostCommit_Python_VR_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/) | `Run Python Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark) |
+| beam_PostCommit_Python2 | [cron](https://builds.apache.org/job/beam_PostCommit_Python2), [phrase](https://builds.apache.org/job/beam_PostCommit_Python2_PR/) | `Run Python 2 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python2) |
+| beam_PostCommit_Python35 | [cron](https://builds.apache.org/job/beam_PostCommit_Python35), [phrase](https://builds.apache.org/job/beam_PostCommit_Python35_PR/) | `Run Python 3.5 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python35) |
+| beam_PostCommit_Python36 | [cron](https://builds.apache.org/job/beam_PostCommit_Python36), [phrase](https://builds.apache.org/job/beam_PostCommit_Python36_PR/) | `Run Python 3.6 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python36) |
+| beam_PostCommit_Python37 | [cron](https://builds.apache.org/job/beam_PostCommit_Python37), [phrase](https://builds.apache.org/job/beam_PostCommit_Python37_PR/) | `Run Python 3.7 PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python37) |
| beam_PostCommit_SQL | [cron](https://builds.apache.org/job/beam_PostCommit_SQL/), [phrase](https://builds.apache.org/job/beam_PostCommit_SQL_PR/) | `Run SQL PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_SQL) |
| beam_PostCommit_Website_Publish | [cron](https://builds.apache.org/job/beam_PostCommit_Website_Publish/) | N/A | [](https://builds.apache.org/job/beam_PostCommit_Website_Publish) |
@@ -83,33 +91,58 @@
| beam_PerformanceTests_JDBC | [cron](https://builds.apache.org/job/beam_PerformanceTests_JDBC/) | `Run Java JdbcIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_JDBC) |
| beam_PerformanceTests_ManyFiles_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS/) | `Run Java ManyFilesTextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_ManyFiles_TextIOIT_HDFS) |
| beam_PerformanceTests_ParquetIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT_HDFS/) | `Run Java ParquetIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_ParquetIOIT_HDFS) |
-| beam_PerformanceTests_Python | [cron](https://builds.apache.org/job/beam_PerformanceTests_Python/) | `Run Python Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Python) |
-| beam_PerformanceTests_Python35 | [cron](https://builds.apache.org/job/beam_PerformanceTests_Python35/) | `Run Python35 Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Python35) |
| beam_PerformanceTests_Spark | [cron](https://builds.apache.org/job/beam_PerformanceTests_Spark/) | `Run Spark Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_Spark) |
| beam_PerformanceTests_TFRecordIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/) | `Run Java JdbcIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT) |
| beam_PerformanceTests_TextIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/) | `Run Java TextIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS) |
+| beam_PerformanceTests_WordCountIT_Py27 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27/) | `Run Python27 WordCountIT Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py27) |
+| beam_PerformanceTests_WordCountIT_Py35 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35/) | `Run Python35 WordCountIT Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py35) |
+| beam_PerformanceTests_WordCountIT_Py36 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/) | `Run Python36 WordCountIT Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36) |
+| beam_PerformanceTests_WordCountIT_Py37 | [cron](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/) | `Run Python37 WordCountIT Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37) |
| beam_PerformanceTests_XmlIOIT | [cron](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT/), [hdfs_cron](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT_HDFS/) | `Run Java XmlIO Performance Test` | [](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT) [](https://builds.apache.org/job/beam_PerformanceTests_XmlIOIT_HDFS) |
+### Load test Jobs
+
+| Name | Link | PR Trigger Phrase | Cron Status |
+|------|------|-------------------|-------------|
+| beam_LoadTests_Python_coGBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch_PR/) | Run Load Tests Python CoGBK Flink Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_coGBK_Flink_Batch/) |
+| beam_LoadTests_Java_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch_PR/) | Run Load Tests Java CoGBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Batch/) |
+| beam_LoadTests_Java_CoGBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming_PR/) | Run Load Tests Java CoGBK Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_Streaming/) |
+| beam_LoadTests_Python_CoGBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch_PR/) | Run Load Tests Python CoGBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_CoGBK_Dataflow_Batch/) |
+| beam_LoadTests_Python_Combine_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch_PR/) | Run Load Tests Python Combine Flink Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Flink_Batch/) |
+| beam_LoadTests_Java_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch_PR/) | Run Load Tests Java Combine Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Batch/) |
+| beam_LoadTests_Java_Combine_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming_PR/) | Run Load Tests Java Combine Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Dataflow_Streaming/) |
+| beam_LoadTests_Python_Combine_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch_PR/) | Run Python Load Tests Combine Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_Combine_Dataflow_Batch/) |
+| beam_LoadTests_Python_GBK_Flink_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/) | Run Load Tests Python GBK Flink Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch/) |
+| beam_LoadTests_Java_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch_PR/) | Run Load Tests Java GBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Batch/) |
+| beam_LoadTests_Java_GBK_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming_PR/) | Run Load Tests Java GBK Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_GBK_Dataflow_Streaming/) |
+| beam_LoadTests_Python_GBK_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch_PR/) | Run Load Tests Python GBK Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_Dataflow_Batch/) |
+| beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch_PR/) | Run Load Tests Python GBK reiterate Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch/) |
+| beam_Java_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Java_LoadTests_Smoke_PR/) | Run Java Load Tests Smoke | |
+| beam_Python_LoadTests_Smoke | [phrase](https://builds.apache.org/job/beam_Python_LoadTests_Smoke_PR/) | Run Python Load Tests Smoke | |
+| beam_LoadTests_Java_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch_PR/) | Run Load Tests Java ParDo Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Batch/) |
+| beam_LoadTests_Java_ParDo_Dataflow_Streaming | [cron](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/), [phrase](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming_PR/) | Run Load Tests Java ParDo Dataflow Streaming | [](https://builds.apache.org/job/beam_LoadTests_Java_ParDo_Dataflow_Streaming/) |
+| beam_LoadTests_Python_ParDo_Dataflow_Batch | [cron](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/), [phrase](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch_PR/) | Run Python Load Tests ParDo Dataflow Batch | [](https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Dataflow_Batch/) |
+
### Inventory Jobs
| Name | Link | PR Trigger Phrase | Cron Status |
|------|------|-------------------|-------------|
-| beam_Inventory_beam1 | [cron](https://builds.apache.org/job/beam_Inventory_beam1/) | `Run inventory on beam1` | [](https://builds.apache.org/job/beam_Inventory_beam1) |
-| beam_Inventory_beam2 | [cron](https://builds.apache.org/job/beam_Inventory_beam2/) | `Run inventory on beam2` | [](https://builds.apache.org/job/beam_Inventory_beam2) |
-| beam_Inventory_beam3 | [cron](https://builds.apache.org/job/beam_Inventory_beam3/) | `Run inventory on beam3` | [](https://builds.apache.org/job/beam_Inventory_beam3) |
-| beam_Inventory_beam4 | [cron](https://builds.apache.org/job/beam_Inventory_beam4/) | `Run inventory on beam4` | [](https://builds.apache.org/job/beam_Inventory_beam4) |
-| beam_Inventory_beam5 | [cron](https://builds.apache.org/job/beam_Inventory_beam5/) | `Run inventory on beam5` | [](https://builds.apache.org/job/beam_Inventory_beam5) |
-| beam_Inventory_beam6 | [cron](https://builds.apache.org/job/beam_Inventory_beam6/) | `Run inventory on beam6` | [](https://builds.apache.org/job/beam_Inventory_beam6) |
-| beam_Inventory_beam7 | [cron](https://builds.apache.org/job/beam_Inventory_beam7/) | `Run inventory on beam7` | [](https://builds.apache.org/job/beam_Inventory_beam7) |
-| beam_Inventory_beam8 | [cron](https://builds.apache.org/job/beam_Inventory_beam8/) | `Run inventory on beam8` | [](https://builds.apache.org/job/beam_Inventory_beam8) |
-| beam_Inventory_beam9 | [cron](https://builds.apache.org/job/beam_Inventory_beam9/) | `Run inventory on beam9` | [](https://builds.apache.org/job/beam_Inventory_beam9) |
-| beam_Inventory_beam10 | [cron](https://builds.apache.org/job/beam_Inventory_beam10/) | `Run inventory on beam10` | [](https://builds.apache.org/job/beam_Inventory_beam10) |
-| beam_Inventory_beam11 | [cron](https://builds.apache.org/job/beam_Inventory_beam11/) | `Run inventory on beam11` | [](https://builds.apache.org/job/beam_Inventory_beam11) |
-| beam_Inventory_beam12 | [cron](https://builds.apache.org/job/beam_Inventory_beam12/) | `Run inventory on beam12` | [](https://builds.apache.org/job/beam_Inventory_beam12) |
-| beam_Inventory_beam13 | [cron](https://builds.apache.org/job/beam_Inventory_beam13/) | `Run inventory on beam13` | [](https://builds.apache.org/job/beam_Inventory_beam13) |
-| beam_Inventory_beam14 | [cron](https://builds.apache.org/job/beam_Inventory_beam14/) | `Run inventory on beam14` | [](https://builds.apache.org/job/beam_Inventory_beam14) |
-| beam_Inventory_beam15 | [cron](https://builds.apache.org/job/beam_Inventory_beam15/) | `Run inventory on beam15` | [](https://builds.apache.org/job/beam_Inventory_beam15) |
-| beam_Inventory_beam16 | [cron](https://builds.apache.org/job/beam_Inventory_beam16/) | `Run inventory on beam16` | [](https://builds.apache.org/job/beam_Inventory_beam16) |
+| beam_Inventory_apache-beam-jenkins-1 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-1/) | `Run inventory apache-beam-jenkins-1` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-1) |
+| beam_Inventory_apache-beam-jenkins-2 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-2/) | `Run inventory apache-beam-jenkins-2` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-2) |
+| beam_Inventory_apache-beam-jenkins-3 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-3/) | `Run inventory apache-beam-jenkins-3` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-3) |
+| beam_Inventory_apache-beam-jenkins-4 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-4/) | `Run inventory apache-beam-jenkins-4` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-4) |
+| beam_Inventory_apache-beam-jenkins-5 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-5/) | `Run inventory apache-beam-jenkins-5` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-5) |
+| beam_Inventory_apache-beam-jenkins-6 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-6/) | `Run inventory apache-beam-jenkins-6` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-6) |
+| beam_Inventory_apache-beam-jenkins-7 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-7/) | `Run inventory apache-beam-jenkins-7` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-7) |
+| beam_Inventory_apache-beam-jenkins-8 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-8/) | `Run inventory apache-beam-jenkins-8` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-8) |
+| beam_Inventory_apache-beam-jenkins-9 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-9/) | `Run inventory apache-beam-jenkins-9` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-9) |
+| beam_Inventory_apache-beam-jenkins-10 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-10/) | `Run inventory apache-beam-jenkins-10` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-10) |
+| beam_Inventory_apache-beam-jenkins-11 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-11/) | `Run inventory apache-beam-jenkins-11` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-11) |
+| beam_Inventory_apache-beam-jenkins-12 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-12/) | `Run inventory apache-beam-jenkins-12` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-12) |
+| beam_Inventory_apache-beam-jenkins-13 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-13/) | `Run inventory apache-beam-jenkins-13` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-13) |
+| beam_Inventory_apache-beam-jenkins-14 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-14/) | `Run inventory apache-beam-jenkins-14` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-14) |
+| beam_Inventory_apache-beam-jenkins-15 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-15/) | `Run inventory apache-beam-jenkins-15` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-15) |
+| beam_Inventory_apache-beam-jenkins-16 | [cron](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-16/) | `Run inventory apache-beam-jenkins-16` | [](https://builds.apache.org/job/beam_Inventory_apache-beam-jenkins-16) |
### Other Jobs
@@ -120,7 +153,7 @@
| beam_Release_Python_NightlySnapshot | [cron](https://builds.apache.org/job/beam_Release_Python_NightlySnapshot/) | `Run Python Publish` | [](https://builds.apache.org/job/beam_Release_Python_NightlySnapshot) |
| beam_PostRelease_NightlySnapshot | [cron](https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/) | `Run Dataflow PostRelease` | [](https://builds.apache.org/job/beam_PostRelease_NightlySnapshot) |
| beam_Prober_CommunityMetrics | [cron](https://builds.apache.org/job/beam_Prober_CommunityMetrics/) | `Run Community Metrics Prober` | [](https://builds.apache.org/job/beam_Prober_CommunityMetrics) |
-| beam_SeedJob | [cron](https://builds.apache.org/job/beam_SeedJob/), [standalone](https://builds.apache.org/job/beam_SeedJob_Standalone/) | `Run Dependency Check` | [](https://builds.apache.org/job/beam_SeedJob) |
+| beam_SeedJob | [cron](https://builds.apache.org/job/beam_SeedJob/), [standalone](https://builds.apache.org/job/beam_SeedJob_Standalone/) | `Run Seed Job` | [](https://builds.apache.org/job/beam_SeedJob) |
| beam_sonarqube_report | [cron](https://builds.apache.org/job/beam_sonarqube_report/)| N/A | [](https://builds.apache.org/job/beam_sonarqube_report/) |
### Notes:
diff --git a/.test-infra/jenkins/dependency_check/dependency_check_report_generator_test.py b/.test-infra/jenkins/dependency_check/dependency_check_report_generator_test.py
index 82f5ffe..0cc0789 100644
--- a/.test-infra/jenkins/dependency_check/dependency_check_report_generator_test.py
+++ b/.test-infra/jenkins/dependency_check/dependency_check_report_generator_test.py
@@ -23,7 +23,7 @@
import unittest, mock
from mock import patch, mock_open
from datetime import datetime
-from dependency_check_report_generator import prioritize_dependencies
+from .dependency_check_report_generator import prioritize_dependencies
_PROJECT_ID = 'mock-apache-beam-testing'
@@ -70,7 +70,7 @@
Test on empty outdated dependencies.
Expect: empty report
"""
- with patch('__builtin__.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
+ with patch('builtins.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
report = prioritize_dependencies([], _SDK_TYPE)
self.assertEqual(len(report), 0)
@@ -95,7 +95,7 @@
" - group3:artifact3 [1.0.0 -> 1.1.0]",
" - group4:artifact4 [1.0.0 -> 1.1.0]"
]
- with patch('__builtin__.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
+ with patch('builtins.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
report = prioritize_dependencies(deps, _SDK_TYPE)
self.assertEqual(len(report), 3)
self.assertIn('group1:artifact1', report[0])
@@ -114,7 +114,7 @@
Expect: group1:artifact1
"""
deps = [" - group1:artifact1 [Release1-123 -> Release2-456]"]
- with patch('__builtin__.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
+ with patch('builtins.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
report = prioritize_dependencies(deps, _SDK_TYPE)
self.assertEqual(len(report), 1)
self.assertIn('group1:artifact1', report[0])
@@ -131,7 +131,7 @@
Expect: group1:artifact1
"""
deps = [" - group1:artifact1 [0.rc1.0 -> 0.rc2.0]"]
- with patch('__builtin__.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
+ with patch('builtins.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
report = prioritize_dependencies(deps, _SDK_TYPE)
self.assertEqual(len(report), 1)
self.assertIn('group1:artifact1', report[0])
@@ -150,7 +150,7 @@
"- group1:artifact1 (1.0.0, 2.0.0)",
" - group2:artifact2 [1.0.0 -> 2.0.0]"
]
- with patch('__builtin__.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
+ with patch('builtins.open', mock_open(read_data=_MOCKED_OWNERS_FILE)):
report = prioritize_dependencies(deps, _SDK_TYPE)
self.assertEqual(len(report), 1)
self.assertIn('group2:artifact2', report[0])
diff --git a/.test-infra/jenkins/dependency_check/generate_report.sh b/.test-infra/jenkins/dependency_check/generate_report.sh
index 734aa44..1ac2746 100755
--- a/.test-infra/jenkins/dependency_check/generate_report.sh
+++ b/.test-infra/jenkins/dependency_check/generate_report.sh
@@ -39,7 +39,7 @@
# Virtualenv for the rest of the script to run setup
-/usr/bin/virtualenv dependency/check
+virtualenv dependency/check
. dependency/check/bin/activate
pip install --upgrade google-cloud-bigquery
pip install --upgrade google-cloud-bigtable
@@ -55,7 +55,7 @@
echo "<html><body>" > $WORKSPACE/src/build/dependencyUpdates/beam-dependency-check-report.html
-python -m dependency_check/dependency_check_report_generator Python
+python -m dependency_check.dependency_check_report_generator Python
python -m dependency_check.dependency_check_report_generator Java
diff --git a/.test-infra/jenkins/dependency_check/version_comparer_test.py b/.test-infra/jenkins/dependency_check/version_comparer_test.py
index 78b67e1..b37fb5f 100644
--- a/.test-infra/jenkins/dependency_check/version_comparer_test.py
+++ b/.test-infra/jenkins/dependency_check/version_comparer_test.py
@@ -16,7 +16,7 @@
# limitations under the License.
#
-import version_comparer
+from . import version_comparer
import unittest
class VersionComparerTest(unittest.TestCase):
diff --git a/.test-infra/jenkins/jira_utils/jira_manager.py b/.test-infra/jenkins/jira_utils/jira_manager.py
index 979ed7c..718f019 100644
--- a/.test-infra/jenkins/jira_utils/jira_manager.py
+++ b/.test-infra/jenkins/jira_utils/jira_manager.py
@@ -23,7 +23,7 @@
import dependency_check.version_comparer as version_comparer
from datetime import datetime
-from jira_client import JiraClient
+from .jira_client import JiraClient
_JIRA_PROJECT_NAME = 'BEAM'
diff --git a/.test-infra/jenkins/jira_utils/jira_manager_test.py b/.test-infra/jenkins/jira_utils/jira_manager_test.py
index 0c9afa6..6514568 100644
--- a/.test-infra/jenkins/jira_utils/jira_manager_test.py
+++ b/.test-infra/jenkins/jira_utils/jira_manager_test.py
@@ -19,7 +19,7 @@
import unittest, mock
import jira_utils
from mock import patch, mock_open, Mock
-from jira_manager import JiraManager
+from .jira_manager import JiraManager
from datetime import datetime
MOCKED_DEP_CURRENT_VERSION = '0.1.0'
@@ -67,7 +67,7 @@
dep0:
owners: owner0, owner1 , owner2,
"""
- with patch('__builtin__.open', mock_open(read_data=owners_yaml)):
+ with patch('builtins.open', mock_open(read_data=owners_yaml)):
manager = JiraManager('url', 'username', 'password', owners_yaml)
manager.run('dep0',
MOCKED_DEP_CURRENT_VERSION,
@@ -94,7 +94,7 @@
summary = self._get_experct_summary(dep_name)
description = self._get_expected_description(dep_name, MOCKED_DEP_LATEST_VERSION, [])
- with patch('__builtin__.open', mock_open(read_data=owners_yaml)):
+ with patch('builtins.open', mock_open(read_data=owners_yaml)):
with patch('jira_utils.jira_manager.JiraManager._search_issues',
return_value=[MockedJiraIssue('BEAM-1000', summary, description, 'Open')]):
manager = JiraManager('url', 'username', 'password', owners_yaml)
@@ -122,7 +122,7 @@
summary = self._get_experct_summary('group0')
description = self._get_expected_description(dep_name, MOCKED_DEP_LATEST_VERSION, [])
- with patch('__builtin__.open', mock_open(read_data=owners_yaml)):
+ with patch('builtins.open', mock_open(read_data=owners_yaml)):
with patch('jira_utils.jira_manager.JiraManager._search_issues',
side_effect = [[MockedJiraIssue('BEAM-1000', summary, description, 'Open')],
[]]):
@@ -156,7 +156,7 @@
"""
summary = self._get_experct_summary('group0')
description = self._get_expected_description(dep_name, MOCKED_DEP_LATEST_VERSION, [])
- with patch('__builtin__.open', mock_open(read_data=owners_yaml)):
+ with patch('builtins.open', mock_open(read_data=owners_yaml)):
with patch('jira_utils.jira_manager.JiraManager._search_issues',
side_effect = [[MockedJiraIssue('BEAM-1000', summary, description, 'Closed')],
[]]):
@@ -185,7 +185,7 @@
"""
summary = self._get_experct_summary(dep_name)
description = self._get_expected_description(dep_name, MOCKED_DEP_LATEST_VERSION, [])
- with patch('__builtin__.open', mock_open(read_data=owners_yaml)):
+ with patch('builtins.open', mock_open(read_data=owners_yaml)):
with patch('jira_utils.jira_manager.JiraManager._search_issues',
side_effect = [[MockedJiraIssue('BEAM-1000', summary, description, 'Closed')],
[]]):
@@ -215,7 +215,7 @@
"""
summary = self._get_experct_summary(dep_name)
description = self._get_expected_description(dep_name, issue_closed_version, [])
- with patch('__builtin__.open', mock_open(read_data=owners_yaml)):
+ with patch('builtins.open', mock_open(read_data=owners_yaml)):
with patch('jira_utils.jira_manager.JiraManager._search_issues',
side_effect = [[MockedJiraIssue('BEAM-1000', summary, description, 'Closed')],
[]]):
diff --git a/.test-infra/jenkins/job_Inventory.groovy b/.test-infra/jenkins/job_Inventory.groovy
index 32e0645..fec1ba2 100644
--- a/.test-infra/jenkins/job_Inventory.groovy
+++ b/.test-infra/jenkins/job_Inventory.groovy
@@ -35,7 +35,7 @@
// Allows triggering this build against pull requests.
commonJobProperties.enablePhraseTriggeringFromPullRequest(
delegate,
- 'Machine Inventory',
+ "Machine Inventory ${machine}",
"Run Inventory ${machine}")
parameters {
@@ -66,6 +66,7 @@
shell('virtualenv -p python3.7 test37 && . ./test37/bin/activate && python --version && deactivate || echo "python 3.7 not found"')
shell('echo "Maven home $MAVEN_HOME"')
shell('env')
+ shell('docker system prune --all --filter until=24h --force')
}
}
}
diff --git a/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy b/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy
new file mode 100644
index 0000000..9c5a535
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def loadTestConfigurations = { mode, isStreaming, datasetName ->
+ [
+ [
+ title : 'Load test: CoGBK 2GB 100 byte records - single key',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ appName : "load_tests_Java_Dataflow_${mode}_CoGBK_1",
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ publishToBigQuery : true,
+ bigQueryDataset : datasetName,
+ bigQueryTable : "java_dataflow_${mode}_CoGBK_1",
+ sourceOptions : """
+ {
+ "numRecords": 20000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 1
+ }
+ """.trim().replaceAll("\\s", ""),
+ coSourceOptions : """
+ {
+ "numRecords": 2000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 1000
+ }
+ """.trim().replaceAll("\\s", ""),
+ iterations : 1,
+ numWorkers : 5,
+ autoscalingAlgorithm : "NONE",
+ streaming : isStreaming
+ ]
+ ],
+ [
+ title : 'Load test: CoGBK 2GB 100 byte records - multiple keys',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ appName : "load_tests_Java_Dataflow_${mode}_CoGBK_2",
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ publishToBigQuery : true,
+ bigQueryDataset : datasetName,
+ bigQueryTable : "java_dataflow_${mode}_CoGBK_2",
+ sourceOptions : """
+ {
+ "numRecords": 20000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 5
+ }
+ """.trim().replaceAll("\\s", ""),
+ coSourceOptions : """
+ {
+ "numRecords": 2000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 1000
+ }
+ """.trim().replaceAll("\\s", ""),
+ iterations : 1,
+ numWorkers : 5,
+ autoscalingAlgorithm : "NONE",
+ streaming : isStreaming
+ ]
+ ],
+ [
+
+ title : 'Load test: CoGBK 2GB reiteration 10kB value',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ appName : "load_tests_Java_Dataflow_${mode}_CoGBK_3",
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ publishToBigQuery : true,
+ bigQueryDataset : datasetName,
+ bigQueryTable : "java_dataflow_${mode}_CoGBK_3",
+ sourceOptions : """
+ {
+ "numRecords": 2000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 200000
+ }
+ """.trim().replaceAll("\\s", ""),
+ coSourceOptions : """
+ {
+ "numRecords": 2000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 1000
+ }
+ """.trim().replaceAll("\\s", ""),
+ iterations : 4,
+ numWorkers : 5,
+ autoscalingAlgorithm : "NONE",
+ streaming : isStreaming
+ ]
+
+ ],
+ [
+ title : 'Load test: CoGBK 2GB reiteration 2MB value',
+ test : 'org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ appName : "load_tests_Java_Dataflow_${mode}_CoGBK_4",
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ publishToBigQuery : true,
+ bigQueryDataset : datasetName,
+ bigQueryTable : "java_dataflow_${mode}_CoGBK_4",
+ sourceOptions : """
+ {
+ "numRecords": 2000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 1000
+ }
+ """.trim().replaceAll("\\s", ""),
+ coSourceOptions : """
+ {
+ "numRecords": 2000000,
+ "keySizeBytes": 10,
+ "valueSizeBytes": 90,
+ "numHotKeys": 1000
+ }
+ """.trim().replaceAll("\\s", ""),
+ iterations : 4,
+ numWorkers : 5,
+ autoscalingAlgorithm : "NONE",
+ streaming : isStreaming
+ ]
+ ]
+ ]
+}
+
+def streamingLoadTestJob = { scope, triggeringContext ->
+ scope.description('Runs Java CoGBK load tests on Dataflow runner in streaming mode')
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in loadTestConfigurations('streaming', true, datasetName)) {
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200, coInputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
+ }
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Java_CoGBK_Dataflow_Streaming', 'H 12 * * *', this) {
+ streamingLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Java_CoGBK_Dataflow_Streaming',
+ 'Run Load Tests Java CoGBK Dataflow Streaming',
+ 'Load Tests Java CoGBK Dataflow Streaming suite',
+ this
+) {
+ streamingLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+
+def batchLoadTestJob = { scope, triggeringContext ->
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.JAVA, loadTestConfigurations('batch', false, datasetName), "CoGBK", "batch")
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Java_CoGBK_Dataflow_Batch', 'H 14 * * *', this) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Java_CoGBK_Dataflow_Batch',
+ 'Run Load Tests Java CoGBK Dataflow Batch',
+ 'Load Tests Java CoGBK Dataflow Batch suite',
+ this
+) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy
new file mode 100644
index 0000000..2f847e7
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import Flink
+import Docker
+
+String now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def scenarios = { datasetName, sdkHarnessImageTag -> [
+ [
+ title : 'Combine Python Load test: 2GB 10 byte records',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-combine-1-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query: false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_combine_1',
+ input_options : '\'{' +
+ '"num_records": 200000000,' +
+ '"key_size": 1,' +
+ '"value_size": 9}\'',
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ top_count : 20,
+ ]
+ ],
+ [
+ title : 'Combine Python Load test: 2GB Fanout 4',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-combine-4-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query: false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_combine_4',
+ input_options : '\'{' +
+ '"num_records": 5000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ parallelism : 16,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ fanout : 4,
+ top_count : 20,
+ ]
+ ],
+ [
+ title : 'Combine Python Load test: 2GB Fanout 8',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-combine-5-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query: false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_combine_5',
+ input_options : '\'{' +
+ '"num_records": 2500000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ parallelism : 16,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ fanout : 8,
+ top_count : 20,
+ ]
+ ]
+]}
+
+def batchLoadTestJob = { scope, triggeringContext ->
+ scope.description('Runs Python Combine load tests on Flink runner in batch mode')
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+
+ Docker publisher = new Docker(scope, loadTestsBuilder.DOCKER_CONTAINER_REGISTRY)
+ String pythonHarnessImageTag = publisher.getFullImageName('python2.7_sdk')
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ def numberOfWorkers = 16
+ List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
+
+ publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
+ publisher.publish(':runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ def flink = new Flink(scope, 'beam_LoadTests_Python_Combine_Flink_Batch')
+ flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
+
+ defineTestSteps(scope, testScenarios, [
+ 'Combine Python Load test: 2GB Fanout 4',
+ 'Combine Python Load test: 2GB Fanout 8'
+ ])
+
+ numberOfWorkers = 5
+ flink.scaleCluster(numberOfWorkers)
+
+ defineTestSteps(scope, testScenarios, ['Combine Python Load test: 2GB 10 byte records'])
+}
+
+private List<Map> defineTestSteps(scope, List<Map> testScenarios, List<String> titles) {
+ return testScenarios
+ .findAll { it.title in titles }
+ .forEach {
+ loadTestsBuilder.loadTest(scope, it.title, it.runner, CommonTestProperties.SDK.PYTHON, it.pipelineOptions, it.test)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_Combine_Flink_Batch',
+ 'Run Load Tests Python Combine Flink Batch',
+ 'Load Tests Python Combine Flink Batch suite',
+ this
+) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_Combine_Flink_Batch', 'H 15 * * *', this) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy b/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy
index 5e41cea..6d35339 100644
--- a/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Combine_Java.groovy
@@ -16,110 +16,52 @@
* limitations under the License.
*/
+
import CommonJobProperties as commonJobProperties
import CommonTestProperties
+import CronJobBuilder
import LoadTestsBuilder as loadTestsBuilder
import PhraseTriggeringPostCommitBuilder
-import CronJobBuilder
-def commonLoadTestConfig = { jobType, isStreaming ->
- [
- [
- title : 'Load test: 2GB of 10B records',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
- project : 'apache-beam-testing',
- appName : "load_tests_Java_Dataflow_${jobType}_Combine_1",
- tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
- publishToBigQuery : true,
- bigQueryDataset : 'load_test',
- bigQueryTable : "java_dataflow_${jobType}_Combine_1",
- sourceOptions : """
+def commonLoadTestConfig = { jobType, isStreaming, datasetName ->
+ [
+ [
+ title : 'Load test: 2GB of 10B records',
+ test : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ appName : "load_tests_Java_Dataflow_${jobType}_Combine_1",
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ publishToBigQuery : true,
+ bigQueryDataset : datasetName,
+ bigQueryTable : "java_dataflow_${jobType}_Combine_1",
+ sourceOptions : """
{
"numRecords": 200000000,
"keySizeBytes": 1,
"valueSizeBytes": 9
}
""".trim().replaceAll("\\s", ""),
- fanout : 1,
- iterations : 1,
- topCount : 20,
- maxNumWorkers : 5,
- numWorkers : 5,
- autoscalingAlgorithm: "NONE",
- perKeyCombiner : "TOP_LARGEST",
- streaming : isStreaming
- ]
- ],
- [
- title : 'Load test: 2GB of 100B records',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
- project : 'apache-beam-testing',
- appName : "load_tests_Java_Dataflow_${jobType}_Combine_2",
- tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
- publishToBigQuery : true,
- bigQueryDataset : 'load_test',
- bigQueryTable : "java_dataflow_${jobType}_Combine_2",
- sourceOptions : """
- {
- "numRecords": 20000000,
- "keySizeBytes": 10,
- "valueSizeBytes": 90
- }
- """.trim().replaceAll("\\s", ""),
- fanout : 1,
- iterations : 1,
- topCount : 20,
- maxNumWorkers : 5,
- numWorkers : 5,
- autoscalingAlgorithm: "NONE",
- perKeyCombiner : "TOP_LARGEST",
- streaming : isStreaming
- ]
- ],
- [
-
- title : 'Load test: 2GB of 100kB records',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
- project : 'apache-beam-testing',
- appName : "load_tests_Java_Dataflow_${jobType}_Combine_3",
- tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
- publishToBigQuery : true,
- bigQueryDataset : 'load_test',
- bigQueryTable : "java_dataflow_${jobType}_Combine_3",
- sourceOptions : """
- {
- "numRecords": 2000,
- "keySizeBytes": 100000,
- "valueSizeBytes": 900000
- }
- """.trim().replaceAll("\\s", ""),
- fanout : 1,
- iterations : 1,
- topCount : 20,
- maxNumWorkers : 5,
- numWorkers : 5,
- autoscalingAlgorithm: "NONE",
- perKeyCombiner : "TOP_LARGEST",
- streaming : isStreaming
- ]
-
- ],
- [
- title : 'Load test: fanout 4 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ fanout : 1,
+ iterations : 1,
+ topCount : 20,
+ numWorkers : 5,
+ autoscalingAlgorithm: "NONE",
+ perKeyCombiner : "TOP_LARGEST",
+ streaming : isStreaming
+ ]
+ ],
+ [
+ title : 'Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_Combine_4",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${jobType}_Combine_4",
sourceOptions : """
{
@@ -131,7 +73,6 @@
fanout : 4,
iterations : 1,
topCount : 20,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm: "NONE",
perKeyCombiner : "TOP_LARGEST",
@@ -139,15 +80,15 @@
]
],
[
- title : 'Load test: fanout 8 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.CombineLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_Combine_5",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${jobType}_Combine_5",
sourceOptions : """
{
@@ -159,7 +100,6 @@
fanout : 8,
iterations : 1,
topCount : 20,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm: "NONE",
perKeyCombiner : "TOP_LARGEST",
@@ -171,21 +111,18 @@
def batchLoadTestJob = { scope, triggeringContext ->
- scope.description('Runs Java Combine load tests on Dataflow runner in batch mode')
- commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
-
- for (testConfiguration in commonLoadTestConfig('batch', false)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, triggeringContext)
- }
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.JAVA, commonLoadTestConfig('batch', false, datasetName), "Combine", "batch")
}
def streamingLoadTestJob = {scope, triggeringContext ->
scope.description('Runs Java Combine load tests on Dataflow runner in streaming mode')
commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
- for (testConfiguration in commonLoadTestConfig('streaming', true)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, triggeringContext)
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in commonLoadTestConfig('streaming', true, datasetName)) {
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
@@ -213,4 +150,4 @@
this
) {
streamingLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
-}
\ No newline at end of file
+}
diff --git a/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy b/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy
new file mode 100644
index 0000000..cf18a58
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = { datasetName -> [
+ [
+ title : 'Combine Python Load test: 2GB 10 byte records',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-combine-1-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_combine_1',
+ input_options : '\'{' +
+ '"num_records": 200000000,' +
+ '"key_size": 1,' +
+ '"value_size": 9}\'',
+ num_workers : 5,
+ autoscaling_algorithm: "NONE",
+ top_count : 20,
+ ]
+ ],
+ [
+ title : 'Combine Python Load test: 2GB Fanout 4',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-combine-4-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_combine_4',
+ input_options : '\'{' +
+ '"num_records": 5000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ num_workers : 16,
+ autoscaling_algorithm: "NONE",
+ fanout : 4,
+ top_count : 20,
+ ]
+ ],
+ [
+ title : 'Combine Python Load test: 2GB Fanout 8',
+ test : 'apache_beam.testing.load_tests.combine_test:CombineTest.testCombineGlobally',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-combine-5-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_combine_5',
+ input_options : '\'{' +
+ '"num_records": 2500000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ num_workers : 16,
+ autoscaling_algorithm: "NONE",
+ fanout : 8,
+ top_count : 20,
+ ]
+ ],
+]}
+
+def batchLoadTestJob = { scope, triggeringContext ->
+ scope.description('Runs Python Combine load tests on Dataflow runner in batch mode')
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 120)
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in loadTestConfigurations(datasetName)) {
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_Combine_Dataflow_Batch',
+ 'Run Python Load Tests Combine Dataflow Batch',
+ 'Load Tests Python Combine Dataflow Batch suite',
+ this
+) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_Combine_Dataflow_Batch', 'H 15 * * *', this) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy
new file mode 100644
index 0000000..7f3bb21
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import Flink
+import Docker
+
+String now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def scenarios = { datasetName, sdkHarnessImageTag -> [
+ [
+ title : 'Load test: 2GB of 10B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_1_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_1",
+ input_options : '\'{"num_records": 200000000,"key_size": 1,"value_size":9}\'',
+ iterations : 1,
+ fanout : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ],
+ [
+ title : 'Load test: 2GB of 100B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_2_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_2",
+ input_options : '\'{"num_records": 20000000,"key_size": 10,"value_size":90}\'',
+ iterations : 1,
+ fanout : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ],
+ [
+ title : 'Load test: 2GB of 100kB records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_3_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_3",
+ input_options : '\'{"num_records": 2000,"key_size": 100000,"value_size":900000}\'',
+ iterations : 1,
+ fanout : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ],
+ [
+ title : 'Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_4_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_4",
+ input_options : '\'{"num_records": 5000000,"key_size": 10,"value_size":90}\'',
+ iterations : 1,
+ fanout : 4,
+ parallelism : 16,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ],
+ [
+ title : 'Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_5_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_5",
+ input_options : '\'{"num_records": 2500000,"key_size": 10,"value_size":90}\'',
+ iterations : 1,
+ fanout : 8,
+ parallelism : 16,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ],
+ [
+ title : 'Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_6_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_6",
+ input_options : '\'{"num_records": 20000000,"key_size": 10,"value_size":90, "num_hot_keys": 200, "hot_key_fraction": 1}\'',
+ iterations : 4,
+ fanout : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ],
+ [
+ title : 'Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : "load_tests_Python_Flink_Batch_GBK_7_${now}",
+ publish_to_big_query: false,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_GBK_7",
+ input_options : '\'{"num_records": 20000000,"key_size": 10,"value_size":90, "num_hot_keys": 10, "hot_key_fraction": 1}\'',
+ iterations : 4,
+ fanout : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER'
+ ]
+ ]
+ ]}
+
+
+def loadTest = { scope, triggeringContext ->
+ Docker publisher = new Docker(scope, loadTestsBuilder.DOCKER_CONTAINER_REGISTRY)
+ def sdk = CommonTestProperties.SDK.PYTHON
+ String pythonHarnessImageTag = publisher.getFullImageName('python2.7_sdk')
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ def numberOfWorkers = 16
+ List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
+
+ publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
+ publisher.publish(':runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ def flink = new Flink(scope, 'beam_LoadTests_Python_GBK_Flink_Batch')
+ flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
+
+ def configurations = testScenarios.findAll { it.pipelineOptions?.parallelism?.value == numberOfWorkers }
+ loadTestsBuilder.loadTests(scope, sdk, configurations, "GBK", "batch")
+
+ numberOfWorkers = 5
+ flink.scaleCluster(numberOfWorkers)
+
+ configurations = testScenarios.findAll { it.pipelineOptions?.parallelism?.value == numberOfWorkers }
+ loadTestsBuilder.loadTests(scope, sdk, configurations, "GBK", "batch")
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_GBK_Flink_Batch',
+ 'Run Load Tests Python GBK Flink Batch',
+ 'Load Tests Python GBK Flink Batch suite',
+ this
+) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_GBK_Flink_Batch', 'H 12 * * *', this) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
index 80c64ff..b048c54 100644
--- a/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Java.groovy
@@ -22,18 +22,18 @@
import PhraseTriggeringPostCommitBuilder
import CronJobBuilder
-def loadTestConfigurations = { mode, isStreaming ->
+def loadTestConfigurations = { mode, isStreaming, datasetName ->
[
[
- title : 'Load test: 2GB of 10B records',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 10B records',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_1",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_1",
sourceOptions : """
{
@@ -44,22 +44,21 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: 2GB of 100B records',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 100B records',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_2",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_2",
sourceOptions : """
{
@@ -70,7 +69,6 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -78,15 +76,15 @@
],
[
- title : 'Load test: 2GB of 100kB records',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: 2GB of 100kB records',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_3",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_3",
sourceOptions : """
{
@@ -97,7 +95,6 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 1,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -105,15 +102,15 @@
],
[
- title : 'Load test: fanout 4 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : 'load_tests_Java_Dataflow_${mode}_GBK_4',
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_4",
sourceOptions : """
{
@@ -124,22 +121,21 @@
""".trim().replaceAll("\\s", ""),
fanout : 4,
iterations : 1,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: fanout 8 times with 2GB 10-byte records total',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_5",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_5",
sourceOptions : """
{
@@ -150,22 +146,21 @@
""".trim().replaceAll("\\s", ""),
fanout : 8,
iterations : 1,
- maxNumWorkers : 16,
numWorkers : 16,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: reiterate 4 times 10kB values',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: reiterate 4 times 10kB values',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_6",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_6",
sourceOptions : """
{
@@ -178,22 +173,21 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 4,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: reiterate 4 times 2MB values',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: reiterate 4 times 2MB values',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${mode}_GBK_7",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${mode}_GBK_7",
sourceOptions : """
{
@@ -206,7 +200,6 @@
""".trim().replaceAll("\\s", ""),
fanout : 1,
iterations : 4,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm : "NONE",
streaming : isStreaming
@@ -219,9 +212,10 @@
scope.description('Runs Java GBK load tests on Dataflow runner in streaming mode')
commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
- for (testConfiguration in loadTestConfigurations('streaming', true)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, triggeringContext)
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in loadTestConfigurations('streaming', true, datasetName)) {
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
@@ -240,12 +234,8 @@
def batchLoadTestJob = { scope, triggeringContext ->
- scope.description('Runs Java GBK load tests on Dataflow runner in batch mode')
- commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
-
- for (testConfiguration in loadTestConfigurations('batch', false)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, triggeringContext)
- }
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.JAVA, loadTestConfigurations('batch', false, datasetName), "GBK", "batch")
}
CronJobBuilder.cronJob('beam_LoadTests_Java_GBK_Dataflow_Batch', 'H 14 * * *', this) {
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy
new file mode 100644
index 0000000..b4b8c68
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = { datasetName -> [
+ [
+ title : 'GroupByKey Python Load test: 2GB of 10B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-gbk-1-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_gbk_1',
+ input_options : '\'{"num_records": 200000000,' +
+ '"key_size": 1,' +
+ '"value_size": 9}\'',
+ iterations : 1,
+ fanout : 1,
+ num_workers : 5,
+ autoscaling_algorithm: "NONE"
+ ]
+ ],
+ [
+ title : 'GroupByKey Python Load test: 2GB of 100B records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-gbk-2-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_gbk_2',
+ input_options : '\'{"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ fanout : 1,
+ num_workers : 5,
+ autoscaling_algorithm: "NONE"
+ ]
+ ],
+ [
+ title : 'GroupByKey Python Load test: 2GB of 100kB records',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-gbk-3-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_gbk_3',
+ input_options : '\'{"num_records": 2000,' +
+ '"key_size": 100000,' +
+ '"value_size": 900000}\'',
+ iterations : 1,
+ fanout : 1,
+ num_workers : 5,
+ autoscaling_algorithm: "NONE"
+ ]
+ ],
+ [
+ title : 'GroupByKey Python Load test: fanout 4 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-gbk-4-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_gbk_4',
+ input_options : '\'{"num_records": 5000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ fanout : 4,
+ num_workers : 5,
+ autoscaling_algorithm: "NONE"
+ ]
+ ],
+ [
+ title : 'GroupByKey Python Load test: fanout 8 times with 2GB 10-byte records total',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-gbk-5-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_gbk_5',
+ input_options : '\'{"num_records": 2500000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ fanout : 8,
+ num_workers : 5,
+ autoscaling_algorithm: "NONE"
+ ]
+ ],
+]}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_GBK_Dataflow_Batch',
+ 'Run Load Tests Python GBK Dataflow Batch',
+ 'Load Tests Python GBK Dataflow Batch suite',
+ this
+) {
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', CommonTestProperties.TriggeringContext.PR)
+ loadTestsBuilder.loadTests(delegate, CommonTestProperties.SDK.PYTHON, loadTestConfigurations(datasetName), "GBK", "batch")
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_GBK_Dataflow_Batch', 'H 12 * * *', this) {
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', CommonTestProperties.TriggeringContext.POST_COMMIT)
+ loadTestsBuilder.loadTests(delegate, CommonTestProperties.SDK.PYTHON, loadTestConfigurations(datasetName), "GBK", "batch")
+}
+
diff --git a/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy b/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy
new file mode 100644
index 0000000..5bb1ac4
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = { datasetName -> [
+ [
+ title : 'GroupByKey Python Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-dataflow-batch-gbk-6-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : "python_dataflow_batch_gbk_6",
+ input_options : '\'{"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 200,' +
+ '"hot_key_fraction": 1}\'',
+ fanout : 1,
+ iterations : 4,
+ num_workers : 5,
+ autoscaling_algorithm: "NONE"
+ ]
+ ],
+ [
+ title : 'GroupByKey Python Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-dataflow-batch-gbk-7-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_gbk_7',
+ input_options : '\'{"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 10,' +
+ '"hot_key_fraction": 1}\'',
+ fanout : 1,
+ iterations : 4,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE'
+ ]
+ ]
+]}
+
+def batchLoadTestJob = { scope, triggeringContext ->
+ scope.description('Runs Python GBK reiterate load tests on Dataflow runner in batch mode')
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in loadTestConfigurations(datasetName)) {
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
+ }
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch', 'H 14 * * *', this) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_GBK_reiterate_Dataflow_Batch',
+ 'Run Load Tests Python GBK reiterate Dataflow Batch',
+ 'Load Tests Python GBK reiterate Dataflow Batch suite',
+ this
+) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy b/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy
index 63a1428..62fc180 100644
--- a/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy
+++ b/.test-infra/jenkins/job_LoadTests_Java_Smoke.groovy
@@ -16,19 +16,18 @@
* limitations under the License.
*/
-import CommonJobProperties as commonJobProperties
import CommonTestProperties
import LoadTestsBuilder as loadTestsBuilder
import PhraseTriggeringPostCommitBuilder
-def smokeTestConfigurations = [
+def smokeTestConfigurations = { datasetName -> [
[
- title : 'GroupByKey load test Direct',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DIRECT,
- jobProperties: [
+ title : 'GroupByKey load test Direct',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DIRECT,
+ pipelineOptions: [
publishToBigQuery: true,
- bigQueryDataset : 'load_test_SMOKE',
+ bigQueryDataset : datasetName,
bigQueryTable : 'direct_gbk',
sourceOptions : '{"numRecords":100000,"splitPointFrequencyRecords":1}',
stepOptions : '{"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true}',
@@ -37,14 +36,14 @@
]
],
[
- title : 'GroupByKey load test Dataflow',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'GroupByKey load test Dataflow',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
tempLocation : 'gs://temp-storage-for-perf-tests/smoketests',
publishToBigQuery: true,
- bigQueryDataset : 'load_test_SMOKE',
+ bigQueryDataset : datasetName,
bigQueryTable : 'dataflow_gbk',
sourceOptions : '{"numRecords":100000,"splitPointFrequencyRecords":1}',
stepOptions : '{"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true}',
@@ -53,12 +52,12 @@
]
],
[
- title : 'GroupByKey load test Flink',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.FLINK,
- jobProperties: [
+ title : 'GroupByKey load test Flink',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.FLINK,
+ pipelineOptions: [
publishToBigQuery: true,
- bigQueryDataset : 'load_test_SMOKE',
+ bigQueryDataset : datasetName,
bigQueryTable : 'flink_gbk',
sourceOptions : '{"numRecords":100000,"splitPointFrequencyRecords":1}',
stepOptions : '{"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true}',
@@ -67,13 +66,13 @@
]
],
[
- title : 'GroupByKey load test Spark',
- itClass : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
- runner : CommonTestProperties.Runner.SPARK,
- jobProperties: [
+ title : 'GroupByKey load test Spark',
+ test : 'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.SPARK,
+ pipelineOptions: [
sparkMaster : 'local[4]',
publishToBigQuery: true,
- bigQueryDataset : 'load_test_SMOKE',
+ bigQueryDataset : datasetName,
bigQueryTable : 'spark_gbk',
sourceOptions : '{"numRecords":100000,"splitPointFrequencyRecords":1}',
stepOptions : '{"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true}',
@@ -81,7 +80,7 @@
iterations : 1,
]
]
-]
+]}
// Runs a tiny version load test suite to ensure nothing is broken.
@@ -91,10 +90,6 @@
'Java Load Tests Smoke',
this
) {
- description("Runs load tests in \"smoke\" mode to check if everything works well")
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
-
- for (testConfiguration in smokeTestConfigurations) {
- loadTestsBuilder.loadTest(delegate, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, CommonTestProperties.TriggeringContext.PR)
- }
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test_SMOKE', CommonTestProperties.TriggeringContext.PR)
+ loadTestsBuilder.loadTests(delegate, CommonTestProperties.SDK.JAVA, smokeTestConfigurations(datasetName), "GBK", "smoke")
}
diff --git a/.test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
new file mode 100644
index 0000000..97c37dd
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import Flink
+import Docker
+
+String now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def scenarios = { datasetName, sdkHarnessImageTag -> [
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-1-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_1',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 10,
+ number_of_counter_operations: 0,
+ number_of_counters : 0,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 200 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-2-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_2',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 200,
+ number_of_counter_operations: 0,
+ number_of_counters : 0,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-3-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_3',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ number_of_counter_operations: 10,
+ number_of_counters : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 100 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ job_name : 'load-tests-python-flink-batch-pardo-4-' + now,
+ project : 'apache-beam-testing',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_pardo_4',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ number_of_counter_operations: 100,
+ number_of_counters : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+]}
+
+def loadTest = { scope, triggeringContext ->
+ Docker publisher = new Docker(scope, loadTestsBuilder.DOCKER_CONTAINER_REGISTRY)
+ String pythonHarnessImageTag = publisher.getFullImageName('python2.7_sdk')
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ def numberOfWorkers = 5
+ List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
+
+ publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
+ publisher.publish(':runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ Flink flink = new Flink(scope, 'beam_LoadTests_Python_ParDo_Flink_Batch')
+ flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
+
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.PYTHON, testScenarios, 'ParDo', 'batch')
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_ParDo_Flink_Batch',
+ 'Run Python Load Tests ParDo Flink Batch',
+ 'Load Tests Python ParDo Flink Batch suite',
+ this
+) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_ParDo_Flink_Batch', 'H 13 * * *', this) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy b/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
index 6729e0f3..2969c72 100644
--- a/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
+++ b/.test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
@@ -22,18 +22,18 @@
import PhraseTriggeringPostCommitBuilder
import CronJobBuilder
-def commonLoadTestConfig = { jobType, isStreaming ->
+def commonLoadTestConfig = { jobType, isStreaming, datasetName ->
[
[
- title : 'Load test: ParDo 2GB 100 byte records 10 times',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 10 times',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_1",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${jobType}_ParDo_1",
sourceOptions : """
{
@@ -45,22 +45,21 @@
iterations : 10,
numberOfCounters : 1,
numberOfCounterOperations: 0,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
]
],
[
- title : 'Load test: ParDo 2GB 100 byte records 200 times',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 200 times',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_2",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${jobType}_ParDo_2",
sourceOptions : """
{
@@ -72,7 +71,6 @@
iterations : 200,
numberOfCounters : 1,
numberOfCounterOperations: 0,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
@@ -80,15 +78,15 @@
],
[
- title : 'Load test: ParDo 2GB 100 byte records 10 counters',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 10 counters',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_3",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${jobType}_ParDo_3",
sourceOptions : """
{
@@ -97,10 +95,9 @@
"valueSizeBytes": 90
}
""".trim().replaceAll("\\s", ""),
- iterations : 10,
+ iterations : 1,
numberOfCounters : 1,
numberOfCounterOperations: 10,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
@@ -108,15 +105,15 @@
],
[
- title : 'Load test: ParDo 2GB 100 byte records 100 counters',
- itClass : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
- runner : CommonTestProperties.Runner.DATAFLOW,
- jobProperties: [
+ title : 'Load test: ParDo 2GB 100 byte records 100 counters',
+ test : 'org.apache.beam.sdk.loadtests.ParDoLoadTest',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
project : 'apache-beam-testing',
appName : "load_tests_Java_Dataflow_${jobType}_ParDo_4",
tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
publishToBigQuery : true,
- bigQueryDataset : 'load_test',
+ bigQueryDataset : datasetName,
bigQueryTable : "java_dataflow_${jobType}_ParDo_4",
sourceOptions : """
{
@@ -125,10 +122,9 @@
"valueSizeBytes": 90
}
""".trim().replaceAll("\\s", ""),
- iterations : 10,
+ iterations : 1,
numberOfCounters : 1,
numberOfCounterOperations: 100,
- maxNumWorkers : 5,
numWorkers : 5,
autoscalingAlgorithm: "NONE",
streaming : isStreaming
@@ -139,21 +135,18 @@
def batchLoadTestJob = { scope, triggeringContext ->
- scope.description('Runs Java ParDo load tests on Dataflow runner in batch mode')
- commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
-
- for (testConfiguration in commonLoadTestConfig('batch', false)) {
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, triggeringContext)
- }
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.JAVA, commonLoadTestConfig('batch', false, datasetName), "ParDo", "batch")
}
def streamingLoadTestJob = {scope, triggeringContext ->
scope.description('Runs Java ParDo load tests on Dataflow runner in streaming mode')
commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
- for (testConfiguration in commonLoadTestConfig('streaming', true)) {
- testConfiguration.jobProperties << [inputWindowDurationSec: 1200]
- loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.jobProperties, testConfiguration.itClass, triggeringContext)
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in commonLoadTestConfig('streaming', true, datasetName)) {
+ testConfiguration.pipelineOptions << [inputWindowDurationSec: 1200]
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.JAVA, testConfiguration.pipelineOptions, testConfiguration.test)
}
}
@@ -181,4 +174,4 @@
this
) {
streamingLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
-}
\ No newline at end of file
+}
diff --git a/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy b/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy
new file mode 100644
index 0000000..f55ed6e
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_ParDo_Python.groovy
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = { datasetName -> [
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-pardo-1-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_pardo_1',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 10,
+ number_of_counter_operations: 0,
+ number_of_counters : 0,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 200 times',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-pardo-2-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_pardo_2',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 200,
+ number_of_counter_operations: 0,
+ number_of_counters : 0,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 10 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-pardo-3-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_pardo_3',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ number_of_counter_operations: 10,
+ number_of_counters : 1,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
+ ],
+ [
+ title : 'ParDo Python Load test: 2GB 100 byte records 100 counters',
+ test : 'apache_beam.testing.load_tests.pardo_test:ParDoTest.testParDo',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-pardo-4-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_pardo_4',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90}\'',
+ iterations : 1,
+ number_of_counter_operations: 100,
+ number_of_counters : 1,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE',
+ ]
+ ],
+]}
+
+def batchLoadTestJob = { scope, triggeringContext ->
+ scope.description('Runs Python ParDo load tests on Dataflow runner in batch mode')
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 120)
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in loadTestConfigurations(datasetName)) {
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_ParDo_Dataflow_Batch',
+ 'Run Python Load Tests ParDo Dataflow Batch',
+ 'Load Tests Python ParDo Dataflow Batch suite',
+ this
+) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_ParDo_Dataflow_Batch', 'H 13 * * *', this) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_Python.groovy b/.test-infra/jenkins/job_LoadTests_Python.groovy
deleted file mode 100644
index fb45104..0000000
--- a/.test-infra/jenkins/job_LoadTests_Python.groovy
+++ /dev/null
@@ -1,80 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-import LoadTestsBuilder as loadTestsBuilder
-import PhraseTriggeringPostCommitBuilder
-
-def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
-
-def smokeTestConfigurations = [
- [
- title : 'GroupByKey Python load test Direct',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DIRECT,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
- publish_to_big_query: true,
- project : 'apache-beam-testing',
- metrics_dataset : 'load_test_SMOKE',
- metrics_table : 'python_direct_gbk',
- input_options : '\'{"num_records": 100000,' +
- '"key_size": 1,' +
- '"value_size":1,' +
- '"bundle_size_distribution_type": "const",' +
- '"bundle_size_distribution_param": 1,' +
- '"force_initial_num_bundles": 10}\'',
-
- ]
- ],
- [
- title : 'GroupByKey Python load test Dataflow',
- itClass : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
- runner : CommonTestProperties.Runner.DATAFLOW,
- sdk : CommonTestProperties.SDK.PYTHON,
- jobProperties: [
- job_name : 'load-tests-python-dataflow-batch-gbk-smoke-' + now,
- project : 'apache-beam-testing',
- temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
- publish_to_big_query: true,
- metrics_dataset : 'load_test_SMOKE',
- metrics_table : 'python_dataflow_gbk',
- input_options : '\'{"num_records": 100000,' +
- '"key_size": 1,' +
- '"value_size":1,' +
- '"bundle_size_distribution_type": "const",' +
- '"bundle_size_distribution_param": 1,' +
- '"force_initial_num_bundles": 10}\'',
- maxNumWorkers : 10,
- ]
- ],
-]
-
-PhraseTriggeringPostCommitBuilder.postCommitJob(
- 'beam_Python_LoadTests_Smoke',
- 'Run Python Load Tests Smoke',
- 'Python Load Tests Smoke',
- this
-) {
- description("Runs Python load tests in \"smoke\" mode to check if everything works well")
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
-
- for (testConfiguration in smokeTestConfigurations) {
- loadTestsBuilder.loadTest(delegate, testConfiguration.title, testConfiguration.runner,testConfiguration.sdk, testConfiguration.jobProperties, testConfiguration.itClass, CommonTestProperties.TriggeringContext.PR)
- }
-}
\ No newline at end of file
diff --git a/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy b/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy
new file mode 100644
index 0000000..e03f7d2
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_Python_Smoke.groovy
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def smokeTestConfigurations = { datasetName -> [
+ [
+ title : 'GroupByKey Python load test Direct',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DIRECT,
+ pipelineOptions: [
+ publish_to_big_query: true,
+ project : 'apache-beam-testing',
+ metrics_dataset : datasetName,
+ metrics_table : 'python_direct_gbk',
+ input_options : '\'{"num_records": 100000,' +
+ '"key_size": 1,' +
+ '"value_size":1}\'',
+
+ ]
+ ],
+ [
+ title : 'GroupByKey Python load test Dataflow',
+ test : 'apache_beam.testing.load_tests.group_by_key_test:GroupByKeyTest.testGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'load-tests-python-dataflow-batch-gbk-smoke-' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/smoketests',
+ publish_to_big_query: true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_gbk',
+ input_options : '\'{"num_records": 100000,' +
+ '"key_size": 1,' +
+ '"value_size":1}\'',
+ max_num_workers : 1,
+ ]
+ ],
+]}
+
+// Runs a tiny version load test suite to ensure nothing is broken.
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Python_LoadTests_Smoke',
+ 'Run Python Load Tests Smoke',
+ 'Python Load Tests Smoke',
+ this
+) {
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test_SMOKE', CommonTestProperties.TriggeringContext.PR)
+ loadTestsBuilder.loadTests(delegate, CommonTestProperties.SDK.PYTHON, smokeTestConfigurations(datasetName), "GBK", "smoke")
+}
diff --git a/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy b/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy
new file mode 100644
index 0000000..026d197
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import Flink
+import Docker
+
+String now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def scenarios = { datasetName, sdkHarnessImageTag -> [
+ [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with a single key',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-flink-batch-cogbk-1-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_cogbk_1",
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with multiple keys',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-flink-batch-cogbk-2-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_cogbk_2',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 5,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 5,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 1,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-flink-batch-cogbk-3-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : "python_flink_batch_cogbk_3",
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 200000,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 200000,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 4,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+ [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.PORTABLE,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-flink-batch-cogbk-4-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : false,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_flink_batch_cogbk_4',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1000,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1000,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 4,
+ parallelism : 5,
+ job_endpoint : 'localhost:8099',
+ environment_config : sdkHarnessImageTag,
+ environment_type : 'DOCKER',
+ ]
+ ],
+]}
+
+def loadTest = { scope, triggeringContext ->
+ Docker publisher = new Docker(scope, loadTestsBuilder.DOCKER_CONTAINER_REGISTRY)
+ String pythonHarnessImageTag = publisher.getFullImageName('python2.7_sdk')
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ def numberOfWorkers = 5
+ List<Map> testScenarios = scenarios(datasetName, pythonHarnessImageTag)
+
+ publisher.publish(':sdks:python:container:py2:docker', 'python2.7_sdk')
+ publisher.publish('runners:flink:1.7:job-server-container:docker', 'flink-job-server')
+ def flink = new Flink(scope, 'beam_LoadTests_Python_CoGBK_Flink_Batch')
+ flink.setUp([pythonHarnessImageTag], numberOfWorkers, publisher.getFullImageName('flink-job-server'))
+
+ loadTestsBuilder.loadTests(scope, CommonTestProperties.SDK.PYTHON, testScenarios, 'CoGBK', 'batch')
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_CoGBK_Flink_Batch',
+ 'Run Load Tests Python CoGBK Flink Batch',
+ 'Load Tests Python CoGBK Flink Batch suite',
+ this
+) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.PR)
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_CoGBK_Flink_Batch', 'H 16 * * *', this) {
+ loadTest(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
diff --git a/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy b/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
new file mode 100644
index 0000000..470602c
--- /dev/null
+++ b/.test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import CommonTestProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = { datasetName -> [
+ [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with a single key',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-dataflow-batch-cogbk-1-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : "python_dataflow_batch_cogbk_1",
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 1,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE'
+ ]
+ ],
+ [
+ title : 'CoGroupByKey Python Load test: 2GB of 100B records with multiple keys',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-dataflow-batch-cogbk-2-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_cogbk_2',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 5,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 5,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 1,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE'
+ ]
+ ],
+ [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 10kB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-dataflow-batch-cogbk-3-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : "python_dataflow_batch_cogbk_3",
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 200000,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 200000,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 4,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE'
+ ]
+ ],
+ [
+ title : 'CoGroupByKey Python Load test: reiterate 4 times 2MB values',
+ test : 'apache_beam.testing.load_tests.co_group_by_key_test:CoGroupByKeyTest.testCoGroupByKey',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ project : 'apache-beam-testing',
+ job_name : 'load-tests-python-dataflow-batch-cogbk-4-' + now,
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ publish_to_big_query : true,
+ metrics_dataset : datasetName,
+ metrics_table : 'python_dataflow_batch_cogbk_4',
+ input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1000,' +
+ '"hot_key_fraction": 1}\'',
+ co_input_options : '\'{' +
+ '"num_records": 20000000,' +
+ '"key_size": 10,' +
+ '"value_size": 90,' +
+ '"num_hot_keys": 1000,' +
+ '"hot_key_fraction": 1}\'',
+ iterations : 4,
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE'
+ ]
+ ],
+]}
+
+def batchLoadTestJob = { scope, triggeringContext ->
+ scope.description('Runs Python CoGBK load tests on Dataflow runner in batch mode')
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+
+ def datasetName = loadTestsBuilder.getBigQueryDataset('load_test', triggeringContext)
+ for (testConfiguration in loadTestConfigurations(datasetName)) {
+ loadTestsBuilder.loadTest(scope, testConfiguration.title, testConfiguration.runner, CommonTestProperties.SDK.PYTHON, testConfiguration.pipelineOptions, testConfiguration.test)
+ }
+}
+
+CronJobBuilder.cronJob('beam_LoadTests_Python_CoGBK_Dataflow_Batch', 'H 16 * * *', this) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.POST_COMMIT)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_LoadTests_Python_CoGBK_Dataflow_Batch',
+ 'Run Load Tests Python CoGBK Dataflow Batch',
+ 'Load Tests Python CoGBK Dataflow Batch suite',
+ this
+) {
+ batchLoadTestJob(delegate, CommonTestProperties.TriggeringContext.PR)
+}
diff --git a/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy
new file mode 100644
index 0000000..7231140
--- /dev/null
+++ b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def bqioStreamTest = [
+ title : 'BigQueryIO Streaming Performance Test Java 10 GB',
+ test : 'org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ jobName : 'performance-tests-bqio-java-stream-10gb' + now,
+ project : 'apache-beam-testing',
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ tempRoot : 'gs://temp-storage-for-perf-tests/loadtests',
+ writeMethod : 'STREAMING_INSERTS',
+ publishToBigQuery : true,
+ testBigQueryDataset : 'beam_performance',
+ testBigQueryTable : 'bqio_write_10GB_java',
+ metricsBigQueryDataset: 'beam_performance',
+ metricsBigQueryTable : 'bqio_10GB_results_java_stream',
+ sourceOptions : '\'{' +
+ '"num_records": 10485760,' +
+ '"key_size": 1,' +
+ '"value_size": 1024}\'',
+ numWorkers : 5,
+ autoscalingAlgorithm : 'NONE', // Disable autoscale the worker pool.
+ ]
+]
+
+def bqioBatchTest = [
+ title : 'BigQueryIO Batch Performance Test Java 10 GB',
+ test : 'org.apache.beam.sdk.bigqueryioperftests.BigQueryIOIT',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ jobName : 'performance-tests-bqio-java-stream-10gb' + now,
+ project : 'apache-beam-testing',
+ tempLocation : 'gs://temp-storage-for-perf-tests/loadtests',
+ tempRoot : 'gs://temp-storage-for-perf-tests/loadtests',
+ writeMethod : 'FILE_LOADS',
+ publishToBigQuery : true,
+ testBigQueryDataset : 'beam_performance',
+ testBigQueryTable : 'bqio_write_10GB_java',
+ metricsBigQueryDataset: 'beam_performance',
+ metricsBigQueryTable : 'bqio_10GB_results_java_batch',
+ sourceOptions : '\'{' +
+ '"num_records": 10485760,' +
+ '"key_size": 1,' +
+ '"value_size": 1024}\'',
+ numWorkers : 5,
+ autoscalingAlgorithm : 'NONE', // Disable autoscale the worker pool.
+ ]
+]
+
+def executeJob = { scope, testConfig ->
+ job(testConfig.title) {
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+ def testTask = ':sdks:java:io:bigquery-io-perf-tests:integrationTest'
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ commonJobProperties.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${commonJobProperties.joinPipelineOptions(testConfig.pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=\'${testConfig.runner}\'")
+ tasks("${testTask} --tests ${testConfig.test}")
+ }
+ }
+
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_BiqQueryIO_Batch_Performance_Test_Java',
+ 'Run BigQueryIO Batch Performance Test Java',
+ 'BigQueryIO Batch Performance Test Java',
+ this
+) {
+ executeJob(delegate, bqioBatchTest)
+}
+
+CronJobBuilder.cronJob('beam_BiqQueryIO_Batch_Performance_Test_Java', 'H 15 * * *', this) {
+ executeJob(delegate, bqioBatchTest)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_BiqQueryIO_Stream_Performance_Test_Java',
+ 'Run BigQueryIO Streaming Performance Test Java',
+ 'BigQueryIO Streaming Performance Test Java',
+ this
+) {
+ executeJob(delegate, bqioStreamTest)
+}
+
+CronJobBuilder.cronJob('beam_BiqQueryIO_Stream_Performance_Test_Java', 'H 15 * * *', this) {
+ executeJob(delegate, bqioStreamTest)
+}
diff --git a/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy
new file mode 100644
index 0000000..471b394
--- /dev/null
+++ b/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def bqio_read_test = [
+ title : 'BigQueryIO Read Performance Test Python 10 GB',
+ test : 'apache_beam.io.gcp.bigquery_read_perf_test:BigQueryReadPerfTest.test',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'performance-tests-bqio-read-python-10gb' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ input_dataset : 'beam_performance',
+ input_table : 'bqio_read_10GB',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_performance',
+ metrics_table : 'bqio_read_10GB_results',
+ input_options : '\'{' +
+ '"num_records": 10485760,' +
+ '"key_size": 1,' +
+ '"value_size": 1024}\'',
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
+ ]
+]
+
+def bqio_write_test = [
+ title : 'BigQueryIO Write Performance Test Python Batch 10 GB',
+ test : 'apache_beam.io.gcp.bigquery_write_perf_test:BigQueryWritePerfTest.test',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'performance-tests-bqio-write-python-batch-10gb' + now,
+ project : 'apache-beam-testing',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ output_dataset : 'beam_performance',
+ output_table : 'bqio_write_10GB',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_performance',
+ metrics_table : 'bqio_write_10GB_results',
+ input_options : '\'{' +
+ '"num_records": 10485760,' +
+ '"key_size": 1,' +
+ '"value_size": 1024}\'',
+ num_workers : 5,
+ autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
+ ]
+]
+
+def executeJob = { scope, testConfig ->
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 240)
+
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_BiqQueryIO_Read_Performance_Test_Python',
+ 'Run BigQueryIO Read Performance Test Python',
+ 'BigQueryIO Read Performance Test Python',
+ this
+) {
+ executeJob(delegate, bqio_read_test)
+}
+
+CronJobBuilder.cronJob('beam_BiqQueryIO_Read_Performance_Test_Python', 'H 15 * * *', this) {
+ executeJob(delegate, bqio_read_test)
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_BiqQueryIO_Write_Performance_Test_Python_Batch',
+ 'Run BigQueryIO Write Performance Test Python Batch',
+ 'BigQueryIO Write Performance Test Python Batch',
+ this
+) {
+ executeJob(delegate, bqio_write_test)
+}
+
+CronJobBuilder.cronJob('beam_BiqQueryIO_Write_Performance_Test_Python_Batch', 'H 15 * * *', this) {
+ executeJob(delegate, bqio_write_test)
+}
diff --git a/.test-infra/jenkins/job_PerformanceTests_Dataflow.groovy b/.test-infra/jenkins/job_PerformanceTests_Dataflow.groovy
deleted file mode 100644
index 6af1723..0000000
--- a/.test-infra/jenkins/job_PerformanceTests_Dataflow.groovy
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-
-// This job runs the Beam performance tests on PerfKit Benchmarker.
-job('beam_PerformanceTests_Dataflow'){
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
-
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
-
- def argMap = [
- benchmarks: 'dpb_wordcount_benchmark',
- dpb_dataflow_staging_location: 'gs://temp-storage-for-perf-tests/staging',
- dpb_wordcount_input: 'dataflow-samples/shakespeare/kinglear.txt',
- config_override: 'dpb_wordcount_benchmark.dpb_service.service_type=dataflow'
- ]
-
- commonJobProperties.buildPerformanceTest(delegate, argMap)
-
- // [BEAM-2141] Perf tests do not pass.
- disabled()
-}
diff --git a/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy b/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
index 39ad3fd..ddccd5d 100644
--- a/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
@@ -16,155 +16,316 @@
* limitations under the License.
*/
-import CommonJobProperties as commonJobProperties
+import CommonJobProperties as common
-def testsConfigurations = [
+def jobs = [
[
- jobName : 'beam_PerformanceTests_TextIOIT',
- jobDescription : 'Runs PerfKit tests for TextIOIT',
- itClass : 'org.apache.beam.sdk.io.text.TextIOIT',
- bqTable : 'beam_performance.textioit_pkb_results',
- prCommitStatusName: 'Java TextIO Performance Test',
- prTriggerPhase : 'Run Java TextIO Performance Test',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'textioit_results',
- numberOfRecords: '1000000'
+ name : 'beam_PerformanceTests_TextIOIT',
+ description : 'Runs performance tests for TextIOIT',
+ test : 'org.apache.beam.sdk.io.text.TextIOIT',
+ githubTitle : 'Java TextIO Performance Test',
+ githubTriggerPhrase: 'Run Java TextIO Performance Test',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'textioit_results',
+ numberOfRecords : '1000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
- jobName : 'beam_PerformanceTests_Compressed_TextIOIT',
- jobDescription : 'Runs PerfKit tests for TextIOIT with GZIP compression',
- itClass : 'org.apache.beam.sdk.io.text.TextIOIT',
- bqTable : 'beam_performance.compressed_textioit_pkb_results',
- prCommitStatusName : 'Java CompressedTextIO Performance Test',
- prTriggerPhase : 'Run Java CompressedTextIO Performance Test',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'compressed_textioit_results',
- numberOfRecords: '1000000',
- compressionType: 'GZIP'
+ name : 'beam_PerformanceTests_Compressed_TextIOIT',
+ description : 'Runs performance tests for TextIOIT with GZIP compression',
+ test : 'org.apache.beam.sdk.io.text.TextIOIT',
+ githubTitle : 'Java CompressedTextIO Performance Test',
+ githubTriggerPhrase: 'Run Java CompressedTextIO Performance Test',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'compressed_textioit_results',
+ numberOfRecords : '1000000',
+ compressionType : 'GZIP',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
- jobName : 'beam_PerformanceTests_ManyFiles_TextIOIT',
- jobDescription : 'Runs PerfKit tests for TextIOIT with many output files',
- itClass : 'org.apache.beam.sdk.io.text.TextIOIT',
- bqTable : 'beam_performance.many_files_textioit_pkb_results',
- prCommitStatusName: 'Java ManyFilesTextIO Performance Test',
- prTriggerPhase : 'Run Java ManyFilesTextIO Performance Test',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'many_files_textioit_results',
+ name : 'beam_PerformanceTests_ManyFiles_TextIOIT',
+ description : 'Runs performance tests for TextIOIT with many output files',
+ test : 'org.apache.beam.sdk.io.text.TextIOIT',
+ githubTitle : 'Java ManyFilesTextIO Performance Test',
+ githubTriggerPhrase: 'Run Java ManyFilesTextIO Performance Test',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'many_files_textioit_results',
reportGcsPerformanceMetrics: 'true',
- gcsPerformanceMetrics: 'true',
- numberOfRecords: '1000000',
- numberOfShards: '1000'
+ gcsPerformanceMetrics : 'true',
+ numberOfRecords : '1000000',
+ numberOfShards : '1000',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
]
],
[
- jobName : 'beam_PerformanceTests_AvroIOIT',
- jobDescription : 'Runs PerfKit tests for AvroIOIT',
- itClass : 'org.apache.beam.sdk.io.avro.AvroIOIT',
- bqTable : 'beam_performance.avroioit_pkb_results',
- prCommitStatusName: 'Java AvroIO Performance Test',
- prTriggerPhase : 'Run Java AvroIO Performance Test',
- extraPipelineArgs: [
- numberOfRecords: '1000000',
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'avroioit_results',
+ name : 'beam_PerformanceTests_AvroIOIT',
+ description : 'Runs performance tests for AvroIOIT',
+ test : 'org.apache.beam.sdk.io.avro.AvroIOIT',
+ githubTitle : 'Java AvroIO Performance Test',
+ githubTriggerPhrase: 'Run Java AvroIO Performance Test',
+ pipelineOptions : [
+ numberOfRecords : '1000000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'avroioit_results',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
- jobName : 'beam_PerformanceTests_TFRecordIOIT',
- jobDescription : 'Runs PerfKit tests for beam_PerformanceTests_TFRecordIOIT',
- itClass : 'org.apache.beam.sdk.io.tfrecord.TFRecordIOIT',
- bqTable : 'beam_performance.tfrecordioit_pkb_results',
- prCommitStatusName: 'Java TFRecordIO Performance Test',
- prTriggerPhase : 'Run Java TFRecordIO Performance Test',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'tfrecordioit_results',
- numberOfRecords: '1000000'
+ name : 'beam_PerformanceTests_TFRecordIOIT',
+ description : 'Runs performance tests for beam_PerformanceTests_TFRecordIOIT',
+ test : 'org.apache.beam.sdk.io.tfrecord.TFRecordIOIT',
+ githubTitle : 'Java TFRecordIO Performance Test',
+ githubTriggerPhrase: 'Run Java TFRecordIO Performance Test',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'tfrecordioit_results',
+ numberOfRecords : '1000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
- jobName : 'beam_PerformanceTests_XmlIOIT',
- jobDescription : 'Runs PerfKit tests for beam_PerformanceTests_XmlIOIT',
- itClass : 'org.apache.beam.sdk.io.xml.XmlIOIT',
- bqTable : 'beam_performance.xmlioit_pkb_results',
- prCommitStatusName: 'Java XmlIOPerformance Test',
- prTriggerPhase : 'Run Java XmlIO Performance Test',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'xmlioit_results',
- numberOfRecords: '100000000',
- charset: 'UTF-8'
+ name : 'beam_PerformanceTests_XmlIOIT',
+ description : 'Runs performance tests for beam_PerformanceTests_XmlIOIT',
+ test : 'org.apache.beam.sdk.io.xml.XmlIOIT',
+ githubTitle : 'Java XmlIOPerformance Test',
+ githubTriggerPhrase: 'Run Java XmlIO Performance Test',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'xmlioit_results',
+ numberOfRecords : '100000000',
+ charset : 'UTF-8',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
],
[
- jobName : 'beam_PerformanceTests_ParquetIOIT',
- jobDescription : 'Runs PerfKit tests for beam_PerformanceTests_ParquetIOIT',
- itClass : 'org.apache.beam.sdk.io.parquet.ParquetIOIT',
- bqTable : 'beam_performance.parquetioit_pkb_results',
- prCommitStatusName: 'Java ParquetIOPerformance Test',
- prTriggerPhase : 'Run Java ParquetIO Performance Test',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'parquetioit_results',
- numberOfRecords: '100000000'
+ name : 'beam_PerformanceTests_ParquetIOIT',
+ description : 'Runs performance tests for beam_PerformanceTests_ParquetIOIT',
+ test : 'org.apache.beam.sdk.io.parquet.ParquetIOIT',
+ githubTitle : 'Java ParquetIOPerformance Test',
+ githubTriggerPhrase: 'Run Java ParquetIO Performance Test',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'parquetioit_results',
+ numberOfRecords : '100000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
+ ]
+ ],
+ [
+ name : 'beam_PerformanceTests_TextIOIT_HDFS',
+ description : 'Runs performance tests for TextIOIT on HDFS',
+ test : 'org.apache.beam.sdk.io.text.TextIOIT',
+ githubTitle : 'Java TextIO Performance Test on HDFS',
+ githubTriggerPhrase: 'Run Java TextIO Performance Test HDFS',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'textioit_hdfs_results',
+ numberOfRecords : '1000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
+ ]
+
+ ],
+ [
+ name : 'beam_PerformanceTests_Compressed_TextIOIT_HDFS',
+ description : 'Runs performance tests for TextIOIT with GZIP compression on HDFS',
+ test : 'org.apache.beam.sdk.io.text.TextIOIT',
+ githubTitle : 'Java CompressedTextIO Performance Test on HDFS',
+ githubTriggerPhrase: 'Run Java CompressedTextIO Performance Test HDFS',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'compressed_textioit_hdfs_results',
+ numberOfRecords : '1000000',
+ compressionType : 'GZIP',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
+ ]
+ ],
+ [
+ name : 'beam_PerformanceTests_ManyFiles_TextIOIT_HDFS',
+ description : 'Runs performance tests for TextIOIT with many output files on HDFS',
+ test : 'org.apache.beam.sdk.io.text.TextIOIT',
+ githubTitle : 'Java ManyFilesTextIO Performance Test on HDFS',
+ githubTriggerPhrase: 'Run Java ManyFilesTextIO Performance Test HDFS',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'many_files_textioit_hdfs_results',
+ reportGcsPerformanceMetrics: 'true',
+ gcsPerformanceMetrics : 'true',
+ numberOfRecords : '1000000',
+ numberOfShards : '1000',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
+ ]
+
+ ],
+ [
+ name : 'beam_PerformanceTests_AvroIOIT_HDFS',
+ description : 'Runs performance tests for AvroIOIT on HDFS',
+ test : 'org.apache.beam.sdk.io.avro.AvroIOIT',
+ githubTitle : 'Java AvroIO Performance Test on HDFS',
+ githubTriggerPhrase: 'Run Java AvroIO Performance Test HDFS',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'avroioit_hdfs_results',
+ numberOfRecords : '1000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
+ ]
+ ],
+ [
+ name : 'beam_PerformanceTests_TFRecordIOIT_HDFS',
+ description : 'Runs performance tests for beam_PerformanceTests_TFRecordIOIT on HDFS',
+ test : 'org.apache.beam.sdk.io.tfrecord.TFRecordIOIT',
+ githubTitle : 'Java TFRecordIO Performance Test on HDFS',
+ githubTriggerPhrase: 'Run Java TFRecordIO Performance Test HDFS',
+ pipelineOptions : [
+ numberOfRecords : '1000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
+ ]
+ ],
+ [
+ name : 'beam_PerformanceTests_XmlIOIT_HDFS',
+ description : 'Runs performance tests for beam_PerformanceTests_XmlIOIT on HDFS',
+ test : 'org.apache.beam.sdk.io.xml.XmlIOIT',
+ githubTitle : 'Java XmlIOPerformance Test on HDFS',
+ githubTriggerPhrase: 'Run Java XmlIO Performance Test HDFS',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'xmlioit_hdfs_results',
+ numberOfRecords : '100000',
+ charset : 'UTF-8',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
+ ]
+ ],
+ [
+ name : 'beam_PerformanceTests_ParquetIOIT_HDFS',
+ description : 'Runs performance tests for beam_PerformanceTests_ParquetIOIT on HDFS',
+ test : 'org.apache.beam.sdk.io.parquet.ParquetIOIT',
+ githubTitle : 'Java ParquetIOPerformance Test on HDFS',
+ githubTriggerPhrase: 'Run Java ParquetIO Performance Test HDFS',
+ pipelineOptions : [
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'parquetioit_hdfs_results',
+ numberOfRecords : '1000000',
+ numWorkers : '5',
+ autoscalingAlgorithm: 'NONE'
]
]
]
-for (testConfiguration in testsConfigurations) {
- create_filebasedio_performance_test_job(testConfiguration)
+jobs.findAll {
+ it.name in [
+ 'beam_PerformanceTests_TextIOIT',
+ 'beam_PerformanceTests_Compressed_TextIOIT',
+ 'beam_PerformanceTests_ManyFiles_TextIOIT',
+ 'beam_PerformanceTests_AvroIOIT',
+ 'beam_PerformanceTests_TFRecordIOIT',
+ 'beam_PerformanceTests_XmlIOIT',
+ 'beam_PerformanceTests_ParquetIOIT'
+ ]
+}.forEach { testJob -> createGCSFileBasedIOITTestJob(testJob) }
+
+private void createGCSFileBasedIOITTestJob(testJob) {
+ job(testJob.name) {
+ description(testJob.description)
+ common.setTopLevelMainJobProperties(delegate)
+ common.enablePhraseTriggeringFromPullRequest(delegate, testJob.githubTitle, testJob.githubTriggerPhrase)
+ common.setAutoJob(delegate, 'H */6 * * *')
+
+ def dataflowSpecificOptions = [
+ runner : 'DataflowRunner',
+ project : 'apache-beam-testing',
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ filenamePrefix: "gs://temp-storage-for-perf-tests/${testJob.name}/\${BUILD_ID}/",
+ ]
+
+ Map allPipelineOptions = dataflowSpecificOptions << testJob.pipelineOptions
+ String runner = "dataflow"
+ String filesystem = "gcs"
+ String testTask = ":sdks:java:io:file-based-io-tests:integrationTest"
+
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(allPipelineOptions)}\'")
+ switches("-Dfilesystem=\'${filesystem}\'")
+ switches("-DintegrationTestRunner=\'${runner}\'")
+ tasks("${testTask} --tests ${testJob.test}")
+ }
+ }
+ }
}
+jobs.findAll {
+ it.name in [
+ 'beam_PerformanceTests_TextIOIT_HDFS',
+ 'beam_PerformanceTests_Compressed_TextIOIT_HDFS',
+ 'beam_PerformanceTests_ManyFiles_TextIOIT_HDFS',
+ // TODO(BEAM-3945) TFRecord performance test is failing only when running on hdfs.
+ // We need to fix this before enabling this job on jenkins.
+ //'beam_PerformanceTests_TFRecordIOIT_HDFS',
+ 'beam_PerformanceTests_AvroIOIT_HDFS',
+ 'beam_PerformanceTests_XmlIOIT_HDFS',
+ 'beam_PerformanceTests_ParquetIOIT_HDFS'
+ ]
+}.forEach { testJob -> createHDFSFileBasedIOITTestJob(testJob) }
-private void create_filebasedio_performance_test_job(testConfiguration) {
+private void createHDFSFileBasedIOITTestJob(testJob) {
+ job(testJob.name) {
+ description(testJob.description)
+ common.setTopLevelMainJobProperties(delegate)
+ common.enablePhraseTriggeringFromPullRequest(delegate, testJob.githubTitle, testJob.githubTriggerPhrase)
+ common.setAutoJob(delegate, 'H */6 * * *')
- // This job runs the file-based IOs performance tests on PerfKit Benchmarker.
- job(testConfiguration.jobName) {
- description(testConfiguration.jobDescription)
+ String namespace = common.getKubernetesNamespace(testJob.name)
+ String kubeconfig = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfig, namespace)
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml"))
+ String hostName = "LOAD_BALANCER_IP"
+ k8s.loadBalancerIP("hadoop", hostName)
- // Allows triggering this build against pull requests.
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- testConfiguration.prCommitStatusName,
- testConfiguration.prTriggerPhase)
+ Map additionalOptions = [
+ runner : 'DataflowRunner',
+ project : 'apache-beam-testing',
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ hdfsConfiguration: /[{\\\"fs.defaultFS\\\":\\\"hdfs:$${hostName}:9000\\\",\\\"dfs.replication\\\":1}]/,
+ filenamePrefix : "hdfs://\$${hostName}:9000/TEXTIO_IT_"
+ ]
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
+ Map allPipelineOptions = testJob.pipelineOptions << additionalOptions
+ String runner = "dataflow"
+ String filesystem = "hdfs"
+ String testTask = ":sdks:java:io:file-based-io-tests:integrationTest"
- def pipelineOptions = [
- project : 'apache-beam-testing',
- tempRoot : 'gs://temp-storage-for-perf-tests',
- filenamePrefix : "gs://temp-storage-for-perf-tests/${testConfiguration.jobName}/\${BUILD_ID}/",
- ]
- if (testConfiguration.containsKey('extraPipelineArgs')) {
- pipelineOptions << testConfiguration.extraPipelineArgs
- }
-
- def argMap = [
- benchmarks : 'beam_integration_benchmark',
- beam_it_timeout : '1200',
- beam_prebuilt : 'false',
- beam_sdk : 'java',
- beam_it_module : 'sdks/java/io/file-based-io-tests',
- beam_it_class : testConfiguration.itClass,
- beam_it_options : commonJobProperties.joinPipelineOptions(pipelineOptions),
- beam_extra_properties: '["filesystem=gcs"]',
- bigquery_table : testConfiguration.bqTable,
- ]
- commonJobProperties.buildPerformanceTest(delegate, argMap)
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(allPipelineOptions)}\'")
+ switches("-Dfilesystem=\'${filesystem}\'")
+ switches("-DintegrationTestRunner=\'${runner}\'")
+ tasks("${testTask} --tests ${testJob.test}")
+ }
}
+ }
}
diff --git a/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT_HDFS.groovy b/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
deleted file mode 100644
index fa94769..0000000
--- a/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
+++ /dev/null
@@ -1,187 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-
-def testsConfigurations = [
- [
- jobName : 'beam_PerformanceTests_TextIOIT_HDFS',
- jobDescription : 'Runs PerfKit tests for TextIOIT on HDFS',
- itClass : 'org.apache.beam.sdk.io.text.TextIOIT',
- bqTable : 'beam_performance.textioit_hdfs_pkb_results',
- prCommitStatusName: 'Java TextIO Performance Test on HDFS',
- prTriggerPhase : 'Run Java TextIO Performance Test HDFS',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'textioit_hdfs_results',
- numberOfRecords: '1000000'
- ]
-
- ],
- [
- jobName : 'beam_PerformanceTests_Compressed_TextIOIT_HDFS',
- jobDescription : 'Runs PerfKit tests for TextIOIT with GZIP compression on HDFS',
- itClass : 'org.apache.beam.sdk.io.text.TextIOIT',
- bqTable : 'beam_performance.compressed_textioit_hdfs_pkb_results',
- prCommitStatusName : 'Java CompressedTextIO Performance Test on HDFS',
- prTriggerPhase : 'Run Java CompressedTextIO Performance Test HDFS',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'compressed_textioit_hdfs_results',
- numberOfRecords: '1000000',
- compressionType: 'GZIP'
- ]
- ],
- [
- jobName : 'beam_PerformanceTests_ManyFiles_TextIOIT_HDFS',
- jobDescription : 'Runs PerfKit tests for TextIOIT with many output files on HDFS',
- itClass : 'org.apache.beam.sdk.io.text.TextIOIT',
- bqTable : 'beam_performance.many_files_textioit_hdfs_pkb_results',
- prCommitStatusName: 'Java ManyFilesTextIO Performance Test on HDFS',
- prTriggerPhase : 'Run Java ManyFilesTextIO Performance Test HDFS',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'many_files_textioit_hdfs_results',
- reportGcsPerformanceMetrics: 'true',
- gcsPerformanceMetrics: 'true',
- numberOfRecords: '1000000',
- numberOfShards: '1000'
- ]
-
- ],
- [
- jobName : 'beam_PerformanceTests_AvroIOIT_HDFS',
- jobDescription : 'Runs PerfKit tests for AvroIOIT on HDFS',
- itClass : 'org.apache.beam.sdk.io.avro.AvroIOIT',
- bqTable : 'beam_performance.avroioit_hdfs_pkb_results',
- prCommitStatusName: 'Java AvroIO Performance Test on HDFS',
- prTriggerPhase : 'Run Java AvroIO Performance Test HDFS',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'avroioit_hdfs_results',
- numberOfRecords: '1000000'
- ]
- ],
-// TODO(BEAM-3945) TFRecord performance test is failing only when running on hdfs.
-// We need to fix this before enabling this job on jenkins.
-// [
-// jobName : 'beam_PerformanceTests_TFRecordIOIT_HDFS',
-// jobDescription : 'Runs PerfKit tests for beam_PerformanceTests_TFRecordIOIT on HDFS',
-// itClass : 'org.apache.beam.sdk.io.tfrecord.TFRecordIOIT',
-// bqTable : 'beam_performance.tfrecordioit_hdfs_pkb_results',
-// prCommitStatusName: 'Java TFRecordIO Performance Test on HDFS',
-// prTriggerPhase : 'Run Java TFRecordIO Performance Test HDFS',
-// extraPipelineArgs: [
-// numberOfRecords: '1000000'
-// ]
-// ],
- [
- jobName : 'beam_PerformanceTests_XmlIOIT_HDFS',
- jobDescription : 'Runs PerfKit tests for beam_PerformanceTests_XmlIOIT on HDFS',
- itClass : 'org.apache.beam.sdk.io.xml.XmlIOIT',
- bqTable : 'beam_performance.xmlioit_hdfs_pkb_results',
- prCommitStatusName: 'Java XmlIOPerformance Test on HDFS',
- prTriggerPhase : 'Run Java XmlIO Performance Test HDFS',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'xmlioit_hdfs_results',
- numberOfRecords: '100000',
- charset: 'UTF-8'
- ]
- ],
- [
- jobName : 'beam_PerformanceTests_ParquetIOIT_HDFS',
- jobDescription : 'Runs PerfKit tests for beam_PerformanceTests_ParquetIOIT on HDFS',
- itClass : 'org.apache.beam.sdk.io.parquet.ParquetIOIT',
- bqTable : 'beam_performance.parquetioit_hdfs_pkb_results',
- prCommitStatusName: 'Java ParquetIOPerformance Test on HDFS',
- prTriggerPhase : 'Run Java ParquetIO Performance Test HDFS',
- extraPipelineArgs: [
- bigQueryDataset: 'beam_performance',
- bigQueryTable: 'parquetioit_hdfs_results',
- numberOfRecords: '1000000'
- ]
- ]
-]
-
-for (testConfiguration in testsConfigurations) {
- create_filebasedio_performance_test_job(testConfiguration)
-}
-
-
-private void create_filebasedio_performance_test_job(testConfiguration) {
-
- // This job runs the file-based IOs performance tests on PerfKit Benchmarker.
- job(testConfiguration.jobName) {
- description(testConfiguration.jobDescription)
-
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
-
- // Allows triggering this build against pull requests.
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- testConfiguration.prCommitStatusName,
- testConfiguration.prTriggerPhase)
-
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
-
- def pipelineArgs = [
- project : 'apache-beam-testing',
- tempRoot : 'gs://temp-storage-for-perf-tests',
- ]
- if (testConfiguration.containsKey('extraPipelineArgs')) {
- pipelineArgs << testConfiguration.extraPipelineArgs
- }
-
- def pipelineArgList = []
- pipelineArgs.each({
- key, value -> pipelineArgList.add("\"--$key=$value\"")
- })
- def pipelineArgsJoined = "[" + pipelineArgList.join(',') + "]"
-
- String namespace = commonJobProperties.getKubernetesNamespace(testConfiguration.jobName)
- String kubeconfig = commonJobProperties.getKubeconfigLocationForNamespace(namespace)
-
- def argMap = [
- kubeconfig : kubeconfig,
- benchmarks : 'beam_integration_benchmark',
- beam_it_timeout : '1200',
- beam_prebuilt : 'false',
- beam_sdk : 'java',
- beam_it_module : 'sdks/java/io/file-based-io-tests',
- beam_it_class : testConfiguration.itClass,
- beam_it_options : pipelineArgsJoined,
- beam_extra_properties : '["filesystem=hdfs"]',
- bigquery_table : testConfiguration.bqTable,
- beam_options_config_file: makePathAbsolute('pkb-config.yml'),
- beam_kubernetes_scripts : makePathAbsolute('hdfs-multi-datanode-cluster.yml')
- ]
- commonJobProperties.setupKubernetes(delegate, namespace, kubeconfig)
- commonJobProperties.buildPerformanceTest(delegate, argMap)
- commonJobProperties.cleanupKubernetes(delegate, namespace, kubeconfig)
- }
-}
-
-static def makePathAbsolute(String path) {
- return '"$WORKSPACE/src/.test-infra/kubernetes/hadoop/LargeITCluster/' + path + '"'
-}
diff --git a/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy b/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy
index 6bbb491..0311859 100644
--- a/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_HadoopFormat.groovy
@@ -15,54 +15,53 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
-import CommonJobProperties as commonJobProperties
+import CommonJobProperties as common
+import Kubernetes
String jobName = "beam_PerformanceTests_HadoopFormat"
job(jobName) {
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ common.setTopLevelMainJobProperties(delegate)
+ common.setAutoJob(delegate, 'H */6 * * *')
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Java HadoopFormatIO Performance Test',
+ 'Run Java HadoopFormatIO Performance Test')
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
+ String namespace = common.getKubernetesNamespace(jobName)
+ String kubeconfig = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfig, namespace)
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Java HadoopFormatIO Performance Test',
- 'Run Java HadoopFormatIO Performance Test')
+ k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml"))
+ String postgresHostName = "LOAD_BALANCER_IP"
+ k8s.loadBalancerIP("postgres-for-dev", postgresHostName)
- def pipelineOptions = [
- tempRoot : 'gs://temp-storage-for-perf-tests',
- project : 'apache-beam-testing',
- postgresPort : '5432',
- numberOfRecords: '600000',
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'hadoopformatioit_results'
- ]
+ Map pipelineOptions = [
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ runner : 'DataflowRunner',
+ numberOfRecords : '600000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'hadoopformatioit_results',
+ postgresUsername : 'postgres',
+ postgresPassword : 'uuinkks',
+ postgresDatabaseName : 'postgres',
+ postgresServerName : "\$${postgresHostName}",
+ postgresSsl : false,
+ postgresPort : '5432',
+ numWorkers : '5',
+ autoscalingAlgorithm : 'NONE'
+ ]
- String namespace = commonJobProperties.getKubernetesNamespace(jobName)
- String kubeconfig = commonJobProperties.getKubeconfigLocationForNamespace(namespace)
-
- def testArgs = [
- kubeconfig : kubeconfig,
- beam_it_timeout : '1200',
- benchmarks : 'beam_integration_benchmark',
- beam_prebuilt : 'false',
- beam_sdk : 'java',
- beam_it_module : 'sdks/java/io/hadoop-format',
- beam_it_class : 'org.apache.beam.sdk.io.hadoop.format.HadoopFormatIOIT',
- beam_it_options : commonJobProperties.joinPipelineOptions(pipelineOptions),
- beam_kubernetes_scripts : commonJobProperties.makePathAbsolute('src/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml'),
- beam_options_config_file: commonJobProperties.makePathAbsolute('src/.test-infra/kubernetes/postgres/pkb-config-local.yml'),
- bigquery_table : 'beam_performance.hadoopformatioit_pkb_results'
- ]
-
- commonJobProperties.setupKubernetes(delegate, namespace, kubeconfig)
- commonJobProperties.buildPerformanceTest(delegate, testArgs)
- commonJobProperties.cleanupKubernetes(delegate, namespace, kubeconfig)
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ tasks(":sdks:java:io:hadoop-format:integrationTest --tests org.apache.beam.sdk.io.hadoop.format.HadoopFormatIOIT")
+ }
+ }
}
diff --git a/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy b/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy
index a8640ba..abc6b98 100644
--- a/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy
@@ -15,54 +15,54 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
-import CommonJobProperties as commonJobProperties
+import CommonJobProperties as common
+import Kubernetes
String jobName = "beam_PerformanceTests_JDBC"
job(jobName) {
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ common.setTopLevelMainJobProperties(delegate)
+ common.setAutoJob(delegate, 'H */6 * * *')
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Java JdbcIO Performance Test',
+ 'Run Java JdbcIO Performance Test')
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Java JdbcIO Performance Test',
- 'Run Java JdbcIO Performance Test')
+ String namespace = common.getKubernetesNamespace(jobName)
+ String kubeconfig = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfig, namespace)
- def pipelineOptions = [
- tempRoot : 'gs://temp-storage-for-perf-tests',
- project : 'apache-beam-testing',
- postgresPort : '5432',
- numberOfRecords: '5000000',
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'jdbcioit_results',
- ]
+ k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml"))
+ String postgresHostName = "LOAD_BALANCER_IP"
+ k8s.loadBalancerIP("postgres-for-dev", postgresHostName)
- String namespace = commonJobProperties.getKubernetesNamespace(jobName)
- String kubeconfig = commonJobProperties.getKubeconfigLocationForNamespace(namespace)
+ Map pipelineOptions = [
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ runner : 'DataflowRunner',
+ numberOfRecords : '5000000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'jdbcioit_results',
+ postgresUsername : 'postgres',
+ postgresPassword : 'uuinkks',
+ postgresDatabaseName : 'postgres',
+ postgresServerName : "\$${postgresHostName}",
+ postgresSsl : false,
+ postgresPort : '5432',
+ autoscalingAlgorithm : 'NONE',
+ numWorkers : '5'
+ ]
- def testArgs = [
- kubeconfig : kubeconfig,
- beam_it_timeout : '1800',
- benchmarks : 'beam_integration_benchmark',
- beam_prebuilt : 'false',
- beam_sdk : 'java',
- beam_it_module : 'sdks/java/io/jdbc',
- beam_it_class : 'org.apache.beam.sdk.io.jdbc.JdbcIOIT',
- beam_it_options : commonJobProperties.joinPipelineOptions(pipelineOptions),
- beam_kubernetes_scripts : commonJobProperties.makePathAbsolute('src/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml'),
- beam_options_config_file: commonJobProperties.makePathAbsolute('src/.test-infra/kubernetes/postgres/pkb-config-local.yml'),
- bigquery_table : 'beam_performance.jdbcioit_pkb_results'
- ]
-
- commonJobProperties.setupKubernetes(delegate, namespace, kubeconfig)
- commonJobProperties.buildPerformanceTest(delegate, testArgs)
- commonJobProperties.cleanupKubernetes(delegate, namespace, kubeconfig)
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ tasks(":sdks:java:io:jdbc:integrationTest --tests org.apache.beam.sdk.io.jdbc.JdbcIOIT")
+ }
+ }
}
diff --git a/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy b/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy
index b20573a..83e1199 100644
--- a/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy
@@ -15,52 +15,49 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-
-import CommonJobProperties as commonJobProperties
+import CommonJobProperties as common
+import Kubernetes
String jobName = "beam_PerformanceTests_MongoDBIO_IT"
job(jobName) {
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ common.setTopLevelMainJobProperties(delegate)
+ common.setAutoJob(delegate,'H */6 * * *')
+ common.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Java MongoDBIO Performance Test',
+ 'Run Java MongoDBIO Performance Test')
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
+ String namespace = common.getKubernetesNamespace(jobName)
+ String kubeconfigPath = common.getKubeconfigLocationForNamespace(namespace)
+ Kubernetes k8s = Kubernetes.create(delegate, kubeconfigPath, namespace)
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Java MongoDBIO Performance Test',
- 'Run Java MongoDBIO Performance Test')
+ k8s.apply(common.makePathAbsolute("src/.test-infra/kubernetes/mongodb/load-balancer/mongo.yml"))
+ String mongoHostName = "LOAD_BALANCER_IP"
+ k8s.loadBalancerIP("mongo-load-balancer-service", mongoHostName)
- def pipelineOptions = [
- tempRoot : 'gs://temp-storage-for-perf-tests',
- project : 'apache-beam-testing',
- numberOfRecords: '10000000',
- bigQueryDataset: 'beam_performance',
- bigQueryTable : 'mongodbioit_results'
- ]
+ Map pipelineOptions = [
+ tempRoot : 'gs://temp-storage-for-perf-tests',
+ project : 'apache-beam-testing',
+ numberOfRecords : '10000000',
+ bigQueryDataset : 'beam_performance',
+ bigQueryTable : 'mongodbioit_results',
+ mongoDBDatabaseName : 'beam',
+ mongoDBHostName : "\$${mongoHostName}",
+ mongoDBPort : 27017,
+ runner : 'DataflowRunner',
+ autoscalingAlgorithm: 'NONE',
+ numWorkers : '5'
+ ]
- String namespace = commonJobProperties.getKubernetesNamespace(jobName)
- String kubeconfig = commonJobProperties.getKubeconfigLocationForNamespace(namespace)
-
- def testArgs = [
- kubeconfig : kubeconfig,
- beam_it_timeout : '1800',
- benchmarks : 'beam_integration_benchmark',
- beam_prebuilt : 'false',
- beam_sdk : 'java',
- beam_it_module : 'sdks/java/io/mongodb',
- beam_it_class : 'org.apache.beam.sdk.io.mongodb.MongoDBIOIT',
- beam_it_options : commonJobProperties.joinPipelineOptions(pipelineOptions),
- beam_kubernetes_scripts : commonJobProperties.makePathAbsolute('src/.test-infra/kubernetes/mongodb/load-balancer/mongo.yml'),
- beam_options_config_file: commonJobProperties.makePathAbsolute('src/.test-infra/kubernetes/mongodb/load-balancer/pkb-config.yml'),
- bigquery_table : 'beam_performance.mongodbioit_pkb_results'
- ]
-
- commonJobProperties.setupKubernetes(delegate, namespace, kubeconfig)
- commonJobProperties.buildPerformanceTest(delegate, testArgs)
- commonJobProperties.cleanupKubernetes(delegate, namespace, kubeconfig)
+ steps {
+ gradle {
+ rootBuildScriptDir(common.checkoutDir)
+ common.setGradleSwitches(delegate)
+ switches("--info")
+ switches("-DintegrationTestPipelineOptions=\'${common.joinPipelineOptions(pipelineOptions)}\'")
+ switches("-DintegrationTestRunner=dataflow")
+ tasks(":sdks:java:io:mongodb:integrationTest --tests org.apache.beam.sdk.io.mongodb.MongoDBIOIT")
+ }
+ }
}
diff --git a/.test-infra/jenkins/job_PerformanceTests_Python.groovy b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
index 0ccd4ea..e10fd41 100644
--- a/.test-infra/jenkins/job_PerformanceTests_Python.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_Python.groovy
@@ -33,7 +33,7 @@
// A benchmark defined flag, will pass to benchmark as "--bigqueryTable"
String resultTable
// A benchmark defined flag, will pass to benchmark as "--beam_it_class"
- String itClass
+ String test
// A benchmark defined flag, will pass to benchmark as "--beam_it_module".
// It's a Gradle project that defines 'integrationTest' task. This task is executed by Perfkit
// Beam benchmark launcher and can be added by enablePythonPerformanceTest() defined in
@@ -41,7 +41,7 @@
String itModule
// A benchmark defined flag, will pass to benchmark as "--beam_python_sdk_location".
// It's the location of Python SDK distribution archive which is required for TestDataflowRunner.
- String pythonSdkLocation = ''
+ String pythonSdkLocation = 'build/apache-beam.tar.gz'
// A benchmark defined flag, will pass to benchmark as "--beam_runner"
String runner = 'TestDataflowRunner'
// A benchmark defined flag, will pass to benchmark as "--beam_it_timeout"
@@ -65,8 +65,8 @@
jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py27 with 1Gb files',
jobTriggerPhrase : 'Run Python27 WordCountIT Performance Test',
resultTable : 'beam_performance.wordcount_py27_pkb_results',
- itClass : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
- itModule : 'sdks/python',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ itModule : ':sdks:python:test-suites:dataflow:py2',
extraPipelineArgs : dataflowPipelineArgs + [
input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
output: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output',
@@ -80,8 +80,8 @@
jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py35 with 1Gb files',
jobTriggerPhrase : 'Run Python35 WordCountIT Performance Test',
resultTable : 'beam_performance.wordcount_py35_pkb_results',
- itClass : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
- itModule : 'sdks/python/test-suites/dataflow/py35',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ itModule : ':sdks:python:test-suites:dataflow:py35',
extraPipelineArgs : dataflowPipelineArgs + [
input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
output: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output',
@@ -89,7 +89,37 @@
num_workers: '10',
autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
],
- )
+ ),
+ new PerformanceTestConfigurations(
+ jobName : 'beam_PerformanceTests_WordCountIT_Py36',
+ jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py36 with 1Gb files',
+ jobTriggerPhrase : 'Run Python36 WordCountIT Performance Test',
+ resultTable : 'beam_performance.wordcount_py36_pkb_results',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ itModule : ':sdks:python:test-suites:dataflow:py36',
+ extraPipelineArgs : dataflowPipelineArgs + [
+ input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
+ output: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output',
+ expect_checksum: 'ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710',
+ num_workers: '10',
+ autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
+ ],
+ ),
+ new PerformanceTestConfigurations(
+ jobName : 'beam_PerformanceTests_WordCountIT_Py37',
+ jobDescription : 'Python SDK Performance Test - Run WordCountIT in Py37 with 1Gb files',
+ jobTriggerPhrase : 'Run Python37 WordCountIT Performance Test',
+ resultTable : 'beam_performance.wordcount_py37_pkb_results',
+ test : 'apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it',
+ itModule : ':sdks:python:test-suites:dataflow:py37',
+ extraPipelineArgs : dataflowPipelineArgs + [
+ input: 'gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.0000*', // 1Gb
+ output: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output',
+ expect_checksum: 'ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710',
+ num_workers: '10',
+ autoscaling_algorithm: 'NONE', // Disable autoscale the worker pool.
+ ],
+ ),
]
@@ -119,11 +149,10 @@
beam_sdk : 'python',
benchmarks : testConfig.benchmarkName,
bigquery_table : testConfig.resultTable,
- beam_it_class : testConfig.itClass,
+ beam_it_class : testConfig.test,
beam_it_module : testConfig.itModule,
beam_prebuilt : 'true', // Python benchmark don't need to prebuild repo before running
- beam_python_sdk_location: getSDKLocationFromModule(testConfig.pythonSdkLocation,
- testConfig.itModule),
+ beam_python_sdk_location: testConfig.pythonSdkLocation,
beam_runner : testConfig.runner,
beam_it_timeout : testConfig.itTimeoutSec.toString(),
beam_it_args : joinPipelineArgs(testConfig.extraPipelineArgs),
@@ -142,12 +171,3 @@
})
return pipelineArgList.join(',')
}
-
-
-// Get relative path of sdk location based on itModule if the location is not provided.
-private static String getSDKLocationFromModule(String pythonSDKLocation, String itModule) {
- if (!pythonSDKLocation && itModule.startsWith("sdks/python")) {
- return (itModule.substring("sdks/python".length()) + "/build/apache-beam.tar.gz").substring(1)
- }
- return pythonSDKLocation
-}
diff --git a/.test-infra/jenkins/job_PerformanceTests_Spark.groovy b/.test-infra/jenkins/job_PerformanceTests_Spark.groovy
deleted file mode 100644
index c7bb483..0000000
--- a/.test-infra/jenkins/job_PerformanceTests_Spark.groovy
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-
-// This job runs the Beam performance tests on PerfKit Benchmarker.
-job('beam_PerformanceTests_Spark'){
- // Set default Beam job properties.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
- commonJobProperties.enablePhraseTriggeringFromPullRequest(
- delegate,
- 'Spark Performance Test',
- 'Run Spark Performance Test')
-
- // Run job in postcommit every 6 hours, don't trigger every push, and
- // don't email individual committers.
- commonJobProperties.setAutoJob(
- delegate,
- 'H */6 * * *')
-
- def argMap = [
- benchmarks: 'dpb_wordcount_benchmark',
- // There are currently problems uploading to Dataproc, so we use a file
- // already present on the machines as input.
- dpb_wordcount_input: '/etc/hosts',
- config_override: 'dpb_wordcount_benchmark.dpb_service.service_type=dataproc',
- bigquery_table: 'beam_performance.spark_pkp_results'
- ]
-
- commonJobProperties.buildPerformanceTest(delegate, argMap)
-}
diff --git a/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
new file mode 100644
index 0000000..e4bf45e
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Flink runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Flink',
+ 'Run XVR_Flink PostCommit', 'Flink CrossLanguageValidatesRunner Tests', this) {
+ description('Runs the CrossLanguageValidatesRunner suite on the Flink runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Publish all test results to Jenkins
+ publishers {
+ archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':runners:flink:1.8:job-server:validatesCrossLanguageRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Flink.groovy
new file mode 100644
index 0000000..a24983c
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Flink.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Go integration tests against the Flink runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Go_VR_Flink',
+ 'Run Go Flink ValidatesRunner', 'Go Flink ValidatesRunner Tests', this) {
+ description('Runs Go integration tests on the Flink runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:go:test:flinkValidatesRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Spark.groovy b/.test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Spark.groovy
new file mode 100644
index 0000000..6f8353d
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Spark.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Go integration tests against the Spark runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Go_VR_Spark',
+ 'Run Go Spark ValidatesRunner', 'Go Spark ValidatesRunner Tests', this) {
+ description('Runs Go integration tests on the Spark runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:go:test:sparkValidatesRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Examples.groovy b/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Examples.groovy
index ad7546e..9e1004c 100644
--- a/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Examples.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Examples.groovy
@@ -33,7 +33,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java-examples:java11PostCommit')
+ tasks(':runners:google-cloud-dataflow-java:examples:java11PostCommit')
// Increase parallel worker threads above processor limit since most time is
// spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are slow
diff --git a/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Portability_Examples.groovy b/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Portability_Examples.groovy
index b670352..eadf255 100644
--- a/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Portability_Examples.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java11_Dataflow_Portability_Examples.groovy
@@ -33,7 +33,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java-examples:verifyPortabilityApi')
+ tasks(':runners:google-cloud-dataflow-java:examples:verifyPortabilityApi')
switches ('-Pdockerfile=Dockerfile-java11')
// Increase parallel worker threads above processor limit since most time is
diff --git a/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy
index 9210139..fa5053a 100644
--- a/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow.groovy
@@ -34,7 +34,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java:validatesRunnerFnApiWorkerTest')
+ tasks(':runners:google-cloud-dataflow-java:validatesRunnerFnApiWorkerTest')
switches ('-Pdockerfile=Dockerfile-java11')
// Increase parallel worker threads above processor limit since most time is
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow.groovy
index 30b59f1..2b6f3c3 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow.groovy
@@ -38,9 +38,9 @@
shell('echo *** RUN NEXMARK IN BATCH MODE USING DATAFLOW RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-google-cloud-dataflow-java"' +
+ switches('-Pnexmark.runner=":runners:google-cloud-dataflow-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DataflowRunner',
@@ -58,9 +58,9 @@
shell('echo *** RUN NEXMARK IN STREAMING MODE USING DATAFLOW RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-google-cloud-dataflow-java"' +
+ switches('-Pnexmark.runner=":runners:google-cloud-dataflow-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DataflowRunner',
@@ -78,9 +78,9 @@
shell('echo *** RUN NEXMARK IN SQL BATCH MODE USING DATAFLOW RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-google-cloud-dataflow-java"' +
+ switches('-Pnexmark.runner=":runners:google-cloud-dataflow-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DataflowRunner',
@@ -99,9 +99,9 @@
shell('echo *** RUN NEXMARK IN SQL STREAMING MODE USING DATAFLOW RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-google-cloud-dataflow-java"' +
+ switches('-Pnexmark.runner=":runners:google-cloud-dataflow-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DataflowRunner',
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
index 1318059..706b536 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Direct.groovy
@@ -37,9 +37,9 @@
shell('echo *** RUN NEXMARK IN BATCH MODE USING DIRECT RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-direct-java"' +
+ switches('-Pnexmark.runner=":runners:direct-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DirectRunner',
@@ -53,9 +53,9 @@
shell('echo *** RUN NEXMARK IN STREAMING MODE USING DIRECT RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-direct-java"' +
+ switches('-Pnexmark.runner=":runners:direct-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DirectRunner',
@@ -69,9 +69,9 @@
shell('echo *** RUN NEXMARK IN SQL BATCH MODE USING DIRECT RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-direct-java"' +
+ switches('-Pnexmark.runner=":runners:direct-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DirectRunner',
@@ -86,9 +86,9 @@
shell('echo *** RUN NEXMARK IN SQL STREAMING MODE USING DIRECT RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-direct-java"' +
+ switches('-Pnexmark.runner=":runners:direct-java"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=DirectRunner',
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
index 63fc48c..038d6b3 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Flink.groovy
@@ -38,9 +38,9 @@
shell('echo *** RUN NEXMARK IN BATCH MODE USING FLINK RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-flink_2.11"' +
+ switches('-Pnexmark.runner=":runners:flink:1.8"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--streaming=false',
@@ -53,9 +53,9 @@
shell('echo *** RUN NEXMARK IN STREAMING MODE USING FLINK RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-flink_2.11"' +
+ switches('-Pnexmark.runner=":runners:flink:1.8"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--streaming=true',
@@ -68,9 +68,9 @@
shell('echo *** RUN NEXMARK IN SQL BATCH MODE USING FLINK RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-flink_2.11"' +
+ switches('-Pnexmark.runner=":runners:flink:1.8"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--queryLanguage=sql',
@@ -83,9 +83,9 @@
shell('echo *** RUN NEXMARK IN SQL STREAMING MODE USING FLINK RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-flink_2.11"' +
+ switches('-Pnexmark.runner=":runners:flink:1.8"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--queryLanguage=sql',
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy
index 20847dc..5fb7d24 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Spark.groovy
@@ -36,9 +36,9 @@
shell('echo *** RUN NEXMARK IN BATCH MODE USING SPARK RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-spark"' +
+ switches('-Pnexmark.runner=":runners:spark"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=SparkRunner',
@@ -51,9 +51,9 @@
shell('echo *** RUN NEXMARK SQL IN BATCH MODE USING SPARK RUNNER ***')
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-java-nexmark:run')
+ tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
- switches('-Pnexmark.runner=":beam-runners-spark"' +
+ switches('-Pnexmark.runner=":runners:spark"' +
' -Pnexmark.args="' +
[NexmarkBigqueryProperties.nexmarkBigQueryArgs,
'--runner=SparkRunner',
diff --git a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
index 7aee97a..4da75f9 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-flink_2.11-job-server:validatesPortableRunnerBatch')
+ tasks(':runners:flink:1.8:job-server:validatesPortableRunnerBatch')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy
index c016a5f..612c154 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Streaming.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-flink_2.11-job-server:validatesPortableRunnerStreaming')
+ tasks(':runners:flink:1.8:job-server:validatesPortableRunnerStreaming')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy
new file mode 100644
index 0000000..1e9728a
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Java ValidatesRunner tests against the Spark runner in batch mode.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_PVR_Spark_Batch',
+ 'Run Java Spark PortableValidatesRunner Batch', 'Java Spark PortableValidatesRunner Batch Tests', this) {
+ description('Runs the Java PortableValidatesRunner suite on the Spark runner in batch mode.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Publish all test results to Jenkins
+ publishers {
+ archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':runners:spark:job-server:validatesPortableRunnerBatch')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy
index faaa7af..3cbf396 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy
@@ -37,7 +37,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-apex:validatesRunner')
+ tasks(':runners:apex:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
index 7280289..530fba6 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
@@ -40,7 +40,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java:validatesRunner')
+ tasks(':runners:google-cloud-dataflow-java:validatesRunner')
// Increase parallel worker threads above processor limit since most time is
// spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are slow
// because each one launches a Dataflow job with about 3 mins of overhead.
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
index 62e7361..3826c78 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
@@ -39,7 +39,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java:validatesRunnerFnApiWorkerExecutableStageTest')
+ tasks(':runners:google-cloud-dataflow-java:validatesRunnerFnApiWorkerExecutableStageTest')
// Increase parallel worker threads above processor limit since most time is
// spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are slow
// because each one launches a Dataflow job with about 3 mins of overhead.
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
index 4e6da6a..74e49b6 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
@@ -34,7 +34,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java:validatesJava11Runner')
+ tasks(':runners:google-cloud-dataflow-java:validatesJava11Runner')
// Increase parallel worker threads above processor limit since most time is
// spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are slow
// because each one launches a Dataflow job with about 3 mins of overhead.
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct.groovy
index f933652..923e9c3 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct.groovy
@@ -36,7 +36,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-direct-java:validatesRunner')
+ tasks(':runners:direct-java:validatesRunner')
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy
index 0df2f84..5e3c8cf 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy
@@ -38,14 +38,14 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-direct-java:shadowJar')
- tasks(':beam-runners-direct-java:shadowTestJar')
+ tasks(':runners:direct-java:shadowJar')
+ tasks(':runners:direct-java:shadowTestJar')
switches("-Dorg.gradle.java.home=${JAVA_8_HOME}")
}
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-direct-java:validatesRunner')
+ tasks(':runners:direct-java:validatesRunner')
switches("-Dorg.gradle.java.home=${JAVA_11_HOME}")
switches('-x shadowJar')
switches('-x shadowTestJar')
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
index d21ae86..d5e6da9 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
@@ -37,7 +37,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-flink_2.11:validatesRunner')
+ tasks(':runners:flink:1.8:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
index 2b0dd7d..e1ef58f 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
@@ -41,7 +41,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-gearpump:validatesRunner')
+ tasks(':runners:gearpump:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
index 587e885..54ad764 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
@@ -40,7 +40,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-google-cloud-dataflow-java:validatesRunnerFnApiWorkerTest')
+ tasks(':runners:google-cloud-dataflow-java:validatesRunnerFnApiWorkerTest')
// Increase parallel worker threads above processor limit since most time is
// spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are slow
// because each one launches a Dataflow job with about 3 mins of overhead.
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy
index fe08abd..c60b36a 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy
@@ -37,7 +37,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-samza:validatesRunner')
+ tasks(':runners:samza:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy
index 8804d0d..6dc0fcf 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy
@@ -37,7 +37,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-runners-spark:validatesRunner')
+ tasks(':runners:spark:validatesRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy b/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
new file mode 100644
index 0000000..a2bc53e
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// Tests creation and execution of portable pipeline Jars on the Flink runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_PortableJar_Flink',
+ 'Run PortableJar_Flink PostCommit', 'Flink Portable Jar Tests', this) {
+ description('Tests creation and execution of portable pipeline Jars on the Flink runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':runners:flink:1.8:job-server:testPipelineJar')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python2.groovy b/.test-infra/jenkins/job_PostCommit_Python2.groovy
new file mode 100644
index 0000000..f8a7a28
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python2.groovy
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job defines the Python postcommit tests.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python2', 'Run Python 2 PostCommit',
+ 'Python2_PC("Run Python 2 PostCommit")', this) {
+ description('Runs Python postcommit tests using Python 2.7.')
+
+ previousNames('/beam_PostCommit_Python_Verify/')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ publishers {
+ archiveJunit('**/nosetests*.xml')
+ }
+
+ // Execute shell command to test Python SDK.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':python2PostCommit')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python35.groovy b/.test-infra/jenkins/job_PostCommit_Python35.groovy
new file mode 100644
index 0000000..7b04b6f
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python35.groovy
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job defines the Python postcommit tests.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python35', 'Run Python 3.5 PostCommit',
+ 'Python35_PC("Run Python 3.5 PostCommit")', this) {
+ description('Runs Python postcommit tests using Python 3.5.')
+
+ previousNames('/beam_PostCommit_Python3_Verify/')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ publishers {
+ archiveJunit('**/nosetests*.xml')
+ }
+
+ // Execute shell command to test Python SDK.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':python35PostCommit')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy
new file mode 100644
index 0000000..8056d9c
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Python ValidatesRunner tests against the Flink runner on Python 3.5.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python35_VR_Flink',
+ 'Run Python 3.5 Flink ValidatesRunner', 'Run Python 3.5 Flink ValidatesRunner', this) {
+ description('Runs the Python 3.5 ValidatesRunner suite on the Flink runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py35:flinkValidatesRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python36.groovy b/.test-infra/jenkins/job_PostCommit_Python36.groovy
new file mode 100644
index 0000000..d6a5095
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python36.groovy
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job defines the Python postcommit tests.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python36', 'Run Python 3.6 PostCommit',
+ 'Python36_PC("Run Python 3.6 PostCommit")', this) {
+ description('Runs Python postcommit tests using Python 3.6.')
+
+ previousNames('/beam_PostCommit_Python3_Verify/')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ publishers {
+ archiveJunit('**/nosetests*.xml')
+ }
+
+ // Execute shell command to test Python SDK.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':python36PostCommit')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python37.groovy b/.test-infra/jenkins/job_PostCommit_Python37.groovy
new file mode 100644
index 0000000..ea511cd
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python37.groovy
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job defines the Python postcommit tests.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python37', 'Run Python 3.7 PostCommit',
+ 'Python37_PC("Run Python 3.7 PostCommit")', this) {
+ description('Runs Python postcommit tests using Python 3.7.')
+
+ previousNames('/beam_PostCommit_Python3_Verify/')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ publishers {
+ archiveJunit('**/nosetests*.xml')
+ }
+
+ // Execute shell command to test Python SDK.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':python37PostCommit')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python3_Verify.groovy b/.test-infra/jenkins/job_PostCommit_Python3_Verify.groovy
deleted file mode 100644
index e51ccce..0000000
--- a/.test-infra/jenkins/job_PostCommit_Python3_Verify.groovy
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-import PostcommitJobBuilder
-
-// This job defines the Python postcommit tests.
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python3_Verify', 'Run Python PostCommit',
- 'Python SDK PostCommit Tests on Python 3', this) {
- description('Runs postcommit tests on the Python SDK on Python 3.')
-
- // Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
-
- // Execute shell command to test Python SDK.
- steps {
- gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':python3PostCommit')
- commonJobProperties.setGradleSwitches(delegate)
- }
- }
-}
diff --git a/.test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
new file mode 100644
index 0000000..c7da4bb
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Dataflow.groovy
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+import CronJobBuilder
+
+
+// This job runs the Chicao Taxi Example script on Dataflow
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Chicago_Taxi_Dataflow',
+ 'Run Chicago Taxi on Dataflow', 'Google Cloud Dataflow Runner Chicago Taxi Example', this) {
+ description('Runs the Chicago Taxi Example on the Dataflow runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:dataflow:py2:dataflowChicagoTaxiExample')
+ switches('-PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi')
+ switches('-Prunner=DataflowRunner')
+ }
+ }
+}
+
+CronJobBuilder.cronJob('beam_PostCommit_Python_Chicago_Taxi_Dataflow', 'H 14 * * *', this) {
+ description('Runs the Chicago Taxi Example on the Dataflow runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:dataflow:py2:dataflowChicagoTaxiExample')
+ switches('-PgcsRoot=gs://temp-storage-for-perf-tests/chicago-taxi')
+ switches('-Prunner=DataflowRunner')
+ }
+ }
+}
\ No newline at end of file
diff --git a/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy b/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
new file mode 100644
index 0000000..175ad68
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the integration test of python mongodbio class.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_MongoDBIO_IT',
+ 'Run Python MongoDBIO_IT', 'Python MongoDBIO Integration Test',this) {
+ description('Runs the Python MongoDBIO Integration Test.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:direct:py2:mongodbioIT')
+ tasks(':sdks:python:test-suites:direct:py35:mongodbioIT')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
index da5094e..f25f133 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
@@ -28,10 +28,16 @@
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate)
+ publishers {
+ archiveJunit('**/nosetests*.xml')
+ }
+
// Execute shell command to test Python SDK.
// TODO: Parallel the script run with Jenkins DSL or Gradle.
steps {
shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python2')
- shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python3')
+ shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python35')
+ shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python36')
+ shell('cd ' + commonJobProperties.checkoutDir + ' && bash sdks/python/container/run_validatescontainer.sh python37')
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
index cfdd01b..f7fd557 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
@@ -28,12 +28,22 @@
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate)
+ publishers {
+ archiveJunit('**/nosetests*.xml')
+ }
+
// Execute gradle task to test Python SDK.
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-python:validatesRunnerBatchTests')
- tasks(':beam-sdks-python:validatesRunnerStreamingTests')
+ tasks(':sdks:python:test-suites:dataflow:py2:validatesRunnerBatchTests')
+ tasks(':sdks:python:test-suites:dataflow:py2:validatesRunnerStreamingTests')
+ tasks(':sdks:python:test-suites:dataflow:py35:validatesRunnerBatchTests')
+ tasks(':sdks:python:test-suites:dataflow:py36:validatesRunnerBatchTests')
+ tasks(':sdks:python:test-suites:dataflow:py37:validatesRunnerBatchTests')
+ tasks(':sdks:python:test-suites:dataflow:py35:validatesRunnerStreamingTests')
+ tasks(':sdks:python:test-suites:dataflow:py36:validatesRunnerStreamingTests')
+ tasks(':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy
new file mode 100644
index 0000000..59b45f7
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Python ValidatesRunner tests against the Spark runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_VR_Spark',
+ 'Run Python Spark ValidatesRunner', 'Python Spark ValidatesRunner Tests', this) {
+ description('Runs the Python ValidatesRunner suite on the Spark runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py2:sparkValidatesRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python_Verify.groovy b/.test-infra/jenkins/job_PostCommit_Python_Verify.groovy
deleted file mode 100644
index 7e2a3f8..0000000
--- a/.test-infra/jenkins/job_PostCommit_Python_Verify.groovy
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-import PostcommitJobBuilder
-
-// This job defines the Python postcommit tests.
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Verify', 'Run Python PostCommit',
- 'Python SDK PostCommit Tests', this) {
- description('Runs postcommit tests on the Python SDK.')
-
- previousNames('beam_PostCommit_PythonVerify')
-
- // Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
-
- // Execute shell command to test Python SDK.
- steps {
- gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':pythonPostCommit')
- commonJobProperties.setGradleSwitches(delegate)
- }
- }
-}
diff --git a/.test-infra/jenkins/job_PostCommit_Website_Publish.groovy b/.test-infra/jenkins/job_PostCommit_Website_Publish.groovy
index 669a84ab..55fad49 100644
--- a/.test-infra/jenkins/job_PostCommit_Website_Publish.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Website_Publish.groovy
@@ -33,8 +33,8 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-website:clean')
- tasks(':beam-website:publishWebsite')
+ tasks(':website:clean')
+ tasks(':website:publishWebsite')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_Website_Test.groovy b/.test-infra/jenkins/job_PostCommit_Website_Test.groovy
index 1abed32..d077e97 100644
--- a/.test-infra/jenkins/job_PostCommit_Website_Test.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Website_Test.groovy
@@ -29,7 +29,7 @@
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-website:testWebsite -PdisableExternal=false')
+ tasks(':website:testWebsite -PdisableExternal=false')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy b/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
new file mode 100644
index 0000000..ec7fef2
--- /dev/null
+++ b/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'JavaBeamZetaSQL',
+ gradleTask: ':javaPreCommitBeamZetaSQL',
+ gradleSwitches: ['-PdisableSpotlessCheck=true'], // spotless checked in separate pre-commit
+ triggerPathPatterns: [
+ '^sdks/java/extensions/sql/.*$',
+ ]
+)
+builder.build {
+ publishers {
+ archiveJunit('**/build/test-results/**/*.xml')
+ }
+}
diff --git a/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy b/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy
index 9ee5f12..ff1fa05 100644
--- a/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy
@@ -16,12 +16,13 @@
* limitations under the License.
*/
+import CommonJobProperties as commonJobProperties
import PrecommitJobBuilder
PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Portable_Python',
- gradleTask: ':portablePythonPreCommit',
+ gradleTask: ':clean', // Do nothing here. Add test configs below.
triggerPathPatterns: [
'^model/.*$',
'^runners/core-construction-java/.*$',
@@ -34,4 +35,31 @@
'^release/.*$',
]
)
-builder.build {}
+
+builder.build {
+ // Due to BEAM-7993, run multiple Python version of portable precommit
+ // tests in parallel could lead python3 container crash. We manually
+ // config gradle steps here to run tests in sequential.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py2:preCommitPy2')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py35:preCommitPy35')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py36:preCommitPy36')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:test-suites:portable:py37:preCommitPy37')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PreCommit_Python.groovy b/.test-infra/jenkins/job_PreCommit_Python.groovy
index 00fc6e1..69be311 100644
--- a/.test-infra/jenkins/job_PreCommit_Python.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Python.groovy
@@ -32,6 +32,6 @@
// Publish all test results to Jenkins. Note that Nose documentation
// specifically mentions that it produces JUnit compatible test results.
publishers {
- archiveJunit('**/nosetests.xml')
+ archiveJunit('**/nosetests*.xml')
}
}
diff --git a/.test-infra/jenkins/job_PreCommit_PythonLint.groovy b/.test-infra/jenkins/job_PreCommit_PythonLint.groovy
new file mode 100644
index 0000000..caab66d
--- /dev/null
+++ b/.test-infra/jenkins/job_PreCommit_PythonLint.groovy
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'PythonLint',
+ gradleTask: ':pythonLintPreCommit',
+ triggerPathPatterns: [
+ '^sdks/python/.*$',
+ '^release/.*$',
+ ]
+)
+builder.build()
diff --git a/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy b/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
index 435c075..2812681 100644
--- a/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
@@ -1,4 +1,3 @@
-
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@@ -19,11 +18,11 @@
import PrecommitJobBuilder
-// This job runs the suite of ValidatesRunner tests against the Flink runner.
+// This job runs the suite of Python ValidatesRunner tests against the Flink runner on Python 2.
PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
- nameBase: 'Python_PVR_Flink',
- gradleTask: ':beam-sdks-python:flinkValidatesRunner',
+ nameBase: 'Python2_PVR_Flink',
+ gradleTask: ':sdks:python:test-suites:portable:py2:flinkValidatesRunner',
triggerPathPatterns: [
'^model/.*$',
'^runners/core-construction-java/.*$',
@@ -34,8 +33,10 @@
'^runners/reference/.*$',
'^sdks/python/.*$',
'^release/.*$',
+ // Test regressions of cross-language KafkaIO test
+ '^sdks/java/io/kafka/.*$',
]
)
builder.build {
- previousNames('beam_PostCommit_Python_VR_Flink')
+ previousNames('beam_PreCommit_Python_PVR_Flink')
}
diff --git a/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy b/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
index c5cf28f..03b6c34 100644
--- a/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
@@ -21,7 +21,7 @@
PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Website_Stage_GCS',
- gradleTask: ':beam-website:stageWebsite',
+ gradleTask: ':website:stageWebsite',
triggerPathPatterns: ['^website/.*$']
)
builder.build {
diff --git a/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy b/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
index 4df59b1..2b7daae 100644
--- a/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
+++ b/.test-infra/jenkins/job_ReleaseCandidate_Python.groovy
@@ -22,7 +22,7 @@
description('Runs verification of the Python release candidate.')
// Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 360)
// Allows triggering this build against pull requests.
commonJobProperties.enablePhraseTriggeringFromPullRequest(
diff --git a/.test-infra/jenkins/job_Release_Gradle_Build.groovy b/.test-infra/jenkins/job_Release_Gradle_Build.groovy
new file mode 100644
index 0000000..dc82585
--- /dev/null
+++ b/.test-infra/jenkins/job_Release_Gradle_Build.groovy
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+
+// This job runs complete cycle of Gradle build against the official release
+// version. Release manager should use this job to verify a release branch
+// after cut.
+job('beam_Release_Gradle_Build') {
+ description('Verify Gradle build against the official release version.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Allows triggering this build against pull requests.
+ commonJobProperties.enablePhraseTriggeringFromPullRequest(
+ delegate,
+ 'Release_Build ("Run Release Gradle Build")',
+ 'Run Release Gradle Build')
+
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks('build')
+ commonJobProperties.setGradleSwitches(delegate)
+ switches('-PisRelease')
+ switches('--stacktrace')
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy b/.test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
index ab02b3a..8548b15 100644
--- a/.test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
+++ b/.test-infra/jenkins/job_Release_Python_NightlySnapshot.groovy
@@ -44,13 +44,13 @@
// Cleanup Python directory.
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-python:clean')
+ tasks(':sdks:python:clean')
commonJobProperties.setGradleSwitches(delegate)
}
// Build snapshot.
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-sdks-python:buildSnapshot')
+ tasks(':sdks:python:buildSnapshot')
commonJobProperties.setGradleSwitches(delegate)
}
// Publish snapshot to a public accessible GCS directory.
diff --git a/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml b/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml
deleted file mode 100644
index c829e05..0000000
--- a/.test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml
+++ /dev/null
@@ -1,44 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is a pkb benchmark configuration file, used when running the IO ITs
-# that use this data store. It allows users to run tests when they are on a
-# separate network from the kubernetes cluster by reading the hadoop namenode IP
-# address from the LoadBalancer service.
-#
-# When running Perfkit with DirectRunner - format pattern must additionally contain
-# dfs.client.use.datanode.hostname set to true:
-# format: '[{\"fs.defaultFS\":\"hdfs://{{LoadBalancerIp}}:9000\",\"dfs.replication\":1,\"dfs.client.use.datanode.hostname\":\"true\" }]'
-# and /etc/hosts should be modified with an entries containing:
-# LoadBalancerIp HadoopMasterPodName
-# LoadBalancerIp FQDN-HadoopDatanode-0
-# LoadBalancerIp FQDN-HadoopDatanode-1
-# LoadBalancerIp FQDN-HadoopDatanode-2
-# otherwise hdfs client won't be able to reach datanodes. Proper configuration to add
-# will be generated when setup-all.sh script will be used to create cluster.
-# FilenamePrefix is used in file-based-io-tests.
-
-static_pipeline_options:
-dynamic_pipeline_options:
- - name: hdfsConfiguration
- format: '[{\"fs.defaultFS\":\"hdfs://{{LoadBalancerIp}}:9000\",\"dfs.replication\":1}]'
- type: LoadBalancerIp
- serviceName: hadoop
- - name: filenamePrefix
- format: 'hdfs://{{LoadBalancerIp}}:9000/TEXTIO_IT_'
- type: LoadBalancerIp
- serviceName: hadoop
diff --git a/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml b/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
deleted file mode 100644
index 72f458a..0000000
--- a/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
+++ /dev/null
@@ -1,40 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is a pkb benchmark configuration file, used when running the IO ITs
-# that use this data store. It allows users to run tests when they are on a
-# separate network from the kubernetes cluster by reading the hadoop namenode IP
-# address from the LoadBalancer service.
-#
-# When running Perfkit with DirectRunner - format pattern must additionally contain
-# dfs.client.use.datanode.hostname set to true:
-# format: '[{\"fs.defaultFS\":\"hdfs://{{LoadBalancerIp}}:9000\",\"dfs.replication\":1,\"dfs.client.use.datanode.hostname\":\"true\" }]'
-# and /etc/hosts should be modified with an entry containing:
-# LoadBalancerIp HadoopMasterPodName
-# otherwise hdfs client won't be able to reach datanode.
-# FilenamePrefix is used in file-based-io-tests.
-
-static_pipeline_options:
-dynamic_pipeline_options:
- - name: hdfsConfiguration
- format: '[{\"fs.defaultFS\":\"hdfs://{{LoadBalancerIp}}:9000\",\"dfs.replication\":1}]'
- type: LoadBalancerIp
- serviceName: hadoop-external
- - name: filenamePrefix
- format: 'hdfs://{{LoadBalancerIp}}:9000/TEXTIO_IT_'
- type: LoadBalancerIp
- serviceName: hadoop-external
diff --git a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml
index 08e7350..d92abc0 100644
--- a/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml
+++ b/.test-infra/kubernetes/kafka-cluster/03-zookeeper/30service.yml
@@ -24,3 +24,4 @@
name: client
selector:
app: zookeeper
+ type: LoadBalancer
diff --git a/.test-infra/kubernetes/kubernetes.sh b/.test-infra/kubernetes/kubernetes.sh
new file mode 100755
index 0000000..ca22066
--- /dev/null
+++ b/.test-infra/kubernetes/kubernetes.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Set of common operations that CI needs to invoke when using Kubernetes.
+# The operations can be invoked using a provided kubeconfig file
+# and kubernetes namespace.
+#
+# Specify the following environment variables to override defaults:
+# - KUBECONFIG: path to .kube/config file (default: $HOME/.kube/config)
+# - KUBERNETES_NAMESPACE: namespace to be used (default: default)
+set -euxo pipefail
+
+KUBECONFIG="${KUBECONFIG:=$HOME/.kube/config}"
+KUBERNETES_NAMESPACE="${KUBERNETES_NAMESPACE:=default}"
+KUBECTL="kubectl --kubeconfig=$KUBECONFIG --namespace=$KUBERNETES_NAMESPACE"
+
+function retry() {
+ local command=$1
+ local max_retries=$2
+ local sleep_time=$3
+
+ for ((i = 1; i <= max_retries; i++)); do
+ local output
+ output=$(eval "${command}")
+
+ local status=$?
+
+ if [[ ${status} == 0 ]] && [[ -n ${output} ]]; then
+ echo "${output}"
+ return 0
+ fi
+
+ if [[ $i == "${max_retries}" ]]; then
+ echo "Command failed after ${max_retries} retries" >&2
+ return 1
+ fi
+
+ sleep "${sleep_time}"
+ done
+}
+
+# Invokes "kubectl apply" using specified kubeconfig and namespace.
+#
+# Usage: ./kubernetes.sh apply <path to .yaml file>
+function apply() {
+ eval "$KUBECTL apply -R -f $1"
+}
+
+# Invokes "kubectl delete" using specified kubeconfig and namespace.
+#
+# Usage: ./kubernetes.sh delete <path to .yaml file>
+function delete() {
+ eval "$KUBECTL delete -R -f $1"
+}
+
+# Creates a namespace.
+#
+# Usage: ./kubernetes.sh createNamespace <namespace name>
+function createNamespace() {
+ eval "kubectl --kubeconfig=${KUBECONFIG} create namespace $1"
+}
+
+# Deletes whole namespace with all its contents.
+#
+# Usage: ./kubernetes.sh deleteNamespace <namespace name>
+function deleteNamespace() {
+ eval "kubectl --kubeconfig=${KUBECONFIG} delete namespace $1"
+}
+
+# Gets Load Balancer Ingress IP address.
+# Blocks and retries until the IP is present or retry limit is exceeded.
+#
+# Usage: ./kubernetes.sh loadBalancerIP <name of the load balancer service>
+function loadBalancerIP() {
+ local name=$1
+ local command="$KUBECTL get svc $name -ojsonpath='{.status.loadBalancer.ingress[0].ip}'"
+ retry "${command}" 36 10
+}
+
+"$@"
diff --git a/.test-infra/kubernetes/mongodb/load-balancer/pkb-config.yml b/.test-infra/kubernetes/mongodb/load-balancer/pkb-config.yml
deleted file mode 100644
index 299de0d..0000000
--- a/.test-infra/kubernetes/mongodb/load-balancer/pkb-config.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is a pkb benchmark configuration file, used when running the IO ITs
-# that use this data store. It allows users to run tests when they are on a
-# separate network from the kubernetes cluster by reading the mongo IP
-# address from the LoadBalancer service.
-#
-# This file defines pipeline options to pass to beam, as well as how to derive
-# the values for those pipeline options from kubernetes (where appropriate.)
-
-static_pipeline_options:
- - mongoDBDatabaseName: beam
- - mongoDBPort: 27017
-dynamic_pipeline_options:
- - name: mongoDBHostName
- type: LoadBalancerIp
- serviceName: mongo-load-balancer-service
diff --git a/.test-infra/kubernetes/postgres/pkb-config-local.yml b/.test-infra/kubernetes/postgres/pkb-config-local.yml
deleted file mode 100644
index 1bac0c4..0000000
--- a/.test-infra/kubernetes/postgres/pkb-config-local.yml
+++ /dev/null
@@ -1,34 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is a pkb benchmark configuration file, used when running the IO ITs
-# that use this data store. It allows users to run tests when they are on a
-# separate network from the kubernetes cluster by reading the postgres IP
-# address from the LoadBalancer service.
-#
-# This file defines pipeline options to pass to beam, as well as how to derive
-# the values for those pipeline options from kubernetes (where appropriate.)
-
-static_pipeline_options:
- - postgresUsername: postgres
- - postgresPassword: uuinkks
- - postgresDatabaseName: postgres
- - postgresSsl: false
-dynamic_pipeline_options:
- - name: postgresServerName
- type: LoadBalancerIp
- serviceName: postgres-for-dev
diff --git a/.test-infra/kubernetes/postgres/pkb-config.yml b/.test-infra/kubernetes/postgres/pkb-config.yml
deleted file mode 100644
index b943b17..0000000
--- a/.test-infra/kubernetes/postgres/pkb-config.yml
+++ /dev/null
@@ -1,32 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is a pkb benchmark configuration file, used when running the IO ITs
-# that use this data store.
-#
-# This file defines pipeline options to pass to beam, as well as how to derive
-# the values for those pipeline options from kubernetes (where appropriate.)
-
-static_pipeline_options:
- - postgresUsername: postgres
- - postgresPassword: uuinkks
- - postgresDatabaseName: postgres
- - postgresSsl: false
-dynamic_pipeline_options:
- - name: postgresServerName
- type: NodePortIp
- podLabel: name=postgres
diff --git a/.test-infra/metrics/README.md b/.test-infra/metrics/README.md
index a029e63..5503d99 100644
--- a/.test-infra/metrics/README.md
+++ b/.test-infra/metrics/README.md
@@ -17,12 +17,14 @@
under the License.
-->
# BeamMonitoring
-This folder contains resources required to deploy the Beam community metrics
-stack.
+This folder contains resources required to deploy the Beam metrics stack.
+There are two types of metrics in Beam project:
+* Community metrics
+* Metrics published by tests (IO Performance tests, Load tests and Nexmark tests)
-[Beam community dashboard is available here.](https://s.apache.org/beam-community-metrics)
+Both types of metrics are presented in [Grafana dashboard available here.](https://s.apache.org/beam-community-metrics)
-Whole stack can be deployed on your local machine as well.
+## Community metrics
This includes
* Python scripts for ingesting data from sources (Jenkins, JIRA,
@@ -30,6 +32,15 @@
* Postgres analytics database
* [Grafana](https://grafana.com) dashboarding UI
+## Test metrics
+Beam uses Prometheus to store metrics published by tests running on Jenkins.
+
+Prometheus stack consists of the following components
+* the main Prometheus server
+* Alertmanager
+* Pushgateway
+
+Both stacks can be deployed on your local machine.
All components run within Docker containers. These are composed together via
docker-compose for local hosting, and Kubernetes for the production instance on
GCP.
@@ -90,17 +101,21 @@
machine:
* Grafana: http://localhost:3000
-* Postgres DB: localhost:5432
+* Postgres DB: http://localhost:5432
+* Prometheus: http://localhost:9090
+* Pushgateway: http://localhost:9091
+* Alertmanager: http://localhost:9093
If you're deploying for the first time on your machine, follow the wiki instructions
on how to manually [configure
Grafana](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics#CommunityMetrics-GrafanaUI).
-Grafana and Postgres containers persist data to Docker volumes, which will be
+Grafana, Postgres and Prometheus containers persist data to Docker volumes, which will be
restored on subsequent runs. To start from a clean state, you must also wipe out
these volumes. (List volumes via `docker volume ls`)
## Kubernetes setup
-Kubernetes deployment instructions are maintained in the
-[wiki](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics).
+Kubernetes deployment instructions are maintained in the wiki:
+* [Community metrics](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics)
+* [Test metrics]() <!-- TODO(BEAM-8130): add a link to instructions -->
diff --git a/.test-infra/metrics/apply_configmaps.sh b/.test-infra/metrics/apply_configmaps.sh
new file mode 100755
index 0000000..2094ad4
--- /dev/null
+++ b/.test-infra/metrics/apply_configmaps.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Creates config maps used by Prometheus deployment and deletes old ones.
+
+set -euxo pipefail
+
+kubectl delete configmap prometheus-config --ignore-not-found=true
+kubectl delete configmap alertmanager-config --ignore-not-found=true
+
+kubectl create configmap prometheus-config --from-file=prometheus/prometheus/config
+kubectl create configmap alertmanager-config --from-file=prometheus/alertmanager/config
diff --git a/.test-infra/metrics/beamgrafana-deploy.yaml b/.test-infra/metrics/beamgrafana-deploy.yaml
index 3cbb2c0..c775649 100644
--- a/.test-infra/metrics/beamgrafana-deploy.yaml
+++ b/.test-infra/metrics/beamgrafana-deploy.yaml
@@ -34,7 +34,7 @@
fsGroup: 1000
containers:
- name: beamgrafana
- image: grafana/grafana
+ image: gcr.io/apache-beam-testing/beamgrafana:beammetrics20190909
securityContext:
runAsUser: 0
env:
@@ -42,19 +42,23 @@
value: "true"
- name: GF_AUTH_ANONYMOUS_ORG_NAME
value: Beam
- - name: GF_INSTALL_PLUGINS
- value: vonage-status-panel
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-admin-pwd
key: grafana_admin_password
- - name: PSQL_DB_USER
+ - name: DB_HOST
+ value: 127.0.0.1
+ - name: DB_PORT
+ value: "5432"
+ - name: DB_DBNAME
+ value: beammetrics
+ - name: DB_DBUSERNAME
valueFrom:
secretKeyRef:
name: beammetrics-psql-db-credentials
key: username
- - name: DB_PASSWORD
+ - name: DB_DBPWD
valueFrom:
secretKeyRef:
name: beammetrics-psql-db-credentials
@@ -81,26 +85,26 @@
mountPath: /secrets/cloudsql
readOnly: true
- name: beammetricssyncjenkins
- image: gcr.io/apache-beam-testing/beammetricssyncjenkins:v20181227
+ image: gcr.io/apache-beam-testing/beammetricssyncjenkins:beammetrics20190909
env:
- - name: JENSYNC_HOST
+ - name: DB_HOST
value: 127.0.0.1
- - name: JENSYNC_PORT
+ - name: DB_PORT
value: "5432"
- - name: JENSYNC_DBNAME
+ - name: DB_DBNAME
value: beammetrics
- - name: JENSYNC_DBUSERNAME
+ - name: DB_DBUSERNAME
valueFrom:
secretKeyRef:
name: beammetrics-psql-db-credentials
key: username
- - name: JENSYNC_DBPWD
+ - name: DB_DBPWD
valueFrom:
secretKeyRef:
name: beammetrics-psql-db-credentials
key: password
- name: beammetricssyncjira
- image: gcr.io/apache-beam-testing/beammetricssyncjira:v20181016
+ image: gcr.io/apache-beam-testing/beammetricssyncjira:beammetrics20190909
env:
- name: DB_HOST
value: 127.0.0.1
@@ -119,7 +123,7 @@
name: beammetrics-psql-db-credentials
key: password
- name: beammetricssyncgithub
- image: gcr.io/apache-beam-testing/beammetricssyncgithub:v20181029
+ image: gcr.io/apache-beam-testing/beammetricssyncgithub:beammetrics20190909
env:
- name: DB_HOST
value: 127.0.0.1
diff --git a/.test-infra/metrics/beamprometheus-deploy.yaml b/.test-infra/metrics/beamprometheus-deploy.yaml
new file mode 100644
index 0000000..40df19d
--- /dev/null
+++ b/.test-infra/metrics/beamprometheus-deploy.yaml
@@ -0,0 +1,125 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: prometheus
+ labels:
+ app: prometheus
+spec:
+ replicas: 1
+ template:
+ metadata:
+ labels:
+ app: prometheus
+ spec:
+ containers:
+ - image: prom/pushgateway
+ name: pushgateway
+ ports:
+ - containerPort: 9091
+ - image: prom/prometheus
+ name: prometheus
+ securityContext:
+ runAsUser: 0
+ ports:
+ - containerPort: 9090
+ args:
+ - --config.file=/etc/prometheus/prometheus.yml
+ - --web.console.libraries=/etc/prometheus/console_libraries
+ - --web.console.templates=/etc/prometheus/consoles
+ - --storage.tsdb.path=/prometheus
+ - --storage.tsdb.retention.time=365d
+ volumeMounts:
+ - mountPath: /prometheus
+ name: prometheus-storage
+ - mountPath: /etc/prometheus
+ name: prometheus-config
+ readOnly: true
+ - image: prom/alertmanager
+ name: alertmanager
+ ports:
+ - containerPort: 9093
+ volumeMounts:
+ - mountPath: /etc/alertmanager
+ name: alertmanager-config
+ readOnly: true
+ restartPolicy: Always
+ volumes:
+ - name: prometheus-storage
+ persistentVolumeClaim:
+ claimName: prometheus-storage
+ - name: prometheus-config
+ configMap:
+ name: prometheus-config
+ - name: alertmanager-config
+ configMap:
+ name: alertmanager-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: prometheus
+ labels:
+ app: prometheus
+spec:
+ ports:
+ - port: 9090
+ targetPort: 9090
+ selector:
+ app: prometheus
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: pushgateway
+ labels:
+ app: prometheus
+spec:
+ type: NodePort
+ ports:
+ - port: 9091
+ targetPort: 9091
+ nodePort: 30000
+ selector:
+ app: prometheus
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: alertmanager
+ labels:
+ app: prometheus
+spec:
+ ports:
+ - port: 9093
+ targetPort: 9093
+ selector:
+ app: prometheus
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: prometheus-storage
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 10Gi
diff --git a/.test-infra/metrics/build.gradle b/.test-infra/metrics/build.gradle
index 64fff94..bdcc040 100644
--- a/.test-infra/metrics/build.gradle
+++ b/.test-infra/metrics/build.gradle
@@ -31,7 +31,8 @@
}
task testMetricsStack {
- doLast { // TODO(BEAM-5837): Add some actual validation of the metrics stack
+ doLast {
+ // TODO(BEAM-5837): Add some actual validation of the metrics stack
println "Hello world!" }
}
dockerCompose.isRequiredBy(testMetricsStack)
diff --git a/.test-infra/metrics/build_and_publish_containers.sh b/.test-infra/metrics/build_and_publish_containers.sh
new file mode 100755
index 0000000..339fa3c
--- /dev/null
+++ b/.test-infra/metrics/build_and_publish_containers.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# This script will:
+# * build containers for beam metrics
+# * publish containers if second parameter is provided
+# * update beamgrafana-deploy.yaml with new container versions
+#
+
+if [ "$#" = 0 ]; then
+ echo "build_and_publish_container.sh <container_tag_name> [<should_publish_containers>]"
+ exit 0;
+fi
+
+echo
+echo ===========Start==========
+CONTAINER_VERSION_NAME=$1
+DO_PUSH=$2
+
+echo
+echo ===========Building containers==========
+docker build -t gcr.io/${PROJECT_ID}/beamgrafana:$CONTAINER_VERSION_NAME ./grafana
+docker build -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:$CONTAINER_VERSION_NAME ./sync/jenkins
+docker build -t gcr.io/${PROJECT_ID}/beammetricssyncjira:$CONTAINER_VERSION_NAME ./sync/jira
+docker build -t gcr.io/${PROJECT_ID}/beammetricssyncgithub:$CONTAINER_VERSION_NAME ./sync/github
+
+if [ "$DO_PUSH" = true ]; then
+ echo
+ echo ===========Publishing containers==========
+ docker push gcr.io/${PROJECT_ID}/beamgrafana:$CONTAINER_VERSION_NAME
+ docker push gcr.io/${PROJECT_ID}/beammetricssyncjenkins:$CONTAINER_VERSION_NAME
+ docker push gcr.io/${PROJECT_ID}/beammetricssyncjira:$CONTAINER_VERSION_NAME
+ docker push gcr.io/${PROJECT_ID}/beammetricssyncgithub:$CONTAINER_VERSION_NAME
+fi
+
+echo
+echo ===========Updating deployment script==========
+set -o xtrace
+sed -i "s/\( *image: gcr.io\/${PROJECT_ID}\/beam.*:\).*$/\1$CONTAINER_VERSION_NAME/g" ./beamgrafana-deploy.yaml
+set +o xtrace
+
diff --git a/.test-infra/metrics/dashboards/Post-Commits_status_dashboard.json b/.test-infra/metrics/dashboards/Post-Commits_status_dashboard.json
deleted file mode 100644
index 1b7e15a..0000000
--- a/.test-infra/metrics/dashboards/Post-Commits_status_dashboard.json
+++ /dev/null
@@ -1,175 +0,0 @@
-{
- "annotations": {
- "list": [
- {
- "builtIn": 1,
- "datasource": "-- Grafana --",
- "enable": true,
- "hide": true,
- "iconColor": "rgba(0, 211, 255, 1)",
- "name": "Annotations & Alerts",
- "type": "dashboard"
- }
- ]
- },
- "editable": true,
- "gnetId": null,
- "graphTooltip": 0,
- "id": 8,
- "links": [],
- "panels": [
- {
- "columns": [],
- "datasource": "BeamPSQL",
- "fontSize": "80%",
- "gridPos": {
- "h": 32,
- "w": 24,
- "x": 0,
- "y": 0
- },
- "id": 2,
- "links": [],
- "pageSize": 1000,
- "scroll": false,
- "showHeader": true,
- "sort": {
- "col": 0,
- "desc": false
- },
- "styles": [
- {
- "alias": "",
- "colorMode": "cell",
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 2,
- "mappingType": 1,
- "pattern": "job_\\d*",
- "preserveFormat": false,
- "thresholds": [
- "0.5",
- "0.1"
- ],
- "type": "string",
- "unit": "short",
- "valueMaps": [
- {
- "text": "Success",
- "value": "1"
- },
- {
- "text": "Fail",
- "value": "0"
- }
- ]
- },
- {
- "alias": "Job Name",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 2,
- "link": true,
- "linkTargetBlank": true,
- "linkUrl": "https://builds.apache.org/job/${__cell}",
- "mappingType": 1,
- "pattern": "job_name",
- "thresholds": [],
- "type": "string",
- "unit": "short"
- }
- ],
- "targets": [
- {
- "aggregation": "Last",
- "alias": "job",
- "decimals": 2,
- "displayAliasType": "Warning / Critical",
- "displayType": "Regular",
- "displayValueWithAlias": "Never",
- "format": "table",
- "group": [],
- "hide": false,
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with src1 as (select * from crosstab ('\n select *\n from (SELECT\n t.job_name,\n ROW_NUMBER() OVER (PARTITION BY job_name ORDER BY build_id desc) AS r,\n case when t.build_result like ''SUCCESS'' then 1 else 0 end as res\n FROM\n jenkins_builds t\n where job_name like ''beam_PostCommit_%'' \n and job_name not like ''%_PR''\n and (t.build_timestamp BETWEEN '$__timeFrom()' AND '$__timeTo()')\n ) x\n where x.r < 11')\n \n AS (\n job_name varchar,\n job_1 integer,\n job_2 integer,\n job_3 integer,\n job_4 integer,\n job_5 integer,\n job_6 integer,\n job_7 integer,\n job_8 integer,\n job_9 integer,\n job_10 integer\n )),\n src2 as (select * from crosstab ('\n select *\n from (SELECT\n t.job_name,\n ROW_NUMBER() OVER (PARTITION BY job_name ORDER BY build_id desc) AS r,\n case when t.build_result like ''SUCCESS'' then 1 else 0 end as res\n FROM\n jenkins_builds t\n where job_name like ''beam_PreCommit_%_Cron''\n and (t.build_timestamp BETWEEN '$__timeFrom()' AND '$__timeTo()')\n ) x\n where x.r < 11')\n \n AS (\n job_name varchar,\n job_1 integer,\n job_2 integer,\n job_3 integer,\n job_4 integer,\n job_5 integer,\n job_6 integer,\n job_7 integer,\n job_8 integer,\n job_9 integer,\n job_10 integer\n ))\n \nselect *\nfrom src2\nunion all\nselect *\nfrom src1;",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "units": "none",
- "valueHandler": "Number Threshold",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Post-Commits statuses",
- "transform": "table",
- "type": "table"
- }
- ],
- "refresh": "30s",
- "schemaVersion": 16,
- "style": "dark",
- "tags": [],
- "templating": {
- "list": []
- },
- "time": {
- "from": "now-7d",
- "to": "now"
- },
- "timepicker": {
- "refresh_intervals": [
- "5s",
- "10s",
- "30s",
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "1d"
- ],
- "time_options": [
- "5m",
- "15m",
- "1h",
- "6h",
- "12h",
- "24h",
- "2d",
- "7d",
- "30d"
- ]
- },
- "timezone": "",
- "title": "Post-Commits status dashboard",
- "uid": "8N6LVeCmk",
- "version": 10
-}
diff --git a/.test-infra/metrics/dashboards/code_velocity.json b/.test-infra/metrics/dashboards/code_velocity.json
deleted file mode 100644
index c9ea8f1..0000000
--- a/.test-infra/metrics/dashboards/code_velocity.json
+++ /dev/null
@@ -1,1167 +0,0 @@
-{
- "__inputs": [
- {
- "name": "DS_BEAMPSQL",
- "label": "BeamPSQL",
- "description": "",
- "type": "datasource",
- "pluginId": "postgres",
- "pluginName": "PostgreSQL"
- }
- ],
- "__requires": [
- {
- "type": "grafana",
- "id": "grafana",
- "name": "Grafana",
- "version": "5.3.2"
- },
- {
- "type": "panel",
- "id": "graph",
- "name": "Graph",
- "version": "5.0.0"
- },
- {
- "type": "datasource",
- "id": "postgres",
- "name": "PostgreSQL",
- "version": "5.0.0"
- },
- {
- "type": "panel",
- "id": "singlestat",
- "name": "Singlestat",
- "version": "5.0.0"
- },
- {
- "type": "panel",
- "id": "table",
- "name": "Table",
- "version": "5.0.0"
- }
- ],
- "annotations": {
- "list": [
- {
- "builtIn": 1,
- "datasource": "-- Grafana --",
- "enable": true,
- "hide": true,
- "iconColor": "rgba(0, 211, 255, 1)",
- "name": "Annotations & Alerts",
- "type": "dashboard"
- }
- ]
- },
- "description": "Shows common Code Velocity metrics, like reviewer load and open/closed PRs.",
- "editable": true,
- "gnetId": null,
- "graphTooltip": 0,
- "id": null,
- "links": [],
- "panels": [
- {
- "cacheTimeout": null,
- "colorBackground": false,
- "colorValue": false,
- "colors": [
- "#299c46",
- "rgba(237, 129, 40, 0.89)",
- "#d44a3a"
- ],
- "datasource": "${DS_BEAMPSQL}",
- "decimals": null,
- "format": "dthms",
- "gauge": {
- "maxValue": 100,
- "minValue": 0,
- "show": false,
- "thresholdLabels": false,
- "thresholdMarkers": true
- },
- "gridPos": {
- "h": 2,
- "w": 3,
- "x": 0,
- "y": 0
- },
- "id": 6,
- "interval": null,
- "links": [
- {
- "targetBlank": true,
- "title": "Link to oldest open PRs",
- "type": "absolute",
- "url": "https://github.com/apache/beam/pulls?q=is%3Apr+is%3Aopen+sort%3Acreated-asc"
- }
- ],
- "mappingType": 1,
- "mappingTypes": [
- {
- "name": "value to text",
- "value": 1
- },
- {
- "name": "range to text",
- "value": 2
- }
- ],
- "maxDataPoints": 100,
- "nullPointMode": "connected",
- "nullText": null,
- "postfix": "",
- "postfixFontSize": "50%",
- "prefix": "",
- "prefixFontSize": "50%",
- "rangeMaps": [
- {
- "from": "null",
- "text": "N/A",
- "to": "null"
- }
- ],
- "sparkline": {
- "fillColor": "rgba(31, 118, 189, 0.18)",
- "full": false,
- "lineColor": "rgb(31, 120, 193)",
- "show": false
- },
- "tableColumn": "maxprage",
- "targets": [
- {
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT (now() - min(created_ts)) as MaxPrAge\nfrom gh_pull_requests\nwhere closed_ts is NULL",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": "",
- "title": "Max Age of Open PR",
- "type": "singlestat",
- "valueFontSize": "80%",
- "valueMaps": [
- {
- "op": "=",
- "text": "N/A",
- "value": "null"
- }
- ],
- "valueName": "avg"
- },
- {
- "cacheTimeout": null,
- "colorBackground": false,
- "colorValue": false,
- "colors": [
- "#299c46",
- "rgba(237, 129, 40, 0.89)",
- "#d44a3a"
- ],
- "datasource": "${DS_BEAMPSQL}",
- "format": "none",
- "gauge": {
- "maxValue": 100,
- "minValue": 0,
- "show": false,
- "thresholdLabels": false,
- "thresholdMarkers": true
- },
- "gridPos": {
- "h": 2,
- "w": 3,
- "x": 3,
- "y": 0
- },
- "id": 13,
- "interval": null,
- "links": [
- {
- "targetBlank": true,
- "title": "Link to oldest open PRs",
- "type": "absolute",
- "url": "https://github.com/apache/beam/pulls?q=is%3Apr+is%3Aopen+sort%3Acreated-asc"
- }
- ],
- "mappingType": 1,
- "mappingTypes": [
- {
- "name": "value to text",
- "value": 1
- },
- {
- "name": "range to text",
- "value": 2
- }
- ],
- "maxDataPoints": 100,
- "nullPointMode": "connected",
- "nullText": null,
- "postfix": "",
- "postfixFontSize": "50%",
- "prefix": "",
- "prefixFontSize": "50%",
- "rangeMaps": [
- {
- "from": "null",
- "text": "N/A",
- "to": "null"
- }
- ],
- "sparkline": {
- "fillColor": "rgba(31, 118, 189, 0.18)",
- "full": false,
- "lineColor": "rgb(31, 120, 193)",
- "show": false
- },
- "tableColumn": "pr_id",
- "targets": [
- {
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT pr_id\nfrom gh_pull_requests\nwhere closed_ts is NULL and created_ts = (select min(created_ts) from gh_pull_requests where closed_ts is NULL)\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": "",
- "title": "Oldest Open PR",
- "transparent": false,
- "type": "singlestat",
- "valueFontSize": "80%",
- "valueMaps": [
- {
- "op": "=",
- "text": "N/A",
- "value": "null"
- }
- ],
- "valueName": "avg"
- },
- {
- "cacheTimeout": null,
- "colorBackground": false,
- "colorValue": false,
- "colors": [
- "#299c46",
- "rgba(237, 129, 40, 0.89)",
- "#d44a3a"
- ],
- "datasource": "${DS_BEAMPSQL}",
- "format": "none",
- "gauge": {
- "maxValue": 100,
- "minValue": 0,
- "show": false,
- "thresholdLabels": false,
- "thresholdMarkers": true
- },
- "gridPos": {
- "h": 2,
- "w": 3,
- "x": 6,
- "y": 0
- },
- "id": 10,
- "interval": null,
- "links": [
- {
- "targetBlank": true,
- "title": "Least updated CLs",
- "type": "absolute",
- "url": "https://github.com/apache/beam/pulls?q=is%3Aopen+is%3Apr+sort%3Aupdated-asc"
- }
- ],
- "mappingType": 1,
- "mappingTypes": [
- {
- "name": "value to text",
- "value": 1
- },
- {
- "name": "range to text",
- "value": 2
- }
- ],
- "maxDataPoints": 100,
- "nullPointMode": "connected",
- "nullText": null,
- "postfix": "",
- "postfixFontSize": "50%",
- "prefix": "",
- "prefixFontSize": "50%",
- "rangeMaps": [
- {
- "from": "null",
- "text": "N/A",
- "to": "null"
- }
- ],
- "sparkline": {
- "fillColor": "rgba(31, 118, 189, 0.18)",
- "full": false,
- "lineColor": "rgb(31, 120, 193)",
- "show": false
- },
- "tableColumn": "pr_id",
- "targets": [
- {
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with d1 as (SELECT (now() - min(updated_ts)) as MaxPrAge, min(updated_ts) as min_updated_ts\n from gh_pull_requests\n where closed_ts is NULL)\nselect pr_id\nfrom gh_pull_requests, d1\nwhere updated_ts = min_updated_ts",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": "",
- "title": "Open PR with Oldest Activity",
- "type": "singlestat",
- "valueFontSize": "80%",
- "valueMaps": [
- {
- "op": "=",
- "text": "N/A",
- "value": "null"
- }
- ],
- "valueName": "avg"
- },
- {
- "cacheTimeout": null,
- "colorBackground": false,
- "colorValue": false,
- "colors": [
- "#299c46",
- "rgba(237, 129, 40, 0.89)",
- "#d44a3a"
- ],
- "datasource": "${DS_BEAMPSQL}",
- "format": "none",
- "gauge": {
- "maxValue": 100,
- "minValue": 0,
- "show": false,
- "thresholdLabels": false,
- "thresholdMarkers": true
- },
- "gridPos": {
- "h": 2,
- "w": 3,
- "x": 9,
- "y": 0
- },
- "id": 12,
- "interval": null,
- "links": [
- {
- "title": "Least updated PRs",
- "type": "absolute",
- "url": "https://github.com/apache/beam/pulls?q=is%3Aopen+is%3Apr+sort%3Aupdated-asc"
- }
- ],
- "mappingType": 1,
- "mappingTypes": [
- {
- "name": "value to text",
- "value": 1
- },
- {
- "name": "range to text",
- "value": 2
- }
- ],
- "maxDataPoints": 100,
- "nullPointMode": "connected",
- "nullText": null,
- "postfix": "",
- "postfixFontSize": "50%",
- "prefix": "",
- "prefixFontSize": "50%",
- "rangeMaps": [
- {
- "from": "null",
- "text": "N/A",
- "to": "null"
- }
- ],
- "sparkline": {
- "fillColor": "rgba(31, 118, 189, 0.18)",
- "full": false,
- "lineColor": "rgb(31, 120, 193)",
- "show": false
- },
- "tableColumn": "maxpractivityage",
- "targets": [
- {
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT (now() - min(updated_ts)) as MaxPrActivityAge\n from gh_pull_requests\n where closed_ts is NULL",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": "",
- "title": "Age of Oldest Activity on Open PR",
- "type": "singlestat",
- "valueFontSize": "80%",
- "valueMaps": [
- {
- "op": "=",
- "text": "N/A",
- "value": "null"
- }
- ],
- "valueName": "avg"
- },
- {
- "columns": [],
- "datasource": "${DS_BEAMPSQL}",
- "description": "Shows open PRs and list of reviewers assigned to PR.\n\nReviewers are calculated in two ways:\n1. beam_reviewers - this is initial algorithm that detects defined patterns in comments.\n2. updated_beam_reviewers_algorithm - this algo lists all mentions (@.*), as well as whoever was assigned as reviewer, reviewed or merged code.",
- "fontSize": "100%",
- "gridPos": {
- "h": 14,
- "w": 12,
- "x": 12,
- "y": 0
- },
- "id": 8,
- "links": [],
- "pageSize": null,
- "scroll": true,
- "showHeader": true,
- "sort": {
- "col": null,
- "desc": false
- },
- "styles": [
- {
- "alias": "Time",
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "pattern": "Time",
- "type": "date"
- },
- {
- "alias": "",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": null,
- "link": true,
- "linkTargetBlank": true,
- "linkTooltip": "PR",
- "linkUrl": "https://github.com/apache/beam/pull/${__cell:raw}",
- "mappingType": 1,
- "pattern": "pr_id",
- "thresholds": [],
- "type": "number",
- "unit": "none"
- },
- {
- "alias": "",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 2,
- "mappingType": 1,
- "pattern": "beam_reviewers",
- "thresholds": [],
- "type": "number",
- "unit": "short"
- },
- {
- "alias": "",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "decimals": 0,
- "pattern": "/.*/",
- "thresholds": [],
- "type": "number",
- "unit": "none"
- }
- ],
- "targets": [
- {
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with combined_reviewers as (select pr_id, array_agg(DISTINCT unnested) as unique_reviewers\n from gh_pull_requests src, unnest(mentioned || requested_reviewers || reviewed_by) as unnested\n group by pr_id)\n\nSELECT gh_pull_requests.pr_id, author, (now() - updated_ts) as last_activity, beam_reviewers, combined_reviewers.unique_reviewers as updated_beam_reviewers_algorithm\nFROM gh_pull_requests left outer join combined_reviewers ON gh_pull_requests.pr_id = combined_reviewers.pr_id\nWHERE gh_pull_requests.closed_ts is null\norder by last_activity desc",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "title": "Currently Open PRs",
- "transform": "table",
- "type": "table"
- },
- {
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "description": "Shows total PRs opened and statistics of open/closed PRs per day.",
- "fill": 1,
- "gridPos": {
- "h": 6,
- "w": 12,
- "x": 0,
- "y": 2
- },
- "id": 15,
- "legend": {
- "avg": false,
- "current": false,
- "max": false,
- "min": false,
- "show": true,
- "total": false,
- "values": false
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day, count(*) as total_open\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.created_ts < days.day AND (gh_pull_requests.closed_ts > days.day OR gh_pull_requests.closed_ts is null)\n GROUP BY days.day\n ORDER BY days.day)\nselect days.day as time, greatest(knowndays.total_open, 0) as total_open\nfrom days left outer join knowndays\non days.day = knowndays.day",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- },
- {
- "format": "time_series",
- "group": [],
- "hide": false,
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day, 0 as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day as day, count(*) as ClosedTickets\n FROM gh_pull_requests, days\n WHERE date_trunc('day', gh_pull_requests.closed_ts) = days.day\n GROUP BY day\n ORDER BY day\n )\nselect days.day as time, greatest(knowndays.ClosedTickets, days.zcnt) as closed_count\nfrom days left outer join knowndays\non days.day = knowndays.day\n",
- "refId": "B",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- },
- {
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day, 0 as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day as day, count(*) as ClosedTickets\n FROM gh_pull_requests, days\n WHERE date_trunc('day', gh_pull_requests.created_ts) = days.day\n GROUP BY day\n ORDER BY day\n )\nselect days.day as time, greatest(knowndays.ClosedTickets, days.zcnt) as craeted_count\nfrom days left outer join knowndays\non days.day = knowndays.day\n",
- "refId": "C",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [],
- "timeFrom": null,
- "timeShift": null,
- "title": "PR Activity Trends",
- "tooltip": {
- "shared": true,
- "sort": 0,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- },
- {
- "alert": {
- "conditions": [
- {
- "evaluator": {
- "params": [
- 72
- ],
- "type": "gt"
- },
- "operator": {
- "type": "and"
- },
- "query": {
- "params": [
- "A",
- "5m",
- "now"
- ]
- },
- "reducer": {
- "params": [],
- "type": "last"
- },
- "type": "query"
- }
- ],
- "executionErrorState": "alerting",
- "frequency": "1h",
- "handler": 1,
- "name": "Time to First Non-Author PR Response on 7 Day Span alert",
- "noDataState": "no_data",
- "notifications": []
- },
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "description": "Shows first non-author activity on PR on sliding window of 7 days.",
- "fill": 0,
- "gridPos": {
- "h": 6,
- "w": 12,
- "x": 0,
- "y": 8
- },
- "id": 17,
- "legend": {
- "avg": false,
- "current": false,
- "max": false,
- "min": false,
- "show": true,
- "total": false,
- "values": false
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day, '0 day'::interval as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n responseTimes as (SELECT (gh_pull_requests.first_non_author_activity_ts - gh_pull_requests.created_ts) as responseTime, days.day as day\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.first_non_author_activity_ts < days.day AND gh_pull_requests.first_non_author_activity_ts > (days.day - '7 day'::interval)\n ),\n percTimes as (SELECT percentile_disc(0.85) WITHIN GROUP (ORDER BY responseTime) as eightyFifthResponseTime, day\n FROM responseTimes\n group by day),\n avgTimes as (SELECT avg(responseTime) as avgResponseTime, day\n FROM responseTimes\n group by day)\nselect days.day as time, EXTRACT(EPOCH FROM percTimes.eightyFifthResponseTime)/3600 as \"85th Percentile\", EXTRACT(EPOCH FROM avgTimes.avgResponseTime)/3600 as \"Mean\"\nfrom days left outer join percTimes on days.day = percTimes.day\n left outer join avgTimes on days.day = avgTimes.day\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [
- {
- "colorMode": "critical",
- "fill": true,
- "line": true,
- "op": "gt",
- "value": 72
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Time to First Non-Author PR Response on 7 Day Span",
- "tooltip": {
- "shared": true,
- "sort": 0,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": null,
- "format": "h",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- },
- {
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "decimals": 0,
- "description": "Assignees with less than 5 PRs are not shown.",
- "fill": 0,
- "gridPos": {
- "h": 6,
- "w": 12,
- "x": 0,
- "y": 14
- },
- "id": 22,
- "legend": {
- "alignAsTable": true,
- "avg": false,
- "current": true,
- "hideEmpty": true,
- "hideZero": true,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": true,
- "total": false,
- "values": true
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null as zero",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day, '0 day'::interval as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n openPRs as (SELECT *\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.created_ts < days.day AND (gh_pull_requests.closed_ts > days.day OR gh_pull_requests.closed_ts is null)),\n flattennedPRs as (SELECT * \n FROM openPRs, unnest(openPRs.beam_reviewers) flattenedMentioned)\nselect day as time, count(*) as PRsAssigned, flattenedMentioned\nfrom flattennedPRs\ngroup by day, flattenedMentioned\nHAVING count(*) > 4\norder by day, flattenedMentioned\n\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [],
- "timeFrom": null,
- "timeShift": null,
- "title": "Reviews per Beam Reviewer",
- "tooltip": {
- "shared": true,
- "sort": 2,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": null,
- "format": "none",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- },
- {
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "decimals": 0,
- "description": "Assignees with less than 5 PRs are not shown.",
- "fill": 0,
- "gridPos": {
- "h": 6,
- "w": 12,
- "x": 12,
- "y": 14
- },
- "id": 18,
- "legend": {
- "alignAsTable": true,
- "avg": false,
- "current": true,
- "hideEmpty": false,
- "hideZero": false,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": true,
- "total": false,
- "values": true
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day, '0 day'::interval as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n openPRs as (SELECT *\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.created_ts < days.day AND (gh_pull_requests.closed_ts > days.day OR gh_pull_requests.closed_ts is null)),\n flattennedPRs as (SELECT * \n FROM openPRs, unnest(openPRs.mentioned) flattenedMentioned)\nselect day as time, count(*) as PRsAssigned, flattenedMentioned\nfrom flattennedPRs\ngroup by day, flattenedMentioned\nHAVING count(*) > 4\norder by day, flattenedMentioned\n\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [],
- "timeFrom": null,
- "timeShift": null,
- "title": "Reviews per Mentioned and Assigned",
- "tooltip": {
- "shared": true,
- "sort": 2,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": null,
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": true
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- }
- ],
- "refresh": false,
- "schemaVersion": 16,
- "style": "dark",
- "tags": [],
- "templating": {
- "list": []
- },
- "time": {
- "from": "now-90d",
- "to": "now"
- },
- "timepicker": {
- "refresh_intervals": [
- "5s",
- "10s",
- "30s",
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "1d"
- ],
- "time_options": [
- "5m",
- "15m",
- "1h",
- "6h",
- "12h",
- "24h",
- "2d",
- "7d",
- "30d"
- ]
- },
- "timezone": "",
- "title": "Code Velocity",
- "uid": "code_velocity",
- "version": 7
-}
\ No newline at end of file
diff --git a/.test-infra/metrics/dashboards/post-commit_tests.json b/.test-infra/metrics/dashboards/post-commit_tests.json
deleted file mode 100644
index 219cace..0000000
--- a/.test-infra/metrics/dashboards/post-commit_tests.json
+++ /dev/null
@@ -1,836 +0,0 @@
-{
- "__inputs": [
- {
- "name": "DS_BEAMPSQL",
- "label": "BeamPSQL",
- "description": "",
- "type": "datasource",
- "pluginId": "postgres",
- "pluginName": "PostgreSQL"
- }
- ],
- "__requires": [
- {
- "type": "grafana",
- "id": "grafana",
- "name": "Grafana",
- "version": "5.3.1"
- },
- {
- "type": "panel",
- "id": "graph",
- "name": "Graph",
- "version": "5.0.0"
- },
- {
- "type": "datasource",
- "id": "postgres",
- "name": "PostgreSQL",
- "version": "5.0.0"
- },
- {
- "type": "panel",
- "id": "table",
- "name": "Table",
- "version": "5.0.0"
- },
- {
- "type": "panel",
- "id": "text",
- "name": "Text",
- "version": "5.0.0"
- }
- ],
- "annotations": {
- "list": [
- {
- "builtIn": 1,
- "datasource": "-- Grafana --",
- "enable": true,
- "hide": true,
- "iconColor": "rgba(0, 211, 255, 1)",
- "limit": 100,
- "name": "Annotations & Alerts",
- "showIn": 0,
- "type": "dashboard"
- }
- ]
- },
- "editable": true,
- "gnetId": null,
- "graphTooltip": 0,
- "id": null,
- "links": [],
- "panels": [
- {
- "content": "This dashboard tracks Post-commit test reliability over-time.\n\n* [Post-commit test policies](https://beam.apache.org/contribute/postcommits-policies/)\n* [Existing test failure issues](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)\n* [File a new test failure issue](https://s.apache.org/beam-test-failure)",
- "gridPos": {
- "h": 4,
- "w": 24,
- "x": 0,
- "y": 0
- },
- "id": 11,
- "links": [],
- "mode": "markdown",
- "title": "Dashboard guidelines",
- "transparent": false,
- "type": "text"
- },
- {
- "alert": {
- "conditions": [
- {
- "evaluator": {
- "params": [
- 0.7
- ],
- "type": "lt"
- },
- "operator": {
- "type": "and"
- },
- "query": {
- "params": [
- "A",
- "5m",
- "now"
- ]
- },
- "reducer": {
- "params": [],
- "type": "min"
- },
- "type": "query"
- }
- ],
- "executionErrorState": "alerting",
- "frequency": "30m",
- "handler": 1,
- "name": "Post-commit reliability per week alert",
- "noDataState": "no_data",
- "notifications": []
- },
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "decimals": 0,
- "description": "Percent reliability of all post-commit job runs for a given week.\n\nUnreliability of a test suite impact developer productivity by forcing contributors to re-run tests. When tests are consistently unreliable, developers will simply ignore them.\n\nWe aim for >= 70% reliability per test suite.",
- "fill": 0,
- "gridPos": {
- "h": 7,
- "w": 24,
- "x": 0,
- "y": 4
- },
- "id": 6,
- "legend": {
- "alignAsTable": true,
- "avg": false,
- "current": true,
- "hideEmpty": false,
- "hideZero": true,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": true,
- "total": false,
- "values": true
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 2,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "alias": "",
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT\n DATE_TRUNC('week', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [
- {
- "colorMode": "critical",
- "fill": true,
- "line": true,
- "op": "lt",
- "value": 0.7
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Post-commit reliability per week",
- "tooltip": {
- "shared": true,
- "sort": 1,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": 1,
- "format": "percentunit",
- "label": "% successful runs",
- "logBase": 1,
- "max": "1",
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- },
- {
- "alert": {
- "conditions": [
- {
- "evaluator": {
- "params": [
- 0.7
- ],
- "type": "lt"
- },
- "operator": {
- "type": "and"
- },
- "query": {
- "params": [
- "A",
- "5m",
- "now"
- ]
- },
- "reducer": {
- "params": [],
- "type": "min"
- },
- "type": "query"
- }
- ],
- "executionErrorState": "alerting",
- "frequency": "30m",
- "handler": 1,
- "name": "Post-commit reliability per day alert",
- "noDataState": "no_data",
- "notifications": []
- },
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "decimals": 0,
- "description": "Percent reliability of all post-commit job runs per-day.\n\nUnreliability of a test suite impact developer productivity by forcing contributors to re-run tests. When tests are consistently unreliable, developers will simply ignore them.\n\nWe aim for >= 70% reliability per test suite.",
- "fill": 0,
- "gridPos": {
- "h": 12,
- "w": 15,
- "x": 0,
- "y": 11
- },
- "id": 9,
- "legend": {
- "alignAsTable": true,
- "avg": false,
- "current": true,
- "hideZero": true,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": false,
- "sideWidth": null,
- "total": false,
- "values": true
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 2,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "alias": "",
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT\n DATE_TRUNC('day', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [
- {
- "colorMode": "critical",
- "fill": true,
- "line": true,
- "op": "lt",
- "value": 0.7
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Post-commit reliability per day",
- "tooltip": {
- "shared": true,
- "sort": 1,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": 1,
- "format": "percentunit",
- "label": "% successful runs",
- "logBase": 1,
- "max": "1",
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- },
- {
- "columns": [],
- "datasource": "${DS_BEAMPSQL}",
- "description": "List of jobs which have failed. Click on the job to view it in Jenkins.",
- "fontSize": "100%",
- "gridPos": {
- "h": 12,
- "w": 9,
- "x": 15,
- "y": 11
- },
- "hideTimeOverride": false,
- "id": 8,
- "links": [
- {
- "includeVars": false,
- "targetBlank": true,
- "title": "Beam Jenkins",
- "type": "absolute",
- "url": "https://builds.apache.org/view/A-D/view/Beam/"
- }
- ],
- "pageSize": null,
- "scroll": true,
- "showHeader": true,
- "sort": {
- "col": 0,
- "desc": true
- },
- "styles": [
- {
- "alias": "Time",
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "link": false,
- "pattern": "Time",
- "type": "date"
- },
- {
- "alias": "Build Url",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 2,
- "link": true,
- "linkTargetBlank": true,
- "linkTooltip": "Link to Jenkins job.",
- "linkUrl": "${__cell:raw}",
- "mappingType": 1,
- "pattern": "build_url",
- "thresholds": [],
- "type": "hidden",
- "unit": "short"
- },
- {
- "alias": "Job Name",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 2,
- "link": true,
- "linkTargetBlank": true,
- "linkTooltip": "View Jenkins job: ${__cell_1}_${__cell_2}",
- "linkUrl": "${__cell_0:raw}",
- "mappingType": 1,
- "pattern": "job_name",
- "thresholds": [],
- "type": "string",
- "unit": "short"
- },
- {
- "alias": "Build ID",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 0,
- "link": true,
- "linkTargetBlank": true,
- "linkTooltip": "View Jenkins job: ${__cell_1}_${__cell_2}",
- "linkUrl": "${__cell_0:raw}",
- "mappingType": 1,
- "pattern": "build_id",
- "thresholds": [],
- "type": "number",
- "unit": "short"
- },
- {
- "alias": "Start Time",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "MM/DD/YY h:mm:ss a",
- "decimals": 2,
- "link": true,
- "linkTargetBlank": true,
- "linkTooltip": "View Jenkins job: ${__cell_1}_${__cell_2}",
- "linkUrl": "${__cell_0:raw}",
- "mappingType": 1,
- "pattern": "build_timestamp",
- "thresholds": [],
- "type": "date",
- "unit": "short"
- }
- ],
- "targets": [
- {
- "alias": "",
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT \n build_url,\n job_name,\n build_id,\n build_timestamp\nFROM jenkins_builds\nWHERE \n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name LIKE '%_PR')\n AND NOT (build_result = 'SUCCESS')\nORDER BY \n build_timestamp",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "timeShift": null,
- "title": "Failed builds",
- "transform": "table",
- "type": "table"
- },
- {
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "decimals": 1,
- "description": "Execution time for each post-commit job",
- "fill": 0,
- "gridPos": {
- "h": 8,
- "w": 15,
- "x": 0,
- "y": 23
- },
- "id": 5,
- "legend": {
- "alignAsTable": true,
- "avg": false,
- "current": true,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": false,
- "total": false,
- "values": true
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "alias": "",
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT\n build_timestamp as time,\n build_duration as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as metric\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name LIKE '%_PR')\nORDER BY\n job_name, time",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [],
- "timeFrom": null,
- "timeShift": null,
- "title": "Post-commit job duration",
- "tooltip": {
- "shared": true,
- "sort": 2,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": null,
- "format": "ms",
- "label": "Average job duration",
- "logBase": 1,
- "max": null,
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- },
- {
- "aliasColors": {},
- "bars": true,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "decimals": 0,
- "description": "Tracks the count of test failure JIRA issues currently open.",
- "fill": 3,
- "gridPos": {
- "h": 8,
- "w": 9,
- "x": 15,
- "y": 23
- },
- "id": 14,
- "legend": {
- "alignAsTable": false,
- "avg": false,
- "current": false,
- "max": false,
- "min": false,
- "rightSide": false,
- "show": true,
- "total": false,
- "values": false
- },
- "lines": false,
- "linewidth": 1,
- "links": [
- {
- "targetBlank": true,
- "title": "Jira tickets",
- "type": "absolute",
- "url": "https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC"
- }
- ],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [
- {
- "alias": "total_open",
- "color": "#eab839"
- },
- {
- "alias": "currently_failing",
- "color": "#bf1b00"
- }
- ],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "format": "time_series",
- "group": [],
- "hide": false,
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day, count(*) as total_open\n FROM jira_issues, days\n WHERE jira_issues.created < days.day AND (jira_issues.resolutiondate > days.day OR jira_issues.resolutiondate is null)\n GROUP BY days.day\n ORDER BY days.day)\nselect days.day as time, greatest(knowndays.total_open, 0) as total_open\nfrom days left outer join knowndays\non days.day = knowndays.day",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- },
- {
- "format": "time_series",
- "group": [],
- "hide": false,
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "with days as (select date_trunc('day', dd) as day from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day, count(*) as currently_failing\n FROM jira_issues, days\n WHERE jira_issues.created < days.day AND (jira_issues.resolutiondate > days.day OR jira_issues.resolutiondate is null) AND (jira_issues.labels LIKE '%currently-failing%')\n GROUP BY days.day\n ORDER BY days.day)\nselect days.day as time, greatest(knowndays.currently_failing, 0) as currently_failing\nfrom days left outer join knowndays\non days.day = knowndays.day",
- "refId": "D",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [],
- "timeFrom": null,
- "timeShift": null,
- "title": "Test Failure JIRA issues",
- "tooltip": {
- "shared": true,
- "sort": 0,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": 0,
- "format": "short",
- "label": "# of JIRA issues",
- "logBase": 1,
- "max": null,
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- }
- ],
- "refresh": false,
- "schemaVersion": 16,
- "style": "dark",
- "tags": [],
- "templating": {
- "list": []
- },
- "time": {
- "from": "now-30d",
- "to": "now"
- },
- "timepicker": {
- "hidden": false,
- "refresh_intervals": [
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "1d"
- ],
- "time_options": [
- "5m",
- "15m",
- "1h",
- "6h",
- "12h",
- "24h",
- "2d",
- "7d",
- "30d"
- ]
- },
- "timezone": "",
- "title": "Post-commit Test Reliability",
- "uid": "D81lW0pmk",
- "version": 45
-}
diff --git a/.test-infra/metrics/dashboards/pre-commit_tests.json b/.test-infra/metrics/dashboards/pre-commit_tests.json
deleted file mode 100644
index d24253c..0000000
--- a/.test-infra/metrics/dashboards/pre-commit_tests.json
+++ /dev/null
@@ -1,241 +0,0 @@
-{
- "__inputs": [
- {
- "name": "DS_BEAMPSQL",
- "label": "BeamPSQL",
- "description": "",
- "type": "datasource",
- "pluginId": "postgres",
- "pluginName": "PostgreSQL"
- }
- ],
- "__requires": [
- {
- "type": "grafana",
- "id": "grafana",
- "name": "Grafana",
- "version": "5.3.1"
- },
- {
- "type": "panel",
- "id": "graph",
- "name": "Graph",
- "version": "5.0.0"
- },
- {
- "type": "datasource",
- "id": "postgres",
- "name": "PostgreSQL",
- "version": "5.0.0"
- }
- ],
- "annotations": {
- "list": [
- {
- "builtIn": 1,
- "datasource": "-- Grafana --",
- "enable": true,
- "hide": true,
- "iconColor": "rgba(0, 211, 255, 1)",
- "name": "Annotations & Alerts",
- "type": "dashboard"
- }
- ]
- },
- "editable": true,
- "gnetId": null,
- "graphTooltip": 0,
- "id": null,
- "links": [],
- "panels": [
- {
- "alert": {
- "conditions": [
- {
- "evaluator": {
- "params": [
- 7200000
- ],
- "type": "gt"
- },
- "operator": {
- "type": "and"
- },
- "query": {
- "params": [
- "A",
- "5m",
- "now"
- ]
- },
- "reducer": {
- "params": [],
- "type": "max"
- },
- "type": "query"
- }
- ],
- "executionErrorState": "alerting",
- "frequency": "30m",
- "handler": 1,
- "name": "Pre-commit job duration per-day alert",
- "noDataState": "no_data",
- "notifications": []
- },
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "description": "Execution time for each pre-commit job.\n\nLong test suite execution impacts developer productivity by delaying the quality signal of a pull request of current HEAD. If tests are consistently slow, developers won't wait for them to complete.\n\nWe aim for under 2 hour execution per test suite, but ideally under 30 mins.",
- "fill": 0,
- "gridPos": {
- "h": 8,
- "w": 24,
- "x": 0,
- "y": 0
- },
- "id": 4,
- "legend": {
- "alignAsTable": true,
- "avg": false,
- "current": true,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": true,
- "sort": "current",
- "sortDesc": true,
- "total": false,
- "values": true
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "alias": "",
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT\n build_timestamp as time,\n build_duration as value,\n substring(job_name from 'beam_PreCommit_#\"%#\"_(Cron|Commit)' for '#') as metric\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND build_result = 'SUCCESS'\n AND ((job_name LIKE 'beam_PreCommit_%_Commit')\n OR (job_name LIKE 'beam_PreCommit_%_Cron'))\nORDER BY\n metric, time",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [
- {
- "colorMode": "critical",
- "fill": true,
- "line": true,
- "op": "gt",
- "value": 7200000
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Pre-commit job duration",
- "tooltip": {
- "shared": true,
- "sort": 0,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "format": "ms",
- "label": "Average job duration",
- "logBase": 1,
- "max": null,
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- }
- ],
- "schemaVersion": 16,
- "style": "dark",
- "tags": [],
- "templating": {
- "list": []
- },
- "time": {
- "from": "now-7d",
- "to": "now"
- },
- "timepicker": {
- "refresh_intervals": [
- "5s",
- "10s",
- "30s",
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "1d"
- ],
- "time_options": [
- "5m",
- "15m",
- "1h",
- "6h",
- "12h",
- "24h",
- "2d",
- "7d",
- "30d"
- ]
- },
- "timezone": "",
- "title": "Pre-commit Test Latency",
- "uid": "_TNndF2iz",
- "version": 13
-}
diff --git a/.test-infra/metrics/dashboards/source_data_freshness.json b/.test-infra/metrics/dashboards/source_data_freshness.json
deleted file mode 100644
index 4383a5d..0000000
--- a/.test-infra/metrics/dashboards/source_data_freshness.json
+++ /dev/null
@@ -1,258 +0,0 @@
-{
- "__inputs": [
- {
- "name": "DS_BEAMPSQL",
- "label": "BeamPSQL",
- "description": "",
- "type": "datasource",
- "pluginId": "postgres",
- "pluginName": "PostgreSQL"
- }
- ],
- "__requires": [
- {
- "type": "grafana",
- "id": "grafana",
- "name": "Grafana",
- "version": "5.3.2"
- },
- {
- "type": "panel",
- "id": "graph",
- "name": "Graph",
- "version": "5.0.0"
- },
- {
- "type": "datasource",
- "id": "postgres",
- "name": "PostgreSQL",
- "version": "5.0.0"
- }
- ],
- "annotations": {
- "list": [
- {
- "builtIn": 1,
- "datasource": "-- Grafana --",
- "enable": true,
- "hide": true,
- "iconColor": "rgba(0, 211, 255, 1)",
- "name": "Annotations & Alerts",
- "type": "dashboard"
- }
- ]
- },
- "editable": true,
- "gnetId": null,
- "graphTooltip": 0,
- "id": null,
- "links": [],
- "panels": [
- {
- "alert": {
- "conditions": [
- {
- "evaluator": {
- "params": [
- 480
- ],
- "type": "gt"
- },
- "operator": {
- "type": "and"
- },
- "query": {
- "params": [
- "A",
- "5m",
- "now"
- ]
- },
- "reducer": {
- "params": [],
- "type": "max"
- },
- "type": "query"
- }
- ],
- "executionErrorState": "alerting",
- "frequency": "5m",
- "handler": 1,
- "name": "Source Data Freshness alert",
- "noDataState": "alerting",
- "notifications": []
- },
- "aliasColors": {},
- "bars": true,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "description": "Data freshness for each metrics input source. Used for health monitoring for other dashboards.",
- "fill": 1,
- "gridPos": {
- "h": 12,
- "w": 6,
- "x": 0,
- "y": 0
- },
- "id": 2,
- "legend": {
- "alignAsTable": false,
- "avg": false,
- "current": true,
- "max": false,
- "min": false,
- "rightSide": false,
- "show": true,
- "total": false,
- "values": true
- },
- "lines": false,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 5,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "format": "time_series",
- "group": [
- {
- "params": [
- "$__interval",
- "none"
- ],
- "type": "time"
- }
- ],
- "hide": false,
- "metricColumn": "build_result",
- "rawQuery": true,
- "rawSql": "WITH sources AS (\n SELECT\n 'Jenkins' as source,\n MAX(build_timestamp + make_interval(secs:= timing_executingtimemillis / 1000.0)) AS last_sync\n FROM jenkins_builds\n UNION SELECT\n 'JIRA' as source,\n lastsynctime AS last_sync\n FROM jira_issues_metadata\n UNION SELECT\n 'GitHub' as source,\n timestamp AS last_sync\n FROM gh_sync_metadata\n)\nSELECT\n current_timestamp AS time,\n source,\n EXTRACT(EPOCH FROM age(current_timestamp, last_sync)) / (60) AS value\nFROM sources",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "build_duration"
- ],
- "type": "column"
- },
- {
- "params": [
- "max"
- ],
- "type": "aggregate"
- },
- {
- "params": [
- "build_duration"
- ],
- "type": "alias"
- }
- ]
- ],
- "table": "jenkins_builds",
- "timeColumn": "build_timestamp",
- "timeColumnType": "timestamp",
- "where": []
- }
- ],
- "thresholds": [
- {
- "colorMode": "critical",
- "fill": true,
- "line": true,
- "op": "gt",
- "value": 480
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Source Data Freshness",
- "tooltip": {
- "shared": false,
- "sort": 1,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "series",
- "name": null,
- "show": false,
- "values": [
- "current"
- ]
- },
- "yaxes": [
- {
- "decimals": 0,
- "format": "m",
- "label": "Staleness",
- "logBase": 1,
- "max": "60",
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- }
- ],
- "schemaVersion": 16,
- "style": "dark",
- "tags": [],
- "templating": {
- "list": []
- },
- "time": {
- "from": "now-30d",
- "to": "now"
- },
- "timepicker": {
- "refresh_intervals": [
- "5s",
- "10s",
- "30s",
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "1d"
- ],
- "time_options": [
- "5m",
- "15m",
- "1h",
- "6h",
- "12h",
- "24h",
- "2d",
- "7d",
- "30d"
- ]
- },
- "timezone": "",
- "title": "Source Data Freshness",
- "uid": "data-freshness",
- "version": 8
-}
diff --git a/.test-infra/metrics/dashboards/stability_critical_jobs_status.json b/.test-infra/metrics/dashboards/stability_critical_jobs_status.json
deleted file mode 100644
index f784ae4..0000000
--- a/.test-infra/metrics/dashboards/stability_critical_jobs_status.json
+++ /dev/null
@@ -1,332 +0,0 @@
-{
- "__inputs": [
- {
- "name": "DS_BEAMPSQL",
- "label": "BeamPSQL",
- "description": "",
- "type": "datasource",
- "pluginId": "postgres",
- "pluginName": "PostgreSQL"
- }
- ],
- "__requires": [
- {
- "type": "grafana",
- "id": "grafana",
- "name": "Grafana",
- "version": "5.3.2"
- },
- {
- "type": "panel",
- "id": "graph",
- "name": "Graph",
- "version": "5.0.0"
- },
- {
- "type": "datasource",
- "id": "postgres",
- "name": "PostgreSQL",
- "version": "5.0.0"
- },
- {
- "type": "panel",
- "id": "table",
- "name": "Table",
- "version": "5.0.0"
- },
- {
- "type": "panel",
- "id": "text",
- "name": "Text",
- "version": "5.0.0"
- }
- ],
- "annotations": {
- "list": [
- {
- "builtIn": 1,
- "datasource": "-- Grafana --",
- "enable": true,
- "hide": true,
- "iconColor": "rgba(0, 211, 255, 1)",
- "name": "Annotations & Alerts",
- "type": "dashboard"
- }
- ]
- },
- "editable": true,
- "gnetId": null,
- "graphTooltip": 0,
- "id": null,
- "links": [],
- "panels": [
- {
- "content": "The graph shows: average greenness of critical post-commit tests jobs per week. This graph show health of our project.\n\nTable shows list of relevant jobs failures during selected time interval (You can change time interval on top-right corner of the dashboard). Please, triage failed jobs and update or create corresponding jira tickets. You can utilized provided links to help with this.",
- "gridPos": {
- "h": 3,
- "w": 10,
- "x": 0,
- "y": 0
- },
- "id": 8,
- "links": [],
- "mode": "markdown",
- "title": "Dashboard guidelines",
- "transparent": false,
- "type": "text"
- },
- {
- "columns": [],
- "datasource": "${DS_BEAMPSQL}",
- "fontSize": "100%",
- "gridPos": {
- "h": 13,
- "w": 14,
- "x": 10,
- "y": 0
- },
- "hideTimeOverride": false,
- "id": 4,
- "links": [],
- "pageSize": null,
- "scroll": true,
- "showHeader": true,
- "sort": {
- "col": 0,
- "desc": true
- },
- "styles": [
- {
- "alias": "Time",
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "link": false,
- "pattern": "Time",
- "type": "date"
- },
- {
- "alias": "Build Url",
- "colorMode": null,
- "colors": [
- "rgba(245, 54, 54, 0.9)",
- "rgba(237, 129, 40, 0.89)",
- "rgba(50, 172, 45, 0.97)"
- ],
- "dateFormat": "YYYY-MM-DD HH:mm:ss",
- "decimals": 2,
- "link": true,
- "linkTargetBlank": true,
- "linkTooltip": "Link to Jenkins job.",
- "linkUrl": "${__cell:raw}",
- "mappingType": 1,
- "pattern": "build_url",
- "thresholds": [],
- "type": "number",
- "unit": "short"
- }
- ],
- "targets": [
- {
- "alias": "",
- "format": "table",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT \n build_timestamp,\n job_name,\n build_url\nFROM jenkins_builds\nWHERE \n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND ((job_name LIKE 'beam_PostCommit_Java') \n OR (job_name LIKE 'beam_PostCommit_Go') \n OR (job_name LIKE 'beam_PostCommit_Python_Verify')\n OR (job_name LIKE 'beam_PostCommit_Website_Publish'))\n AND NOT (job_name LIKE '%_PR')\n AND NOT (build_result = 'SUCCESS')\nORDER BY \n build_timestamp",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "timeShift": null,
- "title": "Failed builds",
- "transform": "table",
- "type": "table"
- },
- {
- "content": "[List existing jira tickets](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)\n\n[Create new Jira ticket](https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12319527&issuetype=1&summary=%5BjobName%5D%5BTestName%5D%5BIsFlake%5D%20Failure%20summary&priority=3&components=12334203&description=%3CFailure%20summary%3E%0AFailing%20job%20url:%0AJob%20history%20url:%0ARelevant%20log:)",
- "gridPos": {
- "h": 3,
- "w": 10,
- "x": 0,
- "y": 3
- },
- "id": 6,
- "links": [],
- "mode": "markdown",
- "title": "Useful links",
- "transparent": false,
- "type": "text"
- },
- {
- "aliasColors": {},
- "bars": false,
- "dashLength": 10,
- "dashes": false,
- "datasource": "${DS_BEAMPSQL}",
- "fill": 0,
- "gridPos": {
- "h": 7,
- "w": 10,
- "x": 0,
- "y": 6
- },
- "id": 2,
- "legend": {
- "avg": false,
- "current": false,
- "max": false,
- "min": false,
- "rightSide": true,
- "show": true,
- "total": false,
- "values": false
- },
- "lines": true,
- "linewidth": 1,
- "links": [],
- "nullPointMode": "null",
- "percentage": false,
- "pointradius": 2,
- "points": false,
- "renderer": "flot",
- "seriesOverrides": [],
- "spaceLength": 10,
- "stack": false,
- "steppedLine": false,
- "targets": [
- {
- "alias": "",
- "format": "time_series",
- "group": [],
- "metricColumn": "none",
- "rawQuery": true,
- "rawSql": "SELECT\n DATE_TRUNC('week', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND ((job_name LIKE 'beam_PostCommit_Java') \n OR (job_name LIKE 'beam_PostCommit_Go') \n OR (job_name LIKE 'beam_PostCommit_Python_Verify')\n OR (job_name LIKE 'beam_PostCommit_Website_Publish'))\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time\n",
- "refId": "A",
- "select": [
- [
- {
- "params": [
- "value"
- ],
- "type": "column"
- }
- ]
- ],
- "timeColumn": "time",
- "where": [
- {
- "name": "$__timeFilter",
- "params": [],
- "type": "macro"
- }
- ]
- }
- ],
- "thresholds": [
- {
- "colorMode": "custom",
- "fill": false,
- "line": true,
- "lineColor": "#3f6833",
- "op": "lt",
- "value": 0.7,
- "yaxis": "left"
- }
- ],
- "timeFrom": null,
- "timeShift": null,
- "title": "Greenness per Week (in %)",
- "tooltip": {
- "shared": true,
- "sort": 1,
- "value_type": "individual"
- },
- "type": "graph",
- "xaxis": {
- "buckets": null,
- "mode": "time",
- "name": null,
- "show": true,
- "values": []
- },
- "yaxes": [
- {
- "decimals": null,
- "format": "percentunit",
- "label": "",
- "logBase": 1,
- "max": "1",
- "min": "0",
- "show": true
- },
- {
- "format": "short",
- "label": null,
- "logBase": 1,
- "max": null,
- "min": null,
- "show": false
- }
- ],
- "yaxis": {
- "align": false,
- "alignLevel": null
- }
- }
- ],
- "schemaVersion": 16,
- "style": "dark",
- "tags": [],
- "templating": {
- "list": []
- },
- "time": {
- "from": "now-30d",
- "to": "now"
- },
- "timepicker": {
- "refresh_intervals": [
- "5s",
- "10s",
- "30s",
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "1d"
- ],
- "time_options": [
- "5m",
- "15m",
- "1h",
- "6h",
- "12h",
- "24h",
- "2d",
- "7d",
- "30d"
- ]
- },
- "timezone": "",
- "title": "Stability critical jobs status",
- "uid": "McTAiu0ik",
- "version": 10
-}
diff --git a/.test-infra/metrics/docker-compose.yml b/.test-infra/metrics/docker-compose.yml
index 546830a..3ec1954 100644
--- a/.test-infra/metrics/docker-compose.yml
+++ b/.test-infra/metrics/docker-compose.yml
@@ -25,12 +25,14 @@
container_name: beampostgresql
volumes:
- beam-postgresql-data:/var/lib/postgresql/data
+ - ./postgres:/docker-entrypoint-initdb.d
environment:
- POSTGRES_USER=admin
- POSTGRES_PASSWORD=<PGPasswordHere>
- POSTGRES_DB=beam_metrics
grafana:
- image: grafana/grafana
+ image: beamgrafana
+ build: ./grafana
ports:
- "3000:3000"
container_name: beamgrafana
@@ -42,7 +44,11 @@
- GF_SECURITY_ADMIN_PASSWORD=<GrafanaPasswordHere>
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_NAME=Beam
- - GF_INSTALL_PLUGINS=vonage-status-panel
+ - DB_HOST=beampostgresql
+ - DB_PORT=5432
+ - DB_DBNAME=beam_metrics
+ - DB_DBUSERNAME=admin
+ - DB_DBPWD=<PGPasswordHere>
syncgithub:
image: syncgithub
container_name: beamsyncgithub
@@ -55,7 +61,7 @@
- DB_DBNAME=beam_metrics
- DB_DBUSERNAME=admin
- DB_DBPWD=<PGPasswordHere>
- - GH_ACCESS_TOKEN=<GitHubAccessToken>
+ - GH_ACCESS_TOKEN=<GithubAccessToken>
syncjenkins:
image: syncjenkins
container_name: beamsyncjenkins
@@ -63,11 +69,11 @@
context: ./sync/jenkins
dockerfile: Dockerfile
environment:
- - JENSYNC_HOST=beampostgresql
- - JENSYNC_PORT=5432
- - JENSYNC_DBNAME=beam_metrics
- - JENSYNC_DBUSERNAME=admin
- - JENSYNC_DBPWD=<PGPasswordHere>
+ - DB_HOST=beampostgresql
+ - DB_PORT=5432
+ - DB_DBNAME=beam_metrics
+ - DB_DBUSERNAME=admin
+ - DB_DBPWD=<PGPasswordHere>
syncjira:
image: syncjira
container_name: beamsyncjira
@@ -80,9 +86,35 @@
- DB_DBNAME=beam_metrics
- DB_DBUSERNAME=admin
- DB_DBPWD=<PGPasswordHere>
+ prometheus:
+ image: prom/prometheus
+ ports:
+ - 9090:9090
+ container_name: prometheus
+ volumes:
+ - ./prometheus/prometheus/config:/etc/prometheus:ro
+ - prometheus-storage:/prometheus
+ command:
+ - --config.file=/etc/prometheus/prometheus.yml
+ - --web.console.libraries=/etc/prometheus/console_libraries
+ - --web.console.templates=/etc/prometheus/consoles
+ - --storage.tsdb.path=/prometheus
+ - --storage.tsdb.retention.time=365d
+ pushgateway:
+ image: prom/pushgateway
+ container_name: pushgateway
+ ports:
+ - 9091:9091
+ alertmanager:
+ image: prom/alertmanager
+ container_name: alertmanager
+ ports:
+ - 9093:9093
+ volumes:
+ - ./prometheus/alertmanager/config:/etc/alertmanager:ro
volumes:
beam-postgresql-data:
beam-grafana-libdata:
beam-grafana-etcdata:
beam-grafana-logdata:
-
+ prometheus-storage:
diff --git a/.test-infra/metrics/grafana/Dockerfile b/.test-infra/metrics/grafana/Dockerfile
new file mode 100644
index 0000000..8951eaf
--- /dev/null
+++ b/.test-infra/metrics/grafana/Dockerfile
@@ -0,0 +1,25 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+FROM grafana/grafana:6.2.5
+
+COPY ./provisioning /etc/beamgrafana/provisioning
+COPY ./dashboards /etc/beamgrafana/dashboards
+
+ENV GF_PATHS_PROVISIONING /etc/beamgrafana/provisioning
+
diff --git a/.test-infra/metrics/grafana/dashboards/Post-Commits_status_dashboard.json b/.test-infra/metrics/grafana/dashboards/Post-Commits_status_dashboard.json
new file mode 100644
index 0000000..0a5e729
--- /dev/null
+++ b/.test-infra/metrics/grafana/dashboards/Post-Commits_status_dashboard.json
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 8,
+ "links": [],
+ "panels": [
+ {
+ "columns": [],
+ "datasource": "BeamPSQL",
+ "fontSize": "80%",
+ "gridPos": {
+ "h": 32,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "links": [],
+ "pageSize": 1000,
+ "scroll": false,
+ "showHeader": true,
+ "sort": {
+ "col": 0,
+ "desc": false
+ },
+ "styles": [
+ {
+ "alias": "",
+ "colorMode": "cell",
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 2,
+ "mappingType": 1,
+ "pattern": "job_\\d*",
+ "preserveFormat": false,
+ "thresholds": [
+ "0.5",
+ "0.1"
+ ],
+ "type": "string",
+ "unit": "short",
+ "valueMaps": [
+ {
+ "text": "Success",
+ "value": "1"
+ },
+ {
+ "text": "Fail",
+ "value": "0"
+ }
+ ]
+ },
+ {
+ "alias": "Job Name",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 2,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkUrl": "https://builds.apache.org/job/${__cell}",
+ "mappingType": 1,
+ "pattern": "job_name",
+ "thresholds": [],
+ "type": "string",
+ "unit": "short"
+ }
+ ],
+ "targets": [
+ {
+ "aggregation": "Last",
+ "alias": "job",
+ "decimals": 2,
+ "displayAliasType": "Warning / Critical",
+ "displayType": "Regular",
+ "displayValueWithAlias": "Never",
+ "format": "table",
+ "group": [],
+ "hide": false,
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with src1 as (select * from crosstab ('\n select *\n from (SELECT\n t.job_name,\n ROW_NUMBER() OVER (PARTITION BY job_name ORDER BY build_id desc) AS r,\n case when t.build_result like ''SUCCESS'' then 1 else 0 end as res\n FROM\n jenkins_builds t\n where job_name like ''beam_PostCommit_%'' \n and job_name not like ''%_PR''\n and (t.build_timestamp BETWEEN '$__timeFrom()' AND '$__timeTo()')\n ) x\n where x.r < 11')\n \n AS (\n job_name varchar,\n job_1 integer,\n job_2 integer,\n job_3 integer,\n job_4 integer,\n job_5 integer,\n job_6 integer,\n job_7 integer,\n job_8 integer,\n job_9 integer,\n job_10 integer\n )),\n src2 as (select * from crosstab ('\n select *\n from (SELECT\n t.job_name,\n ROW_NUMBER() OVER (PARTITION BY job_name ORDER BY build_id desc) AS r,\n case when t.build_result like ''SUCCESS'' then 1 else 0 end as res\n FROM\n jenkins_builds t\n where job_name like ''beam_PreCommit_%_Cron''\n and (t.build_timestamp BETWEEN '$__timeFrom()' AND '$__timeTo()')\n ) x\n where x.r < 11')\n \n AS (\n job_name varchar,\n job_1 integer,\n job_2 integer,\n job_3 integer,\n job_4 integer,\n job_5 integer,\n job_6 integer,\n job_7 integer,\n job_8 integer,\n job_9 integer,\n job_10 integer\n ))\n \nselect *\nfrom src2\nunion all\nselect *\nfrom src1;",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "units": "none",
+ "valueHandler": "Number Threshold",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "timeFrom": null,
+ "timeShift": null,
+ "title": "Post-Commits statuses",
+ "transform": "table",
+ "type": "table"
+ }
+ ],
+ "refresh": "30s",
+ "schemaVersion": 18,
+ "style": "dark",
+ "tags": [],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-7d",
+ "to": "now"
+ },
+ "timepicker": {
+ "refresh_intervals": [
+ "5s",
+ "10s",
+ "30s",
+ "1m",
+ "5m",
+ "15m",
+ "30m",
+ "1h",
+ "2h",
+ "1d"
+ ],
+ "time_options": [
+ "5m",
+ "15m",
+ "1h",
+ "6h",
+ "12h",
+ "24h",
+ "2d",
+ "7d",
+ "30d"
+ ]
+ },
+ "timezone": "",
+ "title": "Post-Commits status dashboard",
+ "uid": "8N6LVeCmk",
+ "version": 10
+}
diff --git a/.test-infra/metrics/grafana/dashboards/code_velocity.json b/.test-infra/metrics/grafana/dashboards/code_velocity.json
new file mode 100644
index 0000000..b750dcb
--- /dev/null
+++ b/.test-infra/metrics/grafana/dashboards/code_velocity.json
@@ -0,0 +1,1128 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "description": "Shows common Code Velocity metrics, like reviewer load and open/closed PRs.",
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 7,
+ "links": [],
+ "panels": [
+ {
+ "cacheTimeout": null,
+ "colorBackground": false,
+ "colorValue": false,
+ "colors": [
+ "#299c46",
+ "rgba(237, 129, 40, 0.89)",
+ "#d44a3a"
+ ],
+ "datasource": "BeamPSQL",
+ "decimals": null,
+ "format": "dthms",
+ "gauge": {
+ "maxValue": 100,
+ "minValue": 0,
+ "show": false,
+ "thresholdLabels": false,
+ "thresholdMarkers": true
+ },
+ "gridPos": {
+ "h": 2,
+ "w": 3,
+ "x": 0,
+ "y": 0
+ },
+ "id": 6,
+ "interval": null,
+ "links": [
+ {
+ "targetBlank": true,
+ "title": "Link to oldest open PRs",
+ "type": "absolute",
+ "url": "https://github.com/apache/beam/pulls?q=is%3Apr+is%3Aopen+sort%3Acreated-asc"
+ }
+ ],
+ "mappingType": 1,
+ "mappingTypes": [
+ {
+ "name": "value to text",
+ "value": 1
+ },
+ {
+ "name": "range to text",
+ "value": 2
+ }
+ ],
+ "maxDataPoints": 100,
+ "nullPointMode": "connected",
+ "nullText": null,
+ "postfix": "",
+ "postfixFontSize": "50%",
+ "prefix": "",
+ "prefixFontSize": "50%",
+ "rangeMaps": [
+ {
+ "from": "null",
+ "text": "N/A",
+ "to": "null"
+ }
+ ],
+ "sparkline": {
+ "fillColor": "rgba(31, 118, 189, 0.18)",
+ "full": false,
+ "lineColor": "rgb(31, 120, 193)",
+ "show": false
+ },
+ "tableColumn": "maxprage",
+ "targets": [
+ {
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT (now() - min(created_ts)) as MaxPrAge\nfrom gh_pull_requests\nwhere closed_ts is NULL",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": "",
+ "title": "Max Age of Open PR",
+ "type": "singlestat",
+ "valueFontSize": "80%",
+ "valueMaps": [
+ {
+ "op": "=",
+ "text": "N/A",
+ "value": "null"
+ }
+ ],
+ "valueName": "avg"
+ },
+ {
+ "cacheTimeout": null,
+ "colorBackground": false,
+ "colorValue": false,
+ "colors": [
+ "#299c46",
+ "rgba(237, 129, 40, 0.89)",
+ "#d44a3a"
+ ],
+ "datasource": "BeamPSQL",
+ "format": "none",
+ "gauge": {
+ "maxValue": 100,
+ "minValue": 0,
+ "show": false,
+ "thresholdLabels": false,
+ "thresholdMarkers": true
+ },
+ "gridPos": {
+ "h": 2,
+ "w": 3,
+ "x": 3,
+ "y": 0
+ },
+ "id": 13,
+ "interval": null,
+ "links": [
+ {
+ "targetBlank": true,
+ "title": "Link to oldest open PRs",
+ "type": "absolute",
+ "url": "https://github.com/apache/beam/pulls?q=is%3Apr+is%3Aopen+sort%3Acreated-asc"
+ }
+ ],
+ "mappingType": 1,
+ "mappingTypes": [
+ {
+ "name": "value to text",
+ "value": 1
+ },
+ {
+ "name": "range to text",
+ "value": 2
+ }
+ ],
+ "maxDataPoints": 100,
+ "nullPointMode": "connected",
+ "nullText": null,
+ "postfix": "",
+ "postfixFontSize": "50%",
+ "prefix": "",
+ "prefixFontSize": "50%",
+ "rangeMaps": [
+ {
+ "from": "null",
+ "text": "N/A",
+ "to": "null"
+ }
+ ],
+ "sparkline": {
+ "fillColor": "rgba(31, 118, 189, 0.18)",
+ "full": false,
+ "lineColor": "rgb(31, 120, 193)",
+ "show": false
+ },
+ "tableColumn": "pr_id",
+ "targets": [
+ {
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT pr_id\nfrom gh_pull_requests\nwhere closed_ts is NULL and created_ts = (select min(created_ts) from gh_pull_requests where closed_ts is NULL)\n",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": "",
+ "title": "Oldest Open PR",
+ "type": "singlestat",
+ "valueFontSize": "80%",
+ "valueMaps": [
+ {
+ "op": "=",
+ "text": "N/A",
+ "value": "null"
+ }
+ ],
+ "valueName": "avg"
+ },
+ {
+ "cacheTimeout": null,
+ "colorBackground": false,
+ "colorValue": false,
+ "colors": [
+ "#299c46",
+ "rgba(237, 129, 40, 0.89)",
+ "#d44a3a"
+ ],
+ "datasource": "BeamPSQL",
+ "format": "none",
+ "gauge": {
+ "maxValue": 100,
+ "minValue": 0,
+ "show": false,
+ "thresholdLabels": false,
+ "thresholdMarkers": true
+ },
+ "gridPos": {
+ "h": 2,
+ "w": 3,
+ "x": 6,
+ "y": 0
+ },
+ "id": 10,
+ "interval": null,
+ "links": [
+ {
+ "targetBlank": true,
+ "title": "Least updated CLs",
+ "type": "absolute",
+ "url": "https://github.com/apache/beam/pulls?q=is%3Aopen+is%3Apr+sort%3Aupdated-asc"
+ }
+ ],
+ "mappingType": 1,
+ "mappingTypes": [
+ {
+ "name": "value to text",
+ "value": 1
+ },
+ {
+ "name": "range to text",
+ "value": 2
+ }
+ ],
+ "maxDataPoints": 100,
+ "nullPointMode": "connected",
+ "nullText": null,
+ "postfix": "",
+ "postfixFontSize": "50%",
+ "prefix": "",
+ "prefixFontSize": "50%",
+ "rangeMaps": [
+ {
+ "from": "null",
+ "text": "N/A",
+ "to": "null"
+ }
+ ],
+ "sparkline": {
+ "fillColor": "rgba(31, 118, 189, 0.18)",
+ "full": false,
+ "lineColor": "rgb(31, 120, 193)",
+ "show": false
+ },
+ "tableColumn": "pr_id",
+ "targets": [
+ {
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with d1 as (SELECT (now() - min(updated_ts)) as MaxPrAge, min(updated_ts) as min_updated_ts\n from gh_pull_requests\n where closed_ts is NULL)\nselect pr_id\nfrom gh_pull_requests, d1\nwhere updated_ts = min_updated_ts",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": "",
+ "title": "Open PR with Oldest Activity",
+ "type": "singlestat",
+ "valueFontSize": "80%",
+ "valueMaps": [
+ {
+ "op": "=",
+ "text": "N/A",
+ "value": "null"
+ }
+ ],
+ "valueName": "avg"
+ },
+ {
+ "cacheTimeout": null,
+ "colorBackground": false,
+ "colorValue": false,
+ "colors": [
+ "#299c46",
+ "rgba(237, 129, 40, 0.89)",
+ "#d44a3a"
+ ],
+ "datasource": "BeamPSQL",
+ "format": "none",
+ "gauge": {
+ "maxValue": 100,
+ "minValue": 0,
+ "show": false,
+ "thresholdLabels": false,
+ "thresholdMarkers": true
+ },
+ "gridPos": {
+ "h": 2,
+ "w": 3,
+ "x": 9,
+ "y": 0
+ },
+ "id": 12,
+ "interval": null,
+ "links": [
+ {
+ "title": "Least updated PRs",
+ "type": "absolute",
+ "url": "https://github.com/apache/beam/pulls?q=is%3Aopen+is%3Apr+sort%3Aupdated-asc"
+ }
+ ],
+ "mappingType": 1,
+ "mappingTypes": [
+ {
+ "name": "value to text",
+ "value": 1
+ },
+ {
+ "name": "range to text",
+ "value": 2
+ }
+ ],
+ "maxDataPoints": 100,
+ "nullPointMode": "connected",
+ "nullText": null,
+ "postfix": "",
+ "postfixFontSize": "50%",
+ "prefix": "",
+ "prefixFontSize": "50%",
+ "rangeMaps": [
+ {
+ "from": "null",
+ "text": "N/A",
+ "to": "null"
+ }
+ ],
+ "sparkline": {
+ "fillColor": "rgba(31, 118, 189, 0.18)",
+ "full": false,
+ "lineColor": "rgb(31, 120, 193)",
+ "show": false
+ },
+ "tableColumn": "maxpractivityage",
+ "targets": [
+ {
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT (now() - min(updated_ts)) as MaxPrActivityAge\n from gh_pull_requests\n where closed_ts is NULL",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": "",
+ "title": "Age of Oldest Activity on Open PR",
+ "type": "singlestat",
+ "valueFontSize": "80%",
+ "valueMaps": [
+ {
+ "op": "=",
+ "text": "N/A",
+ "value": "null"
+ }
+ ],
+ "valueName": "avg"
+ },
+ {
+ "columns": [],
+ "datasource": "BeamPSQL",
+ "description": "Shows open PRs and list of reviewers assigned to PR.\n\nReviewers are calculated in two ways:\n1. beam_reviewers - this is initial algorithm that detects defined patterns in comments.\n2. updated_beam_reviewers_algorithm - this algo lists all mentions (@.*), as well as whoever was assigned as reviewer, reviewed or merged code.",
+ "fontSize": "100%",
+ "gridPos": {
+ "h": 14,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "id": 8,
+ "links": [],
+ "pageSize": null,
+ "scroll": true,
+ "showHeader": true,
+ "sort": {
+ "col": null,
+ "desc": false
+ },
+ "styles": [
+ {
+ "alias": "Time",
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "pattern": "Time",
+ "type": "date"
+ },
+ {
+ "alias": "",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": null,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkTooltip": "PR",
+ "linkUrl": "https://github.com/apache/beam/pull/${__cell:raw}",
+ "mappingType": 1,
+ "pattern": "pr_id",
+ "thresholds": [],
+ "type": "number",
+ "unit": "none"
+ },
+ {
+ "alias": "",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 2,
+ "mappingType": 1,
+ "pattern": "beam_reviewers",
+ "thresholds": [],
+ "type": "number",
+ "unit": "short"
+ },
+ {
+ "alias": "",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "decimals": 0,
+ "pattern": "/.*/",
+ "thresholds": [],
+ "type": "number",
+ "unit": "none"
+ }
+ ],
+ "targets": [
+ {
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with combined_reviewers as (select pr_id, array_agg(DISTINCT unnested) as unique_reviewers\n from gh_pull_requests src, unnest(mentioned || requested_reviewers || reviewed_by) as unnested\n group by pr_id)\n\nSELECT gh_pull_requests.pr_id, author, (now() - updated_ts) as last_activity, beam_reviewers, combined_reviewers.unique_reviewers as updated_beam_reviewers_algorithm\nFROM gh_pull_requests left outer join combined_reviewers ON gh_pull_requests.pr_id = combined_reviewers.pr_id\nWHERE gh_pull_requests.closed_ts is null\norder by last_activity desc",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "title": "Currently Open PRs",
+ "transform": "table",
+ "type": "table"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "description": "Shows total PRs opened and statistics of open/closed PRs per day.",
+ "fill": 1,
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 0,
+ "y": 2
+ },
+ "id": 15,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day, count(*) as total_open\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.created_ts < days.day AND (gh_pull_requests.closed_ts > days.day OR gh_pull_requests.closed_ts is null)\n GROUP BY days.day\n ORDER BY days.day)\nselect days.day as time, greatest(knowndays.total_open, 0) as total_open\nfrom days left outer join knowndays\non days.day = knowndays.day",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ },
+ {
+ "format": "time_series",
+ "group": [],
+ "hide": false,
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day, 0 as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day as day, count(*) as ClosedTickets\n FROM gh_pull_requests, days\n WHERE date_trunc('day', gh_pull_requests.closed_ts) = days.day\n GROUP BY day\n ORDER BY day\n )\nselect days.day as time, greatest(knowndays.ClosedTickets, days.zcnt) as closed_count\nfrom days left outer join knowndays\non days.day = knowndays.day\n",
+ "refId": "B",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ },
+ {
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day, 0 as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day as day, count(*) as ClosedTickets\n FROM gh_pull_requests, days\n WHERE date_trunc('day', gh_pull_requests.created_ts) = days.day\n GROUP BY day\n ORDER BY day\n )\nselect days.day as time, greatest(knowndays.ClosedTickets, days.zcnt) as craeted_count\nfrom days left outer join knowndays\non days.day = knowndays.day\n",
+ "refId": "C",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "PR Activity Trends",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "alert": {
+ "conditions": [
+ {
+ "evaluator": {
+ "params": [
+ 72
+ ],
+ "type": "gt"
+ },
+ "operator": {
+ "type": "and"
+ },
+ "query": {
+ "params": [
+ "A",
+ "5m",
+ "now"
+ ]
+ },
+ "reducer": {
+ "params": [],
+ "type": "last"
+ },
+ "type": "query"
+ }
+ ],
+ "executionErrorState": "alerting",
+ "frequency": "1h",
+ "handler": 1,
+ "name": "Time to First Non-Author PR Response on 7 Day Span alert",
+ "noDataState": "no_data",
+ "notifications": []
+ },
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "description": "Shows first non-author activity on PR on sliding window of 7 days.",
+ "fill": 0,
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 0,
+ "y": 8
+ },
+ "id": 17,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day, '0 day'::interval as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n responseTimes as (SELECT (gh_pull_requests.first_non_author_activity_ts - gh_pull_requests.created_ts) as responseTime, days.day as day\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.first_non_author_activity_ts < days.day AND gh_pull_requests.first_non_author_activity_ts > (days.day - '7 day'::interval)\n ),\n percTimes as (SELECT percentile_disc(0.85) WITHIN GROUP (ORDER BY responseTime) as eightyFifthResponseTime, day\n FROM responseTimes\n group by day),\n avgTimes as (SELECT avg(responseTime) as avgResponseTime, day\n FROM responseTimes\n group by day)\nselect days.day as time, EXTRACT(EPOCH FROM percTimes.eightyFifthResponseTime)/3600 as \"85th Percentile\", EXTRACT(EPOCH FROM avgTimes.avgResponseTime)/3600 as \"Mean\"\nfrom days left outer join percTimes on days.day = percTimes.day\n left outer join avgTimes on days.day = avgTimes.day\n",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "critical",
+ "fill": true,
+ "line": true,
+ "op": "gt",
+ "value": 72
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Time to First Non-Author PR Response on 7 Day Span",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": null,
+ "format": "h",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "decimals": 0,
+ "description": "Assignees with less than 5 PRs are not shown.",
+ "fill": 0,
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 0,
+ "y": 14
+ },
+ "id": 22,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": true,
+ "hideEmpty": true,
+ "hideZero": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": true
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null as zero",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day, '0 day'::interval as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n openPRs as (SELECT *\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.created_ts < days.day AND (gh_pull_requests.closed_ts > days.day OR gh_pull_requests.closed_ts is null)),\n flattennedPRs as (SELECT * \n FROM openPRs, unnest(openPRs.beam_reviewers) flattenedMentioned)\nselect day as time, count(*) as PRsAssigned, flattenedMentioned\nfrom flattennedPRs\ngroup by day, flattenedMentioned\nHAVING count(*) > 4\norder by day, flattenedMentioned\n\n",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Reviews per Beam Reviewer",
+ "tooltip": {
+ "shared": true,
+ "sort": 2,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": null,
+ "format": "none",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "decimals": 0,
+ "description": "Assignees with less than 5 PRs are not shown.",
+ "fill": 0,
+ "gridPos": {
+ "h": 6,
+ "w": 12,
+ "x": 12,
+ "y": 14
+ },
+ "id": 18,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": true,
+ "hideEmpty": false,
+ "hideZero": false,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": true
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day, '0 day'::interval as zcnt from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n openPRs as (SELECT *\n FROM gh_pull_requests, days\n WHERE gh_pull_requests.created_ts < days.day AND (gh_pull_requests.closed_ts > days.day OR gh_pull_requests.closed_ts is null)),\n flattennedPRs as (SELECT * \n FROM openPRs, unnest(openPRs.mentioned) flattenedMentioned)\nselect day as time, count(*) as PRsAssigned, flattenedMentioned\nfrom flattennedPRs\ngroup by day, flattenedMentioned\nHAVING count(*) > 4\norder by day, flattenedMentioned\n\n",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Reviews per Mentioned and Assigned",
+ "tooltip": {
+ "shared": true,
+ "sort": 2,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": null,
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": false,
+ "schemaVersion": 18,
+ "style": "dark",
+ "tags": [],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-90d",
+ "to": "now"
+ },
+ "timepicker": {
+ "refresh_intervals": [
+ "5s",
+ "10s",
+ "30s",
+ "1m",
+ "5m",
+ "15m",
+ "30m",
+ "1h",
+ "2h",
+ "1d"
+ ],
+ "time_options": [
+ "5m",
+ "15m",
+ "1h",
+ "6h",
+ "12h",
+ "24h",
+ "2d",
+ "7d",
+ "30d"
+ ]
+ },
+ "timezone": "",
+ "title": "Code Velocity",
+ "uid": "code_velocity",
+ "version": 7
+}
diff --git a/.test-infra/metrics/grafana/dashboards/post-commit_tests.json b/.test-infra/metrics/grafana/dashboards/post-commit_tests.json
new file mode 100644
index 0000000..cb81dcf
--- /dev/null
+++ b/.test-infra/metrics/grafana/dashboards/post-commit_tests.json
@@ -0,0 +1,797 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "limit": 100,
+ "name": "Annotations & Alerts",
+ "showIn": 0,
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 1,
+ "links": [],
+ "panels": [
+ {
+ "content": "This dashboard tracks Post-commit test reliability over-time.\n\n* [Post-commit test policies](https://beam.apache.org/contribute/postcommits-policies/)\n* [Existing test failure issues](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)\n* [File a new test failure issue](https://s.apache.org/beam-test-failure)",
+ "gridPos": {
+ "h": 4,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 11,
+ "links": [],
+ "mode": "markdown",
+ "title": "Dashboard guidelines",
+ "type": "text"
+ },
+ {
+ "alert": {
+ "conditions": [
+ {
+ "evaluator": {
+ "params": [
+ 0.7
+ ],
+ "type": "lt"
+ },
+ "operator": {
+ "type": "and"
+ },
+ "query": {
+ "params": [
+ "A",
+ "5m",
+ "now"
+ ]
+ },
+ "reducer": {
+ "params": [],
+ "type": "min"
+ },
+ "type": "query"
+ }
+ ],
+ "executionErrorState": "alerting",
+ "frequency": "30m",
+ "handler": 1,
+ "name": "Post-commit reliability per week alert",
+ "noDataState": "keep_state",
+ "notifications": []
+ },
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "decimals": 0,
+ "description": "Percent reliability of all post-commit job runs for a given week.\n\nUnreliability of a test suite impact developer productivity by forcing contributors to re-run tests. When tests are consistently unreliable, developers will simply ignore them.\n\nWe aim for >= 70% reliability per test suite.",
+ "fill": 0,
+ "gridPos": {
+ "h": 7,
+ "w": 24,
+ "x": 0,
+ "y": 4
+ },
+ "id": 6,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": true,
+ "hideEmpty": false,
+ "hideZero": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": true
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n DATE_TRUNC('week', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time\n",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "critical",
+ "fill": true,
+ "line": true,
+ "op": "lt",
+ "value": 0.7
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Post-commit reliability per week",
+ "tooltip": {
+ "shared": true,
+ "sort": 1,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": 1,
+ "format": "percentunit",
+ "label": "% successful runs",
+ "logBase": 1,
+ "max": "1",
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "alert": {
+ "conditions": [
+ {
+ "evaluator": {
+ "params": [
+ 0.7
+ ],
+ "type": "lt"
+ },
+ "operator": {
+ "type": "and"
+ },
+ "query": {
+ "params": [
+ "A",
+ "5m",
+ "now"
+ ]
+ },
+ "reducer": {
+ "params": [],
+ "type": "min"
+ },
+ "type": "query"
+ }
+ ],
+ "executionErrorState": "alerting",
+ "frequency": "30m",
+ "handler": 1,
+ "name": "Post-commit reliability per day alert",
+ "noDataState": "no_data",
+ "notifications": []
+ },
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "decimals": 0,
+ "description": "Percent reliability of all post-commit job runs per-day.\n\nUnreliability of a test suite impact developer productivity by forcing contributors to re-run tests. When tests are consistently unreliable, developers will simply ignore them.\n\nWe aim for >= 70% reliability per test suite.",
+ "fill": 0,
+ "gridPos": {
+ "h": 12,
+ "w": 15,
+ "x": 0,
+ "y": 11
+ },
+ "id": 9,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": true,
+ "hideZero": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": false,
+ "sideWidth": null,
+ "total": false,
+ "values": true
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n DATE_TRUNC('day', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time\n",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "critical",
+ "fill": true,
+ "line": true,
+ "op": "lt",
+ "value": 0.7
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Post-commit reliability per day",
+ "tooltip": {
+ "shared": true,
+ "sort": 1,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": 1,
+ "format": "percentunit",
+ "label": "% successful runs",
+ "logBase": 1,
+ "max": "1",
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "columns": [],
+ "datasource": "BeamPSQL",
+ "description": "List of jobs which have failed. Click on the job to view it in Jenkins.",
+ "fontSize": "100%",
+ "gridPos": {
+ "h": 12,
+ "w": 9,
+ "x": 15,
+ "y": 11
+ },
+ "hideTimeOverride": false,
+ "id": 8,
+ "links": [
+ {
+ "includeVars": false,
+ "targetBlank": true,
+ "title": "Beam Jenkins",
+ "type": "absolute",
+ "url": "https://builds.apache.org/view/A-D/view/Beam/"
+ }
+ ],
+ "pageSize": null,
+ "scroll": true,
+ "showHeader": true,
+ "sort": {
+ "col": 0,
+ "desc": true
+ },
+ "styles": [
+ {
+ "alias": "Time",
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "link": false,
+ "pattern": "Time",
+ "type": "date"
+ },
+ {
+ "alias": "Build Url",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 2,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkTooltip": "Link to Jenkins job.",
+ "linkUrl": "${__cell:raw}",
+ "mappingType": 1,
+ "pattern": "build_url",
+ "thresholds": [],
+ "type": "hidden",
+ "unit": "short"
+ },
+ {
+ "alias": "Job Name",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 2,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkTooltip": "View Jenkins job: ${__cell_1}_${__cell_2}",
+ "linkUrl": "${__cell_0:raw}",
+ "mappingType": 1,
+ "pattern": "job_name",
+ "thresholds": [],
+ "type": "string",
+ "unit": "short"
+ },
+ {
+ "alias": "Build ID",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 0,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkTooltip": "View Jenkins job: ${__cell_1}_${__cell_2}",
+ "linkUrl": "${__cell_0:raw}",
+ "mappingType": 1,
+ "pattern": "build_id",
+ "thresholds": [],
+ "type": "number",
+ "unit": "short"
+ },
+ {
+ "alias": "Start Time",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "MM/DD/YY h:mm:ss a",
+ "decimals": 2,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkTooltip": "View Jenkins job: ${__cell_1}_${__cell_2}",
+ "linkUrl": "${__cell_0:raw}",
+ "mappingType": 1,
+ "pattern": "build_timestamp",
+ "thresholds": [],
+ "type": "date",
+ "unit": "short"
+ }
+ ],
+ "targets": [
+ {
+ "alias": "",
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT \n build_url,\n job_name,\n build_id,\n build_timestamp\nFROM jenkins_builds\nWHERE \n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name LIKE '%_PR')\n AND NOT (build_result = 'SUCCESS')\nORDER BY \n build_timestamp",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "timeShift": null,
+ "title": "Failed builds",
+ "transform": "table",
+ "type": "table"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "decimals": 1,
+ "description": "Execution time for each post-commit job",
+ "fill": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 15,
+ "x": 0,
+ "y": 23
+ },
+ "id": 5,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": false,
+ "total": false,
+ "values": true
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n build_timestamp as time,\n build_duration as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as metric\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND (job_name LIKE 'beam_PostCommit_%')\n AND NOT (job_name LIKE '%_PR')\nORDER BY\n job_name, time",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Post-commit job duration",
+ "tooltip": {
+ "shared": true,
+ "sort": 2,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": null,
+ "format": "ms",
+ "label": "Average job duration",
+ "logBase": 1,
+ "max": null,
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": true,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "decimals": 0,
+ "description": "Tracks the count of test failure JIRA issues currently open.",
+ "fill": 3,
+ "gridPos": {
+ "h": 8,
+ "w": 9,
+ "x": 15,
+ "y": 23
+ },
+ "id": 14,
+ "legend": {
+ "alignAsTable": false,
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "rightSide": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": false,
+ "linewidth": 1,
+ "links": [
+ {
+ "targetBlank": true,
+ "title": "Jira tickets",
+ "type": "absolute",
+ "url": "https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC"
+ }
+ ],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [
+ {
+ "alias": "total_open",
+ "color": "#eab839"
+ },
+ {
+ "alias": "currently_failing",
+ "color": "#bf1b00"
+ }
+ ],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [],
+ "hide": false,
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day, count(*) as total_open\n FROM jira_issues, days\n WHERE jira_issues.created < days.day AND (jira_issues.resolutiondate > days.day OR jira_issues.resolutiondate is null)\n GROUP BY days.day\n ORDER BY days.day)\nselect days.day as time, greatest(knowndays.total_open, 0) as total_open\nfrom days left outer join knowndays\non days.day = knowndays.day",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ },
+ {
+ "format": "time_series",
+ "group": [],
+ "hide": false,
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "with days as (select date_trunc('day', dd) as day from generate_series( $__timeFrom()::timestamp, $__timeTo()::timestamp, '1 day'::interval) as dd),\n knowndays as (SELECT days.day, count(*) as currently_failing\n FROM jira_issues, days\n WHERE jira_issues.created < days.day AND (jira_issues.resolutiondate > days.day OR jira_issues.resolutiondate is null) AND (jira_issues.labels LIKE '%currently-failing%')\n GROUP BY days.day\n ORDER BY days.day)\nselect days.day as time, greatest(knowndays.currently_failing, 0) as currently_failing\nfrom days left outer join knowndays\non days.day = knowndays.day",
+ "refId": "D",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Test Failure JIRA issues",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": 0,
+ "format": "short",
+ "label": "# of JIRA issues",
+ "logBase": 1,
+ "max": null,
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": false,
+ "schemaVersion": 18,
+ "style": "dark",
+ "tags": [],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-30d",
+ "to": "now"
+ },
+ "timepicker": {
+ "hidden": false,
+ "refresh_intervals": [
+ "1m",
+ "5m",
+ "15m",
+ "30m",
+ "1h",
+ "2h",
+ "1d"
+ ],
+ "time_options": [
+ "5m",
+ "15m",
+ "1h",
+ "6h",
+ "12h",
+ "24h",
+ "2d",
+ "7d",
+ "30d"
+ ]
+ },
+ "timezone": "",
+ "title": "Post-commit Test Reliability",
+ "uid": "D81lW0pmk",
+ "version": 46
+}
diff --git a/.test-infra/metrics/grafana/dashboards/pre-commit_tests.json b/.test-infra/metrics/grafana/dashboards/pre-commit_tests.json
new file mode 100644
index 0000000..e5ab46e
--- /dev/null
+++ b/.test-infra/metrics/grafana/dashboards/pre-commit_tests.json
@@ -0,0 +1,435 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 2,
+ "links": [],
+ "panels": [
+ {
+ "alert": {
+ "conditions": [
+ {
+ "evaluator": {
+ "params": [
+ 7200000
+ ],
+ "type": "gt"
+ },
+ "operator": {
+ "type": "and"
+ },
+ "query": {
+ "params": [
+ "A",
+ "5m",
+ "now"
+ ]
+ },
+ "reducer": {
+ "params": [],
+ "type": "max"
+ },
+ "type": "query"
+ }
+ ],
+ "executionErrorState": "alerting",
+ "frequency": "30m",
+ "handler": 1,
+ "name": "Pre-commit job duration per-day alert",
+ "noDataState": "keep_state",
+ "notifications": []
+ },
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "description": "Execution time for each pre-commit job.\n\nLong test suite execution impacts developer productivity by delaying the quality signal of a pull request of current HEAD. If tests are consistently slow, developers won't wait for them to complete.\n\nWe aim for under 2 hour execution per test suite, but ideally under 30 mins.",
+ "fill": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 0
+ },
+ "id": 4,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "sort": "current",
+ "sortDesc": true,
+ "total": false,
+ "values": true
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n build_timestamp as time,\n build_duration as value,\n substring(job_name from 'beam_PreCommit_#\"%#\"_(Cron|Commit)' for '#') as metric\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND build_result = 'SUCCESS'\n AND ((job_name LIKE 'beam_PreCommit_%_Commit')\n OR (job_name LIKE 'beam_PreCommit_%_Cron'))\nORDER BY\n metric, time",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "critical",
+ "fill": true,
+ "line": true,
+ "op": "gt",
+ "value": 7200000
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Pre-commit job duration",
+ "tooltip": {
+ "shared": true,
+ "sort": 0,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "ms",
+ "label": "Average job duration",
+ "logBase": 1,
+ "max": null,
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "fill": 1,
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 8
+ },
+ "id": 6,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": false,
+ "hideEmpty": true,
+ "hideZero": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n build_timestamp as time,\n timing_queuingDurationMillis as value,\n substring(job_name from 'beam_PreCommit_#\"%#\"_(Cron|Commit|Phrase)' for '#') as metric\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND build_result = 'SUCCESS'\n AND ((job_name LIKE 'beam_PreCommit_%_Commit')\n OR (job_name LIKE 'beam_PreCommit_%_Cron')\n OR (job_name LIKE 'beam_PreCommit_%_Phrase'))\nORDER BY\n metric, time",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Time in queue",
+ "tooltip": {
+ "shared": true,
+ "sort": 2,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "dtdurationms",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "fill": 0,
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 16
+ },
+ "id": 8,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": false,
+ "hideEmpty": true,
+ "hideZero": true,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "aggregation": "Last",
+ "decimals": 2,
+ "displayAliasType": "Warning / Critical",
+ "displayType": "Regular",
+ "displayValueWithAlias": "Never",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n DATE_TRUNC('month', build_timestamp) as time,\n percentile_disc(0.9) within group (order by timing_queuingDurationMillis) as value,\n substring(job_name from 'beam_PreCommit_#\"%#\"_(Cron|Commit|Phrase)' for '#') as metric\nFROM\n jenkins_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND build_result = 'SUCCESS'\n AND ((job_name LIKE 'beam_PreCommit_%_Commit')\n OR (job_name LIKE 'beam_PreCommit_%_Cron')\n OR (job_name LIKE 'beam_PreCommit_%_Phrase'))\nGROUP BY\n time, metric\nORDER BY\n time, metric",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "units": "none",
+ "valueHandler": "Number Threshold",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Time in queue: 0.9 percentile on month period",
+ "tooltip": {
+ "shared": true,
+ "sort": 2,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "format": "dtdurationms",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": true
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "schemaVersion": 18,
+ "style": "dark",
+ "tags": [],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-7d",
+ "to": "now"
+ },
+ "timepicker": {
+ "refresh_intervals": [
+ "5s",
+ "10s",
+ "30s",
+ "1m",
+ "5m",
+ "15m",
+ "30m",
+ "1h",
+ "2h",
+ "1d"
+ ],
+ "time_options": [
+ "5m",
+ "15m",
+ "1h",
+ "6h",
+ "12h",
+ "24h",
+ "2d",
+ "7d",
+ "30d"
+ ]
+ },
+ "timezone": "utc",
+ "title": "Pre-commit Test Latency",
+ "uid": "_TNndF2iz",
+ "version": 18
+}
diff --git a/.test-infra/metrics/grafana/dashboards/source_data_freshness.json b/.test-infra/metrics/grafana/dashboards/source_data_freshness.json
new file mode 100644
index 0000000..22a7b55
--- /dev/null
+++ b/.test-infra/metrics/grafana/dashboards/source_data_freshness.json
@@ -0,0 +1,229 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 5,
+ "links": [],
+ "panels": [
+ {
+ "alert": {
+ "conditions": [
+ {
+ "evaluator": {
+ "params": [
+ 480
+ ],
+ "type": "gt"
+ },
+ "operator": {
+ "type": "and"
+ },
+ "query": {
+ "params": [
+ "A",
+ "5m",
+ "now"
+ ]
+ },
+ "reducer": {
+ "params": [],
+ "type": "max"
+ },
+ "type": "query"
+ }
+ ],
+ "executionErrorState": "alerting",
+ "frequency": "5m",
+ "handler": 1,
+ "name": "Source Data Freshness alert",
+ "noDataState": "alerting",
+ "notifications": []
+ },
+ "aliasColors": {},
+ "bars": true,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "description": "Data freshness for each metrics input source. Used for health monitoring for other dashboards.",
+ "fill": 1,
+ "gridPos": {
+ "h": 12,
+ "w": 6,
+ "x": 0,
+ "y": 0
+ },
+ "id": 2,
+ "legend": {
+ "alignAsTable": false,
+ "avg": false,
+ "current": true,
+ "max": false,
+ "min": false,
+ "rightSide": false,
+ "show": true,
+ "total": false,
+ "values": true
+ },
+ "lines": false,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "percentage": false,
+ "pointradius": 5,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "format": "time_series",
+ "group": [
+ {
+ "params": [
+ "$__interval",
+ "none"
+ ],
+ "type": "time"
+ }
+ ],
+ "hide": false,
+ "metricColumn": "build_result",
+ "rawQuery": true,
+ "rawSql": "WITH sources AS (\n SELECT\n 'Jenkins' as source,\n MAX(build_timestamp + make_interval(secs:= timing_executingtimemillis / 1000.0)) AS last_sync\n FROM jenkins_builds\n UNION SELECT\n 'JIRA' as source,\n lastsynctime AS last_sync\n FROM jira_issues_metadata\n UNION SELECT\n 'GitHub' as source,\n timestamp AS last_sync\n FROM gh_sync_metadata\n)\nSELECT\n current_timestamp AS time,\n source,\n EXTRACT(EPOCH FROM age(current_timestamp, last_sync)) / (60) AS value\nFROM sources",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "build_duration"
+ ],
+ "type": "column"
+ },
+ {
+ "params": [
+ "max"
+ ],
+ "type": "aggregate"
+ },
+ {
+ "params": [
+ "build_duration"
+ ],
+ "type": "alias"
+ }
+ ]
+ ],
+ "table": "jenkins_builds",
+ "timeColumn": "build_timestamp",
+ "timeColumnType": "timestamp",
+ "where": []
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "critical",
+ "fill": true,
+ "line": true,
+ "op": "gt",
+ "value": 480
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Source Data Freshness",
+ "tooltip": {
+ "shared": false,
+ "sort": 1,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "series",
+ "name": null,
+ "show": false,
+ "values": [
+ "current"
+ ]
+ },
+ "yaxes": [
+ {
+ "decimals": 0,
+ "format": "m",
+ "label": "Staleness",
+ "logBase": 1,
+ "max": "60",
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "schemaVersion": 18,
+ "style": "dark",
+ "tags": [],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-30d",
+ "to": "now"
+ },
+ "timepicker": {
+ "refresh_intervals": [
+ "5s",
+ "10s",
+ "30s",
+ "1m",
+ "5m",
+ "15m",
+ "30m",
+ "1h",
+ "2h",
+ "1d"
+ ],
+ "time_options": [
+ "5m",
+ "15m",
+ "1h",
+ "6h",
+ "12h",
+ "24h",
+ "2d",
+ "7d",
+ "30d"
+ ]
+ },
+ "timezone": "",
+ "title": "Source Data Freshness",
+ "uid": "data-freshness",
+ "version": 8
+}
diff --git a/.test-infra/metrics/grafana/dashboards/stability_critical_jobs_status.json b/.test-infra/metrics/grafana/dashboards/stability_critical_jobs_status.json
new file mode 100644
index 0000000..83695dc
--- /dev/null
+++ b/.test-infra/metrics/grafana/dashboards/stability_critical_jobs_status.json
@@ -0,0 +1,415 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 3,
+ "links": [],
+ "panels": [
+ {
+ "content": "The graph shows: average greenness of critical post-commit tests jobs per week. This graph show health of our project.\n\nTable shows list of relevant jobs failures during selected time interval (You can change time interval on top-right corner of the dashboard). Please, triage failed jobs and update or create corresponding jira tickets. You can utilized provided links to help with this.",
+ "gridPos": {
+ "h": 3,
+ "w": 10,
+ "x": 0,
+ "y": 0
+ },
+ "id": 8,
+ "links": [],
+ "mode": "markdown",
+ "options": {},
+ "title": "Dashboard guidelines",
+ "type": "text"
+ },
+ {
+ "columns": [],
+ "datasource": "BeamPSQL",
+ "fontSize": "100%",
+ "gridPos": {
+ "h": 6,
+ "w": 14,
+ "x": 10,
+ "y": 0
+ },
+ "hideTimeOverride": false,
+ "id": 4,
+ "links": [],
+ "options": {},
+ "pageSize": null,
+ "scroll": true,
+ "showHeader": true,
+ "sort": {
+ "col": 0,
+ "desc": true
+ },
+ "styles": [
+ {
+ "alias": "Time",
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "link": false,
+ "pattern": "Time",
+ "type": "date"
+ },
+ {
+ "alias": "Build Url",
+ "colorMode": null,
+ "colors": [
+ "rgba(245, 54, 54, 0.9)",
+ "rgba(237, 129, 40, 0.89)",
+ "rgba(50, 172, 45, 0.97)"
+ ],
+ "dateFormat": "YYYY-MM-DD HH:mm:ss",
+ "decimals": 2,
+ "link": true,
+ "linkTargetBlank": true,
+ "linkTooltip": "Link to Jenkins job.",
+ "linkUrl": "${__cell:raw}",
+ "mappingType": 1,
+ "pattern": "build_url",
+ "thresholds": [],
+ "type": "number",
+ "unit": "short"
+ }
+ ],
+ "targets": [
+ {
+ "alias": "",
+ "format": "table",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT \n build_timestamp,\n job_name,\n build_url\nFROM jenkins_builds\nWHERE \n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND ((job_name LIKE 'beam_PostCommit_Java_GradleBuild') \n OR (job_name LIKE 'beam_PostCommit_Go_GradleBuild') \n OR (job_name LIKE 'beam_PostCommit_Python_Verify')\n OR (job_name LIKE 'beam_PostCommit_Website_Publish'))\n AND NOT (job_name LIKE '%_PR')\n AND NOT (build_result = 'SUCCESS')\nORDER BY \n build_timestamp",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "timeShift": null,
+ "title": "Failed builds",
+ "transform": "table",
+ "type": "table"
+ },
+ {
+ "content": "[List existing jira tickets](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20test-failures%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)\n\n[Create new Jira ticket](https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12319527&issuetype=1&summary=%5BjobName%5D%5BTestName%5D%5BIsFlake%5D%20Failure%20summary&priority=3&components=12334203&description=%3CFailure%20summary%3E%0AFailing%20job%20url:%0AJob%20history%20url:%0ARelevant%20log:)",
+ "gridPos": {
+ "h": 3,
+ "w": 10,
+ "x": 0,
+ "y": 3
+ },
+ "id": 6,
+ "links": [],
+ "mode": "markdown",
+ "options": {},
+ "title": "Useful links",
+ "type": "text"
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "description": "Each data point shows aggregation for corresponding week.\nLatest (rightmost) data point aggregates all data available for current week, so it may change based on new data and should not be considered a final value.",
+ "fill": 0,
+ "gridPos": {
+ "h": 7,
+ "w": 10,
+ "x": 0,
+ "y": 6
+ },
+ "id": 2,
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "options": {},
+ "percentage": false,
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n DATE_TRUNC('week', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n /*\n We perform a union here to create a fake \"Python_All\" job_name in\n order to graph a new line for all the python results combined.\n */\n ( SELECT build_timestamp, build_result, job_name\n FROM jenkins_builds\n UNION\n SELECT build_timestamp, build_result, 'beam_PostCommit_Python_All' as job_name\n FROM jenkins_builds\n WHERE \n ((job_name SIMILAR TO 'beam_PostCommit_Python[0-9]+'))\n AND NOT (job_name like '%_PR')\n ) AS critical_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND ((job_name = 'beam_PostCommit_Java') \n OR (job_name = 'beam_PostCommit_Go') \n OR (job_name SIMILAR TO 'beam_PostCommit_Python[0-9]+')\n OR (job_name = 'beam_PostCommit_Python_Verify')\n OR (job_name = 'beam_PostCommit_Python_All')\n OR (job_name = 'beam_PostCommit_Website_Publish'))\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "custom",
+ "fill": false,
+ "line": true,
+ "lineColor": "#3f6833",
+ "op": "lt",
+ "value": 0.7,
+ "yaxis": "left"
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Greenness per Week (in %)",
+ "tooltip": {
+ "shared": true,
+ "sort": 1,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": null,
+ "format": "percentunit",
+ "label": "",
+ "logBase": 1,
+ "max": "1",
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ },
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamPSQL",
+ "description": "Each data point shows aggregation for corresponding month.\nLatest (rightmost) data point aggregates all data available for current month, so it may change based on new data and should not be considered a final value.",
+ "fill": 0,
+ "gridPos": {
+ "h": 7,
+ "w": 14,
+ "x": 10,
+ "y": 6
+ },
+ "id": 10,
+ "legend": {
+ "alignAsTable": true,
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "rightSide": true,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 1,
+ "links": [],
+ "nullPointMode": "null",
+ "options": {},
+ "percentage": false,
+ "pointradius": 2,
+ "points": false,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "",
+ "format": "time_series",
+ "group": [],
+ "metricColumn": "none",
+ "rawQuery": true,
+ "rawSql": "SELECT\n DATE_TRUNC('month', build_timestamp) as time,\n avg(\n case \n when build_result = 'SUCCESS' then 1\n else 0\n end) as value,\n substring(job_name from 'beam_PostCommit_#\"%#\"' for '#') as job_name\nFROM\n /*\n We perform a union here to create a fake \"Python_All\" job_name in\n order to graph a new line for all the python results combined.\n */\n ( SELECT build_timestamp, build_result, job_name\n FROM jenkins_builds\n UNION\n SELECT build_timestamp, build_result, 'beam_PostCommit_Python_All' as job_name\n FROM jenkins_builds\n WHERE \n ((job_name SIMILAR TO 'beam_PostCommit_Python[0-9]+'))\n AND NOT (job_name like '%_PR')\n ) AS critical_builds\nWHERE\n (build_timestamp BETWEEN $__timeFrom() AND $__timeTo())\n AND ((job_name = 'beam_PostCommit_Java') \n OR (job_name = 'beam_PostCommit_Go') \n OR (job_name SIMILAR TO 'beam_PostCommit_Python[0-9]+')\n OR (job_name = 'beam_PostCommit_Python_Verify')\n OR (job_name = 'beam_PostCommit_Python_All')\n OR (job_name = 'beam_PostCommit_Website_Publish'))\n AND NOT (job_name like '%_PR')\nGROUP BY\n time, job_name\norder BY\n job_name, time",
+ "refId": "A",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "column"
+ }
+ ]
+ ],
+ "timeColumn": "time",
+ "where": [
+ {
+ "name": "$__timeFilter",
+ "params": [],
+ "type": "macro"
+ }
+ ]
+ }
+ ],
+ "thresholds": [
+ {
+ "colorMode": "custom",
+ "fill": false,
+ "line": true,
+ "lineColor": "#3f6833",
+ "op": "lt",
+ "value": 0.7,
+ "yaxis": "left"
+ }
+ ],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "Greenness per Month (in %)",
+ "tooltip": {
+ "shared": true,
+ "sort": 1,
+ "value_type": "individual"
+ },
+ "type": "graph",
+ "xaxis": {
+ "buckets": null,
+ "mode": "time",
+ "name": null,
+ "show": true,
+ "values": []
+ },
+ "yaxes": [
+ {
+ "decimals": null,
+ "format": "percentunit",
+ "label": "",
+ "logBase": 1,
+ "max": "1",
+ "min": "0",
+ "show": true
+ },
+ {
+ "format": "short",
+ "label": null,
+ "logBase": 1,
+ "max": null,
+ "min": null,
+ "show": false
+ }
+ ],
+ "yaxis": {
+ "align": false,
+ "alignLevel": null
+ }
+ }
+ ],
+ "refresh": false,
+ "schemaVersion": 18,
+ "style": "dark",
+ "tags": [],
+ "templating": {
+ "list": []
+ },
+ "time": {
+ "from": "now-90d",
+ "to": "now"
+ },
+ "timepicker": {
+ "refresh_intervals": [
+ "5s",
+ "10s",
+ "30s",
+ "1m",
+ "5m",
+ "15m",
+ "30m",
+ "1h",
+ "2h",
+ "1d"
+ ],
+ "time_options": [
+ "5m",
+ "15m",
+ "1h",
+ "6h",
+ "12h",
+ "24h",
+ "2d",
+ "7d",
+ "30d"
+ ]
+ },
+ "timezone": "utc",
+ "title": "Stability critical jobs status",
+ "uid": "McTAiu0ik",
+ "version": 1
+}
\ No newline at end of file
diff --git a/.test-infra/metrics/grafana/provisioning/dashboards/all.yaml b/.test-infra/metrics/grafana/provisioning/dashboards/all.yaml
new file mode 100644
index 0000000..db8972e
--- /dev/null
+++ b/.test-infra/metrics/grafana/provisioning/dashboards/all.yaml
@@ -0,0 +1,32 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+apiVersion: 1
+
+providers:
+ - name: "All Beam dashboards"
+ orgId: 1
+ folder: ''
+ folderUid: ''
+ type: file
+ disableDeletion: true
+ editable: true
+ updateIntervalSeconds: 3600
+ options:
+ path: /etc/beamgrafana/dashboards
+
diff --git a/.test-infra/metrics/grafana/provisioning/datasources/beampostgresql.yaml b/.test-infra/metrics/grafana/provisioning/datasources/beampostgresql.yaml
new file mode 100644
index 0000000..6bcaa5f
--- /dev/null
+++ b/.test-infra/metrics/grafana/provisioning/datasources/beampostgresql.yaml
@@ -0,0 +1,35 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+apiVersion: 1
+
+deleteDatasources:
+
+datasources:
+ - name: BeamPSQL
+ type: postgres
+ orgId: 1
+ url: ${DB_HOST}:${DB_PORT}
+ secureJsonData:
+ password: ${DB_DBPWD}
+ user: ${DB_DBUSERNAME}
+ database: ${DB_DBNAME}
+ jsonData:
+ sslmode: disable
+ editable: false
+
diff --git a/.test-infra/metrics/postgres/init.sql b/.test-infra/metrics/postgres/init.sql
new file mode 100644
index 0000000..e0c9d32
--- /dev/null
+++ b/.test-infra/metrics/postgres/init.sql
@@ -0,0 +1,18 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+
+CREATE extension tablefunc;
+
diff --git a/.test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml b/.test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml
new file mode 100644
index 0000000..fe0b677
--- /dev/null
+++ b/.test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml
@@ -0,0 +1,41 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# A configuration file for alertmanager.
+# Can be reloaded at runtime by sending a SIGHUP signal to the alertmanager
+# process or sending a HTTP POST request to the /reload endpoint.
+
+global:
+ resolve_timeout: 7d
+
+route:
+ receiver: 'default'
+ group_by: ['alertname']
+ group_wait: 0s
+ group_interval: 3d
+ repeat_interval: 3d
+ routes:
+ - match_re:
+ job: 'beam'
+ receiver: 'emails-and-slack'
+ group_by: ['test']
+
+receivers:
+ - name: 'default'
+ # TODO: Add details about emails-and-slack receiver
+ - name: 'emails-and-slack'
diff --git a/.test-infra/metrics/prometheus/prometheus/config/prometheus.yml b/.test-infra/metrics/prometheus/prometheus/config/prometheus.yml
new file mode 100644
index 0000000..c0e5b61
--- /dev/null
+++ b/.test-infra/metrics/prometheus/prometheus/config/prometheus.yml
@@ -0,0 +1,40 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# A configuration file for the main Prometheus server.
+# Can be reloaded at runtime by sending a SIGHUP signal to the Prometheus
+# process.
+
+global:
+ scrape_interval: 6h
+ evaluation_interval: 1m
+
+rule_files:
+ - 'rules.yml'
+
+scrape_configs:
+ - job_name: 'beam'
+ honor_labels: true
+ honor_timestamps: true
+ static_configs:
+ - targets: ['pushgateway:9091']
+
+alerting:
+ alertmanagers:
+ - static_configs:
+ - targets: ['alertmanager:9093']
diff --git a/.test-infra/metrics/prometheus/prometheus/config/rules.yml b/.test-infra/metrics/prometheus/prometheus/config/rules.yml
new file mode 100644
index 0000000..45bcfa4
--- /dev/null
+++ b/.test-infra/metrics/prometheus/prometheus/config/rules.yml
@@ -0,0 +1,35 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# Defines alerting rules used by Prometheus to detect anomalous behaviours
+# among test results.
+# Can be reloaded at runtime by sending a SIGHUP signal to the Prometheus
+# process.
+
+groups:
+- name: beamTests
+ rules:
+ - alert: TestRegression
+ expr: ((avg_over_time({job="beam",instance="",__name__!="push_time_seconds"}[1d])
+ - avg_over_time({job="beam",instance="",__name__!="push_time_seconds"}[6d] offset 1d))
+ / avg_over_time({job="beam",instance="",__name__!="push_time_seconds"}[6d] offset 1d))
+ > 0.2
+ labels:
+ job: beamAlert
+ annotations:
+ summary: 'Average runtime over 24 hours is 20% greater than average from six previous days'
diff --git a/.test-infra/metrics/sync/github/sync.py b/.test-infra/metrics/sync/github/sync.py
index 930275b..c23f4f7 100644
--- a/.test-infra/metrics/sync/github/sync.py
+++ b/.test-infra/metrics/sync/github/sync.py
@@ -38,9 +38,9 @@
return cmd_out.split(" ")[2]
-DB_HOST = findDockerNetworkIP()
+#DB_HOST = findDockerNetworkIP()
-# DB_HOST = os.environ['DB_HOST']
+DB_HOST = os.environ['DB_HOST']
DB_PORT = os.environ['DB_PORT']
DB_NAME = os.environ['DB_DBNAME']
DB_USER_NAME = os.environ['DB_DBUSERNAME']
diff --git a/.test-infra/metrics/sync/jenkins/syncjenkins.py b/.test-infra/metrics/sync/jenkins/syncjenkins.py
index f89c4b4..f4d4fc6 100644
--- a/.test-infra/metrics/sync/jenkins/syncjenkins.py
+++ b/.test-infra/metrics/sync/jenkins/syncjenkins.py
@@ -27,11 +27,12 @@
# cmd_out = subprocess.check_output(["ip", "route", "show"]).decode("utf-8")
# host = cmd_out.split(" ")[2]
-host = os.environ['JENSYNC_HOST']
-port = os.environ['JENSYNC_PORT']
-dbname = os.environ['JENSYNC_DBNAME']
-dbusername = os.environ['JENSYNC_DBUSERNAME']
-dbpassword = os.environ['JENSYNC_DBPWD']
+host = os.environ['DB_HOST']
+port = os.environ['DB_PORT']
+dbname = os.environ['DB_DBNAME']
+dbusername = os.environ['DB_DBUSERNAME']
+dbpassword = os.environ['DB_DBPWD']
+
jenkinsBuildsTableName = 'jenkins_builds'
diff --git a/README.md b/README.md
index 423f8ac..8d7b9ee 100644
--- a/README.md
+++ b/README.md
@@ -19,7 +19,7 @@
# Apache Beam
-[Apache Beam](http://beam.apache.org/) is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends, including [Apache Apex](http://apex.apache.org/), [Apache Flink](http://flink.apache.org/), [Apache Spark](http://spark.apache.org/), and [Google Cloud Dataflow](http://cloud.google.com/dataflow/).
+[Apache Beam](http://beam.apache.org/) is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends, including [Apache Apex](http://apex.apache.org/), [Apache Flink](http://flink.apache.org/), [Apache Spark](http://spark.apache.org/), [Google Cloud Dataflow](http://cloud.google.com/dataflow/) and [Hazelcast Jet](https://jet.hazelcast.org/).
## Status
@@ -27,14 +27,17 @@
[](https://badge.fury.io/py/apache-beam)
[](https://builds.apache.org/job/beam_PostCommit_Java)
[](https://coveralls.io/github/apache/beam?branch=master)
+[](https://python-compatibility-tools.appspot.com/one_badge_target?package=apache-beam%5Bgcp%5D)
+[](https://python-compatibility-tools.appspot.com/one_badge_target?package=git%2Bgit%3A//github.com/apache/beam.git%23subdirectory%3Dsdks/python)
### Post-commit tests status (on master branch)
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
-Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
-Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
-Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
+Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
+Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
+Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
+XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
## Overview
@@ -74,6 +77,7 @@
- The `DataflowRunner` submits the pipeline to the [Google Cloud Dataflow](http://cloud.google.com/dataflow/).
- The `FlinkRunner` runs the pipeline on an Apache Flink cluster. The code has been donated from [dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) and is now part of Beam.
- The `SparkRunner` runs the pipeline on an Apache Spark cluster. The code has been donated from [cloudera/spark-dataflow](https://github.com/cloudera/spark-dataflow) and is now part of Beam.
+- The `JetRunner` runs the pipeline on a Hazelcast Jet cluster. The code has been donated from [hazelcast/hazelcast-jet](https://github.com/hazelcast/hazelcast-jet) and is now part of Beam.
Have ideas for new Runners? See the [JIRA](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-ideas).
diff --git a/build.gradle b/build.gradle
index 9e9fba1..299bde2 100644
--- a/build.gradle
+++ b/build.gradle
@@ -19,12 +19,12 @@
plugins {
id 'base'
// Enable publishing build scans
- id 'com.gradle.build-scan' version '2.1' apply false
+ id 'com.gradle.build-scan' version '2.3' apply false
// This plugin provides a task to determine which dependencies have updates.
// Additionally, the plugin checks for updates to Gradle itself.
//
// See https://github.com/ben-manes/gradle-versions-plugin for further details.
- id 'com.github.ben-manes.versions' version '0.17.0'
+ id 'com.github.ben-manes.versions' version '0.20.0'
// Apply one top level rat plugin to perform any required license enforcement analysis
id 'org.nosphere.apache.rat' version '0.4.0'
// Enable gradle-based release management
@@ -33,10 +33,6 @@
id "org.sonarqube" version "2.7"
}
-// Add performanceTest task to this build.gradle file
-// so that running Performance tests using PerfKitBenchmarker is possible.
-createPerformanceTestHarness()
-
/*************************************************************************************************/
// Configure the root project
@@ -100,12 +96,18 @@
"ownership/**/*",
"**/OWNERS",
- // FIXME add licencse header
+ // FIXME add license header
"project-mappings",
"deprecation-warning.txt",
// Json doesn't support comments.
"**/*.json",
+
+ // Katas files
+ "learning/katas/*/IO/**/*.txt",
+
+ // Mockito extensions
+ "sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker"
]
// Add .gitignore excludes to the Apache Rat exclusion list. We re-create the behavior
@@ -141,6 +143,10 @@
dependsOn ":runners:direct-java:needsRunnerTests"
}
+task javaPreCommitBeamZetaSQL() {
+ dependsOn ":sdks:java:extensions:sql:zetasql:test"
+}
+
task javaPreCommitPortabilityApi() {
dependsOn ":runners:google-cloud-dataflow-java:worker:build"
dependsOn ":runners:google-cloud-dataflow-java:examples:verifyPortabilityApi"
@@ -149,12 +155,14 @@
task javaPostCommit() {
dependsOn ":runners:google-cloud-dataflow-java:postCommit"
dependsOn ":sdks:java:extensions:google-cloud-platform-core:postCommit"
+ dependsOn ":sdks:java:extensions:zetasketch:postCommit"
dependsOn ":sdks:java:io:google-cloud-platform:postCommit"
}
task sqlPostCommit() {
dependsOn ":sdks:java:extensions:sql:postCommit"
dependsOn ":sdks:java:extensions:sql:jdbc:postCommit"
+ dependsOn ":sdks:java:extensions:sql:datacatalog:postCommit"
}
task javaPostCommitPortabilityApi () {
@@ -164,13 +172,13 @@
task goPreCommit() {
dependsOn ":sdks:go:goTest"
- dependsOn ":sdks:go:examples:build"
- dependsOn ":sdks:go:test:build"
+ dependsOn ":sdks:go:examples:goBuild"
+ dependsOn ":sdks:go:test:goBuild"
// Ensure all container Go boot code builds as well.
- dependsOn ":sdks:java:container:build"
- dependsOn ":sdks:python:container:build"
- dependsOn ":sdks:go:container:build"
+ dependsOn ":sdks:java:container:goBuild"
+ dependsOn ":sdks:python:container:goBuild"
+ dependsOn ":sdks:go:container:goBuild"
}
task goPostCommit() {
@@ -189,29 +197,51 @@
}
task pythonPreCommit() {
- dependsOn ":sdks:python:preCommitPy2"
+ dependsOn ":sdks:python:test-suites:tox:py2:preCommitPy2"
dependsOn ":sdks:python:test-suites:tox:py35:preCommitPy35"
dependsOn ":sdks:python:test-suites:tox:py36:preCommitPy36"
dependsOn ":sdks:python:test-suites:tox:py37:preCommitPy37"
- dependsOn ":sdks:python:test-suites:dataflow:preCommitIT"
+ dependsOn ":sdks:python:test-suites:dataflow:py2:preCommitIT"
+ dependsOn ":sdks:python:test-suites:dataflow:py37:preCommitIT"
+ // We don't include Py35, Py36 precommit ITs to reduce quota footprint.
+ // We can reconsider if we ever see an issue that these suites would
+ // have caught. Note that the same tests will still run in postcommit.
}
-task pythonPostCommit() {
- dependsOn ":sdks:python:postCommit"
+task pythonLintPreCommit() {
+ dependsOn ":sdks:python:test-suites:tox:py2:lint"
+ dependsOn ":sdks:python:test-suites:tox:py35:lint"
}
-task python3PostCommit() {
+task python2PostCommit() {
+ dependsOn ":sdks:python:test-suites:portable:py2:crossLanguageTests"
+ dependsOn ":sdks:python:test-suites:dataflow:py2:postCommitIT"
+ dependsOn ":sdks:python:test-suites:direct:py2:directRunnerIT"
+ dependsOn ":sdks:python:test-suites:direct:py2:hdfsIntegrationTest"
+ dependsOn ":sdks:python:test-suites:direct:py2:mongodbioIT"
+}
+
+task python35PostCommit() {
dependsOn ":sdks:python:test-suites:dataflow:py35:postCommitIT"
- dependsOn ":sdks:python:test-suites:dataflow:py36:postCommitIT"
- dependsOn ":sdks:python:test-suites:dataflow:py37:postCommitIT"
- dependsOn ":sdks:python:test-suites:dataflow:py35:validatesRunnerBatchTests"
dependsOn ":sdks:python:test-suites:direct:py35:postCommitIT"
+}
+
+task python36PostCommit() {
+ dependsOn ":sdks:python:test-suites:dataflow:py36:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py36:postCommitIT"
+}
+
+task python37PostCommit() {
+ dependsOn ":sdks:python:test-suites:dataflow:py37:postCommitIT"
dependsOn ":sdks:python:test-suites:direct:py37:postCommitIT"
+ dependsOn ":sdks:python:test-suites:direct:py37:hdfsIntegrationTest"
}
task portablePythonPreCommit() {
- dependsOn ":sdks:python:portablePreCommit"
+ dependsOn ":sdks:python:test-suites:portable:py2:preCommitPy2"
+ dependsOn ":sdks:python:test-suites:portable:py35:preCommitPy35"
+ dependsOn ":sdks:python:test-suites:portable:py36:preCommitPy36"
+ dependsOn ":sdks:python:test-suites:portable:py37:preCommitPy37"
}
task websitePreCommit() {
diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
index 593bf80..27fde76 100644
--- a/buildSrc/build.gradle
+++ b/buildSrc/build.gradle
@@ -20,7 +20,7 @@
plugins {
id 'java-gradle-plugin'
id 'groovy'
- id "com.diffplug.gradle.spotless" version "3.17.0"
+ id "com.diffplug.gradle.spotless" version "3.24.0"
}
// Define the set of repositories required to fetch and enable plugins.
@@ -38,14 +38,14 @@
compile gradleApi()
compile localGroovy()
compile 'com.github.jengelman.gradle.plugins:shadow:4.0.3'
- compile 'gradle.plugin.com.github.spotbugs:spotbugs-gradle-plugin:1.6.9' // Enable spotbugs
+ compile 'gradle.plugin.com.github.spotbugs:spotbugs-gradle-plugin:2.0.0' // Enable spotbugs
runtime "net.ltgt.gradle:gradle-apt-plugin:0.20" // Enable a Java annotation processor
runtime "com.google.protobuf:protobuf-gradle-plugin:0.8.5" // Enable proto code generation
runtime "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE" // Enable provided and optional configurations
runtime "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0" // Enable Avro code generation
- runtime "com.diffplug.spotless:spotless-plugin-gradle:3.17.0" // Enable a code formatting plugin
- runtime "gradle.plugin.com.github.blindpirate:gogradle:0.11.2" // Enable Go code compilation
+ runtime "com.diffplug.spotless:spotless-plugin-gradle:3.24.0" // Enable a code formatting plugin
+ runtime "gradle.plugin.com.github.blindpirate:gogradle:0.11.4" // Enable Go code compilation
runtime "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.20.1" // Enable building Docker containers
runtime "gradle.plugin.com.dorongold.plugins:task-tree:1.3.1" // Adds a 'taskTree' task to print task dependency tree
runtime "com.github.jengelman.gradle.plugins:shadow:4.0.3" // Enable shading Java dependencies
@@ -54,8 +54,8 @@
runtime "net.ltgt.gradle:gradle-errorprone-plugin:0.0.13" // Enable errorprone Java static analysis
runtime "org.ajoberstar.grgit:grgit-gradle:3.0.0" // Enable website git publish to asf-site branch
runtime "com.avast.gradle:gradle-docker-compose-plugin:0.8.8" // Enable docker compose tasks
- runtime "ca.cutterslade.gradle:gradle-dependency-analyze:1.3.0" // Enable dep analysis
- runtime "gradle.plugin.net.ossindex:ossindex-gradle-plugin:0.4.11"
+ runtime "ca.cutterslade.gradle:gradle-dependency-analyze:1.3.1" // Enable dep analysis
+ runtime "gradle.plugin.net.ossindex:ossindex-gradle-plugin:0.4.11" // Enable dep vulnerability analysis
}
// Because buildSrc is built and tested automatically _before_ gradle
diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index fd62f2c..b7e9d2a 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -87,6 +87,9 @@
/** Controls whether the dependency analysis plugin is enabled. */
boolean enableStrictDependencies = false
+ /** Override the default "beam-" + `dash separated path` archivesBaseName. */
+ String archivesBaseName = null;
+
/**
* List of additional lint warnings to disable.
* In addition, defaultLintSuppressions defined below
@@ -115,14 +118,10 @@
List<String> shadowJarValidationExcludes = ["org/apache/beam/**"]
/**
- * The shadowJar / shadowTestJar tasks execute the following closure to configure themselves.
- * Users can compose their closure with the default closure via:
- * DEFAULT_SHADOW_CLOSURE << {
- * dependencies {
- * include(...)
- * }
- * relocate(...)
- * }
+ * If unset, no shading is performed. The jar and test jar archives are used during publishing.
+ * Otherwise the shadowJar and shadowTestJar artifacts are used during publishing.
+ *
+ * The shadowJar / shadowTestJar tasks execute the specified closure to configure themselves.
*/
Closure shadowClosure;
@@ -131,6 +130,14 @@
/** Controls whether javadoc is exported for this project. */
boolean exportJavadoc = true
+
+ /**
+ * Automatic-Module-Name Header value to be set in MANFIEST.MF file.
+ * This is a required parameter unless publishing to Maven is disabled for this project.
+ *
+ * @see: https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md
+ */
+ String automaticModuleName = null
}
/** A class defining the set of configurable properties accepted by applyPortabilityNature. */
@@ -142,6 +149,20 @@
* By default we exclude any class underneath the org.apache.beam namespace.
*/
List<String> shadowJarValidationExcludes = ["org/apache/beam/**"]
+
+ /** Override the default "beam-" + `dash separated path` archivesBaseName. */
+ String archivesBaseName = null;
+
+ /** Controls whether this project is published to Maven. */
+ boolean publish = true
+
+ /**
+ * Automatic-Module-Name Header value to be set in MANFIEST.MF file.
+ * This is a required parameter unless publishing to Maven is disabled for this project.
+ *
+ * @see: https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md
+ */
+ String automaticModuleName
}
// A class defining the set of configurable properties for createJavaExamplesArchetypeValidationTask
@@ -169,72 +190,16 @@
// Reads and contains all necessary performance test parameters
class JavaPerformanceTestConfiguration {
-
- /* Optional properties (set only if needed in your case): */
-
- // Path to PerfKitBenchmarker application (pkb.py).
- // It is only required when running Performance Tests with PerfKitBenchmarker
- String pkbLocation = System.getProperty('pkbLocation')
-
- // Data Processing Backend's log level.
- String logLevel = System.getProperty('logLevel', 'INFO')
-
- // Path to gradle binary.
- String gradleBinary = System.getProperty('gradleBinary', './gradlew')
-
- // If benchmark is official or not.
- // Official benchmark results are meant to be displayed on PerfKitExplorer dashboards.
- String isOfficial = System.getProperty('official', 'false')
-
- // Specifies names of benchmarks to be run by PerfKitBenchmarker.
- String benchmarks = System.getProperty('benchmarks', 'beam_integration_benchmark')
-
- // If beam is not "prebuilt" then PerfKitBenchmarker runs the build task before running the tests.
- String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
-
- // Beam's sdk to be used by PerfKitBenchmarker.
- String beamSdk = System.getProperty('beamSdk', 'java')
-
- // Timeout (in seconds) after which PerfKitBenchmarker will stop executing the benchmark (and will fail).
- String timeout = System.getProperty('itTimeout', '1200')
-
- // Path to kubernetes configuration file.
- String kubeconfig = System.getProperty('kubeconfig', System.getProperty('user.home') + '/.kube/config')
-
- // Path to kubernetes executable.
- String kubectl = System.getProperty('kubectl', 'kubectl')
-
- // Paths to files with kubernetes infrastructure to setup before the test runs.
- // PerfKitBenchmarker will have trouble reading 'null' path. It expects empty string if no scripts are expected.
- String kubernetesScripts = System.getProperty('kubernetesScripts', '')
-
- // Path to file with 'dynamic' and 'static' pipeline options.
- // that will be appended by PerfKitBenchmarker to the test running command.
- // PerfKitBenchmarker will have trouble reading 'null' path. It expects empty string if no config file is expected.
- String optionsConfigFile = System.getProperty('beamITOptions', '')
-
- // Any additional properties to be appended to benchmark execution command.
- String extraProperties = System.getProperty('beamExtraProperties', '')
-
- // Runner which will be used for running the tests. Possible values: dataflow/direct.
+ // Optional. Runner which will be used for running the tests. Possible values: dataflow/direct.
// PerfKitBenchmarker will have trouble reading 'null' value. It expects empty string if no config file is expected.
String runner = System.getProperty('integrationTestRunner', '')
- // Filesystem which will be used for running the tests. Possible values: hdfs.
+ // Optional. Filesystem which will be used for running the tests. Possible values: hdfs.
// if not specified runner's local filesystem will be used.
String filesystem = System.getProperty('filesystem')
- /* Always required properties: */
-
- // Pipeline options to be used by the tested pipeline.
+ // Required. Pipeline options to be used by the tested pipeline.
String integrationTestPipelineOptions = System.getProperty('integrationTestPipelineOptions')
-
- // Fully qualified name of the test to be run, eg:
- // 'org.apache.beam.sdks.java.io.jdbc.JdbcIOIT'.
- String integrationTest = System.getProperty('integrationTest')
-
- // Relative path to module where the test is, eg. 'sdks/java/io/jdbc.
- String itModule = System.getProperty('itModule')
}
// Reads and contains all necessary performance test parameters
@@ -295,12 +260,35 @@
}
}
+ // A class defining the configuration for CrossLanguageValidatesRunner.
+ class CrossLanguageValidatesRunnerConfiguration {
+ // Task name for cross-language validate runner case.
+ String name = 'validatesCrossLanguageRunner'
+ // Fully qualified JobServerClass name to use.
+ String jobServerDriver
+ // A string representing the jobServer Configuration.
+ String jobServerConfig
+ // Number of parallel test runs.
+ Integer numParallelTests = 1
+ // Extra options to pass to TestPipeline
+ String[] pipelineOpts = []
+ // Categories for tests to run.
+ Closure testCategories = {
+ includeCategories 'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
+ // Use the following to include / exclude categories:
+ // includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ // excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
+ }
+ // Configuration for the classpath when running the test.
+ Configuration testClasspathConfiguration
+ }
+
def isRelease(Project project) {
return project.hasProperty('isRelease')
}
- def archivesBaseName(Project p) {
- 'beam' + p.path.replace(':', '-')
+ def defaultArchivesBaseName(Project p) {
+ return 'beam' + p.path.replace(':', '-')
}
void apply(Project project) {
@@ -312,13 +300,15 @@
// Automatically use the official release version if we are performing a release
// otherwise append '-SNAPSHOT'
- project.version = '2.14.0'
+ project.version = '2.17.0'
if (!isRelease(project)) {
project.version += '-SNAPSHOT'
}
+ // Default to dash-separated directories for artifact base name,
+ // which will also be the default artifactId for maven publications
project.apply plugin: 'base'
- project.archivesBaseName = archivesBaseName(project)
+ project.archivesBaseName = defaultArchivesBaseName(project)
project.apply plugin: 'org.apache.beam.jenkins'
@@ -348,7 +338,7 @@
// non-declared dependency, since these can break users (as in BEAM-6558)
//
// Though this is Java-specific, it is required to be applied to the root
- // project due to implemeentation-details of the plugin. It can be enabled/disabled
+ // project due to implementation-details of the plugin. It can be enabled/disabled
// via JavaNatureConfiguration per project. It is disabled by default until we can
// make all of our deps good.
project.apply plugin: "ca.cutterslade.analyze"
@@ -364,8 +354,8 @@
//
// Example usage:
// configuration {
- // shadow library.java.avro
- // shadowTest library.java.junit
+ // compile library.java.avro
+ // testCompile library.java.junit
// }
// These versions are defined here because they represent
@@ -374,10 +364,11 @@
def apex_core_version = "3.7.0"
def apex_malhar_version = "3.4.0"
def aws_java_sdk_version = "1.11.519"
+ def aws_java_sdk2_version = "2.5.71"
def cassandra_driver_version = "3.6.0"
def generated_grpc_beta_version = "0.44.0"
def generated_grpc_ga_version = "1.43.0"
- def generated_grpc_dc_beta_version = "0.4.0-alpha"
+ def generated_grpc_dc_beta_version = "0.27.0-alpha"
def google_auth_version = "0.12.0"
def google_clients_version = "1.27.0"
def google_cloud_bigdataoss_version = "1.9.16"
@@ -387,20 +378,21 @@
def guava_version = "20.0"
def hadoop_version = "2.7.3"
def hamcrest_version = "2.1"
- def jackson_version = "2.9.8"
+ def jackson_version = "2.9.10"
def jaxb_api_version = "2.2.12"
def kafka_version = "1.0.0"
def nemo_version = "0.1"
def netty_version = "4.1.30.Final"
def postgres_version = "42.2.2"
+ def powermock_version = "2.0.2"
def proto_google_common_protos_version = "1.12.0"
def protobuf_version = "3.6.0"
def quickcheck_version = "0.8"
- def spark_version = "2.4.3"
+ def spark_version = "2.4.4"
// A map of maps containing common libraries used per language. To use:
// dependencies {
- // shadow library.java.slf4j_api
+ // compile library.java.slf4j_api
// }
project.ext.library = [
java : [
@@ -420,26 +412,28 @@
avro_tests : "org.apache.avro:avro:1.8.2:tests",
aws_java_sdk_cloudwatch : "com.amazonaws:aws-java-sdk-cloudwatch:$aws_java_sdk_version",
aws_java_sdk_core : "com.amazonaws:aws-java-sdk-core:$aws_java_sdk_version",
+ aws_java_sdk_dynamodb : "com.amazonaws:aws-java-sdk-dynamodb:$aws_java_sdk_version",
aws_java_sdk_kinesis : "com.amazonaws:aws-java-sdk-kinesis:$aws_java_sdk_version",
aws_java_sdk_s3 : "com.amazonaws:aws-java-sdk-s3:$aws_java_sdk_version",
aws_java_sdk_sns : "com.amazonaws:aws-java-sdk-sns:$aws_java_sdk_version",
aws_java_sdk_sqs : "com.amazonaws:aws-java-sdk-sqs:$aws_java_sdk_version",
+ aws_java_sdk2_apache_client : "software.amazon.awssdk:apache-client:$aws_java_sdk2_version",
+ aws_java_sdk2_auth : "software.amazon.awssdk:auth:$aws_java_sdk2_version",
+ aws_java_sdk2_cloudwatch : "software.amazon.awssdk:cloudwatch:$aws_java_sdk2_version",
+ aws_java_sdk2_dynamodb : "software.amazon.awssdk:dynamodb:$aws_java_sdk2_version",
+ aws_java_sdk2_sdk_core : "software.amazon.awssdk:sdk-core:$aws_java_sdk2_version",
+ aws_java_sdk2_sns : "software.amazon.awssdk:sns:$aws_java_sdk2_version",
bigdataoss_gcsio : "com.google.cloud.bigdataoss:gcsio:$google_cloud_bigdataoss_version",
bigdataoss_util : "com.google.cloud.bigdataoss:util:$google_cloud_bigdataoss_version",
- bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.8.0",
- bigtable_protos : "com.google.api.grpc:grpc-google-cloud-bigtable-v2:$generated_grpc_beta_version",
- byte_buddy : "net.bytebuddy:byte-buddy:1.9.3",
cassandra_driver_core : "com.datastax.cassandra:cassandra-driver-core:$cassandra_driver_version",
cassandra_driver_mapping : "com.datastax.cassandra:cassandra-driver-mapping:$cassandra_driver_version",
commons_codec : "commons-codec:commons-codec:1.10",
- commons_compress : "org.apache.commons:commons-compress:1.18",
+ commons_compress : "org.apache.commons:commons-compress:1.19",
commons_csv : "org.apache.commons:commons-csv:1.4",
commons_io_1x : "commons-io:commons-io:1.3.2",
commons_io_2x : "commons-io:commons-io:2.5",
commons_lang3 : "org.apache.commons:commons-lang3:3.6",
commons_math3 : "org.apache.commons:commons-math3:3.6.1",
- datastore_v1_proto_client : "com.google.cloud.datastore:datastore-v1-proto-client:1.6.0",
- datastore_v1_protos : "com.google.api.grpc:proto-google-cloud-datastore-v1:$generated_grpc_beta_version",
error_prone_annotations : "com.google.errorprone:error_prone_annotations:2.0.15",
gax_grpc : "com.google.api:gax-grpc:1.38.0",
google_api_client : "com.google.api-client:google-api-client:$google_clients_version",
@@ -449,17 +443,18 @@
google_api_services_bigquery : "com.google.apis:google-api-services-bigquery:v2-rev20181104-$google_clients_version",
google_api_services_clouddebugger : "com.google.apis:google-api-services-clouddebugger:v2-rev20180801-$google_clients_version",
google_api_services_cloudresourcemanager : "com.google.apis:google-api-services-cloudresourcemanager:v1-rev20181015-$google_clients_version",
- google_api_services_dataflow : "com.google.apis:google-api-services-dataflow:v1b3-rev20190322-$google_clients_version",
+ google_api_services_dataflow : "com.google.apis:google-api-services-dataflow:v1b3-rev20190607-$google_clients_version",
google_api_services_pubsub : "com.google.apis:google-api-services-pubsub:v1-rev20181105-$google_clients_version",
google_api_services_storage : "com.google.apis:google-api-services-storage:v1-rev20181013-$google_clients_version",
google_auth_library_credentials : "com.google.auth:google-auth-library-credentials:$google_auth_version",
google_auth_library_oauth2_http : "com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
google_cloud_bigquery : "com.google.cloud:google-cloud-bigquery:$google_clients_version",
google_cloud_bigquery_storage : "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha",
- google_cloud_bigquery_storage_proto : "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:$generated_grpc_beta_version",
+ google_cloud_bigtable_client_core : "com.google.cloud.bigtable:bigtable-client-core:1.8.0",
google_cloud_core : "com.google.cloud:google-cloud-core:$google_cloud_core_version",
google_cloud_core_grpc : "com.google.cloud:google-cloud-core-grpc:$google_cloud_core_version",
google_cloud_dataflow_java_proto_library_all: "com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
+ google_cloud_datastore_v1_proto_client : "com.google.cloud.datastore:datastore-v1-proto-client:1.6.0",
google_cloud_spanner : "com.google.cloud:google-cloud-spanner:$google_cloud_spanner_version",
google_http_client : "com.google.http-client:google-http-client:$google_clients_version",
google_http_client_jackson : "com.google.http-client:google-http-client-jackson:$google_clients_version",
@@ -495,21 +490,25 @@
jackson_datatype_joda : "com.fasterxml.jackson.datatype:jackson-datatype-joda:$jackson_version",
jackson_module_scala : "com.fasterxml.jackson.module:jackson-module-scala_2.11:$jackson_version",
jaxb_api : "javax.xml.bind:jaxb-api:$jaxb_api_version",
- joda_time : "joda-time:joda-time:2.10.1",
- junit : "junit:junit:4.13-beta-1",
- kafka_2_11 : "org.apache.kafka:kafka_2.11:$kafka_version",
+ joda_time : "joda-time:joda-time:2.10.3",
+ junit : "junit:junit:4.13-beta-3",
+ kafka : "org.apache.kafka:kafka_2.11:$kafka_version",
kafka_clients : "org.apache.kafka:kafka-clients:$kafka_version",
malhar_library : "org.apache.apex:malhar-library:$apex_malhar_version",
- mockito_core : "org.mockito:mockito-core:1.10.19",
+ mockito_core : "org.mockito:mockito-core:3.0.0",
nemo_compiler_frontend_beam : "org.apache.nemo:nemo-compiler-frontend-beam:$nemo_version",
netty_handler : "io.netty:netty-handler:$netty_version",
netty_tcnative_boringssl_static : "io.netty:netty-tcnative-boringssl-static:2.0.17.Final",
netty_transport_native_epoll : "io.netty:netty-transport-native-epoll:$netty_version",
postgres : "org.postgresql:postgresql:$postgres_version",
- powermock : "org.powermock:powermock-mockito-release-full:1.6.4",
+ powermock : "org.powermock:powermock-module-junit4:$powermock_version",
+ powermock_mockito : "org.powermock:powermock-api-mockito2:$powermock_version",
protobuf_java : "com.google.protobuf:protobuf-java:$protobuf_version",
protobuf_java_util : "com.google.protobuf:protobuf-java-util:$protobuf_version",
+ proto_google_cloud_bigquery_storage_v1beta1 : "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:$generated_grpc_beta_version",
+ proto_google_cloud_bigtable_v2 : "com.google.api.grpc:proto-google-cloud-bigtable-v2:$generated_grpc_beta_version",
proto_google_cloud_datacatalog_v1beta1 : "com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1:$generated_grpc_dc_beta_version",
+ proto_google_cloud_datastore_v1 : "com.google.api.grpc:proto-google-cloud-datastore-v1:$generated_grpc_beta_version",
proto_google_cloud_pubsub_v1 : "com.google.api.grpc:proto-google-cloud-pubsub-v1:$generated_grpc_ga_version",
proto_google_cloud_spanner_admin_database_v1: "com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:$google_cloud_spanner_version",
proto_google_common_protos : "com.google.api.grpc:proto-google-common-protos:$proto_google_common_protos_version",
@@ -522,8 +521,10 @@
spark_network_common : "org.apache.spark:spark-network-common_2.11:$spark_version",
spark_streaming : "org.apache.spark:spark-streaming_2.11:$spark_version",
stax2_api : "org.codehaus.woodstox:stax2-api:3.1.4",
- vendored_grpc_1_13_1 : "org.apache.beam:beam-vendor-grpc-1_13_1:0.2",
- vendored_guava_20_0 : "org.apache.beam:beam-vendor-guava-20_0:0.1",
+ vendored_bytebuddy_1_9_3 : "org.apache.beam:beam-vendor-bytebuddy-1_9_3:0.1",
+ vendored_grpc_1_21_0 : "org.apache.beam:beam-vendor-grpc-1_21_0:0.1",
+ vendored_guava_26_0_jre : "org.apache.beam:beam-vendor-guava-26_0-jre:0.1",
+ vendored_calcite_1_20_0 : "org.apache.beam:beam-vendor-calcite-1_20_0:0.1",
woodstox_core_asl : "org.codehaus.woodstox:woodstox-core-asl:4.4.1",
zstd_jni : "com.github.luben:zstd-jni:1.3.8-3",
quickcheck_core : "com.pholser:junit-quickcheck-core:$quickcheck_version",
@@ -552,25 +553,6 @@
+ suffix)
}
- // By default if there is at least one include rule then all included dependencies must be specified.
- // This overrides the default behavior of include all if no includes are specified.
- // See details here:
- // https://github.com/johnrengelman/shadow/blob/98191096a94674245c7b3e63975df9e14f67074e/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/internal/DefaultDependencyFilter.groovy#L123
- project.ext.DEFAULT_SHADOW_CLOSURE = {
- dependencies {
- include(dependency(project.library.java.guava))
- }
- // guava uses the com.google.common and com.google.thirdparty package namespaces
- relocate("com.google.common", project.getJavaRelocatedPath("com.google.common")) {
- // com.google.common is too generic, need to exclude guava-testlib
- exclude "com.google.common.collect.testing.**"
- exclude "com.google.common.escape.testing.**"
- exclude "com.google.common.testing.**"
- exclude "com.google.common.util.concurrent.testing.**"
- }
- relocate "com.google.thirdparty", project.getJavaRelocatedPath("com.google.thirdparty")
- }
-
project.ext.repositories = {
maven {
name "testPublicationLocal"
@@ -580,7 +562,6 @@
url(project.properties['distMgmtSnapshotsUrl'] ?: isRelease(project)
? 'https://repository.apache.org/service/local/staging/deploy/maven2'
: 'https://repository.apache.org/content/repositories/snapshots')
-
// We attempt to find and load credentials from ~/.m2/settings.xml file that a user
// has configured with the Apache release and snapshot staging credentials.
// <settings>
@@ -628,15 +609,19 @@
// * propdeps-idea
// * checkstyle
// * spotbugs
- // * shadow
+ // * shadow (conditional on shadowClosure being specified)
// * com.diffplug.gradle.spotless (code style plugin)
//
// Dependency Management for Java Projects
// ---------------------------------------
//
- // By default, the shadow plugin is enabled to perform shading of commonly found dependencies.
- // Because of this it is important that dependencies are added to the correct configuration.
- // Dependencies should fall into one of these four configurations:
+ // By default, the shadow plugin is not enabled. It is only enabled by specifying a shadowClosure
+ // as an argument. If no shadowClosure has been specified, dependencies should fall into the
+ // configurations as described within the Gradle documentation (https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_plugin_and_dependency_management)
+ //
+ // When the shadowClosure argument is specified, the shadow plugin is enabled to perform shading
+ // of commonly found dependencies. Because of this it is important that dependencies are added
+ // to the correct configuration. Dependencies should fall into one of these four configurations:
// * compile - Required during compilation or runtime of the main source set.
// This configuration represents all dependencies that much also be shaded away
// otherwise the generated Maven pom will be missing this dependency.
@@ -648,21 +633,23 @@
// TODO: Figure out whether this should be a test scope dependency
// of the generated Maven pom.
//
- // When creating a cross-project dependency between two Java projects, one should only rely on the shaded configurations.
- // This allows for compilation/test execution to occur against the final artifact that will be provided to users.
- // This is by done by referencing the "shadow" or "shadowTest" configuration as so:
+ // When creating a cross-project dependency between two Java projects, one should only rely on
+ // the shaded configurations if the project has a shadowClosure being specified. This allows
+ // for compilation/test execution to occur against the final artifact that will be provided to
+ // users. This is by done by referencing the "shadow" or "shadowTest" configuration as so:
// dependencies {
// shadow project(path: "other:java:project1", configuration: "shadow")
// shadowTest project(path: "other:java:project2", configuration: "shadowTest")
// }
- // This will ensure the correct set of transitive dependencies from those projects are correctly added to the
- // main and test source set runtimes.
+ // This will ensure the correct set of transitive dependencies from those projects are correctly
+ // added to the main and test source set runtimes.
project.ext.applyJavaNature = {
// Use the implicit it parameter of the closure to handle zero argument or one argument map calls.
JavaNatureConfiguration configuration = it ? it as JavaNatureConfiguration : new JavaNatureConfiguration()
- if (!configuration.shadowClosure) {
- configuration.shadowClosure = project.DEFAULT_SHADOW_CLOSURE
+
+ if (configuration.archivesBaseName) {
+ project.archivesBaseName = configuration.archivesBaseName
}
project.apply plugin: "java"
@@ -711,6 +698,9 @@
include "**/*Test.class"
include "**/*Tests.class"
include "**/*TestCase.class"
+ // fixes issues with test filtering on multi-module project
+ // see https://discuss.gradle.org/t/multi-module-build-fails-with-tests-filter/25835
+ filter { setFailOnNoMatchingTests(false) }
}
project.tasks.withType(Test) {
@@ -720,52 +710,56 @@
maxHeapSize = '2g'
}
- // Ensure that tests are packaged and part of the artifact set.
- project.task('packageTests', type: Jar) {
- classifier = 'tests-unshaded'
- from project.sourceSets.test.output
+ if (configuration.shadowClosure) {
+ // Ensure that tests are packaged and part of the artifact set.
+ project.task('packageTests', type: Jar) {
+ classifier = 'tests-unshaded'
+ from project.sourceSets.test.output
+ }
+ project.artifacts.archives project.packageTests
}
- project.artifacts.archives project.packageTests
// Configures annotation processing for commonly used annotation processors
// across all Java projects.
project.apply plugin: "net.ltgt.apt"
// let idea apt plugin handle the ide integration
project.apply plugin: "net.ltgt.apt-idea"
- project.dependencies {
- // Note that these plugins specifically use the compileOnly and testCompileOnly
- // configurations because they are never required to be shaded or become a
- // dependency of the output.
- def auto_value = "com.google.auto.value:auto-value:1.6.3"
- def auto_value_annotations = "com.google.auto.value:auto-value-annotations:1.6.3"
- def auto_service = "com.google.auto.service:auto-service:1.0-rc2"
- compileOnly auto_value_annotations
- testCompileOnly auto_value_annotations
- annotationProcessor auto_value
- testAnnotationProcessor auto_value
-
- compileOnly auto_service
- testCompileOnly auto_service
- annotationProcessor auto_service
- testAnnotationProcessor auto_service
-
+ // Note that these plugins specifically use the compileOnly and testCompileOnly
+ // configurations because they are never required to be shaded or become a
+ // dependency of the output.
+ def compileOnlyAnnotationDeps = [
+ "com.google.auto.value:auto-value-annotations:1.6.3",
+ "com.google.auto.service:auto-service-annotations:1.0-rc6",
+ "com.google.j2objc:j2objc-annotations:1.3",
// These dependencies are needed to avoid error-prone warnings on package-info.java files,
// also to include the annotations to suppress warnings.
//
// spotbugs-annotations artifact is licensed under LGPL and cannot be included in the
// Apache Beam distribution, but may be relied on during build.
// See: https://www.apache.org/legal/resolved.html#prohibited
- def spotbugs_annotations = "com.github.spotbugs:spotbugs-annotations:3.1.11"
- def jcip_annotations = "net.jcip:jcip-annotations:1.0"
- compileOnly spotbugs_annotations
- compileOnly jcip_annotations
- testCompileOnly spotbugs_annotations
- testCompileOnly jcip_annotations
- annotationProcessor spotbugs_annotations
- annotationProcessor jcip_annotations
- testAnnotationProcessor spotbugs_annotations
- testAnnotationProcessor jcip_annotations
+ "com.github.spotbugs:spotbugs-annotations:3.1.12",
+ "net.jcip:jcip-annotations:1.0",
+ ]
+
+ project.dependencies {
+ compileOnlyAnnotationDeps.each { dep ->
+ compileOnly dep
+ testCompileOnly dep
+ annotationProcessor dep
+ testAnnotationProcessor dep
+ }
+
+ // Add common annotation processors to all Java projects
+ def annotationProcessorDeps = [
+ "com.google.auto.value:auto-value:1.6.3",
+ "com.google.auto.service:auto-service:1.0-rc6",
+ ]
+
+ annotationProcessorDeps.each { dep ->
+ annotationProcessor dep
+ testAnnotationProcessor dep
+ }
}
// Add the optional and provided configurations for dependencies
@@ -787,7 +781,7 @@
showViolations = true
maxErrors = 0
}
- project.checkstyle { toolVersion = "8.7" }
+ project.checkstyle { toolVersion = "8.23" }
// Configures javadoc plugin and ensure check runs javadoc.
project.tasks.withType(Javadoc) {
@@ -804,7 +798,6 @@
project.apply plugin: "net.ltgt.apt-eclipse"
// Enables a plugin which can apply code formatting to source.
- // TODO(https://issues.apache.org/jira/browse/BEAM-4394): Should this plugin be enabled for all projects?
project.apply plugin: "com.diffplug.gradle.spotless"
// scan CVE
project.apply plugin: "net.ossindex.audit"
@@ -818,13 +811,7 @@
java {
licenseHeader javaLicenseHeader
googleJavaFormat('1.7')
- target project.fileTree(project.projectDir) {
- include '**/*.java'
- exclude '**/archetype-resources/src/**'
- exclude '**/build/generated/**'
- exclude '**/build/generated-src/**'
- exclude '**/build/generated-*-avro-*/**'
- }
+ target project.fileTree(project.projectDir) { include 'src/*/java/**/*.java' }
}
}
@@ -832,6 +819,11 @@
// This plugin is configured to only analyze the "main" source set.
if (configuration.enableSpotbugs) {
project.apply plugin: 'com.github.spotbugs'
+ project.dependencies {
+ spotbugs "com.github.spotbugs:spotbugs:3.1.12"
+ spotbugs "com.google.auto.value:auto-value:1.6.3"
+ compileOnlyAnnotationDeps.each { dep -> spotbugs dep }
+ }
project.spotbugs {
excludeFilter = project.rootProject.file('sdks/java/build-tools/src/main/resources/beam/spotbugs-filter.xml')
sourceSets = [sourceSets.main]
@@ -844,6 +836,15 @@
}
}
+ // Disregard unused but declared (test) compile only dependencies used
+ // for common annotation classes used during compilation such as annotation
+ // processing or post validation such as spotbugs.
+ project.dependencies {
+ compileOnlyAnnotationDeps.each { dep ->
+ permitUnusedDeclared dep
+ permitTestUnusedDeclared dep
+ }
+ }
if (configuration.enableStrictDependencies) {
project.tasks.analyzeClassesDependencies.enabled = true
project.tasks.analyzeDependencies.enabled = true
@@ -859,97 +860,122 @@
project.configurations.errorprone { resolutionStrategy.force 'com.google.errorprone:error_prone_core:2.3.1' }
- // Enables a plugin which can perform shading of classes. See the general comments
- // above about dependency management for Java projects and how the shadow plugin
- // is expected to be used for the different Gradle configurations.
- //
- // TODO: Enforce all relocations are always performed to:
- // getJavaRelocatedPath(package_suffix) where package_suffix is something like "com.google.commmon"
- project.apply plugin: 'com.github.johnrengelman.shadow'
+ if (configuration.shadowClosure) {
+ // Enables a plugin which can perform shading of classes. See the general comments
+ // above about dependency management for Java projects and how the shadow plugin
+ // is expected to be used for the different Gradle configurations.
+ //
+ // TODO: Enforce all relocations are always performed to:
+ // getJavaRelocatedPath(package_suffix) where package_suffix is something like "com.google.commmon"
+ project.apply plugin: 'com.github.johnrengelman.shadow'
- // Create a new configuration 'shadowTest' like 'shadow' for the test scope
- project.configurations {
- shadow { description = "Dependencies for shaded source set 'main'" }
- compile.extendsFrom shadow
- shadowTest {
- description = "Dependencies for shaded source set 'test'"
- extendsFrom shadow
+ // Create a new configuration 'shadowTest' like 'shadow' for the test scope
+ project.configurations {
+ shadow { description = "Dependencies for shaded source set 'main'" }
+ compile.extendsFrom shadow
+ shadowTest {
+ description = "Dependencies for shaded source set 'test'"
+ extendsFrom shadow
+ }
+ testCompile.extendsFrom shadowTest
}
- testCompile.extendsFrom shadowTest
}
project.jar {
- classifier = "unshaded"
- zip64 true
- }
+ setAutomaticModuleNameHeader(configuration, project)
- // Always configure the shadowJar classifier and merge service files.
- project.shadowJar({
- classifier = null
- mergeServiceFiles()
zip64 true
into("META-INF/") {
from "${project.rootProject.projectDir}/LICENSE"
from "${project.rootProject.projectDir}/NOTICE"
}
- } << configuration.shadowClosure)
-
- // Always configure the shadowTestJar classifier and merge service files.
- project.task('shadowTestJar', type: ShadowJar, {
- group = "Shadow"
- description = "Create a combined JAR of project and test dependencies"
- classifier = "tests"
- from project.sourceSets.test.output
- configurations = [
- project.configurations.testRuntime
- ]
- zip64 true
- exclude "META-INF/INDEX.LIST"
- exclude "META-INF/*.SF"
- exclude "META-INF/*.DSA"
- exclude "META-INF/*.RSA"
- } << configuration.shadowClosure)
-
- // Ensure that shaded jar and test-jar are part of the their own configuration artifact sets
- project.artifacts.shadow project.shadowJar
- project.artifacts.shadowTest project.shadowTestJar
-
- if (configuration.testShadowJar) {
- // Use a configuration and dependency set which represents the execution classpath using shaded artifacts for tests.
- project.configurations { shadowTestRuntimeClasspath }
-
- project.dependencies {
- shadowTestRuntimeClasspath it.project(path: project.path, configuration: "shadowTest")
- shadowTestRuntimeClasspath it.project(path: project.path, configuration: "provided")
- }
-
- project.test { classpath = project.configurations.shadowTestRuntimeClasspath }
}
- if (configuration.validateShadowJar) {
- project.task('validateShadedJarDoesntLeakNonProjectClasses', dependsOn: 'shadowJar') {
- ext.outFile = project.file("${project.reportsDir}/${name}.out")
- inputs.files project.configurations.shadow.artifacts.files
- outputs.files outFile
- doLast {
- project.configurations.shadow.artifacts.files.each {
- FileTree exposedClasses = project.zipTree(it).matching {
- include "**/*.class"
- // BEAM-5919: Exclude paths for Java 9 multi-release jars.
- exclude "META-INF/versions/*/module-info.class"
- configuration.shadowJarValidationExcludes.each {
- exclude "$it"
- exclude "META-INF/versions/*/$it"
+ // Always configure the shadowJar classifier and merge service files.
+ if (configuration.shadowClosure) {
+ // Only set the classifer on the unshaded classes if we are shading.
+ project.jar { classifier = "unshaded" }
+
+ project.shadowJar({
+ classifier = null
+ mergeServiceFiles()
+ zip64 true
+ into("META-INF/") {
+ from "${project.rootProject.projectDir}/LICENSE"
+ from "${project.rootProject.projectDir}/NOTICE"
+ }
+ } << configuration.shadowClosure)
+
+ // Always configure the shadowTestJar classifier and merge service files.
+ project.task('shadowTestJar', type: ShadowJar, {
+ group = "Shadow"
+ description = "Create a combined JAR of project and test dependencies"
+ classifier = "tests"
+ from project.sourceSets.test.output
+ configurations = [
+ project.configurations.testRuntime
+ ]
+ zip64 true
+ exclude "META-INF/INDEX.LIST"
+ exclude "META-INF/*.SF"
+ exclude "META-INF/*.DSA"
+ exclude "META-INF/*.RSA"
+ } << configuration.shadowClosure)
+
+ // Ensure that shaded jar and test-jar are part of the their own configuration artifact sets
+ project.artifacts.shadow project.shadowJar
+ project.artifacts.shadowTest project.shadowTestJar
+
+ if (configuration.testShadowJar) {
+ // Use a configuration and dependency set which represents the execution classpath using shaded artifacts for tests.
+ project.configurations { shadowTestRuntimeClasspath }
+
+ project.dependencies {
+ shadowTestRuntimeClasspath it.project(path: project.path, configuration: "shadowTest")
+ shadowTestRuntimeClasspath it.project(path: project.path, configuration: "provided")
+ }
+
+ project.test { classpath = project.configurations.shadowTestRuntimeClasspath }
+ }
+
+ if (configuration.validateShadowJar) {
+ project.task('validateShadedJarDoesntLeakNonProjectClasses', dependsOn: 'shadowJar') {
+ ext.outFile = project.file("${project.reportsDir}/${name}.out")
+ inputs.files project.configurations.shadow.artifacts.files
+ outputs.files outFile
+ doLast {
+ project.configurations.shadow.artifacts.files.each {
+ FileTree exposedClasses = project.zipTree(it).matching {
+ include "**/*.class"
+ // BEAM-5919: Exclude paths for Java 9 multi-release jars.
+ exclude "META-INF/versions/*/module-info.class"
+ configuration.shadowJarValidationExcludes.each {
+ exclude "$it"
+ exclude "META-INF/versions/*/$it"
+ }
}
- }
- outFile.text = exposedClasses.files
- if (exposedClasses.files) {
- throw new GradleException("$it exposed classes outside of ${configuration.shadowJarValidationExcludes}: ${exposedClasses.files}")
+ outFile.text = exposedClasses.files
+ if (exposedClasses.files) {
+ throw new GradleException("$it exposed classes outside of ${configuration.shadowJarValidationExcludes}: ${exposedClasses.files}")
+ }
}
}
}
+ project.tasks.check.dependsOn project.tasks.validateShadedJarDoesntLeakNonProjectClasses
}
- project.tasks.check.dependsOn project.tasks.validateShadedJarDoesntLeakNonProjectClasses
+ } else {
+ project.task("testJar", type: Jar, {
+ group = "Jar"
+ description = "Create a JAR of test classes"
+ classifier = "tests"
+ from project.sourceSets.test.output
+ zip64 true
+ exclude "META-INF/INDEX.LIST"
+ exclude "META-INF/*.SF"
+ exclude "META-INF/*.DSA"
+ exclude "META-INF/*.RSA"
+ })
+ project.artifacts.testRuntime project.testJar
}
project.ext.includeInJavaBom = configuration.publish
@@ -972,9 +998,9 @@
}
}
- // Have the shaded include both the generate pom.xml and its properties file
+ // Have the main artifact jar include both the generate pom.xml and its properties file
// emulating the behavior of the maven-archiver plugin.
- project.shadowJar {
+ project.(configuration.shadowClosure ? 'shadowJar' : 'jar') {
def pomFile = "${project.buildDir}/publications/mavenJava/pom-default.xml"
// Validate that the artifacts exist before copying them into the jar.
@@ -998,8 +1024,13 @@
}
// Only build artifacts for archives if we are publishing
- project.artifacts.archives project.shadowJar
- project.artifacts.archives project.shadowTestJar
+ if (configuration.shadowClosure) {
+ project.artifacts.archives project.shadowJar
+ project.artifacts.archives project.shadowTestJar
+ } else {
+ project.artifacts.archives project.jar
+ project.artifacts.archives project.testJar
+ }
project.task('sourcesJar', type: Jar) {
from project.sourceSets.main.allSource
@@ -1024,8 +1055,13 @@
publications {
mavenJava(MavenPublication) {
- artifact project.shadowJar
- artifact project.shadowTestJar
+ if (configuration.shadowClosure) {
+ artifact project.shadowJar
+ artifact project.shadowTestJar
+ } else {
+ artifact project.jar
+ artifact project.testJar
+ }
artifact project.sourcesJar
artifact project.testSourcesJar
artifact project.javadocJar
@@ -1096,17 +1132,26 @@
def generateDependenciesFromConfiguration = { param ->
project.configurations."${param.configuration}".allDependencies.each {
def dependencyNode = dependenciesNode.appendNode('dependency')
+ def appendClassifier = { dep ->
+ dep.artifacts.each { art ->
+ if (art.hasProperty('classifier')) {
+ dependencyNode.appendNode('classifier', art.classifier)
+ }
+ }
+ }
if (it instanceof ProjectDependency) {
dependencyNode.appendNode('groupId', it.getDependencyProject().mavenGroupId)
- dependencyNode.appendNode('artifactId', archivesBaseName(it.getDependencyProject()))
+ dependencyNode.appendNode('artifactId', it.getDependencyProject().archivesBaseName)
dependencyNode.appendNode('version', it.version)
dependencyNode.appendNode('scope', param.scope)
+ appendClassifier(it)
} else {
dependencyNode.appendNode('groupId', it.group)
dependencyNode.appendNode('artifactId', it.name)
dependencyNode.appendNode('version', it.version)
dependencyNode.appendNode('scope', param.scope)
+ appendClassifier(it)
}
// Start with any exclusions that were added via configuration exclude rules.
@@ -1130,7 +1175,8 @@
// TODO: Should we use the runtime scope instead of the compile scope
// which forces all our consumers to declare what they consume?
- generateDependenciesFromConfiguration(configuration: 'shadow', scope: 'compile')
+ generateDependenciesFromConfiguration(
+ configuration: (configuration.shadowClosure ? 'shadow' : 'compile'), scope: 'compile')
generateDependenciesFromConfiguration(configuration: 'provided', scope: 'provided')
// NB: This must come after asNode() logic, as it seems asNode()
@@ -1179,7 +1225,6 @@
// test libraries classes causing version conflicts. Users should rely
// on using the yyy-core package instead of the yyy-all package.
exclude group: "org.hamcrest", module: "hamcrest-all"
- exclude group: "org.mockito", module: "mockito-all"
}
// Force usage of the libraries defined within our common set found in the root
@@ -1200,7 +1245,7 @@
}
// When applied in a module's build.gradle file, this closure provides task for running
- // IO integration tests (manually, without PerfKitBenchmarker).
+ // IO integration tests.
project.ext.enableJavaPerformanceTesting = {
// Use the implicit it parameter of the closure to handle zero argument or one argument map calls.
@@ -1249,22 +1294,22 @@
/* include dependencies required by runners */
//if (runner?.contains('dataflow')) {
if (runner?.equalsIgnoreCase('dataflow')) {
- testCompile it.project(path: ":runners:google-cloud-dataflow-java", configuration: 'shadowTest')
- shadow it.project(path: ":runners:google-cloud-dataflow-java:worker:legacy-worker", configuration: 'shadow')
+ testRuntime it.project(path: ":runners:google-cloud-dataflow-java", configuration: 'testRuntime')
+ testRuntime it.project(path: ":runners:google-cloud-dataflow-java:worker:legacy-worker", configuration: 'shadow')
}
if (runner?.equalsIgnoreCase('direct')) {
- testCompile it.project(path: ":runners:direct-java", configuration: 'shadowTest')
+ testRuntime it.project(path: ":runners:direct-java", configuration: 'shadowTest')
}
if (runner?.equalsIgnoreCase('flink')) {
- testCompile it.project(path: ":runners:flink:1.5", configuration: 'shadowTest')
+ testRuntime it.project(path: ":runners:flink:1.8", configuration: 'testRuntime')
}
if (runner?.equalsIgnoreCase('spark')) {
- testCompile it.project(path: ":runners:spark", configuration: 'shadowTest')
- testCompile project.library.java.spark_core
- testCompile project.library.java.spark_streaming
+ testRuntime it.project(path: ":runners:spark", configuration: 'testRuntime')
+ testRuntime project.library.java.spark_core
+ testRuntime project.library.java.spark_streaming
// Testing the Spark runner causes a StackOverflowError if slf4j-jdk14 is on the classpath
project.configurations.testRuntimeClasspath {
@@ -1274,71 +1319,18 @@
/* include dependencies required by filesystems */
if (filesystem?.equalsIgnoreCase('hdfs')) {
- testCompile it.project(path: ":sdks:java:io:hadoop-file-system", configuration: 'shadowTest')
- shadowTest project.library.java.hadoop_client
+ testRuntime it.project(path: ":sdks:java:io:hadoop-file-system", configuration: 'testRuntime')
+ testRuntime project.library.java.hadoop_client
}
/* include dependencies required by AWS S3 */
if (filesystem?.equalsIgnoreCase('s3')) {
- testCompile it.project(path: ":sdks:java:io:amazon-web-services", configuration: 'shadowTest')
+ testRuntime it.project(path: ":sdks:java:io:amazon-web-services", configuration: 'testRuntime')
}
}
-
project.task('packageIntegrationTests', type: Jar)
}
- // When applied in a module's build gradle file, this closure provides a task
- // that will involve PerfKitBenchmarker for running integrationTests.
- project.ext.createPerformanceTestHarness = {
-
- // Use the implicit it parameter of the closure to handle zero argument or one argument map calls.
- // See: http://groovy-lang.org/closures.html#implicit-it
- JavaPerformanceTestConfiguration configuration = it ? it as JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
-
- // This task runs PerfKitBenchmarker, which does benchmarking of the IO ITs.
- // The arguments passed to it allows it to invoke gradle again with the desired benchmark.
- //
- // To invoke this, run:
- //
- // ./gradlew performanceTest \
- // -DpkbLocation="<path to pkb.py>"
- // -DintegrationTestPipelineOptions='["--numberOfRecords=1000", "<more options>"]' \
- // -DintegrationTest=<io test, eg. org.apache.beam.sdk.io.text.TextIOIT> \
- // -DitModule=<directory containing desired test, eg. sdks/java/io/file-based-io-tests> \
- // -DintegrationTestRunner=<runner to be used for testing, eg. dataflow>
- //
- // There are more options with default values that can be tweaked if needed (see below).
- project.task('performanceTest', type: Exec) {
-
- // PerfKitBenchmarker needs to work in the Beam's root directory,
- // otherwise it requires absolute paths ./gradlew, kubernetes scripts etc.
- commandLine "${configuration.pkbLocation}",
- "--dpb_log_level=${configuration.logLevel}",
- "--gradle_binary=${configuration.gradleBinary}",
- "--official=${configuration.isOfficial}",
- "--benchmarks=${configuration.benchmarks}",
- "--beam_location=${project.rootProject.projectDir}",
-
- "--beam_prebuilt=${configuration.beamPrebuilt}",
- "--beam_sdk=${configuration.beamSdk}",
-
- "--beam_it_timeout=${configuration.timeout}",
-
- "--kubeconfig=${configuration.kubeconfig}",
- "--kubectl=${configuration.kubectl}",
- "--beam_kubernetes_scripts=${configuration.kubernetesScripts}",
-
- "--beam_it_options=${configuration.integrationTestPipelineOptions}",
- "--beam_options_config_file=${configuration.optionsConfigFile}",
-
- "--beam_it_class=${configuration.integrationTest}",
- "--beam_it_module=${configuration.itModule}",
-
- "--beam_extra_properties=${configuration.extraProperties}",
- "--beam_runner=${configuration.runner}"
- }
- }
-
/** ***********************************************************************************************/
project.ext.applyGoNature = {
@@ -1346,7 +1338,7 @@
project.apply plugin: 'base'
project.apply plugin: "com.github.blindpirate.gogradle"
- project.golang { goVersion = '1.10' }
+ project.golang { goVersion = '1.12' }
project.repositories {
golang {
@@ -1385,7 +1377,6 @@
/** ***********************************************************************************************/
project.ext.applyGroovyNature = {
- println "Applying groovy nature"
project.apply plugin: "groovy"
project.apply plugin: "com.diffplug.gradle.spotless"
@@ -1401,9 +1392,10 @@
}
// containerImageName returns a configurable container image name, by default a
- // development image at bintray.io (see sdks/CONTAINERS.md):
+ // development image at docker.io (see sdks/CONTAINERS.md):
//
- // $USER-docker-apache.bintray.io/beam/$NAME:latest
+ // format: apachebeam/$NAME_sdk:latest
+ // ie: apachebeam/python2.7_sdk:latest apachebeam/java_sdk:latest apachebeam/go_sdk:latest
//
// Both the root and tag can be defined using properties or explicitly provided.
project.ext.containerImageName = {
@@ -1432,7 +1424,8 @@
project.ext.applyGrpcNature = {
project.apply plugin: "com.google.protobuf"
project.protobuf {
- protoc { // The artifact spec for the Protobuf Compiler
+ protoc {
+ // The artifact spec for the Protobuf Compiler
artifact = "com.google.protobuf:protoc:3.6.0" }
// Configure the codegen plugins
@@ -1480,14 +1473,21 @@
project.ext.applyPortabilityNature = {
PortabilityNatureConfiguration configuration = it ? it as PortabilityNatureConfiguration : new PortabilityNatureConfiguration()
+ if (configuration.archivesBaseName) {
+ project.archivesBaseName = configuration.archivesBaseName
+ }
+
project.ext.applyJavaNature(
exportJavadoc: false,
enableSpotbugs: false,
+ publish: configuration.publish,
+ archivesBaseName: configuration.archivesBaseName,
+ automaticModuleName: configuration.automaticModuleName,
shadowJarValidationExcludes: it.shadowJarValidationExcludes,
shadowClosure: GrpcVendoring.shadowClosure() << {
// We perform all the code relocations but don't include
// any of the actual dependencies since they will be supplied
- // by org.apache.beam:beam-vendor-grpc-v1p13p1:0.1
+ // by org.apache.beam:beam-vendor-grpc-v1p21p0:0.1
dependencies {
include(dependency { return false })
}
@@ -1502,15 +1502,16 @@
project.apply plugin: "com.google.protobuf"
project.protobuf {
- protoc { // The artifact spec for the Protobuf Compiler
- artifact = "com.google.protobuf:protoc:3.6.0" }
+ protoc {
+ // The artifact spec for the Protobuf Compiler
+ artifact = "com.google.protobuf:protoc:3.7.1" }
// Configure the codegen plugins
plugins {
// An artifact spec for a protoc plugin, with "grpc" as
// the identifier, which can be referred to in the "plugins"
// container of the "generateProtoTasks" closure.
- grpc { artifact = "io.grpc:protoc-gen-grpc-java:1.13.1" }
+ grpc { artifact = "io.grpc:protoc-gen-grpc-java:1.21.0" }
}
generateProtoTasks {
@@ -1524,7 +1525,7 @@
}
}
- project.dependencies GrpcVendoring.dependenciesClosure() << { shadow project.ext.library.java.vendored_grpc_1_13_1 }
+ project.dependencies GrpcVendoring.dependenciesClosure() << { shadow project.ext.library.java.vendored_grpc_1_21_0 }
}
/** ***********************************************************************************************/
@@ -1591,7 +1592,6 @@
*/
project.evaluationDependsOn(":sdks:java:core")
project.evaluationDependsOn(":runners:core-java")
- project.evaluationDependsOn(":runners:core-construction-java")
def config = it ? it as PortableValidatesRunnerConfiguration : new PortableValidatesRunnerConfiguration()
def name = config.name
def beamTestPipelineOptions = [
@@ -1599,8 +1599,6 @@
"--jobServerDriver=${config.jobServerDriver}",
"--environmentCacheMillis=10000"
]
- def expansionPort = startingExpansionPortNumber.getAndDecrement()
- config.systemProperties.put("expansionPort", expansionPort)
beamTestPipelineOptions.addAll(config.pipelineOpts)
if (config.environment == PortableValidatesRunnerConfiguration.Environment.EMBEDDED) {
beamTestPipelineOptions += "--defaultEnvironmentType=EMBEDDED"
@@ -1628,6 +1626,113 @@
/** ***********************************************************************************************/
+ // Method to create the crossLanguageValidatesRunnerTask.
+ // The method takes crossLanguageValidatesRunnerConfiguration as parameter.
+ project.ext.createCrossLanguageValidatesRunnerTask = {
+ def config = it ? it as CrossLanguageValidatesRunnerConfiguration : new CrossLanguageValidatesRunnerConfiguration()
+
+ project.evaluationDependsOn(":sdks:python")
+ project.evaluationDependsOn(":sdks:java:testing:expansion-service")
+ project.evaluationDependsOn(":runners:core-construction-java")
+
+ // Task for launching expansion services
+ def envDir = project.project(":sdks:python").envdir
+ def pythonDir = project.project(":sdks:python").projectDir
+ def javaPort = startingExpansionPortNumber.getAndDecrement()
+ def pythonPort = startingExpansionPortNumber.getAndDecrement()
+ def expansionJar = project.project(':sdks:java:testing:expansion-service').buildTestExpansionServiceJar.archivePath
+ def expansionServiceOpts = [
+ "group_id": project.name,
+ "java_expansion_service_jar": expansionJar,
+ "java_port": javaPort,
+ "python_virtualenv_dir": envDir,
+ "python_expansion_service_module": "apache_beam.runners.portability.expansion_service_test",
+ "python_port": pythonPort
+ ]
+ def serviceArgs = project.project(':sdks:python').mapToArgString(expansionServiceOpts)
+ def setupTask = project.tasks.create(name: config.name+"Setup", type: Exec) {
+ dependsOn ':sdks:java:container:docker'
+ dependsOn ':sdks:python:container:buildAll'
+ dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+ dependsOn ":sdks:python:installGcpTest"
+ // setup test env
+ executable 'sh'
+ args '-c', "$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name} && $pythonDir/scripts/run_expansion_services.sh start $serviceArgs"
+ }
+
+ def mainTask = project.tasks.create(name: config.name) {
+ group = "Verification"
+ description = "Validates cross-language capability of runner"
+ }
+
+ def cleanupTask = project.tasks.create(name: config.name+'Cleanup', type: Exec) {
+ // teardown test env
+ executable 'sh'
+ args '-c', "$pythonDir/scripts/run_expansion_services.sh stop --group_id ${project.name}"
+ }
+ setupTask.finalizedBy cleanupTask
+
+ // Task for running testcases in Java SDK
+ def beamJavaTestPipelineOptions = [
+ "--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+ "--jobServerDriver=${config.jobServerDriver}",
+ "--environmentCacheMillis=10000"
+ ]
+ beamJavaTestPipelineOptions.addAll(config.pipelineOpts)
+ if (config.jobServerConfig) {
+ beamJavaTestPipelineOptions.add("--jobServerConfig=${config.jobServerConfig}")
+ }
+ ['Java': javaPort, 'Python': pythonPort].each { sdk, port ->
+ def javaTask = project.tasks.create(name: config.name+"JavaUsing"+sdk, type: Test) {
+ group = "Verification"
+ description = "Validates runner for cross-language capability of using ${sdk} transforms from Java SDK"
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(beamJavaTestPipelineOptions)
+ systemProperty "expansionPort", port
+ classpath = config.testClasspathConfiguration
+ testClassesDirs = project.files(project.project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
+ maxParallelForks config.numParallelTests
+ useJUnit(config.testCategories)
+ // increase maxHeapSize as this is directly correlated to direct memory,
+ // see https://issues.apache.org/jira/browse/BEAM-6698
+ maxHeapSize = '4g'
+ dependsOn setupTask
+ }
+ mainTask.dependsOn javaTask
+ cleanupTask.mustRunAfter javaTask
+
+ // Task for running testcases in Python SDK
+ def testOpts = [
+ "--attr=UsesCrossLanguageTransforms"
+ ]
+ def pipelineOpts = [
+ "--runner=PortableRunner",
+ "--environment_cache_millis=10000"
+ ]
+ def beamPythonTestPipelineOptions = [
+ "pipeline_opts": pipelineOpts,
+ "test_opts": testOpts,
+ "suite": "xlangValidateRunner"
+ ]
+ def cmdArgs = project.project(':sdks:python').mapToArgString(beamPythonTestPipelineOptions)
+ def pythonTask = project.tasks.create(name: config.name+"PythonUsing"+sdk, type: Exec) {
+ group = "Verification"
+ description = "Validates runner for cross-language capability of using ${sdk} transforms from Python SDK"
+ environment "EXPANSION_JAR", expansionJar
+ environment "EXPANSION_PORT", port
+ executable 'sh'
+ args '-c', ". $envDir/bin/activate && cd $pythonDir && ./scripts/run_integration_test.sh $cmdArgs"
+ dependsOn setupTask
+ // We need flink-job-server-container dependency since Python PortableRunner automatically
+ // brings the flink-job-server-container up when --job_endpoint is not specified.
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ }
+ mainTask.dependsOn pythonTask
+ cleanupTask.mustRunAfter pythonTask
+ }
+ }
+
+ /** ***********************************************************************************************/
+
project.ext.applyPythonNature = {
// Define common lifecycle tasks and artifact types
@@ -1663,7 +1768,7 @@
project.exec { commandLine virtualenvCmd }
project.exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip install --retries 10 --upgrade tox==3.0.0 grpcio-tools==1.3.5"
+ args '-c', ". ${project.ext.envdir}/bin/activate && pip install --retries 10 --upgrade tox==3.11.1 grpcio-tools==1.3.5"
}
}
// Gradle will delete outputs whenever it thinks they are stale. Putting a
@@ -1672,7 +1777,7 @@
outputs.dirs(project.ext.envdir)
}
- def pythonSdkDeps = project.files(
+ project.ext.pythonSdkDeps = project.files(
project.fileTree(
dir: "${project.rootDir}",
include: ['model/**', 'sdks/python/**'],
@@ -1686,32 +1791,9 @@
)
def copiedSrcRoot = "${project.buildDir}/srcs"
- project.configurations { distConfig }
-
- project.task('sdist', dependsOn: 'setupVirtualenv') {
- doLast {
- // Copy sdk sources to an isolated directory
- project.copy {
- from pythonSdkDeps
- into copiedSrcRoot
- }
-
- // Build artifact
- project.exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedSrcRoot}/sdks/python && python setup.py sdist --formats zip,gztar --dist-dir ${project.buildDir}"
- }
- def collection = project.fileTree("${project.buildDir}"){ include '**/*.tar.gz' exclude '**/apache-beam.tar.gz', 'srcs/**'}
- println "sdist archive name: ${collection.singleFile}"
-
- // we need a fixed name for the artifact
- project.copy { from collection.singleFile; into "${project.buildDir}"; rename { 'apache-beam.tar.gz' } }
- }
- }
-
- project.artifacts {
- distConfig file: project.file("${project.buildDir}/apache-beam.tar.gz"), builtBy: project.sdist
- }
+ // Create new configuration distTarBall which represents Python source
+ // distribution tarball generated by :sdks:python:sdist.
+ project.configurations { distTarBall }
project.task('installGcpTest', dependsOn: 'setupVirtualenv') {
doLast {
@@ -1721,7 +1803,7 @@
}
}
}
- project.installGcpTest.mustRunAfter project.sdist
+ project.installGcpTest.mustRunAfter project.configurations.distTarBall
project.task('cleanPython') {
doLast {
@@ -1759,15 +1841,23 @@
project.ext.toxTask = { name, tox_env ->
project.tasks.create(name) {
- dependsOn = ['sdist']
+ dependsOn 'setupVirtualenv'
+ dependsOn ':sdks:python:sdist'
+
doLast {
+ // Python source directory is also tox execution workspace, We want
+ // to isolate them per tox suite to avoid conflict when running
+ // multiple tox suites in parallel.
+ project.copy { from project.pythonSdkDeps; into copiedSrcRoot }
+
def copiedPyRoot = "${copiedSrcRoot}/sdks/python"
+ def distTarBall = "${pythonRootDir}/build/apache-beam.tar.gz"
project.exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedPyRoot} && scripts/run_tox.sh $tox_env ${project.buildDir}/apache-beam.tar.gz"
+ args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedPyRoot} && scripts/run_tox.sh $tox_env $distTarBall"
}
}
- inputs.files pythonSdkDeps
+ inputs.files project.pythonSdkDeps
outputs.files project.fileTree(dir: "${pythonRootDir}/target/.tox/${tox_env}/log/")
}
}
@@ -1781,7 +1871,7 @@
project.task('integrationTest') {
dependsOn 'installGcpTest'
- dependsOn 'sdist'
+ dependsOn ':sdks:python:sdist'
doLast {
def argMap = [:]
@@ -1800,6 +1890,7 @@
argMap["pipeline_opts"] = config.pipelineOptions
if (config.kmsKeyName)
argMap["kms_key_name"] = config.kmsKeyName
+ argMap["suite"] = "integrationTest-perf"
def cmdArgs = project.mapToArgString(argMap)
def runScriptsDir = "${pythonRootDir}/scripts"
@@ -1810,6 +1901,74 @@
}
}
}
+
+ def addPortableWordCountTask = { boolean isStreaming ->
+ project.task('portableWordCount' + (isStreaming ? 'Streaming' : 'Batch')) {
+ dependsOn = ['installGcpTest']
+ mustRunAfter = [
+ ':runners:flink:1.8:job-server-container:docker',
+ ':sdks:python:container:py2:docker',
+ ':sdks:python:container:py35:docker',
+ ':sdks:python:container:py36:docker',
+ ':sdks:python:container:py37:docker'
+ ]
+ doLast {
+ // TODO: Figure out GCS credentials and use real GCS input and output.
+ def options = [
+ "--input=/etc/profile",
+ "--output=/tmp/py-wordcount-direct",
+ "--runner=PortableRunner",
+ "--experiments=worker_threads=100",
+ "--parallelism=2",
+ "--shutdown_sources_on_final_watermark",
+ "--sdk_worker_parallelism=1",
+ ]
+ if (isStreaming)
+ options += [
+ "--streaming"
+ ]
+ else
+ // workaround for local file output in docker container
+ options += [
+ "--environment_cache_millis=60000"
+ ]
+ if (project.hasProperty("jobEndpoint"))
+ options += [
+ "--job_endpoint=${project.property('jobEndpoint')}"
+ ]
+ if (project.hasProperty("environmentType")) {
+ options += [
+ "--environment_type=${project.property('environmentType')}"
+ ]
+ }
+ if (project.hasProperty("environmentConfig")) {
+ options += [
+ "--environment_config=${project.property('environmentConfig')}"
+ ]
+ }
+ project.exec {
+ executable 'sh'
+ args '-c', ". ${project.ext.envdir}/bin/activate && python -m apache_beam.examples.wordcount ${options.join(' ')}"
+ // TODO: Check that the output file is generated and runs.
+ }
+ }
+ }
+ }
+ project.ext.addPortableWordCountTasks = {
+ ->
+ addPortableWordCountTask(false)
+ addPortableWordCountTask(true)
+ }
+ }
+ }
+
+ private void setAutomaticModuleNameHeader(JavaNatureConfiguration configuration, Project project) {
+ if (configuration.publish && !configuration.automaticModuleName) {
+ throw new GradleException("Expected automaticModuleName to be set for the module that is published to maven repository.")
+ } else if (configuration.automaticModuleName) {
+ project.jar.manifest {
+ attributes 'Automatic-Module-Name': configuration.automaticModuleName
+ }
}
}
}
diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring.groovy
index 102f0a0..96c6bf8 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring.groovy
@@ -27,23 +27,24 @@
/** Returns the list of compile time dependencies. */
static List<String> dependencies() {
return [
- 'com.google.guava:guava:20.0',
- 'com.google.protobuf:protobuf-java:3.6.0',
- 'com.google.protobuf:protobuf-java-util:3.6.0',
+ 'com.google.guava:guava:26.0-jre',
+ 'com.google.protobuf:protobuf-java:3.7.1',
+ 'com.google.protobuf:protobuf-java-util:3.7.1',
'com.google.code.gson:gson:2.7',
- 'io.grpc:grpc-auth:1.13.1',
- 'io.grpc:grpc-core:1.13.1',
- 'io.grpc:grpc-context:1.13.1',
- 'io.grpc:grpc-netty:1.13.1',
- 'io.grpc:grpc-protobuf:1.13.1',
- 'io.grpc:grpc-stub:1.13.1',
- 'io.netty:netty-transport-native-epoll:4.1.25.Final',
- 'io.netty:netty-tcnative-boringssl-static:2.0.8.Final',
- 'com.google.auth:google-auth-library-credentials:0.10.0',
- 'io.grpc:grpc-testing:1.13.1',
+ 'io.grpc:grpc-auth:1.21.0',
+ 'io.grpc:grpc-core:1.21.0',
+ 'io.grpc:grpc-context:1.21.0',
+ 'io.grpc:grpc-netty:1.21.0',
+ 'io.grpc:grpc-protobuf:1.21.0',
+ 'io.grpc:grpc-stub:1.21.0',
+ 'io.netty:netty-transport-native-epoll:4.1.34.Final',
+ // tcnative version from https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty
+ 'io.netty:netty-tcnative-boringssl-static:2.0.22.Final',
+ 'com.google.auth:google-auth-library-credentials:0.13.0',
+ 'io.grpc:grpc-testing:1.21.0',
'com.google.api.grpc:proto-google-common-protos:1.12.0',
- 'io.opencensus:opencensus-api:0.12.3',
- 'io.opencensus:opencensus-contrib-grpc-metrics:0.12.3',
+ 'io.opencensus:opencensus-api:0.21.0',
+ 'io.opencensus:opencensus-contrib-grpc-metrics:0.21.0',
]
}
@@ -53,7 +54,7 @@
*/
static List<String> runtimeDependencies() {
return [
- 'com.google.errorprone:error_prone_annotations:2.1.2'
+ 'com.google.errorprone:error_prone_annotations:2.3.2',
]
}
@@ -72,7 +73,8 @@
// those libraries may provide. The 'validateShadedJarDoesntLeakNonOrgApacheBeamClasses'
// ensures that there are no classes outside of the 'org.apache.beam' namespace.
- String prefix = "org.apache.beam.vendor.grpc.v1p13p1";
+ String version = "v1p21p0";
+ String prefix = "org.apache.beam.vendor.grpc.${version}";
List<String> packagesToRelocate = [
// guava uses the com.google.common and com.google.thirdparty package namespaces
"com.google.common",
@@ -92,25 +94,29 @@
]
return packagesToRelocate.collectEntries {
- [ (it): "org.apache.beam.vendor.grpc.v1p13p1.${it}" ]
+ [ (it): "${prefix}.${it}" ]
} + [
// Adapted from https://github.com/grpc/grpc-java/blob/e283f70ad91f99c7fee8b31b605ef12a4f9b1690/netty/shaded/build.gradle#L41
// We "io.netty": "${prefix}.io.netty",have to be careful with these replacements as they must not match any
// string in NativeLibraryLoader, else they cause corruption. Note that
// this includes concatenation of string literals and constants.
- 'META-INF/native/libnetty': 'META-INF/native/liborg_apache_beam_vendor_grpc_v1p13p1_netty',
- 'META-INF/native/netty': 'META-INF/native/org_apache_beam_vendor_grpc_v1p13p1_netty',
+ 'META-INF/native/libnetty': "META-INF/native/liborg_apache_beam_vendor_grpc_${version}_netty",
+ 'META-INF/native/netty': "META-INF/native/org_apache_beam_vendor_grpc_${version}_netty",
]
}
/** Returns the list of shading exclusions. */
static List<String> exclusions() {
return [
- // Don't include errorprone, JDK8 annotations, objenesis, junit, and mockito in the vendored jar
+ // Don't include android annotations, errorprone, checkerframework, JDK8 annotations, objenesis, junit, and mockito in the vendored jar
+ "android/annotation/**/",
"com/google/errorprone/**",
"com/google/instrumentation/**",
+ "com/google/j2objc/annotations/**",
"javax/annotation/**",
"junit/**",
+ "org/checkerframework/**",
+ "org/codehaus/mojo/animal_sniffer/**",
"org/hamcrest/**",
"org/junit/**",
"org/mockito/**",
diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
index 500303e..24ca6e1 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy
@@ -35,11 +35,11 @@
* <li>Increment the vendored artifact version only if we need to release a new version.
* </ul>
*
- * <p>Example for com.google.guava:guava:20.0:
+ * <p>Example for com.google.guava:guava:26.0-jre:
* <ul>
* <li>groupId: org.apache.beam
- * <li>artifactId: guava-20_0
- * <li>namespace: org.apache.beam.vendor.guava.v20_0
+ * <li>artifactId: guava-26_0-jre
+ * <li>namespace: org.apache.beam.vendor.guava.v26_0_jre
* <li>version: 0.1
* </ul>
*
@@ -70,6 +70,9 @@
project.ext.vendorJava = {
VendorJavaPluginConfig config = it ? it as VendorJavaPluginConfig : new VendorJavaPluginConfig()
+ project.apply plugin: 'base'
+ project.archivesBaseName = config.artifactId
+
if (!isRelease(project)) {
config.version += '-SNAPSHOT'
}
@@ -143,6 +146,8 @@
// Only publish vendored dependencies if specifically requested.
if (project.hasProperty("vendoredDependenciesOnly")) {
project.apply plugin: 'maven-publish'
+ // Ensure that we validate the contents of the jar before publishing
+ project.publish.dependsOn project.check
// Have the shaded jar include both the generate pom.xml and its properties file
// emulating the behavior of the maven-archiver plugin.
diff --git a/examples/java/README.md b/examples/java/README.md
index ff24917..eac6f9c 100644
--- a/examples/java/README.md
+++ b/examples/java/README.md
@@ -30,7 +30,7 @@
1. [`MinimalWordCount`](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java) is the simplest word count pipeline and introduces basic concepts like [Pipelines](https://beam.apache.org/documentation/programming-guide/#pipeline),
[PCollections](https://beam.apache.org/documentation/programming-guide/#pcollection),
-[ParDo](https://beam.apache.org/documentation/programming-guide/#transforms-pardo),
+[ParDo](https://beam.apache.org/documentation/programming-guide/#pardo),
and [reading and writing data](https://beam.apache.org/documentation/programming-guide/#io) from external storage.
1. [`WordCount`](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java) introduces best practices like [PipelineOptions](https://beam.apache.org/documentation/programming-guide/#pipeline) and custom [PTransforms](https://beam.apache.org/documentation/programming-guide/#transforms-composite).
@@ -54,4 +54,7 @@
See the other examples as well. This directory includes a Java 8 version of the
MinimalWordCount example, as well as a series of examples in a simple 'mobile
-gaming' domain. This series introduces some advanced concepts.
+gaming' domain. This series introduces some advanced concepts. Finally, the
+following are user contributed examples:
+
+1. [`CryptoRealTime`](https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/cryptorealtime) Create an unbounded streaming source/reader and manage basic watermarking, checkpointing and record id for data ingestion, stream trading data from exchanges with BEAM + [Medium post](https://medium.com/@igalic/bigtable-beam-dataflow-cryptocurrencies-gcp-terraform-java-maven-4e7873811e86)
diff --git a/examples/java/build.gradle b/examples/java/build.gradle
index 1f2059d..3936398 100644
--- a/examples/java/build.gradle
+++ b/examples/java/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.examples')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -45,45 +45,45 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadow library.java.google_api_client
- shadow library.java.google_api_services_bigquery
- shadow library.java.google_http_client
- shadow library.java.bigdataoss_util
- shadow library.java.google_auth_library_oauth2_http
- shadow library.java.google_auth_library_credentials
- shadow library.java.avro
- shadow library.java.google_api_services_pubsub
- shadow library.java.datastore_v1_proto_client
- shadow library.java.datastore_v1_protos
- shadow library.java.joda_time
- shadow library.java.slf4j_api
- shadow library.java.slf4j_jdk14
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:io:google-cloud-platform")
+ compile library.java.avro
+ compile library.java.bigdataoss_util
+ compile library.java.google_api_client
+ compile library.java.google_api_services_bigquery
+ compile library.java.google_api_services_pubsub
+ compile library.java.google_auth_library_credentials
+ compile library.java.google_auth_library_oauth2_http
+ compile library.java.google_cloud_datastore_v1_proto_client
+ compile library.java.google_http_client
+ compile library.java.joda_time
+ compile library.java.proto_google_cloud_datastore_v1
+ compile library.java.slf4j_api
+ compile library.java.slf4j_jdk14
runtime project(path: ":runners:direct-java", configuration: "shadow")
- shadowTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
+ testCompile project(":sdks:java:io:google-cloud-platform")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
// Add dependencies for the PreCommit configurations
// For each runner a project level dependency on the examples project.
for (String runner : preCommitRunners) {
- delegate.add(runner + "PreCommit", project(path: ":examples:java", configuration: "shadow"))
- delegate.add(runner + "PreCommit", project(path: ":examples:java", configuration: "shadowTest"))
+ delegate.add(runner + "PreCommit", project(":examples:java"))
+ delegate.add(runner + "PreCommit", project(path: ":examples:java", configuration: "testRuntime"))
}
// https://issues.apache.org/jira/browse/BEAM-3583
- // apexRunnerPreCommit project(path: ":runners:apex", configuration: "shadow")
+ // apexRunnerPreCommit project(":runners:apex")
directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow")
- flinkRunnerPreCommit project(path: ":runners:flink:1.5", configuration: "shadow")
+ flinkRunnerPreCommit project(":runners:flink:1.8")
// TODO: Make the netty version used configurable, we add netty-all 4.1.17.Final so it appears on the classpath
// before 4.1.8.Final defined by Apache Beam
sparkRunnerPreCommit "io.netty:netty-all:4.1.17.Final"
- sparkRunnerPreCommit project(path: ":runners:spark", configuration: "shadow")
- sparkRunnerPreCommit project(path: ":sdks:java:io:hadoop-file-system", configuration: "shadow")
+ sparkRunnerPreCommit project(":runners:spark")
+ sparkRunnerPreCommit project(":sdks:java:io:hadoop-file-system")
sparkRunnerPreCommit library.java.spark_streaming
sparkRunnerPreCommit library.java.spark_core
}
diff --git a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
index 181d190..97d3298 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/common/ExampleUtils.java
@@ -50,10 +50,10 @@
import org.apache.beam.sdk.util.BackOffUtils;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.Sleeper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
/**
diff --git a/examples/java/src/main/java/org/apache/beam/examples/common/WriteOneFilePerWindow.java b/examples/java/src/main/java/org/apache/beam/examples/common/WriteOneFilePerWindow.java
index 69a9752..b15eff6 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/common/WriteOneFilePerWindow.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/common/WriteOneFilePerWindow.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.examples.common;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import javax.annotation.Nullable;
import org.apache.beam.sdk.io.FileBasedSink;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
index 2f68824..58ab72b 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
@@ -19,7 +19,7 @@
import static com.google.datastore.v1.client.DatastoreHelper.makeKey;
import static com.google.datastore.v1.client.DatastoreHelper.makeValue;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableReference;
@@ -69,7 +69,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Duration;
/**
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
index 0b0023e..fe176ec 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TfIdf.java
@@ -56,7 +56,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
index faa1b42..01cd3a2 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java b/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java
index 0fa685c..53fa009 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/TrafficRoutes.java
@@ -51,7 +51,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.DateTimeFormat;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
index 27c03b8..e0f56c8 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java
index 07d08e8..e79e5e4 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.examples.complete.game;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import java.util.HashMap;
import java.util.Map;
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Instant;
/**
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
index db5b722..2938fb0 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java
@@ -205,8 +205,11 @@
public interface Options extends PipelineOptions {
@Description("Path to the data file(s) containing game data.")
- // The default maps to two large Google Cloud Storage files (each ~12GB) holding two subsequent
- // day's worth (roughly) of data.
+ /* The default maps to two large Google Cloud Storage files (each ~12GB) holding two subsequent
+ day's worth (roughly) of data.
+
+ Note: You may want to use a small sample dataset to test it locally/quickly : gs://apache-beam-samples/game/small/gaming_data.csv
+ You can also download it via the command line gsutil cp gs://apache-beam-samples/game/small/gaming_data.csv ./destination_folder/gaming_data.csv */
@Default.String("gs://apache-beam-samples/game/gaming_data*.csv")
String getInput();
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
index 4428568..0a779b0 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java
@@ -30,7 +30,7 @@
import java.util.List;
import java.util.Random;
import org.apache.beam.examples.complete.game.utils.GameConstants;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* This is a generator that simulates usage data from a mobile game, and either publishes the data
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/InjectorUtils.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/InjectorUtils.java
index 096e22e..dbefed2 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/InjectorUtils.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/InjectorUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.examples.complete.game.injector;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/RetryHttpInitializerWrapper.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/RetryHttpInitializerWrapper.java
index 6401678..c487157 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/RetryHttpInitializerWrapper.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/injector/RetryHttpInitializerWrapper.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.examples.complete.game.injector;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.http.HttpBackOffIOExceptionHandler;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java
index 2093907..e88484e 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.examples.complete.game.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.util.ArrayList;
diff --git a/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java b/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java
index a4356c7..b4b775e 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java
@@ -20,7 +20,6 @@
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
-import com.google.cloud.bigquery.storage.v1beta1.ReadOptions.TableReadOptions;
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.Pipeline;
@@ -38,7 +37,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* An example that reads the public samples of weather data from BigQuery, counts the number of
@@ -134,7 +133,7 @@
* <p>Inherits standard configuration options.
*/
public interface Options extends PipelineOptions {
- @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Description("Table to read from, specified as <project_id>:<dataset_id>.<table_id>")
@Default.String(WEATHER_SAMPLES_TABLE)
String getInput();
@@ -166,25 +165,23 @@
PCollection<TableRow> rowsFromBigQuery;
- if (options.getReadMethod() == Method.DIRECT_READ) {
- // Build the read options proto for the read operation.
- TableReadOptions tableReadOptions =
- TableReadOptions.newBuilder()
- .addAllSelectedFields(Lists.newArrayList("month", "tornado"))
- .build();
+ switch (options.getReadMethod()) {
+ case DIRECT_READ:
+ rowsFromBigQuery =
+ p.apply(
+ BigQueryIO.readTableRows()
+ .from(options.getInput())
+ .withMethod(Method.DIRECT_READ)
+ .withSelectedFields(Lists.newArrayList("month", "tornado")));
+ break;
- rowsFromBigQuery =
- p.apply(
- BigQueryIO.readTableRows()
- .from(options.getInput())
- .withMethod(Method.DIRECT_READ)
- .withReadOptions(tableReadOptions));
- } else {
- rowsFromBigQuery =
- p.apply(
- BigQueryIO.readTableRows()
- .from(options.getInput())
- .withMethod(options.getReadMethod()));
+ default:
+ rowsFromBigQuery =
+ p.apply(
+ BigQueryIO.readTableRows()
+ .from(options.getInput())
+ .withMethod(options.getReadMethod()));
+ break;
}
rowsFromBigQuery
diff --git a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
index 8216bba..e100166 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java
@@ -23,6 +23,9 @@
import com.google.api.services.bigquery.model.TableSchema;
import com.google.api.services.bigquery.model.TimePartitioning;
import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
@@ -30,6 +33,7 @@
import org.apache.avro.generic.GenericRecord;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.coders.DoubleCoder;
import org.apache.beam.sdk.io.Compression;
@@ -42,6 +46,11 @@
import org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations;
import org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord;
import org.apache.beam.sdk.io.gcp.bigquery.TableDestination;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.MapElements;
@@ -55,16 +64,21 @@
import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.Repeatedly;
import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowFn.MergeContext;
+import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.DateTimeFormat;
@@ -143,6 +157,24 @@
}
{
+ // [START BigQueryDataTypes]
+ TableRow row = new TableRow();
+ row.set("string", "abc");
+ byte[] rawbytes = {(byte) 0xab, (byte) 0xac};
+ row.set("bytes", new String(Base64.getEncoder().encodeToString(rawbytes)));
+ row.set("integer", 5);
+ row.set("float", 0.5);
+ row.set("numeric", 5);
+ row.set("boolean", true);
+ row.set("timestamp", "2018-12-31 12:44:31.744957 UTC");
+ row.set("date", "2018-12-31");
+ row.set("time", "12:44:31");
+ row.set("datetime", "2019-06-11T14:44:31");
+ row.set("geography", "POINT(30 10)");
+ // [END BigQueryDataTypes]
+ }
+
+ {
String tableSpec = "clouddataflow-readonly:samples.weather_stations";
// [START BigQueryReadTable]
PCollection<Double> maxTemperatures =
@@ -504,22 +536,23 @@
TextIO.read()
.from("<path-to-files>/*")
.watchForNewFiles(
- // Check for new files every minute
+ // Check for new files every minute.
Duration.standardMinutes(1),
- // Stop watching the filepattern if no new files appear within an hour
+ // Stop watching the file pattern if no new files appear for an hour.
Watch.Growth.afterTimeSinceNewOutput(Duration.standardHours(1))));
// [END FileProcessPatternProcessNewFilesSnip2]
// [START FileProcessPatternAccessMetadataSnip1]
p.apply(FileIO.match().filepattern("hdfs://path/to/*.gz"))
- // withCompression can be omitted - by default compression is detected from the filename.
+ // The withCompression method is optional. By default, the Beam SDK detects compression from
+ // the filename.
.apply(FileIO.readMatches().withCompression(Compression.GZIP))
.apply(
ParDo.of(
new DoFn<FileIO.ReadableFile, String>() {
@ProcessElement
public void process(@Element FileIO.ReadableFile file) {
- // We now have access to the file and its metadata
+ // We can now access the file and its metadata.
LOG.info("File Metadata resourceId is {} ", file.getMetadata().resourceId());
}
}));
@@ -531,12 +564,11 @@
// [START SideInputPatternSlowUpdateGlobalWindowSnip1]
public static void sideInputPatterns() {
- // Using View.asSingleton, this pipeline uses a dummy external service as illustration.
- // Run in debug mode to see the output
+ // This pipeline uses View.asSingleton for a placeholder external service.
+ // Run in debug mode to see the output.
Pipeline p = Pipeline.create();
- // Create slowly updating sideinput
-
+ // Create a side input that updates each second.
PCollectionView<Map<String, String>> map =
p.apply(GenerateSequence.from(0).withRate(1, Duration.standardSeconds(5L)))
.apply(
@@ -550,20 +582,15 @@
@ProcessElement
public void process(
@Element Long input, OutputReceiver<Map<String, String>> o) {
- // Do any external reads needed here...
- // We will make use of our dummy external service.
- // Every time this triggers, the complete map will be replaced with that
- // read from
- // the service.
- o.output(DummyExternalService.readDummyData());
+ // Replace map with test data from the placeholder external service.
+ // Add external reads here.
+ o.output(PlaceholderExternalService.readTestData());
}
}))
.apply(View.asSingleton());
- // ---- Consume slowly updating sideinput
-
- // GenerateSequence is only used here to generate dummy data for this illustration.
- // You would use your real source for example PubSubIO, KafkaIO etc...
+ // Consume side input. GenerateSequence generates test data.
+ // Use a real source (like PubSubIO or KafkaIO) in production.
p.apply(GenerateSequence.from(0).withRate(1, Duration.standardSeconds(1L)))
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(1))))
.apply(Sum.longsGlobally().withoutDefaults())
@@ -577,7 +604,7 @@
c.outputWithTimestamp(KV.of(1L, c.element()), Instant.now());
LOG.debug(
- "Value is {} key A is {} and key B is {}",
+ "Value is {}, key A is {}, and key B is {}.",
c.element(),
keyMap.get("Key_A"),
keyMap.get("Key_B"));
@@ -586,10 +613,10 @@
.withSideInputs(map));
}
- /** Dummy class representing a pretend external service. */
- public static class DummyExternalService {
+ /** Placeholder class that represents an external service generating test data. */
+ public static class PlaceholderExternalService {
- public static Map<String, String> readDummyData() {
+ public static Map<String, String> readTestData() {
Map<String, String> map = new HashMap<>();
Instant now = Instant.now();
@@ -605,4 +632,157 @@
// [END SideInputPatternSlowUpdateGlobalWindowSnip1]
+ // [START AccessingValueProviderInfoAfterRunSnip1]
+
+ /** Sample of PipelineOptions with a ValueProvider option argument. */
+ public interface MyOptions extends PipelineOptions {
+ @Description("My option")
+ @Default.String("Hello world!")
+ ValueProvider<String> getStringValue();
+
+ void setStringValue(ValueProvider<String> value);
+ }
+
+ public static void accessingValueProviderInfoAfterRunSnip1(String[] args) {
+
+ MyOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(MyOptions.class);
+
+ // Create pipeline.
+ Pipeline p = Pipeline.create(options);
+
+ // Add a branch for logging the ValueProvider value.
+ p.apply(Create.of(1))
+ .apply(
+ ParDo.of(
+ new DoFn<Integer, Integer>() {
+
+ // Define the DoFn that logs the ValueProvider value.
+ @ProcessElement
+ public void process(ProcessContext c) {
+
+ MyOptions ops = c.getPipelineOptions().as(MyOptions.class);
+ // This example logs the ValueProvider value, but you could store it by
+ // pushing it to an external database.
+
+ LOG.info("Option StringValue was {}", ops.getStringValue());
+ }
+ }));
+
+ // The main pipeline.
+ p.apply(Create.of(1, 2, 3, 4)).apply(Sum.integersGlobally());
+
+ p.run();
+ }
+
+ // [END AccessingValueProviderInfoAfterRunSnip1]
+
+ private static final Duration gapDuration = Duration.standardSeconds(10L);
+
+ // [START CustomSessionWindow1]
+
+ public Collection<IntervalWindow> assignWindows(WindowFn.AssignContext c) {
+
+ // Assign each element into a window from its timestamp until gapDuration in the
+ // future. Overlapping windows (representing elements within gapDuration of
+ // each other) will be merged.
+ return Arrays.asList(new IntervalWindow(c.timestamp(), gapDuration));
+ }
+ // [END CustomSessionWindow1]
+
+ // [START CustomSessionWindow2]
+ public static class DynamicSessions extends WindowFn<TableRow, IntervalWindow> {
+ /** Duration of the gaps between sessions. */
+ private final Duration gapDuration;
+
+ /** Creates a {@code DynamicSessions} {@link WindowFn} with the specified gap duration. */
+ private DynamicSessions(Duration gapDuration) {
+ this.gapDuration = gapDuration;
+ }
+
+ // [END CustomSessionWindow2]
+
+ // [START CustomSessionWindow3]
+ @Override
+ public Collection<IntervalWindow> assignWindows(AssignContext c) {
+ // Assign each element into a window from its timestamp until gapDuration in the
+ // future. Overlapping windows (representing elements within gapDuration of
+ // each other) will be merged.
+ Duration dataDrivenGap;
+ TableRow message = c.element();
+
+ try {
+ dataDrivenGap = Duration.standardSeconds(Long.parseLong(message.get("gap").toString()));
+ } catch (Exception e) {
+ dataDrivenGap = gapDuration;
+ }
+ return Arrays.asList(new IntervalWindow(c.timestamp(), dataDrivenGap));
+ }
+ // [END CustomSessionWindow3]
+
+ // [START CustomSessionWindow4]
+ /** Creates a {@code DynamicSessions} {@link WindowFn} with the specified gap duration. */
+ public static DynamicSessions withDefaultGapDuration(Duration gapDuration) {
+ return new DynamicSessions(gapDuration);
+ }
+
+ // [END CustomSessionWindow4]
+
+ @Override
+ public void mergeWindows(MergeContext c) throws Exception {}
+
+ @Override
+ public boolean isCompatible(WindowFn<?, ?> other) {
+ return false;
+ }
+
+ @Override
+ public Coder<IntervalWindow> windowCoder() {
+ return null;
+ }
+
+ @Override
+ public WindowMappingFn<IntervalWindow> getDefaultWindowMappingFn() {
+ return null;
+ }
+ }
+
+ public static class CustomSessionPipeline {
+
+ public static void main(String[] args) {
+
+ // [START CustomSessionWindow5]
+
+ PCollection<TableRow> p =
+ Pipeline.create()
+ .apply(
+ "Create data",
+ Create.timestamped(
+ TimestampedValue.of(
+ new TableRow().set("user", "mobile").set("score", 12).set("gap", 5),
+ new Instant()),
+ TimestampedValue.of(
+ new TableRow().set("user", "desktop").set("score", 4), new Instant()),
+ TimestampedValue.of(
+ new TableRow().set("user", "mobile").set("score", -3).set("gap", 5),
+ new Instant().plus(2000)),
+ TimestampedValue.of(
+ new TableRow().set("user", "mobile").set("score", 2).set("gap", 5),
+ new Instant().plus(9000)),
+ TimestampedValue.of(
+ new TableRow().set("user", "mobile").set("score", 7).set("gap", 5),
+ new Instant().plus(12000)),
+ TimestampedValue.of(
+ new TableRow().set("user", "desktop").set("score", 10),
+ new Instant().plus(12000))));
+ // [END CustomSessionWindow5]
+
+ // [START CustomSessionWindow6]
+ p.apply(
+ "Window into sessions",
+ Window.<TableRow>into(
+ DynamicSessions.withDefaultGapDuration(Duration.standardSeconds(10))));
+ // [END CustomSessionWindow6]
+
+ }
+ }
}
diff --git a/examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessCommandLineArgs.java b/examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessCommandLineArgs.java
index 60d98ca..b29b3c2 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessCommandLineArgs.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessCommandLineArgs.java
@@ -18,7 +18,7 @@
package org.apache.beam.examples.subprocess.kernel;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** Parameters to the sub-process, has tuple of ordinal position and the value. */
public class SubProcessCommandLineArgs {
diff --git a/examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/CallingSubProcessUtils.java b/examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/CallingSubProcessUtils.java
index 9ce8686..a6b1fb4 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/CallingSubProcessUtils.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/CallingSubProcessUtils.java
@@ -22,7 +22,7 @@
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Semaphore;
import org.apache.beam.examples.subprocess.configuration.SubProcessConfiguration;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java b/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java
index ff95248..a767560 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java
@@ -21,7 +21,7 @@
import java.nio.charset.StandardCharsets;
import org.apache.beam.examples.DebuggingWordCount.WordCountOptions;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/MinimalWordCountTest.java b/examples/java/src/test/java/org/apache/beam/examples/MinimalWordCountTest.java
index d7039d3..8f86748 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/MinimalWordCountTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/MinimalWordCountTest.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
index 1933e9a..a758417 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WindowedWordCountIT.java
@@ -45,9 +45,9 @@
import org.apache.beam.sdk.util.NumberedShardedFile;
import org.apache.beam.sdk.util.ShardedFile;
import org.apache.beam.sdk.util.Sleeper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
import org.joda.time.Duration;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java b/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
index ce81aa7..6b9916e 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
@@ -28,11 +28,9 @@
import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnTester;
import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
-import org.hamcrest.CoreMatchers;
-import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -46,15 +44,12 @@
/** Example test that tests a specific {@link DoFn}. */
@Test
public void testExtractWordsFn() throws Exception {
- DoFnTester<String, String> extractWordsFn = DoFnTester.of(new ExtractWordsFn());
-
- Assert.assertThat(
- extractWordsFn.processBundle(" some input words "),
- CoreMatchers.hasItems("some", "input", "words"));
- Assert.assertThat(extractWordsFn.processBundle(" "), CoreMatchers.hasItems());
- Assert.assertThat(
- extractWordsFn.processBundle(" some ", " input", " words"),
- CoreMatchers.hasItems("some", "input", "words"));
+ List<String> words = Arrays.asList(" some input words ", " ", " cool ", " foo", " bar");
+ PCollection<String> output =
+ p.apply(Create.of(words).withCoder(StringUtf8Coder.of()))
+ .apply(ParDo.of(new ExtractWordsFn()));
+ PAssert.that(output).containsInAnyOrder("some", "input", "words", "cool", "foo", "bar");
+ p.run().waitUntilFinish();
}
static final String[] WORDS_ARRAY =
diff --git a/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfTest.java b/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfTest.java
index fda6449..2190f35 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/complete/TfIdfTest.java
@@ -22,7 +22,6 @@
import org.apache.beam.sdk.coders.StringDelegateCoder;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.Distinct;
import org.apache.beam.sdk.transforms.Keys;
@@ -30,7 +29,6 @@
import org.apache.beam.sdk.values.PCollection;
import org.junit.Rule;
import org.junit.Test;
-import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -42,7 +40,6 @@
/** Test that the example runs. */
@Test
- @Category(ValidatesRunner.class)
public void testTfIdf() throws Exception {
pipeline.getCoderRegistry().registerCoderForClass(URI.class, StringDelegateCoder.of(URI.class));
diff --git a/examples/java/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java b/examples/java/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
index 962d5fa..4632887 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/complete/game/UserScoreTest.java b/examples/java/src/test/java/org/apache/beam/examples/complete/game/UserScoreTest.java
index 12956e3..035c710 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/complete/game/UserScoreTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/complete/game/UserScoreTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
index 2f60e0c..a67273d 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesTest.java b/examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesTest.java
index 107097c..82e31f1 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/cookbook/MaxPerKeyExamplesTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java b/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
index 61dc6eb..3108c47 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java b/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java
index ec51801..d7fd06c 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/examples/kotlin/README.md b/examples/kotlin/README.md
index a820a36..6e3cbb8 100644
--- a/examples/kotlin/README.md
+++ b/examples/kotlin/README.md
@@ -30,7 +30,7 @@
1. [`MinimalWordCount`](https://github.com/apache/beam/blob/master/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/MinimalWordCount.kt) is the simplest word count pipeline and introduces basic concepts like [Pipelines](https://beam.apache.org/documentation/programming-guide/#pipeline),
[PCollections](https://beam.apache.org/documentation/programming-guide/#pcollection),
-[ParDo](https://beam.apache.org/documentation/programming-guide/#transforms-pardo),
+[ParDo](https://beam.apache.org/documentation/programming-guide/#pardo),
and [reading and writing data](https://beam.apache.org/documentation/programming-guide/#io) from external storage.
1. [`WordCount`](https://github.com/apache/beam/blob/master/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/WordCount.kt) introduces best practices like [PipelineOptions](https://beam.apache.org/documentation/programming-guide/#pipeline) and custom [PTransforms](https://beam.apache.org/documentation/programming-guide/#transforms-composite).
diff --git a/examples/kotlin/build.gradle b/examples/kotlin/build.gradle
index ff54658..7b55870 100644
--- a/examples/kotlin/build.gradle
+++ b/examples/kotlin/build.gradle
@@ -22,7 +22,7 @@
id 'org.jetbrains.kotlin.jvm' version '1.3.21'
}
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.examples.kotlin')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -48,45 +48,45 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadow library.java.google_api_client
- shadow library.java.google_api_services_bigquery
- shadow library.java.google_http_client
- shadow library.java.bigdataoss_util
- shadow library.java.google_auth_library_oauth2_http
- shadow library.java.google_auth_library_credentials
- shadow library.java.avro
- shadow library.java.google_api_services_pubsub
- shadow library.java.datastore_v1_proto_client
- shadow library.java.datastore_v1_protos
- shadow library.java.joda_time
- shadow library.java.slf4j_api
- shadow library.java.slf4j_jdk14
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:io:google-cloud-platform")
+ compile library.java.avro
+ compile library.java.bigdataoss_util
+ compile library.java.google_api_client
+ compile library.java.google_api_services_bigquery
+ compile library.java.google_api_services_pubsub
+ compile library.java.google_auth_library_credentials
+ compile library.java.google_auth_library_oauth2_http
+ compile library.java.google_cloud_datastore_v1_proto_client
+ compile library.java.google_http_client
+ compile library.java.joda_time
+ compile library.java.proto_google_cloud_datastore_v1
+ compile library.java.slf4j_api
+ compile library.java.slf4j_jdk14
runtime project(path: ":runners:direct-java", configuration: "shadow")
- shadowTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
+ testCompile project(":sdks:java:io:google-cloud-platform")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
// Add dependencies for the PreCommit configurations
// For each runner a project level dependency on the examples project.
for (String runner : preCommitRunners) {
- delegate.add(runner + "PreCommit", project(path: ":examples:kotlin", configuration: "shadow"))
- delegate.add(runner + "PreCommit", project(path: ":examples:kotlin", configuration: "shadowTest"))
+ delegate.add(runner + "PreCommit", project(":examples:kotlin"))
+ delegate.add(runner + "PreCommit", project(path: ":examples:kotlin", configuration: "testRuntime"))
}
// https://issues.apache.org/jira/browse/BEAM-3583
- // apexRunnerPreCommit project(path: ":beam-runners-apex", configuration: "shadow")
+ // apexRunnerPreCommit project(":runners:apex")
directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow")
- flinkRunnerPreCommit project(path: ":runners:flink:1.5", configuration: "shadow")
+ flinkRunnerPreCommit project(":runners:flink:1.8")
// TODO: Make the netty version used configurable, we add netty-all 4.1.17.Final so it appears on the classpath
// before 4.1.8.Final defined by Apache Beam
sparkRunnerPreCommit "io.netty:netty-all:4.1.17.Final"
- sparkRunnerPreCommit project(path: ":runners:spark", configuration: "shadow")
- sparkRunnerPreCommit project(path: ":sdks:java:io:hadoop-file-system", configuration: "shadow")
+ sparkRunnerPreCommit project(":runners:spark")
+ sparkRunnerPreCommit project(":sdks:java:io:hadoop-file-system")
sparkRunnerPreCommit library.java.spark_streaming
sparkRunnerPreCommit library.java.spark_core
implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8"
diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt
index 4eab556..7fd9aeb 100644
--- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt
+++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/ExampleUtils.kt
@@ -38,10 +38,10 @@
import org.apache.beam.sdk.util.BackOffUtils
import org.apache.beam.sdk.util.FluentBackoff
import org.apache.beam.sdk.util.Sleeper
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles
import org.joda.time.Duration
import java.io.IOException
import java.util.concurrent.TimeUnit
diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/WriteOneFilePerWindow.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/WriteOneFilePerWindow.kt
index cbe38d6..a73fc76 100644
--- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/WriteOneFilePerWindow.kt
+++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/common/WriteOneFilePerWindow.kt
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo
import org.apache.beam.sdk.values.PCollection
import org.apache.beam.sdk.values.PDone
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull
import org.joda.time.format.ISODateTimeFormat
/**
diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt
index e4019ba..91a49a4 100644
--- a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt
+++ b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.ParDo
import org.apache.beam.sdk.values.KV
import org.apache.beam.sdk.values.PCollection
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists
/**
diff --git a/examples/kotlin/src/test/java/org/apache/beam/examples/DebuggingWordCountTestKotlin.java b/examples/kotlin/src/test/java/org/apache/beam/examples/DebuggingWordCountTestKotlin.java
index 4da3b35..1632bfc 100644
--- a/examples/kotlin/src/test/java/org/apache/beam/examples/DebuggingWordCountTestKotlin.java
+++ b/examples/kotlin/src/test/java/org/apache/beam/examples/DebuggingWordCountTestKotlin.java
@@ -21,7 +21,7 @@
import java.nio.charset.StandardCharsets;
import org.apache.beam.examples.kotlin.DebuggingWordCount;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
diff --git a/examples/kotlin/src/test/java/org/apache/beam/examples/MinimalWordCountTestKotlin.java b/examples/kotlin/src/test/java/org/apache/beam/examples/MinimalWordCountTestKotlin.java
index 8c029ac..55583b9 100644
--- a/examples/kotlin/src/test/java/org/apache/beam/examples/MinimalWordCountTestKotlin.java
+++ b/examples/kotlin/src/test/java/org/apache/beam/examples/MinimalWordCountTestKotlin.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/examples/kotlin/src/test/java/org/apache/beam/examples/WindowedWordCountITKotlin.kt b/examples/kotlin/src/test/java/org/apache/beam/examples/WindowedWordCountITKotlin.kt
index 8d25bd4..ee2ec23 100644
--- a/examples/kotlin/src/test/java/org/apache/beam/examples/WindowedWordCountITKotlin.kt
+++ b/examples/kotlin/src/test/java/org/apache/beam/examples/WindowedWordCountITKotlin.kt
@@ -32,9 +32,9 @@
import org.apache.beam.sdk.testing.TestPipelineOptions
import org.apache.beam.sdk.transforms.windowing.IntervalWindow
import org.apache.beam.sdk.util.*
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists
import org.hamcrest.Description
import org.hamcrest.Matchers.equalTo
import org.hamcrest.TypeSafeMatcher
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
new file mode 100644
index 0000000..70da228
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
@@ -0,0 +1,512 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/filter\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "filter"
+ },
+ "source": [
+ "# Filter\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Given a predicate, filter out all elements that don't satisfy that predicate.\n",
+ "May also be used to filter based on an inequality with a given value based\n",
+ "on the comparison ordering of the element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Filter` in multiple ways to filter out produce by their duration value.\n",
+ "\n",
+ "`Filter` accepts a function that keeps elements that return `True`, and filters out the remaining elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-filtering-with-a-function"
+ },
+ "source": [
+ "### Example 1: Filtering with a function\n",
+ "\n",
+ "We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-filtering-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def is_perennial(plant):\n",
+ " return plant['duration'] == 'perennial'\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(is_perennial)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-filtering-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-filtering-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 2: Filtering with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 1**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-filtering-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(\n",
+ " lambda plant: plant['duration'] == 'perennial')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-filtering-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-filtering-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 3: Filtering with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Filter`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `has_duration` takes `plant` and `duration` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-filtering-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def has_duration(plant, duration):\n",
+ " return plant['duration'] == duration\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(has_duration, 'perennial')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-filtering-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-filtering-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 4: Filtering with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `'perennial'` as a singleton.\n",
+ "We then use that value to filter out perennials."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-filtering-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " perennial = pipeline | 'Perennial' >> beam.Create(['perennial'])\n",
+ "\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter perennials' >> beam.Filter(\n",
+ " lambda plant, duration: plant['duration'] == duration,\n",
+ " duration=beam.pvalue.AsSingleton(perennial),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-filtering-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-filtering-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 5: Filtering with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-filtering-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " valid_durations = pipeline | 'Valid durations' >> beam.Create([\n",
+ " 'annual',\n",
+ " 'biennial',\n",
+ " 'perennial',\n",
+ " ])\n",
+ "\n",
+ " valid_plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'PERENNIAL'},\n",
+ " ])\n",
+ " | 'Filter valid plants' >> beam.Filter(\n",
+ " lambda plant, valid_durations: plant['duration'] in valid_durations,\n",
+ " valid_durations=beam.pvalue.AsIter(valid_durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-filtering-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-filtering-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 6: Filtering with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-filtering-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " keep_duration = pipeline | 'Duration filters' >> beam.Create([\n",
+ " ('annual', False),\n",
+ " ('biennial', False),\n",
+ " ('perennial', True),\n",
+ " ])\n",
+ "\n",
+ " perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Filter plants by duration' >> beam.Filter(\n",
+ " lambda plant, keep_duration: keep_duration[plant['duration']],\n",
+ " keep_duration=beam.pvalue.AsDict(keep_duration),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-filtering-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it might produce zero or more outputs.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/filter\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Filter - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
new file mode 100644
index 0000000..b99e3e9
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
@@ -0,0 +1,672 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/flatmap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "flatmap"
+ },
+ "source": [
+ "# FlatMap\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Applies a simple 1-to-many mapping function over each element in the collection.\n",
+ "The many elements are flattened into the resulting collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `FlatMap` in multiple ways to yield zero or more elements per each input element into the resulting `PCollection`.\n",
+ "\n",
+ "`FlatMap` accepts a function that returns an `iterable`,\n",
+ "where each of the output `iterable`'s elements is an element of the resulting `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-flatmap-with-a-predefined-function"
+ },
+ "source": [
+ "### Example 1: FlatMap with a predefined function\n",
+ "\n",
+ "We use the function `str.split` which takes a single `str` element and outputs a `list` of `str`s.\n",
+ "This pipeline splits the input element using whitespaces, creating a list of zero or more elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-flatmap-with-a-predefined-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry 🥕Carrot 🍆Eggplant',\n",
+ " '🍅Tomato 🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(str.split)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-flatmap-with-a-predefined-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-flatmap-with-a-function"
+ },
+ "source": [
+ "### Example 2: FlatMap with a function\n",
+ "\n",
+ "We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-flatmap-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def split_words(text):\n",
+ " return text.split(',')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(split_words)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-flatmap-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-flatmap-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 3: FlatMap with a lambda function\n",
+ "\n",
+ "For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.\n",
+ "Each input element is already an `iterable`, where each element is what we want in the resulting `PCollection`.\n",
+ "We use a lambda function that returns the same input element it received."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-flatmap-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],\n",
+ " ['🍅Tomato', '🥔Potato'],\n",
+ " ])\n",
+ " | 'Flatten lists' >> beam.FlatMap(lambda elements: elements)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-flatmap-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-flatmap-with-a-generator"
+ },
+ "source": [
+ "### Example 4: FlatMap with a generator\n",
+ "\n",
+ "For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.\n",
+ "We use a generator to iterate over the input list and yield each of the elements.\n",
+ "Each yielded result in the generator is an element in the resulting `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-flatmap-with-a-generator-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def generate_elements(elements):\n",
+ " for element in elements:\n",
+ " yield element\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],\n",
+ " ['🍅Tomato', '🥔Potato'],\n",
+ " ])\n",
+ " | 'Flatten lists' >> beam.FlatMap(generate_elements)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-flatmap-with-a-generator-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-flatmaptuple-for-key-value-pairs"
+ },
+ "source": [
+ "### Example 5: FlatMapTuple for key-value pairs\n",
+ "\n",
+ "If your `PCollection` consists of `(key, value)` pairs,\n",
+ "you can use `FlatMapTuple` to unpack them into different function arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-flatmaptuple-for-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def format_plant(icon, plant):\n",
+ " if icon:\n",
+ " yield '{}{}'.format(icon, plant)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " (None, 'Invalid'),\n",
+ " ])\n",
+ " | 'Format' >> beam.FlatMapTuple(format_plant)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-flatmaptuple-for-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-flatmap-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 6: FlatMap with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `FlatMap`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `split_words` takes `text` and `delimiter` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-flatmap-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def split_words(text, delimiter=None):\n",
+ " return text.split(delimiter)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(split_words, delimiter=',')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-flatmap-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-flatmap-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 7: FlatMap with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `','` as a singleton.\n",
+ "We then use that value as the delimiter for the `str.split` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-flatmap-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " delimiter = pipeline | 'Create delimiter' >> beam.Create([','])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.FlatMap(\n",
+ " lambda text, delimiter: text.split(delimiter),\n",
+ " delimiter=beam.pvalue.AsSingleton(delimiter),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-flatmap-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-flatmap-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 8: FlatMap with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-flatmap-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def normalize_and_validate_durations(plant, valid_durations):\n",
+ " plant['duration'] = plant['duration'].lower()\n",
+ " if plant['duration'] in valid_durations:\n",
+ " yield plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " valid_durations = pipeline | 'Valid durations' >> beam.Create([\n",
+ " 'annual',\n",
+ " 'biennial',\n",
+ " 'perennial',\n",
+ " ])\n",
+ "\n",
+ " valid_plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'Perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'BIENNIAL'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'unknown'},\n",
+ " ])\n",
+ " | 'Normalize and validate durations' >> beam.FlatMap(\n",
+ " normalize_and_validate_durations,\n",
+ " valid_durations=beam.pvalue.AsIter(valid_durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-flatmap-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-flatmap-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 9: FlatMap with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-9-flatmap-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def replace_duration_if_valid(plant, durations):\n",
+ " if plant['duration'] in durations:\n",
+ " plant['duration'] = durations[plant['duration']]\n",
+ " yield plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " durations = pipeline | 'Durations dict' >> beam.Create([\n",
+ " (0, 'annual'),\n",
+ " (1, 'biennial'),\n",
+ " (2, 'perennial'),\n",
+ " ])\n",
+ "\n",
+ " valid_plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 1},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 0},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': -1},\n",
+ " ])\n",
+ " | 'Replace duration if valid' >> beam.FlatMap(\n",
+ " replace_duration_if_valid,\n",
+ " durations=beam.pvalue.AsDict(durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-flatmap-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/flatmap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "FlatMap - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
new file mode 100644
index 0000000..b0be11e
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/keys\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "keys"
+ },
+ "source": [
+ "# Keys\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs and returns the key of each element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example"
+ },
+ "source": [
+ "## Example\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `Keys` to extract the keys and discard the values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " icons = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Keys' >> beam.Keys()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [KvSwap](https://beam.apache.org/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.\n",
+ "* [Values](https://beam.apache.org/documentation/transforms/python/elementwise/values) for extracting the value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Keys\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/keys\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Keys - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
new file mode 100644
index 0000000..25046a5
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
@@ -0,0 +1,196 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/kvswap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "kvswap"
+ },
+ "source": [
+ "# Kvswap\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs and returns a collection of key-value pairs\n",
+ "which has each key and value swapped."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `KvSwap` to swap the keys and values."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "examples-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Key-Value swap' >> beam.KvSwap()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Keys](https://beam.apache.org/documentation/transforms/python/elementwise/keys) for extracting the key of each component.\n",
+ "* [Values](https://beam.apache.org/documentation/transforms/python/elementwise/values) for extracting the value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.KvSwap\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/kvswap\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "KvSwap - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
new file mode 100644
index 0000000..471bef4
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
@@ -0,0 +1,616 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/map\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "map"
+ },
+ "source": [
+ "# Map\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Applies a simple 1-to-1 mapping function over each element in the collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.\n",
+ "\n",
+ "`Map` accepts a function that returns a single element for every input element in the `PCollection`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function"
+ },
+ "source": [
+ "### Example 1: Map with a predefined function\n",
+ "\n",
+ "We use the function `str.strip` which takes a single `str` element and outputs a `str`.\n",
+ "It strips the input element's whitespaces, including newlines and tabs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ' 🍓Strawberry \\n',\n",
+ " ' 🥕Carrot \\n',\n",
+ " ' 🍆Eggplant \\n',\n",
+ " ' 🍅Tomato \\n',\n",
+ " ' 🥔Potato \\n',\n",
+ " ])\n",
+ " | 'Strip' >> beam.Map(str.strip)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-map-with-a-predefined-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-map-with-a-function"
+ },
+ "source": [
+ "### Example 2: Map with a function\n",
+ "\n",
+ "We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\\n'` characters from each element."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-map-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def strip_header_and_newline(text):\n",
+ " return text.strip('# \\n')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(strip_header_and_newline)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-map-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 3: Map with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 2**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(lambda text: text.strip('# \\n'))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-map-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 4: Map with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Map`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In this example, `strip` takes `text` and `chars` as arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def strip(text, chars=None):\n",
+ " return text.strip(chars)\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(strip, chars='# \\n')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-map-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs"
+ },
+ "source": [
+ "### Example 5: MapTuple for key-value pairs\n",
+ "\n",
+ "If your `PCollection` consists of `(key, value)` pairs,\n",
+ "you can use `MapTuple` to unpack them into different function arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Format' >> beam.MapTuple(\n",
+ " lambda icon, plant: '{}{}'.format(icon, plant))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-maptuple-for-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons"
+ },
+ "source": [
+ "### Example 6: Map with side inputs as singletons\n",
+ "\n",
+ "If the `PCollection` has a single value, such as the average from another computation,\n",
+ "passing the `PCollection` as a *singleton* accesses that value.\n",
+ "\n",
+ "In this example, we pass a `PCollection` the value `'# \\n'` as a singleton.\n",
+ "We then use that value as the characters for the `str.strip` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " chars = pipeline | 'Create chars' >> beam.Create(['# \\n'])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(\n",
+ " lambda text, chars: text.strip(chars),\n",
+ " chars=beam.pvalue.AsSingleton(chars),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-map-with-side-inputs-as-singletons-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators"
+ },
+ "source": [
+ "### Example 7: Map with side inputs as iterators\n",
+ "\n",
+ "If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.\n",
+ "This accesses elements lazily as they are needed,\n",
+ "so it is possible to iterate over large `PCollection`s that won't fit into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " chars = pipeline | 'Create chars' >> beam.Create(['#', ' ', '\\n'])\n",
+ "\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '# 🍓Strawberry\\n',\n",
+ " '# 🥕Carrot\\n',\n",
+ " '# 🍆Eggplant\\n',\n",
+ " '# 🍅Tomato\\n',\n",
+ " '# 🥔Potato\\n',\n",
+ " ])\n",
+ " | 'Strip header' >> beam.Map(\n",
+ " lambda text, chars: text.strip(''.join(chars)),\n",
+ " chars=beam.pvalue.AsIter(chars),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-map-with-side-inputs-as-iterators-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,\n",
+ "> but this requires that all the elements fit into memory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries"
+ },
+ "source": [
+ "### Example 8: Map with side inputs as dictionaries\n",
+ "\n",
+ "If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.\n",
+ "Each element must be a `(key, value)` pair.\n",
+ "Note that all the elements of the `PCollection` must fit into memory for this.\n",
+ "If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "def replace_duration(plant, durations):\n",
+ " plant['duration'] = durations[plant['duration']]\n",
+ " return plant\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " durations = pipeline | 'Durations' >> beam.Create([\n",
+ " (0, 'annual'),\n",
+ " (1, 'biennial'),\n",
+ " (2, 'perennial'),\n",
+ " ])\n",
+ "\n",
+ " plant_details = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 1},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 0},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 2},\n",
+ " ])\n",
+ " | 'Replace duration' >> beam.Map(\n",
+ " replace_duration,\n",
+ " durations=beam.pvalue.AsDict(durations),\n",
+ " )\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-map-with-side-inputs-as-dictionaries-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it may produce zero or more outputs.\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/map\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Map - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
new file mode 100644
index 0000000..1bc48a8
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
@@ -0,0 +1,414 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/pardo\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "pardo"
+ },
+ "source": [
+ "# ParDo\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "A transform for generic parallel processing.\n",
+ "A `ParDo` transform considers each element in the input `PCollection`,\n",
+ "performs some processing function (your user code) on that element,\n",
+ "and emits zero or more elements to an output `PCollection`.\n",
+ "\n",
+ "See more information in the\n",
+ "[Beam Programming Guide](https://beam.apache.org/documentation/programming-guide/#pardo)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we explore how to create custom `DoFn`s and access\n",
+ "the timestamp and windowing information."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn"
+ },
+ "source": [
+ "### Example 1: ParDo with a simple DoFn\n",
+ "\n",
+ "The following example defines a simple `DoFn` class called `SplitWords`\n",
+ "which stores the `delimiter` as an object field.\n",
+ "The `process` method is called once per element,\n",
+ "and it can yield zero or more output elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class SplitWords(beam.DoFn):\n",
+ " def __init__(self, delimiter=','):\n",
+ " self.delimiter = delimiter\n",
+ "\n",
+ " def process(self, text):\n",
+ " for word in text.split(self.delimiter):\n",
+ " yield word\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " '🍓Strawberry,🥕Carrot,🍆Eggplant',\n",
+ " '🍅Tomato,🥔Potato',\n",
+ " ])\n",
+ " | 'Split words' >> beam.ParDo(SplitWords(','))\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-pardo-with-a-simple-dofn-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information"
+ },
+ "source": [
+ "### Example 2: ParDo with timestamp and window information\n",
+ "\n",
+ "In this example, we add new parameters to the `process` method to bind parameter values at runtime.\n",
+ "\n",
+ "* [`beam.DoFn.TimestampParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam)\n",
+ " binds the timestamp information as an\n",
+ " [`apache_beam.utils.timestamp.Timestamp`](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp)\n",
+ " object.\n",
+ "* [`beam.DoFn.WindowParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam)\n",
+ " binds the window information as the appropriate\n",
+ " [`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)\n",
+ " object."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class AnalyzeElement(beam.DoFn):\n",
+ " def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):\n",
+ " yield '\\n'.join([\n",
+ " '# timestamp',\n",
+ " 'type(timestamp) -> ' + repr(type(timestamp)),\n",
+ " 'timestamp.micros -> ' + repr(timestamp.micros),\n",
+ " 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),\n",
+ " 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),\n",
+ " '',\n",
+ " '# window',\n",
+ " 'type(window) -> ' + repr(type(window)),\n",
+ " 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),\n",
+ " 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),\n",
+ " 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),\n",
+ " ])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " dofn_params = (\n",
+ " pipeline\n",
+ " | 'Create a single test element' >> beam.Create([':)'])\n",
+ " | 'Add timestamp (Spring equinox 2020)' >> beam.Map(\n",
+ " lambda elem: beam.window.TimestampedValue(elem, 1584675660))\n",
+ " | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))\n",
+ " | 'Analyze element' >> beam.ParDo(AnalyzeElement())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-pardo-with-timestamp-and-window-information-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods"
+ },
+ "source": [
+ "### Example 3: ParDo with DoFn methods\n",
+ "\n",
+ "A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)\n",
+ "can be customized with a number of methods that can help create more complex behaviors.\n",
+ "You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.\n",
+ "You can also customize what to do when a\n",
+ "[*bundle of elements*](https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence)\n",
+ "starts and finishes with `start_bundle` and `finish_bundle`.\n",
+ "\n",
+ "* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):\n",
+ " Called *once per `DoFn` instance* when the `DoFn` instance is initialized.\n",
+ " `setup` need not to be cached, so it could be called more than once per worker.\n",
+ " This is a good place to connect to database instances, open network connections or other resources.\n",
+ "\n",
+ "* [`DoFn.start_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle):\n",
+ " Called *once per bundle of elements* before calling `process` on the first element of the bundle.\n",
+ " This is a good place to start keeping track of the bundle elements.\n",
+ "\n",
+ "* [**`DoFn.process(element, *args, **kwargs)`**](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process):\n",
+ " Called *once per element*, can *yield zero or more elements*.\n",
+ " Additional `*args` or `**kwargs` can be passed through\n",
+ " [`beam.ParDo()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo).\n",
+ " **[required]**\n",
+ "\n",
+ "* [`DoFn.finish_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle):\n",
+ " Called *once per bundle of elements* after calling `process` after the last element of the bundle,\n",
+ " can *yield zero or more elements*. This is a good place to do batch calls on a bundle of elements,\n",
+ " such as running a database query.\n",
+ "\n",
+ " For example, you can initialize a batch in `start_bundle`,\n",
+ " add elements to the batch in `process` instead of yielding them,\n",
+ " then running a batch query on those elements on `finish_bundle`, and yielding all the results.\n",
+ "\n",
+ " Note that yielded elements from `finish_bundle` must be of the type\n",
+ " [`apache_beam.utils.windowed_value.WindowedValue`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py).\n",
+ " You need to provide a timestamp as a unix timestamp, which you can get from the last processed element.\n",
+ " You also need to provide a window, which you can get from the last processed element like in the example below.\n",
+ "\n",
+ "* [`DoFn.teardown()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown):\n",
+ " Called *once (as a best effort) per `DoFn` instance* when the `DoFn` instance is shutting down.\n",
+ " This is a good place to close database instances, close network connections or other resources.\n",
+ "\n",
+ " Note that `teardown` is called as a *best effort* and is *not guaranteed*.\n",
+ " For example, if the worker crashes, `teardown` might not be called."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class DoFnMethods(beam.DoFn):\n",
+ " def __init__(self):\n",
+ " print('__init__')\n",
+ " self.window = beam.window.GlobalWindow()\n",
+ "\n",
+ " def setup(self):\n",
+ " print('setup')\n",
+ "\n",
+ " def start_bundle(self):\n",
+ " print('start_bundle')\n",
+ "\n",
+ " def process(self, element, window=beam.DoFn.WindowParam):\n",
+ " self.window = window\n",
+ " yield '* process: ' + element\n",
+ "\n",
+ " def finish_bundle(self):\n",
+ " yield beam.utils.windowed_value.WindowedValue(\n",
+ " value='* finish_bundle: 🌱🌳🌍',\n",
+ " timestamp=0,\n",
+ " windows=[self.window],\n",
+ " )\n",
+ "\n",
+ " def teardown(self):\n",
+ " print('teardown')\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " results = (\n",
+ " pipeline\n",
+ " | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])\n",
+ " | 'DoFn methods' >> beam.ParDo(DoFnMethods())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-pardo-with-dofn-methods-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "> *Known issues:*\n",
+ ">\n",
+ "> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)\n",
+ "> `DoFn.setup()` doesn't run for streaming jobs running in the `DirectRunner`.\n",
+ "> * [[BEAM-7340]](https://issues.apache.org/jira/browse/BEAM-7340)\n",
+ "> `DoFn.teardown()` metrics are lost."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`,\n",
+ " but for each input it may produce zero or more outputs.\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/pardo\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "ParDo - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
new file mode 100644
index 0000000..ea520e5
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
@@ -0,0 +1,404 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/partition\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "partition"
+ },
+ "source": [
+ "# Partition\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Separates elements in a collection into multiple output\n",
+ "collections. The partitioning function contains the logic that determines how\n",
+ "to separate the elements of the input collection into each resulting\n",
+ "partition output collection.\n",
+ "\n",
+ "The number of partitions must be determined at graph construction time.\n",
+ "You cannot determine the number of partitions in mid-pipeline\n",
+ "\n",
+ "See more information in the [Beam Programming Guide](https://beam.apache.org/documentation/programming-guide/#partition)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.\n",
+ "Then, we apply `Partition` in multiple ways to split the `PCollection` into multiple `PCollections`.\n",
+ "\n",
+ "`Partition` accepts a function that receives the number of partitions,\n",
+ "and returns the index of the desired partition for the element.\n",
+ "The number of partitions passed must be a positive integer,\n",
+ "and it must return an integer in the range `0` to `num_partitions-1`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-partition-with-a-function"
+ },
+ "source": [
+ "### Example 1: Partition with a function\n",
+ "\n",
+ "In the following example, we have a known list of durations.\n",
+ "We partition the `PCollection` into one `PCollection` for every duration type."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-partition-with-a-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "durations = ['annual', 'biennial', 'perennial']\n",
+ "\n",
+ "def by_duration(plant, num_partitions):\n",
+ " return durations.index(plant['duration'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " annuals, biennials, perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(by_duration, len(durations))\n",
+ " )\n",
+ " _ = (\n",
+ " annuals\n",
+ " | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " biennials\n",
+ " | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " perennials\n",
+ " | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-partition-with-a-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function"
+ },
+ "source": [
+ "### Example 2: Partition with a lambda function\n",
+ "\n",
+ "We can also use lambda functions to simplify **Example 1**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "durations = ['annual', 'biennial', 'perennial']\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " annuals, biennials, perennials = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(\n",
+ " lambda plant, num_partitions: durations.index(plant['duration']),\n",
+ " len(durations),\n",
+ " )\n",
+ " )\n",
+ " _ = (\n",
+ " annuals\n",
+ " | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " biennials\n",
+ " | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " perennials\n",
+ " | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-partition-with-a-lambda-function-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments"
+ },
+ "source": [
+ "### Example 3: Partition with multiple arguments\n",
+ "\n",
+ "You can pass functions with multiple arguments to `Partition`.\n",
+ "They are passed as additional positional arguments or keyword arguments to the function.\n",
+ "\n",
+ "In machine learning, it is a common task to split data into\n",
+ "[training and a testing datasets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).\n",
+ "Typically, 80% of the data is used for training a model and 20% is used for testing.\n",
+ "\n",
+ "In this example, we split a `PCollection` dataset into training and testing datasets.\n",
+ "We define `split_dataset`, which takes the `plant` element, `num_partitions`,\n",
+ "and an additional argument `ratio`.\n",
+ "The `ratio` is a list of numbers which represents the ratio of how many items will go into each partition.\n",
+ "`num_partitions` is used by `Partitions` as a positional argument,\n",
+ "while `plant` and `ratio` are passed to `split_dataset`.\n",
+ "\n",
+ "If we want an 80%/20% split, we can specify a ratio of `[8, 2]`, which means that for every 10 elements,\n",
+ "8 go into the first partition and 2 go into the second.\n",
+ "In order to determine which partition to send each element, we have different buckets.\n",
+ "For our case `[8, 2]` has **10** buckets,\n",
+ "where the first 8 buckets represent the first partition and the last 2 buckets represent the second partition.\n",
+ "\n",
+ "First, we check that the ratio list's length corresponds to the `num_partitions` we pass.\n",
+ "We then get a bucket index for each element, in the range from 0 to 9 (`num_buckets-1`).\n",
+ "We could do `hash(element) % len(ratio)`, but instead we sum all the ASCII characters of the\n",
+ "JSON representation to make it deterministic.\n",
+ "Finally, we loop through all the elements in the ratio and have a running total to\n",
+ "identify the partition index to which that bucket corresponds.\n",
+ "\n",
+ "This `split_dataset` function is generic enough to support any number of partitions by any ratio.\n",
+ "You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "import json\n",
+ "\n",
+ "def split_dataset(plant, num_partitions, ratio):\n",
+ " assert num_partitions == len(ratio)\n",
+ " bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)\n",
+ " total = 0\n",
+ " for i, part in enumerate(ratio):\n",
+ " total += part\n",
+ " if bucket < total:\n",
+ " return i\n",
+ " return len(ratio) - 1\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " train_dataset, test_dataset = (\n",
+ " pipeline\n",
+ " | 'Gardening plants' >> beam.Create([\n",
+ " {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},\n",
+ " {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},\n",
+ " {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},\n",
+ " {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},\n",
+ " {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},\n",
+ " ])\n",
+ " | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])\n",
+ " )\n",
+ " _ = (\n",
+ " train_dataset\n",
+ " | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))\n",
+ " )\n",
+ " _ = (\n",
+ " test_dataset\n",
+ " | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-partition-with-multiple-arguments-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Filter](https://beam.apache.org/documentation/transforms/python/elementwise/filter) is useful if the function is just\n",
+ " deciding whether to output an element or not.\n",
+ "* [ParDo](https://beam.apache.org/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping\n",
+ " operation, and includes other abilities such as multiple output collections and side-inputs.\n",
+ "* [CoGroupByKey](https://beam.apache.org/documentation/transforms/python/aggregation/cogroupbykey)\n",
+ "performs a per-key equijoin.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/partition\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Partition - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
new file mode 100644
index 0000000..705ce90
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
@@ -0,0 +1,719 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/regex\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "regex"
+ },
+ "source": [
+ "# Regex\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Filters input string elements based on a regex. May also transform them based on the matching groups."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` of text strings.\n",
+ "Then, we use the `Regex` transform to search, replace, and split through the text elements using\n",
+ "[regular expressions](https://docs.python.org/3/library/re.html).\n",
+ "\n",
+ "You can use tools to help you create and test your regular expressions, such as\n",
+ "[regex101](https://regex101.com/).\n",
+ "Make sure to specify the Python flavor at the left side bar.\n",
+ "\n",
+ "Lets look at the\n",
+ "[regular expression `(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)`](https://regex101.com/r/Z7hTTj/3)\n",
+ "for example.\n",
+ "It matches anything that is not a whitespace `\\s` (`[ \\t\\n\\r\\f\\v]`) or comma `,`\n",
+ "until a comma is found and stores that in the named group `icon`,\n",
+ "this can match even `utf-8` strings.\n",
+ "Then it matches any number of whitespaces, followed by at least one word character\n",
+ "`\\w` (`[a-zA-Z0-9_]`), which is stored in the second group for the *name*.\n",
+ "It does the same with the third group for the *duration*.\n",
+ "\n",
+ "> *Note:* To avoid unexpected string escaping in your regular expressions,\n",
+ "> it is recommended to use\n",
+ "> [raw strings](https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals)\n",
+ "> such as `r'raw-string'` instead of `'escaped-string'`."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-regex-match"
+ },
+ "source": [
+ "### Example 1: Regex match\n",
+ "\n",
+ "`Regex.matches` keeps only the elements that match the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.matches` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find(regex)`](#example-4-regex-find)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-regex-match-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.matches(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-regex-match-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups"
+ },
+ "source": [
+ "### Example 2: Regex match with all groups\n",
+ "\n",
+ "`Regex.all_matches` keeps only the elements that match the regular expression,\n",
+ "returning *all groups* as a list.\n",
+ "The groups are returned in the order encountered in the regular expression,\n",
+ "including `group 0` (the entire match) as the first group.\n",
+ "\n",
+ "`Regex.all_matches` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_all_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.all_matches(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-regex-match-with-all-groups-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs"
+ },
+ "source": [
+ "### Example 3: Regex match into key-value pairs\n",
+ "\n",
+ "`Regex.matches_kv` keeps only the elements that match the regular expression,\n",
+ "returning a key-value pair using the specified groups.\n",
+ "The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "The argument `valueGroup` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.matches_kv` starts to match the regular expression at the beginning of the string.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "To start matching at any point instead of the beginning of the string, use\n",
+ "[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches_kv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial ignoring trailing words',\n",
+ " '🍆, Eggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " '# 🍌, invalid, format',\n",
+ " 'invalid, 🍉, format',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.matches_kv(regex, keyGroup='icon')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-regex-match-into-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-regex-find"
+ },
+ "source": [
+ "### Example 4: Regex find\n",
+ "\n",
+ "`Regex.find` keeps only the elements that match the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.find` matches the first occurrence of the regular expression in the string.\n",
+ "To start matching at the beginning, add `'^'` at the beginning of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match from the start only, consider using\n",
+ "[`Regex.matches(regex)`](#example-1-regex-match)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-4-regex-find-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-4-regex-find-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-regex-find-all"
+ },
+ "source": [
+ "### Example 5: Regex find all\n",
+ "\n",
+ "`Regex.find_all` returns a list of all the matches of the regular expression,\n",
+ "returning the matched group.\n",
+ "The argument `group` is set to `0` by default, but can be set to a group number like `3`, to a named group like `'icon'`, or to `Regex.ALL` to return all groups.\n",
+ "The argument `outputEmpty` is set to `True` by default, but can be set to `False` to skip elements where no matches were found.\n",
+ "\n",
+ "`Regex.find_all` matches the regular expression anywhere it is found in the string.\n",
+ "To start matching at the beginning, add `'^'` at the start of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match all groups from the start only, consider using\n",
+ "[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-5-regex-find-all-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_find_all = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find_all(regex)\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-5-regex-find-all-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs"
+ },
+ "source": [
+ "### Example 6: Regex find as key-value pairs\n",
+ "\n",
+ "`Regex.find_kv` returns a list of all the matches of the regular expression,\n",
+ "returning a key-value pair using the specified groups.\n",
+ "The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "The argument `valueGroup` is set to `0` (the entire match) by default,\n",
+ "but can be set to a group number like `3`, or to a named group like `'icon'`.\n",
+ "\n",
+ "`Regex.find_kv` matches the first occurrence of the regular expression in the string.\n",
+ "To start matching at the beginning, add `'^'` at the beginning of the regular expression.\n",
+ "To match until the end of the string, add `'$'` at the end of the regular expression.\n",
+ "\n",
+ "If you need to match as key-value pairs from the start only, consider using\n",
+ "[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "# Matches a named group 'icon', and then two comma-separated groups.\n",
+ "regex = r'(?P<icon>[^\\s,]+), *(\\w+), *(\\w+)'\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_matches_kv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '# 🍓, Strawberry, perennial',\n",
+ " '# 🥕, Carrot, biennial ignoring trailing words',\n",
+ " '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',\n",
+ " '# 🍅, Tomato, annual - 🍉, Watermelon, annual',\n",
+ " '# 🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.find_kv(regex, keyGroup='icon')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-6-regex-find-as-key-value-pairs-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-regex-replace-all"
+ },
+ "source": [
+ "### Example 7: Regex replace all\n",
+ "\n",
+ "`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.\n",
+ "You can also use\n",
+ "[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)\n",
+ "on the `replacement`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-7-regex-replace-all-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_replace_all = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓 : Strawberry : perennial',\n",
+ " '🥕 : Carrot : biennial',\n",
+ " '🍆\\t:\\tEggplant\\t:\\tperennial',\n",
+ " '🍅 : Tomato : annual',\n",
+ " '🥔 : Potato : perennial',\n",
+ " ])\n",
+ " | 'To CSV' >> beam.Regex.replace_all(r'\\s*:\\s*', ',')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-7-regex-replace-all-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-regex-replace-first"
+ },
+ "source": [
+ "### Example 8: Regex replace first\n",
+ "\n",
+ "`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.\n",
+ "You can also use\n",
+ "[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)\n",
+ "on the `replacement`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-8-regex-replace-first-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_replace_first = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓, Strawberry, perennial',\n",
+ " '🥕, Carrot, biennial',\n",
+ " '🍆,\\tEggplant, perennial',\n",
+ " '🍅, Tomato, annual',\n",
+ " '🥔, Potato, perennial',\n",
+ " ])\n",
+ " | 'As dictionary' >> beam.Regex.replace_first(r'\\s*,\\s*', ': ')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-8-regex-replace-first-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-regex-split"
+ },
+ "source": [
+ "### Example 9: Regex split\n",
+ "\n",
+ "`Regex.split` returns the list of strings that were delimited by the specified regular expression.\n",
+ "The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-9-regex-split-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_split = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " '🍓 : Strawberry : perennial',\n",
+ " '🥕 : Carrot : biennial',\n",
+ " '🍆\\t:\\tEggplant : perennial',\n",
+ " '🍅 : Tomato : annual',\n",
+ " '🥔 : Potato : perennial',\n",
+ " ])\n",
+ " | 'Parse plants' >> beam.Regex.split(r'\\s*:\\s*')\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-9-regex-split-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [FlatMap](https://beam.apache.org/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for\n",
+ " each input it may produce zero or more outputs.\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Regex\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/></icon>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/regex\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Regex - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
new file mode 100644
index 0000000..69addef
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
@@ -0,0 +1,314 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/tostring\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "tostring"
+ },
+ "source": [
+ "# ToString\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Transforms every element in an input collection to a string."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "Any non-string element can be converted to a string using standard Python functions and methods.\n",
+ "Many I/O transforms, such as\n",
+ "[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),\n",
+ "expect their input elements to be strings."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string"
+ },
+ "source": [
+ "### Example 1: Key-value pairs to string\n",
+ "\n",
+ "The following example converts a `(key, value)` pair into a string delimited by `','`.\n",
+ "You can specify a different delimiter using the `delimiter` argument."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Kvs()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-key-value-pairs-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-elements-to-string"
+ },
+ "source": [
+ "### Example 2: Elements to string\n",
+ "\n",
+ "The following example converts a dictionary into a string.\n",
+ "The string output will be equivalent to `str(element)`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-elements-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_lists = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ['🍓', 'Strawberry', 'perennial'],\n",
+ " ['🥕', 'Carrot', 'biennial'],\n",
+ " ['🍆', 'Eggplant', 'perennial'],\n",
+ " ['🍅', 'Tomato', 'annual'],\n",
+ " ['🥔', 'Potato', 'perennial'],\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Element()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-elements-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-iterables-to-string"
+ },
+ "source": [
+ "### Example 3: Iterables to string\n",
+ "\n",
+ "The following example converts an iterable, in this case a list of strings,\n",
+ "into a string delimited by `','`.\n",
+ "You can specify a different delimiter using the `delimiter` argument.\n",
+ "The string output will be equivalent to `iterable.join(delimiter)`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-iterables-to-string-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants_csv = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ['🍓', 'Strawberry', 'perennial'],\n",
+ " ['🥕', 'Carrot', 'biennial'],\n",
+ " ['🍆', 'Eggplant', 'perennial'],\n",
+ " ['🍅', 'Tomato', 'annual'],\n",
+ " ['🥔', 'Potato', 'perennial'],\n",
+ " ])\n",
+ " | 'To string' >> beam.ToString.Iterables()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-iterables-to-string-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Map](https://beam.apache.org/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.ToString\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/tostring\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "ToString - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
new file mode 100644
index 0000000..db28234
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
@@ -0,0 +1,195 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/values\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "values"
+ },
+ "source": [
+ "# Values\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "Takes a collection of key-value pairs, and returns the value of each element."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example"
+ },
+ "source": [
+ "## Example\n",
+ "\n",
+ "In the following example, we create a pipeline with a `PCollection` of key-value pairs.\n",
+ "Then, we apply `Values` to extract the values and discard the keys."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plants = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " ('🍓', 'Strawberry'),\n",
+ " ('🥕', 'Carrot'),\n",
+ " ('🍆', 'Eggplant'),\n",
+ " ('🍅', 'Tomato'),\n",
+ " ('🥔', 'Potato'),\n",
+ " ])\n",
+ " | 'Values' >> beam.Values()\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Keys](https://beam.apache.org/documentation/transforms/python/elementwise/keys) for extracting the key of each component.\n",
+ "* [KvSwap](https://beam.apache.org/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.\n",
+ "\n",
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Values\"><img src=\"https://beam.apache.org/images/logos/sdks/python.png\" width=\"32px\" height=\"32px\" alt=\"Pydoc\"/> Pydoc</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/values\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "Values - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb b/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
new file mode 100644
index 0000000..3418035
--- /dev/null
+++ b/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
@@ -0,0 +1,369 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-in-github"
+ },
+ "source": [
+ "<a href=\"https://colab.research.google.com/github/apache/beam/blob/master//Users/dcavazos/src/beam/examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-top"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "cellView": "form",
+ "id": "_-code"
+ },
+ "outputs": [],
+ "source": [
+ "#@title Licensed under the Apache License, Version 2.0 (the \"License\")\n",
+ "# Licensed to the Apache Software Foundation (ASF) under one\n",
+ "# or more contributor license agreements. See the NOTICE file\n",
+ "# distributed with this work for additional information\n",
+ "# regarding copyright ownership. The ASF licenses this file\n",
+ "# to you under the Apache License, Version 2.0 (the\n",
+ "# \"License\"); you may not use this file except in compliance\n",
+ "# with the License. You may obtain a copy of the License at\n",
+ "#\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "#\n",
+ "# Unless required by applicable law or agreed to in writing,\n",
+ "# software distributed under the License is distributed on an\n",
+ "# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
+ "# KIND, either express or implied. See the License for the\n",
+ "# specific language governing permissions and limitations\n",
+ "# under the License."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "withtimestamps"
+ },
+ "source": [
+ "# WithTimestamps\n",
+ "\n",
+ "<script type=\"text/javascript\">\n",
+ "localStorage.setItem('language', 'language-py')\n",
+ "</script>\n",
+ "\n",
+ "Assigns timestamps to all the elements of a collection."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "setup"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "To run a code cell, you can click the **Run cell** button at the top left of the cell,\n",
+ "or select it and press **`Shift+Enter`**.\n",
+ "Try modifying a code cell and re-running it to see what happens.\n",
+ "\n",
+ "> To learn more about Colab, see\n",
+ "> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).\n",
+ "\n",
+ "First, let's install the `apache-beam` module."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "setup-code"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install --quiet -U apache-beam"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "examples"
+ },
+ "source": [
+ "## Examples\n",
+ "\n",
+ "In the following examples, we create a pipeline with a `PCollection` and attach a timestamp value to each of its elements.\n",
+ "When windowing and late data play an important role in streaming pipelines, timestamps are especially useful."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time"
+ },
+ "source": [
+ "### Example 1: Timestamp by event time\n",
+ "\n",
+ "The elements themselves often already contain a timestamp field.\n",
+ "`beam.window.TimestampedValue` takes a value and a\n",
+ "[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)\n",
+ "in the form of seconds."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_timestamps = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry', 'season': 1585699200}, # April, 2020\n",
+ " {'name': 'Carrot', 'season': 1590969600}, # June, 2020\n",
+ " {'name': 'Artichoke', 'season': 1583020800}, # March, 2020\n",
+ " {'name': 'Tomato', 'season': 1588291200}, # May, 2020\n",
+ " {'name': 'Potato', 'season': 1598918400}, # September, 2020\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(\n",
+ " lambda plant: beam.window.TimestampedValue(plant, plant['season']))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>\n",
+ "\n",
+ "To convert from a\n",
+ "[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)\n",
+ "to `unix_time` you can use\n",
+ "[`time.mktime`](https://docs.python.org/3/library/time.html#time.mktime).\n",
+ "For more information on time formatting options, see\n",
+ "[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code-2"
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "time_tuple = time.strptime('2020-03-19 20:50:00', '%Y-%m-%d %H:%M:%S')\n",
+ "unix_time = time.mktime(time_tuple)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-3"
+ },
+ "source": [
+ "To convert from a\n",
+ "[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)\n",
+ "to `unix_time` you can use convert it to a `time.struct_time` first with\n",
+ "[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-1-timestamp-by-event-time-code-3"
+ },
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "import datetime\n",
+ "\n",
+ "now = datetime.datetime.now()\n",
+ "time_tuple = now.timetuple()\n",
+ "unix_time = time.mktime(time_tuple)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock"
+ },
+ "source": [
+ "### Example 2: Timestamp by logical clock\n",
+ "\n",
+ "If each element has a chronological number, these numbers can be used as a\n",
+ "[logical clock](https://en.wikipedia.org/wiki/Logical_clock).\n",
+ "These numbers have to be converted to a *\"seconds\"* equivalent, which can be especially important depending on your windowing and late data rules."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " event_id = int(timestamp.micros / 1e6) # equivalent to seconds\n",
+ " yield '{} - {}'.format(event_id, plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_events = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry', 'event_id': 1},\n",
+ " {'name': 'Carrot', 'event_id': 4},\n",
+ " {'name': 'Artichoke', 'event_id': 2},\n",
+ " {'name': 'Tomato', 'event_id': 3},\n",
+ " {'name': 'Potato', 'event_id': 5},\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(lambda plant: \\\n",
+ " beam.window.TimestampedValue(plant, plant['event_id']))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-2-timestamp-by-logical-clock-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time"
+ },
+ "source": [
+ "### Example 3: Timestamp by processing time\n",
+ "\n",
+ "If the elements do not have any time data available, you can also use the current processing time for each element.\n",
+ "Note that this grabs the local time of the *worker* that is processing each element.\n",
+ "Workers might have time deltas, so using this method is not a reliable way to do precise ordering.\n",
+ "\n",
+ "By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time-code"
+ },
+ "outputs": [],
+ "source": [
+ "import apache_beam as beam\n",
+ "import time\n",
+ "\n",
+ "class GetTimestamp(beam.DoFn):\n",
+ " def process(self, plant, timestamp=beam.DoFn.TimestampParam):\n",
+ " yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])\n",
+ "\n",
+ "with beam.Pipeline() as pipeline:\n",
+ " plant_processing_times = (\n",
+ " pipeline\n",
+ " | 'Garden plants' >> beam.Create([\n",
+ " {'name': 'Strawberry'},\n",
+ " {'name': 'Carrot'},\n",
+ " {'name': 'Artichoke'},\n",
+ " {'name': 'Tomato'},\n",
+ " {'name': 'Potato'},\n",
+ " ])\n",
+ " | 'With timestamps' >> beam.Map(lambda plant: \\\n",
+ " beam.window.TimestampedValue(plant, time.time()))\n",
+ " | 'Get timestamp' >> beam.ParDo(GetTimestamp())\n",
+ " | beam.Map(print)\n",
+ " )"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "example-3-timestamp-by-processing-time-2"
+ },
+ "source": [
+ "<table align=\"left\" style=\"margin-right:1em\">\n",
+ " <td>\n",
+ " <a class=\"button\" target=\"_blank\" href=\"https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" width=\"32px\" height=\"32px\" alt=\"View source code\"/> View source code</a>\n",
+ " </td>\n",
+ "</table>\n",
+ "\n",
+ "<br/><br/><br/>"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "related-transforms"
+ },
+ "source": [
+ "## Related transforms\n",
+ "\n",
+ "* [Reify](https://beam.apache.org/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "view-the-docs-bottom"
+ },
+ "source": [
+ "<table align=\"left\"><td><a target=\"_blank\" href=\"https://beam.apache.org/documentation/transforms/python/elementwise/withtimestamps\"><img src=\"https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png\" width=\"32\" height=\"32\" />View the docs</a></td></table>"
+ ]
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "name": "WithTimestamps - element-wise transform",
+ "toc_visible": true
+ },
+ "kernelspec": {
+ "display_name": "python3",
+ "name": "python3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/notebooks/get-started/try-apache-beam-java.ipynb b/examples/notebooks/get-started/try-apache-beam-java.ipynb
index c19762c..40d648a 100644
--- a/examples/notebooks/get-started/try-apache-beam-java.ipynb
+++ b/examples/notebooks/get-started/try-apache-beam-java.ipynb
@@ -593,8 +593,8 @@
"\n",
"> Task :runShadow\n",
"WARNING: An illegal reflective access operation has occurred\n",
- "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil (file:/content/build/install/content-shadow/lib/WordCount.jar) to field java.nio.Buffer.address\n",
- "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil\n",
+ "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.UnsafeUtil (file:/content/build/install/content-shadow/lib/WordCount.jar) to field java.nio.Buffer.address\n",
+ "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.UnsafeUtil\n",
"WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
"WARNING: All illegal access operations will be denied in a future release\n",
"Mar 04, 2019 11:00:24 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes\n",
@@ -735,8 +735,8 @@
"\n",
">> java -jar WordCount.jar\n",
"WARNING: An illegal reflective access operation has occurred\n",
- "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil (file:/content/WordCount.jar) to field java.nio.Buffer.address\n",
- "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil\n",
+ "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.UnsafeUtil (file:/content/WordCount.jar) to field java.nio.Buffer.address\n",
+ "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.UnsafeUtil\n",
"WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
"WARNING: All illegal access operations will be denied in a future release\n",
"Mar 04, 2019 11:00:49 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes\n",
@@ -981,8 +981,8 @@
"\n",
"> Task :runShadow\n",
"WARNING: An illegal reflective access operation has occurred\n",
- "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil (file:/content/build/install/content-shadow/lib/WordCount.jar) to field java.nio.Buffer.address\n",
- "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.UnsafeUtil\n",
+ "WARNING: Illegal reflective access by org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.UnsafeUtil (file:/content/build/install/content-shadow/lib/WordCount.jar) to field java.nio.Buffer.address\n",
+ "WARNING: Please consider reporting this to the maintainers of org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.UnsafeUtil\n",
"WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations\n",
"WARNING: All illegal access operations will be denied in a future release\n",
"Mar 04, 2019 11:01:26 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes\n",
diff --git a/gradle.properties b/gradle.properties
index f0ed4b41..412f5a1 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -23,4 +23,5 @@
signing.gnupg.executable=gpg
signing.gnupg.useLegacyGpg=true
-version=2.14.0-SNAPSHOT
+version=2.17.0-SNAPSHOT
+python_sdk_version=2.17.0.dev
diff --git a/gradlew b/gradlew
index d6cbff0..97e181c 100755
--- a/gradlew
+++ b/gradlew
@@ -18,6 +18,8 @@
# limitations under the License.
################################################################################
+cd $(dirname $0)
+
save () {
for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
echo " "
@@ -26,11 +28,6 @@
orig_args=$(save "$@")
args=$orig_args
-echo "Executing"
-echo
-echo " gradlew $@"
-echo
-
while IFS=$'\n' read -r line_data; do
# echo "$line_data"
set -f
@@ -52,4 +49,4 @@
echo "$message"
fi
-./gradlew_orig "$@"
\ No newline at end of file
+./gradlew_orig "$@"
diff --git a/gradlew.bat b/gradlew.bat
index 911cddc..ae2db48 100644
--- a/gradlew.bat
+++ b/gradlew.bat
@@ -17,14 +17,11 @@
@rem ################################################################################
@echo off
+pushd %~dp0
+
set CMD_LINE_ARGS=%*
set ORG_CMD_LINE_ARGS=%*
-echo Executing
-echo.
-echo gradlew %CMD_LINE_ARGS%
-echo.
-
for /F "tokens=1,2*" %%i in (project-mappings) do call :process %%i %%j
if not "%ORG_CMD_LINE_ARGS%" == "%CMD_LINE_ARGS%" (
@@ -36,7 +33,7 @@
echo.
)
-gradlew_orig.bat %CMD_LINE_ARGS%
+gradlew_orig.bat %CMD_LINE_ARGS% & popd
EXIT /B 0
diff --git a/learning/katas/README.md b/learning/katas/README.md
index 167edc3..d803bf9 100644
--- a/learning/katas/README.md
+++ b/learning/katas/README.md
@@ -17,8 +17,10 @@
under the License.
-->
-# Beam Kata
-Beam Kata is a set of interactive Beam coding exercises (i.e. code kata) that aids in learning Beam programming hands-on. It is built based on [JetBrains Educational Products](https://www.jetbrains.com/education/).
-
-## Objective
-The objective of Beam Kata is to provide a structured hands-on learning experience for people to learn about Beam and its SDK by solving focused mini problems with gradually increasing complexity (e.g. SDK, common transforms, common use-case patterns, common problems like word count, etc).
+# Beam Katas
+Beam Katas are interactive Beam coding exercises (i.e. [code katas](http://codekata.com/))
+that can help you to learn Apache Beam concepts and programming model hands-on.
+Built based on [JetBrains Educational Products](https://www.jetbrains.com/education/), Beam Katas
+objective is to provide a series of structured hands-on learning experiences for learners
+to understand about Apache Beam and its SDKs by solving exercises with gradually increasing
+complexity. Beam Katas are available for both Java and Python SDKs.
diff --git a/learning/katas/java/.idea/study_project.xml b/learning/katas/java/.idea/study_project.xml
index 82cb98c..cee5d67 100644
--- a/learning/katas/java/.idea/study_project.xml
+++ b/learning/katas/java/.idea/study_project.xml
@@ -21,7 +21,7 @@
<option name="courseMode" value="Course Creator" />
<option name="createDate" value="1557823043901" />
<option name="customPresentableName" />
- <option name="description" value="This course provides a series of kata to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" />
+ <option name="description" value="This course provides a series of katas to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" />
<option name="environment" value="" />
<option name="fromZip" value="false" />
<option name="id" value="54530" />
@@ -31,16 +31,16 @@
<option value="48485817" />
</list>
</option>
- <option name="language" value="JAVA" />
+ <option name="language" value="JAVA 8" />
<option name="languageCode" value="en" />
- <option name="name" value="Beam Kata - Java" />
- <option name="public" value="false" />
+ <option name="name" value="Beam Katas - Java" />
+ <option name="public" value="true" />
<option name="sectionIds">
<list />
</option>
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="type" value="pycharm11 JAVA" />
- <option name="updateDate" value="1557823043000" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="type" value="pycharm11 JAVA 8" />
+ <option name="updateDate" value="1560936271000" />
<option name="items">
<list>
<Section>
@@ -49,9 +49,9 @@
<option name="id" value="85639" />
<option name="index" value="1" />
<option name="name" value="Introduction" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823047000" />
+ <option name="position" value="1" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1559325015000" />
<option name="items">
<list>
<Lesson>
@@ -59,15 +59,15 @@
<option name="id" value="229506" />
<option name="index" value="1" />
<option name="name" value="Hello Beam" />
- <option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557823051000" />
- <option name="unitId" value="202031" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1559325015000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p>This kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam"</p> <br> <br> <div class='hint'>Hardcoded input can be created using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Create">Create</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p> Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow. </p> <p> Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. </p> <p> To learn more about Apache Beam, refer to <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>. </p> <p> <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam". </p> <br> <div class="hint"> Hardcoded input can be created using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Create.html"> Create</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory"> "Creating a PCollection from in-memory data"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -79,10 +79,10 @@
<option name="name" value="Hello Beam" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/intro/hello/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -95,7 +95,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1461" />
+ <option name="offset" value="1552" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="pipeline.apply(Create.of("Hello Beam"))" />
@@ -107,7 +107,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/intro/hello/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -115,14 +115,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/intro/hello/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/intro/hello/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -132,7 +132,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823053000" />
+ <option name="updateDate" value="1560936162000" />
</EduTask>
</list>
</option>
@@ -146,9 +146,9 @@
<option name="id" value="85640" />
<option name="index" value="2" />
<option name="name" value="Core Transforms" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823054000" />
+ <option name="position" value="2" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1559325050000" />
<option name="items">
<list>
<Lesson>
@@ -157,14 +157,14 @@
<option name="index" value="1" />
<option name="name" value="Map" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823058000" />
- <option name="unitId" value="202032" />
+ <option name="updateDate" value="1559325026000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p>ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection.</p> <p>For this task, please write a simple ParDo that maps the input element by multiplying it by 10.</p> <br> <br> <div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method</div> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p> ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection. </p> <p> <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -176,10 +176,10 @@
<option name="name" value="ParDo" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -192,7 +192,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1648" />
+ <option name="offset" value="1752" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, OutputReceiver<Integer> out) { out.output(number * 10); } }))" />
@@ -204,7 +204,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -212,14 +212,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -229,12 +229,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823060000" />
+ <option name="updateDate" value="1560936166000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p>For this task, please write a ParDo that maps each input sentence into words tokenized by whitespace (" ").</p> <br> <br> <div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method. You can return an Iterable for multiple elements or call "yield" for each element to return a generator.</div> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p> <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by whitespace (" "). </p> <br> <div class="hint"> You can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html"> OutputReceiver</a> multiple times in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a>. </div> <div class="hint"> If you're using Beam version before v2.5.0, you can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.WindowedContext.html#output-OutputT-"> DoFn.ProcessContext.output(..)</a> multiple times in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>. </div> </html>" />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -246,10 +246,10 @@
<option name="name" value="ParDo OneToMany" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -262,7 +262,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1664" />
+ <option name="offset" value="1777" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(ParDo.of(new DoFn<String, String>() { @ProcessElement public void processElement(@Element String sentence, OutputReceiver<String> out) { String[] words = sentence.split(" "); for (String word : words) { out.output(word); } } }))" />
@@ -274,7 +274,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -282,14 +282,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -299,12 +299,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823062000" />
+ <option name="updateDate" value="1560936169000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p> <p>MapElements can be used to simplify DoFn that maps an element to another element (one to one).</p> <p>In this task, we are going to implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/MapElements.html">MapElements.into(...).via(...)</a></div> </html>" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> MapElements can be used to simplify DoFn that maps an element to another element (one to one). </p> <p> <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"> MapElements.into(...).via(...)</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -316,10 +316,10 @@
<option name="name" value="MapElements" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -332,7 +332,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1666" />
+ <option name="offset" value="1776" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply( MapElements.into(TypeDescriptors.integers()) .via(number -> number * 5) )" />
@@ -344,7 +344,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -352,14 +352,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -369,12 +369,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823065000" />
+ <option name="updateDate" value="1560936172000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p> <p>FlatMapElements can be used to simplify DoFn that maps an element to multiple elements (one to many).</p> <p>In this task, we are going to implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/FlatMapElements.html">FlatMapElements.into(...).via(...)</a> </div> </html>" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> FlatMapElements can be used to simplify DoFn that maps an element to multiple elements (one to many). </p> <p> <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html"> FlatMapElements.into(...).via(...)</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -386,10 +386,10 @@
<option name="name" value="FlatMapElements" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -402,7 +402,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1721" />
+ <option name="offset" value="1835" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply( FlatMapElements.into(TypeDescriptors.strings()) .via(sentence -> Arrays.asList(sentence.split(" "))) )" />
@@ -414,7 +414,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -422,14 +422,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -439,7 +439,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823067000" />
+ <option name="updateDate" value="1560791586000" />
</EduTask>
</list>
</option>
@@ -450,14 +450,14 @@
<option name="index" value="2" />
<option name="name" value="GroupByKey" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823071000" />
- <option name="unitId" value="202033" />
+ <option name="updateDate" value="1559325029000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p>GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <br> <div class='hint'>Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">GroupByKey</a> to solve this problem</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p> GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/KV.html"> KV</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html"> GroupByKey</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey"> "GroupByKey"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -469,10 +469,10 @@
<option name="name" value="GroupByKey" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -485,7 +485,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1920" />
+ <option name="offset" value="2025" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input .apply(MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word))) .apply(GroupByKey.create())" />
@@ -497,7 +497,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -505,14 +505,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -522,7 +522,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823073000" />
+ <option name="updateDate" value="1560936177000" />
</EduTask>
</list>
</option>
@@ -533,14 +533,14 @@
<option name="index" value="3" />
<option name="name" value="CoGroupByKey" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823076000" />
- <option name="unitId" value="202034" />
+ <option name="updateDate" value="1559325032000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p>CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the string representation of the WordsAlphabet model. </p> <br> <br> <div class='hint'>Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">CoGroupByKey</a> to solve this problem</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p> CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the toString() representation of the WordsAlphabet model. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> CoGroupByKey</a>, <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/TupleTag.html"> TupleTag</a>, and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGbkResult.html"> CoGbkResult</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey"> "CoGroupByKey"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -552,10 +552,10 @@
<option name="name" value="CoGroupByKey" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -568,10 +568,10 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="2290" />
+ <option name="offset" value="2418" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
- <option name="possibleAnswer" value="TupleTag<String> fruitsTag = new TupleTag<>(); TupleTag<String> countriesTag = new TupleTag<>(); MapElements<String, KV<String, String>> mapToAlphabetKv = MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word)); PCollection<KV<String, String>> fruitsPColl = fruits.apply("Fruit to KV", mapToAlphabetKv); PCollection<KV<String, String>> countriesPColl = countries .apply("Country to KV", mapToAlphabetKv); return KeyedPCollectionTuple .of(fruitsTag, fruitsPColl) .and(countriesTag, countriesPColl) .apply(CoGroupByKey.create()) .apply(ParDo.of(new DoFn<KV<String, CoGbkResult>, String>() { @ProcessElement public void processElement( @Element KV<String, CoGbkResult> element, OutputReceiver<String> out) { String alphabet = element.getKey(); CoGbkResult coGbkResult = element.getValue(); String fruit = coGbkResult.getOnly(fruitsTag); String country = coGbkResult.getOnly(countriesTag); out.output(new WordsAlphabet(alphabet, fruit, country).toString()); } }))" />
+ <option name="possibleAnswer" value="TupleTag<String> fruitsTag = new TupleTag<>(); TupleTag<String> countriesTag = new TupleTag<>(); MapElements<String, KV<String, String>> mapToAlphabetKv = MapElements.into(kvs(strings(), strings())) .via(word -> KV.of(word.substring(0, 1), word)); PCollection<KV<String, String>> fruitsPColl = fruits.apply("Fruit to KV", mapToAlphabetKv); PCollection<KV<String, String>> countriesPColl = countries .apply("Country to KV", mapToAlphabetKv); return KeyedPCollectionTuple .of(fruitsTag, fruitsPColl) .and(countriesTag, countriesPColl) .apply(CoGroupByKey.create()) .apply(ParDo.of(new DoFn<KV<String, CoGbkResult>, String>() { @ProcessElement public void processElement( @Element KV<String, CoGbkResult> element, OutputReceiver<String> out) { String alphabet = element.getKey(); CoGbkResult coGbkResult = element.getValue(); String fruit = coGbkResult.getOnly(fruitsTag); String country = coGbkResult.getOnly(countriesTag); out.output(new WordsAlphabet(alphabet, fruit, country).toString()); } }));" />
<option name="selected" value="false" />
<option name="status" value="Unchecked" />
<option name="studentAnswer" />
@@ -580,7 +580,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -588,14 +588,14 @@
</TaskFile>
</value>
</entry>
- <entry key="src/WordsAlphabet.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/WordsAlphabet.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -603,14 +603,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -620,7 +620,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823078000" />
+ <option name="updateDate" value="1560936180000" />
</EduTask>
</list>
</option>
@@ -631,14 +631,14 @@
<option name="index" value="4" />
<option name="name" value="Combine" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823081000" />
- <option name="unitId" value="202035" />
+ <option name="updateDate" value="1559325044000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p>Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection.</p> <p>Simple combine operations, such as sums, can usually be implemented as a simple function.</p> <p>In this task, we are going to implement the summation of numbers using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"> CombineGlobally</a>. </p> <br> <br> <div class="hint"> Implement a simple Python function that performs the summation of the values. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Simple combine operations, such as sums, can usually be implemented as a simple function. </p> <p> <b>Kata:</b> Implement the summation of numbers using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html"> Combine.globally(SerializableFunction)</a>. </p> <br> <div class="hint"> Implement the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html#apply-InputT-"> SerializableFunction.apply</a> method that performs the summation of the Iterable. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines"> "Simple combinations using simple functions"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -650,10 +650,10 @@
<option name="name" value="Simple Function" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -666,7 +666,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1814" />
+ <option name="offset" value="1923" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="@Override public Integer apply(Iterable<Integer> input) { int sum = 0; for (int item : input) { sum += item; } return sum; }" />
@@ -678,7 +678,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -686,14 +686,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -703,12 +703,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823083000" />
+ <option name="updateDate" value="1560936184000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p>Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection.</p> <p>Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account.</p> <p>In this task, we are going to implement the average of numbers using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> Combine.CombineFn</a>. </p> <br> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">CombineFn</a> class that counts the average of the number. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account. </p> <p> <b>Kata:</b> Implement the average of numbers using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html"> Combine.CombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html"> Combine.CombineFn</a> class that counts the average of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines"> "Advanced combinations using CombineFn"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -720,10 +720,10 @@
<option name="name" value="CombineFn" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -736,7 +736,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1850" />
+ <option name="offset" value="1962" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="class Accum implements Serializable { int sum = 0; int count = 0; @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Accum accum = (Accum) o; return sum == accum.sum && count == accum.count; } @Override public int hashCode() { return Objects.hash(sum, count); } } @Override public Accum createAccumulator() { return new Accum(); } @Override public Accum addInput(Accum accumulator, Integer input) { accumulator.sum += input; accumulator.count++; return accumulator; } @Override public Accum mergeAccumulators(Iterable<Accum> accumulators) { Accum merged = createAccumulator(); for (Accum accumulator : accumulators) { merged.sum += accumulator.sum; merged.count += accumulator.count; } return merged; } @Override public Double extractOutput(Accum accumulator) { return ((double) accumulator.sum) / accumulator.count; }" />
@@ -748,7 +748,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -756,14 +756,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -773,12 +773,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823086000" />
+ <option name="updateDate" value="1560936188000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn</h2> <p>Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection.</p> <p>BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations.</p> <p>In this task, we are going to implement the summation of BigInteger using <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a>. </p> <br> <br> <div class='hint'>Extend the <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">Combine.BinaryCombineFn</a> class that counts the sum of the number.</div> </html>" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations. </p> <p> <b>Kata:</b> Implement the summation of BigInteger using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combine"> "Combine"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -790,10 +790,10 @@
<option name="name" value="BinaryCombineFn" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Content changed" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -806,7 +806,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="2007" />
+ <option name="offset" value="2125" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="@Override public BigInteger apply(BigInteger left, BigInteger right) { return left.add(right); }" />
@@ -818,7 +818,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -826,14 +826,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -843,27 +843,27 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823088000" />
+ <option name="updateDate" value="1560936191000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p>After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn.</p> <p>In this task, we are going to implement the sum of scores per player using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey"> CombinePerKey</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey">CombinePerKey(CombineFn)</a>.</div> <div class='hint'>Extend the <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">CombineFn</a> class that counts the sum of the number.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - BinaryCombineFn Lambda</h2> <p> BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations. </p> <p> Since Beam v2.13.0, you can also use lambda or method reference in order to create the BinaryCombineFn. </p> <p> <b>Kata:</b> Implement the summation of BigInteger using lambda or method reference. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableBiFunction.html"> SerializableBiFunction</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combine"> "Combine"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713733" />
+ <option name="id" value="750324" />
<option name="index" value="4" />
- <option name="name" value="Combine PerKey" />
+ <option name="name" value="BinaryCombineFn Lambda" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Content changed" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -876,7 +876,77 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="2039" />
+ <option name="offset" value="1922" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="input.apply(Combine.globally(BigInteger::add))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936195000" />
+ </EduTask>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p> After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn. </p> <p> <b>Kata:</b> Implement the sum of scores per player using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html"> Combine.perKey</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html"> Combine.perKey(GlobalCombineFn)</a>. </div> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html"> Combine.BinaryCombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection"> "Combining values in a keyed PCollection"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="713733" />
+ <option name="index" value="5" />
+ <option name="name" value="Combine PerKey" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2155" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Combine.perKey(new SumIntBinaryCombineFn()))" />
@@ -893,7 +963,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="2179" />
+ <option name="offset" value="2295" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="@Override public Integer apply(Integer left, Integer right) { return left + right; }" />
@@ -905,7 +975,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -913,14 +983,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -930,7 +1000,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823090000" />
+ <option name="updateDate" value="1560936199000" />
</EduTask>
</list>
</option>
@@ -941,14 +1011,14 @@
<option name="index" value="5" />
<option name="name" value="Flatten" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823093000" />
- <option name="unitId" value="202036" />
+ <option name="updateDate" value="1559325047000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p>Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten">Flatten</a> to solve this problem. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p> Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html"> Flatten</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#flatten"> "Flatten"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -960,10 +1030,10 @@
<option name="name" value="Flatten" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -976,7 +1046,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1938" />
+ <option name="offset" value="2040" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="PCollectionList.of(words1).and(words2) .apply(Flatten.pCollections())" />
@@ -988,7 +1058,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -996,14 +1066,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1013,7 +1083,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823095000" />
+ <option name="updateDate" value="1560936202000" />
</EduTask>
</list>
</option>
@@ -1024,14 +1094,14 @@
<option name="index" value="6" />
<option name="name" value="Partition" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823098000" />
- <option name="unitId" value="202037" />
+ <option name="updateDate" value="1559325050000" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p>Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections.</p> <p>Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">Partition</a> to solve this problem. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p> Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections. </p> <p> Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html"> Partition</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#partition"> "Partition"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1043,10 +1113,10 @@
<option name="name" value="Partition" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/partition/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1059,7 +1129,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1862" />
+ <option name="offset" value="1966" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input .apply(Partition.of(2, (PartitionFn<Integer>) (number, numPartitions) -> { if (number > 100) { return 0; } else { return 1; } }))" />
@@ -1071,7 +1141,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/partition/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1079,14 +1149,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1096,7 +1166,453 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823100000" />
+ <option name="updateDate" value="1560936206000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237989" />
+ <option name="index" value="7" />
+ <option name="name" value="Side Input" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560791406453" />
+ <option name="unitId" value="-1" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Input</h2> <p> In addition to the main input PCollection, you can provide additional inputs to a ParDo transform in the form of side inputs. A side input is an additional input that your DoFn can access each time it processes an element in the input PCollection. When you specify a side input, you create a view of some other data that can be read from within the ParDo transform’s DoFn while processing each element. </p> <p> Side inputs are useful if your ParDo needs to inject additional data when processing each element in the input PCollection, but the additional data needs to be determined at runtime (and not hard-coded). Such values might be determined by the input data, or depend on a different branch of your pipeline. </p> <p> <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/View.html"> View</a> to create PCollectionView of citiesToCountries. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a> that accepts <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withSideInputs-org.apache.beam.sdk.values.PCollectionView...-"> side input</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="754085" />
+ <option name="index" value="1" />
+ <option name="name" value="Side Input" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2716" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="citiesToCountries.apply(View.asMap())" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2914" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="persons.apply(ParDo.of(new DoFn<Person, Person>() { @ProcessElement public void processElement(@Element Person person, OutputReceiver<Person> out, ProcessContext context) { Map<String, String> citiesToCountries = context.sideInput(citiesToCountriesView); String city = person.getCity(); String country = citiesToCountries.get(city); out.output(new Person(person.getName(), city, country)); } }).withSideInputs(citiesToCountriesView))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936210000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237990" />
+ <option name="index" value="8" />
+ <option name="name" value="Side Output" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560791445676" />
+ <option name="unitId" value="-1" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Output</h2> <p> While ParDo always produces a main output PCollection (as the return value from apply), you can also have your ParDo produce any number of additional output PCollections. If you choose to have multiple outputs, your ParDo returns all of the output PCollections (including the main output) bundled together. </p> <p> <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.MultiOutputReceiver.html"> MultiOutputReceiver</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withOutputTags-org.apache.beam.sdk.values.TupleTag-org.apache.beam.sdk.values.TupleTagList-"> .withOutputTags</a> to output multiple tagged-outputs in a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo.</a> </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs"> "Additional outputs"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="754087" />
+ <option name="index" value="1" />
+ <option name="name" value="Side Output" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2253" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="numbers.apply(ParDo.of(new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, MultiOutputReceiver out) { if (number <= 100) { out.get(numBelow100Tag).output(number); } else { out.get(numAbove100Tag).output(number); } } }).withOutputTags(numBelow100Tag, TupleTagList.of(numAbove100Tag)))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936215000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237991" />
+ <option name="index" value="9" />
+ <option name="name" value="Branching" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560791458069" />
+ <option name="unitId" value="-1" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Branching</h2> <p> You can use the same PCollection as input for multiple transforms without consuming the input or altering it. </p> <p> <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying each number by 5 and the other transform is multiplying each number by 10. </p> <br> <div class="hint"> Refer to the Beam Design Your Pipeline Guide <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection"> "Multiple transforms process the same PCollection"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="754088" />
+ <option name="index" value="1" />
+ <option name="name" value="Branching" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/branching/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1994" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="input.apply("Multiply by 5", MapElements.into(integers()).via(num -> num * 5))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2175" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="input.apply("Multiply by 10", MapElements.into(integers()).via(num -> num * 10))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/branching/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936219000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237192" />
+ <option name="index" value="10" />
+ <option name="name" value="Composite Transform" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560431460000" />
+ <option name="unitId" value="-1" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Composite Transform</h2> <p> Transforms can have a nested structure, where a complex transform performs multiple simpler transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms). These transforms are called composite transforms. Nesting multiple transforms inside a single composite transform can make your code more modular and easier to understand. </p> <p> To create your own composite transform, create a subclass of the PTransform class and override the expand method to specify the actual processing logic. You can then use this transform just as you would a built-in transform from the Beam SDK. For the PTransform class type parameters, you pass the PCollection types that your transform takes as input, and produces as output. Within your PTransform subclass, you’ll need to override the expand method. The expand method is where you add the processing logic for the PTransform. Your override of expand must accept the appropriate type of input PCollection as a parameter, and specify the output PCollection as the return value. </p> <p> <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts numbers from comma separated line and then multiplies each number by 10. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/PTransform.html"> PTransform</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms"> "Composite transforms"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="750323" />
+ <option name="index" value="1" />
+ <option name="name" value="Composite Transform" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/composite/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1929" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="@Override public PCollection<Integer> expand(PCollection<String> input) { return input .apply(ParDo.of(new DoFn<String, Integer>() { @ProcessElement public void processElement(@Element String numbers, OutputReceiver<Integer> out) { Arrays.stream(numbers.split(",")) .forEach(numStr -> out.output(Integer.parseInt(numStr))); } })) .apply(MapElements.into(integers()).via(number -> number * 10)); }" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/composite/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560791618000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237765" />
+ <option name="index" value="11" />
+ <option name="name" value="DoFn Additional Parameters" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="0" />
+ <option name="unitId" value="-1" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>DoFn Additional Parameters</h2> <p> In addition to the element and the OutputReceiver, Beam will populate other parameters to your DoFn’s @ProcessElement method. Any combination of these parameters can be added to your process method in any order. </p> <div> <ul> <li> <b>Timestamp</b>: To access the timestamp of an input element, add a parameter annotated with @Timestamp of type Instant </li> <li> <b>Window</b>: To access the window an input element falls into, add a parameter of the type of the window used for the input PCollection. </li> <li> <b>PaneInfo</b>: When triggers are used, Beam provides a PaneInfo object that contains information about the current firing. Using PaneInfo you can determine whether this is an early or a late firing, and how many times this window has already fired for this key. </li> <li> <b>PipelineOptions</b>: The PipelineOptions for the current pipeline can always be accessed in a process method by adding it as a parameter. </li> </ul> </div> <p> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#other-dofn-parameters"> "Accessing additional parameters in your DoFn"</a> section for more information. </p> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753154" />
+ <option name="index" value="1" />
+ <option name="name" value="DoFn Additional Parameters" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560699463688" />
</EduTask>
</list>
</option>
@@ -1110,9 +1626,9 @@
<option name="id" value="85641" />
<option name="index" value="3" />
<option name="name" value="Common Transforms" />
- <option name="position" value="0" />
+ <option name="position" value="3" />
<option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557823101000" />
+ <option name="updateDate" value="1559325072000" />
<option name="items">
<list>
<Lesson>
@@ -1121,14 +1637,14 @@
<option name="index" value="1" />
<option name="name" value="Filter" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823104000" />
+ <option name="updateDate" value="1559325056000" />
<option name="unitId" value="202038" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p>In this task, we are going to implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a>. </p> <br> <br> <div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method. You can use "yield" for each intended element.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p> <b>Kata:</b> Implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a> and only output the intended element. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1140,10 +1656,10 @@
<option name="name" value="ParDo" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1156,7 +1672,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1656" />
+ <option name="offset" value="1752" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(ParDo.of( new DoFn<Integer, Integer>() { @ProcessElement public void processElement(@Element Integer number, OutputReceiver<Integer> out) { if (number % 2 == 1) { out.output(number); } } }) )" />
@@ -1168,7 +1684,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1176,14 +1692,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1193,12 +1709,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823106000" />
+ <option name="updateDate" value="1560936224000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p> <p>In this task, we are going to implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> Filter</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">Filter</a> with a lambda.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html"> Filter</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html"> Filter.by(...)</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1210,10 +1726,10 @@
<option name="name" value="Filter" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1226,7 +1742,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1621" />
+ <option name="offset" value="1718" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Filter.by(number -> number % 2 == 0))" />
@@ -1238,7 +1754,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1246,14 +1762,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1263,7 +1779,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823108000" />
+ <option name="updateDate" value="1560936227000" />
</EduTask>
</list>
</option>
@@ -1274,14 +1790,14 @@
<option name="index" value="2" />
<option name="name" value="Aggregation" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823111000" />
+ <option name="updateDate" value="1559325072000" />
<option name="unitId" value="202039" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> In this task, we are going to count the number of elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count">Count</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> <b>Kata:</b> Count the number of elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html"> Count</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1293,10 +1809,10 @@
<option name="name" value="Count" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1309,7 +1825,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1606" />
+ <option name="offset" value="1707" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Count.globally())" />
@@ -1321,7 +1837,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1329,14 +1845,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1346,12 +1862,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823113000" />
+ <option name="updateDate" value="1560936231000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> In this task, we are going to compute the sum of all elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">CombineGlobally</a> and Python built-in <a href="https://docs.python.org/2/library/functions.html#sum">sum</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> <b>Kata:</b> Compute the sum of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Sum.html"> Sum</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1363,10 +1879,10 @@
<option name="name" value="Sum" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1379,7 +1895,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1610" />
+ <option name="offset" value="1709" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Sum.integersGlobally())" />
@@ -1391,7 +1907,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1399,14 +1915,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1416,12 +1932,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823115000" />
+ <option name="updateDate" value="1560936235000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p>In this task, we are going to compute the mean/average of all elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Mean">Mean</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p> <b>Kata:</b> Compute the mean/average of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Mean.html"> Mean</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1433,10 +1949,10 @@
<option name="name" value="Mean" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1449,7 +1965,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1609" />
+ <option name="offset" value="1709" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Mean.globally())" />
@@ -1461,7 +1977,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1469,14 +1985,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java" />
<option name="text" value="" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1486,12 +2002,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823117000" />
+ <option name="updateDate" value="1560936238000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Min</h2> <p>In this task, we are going to compute the minimum of the elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Min.html">Min</a></div> </html>" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Min</h2> <p> <b>Kata:</b> Compute the minimum of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Min.html"> Min</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1503,10 +2019,10 @@
<option name="name" value="Min" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1519,7 +2035,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1610" />
+ <option name="offset" value="1709" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Min.integersGlobally())" />
@@ -1531,7 +2047,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1539,14 +2055,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1556,12 +2072,12 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823119000" />
+ <option name="updateDate" value="1560936242000" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Max</h2> <p>In this task, we are going to compute the maximum of the elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Max.html">Max</a></div> </html>" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Max</h2> <p> <b>Kata:</b> Compute the maximum of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Max.html"> Max</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
@@ -1573,10 +2089,10 @@
<option name="name" value="Max" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1589,7 +2105,7 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1610" />
+ <option name="offset" value="1709" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input.apply(Max.integersGlobally())" />
@@ -1601,7 +2117,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1609,14 +2125,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1626,54 +2142,40 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823121000" />
+ <option name="updateDate" value="1560936246000" />
</EduTask>
</list>
</option>
</Lesson>
- </list>
- </option>
- </Section>
- <Section>
- <option name="courseId" value="54530" />
- <option name="customPresentableName" />
- <option name="id" value="85642" />
- <option name="index" value="4" />
- <option name="name" value="Examples" />
- <option name="position" value="0" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823123000" />
- <option name="items">
- <list>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229515" />
- <option name="index" value="1" />
- <option name="name" value="Word Count" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557823126000" />
- <option name="unitId" value="202040" />
+ <option name="id" value="237992" />
+ <option name="index" value="3" />
+ <option name="name" value="WithKeys" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="updateDate" value="1560791491864" />
+ <option name="unitId" value="-1" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p>This kata is to create a pipeline that counts the number of words.</p> <p>For this task, please output the count of each word in the following format:<br/> <pre> word:count ball:5 book:3 </pre> </p> <br> <br> <div class='hint'>Refer to your lessons above.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>WithKeys</h2> <p> <b>Kata:</b> Convert each fruit name into a KV of its first letter and itself, e.g. <code>apple => KV.of("a", "apple")</code> </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithKeys.html"> WithKeys</a>. </div> <div class="hint"> If using a lambda in Java 8, <code>withKeyType(TypeDescriptor)</code> must be called on the result PTransform. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713743" />
+ <option name="id" value="754089" />
<option name="index" value="1" />
- <option name="name" value="Word Count" />
+ <option name="name" value="WithKeys" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
- <entry key="src/Task.java">
+ <entry key="src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
@@ -1686,7 +2188,916 @@
<option name="initialState" />
<option name="initializedFromDependency" value="false" />
<option name="length" value="6" />
- <option name="offset" value="1990" />
+ <option name="offset" value="1875" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="input .apply(WithKeys.<String, String>of(fruit -> fruit.substring(0, 1)) .withKeyType(strings()))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936249000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ </list>
+ </option>
+ </Section>
+ <Section>
+ <option name="courseId" value="54530" />
+ <option name="customPresentableName" />
+ <option name="id" value="88010" />
+ <option name="index" value="4" />
+ <option name="name" value="IO" />
+ <option name="position" value="4" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560431425000" />
+ <option name="items">
+ <list>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237187" />
+ <option name="index" value="1" />
+ <option name="name" value="TextIO" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560431430000" />
+ <option name="unitId" value="209563" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>TextIO Read</h2> <p> When you create a pipeline, you often need to read data from some external source, such as a file or a database. Likewise, you may want your pipeline to output its result data to an external storage system. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms. </p> <p> To read a PCollection from one or more text files, use TextIO.read() to instantiate a transform and use TextIO.Read.from(String) to specify the path of the file(s) to be read. </p> <p> <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html"> TextIO</a> and its corresponding <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html#read--"> TextIO.read()</a> method. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data"> "Reading input data"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="750317" />
+ <option name="index" value="1" />
+ <option name="name" value="TextIO Read" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="countries.txt">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="countries.txt" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/io/textio/read/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1615" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="TextIO.read().from(FILE_PATH)" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1855" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="input.apply(MapElements.into(strings()).via(String::toUpperCase))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/io/textio/read/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936253000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237188" />
+ <option name="index" value="2" />
+ <option name="name" value="Built-in IOs" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560431436000" />
+ <option name="unitId" value="209564" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Built-in I/Os</h2> <p> Beam SDKs provide many out of the box I/O transforms that can be used to read from many different sources and write to many different sinks. </p> <p> See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms. </p> </html>" />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="750319" />
+ <option name="index" value="1" />
+ <option name="name" value="Built-in IOs" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/io/builtinios/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/io/builtinios/Task.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560936257000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ </list>
+ </option>
+ </Section>
+ <Section>
+ <option name="courseId" value="54530" />
+ <option name="customPresentableName" />
+ <option name="id" value="88156" />
+ <option name="index" value="5" />
+ <option name="name" value="Windowing" />
+ <option name="position" value="5" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560698891352" />
+ <option name="items">
+ <list>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237760" />
+ <option name="index" value="1" />
+ <option name="name" value="Adding Timestamp" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="0" />
+ <option name="unitId" value="210092" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Adding Timestamp - ParDo</h2> <p> Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need timestamps, you must add them to your PCollection’s elements. </p> <p> You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that outputs new elements with timestamps that you set. </p> <p> <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html"> ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html"> DoFn</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html#outputWithTimestamp-T-org.joda.time.Instant-"> OutputReceiver.outputWithTimestamp</a> method to assign timestamp to the element. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements"> "Adding timestamps to a PCollection’s elements"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753142" />
+ <option name="index" value="1" />
+ <option name="name" value="ParDo" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2249" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="events.apply(ParDo.of(new DoFn<Event, Event>() { @ProcessElement public void processElement(@Element Event event, OutputReceiver<Event> out) { out.outputWithTimestamp(event, event.getDate().toInstant()); } }))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560698905262" />
+ </EduTask>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Adding Timestamp - WithTimestamps</h2> <p> Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need timestamps, you must add them to your PCollection’s elements. </p> <p> You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that outputs new elements with timestamps that you set. </p> <p> <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithTimestamps.html"> WithTimestamps</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements"> "Adding timestamps to a PCollection’s elements"</a> section for more information. </div> </html>" />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753143" />
+ <option name="index" value="2" />
+ <option name="name" value="WithTimestamps" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2223" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="events.apply(WithTimestamps.of(event -> event.getDate().toInstant()))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560698907450" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237761" />
+ <option name="index" value="2" />
+ <option name="name" value="Fixed Time Window" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="0" />
+ <option name="unitId" value="210093" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Fixed Time Window</h2> <p> Windowing subdivides a PCollection according to the timestamps of its individual elements. Transforms that aggregate multiple elements, such as GroupByKey and Combine, work implicitly on a per-window basis — they process each PCollection as a succession of multiple, finite windows, though the entire collection itself may be of unbounded size. </p> <p> In the Beam model, any PCollection (including unbounded PCollections) can be subdivided into logical windows. Each element in a PCollection is assigned to one or more windows according to the PCollection’s windowing function, and each individual window contains a finite number of elements. Grouping transforms then consider each PCollection’s elements on a per-window basis. GroupByKey, for example, implicitly groups the elements of a PCollection by key and window. </p> <div> Beam provides several windowing functions, including: <ul> <li>Fixed Time Windows</li> <li>Sliding Time Windows</li> <li>Per-Session Windows</li> <li>Single Global Window</li> </ul> </div> <p> The simplest form of windowing is using fixed time windows. A fixed time window represents a consistent duration, non overlapping time interval in the data stream. </p> <p> <b>Kata:</b> Please count the number of events that happened based on fixed window with 1-day duration. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#fixed-time-windows"> "Fixed time windows"</a> section for more information. </div> </html>" />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753144" />
+ <option name="index" value="1" />
+ <option name="name" value="Fixed Time Window" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2906" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="events .apply(Window.into(FixedWindows.of(Duration.standardDays(1)))) .apply(Count.perElement())" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560698912954" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ </list>
+ </option>
+ </Section>
+ <Section>
+ <option name="courseId" value="54530" />
+ <option name="customPresentableName" />
+ <option name="id" value="88157" />
+ <option name="index" value="6" />
+ <option name="name" value="Triggers" />
+ <option name="position" value="6" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560923505422" />
+ <option name="items">
+ <list>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237762" />
+ <option name="index" value="1" />
+ <option name="name" value="Event Time Triggers" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560923508379" />
+ <option name="unitId" value="210094" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Event Time Triggers</h2> <p> When collecting and grouping data into windows, Beam uses triggers to determine when to emit the aggregated results of each window (referred to as a pane). If you use Beam’s default windowing configuration and default trigger, Beam outputs the aggregated result when it estimates all data has arrived, and discards all subsequent data for that window. </p> <p> You can set triggers for your PCollections to change this default behavior. Beam provides a number of pre-built triggers that you can set: </p> <div> <ul> <li>Event time triggers</li> <li>Processing time triggers</li> <li>Data-driven triggers</li> <li>Composite triggers</li> </ul> </div> <p> Event time triggers operate on the event time, as indicated by the timestamp on each data element. Beam’s default trigger is event time-based. </p> <p> The AfterWatermark trigger operates on event time. The AfterWatermark trigger emits the contents of a window after the watermark passes the end of the window, based on the timestamps attached to the data elements. The watermark is a global progress metric, and is Beam’s notion of input completeness within your pipeline at any given point. AfterWatermark.pastEndOfWindow() only fires when the watermark passes the end of the window. </p> <p> <b>Kata:</b> Given that events are being generated every second, please implement a trigger that emits the number of events count within a fixed window of 5-second duration. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 5-second duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0 with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--"> discarding accumulation mode</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753145" />
+ <option name="index" value="1" />
+ <option name="name" value="Event Time Triggers" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1905" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardSeconds(5))) .triggering(AfterWatermark.pastEndOfWindow()) .withAllowedLateness(Duration.ZERO) .discardingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560923517000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237763" />
+ <option name="index" value="2" />
+ <option name="name" value="Early Triggers" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560923523075" />
+ <option name="unitId" value="210095" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Early Triggers</h2> <p> Triggers allow Beam to emit early results, before all the data in a given window has arrived. For example, emitting after a certain amount of time elapses, or after a certain number of elements arrives. </p> <p> <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day duration, please implement an early trigger that emits the number of events count immediately after new element is processed. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-"> withEarlyFirings</a> to set early firing triggers. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0 with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--"> discarding accumulation mode</a>. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html>" />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753146" />
+ <option name="index" value="1" />
+ <option name="name" value="Early Triggers" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="resources/log4j2.xml">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="false" />
+ <option name="name" value="resources/log4j2.xml" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1970" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardDays(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings( AfterProcessingTime.pastFirstElementInPane())) .withAllowedLateness(Duration.ZERO) .discardingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560923531000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="237764" />
+ <option name="index" value="3" />
+ <option name="name" value="Window Accumulation Mode" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560923537697" />
+ <option name="unitId" value="210096" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Window Accumulation Mode</h2> <p> When you specify a trigger, you must also set the the window’s accumulation mode. When a trigger fires, it emits the current contents of the window as a pane. Since a trigger can fire multiple times, the accumulation mode determines whether the system accumulates the window panes as the trigger fires, or discards them. </p> <p> <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day duration, please implement an early trigger that emits the number of events count immediately after new element is processed in accumulating mode. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/windowing/Window.html#accumulatingFiredPanes--"> accumulatingFiredPanes()</a> to set a window to accumulate the panes that are produced when the trigger fires. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-"> withEarlyFirings</a> to set early firing triggers. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html"> FixedWindows</a> with 1-day duration using <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--"> AfterWatermark.pastEndOfWindow()</a> trigger. </div> <div class="hint"> Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-"> allowed lateness</a> to 0. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-"> Combine.globally</a> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--"> Count.combineFn</a> to calculate the count of events. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers"> "Event time triggers"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="753147" />
+ <option name="index" value="1" />
+ <option name="name" value="Window Accumulation Mode" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1972" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="events .apply( Window.<String>into(FixedWindows.of(Duration.standardDays(1))) .triggering( AfterWatermark.pastEndOfWindow() .withEarlyFirings( AfterProcessingTime.pastFirstElementInPane())) .withAllowedLateness(Duration.ZERO) .accumulatingFiredPanes()) .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java" />
+ <option name="text" value="public class Task { //put your task here }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java" />
+ <option name="text" value="import org.junit.Assert; import org.junit.Test; public class Tests { @Test public void testSolution() { // put your test here Assert.fail("Tests not implemented for the task"); } }" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560923544000" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ </list>
+ </option>
+ </Section>
+ <Section>
+ <option name="courseId" value="54530" />
+ <option name="customPresentableName" />
+ <option name="id" value="85642" />
+ <option name="index" value="7" />
+ <option name="name" value="Examples" />
+ <option name="position" value="7" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1557824624000" />
+ <option name="items">
+ <list>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="229515" />
+ <option name="index" value="1" />
+ <option name="name" value="Word Count" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1557824624000" />
+ <option name="unitId" value="202040" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p> <b>Kata:</b> Create a pipeline that counts the number of words. </p> <p> Please output the count of each word in the following format: </p> <pre> word:count ball:5 book:3 </pre> <br> <div class="hint"> Refer to your katas above. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="713743" />
+ <option name="index" value="1" />
+ <option name="name" value="Word Count" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
+ <option name="files">
+ <map>
+ <entry key="src/org/apache/beam/learning/katas/examples/wordcount/Task.java">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2075" />
<option name="placeholderDependency" />
<option name="placeholderText" value="TODO()" />
<option name="possibleAnswer" value="input .apply(FlatMapElements.into(TypeDescriptors.strings()) .via(line -> Arrays.asList(line.split(" ")))) .apply(Count.perElement()) .apply(ParDo.of(new DoFn<KV<String, Long>, String>() { @ProcessElement public void processElement( @Element KV<String, Long> element, OutputReceiver<String> out) { out.output(element.getKey() + ":" + element.getValue()); } }))" />
@@ -1698,7 +3109,7 @@
</list>
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="src/Task.java" />
+ <option name="name" value="src/org/apache/beam/learning/katas/examples/wordcount/Task.java" />
<option name="text" value="class Task { //put your task here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1706,14 +3117,14 @@
</TaskFile>
</value>
</entry>
- <entry key="test/TaskTest.java">
+ <entry key="test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java">
<value>
<TaskFile>
<option name="answerPlaceholders">
<list />
</option>
<option name="highlightErrors" value="true" />
- <option name="name" value="test/TaskTest.java" />
+ <option name="name" value="test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java" />
<option name="text" value="public class Test { // put your test here }" />
<option name="trackChanges" value="true" />
<option name="trackLengths" value="true" />
@@ -1723,7 +3134,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557823128000" />
+ <option name="updateDate" value="1560936261000" />
</EduTask>
</list>
</option>
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/src/Task.java b/learning/katas/java/Common Transforms/Aggregation/Count/src/Task.java
deleted file mode 100644
index c15c928..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Count/src/Task.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Count;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Long> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Long> applyTransform(PCollection<Integer> input) {
- return input.apply(Count.globally());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java b/learning/katas/java/Common Transforms/Aggregation/Count/src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java
new file mode 100644
index 0000000..94430a9
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Count/src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.count;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Long> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Long> applyTransform(PCollection<Integer> input) {
+ return input.apply(Count.globally());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/task.html b/learning/katas/java/Common Transforms/Aggregation/Count/task.html
index ac96243..41fd3be 100644
--- a/learning/katas/java/Common Transforms/Aggregation/Count/task.html
+++ b/learning/katas/java/Common Transforms/Aggregation/Count/task.html
@@ -18,8 +18,12 @@
<html>
<h2>Aggregation - Count</h2>
-<p>In this task, we are going to count the number of elements from an input.</p>
+<p>
+ <b>Kata:</b> Count the number of elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Count.html">Count</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html">
+ Count</a>.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/test/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Count/test/TaskTest.java
deleted file mode 100644
index 16f8a66..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Count/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void count() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Long> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(10L);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Count/test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Count/test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java
new file mode 100644
index 0000000..01529ee
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Count/test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.count;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void count() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Long> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(10L);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/src/Task.java b/learning/katas/java/Common Transforms/Aggregation/Max/src/Task.java
deleted file mode 100644
index d1081eb..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Max/src/Task.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Max;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(Max.integersGlobally());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java b/learning/katas/java/Common Transforms/Aggregation/Max/src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java
new file mode 100644
index 0000000..99cdf7e
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Max/src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.max;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Max;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(Max.integersGlobally());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/task.html b/learning/katas/java/Common Transforms/Aggregation/Max/task.html
index 98b399e..194adc5 100644
--- a/learning/katas/java/Common Transforms/Aggregation/Max/task.html
+++ b/learning/katas/java/Common Transforms/Aggregation/Max/task.html
@@ -18,8 +18,12 @@
<html>
<h2>Aggregation - Max</h2>
-<p>In this task, we are going to compute the maximum of the elements from an input.</p>
+<p>
+ <b>Kata:</b> Compute the maximum of the elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Max.html">Max</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Max.html">
+ Max</a>.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/test/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Max/test/TaskTest.java
deleted file mode 100644
index 0852d60..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Max/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void max() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(10);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Max/test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Max/test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java
new file mode 100644
index 0000000..5741e0c
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Max/test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.max;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void max() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(10);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/src/Task.java b/learning/katas/java/Common Transforms/Aggregation/Mean/src/Task.java
deleted file mode 100644
index f184d2b..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Mean/src/Task.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Mean;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Double> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Double> applyTransform(PCollection<Integer> input) {
- return input.apply(Mean.globally());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java b/learning/katas/java/Common Transforms/Aggregation/Mean/src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java
new file mode 100644
index 0000000..90c2d6d
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Mean/src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.mean;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Mean;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Double> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Double> applyTransform(PCollection<Integer> input) {
+ return input.apply(Mean.globally());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/task.html b/learning/katas/java/Common Transforms/Aggregation/Mean/task.html
index 39348b7..8934abf 100644
--- a/learning/katas/java/Common Transforms/Aggregation/Mean/task.html
+++ b/learning/katas/java/Common Transforms/Aggregation/Mean/task.html
@@ -18,8 +18,12 @@
<html>
<h2>Aggregation - Mean</h2>
-<p>In this task, we are going to compute the mean/average of all elements from an input.</p>
+<p>
+ <b>Kata:</b> Compute the mean/average of all elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Mean.html">Mean</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Mean.html">
+ Mean</a>.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/test/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Mean/test/TaskTest.java
deleted file mode 100644
index 9c77105..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Mean/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void mean() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Double> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(5.5);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Mean/test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Mean/test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java
new file mode 100644
index 0000000..75653a7
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Mean/test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.mean;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void mean() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Double> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(5.5);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/src/Task.java b/learning/katas/java/Common Transforms/Aggregation/Min/src/Task.java
deleted file mode 100644
index 54c4238..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Min/src/Task.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Min;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(Min.integersGlobally());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java b/learning/katas/java/Common Transforms/Aggregation/Min/src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java
new file mode 100644
index 0000000..9fd7ba1
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Min/src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.min;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Min;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(Min.integersGlobally());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/task.html b/learning/katas/java/Common Transforms/Aggregation/Min/task.html
index 1ec26be..157a7e1 100644
--- a/learning/katas/java/Common Transforms/Aggregation/Min/task.html
+++ b/learning/katas/java/Common Transforms/Aggregation/Min/task.html
@@ -18,8 +18,12 @@
<html>
<h2>Aggregation - Min</h2>
-<p>In this task, we are going to compute the minimum of the elements from an input.</p>
+<p>
+ <b>Kata:</b> Compute the minimum of the elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Min.html">Min</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Min.html">
+ Min</a>.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/test/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Min/test/TaskTest.java
deleted file mode 100644
index 8814308..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Min/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void min() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(1);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Min/test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Min/test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java
new file mode 100644
index 0000000..d00b5dd
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Min/test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.min;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void min() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(1);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/src/Task.java b/learning/katas/java/Common Transforms/Aggregation/Sum/src/Task.java
deleted file mode 100644
index ead1ffa..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Sum/src/Task.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Sum;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(Sum.integersGlobally());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java b/learning/katas/java/Common Transforms/Aggregation/Sum/src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java
new file mode 100644
index 0000000..275e7f0
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Sum/src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.sum;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Sum;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(Sum.integersGlobally());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/task.html b/learning/katas/java/Common Transforms/Aggregation/Sum/task.html
index 55692b9..82511e7 100644
--- a/learning/katas/java/Common Transforms/Aggregation/Sum/task.html
+++ b/learning/katas/java/Common Transforms/Aggregation/Sum/task.html
@@ -18,8 +18,12 @@
<html>
<h2>Aggregation - Sum</h2>
-<p>In this task, we are going to compute the sum of all elements from an input.</p>
+<p>
+ <b>Kata:</b> Compute the sum of all elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Sum.html">Sum</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Sum.html">
+ Sum</a>.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/test/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Sum/test/TaskTest.java
deleted file mode 100644
index 99f4b8e..0000000
--- a/learning/katas/java/Common Transforms/Aggregation/Sum/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void sum() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(55);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Aggregation/Sum/test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java b/learning/katas/java/Common Transforms/Aggregation/Sum/test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java
new file mode 100644
index 0000000..c2aadf0
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Aggregation/Sum/test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.aggregation.sum;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void sum() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(55);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/src/Task.java b/learning/katas/java/Common Transforms/Filter/Filter/src/Task.java
deleted file mode 100644
index c567f5b..0000000
--- a/learning/katas/java/Common Transforms/Filter/Filter/src/Task.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Filter;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers =
- pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(Filter.by(number -> number % 2 == 0));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java b/learning/katas/java/Common Transforms/Filter/Filter/src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java
new file mode 100644
index 0000000..1bed0aa
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/Filter/src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.filter.filter;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers =
+ pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(Filter.by(number -> number % 2 == 0));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/task.html b/learning/katas/java/Common Transforms/Filter/Filter/task.html
index 02ed9d9..15eb012 100644
--- a/learning/katas/java/Common Transforms/Filter/Filter/task.html
+++ b/learning/katas/java/Common Transforms/Filter/Filter/task.html
@@ -18,12 +18,17 @@
<html>
<h2>Filter</h2>
-<p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p>
-<p>In this task, we are going to implement a filter function that filters out the odd numbers by using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Filter.html">
- Filter</a>.
+<p>
+ The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+</p>
+<p>
+ <b>Kata:</b> Implement a filter function that filters out the odd numbers by using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html">
+ Filter</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Filter.html">Filter.by(...)</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Filter.html">
+ Filter.by(...)</a>.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/test/TaskTest.java b/learning/katas/java/Common Transforms/Filter/Filter/test/TaskTest.java
deleted file mode 100644
index ccf1bab..0000000
--- a/learning/katas/java/Common Transforms/Filter/Filter/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void filter() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(2, 4, 6, 8, 10);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Filter/Filter/test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java b/learning/katas/java/Common Transforms/Filter/Filter/test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java
new file mode 100644
index 0000000..1f58ef5
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/Filter/test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.filter.filter;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void filter() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(2, 4, 6, 8, 10);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/src/Task.java b/learning/katas/java/Common Transforms/Filter/ParDo/src/Task.java
deleted file mode 100644
index 7b3cb3f..0000000
--- a/learning/katas/java/Common Transforms/Filter/ParDo/src/Task.java
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(ParDo.of(
- new DoFn<Integer, Integer>() {
-
- @ProcessElement
- public void processElement(@Element Integer number, OutputReceiver<Integer> out) {
- if (number % 2 == 1) {
- out.output(number);
- }
- }
- })
- );
- }
-
-}
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java b/learning/katas/java/Common Transforms/Filter/ParDo/src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java
new file mode 100644
index 0000000..741f249
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/ParDo/src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.filter.pardo;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(ParDo.of(
+ new DoFn<Integer, Integer>() {
+
+ @ProcessElement
+ public void processElement(@Element Integer number, OutputReceiver<Integer> out) {
+ if (number % 2 == 1) {
+ out.output(number);
+ }
+ }
+ })
+ );
+ }
+
+}
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/task.html b/learning/katas/java/Common Transforms/Filter/ParDo/task.html
index af3cb62..61adb70 100644
--- a/learning/katas/java/Common Transforms/Filter/ParDo/task.html
+++ b/learning/katas/java/Common Transforms/Filter/ParDo/task.html
@@ -18,13 +18,16 @@
<html>
<h2>Filter using ParDo</h2>
-<p>In this task, we are going to implement a filter function that filters out the even numbers by using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/DoFn.html">
- DoFn</a>.
+<p>
+ <b>Kata:</b> Implement a filter function that filters out the even numbers by using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
+ DoFn</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>
- with <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/DoFn.html">DoFn</a>
- and only output the intended element.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
+ ParDo</a> with
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
+ DoFn</a> and only output the intended element.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/test/TaskTest.java b/learning/katas/java/Common Transforms/Filter/ParDo/test/TaskTest.java
deleted file mode 100644
index d92d0ae..0000000
--- a/learning/katas/java/Common Transforms/Filter/ParDo/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void filter_parDo() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(1, 3, 5, 7, 9);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
diff --git a/learning/katas/java/Common Transforms/Filter/ParDo/test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java b/learning/katas/java/Common Transforms/Filter/ParDo/test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java
new file mode 100644
index 0000000..9db1e47
--- /dev/null
+++ b/learning/katas/java/Common Transforms/Filter/ParDo/test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.filter.pardo;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void filter_parDo() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(1, 3, 5, 7, 9);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
diff --git a/learning/katas/java/Common Transforms/WithKeys/WithKeys/src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java b/learning/katas/java/Common Transforms/WithKeys/WithKeys/src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java
new file mode 100644
index 0000000..5d7dc68
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/WithKeys/src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.withkeys;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> words =
+ pipeline.apply(
+ Create.of("apple", "banana", "cherry", "durian", "guava", "melon"));
+
+ PCollection<KV<String, String>> output = applyTransform(words);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<KV<String, String>> applyTransform(PCollection<String> input) {
+ return input
+ .apply(WithKeys.<String, String>of(fruit -> fruit.substring(0, 1))
+ .withKeyType(strings()));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Common Transforms/WithKeys/WithKeys/task.html b/learning/katas/java/Common Transforms/WithKeys/WithKeys/task.html
new file mode 100644
index 0000000..e95e56e
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/WithKeys/task.html
@@ -0,0 +1,34 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>WithKeys</h2>
+<p>
+ <b>Kata:</b> Convert each fruit name into a KV of its first letter and itself, e.g.
+ <code>apple => KV.of("a", "apple")</code>
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithKeys.html">
+ WithKeys</a>.
+</div>
+<div class="hint">
+ If using a lambda in Java 8, <code>withKeyType(TypeDescriptor)</code> must be called on the
+ result PTransform.
+</div>
+</html>
diff --git a/learning/katas/java/Common Transforms/WithKeys/WithKeys/test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java b/learning/katas/java/Common Transforms/WithKeys/WithKeys/test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java
new file mode 100644
index 0000000..c91cd03
--- /dev/null
+++ b/learning/katas/java/Common Transforms/WithKeys/WithKeys/test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.commontransforms.withkeys;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void groupByKey() {
+ PCollection<String> numbers =
+ testPipeline.apply(
+ Create.of("apple", "banana", "cherry", "durian", "guava", "melon")
+ );
+
+ PCollection<KV<String, String>> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(
+ KV.of("a", "apple"),
+ KV.of("b", "banana"),
+ KV.of("c", "cherry"),
+ KV.of("d", "durian"),
+ KV.of("g", "guava"),
+ KV.of("m", "melon")
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Branching/Branching/src/org/apache/beam/learning/katas/coretransforms/branching/Task.java b/learning/katas/java/Core Transforms/Branching/Branching/src/org/apache/beam/learning/katas/coretransforms/branching/Task.java
new file mode 100644
index 0000000..16e5f2c
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/Branching/src/org/apache/beam/learning/katas/coretransforms/branching/Task.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.branching;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers =
+ pipeline.apply(Create.of(1, 2, 3, 4, 5));
+
+ PCollection<Integer> mult5Results = applyMultiply5Transform(numbers);
+ PCollection<Integer> mult10Results = applyMultiply10Transform(numbers);
+
+ mult5Results.apply("Log multiply 5", Log.ofElements("Multiplied by 5: "));
+ mult10Results.apply("Log multiply 10", Log.ofElements("Multiplied by 10: "));
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyMultiply5Transform(PCollection<Integer> input) {
+ return input.apply("Multiply by 5", MapElements.into(integers()).via(num -> num * 5));
+ }
+
+ static PCollection<Integer> applyMultiply10Transform(PCollection<Integer> input) {
+ return input.apply("Multiply by 10", MapElements.into(integers()).via(num -> num * 10));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Branching/Branching/task.html b/learning/katas/java/Core Transforms/Branching/Branching/task.html
new file mode 100644
index 0000000..12d9645
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/Branching/task.html
@@ -0,0 +1,35 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Branching</h2>
+<p>
+ You can use the same PCollection as input for multiple transforms without consuming the input
+ or altering it.
+</p>
+<p>
+ <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying
+ each number by 5 and the other transform is multiplying each number by 10.
+</p>
+<br>
+<div class="hint">
+ Refer to the Beam Design Your Pipeline Guide
+ <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection">
+ "Multiple transforms process the same PCollection"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Branching/Branching/test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java b/learning/katas/java/Core Transforms/Branching/Branching/test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java
new file mode 100644
index 0000000..0a07ceb
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Branching/Branching/test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.branching;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void branching() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> mult5Results = Task.applyMultiply5Transform(numbers);
+ PCollection<Integer> mult10Results = Task.applyMultiply10Transform(numbers);
+
+ PAssert.that(mult5Results)
+ .containsInAnyOrder(5, 10, 15, 20, 25);
+
+ PAssert.that(mult10Results)
+ .containsInAnyOrder(10, 20, 30, 40, 50);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/Task.java b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/Task.java
deleted file mode 100644
index 01ec78f..0000000
--- a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/Task.java
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
-import static org.apache.beam.sdk.values.TypeDescriptors.strings;
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.join.CoGbkResult;
-import org.apache.beam.sdk.transforms.join.CoGroupByKey;
-import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.TupleTag;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> fruits =
- pipeline.apply(
- Create.of("apple", "banana", "cherry")
- );
-
- PCollection<String> countries =
- pipeline.apply(
- Create.of("australia", "brazil", "canada")
- );
-
- PCollection<String> output = applyTransform(fruits, countries);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<String> applyTransform(
- PCollection<String> fruits, PCollection<String> countries) {
-
- TupleTag<String> fruitsTag = new TupleTag<>();
- TupleTag<String> countriesTag = new TupleTag<>();
-
- MapElements<String, KV<String, String>> mapToAlphabetKv =
- MapElements.into(kvs(strings(), strings()))
- .via(word -> KV.of(word.substring(0, 1), word));
-
- PCollection<KV<String, String>> fruitsPColl = fruits.apply("Fruit to KV", mapToAlphabetKv);
- PCollection<KV<String, String>> countriesPColl = countries
- .apply("Country to KV", mapToAlphabetKv);
-
- return KeyedPCollectionTuple
- .of(fruitsTag, fruitsPColl)
- .and(countriesTag, countriesPColl)
-
- .apply(CoGroupByKey.create())
-
- .apply(ParDo.of(new DoFn<KV<String, CoGbkResult>, String>() {
-
- @ProcessElement
- public void processElement(
- @Element KV<String, CoGbkResult> element, OutputReceiver<String> out) {
-
- String alphabet = element.getKey();
- CoGbkResult coGbkResult = element.getValue();
-
- String fruit = coGbkResult.getOnly(fruitsTag);
- String country = coGbkResult.getOnly(countriesTag);
-
- out.output(new WordsAlphabet(alphabet, fruit, country).toString());
- }
-
- }));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/WordsAlphabet.java b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/WordsAlphabet.java
deleted file mode 100644
index 3870642..0000000
--- a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/WordsAlphabet.java
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-public class WordsAlphabet {
-
- private String alphabet;
- private String fruit;
- private String country;
-
- public WordsAlphabet(String alphabet, String fruit, String country) {
- this.alphabet = alphabet;
- this.fruit = fruit;
- this.country = country;
- }
-
- @Override
- public String toString() {
- return "WordsAlphabet{" +
- "alphabet='" + alphabet + '\'' +
- ", fruit='" + fruit + '\'' +
- ", country='" + country + '\'' +
- '}';
- }
-
-}
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java
new file mode 100644
index 0000000..8d3f999
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.cogroupbykey;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.join.CoGbkResult;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TupleTag;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> fruits =
+ pipeline.apply("Fruits",
+ Create.of("apple", "banana", "cherry")
+ );
+
+ PCollection<String> countries =
+ pipeline.apply("Countries",
+ Create.of("australia", "brazil", "canada")
+ );
+
+ PCollection<String> output = applyTransform(fruits, countries);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> applyTransform(
+ PCollection<String> fruits, PCollection<String> countries) {
+
+ TupleTag<String> fruitsTag = new TupleTag<>();
+ TupleTag<String> countriesTag = new TupleTag<>();
+
+ MapElements<String, KV<String, String>> mapToAlphabetKv =
+ MapElements.into(kvs(strings(), strings()))
+ .via(word -> KV.of(word.substring(0, 1), word));
+
+ PCollection<KV<String, String>> fruitsPColl = fruits.apply("Fruit to KV", mapToAlphabetKv);
+ PCollection<KV<String, String>> countriesPColl = countries
+ .apply("Country to KV", mapToAlphabetKv);
+
+ return KeyedPCollectionTuple
+ .of(fruitsTag, fruitsPColl)
+ .and(countriesTag, countriesPColl)
+
+ .apply(CoGroupByKey.create())
+
+ .apply(ParDo.of(new DoFn<KV<String, CoGbkResult>, String>() {
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<String, CoGbkResult> element, OutputReceiver<String> out) {
+
+ String alphabet = element.getKey();
+ CoGbkResult coGbkResult = element.getValue();
+
+ String fruit = coGbkResult.getOnly(fruitsTag);
+ String country = coGbkResult.getOnly(countriesTag);
+
+ out.output(new WordsAlphabet(alphabet, fruit, country).toString());
+ }
+
+ }));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java
new file mode 100644
index 0000000..59da83a
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.cogroupbykey;
+
+public class WordsAlphabet {
+
+ private String alphabet;
+ private String fruit;
+ private String country;
+
+ public WordsAlphabet(String alphabet, String fruit, String country) {
+ this.alphabet = alphabet;
+ this.fruit = fruit;
+ this.country = country;
+ }
+
+ @Override
+ public String toString() {
+ return "WordsAlphabet{" +
+ "alphabet='" + alphabet + '\'' +
+ ", fruit='" + fruit + '\'' +
+ ", country='" + country + '\'' +
+ '}';
+ }
+
+}
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.html b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
index a6c29a8..29f5322 100644
--- a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
@@ -18,15 +18,28 @@
<html>
<h2>CoGroupByKey</h2>
-<p>CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/join/CoGroupByKey.html">
- CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the toString()
- representation of the WordsAlphabet model.
+<p>
+ CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+ key type.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html">
+ CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces
+ the toString() representation of the WordsAlphabet model.
</p>
<br>
-<br>
-<div class='hint'>Refer to <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/join/CoGroupByKey.html">CoGroupByKey</a>,
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/values/TupleTag.html">TupleTag</a>,
- and <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/join/CoGbkResult.html">CoGbkResult</a>.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGroupByKey.html">
+ CoGroupByKey</a>,
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/TupleTag.html">
+ TupleTag</a>, and
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/join/CoGbkResult.html">
+ CoGbkResult</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/TaskTest.java b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/TaskTest.java
deleted file mode 100644
index 08d8ab9..0000000
--- a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/TaskTest.java
+++ /dev/null
@@ -1,55 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void coGroupByKey() {
- PCollection<String> fruits =
- testPipeline.apply("Fruits",
- Create.of("apple", "banana", "cherry")
- );
-
- PCollection<String> countries =
- testPipeline.apply("Countries",
- Create.of("australia", "brazil", "canada")
- );
-
- PCollection<String> results = Task.applyTransform(fruits, countries);
-
- PAssert.that(results)
- .containsInAnyOrder(
- "WordsAlphabet{alphabet='a', fruit='apple', country='australia'}",
- "WordsAlphabet{alphabet='b', fruit='banana', country='brazil'}",
- "WordsAlphabet{alphabet='c', fruit='cherry', country='canada'}"
- );
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java
new file mode 100644
index 0000000..5d3229a
--- /dev/null
+++ b/learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.cogroupbykey;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void coGroupByKey() {
+ PCollection<String> fruits =
+ testPipeline.apply("Fruits",
+ Create.of("apple", "banana", "cherry")
+ );
+
+ PCollection<String> countries =
+ testPipeline.apply("Countries",
+ Create.of("australia", "brazil", "canada")
+ );
+
+ PCollection<String> results = Task.applyTransform(fruits, countries);
+
+ PAssert.that(results)
+ .containsInAnyOrder(
+ "WordsAlphabet{alphabet='a', fruit='apple', country='australia'}",
+ "WordsAlphabet{alphabet='b', fruit='banana', country='brazil'}",
+ "WordsAlphabet{alphabet='c', fruit='cherry', country='canada'}"
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java
new file mode 100644
index 0000000..80a9046
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.binarycombinefnlambda;
+
+import java.math.BigInteger;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<BigInteger> numbers =
+ pipeline.apply(
+ Create.of(
+ BigInteger.valueOf(10), BigInteger.valueOf(20), BigInteger.valueOf(30),
+ BigInteger.valueOf(40), BigInteger.valueOf(50)
+ ));
+
+ PCollection<BigInteger> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<BigInteger> applyTransform(PCollection<BigInteger> input) {
+ return input.apply(Combine.globally(BigInteger::add));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task.html b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task.html
new file mode 100644
index 0000000..ccafa62
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task.html
@@ -0,0 +1,43 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Combine - BinaryCombineFn Lambda</h2>
+<p>
+ BinaryCombineFn is used for implementing combiners that are more easily expressed as binary
+ operations.
+</p>
+<p>
+ Since Beam v2.13.0, you can also use lambda or method reference in order to create the
+ BinaryCombineFn.
+</p>
+<p>
+ <b>Kata:</b> Implement the summation of BigInteger using lambda or method reference.
+</p>
+<br>
+<div class="hint">
+ Refer to
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableBiFunction.html">
+ SerializableBiFunction</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#combine">
+ "Combine"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java
new file mode 100644
index 0000000..e49c005
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.binarycombinefnlambda;
+
+import java.math.BigInteger;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void combine_binaryCombineFn_lambda() {
+ Create.Values<BigInteger> values = Create.of(
+ BigInteger.valueOf(10), BigInteger.valueOf(20), BigInteger.valueOf(30),
+ BigInteger.valueOf(40), BigInteger.valueOf(50)
+ );
+ PCollection<BigInteger> numbers = testPipeline.apply(values);
+
+ PCollection<BigInteger> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(BigInteger.valueOf(150));
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/Task.java b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/Task.java
deleted file mode 100644
index c3c9ac2..0000000
--- a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/Task.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.math.BigInteger;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Combine.BinaryCombineFn;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<BigInteger> numbers =
- pipeline.apply(
- Create.of(
- BigInteger.valueOf(10), BigInteger.valueOf(20), BigInteger.valueOf(30),
- BigInteger.valueOf(40), BigInteger.valueOf(50)
- ));
-
- PCollection<BigInteger> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<BigInteger> applyTransform(PCollection<BigInteger> input) {
- return input.apply(Combine.globally(new SumBigIntegerFn()));
- }
-
- static class SumBigIntegerFn extends BinaryCombineFn<BigInteger> {
-
- @Override
- public BigInteger apply(BigInteger left, BigInteger right) {
- return left.add(right);
- }
-
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java
new file mode 100644
index 0000000..ea33b52
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.binarycombinefn;
+
+import java.math.BigInteger;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Combine.BinaryCombineFn;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<BigInteger> numbers =
+ pipeline.apply(
+ Create.of(
+ BigInteger.valueOf(10), BigInteger.valueOf(20), BigInteger.valueOf(30),
+ BigInteger.valueOf(40), BigInteger.valueOf(50)
+ ));
+
+ PCollection<BigInteger> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<BigInteger> applyTransform(PCollection<BigInteger> input) {
+ return input.apply(Combine.globally(new SumBigIntegerFn()));
+ }
+
+ static class SumBigIntegerFn extends BinaryCombineFn<BigInteger> {
+
+ @Override
+ public BigInteger apply(BigInteger left, BigInteger right) {
+ return left.add(right);
+ }
+
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.html b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.html
index ee8db27..c18d3ac 100644
--- a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.html
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.html
@@ -18,21 +18,33 @@
<html>
<h2>Combine - BinaryCombineFn</h2>
-<p>Combine is a Beam transform for combining collections of elements or values in your data.
+<p>
+ Combine is a Beam transform for combining collections of elements or values in your data.
When you apply a Combine transform, you must provide the function that contains the logic for
combining the elements or values. The combining function should be commutative and associative,
as the function is not necessarily invoked exactly once on all values with a given key. Because
the input data (including the value collection) may be distributed across multiple workers, the
combining function might be called multiple times to perform partial combining on subsets of
- the value collection.</p>
-<p>BinaryCombineFn is used for implementing combiners that are more easily expressed as binary operations.</p>
-<p>In this task, we are going to implement the summation of BigInteger using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">
+ the value collection.
+</p>
+<p>
+ BinaryCombineFn is used for implementing combiners that are more easily expressed as binary
+ operations.
+</p>
+<p>
+ <b>Kata:</b> Implement the summation of BigInteger using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">
Combine.BinaryCombineFn</a>.
</p>
<br>
-<br>
-<div class='hint'>Extend the
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">Combine.BinaryCombineFn</a>
- class that counts the sum of the number.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Extend the
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">
+ Combine.BinaryCombineFn</a> class that counts the sum of the number.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#combine">
+ "Combine"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/TaskTest.java b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/TaskTest.java
deleted file mode 100644
index d4e4f42..0000000
--- a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/TaskTest.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.math.BigInteger;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void combine_binaryCombineFn() {
- Create.Values<BigInteger> values = Create.of(
- BigInteger.valueOf(10), BigInteger.valueOf(20), BigInteger.valueOf(30),
- BigInteger.valueOf(40), BigInteger.valueOf(50)
- );
- PCollection<BigInteger> numbers = testPipeline.apply(values);
-
- PCollection<BigInteger> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(BigInteger.valueOf(150));
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java
new file mode 100644
index 0000000..68b460e
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.binarycombinefn;
+
+import java.math.BigInteger;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void combine_binaryCombineFn() {
+ Create.Values<BigInteger> values = Create.of(
+ BigInteger.valueOf(10), BigInteger.valueOf(20), BigInteger.valueOf(30),
+ BigInteger.valueOf(40), BigInteger.valueOf(50)
+ );
+ PCollection<BigInteger> numbers = testPipeline.apply(values);
+
+ PCollection<BigInteger> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(BigInteger.valueOf(150));
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/src/Task.java b/learning/katas/java/Core Transforms/Combine/Combine PerKey/src/Task.java
deleted file mode 100644
index 16ca7c5a..0000000
--- a/learning/katas/java/Core Transforms/Combine/Combine PerKey/src/Task.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Combine.BinaryCombineFn;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- static final String PLAYER_1 = "Player 1";
- static final String PLAYER_2 = "Player 2";
- static final String PLAYER_3 = "Player 3";
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<KV<String, Integer>> scores =
- pipeline.apply(
- Create.of(
- KV.of(PLAYER_1, 15), KV.of(PLAYER_2, 10), KV.of(PLAYER_1, 100),
- KV.of(PLAYER_3, 25), KV.of(PLAYER_2, 75)
- ));
-
- PCollection<KV<String, Integer>> output = applyTransform(scores);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<KV<String, Integer>> applyTransform(PCollection<KV<String, Integer>> input) {
- return input.apply(Combine.perKey(new SumIntBinaryCombineFn()));
- }
-
- static class SumIntBinaryCombineFn extends BinaryCombineFn<Integer> {
-
- @Override
- public Integer apply(Integer left, Integer right) {
- return left + right;
- }
-
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java b/learning/katas/java/Core Transforms/Combine/Combine PerKey/src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java
new file mode 100644
index 0000000..92c7981
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Combine PerKey/src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.combineperkey;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Combine.BinaryCombineFn;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ static final String PLAYER_1 = "Player 1";
+ static final String PLAYER_2 = "Player 2";
+ static final String PLAYER_3 = "Player 3";
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<KV<String, Integer>> scores =
+ pipeline.apply(
+ Create.of(
+ KV.of(PLAYER_1, 15), KV.of(PLAYER_2, 10), KV.of(PLAYER_1, 100),
+ KV.of(PLAYER_3, 25), KV.of(PLAYER_2, 75)
+ ));
+
+ PCollection<KV<String, Integer>> output = applyTransform(scores);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<KV<String, Integer>> applyTransform(PCollection<KV<String, Integer>> input) {
+ return input.apply(Combine.perKey(new SumIntBinaryCombineFn()));
+ }
+
+ static class SumIntBinaryCombineFn extends BinaryCombineFn<Integer> {
+
+ @Override
+ public Integer apply(Integer left, Integer right) {
+ return left + right;
+ }
+
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/task.html b/learning/katas/java/Core Transforms/Combine/Combine PerKey/task.html
index 76a591a..62b6afb 100644
--- a/learning/katas/java/Core Transforms/Combine/Combine PerKey/task.html
+++ b/learning/katas/java/Core Transforms/Combine/Combine PerKey/task.html
@@ -18,19 +18,31 @@
<html>
<h2>Combine - Combine PerKey</h2>
-<p>After creating a keyed PCollection (for example, by using a GroupByKey transform), a common
- pattern is to combine the collection of values associated with each key into a single, merged value.
- This pattern of a GroupByKey followed by merging the collection of values is equivalent to
+<p>
+ After creating a keyed PCollection (for example, by using a GroupByKey transform), a common
+ pattern is to combine the collection of values associated with each key into a single, merged
+ value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to
Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative
- reduction function or a subclass of CombineFn.</p>
-<p>In this task, we are going to implement the sum of scores per player using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html">
+ reduction function or a subclass of CombineFn.
+</p>
+<p>
+ <b>Kata:</b> Implement the sum of scores per player using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html">
Combine.perKey</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html">Combine.perKey(GlobalCombineFn)</a>.</div>
-<div class='hint'>Extend the
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">Combine.BinaryCombineFn</a>
- class that counts the sum of the number.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/CombineFnBase.GlobalCombineFn.html">
+ Combine.perKey(GlobalCombineFn)</a>.
+</div>
+<div class="hint">
+ Extend the
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.BinaryCombineFn.html">
+ Combine.BinaryCombineFn</a> class that counts the sum of the number.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection">
+ "Combining values in a keyed PCollection"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/test/TaskTest.java b/learning/katas/java/Core Transforms/Combine/Combine PerKey/test/TaskTest.java
deleted file mode 100644
index ff5df4d..0000000
--- a/learning/katas/java/Core Transforms/Combine/Combine PerKey/test/TaskTest.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @SuppressWarnings("unchecked")
- @Test
- public void combine_combineFn() {
- Create.Values<KV<String, Integer>> values = Create.of(
- KV.of(Task.PLAYER_1, 15), KV.of(Task.PLAYER_2, 10), KV.of(Task.PLAYER_1, 100),
- KV.of(Task.PLAYER_3, 25), KV.of(Task.PLAYER_2, 75)
- );
- PCollection<KV<String, Integer>> numbers = testPipeline.apply(values);
-
- PCollection<KV<String, Integer>> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(
- KV.of(Task.PLAYER_1, 115), KV.of(Task.PLAYER_2, 85), KV.of(Task.PLAYER_3, 25)
- );
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Combine PerKey/test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java b/learning/katas/java/Core Transforms/Combine/Combine PerKey/test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java
new file mode 100644
index 0000000..4be2a6b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Combine PerKey/test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.combineperkey;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void combine_combineFn() {
+ Create.Values<KV<String, Integer>> values = Create.of(
+ KV.of(Task.PLAYER_1, 15), KV.of(Task.PLAYER_2, 10), KV.of(Task.PLAYER_1, 100),
+ KV.of(Task.PLAYER_3, 25), KV.of(Task.PLAYER_2, 75)
+ );
+ PCollection<KV<String, Integer>> numbers = testPipeline.apply(values);
+
+ PCollection<KV<String, Integer>> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(
+ KV.of(Task.PLAYER_1, 115), KV.of(Task.PLAYER_2, 85), KV.of(Task.PLAYER_3, 25)
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/src/Task.java b/learning/katas/java/Core Transforms/Combine/CombineFn/src/Task.java
deleted file mode 100644
index fa2d226..0000000
--- a/learning/katas/java/Core Transforms/Combine/CombineFn/src/Task.java
+++ /dev/null
@@ -1,106 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.io.Serializable;
-import java.util.Objects;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(10, 20, 50, 70, 90));
-
- PCollection<Double> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Double> applyTransform(PCollection<Integer> input) {
- return input.apply(Combine.globally(new AverageFn()));
- }
-
- static class AverageFn extends CombineFn<Integer, AverageFn.Accum, Double> {
-
- class Accum implements Serializable {
- int sum = 0;
- int count = 0;
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (o == null || getClass() != o.getClass()) {
- return false;
- }
- Accum accum = (Accum) o;
- return sum == accum.sum &&
- count == accum.count;
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(sum, count);
- }
- }
-
- @Override
- public Accum createAccumulator() {
- return new Accum();
- }
-
- @Override
- public Accum addInput(Accum accumulator, Integer input) {
- accumulator.sum += input;
- accumulator.count++;
-
- return accumulator;
- }
-
- @Override
- public Accum mergeAccumulators(Iterable<Accum> accumulators) {
- Accum merged = createAccumulator();
-
- for (Accum accumulator : accumulators) {
- merged.sum += accumulator.sum;
- merged.count += accumulator.count;
- }
-
- return merged;
- }
-
- @Override
- public Double extractOutput(Accum accumulator) {
- return ((double) accumulator.sum) / accumulator.count;
- }
-
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java b/learning/katas/java/Core Transforms/Combine/CombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java
new file mode 100644
index 0000000..ac244fb
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/CombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.combinefn;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(10, 20, 50, 70, 90));
+
+ PCollection<Double> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Double> applyTransform(PCollection<Integer> input) {
+ return input.apply(Combine.globally(new AverageFn()));
+ }
+
+ static class AverageFn extends CombineFn<Integer, AverageFn.Accum, Double> {
+
+ class Accum implements Serializable {
+ int sum = 0;
+ int count = 0;
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Accum accum = (Accum) o;
+ return sum == accum.sum &&
+ count == accum.count;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(sum, count);
+ }
+ }
+
+ @Override
+ public Accum createAccumulator() {
+ return new Accum();
+ }
+
+ @Override
+ public Accum addInput(Accum accumulator, Integer input) {
+ accumulator.sum += input;
+ accumulator.count++;
+
+ return accumulator;
+ }
+
+ @Override
+ public Accum mergeAccumulators(Iterable<Accum> accumulators) {
+ Accum merged = createAccumulator();
+
+ for (Accum accumulator : accumulators) {
+ merged.sum += accumulator.sum;
+ merged.count += accumulator.count;
+ }
+
+ return merged;
+ }
+
+ @Override
+ public Double extractOutput(Accum accumulator) {
+ return ((double) accumulator.sum) / accumulator.count;
+ }
+
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/task.html b/learning/katas/java/Core Transforms/Combine/CombineFn/task.html
index 2a31de3..94b6be3 100644
--- a/learning/katas/java/Core Transforms/Combine/CombineFn/task.html
+++ b/learning/katas/java/Core Transforms/Combine/CombineFn/task.html
@@ -18,24 +18,35 @@
<html>
<h2>Combine - CombineFn</h2>
-<p>Combine is a Beam transform for combining collections of elements or values in your data.
+<p>
+ Combine is a Beam transform for combining collections of elements or values in your data.
When you apply a Combine transform, you must provide the function that contains the logic for
combining the elements or values. The combining function should be commutative and associative,
as the function is not necessarily invoked exactly once on all values with a given key. Because
the input data (including the value collection) may be distributed across multiple workers, the
combining function might be called multiple times to perform partial combining on subsets of
- the value collection.</p>
-<p>Complex combination operations might require you to create a subclass of CombineFn that has an
+ the value collection.
+</p>
+<p>
+ Complex combination operations might require you to create a subclass of CombineFn that has an
accumulation type distinct from the input/output type. You should use CombineFn if the combine
function requires a more sophisticated accumulator, must perform additional pre- or
- post-processing, might change the output type, or takes the key into account.</p>
-<p>In this task, we are going to implement the average of numbers using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.CombineFn.html">
+ post-processing, might change the output type, or takes the key into account.
+</p>
+<p>
+ <b>Kata:</b> Implement the average of numbers using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html">
Combine.CombineFn</a>.
</p>
<br>
-<br>
-<div class='hint'>Extend the
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Combine.CombineFn.html">Combine.CombineFn</a>
- class that counts the average of the number.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Extend the
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.CombineFn.html">
+ Combine.CombineFn</a> class that counts the average of the number.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines">
+ "Advanced combinations using CombineFn"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/test/TaskTest.java b/learning/katas/java/Core Transforms/Combine/CombineFn/test/TaskTest.java
deleted file mode 100644
index a6631e4..0000000
--- a/learning/katas/java/Core Transforms/Combine/CombineFn/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void combine_combineFn() {
- Create.Values<Integer> values = Create.of(10, 20, 50, 70, 90);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Double> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(48.0);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/CombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java b/learning/katas/java/Core Transforms/Combine/CombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java
new file mode 100644
index 0000000..6dc06ca
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/CombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.combinefn;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void combine_combineFn() {
+ Create.Values<Integer> values = Create.of(10, 20, 50, 70, 90);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Double> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(48.0);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/src/Task.java b/learning/katas/java/Core Transforms/Combine/Simple Function/src/Task.java
deleted file mode 100644
index 5b4591f..0000000
--- a/learning/katas/java/Core Transforms/Combine/Simple Function/src/Task.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers = pipeline.apply(Create.of(10, 30, 50, 70, 90));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(Combine.globally(new SumIntegerFn()));
- }
-
- static class SumIntegerFn implements SerializableFunction<Iterable<Integer>, Integer> {
-
- @Override
- public Integer apply(Iterable<Integer> input) {
- int sum = 0;
-
- for (int item : input) {
- sum += item;
- }
-
- return sum;
- }
-
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java b/learning/katas/java/Core Transforms/Combine/Simple Function/src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java
new file mode 100644
index 0000000..2c94ef4
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Simple Function/src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.simple;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(10, 30, 50, 70, 90));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(Combine.globally(new SumIntegerFn()));
+ }
+
+ static class SumIntegerFn implements SerializableFunction<Iterable<Integer>, Integer> {
+
+ @Override
+ public Integer apply(Iterable<Integer> input) {
+ int sum = 0;
+
+ for (int item : input) {
+ sum += item;
+ }
+
+ return sum;
+ }
+
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/task.html b/learning/katas/java/Core Transforms/Combine/Simple Function/task.html
index 0006a87..d501be8 100644
--- a/learning/katas/java/Core Transforms/Combine/Simple Function/task.html
+++ b/learning/katas/java/Core Transforms/Combine/Simple Function/task.html
@@ -18,21 +18,32 @@
<html>
<h2>Combine - Simple Function</h2>
-<p>Combine is a Beam transform for combining collections of elements or values in your data.
+<p>
+ Combine is a Beam transform for combining collections of elements or values in your data.
When you apply a Combine transform, you must provide the function that contains the logic for
combining the elements or values. The combining function should be commutative and associative,
as the function is not necessarily invoked exactly once on all values with a given key. Because
the input data (including the value collection) may be distributed across multiple workers, the
combining function might be called multiple times to perform partial combining on subsets of
- the value collection.</p>
-<p>Simple combine operations, such as sums, can usually be implemented as a simple function.</p>
-<p>In this task, we are going to implement the summation of numbers using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/SerializableFunction.html">
+ the value collection.
+</p>
+<p>
+ Simple combine operations, such as sums, can usually be implemented as a simple function.
+</p>
+<p>
+ <b>Kata:</b> Implement the summation of numbers using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html">
Combine.globally(SerializableFunction)</a>.
</p>
<br>
-<br>
-<div class='hint'>Implement the
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/SerializableFunction.html#apply-InputT-">SerializableFunction.apply</a>
- method that performs the summation of the Iterable.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Implement the
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/SerializableFunction.html#apply-InputT-">
+ SerializableFunction.apply</a> method that performs the summation of the Iterable.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines">
+ "Simple combinations using simple functions"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/test/TaskTest.java b/learning/katas/java/Core Transforms/Combine/Simple Function/test/TaskTest.java
deleted file mode 100644
index 85cfcae..0000000
--- a/learning/katas/java/Core Transforms/Combine/Simple Function/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void combine_simpleFn() {
- Create.Values<Integer> values = Create.of(10, 30, 50, 70, 90);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(250);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Combine/Simple Function/test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java b/learning/katas/java/Core Transforms/Combine/Simple Function/test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java
new file mode 100644
index 0000000..1006692
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Combine/Simple Function/test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.combine.simple;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void combine_simpleFn() {
+ Create.Values<Integer> values = Create.of(10, 30, 50, 70, 90);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(250);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/src/org/apache/beam/learning/katas/coretransforms/composite/Task.java b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/src/org/apache/beam/learning/katas/coretransforms/composite/Task.java
new file mode 100644
index 0000000..38ab3bf
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/src/org/apache/beam/learning/katas/coretransforms/composite/Task.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.composite;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+
+import java.util.Arrays;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ pipeline
+ .apply(Create.of("1,2,3,4,5", "6,7,8,9,10"))
+ .apply(new ExtractAndMultiplyNumbers())
+ .apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static class ExtractAndMultiplyNumbers
+ extends PTransform<PCollection<String>, PCollection<Integer>> {
+
+ @Override
+ public PCollection<Integer> expand(PCollection<String> input) {
+ return input
+ .apply(ParDo.of(new DoFn<String, Integer>() {
+
+ @ProcessElement
+ public void processElement(@Element String numbers, OutputReceiver<Integer> out) {
+ Arrays.stream(numbers.split(","))
+ .forEach(numStr -> out.output(Integer.parseInt(numStr)));
+ }
+
+ }))
+
+ .apply(MapElements.into(integers()).via(number -> number * 10));
+ }
+
+ }
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task.html b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task.html
new file mode 100644
index 0000000..52c0f24
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task.html
@@ -0,0 +1,51 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Composite Transform</h2>
+<p>
+ Transforms can have a nested structure, where a complex transform performs multiple simpler
+ transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms).
+ These transforms are called composite transforms. Nesting multiple transforms inside a single
+ composite transform can make your code more modular and easier to understand.
+</p>
+<p>
+ To create your own composite transform, create a subclass of the PTransform class and override
+ the expand method to specify the actual processing logic. You can then use this transform just as
+ you would a built-in transform from the Beam SDK. For the PTransform class type parameters, you
+ pass the PCollection types that your transform takes as input, and produces as output. Within
+ your PTransform subclass, you’ll need to override the expand method. The expand method is where
+ you add the processing logic for the PTransform. Your override of expand must accept the
+ appropriate type of input PCollection as a parameter, and specify the output PCollection as the
+ return value.
+</p>
+<p>
+ <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts
+ numbers from comma separated line and then multiplies each number by 10.
+</p>
+<br>
+<div class="hint">
+ Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/PTransform.html">
+ PTransform</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms">
+ "Composite transforms"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java
new file mode 100644
index 0000000..f5c1156
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Composite Transform/Composite Transform/test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.composite;
+
+import org.apache.beam.learning.katas.coretransforms.composite.Task.ExtractAndMultiplyNumbers;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void compositeTransform() {
+ Create.Values<String> values = Create.of("1,2,3,4,5", "6,7,8,9,10");
+
+ PCollection<Integer> results =
+ testPipeline
+ .apply(values)
+ .apply(new ExtractAndMultiplyNumbers());
+
+ PAssert.that(results)
+ .containsInAnyOrder(10, 20, 30, 40, 50, 60, 70, 80, 90, 100);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java
new file mode 100644
index 0000000..0ad9101
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.dofnadditionalparams;
+
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+
+
+ pipeline.run();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task.html b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task.html
new file mode 100644
index 0000000..c6e38b0
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task.html
@@ -0,0 +1,52 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>DoFn Additional Parameters</h2>
+<p>
+ In addition to the element and the OutputReceiver, Beam will populate other parameters to your
+ DoFn’s @ProcessElement method. Any combination of these parameters can be added to your process
+ method in any order.
+</p>
+<div>
+ <ul>
+ <li>
+ <b>Timestamp</b>: To access the timestamp of an input element, add a parameter annotated with
+ @Timestamp of type Instant
+ </li>
+ <li>
+ <b>Window</b>: To access the window an input element falls into, add a parameter of the type of the
+ window used for the input PCollection.
+ </li>
+ <li>
+ <b>PaneInfo</b>: When triggers are used, Beam provides a PaneInfo object that contains information
+ about the current firing. Using PaneInfo you can determine whether this is an early or a
+ late firing, and how many times this window has already fired for this key.
+ </li>
+ <li>
+ <b>PipelineOptions</b>: The PipelineOptions for the current pipeline can always be accessed in a
+ process method by adding it as a parameter.
+ </li>
+ </ul>
+</div>
+<p>
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#other-dofn-parameters">
+ "Accessing additional parameters in your DoFn"</a> section for more information.
+</p>
+</html>
diff --git a/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java
new file mode 100644
index 0000000..121d45a
--- /dev/null
+++ b/learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.dofnadditionalparams;
+
+public class TaskTest {
+}
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/src/Task.java b/learning/katas/java/Core Transforms/Flatten/Flatten/src/Task.java
deleted file mode 100644
index 1d36f22..0000000
--- a/learning/katas/java/Core Transforms/Flatten/Flatten/src/Task.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionList;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> wordsStartingWithA =
- pipeline.apply("Words starting with A",
- Create.of("apple", "ant", "arrow")
- );
-
- PCollection<String> wordsStartingWithB =
- pipeline.apply("Words starting with B",
- Create.of("ball", "book", "bow")
- );
-
- PCollection<String> output = applyTransform(wordsStartingWithA, wordsStartingWithB);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<String> applyTransform(
- PCollection<String> words1, PCollection<String> words2) {
-
- return PCollectionList.of(words1).and(words2)
- .apply(Flatten.pCollections());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java b/learning/katas/java/Core Transforms/Flatten/Flatten/src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java
new file mode 100644
index 0000000..fb90de7
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Flatten/Flatten/src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.flatten;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> wordsStartingWithA =
+ pipeline.apply("Words starting with A",
+ Create.of("apple", "ant", "arrow")
+ );
+
+ PCollection<String> wordsStartingWithB =
+ pipeline.apply("Words starting with B",
+ Create.of("ball", "book", "bow")
+ );
+
+ PCollection<String> output = applyTransform(wordsStartingWithA, wordsStartingWithB);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> applyTransform(
+ PCollection<String> words1, PCollection<String> words2) {
+
+ return PCollectionList.of(words1).and(words2)
+ .apply(Flatten.pCollections());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/task.html b/learning/katas/java/Core Transforms/Flatten/Flatten/task.html
index 3bef763..f0bf35e 100644
--- a/learning/katas/java/Core Transforms/Flatten/Flatten/task.html
+++ b/learning/katas/java/Core Transforms/Flatten/Flatten/task.html
@@ -18,15 +18,24 @@
<html>
<h2>Flatten</h2>
-<p>Flatten is a Beam transform for PCollection objects that store the same data type.
- Flatten merges multiple PCollection objects into a single logical PCollection.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Flatten.html">
+<p>
+ Flatten is a Beam transform for PCollection objects that store the same data type.
+ Flatten merges multiple PCollection objects into a single logical PCollection.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html">
Flatten</a> transform that merges two PCollection of words into a single PCollection.
</p>
<br>
-<br>
-<div class='hint'>Refer to
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Flatten.html">Flatten</a>
- to solve this problem.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Flatten.html">
+ Flatten</a> to solve this problem.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#flatten">
+ "Flatten"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/test/TaskTest.java b/learning/katas/java/Core Transforms/Flatten/Flatten/test/TaskTest.java
deleted file mode 100644
index d6df982..0000000
--- a/learning/katas/java/Core Transforms/Flatten/Flatten/test/TaskTest.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void flatten() {
- PCollection<String> wordsStartingWithA =
- testPipeline.apply("Words starting with A",
- Create.of("apple", "ant", "arrow"));
- PCollection<String> wordsStartingWithB =
- testPipeline.apply("Words starting with B",
- Create.of("ball", "book", "bow"));
-
- PCollection<String> results = Task.applyTransform(wordsStartingWithA, wordsStartingWithB);
-
- PAssert.that(results)
- .containsInAnyOrder("apple", "ant", "arrow", "ball", "book", "bow");
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Flatten/Flatten/test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java b/learning/katas/java/Core Transforms/Flatten/Flatten/test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java
new file mode 100644
index 0000000..4e59fef
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Flatten/Flatten/test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.flatten;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void flatten() {
+ PCollection<String> wordsStartingWithA =
+ testPipeline.apply("Words starting with A",
+ Create.of("apple", "ant", "arrow"));
+ PCollection<String> wordsStartingWithB =
+ testPipeline.apply("Words starting with B",
+ Create.of("ball", "book", "bow"));
+
+ PCollection<String> results = Task.applyTransform(wordsStartingWithA, wordsStartingWithB);
+
+ PAssert.that(results)
+ .containsInAnyOrder("apple", "ant", "arrow", "ball", "book", "bow");
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/Task.java b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/Task.java
deleted file mode 100644
index 548fd1c..0000000
--- a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/Task.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
-import static org.apache.beam.sdk.values.TypeDescriptors.strings;
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> words =
- pipeline.apply(
- Create.of("apple", "ball", "car", "bear", "cheetah", "ant")
- );
-
- PCollection<KV<String, Iterable<String>>> output = applyTransform(words);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<KV<String, Iterable<String>>> applyTransform(PCollection<String> input) {
- return input
- .apply(MapElements.into(kvs(strings(), strings()))
- .via(word -> KV.of(word.substring(0, 1), word)))
-
- .apply(GroupByKey.create());
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java
new file mode 100644
index 0000000..e9ca7f4
--- /dev/null
+++ b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.groupbykey;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.kvs;
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> words =
+ pipeline.apply(
+ Create.of("apple", "ball", "car", "bear", "cheetah", "ant")
+ );
+
+ PCollection<KV<String, Iterable<String>>> output = applyTransform(words);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<KV<String, Iterable<String>>> applyTransform(PCollection<String> input) {
+ return input
+ .apply(MapElements.into(kvs(strings(), strings()))
+ .via(word -> KV.of(word.substring(0, 1), word)))
+
+ .apply(GroupByKey.create());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task.html b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task.html
index 05469eb..54082b0 100644
--- a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task.html
+++ b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task.html
@@ -18,18 +18,28 @@
<html>
<h2>GroupByKey</h2>
-<p>GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation,
- analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of
- key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key,
- but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each
- unique key.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/GroupByKey.html">
- GroupByKey</a> transform that groups words by its first letter.
+<p>
+ GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel
+ reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The
+ input to GroupByKey is a collection of key/value pairs that represents a multimap, where the
+ collection contains multiple pairs that have the same key, but different values. Given such a
+ collection, you use GroupByKey to collect all of the values associated with each unique key.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html">
+ GroupByKey</a> transform that groups words by its first letter.
</p>
<br>
-<br>
-<div class='hint'>Refer to <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/values/KV.html">KV</a>
- and <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/GroupByKey.html">GroupByKey</a>
- to solve this problem.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/KV.html">
+ KV</a> and
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupByKey.html">
+ GroupByKey</a> to solve this problem.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey">
+ "GroupByKey"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/TaskTest.java b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/TaskTest.java
deleted file mode 100644
index 15cf69e..0000000
--- a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/TaskTest.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import static test.util.ContainsKvs.containsKvs;
-
-import com.google.common.collect.ImmutableList;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void groupByKey() {
- Create.Values<String> values = Create.of("apple", "ball", "car", "bear", "cheetah", "ant");
- PCollection<String> numbers = testPipeline.apply(values);
-
- PCollection<KV<String, Iterable<String>>> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .satisfies(
- containsKvs(
- KV.of("a", ImmutableList.of("apple", "ant")),
- KV.of("b", ImmutableList.of("ball", "bear")),
- KV.of("c", ImmutableList.of("car", "cheetah"))
- )
- );
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java
new file mode 100644
index 0000000..d21d4de
--- /dev/null
+++ b/learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.groupbykey;
+
+import static org.apache.beam.learning.katas.util.ContainsKvs.containsKvs;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void groupByKey() {
+ Create.Values<String> values = Create.of("apple", "ball", "car", "bear", "cheetah", "ant");
+ PCollection<String> numbers = testPipeline.apply(values);
+
+ PCollection<KV<String, Iterable<String>>> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .satisfies(
+ containsKvs(
+ KV.of("a", ImmutableList.of("apple", "ant")),
+ KV.of("b", ImmutableList.of("ball", "bear")),
+ KV.of("c", ImmutableList.of("car", "cheetah"))
+ )
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/src/Task.java b/learning/katas/java/Core Transforms/Map/FlatMapElements/src/Task.java
deleted file mode 100644
index cf442d8..0000000
--- a/learning/katas/java/Core Transforms/Map/FlatMapElements/src/Task.java
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.util.Arrays;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.FlatMapElements;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.TypeDescriptors;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> sentences =
- pipeline.apply(Create.of("Apache Beam", "Unified Batch and Streaming"));
-
- PCollection<String> output = applyTransform(sentences);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<String> applyTransform(PCollection<String> input) {
- return input.apply(
- FlatMapElements.into(TypeDescriptors.strings())
- .via(sentence -> Arrays.asList(sentence.split(" ")))
- );
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java b/learning/katas/java/Core Transforms/Map/FlatMapElements/src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java
new file mode 100644
index 0000000..e2db31b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/FlatMapElements/src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.flatmapelements;
+
+import java.util.Arrays;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.FlatMapElements;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptors;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> sentences =
+ pipeline.apply(Create.of("Apache Beam", "Unified Batch and Streaming"));
+
+ PCollection<String> output = applyTransform(sentences);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> applyTransform(PCollection<String> input) {
+ return input.apply(
+ FlatMapElements.into(TypeDescriptors.strings())
+ .via(sentence -> Arrays.asList(sentence.split(" ")))
+ );
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/task.html b/learning/katas/java/Core Transforms/Map/FlatMapElements/task.html
index 1572a8e..50f1627 100644
--- a/learning/katas/java/Core Transforms/Map/FlatMapElements/task.html
+++ b/learning/katas/java/Core Transforms/Map/FlatMapElements/task.html
@@ -18,15 +18,27 @@
<html>
<h2>FlatMapElements</h2>
-<p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p>
-<p>FlatMapElements can be used to simplify DoFn that maps an element to multiple elements (one to many).</p>
-<p>In this task, we are going to implement a function that maps each input sentence into words tokenized by whitespace (" ") using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/FlatMapElements.html">
- FlatMapElements.into(...).via(...)</a>.
+<p>
+ The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+</p>
+<p>
+ FlatMapElements can be used to simplify a DoFn that maps an element to multiple elements (one to
+ many).
+</p>
+<p>
+ <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace
+ (" ") using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html">
+ FlatMapElements.into(...).via(...)</a>.
</p>
<br>
-<br>
-<div class='hint'>Use
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/FlatMapElements.html">FlatMapElements.into(...).via(...)</a>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/FlatMapElements.html">
+ FlatMapElements.into(...).via(...)</a>.
</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns">
+ "Lightweight DoFns and other abstractions"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/test/TaskTest.java b/learning/katas/java/Core Transforms/Map/FlatMapElements/test/TaskTest.java
deleted file mode 100644
index db5de38..0000000
--- a/learning/katas/java/Core Transforms/Map/FlatMapElements/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void flatMapElements() {
- Create.Values<String> values = Create.of("Apache Beam", "Unified Batch and Streaming");
- PCollection<String> numbers = testPipeline.apply(values);
-
- PCollection<String> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder("Apache", "Beam", "Unified", "Batch", "and", "Streaming");
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/FlatMapElements/test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java b/learning/katas/java/Core Transforms/Map/FlatMapElements/test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java
new file mode 100644
index 0000000..5522f28
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/FlatMapElements/test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.flatmapelements;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void flatMapElements() {
+ Create.Values<String> values = Create.of("Apache Beam", "Unified Batch and Streaming");
+ PCollection<String> numbers = testPipeline.apply(values);
+
+ PCollection<String> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder("Apache", "Beam", "Unified", "Batch", "and", "Streaming");
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/src/Task.java b/learning/katas/java/Core Transforms/Map/MapElements/src/Task.java
deleted file mode 100644
index 4a1b49c..0000000
--- a/learning/katas/java/Core Transforms/Map/MapElements/src/Task.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.TypeDescriptors;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers =
- pipeline.apply(Create.of(10, 20, 30, 40, 50));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(
- MapElements.into(TypeDescriptors.integers())
- .via(number -> number * 5)
- );
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java b/learning/katas/java/Core Transforms/Map/MapElements/src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java
new file mode 100644
index 0000000..16ec895
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/MapElements/src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.mapelements;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptors;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers =
+ pipeline.apply(Create.of(10, 20, 30, 40, 50));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(
+ MapElements.into(TypeDescriptors.integers())
+ .via(number -> number * 5)
+ );
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/task.html b/learning/katas/java/Core Transforms/Map/MapElements/task.html
index 8fbfa97..68ae60c 100644
--- a/learning/katas/java/Core Transforms/Map/MapElements/task.html
+++ b/learning/katas/java/Core Transforms/Map/MapElements/task.html
@@ -18,13 +18,25 @@
<html>
<h2>MapElements</h2>
-<p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p>
-<p>MapElements can be used to simplify DoFn that maps an element to another element (one to one).</p>
-<p>In this task, we are going to implement a simple map function that multiplies all input elements by 5 using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/MapElements.html">
- MapElements.into(...).via(...)</a>.
+<p>
+ The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+</p>
+<p>
+ MapElements can be used to simplify a DoFn that maps an element to another element (one to one).
+</p>
+<p>
+ <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html">
+ MapElements.into(...).via(...)</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/MapElements.html">MapElements.into(...).via(...)</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html">
+ MapElements.into(...).via(...)</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns">
+ "Lightweight DoFns and other abstractions"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/test/TaskTest.java b/learning/katas/java/Core Transforms/Map/MapElements/test/TaskTest.java
deleted file mode 100644
index 9d713a7..0000000
--- a/learning/katas/java/Core Transforms/Map/MapElements/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void mapElements() {
- Create.Values<Integer> values = Create.of(10, 20, 30, 40, 50);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(50, 100, 150, 200, 250);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/MapElements/test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java b/learning/katas/java/Core Transforms/Map/MapElements/test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java
new file mode 100644
index 0000000..2c10ae5
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/MapElements/test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.mapelements;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void mapElements() {
+ Create.Values<Integer> values = Create.of(10, 20, 30, 40, 50);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(50, 100, 150, 200, 250);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/Task.java b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/Task.java
deleted file mode 100644
index 9e52ca6..0000000
--- a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/Task.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> sentences =
- pipeline.apply(Create.of("Hello Beam", "It is awesome"));
-
- PCollection<String> output = applyTransform(sentences);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<String> applyTransform(PCollection<String> input) {
- return input.apply(ParDo.of(new DoFn<String, String>() {
-
- @ProcessElement
- public void processElement(@Element String sentence, OutputReceiver<String> out) {
- String[] words = sentence.split(" ");
-
- for (String word : words) {
- out.output(word);
- }
- }
-
- }));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java
new file mode 100644
index 0000000..69b092f
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.pardoonetomany;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> sentences =
+ pipeline.apply(Create.of("Hello Beam", "It is awesome"));
+
+ PCollection<String> output = applyTransform(sentences);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> applyTransform(PCollection<String> input) {
+ return input.apply(ParDo.of(new DoFn<String, String>() {
+
+ @ProcessElement
+ public void processElement(@Element String sentence, OutputReceiver<String> out) {
+ String[] words = sentence.split(" ");
+
+ for (String word : words) {
+ out.output(word);
+ }
+ }
+
+ }));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task.html b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task.html
index 02167e0..b9e134e 100644
--- a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task.html
+++ b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/task.html
@@ -18,10 +18,21 @@
<html>
<h2>ParDo OneToMany</h2>
-<p>For this task, please write a ParDo that maps each input sentence into words tokenized by whitespace (" ").</p>
+<p>
+ <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by
+ whitespace (" ").
+</p>
<br>
-<br>
-<div class='hint'>Hint: You can call DoFn.ProcessContext.output(..) multiple times in a
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>.
+<div class="hint">
+ You can call <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html">
+ OutputReceiver</a> multiple times in a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
+ ParDo</a>.
+</div>
+<div class="hint">
+ If you're using Beam version before v2.5.0, you can call
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.WindowedContext.html#output-OutputT-">
+ DoFn.ProcessContext.output(..)</a> multiple times in a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>.
</div>
</html>
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/TaskTest.java b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/TaskTest.java
deleted file mode 100644
index eecf2da..0000000
--- a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void parDo_oneToMany() {
- Create.Values<String> values = Create.of("Hello Beam", "It is awesome");
- PCollection<String> numbers = testPipeline.apply(values);
-
- PCollection<String> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder("Hello", "Beam", "It", "is", "awesome");
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java
new file mode 100644
index 0000000..1129220
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.pardoonetomany;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void parDo_oneToMany() {
+ Create.Values<String> values = Create.of("Hello Beam", "It is awesome");
+ PCollection<String> numbers = testPipeline.apply(values);
+
+ PCollection<String> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder("Hello", "Beam", "It", "is", "awesome");
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/src/Task.java b/learning/katas/java/Core Transforms/Map/ParDo/src/Task.java
deleted file mode 100644
index ad9a039..0000000
--- a/learning/katas/java/Core Transforms/Map/ParDo/src/Task.java
+++ /dev/null
@@ -1,55 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers =
- pipeline.apply(Create.of(1, 2, 3, 4, 5));
-
- PCollection<Integer> output = applyTransform(numbers);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<Integer> applyTransform(PCollection<Integer> input) {
- return input.apply(ParDo.of(new DoFn<Integer, Integer>() {
-
- @ProcessElement
- public void processElement(@Element Integer number, OutputReceiver<Integer> out) {
- out.output(number * 10);
- }
-
- }));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java b/learning/katas/java/Core Transforms/Map/ParDo/src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java
new file mode 100644
index 0000000..c82717d
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo/src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.pardo;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers =
+ pipeline.apply(Create.of(1, 2, 3, 4, 5));
+
+ PCollection<Integer> output = applyTransform(numbers);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Integer> applyTransform(PCollection<Integer> input) {
+ return input.apply(ParDo.of(new DoFn<Integer, Integer>() {
+
+ @ProcessElement
+ public void processElement(@Element Integer number, OutputReceiver<Integer> out) {
+ out.output(number * 10);
+ }
+
+ }));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/task.html b/learning/katas/java/Core Transforms/Map/ParDo/task.html
index 06673dd..15a0ea1 100644
--- a/learning/katas/java/Core Transforms/Map/ParDo/task.html
+++ b/learning/katas/java/Core Transforms/Map/ParDo/task.html
@@ -18,13 +18,25 @@
<html>
<h2>ParDo</h2>
-<p>ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the
- “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the
- input PCollection, performs some processing function (your user code) on that element, and emits
- zero, one, or multiple elements to an output PCollection.</p>
-<p>For this task, please write a simple ParDo that maps the input element by multiplying it by 10.</p>
+<p>
+ ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is
+ similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers
+ each element in the input PCollection, performs some processing function (your user code) on
+ that element, and emits zero, one, or multiple elements to an output PCollection.
+</p>
+<p>
+ <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/ParDo.html">ParDo</a>
- with <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/DoFn.html">DoFn</a></div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
+ ParDo</a>
+ with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
+ DoFn</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for
+ more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/test/TaskTest.java b/learning/katas/java/Core Transforms/Map/ParDo/test/TaskTest.java
deleted file mode 100644
index d9b44bd..0000000
--- a/learning/katas/java/Core Transforms/Map/ParDo/test/TaskTest.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void parDo() {
- Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5);
- PCollection<Integer> numbers = testPipeline.apply(values);
-
- PCollection<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results)
- .containsInAnyOrder(10, 20, 30, 40, 50);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Map/ParDo/test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java b/learning/katas/java/Core Transforms/Map/ParDo/test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java
new file mode 100644
index 0000000..5b791df
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Map/ParDo/test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.map.pardo;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void parDo() {
+ Create.Values<Integer> values = Create.of(1, 2, 3, 4, 5);
+ PCollection<Integer> numbers = testPipeline.apply(values);
+
+ PCollection<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results)
+ .containsInAnyOrder(10, 20, 30, 40, 50);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/src/Task.java b/learning/katas/java/Core Transforms/Partition/Partition/src/Task.java
deleted file mode 100644
index 339f30d..0000000
--- a/learning/katas/java/Core Transforms/Partition/Partition/src/Task.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.Partition;
-import org.apache.beam.sdk.transforms.Partition.PartitionFn;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionList;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<Integer> numbers =
- pipeline.apply(
- Create.of(1, 2, 3, 4, 5, 100, 110, 150, 250)
- );
-
- PCollectionList<Integer> partition = applyTransform(numbers);
-
- partition.get(0).apply(Log.ofElements("Number > 100: "));
- partition.get(1).apply(Log.ofElements("Number <= 100: "));
-
- pipeline.run();
- }
-
- static PCollectionList<Integer> applyTransform(PCollection<Integer> input) {
- return input
- .apply(Partition.of(2,
- (PartitionFn<Integer>) (number, numPartitions) -> {
- if (number > 100) {
- return 0;
- } else {
- return 1;
- }
- }));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/src/org/apache/beam/learning/katas/coretransforms/partition/Task.java b/learning/katas/java/Core Transforms/Partition/Partition/src/org/apache/beam/learning/katas/coretransforms/partition/Task.java
new file mode 100644
index 0000000..9c02cb5
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Partition/Partition/src/org/apache/beam/learning/katas/coretransforms/partition/Task.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.partition;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Partition;
+import org.apache.beam.sdk.transforms.Partition.PartitionFn;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers =
+ pipeline.apply(
+ Create.of(1, 2, 3, 4, 5, 100, 110, 150, 250)
+ );
+
+ PCollectionList<Integer> partition = applyTransform(numbers);
+
+ partition.get(0).apply(Log.ofElements("Number > 100: "));
+ partition.get(1).apply(Log.ofElements("Number <= 100: "));
+
+ pipeline.run();
+ }
+
+ static PCollectionList<Integer> applyTransform(PCollection<Integer> input) {
+ return input
+ .apply(Partition.of(2,
+ (PartitionFn<Integer>) (number, numPartitions) -> {
+ if (number > 100) {
+ return 0;
+ } else {
+ return 1;
+ }
+ }));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/task.html b/learning/katas/java/Core Transforms/Partition/Partition/task.html
index 18d7bce..96e559c 100644
--- a/learning/katas/java/Core Transforms/Partition/Partition/task.html
+++ b/learning/katas/java/Core Transforms/Partition/Partition/task.html
@@ -18,21 +18,31 @@
<html>
<h2>Partition</h2>
-<p>Partition is a Beam transform for PCollection objects that store the same data type.
+<p>
+ Partition is a Beam transform for PCollection objects that store the same data type.
Partition splits a single PCollection into a fixed number of smaller collections.
+</p>
+<p>
Partition divides the elements of a PCollection according to a partitioning function
that you provide. The partitioning function contains the logic that determines how to split up
- the elements of the input PCollection into each resulting partition PCollection.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Partition.html">
+ the elements of the input PCollection into each resulting partition PCollection.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html">
Partition</a> transform that splits a PCollection of numbers into two PCollections.
The first PCollection contains numbers greater than 100, and the second PCollection contains
the remaining numbers.
</p>
<br>
-<br>
-<div class='hint'>Refer to
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Partition.html">Partition</a>
- to solve this problem.
+<div class="hint">
+ Refer to
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Partition.html">
+ Partition</a> to solve this problem.
</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#partition">
+ "Partition"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/test/TaskTest.java b/learning/katas/java/Core Transforms/Partition/Partition/test/TaskTest.java
deleted file mode 100644
index cf38042..0000000
--- a/learning/katas/java/Core Transforms/Partition/Partition/test/TaskTest.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionList;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void groupByKey() {
- PCollection<Integer> numbers =
- testPipeline.apply(
- Create.of(1, 2, 3, 4, 5, 100, 110, 150, 250)
- );
-
- PCollectionList<Integer> results = Task.applyTransform(numbers);
-
- PAssert.that(results.get(0))
- .containsInAnyOrder(110, 150, 250);
-
- PAssert.that(results.get(1))
- .containsInAnyOrder(1, 2, 3, 4, 5, 100);
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Partition/Partition/test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java b/learning/katas/java/Core Transforms/Partition/Partition/test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java
new file mode 100644
index 0000000..fc627a0
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Partition/Partition/test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.partition;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void groupByKey() {
+ PCollection<Integer> numbers =
+ testPipeline.apply(
+ Create.of(1, 2, 3, 4, 5, 100, 110, 150, 250)
+ );
+
+ PCollectionList<Integer> results = Task.applyTransform(numbers);
+
+ PAssert.that(results.get(0))
+ .containsInAnyOrder(110, 150, 250);
+
+ PAssert.that(results.get(1))
+ .containsInAnyOrder(1, 2, 3, 4, 5, 100);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java b/learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java
new file mode 100644
index 0000000..36a3a1c
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.sideinput;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+public class Person implements Serializable {
+
+ private String name;
+ private String city;
+ private String country;
+
+ public Person(String name, String city) {
+ this.name = name;
+ this.city = city;
+ }
+
+ public Person(String name, String city, String country) {
+ this.name = name;
+ this.city = city;
+ this.country = country;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public String getCity() {
+ return city;
+ }
+
+ public String getCountry() {
+ return country;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+
+ Person person = (Person) o;
+
+ return Objects.equals(name, person.name) &&
+ Objects.equals(city, person.city) &&
+ Objects.equals(country, person.country);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(name, city, country);
+ }
+
+ @Override
+ public String toString() {
+ return "Person{" +
+ "name='" + name + '\'' +
+ ", city='" + city + '\'' +
+ ", country='" + country + '\'' +
+ '}';
+ }
+
+}
diff --git a/learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java b/learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java
new file mode 100644
index 0000000..faa2943
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.sideinput;
+
+import java.util.Map;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<KV<String, String>> citiesToCountries =
+ pipeline.apply("Cities and Countries",
+ Create.of(
+ KV.of("Beijing", "China"),
+ KV.of("London", "United Kingdom"),
+ KV.of("San Francisco", "United States"),
+ KV.of("Singapore", "Singapore"),
+ KV.of("Sydney", "Australia")
+ ));
+
+ PCollectionView<Map<String, String>> citiesToCountriesView =
+ createView(citiesToCountries);
+
+ PCollection<Person> persons =
+ pipeline.apply("Persons",
+ Create.of(
+ new Person("Henry", "Singapore"),
+ new Person("Jane", "San Francisco"),
+ new Person("Lee", "Beijing"),
+ new Person("John", "Sydney"),
+ new Person("Alfred", "London")
+ ));
+
+ PCollection<Person> output = applyTransform(persons, citiesToCountriesView);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollectionView<Map<String, String>> createView(
+ PCollection<KV<String, String>> citiesToCountries) {
+
+ return citiesToCountries.apply(View.asMap());
+ }
+
+ static PCollection<Person> applyTransform(
+ PCollection<Person> persons, PCollectionView<Map<String, String>> citiesToCountriesView) {
+
+ return persons.apply(ParDo.of(new DoFn<Person, Person>() {
+
+ @ProcessElement
+ public void processElement(@Element Person person, OutputReceiver<Person> out,
+ ProcessContext context) {
+ Map<String, String> citiesToCountries = context.sideInput(citiesToCountriesView);
+ String city = person.getCity();
+ String country = citiesToCountries.get(city);
+
+ out.output(new Person(person.getName(), city, country));
+ }
+
+ }).withSideInputs(citiesToCountriesView));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Side Input/Side Input/task.html b/learning/katas/java/Core Transforms/Side Input/Side Input/task.html
new file mode 100644
index 0000000..9e7045b
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/Side Input/task.html
@@ -0,0 +1,54 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Side Input</h2>
+<p>
+ In addition to the main input PCollection, you can provide additional inputs to a ParDo transform
+ in the form of side inputs. A side input is an additional input that your DoFn can access each
+ time it processes an element in the input PCollection. When you specify a side input, you create
+ a view of some other data that can be read from within the ParDo transform’s DoFn while
+ processing each element.
+</p>
+<p>
+ Side inputs are useful if your ParDo needs to inject additional data when processing each element
+ in the input PCollection, but the additional data needs to be determined at runtime (and not
+ hard-coded). Such values might be determined by the input data, or depend on a different branch
+ of your pipeline.
+</p>
+<p>
+ <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/View.html">
+ View</a> to create PCollectionView of citiesToCountries.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
+ ParDo</a> with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
+ DoFn</a> that accepts
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withSideInputs-org.apache.beam.sdk.values.PCollectionView...-">
+ side input</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a>
+ section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Side Input/Side Input/test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java b/learning/katas/java/Core Transforms/Side Input/Side Input/test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java
new file mode 100644
index 0000000..5d36e59
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Input/Side Input/test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.sideinput;
+
+import java.util.Map;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void sideInput() {
+ PCollection<KV<String, String>> citiesToCountries =
+ testPipeline.apply("Cities and Countries",
+ Create.of(
+ KV.of("Beijing", "China"),
+ KV.of("London", "United Kingdom"),
+ KV.of("San Francisco", "United States"),
+ KV.of("Singapore", "Singapore"),
+ KV.of("Sydney", "Australia")
+ ));
+
+ PCollectionView<Map<String, String>> citiesToCountriesView =
+ Task.createView(citiesToCountries);
+
+ PCollection<Person> persons =
+ testPipeline.apply("Persons",
+ Create.of(
+ new Person("Henry", "Singapore"),
+ new Person("Jane", "San Francisco"),
+ new Person("Lee", "Beijing"),
+ new Person("John", "Sydney"),
+ new Person("Alfred", "London")
+ ));
+
+ PCollection<Person> results = Task.applyTransform(persons, citiesToCountriesView);
+
+ PAssert.that(results)
+ .containsInAnyOrder(
+ new Person("Henry", "Singapore", "Singapore"),
+ new Person("Jane", "San Francisco", "United States"),
+ new Person("Lee", "Beijing", "China"),
+ new Person("John", "Sydney", "Australia"),
+ new Person("Alfred", "London", "United Kingdom")
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Side Output/Side Output/src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java b/learning/katas/java/Core Transforms/Side Output/Side Output/src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java
new file mode 100644
index 0000000..a5bfc80
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/Side Output/src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.sideoutput;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Integer> numbers = pipeline.apply(Create.of(10, 50, 120, 20, 200, 0));
+
+ TupleTag<Integer> numBelow100Tag = new TupleTag<Integer>() {};
+ TupleTag<Integer> numAbove100Tag = new TupleTag<Integer>() {};
+
+ PCollectionTuple outputTuple = applyTransform(numbers, numBelow100Tag, numAbove100Tag);
+
+ outputTuple.get(numBelow100Tag).apply(Log.ofElements("Number <= 100: "));
+ outputTuple.get(numAbove100Tag).apply(Log.ofElements("Number > 100: "));
+
+ pipeline.run();
+ }
+
+ static PCollectionTuple applyTransform(
+ PCollection<Integer> numbers, TupleTag<Integer> numBelow100Tag,
+ TupleTag<Integer> numAbove100Tag) {
+
+ return numbers.apply(ParDo.of(new DoFn<Integer, Integer>() {
+
+ @ProcessElement
+ public void processElement(@Element Integer number, MultiOutputReceiver out) {
+ if (number <= 100) {
+ out.get(numBelow100Tag).output(number);
+ } else {
+ out.get(numAbove100Tag).output(number);
+ }
+ }
+
+ }).withOutputTags(numBelow100Tag, TupleTagList.of(numAbove100Tag)));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Core Transforms/Side Output/Side Output/task.html b/learning/katas/java/Core Transforms/Side Output/Side Output/task.html
new file mode 100644
index 0000000..d24f73d
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/Side Output/task.html
@@ -0,0 +1,44 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Side Output</h2>
+<p>
+ While ParDo always produces a main output PCollection (as the return value from apply), you can
+ also have your ParDo produce any number of additional output PCollections. If you choose to have
+ multiple outputs, your ParDo returns all of the output PCollections (including the main output)
+ bundled together.
+</p>
+<p>
+ <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.MultiOutputReceiver.html">
+ MultiOutputReceiver</a> and
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.SingleOutput.html#withOutputTags-org.apache.beam.sdk.values.TupleTag-org.apache.beam.sdk.values.TupleTagList-">
+ .withOutputTags</a> to output multiple tagged-outputs in a
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
+ ParDo.</a>
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs">
+ "Additional outputs"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Core Transforms/Side Output/Side Output/test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java b/learning/katas/java/Core Transforms/Side Output/Side Output/test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java
new file mode 100644
index 0000000..d35de2f
--- /dev/null
+++ b/learning/katas/java/Core Transforms/Side Output/Side Output/test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.coretransforms.sideoutput;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void sideOutput() {
+ PCollection<Integer> numbers = testPipeline.apply(Create.of(10, 50, 120, 20, 200, 0));
+
+ TupleTag<Integer> numBelow100Tag = new TupleTag<Integer>() {};
+ TupleTag<Integer> numAbove100Tag = new TupleTag<Integer>() {};
+
+ PCollectionTuple resultsTuple = Task.applyTransform(numbers, numBelow100Tag, numAbove100Tag);
+
+ PAssert.that(resultsTuple.get(numBelow100Tag))
+ .containsInAnyOrder(0, 10, 20, 50);
+
+ PAssert.that(resultsTuple.get(numAbove100Tag))
+ .containsInAnyOrder(120, 200);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Examples/Word Count/Word Count/src/Task.java b/learning/katas/java/Examples/Word Count/Word Count/src/Task.java
deleted file mode 100644
index fcfbdf4..0000000
--- a/learning/katas/java/Examples/Word Count/Word Count/src/Task.java
+++ /dev/null
@@ -1,74 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.util.Arrays;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Count;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.FlatMapElements;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.TypeDescriptors;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- String[] lines = {
- "apple orange grape banana apple banana",
- "banana orange banana papaya"
- };
-
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> wordCounts =
- pipeline.apply(Create.of(Arrays.asList(lines)));
-
- PCollection<String> output = applyTransform(wordCounts);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<String> applyTransform(PCollection<String> input) {
- return input
-
- .apply(FlatMapElements.into(TypeDescriptors.strings())
- .via(line -> Arrays.asList(line.split(" "))))
-
- .apply(Count.perElement())
-
- .apply(ParDo.of(new DoFn<KV<String, Long>, String>() {
-
- @ProcessElement
- public void processElement(
- @Element KV<String, Long> element, OutputReceiver<String> out) {
-
- out.output(element.getKey() + ":" + element.getValue());
- }
-
- }));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Examples/Word Count/Word Count/src/org/apache/beam/learning/katas/examples/wordcount/Task.java b/learning/katas/java/Examples/Word Count/Word Count/src/org/apache/beam/learning/katas/examples/wordcount/Task.java
new file mode 100644
index 0000000..f93e68d
--- /dev/null
+++ b/learning/katas/java/Examples/Word Count/Word Count/src/org/apache/beam/learning/katas/examples/wordcount/Task.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.examples.wordcount;
+
+import java.util.Arrays;
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.FlatMapElements;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptors;
+
+public class Task {
+
+ public static void main(String[] args) {
+ String[] lines = {
+ "apple orange grape banana apple banana",
+ "banana orange banana papaya"
+ };
+
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> wordCounts =
+ pipeline.apply(Create.of(Arrays.asList(lines)));
+
+ PCollection<String> output = applyTransform(wordCounts);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> applyTransform(PCollection<String> input) {
+ return input
+
+ .apply(FlatMapElements.into(TypeDescriptors.strings())
+ .via(line -> Arrays.asList(line.split(" "))))
+
+ .apply(Count.perElement())
+
+ .apply(ParDo.of(new DoFn<KV<String, Long>, String>() {
+
+ @ProcessElement
+ public void processElement(
+ @Element KV<String, Long> element, OutputReceiver<String> out) {
+
+ out.output(element.getKey() + ":" + element.getValue());
+ }
+
+ }));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Examples/Word Count/Word Count/task.html b/learning/katas/java/Examples/Word Count/Word Count/task.html
index 6ba9d1b..a963aab 100644
--- a/learning/katas/java/Examples/Word Count/Word Count/task.html
+++ b/learning/katas/java/Examples/Word Count/Word Count/task.html
@@ -18,15 +18,19 @@
<html>
<h2>Word Count Pipeline</h2>
-<p>This kata is to create a pipeline that counts the number of words.</p>
-<p>For this task, please output the count of each word in the following format:<br/>
- <pre>
- word:count
- ball:5
- book:3
- </pre>
+<p>
+ <b>Kata:</b> Create a pipeline that counts the number of words.
</p>
+<p>
+ Please output the count of each word in the following format:
+</p>
+<pre>
+ word:count
+ ball:5
+ book:3
+</pre>
<br>
-<br>
-<div class='hint'>Refer to your lessons above.</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to your katas above.
+</div>
+</html>
diff --git a/learning/katas/java/Examples/Word Count/Word Count/test/TaskTest.java b/learning/katas/java/Examples/Word Count/Word Count/test/TaskTest.java
deleted file mode 100644
index 06c057c..0000000
--- a/learning/katas/java/Examples/Word Count/Word Count/test/TaskTest.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest {
-
- @Rule
- public TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void parDo() {
- Create.Values<String> lines =
- Create.of(
- "apple orange grape banana apple banana",
- "banana orange banana papaya");
-
- PCollection<String> linesPColl = testPipeline.apply(lines);
-
- PCollection<String> results = Task.applyTransform(linesPColl);
-
- PAssert.that(results)
- .containsInAnyOrder(
- "apple:2",
- "banana:4",
- "grape:1",
- "orange:2",
- "papaya:1"
- );
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Examples/Word Count/Word Count/test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java b/learning/katas/java/Examples/Word Count/Word Count/test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java
new file mode 100644
index 0000000..cbde760
--- /dev/null
+++ b/learning/katas/java/Examples/Word Count/Word Count/test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.examples.wordcount;
+
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void parDo() {
+ Create.Values<String> lines =
+ Create.of(
+ "apple orange grape banana apple banana",
+ "banana orange banana papaya");
+
+ PCollection<String> linesPColl = testPipeline.apply(lines);
+
+ PCollection<String> results = Task.applyTransform(linesPColl);
+
+ PAssert.that(results)
+ .containsInAnyOrder(
+ "apple:2",
+ "banana:4",
+ "grape:1",
+ "orange:2",
+ "papaya:1"
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/IO/Built-in IOs/Built-in IOs/src/org/apache/beam/learning/katas/io/builtinios/Task.java b/learning/katas/java/IO/Built-in IOs/Built-in IOs/src/org/apache/beam/learning/katas/io/builtinios/Task.java
new file mode 100644
index 0000000..f3735b9
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/Built-in IOs/src/org/apache/beam/learning/katas/io/builtinios/Task.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.io.builtinios;
+
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+
+
+ pipeline.run();
+ }
+
+}
diff --git a/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html
new file mode 100644
index 0000000..fa59837
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html
@@ -0,0 +1,29 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Built-in I/Os</h2>
+<p>
+ Beam SDKs provide many out of the box I/O transforms that can be used to read from many
+ different sources and write to many different sinks.
+</p>
+<p>
+ See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O
+ Transforms</a> page for a list of the currently available I/O transforms.
+</p>
+</html>
\ No newline at end of file
diff --git a/learning/katas/java/IO/Built-in IOs/Built-in IOs/test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java b/learning/katas/java/IO/Built-in IOs/Built-in IOs/test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java
new file mode 100644
index 0000000..7f1be54
--- /dev/null
+++ b/learning/katas/java/IO/Built-in IOs/Built-in IOs/test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.io.builtinios;
+
+public class TaskTest {
+}
diff --git a/learning/katas/java/IO/TextIO/TextIO Read/countries.txt b/learning/katas/java/IO/TextIO/TextIO Read/countries.txt
new file mode 100644
index 0000000..9d6848f
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/TextIO Read/countries.txt
@@ -0,0 +1,10 @@
+Singapore
+United States
+Australia
+England
+France
+China
+Indonesia
+Mexico
+Germany
+Japan
diff --git a/learning/katas/java/IO/TextIO/TextIO Read/src/org/apache/beam/learning/katas/io/textio/read/Task.java b/learning/katas/java/IO/TextIO/TextIO Read/src/org/apache/beam/learning/katas/io/textio/read/Task.java
new file mode 100644
index 0000000..bdc0839
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/TextIO Read/src/org/apache/beam/learning/katas/io/textio/read/Task.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.io.textio.read;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ private static final String FILE_PATH = "IO/TextIO/TextIO Read/countries.txt";
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> countries =
+ pipeline.apply("Read Countries", TextIO.read().from(FILE_PATH));
+
+ PCollection<String> output = applyTransform(countries);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> applyTransform(PCollection<String> input) {
+ return input.apply(MapElements.into(strings()).via(String::toUpperCase));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/IO/TextIO/TextIO Read/task.html b/learning/katas/java/IO/TextIO/TextIO Read/task.html
new file mode 100644
index 0000000..1ebad84
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/TextIO Read/task.html
@@ -0,0 +1,47 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>TextIO Read</h2>
+<p>
+ When you create a pipeline, you often need to read data from some external source, such as a file
+ or a database. Likewise, you may want your pipeline to output its result data to an external
+ storage system. Beam provides read and write transforms for a number of common data storage types.
+ If you want your pipeline to read from or write to a data storage format that isn’t supported by
+ the built-in transforms, you can implement your own read and write transforms.
+</p>
+<p>
+ To read a PCollection from one or more text files, use TextIO.read() to instantiate a transform
+ and use TextIO.Read.from(String) to specify the path of the file(s) to be read.
+</p>
+<p>
+ <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html">
+ TextIO</a> and its corresponding
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html#read--">
+ TextIO.read()</a> method.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data">
+ "Reading input data"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/IO/TextIO/TextIO Read/test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java b/learning/katas/java/IO/TextIO/TextIO Read/test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java
new file mode 100644
index 0000000..0bcdcfa
--- /dev/null
+++ b/learning/katas/java/IO/TextIO/TextIO Read/test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.io.textio.read;
+
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void textIO() {
+ PCollection<String> countries =
+ testPipeline.apply(TextIO.read().from("countries.txt"));
+
+ PCollection<String> results = Task.applyTransform(countries);
+
+ PAssert.that(results)
+ .containsInAnyOrder(
+ "AUSTRALIA",
+ "CHINA",
+ "ENGLAND",
+ "FRANCE",
+ "GERMANY",
+ "INDONESIA",
+ "JAPAN",
+ "MEXICO",
+ "SINGAPORE",
+ "UNITED STATES"
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/src/Task.java b/learning/katas/java/Introduction/Hello Beam/Hello Beam/src/Task.java
deleted file mode 100644
index bed1a67..0000000
--- a/learning/katas/java/Introduction/Hello Beam/Hello Beam/src/Task.java
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.values.PCollection;
-import util.Log;
-
-class Task {
-
- public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
- Pipeline pipeline = Pipeline.create(options);
-
- PCollection<String> output = setupPipeline(pipeline);
-
- output.apply(Log.ofElements());
-
- pipeline.run();
- }
-
- static PCollection<String> setupPipeline(Pipeline pipeline) {
- return pipeline.apply(Create.of("Hello Beam"));
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/src/org/apache/beam/learning/katas/intro/hello/Task.java b/learning/katas/java/Introduction/Hello Beam/Hello Beam/src/org/apache/beam/learning/katas/intro/hello/Task.java
new file mode 100644
index 0000000..9d85767
--- /dev/null
+++ b/learning/katas/java/Introduction/Hello Beam/Hello Beam/src/org/apache/beam/learning/katas/intro/hello/Task.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.intro.hello;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> output = setupPipeline(pipeline);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<String> setupPipeline(Pipeline pipeline) {
+ return pipeline.apply(Create.of("Hello Beam"));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/task.html b/learning/katas/java/Introduction/Hello Beam/Hello Beam/task.html
index 5d6928b..28e579b 100644
--- a/learning/katas/java/Introduction/Hello Beam/Hello Beam/task.html
+++ b/learning/katas/java/Introduction/Hello Beam/Hello Beam/task.html
@@ -18,10 +18,36 @@
<html>
<h2>Hello Beam Pipeline</h2>
-<p>This kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam"</p>
+<p>
+ Apache Beam is an open source, unified model for defining both batch and streaming data-parallel
+ processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the
+ pipeline. The pipeline is then executed by one of Beam’s supported distributed processing
+ back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
+</p>
+<p>
+ Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the
+ problem can be decomposed into many smaller bundles of data that can be processed independently
+ and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data
+ integration. These tasks are useful for moving data between different storage media and data
+ sources, transforming data into a more desirable format, or loading data onto a new system.
+</p>
+<p>
+ To learn more about Apache Beam, refer to
+ <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>.
+</p>
+<p>
+ <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element
+ "Hello Beam".
+</p>
<br>
-<br>
-<div class='hint'>Hardcoded input can be created using
- <a href="https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/transforms/Create.html">Create</a>
+<div class="hint">
+ Hardcoded input can be created using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Create.html">
+ Create</a>.
</div>
-</html>
\ No newline at end of file
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory">
+ "Creating a PCollection from in-memory data"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/test/TaskTest.java b/learning/katas/java/Introduction/Hello Beam/Hello Beam/test/TaskTest.java
deleted file mode 100644
index 6dd0e0b..0000000
--- a/learning/katas/java/Introduction/Hello Beam/Hello Beam/test/TaskTest.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import java.io.Serializable;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.values.PCollection;
-import org.junit.Rule;
-import org.junit.Test;
-
-public class TaskTest implements Serializable {
-
- @Rule
- public final transient TestPipeline testPipeline = TestPipeline.create();
-
- @Test
- public void helloWorld() {
- PCollection<String> results = Task.setupPipeline(testPipeline);
-
- PAssert.that(results)
- .containsInAnyOrder("Hello Beam");
-
- testPipeline.run().waitUntilFinish();
- }
-
-}
\ No newline at end of file
diff --git a/learning/katas/java/Introduction/Hello Beam/Hello Beam/test/org/apache/beam/learning/katas/intro/hello/TaskTest.java b/learning/katas/java/Introduction/Hello Beam/Hello Beam/test/org/apache/beam/learning/katas/intro/hello/TaskTest.java
new file mode 100644
index 0000000..79d14ef
--- /dev/null
+++ b/learning/katas/java/Introduction/Hello Beam/Hello Beam/test/org/apache/beam/learning/katas/intro/hello/TaskTest.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.intro.hello;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void helloWorld() {
+ PCollection<String> results = Task.setupPipeline(testPipeline);
+
+ PAssert.that(results)
+ .containsInAnyOrder("Hello Beam");
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/README.md b/learning/katas/java/README.md
new file mode 100644
index 0000000..ba22699
--- /dev/null
+++ b/learning/katas/java/README.md
@@ -0,0 +1,31 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+### How to Setup
+Please follow the below steps in order to setup the project properly:
+* Using IntelliJ Education (or IntelliJ with EduTools plugin), select "Open" and select this
+directory (learning/katas/java)
+* "Import Gradle project" when prompted, and configure the Gradle setup
+* Wait for Gradle build to finish
+* Open "Project Structure" and setup the project SDK (e.g. JDK 8)
+* Open the "Project" tool window, and select the "Course" view
+* Your project is ready
+
+For further instructions on how the IntelliJ Education works, you can refer
+[here](https://www.jetbrains.com/help/education/educator-start-guide.html?section=Java).
diff --git a/learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java b/learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java
new file mode 100644
index 0000000..24938cb
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.earlytriggers;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+
+public class GenerateEvent extends PTransform<PBegin, PCollection<String>> {
+
+ static GenerateEvent everySecond() {
+ return new GenerateEvent();
+ }
+
+ public PCollection<String> expand(PBegin input) {
+ return input
+ .apply(GenerateSequence.from(1).withRate(1, Duration.standardSeconds(1)))
+ .apply(MapElements.into(strings()).via(num -> "event"));
+ }
+
+}
diff --git a/learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java b/learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java
new file mode 100644
index 0000000..fb4a3bc
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.earlytriggers;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> events =
+ pipeline.apply(GenerateEvent.everySecond());
+
+ PCollection<Long> output = applyTransform(events);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Long> applyTransform(PCollection<String> events) {
+ return events
+ .apply(
+ Window.<String>into(FixedWindows.of(Duration.standardDays(1)))
+ .triggering(
+ AfterWatermark.pastEndOfWindow()
+ .withEarlyFirings(
+ AfterProcessingTime.pastFirstElementInPane()))
+ .withAllowedLateness(Duration.ZERO)
+ .discardingFiredPanes())
+
+ .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Triggers/Early Triggers/Early Triggers/task.html b/learning/katas/java/Triggers/Early Triggers/Early Triggers/task.html
new file mode 100644
index 0000000..6a7f1cb
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/Early Triggers/task.html
@@ -0,0 +1,59 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Early Triggers</h2>
+<p>
+ Triggers allow Beam to emit early results, before all the data in a given window has arrived.
+ For example, emitting after a certain amount of time elapses, or after a certain number of
+ elements arrives.
+</p>
+<p>
+ <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day
+ duration, please implement an early trigger that emits the number of events count immediately
+ after new element is processed.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-">
+ withEarlyFirings</a> to set early firing triggers.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html">
+ FixedWindows</a> with 1-day duration using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--">
+ AfterWatermark.pastEndOfWindow()</a> trigger.
+</div>
+<div class="hint">
+ Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-">
+ allowed lateness</a> to 0 with
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--">
+ discarding accumulation mode</a>.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-">
+ Combine.globally</a> and
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--">
+ Count.combineFn</a> to calculate the count of events.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers">
+ "Event time triggers"</a> section for more information.
+</div>
+</html>
\ No newline at end of file
diff --git a/learning/katas/java/Triggers/Early Triggers/Early Triggers/test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java b/learning/katas/java/Triggers/Early Triggers/Early Triggers/test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java
new file mode 100644
index 0000000..f52aa34
--- /dev/null
+++ b/learning/katas/java/Triggers/Early Triggers/Early Triggers/test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.earlytriggers;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void eventTimeTrigger() {
+ TestStream<String> testStream =
+ TestStream.create(SerializableCoder.of(String.class))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(0)))
+ .advanceProcessingTime(Duration.standardSeconds(1))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(1)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(2)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(3)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(4)))
+ .advanceProcessingTime(Duration.standardSeconds(1))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(5)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(6)))
+ .advanceWatermarkToInfinity();
+
+ PCollection<String> eventsPColl = testPipeline.apply(testStream);
+
+ PCollection<Long> results = Task.applyTransform(eventsPColl);
+
+ PAssert.that(results)
+ .inEarlyPane(new IntervalWindow(Instant.EPOCH, Instant.parse("1970-01-02T00:00:00+00:00")))
+ .containsInAnyOrder(1L, 4L)
+ .inFinalPane(new IntervalWindow(Instant.EPOCH, Instant.parse("1970-01-02T00:00:00+00:00")))
+ .containsInAnyOrder(2L);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java
new file mode 100644
index 0000000..002179b
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.eventtimetriggers;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+
+public class GenerateEvent extends PTransform<PBegin, PCollection<String>> {
+
+ static GenerateEvent everySecond() {
+ return new GenerateEvent();
+ }
+
+ public PCollection<String> expand(PBegin input) {
+ return input
+ .apply(GenerateSequence.from(1).withRate(1, Duration.standardSeconds(1)))
+ .apply(MapElements.into(strings()).via(num -> "event"));
+ }
+
+}
diff --git a/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java
new file mode 100644
index 0000000..d7a45af
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.eventtimetriggers;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> events =
+ pipeline.apply(GenerateEvent.everySecond());
+
+ PCollection<Long> output = applyTransform(events);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Long> applyTransform(PCollection<String> events) {
+ return events
+ .apply(
+ Window.<String>into(FixedWindows.of(Duration.standardSeconds(5)))
+ .triggering(AfterWatermark.pastEndOfWindow())
+ .withAllowedLateness(Duration.ZERO)
+ .discardingFiredPanes())
+
+ .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task.html b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task.html
new file mode 100644
index 0000000..5a124aa
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task.html
@@ -0,0 +1,78 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Event Time Triggers</h2>
+<p>
+ When collecting and grouping data into windows, Beam uses triggers to determine when to emit the
+ aggregated results of each window (referred to as a pane). If you use Beam’s default windowing
+ configuration and default trigger, Beam outputs the aggregated result when it estimates all data
+ has arrived, and discards all subsequent data for that window.
+</p>
+<p>
+ You can set triggers for your PCollections to change this default behavior. Beam provides a
+ number of pre-built triggers that you can set:
+</p>
+<div>
+ <ul>
+ <li>Event time triggers</li>
+ <li>Processing time triggers</li>
+ <li>Data-driven triggers</li>
+ <li>Composite triggers</li>
+ </ul>
+</div>
+<p>
+ Event time triggers operate on the event time, as indicated by the timestamp on each data
+ element. Beam’s default trigger is event time-based.
+</p>
+<p>
+ The AfterWatermark trigger operates on event time. The AfterWatermark trigger emits the contents
+ of a window after the watermark passes the end of the window, based on the timestamps attached
+ to the data elements. The watermark is a global progress metric, and is Beam’s notion of input
+ completeness within your pipeline at any given point. AfterWatermark.pastEndOfWindow() only fires
+ when the watermark passes the end of the window.
+</p>
+<p>
+ <b>Kata:</b> Given that events are being generated every second, please implement a trigger that
+ emits the number of events count within a fixed window of 5-second duration.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html">
+ FixedWindows</a> with 5-second duration using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--">
+ AfterWatermark.pastEndOfWindow()</a> trigger.
+</div>
+<div class="hint">
+ Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-">
+ allowed lateness</a> to 0 with
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#discardingFiredPanes--">
+ discarding accumulation mode</a>.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-">
+ Combine.globally</a> and
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--">
+ Count.combineFn</a> to calculate the count of events.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers">
+ "Event time triggers"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java
new file mode 100644
index 0000000..95e4a48
--- /dev/null
+++ b/learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.eventtimetriggers;
+
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void eventTimeTrigger() {
+ TestStream<String> testStream =
+ TestStream.create(SerializableCoder.of(String.class))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:01+00:00")))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:02+00:00")))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:03+00:00")))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:04+00:00")))
+ .advanceWatermarkTo(Instant.parse("2019-06-01T00:00:05+00:00"))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:05+00:00")))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:06+00:00")))
+ .addElements(TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:07+00:00")))
+ .advanceWatermarkTo(Instant.parse("2019-06-01T00:00:10+00:00"))
+ .advanceWatermarkToInfinity();
+
+ PCollection<String> eventsPColl = testPipeline.apply(testStream);
+
+ PCollection<Long> results = Task.applyTransform(eventsPColl);
+
+ PAssert.that(results)
+ .inWindow(createIntervalWindow("2019-06-01T00:00:00+00:00", "2019-06-01T00:00:05+00:00"))
+ .containsInAnyOrder(5L)
+ .inWindow(createIntervalWindow("2019-06-01T00:00:05+00:00", "2019-06-01T00:00:10+00:00"))
+ .containsInAnyOrder(3L);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+ private IntervalWindow createIntervalWindow(String startStr, String endStr) {
+ return new IntervalWindow(Instant.parse(startStr), Instant.parse(endStr));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java
new file mode 100644
index 0000000..39e16fe
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.windowaccummode;
+
+import static org.apache.beam.sdk.values.TypeDescriptors.strings;
+
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+
+public class GenerateEvent extends PTransform<PBegin, PCollection<String>> {
+
+ static GenerateEvent everySecond() {
+ return new GenerateEvent();
+ }
+
+ public PCollection<String> expand(PBegin input) {
+ return input
+ .apply(GenerateSequence.from(1).withRate(1, Duration.standardSeconds(1)))
+ .apply(MapElements.into(strings()).via(num -> "event"));
+ }
+
+}
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java
new file mode 100644
index 0000000..5452e2a
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.windowaccummode;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> events =
+ pipeline.apply(GenerateEvent.everySecond());
+
+ PCollection<Long> output = applyTransform(events);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Long> applyTransform(PCollection<String> events) {
+ return events
+ .apply(
+ Window.<String>into(FixedWindows.of(Duration.standardDays(1)))
+ .triggering(
+ AfterWatermark.pastEndOfWindow()
+ .withEarlyFirings(
+ AfterProcessingTime.pastFirstElementInPane()))
+ .withAllowedLateness(Duration.ZERO)
+ .accumulatingFiredPanes())
+
+ .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task.html b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task.html
new file mode 100644
index 0000000..f40784e
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task.html
@@ -0,0 +1,63 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Window Accumulation Mode</h2>
+<p>
+ When you specify a trigger, you must also set the the window’s accumulation mode. When a trigger
+ fires, it emits the current contents of the window as a pane. Since a trigger can fire multiple
+ times, the accumulation mode determines whether the system accumulates the window panes as the
+ trigger fires, or discards them.
+</p>
+<p>
+ <b>Kata:</b> Given that events are being generated every second and a fixed window of 1-day
+ duration, please implement an early trigger that emits the number of events count immediately
+ after new element is processed in accumulating mode.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/windowing/Window.html#accumulatingFiredPanes--">
+ accumulatingFiredPanes()</a> to set a window to accumulate the panes that are produced when the
+ trigger fires.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.AfterWatermarkEarlyAndLate.html#withEarlyFirings-org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger-">
+ withEarlyFirings</a> to set early firing triggers.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html">
+ FixedWindows</a> with 1-day duration using
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/AfterWatermark.html#pastEndOfWindow--">
+ AfterWatermark.pastEndOfWindow()</a> trigger.
+</div>
+<div class="hint">
+ Set the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/Window.html#withAllowedLateness-org.joda.time.Duration-">
+ allowed lateness</a> to 0.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Combine.html#globally-org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn-">
+ Combine.globally</a> and
+ <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Count.html#combineFn--">
+ Count.combineFn</a> to calculate the count of events.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#event-time-triggers">
+ "Event time triggers"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java
new file mode 100644
index 0000000..1f9ad09
--- /dev/null
+++ b/learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.triggers.windowaccummode;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void eventTimeTrigger() {
+ TestStream<String> testStream =
+ TestStream.create(SerializableCoder.of(String.class))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(0)))
+ .advanceProcessingTime(Duration.standardSeconds(1))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(1)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(2)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(3)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(4)))
+ .advanceProcessingTime(Duration.standardSeconds(1))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(5)))
+ .addElements(TimestampedValue.of("event", Instant.ofEpochSecond(6)))
+ .advanceWatermarkToInfinity();
+
+ PCollection<String> eventsPColl = testPipeline.apply(testStream);
+
+ PCollection<Long> results = Task.applyTransform(eventsPColl);
+
+ PAssert.that(results)
+ .inEarlyPane(new IntervalWindow(Instant.EPOCH, Instant.parse("1970-01-02T00:00:00+00:00")))
+ .containsInAnyOrder(1L, 5L)
+ .inFinalPane(new IntervalWindow(Instant.EPOCH, Instant.parse("1970-01-02T00:00:00+00:00")))
+ .containsInAnyOrder(7L);
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
diff --git a/learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java b/learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java
new file mode 100644
index 0000000..2cd1f27
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.addingtimestamp.pardo;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.joda.time.DateTime;
+
+public class Event implements Serializable {
+
+ private String id;
+ private String event;
+ private DateTime date;
+
+ public Event(String id, String event, DateTime date) {
+ this.id = id;
+ this.event = event;
+ this.date = date;
+ }
+
+ public String getId() {
+ return id;
+ }
+
+ public void setId(String id) {
+ this.id = id;
+ }
+
+ public String getEvent() {
+ return event;
+ }
+
+ public void setEvent(String event) {
+ this.event = event;
+ }
+
+ public DateTime getDate() {
+ return date;
+ }
+
+ public void setDate(DateTime date) {
+ this.date = date;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ Event event1 = (Event) o;
+
+ return id.equals(event1.id) &&
+ event.equals(event1.event) &&
+ date.equals(event1.date);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(id, event, date);
+ }
+
+ @Override
+ public String toString() {
+ return "Event{" +
+ "id='" + id + '\'' +
+ ", event='" + event + '\'' +
+ ", date=" + date +
+ '}';
+ }
+
+}
diff --git a/learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java b/learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java
new file mode 100644
index 0000000..dc1742e
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.addingtimestamp.pardo;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.DateTime;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Event> events =
+ pipeline.apply(
+ Create.of(
+ new Event("1", "book-order", DateTime.parse("2019-06-01T00:00:00+00:00")),
+ new Event("2", "pencil-order", DateTime.parse("2019-06-02T00:00:00+00:00")),
+ new Event("3", "paper-order", DateTime.parse("2019-06-03T00:00:00+00:00")),
+ new Event("4", "pencil-order", DateTime.parse("2019-06-04T00:00:00+00:00")),
+ new Event("5", "book-order", DateTime.parse("2019-06-05T00:00:00+00:00"))
+ )
+ );
+
+ PCollection<Event> output = applyTransform(events);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Event> applyTransform(PCollection<Event> events) {
+ return events.apply(ParDo.of(new DoFn<Event, Event>() {
+
+ @ProcessElement
+ public void processElement(@Element Event event, OutputReceiver<Event> out) {
+ out.outputWithTimestamp(event, event.getDate().toInstant());
+ }
+
+ }));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Adding Timestamp/ParDo/task.html b/learning/katas/java/Windowing/Adding Timestamp/ParDo/task.html
new file mode 100644
index 0000000..403fc11
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/ParDo/task.html
@@ -0,0 +1,48 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Adding Timestamp - ParDo</h2>
+<p>
+ Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need
+ timestamps, you must add them to your PCollection’s elements.
+</p>
+<p>
+ You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that
+ outputs new elements with timestamps that you set.
+</p>
+<p>
+ <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/ParDo.html">
+ ParDo</a>
+ with <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.html">
+ DoFn</a>.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.OutputReceiver.html#outputWithTimestamp-T-org.joda.time.Instant-">
+ OutputReceiver.outputWithTimestamp</a> method to assign timestamp to the element.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">
+ "Adding timestamps to a PCollection’s elements"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/java/Windowing/Adding Timestamp/ParDo/test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java b/learning/katas/java/Windowing/Adding Timestamp/ParDo/test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java
new file mode 100644
index 0000000..8e8f0b2
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/ParDo/test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.addingtimestamp.pardo;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.DateTime;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void windowing_addingTimeStamp_parDo() {
+ List<Event> events = Arrays.asList(
+ new Event("1", "book-order", DateTime.parse("2019-06-01T00:00:00+00:00")),
+ new Event("2", "pencil-order", DateTime.parse("2019-06-02T00:00:00+00:00")),
+ new Event("3", "paper-order", DateTime.parse("2019-06-03T00:00:00+00:00")),
+ new Event("4", "pencil-order", DateTime.parse("2019-06-04T00:00:00+00:00")),
+ new Event("5", "book-order", DateTime.parse("2019-06-05T00:00:00+00:00"))
+ );
+
+ PCollection<Event> eventsPColl = testPipeline.apply(Create.of(events));
+
+ PCollection<Event> results = Task.applyTransform(eventsPColl);
+
+ PCollection<KV<Event, Instant>> timestampedResults =
+ results.apply("KV<Event, Instant>",
+ ParDo.of(new DoFn<Event, KV<Event, Instant>>() {
+
+ @ProcessElement
+ public void processElement(@Element Event event, ProcessContext context,
+ OutputReceiver<KV<Event, Instant>> out) {
+
+ out.output(KV.of(event, context.timestamp()));
+ }
+
+ })
+ );
+
+ PAssert.that(results)
+ .containsInAnyOrder(events);
+
+ PAssert.that(timestampedResults)
+ .containsInAnyOrder(
+ KV.of(events.get(0), events.get(0).getDate().toInstant()),
+ KV.of(events.get(1), events.get(1).getDate().toInstant()),
+ KV.of(events.get(2), events.get(2).getDate().toInstant()),
+ KV.of(events.get(3), events.get(3).getDate().toInstant()),
+ KV.of(events.get(4), events.get(4).getDate().toInstant())
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java
new file mode 100644
index 0000000..3fa5f3c
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.addingtimestamp.withtimestamps;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.joda.time.DateTime;
+
+public class Event implements Serializable {
+
+ private String id;
+ private String event;
+ private DateTime date;
+
+ public Event(String id, String event, DateTime date) {
+ this.id = id;
+ this.event = event;
+ this.date = date;
+ }
+
+ public String getId() {
+ return id;
+ }
+
+ public void setId(String id) {
+ this.id = id;
+ }
+
+ public String getEvent() {
+ return event;
+ }
+
+ public void setEvent(String event) {
+ this.event = event;
+ }
+
+ public DateTime getDate() {
+ return date;
+ }
+
+ public void setDate(DateTime date) {
+ this.date = date;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ Event event1 = (Event) o;
+
+ return id.equals(event1.id) &&
+ event.equals(event1.event) &&
+ date.equals(event1.date);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(id, event, date);
+ }
+
+ @Override
+ public String toString() {
+ return "Event{" +
+ "id='" + id + '\'' +
+ ", event='" + event + '\'' +
+ ", date=" + date +
+ '}';
+ }
+
+}
diff --git a/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java
new file mode 100644
index 0000000..f82b1eb
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.addingtimestamp.withtimestamps;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.WithTimestamps;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.DateTime;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<Event> events =
+ pipeline.apply(
+ Create.of(
+ new Event("1", "book-order", DateTime.parse("2019-06-01T00:00:00+00:00")),
+ new Event("2", "pencil-order", DateTime.parse("2019-06-02T00:00:00+00:00")),
+ new Event("3", "paper-order", DateTime.parse("2019-06-03T00:00:00+00:00")),
+ new Event("4", "pencil-order", DateTime.parse("2019-06-04T00:00:00+00:00")),
+ new Event("5", "book-order", DateTime.parse("2019-06-05T00:00:00+00:00"))
+ )
+ );
+
+ PCollection<Event> output = applyTransform(events);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<Event> applyTransform(PCollection<Event> events) {
+ return events.apply(WithTimestamps.of(event -> event.getDate().toInstant()));
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task.html b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task.html
new file mode 100644
index 0000000..bd49a74
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task.html
@@ -0,0 +1,42 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Adding Timestamp - WithTimestamps</h2>
+<p>
+ Bounded sources (such as a file from TextIO) do not provide timestamps for elements. If you need
+ timestamps, you must add them to your PCollection’s elements.
+</p>
+<p>
+ You can assign new timestamps to the elements of a PCollection by applying a ParDo transform that
+ outputs new elements with timestamps that you set.
+</p>
+<p>
+ <b>Kata:</b> Please assign each element a timestamp based on the the <code>Event.getDate()</code>.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/WithTimestamps.html">
+ WithTimestamps</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">
+ "Adding timestamps to a PCollection’s elements"</a> section for more information.
+</div>
+</html>
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java
new file mode 100644
index 0000000..8bbeaad
--- /dev/null
+++ b/learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.addingtimestamp.withtimestamps;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.DateTime;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void windowing_addingTimeStamp_withTimestamps() {
+ List<Event> events = Arrays.asList(
+ new Event("1", "book-order", DateTime.parse("2019-06-01T00:00:00+00:00")),
+ new Event("2", "pencil-order", DateTime.parse("2019-06-02T00:00:00+00:00")),
+ new Event("3", "paper-order", DateTime.parse("2019-06-03T00:00:00+00:00")),
+ new Event("4", "pencil-order", DateTime.parse("2019-06-04T00:00:00+00:00")),
+ new Event("5", "book-order", DateTime.parse("2019-06-05T00:00:00+00:00"))
+ );
+
+ PCollection<Event> eventsPColl = testPipeline.apply(Create.of(events));
+
+ PCollection<Event> results = Task.applyTransform(eventsPColl);
+
+ PCollection<KV<Event, Instant>> timestampedResults =
+ results.apply("KV<Event, Instant>",
+ ParDo.of(new DoFn<Event, KV<Event, Instant>>() {
+
+ @ProcessElement
+ public void processElement(@Element Event event, ProcessContext context,
+ OutputReceiver<KV<Event, Instant>> out) {
+
+ out.output(KV.of(event, context.timestamp()));
+ }
+
+ })
+ );
+
+ PAssert.that(results)
+ .containsInAnyOrder(events);
+
+ PAssert.that(timestampedResults)
+ .containsInAnyOrder(
+ KV.of(events.get(0), events.get(0).getDate().toInstant()),
+ KV.of(events.get(1), events.get(1).getDate().toInstant()),
+ KV.of(events.get(2), events.get(2).getDate().toInstant()),
+ KV.of(events.get(3), events.get(3).getDate().toInstant()),
+ KV.of(events.get(4), events.get(4).getDate().toInstant())
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java
new file mode 100644
index 0000000..26dc1db
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.fixedwindow;
+
+import org.apache.beam.learning.katas.util.Log;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+public class Task {
+
+ public static void main(String[] args) {
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
+
+ PCollection<String> events =
+ pipeline.apply(
+ Create.timestamped(
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-05T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-05T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-08T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-08T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-08T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-10T00:00:00+00:00"))
+ )
+ );
+
+ PCollection<KV<String, Long>> output = applyTransform(events);
+
+ output.apply(Log.ofElements());
+
+ pipeline.run();
+ }
+
+ static PCollection<KV<String, Long>> applyTransform(PCollection<String> events) {
+ return events
+ .apply(Window.into(FixedWindows.of(Duration.standardDays(1))))
+ .apply(Count.perElement());
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task.html b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task.html
new file mode 100644
index 0000000..7f010c7
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task.html
@@ -0,0 +1,61 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Fixed Time Window</h2>
+<p>
+ Windowing subdivides a PCollection according to the timestamps of its individual elements.
+ Transforms that aggregate multiple elements, such as GroupByKey and Combine, work implicitly on
+ a per-window basis — they process each PCollection as a succession of multiple, finite windows,
+ though the entire collection itself may be of unbounded size.
+</p>
+<p>
+ In the Beam model, any PCollection (including unbounded PCollections) can be subdivided into
+ logical windows. Each element in a PCollection is assigned to one or more windows according to
+ the PCollection’s windowing function, and each individual window contains a finite number of
+ elements. Grouping transforms then consider each PCollection’s elements on a per-window basis.
+ GroupByKey, for example, implicitly groups the elements of a PCollection by key and window.
+</p>
+<div>
+ Beam provides several windowing functions, including:
+ <ul>
+ <li>Fixed Time Windows</li>
+ <li>Sliding Time Windows</li>
+ <li>Per-Session Windows</li>
+ <li>Single Global Window</li>
+ </ul>
+</div>
+<p>
+ The simplest form of windowing is using fixed time windows. A fixed time window represents a
+ consistent duration, non overlapping time interval in the data stream.
+</p>
+<p>
+ <b>Kata:</b> Please count the number of events that happened based on fixed window with
+ 1-day duration.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/windowing/FixedWindows.html">
+ FixedWindows</a> with 1-day duration.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#fixed-time-windows">
+ "Fixed time windows"</a> section for more information.
+</div>
+</html>
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java
new file mode 100644
index 0000000..5d3a732
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.fixedwindow;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+
+public class TaskTest implements Serializable {
+
+ @Rule
+ public final transient TestPipeline testPipeline = TestPipeline.create();
+
+ @Test
+ public void fixedWindow() {
+ PCollection<String> eventsPColl =
+ testPipeline.apply(
+ Create.timestamped(
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-01T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-05T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-05T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-08T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-08T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-08T00:00:00+00:00")),
+ TimestampedValue.of("event", Instant.parse("2019-06-10T00:00:00+00:00"))
+ )
+ );
+
+ PCollection<KV<String, Long>> results = Task.applyTransform(eventsPColl);
+
+ PCollection<WindowedEvent> windowedResults =
+ results.apply("WindowedEvent",
+ ParDo.of(new DoFn<KV<String, Long>, WindowedEvent>() {
+
+ @ProcessElement
+ public void processElement(@Element KV<String, Long> element,
+ BoundedWindow window, OutputReceiver<WindowedEvent> out) {
+
+ out.output(
+ new WindowedEvent(element.getKey(), element.getValue(), window.toString()));
+ }
+
+ })
+ );
+
+ PAssert.that(windowedResults)
+ .containsInAnyOrder(
+ new WindowedEvent("event", 4L, "[2019-06-01T00:00:00.000Z..2019-06-02T00:00:00.000Z)"),
+ new WindowedEvent("event", 2L, "[2019-06-05T00:00:00.000Z..2019-06-06T00:00:00.000Z)"),
+ new WindowedEvent("event", 3L, "[2019-06-08T00:00:00.000Z..2019-06-09T00:00:00.000Z)"),
+ new WindowedEvent("event", 1L, "[2019-06-10T00:00:00.000Z..2019-06-11T00:00:00.000Z)")
+ );
+
+ testPipeline.run().waitUntilFinish();
+ }
+
+}
\ No newline at end of file
diff --git a/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java
new file mode 100644
index 0000000..cc51496
--- /dev/null
+++ b/learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.windowing.fixedwindow;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+public class WindowedEvent implements Serializable {
+
+ private String event;
+ private Long count;
+ private String window;
+
+ public WindowedEvent(String event, Long count, String window) {
+ this.event = event;
+ this.count = count;
+ this.window = window;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ WindowedEvent that = (WindowedEvent) o;
+ return event.equals(that.event) &&
+ count.equals(that.count) &&
+ window.equals(that.window);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(event, count, window);
+ }
+
+ @Override
+ public String toString() {
+ return "WindowedEvent{" +
+ "event='" + event + '\'' +
+ ", count=" + count +
+ ", window='" + window + '\'' +
+ '}';
+ }
+
+}
diff --git a/learning/katas/java/build.gradle b/learning/katas/java/build.gradle
index 35ccb2f..4c962ec 100644
--- a/learning/katas/java/build.gradle
+++ b/learning/katas/java/build.gradle
@@ -18,12 +18,13 @@
buildscript {
ext {
- beamVersion = '2.11.0'
- guavaVersion = '27.0-jre'
- jodaTimeVersion = '2.10.1'
- slf4jVersion = '1.7.25'
+ beamVersion = '2.13.0'
+ guavaVersion = '27.1-jre'
+ jodaTimeVersion = '2.10.3'
+ slf4jVersion = '1.7.26'
+ log4jSlf4jImpl = '2.11.2'
- assertjVersion = '3.11.1'
+ assertjVersion = '3.12.2'
hamcrestVersion = '1.3'
junitVersion = '4.12'
}
@@ -57,7 +58,7 @@
compile "joda-time:joda-time:$jodaTimeVersion"
compile "org.slf4j:slf4j-api:$slf4jVersion"
- compile "org.slf4j:slf4j-jdk14:$slf4jVersion"
+ compile "org.apache.logging.log4j:log4j-slf4j-impl:$log4jSlf4jImpl"
compile "com.google.guava:guava:$guavaVersion"
testCompile "junit:junit:$junitVersion"
@@ -69,6 +70,7 @@
sourceSets {
main {
java.srcDir 'src'
+ resources.srcDir 'resources'
}
test {
java.srcDir 'test'
diff --git a/learning/katas/java/util/resources/log4j2.xml b/learning/katas/java/util/resources/log4j2.xml
new file mode 100644
index 0000000..e029836
--- /dev/null
+++ b/learning/katas/java/util/resources/log4j2.xml
@@ -0,0 +1,31 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<Configuration>
+ <Appenders>
+ <Console name="Console" target="SYSTEM_OUT">
+ <PatternLayout pattern="%d{HH:mm:ss.SSS} %-5level : %msg%n" />
+ </Console>
+ </Appenders>
+ <Loggers>
+ <Root level="info">
+ <AppenderRef ref="Console" />
+ </Root>
+ </Loggers>
+</Configuration>
\ No newline at end of file
diff --git a/learning/katas/java/util/src/org/apache/beam/learning/katas/util/Log.java b/learning/katas/java/util/src/org/apache/beam/learning/katas/util/Log.java
new file mode 100644
index 0000000..f389718
--- /dev/null
+++ b/learning/katas/java/util/src/org/apache/beam/learning/katas/util/Log.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.util;
+
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.values.PCollection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class Log {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(Log.class);
+
+ private Log() {
+ }
+
+ public static <T> PTransform<PCollection<T>, PCollection<T>> ofElements() {
+ return new LoggingTransform<>();
+ }
+
+ public static <T> PTransform<PCollection<T>, PCollection<T>> ofElements(String prefix) {
+ return new LoggingTransform<>(prefix);
+ }
+
+ private static class LoggingTransform<T> extends PTransform<PCollection<T>, PCollection<T>> {
+
+ private String prefix;
+
+ private LoggingTransform() {
+ prefix = "";
+ }
+
+ private LoggingTransform(String prefix) {
+ this.prefix = prefix;
+ }
+
+ @Override
+ public PCollection<T> expand(PCollection<T> input) {
+ return input.apply(ParDo.of(new DoFn<T, T>() {
+
+ @ProcessElement
+ public void processElement(@Element T element, OutputReceiver<T> out,
+ BoundedWindow window) {
+
+ String message = prefix + element.toString();
+
+ if (!(window instanceof GlobalWindow)) {
+ message = message + " Window:" + window.toString();
+ }
+
+ LOGGER.info(message);
+
+ out.output(element);
+ }
+
+ }));
+ }
+
+ }
+
+}
diff --git a/learning/katas/java/util/src/util/Log.java b/learning/katas/java/util/src/util/Log.java
deleted file mode 100644
index 69624af..0000000
--- a/learning/katas/java/util/src/util/Log.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package util;
-
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class Log {
-
- private static final Logger LOGGER = LoggerFactory.getLogger(Log.class);
-
- private Log() {
- }
-
- public static <T> PTransform<PCollection<T>, PCollection<T>> ofElements() {
- return new LoggingTransform<>();
- }
-
- public static <T> PTransform<PCollection<T>, PCollection<T>> ofElements(String prefix) {
- return new LoggingTransform<>(prefix);
- }
-
- private static class LoggingTransform<T> extends PTransform<PCollection<T>, PCollection<T>> {
-
- private String prefix;
-
- private LoggingTransform() {
- prefix = "";
- }
-
- private LoggingTransform(String prefix) {
- this.prefix = prefix;
- }
-
- @Override
- public PCollection<T> expand(PCollection<T> input) {
- return input.apply(ParDo.of(new DoFn<T, T>() {
-
- @ProcessElement
- public void processElement(@Element T element, OutputReceiver<T> out) {
- LOGGER.info(prefix + element.toString());
-
- out.output(element);
- }
-
- }));
- }
-
- }
-
-}
diff --git a/learning/katas/java/util/test/org/apache/beam/learning/katas/util/ContainsKvs.java b/learning/katas/java/util/test/org/apache/beam/learning/katas/util/ContainsKvs.java
new file mode 100644
index 0000000..35c5866
--- /dev/null
+++ b/learning/katas/java/util/test/org/apache/beam/learning/katas/util/ContainsKvs.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.util;
+
+import static com.google.common.collect.Iterables.toArray;
+import static org.apache.beam.learning.katas.util.KvMatcher.isKv;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.ImmutableList;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.KV;
+import org.hamcrest.Matcher;
+
+public class ContainsKvs implements
+ SerializableFunction<Iterable<KV<String, Iterable<String>>>, Void> {
+
+ private final List<KV<String, Iterable<String>>> expectedKvs;
+
+ private ContainsKvs(List<KV<String, Iterable<String>>> expectedKvs) {
+ this.expectedKvs = expectedKvs;
+ }
+
+ @SafeVarargs
+ public static SerializableFunction<Iterable<KV<String, Iterable<String>>>, Void> containsKvs(
+ KV<String, Iterable<String>>... kvs) {
+ return new ContainsKvs(ImmutableList.copyOf(kvs));
+ }
+
+ @Override
+ public Void apply(Iterable<KV<String, Iterable<String>>> input) {
+ List<Matcher<? super KV<String, Iterable<String>>>> matchers = new ArrayList<>();
+ for (KV<String, Iterable<String>> expected : expectedKvs) {
+ String[] values = toArray(expected.getValue(), String.class);
+ matchers.add(isKv(equalTo(expected.getKey()), containsInAnyOrder(values)));
+ }
+ assertThat(input, containsInAnyOrder(matchers));
+ return null;
+ }
+
+}
diff --git a/learning/katas/java/util/test/org/apache/beam/learning/katas/util/KvMatcher.java b/learning/katas/java/util/test/org/apache/beam/learning/katas/util/KvMatcher.java
new file mode 100644
index 0000000..3957969
--- /dev/null
+++ b/learning/katas/java/util/test/org/apache/beam/learning/katas/util/KvMatcher.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.learning.katas.util;
+
+import org.apache.beam.sdk.values.KV;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.hamcrest.TypeSafeMatcher;
+
+public class KvMatcher<K, V> extends TypeSafeMatcher<KV<? extends K, ? extends V>> {
+
+ private final Matcher<? super K> keyMatcher;
+ private final Matcher<? super V> valueMatcher;
+
+ public KvMatcher(Matcher<? super K> keyMatcher,
+ Matcher<? super V> valueMatcher) {
+ this.keyMatcher = keyMatcher;
+ this.valueMatcher = valueMatcher;
+ }
+
+ public static <K, V> KvMatcher<K, V> isKv(Matcher<K> keyMatcher,
+ Matcher<V> valueMatcher) {
+ return new KvMatcher<>(keyMatcher, valueMatcher);
+ }
+
+ @Override
+ public boolean matchesSafely(KV<? extends K, ? extends V> kv) {
+ return keyMatcher.matches(kv.getKey())
+ && valueMatcher.matches(kv.getValue());
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description
+ .appendText("a KV(").appendValue(keyMatcher)
+ .appendText(", ").appendValue(valueMatcher)
+ .appendText(")");
+ }
+
+}
diff --git a/learning/katas/java/util/test/test/util/ContainsKvs.java b/learning/katas/java/util/test/test/util/ContainsKvs.java
deleted file mode 100644
index ee08944..0000000
--- a/learning/katas/java/util/test/test/util/ContainsKvs.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package test.util;
-
-import static com.google.common.collect.Iterables.toArray;
-import static org.hamcrest.CoreMatchers.equalTo;
-import static org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
-import static org.junit.Assert.assertThat;
-import static test.util.KvMatcher.isKv;
-
-import com.google.common.collect.ImmutableList;
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.values.KV;
-import org.hamcrest.Matcher;
-
-public class ContainsKvs implements
- SerializableFunction<Iterable<KV<String, Iterable<String>>>, Void> {
-
- private final List<KV<String, Iterable<String>>> expectedKvs;
-
- private ContainsKvs(List<KV<String, Iterable<String>>> expectedKvs) {
- this.expectedKvs = expectedKvs;
- }
-
- @SafeVarargs
- public static SerializableFunction<Iterable<KV<String, Iterable<String>>>, Void> containsKvs(
- KV<String, Iterable<String>>... kvs) {
- return new ContainsKvs(ImmutableList.copyOf(kvs));
- }
-
- @Override
- public Void apply(Iterable<KV<String, Iterable<String>>> input) {
- List<Matcher<? super KV<String, Iterable<String>>>> matchers = new ArrayList<>();
- for (KV<String, Iterable<String>> expected : expectedKvs) {
- String[] values = toArray(expected.getValue(), String.class);
- matchers.add(isKv(equalTo(expected.getKey()), containsInAnyOrder(values)));
- }
- assertThat(input, containsInAnyOrder(matchers));
- return null;
- }
-
-}
diff --git a/learning/katas/java/util/test/test/util/KvMatcher.java b/learning/katas/java/util/test/test/util/KvMatcher.java
deleted file mode 100644
index ecbff08..0000000
--- a/learning/katas/java/util/test/test/util/KvMatcher.java
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package test.util;
-
-import org.apache.beam.sdk.values.KV;
-import org.hamcrest.Description;
-import org.hamcrest.Matcher;
-import org.hamcrest.TypeSafeMatcher;
-
-public class KvMatcher<K, V> extends TypeSafeMatcher<KV<? extends K, ? extends V>> {
-
- private final Matcher<? super K> keyMatcher;
- private final Matcher<? super V> valueMatcher;
-
- public KvMatcher(Matcher<? super K> keyMatcher,
- Matcher<? super V> valueMatcher) {
- this.keyMatcher = keyMatcher;
- this.valueMatcher = valueMatcher;
- }
-
- public static <K, V> KvMatcher<K, V> isKv(Matcher<K> keyMatcher,
- Matcher<V> valueMatcher) {
- return new KvMatcher<>(keyMatcher, valueMatcher);
- }
-
- @Override
- public boolean matchesSafely(KV<? extends K, ? extends V> kv) {
- return keyMatcher.matches(kv.getKey())
- && valueMatcher.matches(kv.getValue());
- }
-
- @Override
- public void describeTo(Description description) {
- description
- .appendText("a KV(").appendValue(keyMatcher)
- .appendText(", ").appendValue(valueMatcher)
- .appendText(")");
- }
-
-}
diff --git a/learning/katas/python/.idea/study_project.xml b/learning/katas/python/.idea/study_project.xml
index d2ea075..84e3db9 100644
--- a/learning/katas/python/.idea/study_project.xml
+++ b/learning/katas/python/.idea/study_project.xml
@@ -21,7 +21,7 @@
<option name="courseMode" value="Course Creator" />
<option name="createDate" value="1557824500323" />
<option name="customPresentableName" />
- <option name="description" value="This course provides a series of kata to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" />
+ <option name="description" value="This course provides a series of katas to get familiar with Apache Beam. Apache Beam website – https://beam.apache.org/" />
<option name="environment" value="" />
<option name="fromZip" value="false" />
<option name="id" value="54532" />
@@ -33,14 +33,14 @@
</option>
<option name="language" value="Python 2.7" />
<option name="languageCode" value="en" />
- <option name="name" value="Beam Kata - Python" />
- <option name="public" value="false" />
+ <option name="name" value="Beam Katas - Python" />
+ <option name="public" value="true" />
<option name="sectionIds">
<list />
</option>
<option name="stepikChangeStatus" value="Up to date" />
<option name="type" value="pycharm11 Python 2.7" />
- <option name="updateDate" value="1557824500000" />
+ <option name="updateDate" value="1560937766000" />
<option name="items">
<list>
<Section>
@@ -50,31 +50,31 @@
<option name="index" value="1" />
<option name="name" value="Introduction" />
<option name="position" value="0" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824504000" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1559325495000" />
<option name="items">
<list>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229517" />
+ <option name="id" value="238426" />
<option name="index" value="1" />
<option name="name" value="Hello Beam" />
<option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557824508000" />
- <option name="unitId" value="202042" />
+ <option name="updateDate" value="1560937886298" />
+ <option name="unitId" value="210886" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p>This kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam"</p> <br> <br> <div class='hint'>Hardcoded input can be created using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Create">Create</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Hello Beam Pipeline</h2> <p> Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow. </p> <p> Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system. </p> <p> To learn more about Apache Beam, refer to <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>. </p> <p> <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam". </p> <br> <div class="hint"> Hardcoded input can be created using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Create"> Create</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory"> "Creating a PCollection from in-memory data"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713750" />
+ <option name="id" value="755575" />
<option name="index" value="1" />
<option name="name" value="Hello Beam" />
<option name="record" value="-1" />
@@ -132,7 +132,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824510000" />
+ <option name="updateDate" value="1560937891911" />
</EduTask>
</list>
</option>
@@ -147,36 +147,36 @@
<option name="index" value="2" />
<option name="name" value="Core Transforms" />
<option name="position" value="0" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824511000" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560432551000" />
<option name="items">
<list>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229518" />
+ <option name="id" value="238427" />
<option name="index" value="1" />
<option name="name" value="Map" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824515000" />
- <option name="unitId" value="202043" />
+ <option name="updateDate" value="1560937929994" />
+ <option name="unitId" value="210887" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p>ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection.</p> <p>For this task, please write a simple ParDo that maps the input element by multiplying it by 10.</p> <br> <br> <div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method</div> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo</h2> <p> ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection. </p> <p> <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10. </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713751" />
+ <option name="id" value="755577" />
<option name="index" value="1" />
<option name="name" value="ParDo" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
<entry key="task.py">
@@ -246,24 +246,24 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824517000" />
+ <option name="updateDate" value="1560937936091" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p>For this task, please write a ParDo that maps each input sentence into words tokenized by whitespace (" ").</p> <br> <br> <div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method. You can return an Iterable for multiple elements or call "yield" for each element to return a generator.</div> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ParDo OneToMany</h2> <p> <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by whitespace (" "). </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method. You can return an Iterable for multiple elements or call "yield" for each element to return a generator. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn"> DoFn</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713752" />
+ <option name="id" value="755578" />
<option name="index" value="2" />
<option name="name" value="ParDo OneToMany" />
<option name="record" value="-1" />
<option name="status" value="Unchecked" />
- <option name="stepikChangeStatus" value="Up to date" />
+ <option name="stepikChangeStatus" value="Info and Content changed" />
<option name="files">
<map>
<entry key="task.py">
@@ -333,19 +333,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824519000" />
+ <option name="updateDate" value="1560937938522" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p> <p>In this task, we are going to implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Map"> Map</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">Map</a> with a lambda.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>MapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map"> Map</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map"> Map</a> with a lambda. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713753" />
+ <option name="id" value="755579" />
<option name="index" value="3" />
<option name="name" value="Map" />
<option name="record" value="-1" />
@@ -403,19 +403,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824521000" />
+ <option name="updateDate" value="1560937942178" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p> <p>FlatMap can be used to simplify DoFn that maps an element to multiple elements (one to many).</p> <p>In this task, we are going to implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap"> FlatMap</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">FlatMap</a> with a lambda.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>FlatMapElements</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> FlatMap can be used to simplify DoFn that maps an element to multiple elements (one to many). </p> <p> <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace (" ") using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap"> FlatMap</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap"> FlatMap</a> with a lambda. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns"> "Lightweight DoFns and other abstractions"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713754" />
+ <option name="id" value="755580" />
<option name="index" value="4" />
<option name="name" value="FlatMap" />
<option name="record" value="-1" />
@@ -473,32 +473,32 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824523000" />
+ <option name="updateDate" value="1560937944601" />
</EduTask>
</list>
</option>
</Lesson>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229519" />
+ <option name="id" value="238428" />
<option name="index" value="2" />
<option name="name" value="GroupByKey" />
<option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557824527000" />
- <option name="unitId" value="202044" />
+ <option name="updateDate" value="1560937980839" />
+ <option name="unitId" value="210888" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p>GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <br> <div class='hint'>Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">GroupByKey</a> to solve this problem</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>GroupByKey</h2> <p> GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key, but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each unique key. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey"> GroupByKey</a> transform that groups words by its first letter. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">GroupByKey</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey"> "GroupByKey"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713756" />
+ <option name="id" value="755582" />
<option name="index" value="1" />
<option name="name" value="GroupByKey" />
<option name="record" value="-1" />
@@ -556,32 +556,32 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824529000" />
+ <option name="updateDate" value="1560937986273" />
</EduTask>
</list>
</option>
</Lesson>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229520" />
+ <option name="id" value="238429" />
<option name="index" value="3" />
<option name="name" value="CoGroupByKey" />
<option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557824533000" />
- <option name="unitId" value="202045" />
+ <option name="updateDate" value="1560938006360" />
+ <option name="unitId" value="210889" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p>CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the string representation of the WordsAlphabet model. </p> <br> <br> <div class='hint'>Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">CoGroupByKey</a> to solve this problem</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>CoGroupByKey</h2> <p> CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"> CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the string representation of the WordsAlphabet model. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey"> CoGroupByKey</a>to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey"> "CoGroupByKey"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713757" />
+ <option name="id" value="755583" />
<option name="index" value="1" />
<option name="name" value="CoGroupByKey" />
<option name="record" value="-1" />
@@ -639,32 +639,32 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824535000" />
+ <option name="updateDate" value="1560938011025" />
</EduTask>
</list>
</option>
</Lesson>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229521" />
+ <option name="id" value="238430" />
<option name="index" value="4" />
<option name="name" value="Combine" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824539000" />
- <option name="unitId" value="202046" />
+ <option name="updateDate" value="1560938016807" />
+ <option name="unitId" value="210890" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p>Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection.</p> <p>Simple combine operations, such as sums, can usually be implemented as a simple function.</p> <p>In this task, we are going to implement the summation of numbers using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"> CombineGlobally</a>. </p> <br> <br> <div class="hint"> Implement a simple Python function that performs the summation of the values. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Simple Function</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Simple combine operations, such as sums, can usually be implemented as a simple function. </p> <p> <b>Kata:</b> Implement the summation of numbers using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"> CombineGlobally</a>. </p> <br> <div class="hint"> Implement a simple Python function that performs the summation of the values. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines"> "Simple combinations using simple functions"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713758" />
+ <option name="id" value="755584" />
<option name="index" value="1" />
<option name="name" value="Simple Function" />
<option name="record" value="-1" />
@@ -739,19 +739,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824541000" />
+ <option name="updateDate" value="1560938025042" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p>Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection.</p> <p>Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account.</p> <p>In this task, we are going to implement the average of numbers using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> Combine.CombineFn</a>. </p> <br> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">CombineFn</a> class that counts the average of the number. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - CombineFn</h2> <p> Combine is a Beam transform for combining collections of elements or values in your data. When you apply a Combine transform, you must provide the function that contains the logic for combining the elements or values. The combining function should be commutative and associative, as the function is not necessarily invoked exactly once on all values with a given key. Because the input data (including the value collection) may be distributed across multiple workers, the combining function might be called multiple times to perform partial combining on subsets of the value collection. </p> <p> Complex combination operations might require you to create a subclass of CombineFn that has an accumulation type distinct from the input/output type. You should use CombineFn if the combine function requires a more sophisticated accumulator, must perform additional pre- or post-processing, might change the output type, or takes the key into account. </p> <p> <b>Kata:</b> Implement the average of numbers using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> Combine.CombineFn</a>. </p> <br> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> CombineFn</a> class that counts the average of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines"> "Advanced combinations using CombineFn"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713759" />
+ <option name="id" value="755585" />
<option name="index" value="2" />
<option name="name" value="CombineFn" />
<option name="record" value="-1" />
@@ -826,19 +826,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824543000" />
+ <option name="updateDate" value="1560938027519" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p>After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn.</p> <p>In this task, we are going to implement the sum of scores per player using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey"> CombinePerKey</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey">CombinePerKey(CombineFn)</a>.</div> <div class='hint'>Extend the <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">CombineFn</a> class that counts the sum of the number.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Combine - Combine PerKey</h2> <p> After creating a keyed PCollection (for example, by using a GroupByKey transform), a common pattern is to combine the collection of values associated with each key into a single, merged value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative reduction function or a subclass of CombineFn. </p> <p> <b>Kata:</b> Implement the sum of scores per player using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey"> CombinePerKey</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey"> CombinePerKey(CombineFn)</a>. </div> <div class="hint"> Extend the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn"> CombineFn</a> class that counts the sum of the number. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection"> "Combining values in a keyed PCollection"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713760" />
+ <option name="id" value="755587" />
<option name="index" value="3" />
<option name="name" value="Combine PerKey" />
<option name="record" value="-1" />
@@ -896,32 +896,32 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824546000" />
+ <option name="updateDate" value="1560938030159" />
</EduTask>
</list>
</option>
</Lesson>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229522" />
+ <option name="id" value="238431" />
<option name="index" value="5" />
<option name="name" value="Flatten" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824550000" />
- <option name="unitId" value="202047" />
+ <option name="updateDate" value="1560938036123" />
+ <option name="unitId" value="210891" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p>Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten">Flatten</a> to solve this problem. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Flatten</h2> <p> Flatten is a Beam transform for PCollection objects that store the same data type. Flatten merges multiple PCollection objects into a single logical PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten"> Flatten</a> transform that merges two PCollection of words into a single PCollection. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten"> Flatten</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#flatten"> "Flatten"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713761" />
+ <option name="id" value="755588" />
<option name="index" value="1" />
<option name="name" value="Flatten" />
<option name="record" value="-1" />
@@ -979,32 +979,32 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824553000" />
+ <option name="updateDate" value="1560938041998" />
</EduTask>
</list>
</option>
</Lesson>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229523" />
+ <option name="id" value="238432" />
<option name="index" value="6" />
<option name="name" value="Partition" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824556000" />
- <option name="unitId" value="202048" />
+ <option name="updateDate" value="1560938052303" />
+ <option name="unitId" value="210892" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p>Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections.</p> <p>Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection.</p> <p>In this task, we are going to implement a <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">Partition</a> to solve this problem. </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Partition</h2> <p> Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections. </p> <p> Partition divides the elements of a PCollection according to a partitioning function that you provide. The partitioning function contains the logic that determines how to split up the elements of the input PCollection into each resulting partition PCollection. </p> <p> <b>Kata:</b> Implement a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition"> Partition</a> transform that splits a PCollection of numbers into two PCollections. The first PCollection contains numbers greater than 100, and the second PCollection contains the remaining numbers. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition"> Partition</a> to solve this problem. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#partition"> "Partition"</a> section for more information. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713762" />
+ <option name="id" value="755589" />
<option name="index" value="1" />
<option name="name" value="Partition" />
<option name="record" value="-1" />
@@ -1079,7 +1079,407 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824558000" />
+ <option name="updateDate" value="1560938058938" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="238433" />
+ <option name="index" value="7" />
+ <option name="name" value="Side Input" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560938065022" />
+ <option name="unitId" value="210893" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Input</h2> <p> In addition to the main input PCollection, you can provide additional inputs to a ParDo transform in the form of side inputs. A side input is an additional input that your DoFn can access each time it processes an element in the input PCollection. When you specify a side input, you create a view of some other data that can be read from within the ParDo transform’s DoFn while processing each element. </p> <p> Side inputs are useful if your ParDo needs to inject additional data when processing each element in the input PCollection, but the additional data needs to be determined at runtime (and not hard-coded). Such values might be determined by the input data, or depend on a different branch of your pipeline. </p> <p> <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in. </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method that also accepts side input argument. </div> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a> with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn"> DoFn</a> that accepts side input. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="755590" />
+ <option name="index" value="1" />
+ <option name="name" value="Side Input" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="task.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1534" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="def process(self, element, cities_to_countries): yield Person(element.name, element.city, cities_to_countries[element.city])" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="2096" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="beam.ParDo(EnrichCountryDoFn(), cities_to_countries)" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="task.py" />
+ <option name="text" value="# TODO: type solution here " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="tests.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="tests.py" />
+ <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560938069904" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="238434" />
+ <option name="index" value="8" />
+ <option name="name" value="Side Output" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560938076976" />
+ <option name="unitId" value="210894" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Side Output</h2> <p> While ParDo always produces a main output PCollection (as the return value from apply), you can also have your ParDo produce any number of additional output PCollections. If you choose to have multiple outputs, your ParDo returns all of the output PCollections (including the main output) bundled together. </p> <p> <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.TaggedOutput"> pvalue.TaggedOutput</a> and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo.with_outputs"> .with_outputs</a> to output multiple tagged-outputs in a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo.</a> </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs"> "Additional outputs"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="755591" />
+ <option name="index" value="1" />
+ <option name="name" value="Side Output" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="task.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1011" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="def process(self, element): if element <= 100: yield element else: yield pvalue.TaggedOutput(num_above_100_tag, element)" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1264" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="beam.ParDo(ProcessNumbersDoFn()) .with_outputs(num_above_100_tag, main=num_below_100_tag))" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="task.py" />
+ <option name="text" value="# TODO: type solution here " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="tests.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="tests.py" />
+ <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560938083234" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="238435" />
+ <option name="index" value="9" />
+ <option name="name" value="Branching" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560938090650" />
+ <option name="unitId" value="210895" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Branching</h2> <p> You can use the same PCollection as input for multiple transforms without consuming the input or altering it. </p> <p> <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying each number by 5 and the other transform is multiplying each number by 10. </p> <br> <div class="hint"> Refer to the Beam Design Your Pipeline Guide <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection"> "Multiple transforms process the same PCollection"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="755592" />
+ <option name="index" value="1" />
+ <option name="name" value="Branching" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="task.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="945" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="numbers | beam.Map(lambda num: num * 5)" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1002" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="numbers | beam.Map(lambda num: num * 10)" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="task.py" />
+ <option name="text" value="# TODO: type solution here " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="tests.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="tests.py" />
+ <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560938095634" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="238436" />
+ <option name="index" value="10" />
+ <option name="name" value="Composite Transform" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560938102699" />
+ <option name="unitId" value="210896" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Composite Transform</h2> <p> Transforms can have a nested structure, where a complex transform performs multiple simpler transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms). These transforms are called composite transforms. Nesting multiple transforms inside a single composite transform can make your code more modular and easier to understand. </p> <p> To create your own composite transform, create a subclass of the PTransform class and override the expand method to specify the actual processing logic. You can then use this transform just as you would a built-in transform from the Beam SDK. Within your PTransform subclass, you’ll need to override the expand method. The expand method is where you add the processing logic for the PTransform. Your override of expand must accept the appropriate type of input PCollection as a parameter, and specify the output PCollection as the return value. </p> <p> <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts numbers from comma separated line and then multiplies each number by 10. </p> <br> <div class="hint"> Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform"> PTransform</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms"> "Composite transforms"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="755593" />
+ <option name="index" value="1" />
+ <option name="name" value="Composite Transform" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="task.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="920" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="def expand(self, pcoll): return (pcoll | beam.FlatMap(lambda line: map(int, line.split(','))) | beam.Map(lambda num: num * 10) )" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="1179" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="ExtractAndMultiplyNumbers()" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="task.py" />
+ <option name="text" value="# TODO: type solution here " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="tests.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="tests.py" />
+ <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560938107880" />
</EduTask>
</list>
</option>
@@ -1095,30 +1495,30 @@
<option name="name" value="Common Transforms" />
<option name="position" value="0" />
<option name="stepikChangeStatus" value="Up to date" />
- <option name="updateDate" value="1557824560000" />
+ <option name="updateDate" value="1560431009000" />
<option name="items">
<list>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229524" />
+ <option name="id" value="238437" />
<option name="index" value="1" />
<option name="name" value="Filter" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824563000" />
- <option name="unitId" value="202049" />
+ <option name="updateDate" value="1560938208485" />
+ <option name="unitId" value="210897" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p>In this task, we are going to implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a>. </p> <br> <br> <div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method. You can use "yield" for each intended element.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter using ParDo</h2> <p> <b>Kata:</b> Implement a filter function that filters out the even numbers by using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo"> ParDo</a>. </p> <br> <div class="hint"> Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process"> process</a> method. You can use "yield" for each intended element. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713763" />
+ <option name="id" value="755595" />
<option name="index" value="1" />
<option name="name" value="ParDo" />
<option name="record" value="-1" />
@@ -1176,19 +1576,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824566000" />
+ <option name="updateDate" value="1560938213611" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p> <p>In this task, we are going to implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> Filter</a>. </p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">Filter</a> with a lambda.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Filter</h2> <p> The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation. </p> <p> <b>Kata:</b> Implement a filter function that filters out the odd numbers by using <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> Filter</a>. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter"> Filter</a> with a lambda. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713764" />
+ <option name="id" value="755596" />
<option name="index" value="2" />
<option name="name" value="Filter" />
<option name="record" value="-1" />
@@ -1246,32 +1646,32 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824567000" />
+ <option name="updateDate" value="1560938217127" />
</EduTask>
</list>
</option>
</Lesson>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229525" />
+ <option name="id" value="238438" />
<option name="index" value="2" />
<option name="name" value="Aggregation" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824571000" />
- <option name="unitId" value="202050" />
+ <option name="updateDate" value="1560938223924" />
+ <option name="unitId" value="210898" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> In this task, we are going to count the number of elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count">Count</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Count</h2> <p> <b>Kata:</b> Count the number of elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count"> Count</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713765" />
+ <option name="id" value="755597" />
<option name="index" value="1" />
<option name="name" value="Count" />
<option name="record" value="-1" />
@@ -1329,19 +1729,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824574000" />
+ <option name="updateDate" value="1560938230679" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> In this task, we are going to compute the sum of all elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">CombineGlobally</a> and Python built-in <a href="https://docs.python.org/2/library/functions.html#sum">sum</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Sum</h2> <p> <b>Kata:</b> Compute the sum of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally"> CombineGlobally</a> and Python built-in <a href="https://docs.python.org/2/library/functions.html#sum">sum</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713766" />
+ <option name="id" value="755598" />
<option name="index" value="2" />
<option name="name" value="Sum" />
<option name="record" value="-1" />
@@ -1399,19 +1799,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824575000" />
+ <option name="updateDate" value="1560938232928" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p>In this task, we are going to compute the mean/average of all elements from an input.</p> <br> <br> <div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Mean">Mean</a></div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Mean</h2> <p> <b>Kata:</b> Compute the mean/average of all elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Mean"> Mean</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713767" />
+ <option name="id" value="755599" />
<option name="index" value="3" />
<option name="name" value="Mean" />
<option name="record" value="-1" />
@@ -1469,19 +1869,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824578000" />
+ <option name="updateDate" value="1560938235730" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Smallest</h2> <p>In this task, we are going to compute the smallest of the elements from an input.</p> <br> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Smallest">Top.Smallest</a> </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Smallest</h2> <p> <b>Kata:</b> Compute the smallest of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Smallest"> Top.Smallest</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713768" />
+ <option name="id" value="755600" />
<option name="index" value="4" />
<option name="name" value="Smallest" />
<option name="record" value="-1" />
@@ -1539,19 +1939,19 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824580000" />
+ <option name="updateDate" value="1560938237747" />
</EduTask>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Largest</h2> <p>In this task, we are going to compute the largest of the elements from an input.</p> <br> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Largest">Top.Largest</a> </div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Aggregation - Largest</h2> <p> <b>Kata:</b> Compute the largest of the elements from an input. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Largest"> Top.Largest</a>. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713769" />
+ <option name="id" value="755601" />
<option name="index" value="5" />
<option name="name" value="Largest" />
<option name="record" value="-1" />
@@ -1609,7 +2009,201 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824582000" />
+ <option name="updateDate" value="1560938239860" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ </list>
+ </option>
+ </Section>
+ <Section>
+ <option name="courseId" value="54532" />
+ <option name="customPresentableName" />
+ <option name="id" value="88017" />
+ <option name="index" value="4" />
+ <option name="name" value="IO" />
+ <option name="position" value="5" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560436240000" />
+ <option name="items">
+ <list>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="238439" />
+ <option name="index" value="1" />
+ <option name="name" value="TextIO" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560938245888" />
+ <option name="unitId" value="210899" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>ReadFromText</h2> <p> When you create a pipeline, you often need to read data from some external source, such as a file or a database. Likewise, you may want your pipeline to output its result data to an external storage system. Beam provides read and write transforms for a number of common data storage types. If you want your pipeline to read from or write to a data storage format that isn’t supported by the built-in transforms, you can implement your own read and write transforms. </p> <p> To read a PCollection from one or more text files, use beam.io.ReadFromText to instantiate a transform and specify the path of the file(s) to be read. </p> <p> <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase. </p> <br> <div class="hint"> Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.ReadFromText"> beam.io.ReadFromText</a>. </div> <div class="hint"> Refer to the Beam Programming Guide <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data"> "Reading input data"</a> section for more information. </div> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="755602" />
+ <option name="index" value="1" />
+ <option name="name" value="ReadFromText" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="task.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="0" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="919" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="beam.io.ReadFromText(file_path)" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ <AnswerPlaceholder>
+ <option name="hints">
+ <list />
+ </option>
+ <option name="index" value="1" />
+ <option name="initialState" />
+ <option name="initializedFromDependency" value="false" />
+ <option name="length" value="6" />
+ <option name="offset" value="956" />
+ <option name="placeholderDependency" />
+ <option name="placeholderText" value="TODO()" />
+ <option name="possibleAnswer" value="beam.Map(lambda country: country.upper())" />
+ <option name="selected" value="false" />
+ <option name="status" value="Unchecked" />
+ <option name="studentAnswer" />
+ <option name="useLength" value="false" />
+ </AnswerPlaceholder>
+ </list>
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="task.py" />
+ <option name="text" value="# TODO: type solution here " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="tests.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="tests.py" />
+ <option name="text" value="from test_helper import run_common_tests, failed, passed, get_answer_placeholders def test_answer_placeholders(): placeholders = get_answer_placeholders() placeholder = placeholders[0] if placeholder == "": # TODO: your condition here passed() else: failed() if __name__ == '__main__': run_common_tests() # test_answer_placeholders() # TODO: uncomment test call " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="countries.txt">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="countries.txt" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560938252130" />
+ </EduTask>
+ </list>
+ </option>
+ </Lesson>
+ <Lesson>
+ <option name="customPresentableName" />
+ <option name="id" value="238440" />
+ <option name="index" value="2" />
+ <option name="name" value="Built-in IOs" />
+ <option name="stepikChangeStatus" value="Content changed" />
+ <option name="updateDate" value="1560938258337" />
+ <option name="unitId" value="210900" />
+ <option name="items">
+ <list>
+ <EduTask>
+ <option name="customPresentableName" />
+ <option name="descriptionFormat" value="HTML" />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Built-in I/Os</h2> <p> Beam SDKs provide many out of the box I/O transforms that can be used to read from many different sources and write to many different sinks. </p> <p> See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O Transforms</a> page for a list of the currently available I/O transforms. </p> </html> " />
+ <option name="feedbackLink">
+ <FeedbackLink>
+ <option name="link" />
+ <option name="type" value="STEPIK" />
+ </FeedbackLink>
+ </option>
+ <option name="id" value="755603" />
+ <option name="index" value="1" />
+ <option name="name" value="Built-in IOs" />
+ <option name="record" value="-1" />
+ <option name="status" value="Unchecked" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="files">
+ <map>
+ <entry key="task.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="task.py" />
+ <option name="text" value="# TODO: type solution here " />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="true" />
+ </TaskFile>
+ </value>
+ </entry>
+ <entry key="tests.py">
+ <value>
+ <TaskFile>
+ <option name="answerPlaceholders">
+ <list />
+ </option>
+ <option name="highlightErrors" value="true" />
+ <option name="name" value="tests.py" />
+ <option name="text" value="" />
+ <option name="trackChanges" value="true" />
+ <option name="trackLengths" value="true" />
+ <option name="visible" value="false" />
+ </TaskFile>
+ </value>
+ </entry>
+ </map>
+ </option>
+ <option name="updateDate" value="1560938263697" />
</EduTask>
</list>
</option>
@@ -1621,34 +2215,34 @@
<option name="courseId" value="54532" />
<option name="customPresentableName" />
<option name="id" value="85647" />
- <option name="index" value="4" />
+ <option name="index" value="5" />
<option name="name" value="Examples" />
<option name="position" value="0" />
- <option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824583000" />
+ <option name="stepikChangeStatus" value="Up to date" />
+ <option name="updateDate" value="1560435414000" />
<option name="items">
<list>
<Lesson>
<option name="customPresentableName" />
- <option name="id" value="229526" />
+ <option name="id" value="238441" />
<option name="index" value="1" />
<option name="name" value="Word Count" />
<option name="stepikChangeStatus" value="Content changed" />
- <option name="updateDate" value="1557824587000" />
- <option name="unitId" value="202051" />
+ <option name="updateDate" value="1560938269193" />
+ <option name="unitId" value="210901" />
<option name="items">
<list>
<EduTask>
<option name="customPresentableName" />
<option name="descriptionFormat" value="HTML" />
- <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p>This kata is to create a pipeline that counts the number of words.</p> <p>For this task, please output the count of each word in the following format:<br/> <pre> word:count ball:5 book:3 </pre> </p> <br> <br> <div class='hint'>Refer to your lessons above.</div> </html> " />
+ <option name="descriptionText" value="<!-- ~ Licensed to the Apache Software Foundation (ASF) under one ~ or more contributor license agreements. See the NOTICE file ~ distributed with this work for additional information ~ regarding copyright ownership. The ASF licenses this file ~ to you under the Apache License, Version 2.0 (the ~ "License"); you may not use this file except in compliance ~ with the License. You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. --> <html> <h2>Word Count Pipeline</h2> <p> <b>Kata:</b> Create a pipeline that counts the number of words. </p> <p> Please output the count of each word in the following format: </p> <pre> word:count ball:5 book:3 </pre> <br> <div class="hint"> Refer to your katas above. </div> </html> " />
<option name="feedbackLink">
<FeedbackLink>
<option name="link" />
<option name="type" value="STEPIK" />
</FeedbackLink>
</option>
- <option name="id" value="713770" />
+ <option name="id" value="755604" />
<option name="index" value="1" />
<option name="name" value="Word Count" />
<option name="record" value="-1" />
@@ -1706,7 +2300,7 @@
</entry>
</map>
</option>
- <option name="updateDate" value="1557824590000" />
+ <option name="updateDate" value="1560938273811" />
</EduTask>
</list>
</option>
diff --git a/learning/katas/python/Common Transforms/Aggregation/Count/task.html b/learning/katas/python/Common Transforms/Aggregation/Count/task.html
index b7ff9d6..b9ad594 100644
--- a/learning/katas/python/Common Transforms/Aggregation/Count/task.html
+++ b/learning/katas/python/Common Transforms/Aggregation/Count/task.html
@@ -19,9 +19,11 @@
<html>
<h2>Aggregation - Count</h2>
<p>
- In this task, we are going to count the number of elements from an input.</p>
+ <b>Kata:</b> Count the number of elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count">Count</a></div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count">
+ Count</a>.
+</div>
</html>
-
diff --git a/learning/katas/python/Common Transforms/Aggregation/Largest/task.html b/learning/katas/python/Common Transforms/Aggregation/Largest/task.html
index ec2fcbb..9c9fe4f 100644
--- a/learning/katas/python/Common Transforms/Aggregation/Largest/task.html
+++ b/learning/katas/python/Common Transforms/Aggregation/Largest/task.html
@@ -18,10 +18,12 @@
<html>
<h2>Aggregation - Largest</h2>
-<p>In this task, we are going to compute the largest of the elements from an input.</p>
-<br>
+<p>
+ <b>Kata:</b> Compute the largest of the elements from an input.
+</p>
<br>
<div class="hint">
- Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Largest">Top.Largest</a>
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Largest">
+ Top.Largest</a>.
</div>
</html>
diff --git a/learning/katas/python/Common Transforms/Aggregation/Mean/task.html b/learning/katas/python/Common Transforms/Aggregation/Mean/task.html
index e3013ca..2434ba2 100644
--- a/learning/katas/python/Common Transforms/Aggregation/Mean/task.html
+++ b/learning/katas/python/Common Transforms/Aggregation/Mean/task.html
@@ -18,9 +18,12 @@
<html>
<h2>Aggregation - Mean</h2>
-<p>In this task, we are going to compute the mean/average of all elements from an input.</p>
+<p>
+ <b>Kata:</b> Compute the mean/average of all elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Mean">Mean</a></div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Mean">
+ Mean</a>.
+</div>
</html>
-
diff --git a/learning/katas/python/Common Transforms/Aggregation/Smallest/task.html b/learning/katas/python/Common Transforms/Aggregation/Smallest/task.html
index 5eac49e..49cc7ad 100644
--- a/learning/katas/python/Common Transforms/Aggregation/Smallest/task.html
+++ b/learning/katas/python/Common Transforms/Aggregation/Smallest/task.html
@@ -18,10 +18,12 @@
<html>
<h2>Aggregation - Smallest</h2>
-<p>In this task, we are going to compute the smallest of the elements from an input.</p>
-<br>
+<p>
+ <b>Kata:</b> Compute the smallest of the elements from an input.
+</p>
<br>
<div class="hint">
- Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Smallest">Top.Smallest</a>
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Top.Smallest">
+ Top.Smallest</a>.
</div>
</html>
diff --git a/learning/katas/python/Common Transforms/Aggregation/Sum/task.html b/learning/katas/python/Common Transforms/Aggregation/Sum/task.html
index fa57f37..50f5b99 100644
--- a/learning/katas/python/Common Transforms/Aggregation/Sum/task.html
+++ b/learning/katas/python/Common Transforms/Aggregation/Sum/task.html
@@ -19,10 +19,12 @@
<html>
<h2>Aggregation - Sum</h2>
<p>
- In this task, we are going to compute the sum of all elements from an input.</p>
+ <b>Kata:</b> Compute the sum of all elements from an input.
+</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">CombineGlobally</a>
- and Python built-in <a href="https://docs.python.org/2/library/functions.html#sum">sum</a></div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">
+ CombineGlobally</a> and Python built-in
+ <a href="https://docs.python.org/2/library/functions.html#sum">sum</a>.
+</div>
</html>
-
diff --git a/learning/katas/python/Common Transforms/Filter/Filter/task.html b/learning/katas/python/Common Transforms/Filter/Filter/task.html
index 2de6da8..797f77c 100644
--- a/learning/katas/python/Common Transforms/Filter/Filter/task.html
+++ b/learning/katas/python/Common Transforms/Filter/Filter/task.html
@@ -18,14 +18,17 @@
<html>
<h2>Filter</h2>
-<p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p>
-<p>In this task, we are going to implement a filter function that filters out the odd numbers by using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">
- Filter</a>.
+<p>
+ The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+</p>
+<p>
+ <b>Kata:</b> Implement a filter function that filters out the odd numbers by using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">
+ Filter</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">Filter</a>
- with a lambda.</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Filter">
+ Filter</a> with a lambda.
+</div>
</html>
-
diff --git a/learning/katas/python/Common Transforms/Filter/ParDo/task.html b/learning/katas/python/Common Transforms/Filter/ParDo/task.html
index 628f63f..1c4ea1b 100644
--- a/learning/katas/python/Common Transforms/Filter/ParDo/task.html
+++ b/learning/katas/python/Common Transforms/Filter/ParDo/task.html
@@ -18,13 +18,14 @@
<html>
<h2>Filter using ParDo</h2>
-<p>In this task, we are going to implement a filter function that filters out the even numbers by using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
- ParDo</a>.
+<p>
+ <b>Kata:</b> Implement a filter function that filters out the even numbers by using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
+ ParDo</a>.
</p>
<br>
-<br>
-<div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a>
- method. You can use "yield" for each intended element.</div>
+<div class="hint">
+ Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">
+ process</a> method. You can use "yield" for each intended element.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Branching/Branching/task.html b/learning/katas/python/Core Transforms/Branching/Branching/task.html
new file mode 100644
index 0000000..12d9645
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/Branching/task.html
@@ -0,0 +1,35 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Branching</h2>
+<p>
+ You can use the same PCollection as input for multiple transforms without consuming the input
+ or altering it.
+</p>
+<p>
+ <b>Kata:</b> Branch out the numbers to two different transforms: one transform is multiplying
+ each number by 5 and the other transform is multiplying each number by 10.
+</p>
+<br>
+<div class="hint">
+ Refer to the Beam Design Your Pipeline Guide
+ <a href="https://beam.apache.org/documentation/pipelines/design-your-pipeline/#multiple-transforms-process-the-same-pcollection">
+ "Multiple transforms process the same PCollection"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/python/Core Transforms/Branching/Branching/task.py b/learning/katas/python/Core Transforms/Branching/Branching/task.py
new file mode 100644
index 0000000..e29b67c
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/Branching/task.py
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import apache_beam as beam
+
+from log_elements import LogElements
+
+p = beam.Pipeline()
+
+numbers = p | beam.Create([1, 2, 3, 4, 5])
+
+mult5_results = numbers | beam.Map(lambda num: num * 5)
+mult10_results = numbers | beam.Map(lambda num: num * 10)
+
+mult5_results | 'Log multiply 5' >> LogElements(prefix='Multiplied by 5: ')
+mult10_results | 'Log multiply 10' >> LogElements(prefix='Multiplied by 10: ')
+
+p.run()
diff --git a/learning/katas/python/Core Transforms/Branching/Branching/tests.py b/learning/katas/python/Core Transforms/Branching/Branching/tests.py
new file mode 100644
index 0000000..8278966
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Branching/Branching/tests.py
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from test_helper import run_common_tests, failed, passed, get_file_output
+
+
+def test_output():
+ output = get_file_output()
+
+ mult5_results = ['5', '10', '15', '20', '25']
+ mult10_results = ['10', '20', '30', '40', '50']
+
+ answers = []
+
+ for num in mult5_results:
+ answers.append('Multiplied by 5: ' + num)
+
+ for num in mult10_results:
+ answers.append('Multiplied by 10: ' + num)
+
+ if all(num in output for num in answers):
+ passed()
+ else:
+ failed('Incorrect output. Branch out the numbers and multiply '
+ 'accordingly.')
+
+
+if __name__ == '__main__':
+ run_common_tests()
+ test_output()
diff --git a/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.html b/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
index 48f2f97..5c7ecf2 100644
--- a/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
+++ b/learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
@@ -18,16 +18,25 @@
<html>
<h2>CoGroupByKey</h2>
-<p>CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">
- CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces the string
- representation of the WordsAlphabet model.
+<p>
+ CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+ key type.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">
+ CoGroupByKey</a> transform that join words by its first alphabetical letter, and then produces
+ the string representation of the WordsAlphabet model.
</p>
<br>
-<br>
-<div class='hint'>Refer to
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">CoGroupByKey</a>
- to solve this problem</div>
+<div class="hint">
+ Refer to
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">
+ CoGroupByKey</a>to solve this problem.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Combine/Combine PerKey/task.html b/learning/katas/python/Core Transforms/Combine/Combine PerKey/task.html
index 523857a..044aae7 100644
--- a/learning/katas/python/Core Transforms/Combine/Combine PerKey/task.html
+++ b/learning/katas/python/Core Transforms/Combine/Combine PerKey/task.html
@@ -18,18 +18,31 @@
<html>
<h2>Combine - Combine PerKey</h2>
-<p>After creating a keyed PCollection (for example, by using a GroupByKey transform), a common
- pattern is to combine the collection of values associated with each key into a single, merged value.
- This pattern of a GroupByKey followed by merging the collection of values is equivalent to
+<p>
+ After creating a keyed PCollection (for example, by using a GroupByKey transform), a common
+ pattern is to combine the collection of values associated with each key into a single, merged
+ value. This pattern of a GroupByKey followed by merging the collection of values is equivalent to
Combine PerKey transform. The combine function you supply to Combine PerKey must be an associative
- reduction function or a subclass of CombineFn.</p>
-<p>In this task, we are going to implement the sum of scores per player using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey">
+ reduction function or a subclass of CombineFn.
+</p>
+<p>
+ <b>Kata:</b> Implement the sum of scores per player using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey">
CombinePerKey</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey">CombinePerKey(CombineFn)</a>.</div>
-<div class='hint'>Extend the <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">CombineFn</a>
- class that counts the sum of the number.</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey">
+ CombinePerKey(CombineFn)</a>.
+</div>
+<div class="hint">
+ Extend the
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">
+ CombineFn</a> class that counts the sum of the number.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#combining-values-in-a-keyed-pcollection">
+ "Combining values in a keyed PCollection"</a> section for more information.
+</div>
</html>
diff --git a/learning/katas/python/Core Transforms/Combine/CombineFn/task.html b/learning/katas/python/Core Transforms/Combine/CombineFn/task.html
index 9918d1b..4828e0f 100644
--- a/learning/katas/python/Core Transforms/Combine/CombineFn/task.html
+++ b/learning/katas/python/Core Transforms/Combine/CombineFn/task.html
@@ -18,25 +18,35 @@
<html>
<h2>Combine - CombineFn</h2>
-<p>Combine is a Beam transform for combining collections of elements or values in your data.
+<p>
+ Combine is a Beam transform for combining collections of elements or values in your data.
When you apply a Combine transform, you must provide the function that contains the logic for
combining the elements or values. The combining function should be commutative and associative,
as the function is not necessarily invoked exactly once on all values with a given key. Because
the input data (including the value collection) may be distributed across multiple workers, the
combining function might be called multiple times to perform partial combining on subsets of
- the value collection.</p>
-<p>Complex combination operations might require you to create a subclass of CombineFn that has an
+ the value collection.
+</p>
+<p>
+ Complex combination operations might require you to create a subclass of CombineFn that has an
accumulation type distinct from the input/output type. You should use CombineFn if the combine
function requires a more sophisticated accumulator, must perform additional pre- or
- post-processing, might change the output type, or takes the key into account.</p>
-<p>In this task, we are going to implement the average of numbers using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">
+ post-processing, might change the output type, or takes the key into account.
+</p>
+<p>
+ <b>Kata:</b> Implement the average of numbers using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">
Combine.CombineFn</a>.
</p>
<br>
-<br>
<div class="hint">
- Extend the <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">CombineFn</a>
- class that counts the average of the number.
+ Extend the
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn">
+ CombineFn</a> class that counts the average of the number.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#advanced-combines">
+ "Advanced combinations using CombineFn"</a> section for more information.
</div>
</html>
diff --git a/learning/katas/python/Core Transforms/Combine/Simple Function/task.html b/learning/katas/python/Core Transforms/Combine/Simple Function/task.html
index cb3e79c..5e4bd02 100644
--- a/learning/katas/python/Core Transforms/Combine/Simple Function/task.html
+++ b/learning/katas/python/Core Transforms/Combine/Simple Function/task.html
@@ -18,21 +18,30 @@
<html>
<h2>Combine - Simple Function</h2>
-<p>Combine is a Beam transform for combining collections of elements or values in your data.
+<p>
+ Combine is a Beam transform for combining collections of elements or values in your data.
When you apply a Combine transform, you must provide the function that contains the logic for
combining the elements or values. The combining function should be commutative and associative,
as the function is not necessarily invoked exactly once on all values with a given key. Because
the input data (including the value collection) may be distributed across multiple workers, the
combining function might be called multiple times to perform partial combining on subsets of
- the value collection.</p>
-<p>Simple combine operations, such as sums, can usually be implemented as a simple function.</p>
-<p>In this task, we are going to implement the summation of numbers using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">
+ the value collection.
+</p>
+<p>
+ Simple combine operations, such as sums, can usually be implemented as a simple function.
+</p>
+<p>
+ <b>Kata:</b> Implement the summation of numbers using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">
CombineGlobally</a>.
</p>
<br>
-<br>
<div class="hint">
Implement a simple Python function that performs the summation of the values.
</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#simple-combines">
+ "Simple combinations using simple functions"</a> section for more information.
+</div>
</html>
diff --git a/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.html b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.html
new file mode 100644
index 0000000..94c0e44
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.html
@@ -0,0 +1,49 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Composite Transform</h2>
+<p>
+ Transforms can have a nested structure, where a complex transform performs multiple simpler
+ transforms (such as more than one ParDo, Combine, GroupByKey, or even other composite transforms).
+ These transforms are called composite transforms. Nesting multiple transforms inside a single
+ composite transform can make your code more modular and easier to understand.
+</p>
+<p>
+ To create your own composite transform, create a subclass of the PTransform class and override
+ the expand method to specify the actual processing logic. You can then use this transform just as
+ you would a built-in transform from the Beam SDK. Within your PTransform subclass, you’ll need to
+ override the expand method. The expand method is where you add the processing logic for the
+ PTransform. Your override of expand must accept the appropriate type of input PCollection as a
+ parameter, and specify the output PCollection as the return value.
+</p>
+<p>
+ <b>Kata:</b> Please implement a composite transform "ExtractAndMultiplyNumbers" that extracts
+ numbers from comma separated line and then multiplies each number by 10.
+</p>
+<br>
+<div class="hint">
+ Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform">
+ PTransform</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#composite-transforms">
+ "Composite transforms"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.py b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.py
new file mode 100644
index 0000000..46396b9
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.py
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import apache_beam as beam
+
+from log_elements import LogElements
+
+
+class ExtractAndMultiplyNumbers(beam.PTransform):
+
+ def expand(self, pcoll):
+ return (pcoll
+ | beam.FlatMap(lambda line: map(int, line.split(',')))
+ | beam.Map(lambda num: num * 10)
+ )
+
+
+p = beam.Pipeline()
+
+(p | beam.Create(['1,2,3,4,5', '6,7,8,9,10'])
+ | ExtractAndMultiplyNumbers()
+ | LogElements())
+
+p.run()
diff --git a/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/tests.py b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/tests.py
new file mode 100644
index 0000000..24477d1
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Composite Transform/Composite Transform/tests.py
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from test_helper import run_common_tests, failed, passed, get_answer_placeholders, get_file_output
+
+
+def test_composite_expand_method():
+ placeholders = get_answer_placeholders()
+ placeholder = placeholders[0]
+
+ if 'def expand(' in placeholder:
+ passed()
+ else:
+ failed('Override "expand" method')
+
+
+def test_output():
+ output = get_file_output()
+
+ answers = ['10', '20', '30', '40', '50', '60', '70', '80', '90', '100']
+
+ if all(num in output for num in answers):
+ passed()
+ else:
+ failed("Incorrect output. Extract the numbers and multiply each by 10.")
+
+
+if __name__ == '__main__':
+ run_common_tests()
+ test_composite_expand_method()
+ test_output()
diff --git a/learning/katas/python/Core Transforms/Flatten/Flatten/task.html b/learning/katas/python/Core Transforms/Flatten/Flatten/task.html
index e7225ef..488c139 100644
--- a/learning/katas/python/Core Transforms/Flatten/Flatten/task.html
+++ b/learning/katas/python/Core Transforms/Flatten/Flatten/task.html
@@ -18,16 +18,23 @@
<html>
<h2>Flatten</h2>
-<p>Flatten is a Beam transform for PCollection objects that store the same data type.
- Flatten merges multiple PCollection objects into a single logical PCollection.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten">
+<p>
+ Flatten is a Beam transform for PCollection objects that store the same data type.
+ Flatten merges multiple PCollection objects into a single logical PCollection.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten">
Flatten</a> transform that merges two PCollection of words into a single PCollection.
</p>
<br>
-<br>
<div class="hint">
- Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten">Flatten</a>
- to solve this problem.
+ Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Flatten">
+ Flatten</a> to solve this problem.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#flatten">
+ "Flatten"</a> section for more information.
</div>
</html>
diff --git a/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.html b/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.html
index f5b6d06..042912a 100644
--- a/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.html
+++ b/learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.html
@@ -18,19 +18,27 @@
<html>
<h2>GroupByKey</h2>
-<p>GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel reduction operation,
- analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The input to GroupByKey is a collection of
- key/value pairs that represents a multimap, where the collection contains multiple pairs that have the same key,
- but different values. Given such a collection, you use GroupByKey to collect all of the values associated with each
- unique key.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">
- GroupByKey</a> transform that groups words by its first letter.
+<p>
+ GroupByKey is a Beam transform for processing collections of key/value pairs. It’s a parallel
+ reduction operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style algorithm. The
+ input to GroupByKey is a collection of key/value pairs that represents a multimap, where the
+ collection contains multiple pairs that have the same key, but different values. Given such a
+ collection, you use GroupByKey to collect all of the values associated with each unique key.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">
+ GroupByKey</a> transform that groups words by its first letter.
</p>
<br>
-<br>
-<div class='hint'>Refer to
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">GroupByKey</a>
- to solve this problem</div>
+<div class="hint">
+ Refer to
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">GroupByKey</a>
+ to solve this problem.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#groupbykey">
+ "GroupByKey"</a> section for more information.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Map/FlatMap/task.html b/learning/katas/python/Core Transforms/Map/FlatMap/task.html
index 3a4f51b..f69fffd 100644
--- a/learning/katas/python/Core Transforms/Map/FlatMap/task.html
+++ b/learning/katas/python/Core Transforms/Map/FlatMap/task.html
@@ -18,15 +18,26 @@
<html>
<h2>FlatMapElements</h2>
-<p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p>
-<p>FlatMap can be used to simplify DoFn that maps an element to multiple elements (one to many).</p>
-<p>In this task, we are going to implement a function that maps each input sentence into words tokenized by whitespace (" ") using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">
- FlatMap</a>.
+<p>
+ The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+</p>
+<p>
+ FlatMap can be used to simplify DoFn that maps an element to multiple elements (one to many).
+</p>
+<p>
+ <b>Kata:</b> Implement a function that maps each input sentence into words tokenized by whitespace
+ (" ") using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">
+ FlatMap</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">FlatMap</a>
- with a lambda.</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.FlatMap">
+ FlatMap</a> with a lambda.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns">
+ "Lightweight DoFns and other abstractions"</a> section for more information.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Map/Map/task.html b/learning/katas/python/Core Transforms/Map/Map/task.html
index d0cf0a7..fee1a4b 100644
--- a/learning/katas/python/Core Transforms/Map/Map/task.html
+++ b/learning/katas/python/Core Transforms/Map/Map/task.html
@@ -18,14 +18,22 @@
<html>
<h2>MapElements</h2>
-<p>The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.</p>
-<p>In this task, we are going to implement a simple map function that multiplies all input elements by 5 using
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
- Map</a>.
+<p>
+ The Beam SDKs provide language-specific ways to simplify how you provide your DoFn implementation.
+</p>
+<p>
+ <b>Kata:</b> Implement a simple map function that multiplies all input elements by 5 using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
+ Map</a>.
</p>
<br>
-<br>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">Map</a>
- with a lambda.</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map">
+ Map</a> with a lambda.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#lightweight-dofns">
+ "Lightweight DoFns and other abstractions"</a> section for more information.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.html b/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.html
index d797cc3..5ff36d0 100644
--- a/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.html
+++ b/learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.html
@@ -18,12 +18,26 @@
<html>
<h2>ParDo OneToMany</h2>
-<p>For this task, please write a ParDo that maps each input sentence into words tokenized by whitespace (" ").</p>
+<p>
+ <b>Kata:</b> Please write a ParDo that maps each input sentence into words tokenized by
+ whitespace (" ").
+</p>
<br>
-<br>
-<div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method.
- You can return an Iterable for multiple elements or call "yield" for each element to return a generator.</div>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">ParDo</a>
- with <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a></div>
+<div class="hint">
+ Override
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">
+ process</a> method. You can return an Iterable for multiple elements or call "yield" for each
+ element to return a generator.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
+ ParDo</a> with
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">
+ DoFn</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for
+ more information.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Map/ParDo/task.html b/learning/katas/python/Core Transforms/Map/ParDo/task.html
index 899c7f1..e6eab7b 100644
--- a/learning/katas/python/Core Transforms/Map/ParDo/task.html
+++ b/learning/katas/python/Core Transforms/Map/ParDo/task.html
@@ -18,12 +18,28 @@
<html>
<h2>ParDo</h2>
-<p>ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection.</p>
-<p>For this task, please write a simple ParDo that maps the input element by multiplying it by 10.</p>
+<p>
+ ParDo is a Beam transform for generic parallel processing. The ParDo processing paradigm is
+ similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers
+ each element in the input PCollection, performs some processing function (your user code) on
+ that element, and emits zero, one, or multiple elements to an output PCollection.
+</p>
+<p>
+ <b>Kata:</b> Please write a simple ParDo that maps the input element by multiplying it by 10.
+</p>
<br>
-<br>
-<div class='hint'>Override <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">process</a> method</div>
-<div class='hint'>Use <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">ParDo</a> with
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a></div>
+<div class="hint">
+ Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">
+ process</a> method.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
+ ParDo</a> with
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">DoFn</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#pardo">"ParDo"</a> section for
+ more information.
+</div>
</html>
-
diff --git a/learning/katas/python/Core Transforms/Partition/Partition/task.html b/learning/katas/python/Core Transforms/Partition/Partition/task.html
index 96a7510..513fd3a 100644
--- a/learning/katas/python/Core Transforms/Partition/Partition/task.html
+++ b/learning/katas/python/Core Transforms/Partition/Partition/task.html
@@ -18,21 +18,30 @@
<html>
<h2>Partition</h2>
-<p>Partition is a Beam transform for PCollection objects that store the same data type.
- Partition splits a single PCollection into a fixed number of smaller collections.</p>
-<p>Partition divides the elements of a PCollection according to a partitioning function
+<p>
+ Partition is a Beam transform for PCollection objects that store the same data type.
+ Partition splits a single PCollection into a fixed number of smaller collections.
+</p>
+<p>
+ Partition divides the elements of a PCollection according to a partitioning function
that you provide. The partitioning function contains the logic that determines how to split up
- the elements of the input PCollection into each resulting partition PCollection.</p>
-<p>In this task, we are going to implement a
- <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
+ the elements of the input PCollection into each resulting partition PCollection.
+</p>
+<p>
+ <b>Kata:</b> Implement a
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
Partition</a> transform that splits a PCollection of numbers into two PCollections.
The first PCollection contains numbers greater than 100, and the second PCollection contains
the remaining numbers.
</p>
<br>
-<br>
<div class="hint">
- Refer to <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">Partition</a>
- to solve this problem.
+ Refer to <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Partition">
+ Partition</a> to solve this problem.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#partition">
+ "Partition"</a> section for more information.
</div>
</html>
diff --git a/learning/katas/python/Core Transforms/Side Input/Side Input/task.html b/learning/katas/python/Core Transforms/Side Input/Side Input/task.html
new file mode 100644
index 0000000..b913627
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/Side Input/task.html
@@ -0,0 +1,53 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Side Input</h2>
+<p>
+ In addition to the main input PCollection, you can provide additional inputs to a ParDo transform
+ in the form of side inputs. A side input is an additional input that your DoFn can access each
+ time it processes an element in the input PCollection. When you specify a side input, you create
+ a view of some other data that can be read from within the ParDo transform’s DoFn while
+ processing each element.
+</p>
+<p>
+ Side inputs are useful if your ParDo needs to inject additional data when processing each element
+ in the input PCollection, but the additional data needs to be determined at runtime (and not
+ hard-coded). Such values might be determined by the input data, or depend on a different branch
+ of your pipeline.
+</p>
+<p>
+ <b>Kata:</b> Please enrich each Person with the country based on the city he/she lives in.
+</p>
+<br>
+<div class="hint">
+ Override <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process">
+ process</a> method that also accepts side input argument.
+</div>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
+ ParDo</a> with
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn">
+ DoFn</a> that accepts side input.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">"Side inputs"</a>
+ section for more information.
+</div>
+</html>
diff --git a/learning/katas/python/Core Transforms/Side Input/Side Input/task.py b/learning/katas/python/Core Transforms/Side Input/Side Input/task.py
new file mode 100644
index 0000000..ec6d39e
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/Side Input/task.py
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import apache_beam as beam
+
+from log_elements import LogElements
+
+
+class Person:
+ def __init__(self, name, city, country=''):
+ self.name = name
+ self.city = city
+ self.country = country
+
+ def __str__(self):
+ return 'Person[' + self.name + ',' + self.city + ',' + self.country + ']'
+
+
+class EnrichCountryDoFn(beam.DoFn):
+
+ def process(self, element, cities_to_countries):
+ yield Person(element.name, element.city,
+ cities_to_countries[element.city])
+
+
+p = beam.Pipeline()
+
+cities_to_countries = {
+ 'Beijing': 'China',
+ 'London': 'United Kingdom',
+ 'San Francisco': 'United States',
+ 'Singapore': 'Singapore',
+ 'Sydney': 'Australia'
+}
+
+persons = [
+ Person('Henry', 'Singapore'),
+ Person('Jane', 'San Francisco'),
+ Person('Lee', 'Beijing'),
+ Person('John', 'Sydney'),
+ Person('Alfred', 'London')
+]
+
+(p | beam.Create(persons)
+ | beam.ParDo(EnrichCountryDoFn(), cities_to_countries)
+ | LogElements())
+
+p.run()
diff --git a/learning/katas/python/Core Transforms/Side Input/Side Input/tests.py b/learning/katas/python/Core Transforms/Side Input/Side Input/tests.py
new file mode 100644
index 0000000..0d7146d
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Input/Side Input/tests.py
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from test_helper import run_common_tests, failed, passed, \
+ get_answer_placeholders, get_file_output
+
+
+def test_dofn_process_method():
+ placeholders = get_answer_placeholders()
+ placeholder = placeholders[0]
+
+ if 'def process(self, element' in placeholder:
+ passed()
+ else:
+ failed('Override "process" method')
+
+
+def test_pardo():
+ placeholders = get_answer_placeholders()
+ placeholder = placeholders[1]
+
+ if 'beam.ParDo(EnrichCountryDoFn(),' in placeholder:
+ passed()
+ else:
+ failed('Use beam.ParDo that accepts side input')
+
+
+def test_output():
+ output = get_file_output()
+
+ answers = [
+ 'Person[Henry,Singapore,Singapore]',
+ 'Person[Jane,San Francisco,United States]',
+ 'Person[Lee,Beijing,China]',
+ 'Person[John,Sydney,Australia]',
+ 'Person[Alfred,London,United Kingdom]'
+ ]
+
+ if all(person in output for person in answers):
+ passed()
+ else:
+ failed("Incorrect output. Enrich the Person's country by the city.")
+
+
+if __name__ == '__main__':
+ run_common_tests()
+ test_dofn_process_method()
+ test_pardo()
+ test_output()
diff --git a/learning/katas/python/Core Transforms/Side Output/Side Output/task.html b/learning/katas/python/Core Transforms/Side Output/Side Output/task.html
new file mode 100644
index 0000000..b6e0543
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/Side Output/task.html
@@ -0,0 +1,44 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Side Output</h2>
+<p>
+ While ParDo always produces a main output PCollection (as the return value from apply), you can
+ also have your ParDo produce any number of additional output PCollections. If you choose to have
+ multiple outputs, your ParDo returns all of the output PCollections (including the main output)
+ bundled together.
+</p>
+<p>
+ <b>Kata:</b> Implement additional output to your ParDo for numbers bigger than 100.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.TaggedOutput">
+ pvalue.TaggedOutput</a> and
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo.with_outputs">
+ .with_outputs</a> to output multiple tagged-outputs in a
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo">
+ ParDo.</a>
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#additional-outputs">
+ "Additional outputs"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/python/Core Transforms/Side Output/Side Output/task.py b/learning/katas/python/Core Transforms/Side Output/Side Output/task.py
new file mode 100644
index 0000000..f587e1c
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/Side Output/task.py
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import apache_beam as beam
+from apache_beam import pvalue
+
+from log_elements import LogElements
+
+num_below_100_tag = 'num_below_100'
+num_above_100_tag = 'num_above_100'
+
+
+class ProcessNumbersDoFn(beam.DoFn):
+
+ def process(self, element):
+ if element <= 100:
+ yield element
+ else:
+ yield pvalue.TaggedOutput(num_above_100_tag, element)
+
+
+p = beam.Pipeline()
+
+results = \
+ (p | beam.Create([10, 50, 120, 20, 200, 0])
+ | beam.ParDo(ProcessNumbersDoFn())
+ .with_outputs(num_above_100_tag, main=num_below_100_tag))
+
+results[num_below_100_tag] | 'Log numbers <= 100' >> LogElements(prefix='Number <= 100: ')
+results[num_above_100_tag] | 'Log numbers > 100' >> LogElements(prefix='Number > 100: ')
+
+p.run()
diff --git a/learning/katas/python/Core Transforms/Side Output/Side Output/tests.py b/learning/katas/python/Core Transforms/Side Output/Side Output/tests.py
new file mode 100644
index 0000000..b525cae
--- /dev/null
+++ b/learning/katas/python/Core Transforms/Side Output/Side Output/tests.py
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from test_helper import run_common_tests, failed, passed, \
+ get_answer_placeholders, get_file_output
+
+
+def test_dofn_process_method():
+ placeholders = get_answer_placeholders()
+ placeholder = placeholders[0]
+
+ if 'pvalue.TaggedOutput' in placeholder:
+ passed()
+ else:
+ failed('Use pvalue.TaggedOutput')
+
+
+def test_pardo():
+ placeholders = get_answer_placeholders()
+ placeholder = placeholders[1]
+
+ if all(['beam.ParDo(ProcessNumbersDoFn())', '.with_outputs,']) in placeholder:
+ passed()
+ else:
+ failed('Use beam.ParDo that outputs multiple outputs')
+
+
+def test_output():
+ output = get_file_output()
+
+ numbers_below_100 = ['0', '10', '20', '50']
+ numbers_above_100 = ['120', '200']
+
+ answers = []
+
+ for num in numbers_below_100:
+ answers.append('Number <= 100: ' + num)
+
+ for num in numbers_above_100:
+ answers.append('Number > 100: ' + num)
+
+ if all(num in output for num in answers):
+ passed()
+ else:
+ failed("Incorrect output. Output the numbers to the output tags accordingly.")
+
+
+if __name__ == '__main__':
+ run_common_tests()
+ test_dofn_process_method()
+ test_pardo()
+ test_output()
diff --git a/learning/katas/python/Examples/Word Count/Word Count/task.html b/learning/katas/python/Examples/Word Count/Word Count/task.html
index 9210a56..a963aab 100644
--- a/learning/katas/python/Examples/Word Count/Word Count/task.html
+++ b/learning/katas/python/Examples/Word Count/Word Count/task.html
@@ -18,15 +18,19 @@
<html>
<h2>Word Count Pipeline</h2>
-<p>This kata is to create a pipeline that counts the number of words.</p>
-<p>For this task, please output the count of each word in the following format:<br/>
- <pre>
- word:count
- ball:5
- book:3
- </pre>
+<p>
+ <b>Kata:</b> Create a pipeline that counts the number of words.
</p>
+<p>
+ Please output the count of each word in the following format:
+</p>
+<pre>
+ word:count
+ ball:5
+ book:3
+</pre>
<br>
-<br>
-<div class='hint'>Refer to your lessons above.</div>
+<div class="hint">
+ Refer to your katas above.
+</div>
</html>
diff --git a/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html
new file mode 100644
index 0000000..55e369f
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html
@@ -0,0 +1,29 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>Built-in I/Os</h2>
+<p>
+ Beam SDKs provide many out of the box I/O transforms that can be used to read from many
+ different sources and write to many different sinks.
+</p>
+<p>
+ See the <a href="https://beam.apache.org/documentation/io/built-in/">Beam-provided I/O
+ Transforms</a> page for a list of the currently available I/O transforms.
+</p>
+</html>
diff --git a/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.py b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.py
new file mode 100644
index 0000000..6894717
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/Built-in IOs/task.py
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import apache_beam as beam
+
+p = beam.Pipeline()
+
+
+p.run()
diff --git a/learning/katas/python/IO/Built-in IOs/Built-in IOs/tests.py b/learning/katas/python/IO/Built-in IOs/Built-in IOs/tests.py
new file mode 100644
index 0000000..826d578
--- /dev/null
+++ b/learning/katas/python/IO/Built-in IOs/Built-in IOs/tests.py
@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
diff --git a/learning/katas/python/IO/TextIO/ReadFromText/countries.txt b/learning/katas/python/IO/TextIO/ReadFromText/countries.txt
new file mode 100644
index 0000000..9d6848f
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/ReadFromText/countries.txt
@@ -0,0 +1,10 @@
+Singapore
+United States
+Australia
+England
+France
+China
+Indonesia
+Mexico
+Germany
+Japan
diff --git a/learning/katas/python/IO/TextIO/ReadFromText/task.html b/learning/katas/python/IO/TextIO/ReadFromText/task.html
new file mode 100644
index 0000000..c4fc0bd
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/ReadFromText/task.html
@@ -0,0 +1,45 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<html>
+<h2>ReadFromText</h2>
+<p>
+ When you create a pipeline, you often need to read data from some external source, such as a file
+ or a database. Likewise, you may want your pipeline to output its result data to an external
+ storage system. Beam provides read and write transforms for a number of common data storage types.
+ If you want your pipeline to read from or write to a data storage format that isn’t supported by
+ the built-in transforms, you can implement your own read and write transforms.
+</p>
+<p>
+ To read a PCollection from one or more text files, use beam.io.ReadFromText to instantiate a
+ transform and specify the path of the file(s) to be read.
+</p>
+<p>
+ <b>Kata:</b> Read the 'countries.txt' file and convert each country name into uppercase.
+</p>
+<br>
+<div class="hint">
+ Use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.ReadFromText">
+ beam.io.ReadFromText</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#pipeline-io-reading-data">
+ "Reading input data"</a> section for more information.
+</div>
+</html>
diff --git a/learning/katas/python/IO/TextIO/ReadFromText/task.py b/learning/katas/python/IO/TextIO/ReadFromText/task.py
new file mode 100644
index 0000000..96dfe6f
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/ReadFromText/task.py
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import apache_beam as beam
+
+from log_elements import LogElements
+
+p = beam.Pipeline()
+
+file_path = 'countries.txt'
+
+(p | beam.io.ReadFromText(file_path)
+ | beam.Map(lambda country: country.upper())
+ | LogElements())
+
+p.run()
diff --git a/learning/katas/python/IO/TextIO/ReadFromText/tests.py b/learning/katas/python/IO/TextIO/ReadFromText/tests.py
new file mode 100644
index 0000000..413bda7
--- /dev/null
+++ b/learning/katas/python/IO/TextIO/ReadFromText/tests.py
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from test_helper import run_common_tests, failed, passed, get_answer_placeholders, get_file_output
+
+
+def test_readfromtext_method():
+ placeholders = get_answer_placeholders()
+ placeholder = placeholders[0]
+
+ if 'ReadFromText(' in placeholder:
+ passed()
+ else:
+ failed('Use beam.io.ReadFromText')
+
+
+def test_output():
+ output = get_file_output()
+
+ answers = [
+ 'AUSTRALIA',
+ 'CHINA',
+ 'ENGLAND',
+ 'FRANCE',
+ 'GERMANY',
+ 'INDONESIA',
+ 'JAPAN',
+ 'MEXICO',
+ 'SINGAPORE',
+ 'UNITED STATES'
+ ]
+
+ if all(num in output for num in answers):
+ passed()
+ else:
+ failed("Incorrect output. Convert each country name to uppercase.")
+
+
+if __name__ == '__main__':
+ run_common_tests()
+ test_readfromtext_method()
+ test_output()
diff --git a/learning/katas/python/Introduction/Hello Beam/Hello Beam/task.html b/learning/katas/python/Introduction/Hello Beam/Hello Beam/task.html
index d61899a..e71982d 100644
--- a/learning/katas/python/Introduction/Hello Beam/Hello Beam/task.html
+++ b/learning/katas/python/Introduction/Hello Beam/Hello Beam/task.html
@@ -18,9 +18,36 @@
<html>
<h2>Hello Beam Pipeline</h2>
-<p>This kata is to create a simple pipeline that takes a hardcoded input element "Hello Beam"</p>
+<p>
+ Apache Beam is an open source, unified model for defining both batch and streaming data-parallel
+ processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the
+ pipeline. The pipeline is then executed by one of Beam’s supported distributed processing
+ back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
+</p>
+<p>
+ Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the
+ problem can be decomposed into many smaller bundles of data that can be processed independently
+ and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data
+ integration. These tasks are useful for moving data between different storage media and data
+ sources, transforming data into a more desirable format, or loading data onto a new system.
+</p>
+<p>
+ To learn more about Apache Beam, refer to
+ <a href="https://beam.apache.org/get-started/beam-overview/">Apache Beam Overview</a>.
+</p>
+<p>
+ <b>Kata:</b> Your first kata is to create a simple pipeline that takes a hardcoded input element
+ "Hello Beam".
+</p>
<br>
-<br>
-<div class='hint'>Hardcoded input can be created using <a href="https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.transforms.core.html#apache_beam.transforms.core.Create">Create</a></div>
+<div class="hint">
+ Hardcoded input can be created using
+ <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Create">
+ Create</a>.
+</div>
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#creating-pcollection-in-memory">
+ "Creating a PCollection from in-memory data"</a> section for more information.
+</div>
</html>
-
diff --git a/learning/katas/python/README.md b/learning/katas/python/README.md
new file mode 100644
index 0000000..ce70831
--- /dev/null
+++ b/learning/katas/python/README.md
@@ -0,0 +1,32 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+### How to Setup
+Please follow the below steps in order to setup the project properly:
+* Using PyCharm Education (or PyCharm with EduTools plugin), select "Create New Project" and select
+this directory (learning/katas/python)
+* Select your project interpreter (e.g. virtualenv), then click "Create"
+* Click "Yes" to create project from existing sources when prompted
+* Wait for indexing to finish
+* Open "Preferences", search for "Project Interpreter", and select/add accordingly (e.g. virtualenv)
+* Open the "Project" tool window, and select the "Course" view
+* Your project is ready
+
+For further instructions on how the PyCharm Education works, you can refer
+[here](https://www.jetbrains.com/help/education/educator-start-guide.html?section=Python).
diff --git a/learning/katas/python/requirements.txt b/learning/katas/python/requirements.txt
index 9066dc8..c53a288 100644
--- a/learning/katas/python/requirements.txt
+++ b/learning/katas/python/requirements.txt
@@ -14,5 +14,5 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-apache-beam==2.11.0
-apache-beam[test]==2.11.0
+apache-beam==2.13.0
+apache-beam[test]==2.13.0
diff --git a/model/fn-execution/build.gradle b/model/fn-execution/build.gradle
index 82f4e81..42f1fe0 100644
--- a/model/fn-execution/build.gradle
+++ b/model/fn-execution/build.gradle
@@ -17,7 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-applyPortabilityNature(shadowJarValidationExcludes: ["org/apache/beam/model/fnexecution/v1/**"])
+applyPortabilityNature(
+ automaticModuleName: 'org.apache.beam.model.fn.execution',
+ shadowJarValidationExcludes: ["org/apache/beam/model/fnexecution/v1/**"]
+)
description = "Apache Beam :: Model :: Fn Execution"
ext.summary = "Portable definitions for execution user-defined functions."
diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto b/model/fn-execution/src/main/proto/beam_fn_api.proto
index c14fe14..ed2f013 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -45,32 +45,6 @@
import "google/protobuf/wrappers.proto";
import "metrics.proto";
-/*
- * Constructs that define the pipeline shape.
- *
- * These are mostly unstable due to the missing pieces to be shared with
- * the Runner Api like windowing strategy, display data, .... There are still
- * some modelling questions related to whether a side input is modelled
- * as another field on a PrimitiveTransform or as part of inputs and we
- * still are missing things like the CompositeTransform.
- */
-
-// A representation of an input or output definition on a primitive transform.
-// Stable
-message Target {
- // A repeated list of target definitions.
- message List {
- repeated Target target = 1;
- }
-
- // (Required) The id of the PrimitiveTransform which is the target.
- string primitive_transform_reference = 1;
-
- // (Required) The local name of an input or output defined on the primitive
- // transform.
- string name = 2;
-}
-
// A descriptor for connecting to a remote port using the Beam Fn Data API.
// Allows for communication between two environments (for example between the
// runner and the SDK).
@@ -196,8 +170,8 @@
// https://docs.google.com/document/d/1tUDb45sStdR8u7-jBkGdw3OGFK7aa2-V7eo86zYSE_4/edit#heading=h.9g3g5weg2u9
// for further details.
message BundleApplication {
- // (Required) The primitive transform to which to pass the element
- string ptransform_id = 1;
+ // (Required) The transform to which to pass the element
+ string transform_id = 1;
// (Required) Name of the transform's input to which to pass the element.
string input_id = 2;
@@ -209,53 +183,23 @@
// value represents a lower bound on the timestamps of elements that
// are produced by this PTransform into each of its output PCollections
// when invoked with this application.
+ //
+ // If there is no watermark reported from RestrictionTracker, the runner will
+ // use MIN_TIMESTAMP by default.
map<string, google.protobuf.Timestamp> output_watermarks = 4;
- // Represents an estimate for the amount of currently outstanding work.
- message Backlog {
- // This informs Runners on how to aggregate the backlog
- // being reported across multiple active bundles. Backlogs
- // are aggregated using the set of partitions.
- //
- // For example SplittableDoFn's which consume elements from:
- // * a globally shared resource such as a Pubsub queue should set this
- // to “”.
- // * a shared partitioned resource should use the partition identifier.
- // * a uniquely partitioned resource such as a file range should set this
- // to
- // file name + start offset.
- bytes partition = 1;
-
- // The estimate for the backlog.
- oneof value {
- // Represents an estimate for the amount of outstanding work. Values
- // compare lexicographically.
- bytes bytes = 1000;
-
- // Whether the backlog is unknown.
- bool is_unknown = 1001;
- }
- }
-
- // (Required) An estimate for the amount outstanding work related to
- // this application.
- Backlog backlog = 5;
-
// (Required) Whether this application potentially produces an unbounded
// amount of data. Note that this should only be set to BOUNDED if and
// only if the application is known to produce a finite amount of output.
- //
- // Note that this is different from the backlog as the backlog represents
- // how much work there is currently outstanding.
- org.apache.beam.model.pipeline.v1.IsBounded.Enum is_bounded = 6;
+ org.apache.beam.model.pipeline.v1.IsBounded.Enum is_bounded = 5;
// Contains additional monitoring information related to this application.
//
// Each application is able to report information that some runners
- // will use consume when providing a UI or for making scaling and performance
+ // will use when providing a UI or for making scaling and performance
// decisions. See https://s.apache.org/beam-bundles-backlog-splitting for
// details about what types of signals may be useful to report.
- repeated org.apache.beam.model.pipeline.v1.MonitoringInfo monitoring_infos = 7;
+ repeated org.apache.beam.model.pipeline.v1.MonitoringInfo monitoring_infos = 6;
}
// An Application should be scheduled for execution after a delay.
@@ -273,11 +217,34 @@
message ProcessBundleRequest {
// (Required) A reference to the process bundle descriptor that must be
// instantiated and executed by the SDK harness.
- string process_bundle_descriptor_reference = 1;
+ string process_bundle_descriptor_id = 1;
+
+ // A cache token which can be used by an SDK to check for the validity
+ // of cached elements which have a cache token associated.
+ message CacheToken {
+
+ // A flag to indicate a cache token is valid for user state.
+ message UserState {}
+
+ // A flag to indicate a cache token is valid for a side input.
+ message SideInput {
+ // The id of a side input.
+ string side_input = 1;
+ }
+
+ // The scope of a cache token.
+ oneof type {
+ UserState user_state = 1;
+ SideInput side_input = 2;
+ }
+
+ // The cache token identifier which should be globally unique.
+ bytes token = 10;
+ }
// (Optional) A list of cache tokens that can be used by an SDK to reuse
// cached data returned by the State API across multiple bundles.
- repeated bytes cache_tokens = 2;
+ repeated CacheToken cache_tokens = 2;
}
message ProcessBundleResponse {
@@ -309,7 +276,7 @@
message ProcessBundleProgressRequest {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- string instruction_reference = 1;
+ string instruction_id = 1;
}
// DEPRECATED
@@ -318,7 +285,7 @@
// These metrics are split into processed and active element groups for
// progress reporting purposes. This allows a Runner to see what is measured,
// what is estimated and what can be extrapolated to be able to accurately
- // estimate the backlog of remaining work.
+ // estimate the amount of remaining work.
message PTransform {
// Metrics that are measured for processed and active element groups.
message Measured {
@@ -446,20 +413,7 @@
message ProcessBundleSplitRequest {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- string instruction_reference = 1;
-
- // (Required) Specifies that the Runner would like the bundle to split itself
- // such that it performs no more work than the backlog specified for each
- // PTransform. The interpretation of how much work should be processed is up
- // to the PTransform.
- //
- // For example, A backlog of "" tells the SDK to perform as little work as
- // possible, effectively checkpointing when able. The remaining backlog
- // will be relative to the backlog reported during processing.
- //
- // If the backlog is unspecified for a PTransform, the runner would like
- // the SDK to process all data received for that PTransform.
- map<string, bytes> backlog_remaining = 2;
+ string instruction_id = 1;
// A message specifying the desired split for a single transform.
message DesiredSplit {
@@ -518,20 +472,17 @@
// as some range in an underlying dataset).
message ChannelSplit {
// (Required) The grpc read transform reading this channel.
- string ptransform_id = 1;
-
- // (Required) Name of the transform's input to which to pass the element.
- string input_id = 2;
+ string transform_id = 1;
// The last element of the input channel that should be entirely considered
// part of the primary, identified by its absolute index in the (ordered)
// channel.
- int32 last_primary_element = 3;
+ int32 last_primary_element = 2;
// The first element of the input channel that should be entirely considered
// part of the residual, identified by its absolute index in the (ordered)
// channel.
- int32 first_residual_element = 4;
+ int32 first_residual_element = 3;
}
// Partitions of input data channels into primary and residual elements,
@@ -544,7 +495,7 @@
message FinalizeBundleRequest {
// (Required) A reference to a completed process bundle request with the given
// instruction id.
- string instruction_reference = 1;
+ string instruction_id = 1;
}
message FinalizeBundleResponse {
@@ -559,11 +510,11 @@
// Stable
message Elements {
// Represents multiple encoded elements in nested context for a given named
- // instruction and target.
+ // instruction and transform.
message Data {
// (Required) A reference to an active instruction request with the given
// instruction id.
- string instruction_reference = 1;
+ string instruction_id = 1;
// (Required) A definition representing a consumer or producer of this data.
// If received by a harness, this represents the consumer within that
@@ -572,16 +523,15 @@
//
// Note that a single element may span multiple Data messages.
//
- // Note that a sending/receiving pair should share the same target
- // identifier.
- Target target = 2;
+ // Note that a sending/receiving pair should share the same identifier.
+ string transform_id = 2;
// (Optional) Represents a part of a logical byte stream. Elements within
// the logical byte stream are encoded in the nested context and
// concatenated together.
//
// An empty data block represents the end of stream for the given
- // instruction and target.
+ // instruction and transform.
bytes data = 3;
}
@@ -613,7 +563,7 @@
// (Required) The associated instruction id of the work that is currently
// being processed. This allows for the runner to associate any modifications
// to state to be committed with the appropriate work execution.
- string instruction_reference = 2;
+ string instruction_id = 2;
// (Required) The state key this request is for.
StateKey state_key = 3;
@@ -642,10 +592,6 @@
// failed.
string error = 2;
- // (Optional) If this is specified, then the result of this state request
- // can be cached using the supplied token.
- bytes cache_token = 3;
-
// A corresponding response matching the request will be populated.
oneof response {
// A response to getting state.
@@ -674,12 +620,17 @@
message Runner {
// (Required) Opaque information supplied by the runner. Used to support
// remote references.
+ // https://s.apache.org/beam-fn-api-send-and-receive-data
+ //
+ // Used by state backed iterable. And in this use case, request type can
+ // only be of type get. Details see:
+ // https://s.apache.org/beam-fn-api-state-backed-iterables
bytes key = 1;
}
message MultimapSideInput {
// (Required) The id of the PTransform containing a side input.
- string ptransform_id = 1;
+ string transform_id = 1;
// (Required) The id of the side input.
string side_input_id = 2;
// (Required) The window (after mapping the currently executing elements
@@ -691,7 +642,7 @@
message BagUserState {
// (Required) The id of the PTransform containing user state.
- string ptransform_id = 1;
+ string transform_id = 1;
// (Required) The id of the user state.
string user_state_id = 2;
// (Required) The window encoded in a nested context.
@@ -818,11 +769,11 @@
// (Optional) A reference to the instruction this log statement is associated
// with.
- string instruction_reference = 5;
+ string instruction_id = 5;
- // (Optional) A reference to the primitive transform this log statement is
+ // (Optional) A reference to the transform this log statement is
// associated with.
- string primitive_transform_reference = 6;
+ string transform_id = 6;
// (Optional) Human-readable name of the function or method being invoked,
// with optional context such as the class or package name. The format can
@@ -851,7 +802,7 @@
stream LogControl) {}
}
-message NotifyRunnerAvailableRequest {
+message StartWorkerRequest {
string worker_id = 1;
org.apache.beam.model.pipeline.v1.ApiServiceDescriptor control_endpoint = 2;
org.apache.beam.model.pipeline.v1.ApiServiceDescriptor logging_endpoint = 3;
@@ -860,11 +811,21 @@
map<string, string> params = 10;
}
-message NotifyRunnerAvailableResponse {
+message StartWorkerResponse {
+ string error = 1;
+}
+
+message StopWorkerRequest {
+ string worker_id = 1;
+}
+
+message StopWorkerResponse {
string error = 1;
}
service BeamFnExternalWorkerPool {
- rpc NotifyRunnerAvailable(NotifyRunnerAvailableRequest)
- returns (NotifyRunnerAvailableResponse) {}
+ // Start the SDK worker with the given ID.
+ rpc StartWorker (StartWorkerRequest) returns (StartWorkerResponse) {}
+ // Stop the SDK worker.
+ rpc StopWorker (StopWorkerRequest) returns (StopWorkerResponse) {}
}
diff --git a/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml b/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
index d53b933..ecae656 100644
--- a/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
+++ b/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
@@ -58,6 +58,14 @@
---
coder:
+ urn: "beam:coder:bool:v1"
+examples:
+ "\0": False
+ "\u0001": True
+
+---
+
+coder:
urn: "beam:coder:string_utf8:v1"
nested: false
examples:
@@ -126,6 +134,16 @@
---
coder:
+ urn: "beam:coder:kv:v1"
+ components: [{urn: "beam:coder:bytes:v1"},
+ {urn: "beam:coder:bool:v1"}]
+examples:
+ "\u0003abc\u0001": {key: abc, value: True}
+ "\u0004ab\0c\0": {key: "ab\0c", value: False}
+
+---
+
+coder:
urn: "beam:coder:interval_window:v1"
examples:
"\u0080\u0000\u0001\u0052\u009a\u00a4\u009b\u0068\u0080\u00dd\u00db\u0001" : {end: 1454293425000, span: 3600000}
@@ -157,6 +175,16 @@
coder:
urn: "beam:coder:iterable:v1"
+ components: [{urn: "beam:coder:bool:v1"}]
+examples:
+ "\0\0\0\u0001\u0001": [True]
+ "\0\0\0\u0002\u0001\0": [True, False]
+ "\0\0\0\0": []
+
+---
+
+coder:
+ urn: "beam:coder:iterable:v1"
components: [{urn: "beam:coder:bytes:v1"}]
# This is for iterables of unknown length, where the encoding is not
# deterministic.
diff --git a/model/job-management/build.gradle b/model/job-management/build.gradle
index 38c7938..1568d8b 100644
--- a/model/job-management/build.gradle
+++ b/model/job-management/build.gradle
@@ -17,10 +17,12 @@
*/
plugins { id 'org.apache.beam.module' }
-applyPortabilityNature(shadowJarValidationExcludes:[
- "org/apache/beam/model/expansion/v1/**",
- "org/apache/beam/model/jobmanagement/v1/**",
-])
+applyPortabilityNature(
+ automaticModuleName: 'org.apache.beam.model.job.management',
+ shadowJarValidationExcludes: [
+ "org/apache/beam/model/expansion/v1/**",
+ "org/apache/beam/model/jobmanagement/v1/**",
+ ])
description = "Apache Beam :: Model :: Job Management"
ext.summary = "Portable definitions for submitting pipelines."
diff --git a/model/job-management/src/main/proto/beam_artifact_api.proto b/model/job-management/src/main/proto/beam_artifact_api.proto
index e036315..34eb389 100644
--- a/model/job-management/src/main/proto/beam_artifact_api.proto
+++ b/model/job-management/src/main/proto/beam_artifact_api.proto
@@ -60,10 +60,6 @@
// (Optional) The Unix-like permissions of the artifact
uint32 permissions = 2;
- // (Optional) The base64-encoded md5 checksum of the artifact. Used, among other things, by
- // harness boot code to validate the integrity of the artifact.
- string md5X = 3;
-
// (Optional) The hex-encoded sha256 checksum of the artifact. Used, among other things, by
// harness boot code to validate the integrity of the artifact.
string sha256 = 4;
diff --git a/model/job-management/src/main/proto/beam_job_api.proto b/model/job-management/src/main/proto/beam_job_api.proto
index c7b972e..2ebc1de 100644
--- a/model/job-management/src/main/proto/beam_job_api.proto
+++ b/model/job-management/src/main/proto/beam_job_api.proto
@@ -43,9 +43,15 @@
// Submit the job for execution
rpc Run (RunJobRequest) returns (RunJobResponse);
+ // Get a list of all invoked jobs
+ rpc GetJobs (GetJobsRequest) returns (GetJobsResponse);
+
// Get the current state of the job
rpc GetState (GetJobStateRequest) returns (GetJobStateResponse);
+ // Get the job's pipeline
+ rpc GetPipeline (GetJobPipelineRequest) returns (GetJobPipelineResponse);
+
// Cancel the job
rpc Cancel (CancelJobRequest) returns (CancelJobResponse);
@@ -120,6 +126,22 @@
JobState.Enum state = 1; // (required)
}
+// A subset of info provided by ProvisionApi.ProvisionInfo
+message JobInfo {
+ string job_id = 1; // (required)
+ string job_name = 2; // (required)
+ google.protobuf.Struct pipeline_options = 3; // (required)
+ JobState.Enum state = 4; // (required)
+}
+
+// GetJobs is a synchronus request that returns a list of invoked jobs back
+// Throws error GRPC_STATUS_UNAVAILABLE if server is down
+message GetJobsRequest { }
+
+message GetJobsResponse {
+ repeated JobInfo job_info = 1; // (required)
+}
+
// GetState is a synchronus request that returns a job state back
// Throws error GRPC_STATUS_UNAVAILABLE if server is down
@@ -134,6 +156,19 @@
}
+// GetPipeline is a synchronus request that returns a pipeline back
+// Throws error GRPC_STATUS_UNAVAILABLE if server is down
+// Throws error NOT_FOUND if the jobId is not found
+message GetJobPipelineRequest {
+ string job_id = 1; // (required)
+
+}
+
+message GetJobPipelineResponse {
+ org.apache.beam.model.pipeline.v1.Pipeline pipeline = 1; // (required)
+}
+
+
// GetJobMessages is a streaming api for streaming job messages from the service
// One request will connect you to the job and you'll get a stream of job state
// and job messages back; one is used for logging and the other for detecting
diff --git a/model/pipeline/build.gradle b/model/pipeline/build.gradle
index a305985..7698e84 100644
--- a/model/pipeline/build.gradle
+++ b/model/pipeline/build.gradle
@@ -17,7 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-applyPortabilityNature(shadowJarValidationExcludes: ["org/apache/beam/model/pipeline/v1/**"])
+applyPortabilityNature(
+ automaticModuleName: 'org.apache.beam.model.pipeline',
+ shadowJarValidationExcludes: ["org/apache/beam/model/pipeline/v1/**"]
+)
description = "Apache Beam :: Model :: Pipeline"
ext.summary = "Portable definitions for building pipelines"
diff --git a/model/pipeline/src/main/proto/beam_runner_api.proto b/model/pipeline/src/main/proto/beam_runner_api.proto
index 75fc0cd..ec05ef0 100644
--- a/model/pipeline/src/main/proto/beam_runner_api.proto
+++ b/model/pipeline/src/main/proto/beam_runner_api.proto
@@ -124,7 +124,7 @@
// For some special composite transforms, the payload is also officially
// defined:
//
- // - when the URN is "urn:beam:transforms:combine" it is a CombinePayload
+ // - when the URN is "beam:transforms:combine" it is a CombinePayload
//
FunctionSpec spec = 1;
@@ -170,7 +170,7 @@
// Represents Beam's parallel do operation.
// Payload: ParDoPayload.
// TODO(BEAM-3595): Change this to beam:transform:pardo:v1.
- PAR_DO = 0 [(beam_urn) = "urn:beam:transform:pardo:v1"];
+ PAR_DO = 0 [(beam_urn) = "beam:transform:pardo:v1"];
// Represents Beam's flatten operation.
// Payload: None.
@@ -189,7 +189,7 @@
// Represents the TestStream.
// Payload: TestStreamPayload
- TEST_STREAM = 5 [(beam_urn) = "urn:beam:transform:teststream:v1"];
+ TEST_STREAM = 5 [(beam_urn) = "beam:transform:teststream:v1"];
// Represents mapping of main input window onto side input window.
//
@@ -251,34 +251,29 @@
}
// Payload for all of these: CombinePayload
enum CombineComponents {
- // TODO(BEAM-6199): Remove these old URNs.
- COMBINE_PGBKCV = 0 [(beam_urn) = "beam:transform:combine_pgbkcv:v1"];
- COMBINE_MERGE_ACCUMULATORS = 1 [(beam_urn) = "beam:transform:combine_merge_accumulators:v1"];
- COMBINE_EXTRACT_OUTPUTS = 2 [(beam_urn) = "beam:transform:combine_extract_outputs:v1"];
-
// Represents the Pre-Combine part of a lifted Combine Per Key, as described
// in the following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.ta0g6ase8z07
// Payload: CombinePayload
- COMBINE_PER_KEY_PRECOMBINE = 3 [(beam_urn) = "beam:transform:combine_per_key_precombine:v1"];
+ COMBINE_PER_KEY_PRECOMBINE = 0 [(beam_urn) = "beam:transform:combine_per_key_precombine:v1"];
// Represents the Merge Accumulators part of a lifted Combine Per Key, as
// described in the following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.jco9rvatld5m
// Payload: CombinePayload
- COMBINE_PER_KEY_MERGE_ACCUMULATORS = 4 [(beam_urn) = "beam:transform:combine_per_key_merge_accumulators:v1"];
+ COMBINE_PER_KEY_MERGE_ACCUMULATORS = 1 [(beam_urn) = "beam:transform:combine_per_key_merge_accumulators:v1"];
// Represents the Extract Outputs part of a lifted Combine Per Key, as
// described in the following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.i9i6p8gtl6ku
// Payload: CombinePayload
- COMBINE_PER_KEY_EXTRACT_OUTPUTS = 5 [(beam_urn) = "beam:transform:combine_per_key_extract_outputs:v1"];
+ COMBINE_PER_KEY_EXTRACT_OUTPUTS = 2 [(beam_urn) = "beam:transform:combine_per_key_extract_outputs:v1"];
// Represents the Combine Grouped Values transform, as described in the
// following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.aj86ew4v1wk
// Payload: CombinePayload
- COMBINE_GROUPED_VALUES = 6 [(beam_urn) = "beam:transform:combine_grouped_values:v1"];
+ COMBINE_GROUPED_VALUES = 3 [(beam_urn) = "beam:transform:combine_grouped_values:v1"];
}
// Payload for all of these: ParDoPayload containing the user's SDF
enum SplittableParDoComponents {
@@ -416,7 +411,7 @@
message StateSpec {
oneof spec {
- ValueStateSpec value_spec = 1;
+ ReadModifyWriteStateSpec read_modify_write_spec = 1;
BagStateSpec bag_spec = 2;
CombiningStateSpec combining_spec = 3;
MapStateSpec map_spec = 4;
@@ -424,7 +419,7 @@
}
}
-message ValueStateSpec {
+message ReadModifyWriteStateSpec {
string coder_id = 1;
}
@@ -545,8 +540,8 @@
// may be a cross-language agreed-upon format, or it may be a "custom coder"
// that can only be used by a particular SDK. It does not include component
// coders, as it is beneficial for these to be comprehensible to a runner
- // regardless of whether the binary format is agree-upon.
- SdkFunctionSpec spec = 1;
+ // regardless of whether the binary format is agreed-upon.
+ FunctionSpec spec = 1;
// (Optional) If this coder is parametric, such as ListCoder(VarIntCoder),
// this is a list of the components. In order for encodings to be identical,
@@ -565,6 +560,9 @@
// Components: The key and value coder, in that order.
KV = 1 [(beam_urn) = "beam:coder:kv:v1"];
+ // Components: None
+ BOOL = 12 [(beam_urn) = "beam:coder:bool:v1"];
+
// Variable length Encodes a 64-bit integer.
// Components: None
VARINT = 2 [(beam_urn) = "beam:coder:varint:v1"];
@@ -584,7 +582,7 @@
// If the length is unknown, it is batched up into groups of size b1..bM
// and encoded as
//
- // fixed32(0)
+ // fixed32(-1)
// varInt64(b1) encode(e1) encode(e2) ... encode(e_b1)
// varInt64(b2) encode(e_(b1+1)) encode(e_(b1+2)) ... encode(e_(b1+b2))
// ...
@@ -650,61 +648,6 @@
}
}
-// Experimental: A representation of a Beam Schema.
-message Schema {
- enum TypeName {
- BYTE = 0;
- INT16 = 1;
- INT32 = 2;
- INT64 = 3;
- DECIMAL = 4;
- FLOAT = 5;
- DOUBLE = 6;
- STRING = 7;
- DATETIME = 8;
- BOOLEAN = 9;
- BYTES = 10;
- ARRAY = 11;
- MAP = 13;
- ROW = 14;
- LOGICAL_TYPE = 15;
- }
-
- message LogicalType {
- string id = 1;
- string args = 2;
- FieldType base_type = 3;
- bytes serialized_class = 4;
- }
-
- message MapType {
- FieldType key_type = 1;
- FieldType value_type = 2;
- }
-
- message FieldType {
- TypeName type_name = 1;
- bool nullable = 2;
- oneof type_info {
- FieldType collection_element_type = 3;
- MapType map_type = 4;
- Schema row_schema = 5;
- LogicalType logical_type = 6;
- }
- }
-
- message Field {
- string name = 1;
- string description = 2;
- FieldType type = 3;
- int32 id = 4;
- int32 encoding_position = 5;
- }
-
- repeated Field fields = 1;
- string id = 2;
-}
-
// A windowing strategy describes the window function, triggering, allowed
// lateness, and accumulation mode for a PCollection.
//
@@ -794,6 +737,9 @@
// The aggregation is accumulated across outputs
ACCUMULATING = 2;
+
+ // The aggregation emits retractions when it is output
+ RETRACTING = 3;
}
}
@@ -1006,7 +952,7 @@
// interface for accessing a side input.
//
// The only access pattern intended for Beam, because of its superior
- // performance possibilities, is "urn:beam:sideinput:multimap" (or some such
+ // performance possibilities, is "beam:sideinput:multimap" (or some such
// URN)
FunctionSpec access_pattern = 1;
@@ -1126,12 +1072,12 @@
// one should bear in mind:
//
// 1. The runner understands the URN. For example, it might be
-// a well-known URN like "urn:beam:transform:Top" or
-// "urn:beam:windowfn:FixedWindows" with
+// a well-known URN like "beam:transform:Top" or
+// "beam:windowfn:FixedWindows" with
// an agreed-upon payload (e.g. a number or duration,
// respectively).
// 2. The runner does not understand the URN. It might be an
-// SDK specific URN such as "urn:beam:dofn:javasdk:1.0"
+// SDK specific URN such as "beam:dofn:javasdk:1.0"
// that indicates to the SDK what the payload is,
// such as a serialized Java DoFn from a particular
// version of the Beam Java SDK. The payload will often
diff --git a/model/pipeline/src/main/proto/schema.proto b/model/pipeline/src/main/proto/schema.proto
new file mode 100644
index 0000000..e420e3c
--- /dev/null
+++ b/model/pipeline/src/main/proto/schema.proto
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// ** Experimental **
+// Protocol Buffers describing Beam Schemas, a portable representation for
+// complex types.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaApi";
+
+message Schema {
+ repeated Field fields = 1;
+ string id = 2;
+}
+
+message Field {
+ string name = 1;
+ string description = 2;
+ FieldType type = 3;
+ int32 id = 4;
+ int32 encoding_position = 5;
+}
+
+message FieldType {
+ bool nullable = 1;
+ oneof type_info {
+ AtomicType atomic_type = 2;
+ ArrayType array_type = 3;
+ MapType map_type = 4;
+ RowType row_type = 5;
+ LogicalType logical_type = 6;
+ }
+}
+
+enum AtomicType {
+ UNSPECIFIED = 0;
+ BYTE = 1;
+ INT16 = 2;
+ INT32 = 3;
+ INT64 = 4;
+ FLOAT = 5;
+ DOUBLE = 6;
+ STRING = 7;
+ BOOLEAN = 8;
+ BYTES = 9;
+}
+
+message ArrayType {
+ FieldType element_type = 1;
+}
+
+message MapType {
+ FieldType key_type = 1;
+ FieldType value_type = 2;
+}
+
+message RowType {
+ Schema schema = 1;
+}
+
+message LogicalType {
+ string urn = 1;
+ bytes payload = 2;
+ FieldType representation = 3;
+}
diff --git a/ownership/JAVA_DEPENDENCY_OWNERS.yaml b/ownership/JAVA_DEPENDENCY_OWNERS.yaml
index d3cf107..6602e4d 100644
--- a/ownership/JAVA_DEPENDENCY_OWNERS.yaml
+++ b/ownership/JAVA_DEPENDENCY_OWNERS.yaml
@@ -69,11 +69,6 @@
artifact: aws-java-sdk-kinesis
owners:
- com.amazonaws:aws-java-sdk-kinesis:
- group: com.amazonaws
- artifact: aws-java-sdk-kinesis
- owners:
-
com.amazonaws:aws-java-sdk-s3:
group: com.amazonaws
artifact: aws-java-sdk-s3
@@ -269,11 +264,6 @@
artifact: google-cloud-spanner
owners:
- com.google.cloud:google-cloud-spanner:
- group: com.google.cloud
- artifact: google-cloud-spanner
- owners:
-
com.google.cloud.bigdataoss:gcsio:
group: com.google.cloud.bigdataoss
artifact: gcsio
@@ -529,11 +519,6 @@
artifact: propdeps-plugin
owners:
- io.spring.gradle:propdeps-plugin:
- group: io.spring.gradle
- artifact: propdeps-plugin
- owners:
-
javax.xml.bind:jaxb-api:
group: javax.xml.bind
artifact: jaxb-api
diff --git a/ownership/PYTHON_DEPENDENCY_OWNERS.yaml b/ownership/PYTHON_DEPENDENCY_OWNERS.yaml
index ba81490..865afe4 100644
--- a/ownership/PYTHON_DEPENDENCY_OWNERS.yaml
+++ b/ownership/PYTHON_DEPENDENCY_OWNERS.yaml
@@ -69,6 +69,9 @@
mock:
owners:
+ pymongo:
+ owners: yichi
+
oauth2client:
owners:
diff --git a/project-mappings b/project-mappings
index a211dcf..f8ac258 100644
--- a/project-mappings
+++ b/project-mappings
@@ -1,18 +1,23 @@
:beam-website :website
:beam-vendor-sdks-java-extensions-protobuf :vendor:sdks-java-extensions-protobuf
-:beam-vendor-guava-20_0 :vendor:guava-20_0
-:beam-vendor-grpc-1_13_1 :vendor:grpc-1_13_1
-:beam-sdks-python-test-suites-tox-py36 :sdks:python:test-suites:tox:py37
-:beam-sdks-python-test-suites-tox-py36 :sdks:python:test-suites:tox:py36
-:beam-sdks-python-test-suites-tox-py35 :sdks:python:test-suites:tox:py35
+:beam-vendor-guava-26_0-jre :vendor:guava-26_0-jre
+:beam-vendor-grpc-1_21_0 :vendor:grpc-1_21_0
:beam-sdks-python-test-suites-direct-py35 :sdks:python:test-suites:direct:py35
:beam-sdks-python-test-suites-direct-py36 :sdks:python:test-suites:direct:py36
+:beam-sdks-python-test-suites-direct-py37 :sdks:python:test-suites:direct:py37
+:beam-sdks-python-test-suites-dataflow :sdks:python:test-suites:dataflow
:beam-sdks-python-test-suites-dataflow-py35 :sdks:python:test-suites:dataflow:py35
:beam-sdks-python-test-suites-dataflow-py36 :sdks:python:test-suites:dataflow:py36
-:beam-sdks-python-test-suites-dataflow :sdks:python:test-suites:dataflow
+:beam-sdks-python-test-suites-dataflow-py37 :sdks:python:test-suites:dataflow:py37
+:beam-sdks-python-test-suites-tox-py35 :sdks:python:test-suites:tox:py35
+:beam-sdks-python-test-suites-tox-py36 :sdks:python:test-suites:tox:py36
+:beam-sdks-python-test-suites-tox-py37 :sdks:python:test-suites:tox:py37
:beam-sdks-python-load-tests :sdks:python:apache_beam:testing:load_tests
-:beam-sdks-python-container-py3 :sdks:python:container:py3
:beam-sdks-python-container :sdks:python:container
+:beam-sdks-python-container-py2 :sdks:python:container:py2
+:beam-sdks-python-container-py35 :sdks:python:container:py35
+:beam-sdks-python-container-py36 :sdks:python:container:py36
+:beam-sdks-python-container-py37 :sdks:python:container:py37
:beam-sdks-python :sdks:python
:beam-sdks-java-test-utils :sdks:java:testing:test-utils
:beam-sdks-java-nexmark :sdks:java:testing:nexmark
@@ -89,9 +94,9 @@
:beam-runners-google-cloud-dataflow-java-examples :runners:google-cloud-dataflow-java:examples
:beam-runners-google-cloud-dataflow-java :runners:google-cloud-dataflow-java
:beam-runners-gearpump :runners:gearpump
-:beam-runners-flink_2.11-job-server-container :runners:flink:1.5:job-server-container
-:beam-runners-flink_2.11-job-server :runners:flink:1.5:job-server
-:beam-runners-flink_2.11 :runners:flink:1.5
+:beam-runners-flink_2.11-job-server-container :runners:flink:1.8:job-server-container
+:beam-runners-flink_2.11-job-server :runners:flink:1.8:job-server
+:beam-runners-flink_2.11 :runners:flink:1.8
:beam-runners-flink-1.7-job-server-container :runners:flink:1.7:job-server-container
:beam-runners-flink-1.7-job-server :runners:flink:1.7:job-server
:beam-runners-flink-1.7 :runners:flink:1.7
diff --git a/release/build.gradle b/release/build.gradle
index c5228ba..44e9f98 100644
--- a/release/build.gradle
+++ b/release/build.gradle
@@ -34,7 +34,7 @@
dependsOn ":runners:google-cloud-dataflow-java:runQuickstartJavaDataflow"
dependsOn ":runners:apex:runQuickstartJavaApex"
dependsOn ":runners:spark:runQuickstartJavaSpark"
- dependsOn ":runners:flink:1.5:runQuickstartJavaFlinkLocal"
+ dependsOn ":runners:flink:1.8:runQuickstartJavaFlinkLocal"
dependsOn ":runners:direct-java:runMobileGamingJavaDirect"
dependsOn ":runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow"
}
diff --git a/release/src/main/groovy/TestScripts.groovy b/release/src/main/groovy/TestScripts.groovy
index a36be99..9edca99 100644
--- a/release/src/main/groovy/TestScripts.groovy
+++ b/release/src/main/groovy/TestScripts.groovy
@@ -157,7 +157,7 @@
pb.redirectErrorStream(true)
def proc = pb.start()
String output_text = ""
- def text = StringBuilder.newInstance()
+ def text = new StringBuilder()
proc.inputStream.eachLine {
println it
text.append(it + "\n")
diff --git a/release/src/main/python-release/python_release_automation.sh b/release/src/main/python-release/python_release_automation.sh
index 8b305ae..4c52762 100755
--- a/release/src/main/python-release/python_release_automation.sh
+++ b/release/src/main/python-release/python_release_automation.sh
@@ -19,12 +19,10 @@
source release/src/main/python-release/run_release_candidate_python_quickstart.sh
source release/src/main/python-release/run_release_candidate_python_mobile_gaming.sh
-run_release_candidate_python_quickstart "tar" "python2.7"
-run_release_candidate_python_mobile_gaming "tar" "python2.7"
-run_release_candidate_python_quickstart "wheel" "python2.7"
-run_release_candidate_python_mobile_gaming "wheel" "python2.7"
-
-run_release_candidate_python_quickstart "tar" "python3.5"
-run_release_candidate_python_mobile_gaming "tar" "python3.5"
-run_release_candidate_python_quickstart "wheel" "python3.5"
-run_release_candidate_python_mobile_gaming "wheel" "python3.5"
+for version in 2.7 3.5 3.6 3.7
+do
+ run_release_candidate_python_quickstart "tar" "python${version}"
+ run_release_candidate_python_mobile_gaming "tar" "python${version}"
+ run_release_candidate_python_quickstart "wheel" "python${version}"
+ run_release_candidate_python_mobile_gaming "wheel" "python${version}"
+done
diff --git a/release/src/main/python-release/python_release_automation_utils.sh b/release/src/main/python-release/python_release_automation_utils.sh
index 07653e6..a715c4c 100644
--- a/release/src/main/python-release/python_release_automation_utils.sh
+++ b/release/src/main/python-release/python_release_automation_utils.sh
@@ -81,11 +81,9 @@
# $2 - python interpreter version: python2.7, python3.5, ...
#######################################
function download_files() {
- VERSION=$(get_version)
-
if [[ $1 = *"wheel"* ]]; then
if [[ $2 == "python2.7" ]]; then
- BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp27-cp27mu-manylinux1_x86_64.whl"
+ BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp27-cp27mu-manylinux1_x86_64.whl"
elif [[ $2 == "python3.5" ]]; then
BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp35-cp35m-manylinux1_x86_64.whl"
elif [[ $2 == "python3.6" ]]; then
@@ -218,9 +216,11 @@
# None
#######################################
function cleanup_pubsub() {
- gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1
- gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2
- gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION
+ # Suppress error and pass quietly if topic/subscription not exists. We don't want the script
+ # to be interrupted in this case.
+ gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC1 2> /dev/null || true
+ gcloud pubsub topics delete --project=$PROJECT_ID $PUBSUB_TOPIC2 2> /dev/null || true
+ gcloud pubsub subscriptions delete --project=$PROJECT_ID $PUBSUB_SUBSCRIPTION 2> /dev/null || true
}
@@ -321,7 +321,7 @@
# Python RC configurations
VERSION=$(get_version)
-RC_STAGING_URL="https://dist.apache.org/repos/dist/dev/beam/$VERSION/"
+RC_STAGING_URL="https://dist.apache.org/repos/dist/dev/beam/$VERSION/python"
# Cloud Configurations
PROJECT_ID='apache-beam-testing'
diff --git a/release/src/main/python-release/run_release_candidate_python_quickstart.sh b/release/src/main/python-release/run_release_candidate_python_quickstart.sh
index 4700d30..6d1d4b3 100755
--- a/release/src/main/python-release/run_release_candidate_python_quickstart.sh
+++ b/release/src/main/python-release/run_release_candidate_python_quickstart.sh
@@ -37,6 +37,9 @@
ASC_FILE_NAME=$BEAM_PYTHON_SDK_ZIP".asc"
SHA512_FILE_NAME=$BEAM_PYTHON_SDK_ZIP".sha512"
+# Cleanup Pubsub once script exits
+trap cleanup_pubsub EXIT
+
#######################################
# Remove temp directory when complete.
@@ -116,7 +119,7 @@
--num_workers $NUM_WORKERS \
--sdk_location $BEAM_PYTHON_SDK
-# verify results.
+ # verify results.
wordcount_output_in_gcs="gs://$BUCKET_NAME/$WORDCOUNT_OUTPUT"
gcs_pull_result=$(gsutil ls gs://$BUCKET_NAME)
if [[ $gcs_pull_result != *$wordcount_output_in_gcs* ]]; then
@@ -139,6 +142,7 @@
# None
#######################################
function verify_streaming_wordcount_direct() {
+ cleanup_pubsub
create_pubsub
print_separator "Running Streaming wordcount example with DirectRunner"
python -m apache_beam.examples.streaming_wordcount \
@@ -148,12 +152,10 @@
pid=$!
sleep 15
-# verify result
+ # verify result
run_pubsub_publish
verify_steaming_result "DirectRunner" $pid
-# Delete the pubsub topics and subscription before running the second job. Will recreate them in the second job.
- cleanup_pubsub
kill -9 $pid
sleep 10
}
@@ -168,6 +170,7 @@
# None
#######################################
function verify_streaming_wordcount_dataflow() {
+ cleanup_pubsub
create_pubsub
print_separator "Running Streaming wordcount example with DataflowRunner "
python -m apache_beam.examples.streaming_wordcount \
@@ -193,7 +196,6 @@
kill -9 $pid
gcloud dataflow jobs cancel $running_job
- cleanup_pubsub
}
diff --git a/release/src/main/scripts/build_release_candidate.sh b/release/src/main/scripts/build_release_candidate.sh
index 0af6bb1..93add23 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -20,7 +20,8 @@
# 1. Build and stage java artifacts
# 2. Stage source release on dist.apache.org
# 3. Stage python binaries
-# 4. Create a PR to update beam-site
+# 4. Stage SDK docker images
+# 5. Create a PR to update beam-site
set -e
@@ -44,6 +45,7 @@
BEAM_ROOT_DIR=beam
WEBSITE_ROOT_DIR=beam-site
+PYTHON_VER=("python2.7" "python3.5" "python3.6" "python3.7")
echo "================Setting Up Environment Variables==========="
echo "Which release version are you working on: "
@@ -198,6 +200,45 @@
rm -rf ~/${PYTHON_ARTIFACTS_DIR}
fi
+echo "[Current Step]: Stage SDK docker images"
+echo "Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+ echo "============Staging SDK docker images on docker hub========="
+ cd ~
+ if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
+ rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+ fi
+ mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
+ cd ${LOCAL_PYTHON_STAGING_DIR}
+
+ echo '-------------------Cloning Beam Release Branch-----------------'
+ git clone ${GIT_REPO_URL}
+ cd ${BEAM_ROOT_DIR}
+ git checkout ${RELEASE_BRANCH}
+
+ echo '-------------------Generating and Pushing Python images-----------------'
+ ./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc${RC_NUM}
+ for ver in "${PYTHON_VER[@]}"; do
+ docker push apachebeam/${ver}_sdk:${RELEASE}_rc${RC_NUM} &
+ done
+
+ echo '-------------------Generating and Pushing Java images-----------------'
+ ./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}_rc${RC_NUM}
+
+ echo '-------------------Generating and Pushing Go images-----------------'
+ ./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}_rc${RC_NUM}
+
+ rm -rf ~/${PYTHON_ARTIFACTS_DIR}
+
+ echo '-------------------Clean up images at local-----------------'
+ for ver in "${PYTHON_VER[@]}"; do
+ docker rmi -f apachebeam/${ver}_sdk:${RELEASE}_rc${RC_NUM}
+ done
+ docker rmi -f apachebeam/java_sdk:${RELEASE}_rc${RC_NUM}
+ docker rmi -f apachebeam/go_sdk:${RELEASE}_rc${RC_NUM}
+fi
+
echo "[Current Step]: Update beam-site"
echo "Do you want to proceed? [y|N]"
read confirmation
@@ -280,15 +321,13 @@
echo "3. Website pull request published the Java API reference manual the Python API reference manual."
echo "==============Things Needed To Be Done Manually=============="
-echo "1.You need to update website updates PR with a new commit: "
+echo "1.Make sure a pull request is created to update the javadoc and pydoc to the beam-site: "
echo " - cd ~/${LOCAL_WEBSITE_UPDATE_DIR}/${LOCAL_WEBSITE_REPO}/${WEBSITE_ROOT_DIR}"
echo " - git checkout updates_release_${RELEASE}"
-echo " - Add new release into src/get-started/downloads.md "
+echo " - Check if both javadoc/ and pydoc/ exist."
echo " - commit your changes"
-echo "2.You need to update website updates PR with another commit: src/get-started/downloads.md"
-echo " - add new release download links like commit: "
-echo " https://github.com/apache/beam-site/commit/29394625ce54f0c5584c3db730d3eb6bf365a80c#diff-abdcc989e94369c2324cf64b66659eda"
-echo " - update last release download links from release to archive like commit: "
-echo " https://github.com/apache/beam-site/commit/6b9bdb31324d5c0250a79224507da0ea7ae8ccbf#diff-abdcc989e94369c2324cf64b66659eda"
+echo "2.Create a pull request to update the release in the beam/website:"
+echo " - An example pull request：https://github.com/apache/beam/pull/9341"
+echo " - You can find the release note in JIRA: https://issues.apache.org/jira/projects/BEAM?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page&status=unreleased"
echo "3.You need to build Python Wheels."
echo "4.Start the review-and-vote thread on the dev@ mailing list."
diff --git a/release/src/main/scripts/cut_release_branch.sh b/release/src/main/scripts/cut_release_branch.sh
index 7a24ae4..b3bdc83 100755
--- a/release/src/main/scripts/cut_release_branch.sh
+++ b/release/src/main/scripts/cut_release_branch.sh
@@ -124,6 +124,7 @@
echo ${RELEASE_BRANCH}
echo "==============================================================="
+sed -i -e "s/${DEV}/${RELEASE}/g" gradle.properties
sed -i -e "s/${DEV}/${RELEASE}/g" sdks/python/apache_beam/version.py
# TODO: [BEAM-4767]
sed -i -e "s/'beam-master-.*'/'beam-${RELEASE}'/g" runners/google-cloud-dataflow-java/build.gradle
@@ -140,6 +141,7 @@
exit
fi
+git add gradle.properties
git add sdks/python/apache_beam/version.py
git add runners/google-cloud-dataflow-java/build.gradle
git commit -m "Create release branch for version ${RELEASE}."
diff --git a/release/src/main/scripts/preparation_before_release.sh b/release/src/main/scripts/preparation_before_release.sh
index 1d02976..9d77d3a 100755
--- a/release/src/main/scripts/preparation_before_release.sh
+++ b/release/src/main/scripts/preparation_before_release.sh
@@ -36,6 +36,7 @@
echo "===============Generating new GPG key================"
sudo apt-get install rng-tools
sudo rngd -r /dev/urandom
+ echo "NOTE: When creating the key, please select the type to be RSA and RSA (default), and the size to be 4096 bit long."
gpg --full-generate-key
fi
diff --git a/release/src/main/scripts/publish_docker_images.sh b/release/src/main/scripts/publish_docker_images.sh
new file mode 100644
index 0000000..c55c79a
--- /dev/null
+++ b/release/src/main/scripts/publish_docker_images.sh
@@ -0,0 +1,77 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script will generate and publish docker images for each language version to Docker Hub:
+# 1. Generate images tagged with :{RELEASE}
+# 2. Publish images tagged with :{RELEASE}
+# 3. Tag images with :latest tag and publish.
+# 4. Clearn up images.
+
+set -e
+
+source release/src/main/scripts/build_release_candidate.sh
+
+echo "Publish SDK docker images to Docker Hub."
+echo "Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+ echo "============Publishing SDK docker images on docker hub========="
+ cd ~
+ if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
+ rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+ fi
+ mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
+ cd ${LOCAL_PYTHON_STAGING_DIR}
+
+ echo '-------------------Cloning Beam Release Branch-----------------'
+ git clone ${GIT_REPO_URL}
+ cd ${BEAM_ROOT_DIR}
+ git checkout ${RELEASE_BRANCH}
+
+ echo '-------------------Generating and Pushing Python images-----------------'
+ ./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}
+ for ver in "${PYTHON_VER[@]}"; do
+ docker push apachebeam/${ver}_sdk:${RELEASE}
+ docker tag apachebeam/${ver}_sdk:${RELEASE} apachebeam/${ver}_sdk:latest
+ docker push apachebeam/${ver}_sdk:latest
+ done
+
+ echo '-------------------Generating and Pushing Java images-----------------'
+ ./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}
+ docker tag apachebeam/java_sdk:${RELEASE} apachebeam/java_sdk:latest
+ docker push apachebeam/java_sdk:latest
+
+ echo '-------------------Generating and Pushing Go images-----------------'
+ ./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}
+ docker tag apachebeam/go_sdk:${RELEASE} apachebeam/go_sdk:latest
+ docker push apachebeam/go_sdk:latest
+
+ rm -rf ~/${PYTHON_ARTIFACTS_DIR}
+
+ echo "-------------------Clean up SDK docker images at local-------------------"
+ for ver in "${PYTHON_VER[@]}"; do
+ docker rmi -f apachebeam/${ver}_sdk:${RELEASE}
+ docker rmi -f apachebeam/${ver}_sdk:latest
+ done
+
+ docker rmi -f apachebeam/java_sdk:${RELEASE}
+ docker rmi -f apachebeam/java_sdk:latest
+
+ docker rmi -f apachebeam/go_sdk:${RELEASE}
+ docker rmi -f apachebeam/go_sdk:latest
+fi
diff --git a/release/src/main/scripts/run_rc_validation.sh b/release/src/main/scripts/run_rc_validation.sh
index 3eab8ca..887e821 100755
--- a/release/src/main/scripts/run_rc_validation.sh
+++ b/release/src/main/scripts/run_rc_validation.sh
@@ -15,348 +15,321 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+# This script automates release candidate validation process.
+#
+# It reads configurations from script.config, checks environment settings and
+# runs a list of validation pipelines against multiple runners one after
+# another.
+#
+# NOTE:
+# 1. Please set all variables in script.config before running this script.
+# 2. Please babysit this script until first pipeline starts.
+
+
+. script.config
+
function clean_up(){
- echo "======================Stopping Pubsub Java Injector========================="
+ echo ""
+ echo "====================Final Steps===================="
+ echo "-----------------Stopping Pubsub Java Injector-----------------"
echo "Please stop java injector manually."
- echo "===========================Signing up Spreadsheet==========================="
+ echo "-----------------Signing up Spreadsheet-----------------"
echo "Please open this spreadsheet: https://s.apache.org/beam-release-validation"
echo "Please sign up your name in the tests you have ran."
- echo "===========================Final Cleanup==========================="
- if [[ -f ~/.m2/settings_backup.xml ]]; then
+ echo "-----------------Final Cleanup-----------------"
+ if [[ -f ~/.m2/$BACKUP_M2 ]]; then
rm ~/.m2/settings.xml
- cp ~/.m2/settings_backup.xml ~/.m2/settings.xml
+ cp ~/.m2/$BACKUP_M2 ~/.m2/settings.xml
+ rm ~/.m2/$BACKUP_M2
echo "* Restored ~/.m2/settings.xml"
fi
- if [[ -f ~/.bashrc_backup ]]; then
+ if [[ -f ~/$BACKUP_BASHRC ]]; then
rm ~/.bashrc
- cp ~/.bashrc_backup ~/.bashrc
+ cp ~/$BACKUP_BASHRC ~/.bashrc
+ rm ~/$BACKUP_BASHRC
echo "* Restored ~/.bashrc"
fi
- rm -rf ~/{LOCAL_CLONE_DIR}
- echo "* Deleted working dir ~/{LOCAL_CLONE_DIR}"
+ rm -rf ${LOCAL_BEAM_DIR}
+ echo "* Deleted workspace ${LOCAL_BEAM_DIR}"
}
+trap clean_up EXIT
-RELEASE=
-REPO_URL=
-RC_NUM=
-RELEASE_BRANCH=
-WORKING_BRANCH=
-LOCAL_CLONE_DIR=rc_validations
-BEAM_ROOT_DIR=beam
+RELEASE_BRANCH=release-${RELEASE_VER}
+WORKING_BRANCH=release-${RELEASE_VER}-RC${RC_NUM}_validations
GIT_REPO_URL=https://github.com/apache/beam.git
PYTHON_RC_DOWNLOAD_URL=https://dist.apache.org/repos/dist/dev/beam
-HUB_VERSION=2.5.0
+HUB_VERSION=2.12.0
HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
-declare -a DEFAULT_PYTHON_VERSIONS_TO_VALIDATE=("python2.7" "python3.5")
+BACKUP_BASHRC=.bashrc_backup_$(date +"%Y%m%d%H%M%S")
+BACKUP_M2=settings_backup_$(date +"%Y%m%d%H%M%S").xml
+declare -a PYTHON_VERSIONS_TO_VALIDATE=("python2.7" "python3.5")
-echo "[Input Required] Please enter the release version: "
-read RELEASE
-RELEASE_BRANCH=release-${RELEASE}
-WORKING_BRANCH=release-${RELEASE}-RC${RC_NUM}_validations
-echo "[Input Required] Please enter the release candidate number(e.g. 1): "
-read RC_NUM
-echo "[Input Required] Please copy the repo URL from the vote email sent out by Release Manager:"
-echo "The URL should look like: https://repository.apache.org/content/repositories/orgapachebeam-0000"
-read REPO_URL
-
-echo "====================Checking Environment Variables================="
-echo "running validations on release ${RELEASE} RC${RC_NUM}."
+echo ""
+echo "====================Checking Environment & Variables================="
+echo "PLEASE update RC_VALIDATE_CONFIGS in file script.config first."
+echo ""
+echo "running validations on release ${RELEASE_VER} RC${RC_NUM}."
echo "repo URL for this RC: ${REPO_URL}"
-echo "[Confirmation Required] Do you confirm all information above are correct? [y|N]"
+echo "using workspace: ${LOCAL_BEAM_DIR}"
+echo "validate Python versions: "$(IFS=$' '; echo "${PYTHON_VERSIONS_TO_VALIDATE[*]}")
+echo ""
+echo "All environment and workflow configurations from RC_VALIDATE_CONFIGS:"
+for i in "${RC_VALIDATE_CONFIGS[@]}"; do
+ echo "$i = ${!i}"
+done
+echo "[Confirmation Required] Are they all provided and correctly set? [y|N]"
read confirmation
if [[ $confirmation != "y" ]]; then
- echo "Please rerun this script and make sure you have the right inputs."
+ echo "Please rerun this script and make sure you have the right configurations."
exit
fi
-echo "=================Checking hub========================"
-if [[ -z `which hub` ]]; then
- echo "There is no hub installed on your machine."
- echo "Would you like to install hub with root permission? [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Refused to install hub. Cannot proceed into next setp."
- exit
+echo "----------------- Checking git -----------------"
+if [[ -z ${GITHUB_TOKEN} ]]; then
+ echo "Error: A Github personal access token is required to perform git push "
+ echo "under a newly cloned directory. Please manually create one from Github "
+ echo "website with guide:"
+ echo "https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line"
+ echo "Note: This token can be reused in other release scripts."
+ exit
+else
+ if [[ -d ${LOCAL_BEAM_DIR} ]]; then
+ rm -rf ${LOCAL_BEAM_DIR}
fi
- echo "=================Installing hub======================="
- wget https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
- tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
- sudo ./${HUB_ARTIFACTS_NAME}/install
- echo "eval "$(hub alias -s)"" >> ~/.bashrc
- rm -rf ${HUB_ARTIFACTS_NAME}*
+ echo "* Creating local Beam workspace: ${LOCAL_BEAM_DIR}"
+ mkdir -p ${LOCAL_BEAM_DIR}
+ echo "* Cloning Beam repo"
+ git clone ${GIT_REPO_URL} ${LOCAL_BEAM_DIR}
+ cd ${LOCAL_BEAM_DIR}
+ git checkout -b ${WORKING_BRANCH} origin/${RELEASE_BRANCH} --quiet
+ echo "* Setting up git config"
+ # Set upstream repo url with access token included.
+ USER_REPO_URL=https://${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com/${GITHUB_USERNAME}/beam.git
+ git remote add ${GITHUB_USERNAME} ${USER_REPO_URL}
+ # For hub access Github API.
+ export GITHUB_TOKEN=${GITHUB_TOKEN}
+ # For local git repo only. Required if global configs are not set.
+ git config user.name "${GITHUB_USERNAME}"
+ git config user.email "${GITHUB_USERNAME}@gmail.com"
+fi
+
+echo "-----------------Checking hub-----------------"
+if [[ -z `which hub` ]]; then
+ if [[ "${INSTALL_HUB}" = true ]]; then
+ echo "-----------------Installing hub-----------------"
+ wget https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
+ tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
+ sudo ./${HUB_ARTIFACTS_NAME}/install
+ echo "eval "$(hub alias -s)"" >> ~/.bashrc
+ rm -rf ${HUB_ARTIFACTS_NAME}*
+ else
+ echo "Hub is not installed. Validation on Python Quickstart and MobileGame will be skipped."
+ fi
fi
hub version
+echo "-----------------Checking Google Cloud SDK-----------------"
+if [[ -z `which gcloud` ]]; then
+ if [[ "${INSTALL_GCLOUD}" = true ]]; then
+ echo "-----------------Installing Google Cloud SDK-----------------"
+ sudo apt-get install google-cloud-sdk
-echo "====================Cloning Beam Release Branch===================="
-cd ~
-if [[ -d ${LOCAL_CLONE_DIR} ]]; then
- rm -rf ${LOCAL_CLONE_DIR}
+ gcloud init
+ gcloud config set project ${USER_GCP_PROJECT}
+
+ echo "-----------------Setting Up Service Account-----------------"
+ if [[ ! -z "${USER_SERVICE_ACCOUNT_EMAIL}" ]]; then
+ SERVICE_ACCOUNT_KEY_JSON=~/google-cloud-sdk/${USER}_json_key.json
+ gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_KEY_JSON} --iam-account ${USER_SERVICE_ACCOUNT_EMAIL}
+ export GOOGLE_APPLICATION_CREDENTIALS=${SERVICE_ACCOUNT_KEY_JSON}
+ else
+ echo "Missing USER_SERVICE_ACCOUNT_EMAIL from config file. Force terminate."
+ exit
+ fi
+ else
+ echo "Google Cloud SDK is not installed."
+ fi
fi
+gcloud --version
-mkdir ${LOCAL_CLONE_DIR}
-cd ${LOCAL_CLONE_DIR}
-git clone ${GIT_REPO_URL}
-cd ${BEAM_ROOT_DIR}
-git checkout ${RELEASE_BRANCH}
-git checkout -b ${WORKING_BRANCH}
+echo "-----------------Checking Bigquery CLI-----------------"
+if [[ ! -f ~/.bigqueryrc ]]; then
+ echo "-----------------Initialing Bigquery CLI-----------------"
+ bq init
+fi
+bq version
+echo "-----------------Checking gnome-terminal-----------------"
+if [[ -z `which gnome-terminal` ]]; then
+ echo "You don't have gnome-terminal installed."
+ if [[ "$INSTALL_GNOME_TERMINAL" != true ]]; then
+ sudo apt-get upgrade
+ sudo apt-get install gnome-terminal
+ else
+ echo "gnome-terminal is not installed. Validation on Python Leaderboard & GameStates will be skipped."
+ exit
+ fi
+fi
+gnome-terminal --version
+
+
+echo ""
+echo ""
echo "====================Starting Java Quickstart======================="
echo "[Current task] Java quickstart with direct runner"
-echo "[Confirmation Required] Do you want to start this task? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
+if [[ "$java_quickstart_direct" = true ]]; then
echo "*************************************************************"
echo "* Running Java Quickstart with DirectRunner"
echo "*************************************************************"
./gradlew :runners:direct-java:runQuickstartJavaDirect \
-Prepourl=${REPO_URL} \
- -Pver=${RELEASE}
+ -Pver=${RELEASE_VER}
+else
+ echo "* Skip Java quickstart with direct runner"
fi
echo "[Current task] Java quickstart with Apex local runner"
-echo "[Confirmation Required] Do you want to start this task? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
+if [[ "$java_quickstart_apex_local" = true ]]; then
echo "*************************************************************"
echo "* Running Java Quickstart with Apex local runner"
echo "*************************************************************"
./gradlew :runners:apex:runQuickstartJavaApex \
-Prepourl=${REPO_URL} \
- -Pver=${RELEASE}
+ -Pver=${RELEASE_VER}
+else
+ echo "* Skip Java quickstart with Apex local runner"
fi
echo "[Current task] Java quickstart with Flink local runner"
-echo "[Confirmation Required] Do you want to start this task? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
+if [[ "$java_quickstart_flink_local" = true ]]; then
echo "*************************************************************"
echo "* Running Java Quickstart with Flink local runner"
echo "*************************************************************"
- ./gradlew :runners:flink:1.5:runQuickstartJavaFlinkLocal \
+ ./gradlew :runners:flink:1.8:runQuickstartJavaFlinkLocal \
-Prepourl=${REPO_URL} \
- -Pver=${RELEASE}
+ -Pver=${RELEASE_VER}
+else
+ echo "* Skip Java quickstart with Flink local runner"
fi
echo "[Current task] Java quickstart with Spark local runner"
-echo "[Confirmation Required] Do you want to start this task? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
+if [[ "$java_quickstart_spark_local" = true ]]; then
echo "*************************************************************"
echo "* Running Java Quickstart with Spark local runner"
echo "*************************************************************"
./gradlew :runners:spark:runQuickstartJavaSpark \
-Prepourl=${REPO_URL} \
- -Pver=${RELEASE}
+ -Pver=${RELEASE_VER}
+else
+ echo "* Skip Java quickstart with Spark local runner"
fi
-echo "====================Checking Google Cloud SDK======================"
-if [[ -z `which gcloud` ]]; then
- echo "You don't have Google Cloud SDK installed."
- echo " Do you want to install gcloud with sudo permission? [y|N]"
- read confirmation
- if [[ $confirmation != 'y' ]]; then
- echo "Exit script without running rest validations."
- exit
- fi
- sudo apt-get install google-cloud-sdk
-fi
-gcloud --version
-
echo "[Current task] Java quickstart with Dataflow runner"
-echo "[Confirmation Required] Do you want to start this task? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
- echo "[GCP Project Required] Please input your GCP project:"
- read USER_GCP_PROJECT
- echo "[GCP GCS Bucket Required] Please input your GCS bucket: "
- read USER_GCS_BUCKET
- echo "[gcloud Login Required] Please login into your gcp account: "
- gcloud auth application-default login
- GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json
-
+if [[ "$java_quickstart_dataflow" = true && ! -z `which gcloud` ]]; then
echo "*************************************************************"
echo "* Running Java Quickstart with DataflowRunner"
echo "*************************************************************"
- gcloud auth application-default login
- gcloud config set project ${USER_GCP_PROJECT}
./gradlew :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow \
-Prepourl=${REPO_URL} \
- -Pver=${RELEASE} \
+ -Pver=${RELEASE_VER} \
-PgcpProject=${USER_GCP_PROJECT} \
- -PgcsBucket=${USER_GCS_BUCKET}
+ -PgcsBucket=${USER_GCS_BUCKET:5} # skip 'gs://' prefix
+else
+ echo "* Skip Java quickstart with Dataflow runner. Google Cloud SDK is required."
fi
-echo "===================Starting Java Mobile Game====================="
-echo "[Confirmation Required] This task asks for GCP resources."
-echo "Do you want to proceed? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
- echo "[GCP Project Required] Please input your GCP project:"
- read USER_GCP_PROJECT
- echo "[GCP GCS Bucket Required] Please input your GCS bucket: "
- read USER_GCS_BUCKET
- MOBILE_GAME_DATASET=${USER}_java_validations
- MOBILE_GAME_PUBSUB_TOPIC=leader_board-${USER}-java-topic-1
- echo "Please review following GCP sources setup: "
+echo ""
+echo "====================Starting Java Mobile Game====================="
+if [[ "$java_mobile_game" = true && ! -z `which gcloud` ]]; then
+ MOBILE_GAME_DATASET=${USER}_java_validations_$(date +%m%d)_$RANDOM
+ MOBILE_GAME_PUBSUB_TOPIC=leader_board-${USER}-java-topic-$(date +%m%d)_$RANDOM
echo "Using GCP project: ${USER_GCP_PROJECT}"
echo "Will create BigQuery dataset: ${MOBILE_GAME_DATASET}"
echo "Will create Pubsub topic: ${MOBILE_GAME_PUBSUB_TOPIC}"
- echo "[Confirmation Required] Do you want to run validations with configurations above? [y|N]"
- read confirmation
- if [[ $confirmation = "y" ]]; then
- gcloud auth login
- gcloud config set project ${USER_GCP_PROJECT}
- echo "-----------------Setting Up Service Account------------------------"
- echo "Please go to GCP IAM console under your project(${USER_GCP_PROJECT})."
- echo "Create a service account as project owner, if you don't have one."
- echo "[Input Required] Please enter your service account email:"
- read USER_SERVICE_ACCOUNT_EMAIL
- SERVICE_ACCOUNT_KEY_JSON=${USER}_json_key.json
- gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_KEY_JSON} --iam-account ${USER_SERVICE_ACCOUNT_EMAIL}
- export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/${SERVICE_ACCOUNT_KEY_JSON}
- echo "-------------------Creating BigQuery Dataset-----------------------"
- bq rm -rf --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
- bq mk --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
+ echo "-----------------Creating BigQuery Dataset-----------------"
+ bq mk --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
- echo "----------------------Creating Pubsub Topic------------------------"
- gcloud alpha pubsub topics delete projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC}
- gcloud alpha pubsub topics create --project=${USER_GCP_PROJECT} ${MOBILE_GAME_PUBSUB_TOPIC}
+ echo "-----------------Creating Pubsub Topic-----------------"
+ gcloud pubsub topics create --project=${USER_GCP_PROJECT} ${MOBILE_GAME_PUBSUB_TOPIC}
- echo "**************************************************************************"
- echo "* Java mobile game validations: UserScore, HourlyTeamScore, Leaderboard"
- echo "**************************************************************************"
- ./gradlew :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow \
- -Prepourl=${REPO_URL} \
- -Pver=${RELEASE} \
- -PgcpProject=${USER_GCP_PROJECT} \
- -PgcsBucket=${USER_GCS_BUCKET} \
- -PbqDataset=${MOBILE_GAME_DATASET} -PpubsubTopic=${MOBILE_GAME_PUBSUB_TOPIC}
- fi
+ echo "**************************************************************************"
+ echo "* Java mobile game validations: UserScore, HourlyTeamScore, Leaderboard"
+ echo "**************************************************************************"
+ ./gradlew :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow \
+ -Prepourl=${REPO_URL} \
+ -Pver=${RELEASE_VER} \
+ -PgcpProject=${USER_GCP_PROJECT} \
+ -PbqDataset=${MOBILE_GAME_DATASET} \
+ -PpubsubTopic=${MOBILE_GAME_PUBSUB_TOPIC} \
+ -PgcsBucket=${USER_GCS_BUCKET:5} # skip 'gs://' prefix
+
+ echo "-----------------Cleaning up BigQuery & Pubsub-----------------"
+ bq rm -rf --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
+ gcloud pubsub topics delete projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC}
+else
+ echo "* Skip Java Mobile Game. Google Cloud SDK is required"
fi
-echo "==================Starting Python Quickstart and MobileGame==================="
+echo ""
+echo "====================Starting Python Quickstart and MobileGame==================="
echo "This task will create a PR against apache/beam, trigger a jenkins job to run:"
echo "1. Python quickstart validations(batch & streaming)"
echo "2. Python MobileGame validations(UserScore, HourlyTeamScore)"
-echo "[Confirmation Required] Do you want to proceed? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
- echo "[Input Required] Please enter your github repo URL forked from apache/beam:"
- read USER_REMOTE_URL
- echo "[Input Required] Please enter your github username:"
- read GITHUB_USERNAME
- WORKING_BRANCH=python_validation_pr
- git checkout -b ${WORKING_BRANCH}
+if [[ "$python_quickstart_mobile_game" = true && ! -z `which hub` ]]; then
touch empty_file.txt
git add empty_file.txt
- git commit -m "Add empty file in order to create PR"
- git push -f ${USER_REMOTE_URL}
- hub pull-request -b apache:${RELEASE_BRANCH} -h ${GITHUB_USERNAME}:${WORKING_BRANCH} -F- <<<"[DO NOT MERGE]Run Python RC Validation Tests
+ git commit -m "Add empty file in order to create PR" --quiet
+ git push -f ${GITHUB_USERNAME} --quiet
+ # Create a test PR
+ PR_URL=$(hub pull-request -b apache:${RELEASE_BRANCH} -h ${GITHUB_USERNAME}:${WORKING_BRANCH} -F- <<<"[DO NOT MERGE] Run Python RC Validation Tests
-
- Run Python ReleaseCandidate"
-
- echo "[NOTE] If there is no jenkins job started, please comment generated PR with: Run Python ReleaseCandidate"
+ Run Python ReleaseCandidate")
+ echo "Created $PR_URL"
+ # Comment on PR to trigger Python ReleaseCandidate Jenkins job.
+ PR_NUM=$(echo $PR_URL | sed 's/.*apache\/beam\/pull\/\([0-9]*\).*/\1/')
+ hub api repos/apache/beam/issues/$PR_NUM/comments --raw-field "body=Run Python ReleaseCandidate" > /dev/null
+ echo ""
+ echo "[NOTE] If there is no jenkins job started, please comment on $PR_URL with: Run Python ReleaseCandidate"
+else
+ echo "* Skip Python Quickstart and MobileGame. Hub is required."
fi
-echo "==============Starting Python Leaderboard & GameStates Validations==============="
-echo "This task asks for GCP resources. Do you want to proceed? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
- cd ~/${LOCAL_CLONE_DIR}
-
- echo "---------------------Checking gnome-terminal----------------------------------"
- if [[ -z `which gnome-terminal` ]]; then
- echo "You don't have gnome-terminal installed."
- echo "Do you want to install gnome-terminal with sudo permission? [y|N]"
- read confirmation
- if [[ $confirmation != 'y' ]]; then
- echo "Exit this script without proceeding to the next step."
- exit
- fi
- fi
- gnome-terminal --version
+echo ""
+echo "====================Starting Python Leaderboard & GameStates Validations==============="
+if [[ ("$python_leaderboard_direct" = true || \
+ "$python_leaderboard_dataflow" = true || \
+ "$python_gamestats_direct" = true || \
+ "$python_gamestats_dataflow" = true) && \
+ ! -z `which gnome-terminal` ]]; then
+ cd ${LOCAL_BEAM_DIR}
echo "---------------------Downloading Python Staging RC----------------------------"
- wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE}/python/apache-beam-${RELEASE}.zip
- wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE}/python/apache-beam-${RELEASE}.zip.sha512
-
- echo "--------------------------Verifying Hashes------------------------------------"
- sha512sum -c apache-beam-${RELEASE}.zip.sha512
-
- sudo `which pip` install --upgrade pip
- sudo `which pip` install --upgrade setuptools
- sudo `which pip` install --upgrade virtualenv
-
- echo "[Input Required] Please enter Python interpreter(s) separated by space to use for running validation steps."
- echo "Sample input: python2.7"
- echo "Enter empty line to repeat validation steps using all of ${DEFAULT_PYTHON_VERSIONS_TO_VALIDATE[@]}."
-
- read -a PYTHON_VERSIONS_TO_VALIDATE
- if [[ -z "$PYTHON_VERSIONS_TO_VALIDATE" ]]; then
- PYTHON_VERSIONS_TO_VALIDATE=${DEFAULT_PYTHON_VERSIONS_TO_VALIDATE[@]}
+ wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.zip
+ wget ${PYTHON_RC_DOWNLOAD_URL}/${RELEASE_VER}/python/apache-beam-${RELEASE_VER}.zip.sha512
+ if [[ ! -f apache-beam-${RELEASE_VER}.zip ]]; then
+ { echo "Fail to download Python Staging RC files." ;exit 1; }
fi
- for py_version in "${PYTHON_VERSIONS_TO_VALIDATE[@]}"
- do
- rm -rf ./beam_env_${py_version}
- echo "--------------Setting up virtualenv with $py_version interpreter----------------"
- virtualenv beam_env_${py_version} -p $py_version
- . beam_env_${py_version}/bin/activate
+ echo "--------------------------Verifying Hashes------------------------------------"
+ sha512sum -c apache-beam-${RELEASE_VER}.zip.sha512
- echo "--------------------------Installing Python SDK-------------------------------"
- pip install apache-beam-${RELEASE}.zip
- pip install apache-beam-${RELEASE}.zip[gcp]
+ `which pip` install --upgrade pip
+ `which pip` install --upgrade setuptools
+ `which pip` install --upgrade virtualenv
- echo "----------------------------Setting up GCP Sources----------------------------"
- echo "[GCP Project Required] Please input your GCP project:"
- read USER_GCP_PROJECT
- gcloud auth login
- gcloud config set project ${USER_GCP_PROJECT}
-
- MOBILE_GAME_GCS_BUCKET=gs://${USER}_python_validations_bucket
- MOBILE_GAME_DATASET=${USER}_python_validations
- MOBILE_GAME_PUBSUB_TOPIC=leader_board-${USER}-python-topic-1
- gsutil mb -p ${USER_GCP_PROJECT} ${MOBILE_GAME_GCS_BUCKET}
- gcloud alpha pubsub topics delete projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC}
- gcloud alpha pubsub topics create --project=${USER_GCP_PROJECT} ${MOBILE_GAME_PUBSUB_TOPIC}
-
- echo "-----------------------Setting Up Service Account-----------------------------"
- echo "Please go to GCP IAM console under your project(${USER_GCP_PROJECT})."
- echo "Create a service account as project owner, if you don't have one."
- echo "[Input Required] Please enter your service account email:"
- read USER_SERVICE_ACCOUNT_EMAIL
- SERVICE_ACCOUNT_KEY_JSON=${USER}_json_key.json
- gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_KEY_JSON} --iam-account ${USER_SERVICE_ACCOUNT_EMAIL}
- export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/${SERVICE_ACCOUNT_KEY_JSON}
-
- echo "-----------------------Setting up Shell Env Vars------------------------------"
- # [BEAM-4518]
- FIXED_WINDOW_DURATION=20
- cp ~/.bashrc ~/.bashrc_backup
- echo "export USER_GCP_PROJECT=${USER_GCP_PROJECT}" >> ~/.bashrc
- echo "export MOBILE_GAME_DATASET=${MOBILE_GAME_DATASET}" >> ~/.bashrc
- echo "export MOBILE_GAME_PUBSUB_TOPIC=${MOBILE_GAME_PUBSUB_TOPIC}" >> ~/.bashrc
- echo "export MOBILE_GAME_GCS_BUCKET=${MOBILE_GAME_GCS_BUCKET}" >> ~/.bashrc
- echo "export GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}" >> ~/.bashrc
- echo "export RELEASE=${RELEASE}" >> ~/.bashrc
- echo "export FIXED_WINDOW_DURATION=${FIXED_WINDOW_DURATION}" >> ~/.bashrc
-
-
- echo "--------------------------Updating ~/.m2/settings.xml-------------------------"
+ echo "--------------------------Updating ~/.m2/settings.xml-------------------------"
cd ~
- if [[ -d .m2 ]]; then
+ if [[ ! -d .m2 ]]; then
mkdir .m2
fi
cd .m2
if [[ -f ~/.m2/settings.xml ]]; then
- mv settings.xml settings_backup.xml
+ mv settings.xml $BACKUP_M2
fi
touch settings.xml
echo "<settings>" >> settings.xml
@@ -368,8 +341,8 @@
echo " </activation>" >> settings.xml
echo " <repositories>" >> settings.xml
echo " <repository>" >> settings.xml
- echo " <id>Release ${RELEASE} RC${RC_NUM}</id>" >> settings.xml
- echo " <name>Release ${RELEASE} RC${RC_NUM}</name>" >> settings.xml
+ echo " <id>Release ${RELEASE_VER} RC${RC_NUM}</id>" >> settings.xml
+ echo " <name>Release ${RELEASE_VER} RC${RC_NUM}</name>" >> settings.xml
echo " <url>${REPO_URL}</url>" >> settings.xml
echo " </repository>" >> settings.xml
echo " </repositories>" >> settings.xml
@@ -377,54 +350,73 @@
echo " </profiles>" >> settings.xml
echo "</settings>" >> settings.xml
- echo "----------------------Starting Pubsub Java Injector--------------------------"
- cd ~/${LOCAL_CLONE_DIR}
- mvn archetype:generate \
- -DarchetypeGroupId=org.apache.beam \
- -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
- -DarchetypeVersion=${RELEASE} \
- -DgroupId=org.example \
- -DartifactId=word-count-beam \
- -Dversion="0.1" \
- -Dpackage=org.apache.beam.examples \
- -DinteractiveMode=false \
- -DarchetypeCatalog=internal
+ echo "-----------------------Setting up Shell Env Vars------------------------------"
+ # [BEAM-4518]
+ FIXED_WINDOW_DURATION=20
+ cp ~/.bashrc ~/$BACKUP_BASHRC
+ echo "export USER_GCP_PROJECT=${USER_GCP_PROJECT}" >> ~/.bashrc
+ echo "export USER_GCS_BUCKET=${USER_GCS_BUCKET}" >> ~/.bashrc
+ echo "export SHARED_PUBSUB_TOPIC=${SHARED_PUBSUB_TOPIC}" >> ~/.bashrc
+ echo "export GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}" >> ~/.bashrc
+ echo "export RELEASE_VER=${RELEASE_VER}" >> ~/.bashrc
+ echo "export FIXED_WINDOW_DURATION=${FIXED_WINDOW_DURATION}" >> ~/.bashrc
+ echo "export LOCAL_BEAM_DIR=${LOCAL_BEAM_DIR}" >> ~/.bashrc
- cd word-count-beam
- echo "A new terminal will pop up and start a java top injector."
- gnome-terminal -x sh -c \
- "echo '******************************************************';
- echo '* Running Pubsub Java Injector';
- echo '******************************************************';
- mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.complete.game.injector.Injector \
- -Dexec.args='${USER_GCP_PROJECT} ${MOBILE_GAME_PUBSUB_TOPIC} none';
- exec bash"
+ echo "----------------------Starting Pubsub Java Injector--------------------------"
+ cd ${LOCAL_BEAM_DIR}
+ mvn archetype:generate \
+ -DarchetypeGroupId=org.apache.beam \
+ -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+ -DarchetypeVersion=${RELEASE_VER} \
+ -DgroupId=org.example \
+ -DartifactId=word-count-beam \
+ -Dversion="0.1" \
+ -Dpackage=org.apache.beam.examples \
+ -DinteractiveMode=false \
+ -DarchetypeCatalog=internal
- echo "[Confirmation Required] Please enter y to confirm injector running:"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Following tests only can be ran when java injector running."
- clean_up
- exit
- fi
+ # Create a pubsub topic as a input source shared to all Python pipelines.
+ SHARED_PUBSUB_TOPIC=leader_board-${USER}-python-topic-$(date +%m%d)_$RANDOM
+ gcloud pubsub topics create --project=${USER_GCP_PROJECT} ${SHARED_PUBSUB_TOPIC}
- cd ~/${LOCAL_CLONE_DIR}/
+ cd word-count-beam
+ echo "A new terminal will pop up and start a java top injector."
+ gnome-terminal -x sh -c \
+ "echo '******************************************************';
+ echo '* Running Pubsub Java Injector';
+ echo '******************************************************';
+ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.complete.game.injector.Injector \
+ -Dexec.args='${USER_GCP_PROJECT} ${SHARED_PUBSUB_TOPIC} none';
+ exec bash"
+
+ # Run Leaderboard & GameStates pipelines under multiple versions of Python
+ cd ${LOCAL_BEAM_DIR}
+ for py_version in "${PYTHON_VERSIONS_TO_VALIDATE[@]}"
+ do
+ rm -rf ./beam_env_${py_version}
+ echo "--------------Setting up virtualenv with $py_version interpreter----------------"
+ virtualenv beam_env_${py_version} -p $py_version
+ . beam_env_${py_version}/bin/activate
+
+ echo "--------------------------Installing Python SDK-------------------------------"
+ pip install apache-beam-${RELEASE_VER}.zip[gcp]
echo "----------------Starting Leaderboard with DirectRunner-----------------------"
- echo "[Confirmation Required] Do you want to proceed? [y|N]"
- read confirmation
- if [[ $confirmation = "y" ]]; then
- bq rm -rf --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
- bq mk --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
+ if [[ "$python_leaderboard_direct" = true ]]; then
+ LEADERBOARD_DIRECT_DATASET=${USER}_python_validations_$(date +%m%d)_$RANDOM
+ bq mk --project=${USER_GCP_PROJECT} ${LEADERBOARD_DIRECT_DATASET}
+ echo "export LEADERBOARD_DIRECT_DATASET=${LEADERBOARD_DIRECT_DATASET}" >> ~/.bashrc
+
echo "This is a streaming job. This task will be launched in a separate terminal."
gnome-terminal -x sh -c \
"echo '*****************************************************';
echo '* Running Python Leaderboard with DirectRunner';
echo '*****************************************************';
+ . ${LOCAL_BEAM_DIR}/beam_env_${py_version}/bin/activate
python -m apache_beam.examples.complete.game.leader_board \
--project=${USER_GCP_PROJECT} \
- --topic projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC} \
- --dataset ${MOBILE_GAME_DATASET};
+ --topic projects/${USER_GCP_PROJECT}/topics/${SHARED_PUBSUB_TOPIC} \
+ --dataset ${LEADERBOARD_DIRECT_DATASET};
exec bash"
echo "***************************************************************"
@@ -434,41 +426,35 @@
echo "***************************************************************"
echo "* How to verify results:"
echo "* 1. Check whether there is any error messages in the task running terminal."
- echo "* 2. Goto your BigQuery console and check whether your ${MOBILE_GAME_DATASET} has leader_board_users and leader_board_teams table."
+ echo "* 2. Goto your BigQuery console and check whether your ${LEADERBOARD_DIRECT_DATASET} has leader_board_users and leader_board_teams table."
echo "* 3. Check whether leader_board_users has data, retrieving BigQuery data as below: "
- bq head -n 10 ${MOBILE_GAME_DATASET}.leader_board_users
+ bq head -n 10 ${LEADERBOARD_DIRECT_DATASET}.leader_board_users
echo "* 4. Check whether leader_board_teams has data, retrieving BigQuery data as below:"
- bq head -n 10 ${MOBILE_GAME_DATASET}.leader_board_teams
+ bq head -n 10 ${LEADERBOARD_DIRECT_DATASET}.leader_board_teams
echo "***************************************************************"
-
- echo "If you have verified all items listed above, please terminate the python job."
- echo "[Confirmation Required] Please confirm whether you have stopped this job: [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Current job must be terminated in order to proceed into next test."
- clean_up
- exit
- fi
+ else
+ echo "* Skip Python Leaderboard with DirectRunner"
fi
echo "----------------Starting Leaderboard with DataflowRunner---------------------"
- echo "[Confirmation Required] Do you want to proceed? [y|N]"
- read confirmation
- if [[ $confirmation = "y" ]]; then
- bq rm -rf --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
- bq mk --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
+ if [[ "$python_leaderboard_dataflow" = true ]]; then
+ LEADERBOARD_DF_DATASET=${USER}_python_validations_$(date +%m%d)_$RANDOM
+ bq mk --project=${USER_GCP_PROJECT} ${LEADERBOARD_DF_DATASET}
+ echo "export LEADERBOARD_DF_DATASET=${LEADERBOARD_DF_DATASET}" >> ~/.bashrc
+
echo "This is a streaming job. This task will be launched in a separate terminal."
gnome-terminal -x sh -c \
"echo '*****************************************************';
echo '* Running Python Leaderboard with DataflowRunner';
echo '*****************************************************';
+ . ${LOCAL_BEAM_DIR}/beam_env_${py_version}/bin/activate
python -m apache_beam.examples.complete.game.leader_board \
--project=${USER_GCP_PROJECT} \
- --topic projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC} \
- --dataset ${MOBILE_GAME_DATASET} \
+ --topic projects/${USER_GCP_PROJECT}/topics/${SHARED_PUBSUB_TOPIC} \
+ --dataset ${LEADERBOARD_DF_DATASET} \
--runner DataflowRunner \
- --temp_location=${MOBILE_GAME_GCS_BUCKET}/temp/ \
- --sdk_location apache-beam-${RELEASE}.zip; \
+ --temp_location=${USER_GCS_BUCKET}/temp/ \
+ --sdk_location apache-beam-${RELEASE_VER}.zip; \
exec bash"
echo "***************************************************************"
@@ -477,29 +463,21 @@
sleep 10m
echo "* How to verify results:"
echo "* 1. Goto your Dataflow job console and check whether there is any error."
- echo "* 2. Goto your BigQuery console and check whether your ${MOBILE_GAME_DATASET} has leader_board_users and leader_board_teams table."
+ echo "* 2. Goto your BigQuery console and check whether your ${LEADERBOARD_DF_DATASET} has leader_board_users and leader_board_teams table."
echo "* 3. Check whether leader_board_users has data, retrieving BigQuery data as below: "
- bq head -n 10 ${MOBILE_GAME_DATASET}.leader_board_users
+ bq head -n 10 ${LEADERBOARD_DF_DATASET}.leader_board_users
echo "* 4. Check whether leader_board_teams has data, retrieving BigQuery data as below:"
- bq head -n 10 ${MOBILE_GAME_DATASET}.leader_board_teams
+ bq head -n 10 ${LEADERBOARD_DF_DATASET}.leader_board_teams
echo "***************************************************************"
-
- echo "If you have verified all items listed above, please terminate this job in Dataflow Console."
- echo "[Confirmation Required] Please confirm whether you have stopped this job: [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Current job must be terminated in order to proceed into next test."
- clean_up
- exit
- fi
+ else
+ echo "* Skip Python Leaderboard with DataflowRunner"
fi
echo "------------------Starting GameStats with DirectRunner-----------------------"
- echo "[Confirmation Required] Do you want to proceed? [y|N]"
- read confirmation
- if [[ $confirmation = "y" ]]; then
- bq rm -rf --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
- bq mk --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
+ if [[ "$python_gamestats_direct" = true ]]; then
+ GAMESTATS_DIRECT_DATASET=${USER}_python_validations_$(date +%m%d)_$RANDOM
+ bq mk --project=${USER_GCP_PROJECT} ${GAMESTATS_DIRECT_DATASET}
+ echo "export GAMESTATS_DIRECT_DATASET=${GAMESTATS_DIRECT_DATASET}" >> ~/.bashrc
echo "This is a streaming job. This task will be launched in a separate terminal."
echo "Streaming job is running with fixed_window_duration=${FIXED_WINDOW_DURATION}"
@@ -507,10 +485,11 @@
"echo '*****************************************************';
echo '* Running GameStats with DirectRunner';
echo '*****************************************************';
+ . ${LOCAL_BEAM_DIR}/beam_env_${py_version}/bin/activate
python -m apache_beam.examples.complete.game.game_stats \
--project=${USER_GCP_PROJECT} \
- --topic projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC} \
- --dataset ${MOBILE_GAME_DATASET} \
+ --topic projects/${USER_GCP_PROJECT}/topics/${SHARED_PUBSUB_TOPIC} \
+ --dataset ${GAMESTATS_DIRECT_DATASET} \
--fixed_window_duration ${FIXED_WINDOW_DURATION}; \
exec bash"
@@ -520,42 +499,36 @@
sleep 25m
echo "* How to verify results:"
echo "* 1. Check whether there is any error messages in the task running terminal."
- echo "* 2. Goto your BigQuery console and check whether your ${MOBILE_GAME_DATASET} has game_stats_teams and game_stats_sessions table."
+ echo "* 2. Goto your BigQuery console and check whether your ${GAMESTATS_DIRECT_DATASET} has game_stats_teams and game_stats_sessions table."
echo "* 3. Check whether game_stats_teams has data, retrieving BigQuery data as below: "
- bq head -n 10 ${MOBILE_GAME_DATASET}.game_stats_teams
+ bq head -n 10 ${GAMESTATS_DIRECT_DATASET}.game_stats_teams
echo "* 4. Check whether game_stats_sessions has data, retrieving BigQuery data as below:"
- bq head -n 10 ${MOBILE_GAME_DATASET}.game_stats_sessions
+ bq head -n 10 ${GAMESTATS_DIRECT_DATASET}.game_stats_sessions
echo "***************************************************************"
-
- echo "If you have verified all items listed above, please terminate the python job."
- echo "[Confirmation Required] Please confirm whether you have stopped this job: [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Current job must be terminated in order to proceed into next test."
- clean_up
- exit
- fi
+ else
+ echo "* Skip Python GameStats with DirectRunner"
fi
echo "-------------------Starting GameStats with DataflowRunner--------------------"
- echo "[Confirmation Required] Do you want to proceed? [y|N]"
- read confirmation
- if [[ $confirmation = "y" ]]; then
- bq rm -rf --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
- bq mk --project=${USER_GCP_PROJECT} ${MOBILE_GAME_DATASET}
+ if [[ "$python_gamestats_dataflow" = true ]]; then
+ GAMESTATS_DF_DATASET=${USER}_python_validations_$(date +%m%d)_$RANDOM
+ bq mk --project=${USER_GCP_PROJECT} ${GAMESTATS_DF_DATASET}
+ echo "export GAMESTATS_DF_DATASET=${GAMESTATS_DF_DATASET}" >> ~/.bashrc
+
echo "This is a streaming job. This task will be launched in a separate terminal."
echo "Streaming job is running with fixed_window_duration=${FIXED_WINDOW_DURATION}"
gnome-terminal -x sh -c \
"echo '*****************************************************';
echo '* Running GameStats with DataflowRunner';
echo '*****************************************************';
+ . ${LOCAL_BEAM_DIR}/beam_env_${py_version}/bin/activate
python -m apache_beam.examples.complete.game.game_stats \
--project=${USER_GCP_PROJECT} \
- --topic projects/${USER_GCP_PROJECT}/topics/${MOBILE_GAME_PUBSUB_TOPIC} \
- --dataset ${MOBILE_GAME_DATASET} \
+ --topic projects/${USER_GCP_PROJECT}/topics/${SHARED_PUBSUB_TOPIC} \
+ --dataset ${GAMESTATS_DF_DATASET} \
--runner DataflowRunner \
- --temp_location=${MOBILE_GAME_GCS_BUCKET}/temp/ \
- --sdk_location apache-beam-${RELEASE}.zip \
+ --temp_location=${USER_GCS_BUCKET}/temp/ \
+ --sdk_location apache-beam-${RELEASE_VER}.zip \
--fixed_window_duration ${FIXED_WINDOW_DURATION}; exec bash"
echo "***************************************************************"
@@ -564,23 +537,16 @@
sleep 30m
echo "* How to verify results:"
echo "* 1. Goto your Dataflow job console and check whether there is any error."
- echo "* 2. Goto your BigQuery console and check whether your ${MOBILE_GAME_DATASET} has game_stats_teams and game_stats_sessions table."
+ echo "* 2. Goto your BigQuery console and check whether your ${GAMESTATS_DF_DATASET} has game_stats_teams and game_stats_sessions table."
echo "* 3. Check whether game_stats_teams has data, retrieving BigQuery data as below: "
- bq head -n 10 ${MOBILE_GAME_DATASET}.game_stats_teams
+ bq head -n 10 ${GAMESTATS_DF_DATASET}.game_stats_teams
echo "* 4. Check whether game_stats_sessions has data, retrieving BigQuery data as below:"
- bq head -n 10 ${MOBILE_GAME_DATASET}.game_stats_sessions
+ bq head -n 10 ${GAMESTATS_DF_DATASET}.game_stats_sessions
echo "***************************************************************"
-
- echo "If you have verified all items listed above, please terminate the python job."
- echo "[Confirmation Required] Please confirm whether you have stopped this job: [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Current job must be terminated in order to proceed into next test."
- clean_up
- exit
- fi
+ else
+ echo "* Skip Python GameStats with DataflowRunner"
fi
done # Loop over Python versions.
+else
+ echo "* Skip Python Leaderboard & GameStates Validations"
fi
-
-clean_up
diff --git a/release/src/main/scripts/script.config b/release/src/main/scripts/script.config
new file mode 100755
index 0000000..e8c510f
--- /dev/null
+++ b/release/src/main/scripts/script.config
@@ -0,0 +1,148 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file contains all configurations required by release scripts.
+#
+# Please update Environment Configurations and/or Workflow Validation
+# Configurations sections before running the script.
+
+
+##############################################################################
+# Register configs in following lists that corresponds to release scripts
+##############################################################################
+
+# List of required configurations for run_rc_validation.sh
+RC_VALIDATE_CONFIGS=(
+ RELEASE_VER
+ RC_NUM
+ REPO_URL
+ INSTALL_GCLOUD
+ USER_GCP_PROJECT
+ USER_GCS_BUCKET
+ USER_SERVICE_ACCOUNT_EMAIL
+ INSTALL_HUB
+ GITHUB_USERNAME
+ GITHUB_TOKEN
+ INSTALL_GNOME_TERMINAL
+ LOCAL_BEAM_DIR
+ java_quickstart_direct
+ java_quickstart_apex_local
+ java_quickstart_flink_local
+ java_quickstart_spark_local
+ java_quickstart_dataflow
+ java_mobile_game
+ python_quickstart_mobile_game
+ python_leaderboard_direct
+ python_leaderboard_dataflow
+ python_gamestats_direct
+ python_gamestats_dataflow
+)
+
+# List of required configurations for verify_release_build.sh
+RELEASE_BUILD_CONFIGS=(
+ RELEASE_VER
+ INSTALL_HUB
+ GITHUB_USERNAME
+ GITHUB_TOKEN
+ LOCAL_BEAM_DIR
+)
+
+
+##############################################################################
+# Environment Configurations
+##############################################################################
+
+# Beam version of current release
+# e.g. 2.14.0
+RELEASE_VER=
+
+# Release candidate number
+# This is an identifier for each candidate we built for release. Start from 1
+# and increment if new candidate is built.
+# e.g. 1
+RC_NUM=
+
+# The repo URL from the vote email sent by Release Manager
+# e.g. https://repository.apache.org/content/repositories/orgapachebeam-0000
+REPO_URL=
+
+# Install Google Cloud SDK
+# Google Cloud SDK is required to run validation pipeline on DataflowRunner.
+# Set to true so that it will be installed if not found from local.
+INSTALL_GCLOUD=true
+
+# GCP project id
+# Required for running pipeline with DataflowRunner for validation.
+# e.g. apache-beam-testing
+USER_GCP_PROJECT=
+
+# GCS bucket name
+# Required for running pipeline with DataflowRunner for validation.
+# e.g. gs://bucket-name (please include 'gs://' prefix)
+USER_GCS_BUCKET=
+
+# GCP service account email
+# This service account should be under your project ${USER_GCP_PROJECT}. If you
+# don't have one, create it from GCP IAM console and set as project owner. You
+# can leave it empty if you have gcloud setup before.
+USER_SERVICE_ACCOUNT_EMAIL=
+
+# Install hub
+# hub is a tool used in run_rc_validation.sh and verify_release_build.sh to
+# create a Github PR in an automatic way. Set to true so that it will be
+# installed if not found from local.
+INSTALL_HUB=true
+
+# Your github username
+# Used by hub to create a PR for validation.
+GITHUB_USERNAME=
+
+# Your Github personal access token
+# Allow git push to personal repo in order to create a PR for
+# validation. This token is required when two-factor authentication is enabled
+# in Github account. You can manually create it following
+# https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line
+GITHUB_TOKEN=
+
+# Install gnome-terminal
+# Used in Python Leaderboard & GameStates to launches data injection pipeline
+# in a separate terminal. Set to true so that it will be installed if not found
+# from local. Otherwise, validation on Python Leaderboard & GameStates will be
+# skipped.
+INSTALL_GNOME_TERMINAL=true
+
+# Local Beam directory
+# This is a local workspace used by validation scripts.
+# Default to a temporary directory created uniquely in each run.
+LOCAL_BEAM_DIR="$(mktemp -d -t beam-release.${RELEASE_VER}.XXXXXX)"
+
+
+##############################################################################
+# Workflow Validation Configurations
+#
+# Whether to start certain validation pipeline.
+##############################################################################
+java_quickstart_direct=true
+java_quickstart_apex_local=true
+java_quickstart_flink_local=true
+java_quickstart_spark_local=true
+java_quickstart_dataflow=true
+java_mobile_game=true
+python_quickstart_mobile_game=true
+python_leaderboard_direct=true
+python_leaderboard_dataflow=true
+python_gamestats_direct=true
+python_gamestats_dataflow=true
diff --git a/release/src/main/scripts/set_version.sh b/release/src/main/scripts/set_version.sh
index 7943e15..5844b73 100755
--- a/release/src/main/scripts/set_version.sh
+++ b/release/src/main/scripts/set_version.sh
@@ -67,6 +67,7 @@
sed -i -e "s/version=.*/version=$TARGET_VERSION/" gradle.properties
sed -i -e "s/project.version = .*/project.version = '$TARGET_VERSION'/" buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
sed -i -e "s/^__version__ = .*/__version__ = '${TARGET_VERSION}'/" sdks/python/apache_beam/version.py
+ sed -i -e "s/python_sdk_version=.*/python_sdk_version=$TARGET_VERSION/" gradle.properties
# TODO: [BEAM-4767]
sed -i -e "s/'dataflow.container_version' : .*/'dataflow.container_version' : 'beam-${RELEASE}'/" runners/google-cloud-dataflow-java/build.gradle
else
@@ -78,6 +79,7 @@
sed -i -e "s/version=.*/version=$TARGET_VERSION-SNAPSHOT/" gradle.properties
sed -i -e "s/project.version = .*/project.version = '$TARGET_VERSION'/" buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
sed -i -e "s/^__version__ = .*/__version__ = '${TARGET_VERSION}.dev'/" sdks/python/apache_beam/version.py
+ sed -i -e "s/python_sdk_version=.*/python_sdk_version=$TARGET_VERSION.dev/" gradle.properties
sed -i -e "s/'dataflow.container_version' : .*/'dataflow.container_version' : 'beam-master-.*'/" runners/google-cloud-dataflow-java/build.gradle
fi
diff --git a/release/src/main/scripts/verify_release_build.sh b/release/src/main/scripts/verify_release_build.sh
index 42a47d1..8442e9f 100755
--- a/release/src/main/scripts/verify_release_build.sh
+++ b/release/src/main/scripts/verify_release_build.sh
@@ -16,190 +16,155 @@
# limitations under the License.
#
-# This script will run pre-installations and run release build.
+# This script helps to verify full life cycle of Gradle build and all
+# PostCommit tests against release branch on Jenkins.
+#
+# It reads configurations from script.config, setup environment and finally
+# create a test PR to run Jenkins jobs.
+#
+# NOTE:
+# 1. Please create a personal access token from your Github account first.
+# Instructions: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line
+# 2. Please set RELEASE_BUILD_CONFIGS in script.config before running this
+# script.
+# 3. Please manually comment trigger phrases to created PR to start Gradle
+# release build and all PostCommit jobs. Phrases are listed in
+# JOB_TRIGGER_PHRASES below.
+
+
+. script.config
set -e
-GIT_REPO_URL=https://github.com/apache/beam.git
-LOCAL_CLONE_DIR=release_build
-BEAM_ROOT_DIR=beam
+BEAM_REPO_URL=https://github.com/apache/beam.git
+RELEASE_BRANCH=release-${RELEASE_VER}
+WORKING_BRANCH=postcommit_validation_pr
-echo "Which branch you want to verify release build: "
-read branch
+JOB_TRIGGER_PHRASES=(
+ # To verify Gradle release build
+ "**Run Release Gradle Build**"
+ # To run all PostCommit jobs
+ "Run Go PostCommit"
+ "Run Java PostCommit"
+ "Run Java PostCommit"
+ "Run Java PortabilityApi PostCommit"
+ "Run Java Flink PortableValidatesRunner Batch"
+ "Run Java Flink PortableValidatesRunner Streaming"
+ "Run Apex ValidatesRunner"
+ "Run Dataflow ValidatesRunner"
+ "Run Flink ValidatesRunner"
+ "Run Gearpump ValidatesRunner"
+ "Run Dataflow PortabilityApi ValidatesRunner"
+ "Run Samza ValidatesRunner"
+ "Run Spark ValidatesRunner"
+ "Run Python Dataflow ValidatesContainer"
+ "Run Python Dataflow ValidatesRunner"
+ "Run Python Flink ValidatesRunner"
+ "Run Python PostCommit"
+ "Run SQL PostCommit"
+ "Run Go PreCommit"
+ "Run Java PreCommit"
+ "Run Java_Examples_Dataflow PreCommit"
+ "Run JavaPortabilityApi PreCommit"
+ "Run Portable_Python PreCommit"
+ "Run PythonLint PreCommit"
+ "Run Python PreCommit"
+)
-echo "=====================Environment Variables====================="
-echo "working branch: ${branch}"
-echo "local repo dir: ~/${LOCAL_CLONE_DIR}/${BEAM_ROOT_DIR}"
-echo "====================Checking Requirement======================="
+function clean_up(){
+ echo ""
+ echo "==================== Final Cleanup ===================="
+ rm -rf ${LOCAL_BEAM_DIR}
+ echo "* Deleted workspace ${LOCAL_BEAM_DIR}"
+}
+trap clean_up EXIT
-echo "====================Checking pip==============================="
-if [[ -z `which pip` ]]; then
- echo "There is no pip installed on your machine."
- echo "Would you like to install pip with root permission? [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Refused to install pip on your machine. Exit."
- exit
- else
- echo "==================Installing pip========================="
- curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
- sudo python get-pip.py
- rm get-pip.py
- fi
-else
- pip --version
+
+echo ""
+echo "==================== 1 Checking Environment Variables ================="
+echo "* PLEASE update RELEASE_BUILD_CONFIGS in file script.config first *"
+echo ""
+echo "Verify release build against branch: ${RELEASE_BRANCH}."
+echo "Use workspace: ${LOCAL_BEAM_DIR}"
+echo ""
+echo "All environment and workflow configurations from RELEASE_BUILD_CONFIGS:"
+for i in "${RELEASE_BUILD_CONFIGS[@]}"; do
+ echo "$i = ${!i}"
+done
+echo "[Confirmation Required] Are they all provided and correctly set? [y|N]"
+read confirmation
+if [[ $confirmation != "y" ]]; then
+ echo "Please rerun this script and make sure you have the right configurations."
+ exit
fi
-echo "====================Checking virtualenv========================"
-if [[ -z `which virtualenv` ]]; then
- echo "There is no virtualenv installed on your machine."
- echo "Would you like to install virtualenv with root perrission? [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Refused to install virtualenv on your machine. Exit."
- exit
- else
- echo "==================Installing virtualenv==================="
- sudo `which pip` install --upgrade virtualenv
- fi
+
+echo ""
+echo "==================== 2 Checking Requirements ======================="
+
+echo "====================== 2.1 Checking git ========================"
+if [[ -z ${GITHUB_TOKEN} ]]; then
+ echo "Error: A Github personal access token is required to perform git push "
+ echo "under a newly cloned directory. Please manually create one from Github "
+ echo "website with guide:"
+ echo "https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line"
+ echo "Note: This token can be reused in other release scripts."
+ exit
else
- virtualenv --version
+ echo "====================== Cloning repo ======================"
+ git clone ${BEAM_REPO_URL} ${LOCAL_BEAM_DIR}
+ cd ${LOCAL_BEAM_DIR}
+ # Set upstream repo url with access token included.
+ USER_REPO_URL=https://${GITHUB_USERNAME}:${GITHUB_TOKEN}@github.com/${GITHUB_USERNAME}/beam.git
+ git remote add ${GITHUB_USERNAME} ${USER_REPO_URL}
+ # For hub access Github API.
+ export GITHUB_TOKEN=${GITHUB_TOKEN}
+ # For local git repo only. Required if global configs are not set.
+ git config user.name "${GITHUB_USERNAME}"
+ git config user.email "${GITHUB_USERNAME}@gmail.com"
fi
-echo "=====================Checking cython==========================="
-if [[ -z `which cython` ]]; then
- echo "There is no cython installed on your machine."
- echo "Would you like to install cython with root permission? [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Refused to install cython on your machine. Exit."
- exit
- else
- echo "==================Installing cython======================="
- sudo `which pip` install cython
- sudo apt-get install gcc
- sudo apt-get install python-dev
- sudo apt-get install python3-dev
- sudo apt-get install python3.5-dev
- sudo apt-get install python3.6-dev
- sudo apt-get install python3.7-dev
- fi
-else
- cython --version
-fi
-
-echo "==================Checking /usr/bin/time========================"
-if [[ `which time` != "/usr/bin/time" ]]; then
- echo "There is no usr/bin/time installed on your machine."
- echo "Would you like to install time on your machine with root permission? [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Refused to install time on your machine. Exit."
- exit
- else
- echo "==================Installing time========================="
- sudo apt-get install time
- alias time='/usr/bin/time'
- fi
-else
- which time
-fi
-
-echo "=================Checking hub========================"
-HUB_VERSION=2.5.0
+echo "====================== 2.2 Checking hub ========================"
+HUB_VERSION=2.12.0
HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
if [[ -z `which hub` ]]; then
echo "There is no hub installed on your machine."
- echo "Would you like to install hub with root permission? [y|N]"
- read confirmation
- if [[ $confirmation != "y" ]]; then
- echo "Refused to install hub. Cannot proceed into next setp."
- exit
+ if [[ "${INSTALL_HUB}" = true ]]; then
+ echo "====================== Installing hub ======================="
+ wget https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
+ tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
+ sudo ./${HUB_ARTIFACTS_NAME}/install
+ echo "eval "$(hub alias -s)"" >> ~/.bashrc
+ rm -rf ${HUB_ARTIFACTS_NAME}*
+ else
+ echo "Refused to install hub. Cannot proceed into next setp."; exit
fi
- echo "=================Installing hub======================="
- wget https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
- tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
- sudo ./${HUB_ARTIFACTS_NAME}/install
- echo "eval "$(hub alias -s)"" >> ~/.bashrc
- rm -rf ${HUB_ARTIFACTS_NAME}*
fi
hub version
-cd ~
-echo "======================Starting Clone Repo======================"
-if [[ -d ${LOCAL_CLONE_DIR} ]]; then
- rm -rf ${LOCAL_CLONE_DIR}
-fi
-mkdir ${LOCAL_CLONE_DIR}
-cd ${LOCAL_CLONE_DIR}
-git clone ${GIT_REPO_URL}
-cd ${BEAM_ROOT_DIR}
-git checkout ${branch}
-echo "==============================================================="
-echo "======================Starting Release Build==================="
-git clean -fdx
-./gradlew clean
-# If build fails, we want to catch as much errors as possible once.
-./gradlew build -PisRelease --scan --stacktrace --no-parallel --continue
-echo "==============================================================="
-
-echo "[Current Task] Run All PostCommit Tests against Release Branch"
+echo ""
+echo "==================== 3 Run Gradle Release Build & PostCommit Tests on Jenkins ==================="
+echo "[Current Task] Run Gradle release build and all PostCommit Tests against Release Branch on Jenkins."
echo "This task will create a PR against apache/beam."
echo "After PR created, you need to comment phrases listed in description in the created PR:"
-echo "[Confirmation Required] Do you want to proceed? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
- echo "[Input Required] Please enter your github repo URL forked from apache/beam:"
- read USER_REMOTE_URL
- echo "[Input Required] Please enter your github username:"
- read GITHUB_USERNAME
- echo "[Input Required] Please enter your github token:"
- read GITHUB_TOKEN
- export GITHUB_TOKEN=${GITHUB_TOKEN}
- WORKING_BRANCH=postcommit_validation_pr
- git checkout -b ${WORKING_BRANCH}
+if [[ ! -z `which hub` ]]; then
+ git checkout -b ${WORKING_BRANCH} origin/${RELEASE_BRANCH} --quiet
touch empty_file.txt
- git add empty_file.txt
- git commit -m "Add empty file in order to create PR"
- git push -f ${USER_REMOTE_URL}
- hub pull-request -o -b apache:${branch} -h ${GITHUB_USERNAME}:${WORKING_BRANCH} -F- <<<"[DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch
+ git add .
+ git commit -m "Add empty file in order to create a test PR" --quiet
+ git push -f ${GITHUB_USERNAME} --quiet
- Please comment as instructions below, one phrase per comment please:
- Run Go PostCommit
- Run Java PostCommit
- Run Java PortabilityApi PostCommit
- Run Java Flink PortableValidatesRunner Batch
- Run Java Flink PortableValidatesRunner Streaming'
- Run Apex ValidatesRunner
- Run Dataflow ValidatesRunner
- Run Flink ValidatesRunner
- Run Gearpump ValidatesRunner
- Run Dataflow PortabilityApi ValidatesRunner
- Run Samza ValidatesRunner
- Run Spark ValidatesRunner
- Run Python Dataflow ValidatesContainer
- Run Python Dataflow ValidatesRunner
- Run Python Flink ValidatesRunner
- Run Python PostCommit
- Run SQL PostCommit
- Run Go PreCommit
- Run Java PreCommit
- Run Java_Examples_Dataflow PreCommit
- Run JavaPortabilityApi PreCommit
- Run Portable_Python PreCommit
- Run Python PreCommit"
+ trigger_phrases=$(IFS=$'\n'; echo "${JOB_TRIGGER_PHRASES[*]}")
+ hub pull-request -b apache:${RELEASE_BRANCH} -h ${GITHUB_USERNAME}:${WORKING_BRANCH} -F- <<<"[DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch
+ Please comment as instructions below, one phrase per comment:
+
+ ${trigger_phrases}"
+
+ echo ""
echo "[NOTE]: Please make sure all test targets have been invoked."
echo "Please check the test results. If there is any failure, follow the policy in release guide."
fi
-
-echo "Do you want to clean local clone repo? [y|N]"
-read confirmation
-if [[ $confirmation = "y" ]]; then
- cd ~
- rm -rf ${LOCAL_CLONE_DIR}
- echo "Clean up local repo."
-fi
diff --git a/runners/apex/build.gradle b/runners/apex/build.gradle
index 988847d..06a3d33 100644
--- a/runners/apex/build.gradle
+++ b/runners/apex/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.apex')
description = "Apache Beam :: Runners :: Apex"
@@ -35,25 +35,25 @@
}
dependencies {
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":runners:core-java", configuration: "shadow")
- shadow library.java.apex_common
- shadow library.java.malhar_library
- shadow library.java.apex_engine
- shadow library.java.commons_lang3
- shadow library.java.apex_engine
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-construction-java")
+ compile project(":runners:core-java")
+ compile library.java.apex_common
+ compile library.java.malhar_library
+ compile library.java.apex_engine
+ compile library.java.commons_lang3
+ compile library.java.apex_engine
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
// ApexStateInternalsTest extends abstract StateInternalsTest
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.jackson_dataformat_yaml
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.jackson_dataformat_yaml
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesRunner project(project.path)
}
// TODO: Update this so that the generated file is added to the explicitly added instead of
@@ -98,6 +98,12 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesMetricsPusher'
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedPCollections'
+ // TODO[BEAM-8204]: figure out if we can make the test work on Apex runner, or maybe create a
+ // more meaningful category tag.
+ excludeCategories 'org.apache.beam.sdk.testing.UsesSideInputsWithDifferentCoders'
+ // TODO[BEAM-8204]: figure out if we can make the test work on Apex runner, or maybe create a
+ // new category tag and change the following line to: excludeCategories '<category tag>'.
+ exclude '**/AvroSchemaTest.class'
}
// apex runner is run in embedded mode. Increase default HeapSize
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index dfec1a0..e728460 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -62,9 +62,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PCollectionViews;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.apache.commons.io.FileUtils;
import org.apache.hadoop.conf.Configuration;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunnerRegistrar.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunnerRegistrar.java
index 577a6bb..ad89723 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunnerRegistrar.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunnerRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexYarnLauncher.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexYarnLauncher.java
index 7d05a76..d1ae4ec 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexYarnLauncher.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexYarnLauncher.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.apex;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.datatorrent.api.Attribute;
import com.datatorrent.api.Attribute.AttributeMap;
@@ -62,9 +62,9 @@
import org.apache.apex.api.Launcher.LauncherException;
import org.apache.apex.api.Launcher.ShutdownMode;
import org.apache.apex.api.YarnAppLauncher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.SerializationUtils;
import org.apache.hadoop.conf.Configuration;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslator.java
index d368606..ff22e9d 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.apex.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collections;
import java.util.List;
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** {@link Flatten.PCollections} translation to Apex operator. */
class FlattenPCollectionTranslator<T> implements TransformTranslator<Flatten.PCollections<T>> {
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
index 9f20718..cf5d11b 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.apex.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.datatorrent.api.Operator;
import com.datatorrent.api.Operator.OutputPort;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -41,7 +42,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -76,11 +78,14 @@
Map<TupleTag<?>, PValue> outputs = context.getOutputs();
PCollection<InputT> input = context.getInput();
- List<PCollectionView<?>> sideInputs = transform.getSideInputs();
+ Iterable<PCollectionView<?>> sideInputs = transform.getSideInputs().values();
DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
+
Map<TupleTag<?>, Coder<?>> outputCoders =
outputs.entrySet().stream()
.filter(e -> e.getValue() instanceof PCollection)
@@ -97,6 +102,7 @@
input.getCoder(),
outputCoders,
doFnSchemaInformation,
+ sideInputMapping,
context.getStateBackend());
Map<PCollection<?>, OutputPort<?>> ports = Maps.newHashMapWithExpectedSize(outputs.size());
@@ -124,7 +130,7 @@
}
context.addOperator(operator, ports);
context.addStream(context.getInput(), operator.input);
- if (!sideInputs.isEmpty()) {
+ if (!Iterables.isEmpty(sideInputs)) {
addSideInputs(operator.sideInput1, sideInputs, context);
}
}
@@ -139,7 +145,7 @@
Map<TupleTag<?>, PValue> outputs = context.getOutputs();
PCollection<InputT> input = context.getInput();
- List<PCollectionView<?>> sideInputs = transform.getSideInputs();
+ Iterable<PCollectionView<?>> sideInputs = transform.getSideInputs();
Map<TupleTag<?>, Coder<?>> outputCoders =
outputs.entrySet().stream()
@@ -160,6 +166,7 @@
input.getCoder(),
outputCoders,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
context.getStateBackend());
Map<PCollection<?>, OutputPort<?>> ports = Maps.newHashMapWithExpectedSize(outputs.size());
@@ -188,7 +195,7 @@
context.addOperator(operator, ports);
context.addStream(context.getInput(), operator.input);
- if (!sideInputs.isEmpty()) {
+ if (!Iterables.isEmpty(sideInputs)) {
addSideInputs(operator.sideInput1, sideInputs, context);
}
}
@@ -196,29 +203,29 @@
static void addSideInputs(
Operator.InputPort<?> sideInputPort,
- List<PCollectionView<?>> sideInputs,
+ Iterable<PCollectionView<?>> sideInputs,
TranslationContext context) {
Operator.InputPort<?>[] sideInputPorts = {sideInputPort};
- if (sideInputs.size() > sideInputPorts.length) {
+ if (Iterables.size(sideInputs) > sideInputPorts.length) {
PCollection<?> unionCollection = unionSideInputs(sideInputs, context);
context.addStream(unionCollection, sideInputPorts[0]);
} else {
// the number of ports for side inputs is fixed and each port can only take one input.
- for (int i = 0; i < sideInputs.size(); i++) {
- context.addStream(context.getViewInput(sideInputs.get(i)), sideInputPorts[i]);
+ for (int i = 0; i < Iterables.size(sideInputs); i++) {
+ context.addStream(context.getViewInput(Iterables.get(sideInputs, i)), sideInputPorts[i]);
}
}
}
private static PCollection<?> unionSideInputs(
- List<PCollectionView<?>> sideInputs, TranslationContext context) {
- checkArgument(sideInputs.size() > 1, "requires multiple side inputs");
+ Iterable<PCollectionView<?>> sideInputs, TranslationContext context) {
+ checkArgument(Iterables.size(sideInputs) > 1, "requires multiple side inputs");
// flatten and assign union tag
List<PCollection<Object>> sourceCollections = new ArrayList<>();
Map<PCollection<?>, Integer> unionTags = new HashMap<>();
- PCollection<Object> firstSideInput = context.getViewInput(sideInputs.get(0));
- for (int i = 0; i < sideInputs.size(); i++) {
- PCollectionView<?> sideInput = sideInputs.get(i);
+ PCollection<Object> firstSideInput = context.getViewInput(Iterables.get(sideInputs, 0));
+ for (int i = 0; i < Iterables.size(sideInputs); i++) {
+ PCollectionView<?> sideInput = Iterables.get(sideInputs, i);
PCollection<?> sideInputCollection = context.getViewInput(sideInput);
if (!sideInputCollection
.getWindowingStrategy()
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
index 3621355..6d1f4b0 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.apex.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.datatorrent.api.Context.PortContext;
import com.datatorrent.api.DAG;
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 009a6c2..9a56496 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.apex.translation.operators;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.datatorrent.api.Context.OperatorContext;
import com.datatorrent.api.DefaultInputPort;
@@ -54,7 +54,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index e3a9e7d..9d4b110 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.apex.translation.operators;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.datatorrent.api.Context.OperatorContext;
import com.datatorrent.api.DefaultInputPort;
@@ -86,8 +86,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -115,7 +116,7 @@
private final WindowingStrategy<?, ?> windowingStrategy;
@Bind(JavaSerializer.class)
- private final List<PCollectionView<?>> sideInputs;
+ private final Iterable<PCollectionView<?>> sideInputs;
@Bind(JavaSerializer.class)
private final Coder<WindowedValue<InputT>> windowedInputCoder;
@@ -129,6 +130,9 @@
@Bind(JavaSerializer.class)
private final DoFnSchemaInformation doFnSchemaInformation;
+ @Bind(JavaSerializer.class)
+ private final Map<String, PCollectionView<?>> sideInputMapping;
+
private StateInternalsProxy<?> currentKeyStateInternals;
private final ApexTimerInternals<Object> currentKeyTimerInternals;
@@ -152,10 +156,11 @@
TupleTag<OutputT> mainOutputTag,
List<TupleTag<?>> additionalOutputTags,
WindowingStrategy<?, ?> windowingStrategy,
- List<PCollectionView<?>> sideInputs,
+ Iterable<PCollectionView<?>> sideInputs,
Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping,
ApexStateBackend stateBackend) {
this.pipelineOptions = new SerializablePipelineOptions(pipelineOptions);
this.doFn = doFn;
@@ -186,6 +191,7 @@
TimerInternals.TimerDataCoder.of(windowingStrategy.getWindowFn().windowCoder());
this.currentKeyTimerInternals = new ApexTimerInternals<>(timerCoder);
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
if (doFn instanceof ProcessFn) {
// we know that it is keyed on byte[]
@@ -219,6 +225,7 @@
this.outputCoders = Collections.emptyMap();
this.currentKeyTimerInternals = null;
this.doFnSchemaInformation = null;
+ this.sideInputMapping = null;
}
public final transient DefaultInputPort<ApexStreamTuple<WindowedValue<InputT>>> input =
@@ -260,7 +267,7 @@
LOG.debug("\nsideInput {} {}\n", sideInputIndex, t.getValue());
}
- PCollectionView<?> sideInput = sideInputs.get(sideInputIndex);
+ PCollectionView<?> sideInput = Iterables.get(sideInputs, sideInputIndex);
sideInputHandler.addSideInputValue(sideInput, t.getValue());
List<WindowedValue<InputT>> newPushedBack = new ArrayList<>();
@@ -408,7 +415,7 @@
currentInputWatermark);
}
}
- if (sideInputs.isEmpty()) {
+ if (Iterables.isEmpty(sideInputs)) {
outputWatermark(mark);
return;
}
@@ -439,8 +446,9 @@
ApexStreamTuple.Logging.isDebugEnabled(
pipelineOptions.get().as(ApexPipelineOptions.class), this);
SideInputReader sideInputReader = NullSideInputReader.of(sideInputs);
- if (!sideInputs.isEmpty()) {
- sideInputHandler = new SideInputHandler(sideInputs, sideInputStateInternals);
+ if (!Iterables.isEmpty(sideInputs)) {
+ sideInputHandler =
+ new SideInputHandler(Lists.newArrayList(sideInputs), sideInputStateInternals);
sideInputReader = sideInputHandler;
}
@@ -476,7 +484,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
doFnInvoker = DoFnInvokers.invokerFor(doFn);
doFnInvoker.invokeSetup();
@@ -501,7 +510,8 @@
}
pushbackDoFnRunner =
- SimplePushbackSideInputDoFnRunner.create(doFnRunner, sideInputs, sideInputHandler);
+ SimplePushbackSideInputDoFnRunner.create(
+ doFnRunner, Lists.newArrayList(sideInputs), sideInputHandler);
if (doFn instanceof ProcessFn) {
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexProcessFnOperator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexProcessFnOperator.java
index 6b91bcb..f3c4078 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexProcessFnOperator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexProcessFnOperator.java
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexReadUnboundedInputOperator.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexReadUnboundedInputOperator.java
index 14952e3..a9646b5 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexReadUnboundedInputOperator.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexReadUnboundedInputOperator.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternals.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternals.java
index 64b2cff..b4028e7 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternals.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternals.java
@@ -33,10 +33,10 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.joda.time.Instant;
/**
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
index ac7a194..998007b 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
@@ -53,8 +53,8 @@
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
import org.joda.time.Instant;
/**
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
index 69be44f..3f1a9cf 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStreamTuple.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.apex.translation.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.datatorrent.api.Operator;
import java.io.DataInputStream;
diff --git a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/CoderAdapterStreamCodec.java b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/CoderAdapterStreamCodec.java
index 36869c2..3942e18 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/CoderAdapterStreamCodec.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/CoderAdapterStreamCodec.java
@@ -25,7 +25,7 @@
import java.io.Serializable;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.Coder.Context;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** The Apex {@link StreamCodec} adapter for using Beam {@link Coder}. */
public class CoderAdapterStreamCodec implements StreamCodec<Object>, Serializable {
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexRunnerTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexRunnerTest.java
index 539fc75..5d5e3a3 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexRunnerTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/ApexRunnerTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.Matchers;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
index fbb1eb9..d1ba0ac 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/examples/WordCountTest.java
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ApexGroupByKeyOperatorTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ApexGroupByKeyOperatorTest.java
index 8789bfb..40bdfe0 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ApexGroupByKeyOperatorTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ApexGroupByKeyOperatorTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Assert;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
index 704f841..c36491a 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Assert;
import org.junit.Test;
import org.slf4j.Logger;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/GroupByKeyTranslatorTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/GroupByKeyTranslatorTest.java
index 6458801..11a7327 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/GroupByKeyTranslatorTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/GroupByKeyTranslatorTest.java
@@ -43,8 +43,8 @@
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Assert;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ParDoTranslatorTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ParDoTranslatorTest.java
index e25315a..cda57aa 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ParDoTranslatorTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ParDoTranslatorTest.java
@@ -58,8 +58,8 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Assert;
import org.junit.Ignore;
import org.junit.Test;
@@ -214,6 +214,7 @@
VarIntCoder.of(),
Collections.emptyMap(),
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
new ApexStateInternals.ApexStateBackend());
operator.setup(null);
operator.beginWindow(0);
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ReadUnboundTranslatorTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ReadUnboundTranslatorTest.java
index 9b824fd..4c7dc26 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ReadUnboundTranslatorTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/ReadUnboundTranslatorTest.java
@@ -34,11 +34,11 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ContiguousSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.DiscreteDomain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Range;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ContiguousSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.DiscreteDomain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Range;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Assert;
import org.junit.Test;
import org.slf4j.Logger;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/SideInputTranslationTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/SideInputTranslationTest.java
index f2d6c0e..f579096 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/SideInputTranslationTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/SideInputTranslationTest.java
@@ -52,7 +52,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternalsTest.java b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternalsTest.java
index 29a9bdd..0bd890a 100644
--- a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternalsTest.java
+++ b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/operators/ApexTimerInternalsTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/core-construction-java/build.gradle b/runners/core-construction-java/build.gradle
index b24927d..e7f4899 100644
--- a/runners/core-construction-java/build.gradle
+++ b/runners/core-construction-java/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.core.construction')
description = "Apache Beam :: Runners :: Core Construction Java"
ext.summary = """Beam Runners Core provides utilities to aid runner authors interact with a Pipeline
@@ -33,22 +33,23 @@
}
dependencies {
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":model:job-management", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.vendored_guava_20_0
- shadow library.java.jackson_core
- shadow library.java.jackson_databind
- shadow library.java.joda_time
- shadow library.java.slf4j_api
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.jackson_annotations
- shadowTest library.java.jackson_dataformat_yaml
- shadowTest project(path: ":model:fn-execution", configuration: "shadow")
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":model:job-management", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_grpc_1_21_0
+ compile library.java.vendored_guava_26_0_jre
+ compile library.java.jackson_core
+ compile library.java.jackson_databind
+ compile library.java.joda_time
+ compile library.java.slf4j_api
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.jackson_annotations
+ testCompile library.java.jackson_dataformat_yaml
+ testCompile project(path: ":model:fn-execution", configuration: "shadow")
+ testRuntimeOnly library.java.slf4j_jdk14
}
task runExpansionService (type: JavaExec) {
@@ -56,24 +57,3 @@
classpath = sourceSets.main.runtimeClasspath
args = [project.findProperty("constructionService.port") ?: "8097"]
}
-
-task runTestExpansionService (type: JavaExec) {
- main = "org.apache.beam.runners.core.construction.expansion.TestExpansionService"
- classpath = sourceSets.test.runtimeClasspath
- args = [project.findProperty("constructionService.port") ?: "8097"]
-}
-
-task buildTestExpansionServiceJar(type: Jar) {
- dependsOn = [shadowJar, shadowTestJar]
- appendix = "testExpansionService"
- // Use zip64 mode to avoid "Archive contains more than 65535 entries".
- zip64 = true
- manifest {
- attributes(
- 'Main-Class': 'org.apache.beam.runners.core.construction.expansion.TestExpansionService'
- )
- }
- from { configurations.testRuntime.collect { it.isDirectory() ? it : zipTree(it) }}
- from sourceSets.main.output
- from sourceSets.test.output
-}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
index 53b25f7..4212916 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactServiceStager.java
@@ -50,13 +50,13 @@
import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub;
import org.apache.beam.sdk.util.MoreFutures;
import org.apache.beam.sdk.util.ThrowingSupplier;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Channel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderRegistrar.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderRegistrar.java
index 6cc195d..565bdbf 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderRegistrar.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderRegistrar.java
@@ -21,7 +21,7 @@
import java.util.Map;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Coder registrar for AvroCoder. */
@AutoService(CoderTranslatorRegistrar.class)
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderTranslator.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderTranslator.java
index 0a59c35..93fca3d 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderTranslator.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderTranslator.java
@@ -22,7 +22,7 @@
import org.apache.avro.Schema;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
/** Coder translator for AvroCoder. */
public class AvroCoderTranslator implements CoderTranslator<AvroCoder<?>> {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java
index 0729ce9..f1f30dc 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.core.construction;
import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ProtocolMessageEnum;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ProtocolMessageEnum;
/** Returns the standard URN of a given enum annotated with [(standard_urn)]. */
public class BeamUrns {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
index 30c39bf..8e1021d 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.ArrayList;
@@ -26,20 +26,19 @@
import java.util.ServiceLoader;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.SdkFunctionSpec;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Converts to and from Beam Runner API representations of {@link Coder Coders}. */
public class CoderTranslation {
// This URN says that the coder is just a UDF blob this SDK understands
// TODO: standardize such things
- public static final String JAVA_SERIALIZED_CODER_URN = "urn:beam:coders:javasdk:0.1";
+ public static final String JAVA_SERIALIZED_CODER_URN = "beam:coders:javasdk:0.1";
@VisibleForTesting
static final BiMap<Class<? extends Coder>, String> KNOWN_CODER_URNS = loadCoderURNs();
@@ -90,11 +89,9 @@
return RunnerApi.Coder.newBuilder()
.addAllComponentCoderIds(componentIds)
.setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(KNOWN_CODER_URNS.get(coder.getClass()))
- .setPayload(ByteString.copyFrom(translator.getPayload(coder)))))
+ FunctionSpec.newBuilder()
+ .setUrn(KNOWN_CODER_URNS.get(coder.getClass()))
+ .setPayload(ByteString.copyFrom(translator.getPayload(coder))))
.build();
}
@@ -111,19 +108,16 @@
RunnerApi.Coder.Builder coderBuilder = RunnerApi.Coder.newBuilder();
return coderBuilder
.setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(JAVA_SERIALIZED_CODER_URN)
- .setPayload(
- ByteString.copyFrom(SerializableUtils.serializeToByteArray(coder)))
- .build()))
+ FunctionSpec.newBuilder()
+ .setUrn(JAVA_SERIALIZED_CODER_URN)
+ .setPayload(ByteString.copyFrom(SerializableUtils.serializeToByteArray(coder)))
+ .build())
.build();
}
public static Coder<?> fromProto(RunnerApi.Coder protoCoder, RehydratedComponents components)
throws IOException {
- String coderSpecUrn = protoCoder.getSpec().getSpec().getUrn();
+ String coderSpecUrn = protoCoder.getSpec().getUrn();
if (coderSpecUrn.equals(JAVA_SERIALIZED_CODER_URN)) {
return fromCustomCoder(protoCoder);
}
@@ -132,7 +126,7 @@
private static Coder<?> fromKnownCoder(RunnerApi.Coder coder, RehydratedComponents components)
throws IOException {
- String coderUrn = coder.getSpec().getSpec().getUrn();
+ String coderUrn = coder.getSpec().getUrn();
List<Coder<?>> coderComponents = new ArrayList<>();
for (String componentId : coder.getComponentCoderIdsList()) {
Coder<?> innerCoder = components.getCoder(componentId);
@@ -145,13 +139,12 @@
"Unknown Coder URN %s. Known URNs: %s",
coderUrn,
KNOWN_CODER_URNS.values());
- return translator.fromComponents(
- coderComponents, coder.getSpec().getSpec().getPayload().toByteArray());
+ return translator.fromComponents(coderComponents, coder.getSpec().getPayload().toByteArray());
}
private static Coder<?> fromCustomCoder(RunnerApi.Coder protoCoder) throws IOException {
return (Coder<?>)
SerializableUtils.deserializeFromByteArray(
- protoCoder.getSpec().getSpec().getPayload().toByteArray(), "Custom Coder Bytes");
+ protoCoder.getSpec().getPayload().toByteArray(), "Custom Coder Bytes");
}
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
index 93b94d5..9c4e232 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CoderTranslators.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link CoderTranslator} implementations for known coder types. */
class CoderTranslators {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
index 70c9276..76881e2 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CombineTranslation.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.runners.core.construction;
+import static org.apache.beam.runners.core.construction.PTransformTranslation.COMBINE_GLOBALLY_TRANSFORM_URN;
+import static org.apache.beam.runners.core.construction.PTransformTranslation.COMBINE_GROUPED_VALUES_TRANSFORM_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.COMBINE_PER_KEY_TRANSFORM_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
-import java.util.Collections;
import java.util.Map;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.CombinePayload;
@@ -31,31 +31,32 @@
import org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.util.AppliedCombineFn;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
- * Methods for translating between {@link Combine.PerKey} {@link PTransform PTransforms} and {@link
+ * Methods for translating between {@link Combine} {@link PTransform PTransforms} and {@link
* RunnerApi.CombinePayload} protos.
*/
public class CombineTranslation {
- public static final String JAVA_SERIALIZED_COMBINE_FN_URN = "urn:beam:combinefn:javasdk:v1";
+ static final String JAVA_SERIALIZED_COMBINE_FN_URN = "beam:combinefn:javasdk:v1";
/** A {@link TransformPayloadTranslator} for {@link Combine.PerKey}. */
- public static class CombinePayloadTranslator
+ public static class CombinePerKeyPayloadTranslator
implements PTransformTranslation.TransformPayloadTranslator<Combine.PerKey<?, ?, ?>> {
- private CombinePayloadTranslator() {}
+ private CombinePerKeyPayloadTranslator() {}
@Override
public String getUrn(Combine.PerKey<?, ?, ?> transform) {
@@ -67,9 +68,12 @@
AppliedPTransform<?, ?, Combine.PerKey<?, ?, ?>> transform, SdkComponents components)
throws IOException {
if (transform.getTransform().getSideInputs().isEmpty()) {
+ GlobalCombineFn<?, ?, ?> combineFn = transform.getTransform().getFn();
+ Coder<?> accumulatorCoder =
+ extractAccumulatorCoder(combineFn, (AppliedPTransform) transform);
return FunctionSpec.newBuilder()
- .setUrn(COMBINE_PER_KEY_TRANSFORM_URN)
- .setPayload(payloadForCombine((AppliedPTransform) transform, components).toByteString())
+ .setUrn(getUrn(transform.getTransform()))
+ .setPayload(combinePayload(combineFn, accumulatorCoder, components).toByteString())
.build();
} else {
// Combines with side inputs are translated as generic composites, which have a blank
@@ -78,87 +82,156 @@
}
}
- /** Registers {@link CombinePayloadTranslator}. */
- @AutoService(TransformPayloadTranslatorRegistrar.class)
- public static class Registrar implements TransformPayloadTranslatorRegistrar {
- @Override
- public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
- getTransformPayloadTranslators() {
- return Collections.singletonMap(Combine.PerKey.class, new CombinePayloadTranslator());
+ private static <K, InputT, AccumT> Coder<AccumT> extractAccumulatorCoder(
+ GlobalCombineFn<InputT, AccumT, ?> combineFn,
+ AppliedPTransform<PCollection<KV<K, InputT>>, ?, Combine.PerKey<K, InputT, ?>> transform)
+ throws IOException {
+ try {
+ @SuppressWarnings("unchecked")
+ PCollection<KV<K, InputT>> mainInput =
+ (PCollection<KV<K, InputT>>)
+ Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(transform));
+ return combineFn.getAccumulatorCoder(
+ transform.getPipeline().getCoderRegistry(),
+ ((KvCoder<K, InputT>) mainInput.getCoder()).getValueCoder());
+ } catch (CannotProvideCoderException e) {
+ throw new IOException("Could not obtain a Coder for the accumulator", e);
}
}
}
- /** Produces a {@link RunnerApi.CombinePayload} from a {@link Combine}. */
- static <K, InputT, OutputT> CombinePayload payloadForCombine(
- final AppliedPTransform<
- PCollection<KV<K, InputT>>,
- PCollection<KV<K, OutputT>>,
- Combine.PerKey<K, InputT, OutputT>>
- combine,
- final SdkComponents components)
- throws IOException {
+ /** A {@link TransformPayloadTranslator} for {@link Combine.Globally}. */
+ public static class CombineGloballyPayloadTranslator
+ implements PTransformTranslation.TransformPayloadTranslator<Combine.Globally<?, ?>> {
+ private CombineGloballyPayloadTranslator() {}
- GlobalCombineFn<?, ?, ?> combineFn = combine.getTransform().getFn();
- try {
- return RunnerApi.CombinePayload.newBuilder()
- .setAccumulatorCoderId(
- components.registerCoder(
- extractAccumulatorCoder(combineFn, (AppliedPTransform) combine)))
- .setCombineFn(
- SdkFunctionSpec.newBuilder()
- .setEnvironmentId(components.getOnlyEnvironmentId())
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(JAVA_SERIALIZED_COMBINE_FN_URN)
- .setPayload(
- ByteString.copyFrom(
- SerializableUtils.serializeToByteArray(
- combine.getTransform().getFn())))
- .build())
- .build())
- .build();
- } catch (CannotProvideCoderException e) {
- throw new IllegalArgumentException(e);
+ @Override
+ public String getUrn(Combine.Globally<?, ?> transform) {
+ return COMBINE_GLOBALLY_TRANSFORM_URN;
+ }
+
+ @Override
+ public FunctionSpec translate(
+ AppliedPTransform<?, ?, Combine.Globally<?, ?>> transform, SdkComponents components)
+ throws IOException {
+ if (transform.getTransform().getSideInputs().isEmpty()) {
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn(transform.getTransform()))
+ .setPayload(
+ payloadForCombineGlobally((AppliedPTransform) transform, components).toByteString())
+ .build();
+ } else {
+ // Combines with side inputs are translated as generic composites, which have a blank
+ // FunctionSpec.
+ return null;
+ }
+ }
+
+ private static <InputT, AccumT> Coder<AccumT> extractAccumulatorCoder(
+ GlobalCombineFn<InputT, AccumT, ?> combineFn,
+ AppliedPTransform<PCollection<InputT>, ?, Combine.Globally<InputT, ?>> transform)
+ throws IOException {
+ try {
+ @SuppressWarnings("unchecked")
+ PCollection<InputT> mainInput =
+ (PCollection<InputT>)
+ Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(transform));
+ return combineFn.getAccumulatorCoder(
+ transform.getPipeline().getCoderRegistry(), mainInput.getCoder());
+ } catch (CannotProvideCoderException e) {
+ throw new IOException("Could not obtain a Coder for the accumulator", e);
+ }
+ }
+
+ /** Produces a {@link RunnerApi.CombinePayload} from a {@link Combine.Globally}. */
+ @VisibleForTesting
+ static <InputT, OutputT> CombinePayload payloadForCombineGlobally(
+ final AppliedPTransform<
+ PCollection<InputT>, PCollection<OutputT>, Combine.Globally<InputT, OutputT>>
+ transform,
+ final SdkComponents components)
+ throws IOException {
+ GlobalCombineFn<?, ?, ?> combineFn = transform.getTransform().getFn();
+ Coder<?> accumulatorCoder = extractAccumulatorCoder(combineFn, (AppliedPTransform) transform);
+ return combinePayload(combineFn, accumulatorCoder, components);
}
}
- @VisibleForTesting
- static CombinePayload toProto(
- AppliedPTransform<?, ?, Combine.PerKey<?, ?, ?>> combine, SdkComponents sdkComponents)
- throws IOException {
- checkArgument(
- combine.getTransform().getSideInputs().isEmpty(),
- "CombineTranslation.toProto cannot translate Combines with side inputs.");
- GlobalCombineFn<?, ?, ?> combineFn = combine.getTransform().getFn();
- try {
- Coder<?> accumulatorCoder = extractAccumulatorCoder(combineFn, (AppliedPTransform) combine);
- return RunnerApi.CombinePayload.newBuilder()
- .setAccumulatorCoderId(sdkComponents.registerCoder(accumulatorCoder))
- .setCombineFn(toProto(combineFn, sdkComponents))
- .build();
- } catch (CannotProvideCoderException e) {
- throw new IllegalArgumentException(e);
+ /** A {@link TransformPayloadTranslator} for {@link Combine.GroupedValues}. */
+ public static class CombineGroupedValuesPayloadTranslator
+ implements PTransformTranslation.TransformPayloadTranslator<Combine.GroupedValues<?, ?, ?>> {
+ private CombineGroupedValuesPayloadTranslator() {}
+
+ @Override
+ public String getUrn(Combine.GroupedValues<?, ?, ?> transform) {
+ return COMBINE_GROUPED_VALUES_TRANSFORM_URN;
+ }
+
+ @Override
+ public FunctionSpec translate(
+ AppliedPTransform<?, ?, Combine.GroupedValues<?, ?, ?>> transform, SdkComponents components)
+ throws IOException {
+ if (transform.getTransform().getSideInputs().isEmpty()) {
+ GlobalCombineFn<?, ?, ?> combineFn = transform.getTransform().getFn();
+ Coder<?> accumulatorCoder =
+ extractAccumulatorCoder(combineFn, (AppliedPTransform) transform);
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn(transform.getTransform()))
+ .setPayload(combinePayload(combineFn, accumulatorCoder, components).toByteString())
+ .build();
+ } else {
+ // Combines with side inputs are translated as generic composites, which have a blank
+ // FunctionSpec.
+ return null;
+ }
+ }
+
+ private static <K, InputT, AccumT> Coder<AccumT> extractAccumulatorCoder(
+ GlobalCombineFn<InputT, AccumT, ?> combineFn,
+ AppliedPTransform<
+ PCollection<KV<K, Iterable<InputT>>>, ?, Combine.GroupedValues<K, InputT, ?>>
+ transform)
+ throws IOException {
+ try {
+ @SuppressWarnings("unchecked")
+ PCollection<KV<K, Iterable<InputT>>> mainInput =
+ (PCollection<KV<K, Iterable<InputT>>>)
+ Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(transform));
+ KvCoder<K, Iterable<InputT>> kvCoder = (KvCoder<K, Iterable<InputT>>) mainInput.getCoder();
+ IterableCoder<InputT> iterCoder = (IterableCoder<InputT>) kvCoder.getValueCoder();
+ return combineFn.getAccumulatorCoder(
+ transform.getPipeline().getCoderRegistry(), iterCoder.getElemCoder());
+ } catch (CannotProvideCoderException e) {
+ throw new IOException("Could not obtain a Coder for the accumulator", e);
+ }
}
}
- private static <K, InputT, AccumT> Coder<AccumT> extractAccumulatorCoder(
- GlobalCombineFn<InputT, AccumT, ?> combineFn,
- AppliedPTransform<PCollection<KV<K, InputT>>, ?, Combine.PerKey<K, InputT, ?>> transform)
- throws CannotProvideCoderException {
- @SuppressWarnings("unchecked")
- PCollection<KV<K, InputT>> mainInput =
- (PCollection<KV<K, InputT>>)
- Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(transform));
- KvCoder<K, InputT> inputCoder = (KvCoder<K, InputT>) mainInput.getCoder();
- return AppliedCombineFn.withInputCoder(
- combineFn,
- transform.getPipeline().getCoderRegistry(),
- inputCoder,
- transform.getTransform().getSideInputs(),
- ((PCollection<?>) Iterables.getOnlyElement(transform.getOutputs().values()))
- .getWindowingStrategy())
- .getAccumulatorCoder();
+ /**
+ * Registers {@link TransformPayloadTranslator TransformPayloadTranslators} for {@link Combine
+ * Combines}.
+ */
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class Registrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return ImmutableMap.<Class<? extends PTransform>, TransformPayloadTranslator>builder()
+ .put(Combine.Globally.class, new CombineGloballyPayloadTranslator())
+ .put(Combine.GroupedValues.class, new CombineGroupedValuesPayloadTranslator())
+ .put(Combine.PerKey.class, new CombinePerKeyPayloadTranslator())
+ .build();
+ }
+ }
+
+ /** Produces a {@link RunnerApi.CombinePayload} from a {@link GlobalCombineFn}. */
+ private static CombinePayload combinePayload(
+ GlobalCombineFn<?, ?, ?> combineFn, Coder<?> accumulatorCoder, final SdkComponents components)
+ throws IOException {
+ return RunnerApi.CombinePayload.newBuilder()
+ .setAccumulatorCoderId(components.registerCoder(accumulatorCoder))
+ .setCombineFn(toProto(combineFn, components))
+ .build();
}
public static SdkFunctionSpec toProto(
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
index daf9c48..b89c5b6 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
/**
* Utility methods for translating a {@link View} transforms to and from {@link RunnerApi}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DeduplicatedFlattenFactory.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DeduplicatedFlattenFactory.java
index 4a8f24e..20f16c2 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DeduplicatedFlattenFactory.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DeduplicatedFlattenFactory.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link PTransformOverrideFactory} that will apply a flatten where no element appears in the
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultExpansionServiceClientFactory.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultExpansionServiceClientFactory.java
index a1425cb..1586be8 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultExpansionServiceClientFactory.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultExpansionServiceClientFactory.java
@@ -23,7 +23,7 @@
import org.apache.beam.model.expansion.v1.ExpansionApi;
import org.apache.beam.model.expansion.v1.ExpansionServiceGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
/** Default factory for ExpansionServiceClient used by External transform. */
public class DefaultExpansionServiceClientFactory implements ExpansionServiceClientFactory {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DisplayDataTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DisplayDataTranslation.java
index d1eb58b..c7cd235 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DisplayDataTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DisplayDataTranslation.java
@@ -19,8 +19,8 @@
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Any;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.BoolValue;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Any;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.BoolValue;
/** Utilities for going to/from DisplayData protos. */
public class DisplayDataTranslation {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/EmptyFlattenAsCreateFactory.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/EmptyFlattenAsCreateFactory.java
index 042eab0..82d0ace 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/EmptyFlattenAsCreateFactory.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/EmptyFlattenAsCreateFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
import org.apache.beam.sdk.coders.VoidCoder;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
index c00bb35..fc5b5f3 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java
@@ -35,11 +35,11 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardEnvironments;
import org.apache.beam.model.pipeline.v1.RunnerApi.WindowIntoPayload;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Utilities for interacting with portability {@link Environment environments}. */
public class Environments {
@@ -88,8 +88,7 @@
* See https://beam.apache.org/contribute/docker-images/ for more information on how to build a
* container.
*/
- private static final String JAVA_SDK_HARNESS_CONTAINER_URL =
- String.format("%s-docker-apache.bintray.io/beam/java", System.getenv("USER"));
+ private static final String JAVA_SDK_HARNESS_CONTAINER_URL = "apachebeam/java_sdk";
public static final Environment JAVA_SDK_HARNESS_ENVIRONMENT =
createDockerEnvironment(JAVA_SDK_HARNESS_CONTAINER_URL);
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExecutableStageTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExecutableStageTranslation.java
index bfa926d..b445e0b 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExecutableStageTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExecutableStageTranslation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.ArrayList;
@@ -28,10 +28,10 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.LinkedHashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.LinkedHashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
/**
* Utilities for converting {@link ExecutableStage}s to and from {@link RunnerApi} protocol buffers.
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java
index 6fd5c00..1a842b6 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.Collections;
@@ -38,10 +38,11 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannelBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* Cross-language external transform.
@@ -186,6 +187,11 @@
ExpansionApi.ExpansionResponse response =
DEFAULT.getExpansionServiceClient(endpoint).expand(request);
+ if (!Strings.isNullOrEmpty(response.getError())) {
+ throw new RuntimeException(
+ String.format("expansion service error: %s", response.getError()));
+ }
+
expandedComponents = response.getComponents();
expandedTransform = response.getTransform();
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslation.java
index 9e1bafa..937e982 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslation.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.List;
@@ -29,11 +29,11 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Translating External transforms to proto. */
public class ExternalTranslation {
- public static final String EXTERNAL_TRANSFORM_URN = "urn:beam:transform:external:v1";
+ public static final String EXTERNAL_TRANSFORM_URN = "beam:transform:external:v1";
/** Translator for ExpandableTransform. */
public static class ExternalTranslator
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/JavaReadViaImpulse.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/JavaReadViaImpulse.java
index 424d9b6..29c714d 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/JavaReadViaImpulse.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/JavaReadViaImpulse.java
@@ -40,14 +40,14 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Read from a Java {@link BoundedSource} via the {@link Impulse} and {@link ParDo} primitive
* transforms.
*/
public class JavaReadViaImpulse {
- private static final long DEFAULT_BUNDLE_SIZE = 64L << 20;
+ private static final long DEFAULT_BUNDLE_SIZE_BYTES = 64 * 1024 * 1024L;
public static <T> PTransform<PBegin, PCollection<T>> bounded(BoundedSource<T> source) {
return new BoundedReadViaImpulse<>(source);
@@ -77,7 +77,7 @@
public PCollection<T> expand(PBegin input) {
return input
.apply(Impulse.create())
- .apply(ParDo.of(new SplitBoundedSourceFn<>(source, DEFAULT_BUNDLE_SIZE)))
+ .apply(ParDo.of(new SplitBoundedSourceFn<>(source, DEFAULT_BUNDLE_SIZE_BYTES)))
.setCoder(new BoundedSourceCoder<>())
.apply(Reshuffle.viaRandomKey())
.apply(ParDo.of(new ReadFromBoundedSourceFn<>()))
@@ -132,11 +132,12 @@
@VisibleForTesting
static class ReadFromBoundedSourceFn<T> extends DoFn<BoundedSource<T>, T> {
@ProcessElement
- public void readSoruce(ProcessContext ctxt) throws IOException {
- BoundedSource.BoundedReader<T> reader =
- ctxt.element().createReader(ctxt.getPipelineOptions());
- for (boolean more = reader.start(); more; more = reader.advance()) {
- ctxt.outputWithTimestamp(reader.getCurrent(), reader.getCurrentTimestamp());
+ public void readSource(ProcessContext ctxt) throws IOException {
+ try (BoundedSource.BoundedReader<T> reader =
+ ctxt.element().createReader(ctxt.getPipelineOptions())) {
+ for (boolean more = reader.start(); more; more = reader.advance()) {
+ ctxt.outputWithTimestamp(reader.getCurrent(), reader.getCurrentTimestamp());
+ }
}
}
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
index 66698fd..8294fe0 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.service.AutoService;
import java.util.Map;
import java.util.Set;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.DoubleCoder;
@@ -33,11 +34,11 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** The {@link CoderTranslatorRegistrar} for coders which are shared across languages. */
@AutoService(CoderTranslatorRegistrar.class)
@@ -48,6 +49,7 @@
static final BiMap<Class<? extends Coder>, String> BEAM_MODEL_CODER_URNS =
ImmutableBiMap.<Class<? extends Coder>, String>builder()
.put(ByteArrayCoder.class, ModelCoders.BYTES_CODER_URN)
+ .put(BooleanCoder.class, ModelCoders.BOOL_CODER_URN)
.put(StringUtf8Coder.class, ModelCoders.STRING_UTF8_CODER_URN)
.put(KvCoder.class, ModelCoders.KV_CODER_URN)
.put(VarLongCoder.class, ModelCoders.INT64_CODER_URN)
@@ -66,6 +68,7 @@
static final Map<Class<? extends Coder>, CoderTranslator<? extends Coder>> BEAM_MODEL_CODERS =
ImmutableMap.<Class<? extends Coder>, CoderTranslator<? extends Coder>>builder()
.put(ByteArrayCoder.class, CoderTranslators.atomic(ByteArrayCoder.class))
+ .put(BooleanCoder.class, CoderTranslators.atomic(BooleanCoder.class))
.put(StringUtf8Coder.class, CoderTranslators.atomic(StringUtf8Coder.class))
.put(VarLongCoder.class, CoderTranslators.atomic(VarLongCoder.class))
.put(IntervalWindowCoder.class, CoderTranslators.atomic(IntervalWindowCoder.class))
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
index b925af2..8d1265c 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java
@@ -18,21 +18,21 @@
package org.apache.beam.runners.core.construction;
import static org.apache.beam.runners.core.construction.BeamUrns.getUrn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.util.Set;
import org.apache.beam.model.pipeline.v1.RunnerApi.Coder;
import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.SdkFunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardCoders;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/** Utilities and constants ot interact with coders that are part of the Beam Model. */
public class ModelCoders {
private ModelCoders() {}
public static final String BYTES_CODER_URN = getUrn(StandardCoders.Enum.BYTES);
+ public static final String BOOL_CODER_URN = getUrn(StandardCoders.Enum.BOOL);
// Where is this required explicitly, instead of implicit within WindowedValue and LengthPrefix
// coders?
public static final String INT64_CODER_URN = getUrn(StandardCoders.Enum.VARINT);
@@ -57,6 +57,7 @@
private static final Set<String> MODEL_CODER_URNS =
ImmutableSet.of(
BYTES_CODER_URN,
+ BOOL_CODER_URN,
INT64_CODER_URN,
STRING_UTF8_CODER_URN,
ITERABLE_CODER_URN,
@@ -73,16 +74,14 @@
}
public static WindowedValueCoderComponents getWindowedValueCoderComponents(Coder coder) {
- checkArgument(WINDOWED_VALUE_CODER_URN.equals(coder.getSpec().getSpec().getUrn()));
+ checkArgument(WINDOWED_VALUE_CODER_URN.equals(coder.getSpec().getUrn()));
return new AutoValue_ModelCoders_WindowedValueCoderComponents(
coder.getComponentCoderIds(0), coder.getComponentCoderIds(1));
}
public static Coder windowedValueCoder(String elementCoderId, String windowCoderId) {
return Coder.newBuilder()
- .setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(FunctionSpec.newBuilder().setUrn(WINDOWED_VALUE_CODER_URN)))
+ .setSpec(FunctionSpec.newBuilder().setUrn(WINDOWED_VALUE_CODER_URN))
.addComponentCoderIds(elementCoderId)
.addComponentCoderIds(windowCoderId)
.build();
@@ -97,15 +96,14 @@
}
public static KvCoderComponents getKvCoderComponents(Coder coder) {
- checkArgument(KV_CODER_URN.equals(coder.getSpec().getSpec().getUrn()));
+ checkArgument(KV_CODER_URN.equals(coder.getSpec().getUrn()));
return new AutoValue_ModelCoders_KvCoderComponents(
coder.getComponentCoderIds(0), coder.getComponentCoderIds(1));
}
public static Coder kvCoder(String keyCoderId, String valueCoderId) {
return Coder.newBuilder()
- .setSpec(
- SdkFunctionSpec.newBuilder().setSpec(FunctionSpec.newBuilder().setUrn(KV_CODER_URN)))
+ .setSpec(FunctionSpec.newBuilder().setUrn(KV_CODER_URN))
.addComponentCoderIds(keyCoderId)
.addComponentCoderIds(valueCoderId)
.build();
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionViewTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionViewTranslation.java
index be701eb..d9ad758 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionViewTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PCollectionViewTranslation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import org.apache.beam.model.pipeline.v1.RunnerApi;
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
/** Utilities for interacting with PCollection view protos. */
public class PCollectionViewTranslation {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
index 6af3f03..7ee07f5 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformMatchers.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A {@link PTransformMatcher} that matches {@link PTransform PTransforms} based on the class of the
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformReplacements.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformReplacements.java
index eb66627..39b586e 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformReplacements.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformReplacements.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
import java.util.Set;
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** */
public class PTransformReplacements {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 92a0350..8c73964 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.core.construction;
import static org.apache.beam.runners.core.construction.BeamUrns.getUrn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.Collection;
@@ -47,11 +47,11 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSortedSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* Utilities for converting {@link PTransform PTransforms} to {@link RunnerApi Runner API protocol
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
index b68b63f..280e2f3 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ParDoTranslation.java
@@ -20,8 +20,8 @@
import static org.apache.beam.runners.core.construction.PTransformTranslation.PAR_DO_TRANSFORM_URN;
import static org.apache.beam.sdk.transforms.reflect.DoFnSignatures.getStateSpecOrThrow;
import static org.apache.beam.sdk.transforms.reflect.DoFnSignatures.getTimerSpecOrThrow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.ArrayList;
@@ -75,21 +75,20 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** Utilities for interacting with {@link ParDo} instances and {@link ParDoPayload} protos. */
public class ParDoTranslation {
/** The URN for an unknown Java {@link DoFn}. */
- public static final String CUSTOM_JAVA_DO_FN_URN = "urn:beam:dofn:javasdk:0.1";
+ public static final String CUSTOM_JAVA_DO_FN_URN = "beam:dofn:javasdk:0.1";
/** The URN for an unknown Java {@link ViewFn}. */
- public static final String CUSTOM_JAVA_VIEW_FN_URN = "urn:beam:viewfn:javasdk:0.1";
+ public static final String CUSTOM_JAVA_VIEW_FN_URN = "beam:viewfn:javasdk:0.1";
/** The URN for an unknown Java {@link WindowMappingFn}. */
- public static final String CUSTOM_JAVA_WINDOW_MAPPING_FN_URN =
- "urn:beam:windowmappingfn:javasdk:0.1";
+ public static final String CUSTOM_JAVA_WINDOW_MAPPING_FN_URN = "beam:windowmappingfn:javasdk:0.1";
/** A {@link TransformPayloadTranslator} for {@link ParDo}. */
public static class ParDoTranslator implements TransformTranslator<MultiOutput<?, ?>> {
@@ -173,7 +172,7 @@
.map(TupleTag::getId)
.collect(Collectors.toSet());
Set<String> sideInputs =
- parDo.getSideInputs().stream()
+ parDo.getSideInputs().values().stream()
.map(s -> s.getTagInternal().getId())
.collect(Collectors.toSet());
Set<String> timerInputs = signature.timerDeclarations().keySet();
@@ -210,7 +209,11 @@
@Override
public SdkFunctionSpec translateDoFn(SdkComponents newComponents) {
return ParDoTranslation.translateDoFn(
- parDo.getFn(), parDo.getMainOutputTag(), doFnSchemaInformation, newComponents);
+ parDo.getFn(),
+ parDo.getMainOutputTag(),
+ parDo.getSideInputs(),
+ doFnSchemaInformation,
+ newComponents);
}
@Override
@@ -222,7 +225,7 @@
@Override
public Map<String, SideInput> translateSideInputs(SdkComponents components) {
Map<String, SideInput> sideInputs = new HashMap<>();
- for (PCollectionView<?> sideInput : parDo.getSideInputs()) {
+ for (PCollectionView<?> sideInput : parDo.getSideInputs().values()) {
sideInputs.put(
sideInput.getTagInternal().getId(), translateView(sideInput, components));
}
@@ -316,6 +319,28 @@
return doFnWithExecutionInformationFromProto(payload.getDoFn()).getMainOutputTag();
}
+ public static Map<String, PCollectionView<?>> getSideInputMapping(
+ AppliedPTransform<?, ?, ?> application) {
+ try {
+ return getSideInputMapping(getParDoPayload(application));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static Map<String, PCollectionView<?>> getSideInputMapping(
+ RunnerApi.PTransform pTransform) {
+ try {
+ return getSideInputMapping(getParDoPayload(pTransform));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static Map<String, PCollectionView<?>> getSideInputMapping(ParDoPayload payload) {
+ return doFnWithExecutionInformationFromProto(payload.getDoFn()).getSideInputMapping();
+ }
+
public static TupleTag<?> getMainOutputTag(AppliedPTransform<?, ?, ?> application)
throws IOException {
PTransform<?, ?> transform = application.getTransform();
@@ -360,7 +385,8 @@
throws IOException {
PTransform<?, ?> transform = application.getTransform();
if (transform instanceof ParDo.MultiOutput) {
- return ((ParDo.MultiOutput<?, ?>) transform).getSideInputs();
+ return ((ParDo.MultiOutput<?, ?>) transform)
+ .getSideInputs().values().stream().collect(Collectors.toList());
}
SdkComponents sdkComponents = SdkComponents.create(application.getPipeline().getOptions());
@@ -425,8 +451,8 @@
@Override
public RunnerApi.StateSpec dispatchValue(Coder<?> valueCoder) {
return builder
- .setValueSpec(
- RunnerApi.ValueStateSpec.newBuilder()
+ .setReadModifyWriteSpec(
+ RunnerApi.ReadModifyWriteStateSpec.newBuilder()
.setCoderId(registerCoderOrThrow(components, valueCoder)))
.build();
}
@@ -476,8 +502,9 @@
static StateSpec<?> fromProto(RunnerApi.StateSpec stateSpec, RehydratedComponents components)
throws IOException {
switch (stateSpec.getSpecCase()) {
- case VALUE_SPEC:
- return StateSpecs.value(components.getCoder(stateSpec.getValueSpec().getCoderId()));
+ case READ_MODIFY_WRITE_SPEC:
+ return StateSpecs.value(
+ components.getCoder(stateSpec.getReadModifyWriteSpec().getCoderId()));
case BAG_SPEC:
return StateSpecs.bag(components.getCoder(stateSpec.getBagSpec().getElementCoderId()));
case COMBINING_SPEC:
@@ -552,6 +579,7 @@
public static SdkFunctionSpec translateDoFn(
DoFn<?, ?> fn,
TupleTag<?> tag,
+ Map<String, PCollectionView<?>> sideInputMapping,
DoFnSchemaInformation doFnSchemaInformation,
SdkComponents components) {
return SdkFunctionSpec.newBuilder()
@@ -562,7 +590,8 @@
.setPayload(
ByteString.copyFrom(
SerializableUtils.serializeToByteArray(
- DoFnWithExecutionInformation.of(fn, tag, doFnSchemaInformation))))
+ DoFnWithExecutionInformation.of(
+ fn, tag, sideInputMapping, doFnSchemaInformation))))
.build())
.build();
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
index 23a23d1..b87f0b6 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
@@ -26,11 +26,11 @@
import java.util.Map;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.JsonFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.CaseFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.CaseFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Utilities for going to/from Runner API pipeline options.
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineResources.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineResources.java
index dd6384a..f63e082 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineResources.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineResources.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.File;
import java.io.FileOutputStream;
@@ -30,10 +30,11 @@
import java.util.List;
import java.util.stream.Collectors;
import org.apache.beam.sdk.util.ZipFiles;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnels;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnels;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
/** Utilities for working with classpath resources for pipelines. */
public class PipelineResources {
@@ -82,12 +83,14 @@
List<String> resourcesToStage, String tmpJarLocation) {
return resourcesToStage.stream()
.map(File::new)
- .filter(File::exists)
.map(
- file ->
- file.isDirectory()
- ? packageDirectoriesToStage(file, tmpJarLocation)
- : file.getAbsolutePath())
+ file -> {
+ Preconditions.checkState(
+ file.exists(), "To-be-staged file does not exist: '%s'", file);
+ return file.isDirectory()
+ ? packageDirectoriesToStage(file, tmpJarLocation)
+ : file.getAbsolutePath();
+ })
.collect(Collectors.toList());
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineTranslation.java
index 8863c66..80cfc56 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineTranslation.java
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.Pipeline.PipelineVisitor;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.runners.TransformHierarchy.Node;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
/** Utilities for going to/from Runner API pipelines. */
public class PipelineTranslation {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PrimitiveCreate.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PrimitiveCreate.java
index be5ad2b..3ffee9e 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PrimitiveCreate.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PrimitiveCreate.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** An implementation of {@link Create} that returns a primitive {@link PCollection}. */
public class PrimitiveCreate<T> extends PTransform<PBegin, PCollection<T>> {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
index 74a09dd..f5b9c7f 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReadTranslation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -39,17 +39,17 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Methods for translating {@link Read.Bounded} and {@link Read.Unbounded} {@link PTransform
* PTransformTranslation} into {@link ReadPayload} protos.
*/
public class ReadTranslation {
- private static final String JAVA_SERIALIZED_BOUNDED_SOURCE = "urn:beam:java:boundedsource:v1";
- private static final String JAVA_SERIALIZED_UNBOUNDED_SOURCE = "urn:beam:java:unboundedsource:v1";
+ private static final String JAVA_SERIALIZED_BOUNDED_SOURCE = "beam:java:boundedsource:v1";
+ private static final String JAVA_SERIALIZED_UNBOUNDED_SOURCE = "beam:java:unboundedsource:v1";
public static ReadPayload toProto(Read.Bounded<?> read, SdkComponents components) {
return ReadPayload.newBuilder()
@@ -120,7 +120,8 @@
private static SdkFunctionSpec toProto(UnboundedSource<?, ?> source, SdkComponents components) {
return SdkFunctionSpec.newBuilder()
- .setEnvironmentId(components.getOnlyEnvironmentId())
+ // Do not assign an environment. Unbounded reads are a Runner translated transform,
+ // unless, in the future, we have an adapter available for splittable DoFn.
.setSpec(
FunctionSpec.newBuilder()
.setUrn(JAVA_SERIALIZED_UNBOUNDED_SOURCE)
@@ -129,8 +130,7 @@
.build();
}
- public static UnboundedSource<?, ?> unboundedSourceFromProto(ReadPayload payload)
- throws InvalidProtocolBufferException {
+ public static UnboundedSource<?, ?> unboundedSourceFromProto(ReadPayload payload) {
checkArgument(payload.getIsBounded().equals(IsBounded.Enum.UNBOUNDED));
return (UnboundedSource<?, ?>)
SerializableUtils.deserializeFromByteArray(
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RehydratedComponents.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RehydratedComponents.java
index 6265894..fa5f700 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RehydratedComponents.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/RehydratedComponents.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.Collections;
@@ -30,9 +30,9 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
/**
* Vends Java SDK objects rehydrated from a Runner API {@link Components} collection.
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReplacementOutputs.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReplacementOutputs.java
index 9bb1d47..6f48fb7 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReplacementOutputs.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReplacementOutputs.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collections;
import java.util.HashMap;
@@ -30,8 +30,8 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TaggedPValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Utility methods for creating {@link ReplacementOutput} for known styles of {@link POutput}. */
public class ReplacementOutputs {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReshuffleTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReshuffleTranslation.java
new file mode 100644
index 0000000..954798a
--- /dev/null
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReshuffleTranslation.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import com.google.auto.service.AutoService;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.runners.core.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.Reshuffle;
+
+/**
+ * Utility methods for translating a {@link Reshuffle} to and from {@link RunnerApi}
+ * representations.
+ */
+public class ReshuffleTranslation {
+
+ static class ReshuffleTranslator implements TransformPayloadTranslator<Reshuffle<?, ?>> {
+ @Override
+ public String getUrn(Reshuffle<?, ?> transform) {
+ return PTransformTranslation.RESHUFFLE_URN;
+ }
+
+ @Override
+ public FunctionSpec translate(
+ AppliedPTransform<?, ?, Reshuffle<?, ?>> transform, SdkComponents components) {
+ return FunctionSpec.newBuilder().setUrn(getUrn(transform.getTransform())).build();
+ }
+ }
+
+ /** Registers {@link ReshuffleTranslator}. */
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class Registrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return Collections.singletonMap(Reshuffle.class, new ReshuffleTranslator());
+ }
+ }
+}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java
index 90af770..6d6eb57 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SchemaTranslation.java
@@ -19,45 +19,29 @@
import java.util.Map;
import java.util.UUID;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.LogicalType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Utility methods for translating schemas. */
public class SchemaTranslation {
- private static final BiMap<TypeName, RunnerApi.Schema.TypeName> TYPE_NAME_MAPPING =
- ImmutableBiMap.<TypeName, RunnerApi.Schema.TypeName>builder()
- .put(TypeName.BYTE, RunnerApi.Schema.TypeName.BYTE)
- .put(TypeName.INT16, RunnerApi.Schema.TypeName.INT16)
- .put(TypeName.INT32, RunnerApi.Schema.TypeName.INT32)
- .put(TypeName.INT64, RunnerApi.Schema.TypeName.INT64)
- .put(TypeName.DECIMAL, RunnerApi.Schema.TypeName.DECIMAL)
- .put(TypeName.FLOAT, RunnerApi.Schema.TypeName.FLOAT)
- .put(TypeName.DOUBLE, RunnerApi.Schema.TypeName.DOUBLE)
- .put(TypeName.STRING, RunnerApi.Schema.TypeName.STRING)
- .put(TypeName.DATETIME, RunnerApi.Schema.TypeName.DATETIME)
- .put(TypeName.BOOLEAN, RunnerApi.Schema.TypeName.BOOLEAN)
- .put(TypeName.BYTES, RunnerApi.Schema.TypeName.BYTES)
- .put(TypeName.ARRAY, RunnerApi.Schema.TypeName.ARRAY)
- .put(TypeName.MAP, RunnerApi.Schema.TypeName.MAP)
- .put(TypeName.ROW, RunnerApi.Schema.TypeName.ROW)
- .put(TypeName.LOGICAL_TYPE, RunnerApi.Schema.TypeName.LOGICAL_TYPE)
- .build();
- public static RunnerApi.Schema toProto(Schema schema) {
+ private static final String URN_BEAM_LOGICAL_DATETIME = "beam:logical_type:datetime:v1";
+ private static final String URN_BEAM_LOGICAL_DECIMAL = "beam:logical_type:decimal:v1";
+ private static final String URN_BEAM_LOGICAL_JAVASDK = "beam:logical_type:javasdk:v1";
+
+ public static SchemaApi.Schema schemaToProto(Schema schema) {
String uuid = schema.getUUID() != null ? schema.getUUID().toString() : "";
- RunnerApi.Schema.Builder builder = RunnerApi.Schema.newBuilder().setId(uuid);
+ SchemaApi.Schema.Builder builder = SchemaApi.Schema.newBuilder().setId(uuid);
for (Field field : schema.getFields()) {
- RunnerApi.Schema.Field protoField =
- toProto(
+ SchemaApi.Field protoField =
+ fieldToProto(
field,
schema.indexOf(field.getName()),
schema.getEncodingPositions().get(field.getName()));
@@ -66,60 +50,103 @@
return builder.build();
}
- private static RunnerApi.Schema.Field toProto(Field field, int fieldId, int position) {
- return RunnerApi.Schema.Field.newBuilder()
+ private static SchemaApi.Field fieldToProto(Field field, int fieldId, int position) {
+ return SchemaApi.Field.newBuilder()
.setName(field.getName())
.setDescription(field.getDescription())
- .setType(toProto(field.getType()))
+ .setType(fieldTypeToProto(field.getType()))
.setId(fieldId)
.setEncodingPosition(position)
.build();
}
- private static RunnerApi.Schema.FieldType toProto(FieldType fieldType) {
- RunnerApi.Schema.FieldType.Builder builder =
- RunnerApi.Schema.FieldType.newBuilder()
- .setTypeName(TYPE_NAME_MAPPING.get(fieldType.getTypeName()));
+ private static SchemaApi.FieldType fieldTypeToProto(FieldType fieldType) {
+ SchemaApi.FieldType.Builder builder = SchemaApi.FieldType.newBuilder();
switch (fieldType.getTypeName()) {
case ROW:
- builder.setRowSchema(toProto(fieldType.getRowSchema()));
+ builder.setRowType(
+ SchemaApi.RowType.newBuilder().setSchema(schemaToProto(fieldType.getRowSchema())));
break;
case ARRAY:
- builder.setCollectionElementType(toProto(fieldType.getCollectionElementType()));
+ builder.setArrayType(
+ SchemaApi.ArrayType.newBuilder()
+ .setElementType(fieldTypeToProto(fieldType.getCollectionElementType())));
break;
case MAP:
builder.setMapType(
- RunnerApi.Schema.MapType.newBuilder()
- .setKeyType(toProto(fieldType.getMapKeyType()))
- .setValueType(toProto(fieldType.getMapValueType()))
+ SchemaApi.MapType.newBuilder()
+ .setKeyType(fieldTypeToProto(fieldType.getMapKeyType()))
+ .setValueType(fieldTypeToProto(fieldType.getMapValueType()))
.build());
break;
case LOGICAL_TYPE:
LogicalType logicalType = fieldType.getLogicalType();
builder.setLogicalType(
- RunnerApi.Schema.LogicalType.newBuilder()
- .setId(logicalType.getIdentifier())
- .setArgs(logicalType.getArgument())
- .setBaseType(toProto(logicalType.getBaseType()))
- .setSerializedClass(
+ SchemaApi.LogicalType.newBuilder()
+ // TODO(BEAM-7855): "javasdk" types should only be a last resort. Types defined in
+ // Beam should have their own URN, and there should be a mechanism for users to
+ // register their own types by URN.
+ .setUrn(URN_BEAM_LOGICAL_JAVASDK)
+ .setPayload(
ByteString.copyFrom(SerializableUtils.serializeToByteArray(logicalType)))
+ .setRepresentation(fieldTypeToProto(logicalType.getBaseType()))
.build());
break;
-
- default:
+ // Special-case for DATETIME and DECIMAL which are logical types in portable representation,
+ // but not yet in Java. (BEAM-7554)
+ case DATETIME:
+ builder.setLogicalType(
+ SchemaApi.LogicalType.newBuilder()
+ .setUrn(URN_BEAM_LOGICAL_DATETIME)
+ .setRepresentation(fieldTypeToProto(FieldType.INT64))
+ .build());
+ break;
+ case DECIMAL:
+ builder.setLogicalType(
+ SchemaApi.LogicalType.newBuilder()
+ .setUrn(URN_BEAM_LOGICAL_DECIMAL)
+ .setRepresentation(fieldTypeToProto(FieldType.BYTES))
+ .build());
+ break;
+ case BYTE:
+ builder.setAtomicType(SchemaApi.AtomicType.BYTE);
+ break;
+ case INT16:
+ builder.setAtomicType(SchemaApi.AtomicType.INT16);
+ break;
+ case INT32:
+ builder.setAtomicType(SchemaApi.AtomicType.INT32);
+ break;
+ case INT64:
+ builder.setAtomicType(SchemaApi.AtomicType.INT64);
+ break;
+ case FLOAT:
+ builder.setAtomicType(SchemaApi.AtomicType.FLOAT);
+ break;
+ case DOUBLE:
+ builder.setAtomicType(SchemaApi.AtomicType.DOUBLE);
+ break;
+ case STRING:
+ builder.setAtomicType(SchemaApi.AtomicType.STRING);
+ break;
+ case BOOLEAN:
+ builder.setAtomicType(SchemaApi.AtomicType.BOOLEAN);
+ break;
+ case BYTES:
+ builder.setAtomicType(SchemaApi.AtomicType.BYTES);
break;
}
builder.setNullable(fieldType.getNullable());
return builder.build();
}
- public static Schema fromProto(RunnerApi.Schema protoSchema) {
+ public static Schema fromProto(SchemaApi.Schema protoSchema) {
Schema.Builder builder = Schema.builder();
Map<String, Integer> encodingLocationMap = Maps.newHashMap();
- for (RunnerApi.Schema.Field protoField : protoSchema.getFieldsList()) {
+ for (SchemaApi.Field protoField : protoSchema.getFieldsList()) {
Field field = fieldFromProto(protoField);
builder.addField(field);
encodingLocationMap.put(protoField.getName(), protoField.getEncodingPosition());
@@ -133,41 +160,76 @@
return schema;
}
- private static Field fieldFromProto(RunnerApi.Schema.Field protoField) {
+ private static Field fieldFromProto(SchemaApi.Field protoField) {
return Field.of(protoField.getName(), fieldTypeFromProto(protoField.getType()))
.withDescription(protoField.getDescription());
}
- private static FieldType fieldTypeFromProto(RunnerApi.Schema.FieldType protoFieldType) {
- TypeName typeName = TYPE_NAME_MAPPING.inverse().get(protoFieldType.getTypeName());
- FieldType fieldType;
- switch (typeName) {
- case ROW:
- fieldType = FieldType.row(fromProto(protoFieldType.getRowSchema()));
- break;
- case ARRAY:
- fieldType = FieldType.array(fieldTypeFromProto(protoFieldType.getCollectionElementType()));
- break;
- case MAP:
- fieldType =
- FieldType.map(
- fieldTypeFromProto(protoFieldType.getMapType().getKeyType()),
- fieldTypeFromProto(protoFieldType.getMapType().getValueType()));
- break;
- case LOGICAL_TYPE:
- LogicalType logicalType =
- (LogicalType)
- SerializableUtils.deserializeFromByteArray(
- protoFieldType.getLogicalType().getSerializedClass().toByteArray(),
- "logicalType");
- fieldType = FieldType.logicalType(logicalType);
- break;
- default:
- fieldType = FieldType.of(typeName);
- }
+ private static FieldType fieldTypeFromProto(SchemaApi.FieldType protoFieldType) {
+ FieldType fieldType = fieldTypeFromProtoWithoutNullable(protoFieldType);
+
if (protoFieldType.getNullable()) {
fieldType = fieldType.withNullable(true);
}
+
return fieldType;
}
+
+ private static FieldType fieldTypeFromProtoWithoutNullable(SchemaApi.FieldType protoFieldType) {
+ switch (protoFieldType.getTypeInfoCase()) {
+ case ATOMIC_TYPE:
+ switch (protoFieldType.getAtomicType()) {
+ case BYTE:
+ return FieldType.of(TypeName.BYTE);
+ case INT16:
+ return FieldType.of(TypeName.INT16);
+ case INT32:
+ return FieldType.of(TypeName.INT32);
+ case INT64:
+ return FieldType.of(TypeName.INT64);
+ case FLOAT:
+ return FieldType.of(TypeName.FLOAT);
+ case DOUBLE:
+ return FieldType.of(TypeName.DOUBLE);
+ case STRING:
+ return FieldType.of(TypeName.STRING);
+ case BOOLEAN:
+ return FieldType.of(TypeName.BOOLEAN);
+ case BYTES:
+ return FieldType.of(TypeName.BYTES);
+ case UNSPECIFIED:
+ throw new IllegalArgumentException("Encountered UNSPECIFIED AtomicType");
+ default:
+ throw new IllegalArgumentException(
+ "Encountered unknown AtomicType: " + protoFieldType.getAtomicType());
+ }
+ case ROW_TYPE:
+ return FieldType.row(fromProto(protoFieldType.getRowType().getSchema()));
+ case ARRAY_TYPE:
+ return FieldType.array(fieldTypeFromProto(protoFieldType.getArrayType().getElementType()));
+ case MAP_TYPE:
+ return FieldType.map(
+ fieldTypeFromProto(protoFieldType.getMapType().getKeyType()),
+ fieldTypeFromProto(protoFieldType.getMapType().getValueType()));
+ case LOGICAL_TYPE:
+ // Special-case for DATETIME and DECIMAL which are logical types in portable representation,
+ // but not yet in Java. (BEAM-7554)
+ String urn = protoFieldType.getLogicalType().getUrn();
+ if (urn.equals(URN_BEAM_LOGICAL_DATETIME)) {
+ return FieldType.DATETIME;
+ } else if (urn.equals(URN_BEAM_LOGICAL_DECIMAL)) {
+ return FieldType.DECIMAL;
+ } else if (urn.equals(URN_BEAM_LOGICAL_JAVASDK)) {
+ return FieldType.logicalType(
+ (LogicalType)
+ SerializableUtils.deserializeFromByteArray(
+ protoFieldType.getLogicalType().getPayload().toByteArray(), "logicalType"));
+ } else {
+ throw new IllegalArgumentException("Encountered unsupported logical type URN: " + urn);
+ }
+ default:
+ throw new IllegalArgumentException(
+ "Unexpected type_info: " + protoFieldType.getTypeInfoCase());
+ }
+ }
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
index 8362bd6..81d2cbc 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SdkComponents.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.HashSet;
@@ -37,9 +37,9 @@
import org.apache.beam.sdk.util.NameUtils;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** SDK objects that will be represented at some later point within a {@link Components} object. */
public class SdkComponents {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
index 79b92dd..2373188 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDo.java
@@ -17,10 +17,11 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
+import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
@@ -63,8 +64,8 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Instant;
/**
@@ -368,7 +369,11 @@
public SdkFunctionSpec translateDoFn(SdkComponents newComponents) {
// Schemas not yet supported on splittable DoFn.
return ParDoTranslation.translateDoFn(
- fn, pke.getMainOutputTag(), DoFnSchemaInformation.create(), newComponents);
+ fn,
+ pke.getMainOutputTag(),
+ Collections.emptyMap(),
+ DoFnSchemaInformation.create(),
+ newComponents);
}
@Override
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
index ef55460..340889d 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
import java.util.concurrent.TimeUnit;
@@ -49,7 +49,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Instant;
/**
@@ -238,6 +238,11 @@
}
@Override
+ public Object sideInput(String tagId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
public Object schemaElement(int index) {
throw new UnsupportedOperationException();
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
index e5b68c0..1b747c1 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TestStreamTranslation.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.core.construction;
import static org.apache.beam.runners.core.construction.PTransformTranslation.TEST_STREAM_TRANSFORM_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
index 0f7375a..6fefe87 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformInputs.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Map;
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Utilities for extracting subsets of inputs from an {@link AppliedPTransform}. */
public class TransformInputs {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
index 2200ef5..1d34148 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TriggerTranslation.java
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.transforms.windowing.TimestampTransform;
import org.apache.beam.sdk.transforms.windowing.Trigger;
import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index 354d08b..739fd3b 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.InputStream;
@@ -48,9 +48,9 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
index 3ab85b3..0d72861 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowIntoTranslation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.Window.Assign;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
/**
* Utility methods for translating a {@link Window.Assign} to and from {@link RunnerApi}
@@ -52,7 +52,7 @@
public FunctionSpec translate(
AppliedPTransform<?, ?, Window.Assign<?>> transform, SdkComponents components) {
return FunctionSpec.newBuilder()
- .setUrn("urn:beam:transform:window:v1")
+ .setUrn("beam:transform:window:v1")
.setPayload(
WindowIntoTranslation.toProto(transform.getTransform(), components).toByteString())
.build();
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
index 0838b27..a57aa9b 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
@@ -42,10 +42,10 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.Durations;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.Timestamps;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.Durations;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.Timestamps;
import org.joda.time.Duration;
/** Utilities for working with {@link WindowingStrategy WindowingStrategies}. */
@@ -57,6 +57,8 @@
return AccumulationMode.DISCARDING_FIRED_PANES;
case ACCUMULATING:
return AccumulationMode.ACCUMULATING_FIRED_PANES;
+ case RETRACTING:
+ return AccumulationMode.RETRACTING_FIRED_PANES;
case UNRECOGNIZED:
default:
// Whether or not it is proto that cannot recognize it (due to the version of the
@@ -77,6 +79,8 @@
return RunnerApi.AccumulationMode.Enum.DISCARDING;
case ACCUMULATING_FIRED_PANES:
return RunnerApi.AccumulationMode.Enum.ACCUMULATING;
+ case RETRACTING_FIRED_PANES:
+ return RunnerApi.AccumulationMode.Enum.RETRACTING;
default:
throw new IllegalArgumentException(
String.format(
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
index 817f2d2..e86c450 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
@@ -18,8 +18,8 @@
package org.apache.beam.runners.core.construction;
import static org.apache.beam.runners.core.construction.PTransformTranslation.WRITE_FILES_TRANSFORM_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -46,10 +46,10 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* Utility methods for translating a {@link WriteFiles} to and from {@link RunnerApi}
@@ -58,8 +58,7 @@
public class WriteFilesTranslation {
/** The URN for an unknown Java {@link FileBasedSink}. */
- public static final String CUSTOM_JAVA_FILE_BASED_SINK_URN =
- "urn:beam:file_based_sink:javasdk:0.1";
+ public static final String CUSTOM_JAVA_FILE_BASED_SINK_URN = "beam:file_based_sink:javasdk:0.1";
@VisibleForTesting
static WriteFilesPayload payloadForWriteFiles(
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionServer.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionServer.java
index b8ec70c..12b52f4 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionServer.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionServer.java
@@ -20,9 +20,9 @@
import java.io.IOException;
import java.net.InetSocketAddress;
import java.util.concurrent.TimeUnit;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.netty.NettyServerBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NettyServerBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/** A {@link Server gRPC Server} for an ExpansionService. */
public class ExpansionServer implements AutoCloseable {
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionService.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionService.java
index 7cfc5ac..ae1a3d7 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionService.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/expansion/ExpansionService.java
@@ -54,16 +54,16 @@
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.CaseFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Converter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.CaseFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Converter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -179,11 +179,15 @@
} catch (NoSuchMethodException e) {
throw new RuntimeException(
String.format(
- "The configuration class %s is missing a setter %s for %s",
- config.getClass(), setterName, fieldName),
+ "The configuration class %s is missing a setter %s for %s with type %s",
+ config.getClass(),
+ setterName,
+ fieldName,
+ coder.getEncodedTypeDescriptor().getType().getTypeName()),
e);
}
- method.invoke(config, coder.decode(entry.getValue().getPayload().newInput()));
+ method.invoke(
+ config, coder.decode(entry.getValue().getPayload().newInput(), Coder.Context.NESTED));
}
}
@@ -205,10 +209,7 @@
final String coderUrn = coderUrns.pop();
RunnerApi.Coder.Builder coderBuilder =
RunnerApi.Coder.newBuilder()
- .setSpec(
- RunnerApi.SdkFunctionSpec.newBuilder()
- .setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn(coderUrn).build())
- .build());
+ .setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn(coderUrn).build());
if (coderUrn.equals(BeamUrns.getUrn(RunnerApi.StandardCoders.Enum.ITERABLE))) {
RunnerApi.Coder elementCoder = buildProto(coderUrns, componentsBuilder);
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
index e0d6828..a67c36d 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/FusedPipeline.java
@@ -30,7 +30,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
import org.apache.beam.runners.core.construction.SyntheticComponents;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** A {@link Pipeline} which has been separated into collections of executable components. */
@AutoValue
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
index 61875c6..1a6fee4 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Map;
@@ -31,9 +31,9 @@
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -253,13 +253,13 @@
*/
private static boolean canFuseCompatibleEnvironment(
PTransformNode operation,
- Environment environmemnt,
+ Environment environment,
@SuppressWarnings("unused") PCollectionNode candidate,
@SuppressWarnings("unused") Collection<PCollectionNode> stagePCollections,
QueryablePipeline pipeline) {
// WindowInto transforms may not have an environment
Optional<Environment> operationEnvironment = pipeline.getEnvironment(operation);
- return environmemnt.equals(operationEnvironment.orElse(null));
+ return environment.equals(operationEnvironment.orElse(null));
}
private static boolean compatibleEnvironments(
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
index ecff9c7..61d9546 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.util.ArrayDeque;
@@ -43,11 +43,11 @@
import org.apache.beam.runners.core.construction.graph.OutputDeduplicator.DeduplicationResult;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -187,14 +187,7 @@
rootNode.getId(),
rootNode.getTransform().getInputsMap());
checkArgument(
- !pipeline.getEnvironment(rootNode).isPresent()
- // We allow Read transforms as root transforms. The Runner can choose whether it
- // wants to translate them natively (e.g. Java Read) or through an environment.
- || rootNode
- .getTransform()
- .getSpec()
- .getUrn()
- .equals(PTransformTranslation.READ_TRANSFORM_URN),
+ !pipeline.getEnvironment(rootNode).isPresent(),
"%s requires all root nodes to be runner-implemented %s or %s primitives, "
+ "but transform %s executes in environment %s",
GreedyPipelineFuser.class.getSimpleName(),
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
index 605e995..2334458 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayDeque;
import java.util.LinkedHashSet;
@@ -28,7 +28,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
index 9546d97..a996ac0 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
@@ -24,7 +24,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/** An {@link ExecutableStage} which is constructed with all of its initial state. */
@AutoValue
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
index c179372..fd90cb2 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
@@ -17,9 +17,10 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Comparator;
@@ -32,16 +33,17 @@
import java.util.Queue;
import java.util.Set;
import java.util.function.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ElementOrder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.EndpointPair;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import javax.annotation.Nonnull;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ElementOrder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.EndpointPair;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Graphs;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
/** Static utility methods for {@link Network} instances that are directed. */
public class Networks {
@@ -224,8 +226,11 @@
Ordering<NodeT> maximumOrdering =
new Ordering<NodeT>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public int compare(NodeT t0, NodeT t1) {
+ public int compare(@Nonnull NodeT t0, @Nonnull NodeT t1) {
return (network.outDegree(t0) - network.inDegree(t0))
- (network.outDegree(t1) - network.inDegree(t1));
}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
index f64d94d..157ece0 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.util.ArrayList;
@@ -39,8 +39,8 @@
import org.apache.beam.runners.core.construction.SyntheticComponents;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
/**
* Utilities to insert synthetic {@link PCollectionNode PCollections} for {@link PCollection
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
index aa14707..e46612c 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.core.construction.graph;
import static org.apache.beam.runners.core.construction.BeamUrns.getUrn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
import org.apache.beam.model.pipeline.v1.RunnerApi;
@@ -35,8 +35,8 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.TestStreamPayload;
import org.apache.beam.model.pipeline.v1.RunnerApi.WindowIntoPayload;
import org.apache.beam.model.pipeline.v1.RunnerApi.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* Validates well-formedness of a pipeline. It is recommended to use this class on any user-supplied
@@ -59,12 +59,6 @@
.put(getUrn(Composites.COMBINE_PER_KEY), PipelineValidator::validateCombine)
.put(getUrn(Composites.COMBINE_GLOBALLY), PipelineValidator::validateCombine)
// Nothing to validate for RESHUFFLE and WRITE_FILES
- .put(getUrn(CombineComponents.COMBINE_PGBKCV), PipelineValidator::validateCombine)
- .put(
- getUrn(CombineComponents.COMBINE_MERGE_ACCUMULATORS),
- PipelineValidator::validateCombine)
- .put(
- getUrn(CombineComponents.COMBINE_EXTRACT_OUTPUTS), PipelineValidator::validateCombine)
.put(
getUrn(CombineComponents.COMBINE_PER_KEY_PRECOMBINE),
PipelineValidator::validateCombine)
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
index c38487c..5ea867e 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ProtoOverrides.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.List;
import java.util.Map;
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
index 81b49f1..382f68a 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
@@ -33,7 +33,7 @@
import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.SPLITTABLE_SPLIT_AND_SIZE_RESTRICTIONS_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.TEST_STREAM_TRANSFORM_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayDeque;
import java.util.Collection;
@@ -60,14 +60,14 @@
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
/**
* A {@link Pipeline} which has additional methods to relate nodes in the graph relative to each
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
index d3ce0e0..3c4784f 100644
--- a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
@@ -24,7 +24,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A reference to a side input. This includes the PTransform that references the side input as well
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRenderer.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRenderer.java
new file mode 100644
index 0000000..bb30dae
--- /dev/null
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRenderer.java
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.renderer;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.runners.TransformHierarchy;
+import org.apache.beam.sdk.values.PValue;
+
+/** A DOT renderer for BEAM {@link Pipeline} DAG. */
+public class PipelineDotRenderer implements Pipeline.PipelineVisitor {
+ public static String toDotString(Pipeline pipeline) {
+ final PipelineDotRenderer visitor = new PipelineDotRenderer();
+ visitor.begin();
+ pipeline.traverseTopologically(visitor);
+ visitor.end();
+ return visitor.dotBuilder.toString();
+ }
+
+ public static String toDotString(RunnerApi.Pipeline pipeline) {
+ return PortablePipelineDotRenderer.toDotString(pipeline);
+ }
+
+ private final StringBuilder dotBuilder = new StringBuilder();
+ private final Map<TransformHierarchy.Node, Integer> nodeToId = new HashMap<>();
+ private final Map<PValue, Integer> valueToProducerNodeId = new HashMap<>();
+
+ private int indent;
+ private int nextNodeId;
+
+ private PipelineDotRenderer() {}
+
+ @Override
+ public void enterPipeline(Pipeline p) {}
+
+ @Override
+ public void leavePipeline(Pipeline pipeline) {}
+
+ @Override
+ public CompositeBehavior enterCompositeTransform(TransformHierarchy.Node node) {
+ writeLine("subgraph cluster_%d {", nextNodeId++);
+ enterBlock();
+ writeLine("label = \"%s\"", escapeString(node.getFullName()));
+ return CompositeBehavior.ENTER_TRANSFORM;
+ }
+
+ @Override
+ public void leaveCompositeTransform(TransformHierarchy.Node node) {
+ exitBlock();
+ writeLine("}");
+ }
+
+ @Override
+ public void visitPrimitiveTransform(TransformHierarchy.Node node) {
+ final int nodeId = nextNodeId++;
+ writeLine("%d [label=\"%s\"]", nodeId, escapeString(node.getTransform().getName()));
+
+ node.getOutputs().values().forEach(x -> valueToProducerNodeId.put(x, nodeId));
+
+ node.getInputs()
+ .forEach(
+ (key, value) -> {
+ final int producerId = valueToProducerNodeId.get(value);
+ String style = "solid";
+ if (node.getTransform().getAdditionalInputs().containsKey(key)) {
+ style = "dashed";
+ }
+ writeLine("%d -> %d [style=%s label=\"%s\"]", producerId, nodeId, style, "");
+ });
+ }
+
+ @Override
+ public void visitValue(PValue value, TransformHierarchy.Node producer) {}
+
+ private void begin() {
+ writeLine("digraph {");
+ enterBlock();
+ writeLine("rankdir=LR");
+ }
+
+ private void end() {
+ exitBlock();
+ writeLine("}");
+ }
+
+ private void enterBlock() {
+ indent += 4;
+ }
+
+ private void exitBlock() {
+ indent -= 4;
+ }
+
+ private void writeLine(String format, Object... args) {
+ if (indent != 0) {
+ dotBuilder.append(String.format("%-" + indent + "s", ""));
+ }
+ dotBuilder.append(String.format(format, args));
+ dotBuilder.append(System.lineSeparator());
+ }
+
+ private static String escapeString(String x) {
+ return x.replace("\"", "\\\"");
+ }
+
+ private static String shortenTag(String tag) {
+ return tag.replaceFirst(".*:([a-zA-Z#0-9]+).*", "$1");
+ }
+}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRenderer.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRenderer.java
new file mode 100644
index 0000000..415ef66
--- /dev/null
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRenderer.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.renderer;
+
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+
+/**
+ * A DOT renderer for BEAM portable {@link org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline}.
+ */
+class PortablePipelineDotRenderer {
+ private final StringBuilder dotBuilder = new StringBuilder();
+ private final Map<String, Integer> valueToProducerNodeId = new HashMap<>();
+ private int indent;
+ private int nextNodeId;
+
+ private PortablePipelineDotRenderer() {}
+
+ static String toDotString(RunnerApi.Pipeline pipeline) {
+ return new PortablePipelineDotRenderer().toDot(pipeline);
+ }
+
+ private String toDot(RunnerApi.Pipeline pipeline) {
+ final QueryablePipeline p =
+ QueryablePipeline.forTransforms(
+ pipeline.getRootTransformIdsList(), pipeline.getComponents());
+
+ begin();
+
+ for (PipelineNode.PTransformNode transform : p.getTopologicallyOrderedTransforms()) {
+ visitTransform(transform);
+ }
+
+ end();
+
+ return dotBuilder.toString();
+ }
+
+ private void visitTransform(PipelineNode.PTransformNode node) {
+ final int nodeId = nextNodeId++;
+ final RunnerApi.PTransform transform = node.getTransform();
+ writeLine(
+ "%d [label=\"%s\\n%s\"]",
+ nodeId,
+ escapeString(transform.getUniqueName()),
+ escapeString(transform.getSpec().getUrn()));
+
+ transform.getOutputsMap().values().forEach(x -> valueToProducerNodeId.put(x, nodeId));
+
+ transform
+ .getInputsMap()
+ .forEach(
+ (key, value) -> {
+ final int producerId = valueToProducerNodeId.get(value);
+ String style = "solid";
+ writeLine(
+ "%d -> %d [style=%s label=\"%s\"]",
+ producerId,
+ nodeId,
+ style,
+ escapeString(value.substring(value.lastIndexOf('_') + 1)));
+ });
+ }
+
+ private void begin() {
+ writeLine("digraph {");
+ enterBlock();
+ writeLine("rankdir=LR");
+ }
+
+ private void end() {
+ exitBlock();
+ writeLine("}");
+ }
+
+ private void enterBlock() {
+ indent += 4;
+ }
+
+ private void exitBlock() {
+ indent -= 4;
+ }
+
+ private void writeLine(String format, Object... args) {
+ if (indent != 0) {
+ dotBuilder.append(String.format("%-" + indent + "s", ""));
+ }
+ dotBuilder.append(String.format(format, args));
+ dotBuilder.append(System.lineSeparator());
+ }
+
+ private static String escapeString(String x) {
+ return x.replace("\"", "\\\"");
+ }
+}
diff --git a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/package-info.java b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/package-info.java
new file mode 100644
index 0000000..a557304
--- /dev/null
+++ b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Classes used to render Pipelines. */
+package org.apache.beam.runners.core.construction.renderer;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
index c2f3802..99c14e5 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
@@ -33,12 +33,12 @@
import java.util.Set;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
import org.apache.beam.runners.core.construction.ArtifactServiceStager.StagedFile;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
index b1bc3af..dc28b79 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CoderTranslationTest.java
@@ -33,6 +33,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
@@ -47,8 +48,8 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -62,6 +63,7 @@
private static final Set<StructuredCoder<?>> KNOWN_CODERS =
ImmutableSet.<StructuredCoder<?>>builder()
.add(ByteArrayCoder.of())
+ .add(BooleanCoder.of())
.add(KvCoder.of(VarLongCoder.of(), VarLongCoder.of()))
.add(VarLongCoder.of())
.add(StringUtf8Coder.of())
@@ -151,7 +153,7 @@
if (KNOWN_CODERS.contains(coder)) {
for (RunnerApi.Coder encodedCoder : encodedComponents.getCodersMap().values()) {
assertThat(
- encodedCoder.getSpec().getSpec().getUrn(),
+ encodedCoder.getSpec().getUrn(),
not(equalTo(CoderTranslation.JAVA_SERIALIZED_CODER_URN)));
}
}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CombineTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CombineTranslationTest.java
index ef93515..73b26a3 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CombineTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CombineTranslationTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.junit.Assert.assertEquals;
import java.io.IOException;
@@ -43,7 +43,7 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -80,13 +80,13 @@
public void testToProto() throws Exception {
PCollection<Integer> input = pipeline.apply(Create.of(1, 2, 3));
input.apply(Combine.globally(combineFn));
- final AtomicReference<AppliedPTransform<?, ?, Combine.PerKey<?, ?, ?>>> combine =
+ final AtomicReference<AppliedPTransform<?, ?, Combine.Globally<?, ?>>> combine =
new AtomicReference<>();
pipeline.traverseTopologically(
new PipelineVisitor.Defaults() {
@Override
public void leaveCompositeTransform(Node node) {
- if (node.getTransform() instanceof Combine.PerKey) {
+ if (node.getTransform() instanceof Combine.Globally) {
checkState(combine.get() == null);
combine.set((AppliedPTransform) node.toAppliedPTransform(getPipeline()));
}
@@ -97,7 +97,9 @@
SdkComponents sdkComponents = SdkComponents.create();
sdkComponents.registerEnvironment(Environments.createDockerEnvironment("java"));
- CombinePayload combineProto = CombineTranslation.toProto(combine.get(), sdkComponents);
+ CombinePayload combineProto =
+ CombineTranslation.CombineGloballyPayloadTranslator.payloadForCombineGlobally(
+ (AppliedPTransform) combine.get(), sdkComponents);
RunnerApi.Components componentsProto = sdkComponents.toComponents();
assertEquals(
@@ -121,13 +123,13 @@
PCollection<Integer> input = pipeline.apply(Create.of(1, 2, 3));
CombineFnWithContext<Integer, int[], Integer> combineFn = new TestCombineFnWithContext();
input.apply(Combine.globally(combineFn).withoutDefaults());
- final AtomicReference<AppliedPTransform<?, ?, Combine.PerKey<?, ?, ?>>> combine =
+ final AtomicReference<AppliedPTransform<?, ?, Combine.Globally<?, ?>>> combine =
new AtomicReference<>();
pipeline.traverseTopologically(
new PipelineVisitor.Defaults() {
@Override
public void leaveCompositeTransform(Node node) {
- if (node.getTransform() instanceof Combine.PerKey) {
+ if (node.getTransform() instanceof Combine.Globally) {
checkState(combine.get() == null);
combine.set((AppliedPTransform) node.toAppliedPTransform(getPipeline()));
}
@@ -138,7 +140,9 @@
SdkComponents sdkComponents = SdkComponents.create();
sdkComponents.registerEnvironment(Environments.createDockerEnvironment("java"));
- CombinePayload combineProto = CombineTranslation.toProto(combine.get(), sdkComponents);
+ CombinePayload combineProto =
+ CombineTranslation.CombineGloballyPayloadTranslator.payloadForCombineGlobally(
+ (AppliedPTransform) combine.get(), sdkComponents);
RunnerApi.Components componentsProto = sdkComponents.toComponents();
assertEquals(
@@ -168,13 +172,13 @@
};
input.apply(Combine.globally(combineFn).withSideInputs(sideInputs).withoutDefaults());
- final AtomicReference<AppliedPTransform<?, ?, Combine.PerKey<?, ?, ?>>> combine =
+ final AtomicReference<AppliedPTransform<?, ?, Combine.Globally<?, ?>>> combine =
new AtomicReference<>();
pipeline.traverseTopologically(
new PipelineVisitor.Defaults() {
@Override
public void leaveCompositeTransform(Node node) {
- if (node.getTransform() instanceof Combine.PerKey) {
+ if (node.getTransform() instanceof Combine.Globally) {
checkState(combine.get() == null);
combine.set((AppliedPTransform) node.toAppliedPTransform(getPipeline()));
}
@@ -183,7 +187,9 @@
SdkComponents sdkComponents = SdkComponents.create();
sdkComponents.registerEnvironment(Environments.createDockerEnvironment("java"));
- CombinePayload payload = CombineTranslation.toProto(combine.get(), sdkComponents);
+ CombinePayload payload =
+ CombineTranslation.CombineGloballyPayloadTranslator.payloadForCombineGlobally(
+ (AppliedPTransform) combine.get(), sdkComponents);
}
}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
index 251ea12..1cedd5d 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CommonCoderTest.java
@@ -18,8 +18,8 @@
package org.apache.beam.runners.core.construction;
import static org.apache.beam.runners.core.construction.BeamUrns.getUrn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.instanceOf;
@@ -46,6 +46,7 @@
import java.util.Map;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardCoders;
+import org.apache.beam.sdk.coders.BooleanCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.ByteCoder;
import org.apache.beam.sdk.coders.Coder;
@@ -64,11 +65,11 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharStreams;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
@@ -86,6 +87,7 @@
private static final Map<String, Class<?>> coders =
ImmutableMap.<String, Class<?>>builder()
.put(getUrn(StandardCoders.Enum.BYTES), ByteCoder.class)
+ .put(getUrn(StandardCoders.Enum.BOOL), BooleanCoder.class)
.put(getUrn(StandardCoders.Enum.STRING_UTF8), StringUtf8Coder.class)
.put(getUrn(StandardCoders.Enum.KV), KvCoder.class)
.put(getUrn(StandardCoders.Enum.VARINT), VarLongCoder.class)
@@ -222,6 +224,8 @@
String s = coderSpec.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
return ((String) value).getBytes(StandardCharsets.ISO_8859_1);
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ return value;
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
return value;
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
@@ -291,6 +295,8 @@
String s = coder.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
return ByteArrayCoder.of();
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ return BooleanCoder.of();
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
return StringUtf8Coder.of();
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
@@ -334,6 +340,8 @@
String s = coder.getUrn();
if (s.equals(getUrn(StandardCoders.Enum.BYTES))) {
assertThat(expectedValue, equalTo(actualValue));
+ } else if (s.equals(getUrn(StandardCoders.Enum.BOOL))) {
+ assertEquals(expectedValue, actualValue);
} else if (s.equals(getUrn(StandardCoders.Enum.STRING_UTF8))) {
assertEquals(expectedValue, actualValue);
} else if (s.equals(getUrn(StandardCoders.Enum.KV))) {
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslationTest.java
index eb30869..bf47ff5 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/CreatePCollectionViewTranslationTest.java
@@ -30,7 +30,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PCollectionViews;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.values.PCollectionViews.TypeDescriptorSupplier;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -50,12 +52,16 @@
CreatePCollectionView.of(
PCollectionViews.singletonView(
testPCollection,
+ (TypeDescriptorSupplier<String>) () -> TypeDescriptors.strings(),
testPCollection.getWindowingStrategy(),
false,
null,
StringUtf8Coder.of())),
CreatePCollectionView.of(
- PCollectionViews.listView(testPCollection, testPCollection.getWindowingStrategy())));
+ PCollectionViews.listView(
+ testPCollection,
+ (TypeDescriptorSupplier<String>) () -> TypeDescriptors.strings(),
+ testPCollection.getWindowingStrategy())));
}
@Parameter(0)
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/EnvironmentsTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/EnvironmentsTest.java
index 3df9920..5cbe98a 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/EnvironmentsTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/EnvironmentsTest.java
@@ -223,8 +223,7 @@
public void getEnvironmentRead() throws IOException {
SdkComponents components = SdkComponents.create();
components.registerEnvironment(Environments.createDockerEnvironment("java"));
- ReadPayload payload =
- ReadTranslation.toProto(Read.from(CountingSource.unbounded()), components);
+ ReadPayload payload = ReadTranslation.toProto(Read.from(CountingSource.upTo(10)), components);
RehydratedComponents rehydratedComponents =
RehydratedComponents.forComponents(components.toComponents());
PTransform builder =
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExternalTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExternalTest.java
index 1e35bd4..397b444 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExternalTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExternalTest.java
@@ -32,18 +32,18 @@
import org.apache.beam.sdk.transforms.Filter;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ConnectivityState;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ConnectivityState;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Rule;
@@ -61,14 +61,12 @@
private static final String TEST_URN_LE = "le";
private static final String TEST_URN_MULTI = "multi";
- private static String pythonServerCommand;
private static Integer expansionPort;
private static String localExpansionAddr;
private static Server localExpansionServer;
@BeforeClass
public static void setUp() throws IOException {
- pythonServerCommand = System.getProperty("pythonTestExpansionCommand");
expansionPort = Integer.valueOf(System.getProperty("expansionPort"));
int localExpansionPort = expansionPort + 100;
localExpansionAddr = String.format("localhost:%s", localExpansionPort);
@@ -86,26 +84,27 @@
@Test
@Category({ValidatesRunner.class, UsesCrossLanguageTransforms.class})
public void expandSingleTest() {
- PCollection<Integer> col =
+ PCollection<String> col =
testPipeline
- .apply(Create.of(1, 2, 3))
+ .apply(Create.of("1", "2", "3"))
.apply(External.of(TEST_URN_SIMPLE, new byte[] {}, localExpansionAddr));
- PAssert.that(col).containsInAnyOrder(2, 3, 4);
+ PAssert.that(col).containsInAnyOrder("Simple(1)", "Simple(2)", "Simple(3)");
testPipeline.run();
}
@Test
@Category({ValidatesRunner.class, UsesCrossLanguageTransforms.class})
public void expandMultipleTest() {
- PCollection<Integer> pcol =
+ PCollection<String> pcol =
testPipeline
- .apply(Create.of(1, 2, 3))
- .apply("add one", External.of(TEST_URN_SIMPLE, new byte[] {}, localExpansionAddr))
+ .apply(Create.of(1, 2, 3, 4, 5, 6))
.apply(
"filter <=3",
- External.of(TEST_URN_LE, "3".getBytes(StandardCharsets.UTF_8), localExpansionAddr));
+ External.of(TEST_URN_LE, "3".getBytes(StandardCharsets.UTF_8), localExpansionAddr))
+ .apply(MapElements.into(TypeDescriptors.strings()).via(Object::toString))
+ .apply("put simple", External.of(TEST_URN_SIMPLE, new byte[] {}, localExpansionAddr));
- PAssert.that(pcol).containsInAnyOrder(2, 3);
+ PAssert.that(pcol).containsInAnyOrder("Simple(1)", "Simple(2)", "Simple(3)");
testPipeline.run();
}
@@ -123,20 +122,10 @@
testPipeline.run();
}
- private Process runCommandline(String command) {
- ProcessBuilder builder = new ProcessBuilder("sh", "-c", command);
- try {
- return builder.start();
- } catch (IOException e) {
- throw new AssertionError("process launch failed.");
- }
- }
-
@Test
@Category({ValidatesRunner.class, UsesCrossLanguageTransforms.class})
public void expandPythonTest() {
String target = String.format("localhost:%s", expansionPort);
- Process p = runCommandline(String.format("%s -p %s", pythonServerCommand, expansionPort));
try {
ManagedChannel channel = ManagedChannelBuilder.forTarget(target).build();
ConnectivityState state = channel.getState(true);
@@ -150,17 +139,17 @@
testPipeline
.apply(Create.of("1", "2", "2", "3", "3", "3"))
.apply(
- "toBytes",
- MapElements.into(new TypeDescriptor<byte[]>() {}).via(String::getBytes))
- .apply(External.<byte[]>of("count_per_element_bytes", new byte[] {}, target))
- .apply("toString", MapElements.into(TypeDescriptors.strings()).via(String::new));
+ External.<KV<String, Integer>>of(
+ "beam:transforms:xlang:count", new byte[] {}, target))
+ .apply(
+ "toString",
+ MapElements.into(TypeDescriptors.strings())
+ .via(x -> String.format("%s->%s", x.getKey(), x.getValue())));
PAssert.that(pCol).containsInAnyOrder("1->1", "2->2", "3->3");
testPipeline.run();
} catch (InterruptedException e) {
throw new RuntimeException("interrupted.");
- } finally {
- p.destroyForcibly();
}
}
@@ -175,7 +164,9 @@
public Map<String, ExpansionService.TransformProvider> knownTransforms() {
return ImmutableMap.of(
TEST_URN_SIMPLE,
- spec -> MapElements.into(TypeDescriptors.integers()).via((Integer x) -> x + 1),
+ spec ->
+ MapElements.into(TypeDescriptors.strings())
+ .via((String x) -> String.format("Simple(%s)", x)),
TEST_URN_LE,
spec -> Filter.lessThanEq(Integer.parseInt(spec.getPayload().toStringUtf8())),
TEST_URN_MULTI,
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ForwardingPTransformTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ForwardingPTransformTest.java
index 6b42710..ca61ae1 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ForwardingPTransformTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ForwardingPTransformTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/InMemoryArtifactStagerService.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/InMemoryArtifactStagerService.java
index b49fe03..cb850bd 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/InMemoryArtifactStagerService.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/InMemoryArtifactStagerService.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
@@ -35,8 +35,8 @@
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest.ContentCase;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
/**
* An {@link ArtifactStagingServiceImplBase ArtifactStagingService} which stores the bytes of the
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/JavaReadViaImpulseTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/JavaReadViaImpulseTest.java
index 292d467..7bdb156 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/JavaReadViaImpulseTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/JavaReadViaImpulseTest.java
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ModelCodersTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ModelCodersTest.java
index b63b712..74912fc 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ModelCodersTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ModelCodersTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Coder;
import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.MessageWithComponents;
-import org.apache.beam.model.pipeline.v1.RunnerApi.SdkFunctionSpec;
import org.apache.beam.runners.core.construction.ModelCoders.KvCoderComponents;
import org.apache.beam.runners.core.construction.ModelCoders.WindowedValueCoderComponents;
import org.apache.beam.sdk.coders.ByteArrayCoder;
@@ -72,9 +71,7 @@
thrown.expect(IllegalArgumentException.class);
ModelCoders.getWindowedValueCoderComponents(
Coder.newBuilder()
- .setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(FunctionSpec.newBuilder().setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN)))
+ .setSpec(FunctionSpec.newBuilder().setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN))
.build());
}
@@ -82,7 +79,7 @@
public void windowedValueCoderComponentsNoUrn() {
thrown.expect(IllegalArgumentException.class);
ModelCoders.getWindowedValueCoderComponents(
- Coder.newBuilder().setSpec(SdkFunctionSpec.getDefaultInstance()).build());
+ Coder.newBuilder().setSpec(FunctionSpec.getDefaultInstance()).build());
}
@Test
@@ -104,9 +101,7 @@
thrown.expect(IllegalArgumentException.class);
ModelCoders.getKvCoderComponents(
Coder.newBuilder()
- .setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(FunctionSpec.newBuilder().setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN)))
+ .setSpec(FunctionSpec.newBuilder().setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN))
.build());
}
@@ -114,6 +109,6 @@
public void kvCoderComponentsNoUrn() {
thrown.expect(IllegalArgumentException.class);
ModelCoders.getKvCoderComponents(
- Coder.newBuilder().setSpec(SdkFunctionSpec.getDefaultInstance()).build());
+ Coder.newBuilder().setSpec(FunctionSpec.getDefaultInstance()).build());
}
}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionTranslationTest.java
index e21de0e..3ff1b67 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionTranslationTest.java
@@ -52,7 +52,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionViewTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionViewTranslationTest.java
index 1b711c7..765a339 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionViewTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PCollectionViewTranslationTest.java
@@ -22,6 +22,7 @@
import org.apache.beam.sdk.transforms.Materialization;
import org.apache.beam.sdk.transforms.ViewFn;
import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -63,6 +64,11 @@
}
@Override
+ public TypeDescriptor<Object> getTypeDescriptor() {
+ return new TypeDescriptor<Object>() {};
+ }
+
+ @Override
public boolean equals(Object obj) {
return obj instanceof TestViewFn;
}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
index 7f4ebda..3260d81 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformMatchersTest.java
@@ -66,12 +66,14 @@
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PCollectionViews;
+import org.apache.beam.sdk.values.PCollectionViews.IterableViewFn;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.junit.Rule;
@@ -405,7 +407,8 @@
PCollectionView<Iterable<Integer>> view = input.apply(View.asIterable());
// Purposely create a subclass to get a different class then what was expected.
- ViewFn<?, ?> viewFn = new PCollectionViews.IterableViewFn() {};
+ IterableViewFn<Integer> viewFn =
+ new PCollectionViews.IterableViewFn<Integer>(() -> TypeDescriptors.integers()) {};
CreatePCollectionView<?, ?> createView = CreatePCollectionView.of(view);
PTransformMatcher matcher = PTransformMatchers.createViewWithViewFn(viewFn.getClass());
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformReplacementsTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformReplacementsTest.java
index d3cd0fd..bc26aa3 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformReplacementsTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformReplacementsTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
index f0cc96a..2c422a5 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
@@ -52,7 +52,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ParDoTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ParDoTranslationTest.java
index a5f6620..caff55a 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ParDoTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ParDoTranslationTest.java
@@ -62,7 +62,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
@@ -122,7 +122,7 @@
assertThat(ParDoTranslation.getDoFn(payload), equalTo(parDo.getFn()));
assertThat(ParDoTranslation.getMainOutputTag(payload), equalTo(parDo.getMainOutputTag()));
- for (PCollectionView<?> view : parDo.getSideInputs()) {
+ for (PCollectionView<?> view : parDo.getSideInputs().values()) {
payload.getSideInputsOrThrow(view.getTagInternal().getId());
}
}
@@ -148,7 +148,7 @@
// Decode
ParDoPayload parDoPayload = ParDoPayload.parseFrom(protoTransform.getSpec().getPayload());
- for (PCollectionView<?> view : parDo.getSideInputs()) {
+ for (PCollectionView<?> view : parDo.getSideInputs().values()) {
SideInput sideInput = parDoPayload.getSideInputsOrThrow(view.getTagInternal().getId());
PCollectionView<?> restoredView =
PCollectionViewTranslation.viewFromProto(
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
index 022ee39..a482d02 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
@@ -30,10 +30,10 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.NullValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Value;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.NullValue;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Value;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineResourcesTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineResourcesTest.java
index 2921b76..27b42e2 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineResourcesTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineResourcesTest.java
@@ -18,9 +18,8 @@
package org.apache.beam.runners.core.construction;
import static junit.framework.TestCase.assertTrue;
-import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertThrows;
import java.io.File;
import java.io.IOException;
@@ -29,7 +28,7 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -77,17 +76,17 @@
}
@Test
- public void testRemovingNonexistentFilesFromFilesToStage() throws IOException {
+ public void testFailOnNonExistingPaths() throws IOException {
String nonexistentFilePath = tmpFolder.getRoot().getPath() + "/nonexistent/file";
String existingFilePath = tmpFolder.newFile("existingFile").getAbsolutePath();
String temporaryLocation = tmpFolder.newFolder().getAbsolutePath();
List<String> filesToStage = Arrays.asList(nonexistentFilePath, existingFilePath);
- List<String> expectedFilesToStage = Arrays.asList(existingFilePath);
- List<String> result = PipelineResources.prepareFilesForStaging(filesToStage, temporaryLocation);
-
- assertThat(result, is(expectedFilesToStage));
+ assertThrows(
+ "To-be-staged file does not exist: ",
+ IllegalStateException.class,
+ () -> PipelineResources.prepareFilesForStaging(filesToStage, temporaryLocation));
}
@Test
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java
index 131118a..e149e66 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineTranslationTest.java
@@ -52,7 +52,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReadTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReadTranslationTest.java
index 2709cf2..3a29e73 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReadTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReadTranslationTest.java
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.io.UnboundedSource.CheckpointMark;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
@@ -84,7 +84,7 @@
UnboundedSource<?, ?> unboundedSource = (UnboundedSource<?, ?>) this.source;
Read.Unbounded<?> unboundedRead = Read.from(unboundedSource);
SdkComponents components = SdkComponents.create();
- components.registerEnvironment(Environments.createDockerEnvironment("java"));
+ // No environment set for unbounded sources
ReadPayload payload = ReadTranslation.toProto(unboundedRead, components);
assertThat(payload.getIsBounded(), equalTo(RunnerApi.IsBounded.Enum.UNBOUNDED));
UnboundedSource<?, ?> deserializedSource = ReadTranslation.unboundedSourceFromProto(payload);
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReplacementOutputsTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReplacementOutputsTest.java
index 982df82..6ca2fe4 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReplacementOutputsTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReplacementOutputsTest.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.values.TaggedPValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReshuffleTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReshuffleTranslationTest.java
new file mode 100644
index 0000000..48e849e
--- /dev/null
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReshuffleTranslationTest.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.apache.beam.runners.core.construction.PTransformTranslation.RESHUFFLE_URN;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link ReshuffleTranslation}. */
+@RunWith(JUnit4.class)
+public class ReshuffleTranslationTest {
+
+ /**
+ * Tests that the translator is registered so the URN can be retrieved (the only thing you can
+ * meaningfully do with a {@link Reshuffle}).
+ */
+ @Test
+ public void testUrnRetrievable() throws Exception {
+ assertThat(PTransformTranslation.urnForTransform(Reshuffle.of()), equalTo(RESHUFFLE_URN));
+ }
+}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
new file mode 100644
index 0000000..2020814
--- /dev/null
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.schemas.LogicalTypes;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+/** Tests for {@link SchemaTranslation}. */
+public class SchemaTranslationTest {
+
+ /** Tests round-trip proto encodings for {@link Schema}. */
+ @RunWith(Parameterized.class)
+ public static class ToFromProtoTest {
+ @Parameters(name = "{index}: {0}")
+ public static Iterable<Schema> data() {
+ return ImmutableList.<Schema>builder()
+ .add(Schema.of(Field.of("string", FieldType.STRING)))
+ .add(
+ Schema.of(
+ Field.of("boolean", FieldType.BOOLEAN),
+ Field.of("byte", FieldType.BYTE),
+ Field.of("int16", FieldType.INT16),
+ Field.of("int32", FieldType.INT32),
+ Field.of("int64", FieldType.INT64)))
+ .add(
+ Schema.of(
+ Field.of(
+ "row",
+ FieldType.row(
+ Schema.of(
+ Field.of("foo", FieldType.STRING),
+ Field.of("bar", FieldType.DOUBLE),
+ Field.of("baz", FieldType.BOOLEAN))))))
+ .add(
+ Schema.of(
+ Field.of(
+ "array(array(int64)))",
+ FieldType.array(FieldType.array(FieldType.INT64.withNullable(true))))))
+ .add(
+ Schema.of(
+ Field.of("nullable", FieldType.STRING.withNullable(true)),
+ Field.of("non_nullable", FieldType.STRING.withNullable(false))))
+ .add(
+ Schema.of(
+ Field.of("decimal", FieldType.DECIMAL), Field.of("datetime", FieldType.DATETIME)))
+ .add(
+ Schema.of(Field.of("logical", FieldType.logicalType(LogicalTypes.FixedBytes.of(24)))))
+ .build();
+ }
+
+ @Parameter(0)
+ public Schema schema;
+
+ @Test
+ public void toAndFromProto() throws Exception {
+ SchemaApi.Schema schemaProto = SchemaTranslation.schemaToProto(schema);
+
+ Schema decodedSchema = SchemaTranslation.fromProto(schemaProto);
+ assertThat(decodedSchema, equalTo(schema));
+ }
+ }
+}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TestStreamTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TestStreamTranslationTest.java
index 4f5006a..98960b6 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TestStreamTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TestStreamTranslationTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TriggerTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TriggerTranslationTest.java
index 8fd2583..3dfc233 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TriggerTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TriggerTranslationTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.windowing.Never;
import org.apache.beam.sdk.transforms.windowing.Repeatedly;
import org.apache.beam.sdk.transforms.windowing.Trigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
index 49df063..57aadc7 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
@@ -58,9 +58,9 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslationTest.java
index ddf92fd..2db4d70 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowIntoTranslationTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.junit.Assert.assertEquals;
import java.util.concurrent.atomic.AtomicReference;
@@ -38,8 +38,8 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.Window.Assign;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
index d61fb34..9f50c0b 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -85,6 +85,13 @@
.withMode(AccumulationMode.DISCARDING_FIRED_PANES)
.withTrigger(REPRESENTATIVE_TRIGGER)
.withAllowedLateness(Duration.millis(93))
+ .withTimestampCombiner(TimestampCombiner.LATEST)),
+ toProtoAndBackSpec(
+ WindowingStrategy.of(REPRESENTATIVE_WINDOW_FN)
+ .withClosingBehavior(ClosingBehavior.FIRE_IF_NON_EMPTY)
+ .withMode(AccumulationMode.RETRACTING_FIRED_PANES)
+ .withTrigger(REPRESENTATIVE_TRIGGER)
+ .withAllowedLateness(Duration.millis(100))
.withTimestampCombiner(TimestampCombiner.LATEST)));
}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WriteFilesTranslationTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WriteFilesTranslationTest.java
index 6c42246..8b7005a 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WriteFilesTranslationTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WriteFilesTranslationTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/ExpansionServiceTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/ExpansionServiceTest.java
index 0005147..3e5cf51 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/ExpansionServiceTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/ExpansionServiceTest.java
@@ -46,11 +46,11 @@
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Impulse;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/TestExpansionService.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/TestExpansionService.java
deleted file mode 100644
index 3ec867a..0000000
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/TestExpansionService.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.core.construction.expansion;
-
-import com.google.auto.service.AutoService;
-import java.util.Map;
-import org.apache.beam.sdk.transforms.Count;
-import org.apache.beam.sdk.transforms.Filter;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-
-/**
- * An {@link org.apache.beam.runners.core.construction.expansion.ExpansionService} useful for tests.
- */
-public class TestExpansionService {
-
- private static final String TEST_COUNT_URN = "pytest:beam:transforms:count";
- private static final String TEST_FILTER_URN = "pytest:beam:transforms:filter_less_than";
-
- /** Registers a single test transformation. */
- @AutoService(ExpansionService.ExpansionServiceRegistrar.class)
- public static class TestTransforms implements ExpansionService.ExpansionServiceRegistrar {
- @Override
- public Map<String, ExpansionService.TransformProvider> knownTransforms() {
- return ImmutableMap.of(
- TEST_COUNT_URN, spec -> Count.perElement(),
- TEST_FILTER_URN,
- spec ->
- Filter.lessThanEq(
- // TODO(BEAM-6587): Use strings directly rather than longs.
- (long) spec.getPayload().toStringUtf8().charAt(0)));
- }
- }
-
- public static void main(String[] args) throws Exception {
- int port = Integer.parseInt(args[0]);
- System.out.println("Starting expansion service at localhost:" + port);
- Server server = ServerBuilder.forPort(port).addService(new ExpansionService()).build();
- server.start();
- server.awaitTermination();
- }
-}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageMatcher.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageMatcher.java
index 3c47718..a76b5dc 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageMatcher.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageMatcher.java
@@ -26,7 +26,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageTest.java
index 6602f4c..f7a0509 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ExecutableStageTest.java
@@ -41,7 +41,7 @@
import org.apache.beam.runners.core.construction.Environments;
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/FusedPipelineTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/FusedPipelineTest.java
index 27b096c..eda5f8d 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/FusedPipelineTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/FusedPipelineTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuserTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuserTest.java
index e329c00..87c0f72 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuserTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuserTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import static org.hamcrest.Matchers.anyOf;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuserTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuserTest.java
index 94ad3e8..af918d4 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuserTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuserTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
@@ -43,7 +43,7 @@
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
import org.junit.Before;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
index a2a9991..5ca1806 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
@@ -34,15 +34,15 @@
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ElementOrder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.EndpointPair;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ElementOrder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.EndpointPair;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java
index 023cc27..daae554 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/OutputDeduplicatorTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.empty;
import static org.hamcrest.Matchers.equalTo;
@@ -41,7 +41,7 @@
import org.apache.beam.runners.core.construction.graph.OutputDeduplicator.DeduplicationResult;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ProtoOverridesTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ProtoOverridesTest.java
index cfac0fa..5215d72 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ProtoOverridesTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/ProtoOverridesTest.java
@@ -34,11 +34,10 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.model.pipeline.v1.RunnerApi.SdkFunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.WindowingStrategy;
import org.apache.beam.runners.core.construction.graph.ProtoOverrides.TransformReplacement;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -68,7 +67,7 @@
PCollection.newBuilder().setUniqueName("intermediate").build())
.putCoders(
"coder",
- Coder.newBuilder().setSpec(SdkFunctionSpec.getDefaultInstance()).build()))
+ Coder.newBuilder().setSpec(FunctionSpec.getDefaultInstance()).build()))
.build();
PTransform secondReplacement =
@@ -146,7 +145,7 @@
PCollection.newBuilder().setUniqueName("intermediate").build())
.putCoders(
"coder",
- Coder.newBuilder().setSpec(SdkFunctionSpec.getDefaultInstance()).build()))
+ Coder.newBuilder().setSpec(FunctionSpec.getDefaultInstance()).build()))
.build();
ByteString newPayload = ByteString.copyFrom("foo-bar-baz".getBytes(StandardCharsets.UTF_8));
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/QueryablePipelineTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/QueryablePipelineTest.java
index c239954..b4bcc07 100644
--- a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/QueryablePipelineTest.java
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/QueryablePipelineTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.construction.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
@@ -62,7 +62,7 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRendererTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRendererTest.java
new file mode 100644
index 0000000..50a574a
--- /dev/null
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRendererTest.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.renderer;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Sum;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link PipelineDotRenderer}. */
+@RunWith(JUnit4.class)
+public class PipelineDotRendererTest {
+ @Rule public final transient TestPipeline p = TestPipeline.create();
+
+ @Test
+ public void testEmptyPipeline() {
+ assertEquals(
+ "digraph {"
+ + " rankdir=LR"
+ + " subgraph cluster_0 {"
+ + " label = \"\""
+ + " }"
+ + "}",
+ PipelineDotRenderer.toDotString(p).replaceAll(System.lineSeparator(), ""));
+ }
+
+ @Test
+ public void testCompositePipeline() {
+ p.apply(Create.timestamped(TimestampedValue.of(KV.of(1, 1), new Instant(1))))
+ .apply(Window.into(FixedWindows.of(Duration.millis(10))))
+ .apply(Sum.integersPerKey());
+ assertEquals(
+ "digraph {"
+ + " rankdir=LR"
+ + " subgraph cluster_0 {"
+ + " label = \"\""
+ + " subgraph cluster_1 {"
+ + " label = \"Create.TimestampedValues\""
+ + " subgraph cluster_2 {"
+ + " label = \"Create.TimestampedValues/Create.Values\""
+ + " 3 [label=\"Read(CreateSource)\"]"
+ + " }"
+ + " subgraph cluster_4 {"
+ + " label = \"Create.TimestampedValues/ParDo(ConvertTimestamps)\""
+ + " 5 [label=\"ParMultiDo(ConvertTimestamps)\"]"
+ + " 3 -> 5 [style=solid label=\"\"]"
+ + " }"
+ + " }"
+ + " subgraph cluster_6 {"
+ + " label = \"Window.Into()\""
+ + " 7 [label=\"Window.Assign\"]"
+ + " 5 -> 7 [style=solid label=\"\"]"
+ + " }"
+ + " subgraph cluster_8 {"
+ + " label = \"Combine.perKey(SumInteger)\""
+ + " 9 [label=\"GroupByKey\"]"
+ + " 7 -> 9 [style=solid label=\"\"]"
+ + " subgraph cluster_10 {"
+ + " label = \"Combine.perKey(SumInteger)/Combine.GroupedValues\""
+ + " subgraph cluster_11 {"
+ + " label = \"Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)\""
+ + " 12 [label=\"ParMultiDo(Anonymous)\"]"
+ + " 9 -> 12 [style=solid label=\"\"]"
+ + " }"
+ + " }"
+ + " }"
+ + " }"
+ + "}",
+ PipelineDotRenderer.toDotString(p).replaceAll(System.lineSeparator(), ""));
+ }
+}
diff --git a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRendererTest.java b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRendererTest.java
new file mode 100644
index 0000000..0be7117
--- /dev/null
+++ b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRendererTest.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.renderer;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Sum;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TimestampedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link PortablePipelineDotRenderer}. */
+@RunWith(JUnit4.class)
+public class PortablePipelineDotRendererTest {
+ @Rule public final transient TestPipeline p = TestPipeline.create();
+
+ @Test
+ public void testEmptyPipeline() {
+ assertEquals(
+ "digraph {" + " rankdir=LR" + "}",
+ PortablePipelineDotRenderer.toDotString(PipelineTranslation.toProto(p))
+ .replaceAll(System.lineSeparator(), ""));
+ }
+
+ @Test
+ public void testCompositePipeline() {
+ p.apply(Create.timestamped(TimestampedValue.of(KV.of(1, 1), new Instant(1))))
+ .apply(Window.into(FixedWindows.of(Duration.millis(10))))
+ .apply(Sum.integersPerKey());
+
+ assertEquals(
+ "digraph {"
+ + " rankdir=LR"
+ + " 0 [label=\"Create.TimestampedValues\\n\"]"
+ + " 1 [label=\"Window.Into()\\n\"]"
+ + " 0 -> 1 [style=solid label=\"Create.TimestampedValues/ParDo(ConvertTimestamps)/ParMultiDo(ConvertTimestamps).output\"]"
+ + " 2 [label=\"Combine.perKey(SumInteger)\\nbeam:transform:combine_per_key:v1\"]"
+ + " 1 -> 2 [style=solid label=\"Window.Into()/Window.Assign.out\"]"
+ + "}",
+ PortablePipelineDotRenderer.toDotString(PipelineTranslation.toProto(p))
+ .replaceAll(System.lineSeparator(), ""));
+ }
+}
diff --git a/runners/core-java/build.gradle b/runners/core-java/build.gradle
index f5ab4ae..99795ce 100644
--- a/runners/core-java/build.gradle
+++ b/runners/core-java/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.core')
description = "Apache Beam :: Runners :: Core Java"
ext.summary = "Beam Runners Core provides utilities to aid runner authors."
@@ -31,20 +31,19 @@
}
dependencies {
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":model:fn-execution", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":sdks:java:fn-execution", configuration: "shadow")
- shadow library.java.vendored_guava_20_0
- shadow library.java.joda_time
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest library.java.junit
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.mockito_core
- shadowTest library.java.junit
- shadowTest library.java.slf4j_api
- shadowTest library.java.jackson_dataformat_yaml
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(path: ":model:fn-execution", configuration: "shadow")
+ compile project(":runners:core-construction-java")
+ compile project(":sdks:java:fn-execution")
+ compile library.java.vendored_guava_26_0_jre
+ compile library.java.joda_time
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.slf4j_api
+ testCompile library.java.jackson_dataformat_yaml
testRuntimeOnly library.java.slf4j_simple
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/ActiveWindowSet.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/ActiveWindowSet.java
index 74b637d..03992e3 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/ActiveWindowSet.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/ActiveWindowSet.java
@@ -21,7 +21,7 @@
import java.util.Set;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Track which windows are <i>active</i>, and the <i>state address window(s)</i> under which their
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index 3d929d7..6496561 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -61,7 +61,8 @@
@Nullable Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
return new SimpleDoFnRunner<>(
options,
fn,
@@ -73,7 +74,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
}
/**
@@ -118,7 +120,8 @@
@Nullable Coder<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
return new ProcessFnRunner<>(
simpleRunner(
options,
@@ -131,7 +134,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation),
+ doFnSchemaInformation,
+ sideInputMapping),
views,
sideInputReader);
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/GlobalCombineFnRunners.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/GlobalCombineFnRunners.java
index fbfdb29..94e427e 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/GlobalCombineFnRunners.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/GlobalCombineFnRunners.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* Static utility methods that provide {@link GlobalCombineFnRunner} implementations for different
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
index 5a79343..48a2b58 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Comparator;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryMultimapSideInputView.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryMultimapSideInputView.java
index 9a59d48..e00f330 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryMultimapSideInputView.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryMultimapSideInputView.java
@@ -21,9 +21,9 @@
import org.apache.beam.sdk.transforms.Materializations;
import org.apache.beam.sdk.transforms.Materializations.MultimapView;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimaps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimaps;
/** An in-memory representation of {@link MultimapView}. */
public class InMemoryMultimapSideInputView<K, V> implements Materializations.MultimapView<K, V> {
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
index 750d0bf..9628cff 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
@@ -48,9 +48,9 @@
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
index 5e4eccf..7b01c04 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.NavigableSet;
import java.util.NoSuchElementException;
@@ -28,9 +28,9 @@
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowTracing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
import org.joda.time.Instant;
/** {@link TimerInternals} with all watermarks and processing clock simulated in-memory. */
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItemCoder.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItemCoder.java
index a1777d0..2949548 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItemCoder.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItemCoder.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A {@link Coder} for {@link KeyedWorkItem KeyedWorkItems}. */
public class KeyedWorkItemCoder<K, ElemT> extends StructuredCoder<KeyedWorkItem<K, ElemT>> {
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItems.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItems.java
index b916cf0..ecdb203 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItems.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/KeyedWorkItems.java
@@ -21,8 +21,8 @@
import java.util.Objects;
import org.apache.beam.runners.core.TimerInternals.TimerData;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Static utility methods that provide {@link KeyedWorkItem} implementations. */
public class KeyedWorkItems {
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
index eafc9b8..4865e82 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunner.java
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataUtils.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataUtils.java
index da2b630..0e0eab7 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataUtils.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/LateDataUtils.java
@@ -23,8 +23,8 @@
import org.apache.beam.sdk.util.WindowTracing;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/MergingActiveWindowSet.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/MergingActiveWindowSet.java
index db19a78..9d2d847 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/MergingActiveWindowSet.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/MergingActiveWindowSet.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collection;
@@ -35,9 +35,9 @@
import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** An {@link ActiveWindowSet} for merging {@link WindowFn} implementations. */
public class MergingActiveWindowSet<W extends BoundedWindow> implements ActiveWindowSet<W> {
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/NonMergingActiveWindowSet.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/NonMergingActiveWindowSet.java
index 8540ba1..03efe29 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/NonMergingActiveWindowSet.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/NonMergingActiveWindowSet.java
@@ -21,8 +21,8 @@
import java.util.Set;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/**
* Implementation of {@link ActiveWindowSet} used with {@link WindowFn WindowFns} that don't support
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/NullSideInputReader.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/NullSideInputReader.java
index 4dbeb69..0cd0f6f 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/NullSideInputReader.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/NullSideInputReader.java
@@ -21,7 +21,7 @@
import java.util.Set;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A {@link SideInputReader} representing a well-defined set of views, but not storing any values
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
index 7a7bc60..67982e1 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvoker.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.concurrent.CancellationException;
import java.util.concurrent.Future;
@@ -45,8 +45,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -121,6 +121,11 @@
}
@Override
+ public Object sideInput(String tagId) {
+ throw new UnsupportedOperationException("Not supported in SplittableDoFn");
+ }
+
+ @Override
public Object schemaElement(int index) {
throw new UnsupportedOperationException("Not supported in SplittableDoFn");
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
index 48289b8..1537ad5 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/PaneInfoTracker.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.sdk.state.ReadableState;
import org.apache.beam.sdk.state.ValueState;
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo.PaneInfoCoder;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
import org.apache.beam.sdk.util.WindowTracing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/PeekingReiterator.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/PeekingReiterator.java
index c767c54..8628d9e 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/PeekingReiterator.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/PeekingReiterator.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.NoSuchElementException;
import javax.annotation.Nullable;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
index da4f60d..d65b5f4 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/ProcessFnRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Collections;
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
index b2f575f..ac6185b 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Collection;
import java.util.Map;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
/** Factory for creating instances of the various {@link ReduceFn} contexts. */
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
index 1bc8593..1143974 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collection;
@@ -53,9 +53,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
index b17a074..41f72a7 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Collections;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 8fd2ad3..2b105fb 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Collection;
import java.util.List;
@@ -54,9 +54,9 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.PeriodFormat;
@@ -106,6 +106,8 @@
@Nullable private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
+
/** Constructor. */
public SimpleDoFnRunner(
PipelineOptions options,
@@ -118,7 +120,8 @@
@Nullable Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.options = options;
this.fn = fn;
this.signature = DoFnSignatures.getSignature(fn.getClass());
@@ -151,6 +154,7 @@
this.windowCoder = untypedCoder;
this.allowedLateness = windowingStrategy.getAllowedLateness();
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
@Override
@@ -302,6 +306,12 @@
}
@Override
+ public InputT sideInput(String tagId) {
+ throw new UnsupportedOperationException(
+ "SideInput parameters are not supported outside of @ProcessElement method.");
+ }
+
+ @Override
public Object schemaElement(int index) {
throw new UnsupportedOperationException(
"Element parameters are not supported outside of @ProcessElement method.");
@@ -416,6 +426,12 @@
}
@Override
+ public InputT sideInput(String tagId) {
+ throw new UnsupportedOperationException(
+ "Cannot access sideInput outside of @ProcessElement method.");
+ }
+
+ @Override
public Object schemaElement(int index) {
throw new UnsupportedOperationException(
"Cannot access element outside of @ProcessElement method.");
@@ -632,6 +648,11 @@
}
@Override
+ public Object sideInput(String tagId) {
+ return sideInput(sideInputMapping.get(tagId));
+ }
+
+ @Override
public Object schemaElement(int index) {
SerializableFunction converter = doFnSchemaInformation.getElementConverters().get(index);
return converter.apply(element());
@@ -782,6 +803,11 @@
}
@Override
+ public Object sideInput(String tagId) {
+ throw new UnsupportedOperationException("SideInput parameters are not supported.");
+ }
+
+ @Override
public Object schemaElement(int index) {
throw new UnsupportedOperationException("Element parameters are not supported.");
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunner.java
index dad1f96..36a89fe 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunner.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java
index 6e30ab1..ec73e7b 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java
@@ -53,8 +53,8 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableProcessElementInvoker.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableProcessElementInvoker.java
index d877970..68f0c1f 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableProcessElementInvoker.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableProcessElementInvoker.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import javax.annotation.Nullable;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateNamespaces.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateNamespaces.java
index 03a271f..ae6ae44 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateNamespaces.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateNamespaces.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
/** Factory methods for creating the {@link StateNamespace StateNamespaces}. */
public class StateNamespaces {
@@ -89,8 +89,6 @@
/** {@link StateNamespace} that is scoped to a specific window. */
public static class WindowNamespace<W extends BoundedWindow> implements StateNamespace {
- private static final String WINDOW_FORMAT = "/%s/";
-
private Coder<W> windowCoder;
private W window;
@@ -106,7 +104,8 @@
@Override
public String stringKey() {
try {
- return String.format(WINDOW_FORMAT, CoderUtils.encodeToBase64(windowCoder, window));
+ // equivalent to String.format("/%s/", ...)
+ return "/" + CoderUtils.encodeToBase64(windowCoder, window) + "/";
} catch (CoderException e) {
throw new RuntimeException("Unable to generate string key from window " + window, e);
}
@@ -155,8 +154,6 @@
/** {@link StateNamespace} that is scoped to a particular window and trigger index. */
public static class WindowAndTriggerNamespace<W extends BoundedWindow> implements StateNamespace {
- private static final String WINDOW_AND_TRIGGER_FORMAT = "/%s/%s/";
-
private static final int TRIGGER_RADIX = 36;
private Coder<W> windowCoder;
private W window;
@@ -179,12 +176,15 @@
@Override
public String stringKey() {
try {
- return String.format(
- WINDOW_AND_TRIGGER_FORMAT,
- CoderUtils.encodeToBase64(windowCoder, window),
+ // equivalent to String.format("/%s/%s/", ...)
+ return "/"
+ + CoderUtils.encodeToBase64(windowCoder, window)
+ +
// Use base 36 so that can address 36 triggers in a single byte and still be human
// readable.
- Integer.toString(triggerIndex, TRIGGER_RADIX).toUpperCase());
+ "/"
+ + Integer.toString(triggerIndex, TRIGGER_RADIX).toUpperCase()
+ + "/";
} catch (CoderException e) {
throw new RuntimeException("Unable to generate string key from window " + window, e);
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTable.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTable.java
index 7f75e45..1f971c7 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTable.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTable.java
@@ -24,9 +24,9 @@
import org.apache.beam.runners.core.StateTag.StateBinder;
import org.apache.beam.sdk.state.State;
import org.apache.beam.sdk.state.StateContext;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Equivalence;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
/**
* Table mapping {@code StateNamespace} and {@code StateTag<?>} to a {@code State} instance.
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
index 4b17fc7..8dd84c1 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/StateTags.java
@@ -38,8 +38,8 @@
import org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Equivalence;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Static utility methods for creating {@link StateTag} instances. */
@Experimental(Kind.STATE)
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/SystemReduceFn.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/SystemReduceFn.java
index bfb198a..a4d51ba 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/SystemReduceFn.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/SystemReduceFn.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.AppliedCombineFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* {@link ReduceFn} implementing the default reduction behaviors of {@link GroupByKey}.
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java
index 6ae16fc..a766143 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternals.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/WatermarkHold.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/WatermarkHold.java
index 02626b3..5219659 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/WatermarkHold.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/WatermarkHold.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Serializable;
import java.util.Collection;
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowTracing;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DefaultMetricResults.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DefaultMetricResults.java
index f2b8f6c..9231688 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DefaultMetricResults.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DefaultMetricResults.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* Default implementation of {@link org.apache.beam.sdk.metrics.MetricResults}, which takes static
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java
index c3fb816..7bc8f68 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.metrics;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.Closeable;
import java.util.concurrent.CancellationException;
@@ -29,8 +29,8 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.joda.time.DateTimeUtils.MillisProvider;
/** Monitors the execution of one or more execution threads. */
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateTracker.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateTracker.java
index a559507..e14d597 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateTracker.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateTracker.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.metrics;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.Closeable;
@@ -26,8 +26,8 @@
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Tracks the current state of a single execution thread. */
@SuppressFBWarnings(value = "IS2_INCONSISTENT_SYNC", justification = "Intentional for performance.")
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUpdates.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUpdates.java
index 427c30d..5bbf415 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUpdates.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUpdates.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.metrics.MetricKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Representation of multiple metric updates. */
@Experimental(Kind.METRICS)
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
index a2cd286..bcca019 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.metrics;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.Serializable;
import java.util.ArrayList;
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.metrics.MetricKey;
import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
index febba8f..5a8e89c 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
@@ -19,7 +19,7 @@
import static java.util.Collections.singleton;
import static java.util.stream.Collectors.toList;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.concat;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.concat;
import java.io.Serializable;
import java.util.ArrayList;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsMap.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsMap.java
index d1b9775..d3f416e 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsMap.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsMap.java
@@ -25,8 +25,8 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* A map from {@code K} to {@code T} that supports getting or creating values associated with a key
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
index 68e6f19..3695e37 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.metrics.MetricsOptions;
import org.apache.beam.sdk.metrics.MetricsSink;
import org.apache.beam.sdk.util.InstanceBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
/** Component that regularly merges metrics and pushes them to a metrics sink. */
@Experimental(Experimental.Kind.METRICS)
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
index 5bf435e..1976165 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsTranslation.java
@@ -26,9 +26,9 @@
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.metrics.MetricKey;
import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
/** Translation utilities for metrics classes to/from Fn API. */
@Experimental(Kind.METRICS)
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
index 22f7619..58e8580 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricName.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.metrics;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.HashMap;
import java.util.Map;
@@ -25,7 +25,7 @@
import java.util.Objects;
import org.apache.beam.model.pipeline.v1.MetricsApi;
import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
/**
* An implementation of {@code MetricKey} based on a MonitoringInfo's URN and label to represent the
@@ -51,7 +51,14 @@
@Override
public String getNamespace() {
if (labels.containsKey(MonitoringInfoConstants.Labels.NAMESPACE)) {
+ // User-generated metric
return labels.getOrDefault(MonitoringInfoConstants.Labels.NAMESPACE, null);
+ } else if (labels.containsKey(MonitoringInfoConstants.Labels.PCOLLECTION)) {
+ // System-generated metric
+ return labels.getOrDefault(MonitoringInfoConstants.Labels.PCOLLECTION, null);
+ } else if (labels.containsKey(MonitoringInfoConstants.Labels.PTRANSFORM)) {
+ // System-generated metric
+ return labels.getOrDefault(MonitoringInfoConstants.Labels.PTRANSFORM, null);
} else {
return urn.split(":", 2)[0];
}
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
index 135e5c3..f0b6c46 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
@@ -21,7 +21,7 @@
import java.util.HashMap;
import java.util.Map;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.joda.time.format.PeriodFormatter;
import org.joda.time.format.PeriodFormatterBuilder;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
index d99ee84..65fbdf2 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java
@@ -25,7 +25,7 @@
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfoSpec;
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfoSpecs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Simplified building of MonitoringInfo fields, allows setting one field at a time with simpler
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
index 44ad2af..aecc14b 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
@@ -17,13 +17,13 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link TriggerStateMachine} that fires and finishes once after all of its sub-triggers have
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
index a766449..c54bf09 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.transforms.Combine.Holder;
import org.apache.beam.sdk.transforms.Min;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.PeriodFormat;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
index c651e40..5f039d8 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A composite {@link TriggerStateMachine} that executes its sub-triggers in order. Only one
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
index c094e16..46db56d 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
@@ -17,13 +17,13 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Create a composite {@link TriggerStateMachine} that fires once after at least one of its
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
index 53ae851..abaf5a7 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
@@ -22,7 +22,7 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
import org.joda.time.Instant;
// This should not really have the superclass https://issues.apache.org/jira/browse/BEAM-1486
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
index e8c863d..ff99930 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
@@ -17,13 +17,13 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Objects;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.state.TimeDomain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* {@code AfterWatermark} triggers fire based on progress of the system watermark. This time is a
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/ExecutableTriggerStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/ExecutableTriggerStateMachine.java
index e2252aa..9c14e5a 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/ExecutableTriggerStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/ExecutableTriggerStateMachine.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Serializable;
import java.util.ArrayList;
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/FinishedTriggersSet.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/FinishedTriggersSet.java
index b173835..99c6ac5 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/FinishedTriggersSet.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/FinishedTriggersSet.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.core.triggers;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** An implementation of {@link FinishedTriggers} atop a user-provided mutable {@link Set}. */
public class FinishedTriggersSet implements FinishedTriggers {
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/OrFinallyStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/OrFinallyStateMachine.java
index 7ee5f7d..3a6c1cd 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/OrFinallyStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/OrFinallyStateMachine.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.core.triggers;
import java.util.Arrays;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Executes the {@code actual} trigger until it finishes or until the {@code until} trigger fires.
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachine.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachine.java
index 6d05cae..719d0fb 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachine.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachine.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineContextFactory.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineContextFactory.java
index e0ecbed..f91c540 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineContextFactory.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineContextFactory.java
@@ -35,9 +35,9 @@
import org.apache.beam.sdk.state.Timers;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
index 19402b5..ac642ce 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.BitSet;
import java.util.Collection;
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.state.Timers;
import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
/**
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryMultimapSideInputViewTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryMultimapSideInputViewTest.java
index 2df15dd..4de5c7f 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryMultimapSideInputViewTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/InMemoryMultimapSideInputViewTest.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.transforms.Materializations.MultimapView;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/KeyedWorkItemCoderTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/KeyedWorkItemCoderTest.java
index 7a2c312..1aebb03 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/KeyedWorkItemCoderTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/KeyedWorkItemCoderTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunnerTest.java
index 490ab1f..9ae6bbe 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/LateDataDroppingDoFnRunnerTest.java
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/MergingActiveWindowSetTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/MergingActiveWindowSetTest.java
index 0857fa0..b9dd372 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/MergingActiveWindowSetTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/MergingActiveWindowSetTest.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.Sessions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.After;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvokerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvokerTest.java
index a05aa8d..e8ca98b 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvokerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/OutputAndTimeBoundedSplittableProcessElementInvokerTest.java
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index 7da0461..1c0d2a3 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -83,7 +83,7 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
index f03f71f..f316d07 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnTester.java
@@ -65,11 +65,11 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Equivalence;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/SideInputHandlerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/SideInputHandlerTest.java
index 4376676..e9edbb2 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/SideInputHandlerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/SideInputHandlerTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
index 1e7ca93..b790314 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleDoFnRunnerTest.java
@@ -44,8 +44,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.PeriodFormat;
@@ -88,7 +88,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
thrown.expect(UserCodeException.class);
thrown.expectCause(is(fn.exceptionToThrow));
@@ -111,7 +112,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
thrown.expect(UserCodeException.class);
thrown.expectCause(is(fn.exceptionToThrow));
@@ -141,7 +143,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
// Setting the timer needs the current time, as it is set relative
Instant currentTime = new Instant(42);
@@ -172,7 +175,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
thrown.expect(UserCodeException.class);
thrown.expectCause(is(fn.exceptionToThrow));
@@ -195,7 +199,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
thrown.expect(UserCodeException.class);
thrown.expectCause(is(fn.exceptionToThrow));
@@ -222,7 +227,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(windowFn),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
Instant currentTime = new Instant(42);
Duration offset = Duration.millis(37);
@@ -264,7 +270,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
runner.startBundle();
// An element output at the current timestamp is fine.
@@ -303,7 +310,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
runner.startBundle();
// Outputting between "now" and "now - allowed skew" succeeds.
@@ -343,7 +351,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(new GlobalWindows()),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
runner.startBundle();
runner.processElement(
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
index c098282..28b387e 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimplePushbackSideInputDoFnRunnerTest.java
@@ -17,36 +17,56 @@
*/
package org.apache.beam.runners.core;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.when;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import org.apache.beam.runners.core.TimerInternals.TimerData;
+import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.metrics.MetricName;
+import org.apache.beam.sdk.metrics.MetricsEnvironment;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.IdentitySideInputWindowFn;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.mockito.Mock;
@@ -56,9 +76,27 @@
/** Tests for {@link SimplePushbackSideInputDoFnRunner}. */
@RunWith(JUnit4.class)
public class SimplePushbackSideInputDoFnRunnerTest {
+ @Mock StepContext mockStepContext;
@Mock private ReadyCheckingSideInputReader reader;
private TestDoFnRunner<Integer, Integer> underlying;
private PCollectionView<Integer> singletonView;
+ private DoFnRunner<KV<String, Integer>, Integer> statefulRunner;
+
+ private static final long WINDOW_SIZE = 10;
+ private static final long ALLOWED_LATENESS = 1;
+
+ private static final IntervalWindow WINDOW_1 =
+ new IntervalWindow(new Instant(0), new Instant(10));
+
+ private static final IntervalWindow WINDOW_2 =
+ new IntervalWindow(new Instant(10), new Instant(20));
+
+ private static final WindowingStrategy<?, ?> WINDOWING_STRATEGY =
+ WindowingStrategy.of(FixedWindows.of(Duration.millis(WINDOW_SIZE)))
+ .withAllowedLateness(Duration.millis(ALLOWED_LATENESS));
+
+ private InMemoryStateInternals<String> stateInternals;
+ private InMemoryTimerInternals timerInternals;
@Rule public TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
@@ -72,6 +110,26 @@
.apply(Sum.integersGlobally().asSingletonView());
underlying = new TestDoFnRunner<>();
+
+ DoFn<KV<String, Integer>, Integer> fn = new MyDoFn();
+
+ MockitoAnnotations.initMocks(this);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
+
+ stateInternals = new InMemoryStateInternals<>("hello");
+ timerInternals = new InMemoryTimerInternals();
+
+ when(mockStepContext.stateInternals()).thenReturn((StateInternals) stateInternals);
+ when(mockStepContext.timerInternals()).thenReturn(timerInternals);
+
+ statefulRunner =
+ DoFnRunners.defaultStatefulDoFnRunner(
+ fn,
+ getDoFnRunner(fn),
+ WINDOWING_STRATEGY,
+ new StatefulDoFnRunner.TimeInternalsCleanupTimer(timerInternals, WINDOWING_STRATEGY),
+ new StatefulDoFnRunner.StateInternalsStateCleaner<>(
+ fn, stateInternals, (Coder) WINDOWING_STRATEGY.getWindowFn().windowCoder()));
}
private SimplePushbackSideInputDoFnRunner<Integer, Integer> createRunner(
@@ -276,4 +334,166 @@
finished = true;
}
}
+
+ private SimplePushbackSideInputDoFnRunner<KV<String, Integer>, Integer> createRunner(
+ DoFnRunner<KV<String, Integer>, Integer> doFnRunner,
+ ImmutableList<PCollectionView<?>> views) {
+ SimplePushbackSideInputDoFnRunner<KV<String, Integer>, Integer> runner =
+ SimplePushbackSideInputDoFnRunner.create(doFnRunner, views, reader);
+ runner.startBundle();
+ return runner;
+ }
+
+ @Test
+ @Category({ValidatesRunner.class})
+ public void testLateDroppingForStatefulDoFnRunner() throws Exception {
+ MetricsContainerImpl container = new MetricsContainerImpl("any");
+ MetricsEnvironment.setCurrentContainer(container);
+
+ timerInternals.advanceInputWatermark(new Instant(BoundedWindow.TIMESTAMP_MAX_VALUE));
+ timerInternals.advanceOutputWatermark(new Instant(BoundedWindow.TIMESTAMP_MAX_VALUE));
+
+ PushbackSideInputDoFnRunner runner =
+ createRunner(statefulRunner, ImmutableList.of(singletonView));
+
+ runner.startBundle();
+
+ when(reader.isReady(Mockito.eq(singletonView), Mockito.any(BoundedWindow.class)))
+ .thenReturn(true);
+
+ WindowedValue<Integer> multiWindow =
+ WindowedValue.of(
+ 1,
+ new Instant(0),
+ ImmutableList.of(new IntervalWindow(new Instant(0), new Instant(0L + WINDOW_SIZE))),
+ PaneInfo.ON_TIME_AND_ONLY_FIRING);
+
+ runner.processElementInReadyWindows(multiWindow);
+
+ long droppedValues =
+ container
+ .getCounter(
+ MetricName.named(
+ StatefulDoFnRunner.class, StatefulDoFnRunner.DROPPED_DUE_TO_LATENESS_COUNTER))
+ .getCumulative();
+ assertEquals(1L, droppedValues);
+
+ runner.finishBundle();
+ }
+
+ @Test
+ @Category({ValidatesRunner.class})
+ public void testGarbageCollectForStatefulDoFnRunner() throws Exception {
+ timerInternals.advanceInputWatermark(new Instant(1L));
+
+ MyDoFn fn = new MyDoFn();
+ StateTag<ValueState<Integer>> stateTag = StateTags.tagForSpec(fn.stateId, fn.intState);
+
+ PushbackSideInputDoFnRunner runner =
+ createRunner(statefulRunner, ImmutableList.of(singletonView));
+
+ Instant elementTime = new Instant(1);
+
+ when(reader.isReady(Mockito.eq(singletonView), Mockito.any(BoundedWindow.class)))
+ .thenReturn(true);
+
+ // first element, key is hello, WINDOW_1
+ runner.processElementInReadyWindows(
+ WindowedValue.of(KV.of("hello", 1), elementTime, WINDOW_1, PaneInfo.NO_FIRING));
+
+ assertEquals(1, (int) stateInternals.state(windowNamespace(WINDOW_1), stateTag).read());
+
+ // second element, key is hello, WINDOW_2
+ runner.processElementInReadyWindows(
+ WindowedValue.of(
+ KV.of("hello", 1), elementTime.plus(WINDOW_SIZE), WINDOW_2, PaneInfo.NO_FIRING));
+
+ runner.processElementInReadyWindows(
+ WindowedValue.of(
+ KV.of("hello", 1), elementTime.plus(WINDOW_SIZE), WINDOW_2, PaneInfo.NO_FIRING));
+
+ assertEquals(2, (int) stateInternals.state(windowNamespace(WINDOW_2), stateTag).read());
+
+ // advance watermark past end of WINDOW_1 + allowed lateness
+ // the cleanup timer is set to window.maxTimestamp() + allowed lateness + 1
+ // to ensure that state is still available when a user timer for window.maxTimestamp() fires
+ advanceInputWatermark(
+ timerInternals,
+ WINDOW_1
+ .maxTimestamp()
+ .plus(ALLOWED_LATENESS)
+ .plus(StatefulDoFnRunner.TimeInternalsCleanupTimer.GC_DELAY_MS)
+ .plus(1), // so the watermark is past the GC horizon, not on it
+ runner);
+
+ assertTrue(
+ stateInternals.isEmptyForTesting(
+ stateInternals.state(windowNamespace(WINDOW_1), stateTag)));
+
+ assertEquals(2, (int) stateInternals.state(windowNamespace(WINDOW_2), stateTag).read());
+
+ // advance watermark past end of WINDOW_2 + allowed lateness
+ advanceInputWatermark(
+ timerInternals,
+ WINDOW_2
+ .maxTimestamp()
+ .plus(ALLOWED_LATENESS)
+ .plus(StatefulDoFnRunner.TimeInternalsCleanupTimer.GC_DELAY_MS)
+ .plus(1), // so the watermark is past the GC horizon, not on it
+ runner);
+
+ assertTrue(
+ stateInternals.isEmptyForTesting(
+ stateInternals.state(windowNamespace(WINDOW_2), stateTag)));
+ }
+
+ private static void advanceInputWatermark(
+ InMemoryTimerInternals timerInternals,
+ Instant newInputWatermark,
+ PushbackSideInputDoFnRunner<?, ?> toTrigger)
+ throws Exception {
+ timerInternals.advanceInputWatermark(newInputWatermark);
+ TimerInternals.TimerData timer;
+ while ((timer = timerInternals.removeNextEventTimer()) != null) {
+ StateNamespace namespace = timer.getNamespace();
+ checkArgument(namespace instanceof StateNamespaces.WindowNamespace);
+ BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
+ toTrigger.onTimer(timer.getTimerId(), window, timer.getTimestamp(), timer.getDomain());
+ }
+ }
+
+ private static StateNamespace windowNamespace(IntervalWindow window) {
+ return StateNamespaces.window((Coder) WINDOWING_STRATEGY.getWindowFn().windowCoder(), window);
+ }
+
+ private static class MyDoFn extends DoFn<KV<String, Integer>, Integer> {
+
+ public final String stateId = "foo";
+
+ @StateId(stateId)
+ public final StateSpec<ValueState<Integer>> intState = StateSpecs.value(VarIntCoder.of());
+
+ @ProcessElement
+ public void processElement(ProcessContext c, @StateId(stateId) ValueState<Integer> state) {
+ Integer currentValue = MoreObjects.firstNonNull(state.read(), 0);
+ state.write(currentValue + 1);
+ }
+ }
+
+ private DoFnRunner<KV<String, Integer>, Integer> getDoFnRunner(
+ DoFn<KV<String, Integer>, Integer> fn) {
+ return new SimpleDoFnRunner<>(
+ null,
+ fn,
+ NullSideInputReader.empty(),
+ null,
+ null,
+ Collections.emptyList(),
+ mockStepContext,
+ null,
+ Collections.emptyMap(),
+ WINDOWING_STRATEGY,
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
+ }
}
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoProcessFnTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoProcessFnTest.java
index ae383a6..43f9f21 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoProcessFnTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoProcessFnTest.java
@@ -19,7 +19,7 @@
import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.resume;
import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.stop;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.hasItem;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/StateInternalsTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/StateInternalsTest.java
index fced746..10bf0d3 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/StateInternalsTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/StateInternalsTest.java
@@ -52,7 +52,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/StatefulDoFnRunnerTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/StatefulDoFnRunnerTest.java
index 3e0f1aa..85b3c0b 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/StatefulDoFnRunnerTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/StatefulDoFnRunnerTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.when;
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
@@ -207,7 +207,8 @@
null,
Collections.emptyMap(),
WINDOWING_STRATEGY,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
private static void advanceInputWatermark(
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
index 5c3c999..d934f72 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchersTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchersTest.java
index f91603d..bc3a0c9 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchersTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchersTest.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
index aa1016a..0e7c7f9 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsTranslationTest.java
@@ -24,8 +24,8 @@
import java.util.Collection;
import java.util.Map;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
index 33ba9cd..21f0993 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMetricNameTest.java
@@ -60,13 +60,31 @@
@Test
public void testGetNamespaceReturnsNamespaceIfLabelIsPresent() {
- HashMap<String, String> labels = new HashMap<String, String>();
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "anyTransform");
labels.put(MonitoringInfoConstants.Labels.NAMESPACE, "anyNamespace");
+ labels.put(MonitoringInfoConstants.Labels.PCOLLECTION, "anyPCollection");
MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
assertEquals("anyNamespace", name.getNamespace());
}
@Test
+ public void testGetNamespaceReturnsTransformIfNamespaceLabelIsNotPresent() {
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PTRANSFORM, "anyTransform");
+ MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
+ assertEquals("anyTransform", name.getNamespace());
+ }
+
+ @Test
+ public void testGetNamespaceReturnsPCollectionIfNamespaceLabelIsNotPresent() {
+ HashMap<String, String> labels = new HashMap<>();
+ labels.put(MonitoringInfoConstants.Labels.PCOLLECTION, "anyPCollection");
+ MonitoringInfoMetricName name = MonitoringInfoMetricName.named("anyUrn", labels);
+ assertEquals("anyPCollection", name.getNamespace());
+ }
+
+ @Test
public void testNotEqualsDiffLabels() {
HashMap<String, String> labels = new HashMap<String, String>();
String urn = MonitoringInfoConstants.Urns.ELEMENT_COUNT;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/StubTriggerStateMachine.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/StubTriggerStateMachine.java
index 3b44e45..eea3c04 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/StubTriggerStateMachine.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/StubTriggerStateMachine.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.triggers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** No-op {@link TriggerStateMachine} implementation for testing. */
abstract class StubTriggerStateMachine extends TriggerStateMachine {
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
index 7552776..21f2d13 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineTester.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.junit.Assert.assertTrue;
import java.util.ArrayList;
@@ -52,8 +52,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachinesTest.java b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachinesTest.java
index 70888f2..4082fda 100644
--- a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachinesTest.java
+++ b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachinesTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.core.triggers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.instanceOf;
import static org.junit.Assert.assertThat;
diff --git a/runners/direct-java/build.gradle b/runners/direct-java/build.gradle
index 328de71..b8836a8 100644
--- a/runners/direct-java/build.gradle
+++ b/runners/direct-java/build.gradle
@@ -28,17 +28,19 @@
":runners:java-fn-execution",
":sdks:java:fn-execution"]
-applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE << {
- dependencies {
- dependOnProjects.each {
- include(project(path: it, configuration: "shadow"))
- }
- }
- relocate "org.apache.beam.runners.core", getJavaRelocatedPath("runners.core")
- relocate "org.apache.beam.runners.fnexecution", getJavaRelocatedPath("runners.fnexecution")
- relocate "org.apache.beam.sdk.fn", getJavaRelocatedPath("sdk.fn")
- relocate "org.apache.beam.runners.local", getJavaRelocatedPath("runners.local")
-})
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.direct',
+ shadowClosure: {
+ dependencies {
+ dependOnProjects.each {
+ include(project(path: it, configuration: "shadow"))
+ }
+ }
+ relocate "org.apache.beam.runners.core", getJavaRelocatedPath("runners.core")
+ relocate "org.apache.beam.runners.fnexecution", getJavaRelocatedPath("runners.fnexecution")
+ relocate "org.apache.beam.sdk.fn", getJavaRelocatedPath("sdk.fn")
+ relocate "org.apache.beam.runners.local", getJavaRelocatedPath("runners.local")
+ })
description = "Apache Beam :: Runners :: Direct Java"
@@ -54,49 +56,39 @@
configurations {
needsRunner
validatesRunner
- validatesPortableRunner
}
dependencies {
- shadow library.java.vendored_guava_20_0
+ shadow library.java.vendored_guava_26_0_jre
shadow project(path: ":model:pipeline", configuration: "shadow")
dependOnProjects.each {
- compile project(path: it, configuration: "shadow")
+ compile project(it)
}
shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
+ shadow library.java.vendored_grpc_1_21_0
shadow library.java.joda_time
shadow library.java.slf4j_api
shadow library.java.args4j
provided library.java.hamcrest_core
provided library.java.junit
- testRuntime project(path: ":sdks:java:harness")
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest library.java.guava_testlib
+ shadowTest project(path: ":runners:core-java", configuration: "testRuntime")
shadowTest library.java.slf4j_jdk14
shadowTest library.java.mockito_core
shadowTest library.java.stax2_api
shadowTest library.java.woodstox_core_asl
shadowTest library.java.google_cloud_dataflow_java_proto_library_all
shadowTest library.java.jackson_dataformat_yaml
- needsRunner project(path: ":runners:core-construction-java", configuration: "shadowTest")
- needsRunner project(path: ":runners:core-java", configuration: "shadowTest")
+ needsRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ needsRunner project(path: ":runners:core-java", configuration: "testRuntime")
needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
needsRunner project(path: project.path, configuration: "shadow")
needsRunner project(path: project.path, configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-construction-java", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "shadowTest")
+ validatesRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
validatesRunner project(path: project.path, configuration: "shadow")
validatesRunner project(path: project.path, configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:core-construction-java", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:java-fn-execution", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:reference:java", configuration: "shadowTest")
- validatesPortableRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesPortableRunner project(path: project.path, configuration: "shadow")
- validatesPortableRunner project(path: project.path, configuration: "shadowTest")
}
task needsRunnerTests(type: Test) {
@@ -160,37 +152,3 @@
gcsBucket: gcsBucket,
bqDataset: bqDataset,
pubsubTopic: pubsubTopic)
-
-createPortableValidatesRunnerTask(
- jobServerDriver: "org.apache.beam.runners.direct.portable.job.ReferenceRunnerJobServer",
- testClasspathConfiguration: configurations.validatesPortableRunner,
- jobServerConfig: "--port=0",
- testCategories: {
- includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- excludeCategories 'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
- excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCustomWindowMerging'
- excludeCategories 'org.apache.beam.sdk.testing.UsesDistributionMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesFailureMessage'
- excludeCategories 'org.apache.beam.sdk.testing.UsesParDoLifecycle'
- excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
- // TODO(BEAM-5452): Support metrics.
- excludeCategories 'org.apache.beam.sdk.testing.UsesGaugeMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesAttemptedMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
- excludeCategories 'org.apache.beam.sdk.testing.UsesCounterMetrics'
- // TODO(BEAM-3743): Support SplittableDoFn
- excludeCategories 'org.apache.beam.sdk.testing.UsesBoundedSplittableParDo'
- excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
- excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
- // TODO(BEAM-2928): Support sideinput.
- excludeCategories 'org.apache.beam.sdk.testing.UsesSideInputs'
- // TODO(BEAM-2917): Support user state.
- excludeCategories 'org.apache.beam.sdk.testing.UsesStatefulParDo'
- excludeCategories 'org.apache.beam.sdk.testing.UsesMapState'
- excludeCategories 'org.apache.beam.sdk.testing.UsesSetState'
- // TODO(BEAM-4680): Support user timers.
- excludeCategories 'org.apache.beam.sdk.testing.UsesTimersInParDo'
- },
-)
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index d935b6c..f4108cf 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -40,12 +40,12 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.SettableFuture;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
/**
* A {@link TransformEvaluatorFactory} that produces {@link TransformEvaluator TransformEvaluators}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
index 9f1605d..253c390 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Instant;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
index 1f4195c..16ff95b 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CommittedResult.java
@@ -18,10 +18,10 @@
package org.apache.beam.runners.direct;
import com.google.auto.value.AutoValue;
+import java.util.Optional;
import java.util.Set;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
/** A {@link TransformResult} that has been committed. */
@AutoValue
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
index ca4e8ec..1153c1f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternals.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Collection;
import java.util.HashSet;
import java.util.Map;
+import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryBag;
import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryCombiningState;
@@ -51,8 +52,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
@@ -162,9 +162,9 @@
private Optional<Instant> earliestWatermarkHold;
public CopyOnAccessInMemoryStateTable(StateTable underlying) {
- this.underlying = Optional.fromNullable(underlying);
+ this.underlying = Optional.ofNullable(underlying);
binderFactory = new CopyOnBindBinderFactory(this.underlying);
- earliestWatermarkHold = Optional.absent();
+ earliestWatermarkHold = Optional.empty();
}
/**
@@ -193,7 +193,7 @@
earliestWatermarkHold = Optional.of(earliestHold);
clearEmpty();
binderFactory = new InMemoryStateBinderFactory();
- underlying = Optional.absent();
+ underlying = Optional.empty();
}
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
index d165697..1681f7a 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraph.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
/**
* Methods for interacting with the underlying structure of a {@link Pipeline} that is being
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
index 1ce7807..df4bc55 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGraphVisitor.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Collection;
import java.util.HashMap;
@@ -36,9 +36,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -117,7 +117,8 @@
}
}
if (node.getTransform() instanceof ParDo.MultiOutput) {
- consumedViews.addAll(((ParDo.MultiOutput<?, ?>) node.getTransform()).getSideInputs());
+ consumedViews.addAll(
+ ((ParDo.MultiOutput<?, ?>) node.getTransform()).getSideInputs().values());
} else if (node.getTransform() instanceof ViewOverrideFactory.WriteView) {
viewWriters.put(
((WriteView) node.getTransform()).getView(), node.toAppliedPTransform(getPipeline()));
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
index 61061f1..ae6f34a 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKey.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItemCoder;
@@ -35,8 +35,8 @@
extends ForwardingPTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> {
private final PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> original;
- static final String DIRECT_GBKO_URN = "urn:beam:directrunner:transforms:gbko:v1";
- static final String DIRECT_GABW_URN = "urn:beam:directrunner:transforms:gabw:v1";
+ static final String DIRECT_GBKO_URN = "beam:directrunner:transforms:gbko:v1";
+ static final String DIRECT_GABW_URN = "beam:directrunner:transforms:gabw:v1";
private final WindowingStrategy<?, ?> outputWindowingStrategy;
DirectGroupByKey(
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
index 873eb1f..ef4a3aa 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** A {@link PTransformOverrideFactory} for {@link GroupByKey} PTransforms. */
final class DirectGroupByKeyOverrideFactory<K, V>
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
index ab19406..d2ae615 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Implementation of {@link MetricResults} for the Direct Runner. */
class DirectMetrics extends MetricResults {
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRegistrar.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRegistrar.java
index 87e4243..f208c02 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRegistrar.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 757ac3b..988921e 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -45,13 +45,13 @@
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.joda.time.Duration;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTransformExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTransformExecutor.java
index 5ac6cba..a3f3c0f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTransformExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectTransformExecutor.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.metrics.MetricsEnvironment;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java
index d9fed6a..788a8a0 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DoFnLifecycleManager.java
@@ -26,11 +26,11 @@
import org.apache.beam.sdk.transforms.DoFn.Teardown;
import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalListener;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalNotification;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalListener;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalNotification;
/**
* Manages {@link DoFn} setup, teardown, and serialization.
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index ac3d96d..c5ebfaf 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -17,12 +17,13 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Collection;
import java.util.EnumSet;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
@@ -44,11 +45,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.joda.time.Instant;
/**
@@ -178,7 +178,7 @@
completedBundle,
result.getTimerUpdate().withCompletedTimers(completedTimers),
committedResult.getExecutable(),
- committedResult.getUnprocessedInputs().orNull(),
+ committedResult.getUnprocessedInputs().orElse(null),
committedResult.getOutputs(),
result.getWatermarkHold());
return committedResult;
@@ -193,7 +193,7 @@
private Optional<? extends CommittedBundle<?>> getUnprocessedInput(
CommittedBundle<?> completedBundle, TransformResult<?> result) {
if (completedBundle == null || Iterables.isEmpty(result.getUnprocessedElements())) {
- return Optional.absent();
+ return Optional.empty();
}
CommittedBundle<?> residual =
completedBundle.withElements((Iterable) result.getUnprocessedElements());
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 5b026cb..44ec35f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -20,6 +20,7 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
@@ -38,14 +39,13 @@
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalListener;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalListener;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -357,7 +357,7 @@
}
private VisibleExecutorUpdate(State newState, @Nullable Throwable exception) {
- this.thrown = Optional.fromNullable(exception);
+ this.thrown = Optional.ofNullable(exception);
this.newState = newState;
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
index ea1cc59..c808062 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@link Flatten} {@link
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index 77aa9fc..56220ea 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -50,8 +50,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
index c6b2417..c4e2e80 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.HashMap;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@link GroupByKeyOnly} {@link
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutabilityCheckingBundleFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutabilityCheckingBundleFactory.java
index f582c05..08430ce 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutabilityCheckingBundleFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutabilityCheckingBundleFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.runners.direct.DirectRunner.Enforcement;
import org.apache.beam.runners.local.StructuralKey;
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.util.MutationDetectors;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SetMultimap;
import org.joda.time.Instant;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
index bf75669..68664df 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.util.Iterator;
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
/** A factory that produces bundles that perform no additional validation. */
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactory.java
index 612a967..374855d 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactory.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** The evaluator for the {@link Impulse} transform. Produces only empty byte arrays. */
class ImpulseEvaluatorFactory implements TransformEvaluatorFactory {
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
index 6817c8c..6ba09f5 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/KeyedPValueTrackingVisitor.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.HashSet;
import java.util.Map;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/**
* A pipeline visitor that tracks all keyed {@link PValue PValues}. A {@link PValue} is keyed if it
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/MultiStepCombine.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/MultiStepCombine.java
index 4f0594a..f79ce06 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/MultiStepCombine.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/MultiStepCombine.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Collections;
import java.util.LinkedHashMap;
@@ -56,7 +56,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/** A {@link Combine} that performs the combine in multiple steps. */
@@ -174,7 +174,7 @@
@Nonnull
@Override
public String getUrn() {
- return "urn:beam:directrunner:transforms:multistepcombine:v1";
+ return "beam:directrunner:transforms:multistepcombine:v1";
}
@Nullable
@@ -310,7 +310,7 @@
}
static final String DIRECT_MERGE_ACCUMULATORS_EXTRACT_OUTPUT_URN =
- "urn:beam:directrunner:transforms:merge_accumulators_extract_output:v1";
+ "beam:directrunner:transforms:merge_accumulators_extract_output:v1";
/**
* A primitive {@link PTransform} that merges iterables of accumulators and extracts the output.
*
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
index 489561c..31eb80b 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.HashMap;
import java.util.List;
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
class ParDoEvaluator<InputT> implements TransformEvaluator<InputT> {
@@ -62,7 +62,8 @@
@Nullable Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
WindowingStrategy<?, ? extends BoundedWindow> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation);
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping);
}
public static <InputT, OutputT> DoFnRunnerFactory<InputT, OutputT> defaultRunnerFactory() {
@@ -77,7 +78,8 @@
schemaCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation) -> {
+ doFnSchemaInformation,
+ sideInputMapping) -> {
DoFnRunner<InputT, OutputT> underlying =
DoFnRunners.simpleRunner(
options,
@@ -90,7 +92,8 @@
schemaCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
return SimplePushbackSideInputDoFnRunner.create(underlying, sideInputs, sideInputReader);
};
}
@@ -109,6 +112,7 @@
List<TupleTag<?>> additionalOutputTags,
Map<TupleTag<?>, PCollection<?>> outputs,
DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping,
DoFnRunnerFactory<InputT, OutputT> runnerFactory) {
BundleOutputManager outputManager = createOutputManager(evaluationContext, key, outputs);
@@ -133,7 +137,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
return create(runner, stepContext, application, outputManager);
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
index 552a24a..9d25215 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
@@ -34,9 +34,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -83,7 +83,8 @@
ParDoTranslation.getSideInputs(application),
(TupleTag<OutputT>) ParDoTranslation.getMainOutputTag(application),
ParDoTranslation.getAdditionalOutputTags(application).getAll(),
- ParDoTranslation.getSchemaInformation(application));
+ ParDoTranslation.getSchemaInformation(application),
+ ParDoTranslation.getSideInputMapping(application));
return evaluator;
}
@@ -107,7 +108,8 @@
List<PCollectionView<?>> sideInputs,
TupleTag<OutputT> mainOutputTag,
List<TupleTag<?>> additionalOutputTags,
- DoFnSchemaInformation doFnSchemaInformation)
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping)
throws Exception {
String stepName = evaluationContext.getStepName(application);
DirectStepContext stepContext =
@@ -126,6 +128,7 @@
stepContext,
fnManager.get(),
doFnSchemaInformation,
+ sideInputMapping,
fnManager),
fnManager);
}
@@ -140,6 +143,7 @@
DirectStepContext stepContext,
DoFn<InputT, OutputT> fn,
DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping,
DoFnLifecycleManager fnManager)
throws Exception {
try {
@@ -157,6 +161,7 @@
additionalOutputTags,
pcollections(application.getOutputs()),
doFnSchemaInformation,
+ sideInputMapping,
runnerFactory);
} catch (Exception e) {
try {
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
index 9ba9ebd..f99d07f 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.Collections;
@@ -56,7 +56,7 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link PTransformOverrideFactory} that provides overrides for applications of a {@link ParDo}
@@ -109,7 +109,8 @@
ParDoTranslation.getMainOutputTag(application),
ParDoTranslation.getAdditionalOutputTags(application),
ParDoTranslation.getSideInputs(application),
- ParDoTranslation.getSchemaInformation(application));
+ ParDoTranslation.getSchemaInformation(application),
+ ParDoTranslation.getSideInputMapping(application));
} else {
return application.getTransform();
}
@@ -128,18 +129,21 @@
private final TupleTag<OutputT> mainOutputTag;
private final List<PCollectionView<?>> sideInputs;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
public GbkThenStatefulParDo(
DoFn<KV<K, InputT>, OutputT> doFn,
TupleTag<OutputT> mainOutputTag,
TupleTagList additionalOutputTags,
List<PCollectionView<?>> sideInputs,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.doFn = doFn;
this.additionalOutputTags = additionalOutputTags;
this.mainOutputTag = mainOutputTag;
this.sideInputs = sideInputs;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
@Override
@@ -202,12 +206,16 @@
.apply(
"Stateful ParDo",
new StatefulParDo<>(
- doFn, mainOutputTag, additionalOutputTags, sideInputs, doFnSchemaInformation));
+ doFn,
+ mainOutputTag,
+ additionalOutputTags,
+ sideInputs,
+ doFnSchemaInformation,
+ sideInputMapping));
}
}
- static final String DIRECT_STATEFUL_PAR_DO_URN =
- "urn:beam:directrunner:transforms:stateful_pardo:v1";
+ static final String DIRECT_STATEFUL_PAR_DO_URN = "beam:directrunner:transforms:stateful_pardo:v1";
static class StatefulParDo<K, InputT, OutputT>
extends PTransform<PCollection<? extends KeyedWorkItem<K, KV<K, InputT>>>, PCollectionTuple> {
@@ -216,18 +224,21 @@
private final TupleTag<OutputT> mainOutputTag;
private final List<PCollectionView<?>> sideInputs;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
public StatefulParDo(
DoFn<KV<K, InputT>, OutputT> doFn,
TupleTag<OutputT> mainOutputTag,
TupleTagList additionalOutputTags,
List<PCollectionView<?>> sideInputs,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.doFn = doFn;
this.mainOutputTag = mainOutputTag;
this.additionalOutputTags = additionalOutputTags;
this.sideInputs = sideInputs;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
public DoFn<KV<K, InputT>, OutputT> getDoFn() {
@@ -250,6 +261,10 @@
return doFnSchemaInformation;
}
+ public Map<String, PCollectionView<?>> getSideInputMapping() {
+ return sideInputMapping;
+ }
+
@Override
public Map<TupleTag<?>, PValue> getAdditionalInputs() {
return PCollectionViews.toAdditionalInputs(sideInputs);
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
index 7041ba5..0802997 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java
@@ -22,6 +22,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
+import java.util.Optional;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicLong;
@@ -36,8 +37,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -311,12 +311,12 @@
private static WorkUpdate fromBundle(
CommittedBundle<?> bundle, Collection<AppliedPTransform<?, ?, ?>> consumers) {
return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.of(bundle), consumers, Optional.absent());
+ Optional.of(bundle), consumers, Optional.empty());
}
private static WorkUpdate fromException(Exception e) {
return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.absent(), Collections.emptyList(), Optional.of(e));
+ Optional.empty(), Collections.emptyList(), Optional.of(e));
}
/** Returns the bundle that produced this update. */
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/RootProviderRegistry.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/RootProviderRegistry.java
index 45e37e8..42f8513 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/RootProviderRegistry.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/RootProviderRegistry.java
@@ -20,7 +20,7 @@
import static org.apache.beam.runners.core.construction.PTransformTranslation.FLATTEN_TRANSFORM_URN;
import static org.apache.beam.runners.core.construction.PTransformTranslation.IMPULSE_TRANSFORM_URN;
import static org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory.DIRECT_TEST_STREAM_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Collection;
import java.util.Map;
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.Impulse;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A {@link RootInputProvider} that delegates to primitive {@link RootInputProvider} implementations
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
index 94d4d46..975d1dc 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SideInputContainer.java
@@ -17,13 +17,14 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.atomic.AtomicReference;
import javax.annotation.Nullable;
@@ -41,14 +42,13 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* An in-process container for {@link PCollectionView PCollectionViews}, which provides methods for
@@ -283,7 +283,7 @@
@Override
public Optional<? extends Iterable<? extends WindowedValue<?>>> load(
PCollectionViewWindow<?> key) {
- return Optional.fromNullable(viewByWindows.getUnchecked(key).get());
+ return Optional.ofNullable(viewByWindows.getUnchecked(key).get());
}
}
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SplittableProcessElementsEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SplittableProcessElementsEvaluatorFactory.java
index 497d4e8..b0ff2d3 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SplittableProcessElementsEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SplittableProcessElementsEvaluatorFactory.java
@@ -17,9 +17,10 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
+import java.util.Collections;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import org.apache.beam.runners.core.DoFnRunners;
@@ -40,9 +41,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -124,7 +125,8 @@
application.getTransform().getSideInputs(),
application.getTransform().getMainOutputTag(),
application.getTransform().getAdditionalOutputTags().getAll(),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
final ParDoEvaluator<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>> pde =
evaluator.getParDoEvaluator();
final ProcessFn<InputT, OutputT, RestrictionT, PositionT> processFn =
@@ -188,7 +190,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation) -> {
+ doFnSchemaInformation,
+ sideInputMapping) -> {
ProcessFn<InputT, OutputT, RestrictionT, ?> processFn = (ProcessFn) fn;
return DoFnRunners.newProcessFnRunner(
processFn,
@@ -202,7 +205,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
};
}
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java
index 7c4a6e5b..366ca05 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.util.Collections;
@@ -52,10 +52,10 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** A {@link TransformEvaluatorFactory} for stateful {@link ParDo}. */
final class StatefulParDoEvaluatorFactory<K, InputT, OutputT> implements TransformEvaluatorFactory {
@@ -137,7 +137,8 @@
application.getTransform().getSideInputs(),
application.getTransform().getMainOutputTag(),
application.getTransform().getAdditionalOutputTags().getAll(),
- application.getTransform().getSchemaInformation());
+ application.getTransform().getSchemaInformation(),
+ application.getTransform().getSideInputMapping());
return new StatefulParDoEvaluator<>(delegateEvaluator);
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepAndKey.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepAndKey.java
index 610fd22..dcd7d43 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepAndKey.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepAndKey.java
@@ -20,7 +20,7 @@
import java.util.Objects;
import org.apache.beam.runners.local.StructuralKey;
import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A (Step, Key) pair. This is useful as a map key or cache key for things that are available
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
index 9369154..0468025 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/StepTransformResult.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
/** An immutable {@link TransformResult}. */
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
index 0ce5de8..b7dbf66 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java
@@ -45,9 +45,9 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -189,7 +189,7 @@
return ReplacementOutputs.singleton(outputs, newOutput);
}
- static final String DIRECT_TEST_STREAM_URN = "urn:beam:directrunner:transforms:test_stream:v1";
+ static final String DIRECT_TEST_STREAM_URN = "beam:directrunner:transforms:test_stream:v1";
static class DirectTestStream<T> extends PTransform<PBegin, PCollection<T>> {
private final transient DirectRunner runner;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
index 4b96ca4..a03cb38 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
@@ -29,8 +29,8 @@
import static org.apache.beam.runners.direct.ParDoMultiOverrideFactory.DIRECT_STATEFUL_PAR_DO_URN;
import static org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory.DIRECT_TEST_STREAM_URN;
import static org.apache.beam.runners.direct.ViewOverrideFactory.DIRECT_WRITE_VIEW_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.service.AutoService;
import java.util.ArrayList;
@@ -47,7 +47,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
index ce3bd34..39499c8 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
@@ -23,7 +23,7 @@
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadDeduplicator.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadDeduplicator.java
index 1507573..6193e6c 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadDeduplicator.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadDeduplicator.java
@@ -23,9 +23,9 @@
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.io.Read.Unbounded;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
import org.joda.time.Duration;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index 7cc9dc0..787dab1 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
index 376575b..5a6ce31 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@link CreatePCollectionView}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
index 397b067..8fb14f0 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewOverrideFactory.java
@@ -124,6 +124,5 @@
}
}
- public static final String DIRECT_WRITE_VIEW_URN =
- "urn:beam:directrunner:transforms:write_view:v1";
+ public static final String DIRECT_WRITE_VIEW_URN = "beam:directrunner:transforms:write_view:v1";
}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
index 00f0eb9..7f6800e 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
@@ -17,16 +17,18 @@
*/
package org.apache.beam.runners.direct;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.Serializable;
import java.util.PriorityQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.Executor;
+import javax.annotation.Nonnull;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Instant;
/**
@@ -162,8 +164,11 @@
private static class CallbackOrdering extends Ordering<WatermarkCallback>
implements Serializable {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public int compare(WatermarkCallback left, WatermarkCallback right) {
+ public int compare(@Nonnull WatermarkCallback left, @Nonnull WatermarkCallback right) {
return ComparisonChain.start()
.compare(left.fireAfter, right.fireAfter)
.compare(left.callback, right.callback, Ordering.arbitrary())
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
index 5fb6b4d..82dc0ae 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
@@ -17,10 +17,11 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collection;
@@ -40,6 +41,7 @@
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Function;
+import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import javax.annotation.concurrent.GuardedBy;
import org.apache.beam.runners.core.StateNamespace;
@@ -55,16 +57,16 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowTracing;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SortedMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TreeMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SortedMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeMultiset;
import org.joda.time.Instant;
/**
@@ -1584,8 +1586,12 @@
private static class BundleByElementTimestampComparator extends Ordering<Bundle<?, ?>>
implements Serializable {
+
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public int compare(Bundle<?, ?> o1, Bundle<?, ?> o2) {
+ public int compare(@Nonnull Bundle<?, ?> o1, @Nonnull Bundle<?, ?> o2) {
return ComparisonChain.start()
.compare(o1.getMinimumTimestamp(), o2.getMinimumTimestamp())
.result();
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
index 701b73e..2826261 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
index 62cc673..4fd898e 100644
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
+++ b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WriteWithShardingFactory.java
@@ -39,9 +39,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
/**
* A {@link PTransformOverrideFactory} that overrides {@link WriteFiles} {@link PTransform
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactory.java
deleted file mode 100644
index 118892c..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactory.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.values.PCollection;
-
-/** A factory that creates {@link UncommittedBundle UncommittedBundles}. */
-interface BundleFactory {
- /**
- * Create an {@link UncommittedBundle} from an empty input. Elements added to the bundle do not
- * belong to a {@link PCollection}.
- *
- * <p>For use in creating inputs to root transforms.
- */
- <T> UncommittedBundle<T> createRootBundle();
-
- /**
- * Create an {@link UncommittedBundle} from the specified input. Elements added to the bundle
- * belong to the {@code output} {@link PCollection}.
- */
- <T> UncommittedBundle<T> createBundle(PCollectionNode output);
-
- /**
- * Create an {@link UncommittedBundle} with the specified keys at the specified step. For use by
- * {@code DirectGroupByKeyOnly} {@link PTransform PTransforms}. Elements added to the bundle
- * belong to the {@code output} {@link PCollection}.
- */
- <K, T> UncommittedBundle<T> createKeyedBundle(StructuralKey<K> key, PCollectionNode output);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactory.java
deleted file mode 100644
index 53c2cb9..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactory.java
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.function.Consumer;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
-import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.sdk.util.WindowedValue;
-
-/**
- * An {@link OutputReceiverFactory} which adds received elements to {@link UncommittedBundle}
- * instances. The produced {@link UncommittedBundle bundles} are added to a provided {@link
- * StepTransformResult.Builder StepTransformResult Builder}.
- */
-class BundleFactoryOutputReceiverFactory implements OutputReceiverFactory {
- private final BundleFactory bundleFactory;
- private final RunnerApi.Components components;
-
- private final Consumer<UncommittedBundle<?>> bundleConsumer;
-
- private BundleFactoryOutputReceiverFactory(
- BundleFactory bundleFactory,
- Components components,
- Consumer<UncommittedBundle<?>> bundleConsumer) {
- this.bundleFactory = bundleFactory;
- this.components = components;
- this.bundleConsumer = bundleConsumer;
- }
-
- public static OutputReceiverFactory create(
- BundleFactory bundleFactory,
- Components components,
- Consumer<UncommittedBundle<?>> resultBuilder) {
- return new BundleFactoryOutputReceiverFactory(bundleFactory, components, resultBuilder);
- }
-
- @Override
- public <OutputT> FnDataReceiver<OutputT> create(String pCollectionId) {
- PCollectionNode pcollection =
- PipelineNode.pCollection(pCollectionId, components.getPcollectionsOrThrow(pCollectionId));
- return create(pcollection);
- }
-
- private <ElemT, OutputT> FnDataReceiver<OutputT> create(PCollectionNode pcollection) {
- UncommittedBundle<ElemT> bundle = bundleFactory.createBundle(pcollection);
- bundleConsumer.accept(bundle);
- return input -> bundle.add((WindowedValue<ElemT>) input);
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleProcessor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleProcessor.java
deleted file mode 100644
index a08e1b8..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleProcessor.java
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.local.Bundle;
-
-/**
- * An executor that is capable of processing some bundle of input over some executable stage or
- * step.
- */
-interface BundleProcessor<
- CollectionT, BundleT extends Bundle<?, ? extends CollectionT>, ExecutableT> {
- /**
- * Execute the provided bundle using the provided Executable, calling back to the {@link
- * CompletionCallback} when execution completes.
- */
- void process(BundleT bundle, ExecutableT consumer, CompletionCallback onComplete);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedBundle.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedBundle.java
deleted file mode 100644
index 93926d6..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedBundle.java
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.local.Bundle;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.joda.time.Instant;
-
-/**
- * Part of a {@link PCollection}. Elements are output to an {@link UncommittedBundle}, which will
- * eventually committed. Committed elements are executed by the {@link PTransform PTransforms} that
- * consume the {@link PCollection} this bundle is a part of at a later point.
- *
- * @param <T> the type of elements contained within this bundle
- */
-interface CommittedBundle<T> extends Bundle<T, PCollectionNode> {
- /** Returns the PCollection that the elements of this bundle belong to. */
- @Nullable
- @Override
- PCollectionNode getPCollection();
-
- /**
- * Returns the key that was output in the most recent {@code GroupByKey} in the execution of this
- * bundle.
- */
- @Override
- StructuralKey<?> getKey();
-
- /**
- * Returns an {@link Iterable} containing all of the elements that have been added to this {@link
- * CommittedBundle}.
- */
- Iterable<WindowedValue<T>> getElements();
-
- /**
- * Return the minimum timestamp among elements in this bundle.
- *
- * <p>This should be equivalent to iterating over all of the elements within a bundle and
- * selecting the minimum timestamp from among them.
- */
- @Override
- Instant getMinimumTimestamp();
-
- /**
- * Returns the processing time output watermark at the time the producing {@code Executable}
- * committed this bundle. Downstream synchronized processing time watermarks cannot progress past
- * this point before consuming this bundle.
- *
- * <p>This value is no greater than the earliest incomplete processing time or synchronized
- * processing time {@link TimerData timer} at the time this bundle was committed, including any
- * timers that fired to produce this bundle.
- */
- @Override
- Instant getSynchronizedProcessingOutputWatermark();
- /**
- * Return a new {@link CommittedBundle} that is like this one, except calls to {@link
- * #getElements()} will return the provided elements. This bundle is unchanged.
- *
- * <p>The value of the {@link #getSynchronizedProcessingOutputWatermark() synchronized processing
- * output watermark} of the returned {@link CommittedBundle} is equal to the value returned from
- * the current bundle. This is used to ensure a {@link PTransform} that could not complete
- * processing on input elements properly holds the synchronized processing time to the appropriate
- * value.
- */
- CommittedBundle<T> withElements(Iterable<WindowedValue<T>> elements);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedResult.java
deleted file mode 100644
index 5487118..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedResult.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import com.google.auto.value.AutoValue;
-import java.util.Set;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.transforms.View.CreatePCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-
-/** A {@link TransformResult} that has been committed. */
-@AutoValue
-abstract class CommittedResult<ExecutableT> {
- /** Returns the {@link PTransformNode} that produced this result. */
- public abstract ExecutableT getExecutable();
-
- /**
- * Returns the {@link CommittedBundle} that contains the input elements that could not be
- * processed by the evaluation. The returned optional is present if there were any unprocessed
- * input elements, and absent otherwise.
- */
- public abstract Optional<? extends CommittedBundle<?>> getUnprocessedInputs();
- /** Returns the outputs produced by the transform. */
- public abstract Iterable<? extends CommittedBundle<?>> getOutputs();
-
- /**
- * Returns if the transform that produced this result produced outputs.
- *
- * <p>Transforms that produce output via modifying the state of the runner (e.g. {@link
- * CreatePCollectionView}) should explicitly set this to true. If {@link #getOutputs()} returns a
- * nonempty iterable, this will also return true.
- */
- public abstract Set<OutputType> getProducedOutputTypes();
-
- public static CommittedResult<PTransformNode> create(
- TransformResult<?> original,
- Optional<? extends CommittedBundle<?>> unprocessedElements,
- Iterable<? extends CommittedBundle<?>> outputs,
- Set<OutputType> producedOutputs) {
- return new AutoValue_CommittedResult<>(
- original.getTransform(), unprocessedElements, outputs, producedOutputs);
- }
-
- enum OutputType {
- PCOLLECTION_VIEW,
- BUNDLE
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CompletionCallback.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CompletionCallback.java
deleted file mode 100644
index 4d69044..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CompletionCallback.java
+++ /dev/null
@@ -1,42 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-
-/** A callback for completing a bundle of input. */
-interface CompletionCallback {
- /** Handle a successful result, returning the committed outputs of the result. */
- CommittedResult handleResult(CommittedBundle<?> inputBundle, TransformResult<?> result);
-
- /**
- * Handle an input bundle that did not require processing.
- *
- * <p>This occurs when a Source has no splits that can currently produce outputs.
- */
- void handleEmpty(PTransformNode transform);
-
- /** Handle a result that terminated abnormally due to the provided {@link Exception}. */
- void handleException(CommittedBundle<?> inputBundle, Exception t);
-
- /**
- * Handle a result that terminated abnormally due to the provided {@link Error}. The pipeline
- * should be shut down, and the Error propagated.
- */
- void handleError(Error err);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternals.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternals.java
deleted file mode 100644
index d217121..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternals.java
+++ /dev/null
@@ -1,463 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import java.util.Collection;
-import java.util.HashSet;
-import java.util.Map;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryBag;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryCombiningState;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryMap;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemorySet;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryState;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryStateBinder;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryValue;
-import org.apache.beam.runners.core.InMemoryStateInternals.InMemoryWatermarkHold;
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.StateNamespace;
-import org.apache.beam.runners.core.StateTable;
-import org.apache.beam.runners.core.StateTag;
-import org.apache.beam.runners.core.StateTag.StateBinder;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.state.BagState;
-import org.apache.beam.sdk.state.CombiningState;
-import org.apache.beam.sdk.state.MapState;
-import org.apache.beam.sdk.state.SetState;
-import org.apache.beam.sdk.state.State;
-import org.apache.beam.sdk.state.StateContext;
-import org.apache.beam.sdk.state.StateContexts;
-import org.apache.beam.sdk.state.ValueState;
-import org.apache.beam.sdk.state.WatermarkHoldState;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Instant;
-
-/**
- * {@link StateInternals} built on top of an underlying {@link StateTable} that contains instances
- * of {@link InMemoryState}. Whenever state that exists in the underlying {@link StateTable} is
- * accessed, an independent copy will be created within this table.
- */
-class CopyOnAccessInMemoryStateInternals<K> implements StateInternals {
- private final CopyOnAccessInMemoryStateTable table;
-
- private K key;
-
- /**
- * Creates a new {@link CopyOnAccessInMemoryStateInternals} with the underlying (possibly null)
- * StateInternals.
- */
- public static <K> CopyOnAccessInMemoryStateInternals withUnderlying(
- K key, @Nullable CopyOnAccessInMemoryStateInternals underlying) {
- return new CopyOnAccessInMemoryStateInternals<>(key, underlying);
- }
-
- private CopyOnAccessInMemoryStateInternals(K key, CopyOnAccessInMemoryStateInternals underlying) {
- this.key = key;
- table = new CopyOnAccessInMemoryStateTable(underlying == null ? null : underlying.table);
- }
-
- /**
- * Ensures this {@link CopyOnAccessInMemoryStateInternals} is complete. Other copies of state for
- * the same Step and Key may be discarded after invoking this method.
- *
- * <p>For each {@link StateNamespace}, for each {@link StateTag address} in that namespace that
- * has not been bound in this {@link CopyOnAccessInMemoryStateInternals}, put a reference to that
- * state within this {@link StateInternals}.
- *
- * <p>Additionally, stores the {@link WatermarkHoldState} with the earliest time bound in the
- * state table after the commit is completed, enabling calls to {@link
- * #getEarliestWatermarkHold()}.
- *
- * @return this table
- */
- public CopyOnAccessInMemoryStateInternals<K> commit() {
- table.commit();
- return this;
- }
-
- /**
- * Gets the earliest Watermark Hold present in this table.
- *
- * <p>Must be called after this state has been committed. Will throw an {@link
- * IllegalStateException} if the state has not been committed.
- */
- public Instant getEarliestWatermarkHold() {
- // After commit, the watermark hold is always present, but may be
- // BoundedWindow#TIMESTAMP_MAX_VALUE if there is no hold set.
- checkState(
- table.earliestWatermarkHold.isPresent(),
- "Can't get the earliest watermark hold in a %s before it is committed",
- getClass().getSimpleName());
- return table.earliestWatermarkHold.get();
- }
-
- @Override
- public <T extends State> T state(
- StateNamespace namespace, StateTag<T> address, StateContext<?> c) {
- return table.get(namespace, address, c);
- }
-
- @Override
- public Object getKey() {
- return key;
- }
-
- public boolean isEmpty() {
- return Iterables.isEmpty(table.values());
- }
-
- /**
- * A {@link StateTable} that, when a value is retrieved with {@link StateTable#get(StateNamespace,
- * StateTag, StateContext)}, first attempts to obtain a copy of existing {@link State} from an
- * underlying {@link StateTable}.
- */
- private static class CopyOnAccessInMemoryStateTable extends StateTable {
- private Optional<StateTable> underlying;
-
- /**
- * The StateBinderFactory currently in use by this {@link CopyOnAccessInMemoryStateTable}.
- *
- * <p>There are three {@link StateBinderFactory} implementations used by the {@link
- * CopyOnAccessInMemoryStateTable}.
- *
- * <ul>
- * <li>The default {@link StateBinderFactory} is a {@link CopyOnBindBinderFactory}, allowing
- * the table to copy any existing {@link State} values to this {@link StateTable} from the
- * underlying table when accessed, at which point mutations will not be visible to the
- * underlying table - effectively a "Copy by Value" binder.
- * <li>During the execution of the {@link #commit()} method, this is a {@link
- * ReadThroughBinderFactory}, which copies the references to the existing {@link State}
- * objects to this {@link StateTable}.
- * <li>After the execution of the {@link #commit()} method, this is an instance of {@link
- * InMemoryStateBinderFactory}, which constructs new instances of state when a {@link
- * StateTag} is bound.
- * </ul>
- */
- private StateBinderFactory binderFactory;
-
- /** The earliest watermark hold in this table. */
- private Optional<Instant> earliestWatermarkHold;
-
- public CopyOnAccessInMemoryStateTable(StateTable underlying) {
- this.underlying = Optional.fromNullable(underlying);
- binderFactory = new CopyOnBindBinderFactory(this.underlying);
- earliestWatermarkHold = Optional.absent();
- }
-
- /**
- * Copies all values in the underlying table to this table, then discards the underlying table.
- *
- * <p>If there is an underlying table, this replaces the existing {@link
- * CopyOnBindBinderFactory} with a {@link ReadThroughBinderFactory}, then reads all of the
- * values in the existing table, binding the state values to this table. The old StateTable
- * should be discarded after the call to {@link #commit()}.
- *
- * <p>After copying all of the existing values, replace the binder factory with an instance of
- * {@link InMemoryStateBinderFactory} to construct new values, since all existing values are
- * bound in this {@link StateTable table} and this table represents the canonical state.
- */
- private void commit() {
- Instant earliestHold = getEarliestWatermarkHold();
- if (underlying.isPresent()) {
- ReadThroughBinderFactory readThroughBinder =
- new ReadThroughBinderFactory<>(underlying.get());
- binderFactory = readThroughBinder;
- Instant earliestUnderlyingHold = readThroughBinder.readThroughAndGetEarliestHold(this);
- if (earliestUnderlyingHold.isBefore(earliestHold)) {
- earliestHold = earliestUnderlyingHold;
- }
- }
- earliestWatermarkHold = Optional.of(earliestHold);
- clearEmpty();
- binderFactory = new InMemoryStateBinderFactory();
- underlying = Optional.absent();
- }
-
- /**
- * Get the earliest watermark hold in this table. Ignores the contents of any underlying table.
- */
- private Instant getEarliestWatermarkHold() {
- Instant earliest = BoundedWindow.TIMESTAMP_MAX_VALUE;
- for (State existingState : this.values()) {
- if (existingState instanceof WatermarkHoldState) {
- Instant hold = ((WatermarkHoldState) existingState).read();
- if (hold != null && hold.isBefore(earliest)) {
- earliest = hold;
- }
- }
- }
- return earliest;
- }
-
- /**
- * Clear all empty {@link StateNamespace StateNamespaces} from this table. If all states are
- * empty, clear the entire table.
- *
- * <p>Because {@link InMemoryState} is not removed from the {@link StateTable} after it is
- * cleared, in case contents are modified after being cleared, the table must be explicitly
- * checked to ensure that it contains state and removed if not (otherwise we may never use the
- * table again).
- */
- private void clearEmpty() {
- Collection<StateNamespace> emptyNamespaces = new HashSet<>(this.getNamespacesInUse());
- for (StateNamespace namespace : this.getNamespacesInUse()) {
- for (State existingState : this.getTagsInUse(namespace).values()) {
- if (!((InMemoryState<?>) existingState).isCleared()) {
- emptyNamespaces.remove(namespace);
- break;
- }
- }
- }
- for (StateNamespace empty : emptyNamespaces) {
- this.clearNamespace(empty);
- }
- }
-
- @Override
- protected StateBinder binderForNamespace(final StateNamespace namespace, StateContext<?> c) {
- return binderFactory.forNamespace(namespace, c);
- }
-
- private interface StateBinderFactory {
- StateBinder forNamespace(StateNamespace namespace, StateContext<?> c);
- }
-
- /**
- * {@link StateBinderFactory} that creates a copy of any existing state when the state is bound.
- */
- private static class CopyOnBindBinderFactory implements StateBinderFactory {
- private final Optional<StateTable> underlying;
-
- public CopyOnBindBinderFactory(Optional<StateTable> underlying) {
- this.underlying = underlying;
- }
-
- private boolean containedInUnderlying(StateNamespace namespace, StateTag<?> tag) {
- return underlying.isPresent()
- && underlying.get().isNamespaceInUse(namespace)
- && underlying.get().getTagsInUse(namespace).containsKey(tag);
- }
-
- @Override
- public StateBinder forNamespace(final StateNamespace namespace, final StateContext<?> c) {
- return new StateBinder() {
- @Override
- public WatermarkHoldState bindWatermark(
- StateTag<WatermarkHoldState> address, TimestampCombiner timestampCombiner) {
- if (containedInUnderlying(namespace, address)) {
- @SuppressWarnings("unchecked")
- InMemoryState<? extends WatermarkHoldState> existingState =
- (InMemoryState<? extends WatermarkHoldState>)
- underlying.get().get(namespace, address, c);
- return existingState.copy();
- } else {
- return new InMemoryWatermarkHold<>(timestampCombiner);
- }
- }
-
- @Override
- public <T> ValueState<T> bindValue(StateTag<ValueState<T>> address, Coder<T> coder) {
- if (containedInUnderlying(namespace, address)) {
- @SuppressWarnings("unchecked")
- InMemoryState<? extends ValueState<T>> existingState =
- (InMemoryState<? extends ValueState<T>>)
- underlying.get().get(namespace, address, c);
- return existingState.copy();
- } else {
- return new InMemoryValue<>(coder);
- }
- }
-
- @Override
- public <InputT, AccumT, OutputT>
- CombiningState<InputT, AccumT, OutputT> bindCombiningValue(
- StateTag<CombiningState<InputT, AccumT, OutputT>> address,
- Coder<AccumT> accumCoder,
- CombineFn<InputT, AccumT, OutputT> combineFn) {
- if (containedInUnderlying(namespace, address)) {
- @SuppressWarnings("unchecked")
- InMemoryState<? extends CombiningState<InputT, AccumT, OutputT>> existingState =
- (InMemoryState<? extends CombiningState<InputT, AccumT, OutputT>>)
- underlying.get().get(namespace, address, c);
- return existingState.copy();
- } else {
- return new InMemoryCombiningState<>(combineFn, accumCoder);
- }
- }
-
- @Override
- public <T> BagState<T> bindBag(StateTag<BagState<T>> address, Coder<T> elemCoder) {
- if (containedInUnderlying(namespace, address)) {
- @SuppressWarnings("unchecked")
- InMemoryState<? extends BagState<T>> existingState =
- (InMemoryState<? extends BagState<T>>)
- underlying.get().get(namespace, address, c);
- return existingState.copy();
- } else {
- return new InMemoryBag<>(elemCoder);
- }
- }
-
- @Override
- public <T> SetState<T> bindSet(StateTag<SetState<T>> address, Coder<T> elemCoder) {
- if (containedInUnderlying(namespace, address)) {
- @SuppressWarnings("unchecked")
- InMemoryState<? extends SetState<T>> existingState =
- (InMemoryState<? extends SetState<T>>)
- underlying.get().get(namespace, address, c);
- return existingState.copy();
- } else {
- return new InMemorySet<>(elemCoder);
- }
- }
-
- @Override
- public <KeyT, ValueT> MapState<KeyT, ValueT> bindMap(
- StateTag<MapState<KeyT, ValueT>> address,
- Coder<KeyT> mapKeyCoder,
- Coder<ValueT> mapValueCoder) {
- if (containedInUnderlying(namespace, address)) {
- @SuppressWarnings("unchecked")
- InMemoryState<? extends MapState<KeyT, ValueT>> existingState =
- (InMemoryState<? extends MapState<KeyT, ValueT>>)
- underlying.get().get(namespace, address, c);
- return existingState.copy();
- } else {
- return new InMemoryMap<>(mapKeyCoder, mapValueCoder);
- }
- }
-
- @Override
- public <InputT, AccumT, OutputT>
- CombiningState<InputT, AccumT, OutputT> bindCombiningValueWithContext(
- StateTag<CombiningState<InputT, AccumT, OutputT>> address,
- Coder<AccumT> accumCoder,
- CombineFnWithContext<InputT, AccumT, OutputT> combineFn) {
- return bindCombiningValue(address, accumCoder, CombineFnUtil.bindContext(combineFn, c));
- }
- };
- }
- }
-
- /**
- * {@link StateBinderFactory} that reads directly from the underlying table. Used during calls
- * to {@link CopyOnAccessInMemoryStateTable#commit()} to read all values from the underlying
- * table.
- */
- private static class ReadThroughBinderFactory<K> implements StateBinderFactory {
- private final StateTable underlying;
-
- public ReadThroughBinderFactory(StateTable underlying) {
- this.underlying = underlying;
- }
-
- public Instant readThroughAndGetEarliestHold(StateTable readTo) {
- Instant earliestHold = BoundedWindow.TIMESTAMP_MAX_VALUE;
- for (StateNamespace namespace : underlying.getNamespacesInUse()) {
- for (Map.Entry<StateTag, State> existingState :
- underlying.getTagsInUse(namespace).entrySet()) {
- if (!((InMemoryState<?>) existingState.getValue()).isCleared()) {
- // Only read through non-cleared values to ensure that completed windows are
- // eventually discarded, and remember the earliest watermark hold from among those
- // values.
- State state =
- readTo.get(namespace, existingState.getKey(), StateContexts.nullContext());
- if (state instanceof WatermarkHoldState) {
- Instant hold = ((WatermarkHoldState) state).read();
- if (hold != null && hold.isBefore(earliestHold)) {
- earliestHold = hold;
- }
- }
- }
- }
- }
- return earliestHold;
- }
-
- @Override
- public StateBinder forNamespace(final StateNamespace namespace, final StateContext<?> c) {
- return new StateBinder() {
- @Override
- public WatermarkHoldState bindWatermark(
- StateTag<WatermarkHoldState> address, TimestampCombiner timestampCombiner) {
- return underlying.get(namespace, address, c);
- }
-
- @Override
- public <T> ValueState<T> bindValue(StateTag<ValueState<T>> address, Coder<T> coder) {
- return underlying.get(namespace, address, c);
- }
-
- @Override
- public <InputT, AccumT, OutputT>
- CombiningState<InputT, AccumT, OutputT> bindCombiningValue(
- StateTag<CombiningState<InputT, AccumT, OutputT>> address,
- Coder<AccumT> accumCoder,
- CombineFn<InputT, AccumT, OutputT> combineFn) {
- return underlying.get(namespace, address, c);
- }
-
- @Override
- public <T> BagState<T> bindBag(StateTag<BagState<T>> address, Coder<T> elemCoder) {
- return underlying.get(namespace, address, c);
- }
-
- @Override
- public <T> SetState<T> bindSet(StateTag<SetState<T>> address, Coder<T> elemCoder) {
- return underlying.get(namespace, address, c);
- }
-
- @Override
- public <KeyT, ValueT> MapState<KeyT, ValueT> bindMap(
- StateTag<MapState<KeyT, ValueT>> address,
- Coder<KeyT> mapKeyCoder,
- Coder<ValueT> mapValueCoder) {
- return underlying.get(namespace, address, c);
- }
-
- @Override
- public <InputT, AccumT, OutputT>
- CombiningState<InputT, AccumT, OutputT> bindCombiningValueWithContext(
- StateTag<CombiningState<InputT, AccumT, OutputT>> address,
- Coder<AccumT> accumCoder,
- CombineFnWithContext<InputT, AccumT, OutputT> combineFn) {
- return bindCombiningValue(address, accumCoder, CombineFnUtil.bindContext(combineFn, c));
- }
- };
- }
- }
-
- private static class InMemoryStateBinderFactory implements StateBinderFactory {
-
- public InMemoryStateBinderFactory() {}
-
- @Override
- public StateBinder forNamespace(StateNamespace namespace, StateContext<?> c) {
- return new InMemoryStateBinder(c);
- }
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectGroupByKey.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectGroupByKey.java
deleted file mode 100644
index 37b5430..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectGroupByKey.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-
-import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.KeyedWorkItemCoder;
-import org.apache.beam.runners.core.construction.ForwardingPTransform;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.WindowingStrategy;
-
-class DirectGroupByKey<K, V>
- extends ForwardingPTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> {
- private final PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> original;
-
- static final String DIRECT_GBKO_URN = "urn:beam:directrunner:transforms:gbko:v1";
- static final String DIRECT_GABW_URN = "urn:beam:directrunner:transforms:gabw:v1";
- private final WindowingStrategy<?, ?> outputWindowingStrategy;
-
- DirectGroupByKey(
- PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> original,
- WindowingStrategy<?, ?> outputWindowingStrategy) {
- this.original = original;
- this.outputWindowingStrategy = outputWindowingStrategy;
- }
-
- @Override
- public PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> delegate() {
- return original;
- }
-
- @Override
- public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
- // This operation groups by the combination of key and window,
- // merging windows as needed, using the windows assigned to the
- // key/value input elements and the window merge operation of the
- // window function associated with the input PCollection.
- WindowingStrategy<?, ?> inputWindowingStrategy = input.getWindowingStrategy();
-
- // By default, implement GroupByKey via a series of lower-level operations.
- return input
- .apply(new DirectGroupByKeyOnly<>())
-
- // Group each key's values by window, merging windows as needed.
- .apply(
- "GroupAlsoByWindow",
- new DirectGroupAlsoByWindow<>(inputWindowingStrategy, outputWindowingStrategy));
- }
-
- static final class DirectGroupByKeyOnly<K, V>
- extends PTransform<PCollection<KV<K, V>>, PCollection<KeyedWorkItem<K, V>>> {
- @Override
- public PCollection<KeyedWorkItem<K, V>> expand(PCollection<KV<K, V>> input) {
- return PCollection.createPrimitiveOutputInternal(
- input.getPipeline(),
- WindowingStrategy.globalDefault(),
- input.isBounded(),
- KeyedWorkItemCoder.of(
- GroupByKey.getKeyCoder(input.getCoder()),
- GroupByKey.getInputValueCoder(input.getCoder()),
- input.getWindowingStrategy().getWindowFn().windowCoder()));
- }
-
- DirectGroupByKeyOnly() {}
- }
-
- static final class DirectGroupAlsoByWindow<K, V>
- extends PTransform<PCollection<KeyedWorkItem<K, V>>, PCollection<KV<K, Iterable<V>>>> {
-
- private final WindowingStrategy<?, ?> inputWindowingStrategy;
- private final WindowingStrategy<?, ?> outputWindowingStrategy;
-
- public DirectGroupAlsoByWindow(
- WindowingStrategy<?, ?> inputWindowingStrategy,
- WindowingStrategy<?, ?> outputWindowingStrategy) {
- this.inputWindowingStrategy = inputWindowingStrategy;
- this.outputWindowingStrategy = outputWindowingStrategy;
- }
-
- public WindowingStrategy<?, ?> getInputWindowingStrategy() {
- return inputWindowingStrategy;
- }
-
- private KeyedWorkItemCoder<K, V> getKeyedWorkItemCoder(Coder<KeyedWorkItem<K, V>> inputCoder) {
- // Coder<KV<...>> --> KvCoder<...>
- checkArgument(
- inputCoder instanceof KeyedWorkItemCoder,
- "%s requires a %s<...> but got %s",
- getClass().getSimpleName(),
- KvCoder.class.getSimpleName(),
- inputCoder);
- @SuppressWarnings("unchecked")
- KeyedWorkItemCoder<K, V> kvCoder = (KeyedWorkItemCoder<K, V>) inputCoder;
- return kvCoder;
- }
-
- public Coder<V> getValueCoder(Coder<KeyedWorkItem<K, V>> inputCoder) {
- return getKeyedWorkItemCoder(inputCoder).getElementCoder();
- }
-
- @Override
- public PCollection<KV<K, Iterable<V>>> expand(PCollection<KeyedWorkItem<K, V>> input) {
- KeyedWorkItemCoder<K, V> inputCoder = getKeyedWorkItemCoder(input.getCoder());
- return PCollection.createPrimitiveOutputInternal(
- input.getPipeline(),
- outputWindowingStrategy,
- input.isBounded(),
- KvCoder.of(inputCoder.getKeyCoder(), IterableCoder.of(inputCoder.getElementCoder())));
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
deleted file mode 100644
index a057260..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
+++ /dev/null
@@ -1,307 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static java.util.Arrays.asList;
-
-import java.util.ArrayList;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.atomic.AtomicReference;
-import javax.annotation.Nullable;
-import javax.annotation.concurrent.GuardedBy;
-import org.apache.beam.runners.core.metrics.DistributionData;
-import org.apache.beam.runners.core.metrics.GaugeData;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
-import org.apache.beam.runners.core.metrics.MetricsMap;
-import org.apache.beam.sdk.metrics.DistributionResult;
-import org.apache.beam.sdk.metrics.GaugeResult;
-import org.apache.beam.sdk.metrics.MetricFiltering;
-import org.apache.beam.sdk.metrics.MetricKey;
-import org.apache.beam.sdk.metrics.MetricQueryResults;
-import org.apache.beam.sdk.metrics.MetricResult;
-import org.apache.beam.sdk.metrics.MetricResults;
-import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
-
-/** Implementation of {@link MetricResults} for the Direct Runner. */
-class DirectMetrics extends MetricResults {
-
- // TODO: (BEAM-723) Create a shared ExecutorService for maintenance tasks in the DirectRunner.
- private static final ExecutorService COUNTER_COMMITTER =
- Executors.newCachedThreadPool(
- new ThreadFactoryBuilder()
- .setThreadFactory(MoreExecutors.platformThreadFactory())
- .setDaemon(true)
- .setNameFormat("direct-metrics-counter-committer")
- .build());
-
- private interface MetricAggregation<UpdateT, ResultT> {
- UpdateT zero();
-
- UpdateT combine(Iterable<UpdateT> updates);
-
- ResultT extract(UpdateT data);
- }
-
- /**
- * Implementation of a metric in the direct runner.
- *
- * @param <UpdateT> The type of raw data received and aggregated across updates.
- * @param <ResultT> The type of result extracted from the data.
- */
- private static class DirectMetric<UpdateT, ResultT> {
- private final MetricAggregation<UpdateT, ResultT> aggregation;
-
- private final AtomicReference<UpdateT> finishedCommitted;
-
- private final Object attemptedLock = new Object();
-
- @GuardedBy("attemptedLock")
- private volatile UpdateT finishedAttempted;
-
- private final ConcurrentMap<CommittedBundle<?>, UpdateT> inflightAttempted =
- new ConcurrentHashMap<>();
-
- public DirectMetric(MetricAggregation<UpdateT, ResultT> aggregation) {
- this.aggregation = aggregation;
- finishedCommitted = new AtomicReference<>(aggregation.zero());
- finishedAttempted = aggregation.zero();
- }
-
- /**
- * Add the given {@code tentativeCumulative} update to the physical aggregate.
- *
- * @param bundle The bundle receiving an update.
- * @param tentativeCumulative The new cumulative value for the given bundle.
- */
- public void updatePhysical(CommittedBundle<?> bundle, UpdateT tentativeCumulative) {
- // Add (or update) the cumulatiev value for the given bundle.
- inflightAttempted.put(bundle, tentativeCumulative);
- }
-
- /**
- * Commit a physical value for the given {@code bundle}.
- *
- * @param bundle The bundle being committed.
- * @param finalCumulative The final cumulative value for the given bundle.
- */
- @SuppressWarnings("FutureReturnValueIgnored") // direct runner metrics are best-effort;
- // we choose not to block on async commit
- public void commitPhysical(final CommittedBundle<?> bundle, final UpdateT finalCumulative) {
- // To prevent a query from blocking the commit, we perform the commit in two steps.
- // 1. We perform a non-blocking write to the uncommitted table to make the new value
- // available immediately.
- // 2. We submit a runnable that will commit the update and remove the tentative value in
- // a synchronized block.
- inflightAttempted.put(bundle, finalCumulative);
- COUNTER_COMMITTER.submit(
- () -> {
- synchronized (attemptedLock) {
- finishedAttempted = aggregation.combine(asList(finishedAttempted, finalCumulative));
- inflightAttempted.remove(bundle);
- }
- });
- }
-
- /** Extract the latest values from all attempted and in-progress bundles. */
- public ResultT extractLatestAttempted() {
- ArrayList<UpdateT> updates = new ArrayList<>(inflightAttempted.size() + 1);
- // Within this block we know that will be consistent. Specifically, the only change that can
- // happen concurrently is the addition of new (larger) values to inflightAttempted.
- synchronized (attemptedLock) {
- updates.add(finishedAttempted);
- updates.addAll(inflightAttempted.values());
- }
- return aggregation.extract(aggregation.combine(updates));
- }
-
- /**
- * Commit a logical value for the given {@code bundle}.
- *
- * @param bundle The bundle being committed.
- * @param finalCumulative The final cumulative value for the given bundle.
- */
- public void commitLogical(final CommittedBundle<?> bundle, final UpdateT finalCumulative) {
- UpdateT current;
- do {
- current = finishedCommitted.get();
- } while (!finishedCommitted.compareAndSet(
- current, aggregation.combine(asList(current, finalCumulative))));
- }
-
- /** Extract the value from all successfully committed bundles. */
- public ResultT extractCommitted() {
- return aggregation.extract(finishedCommitted.get());
- }
- }
-
- private static final MetricAggregation<Long, Long> COUNTER =
- new MetricAggregation<Long, Long>() {
- @Override
- public Long zero() {
- return 0L;
- }
-
- @Override
- public Long combine(Iterable<Long> updates) {
- long value = 0;
- for (long update : updates) {
- value += update;
- }
- return value;
- }
-
- @Override
- public Long extract(Long data) {
- return data;
- }
- };
-
- private static final MetricAggregation<DistributionData, DistributionResult> DISTRIBUTION =
- new MetricAggregation<DistributionData, DistributionResult>() {
- @Override
- public DistributionData zero() {
- return DistributionData.EMPTY;
- }
-
- @Override
- public DistributionData combine(Iterable<DistributionData> updates) {
- DistributionData result = DistributionData.EMPTY;
- for (DistributionData update : updates) {
- result = result.combine(update);
- }
- return result;
- }
-
- @Override
- public DistributionResult extract(DistributionData data) {
- return data.extractResult();
- }
- };
-
- private static final MetricAggregation<GaugeData, GaugeResult> GAUGE =
- new MetricAggregation<GaugeData, GaugeResult>() {
- @Override
- public GaugeData zero() {
- return GaugeData.empty();
- }
-
- @Override
- public GaugeData combine(Iterable<GaugeData> updates) {
- GaugeData result = GaugeData.empty();
- for (GaugeData update : updates) {
- result = result.combine(update);
- }
- return result;
- }
-
- @Override
- public GaugeResult extract(GaugeData data) {
- return data.extractResult();
- }
- };
-
- /** The current values of counters in memory. */
- private MetricsMap<MetricKey, DirectMetric<Long, Long>> counters =
- new MetricsMap<>(unusedKey -> new DirectMetric<>(COUNTER));
-
- private MetricsMap<MetricKey, DirectMetric<DistributionData, DistributionResult>> distributions =
- new MetricsMap<>(unusedKey -> new DirectMetric<>(DISTRIBUTION));
- private MetricsMap<MetricKey, DirectMetric<GaugeData, GaugeResult>> gauges =
- new MetricsMap<>(unusedKey -> new DirectMetric<>(GAUGE));
-
- @Override
- public MetricQueryResults queryMetrics(@Nullable MetricsFilter filter) {
- ImmutableList.Builder<MetricResult<Long>> counterResults = ImmutableList.builder();
- for (Entry<MetricKey, DirectMetric<Long, Long>> counter : counters.entries()) {
- maybeExtractResult(filter, counterResults, counter);
- }
- ImmutableList.Builder<MetricResult<DistributionResult>> distributionResults =
- ImmutableList.builder();
- for (Entry<MetricKey, DirectMetric<DistributionData, DistributionResult>> distribution :
- distributions.entries()) {
- maybeExtractResult(filter, distributionResults, distribution);
- }
- ImmutableList.Builder<MetricResult<GaugeResult>> gaugeResults = ImmutableList.builder();
- for (Entry<MetricKey, DirectMetric<GaugeData, GaugeResult>> gauge : gauges.entries()) {
- maybeExtractResult(filter, gaugeResults, gauge);
- }
-
- return MetricQueryResults.create(
- counterResults.build(), distributionResults.build(), gaugeResults.build());
- }
-
- private <ResultT> void maybeExtractResult(
- MetricsFilter filter,
- ImmutableList.Builder<MetricResult<ResultT>> resultsBuilder,
- Map.Entry<MetricKey, ? extends DirectMetric<?, ResultT>> entry) {
- if (MetricFiltering.matches(filter, entry.getKey())) {
- resultsBuilder.add(
- MetricResult.create(
- entry.getKey(),
- entry.getValue().extractCommitted(),
- entry.getValue().extractLatestAttempted()));
- }
- }
-
- /** Apply metric updates that represent physical counter deltas to the current metric values. */
- public void updatePhysical(CommittedBundle<?> bundle, MetricUpdates updates) {
- for (MetricUpdate<Long> counter : updates.counterUpdates()) {
- counters.get(counter.getKey()).updatePhysical(bundle, counter.getUpdate());
- }
- for (MetricUpdate<DistributionData> distribution : updates.distributionUpdates()) {
- distributions.get(distribution.getKey()).updatePhysical(bundle, distribution.getUpdate());
- }
- for (MetricUpdate<GaugeData> gauge : updates.gaugeUpdates()) {
- gauges.get(gauge.getKey()).updatePhysical(bundle, gauge.getUpdate());
- }
- }
-
- public void commitPhysical(CommittedBundle<?> bundle, MetricUpdates updates) {
- for (MetricUpdate<Long> counter : updates.counterUpdates()) {
- counters.get(counter.getKey()).commitPhysical(bundle, counter.getUpdate());
- }
- for (MetricUpdate<DistributionData> distribution : updates.distributionUpdates()) {
- distributions.get(distribution.getKey()).commitPhysical(bundle, distribution.getUpdate());
- }
- for (MetricUpdate<GaugeData> gauge : updates.gaugeUpdates()) {
- gauges.get(gauge.getKey()).commitPhysical(bundle, gauge.getUpdate());
- }
- }
-
- /** Apply metric updates that represent new logical values from a bundle being committed. */
- public void commitLogical(CommittedBundle<?> bundle, MetricUpdates updates) {
- for (MetricUpdate<Long> counter : updates.counterUpdates()) {
- counters.get(counter.getKey()).commitLogical(bundle, counter.getUpdate());
- }
- for (MetricUpdate<DistributionData> distribution : updates.distributionUpdates()) {
- distributions.get(distribution.getKey()).commitLogical(bundle, distribution.getUpdate());
- }
- for (MetricUpdate<GaugeData> gauge : updates.gaugeUpdates()) {
- gauges.get(gauge.getKey()).commitLogical(bundle, gauge.getUpdate());
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectStateAndTimers.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectStateAndTimers.java
deleted file mode 100644
index ca36781..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectStateAndTimers.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.direct.Clock;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
-import org.apache.beam.runners.local.StructuralKey;
-
-/**
- * State and Timer access for the {@link ReferenceRunner}.
- *
- * <p>This provides per-key, per-stage access to {@link CopyOnAccessInMemoryStateInternals} and
- * {@link DirectTimerInternals} for transforms that require access to state or timers.
- *
- * <p>This implementation is not thread safe. A new {@link DirectStateAndTimers} must be created for
- * each thread that requires it.
- */
-class DirectStateAndTimers<K> implements StepStateAndTimers<K> {
- private final StructuralKey<K> key;
- private final CopyOnAccessInMemoryStateInternals existingState;
-
- private final Clock clock;
- private final TransformWatermarks watermarks;
-
- private CopyOnAccessInMemoryStateInternals<K> stateInternals;
- private DirectTimerInternals timerInternals;
-
- DirectStateAndTimers(
- StructuralKey<K> key,
- CopyOnAccessInMemoryStateInternals existingState,
- Clock clock,
- TransformWatermarks watermarks) {
- this.key = key;
- this.existingState = existingState;
- this.clock = clock;
- this.watermarks = watermarks;
- }
-
- @Override
- public CopyOnAccessInMemoryStateInternals<K> stateInternals() {
- if (stateInternals == null) {
- stateInternals = CopyOnAccessInMemoryStateInternals.withUnderlying(key, existingState);
- }
- return stateInternals;
- }
-
- @Override
- public DirectTimerInternals timerInternals() {
- if (timerInternals == null) {
- timerInternals = DirectTimerInternals.create(clock, watermarks, TimerUpdate.builder(key));
- }
- return timerInternals;
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTimerInternals.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTimerInternals.java
deleted file mode 100644
index 15d7b6c..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTimerInternals.java
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.StateNamespace;
-import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.direct.Clock;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate.TimerUpdateBuilder;
-import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
-import org.apache.beam.sdk.state.TimeDomain;
-import org.joda.time.Instant;
-
-/** An implementation of {@link TimerInternals} where all relevant data exists in memory. */
-class DirectTimerInternals implements TimerInternals {
- private final Clock processingTimeClock;
- private final TransformWatermarks watermarks;
- private final TimerUpdateBuilder timerUpdateBuilder;
-
- public static DirectTimerInternals create(
- Clock clock, TransformWatermarks watermarks, TimerUpdateBuilder timerUpdateBuilder) {
- return new DirectTimerInternals(clock, watermarks, timerUpdateBuilder);
- }
-
- private DirectTimerInternals(
- Clock clock, TransformWatermarks watermarks, TimerUpdateBuilder timerUpdateBuilder) {
- this.processingTimeClock = clock;
- this.watermarks = watermarks;
- this.timerUpdateBuilder = timerUpdateBuilder;
- }
-
- @Override
- public void setTimer(
- StateNamespace namespace, String timerId, Instant target, TimeDomain timeDomain) {
- timerUpdateBuilder.setTimer(TimerData.of(timerId, namespace, target, timeDomain));
- }
-
- /** @deprecated use {@link #setTimer(StateNamespace, String, Instant, TimeDomain)}. */
- @Deprecated
- @Override
- public void setTimer(TimerData timerData) {
- timerUpdateBuilder.setTimer(timerData);
- }
-
- @Override
- public void deleteTimer(StateNamespace namespace, String timerId, TimeDomain timeDomain) {
- throw new UnsupportedOperationException("Canceling of timer by ID is not yet supported.");
- }
-
- /** @deprecated use {@link #deleteTimer(StateNamespace, String, TimeDomain)}. */
- @Deprecated
- @Override
- public void deleteTimer(StateNamespace namespace, String timerId) {
- throw new UnsupportedOperationException("Canceling of timer by ID is not yet supported.");
- }
-
- /** @deprecated use {@link #deleteTimer(StateNamespace, String, TimeDomain)}. */
- @Deprecated
- @Override
- public void deleteTimer(TimerData timerKey) {
- timerUpdateBuilder.deletedTimer(timerKey);
- }
-
- public TimerUpdate getTimerUpdate() {
- return timerUpdateBuilder.build();
- }
-
- @Override
- public Instant currentProcessingTime() {
- return processingTimeClock.now();
- }
-
- @Override
- @Nullable
- public Instant currentSynchronizedProcessingTime() {
- return watermarks.getSynchronizedProcessingInputTime();
- }
-
- @Override
- public Instant currentInputWatermarkTime() {
- return watermarks.getInputWatermark();
- }
-
- @Override
- @Nullable
- public Instant currentOutputWatermarkTime() {
- return watermarks.getOutputWatermark();
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTransformExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTransformExecutor.java
deleted file mode 100644
index b4e742a..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTransformExecutor.java
+++ /dev/null
@@ -1,151 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.io.Closeable;
-import java.util.concurrent.Callable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
-import org.apache.beam.sdk.metrics.MetricsEnvironment;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * A {@link Callable} responsible for constructing a {@link TransformEvaluator} from a {@link
- * TransformEvaluatorFactory} and evaluating it on some bundle of input, and registering the result
- * using a registered {@link CompletionCallback}.
- */
-class DirectTransformExecutor<T> implements TransformExecutor {
- private static final Logger LOG = LoggerFactory.getLogger(DirectTransformExecutor.class);
-
- static class Factory implements TransformExecutorFactory {
- private final EvaluationContext context;
- private final TransformEvaluatorRegistry registry;
-
- Factory(EvaluationContext context, TransformEvaluatorRegistry registry) {
- this.context = context;
- this.registry = registry;
- }
-
- @Override
- public TransformExecutor create(
- CommittedBundle<?> bundle,
- PTransformNode transform,
- CompletionCallback onComplete,
- TransformExecutorService executorService) {
- return new DirectTransformExecutor<>(
- context, registry, bundle, transform, onComplete, executorService);
- }
- }
-
- private final TransformEvaluatorRegistry evaluatorRegistry;
-
- /** The transform that will be evaluated. */
- private final PTransformNode transform;
- /** The inputs this {@link DirectTransformExecutor} will deliver to the transform. */
- private final CommittedBundle<T> inputBundle;
-
- private final CompletionCallback onComplete;
- private final TransformExecutorService transformEvaluationState;
- private final EvaluationContext context;
-
- @VisibleForTesting
- DirectTransformExecutor(
- EvaluationContext context,
- TransformEvaluatorRegistry factory,
- CommittedBundle<T> inputBundle,
- PTransformNode transform,
- CompletionCallback completionCallback,
- TransformExecutorService transformEvaluationState) {
- this.evaluatorRegistry = factory;
-
- this.inputBundle = inputBundle;
- this.transform = transform;
-
- this.onComplete = completionCallback;
-
- this.transformEvaluationState = transformEvaluationState;
- this.context = context;
- }
-
- @Override
- public void run() {
- MetricsContainerImpl metricsContainer = new MetricsContainerImpl(transform.getId());
- try (Closeable metricsScope = MetricsEnvironment.scopedMetricsContainer(metricsContainer)) {
- TransformEvaluator<T> evaluator = evaluatorRegistry.forApplication(transform, inputBundle);
- if (evaluator == null) {
- onComplete.handleEmpty(transform);
- // Nothing to do
- return;
- }
-
- processElements(evaluator, metricsContainer);
-
- finishBundle(evaluator, metricsContainer);
- } catch (Exception e) {
- onComplete.handleException(inputBundle, e);
- if (e instanceof RuntimeException) {
- throw (RuntimeException) e;
- }
- throw new RuntimeException(e);
- } catch (Error err) {
- LOG.error("Error occurred within {}", this, err);
- onComplete.handleError(err);
- throw err;
- } finally {
- // Report the physical metrics from the end of this step.
- context.getMetrics().commitPhysical(inputBundle, metricsContainer.getCumulative());
-
- transformEvaluationState.complete(this);
- }
- }
-
- /** Processes all the elements in the input bundle using the transform evaluator. */
- private void processElements(
- TransformEvaluator<T> evaluator, MetricsContainerImpl metricsContainer) throws Exception {
- if (inputBundle != null) {
- for (WindowedValue<T> value : inputBundle.getElements()) {
- evaluator.processElement(value);
-
- // Report the physical metrics after each element
- MetricUpdates deltas = metricsContainer.getUpdates();
- if (deltas != null) {
- context.getMetrics().updatePhysical(inputBundle, deltas);
- metricsContainer.commitUpdates();
- }
- }
- }
- }
-
- /**
- * Finishes processing the input bundle and commit the result using the {@link
- * CompletionCallback}.
- *
- * @return the {@link TransformResult} produced by {@link TransformEvaluator#finishBundle()}
- */
- private TransformResult<T> finishBundle(
- TransformEvaluator<T> evaluator, MetricsContainerImpl metricsContainer) throws Exception {
- TransformResult<T> result =
- evaluator.finishBundle().withLogicalMetricUpdates(metricsContainer.getCumulative());
- onComplete.handleResult(inputBundle, result);
- return result;
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DisplayDataValidator.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DisplayDataValidator.java
deleted file mode 100644
index c01dd62..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DisplayDataValidator.java
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.runners.TransformHierarchy;
-import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.sdk.transforms.display.HasDisplayData;
-
-/**
- * Validate correct implementation of {@link DisplayData} by evaluating {@link
- * HasDisplayData#populateDisplayData(DisplayData.Builder)} during pipeline construction.
- */
-class DisplayDataValidator {
- // Do not instantiate
- private DisplayDataValidator() {}
-
- static void validatePipeline(Pipeline pipeline) {
- validateTransforms(pipeline);
- }
-
- static void validateOptions(PipelineOptions options) {
- evaluateDisplayData(options);
- }
-
- private static void validateTransforms(Pipeline pipeline) {
- pipeline.traverseTopologically(Visitor.INSTANCE);
- }
-
- private static void evaluateDisplayData(HasDisplayData component) {
- DisplayData.from(component);
- }
-
- private static class Visitor extends Pipeline.PipelineVisitor.Defaults {
- private static final Visitor INSTANCE = new Visitor();
-
- @Override
- public CompositeBehavior enterCompositeTransform(TransformHierarchy.Node node) {
- if (!node.isRootNode()) {
- evaluateDisplayData(node.getTransform());
- }
-
- return CompositeBehavior.ENTER_TRANSFORM;
- }
-
- @Override
- public void visitPrimitiveTransform(TransformHierarchy.Node node) {
- evaluateDisplayData(node.getTransform());
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EmptyInputProvider.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EmptyInputProvider.java
deleted file mode 100644
index 82edcef..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EmptyInputProvider.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Collection;
-import java.util.Collections;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-
-/** A {@link RootInputProvider} that provides no input bundles. */
-class EmptyInputProvider implements RootInputProvider<Void> {
- EmptyInputProvider() {}
-
- /**
- * {@inheritDoc}.
- *
- * <p>Returns an empty collection.
- */
- @Override
- public Collection<CommittedBundle<Void>> getInitialInputs(
- PTransformNode transform, int targetParallelism) {
- return Collections.emptyList();
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContext.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContext.java
deleted file mode 100644
index a1b57a9..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContext.java
+++ /dev/null
@@ -1,333 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-
-import java.util.Collection;
-import java.util.EnumSet;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import org.apache.beam.runners.core.TimerInternals.TimerData;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.Clock;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.direct.WatermarkManager;
-import org.apache.beam.runners.direct.WatermarkManager.FiredTimers;
-import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
-import org.apache.beam.runners.direct.portable.CommittedResult.OutputType;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.Trigger;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.joda.time.Instant;
-
-/**
- * The evaluation context for a specific pipeline being executed by the {@code DirectRunner}.
- * Contains state shared within the execution across all transforms.
- *
- * <p>{@link EvaluationContext} contains shared state for an execution of the {@code DirectRunner}
- * that can be used while evaluating a {@link PTransform}. This consists of views into underlying
- * state and watermark implementations, access to read and write {@link PCollectionView
- * PCollectionViews}, and managing the {@link DirectStateAndTimers ExecutionContexts}. This includes
- * executing callbacks asynchronously when state changes to the appropriate point (e.g. when a
- * {@link PCollectionView} is requested and known to be empty).
- *
- * <p>{@link EvaluationContext} also handles results by committing finalizing bundles based on the
- * current global state and updating the global state appropriately. This includes updating the
- * per-{@link StepAndKey} state, updating global watermarks, and executing any callbacks that can be
- * executed.
- */
-class EvaluationContext {
- /** The graph representing this {@link Pipeline}. */
- private final ExecutableGraph<PTransformNode, ? super PCollectionNode> graph;
-
- private final Clock clock;
-
- private final BundleFactory bundleFactory;
-
- /** The current processing time and event time watermarks and timers. */
- private final WatermarkManager<PTransformNode, ? super PCollectionNode> watermarkManager;
-
- /** Executes callbacks based on the progression of the watermark. */
- private final WatermarkCallbackExecutor callbackExecutor;
-
- /** The stateInternals of the world, by applied PTransform and key. */
- private final ConcurrentMap<StepAndKey, CopyOnAccessInMemoryStateInternals>
- applicationStateInternals;
-
- private final DirectMetrics metrics;
-
- private final Set<PCollectionNode> keyedPValues;
-
- public static EvaluationContext create(
- Clock clock,
- BundleFactory bundleFactory,
- ExecutableGraph<PTransformNode, ? super PCollectionNode> graph,
- Set<PCollectionNode> keyedPValues) {
- return new EvaluationContext(clock, bundleFactory, graph, keyedPValues);
- }
-
- private EvaluationContext(
- Clock clock,
- BundleFactory bundleFactory,
- ExecutableGraph<PTransformNode, ? super PCollectionNode> graph,
- Set<PCollectionNode> keyedPValues) {
- this.clock = clock;
- this.bundleFactory = checkNotNull(bundleFactory);
- this.graph = checkNotNull(graph);
- this.keyedPValues = keyedPValues;
-
- this.watermarkManager = WatermarkManager.create(clock, graph, PTransformNode::getId);
-
- this.applicationStateInternals = new ConcurrentHashMap<>();
- this.metrics = new DirectMetrics();
-
- this.callbackExecutor = WatermarkCallbackExecutor.create(MoreExecutors.directExecutor());
- }
-
- public void initialize(
- Map<PTransformNode, ? extends Iterable<CommittedBundle<?>>> initialInputs) {
- watermarkManager.initialize((Map) initialInputs);
- }
-
- /**
- * Handle the provided {@link TransformResult}, produced after evaluating the provided {@link
- * CommittedBundle} (potentially null, if the result of a root {@link PTransform}).
- *
- * <p>The result is the output of running the transform contained in the {@link TransformResult}
- * on the contents of the provided bundle.
- *
- * @param completedBundle the bundle that was processed to produce the result. Potentially {@code
- * null} if the transform that produced the result is a root transform
- * @param completedTimers the timers that were delivered to produce the {@code completedBundle},
- * or an empty iterable if no timers were delivered
- * @param result the result of evaluating the input bundle
- * @return the committed bundles contained within the handled {@code result}
- */
- public CommittedResult<PTransformNode> handleResult(
- CommittedBundle<?> completedBundle,
- Iterable<TimerData> completedTimers,
- TransformResult<?> result) {
- Iterable<? extends CommittedBundle<?>> committedBundles =
- commitBundles(result.getOutputBundles());
- metrics.commitLogical(completedBundle, result.getLogicalMetricUpdates());
-
- // Update watermarks and timers
- EnumSet<OutputType> outputTypes = EnumSet.copyOf(result.getOutputTypes());
- if (Iterables.isEmpty(committedBundles)) {
- outputTypes.remove(OutputType.BUNDLE);
- } else {
- outputTypes.add(OutputType.BUNDLE);
- }
- CommittedResult<PTransformNode> committedResult =
- CommittedResult.create(
- result, getUnprocessedInput(completedBundle, result), committedBundles, outputTypes);
- // Update state internals
- CopyOnAccessInMemoryStateInternals theirState = result.getState();
- if (theirState != null) {
- CopyOnAccessInMemoryStateInternals committedState = theirState.commit();
- StepAndKey stepAndKey = StepAndKey.of(result.getTransform(), completedBundle.getKey());
- if (!committedState.isEmpty()) {
- applicationStateInternals.put(stepAndKey, committedState);
- } else {
- applicationStateInternals.remove(stepAndKey);
- }
- }
- // Watermarks are updated last to ensure visibility of any global state before progress is
- // permitted
- watermarkManager.updateWatermarks(
- completedBundle,
- result.getTimerUpdate().withCompletedTimers(completedTimers),
- committedResult.getExecutable(),
- committedResult.getUnprocessedInputs().orNull(),
- committedResult.getOutputs(),
- result.getWatermarkHold());
- return committedResult;
- }
-
- /**
- * Returns an {@link Optional} containing a bundle which contains all of the unprocessed elements
- * that were not processed from the {@code completedBundle}. If all of the elements of the {@code
- * completedBundle} were processed, or if {@code completedBundle} is null, returns an absent
- * {@link Optional}.
- */
- private Optional<? extends CommittedBundle<?>> getUnprocessedInput(
- CommittedBundle<?> completedBundle, TransformResult<?> result) {
- if (completedBundle == null || Iterables.isEmpty(result.getUnprocessedElements())) {
- return Optional.absent();
- }
- CommittedBundle<?> residual =
- completedBundle.withElements((Iterable) result.getUnprocessedElements());
- return Optional.of(residual);
- }
-
- private Iterable<? extends CommittedBundle<?>> commitBundles(
- Iterable<? extends UncommittedBundle<?>> bundles) {
- ImmutableList.Builder<CommittedBundle<?>> completed = ImmutableList.builder();
- for (UncommittedBundle<?> inProgress : bundles) {
- PTransformNode producing = graph.getProducer(inProgress.getPCollection());
- TransformWatermarks watermarks = watermarkManager.getWatermarks(producing);
- CommittedBundle<?> committed =
- inProgress.commit(watermarks.getSynchronizedProcessingOutputTime());
- // Empty bundles don't impact watermarks and shouldn't trigger downstream execution, so
- // filter them out
- if (!Iterables.isEmpty(committed.getElements())) {
- completed.add(committed);
- }
- }
- return completed.build();
- }
-
- private void fireAllAvailableCallbacks() {
- for (PTransformNode transform : graph.getExecutables()) {
- fireAvailableCallbacks(transform);
- }
- }
-
- private void fireAvailableCallbacks(PTransformNode producingTransform) {
- TransformWatermarks watermarks = watermarkManager.getWatermarks(producingTransform);
- Instant outputWatermark = watermarks.getOutputWatermark();
- callbackExecutor.fireForWatermark(producingTransform, outputWatermark);
- }
-
- /** Create a {@link UncommittedBundle} for use by a source. */
- public <T> UncommittedBundle<T> createRootBundle() {
- return bundleFactory.createRootBundle();
- }
-
- /**
- * Create a {@link UncommittedBundle} whose elements belong to the specified {@link PCollection}.
- */
- public <T> UncommittedBundle<T> createBundle(PCollectionNode output) {
- return bundleFactory.createBundle(output);
- }
-
- /**
- * Create a {@link UncommittedBundle} with the specified keys at the specified step. For use by
- * {@code DirectGroupByKeyOnly} {@link PTransform PTransforms}.
- */
- public <K, T> UncommittedBundle<T> createKeyedBundle(
- StructuralKey<K> key, PCollectionNode output) {
- return bundleFactory.createKeyedBundle(key, output);
- }
-
- /** Indicate whether or not this {@link PCollection} has been determined to be keyed. */
- public <T> boolean isKeyed(PCollectionNode pValue) {
- return keyedPValues.contains(pValue);
- }
-
- /**
- * Schedule a callback to be executed after output would be produced for the given window if there
- * had been input.
- *
- * <p>Output would be produced when the watermark for a {@link PValue} passes the point at which
- * the trigger for the specified window (with the specified windowing strategy) must have fired
- * from the perspective of that {@link PValue}, as specified by the value of {@link
- * Trigger#getWatermarkThatGuaranteesFiring(BoundedWindow)} for the trigger of the {@link
- * WindowingStrategy}. When the callback has fired, either values will have been produced for a
- * key in that window, the window is empty, or all elements in the window are late. The callback
- * will be executed regardless of whether values have been produced.
- */
- public void scheduleAfterOutputWouldBeProduced(
- PCollectionNode value,
- BoundedWindow window,
- WindowingStrategy<?, ?> windowingStrategy,
- Runnable runnable) {
- PTransformNode producing = graph.getProducer(value);
- callbackExecutor.callOnWindowExpiration(producing, window, windowingStrategy, runnable);
-
- fireAvailableCallbacks(producing);
- }
-
- /** Get a {@link DirectStateAndTimers} for the provided {@link PTransformNode} and key. */
- public <K> StepStateAndTimers<K> getStateAndTimers(
- PTransformNode application, StructuralKey<K> key) {
- StepAndKey stepAndKey = StepAndKey.of(application, key);
- return new DirectStateAndTimers<>(
- key,
- applicationStateInternals.get(stepAndKey),
- clock,
- watermarkManager.getWatermarks(application));
- }
-
- /** Returns all of the steps in this {@link Pipeline}. */
- Collection<PTransformNode> getSteps() {
- return graph.getExecutables();
- }
-
- /** Returns the metrics container for this pipeline. */
- public DirectMetrics getMetrics() {
- return metrics;
- }
-
- @VisibleForTesting
- void forceRefresh() {
- watermarkManager.refreshAll();
- fireAllAvailableCallbacks();
- }
-
- /**
- * Extracts all timers that have been fired and have not already been extracted.
- *
- * <p>This is a destructive operation. Timers will only appear in the result of this method once
- * for each time they are set.
- */
- public Collection<FiredTimers<PTransformNode>> extractFiredTimers() {
- forceRefresh();
- return watermarkManager.extractFiredTimers();
- }
-
- /** Returns true if the step will not produce additional output. */
- public boolean isDone(PTransformNode transform) {
- // the PTransform is done only if watermark is at the max value
- Instant stepWatermark = watermarkManager.getWatermarks(transform).getOutputWatermark();
- return !stepWatermark.isBefore(BoundedWindow.TIMESTAMP_MAX_VALUE);
- }
-
- /** Returns true if all steps are done. */
- public boolean isDone() {
- for (PTransformNode transform : graph.getExecutables()) {
- if (!isDone(transform)) {
- return false;
- }
- }
- return true;
- }
-
- public Instant now() {
- return clock.now();
- }
-
- Clock getClock() {
- return clock;
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContextStepStateAndTimersProvider.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContextStepStateAndTimersProvider.java
deleted file mode 100644
index f6ef49c..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContextStepStateAndTimersProvider.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.local.StructuralKey;
-
-/** A {@link StepStateAndTimers.Provider} that uses an {@link EvaluationContext}. */
-class EvaluationContextStepStateAndTimersProvider implements StepStateAndTimers.Provider {
- public static StepStateAndTimers.Provider forContext(EvaluationContext context) {
- return new EvaluationContextStepStateAndTimersProvider(context);
- }
-
- private final EvaluationContext context;
-
- private EvaluationContextStepStateAndTimersProvider(EvaluationContext context) {
- this.context = context;
- }
-
- @Override
- public <K> StepStateAndTimers<K> forStepAndKey(PTransformNode transform, StructuralKey<K> key) {
- return context.getStateAndTimers(transform, key);
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceFactory.java
deleted file mode 100644
index 8b097ad..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceFactory.java
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.concurrent.ExecutorService;
-
-/**
- * A factory that creates {@link ExecutorService ExecutorServices}. {@link ExecutorService
- * ExecutorServices} created by this factory should be independent of one another (e.g., if any
- * executor is shut down the remaining executors should continue to process work).
- */
-interface ExecutorServiceFactory {
- /** Create a new {@link ExecutorService}. */
- ExecutorService create();
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceParallelExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceParallelExecutor.java
deleted file mode 100644
index 1a7a99a..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceParallelExecutor.java
+++ /dev/null
@@ -1,392 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.LinkedBlockingQueue;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.stream.Collectors;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.local.ExecutionDriver;
-import org.apache.beam.runners.local.ExecutionDriver.DriverState;
-import org.apache.beam.runners.local.PipelineMessageReceiver;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.util.UserCodeException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalListener;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * An {@link PipelineExecutor} that uses an underlying {@link ExecutorService} and {@link
- * EvaluationContext} to execute a {@link Pipeline}.
- */
-final class ExecutorServiceParallelExecutor
- implements PipelineExecutor,
- BundleProcessor<PCollectionNode, CommittedBundle<?>, PTransformNode> {
- private static final Logger LOG = LoggerFactory.getLogger(ExecutorServiceParallelExecutor.class);
-
- private final int targetParallelism;
- private final ExecutorService executorService;
-
- private final RootProviderRegistry rootRegistry;
- private final TransformEvaluatorRegistry transformRegistry;
-
- private final ExecutableGraph<PTransformNode, PCollectionNode> graph;
- private final EvaluationContext evaluationContext;
-
- private final TransformExecutorFactory executorFactory;
- private final TransformExecutorService parallelExecutorService;
- private final LoadingCache<StepAndKey, TransformExecutorService> serialExecutorServices;
-
- private final QueueMessageReceiver visibleUpdates;
-
- private AtomicReference<State> pipelineState = new AtomicReference<>(State.RUNNING);
-
- public static ExecutorServiceParallelExecutor create(
- int targetParallelism,
- RootProviderRegistry rootRegistry,
- TransformEvaluatorRegistry transformRegistry,
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- EvaluationContext context) {
- return new ExecutorServiceParallelExecutor(
- targetParallelism, rootRegistry, transformRegistry, graph, context);
- }
-
- private ExecutorServiceParallelExecutor(
- int targetParallelism,
- RootProviderRegistry rootRegistry,
- TransformEvaluatorRegistry transformRegistry,
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- EvaluationContext context) {
- this.targetParallelism = targetParallelism;
- // Don't use Daemon threads for workers. The Pipeline should continue to execute even if there
- // are no other active threads (for example, because waitUntilFinish was not called)
- this.executorService =
- Executors.newFixedThreadPool(
- targetParallelism,
- new ThreadFactoryBuilder()
- .setThreadFactory(MoreExecutors.platformThreadFactory())
- .setNameFormat("direct-runner-worker")
- .build());
- this.rootRegistry = rootRegistry;
- this.transformRegistry = transformRegistry;
- this.graph = graph;
- this.evaluationContext = context;
-
- // Weak Values allows TransformExecutorServices that are no longer in use to be reclaimed.
- // Executing TransformExecutorServices have a strong reference to their TransformExecutorService
- // which stops the TransformExecutorServices from being prematurely garbage collected
- serialExecutorServices =
- CacheBuilder.newBuilder()
- .weakValues()
- .removalListener(shutdownExecutorServiceListener())
- .build(serialTransformExecutorServiceCacheLoader());
-
- this.visibleUpdates = new QueueMessageReceiver();
-
- parallelExecutorService = TransformExecutorServices.parallel(executorService);
- executorFactory = new DirectTransformExecutor.Factory(context, transformRegistry);
- }
-
- private CacheLoader<StepAndKey, TransformExecutorService>
- serialTransformExecutorServiceCacheLoader() {
- return new CacheLoader<StepAndKey, TransformExecutorService>() {
- @Override
- public TransformExecutorService load(StepAndKey stepAndKey) throws Exception {
- return TransformExecutorServices.serial(executorService);
- }
- };
- }
-
- private RemovalListener<StepAndKey, TransformExecutorService> shutdownExecutorServiceListener() {
- return notification -> {
- TransformExecutorService service = notification.getValue();
- if (service != null) {
- service.shutdown();
- }
- };
- }
-
- @Override
- // TODO: [BEAM-4563] Pass Future back to consumer to check for async errors
- @SuppressWarnings("FutureReturnValueIgnored")
- public void start() {
- int numTargetSplits = Math.max(3, targetParallelism);
- ImmutableMap.Builder<PTransformNode, ConcurrentLinkedQueue<CommittedBundle<?>>>
- pendingRootBundles = ImmutableMap.builder();
- for (PTransformNode root : graph.getRootTransforms()) {
- ConcurrentLinkedQueue<CommittedBundle<?>> pending = new ConcurrentLinkedQueue<>();
- try {
- Collection<CommittedBundle<?>> initialInputs =
- rootRegistry.getInitialInputs(root, numTargetSplits);
- pending.addAll(initialInputs);
- } catch (Exception e) {
- throw UserCodeException.wrap(e);
- }
- pendingRootBundles.put(root, pending);
- }
- evaluationContext.initialize(pendingRootBundles.build());
- final ExecutionDriver executionDriver =
- QuiescenceDriver.create(
- evaluationContext, graph, this, visibleUpdates, pendingRootBundles.build());
- executorService.submit(
- new Runnable() {
- @Override
- public void run() {
- DriverState drive = executionDriver.drive();
- if (drive.isTermainal()) {
- State newPipelineState = State.UNKNOWN;
- switch (drive) {
- case FAILED:
- newPipelineState = State.FAILED;
- break;
- case SHUTDOWN:
- newPipelineState = State.DONE;
- break;
- case CONTINUE:
- throw new IllegalStateException(
- String.format("%s should not be a terminal state", DriverState.CONTINUE));
- default:
- throw new IllegalArgumentException(
- String.format("Unknown %s %s", DriverState.class.getSimpleName(), drive));
- }
- shutdownIfNecessary(newPipelineState);
- } else {
- executorService.submit(this);
- }
- }
- });
- }
-
- @SuppressWarnings("unchecked")
- @Override
- public void process(
- CommittedBundle<?> bundle, PTransformNode consumer, CompletionCallback onComplete) {
- evaluateBundle(consumer, bundle, onComplete);
- }
-
- private <T> void evaluateBundle(
- final PTransformNode transform,
- final CommittedBundle<T> bundle,
- final CompletionCallback onComplete) {
- TransformExecutorService transformExecutor;
-
- if (isKeyed(bundle.getPCollection())) {
- final StepAndKey stepAndKey = StepAndKey.of(transform, bundle.getKey());
- // This executor will remain reachable until it has executed all scheduled transforms.
- // The TransformExecutors keep a strong reference to the Executor, the ExecutorService keeps
- // a reference to the scheduled DirectTransformExecutor callable. Follow-up TransformExecutors
- // (scheduled due to the completion of another DirectTransformExecutor) are provided to the
- // ExecutorService before the Earlier DirectTransformExecutor callable completes.
- transformExecutor = serialExecutorServices.getUnchecked(stepAndKey);
- } else {
- transformExecutor = parallelExecutorService;
- }
-
- TransformExecutor callable =
- executorFactory.create(bundle, transform, onComplete, transformExecutor);
- if (!pipelineState.get().isTerminal()) {
- transformExecutor.schedule(callable);
- }
- }
-
- private boolean isKeyed(PCollectionNode pvalue) {
- return evaluationContext.isKeyed(pvalue);
- }
-
- @Override
- public State waitUntilFinish(Duration duration) throws Exception {
- Instant completionTime;
- if (duration.equals(Duration.ZERO)) {
- completionTime = new Instant(Long.MAX_VALUE);
- } else {
- completionTime = Instant.now().plus(duration);
- }
-
- VisibleExecutorUpdate update = null;
- while (Instant.now().isBefore(completionTime)
- && (update == null || isTerminalStateUpdate(update))) {
- // Get an update; don't block forever if another thread has handled it. The call to poll will
- // wait the entire timeout; this call primarily exists to relinquish any core.
- update = visibleUpdates.tryNext(Duration.millis(25L));
- if (update == null && pipelineState.get().isTerminal()) {
- // there are no updates to process and no updates will ever be published because the
- // executor is shutdown
- return pipelineState.get();
- } else if (update != null && update.thrown.isPresent()) {
- Throwable thrown = update.thrown.get();
- if (thrown instanceof Exception) {
- throw (Exception) thrown;
- } else if (thrown instanceof Error) {
- throw (Error) thrown;
- } else {
- throw new Exception("Unknown Type of Throwable", thrown);
- }
- }
- }
- return pipelineState.get();
- }
-
- @Override
- public State getPipelineState() {
- return pipelineState.get();
- }
-
- private boolean isTerminalStateUpdate(VisibleExecutorUpdate update) {
- return !(update.getNewState() == null && update.getNewState().isTerminal());
- }
-
- @Override
- public void stop() {
- shutdownIfNecessary(State.CANCELLED);
- visibleUpdates.cancelled();
- }
-
- private void shutdownIfNecessary(State newState) {
- if (!newState.isTerminal()) {
- return;
- }
- LOG.debug("Pipeline has terminated. Shutting down.");
-
- final Collection<Exception> errors = new ArrayList<>();
- // Stop accepting new work before shutting down the executor. This ensures that thread don't try
- // to add work to the shutdown executor.
- try {
- serialExecutorServices.invalidateAll();
- } catch (final RuntimeException re) {
- errors.add(re);
- }
- try {
- serialExecutorServices.cleanUp();
- } catch (final RuntimeException re) {
- errors.add(re);
- }
- try {
- parallelExecutorService.shutdown();
- } catch (final RuntimeException re) {
- errors.add(re);
- }
- try {
- executorService.shutdown();
- } catch (final RuntimeException re) {
- errors.add(re);
- }
- try {
- transformRegistry.cleanup();
- } catch (final Exception e) {
- errors.add(e);
- }
- pipelineState.compareAndSet(State.RUNNING, newState); // ensure we hit a terminal node
- if (!errors.isEmpty()) {
- final IllegalStateException exception =
- new IllegalStateException(
- "Error"
- + (errors.size() == 1 ? "" : "s")
- + " during executor shutdown:\n"
- + errors.stream()
- .map(Exception::getMessage)
- .collect(Collectors.joining("\n- ", "- ", "")));
- visibleUpdates.failed(exception);
- throw exception;
- }
- }
-
- /**
- * An update of interest to the user. Used in {@link #waitUntilFinish} to decide whether to return
- * normally or throw an exception.
- */
- private static class VisibleExecutorUpdate {
- private final Optional<? extends Throwable> thrown;
- @Nullable private final State newState;
-
- public static VisibleExecutorUpdate fromException(Exception e) {
- return new VisibleExecutorUpdate(null, e);
- }
-
- public static VisibleExecutorUpdate fromError(Error err) {
- return new VisibleExecutorUpdate(State.FAILED, err);
- }
-
- public static VisibleExecutorUpdate finished() {
- return new VisibleExecutorUpdate(State.DONE, null);
- }
-
- public static VisibleExecutorUpdate cancelled() {
- return new VisibleExecutorUpdate(State.CANCELLED, null);
- }
-
- private VisibleExecutorUpdate(State newState, @Nullable Throwable exception) {
- this.thrown = Optional.fromNullable(exception);
- this.newState = newState;
- }
-
- State getNewState() {
- return newState;
- }
- }
-
- private static class QueueMessageReceiver implements PipelineMessageReceiver {
- // If the type of BlockingQueue changes, ensure the findbugs filter is updated appropriately
- private final BlockingQueue<VisibleExecutorUpdate> updates = new LinkedBlockingQueue<>();
-
- @Override
- public void failed(Exception e) {
- updates.offer(VisibleExecutorUpdate.fromException(e));
- }
-
- @Override
- public void failed(Error e) {
- updates.offer(VisibleExecutorUpdate.fromError(e));
- }
-
- @Override
- public void cancelled() {
- updates.offer(VisibleExecutorUpdate.cancelled());
- }
-
- @Override
- public void completed() {
- updates.offer(VisibleExecutorUpdate.finished());
- }
-
- /** Try to get the next unconsumed message in this {@link QueueMessageReceiver}. */
- @Nullable
- private VisibleExecutorUpdate tryNext(Duration timeout) throws InterruptedException {
- return updates.poll(timeout.getMillis(), TimeUnit.MILLISECONDS);
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactory.java
deleted file mode 100644
index 31654e0..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactory.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.util.WindowedValue;
-
-/**
- * The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@link Flatten} {@link
- * PTransform}.
- */
-class FlattenEvaluatorFactory implements TransformEvaluatorFactory {
- private final BundleFactory bundleFactory;
- private final ExecutableGraph<PTransformNode, PCollectionNode> graph;
-
- FlattenEvaluatorFactory(
- ExecutableGraph<PTransformNode, PCollectionNode> graph, BundleFactory bundleFactory) {
- this.bundleFactory = bundleFactory;
- this.graph = graph;
- }
-
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) {
- @SuppressWarnings({"cast", "unchecked", "rawtypes"})
- TransformEvaluator<InputT> evaluator = createInMemoryEvaluator(application);
- return evaluator;
- }
-
- @Override
- public void cleanup() {}
-
- private <InputT> TransformEvaluator<InputT> createInMemoryEvaluator(
- final PTransformNode transform) {
- return new FlattenEvaluator<>(transform);
- }
-
- private class FlattenEvaluator<InputT> implements TransformEvaluator<InputT> {
- private final PTransformNode transform;
- private final UncommittedBundle<InputT> bundle;
-
- FlattenEvaluator(PTransformNode transform) {
- this.transform = transform;
- PCollectionNode output = getOnlyElement(graph.getProduced(transform));
- bundle = bundleFactory.createBundle(output);
- }
-
- @Override
- public void processElement(WindowedValue<InputT> element) {
- bundle.add(element);
- }
-
- @Override
- public TransformResult<InputT> finishBundle() {
- return StepTransformResult.<InputT>withoutHold(transform).addOutput(bundle).build();
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupAlsoByWindowEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupAlsoByWindowEvaluatorFactory.java
deleted file mode 100644
index 1c73908..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupAlsoByWindowEvaluatorFactory.java
+++ /dev/null
@@ -1,282 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.stream.Collectors;
-import java.util.stream.StreamSupport;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.runners.core.GroupAlsoByWindowsAggregators;
-import org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly;
-import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.OutputWindowedValue;
-import org.apache.beam.runners.core.ReduceFnRunner;
-import org.apache.beam.runners.core.SystemReduceFn;
-import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.RehydratedComponents;
-import org.apache.beam.runners.core.construction.TriggerTranslation;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
-import org.apache.beam.runners.core.triggers.TriggerStateMachines;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.direct.portable.DirectGroupByKey.DirectGroupAlsoByWindow;
-import org.apache.beam.runners.fnexecution.wire.WireCoders;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.IterableLikeCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.metrics.Counter;
-import org.apache.beam.sdk.metrics.Metrics;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowTracing;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.joda.time.Instant;
-
-/**
- * The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@code
- * DirectGroupAlsoByWindow} {@link PTransform}.
- */
-class GroupAlsoByWindowEvaluatorFactory implements TransformEvaluatorFactory {
- private final BundleFactory bundleFactory;
- private final ExecutableGraph<PTransformNode, PCollectionNode> graph;
- private final Components components;
- private final StepStateAndTimers.Provider stp;
-
- GroupAlsoByWindowEvaluatorFactory(
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- Components components,
- BundleFactory bundleFactory,
- StepStateAndTimers.Provider stp) {
- this.bundleFactory = bundleFactory;
- this.graph = graph;
- this.components = components;
- this.stp = stp;
- }
-
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) {
- @SuppressWarnings({"unchecked", "rawtypes"})
- TransformEvaluator<InputT> evaluator =
- createEvaluator(application, (CommittedBundle) inputBundle);
- return evaluator;
- }
-
- @Override
- public void cleanup() {}
-
- private <K, V> TransformEvaluator<KeyedWorkItem<K, V>> createEvaluator(
- PTransformNode application, CommittedBundle<KeyedWorkItem<K, V>> inputBundle) {
- @SuppressWarnings("unchecked")
- StructuralKey<K> key = (StructuralKey<K>) inputBundle.getKey();
- return new GroupAlsoByWindowEvaluator<>(
- bundleFactory, key, application, graph, components, stp.forStepAndKey(application, key));
- }
-
- /**
- * A transform evaluator for the pseudo-primitive {@code DirectGroupAlsoByWindow}. The window of
- * the input {@link KeyedWorkItem} is ignored; it should be in the global window, as element
- * windows are reified in the {@link KeyedWorkItem#elementsIterable()}.
- *
- * @see GroupByKeyViaGroupByKeyOnly
- */
- private static class GroupAlsoByWindowEvaluator<K, V>
- implements TransformEvaluator<KeyedWorkItem<K, V>> {
- private final BundleFactory bundleFactory;
-
- private final PTransformNode application;
- private final PCollectionNode outputCollection;
-
- private final StructuralKey<?> key;
-
- private final CopyOnAccessInMemoryStateInternals<K> stateInternals;
- private final DirectTimerInternals timerInternals;
- private final WindowingStrategy<?, BoundedWindow> windowingStrategy;
-
- private final Collection<UncommittedBundle<?>> outputBundles;
-
- private final SystemReduceFn<K, V, Iterable<V>, Iterable<V>, BoundedWindow> reduceFn;
- private final Counter droppedDueToLateness;
-
- private GroupAlsoByWindowEvaluator(
- BundleFactory bundleFactory,
- StructuralKey<K> key,
- PTransformNode application,
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- Components components,
- StepStateAndTimers<K> stp) {
- this.bundleFactory = bundleFactory;
- this.application = application;
- this.outputCollection = getOnlyElement(graph.getProduced(application));
- this.key = key;
-
- this.stateInternals = stp.stateInternals();
- this.timerInternals = stp.timerInternals();
-
- PCollectionNode inputCollection = getOnlyElement(graph.getPerElementInputs(application));
- try {
- windowingStrategy =
- (WindowingStrategy<?, BoundedWindow>)
- RehydratedComponents.forComponents(components)
- .getWindowingStrategy(
- inputCollection.getPCollection().getWindowingStrategyId());
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- outputBundles = new ArrayList<>();
-
- Coder<V> valueCoder;
- try {
- Coder<WindowedValue<KV<K, Iterable<V>>>> windowedValueCoder =
- WireCoders.instantiateRunnerWireCoder(outputCollection, components);
- checkArgument(windowedValueCoder instanceof WindowedValue.WindowedValueCoder);
- Coder<KV<K, Iterable<V>>> outputKvCoder =
- ((WindowedValueCoder<KV<K, Iterable<V>>>) windowedValueCoder).getValueCoder();
- checkArgument(outputKvCoder instanceof KvCoder);
- Coder<Iterable<V>> iterVCoder = ((KvCoder<K, Iterable<V>>) outputKvCoder).getValueCoder();
- checkArgument(iterVCoder instanceof IterableLikeCoder);
- valueCoder = ((IterableLikeCoder<V, ?>) iterVCoder).getElemCoder();
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
-
- reduceFn = SystemReduceFn.buffering(valueCoder);
- droppedDueToLateness =
- Metrics.counter(
- GroupAlsoByWindowEvaluator.class,
- GroupAlsoByWindowsAggregators.DROPPED_DUE_TO_LATENESS_COUNTER);
- }
-
- @Override
- public void processElement(WindowedValue<KeyedWorkItem<K, V>> element) throws Exception {
- KeyedWorkItem<K, V> workItem = element.getValue();
-
- UncommittedBundle<KV<K, Iterable<V>>> bundle =
- bundleFactory.createKeyedBundle(this.key, outputCollection);
- outputBundles.add(bundle);
- RunnerApi.Trigger runnerApiTrigger =
- TriggerTranslation.toProto(windowingStrategy.getTrigger());
- ReduceFnRunner<K, V, Iterable<V>, BoundedWindow> reduceFnRunner =
- new ReduceFnRunner<>(
- workItem.key(),
- windowingStrategy,
- ExecutableTriggerStateMachine.create(
- TriggerStateMachines.stateMachineForTrigger(runnerApiTrigger)),
- stateInternals,
- timerInternals,
- new OutputWindowedValueToBundle<>(bundle),
- null,
- reduceFn,
- null);
-
- // Drop any elements within expired windows
- reduceFnRunner.processElements(
- dropExpiredWindows(workItem.key(), workItem.elementsIterable(), timerInternals));
- reduceFnRunner.onTimers(workItem.timersIterable());
- reduceFnRunner.persist();
- }
-
- @Override
- public TransformResult<KeyedWorkItem<K, V>> finishBundle() throws Exception {
- // State is initialized within the constructor. It can never be null.
- CopyOnAccessInMemoryStateInternals<?> state = stateInternals.commit();
- return StepTransformResult.<KeyedWorkItem<K, V>>withHold(
- application, state.getEarliestWatermarkHold())
- .withState(state)
- .addOutput(outputBundles)
- .withTimerUpdate(timerInternals.getTimerUpdate())
- .build();
- }
-
- /**
- * Returns an {@code Iterable<WindowedValue<InputT>>} that only contains non-late input
- * elements.
- */
- Iterable<WindowedValue<V>> dropExpiredWindows(
- final K key, Iterable<WindowedValue<V>> elements, final TimerInternals timerInternals) {
- return StreamSupport.stream(elements.spliterator(), false)
- .flatMap(wv -> StreamSupport.stream(wv.explodeWindows().spliterator(), false))
- .filter(
- input -> {
- BoundedWindow window = getOnlyElement(input.getWindows());
- boolean expired =
- window
- .maxTimestamp()
- .plus(windowingStrategy.getAllowedLateness())
- .isBefore(timerInternals.currentInputWatermarkTime());
- if (expired) {
- // The element is too late for this window.
- droppedDueToLateness.inc();
- WindowTracing.debug(
- "{}: Dropping element at {} for key: {}; "
- + "window: {} since it is too far behind inputWatermark: {}",
- DirectGroupAlsoByWindow.class.getSimpleName(),
- input.getTimestamp(),
- key,
- window,
- timerInternals.currentInputWatermarkTime());
- }
- // Keep the element if the window is not expired.
- return !expired;
- })
- .collect(Collectors.toList());
- }
- }
-
- private static class OutputWindowedValueToBundle<K, V>
- implements OutputWindowedValue<KV<K, Iterable<V>>> {
- private final UncommittedBundle<KV<K, Iterable<V>>> bundle;
-
- private OutputWindowedValueToBundle(UncommittedBundle<KV<K, Iterable<V>>> bundle) {
- this.bundle = bundle;
- }
-
- @Override
- public void outputWindowedValue(
- KV<K, Iterable<V>> output,
- Instant timestamp,
- Collection<? extends BoundedWindow> windows,
- PaneInfo pane) {
- bundle.add(WindowedValue.of(output, timestamp, windows, pane));
- }
-
- @Override
- public <AdditionalOutputT> void outputWindowedValue(
- TupleTag<AdditionalOutputT> tag,
- AdditionalOutputT output,
- Instant timestamp,
- Collection<? extends BoundedWindow> windows,
- PaneInfo pane) {
- throw new UnsupportedOperationException(
- String.format("%s should not use tagged outputs", "DirectGroupAlsoByWindow"));
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactory.java
deleted file mode 100644
index 46fee32..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactory.java
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components.Builder;
-import org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly;
-import org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly;
-import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.KeyedWorkItems;
-import org.apache.beam.runners.core.construction.RehydratedComponents;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.fnexecution.wire.WireCoders;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.sdk.values.KV;
-
-/**
- * The {@code DirectRunner} {@link TransformEvaluatorFactory} for the {@link GroupByKeyOnly} {@link
- * PTransform}.
- */
-class GroupByKeyOnlyEvaluatorFactory implements TransformEvaluatorFactory {
- private final Components components;
-
- private final BundleFactory bundleFactory;
- private final ExecutableGraph<PTransformNode, PCollectionNode> graph;
-
- GroupByKeyOnlyEvaluatorFactory(
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- Components components,
- BundleFactory bundleFactory) {
- this.components = components;
- this.bundleFactory = bundleFactory;
- this.graph = graph;
- }
-
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) {
- @SuppressWarnings({"cast", "unchecked", "rawtypes"})
- TransformEvaluator<InputT> evaluator = (TransformEvaluator) createEvaluator(application);
- return evaluator;
- }
-
- @Override
- public void cleanup() {}
-
- private <K, V> TransformEvaluator<KV<K, V>> createEvaluator(final PTransformNode application) {
- return new GroupByKeyOnlyEvaluator<>(application);
- }
-
- /**
- * A transform evaluator for the pseudo-primitive {@link GroupByKeyOnly}. Windowing is ignored;
- * all input should be in the global window since all output will be as well.
- *
- * @see GroupByKeyViaGroupByKeyOnly
- */
- private class GroupByKeyOnlyEvaluator<K, V> implements TransformEvaluator<KV<K, V>> {
- private final Coder<K> keyCoder;
- private final Map<StructuralKey<K>, List<WindowedValue<V>>> groupingMap;
-
- private final PCollectionNode outputPCollection;
- private final StepTransformResult.Builder<KV<K, V>> resultBuilder;
-
- private GroupByKeyOnlyEvaluator(PTransformNode application) {
- keyCoder = getKeyCoder(application);
- groupingMap = new HashMap<>();
- outputPCollection = getOnlyElement(graph.getProduced(application));
- resultBuilder = StepTransformResult.withoutHold(application);
- }
-
- private Coder<K> getKeyCoder(PTransformNode application) {
- PCollectionNode inputPCollection = getOnlyElement(graph.getPerElementInputs(application));
- try {
- // We know the type restrictions on the input PCollection, and the restrictions on the
- // Wire coder
- Builder builder = GroupByKeyOnlyEvaluatorFactory.this.components.toBuilder();
- String wireCoderId = WireCoders.addRunnerWireCoder(inputPCollection, builder);
- Coder<WindowedValue<KV<K, V>>> wireCoder =
- (Coder<WindowedValue<KV<K, V>>>)
- RehydratedComponents.forComponents(builder.build()).getCoder(wireCoderId);
-
- checkArgument(
- wireCoder instanceof WindowedValue.WindowedValueCoder,
- "Wire %s must be a %s",
- Coder.class.getSimpleName(),
- WindowedValueCoder.class.getSimpleName());
- WindowedValueCoder<KV<K, V>> windowedValueCoder = (WindowedValueCoder<KV<K, V>>) wireCoder;
-
- checkArgument(
- windowedValueCoder.getValueCoder() instanceof KvCoder,
- "Input elements to %s must be encoded with a %s",
- DirectGroupByKey.DirectGroupByKeyOnly.class.getSimpleName(),
- KvCoder.class.getSimpleName());
- KvCoder<K, V> kvCoder = (KvCoder<K, V>) windowedValueCoder.getValueCoder();
-
- return kvCoder.getKeyCoder();
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- @Override
- public void processElement(WindowedValue<KV<K, V>> element) {
- KV<K, V> kv = element.getValue();
- K key = kv.getKey();
- StructuralKey<K> groupingKey = StructuralKey.of(key, keyCoder);
- List<WindowedValue<V>> values =
- groupingMap.computeIfAbsent(groupingKey, k -> new ArrayList<>());
- values.add(element.withValue(kv.getValue()));
- }
-
- @Override
- public TransformResult<KV<K, V>> finishBundle() {
- for (Map.Entry<StructuralKey<K>, List<WindowedValue<V>>> groupedEntry :
- groupingMap.entrySet()) {
- K key = groupedEntry.getKey().getKey();
- KeyedWorkItem<K, V> groupedKv =
- KeyedWorkItems.elementsWorkItem(key, groupedEntry.getValue());
- UncommittedBundle<KeyedWorkItem<K, V>> bundle =
- bundleFactory.createKeyedBundle(StructuralKey.of(key, keyCoder), outputPCollection);
- bundle.add(WindowedValue.valueInGlobalWindow(groupedKv));
- resultBuilder.addOutput(bundle);
- }
- return resultBuilder.build();
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactory.java
deleted file mode 100644
index 1531e25..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactory.java
+++ /dev/null
@@ -1,171 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import com.google.auto.value.AutoValue;
-import java.util.Iterator;
-import javax.annotation.Nonnull;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.joda.time.Instant;
-
-/** A factory that produces bundles that perform no additional validation. */
-class ImmutableListBundleFactory implements BundleFactory {
- private static final ImmutableListBundleFactory FACTORY = new ImmutableListBundleFactory();
-
- public static ImmutableListBundleFactory create() {
- return FACTORY;
- }
-
- private ImmutableListBundleFactory() {}
-
- @Override
- public <T> UncommittedBundle<T> createRootBundle() {
- return UncommittedImmutableListBundle.create(null, StructuralKey.empty());
- }
-
- @Override
- public <T> UncommittedBundle<T> createBundle(PCollectionNode output) {
- return UncommittedImmutableListBundle.create(output, StructuralKey.empty());
- }
-
- @Override
- public <K, T> UncommittedBundle<T> createKeyedBundle(
- StructuralKey<K> key, PCollectionNode output) {
- return UncommittedImmutableListBundle.create(output, key);
- }
-
- /** A {@link UncommittedBundle} that buffers elements in memory. */
- private static final class UncommittedImmutableListBundle<T> implements UncommittedBundle<T> {
- private final PCollectionNode pcollection;
- private final StructuralKey<?> key;
- private boolean committed = false;
- private ImmutableList.Builder<WindowedValue<T>> elements;
- private Instant minSoFar = BoundedWindow.TIMESTAMP_MAX_VALUE;
-
- /**
- * Create a new {@link UncommittedImmutableListBundle} for the specified {@link PCollection}.
- */
- public static <T> UncommittedImmutableListBundle<T> create(
- PCollectionNode pcollection, StructuralKey<?> key) {
- return new UncommittedImmutableListBundle<>(pcollection, key);
- }
-
- private UncommittedImmutableListBundle(PCollectionNode pcollection, StructuralKey<?> key) {
- this.pcollection = pcollection;
- this.key = key;
- this.elements = ImmutableList.builder();
- }
-
- @Override
- public PCollectionNode getPCollection() {
- return pcollection;
- }
-
- @Override
- public UncommittedImmutableListBundle<T> add(WindowedValue<T> element) {
- checkState(
- !committed,
- "Can't add element %s to committed bundle in PCollection %s",
- element,
- pcollection);
- checkArgument(
- element.getTimestamp().isBefore(BoundedWindow.TIMESTAMP_MAX_VALUE),
- "Can't add an element past the end of time (%s), got timestamp %s",
- BoundedWindow.TIMESTAMP_MAX_VALUE,
- element.getTimestamp());
- elements.add(element);
- if (element.getTimestamp().isBefore(minSoFar)) {
- minSoFar = element.getTimestamp();
- }
- return this;
- }
-
- @Override
- public CommittedBundle<T> commit(final Instant synchronizedCompletionTime) {
- checkState(!committed, "Can't commit already committed bundle %s", this);
- committed = true;
- final Iterable<WindowedValue<T>> committedElements = elements.build();
- return CommittedImmutableListBundle.create(
- pcollection, key, committedElements, minSoFar, synchronizedCompletionTime);
- }
-
- @Override
- public String toString() {
- return MoreObjects.toStringHelper(this).add("elements", elements.build()).toString();
- }
- }
-
- @AutoValue
- abstract static class CommittedImmutableListBundle<T> implements CommittedBundle<T> {
- public static <T> CommittedImmutableListBundle<T> create(
- @Nullable PCollectionNode pcollection,
- StructuralKey<?> key,
- Iterable<WindowedValue<T>> committedElements,
- Instant minElementTimestamp,
- Instant synchronizedCompletionTime) {
- return new AutoValue_ImmutableListBundleFactory_CommittedImmutableListBundle<>(
- pcollection, key, committedElements, minElementTimestamp, synchronizedCompletionTime);
- }
-
- @Override
- @Nonnull
- public Iterator<WindowedValue<T>> iterator() {
- return getElements().iterator();
- }
-
- @Override
- public CommittedBundle<T> withElements(Iterable<WindowedValue<T>> elements) {
- return create(
- getPCollection(),
- getKey(),
- ImmutableList.copyOf(elements),
- minTimestamp(elements),
- getSynchronizedProcessingOutputWatermark());
- }
-
- @Override
- public int hashCode() {
- return System.identityHashCode(this);
- }
-
- @Override
- public boolean equals(Object obj) {
- return this == obj;
- }
- }
-
- private static Instant minTimestamp(Iterable<? extends WindowedValue<?>> elements) {
- Instant minTs = BoundedWindow.TIMESTAMP_MAX_VALUE;
- for (WindowedValue<?> element : elements) {
- if (element.getTimestamp().isBefore(minTs)) {
- minTs = element.getTimestamp();
- }
- }
- return minTs;
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactory.java
deleted file mode 100644
index c2997ef..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactory.java
+++ /dev/null
@@ -1,110 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-
-import java.util.Collection;
-import java.util.Collections;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.sdk.transforms.Impulse;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-
-/** The evaluator for the {@link Impulse} transform. Produces only empty byte arrays. */
-class ImpulseEvaluatorFactory implements TransformEvaluatorFactory {
- private final BundleFactory bundleFactory;
- private final ExecutableGraph<PTransformNode, PCollectionNode> graph;
-
- ImpulseEvaluatorFactory(
- ExecutableGraph<PTransformNode, PCollectionNode> graph, BundleFactory bundleFactory) {
- this.bundleFactory = bundleFactory;
- this.graph = graph;
- }
-
- @Nullable
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) {
- return (TransformEvaluator<InputT>)
- new ImpulseEvaluator(
- bundleFactory, application, getOnlyElement(graph.getProduced(application)));
- }
-
- @Override
- public void cleanup() {
- // Impulse has no state, so do nothing.
- }
-
- private static class ImpulseEvaluator implements TransformEvaluator<ImpulseShard> {
- private final StepTransformResult.Builder<ImpulseShard> result;
-
- private final BundleFactory factory;
- private final PCollectionNode outputPCollection;
-
- private ImpulseEvaluator(
- BundleFactory factory, PTransformNode application, PCollectionNode outputPCollection) {
- this.factory = factory;
- result = StepTransformResult.withoutHold(application);
- this.outputPCollection = outputPCollection;
- }
-
- @Override
- public void processElement(WindowedValue<ImpulseShard> element) throws Exception {
- result.addOutput(
- factory
- .createBundle(outputPCollection)
- .add(WindowedValue.valueInGlobalWindow(new byte[0])));
- }
-
- @Override
- public TransformResult<ImpulseShard> finishBundle() throws Exception {
- return result.build();
- }
- }
-
- /**
- * The {@link RootInputProvider} for the {@link Impulse} {@link PTransform}. Produces a single
- * {@link ImpulseShard}.
- */
- static class ImpulseRootProvider implements RootInputProvider<ImpulseShard> {
- private final BundleFactory bundleFactory;
-
- ImpulseRootProvider(BundleFactory bundleFactory) {
- this.bundleFactory = bundleFactory;
- }
-
- @Override
- public Collection<CommittedBundle<ImpulseShard>> getInitialInputs(
- PTransformNode transform, int targetParallelism) {
- return Collections.singleton(
- bundleFactory
- .<ImpulseShard>createRootBundle()
- .add(WindowedValue.valueInGlobalWindow(new ImpulseShard()))
- .commit(BoundedWindow.TIMESTAMP_MIN_VALUE));
- }
- }
-
- @VisibleForTesting
- static class ImpulseShard {}
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWindow.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWindow.java
deleted file mode 100644
index a632daf..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWindow.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Objects;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.values.PCollectionView;
-
-/**
- * A pair of a {@link PCollectionView} and a {@link BoundedWindow}, which can be thought of as
- * window "of" the view. This is a value class for use e.g. as a compound cache key.
- *
- * @param <T> the type of the underlying PCollectionView
- */
-final class PCollectionViewWindow<T> {
-
- private final PCollectionView<T> view;
- private final BoundedWindow window;
-
- private PCollectionViewWindow(PCollectionView<T> view, BoundedWindow window) {
- this.view = view;
- this.window = window;
- }
-
- public static <T> PCollectionViewWindow<T> of(PCollectionView<T> view, BoundedWindow window) {
- return new PCollectionViewWindow<>(view, window);
- }
-
- public PCollectionView<T> getView() {
- return view;
- }
-
- public BoundedWindow getWindow() {
- return window;
- }
-
- @Override
- public boolean equals(Object otherObject) {
- if (!(otherObject instanceof PCollectionViewWindow)) {
- return false;
- }
- @SuppressWarnings("unchecked")
- PCollectionViewWindow<T> other = (PCollectionViewWindow<T>) otherObject;
- return getView().equals(other.getView()) && getWindow().equals(other.getWindow());
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(getView(), getWindow());
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWriter.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWriter.java
deleted file mode 100644
index 944caca..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWriter.java
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionView;
-
-/**
- * A {@link PCollectionViewWriter} is responsible for writing contents of a {@link PCollection} to a
- * storage mechanism that can be read from while constructing a {@link PCollectionView}.
- *
- * @param <ElemT> the type of elements the input {@link PCollection} contains.
- * @param <ViewT> the type of the PCollectionView this writer writes to.
- */
-@FunctionalInterface
-interface PCollectionViewWriter<ElemT, ViewT> {
- void add(Iterable<WindowedValue<ElemT>> values);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PassthroughTransformEvaluator.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PassthroughTransformEvaluator.java
deleted file mode 100644
index 831bb93..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PassthroughTransformEvaluator.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.util.WindowedValue;
-
-class PassthroughTransformEvaluator<InputT> implements TransformEvaluator<InputT> {
- public static <InputT> PassthroughTransformEvaluator<InputT> create(
- PTransformNode transform, UncommittedBundle<InputT> output) {
- return new PassthroughTransformEvaluator<>(transform, output);
- }
-
- private final PTransformNode transform;
- private final UncommittedBundle<InputT> output;
-
- private PassthroughTransformEvaluator(
- PTransformNode transform, UncommittedBundle<InputT> output) {
- this.transform = transform;
- this.output = output;
- }
-
- @Override
- public void processElement(WindowedValue<InputT> element) throws Exception {
- output.add(element);
- }
-
- @Override
- public TransformResult<InputT> finishBundle() throws Exception {
- return StepTransformResult.<InputT>withoutHold(transform).addOutput(output).build();
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PipelineExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PipelineExecutor.java
deleted file mode 100644
index f169f3e..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PipelineExecutor.java
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.joda.time.Duration;
-
-/**
- * An executor that schedules and executes {@link PTransformNode PTransformNodes} for both source
- * and intermediate {@link PTransform PTransforms}.
- */
-interface PipelineExecutor {
- /**
- * Starts this executor on the provided graph. The {@link RootProviderRegistry} will be used to
- * create initial inputs for the provide {@link ExecutableGraph graph}.
- */
- void start();
-
- /**
- * Blocks until the job being executed enters a terminal state. A job is completed after all root
- * {@link PTransformNode PTransformNodes} have completed, and all {@link CommittedBundle Bundles}
- * have been consumed. Jobs may also terminate abnormally.
- *
- * <p>Waits for up to the provided duration, or forever if the provided duration is less than or
- * equal to zero.
- *
- * @return The terminal state of the Pipeline.
- * @throws Exception whenever an executor thread throws anything, transfers to the waiting thread
- * and rethrows it
- */
- State waitUntilFinish(Duration duration) throws Exception;
-
- /** Gets the current state of the {@link Pipeline}. */
- State getPipelineState();
-
- /**
- * Shuts down the executor.
- *
- * <p>The executor may continue to run for a short time after this method returns.
- */
- void stop();
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PortableGraph.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PortableGraph.java
deleted file mode 100644
index 88b6e8c..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PortableGraph.java
+++ /dev/null
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Collection;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
-import org.apache.beam.runners.direct.ExecutableGraph;
-
-/** A {@link ExecutableGraph} for a Portable {@link RunnerApi.Pipeline}. */
-class PortableGraph implements ExecutableGraph<PTransformNode, PCollectionNode> {
- private final QueryablePipeline queryablePipeline;
-
- public static PortableGraph forPipeline(RunnerApi.Pipeline p) {
- return new PortableGraph(p);
- }
-
- private PortableGraph(RunnerApi.Pipeline p) {
- this.queryablePipeline =
- QueryablePipeline.forTransforms(p.getRootTransformIdsList(), p.getComponents());
- }
-
- @Override
- public Collection<PTransformNode> getRootTransforms() {
- return queryablePipeline.getRootTransforms();
- }
-
- @Override
- public Collection<PTransformNode> getExecutables() {
- return queryablePipeline.getTransforms();
- }
-
- @Override
- public PTransformNode getProducer(PCollectionNode collection) {
- return queryablePipeline.getProducer(collection);
- }
-
- @Override
- public Collection<PCollectionNode> getProduced(PTransformNode producer) {
- return queryablePipeline.getOutputPCollections(producer);
- }
-
- @Override
- public Collection<PCollectionNode> getPerElementInputs(PTransformNode transform) {
- return queryablePipeline.getPerElementInputPCollections(transform);
- }
-
- @Override
- public Collection<PTransformNode> getPerElementConsumers(PCollectionNode pCollection) {
- return queryablePipeline.getPerElementConsumers(pCollection);
- }
-
- public QueryablePipeline getQueryablePipeline() {
- return this.queryablePipeline;
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/QuiescenceDriver.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/QuiescenceDriver.java
deleted file mode 100644
index b11bc91..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/QuiescenceDriver.java
+++ /dev/null
@@ -1,328 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import com.google.auto.value.AutoValue;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.Map;
-import java.util.Queue;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.KeyedWorkItems;
-import org.apache.beam.runners.core.TimerInternals.TimerData;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.direct.WatermarkManager.FiredTimers;
-import org.apache.beam.runners.local.ExecutionDriver;
-import org.apache.beam.runners.local.PipelineMessageReceiver;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * Pushes additional work onto a {@link BundleProcessor} based on the fact that a pipeline has
- * quiesced.
- */
-class QuiescenceDriver implements ExecutionDriver {
- private static final Logger LOG = LoggerFactory.getLogger(QuiescenceDriver.class);
-
- public static ExecutionDriver create(
- EvaluationContext context,
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- BundleProcessor<PCollectionNode, CommittedBundle<?>, PTransformNode> bundleProcessor,
- PipelineMessageReceiver messageReceiver,
- Map<PTransformNode, ConcurrentLinkedQueue<CommittedBundle<?>>> initialBundles) {
- return new QuiescenceDriver(context, graph, bundleProcessor, messageReceiver, initialBundles);
- }
-
- private final EvaluationContext evaluationContext;
- private final ExecutableGraph<PTransformNode, PCollectionNode> graph;
- private final BundleProcessor<PCollectionNode, CommittedBundle<?>, PTransformNode>
- bundleProcessor;
- private final PipelineMessageReceiver pipelineMessageReceiver;
-
- private final CompletionCallback defaultCompletionCallback =
- new TimerIterableCompletionCallback(Collections.emptyList());
-
- private final Map<PTransformNode, ConcurrentLinkedQueue<CommittedBundle<?>>> pendingRootBundles;
- private final Queue<WorkUpdate> pendingWork = new ConcurrentLinkedQueue<>();
-
- private final AtomicReference<ExecutorState> state =
- new AtomicReference<>(ExecutorState.QUIESCENT);
- private final AtomicLong outstandingWork = new AtomicLong(0L);
- private boolean exceptionThrown = false;
-
- private QuiescenceDriver(
- EvaluationContext evaluationContext,
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- BundleProcessor<PCollectionNode, CommittedBundle<?>, PTransformNode> bundleProcessor,
- PipelineMessageReceiver pipelineMessageReceiver,
- Map<PTransformNode, ConcurrentLinkedQueue<CommittedBundle<?>>> pendingRootBundles) {
- this.evaluationContext = evaluationContext;
- this.graph = graph;
- this.bundleProcessor = bundleProcessor;
- this.pipelineMessageReceiver = pipelineMessageReceiver;
- this.pendingRootBundles = pendingRootBundles;
- }
-
- @Override
- public DriverState drive() {
- boolean noWorkOutstanding = outstandingWork.get() == 0L;
- ExecutorState startingState = state.get();
- if (startingState == ExecutorState.ACTIVE) {
- // The remainder of this call will add all available work to the Executor, and there will
- // be no new work available
- state.compareAndSet(ExecutorState.ACTIVE, ExecutorState.PROCESSING);
- } else if (startingState == ExecutorState.PROCESSING && noWorkOutstanding) {
- // The executor has consumed all new work and no new work was added
- state.compareAndSet(ExecutorState.PROCESSING, ExecutorState.QUIESCING);
- } else if (startingState == ExecutorState.QUIESCING && noWorkOutstanding) {
- // The executor re-ran all blocked work and nothing could make progress.
- state.compareAndSet(ExecutorState.QUIESCING, ExecutorState.QUIESCENT);
- }
- fireTimers();
- Collection<WorkUpdate> updates = new ArrayList<>();
- // Pull all available updates off of the queue before adding additional work. This ensures
- // both loops terminate.
- WorkUpdate pendingUpdate = pendingWork.poll();
- while (pendingUpdate != null) {
- updates.add(pendingUpdate);
- pendingUpdate = pendingWork.poll();
- }
- for (WorkUpdate update : updates) {
- applyUpdate(noWorkOutstanding, startingState, update);
- }
- addWorkIfNecessary();
-
- if (exceptionThrown) {
- return DriverState.FAILED;
- } else if (evaluationContext.isDone()) {
- return DriverState.SHUTDOWN;
- } else {
- return DriverState.CONTINUE;
- }
- }
-
- private void applyUpdate(
- boolean noWorkOutstanding, ExecutorState startingState, WorkUpdate update) {
- LOG.debug("Executor Update: {}", update);
- if (update.getBundle().isPresent()) {
- if (ExecutorState.ACTIVE == startingState
- || (ExecutorState.PROCESSING == startingState && noWorkOutstanding)) {
- CommittedBundle<?> bundle = update.getBundle().get();
- for (PTransformNode consumer : update.getConsumers()) {
- outstandingWork.incrementAndGet();
- bundleProcessor.process(bundle, consumer, defaultCompletionCallback);
- }
- } else {
- pendingWork.offer(update);
- }
- } else if (update.getException().isPresent()) {
- pipelineMessageReceiver.failed(update.getException().get());
- exceptionThrown = true;
- }
- }
-
- /** Fires any available timers. */
- private void fireTimers() {
- try {
- for (FiredTimers<PTransformNode> transformTimers : evaluationContext.extractFiredTimers()) {
- Collection<TimerData> delivery = transformTimers.getTimers();
- KeyedWorkItem<?, Object> work =
- KeyedWorkItems.timersWorkItem(transformTimers.getKey().getKey(), delivery);
- PCollectionNode inputPCollection =
- Iterables.getOnlyElement(graph.getPerElementInputs(transformTimers.getExecutable()));
- @SuppressWarnings({"unchecked", "rawtypes"})
- CommittedBundle<?> bundle =
- evaluationContext
- .createKeyedBundle(transformTimers.getKey(), inputPCollection)
- .add(WindowedValue.valueInGlobalWindow(work))
- .commit(evaluationContext.now());
- outstandingWork.incrementAndGet();
- bundleProcessor.process(
- bundle, transformTimers.getExecutable(), new TimerIterableCompletionCallback(delivery));
- state.set(ExecutorState.ACTIVE);
- }
- } catch (Exception e) {
- LOG.error("Internal Error while delivering timers", e);
- pipelineMessageReceiver.failed(e);
- exceptionThrown = true;
- }
- }
-
- /**
- * If all active {@link DirectTransformExecutor TransformExecutors} are in a blocked state, add
- * more work from root nodes that may have additional work. This ensures that if a pipeline has
- * elements available from the root nodes it will add those elements when necessary.
- */
- private void addWorkIfNecessary() {
- // If any timers have fired, they will add more work; We don't need to add more
- if (state.get() == ExecutorState.QUIESCENT) {
- // All current TransformExecutors are blocked; add more work from the roots.
- for (Map.Entry<PTransformNode, ConcurrentLinkedQueue<CommittedBundle<?>>> pendingRootEntry :
- pendingRootBundles.entrySet()) {
- Collection<CommittedBundle<?>> bundles = new ArrayList<>();
- // Pull all available work off of the queue, then schedule it all, so this loop
- // terminates
- while (!pendingRootEntry.getValue().isEmpty()) {
- CommittedBundle<?> bundle = pendingRootEntry.getValue().poll();
- bundles.add(bundle);
- }
- for (CommittedBundle<?> bundle : bundles) {
- outstandingWork.incrementAndGet();
- bundleProcessor.process(bundle, pendingRootEntry.getKey(), defaultCompletionCallback);
- state.set(ExecutorState.ACTIVE);
- }
- }
- }
- }
-
- /**
- * The state of the executor. The state of the executor determines the behavior of the {@link
- * QuiescenceDriver} when it runs.
- */
- private enum ExecutorState {
- /**
- * Output has been produced since the last time the monitor ran. Work exists that has not yet
- * been evaluated, and all pending, including potentially blocked work, should be evaluated.
- *
- * <p>The executor becomes active whenever a timer fires, a {@link PCollectionView} is updated,
- * or output is produced by the evaluation of a {@link DirectTransformExecutor}.
- */
- ACTIVE,
- /**
- * The Executor does not have any unevaluated work available to it, but work is in progress.
- * Work should not be added until the Executor becomes active or no work is outstanding.
- *
- * <p>If all outstanding work completes without the executor becoming {@code ACTIVE}, the
- * Executor enters state {@code QUIESCING}. Previously evaluated work must be reevaluated, in
- * case a side input has made progress.
- */
- PROCESSING,
- /**
- * All outstanding work is work that may be blocked on a side input. When there is no
- * outstanding work, the executor becomes {@code QUIESCENT}.
- */
- QUIESCING,
- /**
- * All elements are either buffered in state or are blocked on a side input. There are no timers
- * that are permitted to fire but have not. There is no outstanding work.
- *
- * <p>The pipeline will not make progress without the progression of watermarks, the progression
- * of processing time, or the addition of elements.
- */
- QUIESCENT
- }
-
- /**
- * The base implementation of {@link CompletionCallback} that provides implementations for {@link
- * #handleResult(CommittedBundle, TransformResult)} and {@link #handleException(CommittedBundle,
- * Exception)}.
- */
- private class TimerIterableCompletionCallback implements CompletionCallback {
- private final Iterable<TimerData> timers;
-
- TimerIterableCompletionCallback(Iterable<TimerData> timers) {
- this.timers = timers;
- }
-
- @Override
- public final CommittedResult handleResult(
- CommittedBundle<?> inputBundle, TransformResult<?> result) {
- CommittedResult<PTransformNode> committedResult =
- evaluationContext.handleResult(inputBundle, timers, result);
- for (CommittedBundle<?> outputBundle : committedResult.getOutputs()) {
- pendingWork.offer(
- WorkUpdate.fromBundle(
- outputBundle, graph.getPerElementConsumers(outputBundle.getPCollection())));
- }
- Optional<? extends CommittedBundle<?>> unprocessedInputs =
- committedResult.getUnprocessedInputs();
- if (unprocessedInputs.isPresent()) {
- if (inputBundle.getPCollection() == null) {
- // TODO: Split this logic out of an if statement
- pendingRootBundles.get(result.getTransform()).offer(unprocessedInputs.get());
- } else {
- pendingWork.offer(
- WorkUpdate.fromBundle(
- unprocessedInputs.get(), Collections.singleton(committedResult.getExecutable())));
- }
- }
- if (!committedResult.getProducedOutputTypes().isEmpty()) {
- state.set(ExecutorState.ACTIVE);
- }
- outstandingWork.decrementAndGet();
- return committedResult;
- }
-
- @Override
- public void handleEmpty(PTransformNode transform) {
- outstandingWork.decrementAndGet();
- }
-
- @Override
- public final void handleException(CommittedBundle<?> inputBundle, Exception e) {
- pendingWork.offer(WorkUpdate.fromException(e));
- outstandingWork.decrementAndGet();
- }
-
- @Override
- public void handleError(Error err) {
- outstandingWork.decrementAndGet();
- pipelineMessageReceiver.failed(err);
- }
- }
-
- /**
- * An internal status update on the state of the executor.
- *
- * <p>Used to signal when the executor should be shut down (due to an exception).
- */
- @AutoValue
- abstract static class WorkUpdate {
- private static WorkUpdate fromBundle(
- CommittedBundle<?> bundle, Collection<PTransformNode> consumers) {
- return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.of(bundle), consumers, Optional.absent());
- }
-
- private static WorkUpdate fromException(Exception e) {
- return new AutoValue_QuiescenceDriver_WorkUpdate(
- Optional.absent(), Collections.emptyList(), Optional.of(e));
- }
-
- /** Returns the bundle that produced this update. */
- public abstract Optional<? extends CommittedBundle<?>> getBundle();
-
- /**
- * Returns the transforms to process the bundle. If nonempty, {@link #getBundle()} will return a
- * present {@link Optional}.
- */
- public abstract Collection<PTransformNode> getConsumers();
-
- public abstract Optional<? extends Exception> getException();
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
deleted file mode 100644
index eee804c..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
+++ /dev/null
@@ -1,559 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.runners.core.construction.SyntheticComponents.uniqueId;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.stream.Collectors;
-import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
-import org.apache.beam.model.fnexecution.v1.ProvisionApi.Resources;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Coder;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.ComponentsOrBuilder;
-import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.MessageWithComponents;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform.Builder;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.model.pipeline.v1.RunnerApi.SdkFunctionSpec;
-import org.apache.beam.runners.core.construction.ModelCoders;
-import org.apache.beam.runners.core.construction.ModelCoders.KvCoderComponents;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.construction.graph.PipelineValidator;
-import org.apache.beam.runners.core.construction.graph.ProtoOverrides;
-import org.apache.beam.runners.core.construction.graph.ProtoOverrides.TransformReplacement;
-import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
-import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService;
-import org.apache.beam.runners.fnexecution.control.ControlClientPool;
-import org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
-import org.apache.beam.runners.fnexecution.control.MapControlClientPool;
-import org.apache.beam.runners.fnexecution.control.SingleEnvironmentInstanceJobBundleFactory;
-import org.apache.beam.runners.fnexecution.data.GrpcDataService;
-import org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
-import org.apache.beam.runners.fnexecution.environment.EmbeddedEnvironmentFactory;
-import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
-import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
-import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
-import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
-import org.apache.beam.runners.fnexecution.state.GrpcStateService;
-import org.apache.beam.runners.fnexecution.wire.LengthPrefixUnknownCoders;
-import org.apache.beam.sdk.fn.IdGenerator;
-import org.apache.beam.sdk.fn.IdGenerators;
-import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-
-/**
- * The "ReferenceRunner" engine implementation. The ReferenceRunner uses the portability framework
- * to execute a Pipeline on a single machine.
- */
-public class ReferenceRunner {
- private final RunnerApi.Pipeline pipeline;
- private final Struct options;
- private final String artifactRetrievalToken;
-
- private final EnvironmentType environmentType;
-
- private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
-
- /** @param environmentType The environment to use for the SDK Harness. */
- private ReferenceRunner(
- Pipeline p, Struct options, String artifactRetrievalToken, EnvironmentType environmentType) {
- this.pipeline = executable(p);
- this.options = options;
- this.environmentType = environmentType;
- this.artifactRetrievalToken = artifactRetrievalToken;
- }
-
- /**
- * Creates a "ReferenceRunner" engine for a single pipeline with a Dockerized SDK harness.
- *
- * @param p Pipeline being executed for this job.
- * @param options PipelineOptions for this job.
- * @param artifactRetrievalToken Token to retrieve artifacts that have been staged.
- */
- public static ReferenceRunner forPipeline(
- RunnerApi.Pipeline p, Struct options, String artifactRetrievalToken) {
- return new ReferenceRunner(p, options, artifactRetrievalToken, EnvironmentType.DOCKER);
- }
-
- static ReferenceRunner forInProcessPipeline(RunnerApi.Pipeline p, Struct options) {
- return new ReferenceRunner(p, options, "", EnvironmentType.IN_PROCESS);
- }
-
- private RunnerApi.Pipeline executable(RunnerApi.Pipeline original) {
- RunnerApi.Pipeline p = original;
- PipelineValidator.validate(p);
- p =
- ProtoOverrides.updateTransform(
- PTransformTranslation.SPLITTABLE_PROCESS_KEYED_URN,
- p,
- new SplittableProcessKeyedReplacer());
- p =
- ProtoOverrides.updateTransform(
- PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN, p, new PortableGroupByKeyReplacer());
- p = GreedyPipelineFuser.fuse(p).toPipeline();
-
- p = foldFeedSDFIntoExecutableStage(p);
- PipelineValidator.validate(p);
-
- return p;
- }
-
- private static Set<PCollectionNode> getKeyedPCollections(
- ExecutableGraph<PTransformNode, PCollectionNode> graph) {
- // This mimics KeyedPValueTrackingVisitor behavior in regular direct runner,
- // but without propagating keyed-ness through key-preserving DoFn's.
- // That is not yet necessary, but will be necessary once we implement state and timers.
- // See https://issues.apache.org/jira/browse/BEAM-4557.
- Set<PCollectionNode> res = Sets.newHashSet();
- Set<String> keyedProducers =
- Sets.newHashSet(DirectGroupByKey.DIRECT_GBKO_URN, DirectGroupByKey.DIRECT_GABW_URN);
- for (PTransformNode transform : graph.getExecutables()) {
- if (keyedProducers.contains(transform.getTransform().getSpec().getUrn())) {
- res.addAll(graph.getProduced(transform));
- }
- }
- return res;
- }
-
- /**
- * First starts all the services needed, then configures and starts the {@link
- * ExecutorServiceParallelExecutor}.
- */
- public void execute() throws Exception {
- ExecutableGraph<PTransformNode, PCollectionNode> graph = PortableGraph.forPipeline(pipeline);
- BundleFactory bundleFactory = ImmutableListBundleFactory.create();
- EvaluationContext ctxt =
- EvaluationContext.create(Instant::new, bundleFactory, graph, getKeyedPCollections(graph));
- RootProviderRegistry rootRegistry = RootProviderRegistry.javaPortableRegistry(bundleFactory);
- int targetParallelism = Math.max(Runtime.getRuntime().availableProcessors(), 3);
- ServerFactory serverFactory = createServerFactory();
- ControlClientPool controlClientPool = MapControlClientPool.create();
- ExecutorService dataExecutor = Executors.newCachedThreadPool();
- ProvisionInfo provisionInfo =
- ProvisionInfo.newBuilder()
- .setJobId("id")
- .setJobName("reference")
- .setPipelineOptions(options)
- .setWorkerId("foo")
- .setResourceLimits(Resources.getDefaultInstance())
- .setRetrievalToken(artifactRetrievalToken)
- .build();
- try (GrpcFnServer<GrpcLoggingService> logging =
- GrpcFnServer.allocatePortAndCreateFor(
- GrpcLoggingService.forWriter(Slf4jLogWriter.getDefault()), serverFactory);
- GrpcFnServer<ArtifactRetrievalService> artifact =
- GrpcFnServer.allocatePortAndCreateFor(
- BeamFileSystemArtifactRetrievalService.create(), serverFactory);
- GrpcFnServer<StaticGrpcProvisionService> provisioning =
- GrpcFnServer.allocatePortAndCreateFor(
- StaticGrpcProvisionService.create(provisionInfo), serverFactory);
- GrpcFnServer<FnApiControlClientPoolService> control =
- GrpcFnServer.allocatePortAndCreateFor(
- FnApiControlClientPoolService.offeringClientsToPool(
- controlClientPool.getSink(),
- GrpcContextHeaderAccessorProvider.getHeaderAccessor()),
- serverFactory);
- GrpcFnServer<GrpcDataService> data =
- GrpcFnServer.allocatePortAndCreateFor(
- GrpcDataService.create(dataExecutor, OutboundObserverFactory.serverDirect()),
- serverFactory);
- GrpcFnServer<GrpcStateService> state =
- GrpcFnServer.allocatePortAndCreateFor(GrpcStateService.create(), serverFactory)) {
-
- EnvironmentFactory environmentFactory =
- createEnvironmentFactory(control, logging, artifact, provisioning, controlClientPool);
- JobBundleFactory jobBundleFactory =
- SingleEnvironmentInstanceJobBundleFactory.create(
- environmentFactory, data, state, idGenerator);
-
- TransformEvaluatorRegistry transformRegistry =
- TransformEvaluatorRegistry.portableRegistry(
- graph,
- pipeline.getComponents(),
- bundleFactory,
- jobBundleFactory,
- EvaluationContextStepStateAndTimersProvider.forContext(ctxt));
- ExecutorServiceParallelExecutor executor =
- ExecutorServiceParallelExecutor.create(
- targetParallelism, rootRegistry, transformRegistry, graph, ctxt);
- executor.start();
- executor.waitUntilFinish(Duration.ZERO);
- } finally {
- dataExecutor.shutdown();
- }
- }
-
- private ServerFactory createServerFactory() {
- switch (environmentType) {
- case DOCKER:
- return ServerFactory.createDefault();
- case IN_PROCESS:
- return InProcessServerFactory.create();
- default:
- throw new IllegalArgumentException(
- String.format("Unknown %s %s", EnvironmentType.class.getSimpleName(), environmentType));
- }
- }
-
- private EnvironmentFactory createEnvironmentFactory(
- GrpcFnServer<FnApiControlClientPoolService> control,
- GrpcFnServer<GrpcLoggingService> logging,
- GrpcFnServer<ArtifactRetrievalService> artifact,
- GrpcFnServer<StaticGrpcProvisionService> provisioning,
- ControlClientPool controlClient) {
- switch (environmentType) {
- case DOCKER:
- return new DockerEnvironmentFactory.Provider(PipelineOptionsTranslation.fromProto(options))
- .createEnvironmentFactory(
- control, logging, artifact, provisioning, controlClient, idGenerator);
- case IN_PROCESS:
- return EmbeddedEnvironmentFactory.create(
- PipelineOptionsFactory.create(), logging, control, controlClient.getSource());
- default:
- throw new IllegalArgumentException(
- String.format("Unknown %s %s", EnvironmentType.class.getSimpleName(), environmentType));
- }
- }
-
- @VisibleForTesting
- static class PortableGroupByKeyReplacer implements TransformReplacement {
- @Override
- public MessageWithComponents getReplacement(String gbkId, ComponentsOrBuilder components) {
- PTransform gbk = components.getTransformsOrThrow(gbkId);
- checkArgument(
- PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN.equals(gbk.getSpec().getUrn()),
- "URN must be %s, got %s",
- PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN,
- gbk.getSpec().getUrn());
-
- PTransform.Builder newTransform = gbk.toBuilder();
- Components.Builder newComponents = Components.newBuilder();
- String inputId = getOnlyElement(gbk.getInputsMap().values());
-
- // Add the GBKO transform
- String kwiCollectionId =
- uniqueId(String.format("%s.%s", inputId, "kwi"), components::containsPcollections);
- {
- PCollection input = components.getPcollectionsOrThrow(inputId);
- Coder inputCoder = components.getCodersOrThrow(input.getCoderId());
- KvCoderComponents kvComponents = ModelCoders.getKvCoderComponents(inputCoder);
- String windowCoderId =
- components
- .getWindowingStrategiesOrThrow(input.getWindowingStrategyId())
- .getWindowCoderId();
- // This coder isn't actually required for the pipeline to function properly - the KWIs can
- // be passed around as pure java objects with no coding of the values, but it approximates
- // a full pipeline.
- Coder kwiCoder =
- Coder.newBuilder()
- .setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(FunctionSpec.newBuilder().setUrn("beam:direct:keyedworkitem:v1")))
- .addAllComponentCoderIds(
- ImmutableList.of(
- kvComponents.keyCoderId(), kvComponents.valueCoderId(), windowCoderId))
- .build();
- String kwiCoderId =
- uniqueId(
- String.format("kwi(%s:%s)", kvComponents.keyCoderId(), kvComponents.valueCoderId()),
- components::containsCoders);
- // The kwi PCollection has the same WindowingStrategy as the input, as no merging will
- // have been performed, so elements remain in their original windows
- PCollection kwi =
- input.toBuilder().setUniqueName(kwiCollectionId).setCoderId(kwiCoderId).build();
- String gbkoId = uniqueId(String.format("%s/GBKO", gbkId), components::containsTransforms);
- PTransform gbko =
- PTransform.newBuilder()
- .setUniqueName(String.format("%s/GBKO", gbk.getUniqueName()))
- .putAllInputs(gbk.getInputsMap())
- .setSpec(FunctionSpec.newBuilder().setUrn(DirectGroupByKey.DIRECT_GBKO_URN))
- .putOutputs("output", kwiCollectionId)
- .build();
-
- newTransform.addSubtransforms(gbkoId);
- newComponents
- .putCoders(kwiCoderId, kwiCoder)
- .putPcollections(kwiCollectionId, kwi)
- .putTransforms(gbkoId, gbko);
- }
-
- // Add the GABW transform
- {
- String gabwId = uniqueId(String.format("%s/GABW", gbkId), components::containsTransforms);
- PTransform gabw =
- PTransform.newBuilder()
- .setUniqueName(String.format("%s/GABW", gbk.getUniqueName()))
- .putInputs("input", kwiCollectionId)
- .setSpec(FunctionSpec.newBuilder().setUrn(DirectGroupByKey.DIRECT_GABW_URN))
- .putAllOutputs(gbk.getOutputsMap())
- .build();
- newTransform.addSubtransforms(gabwId);
- newComponents.putTransforms(gabwId, gabw);
- }
-
- return MessageWithComponents.newBuilder()
- .setPtransform(newTransform)
- .setComponents(newComponents)
- .build();
- }
- }
-
- /**
- * Replaces the {@link PTransformTranslation#SPLITTABLE_PROCESS_KEYED_URN} with a {@link
- * DirectGroupByKey#DIRECT_GBKO_URN} (construct keyed work items) followed by a {@link
- * SplittableRemoteStageEvaluatorFactory#FEED_SDF_URN} (convert the keyed work items to
- * element/restriction pairs that later go into {@link
- * PTransformTranslation#SPLITTABLE_PROCESS_ELEMENTS_URN}).
- */
- @VisibleForTesting
- static class SplittableProcessKeyedReplacer implements TransformReplacement {
- @Override
- public MessageWithComponents getReplacement(String spkId, ComponentsOrBuilder components) {
- PTransform spk = components.getTransformsOrThrow(spkId);
- checkArgument(
- PTransformTranslation.SPLITTABLE_PROCESS_KEYED_URN.equals(spk.getSpec().getUrn()),
- "URN must be %s, got %s",
- PTransformTranslation.SPLITTABLE_PROCESS_KEYED_URN,
- spk.getSpec().getUrn());
-
- Components.Builder newComponents = Components.newBuilder();
- newComponents.putAllCoders(components.getCodersMap());
-
- Builder newPTransform = spk.toBuilder();
-
- String inputId = getOnlyElement(spk.getInputsMap().values());
- PCollection input = components.getPcollectionsOrThrow(inputId);
-
- // This is a Coder<KV<String, KV<ElementT, RestrictionT>>>
- Coder inputCoder = components.getCodersOrThrow(input.getCoderId());
- KvCoderComponents kvComponents = ModelCoders.getKvCoderComponents(inputCoder);
- String windowCoderId =
- components
- .getWindowingStrategiesOrThrow(input.getWindowingStrategyId())
- .getWindowCoderId();
-
- // === Construct a raw GBK returning KeyedWorkItem's ===
- String kwiCollectionId =
- uniqueId(String.format("%s.kwi", spkId), components::containsPcollections);
- {
- // This coder isn't actually required for the pipeline to function properly - the KWIs can
- // be passed around as pure java objects with no coding of the values, but it approximates a
- // full pipeline.
- Coder kwiCoder =
- Coder.newBuilder()
- .setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(FunctionSpec.newBuilder().setUrn("beam:direct:keyedworkitem:v1")))
- .addAllComponentCoderIds(
- ImmutableList.of(
- kvComponents.keyCoderId(), kvComponents.valueCoderId(), windowCoderId))
- .build();
- String kwiCoderId =
- uniqueId(
- String.format(
- "keyed_work_item(%s:%s)",
- kvComponents.keyCoderId(), kvComponents.valueCoderId()),
- components::containsCoders);
-
- PCollection kwiCollection =
- input.toBuilder().setUniqueName(kwiCollectionId).setCoderId(kwiCoderId).build();
- String rawGbkId =
- uniqueId(String.format("%s/RawGBK", spkId), components::containsTransforms);
- PTransform rawGbk =
- PTransform.newBuilder()
- .setUniqueName(String.format("%s/RawGBK", spk.getUniqueName()))
- .putAllInputs(spk.getInputsMap())
- .setSpec(FunctionSpec.newBuilder().setUrn(DirectGroupByKey.DIRECT_GBKO_URN))
- .putOutputs("output", kwiCollectionId)
- .build();
-
- newComponents
- .putCoders(kwiCoderId, kwiCoder)
- .putPcollections(kwiCollectionId, kwiCollection)
- .putTransforms(rawGbkId, rawGbk);
- newPTransform.addSubtransforms(rawGbkId);
- }
-
- // === Construct a "Feed SDF" operation that converts KWI to KV<ElementT, RestrictionT> ===
- String feedSDFCollectionId =
- uniqueId(String.format("%s.feed", spkId), components::containsPcollections);
- {
- String elementRestrictionCoderId = kvComponents.valueCoderId();
- String feedSDFCoderId =
- LengthPrefixUnknownCoders.addLengthPrefixedCoder(
- elementRestrictionCoderId, newComponents, false);
-
- PCollection feedSDFCollection =
- input.toBuilder().setUniqueName(feedSDFCollectionId).setCoderId(feedSDFCoderId).build();
- String feedSDFId =
- uniqueId(String.format("%s/FeedSDF", spkId), components::containsTransforms);
- PTransform feedSDF =
- PTransform.newBuilder()
- .setUniqueName(String.format("%s/FeedSDF", spk.getUniqueName()))
- .putInputs("input", kwiCollectionId)
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(SplittableRemoteStageEvaluatorFactory.FEED_SDF_URN))
- .putOutputs("output", feedSDFCollectionId)
- .build();
-
- newComponents
- .putPcollections(feedSDFCollectionId, feedSDFCollection)
- .putTransforms(feedSDFId, feedSDF);
- newPTransform.addSubtransforms(feedSDFId);
- }
-
- // === Construct the SPLITTABLE_PROCESS_ELEMENTS operation
- {
- String runSDFId =
- uniqueId(String.format("%s/RunSDF", spkId), components::containsTransforms);
- PTransform runSDF =
- PTransform.newBuilder()
- .setUniqueName(String.format("%s/RunSDF", spk.getUniqueName()))
- .putInputs("input", feedSDFCollectionId)
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(PTransformTranslation.SPLITTABLE_PROCESS_ELEMENTS_URN)
- .setPayload(spk.getSpec().getPayload()))
- .putAllOutputs(spk.getOutputsMap())
- .build();
- newComponents.putTransforms(runSDFId, runSDF);
- newPTransform.addSubtransforms(runSDFId);
- }
-
- return MessageWithComponents.newBuilder()
- .setPtransform(newPTransform.build())
- .setComponents(newComponents)
- .build();
- }
- }
-
- /**
- * Finds FEED_SDF nodes followed by an ExecutableStage and replaces them by a single {@link
- * SplittableRemoteStageEvaluatorFactory#URN} stage that feeds the ExecutableStage knowing that
- * the first instruction in the stage is an SDF.
- */
- private static Pipeline foldFeedSDFIntoExecutableStage(Pipeline p) {
- Pipeline.Builder newPipeline = p.toBuilder();
- Components.Builder newPipelineComponents = newPipeline.getComponentsBuilder();
-
- QueryablePipeline q = QueryablePipeline.forPipeline(p);
- String feedSdfUrn = SplittableRemoteStageEvaluatorFactory.FEED_SDF_URN;
- List<PTransformNode> feedSDFNodes =
- q.getTransforms().stream()
- .filter(node -> node.getTransform().getSpec().getUrn().equals(feedSdfUrn))
- .collect(Collectors.toList());
- Map<String, PTransformNode> stageToFeeder = Maps.newHashMap();
- for (PTransformNode node : feedSDFNodes) {
- PCollectionNode output = Iterables.getOnlyElement(q.getOutputPCollections(node));
- PTransformNode consumer = Iterables.getOnlyElement(q.getPerElementConsumers(output));
- String consumerUrn = consumer.getTransform().getSpec().getUrn();
- checkState(
- consumerUrn.equals(ExecutableStage.URN),
- "Expected all FeedSDF nodes to be consumed by an ExecutableStage, "
- + "but %s is consumed by %s which is %s",
- node.getId(),
- consumer.getId(),
- consumerUrn);
- stageToFeeder.put(consumer.getId(), node);
- }
-
- // Copy over root transforms except for the excluded FEED_SDF transforms.
- Set<String> feedSDFIds =
- feedSDFNodes.stream().map(PTransformNode::getId).collect(Collectors.toSet());
- newPipeline.clearRootTransformIds();
- for (String rootId : p.getRootTransformIdsList()) {
- if (!feedSDFIds.contains(rootId)) {
- newPipeline.addRootTransformIds(rootId);
- }
- }
- // Copy over all transforms, except FEED_SDF transforms are skipped, and ExecutableStage's
- // feeding from them are replaced.
- for (PTransformNode node : q.getTransforms()) {
- if (feedSDFNodes.contains(node)) {
- // These transforms are skipped and handled separately.
- continue;
- }
- if (!stageToFeeder.containsKey(node.getId())) {
- // This transform is unchanged
- newPipelineComponents.putTransforms(node.getId(), node.getTransform());
- continue;
- }
- // "node" is an ExecutableStage transform feeding from an SDF.
- PTransformNode feedSDFNode = stageToFeeder.get(node.getId());
- PCollectionNode rawGBKOutput =
- Iterables.getOnlyElement(q.getPerElementInputPCollections(feedSDFNode));
-
- // Replace the ExecutableStage transform.
- newPipelineComponents.putTransforms(
- node.getId(),
- node.getTransform()
- .toBuilder()
- .mergeSpec(
- // Change URN from ExecutableStage.URN to URN of the ULR's splittable executable
- // stage evaluator.
- FunctionSpec.newBuilder()
- .setUrn(SplittableRemoteStageEvaluatorFactory.URN)
- .build())
- .putInputs(
- // The splittable executable stage now reads from the raw GBK, instead of
- // from the now non-existent FEED_SDF.
- Iterables.getOnlyElement(node.getTransform().getInputsMap().keySet()),
- rawGBKOutput.getId())
- .build());
- }
-
- return newPipeline.build();
- }
-
- private enum EnvironmentType {
- DOCKER,
- IN_PROCESS
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactory.java
deleted file mode 100644
index 335c1e7..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactory.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import javax.annotation.Nullable;
-import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
-import org.apache.beam.runners.fnexecution.control.RemoteBundle;
-import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
-import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
-import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-
-/**
- * The {@link TransformEvaluatorFactory} which produces {@link TransformEvaluator evaluators} for
- * stages which execute on an SDK harness via the Fn Execution APIs.
- */
-class RemoteStageEvaluatorFactory implements TransformEvaluatorFactory {
- private final BundleFactory bundleFactory;
-
- private final JobBundleFactory jobFactory;
-
- RemoteStageEvaluatorFactory(BundleFactory bundleFactory, JobBundleFactory jobFactory) {
- this.bundleFactory = bundleFactory;
- this.jobFactory = jobFactory;
- }
-
- @Nullable
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) throws Exception {
- return new RemoteStageEvaluator<>(application);
- }
-
- @Override
- public void cleanup() throws Exception {
- jobFactory.close();
- }
-
- private class RemoteStageEvaluator<T> implements TransformEvaluator<T> {
- private final PTransformNode transform;
- private final RemoteBundle bundle;
- private final FnDataReceiver<WindowedValue<?>> mainInput;
- private final Collection<UncommittedBundle<?>> outputs;
-
- private RemoteStageEvaluator(PTransformNode transform) throws Exception {
- this.transform = transform;
- ExecutableStage stage =
- ExecutableStage.fromPayload(
- ExecutableStagePayload.parseFrom(transform.getTransform().getSpec().getPayload()));
- this.outputs = new ArrayList<>();
- StageBundleFactory stageFactory = jobFactory.forStage(stage);
- this.bundle =
- stageFactory.getBundle(
- BundleFactoryOutputReceiverFactory.create(
- bundleFactory, stage.getComponents(), outputs::add),
- StateRequestHandler.unsupported(),
- BundleProgressHandler.ignored());
- // TODO(BEAM-4680): Add support for timers as inputs to the ULR
- this.mainInput = Iterables.getOnlyElement(bundle.getInputReceivers().values());
- }
-
- @Override
- public void processElement(WindowedValue<T> element) throws Exception {
- mainInput.accept(element);
- }
-
- @Override
- public TransformResult<T> finishBundle() throws Exception {
- bundle.close();
- return StepTransformResult.<T>withoutHold(transform).addOutput(outputs).build();
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootInputProvider.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootInputProvider.java
deleted file mode 100644
index 9f904c7..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootInputProvider.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Collection;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.PTransform;
-
-/**
- * Provides {@link CommittedBundle bundles} that will be provided to the {@link PTransform
- * PTransforms} that are at the root of a {@link Pipeline}.
- */
-interface RootInputProvider<ShardT> {
- /**
- * Get the initial inputs for the {@link PTransformNode}. The {@link PTransformNode} will be
- * provided with these {@link CommittedBundle bundles} as input when the {@link Pipeline} runs.
- *
- * <p>For source transforms, these should be sufficient that, when provided to the evaluators
- * produced by {@link TransformEvaluatorFactory#forApplication(PTransformNode, CommittedBundle)},
- * all of the elements contained in the source are eventually produced.
- *
- * @param transform the {@link PTransformNode} to get initial inputs for.
- * @param targetParallelism the target amount of parallelism to obtain from the source. Must be
- * greater than or equal to 1.
- */
- Collection<CommittedBundle<ShardT>> getInitialInputs(
- PTransformNode transform, int targetParallelism) throws Exception;
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootProviderRegistry.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootProviderRegistry.java
deleted file mode 100644
index 054a04d..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootProviderRegistry.java
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.runners.core.construction.PTransformTranslation.FLATTEN_TRANSFORM_URN;
-import static org.apache.beam.runners.core.construction.PTransformTranslation.IMPULSE_TRANSFORM_URN;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-
-import java.util.Collection;
-import java.util.Map;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-
-/**
- * A {@link RootInputProvider} that delegates to primitive {@link RootInputProvider} implementations
- * based on the type of {@link PTransform} of the application.
- */
-class RootProviderRegistry {
-
- /**
- * Returns a {@link RootProviderRegistry} that supports the {@link Impulse} and {@link Flatten}
- * primitives.
- */
- static RootProviderRegistry javaPortableRegistry(BundleFactory bundleFactory) {
- return new RootProviderRegistry(
- ImmutableMap.<String, RootInputProvider<?>>builder()
- .put(
- IMPULSE_TRANSFORM_URN,
- new ImpulseEvaluatorFactory.ImpulseRootProvider(bundleFactory))
- .put(FLATTEN_TRANSFORM_URN, new EmptyInputProvider())
- .build());
- }
-
- private final Map<String, RootInputProvider<?>> providers;
-
- private RootProviderRegistry(Map<String, RootInputProvider<?>> providers) {
- this.providers = providers;
- }
-
- public Collection<CommittedBundle<?>> getInitialInputs(
- PTransformNode transform, int targetParallelism) throws Exception {
- String transformUrn = PTransformTranslation.urnForTransformOrNull(transform.getTransform());
- RootInputProvider provider =
- checkNotNull(
- providers.get(transformUrn),
- "Tried to get a %s for a transform \"%s\", but there is no such provider",
- RootInputProvider.class.getSimpleName(),
- transformUrn);
- return provider.getInitialInputs(transform, targetParallelism);
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SourceShard.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SourceShard.java
deleted file mode 100644
index cf978b8..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SourceShard.java
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.sdk.io.BoundedSource;
-import org.apache.beam.sdk.io.Read;
-import org.apache.beam.sdk.io.Source;
-import org.apache.beam.sdk.io.UnboundedSource;
-
-/**
- * A shard for a source in the {@link Read} transform.
- *
- * <p>Since {@link UnboundedSource} and {@link BoundedSource} have radically different needs, this
- * is a mostly-empty interface.
- */
-interface SourceShard<T> {
- Source<T> getSource();
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SplittableRemoteStageEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SplittableRemoteStageEvaluatorFactory.java
deleted file mode 100644
index 28f8789..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SplittableRemoteStageEvaluatorFactory.java
+++ /dev/null
@@ -1,174 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import javax.annotation.Nullable;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
-import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
-import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
-import org.apache.beam.runners.fnexecution.control.RemoteBundle;
-import org.apache.beam.runners.fnexecution.splittabledofn.SDFFeederViaStateAndTimers;
-import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
-import org.apache.beam.runners.fnexecution.wire.WireCoders;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-
-/**
- * The {@link TransformEvaluatorFactory} for {@link #URN}, which reads from a {@link
- * DirectGroupByKey#DIRECT_GBKO_URN} and feeds the data, using state and timers, to a {@link
- * ExecutableStage} whose first instruction is an SDF.
- */
-class SplittableRemoteStageEvaluatorFactory implements TransformEvaluatorFactory {
- public static final String URN = "beam:directrunner:transforms:splittable_remote_stage:v1";
-
- // A fictional transform that transforms from KWI<unique key, KV<element, restriction>>
- // to simply KV<element, restriction> taken by the SDF inside the ExecutableStage.
- public static final String FEED_SDF_URN = "beam:directrunner:transforms:feed_sdf:v1";
-
- private final BundleFactory bundleFactory;
- private final JobBundleFactory jobBundleFactory;
- private final StepStateAndTimers.Provider stp;
-
- SplittableRemoteStageEvaluatorFactory(
- BundleFactory bundleFactory,
- JobBundleFactory jobBundleFactory,
- StepStateAndTimers.Provider stepStateAndTimers) {
- this.bundleFactory = bundleFactory;
- this.jobBundleFactory = jobBundleFactory;
- this.stp = stepStateAndTimers;
- }
-
- @Nullable
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) throws Exception {
- return new SplittableRemoteStageEvaluator(
- bundleFactory,
- jobBundleFactory,
- stp.forStepAndKey(application, inputBundle.getKey()),
- application);
- }
-
- @Override
- public void cleanup() throws Exception {
- jobBundleFactory.close();
- }
-
- private static class SplittableRemoteStageEvaluator<InputT, RestrictionT>
- implements TransformEvaluator<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>> {
- private final PTransformNode transform;
- private final ExecutableStage stage;
-
- private final CopyOnAccessInMemoryStateInternals<byte[]> stateInternals;
- private final DirectTimerInternals timerInternals;
- private final RemoteBundle bundle;
- private final FnDataReceiver<WindowedValue<?>> mainInput;
- private final Collection<UncommittedBundle<?>> outputs;
-
- private final SDFFeederViaStateAndTimers<InputT, RestrictionT> feeder;
-
- private SplittableRemoteStageEvaluator(
- BundleFactory bundleFactory,
- JobBundleFactory jobBundleFactory,
- StepStateAndTimers<byte[]> stp,
- PTransformNode transform)
- throws Exception {
- this.stateInternals = stp.stateInternals();
- this.timerInternals = stp.timerInternals();
- this.transform = transform;
- this.stage =
- ExecutableStage.fromPayload(
- ExecutableStagePayload.parseFrom(transform.getTransform().getSpec().getPayload()));
- this.outputs = new ArrayList<>();
-
- FullWindowedValueCoder<KV<InputT, RestrictionT>> windowedValueCoder =
- (FullWindowedValueCoder<KV<InputT, RestrictionT>>)
- WireCoders.<KV<InputT, RestrictionT>>instantiateRunnerWireCoder(
- stage.getInputPCollection(), stage.getComponents());
- KvCoder<InputT, RestrictionT> kvCoder =
- (KvCoder<InputT, RestrictionT>) windowedValueCoder.getValueCoder();
- this.feeder =
- new SDFFeederViaStateAndTimers<>(
- stateInternals,
- timerInternals,
- kvCoder.getKeyCoder(),
- kvCoder.getValueCoder(),
- (Coder<BoundedWindow>) windowedValueCoder.getWindowCoder());
-
- this.bundle =
- jobBundleFactory
- .forStage(stage)
- .getBundle(
- BundleFactoryOutputReceiverFactory.create(
- bundleFactory, stage.getComponents(), outputs::add),
- StateRequestHandler.unsupported(),
- // TODO: Wire in splitting via a split listener
- new BundleProgressHandler() {
- @Override
- public void onProgress(ProcessBundleProgressResponse progress) {}
-
- @Override
- public void onCompleted(ProcessBundleResponse response) {}
- });
- this.mainInput = Iterables.getOnlyElement(bundle.getInputReceivers().values());
- }
-
- @Override
- public void processElement(
- WindowedValue<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>> windowedWorkItem)
- throws Exception {
- KeyedWorkItem<byte[], KV<InputT, RestrictionT>> kwi = windowedWorkItem.getValue();
- WindowedValue<KV<InputT, RestrictionT>> elementRestriction =
- Iterables.getOnlyElement(kwi.elementsIterable(), null);
- if (elementRestriction != null) {
- feeder.seed(elementRestriction);
- } else {
- elementRestriction = feeder.resume(Iterables.getOnlyElement(kwi.timersIterable()));
- }
- mainInput.accept(elementRestriction);
- }
-
- @Override
- public TransformResult<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>> finishBundle()
- throws Exception {
- bundle.close();
- feeder.commit();
- CopyOnAccessInMemoryStateInternals<byte[]> state = stateInternals.commit();
- StepTransformResult.Builder<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>> result =
- StepTransformResult.withHold(transform, state.getEarliestWatermarkHold());
- return result
- .addOutput(outputs)
- .withState(state)
- .withTimerUpdate(timerInternals.getTimerUpdate())
- .build();
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepAndKey.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepAndKey.java
deleted file mode 100644
index 4c4a48b..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepAndKey.java
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Objects;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-
-/**
- * A (Step, Key) pair. This is useful as a map key or cache key for things that are available
- * per-step in a keyed manner (e.g. State).
- */
-final class StepAndKey {
- private final PTransformNode step;
- private final StructuralKey<?> key;
-
- /** Create a new {@link StepAndKey} with the provided step and key. */
- public static StepAndKey of(PTransformNode step, StructuralKey<?> key) {
- return new StepAndKey(step, key);
- }
-
- private StepAndKey(PTransformNode step, StructuralKey<?> key) {
- this.step = step;
- this.key = key;
- }
-
- @Override
- public String toString() {
- return MoreObjects.toStringHelper(StepAndKey.class)
- .add("step", step.getId())
- .add("key", key.getKey())
- .toString();
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(step, key);
- }
-
- @Override
- public boolean equals(Object other) {
- if (other == this) {
- return true;
- } else if (!(other instanceof StepAndKey)) {
- return false;
- } else {
- StepAndKey that = (StepAndKey) other;
- return Objects.equals(this.step, that.step) && Objects.equals(this.key, that.key);
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepStateAndTimers.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepStateAndTimers.java
deleted file mode 100644
index e5cfe55..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepStateAndTimers.java
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.local.StructuralKey;
-
-/** A provider of {@link StateInternals} and {@link TimerInternals}. */
-interface StepStateAndTimers<K> {
- interface Provider {
- <K> StepStateAndTimers<K> forStepAndKey(PTransformNode transform, StructuralKey<K> key);
- }
-
- CopyOnAccessInMemoryStateInternals<K> stateInternals();
-
- DirectTimerInternals timerInternals();
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepTransformResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepTransformResult.java
deleted file mode 100644
index b694d3b..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepTransformResult.java
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import com.google.auto.value.AutoValue;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.EnumSet;
-import java.util.Set;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.runners.direct.portable.CommittedResult.OutputType;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.joda.time.Instant;
-
-/** An immutable {@link TransformResult}. */
-@AutoValue
-abstract class StepTransformResult<InputT> implements TransformResult<InputT> {
-
- public static <InputT> Builder<InputT> withHold(PTransformNode transform, Instant watermarkHold) {
- return new Builder(transform, watermarkHold);
- }
-
- public static <InputT> Builder<InputT> withoutHold(PTransformNode transform) {
- return new Builder(transform, BoundedWindow.TIMESTAMP_MAX_VALUE);
- }
-
- @Override
- public TransformResult<InputT> withLogicalMetricUpdates(MetricUpdates metricUpdates) {
- return new AutoValue_StepTransformResult(
- getTransform(),
- getOutputBundles(),
- getUnprocessedElements(),
- metricUpdates,
- getWatermarkHold(),
- getState(),
- getTimerUpdate(),
- getOutputTypes());
- }
-
- /** A builder for creating instances of {@link StepTransformResult}. */
- public static class Builder<InputT> {
- private final PTransformNode transform;
- private final ImmutableList.Builder<UncommittedBundle<?>> bundlesBuilder;
- private final ImmutableList.Builder<WindowedValue<InputT>> unprocessedElementsBuilder;
- private MetricUpdates metricUpdates;
- private CopyOnAccessInMemoryStateInternals state;
- private TimerUpdate timerUpdate;
- private final Set<OutputType> producedOutputs;
- private final Instant watermarkHold;
-
- private Builder(PTransformNode transform, Instant watermarkHold) {
- this.transform = transform;
- this.watermarkHold = watermarkHold;
- this.bundlesBuilder = ImmutableList.builder();
- this.producedOutputs = EnumSet.noneOf(OutputType.class);
- this.unprocessedElementsBuilder = ImmutableList.builder();
- this.timerUpdate = TimerUpdate.builder(null).build();
- this.metricUpdates = MetricUpdates.EMPTY;
- }
-
- public StepTransformResult<InputT> build() {
- return new AutoValue_StepTransformResult<>(
- transform,
- bundlesBuilder.build(),
- unprocessedElementsBuilder.build(),
- metricUpdates,
- watermarkHold,
- state,
- timerUpdate,
- producedOutputs);
- }
-
- public Builder<InputT> withMetricUpdates(MetricUpdates metricUpdates) {
- this.metricUpdates = metricUpdates;
- return this;
- }
-
- public Builder<InputT> withState(CopyOnAccessInMemoryStateInternals state) {
- this.state = state;
- return this;
- }
-
- public Builder<InputT> withTimerUpdate(TimerUpdate timerUpdate) {
- this.timerUpdate = timerUpdate;
- return this;
- }
-
- public Builder<InputT> addUnprocessedElements(WindowedValue<InputT>... unprocessed) {
- unprocessedElementsBuilder.addAll(Arrays.asList(unprocessed));
- return this;
- }
-
- public Builder<InputT> addUnprocessedElements(
- Iterable<? extends WindowedValue<InputT>> unprocessed) {
- unprocessedElementsBuilder.addAll(unprocessed);
- return this;
- }
-
- public Builder<InputT> addOutput(
- UncommittedBundle<?> outputBundle, UncommittedBundle<?>... outputBundles) {
- bundlesBuilder.add(outputBundle);
- bundlesBuilder.add(outputBundles);
- return this;
- }
-
- public Builder<InputT> addOutput(Collection<UncommittedBundle<?>> outputBundles) {
- bundlesBuilder.addAll(outputBundles);
- return this;
- }
-
- public Builder<InputT> withAdditionalOutput(OutputType producedAdditionalOutput) {
- producedOutputs.add(producedAdditionalOutput);
- return this;
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluator.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluator.java
deleted file mode 100644
index 2579bcc..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluator.java
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.sdk.util.WindowedValue;
-
-/**
- * An evaluator of a specific application of a transform. Will be used for at least one {@link
- * CommittedBundle}.
- *
- * @param <InputT> the type of elements that will be passed to {@link #processElement}
- */
-interface TransformEvaluator<InputT> {
- /**
- * Process an element in the input {@link CommittedBundle}.
- *
- * @param element the element to process
- */
- void processElement(WindowedValue<InputT> element) throws Exception;
-
- /**
- * Finish processing the bundle of this {@link TransformEvaluator}.
- *
- * <p>After {@link #finishBundle()} is called, the {@link TransformEvaluator} will not be reused,
- * and no more elements will be processed.
- *
- * @return an {@link TransformResult} containing the results of this bundle evaluation.
- */
- TransformResult<InputT> finishBundle() throws Exception;
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorFactory.java
deleted file mode 100644
index 9892c36..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorFactory.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.io.Read;
-import org.apache.beam.sdk.transforms.PTransform;
-
-/**
- * A factory for creating instances of {@link TransformEvaluator} for the application of a {@link
- * PTransform}.
- *
- * <p>{@link TransformEvaluatorFactory TransformEvaluatorFactories} will be reused within a single
- * execution of a {@link Pipeline} but will not be reused across executions.
- */
-interface TransformEvaluatorFactory {
- /**
- * Create a new {@link TransformEvaluator} for the application of the {@link PTransform}.
- *
- * <p>Any work that must be done before input elements are processed (such as calling {@code
- * DoFn.StartBundle}) must be done before the {@link TransformEvaluator} is made available to the
- * caller.
- *
- * <p>May return null if the application cannot produce an evaluator (for example, it is a {@link
- * Read} {@link PTransform} where all evaluators are in-use).
- *
- * @return An evaluator capable of processing the transform on the bundle, or null if no evaluator
- * can be constructed.
- * @throws Exception whenever constructing the underlying evaluator throws an exception
- */
- @Nullable
- <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) throws Exception;
-
- /**
- * Cleans up any state maintained by this {@link TransformEvaluatorFactory}. Called after a {@link
- * Pipeline} is shut down. No more calls to {@link #forApplication(PTransformNode,
- * CommittedBundle)} will be made after a call to {@link #cleanup()}.
- */
- void cleanup() throws Exception;
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorRegistry.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorRegistry.java
deleted file mode 100644
index 6587882..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorRegistry.java
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Map;
-import java.util.concurrent.atomic.AtomicBoolean;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * A {@link TransformEvaluatorFactory} that delegates to primitive {@link TransformEvaluatorFactory}
- * implementations based on the type of {@link PTransform} of the application.
- */
-class TransformEvaluatorRegistry {
- private static final Logger LOG = LoggerFactory.getLogger(TransformEvaluatorRegistry.class);
-
- static TransformEvaluatorRegistry portableRegistry(
- ExecutableGraph<PTransformNode, PCollectionNode> graph,
- Components components,
- BundleFactory bundleFactory,
- JobBundleFactory jobBundleFactory,
- StepStateAndTimers.Provider stepStateAndTimers) {
- return new TransformEvaluatorRegistry(
- ImmutableMap.<String, TransformEvaluatorFactory>builder()
- .put(
- PTransformTranslation.IMPULSE_TRANSFORM_URN,
- new ImpulseEvaluatorFactory(graph, bundleFactory))
- .put(
- PTransformTranslation.FLATTEN_TRANSFORM_URN,
- new FlattenEvaluatorFactory(graph, bundleFactory))
- .put(
- DirectGroupByKey.DIRECT_GBKO_URN,
- new GroupByKeyOnlyEvaluatorFactory(graph, components, bundleFactory))
- .put(
- DirectGroupByKey.DIRECT_GABW_URN,
- new GroupAlsoByWindowEvaluatorFactory(
- graph, components, bundleFactory, stepStateAndTimers))
- .put(
- ExecutableStage.URN,
- new RemoteStageEvaluatorFactory(bundleFactory, jobBundleFactory))
- .put(
- SplittableRemoteStageEvaluatorFactory.URN,
- new SplittableRemoteStageEvaluatorFactory(
- bundleFactory, jobBundleFactory, stepStateAndTimers))
- .build());
- }
-
- // the TransformEvaluatorFactories can construct instances of all generic types of transform,
- // so all instances of a primitive can be handled with the same evaluator factory.
- private final Map<String, TransformEvaluatorFactory> factories;
-
- private final AtomicBoolean finished = new AtomicBoolean(false);
-
- private TransformEvaluatorRegistry(
- @SuppressWarnings("rawtypes") Map<String, TransformEvaluatorFactory> factories) {
- this.factories = factories;
- }
-
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, CommittedBundle<?> inputBundle) throws Exception {
- checkState(
- !finished.get(), "Tried to get an evaluator for a finished TransformEvaluatorRegistry");
-
- String urn = PTransformTranslation.urnForTransformOrNull(application.getTransform());
-
- TransformEvaluatorFactory factory =
- checkNotNull(factories.get(urn), "No evaluator for PTransform \"%s\"", urn);
- return factory.forApplication(application, inputBundle);
- }
-
- public void cleanup() throws Exception {
- Collection<Exception> thrownInCleanup = new ArrayList<>();
- for (TransformEvaluatorFactory factory : factories.values()) {
- try {
- factory.cleanup();
- } catch (Exception e) {
- if (e instanceof InterruptedException) {
- Thread.currentThread().interrupt();
- }
- thrownInCleanup.add(e);
- }
- }
- finished.set(true);
- if (!thrownInCleanup.isEmpty()) {
- LOG.error("Exceptions {} thrown while cleaning up evaluators", thrownInCleanup);
- Exception toThrow = null;
- for (Exception e : thrownInCleanup) {
- if (toThrow == null) {
- toThrow = e;
- } else {
- toThrow.addSuppressed(e);
- }
- }
- throw toThrow;
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutor.java
deleted file mode 100644
index f4f34cc..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutor.java
+++ /dev/null
@@ -1,21 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-/** A {@link Runnable} that will execute a {@code PTransform} on some bundle of input. */
-public interface TransformExecutor extends Runnable {}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorFactory.java
deleted file mode 100644
index fe29aee..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorFactory.java
+++ /dev/null
@@ -1,29 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-
-/** A Factory for creating {@link TransformExecutor Transform Executors} on an input. */
-interface TransformExecutorFactory {
- TransformExecutor create(
- CommittedBundle<?> bundle,
- PTransformNode transform,
- CompletionCallback onComplete,
- TransformExecutorService executorService);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorService.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorService.java
deleted file mode 100644
index f8dcf8d..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorService.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-/**
- * Schedules and completes {@link TransformExecutor TransformExecutors}, controlling concurrency as
- * appropriate for the {@link StepAndKey} the executor exists for.
- */
-interface TransformExecutorService {
- /** Schedule the provided work to be eventually executed. */
- void schedule(TransformExecutor work);
-
- /**
- * Finish executing the provided work. This may cause additional {@link DirectTransformExecutor
- * TransformExecutors} to be evaluated.
- */
- void complete(TransformExecutor completed);
-
- /**
- * Cancel any outstanding work, if possible. Any future calls to schedule should ignore any work.
- */
- void shutdown();
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorServices.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorServices.java
deleted file mode 100644
index 24d7694..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorServices.java
+++ /dev/null
@@ -1,180 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Queue;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.RejectedExecutionException;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** Static factory methods for constructing instances of {@link TransformExecutorService}. */
-final class TransformExecutorServices {
- private TransformExecutorServices() {
- // Do not instantiate
- }
-
- /**
- * Returns an EvaluationState that evaluates {@link TransformExecutor TransformExecutors} in
- * parallel.
- */
- public static TransformExecutorService parallel(ExecutorService executor) {
- return new ParallelTransformExecutor(executor);
- }
-
- /**
- * Returns an EvaluationState that evaluates {@link TransformExecutor TransformExecutors} in
- * serial.
- */
- public static TransformExecutorService serial(ExecutorService executor) {
- return new SerialTransformExecutor(executor);
- }
-
- /**
- * A {@link TransformExecutorService} with unlimited parallelism. Any {@link TransformExecutor}
- * scheduled will be immediately submitted to the {@link ExecutorService}.
- *
- * <p>A principal use of this is for the evaluation of an unkeyed Step. Unkeyed computations are
- * processed in parallel.
- */
- private static class ParallelTransformExecutor implements TransformExecutorService {
- private static final Logger LOG = LoggerFactory.getLogger(ParallelTransformExecutor.class);
-
- private final ExecutorService executor;
- private final AtomicBoolean active = new AtomicBoolean(true);
-
- private ParallelTransformExecutor(ExecutorService executor) {
- this.executor = executor;
- }
-
- @Override
- // TODO: [BEAM-4563] Pass Future back to consumer to check for async errors
- @SuppressWarnings("FutureReturnValueIgnored")
- public void schedule(TransformExecutor work) {
- if (active.get()) {
- try {
- executor.submit(work);
- } catch (RejectedExecutionException rejected) {
- boolean stillActive = active.get();
- if (stillActive) {
- throw new IllegalStateException(
- String.format(
- "Execution of Work %s was rejected, but the %s is still active",
- work, ParallelTransformExecutor.class.getSimpleName()));
- } else {
- LOG.debug(
- "Rejected execution of Work {} on executor {}. "
- + "Suppressed exception because evaluator is not active",
- work,
- this);
- }
- }
- }
- }
-
- @Override
- public void complete(TransformExecutor completed) {}
-
- @Override
- public void shutdown() {
- active.set(false);
- }
- }
-
- /**
- * A {@link TransformExecutorService} with a single work queue. Any {@link TransformExecutor}
- * scheduled will be placed on the work queue. Only one item of work will be submitted to the
- * {@link ExecutorService} at any time.
- *
- * <p>A principal use of this is for the serial evaluation of a (Step, Key) pair. Keyed
- * computations are processed serially per step.
- */
- private static class SerialTransformExecutor implements TransformExecutorService {
- private final ExecutorService executor;
-
- private AtomicReference<TransformExecutor> currentlyEvaluating;
- private final Queue<TransformExecutor> workQueue;
- private boolean active = true;
-
- private SerialTransformExecutor(ExecutorService executor) {
- this.executor = executor;
- this.currentlyEvaluating = new AtomicReference<>();
- this.workQueue = new ConcurrentLinkedQueue<>();
- }
-
- /**
- * Schedules the work, adding it to the work queue if there is a bundle currently being
- * evaluated and scheduling it immediately otherwise.
- */
- @Override
- public void schedule(TransformExecutor work) {
- workQueue.offer(work);
- updateCurrentlyEvaluating();
- }
-
- @Override
- public void complete(TransformExecutor completed) {
- if (!currentlyEvaluating.compareAndSet(completed, null)) {
- throw new IllegalStateException(
- "Finished work "
- + completed
- + " but could not complete due to unexpected currently executing "
- + currentlyEvaluating.get());
- }
- updateCurrentlyEvaluating();
- }
-
- @Override
- public void shutdown() {
- synchronized (this) {
- active = false;
- }
- workQueue.clear();
- }
-
- // TODO: [BEAM-4563] Pass Future back to consumer to check for async errors
- @SuppressWarnings("FutureReturnValueIgnored")
- private void updateCurrentlyEvaluating() {
- if (currentlyEvaluating.get() == null) {
- // Only synchronize if we need to update what's currently evaluating
- synchronized (this) {
- TransformExecutor newWork = workQueue.poll();
- if (active && newWork != null) {
- if (currentlyEvaluating.compareAndSet(null, newWork)) {
- executor.submit(newWork);
- } else {
- workQueue.offer(newWork);
- }
- }
- }
- }
- }
-
- @Override
- public String toString() {
- return MoreObjects.toStringHelper(SerialTransformExecutor.class)
- .add("currentlyEvaluating", currentlyEvaluating)
- .add("workQueue", workQueue)
- .toString();
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformResult.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformResult.java
deleted file mode 100644
index 3f055e4..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformResult.java
+++ /dev/null
@@ -1,101 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.runners.direct.portable.CommittedResult.OutputType;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.joda.time.Instant;
-
-/**
- * The result of evaluating an {@link PTransformNode} with a {@link TransformEvaluator}.
- *
- * <p>Every transform evaluator has a defined input type, but {@link ParDo} has multiple outputs so
- * there is not necesssarily a defined output type.
- */
-interface TransformResult<InputT> {
- /**
- * Returns the {@link PTransformNode} that produced this result.
- *
- * <p>This is treated as an opaque identifier so evaluators can delegate to other evaluators that
- * may not have compatible types.
- */
- PTransformNode getTransform();
-
- /**
- * Returns the {@link UncommittedBundle (uncommitted) Bundles} output by this transform. These
- * will be committed by the evaluation context as part of completing this result.
- *
- * <p>Note that the bundles need not have a uniform type, for example in the case of multi-output
- * {@link ParDo}.
- */
- Iterable<? extends UncommittedBundle<?>> getOutputBundles();
-
- /**
- * Returns elements that were provided to the {@link TransformEvaluator} as input but were not
- * processed.
- */
- Iterable<? extends WindowedValue<InputT>> getUnprocessedElements();
-
- /** Returns the logical metric updates. */
- MetricUpdates getLogicalMetricUpdates();
-
- /**
- * Returns the Watermark Hold for the transform at the time this result was produced.
- *
- * <p>If the transform does not set any watermark hold, returns {@link
- * BoundedWindow#TIMESTAMP_MAX_VALUE}.
- */
- Instant getWatermarkHold();
-
- /**
- * Returns the State used by the transform.
- *
- * <p>If this evaluation did not access state, this may return null.
- */
- @Nullable
- CopyOnAccessInMemoryStateInternals getState();
-
- /**
- * Returns a TimerUpdateBuilder that was produced as a result of this evaluation. If the
- * evaluation was triggered due to the delivery of one or more timers, those timers must be added
- * to the builder before it is complete.
- *
- * <p>If this evaluation did not add or remove any timers, returns an empty TimerUpdate.
- */
- TimerUpdate getTimerUpdate();
-
- /**
- * Returns the types of output produced by this {@link PTransform}. This may not include {@link
- * OutputType#BUNDLE}, as empty bundles may be dropped when the transform is committed.
- */
- Set<OutputType> getOutputTypes();
-
- /**
- * Returns a new TransformResult based on this one but overwriting any existing logical metric
- * updates with {@code metricUpdates}.
- */
- TransformResult<InputT> withLogicalMetricUpdates(MetricUpdates metricUpdates);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/UncommittedBundle.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/UncommittedBundle.java
deleted file mode 100644
index a55d415..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/UncommittedBundle.java
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.joda.time.Instant;
-
-/**
- * Part of a {@link PCollection}. Elements are output to a bundle, which will cause them to be
- * executed by {@link PTransform PTransforms} that consume the {@link PCollection} this bundle is a
- * part of at a later point. This is an uncommitted bundle and can have elements added to it.
- *
- * @param <T> the type of elements that can be added to this bundle
- */
-interface UncommittedBundle<T> {
- /** Returns the PCollection that the elements of this {@link UncommittedBundle} belong to. */
- @Nullable
- PCollectionNode getPCollection();
-
- /**
- * Outputs an element to this bundle.
- *
- * @param element the element to add to this bundle
- * @return this bundle
- */
- UncommittedBundle<T> add(WindowedValue<T> element);
-
- /**
- * Commits this {@link UncommittedBundle}, returning an immutable {@link CommittedBundle}
- * containing all of the elements that were added to it. The {@link #add(WindowedValue)} method
- * will throw an {@link IllegalStateException} if called after a call to commit.
- *
- * @param synchronizedProcessingTime the synchronized processing time at which this bundle was
- * committed
- */
- CommittedBundle<T> commit(Instant synchronizedProcessingTime);
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutor.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutor.java
deleted file mode 100644
index b7c8a83..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutor.java
+++ /dev/null
@@ -1,171 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.io.Serializable;
-import java.util.PriorityQueue;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.Executor;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.joda.time.Instant;
-
-/**
- * Executes callbacks that occur based on the progression of the watermark per-step.
- *
- * <p>Callbacks are registered by calls to {@link #callOnGuaranteedFiring(PTransformNode,
- * BoundedWindow, WindowingStrategy, Runnable)}, and are executed after a call to {@link
- * #fireForWatermark(PTransformNode, Instant)} with the same {@link PTransformNode} and a watermark
- * sufficient to ensure that the trigger for the windowing strategy would have been produced.
- *
- * <p>NOTE: {@link WatermarkCallbackExecutor} does not track the latest observed watermark for any
- * {@link PTransformNode} - any call to {@link #callOnGuaranteedFiring(PTransformNode,
- * BoundedWindow, WindowingStrategy, Runnable)} that could have potentially already fired should be
- * followed by a call to {@link #fireForWatermark(PTransformNode, Instant)} for the same transform
- * with the current value of the watermark.
- */
-class WatermarkCallbackExecutor {
- /** Create a new {@link WatermarkCallbackExecutor}. */
- public static WatermarkCallbackExecutor create(Executor executor) {
- return new WatermarkCallbackExecutor(executor);
- }
-
- private final ConcurrentMap<PTransformNode, PriorityQueue<WatermarkCallback>> callbacks;
- private final Executor executor;
-
- private WatermarkCallbackExecutor(Executor executor) {
- this.callbacks = new ConcurrentHashMap<>();
- this.executor = executor;
- }
-
- /**
- * Execute the provided {@link Runnable} after the next call to {@link
- * #fireForWatermark(PTransformNode, Instant)} where the window is guaranteed to have produced
- * output.
- */
- public void callOnGuaranteedFiring(
- PTransformNode step,
- BoundedWindow window,
- WindowingStrategy<?, ?> windowingStrategy,
- Runnable runnable) {
- WatermarkCallback callback =
- WatermarkCallback.onGuaranteedFiring(window, windowingStrategy, runnable);
-
- PriorityQueue<WatermarkCallback> callbackQueue = callbacks.get(step);
- if (callbackQueue == null) {
- callbackQueue = new PriorityQueue<>(11, new CallbackOrdering());
- if (callbacks.putIfAbsent(step, callbackQueue) != null) {
- callbackQueue = callbacks.get(step);
- }
- }
-
- synchronized (callbackQueue) {
- callbackQueue.offer(callback);
- }
- }
-
- /**
- * Execute the provided {@link Runnable} after the next call to {@link
- * #fireForWatermark(PTransformNode, Instant)} where the window is guaranteed to be expired.
- */
- public void callOnWindowExpiration(
- PTransformNode step,
- BoundedWindow window,
- WindowingStrategy<?, ?> windowingStrategy,
- Runnable runnable) {
- WatermarkCallback callback =
- WatermarkCallback.afterWindowExpiration(window, windowingStrategy, runnable);
-
- PriorityQueue<WatermarkCallback> callbackQueue = callbacks.get(step);
- if (callbackQueue == null) {
- callbackQueue = new PriorityQueue<>(11, new CallbackOrdering());
- if (callbacks.putIfAbsent(step, callbackQueue) != null) {
- callbackQueue = callbacks.get(step);
- }
- }
-
- synchronized (callbackQueue) {
- callbackQueue.offer(callback);
- }
- }
-
- /**
- * Schedule all pending callbacks that must have produced output by the time of the provided
- * watermark.
- */
- public void fireForWatermark(PTransformNode step, Instant watermark) {
- PriorityQueue<WatermarkCallback> callbackQueue = callbacks.get(step);
- if (callbackQueue == null) {
- return;
- }
- synchronized (callbackQueue) {
- while (!callbackQueue.isEmpty() && callbackQueue.peek().shouldFire(watermark)) {
- executor.execute(callbackQueue.poll().getCallback());
- }
- }
- }
-
- private static class WatermarkCallback {
- public static <W extends BoundedWindow> WatermarkCallback onGuaranteedFiring(
- BoundedWindow window, WindowingStrategy<?, W> strategy, Runnable callback) {
- @SuppressWarnings("unchecked")
- Instant firingAfter = strategy.getTrigger().getWatermarkThatGuaranteesFiring((W) window);
- return new WatermarkCallback(firingAfter, callback);
- }
-
- public static <W extends BoundedWindow> WatermarkCallback afterWindowExpiration(
- BoundedWindow window, WindowingStrategy<?, W> strategy, Runnable callback) {
- // Fire one milli past the end of the window. This ensures that all window expiration
- // timers are delivered first
- Instant firingAfter = window.maxTimestamp().plus(strategy.getAllowedLateness()).plus(1L);
- return new WatermarkCallback(firingAfter, callback);
- }
-
- private final Instant fireAfter;
- private final Runnable callback;
-
- private WatermarkCallback(Instant fireAfter, Runnable callback) {
- this.fireAfter = fireAfter;
- this.callback = callback;
- }
-
- public boolean shouldFire(Instant currentWatermark) {
- return currentWatermark.isAfter(fireAfter)
- || currentWatermark.equals(BoundedWindow.TIMESTAMP_MAX_VALUE);
- }
-
- public Runnable getCallback() {
- return callback;
- }
- }
-
- private static class CallbackOrdering extends Ordering<WatermarkCallback>
- implements Serializable {
- @Override
- public int compare(WatermarkCallback left, WatermarkCallback right) {
- return ComparisonChain.start()
- .compare(left.fireAfter, right.fireAfter)
- .compare(left.callback, right.callback, Ordering.arbitrary())
- .result();
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactory.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactory.java
deleted file mode 100644
index 83d745d..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactory.java
+++ /dev/null
@@ -1,123 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import java.util.Collection;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Instant;
-
-/**
- * The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@link Window.Assign}
- * primitive {@link PTransform}.
- */
-class WindowEvaluatorFactory implements TransformEvaluatorFactory {
- private final EvaluationContext evaluationContext;
-
- WindowEvaluatorFactory(EvaluationContext evaluationContext) {
- this.evaluationContext = evaluationContext;
- }
-
- @Override
- public <InputT> TransformEvaluator<InputT> forApplication(
- PTransformNode application, @Nullable CommittedBundle<?> inputBundle) {
- return createTransformEvaluator(application);
- }
-
- private <InputT> TransformEvaluator<InputT> createTransformEvaluator(PTransformNode transform) {
- WindowFn<? super InputT, ?> fn = null;
-
- PCollectionNode outputPCollection = null;
- evaluationContext.createBundle(outputPCollection);
- throw new UnsupportedOperationException("Not yet migrated");
- }
-
- @Override
- public void cleanup() {}
-
- private static class WindowIntoEvaluator<InputT> implements TransformEvaluator<InputT> {
- private final PTransformNode transform;
- private final WindowFn<InputT, ?> windowFn;
- private final UncommittedBundle<InputT> outputBundle;
-
- @SuppressWarnings("unchecked")
- public WindowIntoEvaluator(
- PTransformNode transform,
- WindowFn<? super InputT, ?> windowFn,
- UncommittedBundle<InputT> outputBundle) {
- this.outputBundle = outputBundle;
- this.transform = transform;
- // Safe contravariant cast
- this.windowFn = (WindowFn<InputT, ?>) windowFn;
- }
-
- @Override
- public void processElement(WindowedValue<InputT> compressedElement) throws Exception {
- for (WindowedValue<InputT> element : compressedElement.explodeWindows()) {
- Collection<? extends BoundedWindow> windows = assignWindows(windowFn, element);
- outputBundle.add(
- WindowedValue.of(
- element.getValue(), element.getTimestamp(), windows, element.getPane()));
- }
- }
-
- private <W extends BoundedWindow> Collection<? extends BoundedWindow> assignWindows(
- WindowFn<InputT, W> windowFn, WindowedValue<InputT> element) throws Exception {
- WindowFn<InputT, W>.AssignContext assignContext =
- new DirectAssignContext<>(windowFn, element);
- return windowFn.assignWindows(assignContext);
- }
-
- @Override
- public TransformResult<InputT> finishBundle() throws Exception {
- return StepTransformResult.<InputT>withoutHold(transform).addOutput(outputBundle).build();
- }
- }
-
- private static class DirectAssignContext<InputT, W extends BoundedWindow>
- extends WindowFn<InputT, W>.AssignContext {
- private final WindowedValue<InputT> value;
-
- public DirectAssignContext(WindowFn<InputT, W> fn, WindowedValue<InputT> value) {
- fn.super();
- this.value = value;
- }
-
- @Override
- public InputT element() {
- return value.getValue();
- }
-
- @Override
- public Instant timestamp() {
- return value.getTimestamp();
- }
-
- @Override
- public BoundedWindow window() {
- return Iterables.getOnlyElement(value.getWindows());
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
deleted file mode 100644
index eb9b33b..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
+++ /dev/null
@@ -1,116 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import java.io.File;
-import java.io.IOException;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
-
-/**
- * A location where the results of an {@link LocalFileSystemArtifactStagerService} are stored and
- * where the retrieval service retrieves them from.
- */
-class LocalArtifactStagingLocation {
- /**
- * Create a new {@link LocalArtifactStagingLocation} rooted at the specified location, creating
- * any directories or subdirectories as necessary.
- */
- public static LocalArtifactStagingLocation createAt(File rootDirectory) {
- return new LocalArtifactStagingLocation(rootDirectory).createDirectories();
- }
-
- /**
- * Create a {@link LocalArtifactStagingLocation} for an existing directory. The directory must
- * contain a manifest and an artifact directory.
- */
- public static LocalArtifactStagingLocation forExistingDirectory(File rootDirectory) {
- return new LocalArtifactStagingLocation(rootDirectory).verifyExistence();
- }
-
- private final File rootDirectory;
- private final File artifactsDirectory;
-
- private LocalArtifactStagingLocation(File base) {
- this.rootDirectory = base;
- this.artifactsDirectory = new File(base, "artifacts");
- }
-
- private LocalArtifactStagingLocation createDirectories() {
- if (((rootDirectory.exists() && rootDirectory.isDirectory()) || rootDirectory.mkdirs())
- && rootDirectory.canWrite()) {
- checkState(
- ((artifactsDirectory.exists() && artifactsDirectory.isDirectory())
- || artifactsDirectory.mkdir())
- && artifactsDirectory.canWrite(),
- "Could not create artifact staging directory at %s",
- artifactsDirectory);
- } else {
- throw new IllegalStateException(
- String.format("Could not create staging directory structure at root %s", rootDirectory));
- }
- return this;
- }
-
- private LocalArtifactStagingLocation verifyExistence() {
- checkArgument(rootDirectory.exists(), "Nonexistent staging location root %s", rootDirectory);
- checkArgument(
- rootDirectory.isDirectory(), "Staging location %s is not a directory", rootDirectory);
- checkArgument(
- artifactsDirectory.exists(), "Nonexistent artifact directory %s", artifactsDirectory);
- checkArgument(
- artifactsDirectory.isDirectory(),
- "Artifact location %s is not a directory",
- artifactsDirectory);
- checkArgument(getManifestFile().exists(), "No Manifest in existing location %s", rootDirectory);
- return this;
- }
-
- /**
- * Returns the {@link File} which contains the artifact with the provided name.
- *
- * <p>The file may not exist.
- */
- public File getArtifactFile(String artifactName) {
- return new File(artifactsDirectory, artifactName);
- }
-
- /**
- * Returns the {@link File} which contains the {@link Manifest}.
- *
- * <p>The file may not exist.
- */
- public File getManifestFile() {
- return new File(rootDirectory, "MANIFEST");
- }
-
- /**
- * Returns the local location of this {@link LocalArtifactStagingLocation}.
- *
- * <p>This can be used to refer to the staging location when creating a retrieval service.
- */
- public String getRootPath() {
- try {
- return rootDirectory.getCanonicalPath();
- } catch (IOException e) {
- throw new IllegalStateException(e);
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java
deleted file mode 100644
index 41a3d8a..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java
+++ /dev/null
@@ -1,119 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.channels.FileChannel;
-import java.nio.channels.FileChannel.MapMode;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestResponse;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
-import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-
-/** An {@code ArtifactRetrievalService} which stages files to a local temp directory. */
-public class LocalFileSystemArtifactRetrievalService
- extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
- implements ArtifactRetrievalService {
- private static final int DEFAULT_CHUNK_SIZE = 2 * 1024 * 1024;
-
- public static LocalFileSystemArtifactRetrievalService forRootDirectory(File base) {
- return new LocalFileSystemArtifactRetrievalService(base);
- }
-
- private final LocalArtifactStagingLocation location;
- private final Manifest manifest;
-
- private LocalFileSystemArtifactRetrievalService(File rootDirectory) {
- this.location = LocalArtifactStagingLocation.forExistingDirectory(rootDirectory);
- try (FileInputStream manifestStream = new FileInputStream(location.getManifestFile())) {
- this.manifest = ArtifactApi.Manifest.parseFrom(manifestStream);
- } catch (FileNotFoundException e) {
- throw new IllegalArgumentException(
- String.format(
- "No %s in root directory %s", Manifest.class.getSimpleName(), rootDirectory),
- e);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- @Override
- public final void getManifest(
- ArtifactApi.GetManifestRequest request,
- StreamObserver<GetManifestResponse> responseObserver) {
- try {
- responseObserver.onNext(GetManifestResponse.newBuilder().setManifest(manifest).build());
- responseObserver.onCompleted();
- } catch (Exception e) {
- responseObserver.onError(Status.INTERNAL.withCause(e).asException());
- }
- }
-
- /** Get the artifact with the provided name as a sequence of bytes. */
- private ByteBuffer getArtifact(String name) throws IOException {
- File artifact = location.getArtifactFile(name);
- if (!artifact.exists()) {
- throw new FileNotFoundException(String.format("No such artifact %s", name));
- }
- FileChannel input = new FileInputStream(artifact).getChannel();
- return input.map(MapMode.READ_ONLY, 0L, input.size());
- }
-
- @Override
- public void getArtifact(
- ArtifactApi.GetArtifactRequest request,
- StreamObserver<ArtifactApi.ArtifactChunk> responseObserver) {
- try {
- ByteBuffer artifact = getArtifact(request.getName());
- do {
- responseObserver.onNext(
- ArtifactChunk.newBuilder()
- .setData(
- ByteString.copyFrom(
- artifact, Math.min(artifact.remaining(), DEFAULT_CHUNK_SIZE)))
- .build());
- } while (artifact.hasRemaining());
- responseObserver.onCompleted();
- } catch (FileNotFoundException e) {
- responseObserver.onError(
- Status.INVALID_ARGUMENT
- .withDescription(String.format("No such artifact %s", request.getName()))
- .withCause(e)
- .asException());
- } catch (Exception e) {
- responseObserver.onError(
- Status.INTERNAL
- .withDescription(
- String.format("Could not retrieve artifact with name %s", request.getName()))
- .withCause(e)
- .asException());
- }
- }
-
- @Override
- public void close() throws Exception {}
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
deleted file mode 100644
index 3df1f54..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
+++ /dev/null
@@ -1,282 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.OutputStream;
-import java.util.ArrayList;
-import java.util.Collection;
-import javax.annotation.Nullable;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
-import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc;
-import org.apache.beam.runners.fnexecution.FnService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * An {@code ArtifactStagingService} which stages files to a local temp directory. TODO: refactor to
- * use staging session tokens
- */
-public class LocalFileSystemArtifactStagerService
- extends ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase implements FnService {
- private static final Logger LOG =
- LoggerFactory.getLogger(LocalFileSystemArtifactStagerService.class);
-
- public static LocalFileSystemArtifactStagerService forRootDirectory(File base) {
- return new LocalFileSystemArtifactStagerService(base);
- }
-
- private final LocalArtifactStagingLocation location;
-
- private LocalFileSystemArtifactStagerService(File stagingBase) {
- this.location = LocalArtifactStagingLocation.createAt(stagingBase);
- }
-
- @Override
- public StreamObserver<ArtifactApi.PutArtifactRequest> putArtifact(
- final StreamObserver<ArtifactApi.PutArtifactResponse> responseObserver) {
- return new CreateAndWriteFileObserver(responseObserver);
- }
-
- @Override
- public void commitManifest(
- ArtifactApi.CommitManifestRequest request,
- StreamObserver<ArtifactApi.CommitManifestResponse> responseObserver) {
- try {
- commitManifestOrThrow(request, responseObserver);
- } catch (StatusRuntimeException e) {
- responseObserver.onError(e);
- LOG.error("Failed to commit Manifest {}", request.getManifest(), e);
- } catch (Exception e) {
- responseObserver.onError(
- Status.INTERNAL
- .withCause(e)
- .withDescription(Throwables.getStackTraceAsString(e))
- .asRuntimeException());
- LOG.error("Failed to commit Manifest {}", request.getManifest(), e);
- }
- }
-
- private void commitManifestOrThrow(
- ArtifactApi.CommitManifestRequest request,
- StreamObserver<ArtifactApi.CommitManifestResponse> responseObserver)
- throws IOException {
- Collection<ArtifactApi.ArtifactMetadata> missing = new ArrayList<>();
- for (ArtifactApi.ArtifactMetadata artifact : request.getManifest().getArtifactList()) {
- // TODO: Validate the checksums on the server side, to fail more aggressively if require
- if (!location.getArtifactFile(artifact.getName()).exists()) {
- missing.add(artifact);
- }
- }
- if (!missing.isEmpty()) {
- throw Status.INVALID_ARGUMENT
- .withDescription(
- String.format("Attempted to commit manifest with missing Artifacts: [%s]", missing))
- .asRuntimeException();
- }
- File mf = location.getManifestFile();
- checkState(mf.createNewFile(), "Could not create file to store manifest");
- try (OutputStream mfOut = new FileOutputStream(mf)) {
- request.getManifest().writeTo(mfOut);
- }
- responseObserver.onNext(
- ArtifactApi.CommitManifestResponse.newBuilder()
- .setRetrievalToken(location.getRootPath())
- .build());
- responseObserver.onCompleted();
- }
-
- @Override
- public void close() throws Exception {
- // TODO: Close all active staging calls, signalling errors to the caller.
- }
-
- @VisibleForTesting
- LocalArtifactStagingLocation getLocation() {
- return location;
- }
-
- private class CreateAndWriteFileObserver
- implements StreamObserver<ArtifactApi.PutArtifactRequest> {
- private final StreamObserver<ArtifactApi.PutArtifactResponse> responseObserver;
- private FileWritingObserver writer;
-
- private CreateAndWriteFileObserver(
- StreamObserver<ArtifactApi.PutArtifactResponse> responseObserver) {
- this.responseObserver = responseObserver;
- }
-
- @Override
- public void onNext(ArtifactApi.PutArtifactRequest value) {
- try {
- if (writer == null) {
- if (!value.getContentCase().equals(ArtifactApi.PutArtifactRequest.ContentCase.METADATA)) {
- throw Status.INVALID_ARGUMENT
- .withDescription(
- String.format(
- "Expected the first %s to contain the Artifact Name, got %s",
- ArtifactApi.PutArtifactRequest.class.getSimpleName(),
- value.getContentCase()))
- .asRuntimeException();
- }
- writer = createFile(value.getMetadata().getMetadata());
- } else {
- writer.onNext(value);
- }
- } catch (StatusRuntimeException e) {
- responseObserver.onError(e);
- } catch (Exception e) {
- responseObserver.onError(
- Status.INTERNAL
- .withCause(e)
- .withDescription(Throwables.getStackTraceAsString(e))
- .asRuntimeException());
- }
- }
-
- private FileWritingObserver createFile(ArtifactApi.ArtifactMetadata metadata)
- throws IOException {
- File destination = location.getArtifactFile(metadata.getName());
- if (!destination.createNewFile()) {
- throw Status.ALREADY_EXISTS
- .withDescription(String.format("Artifact with name %s already exists", metadata))
- .asRuntimeException();
- }
- return new FileWritingObserver(
- destination, new FileOutputStream(destination), responseObserver);
- }
-
- @Override
- public void onError(Throwable t) {
- if (writer != null) {
- writer.onError(t);
- } else {
- responseObserver.onCompleted();
- }
- }
-
- @Override
- public void onCompleted() {
- if (writer != null) {
- writer.onCompleted();
- } else {
- responseObserver.onCompleted();
- }
- }
- }
-
- private static class FileWritingObserver
- implements StreamObserver<ArtifactApi.PutArtifactRequest> {
- private final File destination;
- private final OutputStream target;
- private final StreamObserver<ArtifactApi.PutArtifactResponse> responseObserver;
-
- private FileWritingObserver(
- File destination,
- OutputStream target,
- StreamObserver<ArtifactApi.PutArtifactResponse> responseObserver) {
- this.destination = destination;
- this.target = target;
- this.responseObserver = responseObserver;
- }
-
- @Override
- public void onNext(ArtifactApi.PutArtifactRequest value) {
- try {
- if (value.getData() == null) {
- StatusRuntimeException e =
- Status.INVALID_ARGUMENT
- .withDescription(
- String.format(
- "Expected all chunks in the current stream state to contain data, got %s",
- value.getContentCase()))
- .asRuntimeException();
- throw e;
- }
- value.getData().getData().writeTo(target);
- } catch (Exception e) {
- cleanedUp(e);
- }
- }
-
- @Override
- public void onError(Throwable t) {
- if (cleanedUp(null)) {
- responseObserver.onCompleted();
- }
- }
-
- @Override
- public void onCompleted() {
- try {
- target.close();
- } catch (IOException e) {
- LOG.error("Failed to complete writing file {}", destination, e);
- cleanedUp(e);
- return;
- }
- responseObserver.onNext(ArtifactApi.PutArtifactResponse.getDefaultInstance());
- responseObserver.onCompleted();
- }
-
- /**
- * Cleans up after the file writing failed exceptionally, due to an error either in the service
- * or sent from the client.
- *
- * @return false if an error was reported, true otherwise
- */
- private boolean cleanedUp(@Nullable Throwable whyFailed) {
- Throwable actual = whyFailed;
- try {
- target.close();
- if (!destination.delete()) {
- LOG.debug("Couldn't delete failed write at {}", destination);
- }
- } catch (IOException e) {
- if (whyFailed == null) {
- actual = e;
- } else {
- actual.addSuppressed(e);
- }
- LOG.error("Failed to clean up after writing file {}", destination, e);
- }
- if (actual != null) {
- if (actual instanceof StatusException || actual instanceof StatusRuntimeException) {
- responseObserver.onError(actual);
- } else {
- Status status =
- Status.INTERNAL
- .withCause(actual)
- .withDescription(Throwables.getStackTraceAsString(actual));
- responseObserver.onError(status.asException());
- }
- }
- return actual == null;
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalService.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalService.java
deleted file mode 100644
index f0d56ba..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalService.java
+++ /dev/null
@@ -1,42 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
-import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
-
-/**
- * An {@link ArtifactRetrievalService} which has not implemented any methods.
- *
- * <p>For use with an in-process SDK harness.
- */
-public class UnsupportedArtifactRetrievalService
- extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
- implements ArtifactRetrievalService {
-
- public static ArtifactRetrievalService create() {
- return new UnsupportedArtifactRetrievalService();
- }
-
- private UnsupportedArtifactRetrievalService() {}
-
- @Override
- public void close() {
- // Do nothing.
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/package-info.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/package-info.java
deleted file mode 100644
index 1ec0da5..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/package-info.java
+++ /dev/null
@@ -1,20 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/** Provides local implementations of the Artifact API services. */
-package org.apache.beam.runners.direct.portable.artifact;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/PreparingJob.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/PreparingJob.java
deleted file mode 100644
index 077a517..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/PreparingJob.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.job;
-
-import com.google.auto.value.AutoValue;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-
-/** A Job with a {@code prepare} call but no corresponding {@code run} call. */
-@AutoValue
-abstract class PreparingJob implements AutoCloseable {
- public static Builder builder() {
- return new AutoValue_PreparingJob.Builder();
- }
-
- abstract Pipeline getPipeline();
-
- abstract Struct getOptions();
-
- abstract String getStagingSessionToken();
-
- abstract GrpcFnServer<BeamFileSystemArtifactStagingService> getArtifactStagingServer();
-
- @Override
- public void close() throws Exception {
- getArtifactStagingServer().close();
- }
-
- @AutoValue.Builder
- public abstract static class Builder {
- abstract Builder setPipeline(Pipeline pipeline);
-
- abstract Builder setOptions(Struct options);
-
- abstract Builder setStagingSessionToken(String stagingSessionToken);
-
- abstract Builder setArtifactStagingServer(
- GrpcFnServer<BeamFileSystemArtifactStagingService> server);
-
- abstract PreparingJob build();
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
deleted file mode 100644
index e2eac87..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
+++ /dev/null
@@ -1,164 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.job;
-
-import java.io.IOException;
-import java.nio.file.Paths;
-import java.util.Arrays;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.kohsuke.args4j.CmdLineException;
-import org.kohsuke.args4j.CmdLineParser;
-import org.kohsuke.args4j.Option;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** A program that runs a {@link ReferenceRunnerJobService}. */
-public class ReferenceRunnerJobServer {
- private static final Logger LOG = LoggerFactory.getLogger(ReferenceRunnerJobServer.class);
- private final ServerConfiguration configuration;
- private GrpcFnServer<ReferenceRunnerJobService> server;
-
- private ReferenceRunnerJobServer(ServerConfiguration configuration) {
- this.configuration = configuration;
- }
-
- public static void main(String[] args) throws Exception {
- try {
- runServer(parseConfiguration(args));
- } catch (CmdLineException ignored) {
- }
- }
-
- private static ServerConfiguration parseConfiguration(String[] args) throws CmdLineException {
- ServerConfiguration configuration = new ServerConfiguration();
- CmdLineParser parser = new CmdLineParser(configuration);
- try {
- parser.parseArgument(args);
- } catch (CmdLineException e) {
- e.printStackTrace(System.err);
- printUsage(parser);
- throw e;
- }
- return configuration;
- }
-
- private static void printUsage(CmdLineParser parser) {
- System.err.println(
- String.format(
- "Usage: java %s arguments...", ReferenceRunnerJobService.class.getSimpleName()));
- parser.printUsage(System.err);
- System.err.println();
- }
-
- private static void runServer(ServerConfiguration configuration) throws Exception {
- ServerFactory serverFactory = ServerFactory.createDefault();
- ReferenceRunnerJobService.Configuration jobServiceConfig =
- createJobServiceConfig(configuration);
- ReferenceRunnerJobService service =
- ReferenceRunnerJobService.create(serverFactory, jobServiceConfig);
- try (GrpcFnServer<ReferenceRunnerJobService> server =
- createServer(configuration, serverFactory, service)) {
- System.out.println(
- String.format(
- "Started %s at %s",
- ReferenceRunnerJobService.class.getSimpleName(),
- server.getApiServiceDescriptor().getUrl()));
- server.getServer().awaitTermination();
- }
- System.out.println("Server shut down, exiting");
- }
-
- public static ReferenceRunnerJobServer fromParams(String[] args) {
- try {
- return new ReferenceRunnerJobServer(parseConfiguration(args));
- } catch (CmdLineException e) {
- throw new IllegalArgumentException(
- "Unable to parse command line arguments " + Arrays.asList(args), e);
- }
- }
-
- public String start() throws Exception {
- ServerFactory serverFactory = ServerFactory.createDefault();
- ReferenceRunnerJobService.Configuration jobServiceConfig =
- createJobServiceConfig(configuration);
- server =
- createServer(
- configuration,
- serverFactory,
- ReferenceRunnerJobService.create(serverFactory, jobServiceConfig));
-
- return server.getApiServiceDescriptor().getUrl();
- }
-
- public void stop() {
- if (server != null) {
- try {
- server.close();
- } catch (Exception e) {
- LOG.error("Unable to stop job server.", e);
- }
- }
- }
-
- private static GrpcFnServer<ReferenceRunnerJobService> createServer(
- ServerConfiguration configuration,
- ServerFactory serverFactory,
- ReferenceRunnerJobService service)
- throws IOException {
- if (configuration.port <= 0) {
- return GrpcFnServer.allocatePortAndCreateFor(service, serverFactory);
- }
- return GrpcFnServer.create(
- service,
- ApiServiceDescriptor.newBuilder().setUrl("localhost:" + configuration.port).build(),
- serverFactory);
- }
-
- /**
- * Helper function to fill out a {@code ReferenceRunnerJobService.Configuration Configuration}
- * object for {@code ReferenceRunnerJobService}.
- */
- private static ReferenceRunnerJobService.Configuration createJobServiceConfig(
- ServerConfiguration configuration) {
- ReferenceRunnerJobService.Configuration jobServiceConfig =
- new ReferenceRunnerJobService.Configuration();
- jobServiceConfig.artifactStagingPath = configuration.artifactStagingPath;
- jobServiceConfig.keepArtifacts = configuration.keepArtifacts;
- return jobServiceConfig;
- }
-
- /** Command-line options to configure the JobServer. */
- public static class ServerConfiguration {
- @Option(
- name = "-p",
- aliases = {"--port"},
- usage = "The local port to expose the server on. 0 to use a dynamic port. (Default: 8099)")
- private int port = 8099;
-
- @Option(name = "--artifacts-dir", usage = "The location to store staged artifact files")
- String artifactStagingPath =
- Paths.get(System.getProperty("java.io.tmpdir"), "beam-artifact-staging").toString();
-
- @Option(
- name = "--keep-artifacts",
- usage = "When enabled, do not delete staged artifacts when a job completes")
- boolean keepArtifacts;
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
deleted file mode 100644
index 07517af..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
+++ /dev/null
@@ -1,308 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.job;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ThreadLocalRandom;
-import org.apache.beam.model.jobmanagement.v1.JobApi;
-import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
-import org.apache.beam.model.jobmanagement.v1.JobApi.JobState.Enum;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase;
-import org.apache.beam.runners.direct.portable.ReferenceRunner;
-import org.apache.beam.runners.fnexecution.FnService;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * This JobService implements the grpc calls for running jobs by using the {@code ReferenceRunner}
- * as an engine.
- */
-public class ReferenceRunnerJobService extends JobServiceImplBase implements FnService {
-
- /** A configuration object for constructing the {@code ReferenceRunnerJobService}. */
- public static class Configuration {
- public String artifactStagingPath;
- public boolean keepArtifacts;
- }
-
- private static final Logger LOG = LoggerFactory.getLogger(ReferenceRunnerJobService.class);
- private static final int WAIT_MS = 1000;
-
- public static ReferenceRunnerJobService create(
- final ServerFactory serverFactory, Configuration configuration) {
- LOG.info("Starting {}", ReferenceRunnerJobService.class);
- return new ReferenceRunnerJobService(serverFactory, configuration);
- }
-
- private final ServerFactory serverFactory;
- private final Configuration configuration;
-
- private final ConcurrentMap<String, PreparingJob> unpreparedJobs;
- private final ConcurrentMap<String, ReferenceRunner> runningJobs;
- private final ConcurrentMap<String, JobState.Enum> jobStates;
- private final ExecutorService executor;
- private final ConcurrentLinkedQueue<GrpcFnServer<BeamFileSystemArtifactStagingService>>
- artifactStagingServices;
-
- private ReferenceRunnerJobService(ServerFactory serverFactory, Configuration configuration) {
- this.serverFactory = serverFactory;
- this.configuration = configuration;
- unpreparedJobs = new ConcurrentHashMap<>();
- runningJobs = new ConcurrentHashMap<>();
- jobStates = new ConcurrentHashMap<>();
- executor =
- Executors.newCachedThreadPool(
- new ThreadFactoryBuilder()
- .setDaemon(false)
- .setNameFormat("reference-runner-pipeline-%s")
- .build());
- artifactStagingServices = new ConcurrentLinkedQueue<>();
- }
-
- @Override
- public void prepare(
- JobApi.PrepareJobRequest request,
- StreamObserver<JobApi.PrepareJobResponse> responseObserver) {
- try {
- LOG.trace("{} {}", PrepareJobResponse.class.getSimpleName(), request);
-
- String preparationId = request.getJobName() + ThreadLocalRandom.current().nextInt();
- GrpcFnServer<BeamFileSystemArtifactStagingService> artifactStagingService =
- createArtifactStagingService();
- artifactStagingServices.add(artifactStagingService);
- String stagingSessionToken =
- BeamFileSystemArtifactStagingService.generateStagingSessionToken(
- preparationId, configuration.artifactStagingPath);
- PreparingJob existingJob =
- unpreparedJobs.putIfAbsent(
- preparationId,
- PreparingJob.builder()
- .setArtifactStagingServer(artifactStagingService)
- .setPipeline(request.getPipeline())
- .setOptions(request.getPipelineOptions())
- .setStagingSessionToken(stagingSessionToken)
- .build());
- checkArgument(
- existingJob == null, "Unexpected existing job with preparation ID %s", preparationId);
-
- responseObserver.onNext(
- PrepareJobResponse.newBuilder()
- .setPreparationId(preparationId)
- .setArtifactStagingEndpoint(artifactStagingService.getApiServiceDescriptor())
- .setStagingSessionToken(stagingSessionToken)
- .build());
- responseObserver.onCompleted();
- } catch (Exception e) {
- LOG.error("Could not prepare job with name {}", request.getJobName(), e);
- responseObserver.onError(Status.INTERNAL.withCause(e).asException());
- }
- }
-
- private GrpcFnServer<BeamFileSystemArtifactStagingService> createArtifactStagingService()
- throws Exception {
- BeamFileSystemArtifactStagingService service = new BeamFileSystemArtifactStagingService();
- return GrpcFnServer.allocatePortAndCreateFor(service, serverFactory);
- }
-
- @Override
- @SuppressWarnings("FutureReturnValueIgnored") // Run API does not block on execution
- public void run(
- JobApi.RunJobRequest request, StreamObserver<JobApi.RunJobResponse> responseObserver) {
- try {
- LOG.trace("{} {}", RunJobRequest.class.getSimpleName(), request);
- String preparationId = request.getPreparationId();
- PreparingJob preparingJob = unpreparedJobs.get(preparationId);
- if (preparingJob == null) {
- responseObserver.onError(
- Status.INVALID_ARGUMENT
- .withDescription(String.format("Unknown Preparation Id %s", preparationId))
- .asException());
- return;
- }
- try {
- // Close any preparation-time only resources.
- preparingJob.close();
- } catch (Exception e) {
- responseObserver.onError(e);
- }
-
- ReferenceRunner runner =
- ReferenceRunner.forPipeline(
- preparingJob.getPipeline(), preparingJob.getOptions(), request.getRetrievalToken());
- String jobId = "job-" + Integer.toString(ThreadLocalRandom.current().nextInt());
- responseObserver.onNext(RunJobResponse.newBuilder().setJobId(jobId).build());
- responseObserver.onCompleted();
- runningJobs.put(jobId, runner);
- jobStates.putIfAbsent(jobId, Enum.RUNNING);
- executor.submit(
- () -> {
- try {
- jobStates.computeIfPresent(jobId, (id, status) -> Enum.RUNNING);
- runner.execute();
- jobStates.computeIfPresent(jobId, (id, status) -> Enum.DONE);
- } catch (Exception e) {
- jobStates.computeIfPresent(jobId, (id, status) -> Enum.FAILED);
- throw e;
- }
-
- // Delete artifacts after job is done.
- if (!configuration.keepArtifacts) {
- String stagingSessionToken = preparingJob.getStagingSessionToken();
- try {
- preparingJob
- .getArtifactStagingServer()
- .getService()
- .removeArtifacts(stagingSessionToken);
- } catch (Exception e) {
- LOG.error(
- "Failed to remove job staging directory for token {}: {}",
- stagingSessionToken,
- e);
- }
- }
- return null;
- });
- } catch (StatusRuntimeException e) {
- responseObserver.onError(e);
- } catch (Exception e) {
- responseObserver.onError(Status.INTERNAL.withCause(e).asException());
- }
- }
-
- @Override
- public void getState(
- GetJobStateRequest request, StreamObserver<GetJobStateResponse> responseObserver) {
- LOG.trace("{} {}", GetJobStateRequest.class.getSimpleName(), request);
- try {
- responseObserver.onNext(
- GetJobStateResponse.newBuilder()
- .setState(jobStates.getOrDefault(request.getJobId(), Enum.UNRECOGNIZED))
- .build());
- responseObserver.onCompleted();
- } catch (Exception e) {
- String errMessage =
- String.format("Encountered Unexpected Exception for Invocation %s", request.getJobId());
- LOG.error(errMessage, e);
- responseObserver.onError(Status.INTERNAL.withCause(e).asException());
- }
- }
-
- @Override
- public void getStateStream(
- GetJobStateRequest request, StreamObserver<GetJobStateResponse> responseObserver) {
- LOG.trace("{} {}", GetJobStateRequest.class.getSimpleName(), request);
- String invocationId = request.getJobId();
- try {
- Thread.sleep(WAIT_MS);
- Enum state = jobStates.getOrDefault(request.getJobId(), Enum.UNRECOGNIZED);
- responseObserver.onNext(GetJobStateResponse.newBuilder().setState(state).build());
- while (Enum.RUNNING.equals(state)) {
- Thread.sleep(WAIT_MS);
- state = jobStates.getOrDefault(request.getJobId(), Enum.UNRECOGNIZED);
- }
- responseObserver.onNext(GetJobStateResponse.newBuilder().setState(state).build());
- } catch (Exception e) {
- String errMessage =
- String.format("Encountered Unexpected Exception for Invocation %s", invocationId);
- LOG.error(errMessage, e);
- responseObserver.onError(Status.INTERNAL.withCause(e).asException());
- }
- responseObserver.onCompleted();
- }
-
- @Override
- public void describePipelineOptions(
- JobApi.DescribePipelineOptionsRequest request,
- StreamObserver<JobApi.DescribePipelineOptionsResponse> responseObserver) {
- LOG.trace("{} {}", JobApi.DescribePipelineOptionsRequest.class.getSimpleName(), request);
- try {
- JobApi.DescribePipelineOptionsResponse response =
- JobApi.DescribePipelineOptionsResponse.newBuilder()
- .addAllOptions(
- PipelineOptionsFactory.describe(PipelineOptionsFactory.getRegisteredOptions()))
- .build();
- responseObserver.onNext(response);
- responseObserver.onCompleted();
- } catch (Exception e) {
- LOG.error("Error describing pipeline options", e);
- responseObserver.onError(Status.INTERNAL.withCause(e).asException());
- }
- }
-
- @Override
- public void getMessageStream(
- JobMessagesRequest request, StreamObserver<JobMessagesResponse> responseObserver) {
- // Not implemented
- LOG.trace("{} {}", JobMessagesRequest.class.getSimpleName(), request);
- }
-
- @Override
- public void cancel(CancelJobRequest request, StreamObserver<CancelJobResponse> responseObserver) {
- LOG.trace("{} {}", CancelJobRequest.class.getSimpleName(), request);
- responseObserver.onError(
- Status.NOT_FOUND
- .withDescription(String.format("Unknown Job ID %s", request.getJobId()))
- .asException());
- }
-
- @Override
- public void close() throws Exception {
- for (PreparingJob preparingJob : ImmutableList.copyOf(unpreparedJobs.values())) {
- try {
- preparingJob.close();
- } catch (Exception e) {
- LOG.warn("Exception while closing preparing job {}", preparingJob);
- }
- }
- while (!artifactStagingServices.isEmpty()) {
- GrpcFnServer<BeamFileSystemArtifactStagingService> artifactStagingService =
- artifactStagingServices.remove();
- try {
- artifactStagingService.close();
- } catch (Exception e) {
- LOG.error(
- "Unable to close staging sevice started on %s",
- artifactStagingService.getApiServiceDescriptor().getUrl(), e);
- }
- }
- }
-}
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/package-info.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/package-info.java
deleted file mode 100644
index 9de085b..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/package-info.java
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * An execution engine for Beam Pipelines that uses the Java Runner harness and the Fn API to
- * execute.
- */
-package org.apache.beam.runners.direct.portable.job;
diff --git a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/package-info.java b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/package-info.java
deleted file mode 100644
index 1a51b2f..0000000
--- a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-/**
- * Defines the {@link org.apache.beam.sdk.options.PipelineOptions.DirectRunner} which executes both
- * Bounded and Unbounded {@code Pipelines} on the local machine.
- *
- * <p>See {@link org.apache.beam.sdk.runners} for more information about Pipeline Runners.
- */
-package org.apache.beam.runners.direct.portable;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
index 17ad8fc..52334cf 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
@@ -55,8 +55,8 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
index fdbc5f3..5d36b92 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CloningBundleFactoryTest.java
@@ -44,8 +44,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
index cc0ebad..16cb694 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CommittedResultTest.java
@@ -23,6 +23,7 @@
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Optional;
import org.apache.beam.runners.direct.CommittedResult.OutputType;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.runners.AppliedPTransform;
@@ -34,8 +35,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Rule;
@@ -70,7 +70,7 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
Collections.emptyList(),
EnumSet.noneOf(OutputType.class));
@@ -99,11 +99,11 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
Collections.emptyList(),
EnumSet.noneOf(OutputType.class));
- assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.absent()));
+ assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.empty()));
}
@Test
@@ -129,7 +129,7 @@
CommittedResult<AppliedPTransform<?, ?, ?>> result =
CommittedResult.create(
StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
+ Optional.empty(),
outputs,
EnumSet.of(OutputType.BUNDLE, OutputType.PCOLLECTION_VIEW));
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternalsTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternalsTest.java
index 0e5305d..5e53d58 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternalsTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/CopyOnAccessInMemoryStateInternalsTest.java
@@ -50,7 +50,7 @@
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
index 120dfd6..c7536aa 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
@@ -43,7 +43,7 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectMetricsTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectMetricsTest.java
index a7c8c3a..7957d52 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectMetricsTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectMetricsTest.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.metrics.MetricQueryResults;
import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.After;
import org.junit.Before;
@@ -51,7 +51,6 @@
public class DirectMetricsTest {
@Mock private CommittedBundle<Object> bundle1;
- @Mock private CommittedBundle<Object> bundle2;
private static final MetricName NAME1 = MetricName.named("ns1", "name1");
private static final MetricName NAME2 = MetricName.named("ns1", "name2");
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRegistrarTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRegistrarTest.java
index b8c8696..d7275ec 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRegistrarTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRegistrarTest.java
@@ -25,8 +25,8 @@
import org.apache.beam.runners.direct.DirectRegistrar.Runner;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerApiSurfaceTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerApiSurfaceTest.java
index 06c8781..a33f5bc 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerApiSurfaceTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerApiSurfaceTest.java
@@ -21,7 +21,6 @@
import static org.junit.Assert.assertThat;
import java.util.Set;
-import org.apache.beam.runners.direct.portable.ExecutableGraphBuilder;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.metrics.MetricResults;
@@ -29,7 +28,7 @@
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.util.ApiSurface;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -65,7 +64,6 @@
.pruningClass(DirectGraphs.class)
.pruningClass(
WatermarkManager.class /* TODO: BEAM-4237 Consider moving to local-java */)
- .pruningClass(ExecutableGraphBuilder.class)
.pruningPattern(
"org[.]apache[.]beam[.]runners[.]direct[.]portable.*"
/* TODO: BEAM-4237 reconsider package layout with the ReferenceRunner */ )
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
index 2d951ef..9cc5a87 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectRunnerTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.is;
@@ -77,7 +77,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.junit.Rule;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
index ed35aad..b28333b 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTransformExecutorTest.java
@@ -30,6 +30,7 @@
import java.util.Collections;
import java.util.EnumSet;
import java.util.List;
+import java.util.Optional;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
@@ -42,9 +43,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
@@ -414,7 +414,7 @@
Optional<? extends CommittedBundle<?>> unprocessedBundle;
if (inputBundle == null || Iterables.isEmpty(unprocessedElements)) {
- unprocessedBundle = Optional.absent();
+ unprocessedBundle = Optional.empty();
} else {
unprocessedBundle =
Optional.<CommittedBundle<?>>of(inputBundle.withElements(unprocessedElements));
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerRemovingTransformEvaluatorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerRemovingTransformEvaluatorTest.java
index 0df2023..da93882 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerRemovingTransformEvaluatorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerRemovingTransformEvaluatorTest.java
@@ -72,7 +72,9 @@
@Test
public void removesOnExceptionInProcessElement() throws Exception {
ParDoEvaluator<Object> underlying = mock(ParDoEvaluator.class);
- doThrow(Exception.class).when(underlying).processElement(any(WindowedValue.class));
+ doThrow(IllegalArgumentException.class)
+ .when(underlying)
+ .processElement(any(WindowedValue.class));
DoFn<?, ?> original = lifecycleManager.get();
assertThat(original, not(nullValue()));
@@ -91,7 +93,7 @@
@Test
public void removesOnExceptionInOnTimer() throws Exception {
ParDoEvaluator<Object> underlying = mock(ParDoEvaluator.class);
- doThrow(Exception.class)
+ doThrow(IllegalArgumentException.class)
.when(underlying)
.onTimer(any(TimerData.class), any(BoundedWindow.class));
@@ -114,7 +116,7 @@
@Test
public void removesOnExceptionInFinishBundle() throws Exception {
ParDoEvaluator<Object> underlying = mock(ParDoEvaluator.class);
- doThrow(Exception.class).when(underlying).finishBundle();
+ doThrow(IllegalArgumentException.class).when(underlying).finishBundle();
DoFn<?, ?> original = lifecycleManager.get();
// the LifecycleManager is set when the evaluator starts
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerTest.java
index 1adbb7e..1114511 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagerTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.isA;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagersTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagersTest.java
index 0447220..dfa24af 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagersTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DoFnLifecycleManagersTest.java
@@ -25,7 +25,7 @@
import java.util.Collection;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.util.UserCodeException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
index f4322ee..b2262af 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/EvaluationContextTest.java
@@ -65,8 +65,8 @@
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Rule;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutorTest.java
index b70ec51..274550a 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutorTest.java
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.testing.ThreadLeakTracker;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.LinkedListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.LinkedListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.joda.time.Instant;
import org.junit.Ignore;
import org.junit.Rule;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
index f4a6328..ba9e7bc 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
index 818716c..a9c6524 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyEvaluatorFactoryTest.java
@@ -34,9 +34,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multiset;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
index 89338b9..9847dae 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/GroupByKeyOnlyEvaluatorFactoryTest.java
@@ -34,9 +34,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multiset;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
index be6969a..21d6e65 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImmutableListBundleFactoryTest.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matcher;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactoryTest.java
index aaec082..3a1e5b6 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ImpulseEvaluatorFactoryTest.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/MockClock.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/MockClock.java
index a39d0cf..faa5a40 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/MockClock.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/MockClock.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.direct;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
index 672bad2..90be7fd 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
@@ -25,6 +25,7 @@
import java.util.ArrayList;
import java.util.Collection;
+import java.util.Collections;
import java.util.List;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.ReadyCheckingSideInputReader;
@@ -50,9 +51,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
@@ -161,6 +162,7 @@
additionalOutputTags,
ImmutableMap.of(mainOutputTag, output),
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
ParDoEvaluator.defaultRunnerFactory());
}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/SideInputContainerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/SideInputContainerTest.java
index 1d18fdc..0c29a60 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/SideInputContainerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/SideInputContainerTest.java
@@ -48,9 +48,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListenableFuture;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListenableFuture;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Rule;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
index 57aff5b..171d9dd 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
@@ -21,8 +21,8 @@
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.nullValue;
import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Matchers.anyList;
-import static org.mockito.Matchers.anyString;
import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -68,8 +68,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
@@ -142,7 +142,8 @@
mainOutput,
TupleTagList.empty(),
Collections.emptyList(),
- DoFnSchemaInformation.create()))
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap()))
.get(mainOutput)
.setCoder(VarIntCoder.of());
@@ -159,7 +160,7 @@
when(mockEvaluationContext.getExecutionContext(
eq(producingTransform), Mockito.<StructuralKey>any()))
.thenReturn(mockExecutionContext);
- when(mockExecutionContext.getStepContext(anyString())).thenReturn(mockStepContext);
+ when(mockExecutionContext.getStepContext(any())).thenReturn(mockStepContext);
IntervalWindow firstWindow = new IntervalWindow(new Instant(0), new Instant(9));
IntervalWindow secondWindow = new IntervalWindow(new Instant(10), new Instant(19));
@@ -251,7 +252,8 @@
mainOutput,
TupleTagList.empty(),
Collections.singletonList(sideInput),
- DoFnSchemaInformation.create()))
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap()))
.get(mainOutput)
.setCoder(VarIntCoder.of());
@@ -269,7 +271,7 @@
when(mockEvaluationContext.getExecutionContext(
eq(producingTransform), Mockito.<StructuralKey>any()))
.thenReturn(mockExecutionContext);
- when(mockExecutionContext.getStepContext(anyString())).thenReturn(mockStepContext);
+ when(mockExecutionContext.getStepContext(any())).thenReturn(mockStepContext);
when(mockEvaluationContext.createBundle(Matchers.<PCollection<Integer>>any()))
.thenReturn(mockUncommittedBundle);
when(mockStepContext.getTimerUpdate()).thenReturn(TimerUpdate.empty());
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
index 269be29..5def3b8 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorServicesTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorServicesTest.java
index 2dcd998..4105616 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorServicesTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TransformExecutorServicesTest.java
@@ -22,7 +22,7 @@
import static org.mockito.Mockito.verify;
import java.util.concurrent.ExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
index b057745..3575f47 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
@@ -31,10 +31,10 @@
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.runners.direct.UnboundedReadDeduplicator.CachedIdDeduplicator;
import org.apache.beam.runners.direct.UnboundedReadDeduplicator.NeverDeduplicator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListenableFuture;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListenableFuture;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
index 983cbde..a45a563 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactoryTest.java
@@ -21,7 +21,7 @@
import static java.util.Collections.emptySet;
import static java.util.Collections.singletonMap;
import static org.apache.beam.runners.direct.DirectGraphs.getProducer;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasSize;
@@ -76,12 +76,12 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ContiguousSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.DiscreteDomain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.LinkedListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Range;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ContiguousSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.DiscreteDomain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.LinkedListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Range;
import org.hamcrest.Matchers;
import org.joda.time.DateTime;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ViewEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ViewEvaluatorFactoryTest.java
index 3c4ab31..3ebe978 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ViewEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ViewEvaluatorFactoryTest.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
index d253734..54a5ff6 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
@@ -65,9 +65,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
index 59b1064..c9cb86c 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
@@ -43,9 +43,9 @@
import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WriteWithShardingFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WriteWithShardingFactoryTest.java
index 684ac02..dd6dc4d 100644
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WriteWithShardingFactoryTest.java
+++ b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WriteWithShardingFactoryTest.java
@@ -61,7 +61,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactoryTest.java
deleted file mode 100644
index 0236aef..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactoryTest.java
+++ /dev/null
@@ -1,262 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.fail;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collection;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components.Builder;
-import org.apache.beam.runners.core.construction.Environments;
-import org.apache.beam.runners.core.construction.RehydratedComponents;
-import org.apache.beam.runners.core.construction.SdkComponents;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
-import org.apache.beam.runners.fnexecution.wire.WireCoders;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.windowing.FixedWindows;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link BundleFactoryOutputReceiverFactory}. */
-@RunWith(JUnit4.class)
-public class BundleFactoryOutputReceiverFactoryTest {
- private final BundleFactory bundleFactory = ImmutableListBundleFactory.create();
- private PCollectionNode fooPC;
- private PCollectionNode barPC;
- private RunnerApi.Components baseComponents;
-
- private OutputReceiverFactory factory;
- private Collection<UncommittedBundle<?>> outputBundles;
-
- @Before
- public void setup() throws IOException {
- Pipeline p = Pipeline.create();
- PCollection<String> foo =
- p.apply("createFoo", Create.of("1", "2", "3"))
- .apply("windowFoo", Window.into(FixedWindows.of(Duration.standardMinutes(5L))));
- PCollection<Integer> bar = p.apply("bar", Create.of(1, 2, 3));
-
- SdkComponents sdkComponents = SdkComponents.create();
- sdkComponents.registerEnvironment(Environments.createDockerEnvironment("java"));
- String fooId = sdkComponents.registerPCollection(foo);
- String barId = sdkComponents.registerPCollection(bar);
- baseComponents = sdkComponents.toComponents();
-
- fooPC = PipelineNode.pCollection(fooId, baseComponents.getPcollectionsOrThrow(fooId));
- barPC = PipelineNode.pCollection(barId, baseComponents.getPcollectionsOrThrow(barId));
-
- outputBundles = new ArrayList<>();
- factory =
- BundleFactoryOutputReceiverFactory.create(
- bundleFactory, baseComponents, outputBundles::add);
- }
-
- @Test
- public void addsBundlesToResult() {
- factory.create(fooPC.getId());
- factory.create(barPC.getId());
-
- assertThat(Iterables.size(outputBundles), equalTo(2));
-
- Collection<PCollectionNode> pcollections = new ArrayList<>();
- for (UncommittedBundle<?> bundle : outputBundles) {
- pcollections.add(bundle.getPCollection());
- }
- assertThat(pcollections, containsInAnyOrder(fooPC, barPC));
- }
-
- @Test
- public void receiverAddsElementsToBundle() throws Exception {
- FnDataReceiver<WindowedValue<byte[]>> receiver = factory.create(fooPC.getId());
-
- Builder builder = baseComponents.toBuilder();
- String sdkWireCoderId = WireCoders.addSdkWireCoder(fooPC, builder);
- Components components = builder.build();
-
- Coder<WindowedValue<String>> sdkCoder =
- (Coder<WindowedValue<String>>)
- RehydratedComponents.forComponents(components).getCoder(sdkWireCoderId);
- Coder<WindowedValue<byte[]>> runnerCoder =
- WireCoders.instantiateRunnerWireCoder(fooPC, components);
-
- WindowedValue<byte[]> firstElem =
- CoderUtils.decodeFromByteArray(
- runnerCoder,
- CoderUtils.encodeToByteArray(
- sdkCoder,
- WindowedValue.of(
- "1",
- new Instant(120),
- new IntervalWindow(new Instant(0), Duration.standardMinutes(5)),
- PaneInfo.NO_FIRING)));
- WindowedValue<byte[]> secondElem =
- CoderUtils.decodeFromByteArray(
- runnerCoder,
- CoderUtils.encodeToByteArray(
- sdkCoder,
- WindowedValue.of(
- "2",
- new Instant(240),
- new IntervalWindow(new Instant(0), Duration.standardMinutes(5)),
- PaneInfo.NO_FIRING)));
- receiver.accept(firstElem);
- receiver.accept(secondElem);
-
- CommittedBundle<?> output = getOnlyElement(outputBundles).commit(Instant.now());
- assertThat(output, containsInAnyOrder(firstElem, secondElem));
- }
-
- /**
- * Tests that if a {@link
- * org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode} is provided
- * multiple times, the returned {@link
- * org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver} instances are independent.
- */
- @Test
- public void multipleInstancesOfPCollectionIndependent() throws Exception {
- FnDataReceiver<WindowedValue<byte[]>> firstReceiver = factory.create(fooPC.getId());
- FnDataReceiver<WindowedValue<byte[]>> secondReceiver = factory.create(fooPC.getId());
-
- Components.Builder builder = baseComponents.toBuilder();
- String sdkWireCoderId = WireCoders.addSdkWireCoder(fooPC, builder);
- Components components = builder.build();
-
- Coder<WindowedValue<String>> sdkCoder =
- (Coder<WindowedValue<String>>)
- RehydratedComponents.forComponents(components).getCoder(sdkWireCoderId);
-
- Coder<WindowedValue<byte[]>> runnerCoder =
- WireCoders.instantiateRunnerWireCoder(fooPC, components);
-
- WindowedValue<byte[]> firstElem =
- CoderUtils.decodeFromByteArray(
- runnerCoder,
- CoderUtils.encodeToByteArray(
- sdkCoder,
- WindowedValue.of(
- "1",
- new Instant(120),
- new IntervalWindow(new Instant(0), Duration.standardMinutes(5)),
- PaneInfo.NO_FIRING)));
- firstReceiver.accept(firstElem);
-
- WindowedValue<byte[]> secondElem =
- CoderUtils.decodeFromByteArray(
- runnerCoder,
- CoderUtils.encodeToByteArray(
- sdkCoder,
- WindowedValue.of(
- "2",
- new Instant(240),
- new IntervalWindow(new Instant(0), Duration.standardMinutes(5)),
- PaneInfo.NO_FIRING)));
- secondReceiver.accept(secondElem);
-
- Collection<WindowedValue<?>> outputs = new ArrayList<>();
- for (UncommittedBundle<?> uncommitted : outputBundles) {
- assertThat(uncommitted.getPCollection(), equalTo(fooPC));
- Iterable<? extends WindowedValue<?>> elements =
- uncommitted.commit(Instant.now()).getElements();
- Iterables.addAll(outputs, elements);
- assertThat(Iterables.size(elements), equalTo(1));
- }
- assertThat(outputs, containsInAnyOrder(firstElem, secondElem));
- }
-
- @Test
- public void differentPCollectionsIndependent() throws Exception {
- FnDataReceiver<WindowedValue<byte[]>> fooReceiver = factory.create(fooPC.getId());
-
- Components.Builder builder = baseComponents.toBuilder();
- String sdkWireCoderId = WireCoders.addSdkWireCoder(fooPC, builder);
- String barSdkWireCoderId = WireCoders.addSdkWireCoder(barPC, builder);
- Components components = builder.build();
-
- Coder<WindowedValue<String>> fooSdkCoder =
- (Coder<WindowedValue<String>>)
- RehydratedComponents.forComponents(components).getCoder(sdkWireCoderId);
- Coder<WindowedValue<byte[]>> fooRunnerCoder =
- WireCoders.instantiateRunnerWireCoder(fooPC, components);
-
- FnDataReceiver<WindowedValue<byte[]>> barReceiver = factory.create(barPC.getId());
- Coder<WindowedValue<Integer>> barSdkCoder =
- (Coder<WindowedValue<Integer>>)
- RehydratedComponents.forComponents(components).getCoder(barSdkWireCoderId);
- Coder<WindowedValue<byte[]>> barRunnerCoder =
- WireCoders.instantiateRunnerWireCoder(barPC, components);
-
- WindowedValue<byte[]> fooElem =
- CoderUtils.decodeFromByteArray(
- fooRunnerCoder,
- CoderUtils.encodeToByteArray(
- fooSdkCoder,
- WindowedValue.of(
- "1",
- new Instant(120),
- new IntervalWindow(new Instant(0), Duration.standardMinutes(5)),
- PaneInfo.NO_FIRING)));
- fooReceiver.accept(fooElem);
-
- WindowedValue<byte[]> barElem =
- CoderUtils.decodeFromByteArray(
- barRunnerCoder,
- CoderUtils.encodeToByteArray(
- barSdkCoder, WindowedValue.timestampedValueInGlobalWindow(2, new Instant(240))));
- barReceiver.accept(barElem);
-
- Collection<? super WindowedValue<?>> outputs = new ArrayList<>();
- for (UncommittedBundle<?> uncommitted : outputBundles) {
- WindowedValue<?> output = getOnlyElement(uncommitted.commit(Instant.now()).getElements());
- if (fooPC.equals(uncommitted.getPCollection())) {
- assertThat(output, equalTo(fooElem));
- } else if (barPC.equals(uncommitted.getPCollection())) {
- assertThat(output, equalTo(barElem));
- } else {
- fail(
- String.format(
- "Output %s should be either 'foo' or 'bar', got '%s",
- PCollection.class.getSimpleName(), uncommitted.getPCollection().getId()));
- }
- outputs.add(output);
- }
- assertThat(outputs, containsInAnyOrder(fooElem, barElem));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CommittedResultTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CommittedResultTest.java
deleted file mode 100644
index 357efb3..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CommittedResultTest.java
+++ /dev/null
@@ -1,129 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.junit.Assert.assertThat;
-
-import java.io.Serializable;
-import java.util.Collections;
-import java.util.EnumSet;
-import java.util.List;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.IsBounded.Enum;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.portable.CommittedResult.OutputType;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.hamcrest.Matchers;
-import org.joda.time.Instant;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link CommittedResult}. */
-@RunWith(JUnit4.class)
-public class CommittedResultTest implements Serializable {
-
- @Rule
- public transient TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
-
- private transient PCollectionNode created =
- PipelineNode.pCollection(
- "created", RunnerApi.PCollection.newBuilder().setUniqueName("created").build());
- private transient PTransformNode transform =
- PipelineNode.pTransform("foo", RunnerApi.PTransform.getDefaultInstance());
- private transient BundleFactory bundleFactory = ImmutableListBundleFactory.create();
-
- @Test
- public void getTransformExtractsFromResult() {
- CommittedResult<PTransformNode> result =
- CommittedResult.create(
- StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
- Collections.emptyList(),
- EnumSet.noneOf(OutputType.class));
-
- assertThat(result.getExecutable(), Matchers.equalTo(transform));
- }
-
- @Test
- public void getUncommittedElementsEqualInput() {
- CommittedBundle<Integer> bundle =
- bundleFactory
- .<Integer>createBundle(created)
- .add(WindowedValue.valueInGlobalWindow(2))
- .commit(Instant.now());
- CommittedResult<PTransformNode> result =
- CommittedResult.create(
- StepTransformResult.withoutHold(transform).build(),
- Optional.of(bundle),
- Collections.emptyList(),
- EnumSet.noneOf(OutputType.class));
-
- assertThat(result.getUnprocessedInputs().get(), Matchers.equalTo(bundle));
- }
-
- @Test
- public void getUncommittedElementsNull() {
- CommittedResult<PTransformNode> result =
- CommittedResult.create(
- StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
- Collections.emptyList(),
- EnumSet.noneOf(OutputType.class));
-
- assertThat(result.getUnprocessedInputs(), Matchers.equalTo(Optional.absent()));
- }
-
- @Test
- public void getOutputsEqualInput() {
- List<? extends CommittedBundle<Integer>> outputs =
- ImmutableList.of(
- bundleFactory
- .<Integer>createBundle(
- PipelineNode.pCollection(
- "bounded",
- RunnerApi.PCollection.newBuilder()
- .setUniqueName("bounded")
- .setIsBounded(Enum.BOUNDED)
- .build()))
- .commit(Instant.now()),
- bundleFactory
- .<Integer>createBundle(
- PipelineNode.pCollection(
- "unbounded",
- RunnerApi.PCollection.newBuilder()
- .setUniqueName("unbounded")
- .setIsBounded(Enum.UNBOUNDED)
- .build()))
- .commit(Instant.now()));
- CommittedResult<PTransformNode> result =
- CommittedResult.create(
- StepTransformResult.withoutHold(transform).build(),
- Optional.absent(),
- outputs,
- EnumSet.of(OutputType.BUNDLE, OutputType.PCOLLECTION_VIEW));
-
- assertThat(result.getOutputs(), Matchers.containsInAnyOrder(outputs.toArray()));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternalsTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternalsTest.java
deleted file mode 100644
index 125860a..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternalsTest.java
+++ /dev/null
@@ -1,591 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.emptyIterable;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.nullValue;
-import static org.hamcrest.Matchers.theInstance;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertThat;
-import static org.mockito.Mockito.never;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.verify;
-
-import org.apache.beam.runners.core.StateNamespace;
-import org.apache.beam.runners.core.StateNamespaceForTest;
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.StateTag;
-import org.apache.beam.runners.core.StateTags;
-import org.apache.beam.sdk.coders.CannotProvideCoderException;
-import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.state.BagState;
-import org.apache.beam.sdk.state.CombiningState;
-import org.apache.beam.sdk.state.GroupingState;
-import org.apache.beam.sdk.state.MapState;
-import org.apache.beam.sdk.state.SetState;
-import org.apache.beam.sdk.state.ValueState;
-import org.apache.beam.sdk.state.WatermarkHoldState;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.Sum;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.joda.time.Instant;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link CopyOnAccessInMemoryStateInternals}. */
-@RunWith(JUnit4.class)
-public class CopyOnAccessInMemoryStateInternalsTest {
-
- @Rule public final TestPipeline pipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
- private String key = "foo";
-
- @Test
- public void testGetWithEmpty() {
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = internals.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
- assertThat(stringBag.read(), containsInAnyOrder("baz", "bar"));
-
- BagState<String> reReadStringBag = internals.state(namespace, bagTag);
- assertThat(reReadStringBag.read(), containsInAnyOrder("baz", "bar"));
- }
-
- @Test
- public void testGetWithAbsentInUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = internals.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
- assertThat(stringBag.read(), containsInAnyOrder("baz", "bar"));
-
- BagState<String> reReadVoidBag = internals.state(namespace, bagTag);
- assertThat(reReadVoidBag.read(), containsInAnyOrder("baz", "bar"));
-
- BagState<String> underlyingState = underlying.state(namespace, bagTag);
- assertThat(underlyingState.read(), emptyIterable());
- }
-
- /**
- * Tests that retrieving state with an underlying StateInternals with an existing value returns a
- * value that initially has equal value to the provided state but can be modified without
- * modifying the existing state.
- */
- @Test
- public void testGetWithPresentInUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<ValueState<String>> valueTag = StateTags.value("foo", StringUtf8Coder.of());
- ValueState<String> underlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.read(), nullValue(String.class));
-
- underlyingValue.write("bar");
- assertThat(underlyingValue.read(), equalTo("bar"));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
- ValueState<String> copyOnAccessState = internals.state(namespace, valueTag);
- assertThat(copyOnAccessState.read(), equalTo("bar"));
-
- copyOnAccessState.write("baz");
- assertThat(copyOnAccessState.read(), equalTo("baz"));
- assertThat(underlyingValue.read(), equalTo("bar"));
-
- ValueState<String> reReadUnderlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.read(), equalTo(reReadUnderlyingValue.read()));
- }
-
- @Test
- public void testBagStateWithUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<Integer>> valueTag = StateTags.bag("foo", VarIntCoder.of());
- BagState<Integer> underlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.read(), emptyIterable());
-
- underlyingValue.add(1);
- assertThat(underlyingValue.read(), containsInAnyOrder(1));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
- BagState<Integer> copyOnAccessState = internals.state(namespace, valueTag);
- assertThat(copyOnAccessState.read(), containsInAnyOrder(1));
-
- copyOnAccessState.add(4);
- assertThat(copyOnAccessState.read(), containsInAnyOrder(4, 1));
- assertThat(underlyingValue.read(), containsInAnyOrder(1));
-
- BagState<Integer> reReadUnderlyingValue = underlying.state(namespace, valueTag);
- assertThat(
- Lists.newArrayList(underlyingValue.read()),
- equalTo(Lists.newArrayList(reReadUnderlyingValue.read())));
- }
-
- @Test
- public void testSetStateWithUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<SetState<Integer>> valueTag = StateTags.set("foo", VarIntCoder.of());
- SetState<Integer> underlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.read(), emptyIterable());
-
- underlyingValue.add(1);
- assertThat(underlyingValue.read(), containsInAnyOrder(1));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
- SetState<Integer> copyOnAccessState = internals.state(namespace, valueTag);
- assertThat(copyOnAccessState.read(), containsInAnyOrder(1));
-
- copyOnAccessState.add(4);
- assertThat(copyOnAccessState.read(), containsInAnyOrder(4, 1));
- assertThat(underlyingValue.read(), containsInAnyOrder(1));
-
- SetState<Integer> reReadUnderlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.read(), equalTo(reReadUnderlyingValue.read()));
- }
-
- @Test
- public void testMapStateWithUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<MapState<String, Integer>> valueTag =
- StateTags.map("foo", StringUtf8Coder.of(), VarIntCoder.of());
- MapState<String, Integer> underlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.entries().read(), emptyIterable());
-
- underlyingValue.put("hello", 1);
- assertThat(underlyingValue.get("hello").read(), equalTo(1));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
- MapState<String, Integer> copyOnAccessState = internals.state(namespace, valueTag);
- assertThat(copyOnAccessState.get("hello").read(), equalTo(1));
-
- copyOnAccessState.put("world", 4);
- assertThat(copyOnAccessState.get("hello").read(), equalTo(1));
- assertThat(copyOnAccessState.get("world").read(), equalTo(4));
- assertThat(underlyingValue.get("hello").read(), equalTo(1));
- assertNull(underlyingValue.get("world").read());
-
- MapState<String, Integer> reReadUnderlyingValue = underlying.state(namespace, valueTag);
- assertThat(underlyingValue.entries().read(), equalTo(reReadUnderlyingValue.entries().read()));
- }
-
- @Test
- public void testAccumulatorCombiningStateWithUnderlying() throws CannotProvideCoderException {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CombineFn<Long, long[], Long> sumLongFn = Sum.ofLongs();
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- CoderRegistry reg = pipeline.getCoderRegistry();
- StateTag<CombiningState<Long, long[], Long>> stateTag =
- StateTags.combiningValue(
- "summer", sumLongFn.getAccumulatorCoder(reg, reg.getCoder(Long.class)), sumLongFn);
- GroupingState<Long, Long> underlyingValue = underlying.state(namespace, stateTag);
- assertThat(underlyingValue.read(), equalTo(0L));
-
- underlyingValue.add(1L);
- assertThat(underlyingValue.read(), equalTo(1L));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
- GroupingState<Long, Long> copyOnAccessState = internals.state(namespace, stateTag);
- assertThat(copyOnAccessState.read(), equalTo(1L));
-
- copyOnAccessState.add(4L);
- assertThat(copyOnAccessState.read(), equalTo(5L));
- assertThat(underlyingValue.read(), equalTo(1L));
-
- GroupingState<Long, Long> reReadUnderlyingValue = underlying.state(namespace, stateTag);
- assertThat(underlyingValue.read(), equalTo(reReadUnderlyingValue.read()));
- }
-
- @Test
- public void testWatermarkHoldStateWithUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- TimestampCombiner timestampCombiner = TimestampCombiner.EARLIEST;
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<WatermarkHoldState> stateTag =
- StateTags.watermarkStateInternal("wmstate", timestampCombiner);
- WatermarkHoldState underlyingValue = underlying.state(namespace, stateTag);
- assertThat(underlyingValue.read(), nullValue());
-
- underlyingValue.add(new Instant(250L));
- assertThat(underlyingValue.read(), equalTo(new Instant(250L)));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
- WatermarkHoldState copyOnAccessState = internals.state(namespace, stateTag);
- assertThat(copyOnAccessState.read(), equalTo(new Instant(250L)));
-
- copyOnAccessState.add(new Instant(100L));
- assertThat(copyOnAccessState.read(), equalTo(new Instant(100L)));
- assertThat(underlyingValue.read(), equalTo(new Instant(250L)));
-
- copyOnAccessState.add(new Instant(500L));
- assertThat(copyOnAccessState.read(), equalTo(new Instant(100L)));
-
- WatermarkHoldState reReadUnderlyingValue = underlying.state(namespace, stateTag);
- assertThat(underlyingValue.read(), equalTo(reReadUnderlyingValue.read()));
- }
-
- @Test
- public void testCommitWithoutUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = internals.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
- assertThat(stringBag.read(), containsInAnyOrder("baz", "bar"));
-
- internals.commit();
-
- BagState<String> reReadStringBag = internals.state(namespace, bagTag);
- assertThat(reReadStringBag.read(), containsInAnyOrder("baz", "bar"));
- assertThat(internals.isEmpty(), is(false));
- }
-
- @Test
- public void testCommitWithUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = underlying.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
-
- internals.commit();
- BagState<String> reReadStringBag = internals.state(namespace, bagTag);
- assertThat(reReadStringBag.read(), containsInAnyOrder("baz", "bar"));
-
- reReadStringBag.add("spam");
-
- BagState<String> underlyingState = underlying.state(namespace, bagTag);
- assertThat(underlyingState.read(), containsInAnyOrder("spam", "bar", "baz"));
- assertThat(underlyingState, is(theInstance(stringBag)));
- assertThat(internals.isEmpty(), is(false));
- }
-
- @Test
- public void testCommitWithClearedInUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CopyOnAccessInMemoryStateInternals<String> secondUnderlying =
- spy(CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying));
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, secondUnderlying);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = underlying.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
- stringBag.clear();
- // We should not read through the cleared bag
- secondUnderlying.commit();
-
- // Should not be visible
- stringBag.add("foo");
-
- internals.commit();
- BagState<String> internalsStringBag = internals.state(namespace, bagTag);
- assertThat(internalsStringBag.read(), emptyIterable());
- verify(secondUnderlying, never()).state(namespace, bagTag);
- assertThat(internals.isEmpty(), is(false));
- }
-
- @Test
- public void testCommitWithOverwrittenUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = underlying.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
-
- BagState<String> internalsState = internals.state(namespace, bagTag);
- internalsState.add("eggs");
- internalsState.add("ham");
- internalsState.add("0x00ff00");
- internalsState.add("&");
-
- internals.commit();
-
- BagState<String> reReadInternalState = internals.state(namespace, bagTag);
- assertThat(
- reReadInternalState.read(),
- containsInAnyOrder("bar", "baz", "0x00ff00", "eggs", "&", "ham"));
- BagState<String> reReadUnderlyingState = underlying.state(namespace, bagTag);
- assertThat(reReadUnderlyingState.read(), containsInAnyOrder("bar", "baz"));
- }
-
- @Test
- public void testCommitWithAddedUnderlying() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
-
- internals.commit();
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = underlying.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
-
- BagState<String> internalState = internals.state(namespace, bagTag);
- assertThat(internalState.read(), emptyIterable());
-
- BagState<String> reReadUnderlyingState = underlying.state(namespace, bagTag);
- assertThat(reReadUnderlyingState.read(), containsInAnyOrder("bar", "baz"));
- }
-
- @Test
- public void testCommitWithEmptyTableIsEmpty() {
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- internals.commit();
-
- assertThat(internals.isEmpty(), is(true));
- }
-
- @Test
- public void testCommitWithOnlyClearedValuesIsEmpty() {
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = internals.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("foo");
- stringBag.clear();
-
- internals.commit();
-
- assertThat(internals.isEmpty(), is(true));
- }
-
- @Test
- public void testCommitWithEmptyNewAndFullUnderlyingIsNotEmpty() {
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, underlying);
-
- StateNamespace namespace = new StateNamespaceForTest("foo");
- StateTag<BagState<String>> bagTag = StateTags.bag("foo", StringUtf8Coder.of());
- BagState<String> stringBag = underlying.state(namespace, bagTag);
- assertThat(stringBag.read(), emptyIterable());
-
- stringBag.add("bar");
- stringBag.add("baz");
-
- internals.commit();
- assertThat(internals.isEmpty(), is(false));
- }
-
- @Test
- public void testGetEarliestWatermarkHoldAfterCommit() {
- BoundedWindow first =
- new BoundedWindow() {
- @Override
- public Instant maxTimestamp() {
- return new Instant(2048L);
- }
- };
- BoundedWindow second =
- new BoundedWindow() {
- @Override
- public Instant maxTimestamp() {
- return new Instant(689743L);
- }
- };
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying("foo", null);
-
- StateTag<WatermarkHoldState> firstHoldAddress =
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST);
- WatermarkHoldState firstHold =
- internals.state(StateNamespaces.window(null, first), firstHoldAddress);
- firstHold.add(new Instant(22L));
-
- StateTag<WatermarkHoldState> secondHoldAddress =
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST);
- WatermarkHoldState secondHold =
- internals.state(StateNamespaces.window(null, second), secondHoldAddress);
- secondHold.add(new Instant(2L));
-
- internals.commit();
- assertThat(internals.getEarliestWatermarkHold(), equalTo(new Instant(2L)));
- }
-
- @Test
- public void testGetEarliestWatermarkHoldWithEarliestInUnderlyingTable() {
- BoundedWindow first =
- new BoundedWindow() {
- @Override
- public Instant maxTimestamp() {
- return new Instant(2048L);
- }
- };
- BoundedWindow second =
- new BoundedWindow() {
- @Override
- public Instant maxTimestamp() {
- return new Instant(689743L);
- }
- };
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying("foo", null);
- StateTag<WatermarkHoldState> firstHoldAddress =
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST);
- WatermarkHoldState firstHold =
- underlying.state(StateNamespaces.window(null, first), firstHoldAddress);
- firstHold.add(new Instant(22L));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying("foo", underlying.commit());
-
- StateTag<WatermarkHoldState> secondHoldAddress =
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST);
- WatermarkHoldState secondHold =
- internals.state(StateNamespaces.window(null, second), secondHoldAddress);
- secondHold.add(new Instant(244L));
-
- internals.commit();
- assertThat(internals.getEarliestWatermarkHold(), equalTo(new Instant(22L)));
- }
-
- @Test
- public void testGetEarliestWatermarkHoldWithEarliestInNewTable() {
- BoundedWindow first =
- new BoundedWindow() {
- @Override
- public Instant maxTimestamp() {
- return new Instant(2048L);
- }
- };
- BoundedWindow second =
- new BoundedWindow() {
- @Override
- public Instant maxTimestamp() {
- return new Instant(689743L);
- }
- };
- CopyOnAccessInMemoryStateInternals<String> underlying =
- CopyOnAccessInMemoryStateInternals.withUnderlying("foo", null);
- StateTag<WatermarkHoldState> firstHoldAddress =
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST);
- WatermarkHoldState firstHold =
- underlying.state(StateNamespaces.window(null, first), firstHoldAddress);
- firstHold.add(new Instant(224L));
-
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying("foo", underlying.commit());
-
- StateTag<WatermarkHoldState> secondHoldAddress =
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST);
- WatermarkHoldState secondHold =
- internals.state(StateNamespaces.window(null, second), secondHoldAddress);
- secondHold.add(new Instant(24L));
-
- internals.commit();
- assertThat(internals.getEarliestWatermarkHold(), equalTo(new Instant(24L)));
- }
-
- @Test
- public void testGetEarliestHoldBeforeCommit() {
- CopyOnAccessInMemoryStateInternals<String> internals =
- CopyOnAccessInMemoryStateInternals.withUnderlying(key, null);
-
- internals
- .state(
- StateNamespaces.global(),
- StateTags.watermarkStateInternal("foo", TimestampCombiner.EARLIEST))
- .add(new Instant(1234L));
-
- thrown.expect(IllegalStateException.class);
- thrown.expectMessage(CopyOnAccessInMemoryStateInternals.class.getSimpleName());
- thrown.expectMessage("Can't get the earliest watermark hold");
- thrown.expectMessage("before it is committed");
-
- internals.getEarliestWatermarkHold();
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectMetricsTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectMetricsTest.java
deleted file mode 100644
index 59faca3..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectMetricsTest.java
+++ /dev/null
@@ -1,232 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.sdk.metrics.MetricNameFilter.inNamespace;
-import static org.apache.beam.sdk.metrics.MetricResultsMatchers.attemptedMetricsResult;
-import static org.apache.beam.sdk.metrics.MetricResultsMatchers.committedMetricsResult;
-import static org.hamcrest.Matchers.contains;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.junit.Assert.assertThat;
-
-import org.apache.beam.runners.core.metrics.DistributionData;
-import org.apache.beam.runners.core.metrics.GaugeData;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
-import org.apache.beam.sdk.metrics.DistributionResult;
-import org.apache.beam.sdk.metrics.GaugeResult;
-import org.apache.beam.sdk.metrics.MetricKey;
-import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.sdk.metrics.MetricQueryResults;
-import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.MockitoAnnotations;
-
-/** Tests for {@link DirectMetrics}. */
-@RunWith(JUnit4.class)
-public class DirectMetricsTest {
-
- @Mock private CommittedBundle<Object> bundle1;
- @Mock private CommittedBundle<Object> bundle2;
-
- private static final MetricName NAME1 = MetricName.named("ns1", "name1");
- private static final MetricName NAME2 = MetricName.named("ns1", "name2");
- private static final MetricName NAME3 = MetricName.named("ns2", "name1");
- private static final MetricName NAME4 = MetricName.named("ns2", "name2");
-
- private DirectMetrics metrics = new DirectMetrics();
-
- @Before
- public void setUp() {
- MockitoAnnotations.initMocks(this);
- }
-
- @SuppressWarnings("unchecked")
- @Test
- public void testApplyCommittedNoFilter() {
- metrics.commitLogical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("step1", NAME1), 5L),
- MetricUpdate.create(MetricKey.create("step1", NAME2), 8L)),
- ImmutableList.of(
- MetricUpdate.create(
- MetricKey.create("step1", NAME1), DistributionData.create(8, 2, 3, 5))),
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("step1", NAME4), GaugeData.create(15L)))));
- metrics.commitLogical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("step2", NAME1), 7L),
- MetricUpdate.create(MetricKey.create("step1", NAME2), 4L)),
- ImmutableList.of(
- MetricUpdate.create(
- MetricKey.create("step1", NAME1), DistributionData.create(4, 1, 4, 4))),
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("step1", NAME4), GaugeData.create(27L)))));
-
- MetricQueryResults results = metrics.allMetrics();
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- attemptedMetricsResult("ns1", "name1", "step1", 0L),
- attemptedMetricsResult("ns1", "name2", "step1", 0L),
- attemptedMetricsResult("ns1", "name1", "step2", 0L)));
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- committedMetricsResult("ns1", "name1", "step1", 5L),
- committedMetricsResult("ns1", "name2", "step1", 12L),
- committedMetricsResult("ns1", "name1", "step2", 7L)));
- assertThat(
- results.getDistributions(),
- contains(
- attemptedMetricsResult("ns1", "name1", "step1", DistributionResult.IDENTITY_ELEMENT)));
- assertThat(
- results.getDistributions(),
- contains(
- committedMetricsResult(
- "ns1", "name1", "step1", DistributionResult.create(12, 3, 3, 5))));
- assertThat(
- results.getGauges(),
- contains(attemptedMetricsResult("ns2", "name2", "step1", GaugeResult.empty())));
- assertThat(
- results.getGauges(),
- contains(
- committedMetricsResult(
- "ns2", "name2", "step1", GaugeResult.create(27L, Instant.now()))));
- }
-
- @SuppressWarnings("unchecked")
- @Test
- public void testApplyAttemptedCountersQueryOneNamespace() {
- metrics.updatePhysical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("step1", NAME1), 5L),
- MetricUpdate.create(MetricKey.create("step1", NAME3), 8L)),
- ImmutableList.of(),
- ImmutableList.of()));
- metrics.updatePhysical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("step2", NAME1), 7L),
- MetricUpdate.create(MetricKey.create("step1", NAME3), 4L)),
- ImmutableList.of(),
- ImmutableList.of()));
-
- MetricQueryResults results =
- metrics.queryMetrics(MetricsFilter.builder().addNameFilter(inNamespace("ns1")).build());
-
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- attemptedMetricsResult("ns1", "name1", "step1", 5L),
- attemptedMetricsResult("ns1", "name1", "step2", 7L)));
-
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- committedMetricsResult("ns1", "name1", "step1", 0L),
- committedMetricsResult("ns1", "name1", "step2", 0L)));
- }
-
- @SuppressWarnings("unchecked")
- @Test
- public void testApplyAttemptedQueryCompositeScope() {
- metrics.updatePhysical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("Outer1/Inner1", NAME1), 5L),
- MetricUpdate.create(MetricKey.create("Outer1/Inner2", NAME1), 8L)),
- ImmutableList.of(),
- ImmutableList.of()));
- metrics.updatePhysical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("Outer1/Inner1", NAME1), 12L),
- MetricUpdate.create(MetricKey.create("Outer2/Inner2", NAME1), 18L)),
- ImmutableList.of(),
- ImmutableList.of()));
-
- MetricQueryResults results =
- metrics.queryMetrics(MetricsFilter.builder().addStep("Outer1").build());
-
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- attemptedMetricsResult("ns1", "name1", "Outer1/Inner1", 12L),
- attemptedMetricsResult("ns1", "name1", "Outer1/Inner2", 8L)));
-
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- committedMetricsResult("ns1", "name1", "Outer1/Inner1", 0L),
- committedMetricsResult("ns1", "name1", "Outer1/Inner2", 0L)));
- }
-
- @SuppressWarnings("unchecked")
- @Test
- public void testPartialScopeMatchingInMetricsQuery() {
- metrics.updatePhysical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("Top1/Outer1/Inner1", NAME1), 5L),
- MetricUpdate.create(MetricKey.create("Top1/Outer1/Inner2", NAME1), 8L)),
- ImmutableList.of(),
- ImmutableList.of()));
- metrics.updatePhysical(
- bundle1,
- MetricUpdates.create(
- ImmutableList.of(
- MetricUpdate.create(MetricKey.create("Top2/Outer1/Inner1", NAME1), 12L),
- MetricUpdate.create(MetricKey.create("Top1/Outer2/Inner2", NAME1), 18L)),
- ImmutableList.of(),
- ImmutableList.of()));
-
- MetricQueryResults results =
- metrics.queryMetrics(MetricsFilter.builder().addStep("Top1/Outer1").build());
-
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- attemptedMetricsResult("ns1", "name1", "Top1/Outer1/Inner1", 5L),
- attemptedMetricsResult("ns1", "name1", "Top1/Outer1/Inner2", 8L)));
-
- results = metrics.queryMetrics(MetricsFilter.builder().addStep("Inner2").build());
-
- assertThat(
- results.getCounters(),
- containsInAnyOrder(
- attemptedMetricsResult("ns1", "name1", "Top1/Outer1/Inner2", 8L),
- attemptedMetricsResult("ns1", "name1", "Top1/Outer2/Inner2", 18L)));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTimerInternalsTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTimerInternalsTest.java
deleted file mode 100644
index f5c73fd..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTimerInternalsTest.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-import static org.mockito.Mockito.when;
-
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.TimerInternals.TimerData;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate.TimerUpdateBuilder;
-import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.state.TimeDomain;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.MockitoAnnotations;
-
-/** Tests for {@link DirectTimerInternals}. */
-@RunWith(JUnit4.class)
-public class DirectTimerInternalsTest {
- private MockClock clock;
- @Mock private TransformWatermarks watermarks;
-
- private TimerUpdateBuilder timerUpdateBuilder;
-
- private DirectTimerInternals internals;
-
- @Before
- public void setup() {
- MockitoAnnotations.initMocks(this);
- clock = MockClock.fromInstant(new Instant(0));
-
- timerUpdateBuilder = TimerUpdate.builder(StructuralKey.of(1234, VarIntCoder.of()));
-
- internals = DirectTimerInternals.create(clock, watermarks, timerUpdateBuilder);
- }
-
- @Test
- public void setTimerAddsToBuilder() {
- TimerData eventTimer =
- TimerData.of(StateNamespaces.global(), new Instant(20145L), TimeDomain.EVENT_TIME);
- TimerData processingTimer =
- TimerData.of(StateNamespaces.global(), new Instant(125555555L), TimeDomain.PROCESSING_TIME);
- TimerData synchronizedProcessingTimer =
- TimerData.of(
- StateNamespaces.global(),
- new Instant(98745632189L),
- TimeDomain.SYNCHRONIZED_PROCESSING_TIME);
- internals.setTimer(eventTimer);
- internals.setTimer(processingTimer);
- internals.setTimer(synchronizedProcessingTimer);
-
- assertThat(
- internals.getTimerUpdate().getSetTimers(),
- containsInAnyOrder(eventTimer, synchronizedProcessingTimer, processingTimer));
- }
-
- @Test
- public void deleteTimerDeletesOnBuilder() {
- TimerData eventTimer =
- TimerData.of(StateNamespaces.global(), new Instant(20145L), TimeDomain.EVENT_TIME);
- TimerData processingTimer =
- TimerData.of(StateNamespaces.global(), new Instant(125555555L), TimeDomain.PROCESSING_TIME);
- TimerData synchronizedProcessingTimer =
- TimerData.of(
- StateNamespaces.global(),
- new Instant(98745632189L),
- TimeDomain.SYNCHRONIZED_PROCESSING_TIME);
- internals.deleteTimer(eventTimer);
- internals.deleteTimer(processingTimer);
- internals.deleteTimer(synchronizedProcessingTimer);
-
- assertThat(
- internals.getTimerUpdate().getDeletedTimers(),
- containsInAnyOrder(eventTimer, synchronizedProcessingTimer, processingTimer));
- }
-
- @Test
- public void getProcessingTimeIsClockNow() {
- assertThat(internals.currentProcessingTime(), equalTo(clock.now()));
- Instant oldProcessingTime = internals.currentProcessingTime();
-
- clock.advance(Duration.standardHours(12));
-
- assertThat(internals.currentProcessingTime(), equalTo(clock.now()));
- assertThat(
- internals.currentProcessingTime(),
- equalTo(oldProcessingTime.plus(Duration.standardHours(12))));
- }
-
- @Test
- public void getSynchronizedProcessingTimeIsWatermarkSynchronizedInputTime() {
- when(watermarks.getSynchronizedProcessingInputTime()).thenReturn(new Instant(12345L));
- assertThat(internals.currentSynchronizedProcessingTime(), equalTo(new Instant(12345L)));
- }
-
- @Test
- public void getInputWatermarkTimeUsesWatermarkTime() {
- when(watermarks.getInputWatermark()).thenReturn(new Instant(8765L));
- assertThat(internals.currentInputWatermarkTime(), equalTo(new Instant(8765L)));
- }
-
- @Test
- public void getOutputWatermarkTimeUsesWatermarkTime() {
- when(watermarks.getOutputWatermark()).thenReturn(new Instant(25525L));
- assertThat(internals.currentOutputWatermarkTime(), equalTo(new Instant(25525L)));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTransformExecutorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTransformExecutorTest.java
deleted file mode 100644
index 0cae265..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTransformExecutorTest.java
+++ /dev/null
@@ -1,330 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.nullValue;
-import static org.junit.Assert.assertThat;
-import static org.mockito.Mockito.when;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.EnumSet;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.atomic.AtomicBoolean;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.portable.CommittedResult.OutputType;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.hamcrest.Matchers;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.MockitoAnnotations;
-
-/** Tests for {@link DirectTransformExecutor}. */
-@RunWith(JUnit4.class)
-public class DirectTransformExecutorTest {
- @Rule public ExpectedException thrown = ExpectedException.none();
- private final PCollectionNode created =
- PipelineNode.pCollection(
- "created", PCollection.newBuilder().setUniqueName("created").build());
-
- private final PTransformNode createdProducer =
- PipelineNode.pTransform(
- "create",
- PTransform.newBuilder().putOutputs("created", "created").setUniqueName("create").build());
- private final PTransformNode downstreamProducer =
- PipelineNode.pTransform(
- "downstream",
- PTransform.newBuilder().putInputs("input", "created").setUniqueName("create").build());
-
- private CountDownLatch evaluatorCompleted;
-
- private RegisteringCompletionCallback completionCallback;
- private TransformExecutorService transformEvaluationState;
- private BundleFactory bundleFactory;
- @Mock private DirectMetrics metrics;
- @Mock private EvaluationContext evaluationContext;
- @Mock private TransformEvaluatorRegistry registry;
-
- @Rule public TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
-
- @Before
- public void setup() {
- MockitoAnnotations.initMocks(this);
-
- bundleFactory = ImmutableListBundleFactory.create();
-
- transformEvaluationState =
- TransformExecutorServices.parallel(MoreExecutors.newDirectExecutorService());
-
- evaluatorCompleted = new CountDownLatch(1);
- completionCallback = new RegisteringCompletionCallback(evaluatorCompleted);
-
- PipelineNode.pCollection(
- "created", RunnerApi.PCollection.newBuilder().setUniqueName("created").build());
-
- when(evaluationContext.getMetrics()).thenReturn(metrics);
- }
-
- @Test
- public void callWithNullInputBundleFinishesBundleAndCompletes() throws Exception {
- final TransformResult<Object> result = StepTransformResult.withoutHold(createdProducer).build();
- final AtomicBoolean finishCalled = new AtomicBoolean(false);
- TransformEvaluator<Object> evaluator =
- new TransformEvaluator<Object>() {
- @Override
- public void processElement(WindowedValue<Object> element) throws Exception {
- throw new IllegalArgumentException("Shouldn't be called");
- }
-
- @Override
- public TransformResult<Object> finishBundle() throws Exception {
- finishCalled.set(true);
- return result;
- }
- };
-
- when(registry.forApplication(createdProducer, null)).thenReturn(evaluator);
-
- DirectTransformExecutor<Object> executor =
- new DirectTransformExecutor<>(
- evaluationContext,
- registry,
- null,
- createdProducer,
- completionCallback,
- transformEvaluationState);
- executor.run();
-
- assertThat(finishCalled.get(), is(true));
- assertThat(completionCallback.handledResult, Matchers.equalTo(result));
- assertThat(completionCallback.handledException, is(nullValue()));
- }
-
- @Test
- public void nullTransformEvaluatorTerminates() throws Exception {
- when(registry.forApplication(createdProducer, null)).thenReturn(null);
-
- DirectTransformExecutor<Object> executor =
- new DirectTransformExecutor<>(
- evaluationContext,
- registry,
- null,
- createdProducer,
- completionCallback,
- transformEvaluationState);
- executor.run();
-
- assertThat(completionCallback.handledResult, is(nullValue()));
- assertThat(completionCallback.handledEmpty, equalTo(true));
- assertThat(completionCallback.handledException, is(nullValue()));
- }
-
- @Test
- public void inputBundleProcessesEachElementFinishesAndCompletes() throws Exception {
- final TransformResult<String> result =
- StepTransformResult.<String>withoutHold(downstreamProducer).build();
- final Collection<WindowedValue<String>> elementsProcessed = new ArrayList<>();
- TransformEvaluator<String> evaluator =
- new TransformEvaluator<String>() {
- @Override
- public void processElement(WindowedValue<String> element) throws Exception {
- elementsProcessed.add(element);
- }
-
- @Override
- public TransformResult<String> finishBundle() throws Exception {
- return result;
- }
- };
-
- WindowedValue<String> foo = WindowedValue.valueInGlobalWindow("foo");
- WindowedValue<String> spam = WindowedValue.valueInGlobalWindow("spam");
- WindowedValue<String> third = WindowedValue.valueInGlobalWindow("third");
- CommittedBundle<String> inputBundle =
- bundleFactory
- .<String>createBundle(created)
- .add(foo)
- .add(spam)
- .add(third)
- .commit(Instant.now());
- when(registry.<String>forApplication(downstreamProducer, inputBundle)).thenReturn(evaluator);
-
- DirectTransformExecutor<String> executor =
- new DirectTransformExecutor<>(
- evaluationContext,
- registry,
- inputBundle,
- downstreamProducer,
- completionCallback,
- transformEvaluationState);
-
- Future<?> future = Executors.newSingleThreadExecutor().submit(executor);
-
- evaluatorCompleted.await();
- future.get();
-
- assertThat(elementsProcessed, containsInAnyOrder(spam, third, foo));
- assertThat(completionCallback.handledResult, Matchers.equalTo(result));
- assertThat(completionCallback.handledException, is(nullValue()));
- }
-
- @Test
- @SuppressWarnings("FutureReturnValueIgnored") // expected exception checked via completionCallback
- public void processElementThrowsExceptionCallsback() throws Exception {
- final TransformResult<String> result =
- StepTransformResult.<String>withoutHold(downstreamProducer).build();
- final Exception exception = new Exception();
- TransformEvaluator<String> evaluator =
- new TransformEvaluator<String>() {
- @Override
- public void processElement(WindowedValue<String> element) throws Exception {
- throw exception;
- }
-
- @Override
- public TransformResult<String> finishBundle() throws Exception {
- return result;
- }
- };
-
- WindowedValue<String> foo = WindowedValue.valueInGlobalWindow("foo");
- CommittedBundle<String> inputBundle =
- bundleFactory.<String>createBundle(created).add(foo).commit(Instant.now());
- when(registry.<String>forApplication(downstreamProducer, inputBundle)).thenReturn(evaluator);
-
- DirectTransformExecutor<String> executor =
- new DirectTransformExecutor<>(
- evaluationContext,
- registry,
- inputBundle,
- downstreamProducer,
- completionCallback,
- transformEvaluationState);
- Executors.newSingleThreadExecutor().submit(executor);
-
- evaluatorCompleted.await();
-
- assertThat(completionCallback.handledResult, is(nullValue()));
- assertThat(completionCallback.handledException, Matchers.<Throwable>equalTo(exception));
- }
-
- @Test
- @SuppressWarnings("FutureReturnValueIgnored") // expected exception checked via completionCallback
- public void finishBundleThrowsExceptionCallsback() throws Exception {
- final Exception exception = new Exception();
- TransformEvaluator<String> evaluator =
- new TransformEvaluator<String>() {
- @Override
- public void processElement(WindowedValue<String> element) throws Exception {}
-
- @Override
- public TransformResult<String> finishBundle() throws Exception {
- throw exception;
- }
- };
-
- CommittedBundle<String> inputBundle =
- bundleFactory.<String>createBundle(created).commit(Instant.now());
- when(registry.<String>forApplication(downstreamProducer, inputBundle)).thenReturn(evaluator);
-
- DirectTransformExecutor<String> executor =
- new DirectTransformExecutor<>(
- evaluationContext,
- registry,
- inputBundle,
- downstreamProducer,
- completionCallback,
- transformEvaluationState);
- Executors.newSingleThreadExecutor().submit(executor);
-
- evaluatorCompleted.await();
-
- assertThat(completionCallback.handledResult, is(nullValue()));
- assertThat(completionCallback.handledException, Matchers.<Throwable>equalTo(exception));
- }
-
- private static class RegisteringCompletionCallback implements CompletionCallback {
- private TransformResult<?> handledResult = null;
- private boolean handledEmpty = false;
- private Exception handledException = null;
- private final CountDownLatch onMethod;
-
- private RegisteringCompletionCallback(CountDownLatch onMethod) {
- this.onMethod = onMethod;
- }
-
- @Override
- public CommittedResult handleResult(CommittedBundle<?> inputBundle, TransformResult<?> result) {
- handledResult = result;
- onMethod.countDown();
- @SuppressWarnings("rawtypes")
- Iterable unprocessedElements =
- result.getUnprocessedElements() == null
- ? Collections.emptyList()
- : result.getUnprocessedElements();
-
- Optional<? extends CommittedBundle<?>> unprocessedBundle;
- if (inputBundle == null || Iterables.isEmpty(unprocessedElements)) {
- unprocessedBundle = Optional.absent();
- } else {
- unprocessedBundle =
- Optional.<CommittedBundle<?>>of(inputBundle.withElements(unprocessedElements));
- }
- return CommittedResult.create(
- result, unprocessedBundle, Collections.emptyList(), EnumSet.noneOf(OutputType.class));
- }
-
- @Override
- public void handleEmpty(PTransformNode transform) {
- handledEmpty = true;
- onMethod.countDown();
- }
-
- @Override
- public void handleException(CommittedBundle<?> inputBundle, Exception e) {
- handledException = e;
- onMethod.countDown();
- }
-
- @Override
- public void handleError(Error err) {
- throw err;
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/EvaluationContextTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/EvaluationContextTest.java
deleted file mode 100644
index 46df904..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/EvaluationContextTest.java
+++ /dev/null
@@ -1,315 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.hamcrest.Matchers.contains;
-import static org.hamcrest.Matchers.emptyIterable;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.not;
-import static org.junit.Assert.assertThat;
-
-import java.util.Collection;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.TimeUnit;
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.StateTag;
-import org.apache.beam.runners.core.StateTags;
-import org.apache.beam.runners.core.TimerInternals.TimerData;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.direct.WatermarkManager.FiredTimers;
-import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.state.BagState;
-import org.apache.beam.sdk.state.TimeDomain;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.hamcrest.Matchers;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link EvaluationContext}. */
-@RunWith(JUnit4.class)
-public class EvaluationContextTest {
- private EvaluationContext context;
-
- private PCollectionNode created;
- private PCollectionNode downstream;
-
- private ExecutableGraph<PTransformNode, PCollectionNode> graph;
-
- private PTransformNode createdProducer;
- private PTransformNode downstreamProducer;
- private PTransformNode unboundedProducer;
-
- @Before
- public void setup() {
- ExecutableGraphBuilder graphBuilder =
- ExecutableGraphBuilder.create()
- .addTransform("create", null, "created")
- .addTransform("downstream", "created", "downstream.out")
- .addTransform("unbounded", null, "unbounded.out");
-
- graph = graphBuilder.toGraph();
- created = graphBuilder.collectionNode("created");
- downstream = graphBuilder.collectionNode("downstream.out");
- createdProducer = graphBuilder.transformNode("create");
- downstreamProducer = graphBuilder.transformNode("downstream");
- unboundedProducer = graphBuilder.transformNode("unbounded");
-
- BundleFactory bundleFactory = ImmutableListBundleFactory.create();
- context = EvaluationContext.create(Instant::new, bundleFactory, graph, ImmutableSet.of());
- }
-
- @Test
- public void getExecutionContextSameStepSameKeyState() {
- StepStateAndTimers<String> fooContext =
- context.getStateAndTimers(createdProducer, StructuralKey.of("foo", StringUtf8Coder.of()));
-
- StateTag<BagState<Integer>> intBag = StateTags.bag("myBag", VarIntCoder.of());
-
- fooContext.stateInternals().state(StateNamespaces.global(), intBag).add(1);
-
- context.handleResult(
- ImmutableListBundleFactory.create()
- .createKeyedBundle(StructuralKey.of("foo", StringUtf8Coder.of()), created)
- .commit(Instant.now()),
- ImmutableList.of(),
- StepTransformResult.withoutHold(createdProducer)
- .withState(fooContext.stateInternals().commit())
- .build());
-
- StepStateAndTimers secondFooContext =
- context.getStateAndTimers(createdProducer, StructuralKey.of("foo", StringUtf8Coder.of()));
- assertThat(
- secondFooContext.stateInternals().state(StateNamespaces.global(), intBag).read(),
- contains(1));
- }
-
- @Test
- public void getExecutionContextDifferentKeysIndependentState() {
- StepStateAndTimers fooContext =
- context.getStateAndTimers(createdProducer, StructuralKey.of("foo", StringUtf8Coder.of()));
-
- StateTag<BagState<Integer>> intBag = StateTags.bag("myBag", VarIntCoder.of());
-
- fooContext.stateInternals().state(StateNamespaces.global(), intBag).add(1);
-
- StepStateAndTimers barContext =
- context.getStateAndTimers(createdProducer, StructuralKey.of("bar", StringUtf8Coder.of()));
- assertThat(barContext, not(equalTo(fooContext)));
- assertThat(
- barContext.stateInternals().state(StateNamespaces.global(), intBag).read(),
- emptyIterable());
- }
-
- @Test
- public void getExecutionContextDifferentStepsIndependentState() {
- StructuralKey<?> myKey = StructuralKey.of("foo", StringUtf8Coder.of());
- StepStateAndTimers fooContext = context.getStateAndTimers(createdProducer, myKey);
-
- StateTag<BagState<Integer>> intBag = StateTags.bag("myBag", VarIntCoder.of());
-
- fooContext.stateInternals().state(StateNamespaces.global(), intBag).add(1);
-
- StepStateAndTimers barContext = context.getStateAndTimers(downstreamProducer, myKey);
- assertThat(
- barContext.stateInternals().state(StateNamespaces.global(), intBag).read(),
- emptyIterable());
- }
-
- @Test
- public void handleResultStoresState() {
- StructuralKey<?> myKey = StructuralKey.of("foo".getBytes(UTF_8), ByteArrayCoder.of());
- StepStateAndTimers fooContext = context.getStateAndTimers(downstreamProducer, myKey);
-
- StateTag<BagState<Integer>> intBag = StateTags.bag("myBag", VarIntCoder.of());
-
- CopyOnAccessInMemoryStateInternals<?> state = fooContext.stateInternals();
- BagState<Integer> bag = state.state(StateNamespaces.global(), intBag);
- bag.add(1);
- bag.add(2);
- bag.add(4);
-
- TransformResult<?> stateResult =
- StepTransformResult.withoutHold(downstreamProducer).withState(state).build();
-
- context.handleResult(
- context.createKeyedBundle(myKey, created).commit(Instant.now()),
- ImmutableList.of(),
- stateResult);
-
- StepStateAndTimers afterResultContext = context.getStateAndTimers(downstreamProducer, myKey);
-
- CopyOnAccessInMemoryStateInternals<?> afterResultState = afterResultContext.stateInternals();
- assertThat(afterResultState.state(StateNamespaces.global(), intBag).read(), contains(1, 2, 4));
- }
-
- @Test
- public void callAfterOutputMustHaveBeenProducedAfterEndOfWatermarkCallsback() throws Exception {
- final CountDownLatch callLatch = new CountDownLatch(1);
- Runnable callback = callLatch::countDown;
-
- // Should call back after the end of the global window
- context.scheduleAfterOutputWouldBeProduced(
- downstream, GlobalWindow.INSTANCE, WindowingStrategy.globalDefault(), callback);
-
- TransformResult<?> result =
- StepTransformResult.withHold(createdProducer, new Instant(0)).build();
-
- context.handleResult(null, ImmutableList.of(), result);
- // Difficult to demonstrate that we took no action in a multithreaded world; poll for a bit
- // will likely be flaky if this logic is broken
- assertThat(callLatch.await(500L, TimeUnit.MILLISECONDS), is(false));
-
- TransformResult<?> finishedResult = StepTransformResult.withoutHold(createdProducer).build();
- context.handleResult(null, ImmutableList.of(), finishedResult);
- context.forceRefresh();
- // Obtain the value via blocking call
- assertThat(callLatch.await(1, TimeUnit.SECONDS), is(true));
- }
-
- @Test
- public void callAfterOutputMustHaveBeenProducedAlreadyAfterCallsImmediately() throws Exception {
- TransformResult<?> finishedResult = StepTransformResult.withoutHold(createdProducer).build();
- context.handleResult(null, ImmutableList.of(), finishedResult);
-
- final CountDownLatch callLatch = new CountDownLatch(1);
- context.extractFiredTimers();
- Runnable callback = callLatch::countDown;
- context.scheduleAfterOutputWouldBeProduced(
- downstream, GlobalWindow.INSTANCE, WindowingStrategy.globalDefault(), callback);
- assertThat(callLatch.await(1, TimeUnit.SECONDS), is(true));
- }
-
- @Test
- public void extractFiredTimersExtractsTimers() {
- TransformResult<?> holdResult =
- StepTransformResult.withHold(createdProducer, new Instant(0)).build();
- context.handleResult(null, ImmutableList.of(), holdResult);
-
- StructuralKey<?> key = StructuralKey.of("foo".length(), VarIntCoder.of());
- TimerData toFire =
- TimerData.of(StateNamespaces.global(), new Instant(100L), TimeDomain.EVENT_TIME);
- TransformResult<?> timerResult =
- StepTransformResult.withoutHold(downstreamProducer)
- .withState(CopyOnAccessInMemoryStateInternals.withUnderlying(key, null))
- .withTimerUpdate(TimerUpdate.builder(key).setTimer(toFire).build())
- .build();
-
- // haven't added any timers, must be empty
- assertThat(context.extractFiredTimers(), emptyIterable());
- context.handleResult(
- context.createKeyedBundle(key, created).commit(Instant.now()),
- ImmutableList.of(),
- timerResult);
-
- // timer hasn't fired
- assertThat(context.extractFiredTimers(), emptyIterable());
-
- TransformResult<?> advanceResult = StepTransformResult.withoutHold(createdProducer).build();
- // Should cause the downstream timer to fire
- context.handleResult(null, ImmutableList.of(), advanceResult);
-
- Collection<FiredTimers<PTransformNode>> fired = context.extractFiredTimers();
- assertThat(Iterables.getOnlyElement(fired).getKey(), Matchers.equalTo(key));
-
- FiredTimers<PTransformNode> firedForKey = Iterables.getOnlyElement(fired);
- // Contains exclusively the fired timer
- assertThat(firedForKey.getTimers(), contains(toFire));
-
- // Don't reextract timers
- assertThat(context.extractFiredTimers(), emptyIterable());
- }
-
- @Test
- public void createKeyedBundleKeyed() {
- StructuralKey<String> key = StructuralKey.of("foo", StringUtf8Coder.of());
- CommittedBundle<KV<String, Integer>> keyedBundle =
- context
- .<String, KV<String, Integer>>createKeyedBundle(key, downstream)
- .commit(Instant.now());
- assertThat(keyedBundle.getKey(), Matchers.equalTo(key));
- }
-
- @Test
- public void isDoneWithUnboundedPCollection() {
- assertThat(context.isDone(unboundedProducer), is(false));
-
- context.handleResult(
- null, ImmutableList.of(), StepTransformResult.withoutHold(unboundedProducer).build());
- context.extractFiredTimers();
- assertThat(context.isDone(unboundedProducer), is(true));
- }
-
- @Test
- public void isDoneWithPartiallyDone() {
- assertThat(context.isDone(), is(false));
-
- UncommittedBundle<Integer> rootBundle = context.createBundle(created);
- rootBundle.add(WindowedValue.valueInGlobalWindow(1));
- CommittedResult handleResult =
- context.handleResult(
- null,
- ImmutableList.of(),
- StepTransformResult.<Integer>withoutHold(createdProducer)
- .addOutput(rootBundle)
- .build());
- @SuppressWarnings("unchecked")
- CommittedBundle<Integer> committedBundle =
- (CommittedBundle<Integer>) Iterables.getOnlyElement(handleResult.getOutputs());
- context.handleResult(
- null, ImmutableList.of(), StepTransformResult.withoutHold(unboundedProducer).build());
- assertThat(context.isDone(), is(false));
-
- for (PTransformNode consumers : graph.getPerElementConsumers(created)) {
- context.handleResult(
- committedBundle, ImmutableList.of(), StepTransformResult.withoutHold(consumers).build());
- }
- context.extractFiredTimers();
- assertThat(context.isDone(), is(true));
- }
-
- private static class TestBoundedWindow extends BoundedWindow {
- private final Instant ts;
-
- public TestBoundedWindow(Instant ts) {
- this.ts = ts;
- }
-
- @Override
- public Instant maxTimestamp() {
- return ts;
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ExecutableGraphBuilder.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ExecutableGraphBuilder.java
deleted file mode 100644
index f0f5b70..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ExecutableGraphBuilder.java
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import javax.annotation.Nullable;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-
-/**
- * A builder of simple {@link ExecutableGraph ExecutableGraphs} suitable for use in the portable
- * direct runner, to reduce verbosity of creating a graph with no payloads of any meaning.
- */
-public class ExecutableGraphBuilder {
- private final RunnerApi.Components.Builder components;
-
- private ExecutableGraphBuilder() {
- components = Components.newBuilder();
- }
-
- public static ExecutableGraphBuilder create() {
- return new ExecutableGraphBuilder();
- }
-
- public ExecutableGraphBuilder addTransform(
- String name, @Nullable String input, String... outputs) {
- PTransform.Builder pt = PTransform.newBuilder().setUniqueName(name);
- if (input != null) {
- pt = pt.putInputs("input", input);
- addPCollection(input);
- }
- for (String output : outputs) {
- pt = pt.putOutputs(output, output);
- addPCollection(output);
- }
- components.putTransforms(name, pt.build());
- return this;
- }
-
- private ExecutableGraphBuilder addPCollection(String name) {
- components.putPcollections(name, PCollection.newBuilder().setUniqueName(name).build());
- return this;
- }
-
- public PTransformNode transformNode(String name) {
- return PipelineNode.pTransform(name, components.getTransformsOrThrow(name));
- }
-
- public PCollectionNode collectionNode(String name) {
- return PipelineNode.pCollection(name, components.getPcollectionsOrThrow(name));
- }
-
- public ExecutableGraph<PTransformNode, PCollectionNode> toGraph() {
- return PortableGraph.forPipeline(
- Pipeline.newBuilder()
- .setComponents(components)
- .addAllRootTransformIds(components.getTransformsMap().keySet())
- .build());
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactoryTest.java
deleted file mode 100644
index e183895..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactoryTest.java
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.junit.Assert.assertThat;
-
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.joda.time.Instant;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link FlattenEvaluatorFactory}. */
-@RunWith(JUnit4.class)
-public class FlattenEvaluatorFactoryTest {
- private BundleFactory bundleFactory = ImmutableListBundleFactory.create();
-
- @Test
- public void testFlattenInMemoryEvaluator() throws Exception {
- PCollectionNode left =
- PipelineNode.pCollection("left", PCollection.newBuilder().setUniqueName("left").build());
- PCollectionNode right =
- PipelineNode.pCollection("right", PCollection.newBuilder().setUniqueName("right").build());
- // Include a root node for a sane-looking graph
- PTransformNode source =
- PipelineNode.pTransform(
- "source",
- PTransform.newBuilder()
- .putOutputs("left", left.getId())
- .putOutputs("right", right.getId())
- .build());
-
- PCollectionNode flattened =
- PipelineNode.pCollection("flat", PCollection.newBuilder().setUniqueName("flat").build());
- PTransformNode flatten =
- PipelineNode.pTransform(
- "flatten",
- PTransform.newBuilder()
- .setUniqueName("flatten")
- .putInputs("left", left.getId())
- .putInputs("right", right.getId())
- .putOutputs("out", flattened.getId())
- .setSpec(
- FunctionSpec.newBuilder().setUrn(PTransformTranslation.FLATTEN_TRANSFORM_URN))
- .build());
-
- PortableGraph graph =
- PortableGraph.forPipeline(
- RunnerApi.Pipeline.newBuilder()
- .addRootTransformIds(source.getId())
- .addRootTransformIds(flatten.getId())
- .setComponents(
- RunnerApi.Components.newBuilder()
- .putTransforms(source.getId(), source.getTransform())
- .putPcollections(left.getId(), left.getPCollection())
- .putPcollections(right.getId(), right.getPCollection())
- .putTransforms(flatten.getId(), flatten.getTransform())
- .putPcollections(flattened.getId(), flattened.getPCollection()))
- .build());
-
- CommittedBundle<Integer> leftBundle =
- bundleFactory.<Integer>createBundle(left).commit(Instant.now());
- CommittedBundle<Integer> rightBundle =
- bundleFactory.<Integer>createBundle(right).commit(Instant.now());
-
- FlattenEvaluatorFactory factory = new FlattenEvaluatorFactory(graph, bundleFactory);
- TransformEvaluator<Integer> leftSideEvaluator = factory.forApplication(flatten, leftBundle);
- TransformEvaluator<Integer> rightSideEvaluator = factory.forApplication(flatten, rightBundle);
-
- leftSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(1));
- rightSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(-1));
- leftSideEvaluator.processElement(
- WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1024)));
- leftSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(4, PaneInfo.NO_FIRING));
- rightSideEvaluator.processElement(
- WindowedValue.valueInGlobalWindow(2, PaneInfo.ON_TIME_AND_ONLY_FIRING));
- rightSideEvaluator.processElement(
- WindowedValue.timestampedValueInGlobalWindow(-4, new Instant(-4096)));
-
- TransformResult<Integer> rightSideResult = rightSideEvaluator.finishBundle();
- TransformResult<Integer> leftSideResult = leftSideEvaluator.finishBundle();
-
- assertThat(
- getOnlyElement(leftSideResult.getOutputBundles()).commit(Instant.now()),
- containsInAnyOrder(
- WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1024)),
- WindowedValue.valueInGlobalWindow(4, PaneInfo.NO_FIRING),
- WindowedValue.valueInGlobalWindow(1)));
- assertThat(
- getOnlyElement(rightSideResult.getOutputBundles()).commit(Instant.now()),
- containsInAnyOrder(
- WindowedValue.valueInGlobalWindow(2, PaneInfo.ON_TIME_AND_ONLY_FIRING),
- WindowedValue.timestampedValueInGlobalWindow(-4, new Instant(-4096)),
- WindowedValue.valueInGlobalWindow(-1)));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactoryTest.java
deleted file mode 100644
index 10db613..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactoryTest.java
+++ /dev/null
@@ -1,247 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.contains;
-import static org.junit.Assert.assertThat;
-
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.KeyedWorkItems;
-import org.apache.beam.runners.core.construction.Environments;
-import org.apache.beam.runners.core.construction.RehydratedComponents;
-import org.apache.beam.runners.core.construction.SdkComponents;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.fnexecution.wire.LengthPrefixUnknownCoders;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multiset;
-import org.hamcrest.BaseMatcher;
-import org.hamcrest.Description;
-import org.joda.time.Instant;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link GroupByKeyOnlyEvaluatorFactory}. */
-@RunWith(JUnit4.class)
-public class GroupByKeyOnlyEvaluatorFactoryTest {
- private BundleFactory bundleFactory = ImmutableListBundleFactory.create();
-
- @Test
- public void testInMemoryEvaluator() throws Exception {
- KvCoder<String, Integer> javaCoder = KvCoder.of(StringUtf8Coder.of(), VarIntCoder.of());
- SdkComponents sdkComponents = SdkComponents.create();
- sdkComponents.registerEnvironment(Environments.createDockerEnvironment("java"));
- String windowingStrategyId =
- sdkComponents.registerWindowingStrategy(WindowingStrategy.globalDefault());
- String coderId = sdkComponents.registerCoder(javaCoder);
- Components.Builder builder = sdkComponents.toComponents().toBuilder();
- String javaWireCoderId =
- LengthPrefixUnknownCoders.addLengthPrefixedCoder(coderId, builder, false);
- String runnerWireCoderId =
- LengthPrefixUnknownCoders.addLengthPrefixedCoder(coderId, builder, true);
- RehydratedComponents components = RehydratedComponents.forComponents(builder.build());
- Coder<KV<String, Integer>> javaWireCoder =
- (Coder<KV<String, Integer>>) components.getCoder(javaWireCoderId);
- Coder<KV<?, ?>> runnerWireCoder = (Coder<KV<?, ?>>) components.getCoder(runnerWireCoderId);
-
- KV<?, ?> firstFoo = asRunnerKV(javaWireCoder, runnerWireCoder, KV.of("foo", -1));
- KV<?, ?> secondFoo = asRunnerKV(javaWireCoder, runnerWireCoder, KV.of("foo", 1));
- KV<?, ?> thirdFoo = asRunnerKV(javaWireCoder, runnerWireCoder, KV.of("foo", 3));
- KV<?, ?> firstBar = asRunnerKV(javaWireCoder, runnerWireCoder, KV.of("bar", 22));
- KV<?, ?> secondBar = asRunnerKV(javaWireCoder, runnerWireCoder, KV.of("bar", 12));
- KV<?, ?> firstBaz = asRunnerKV(javaWireCoder, runnerWireCoder, KV.of("baz", Integer.MAX_VALUE));
-
- PTransformNode inputTransform =
- PipelineNode.pTransform(
- "source", PTransform.newBuilder().putOutputs("out", "values").build());
- PCollectionNode values =
- PipelineNode.pCollection(
- "values",
- RunnerApi.PCollection.newBuilder()
- .setUniqueName("values")
- .setCoderId(coderId)
- .setWindowingStrategyId(windowingStrategyId)
- .build());
- PCollectionNode groupedKvs =
- PipelineNode.pCollection(
- "groupedKvs", RunnerApi.PCollection.newBuilder().setUniqueName("groupedKvs").build());
- PTransformNode groupByKeyOnly =
- PipelineNode.pTransform(
- "gbko",
- PTransform.newBuilder()
- .putInputs("input", "values")
- .putOutputs("output", "groupedKvs")
- .setSpec(FunctionSpec.newBuilder().setUrn(DirectGroupByKey.DIRECT_GBKO_URN).build())
- .build());
- Pipeline pipeline =
- Pipeline.newBuilder()
- .addRootTransformIds(inputTransform.getId())
- .addRootTransformIds(groupByKeyOnly.getId())
- .setComponents(
- builder
- .putTransforms(inputTransform.getId(), inputTransform.getTransform())
- .putTransforms(groupByKeyOnly.getId(), groupByKeyOnly.getTransform())
- .putPcollections(values.getId(), values.getPCollection())
- .putPcollections(groupedKvs.getId(), groupedKvs.getPCollection()))
- .build();
-
- PortableGraph graph = PortableGraph.forPipeline(pipeline);
-
- CommittedBundle<KV<String, Integer>> inputBundle =
- bundleFactory.<KV<String, Integer>>createBundle(values).commit(Instant.now());
-
- TransformEvaluator<KV<?, ?>> evaluator =
- new GroupByKeyOnlyEvaluatorFactory(graph, pipeline.getComponents(), bundleFactory)
- .forApplication(groupByKeyOnly, inputBundle);
-
- evaluator.processElement(WindowedValue.valueInGlobalWindow(firstFoo));
- evaluator.processElement(WindowedValue.valueInGlobalWindow(secondFoo));
- evaluator.processElement(WindowedValue.valueInGlobalWindow(thirdFoo));
- evaluator.processElement(WindowedValue.valueInGlobalWindow(firstBar));
- evaluator.processElement(WindowedValue.valueInGlobalWindow(secondBar));
- evaluator.processElement(WindowedValue.valueInGlobalWindow(firstBaz));
-
- TransformResult<KV<?, ?>> result = evaluator.finishBundle();
-
- // The input to a GroupByKey is assumed to be a KvCoder
- @SuppressWarnings({"rawtypes", "unchecked"})
- Coder runnerKeyCoder = ((KvCoder) runnerWireCoder).getKeyCoder();
- CommittedBundle<?> fooBundle = null;
- CommittedBundle<?> barBundle = null;
- CommittedBundle<?> bazBundle = null;
- StructuralKey fooKey = StructuralKey.of(firstFoo.getKey(), runnerKeyCoder);
- StructuralKey barKey = StructuralKey.of(firstBar.getKey(), runnerKeyCoder);
- StructuralKey bazKey = StructuralKey.of(firstBaz.getKey(), runnerKeyCoder);
- for (UncommittedBundle<?> groupedBundle : result.getOutputBundles()) {
- CommittedBundle<?> groupedCommitted = groupedBundle.commit(Instant.now());
- if (fooKey.equals(groupedCommitted.getKey())) {
- fooBundle = groupedCommitted;
- } else if (barKey.equals(groupedCommitted.getKey())) {
- barBundle = groupedCommitted;
- } else if (bazKey.equals(groupedCommitted.getKey())) {
- bazBundle = groupedCommitted;
- } else {
- throw new IllegalArgumentException(
- String.format("Unknown Key %s", groupedCommitted.getKey()));
- }
- }
- assertThat(
- fooBundle,
- contains(
- new KeyedWorkItemMatcher(
- KeyedWorkItems.elementsWorkItem(
- fooKey.getKey(),
- ImmutableSet.of(
- WindowedValue.valueInGlobalWindow(firstFoo.getValue()),
- WindowedValue.valueInGlobalWindow(secondFoo.getValue()),
- WindowedValue.valueInGlobalWindow(thirdFoo.getValue()))),
- runnerKeyCoder)));
- assertThat(
- barBundle,
- contains(
- new KeyedWorkItemMatcher<>(
- KeyedWorkItems.elementsWorkItem(
- barKey.getKey(),
- ImmutableSet.of(
- WindowedValue.valueInGlobalWindow(firstBar.getValue()),
- WindowedValue.valueInGlobalWindow(secondBar.getValue()))),
- runnerKeyCoder)));
- assertThat(
- bazBundle,
- contains(
- new KeyedWorkItemMatcher<>(
- KeyedWorkItems.elementsWorkItem(
- bazKey.getKey(),
- ImmutableSet.of(WindowedValue.valueInGlobalWindow(firstBaz.getValue()))),
- runnerKeyCoder)));
- }
-
- private KV<?, ?> asRunnerKV(
- Coder<KV<String, Integer>> javaWireCoder,
- Coder<KV<?, ?>> runnerWireCoder,
- KV<String, Integer> value)
- throws org.apache.beam.sdk.coders.CoderException {
- return CoderUtils.decodeFromByteArray(
- runnerWireCoder, CoderUtils.encodeToByteArray(javaWireCoder, value));
- }
-
- private <K, V> KV<K, WindowedValue<V>> gwValue(KV<K, V> kv) {
- return KV.of(kv.getKey(), WindowedValue.valueInGlobalWindow(kv.getValue()));
- }
-
- private static class KeyedWorkItemMatcher<K, V>
- extends BaseMatcher<WindowedValue<KeyedWorkItem<K, V>>> {
- private final KeyedWorkItem<K, V> myWorkItem;
- private final Coder<K> keyCoder;
-
- public KeyedWorkItemMatcher(KeyedWorkItem<K, V> myWorkItem, Coder<K> keyCoder) {
- this.myWorkItem = myWorkItem;
- this.keyCoder = keyCoder;
- }
-
- @Override
- public boolean matches(Object item) {
- if (item == null || !(item instanceof WindowedValue)) {
- return false;
- }
- WindowedValue<KeyedWorkItem<K, V>> that = (WindowedValue<KeyedWorkItem<K, V>>) item;
- Multiset<WindowedValue<V>> myValues = HashMultiset.create();
- Multiset<WindowedValue<V>> thatValues = HashMultiset.create();
- for (WindowedValue<V> value : myWorkItem.elementsIterable()) {
- myValues.add(value);
- }
- for (WindowedValue<V> value : that.getValue().elementsIterable()) {
- thatValues.add(value);
- }
- try {
- return myValues.equals(thatValues)
- && keyCoder
- .structuralValue(myWorkItem.key())
- .equals(keyCoder.structuralValue(that.getValue().key()));
- } catch (Exception e) {
- return false;
- }
- }
-
- @Override
- public void describeTo(Description description) {
- description
- .appendText("KeyedWorkItem<K, V> containing key ")
- .appendValue(myWorkItem.key())
- .appendText(" and values ")
- .appendValueList("[", ", ", "]", myWorkItem.elementsIterable());
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactoryTest.java
deleted file mode 100644
index 232cb82..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactoryTest.java
+++ /dev/null
@@ -1,234 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.not;
-import static org.junit.Assert.assertThat;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.local.StructuralKey;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.coders.VoidCoder;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.hamcrest.Matcher;
-import org.hamcrest.Matchers;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link ImmutableListBundleFactory}. */
-@RunWith(JUnit4.class)
-public class ImmutableListBundleFactoryTest {
- @Rule public final TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- private ImmutableListBundleFactory bundleFactory = ImmutableListBundleFactory.create();
-
- private PCollectionNode created;
- private PCollectionNode downstream;
-
- @Before
- public void setup() {
- created =
- PipelineNode.pCollection(
- "created", RunnerApi.PCollection.newBuilder().setUniqueName("created").build());
- created =
- PipelineNode.pCollection(
- "downstream", RunnerApi.PCollection.newBuilder().setUniqueName("downstream").build());
- }
-
- private <T> void createKeyedBundle(Coder<T> coder, T key) throws Exception {
- StructuralKey<?> skey = StructuralKey.of(key, coder);
-
- UncommittedBundle<Integer> inFlightBundle = bundleFactory.createKeyedBundle(skey, created);
-
- CommittedBundle<Integer> bundle = inFlightBundle.commit(Instant.now());
- assertThat(bundle.getKey(), Matchers.equalTo(skey));
- }
-
- @Test
- public void keyedWithNullKeyShouldCreateKeyedBundle() throws Exception {
- createKeyedBundle(VoidCoder.of(), null);
- }
-
- @Test
- public void keyedWithStringKeyShouldCreateKeyedBundle() throws Exception {
- createKeyedBundle(StringUtf8Coder.of(), "foo");
- }
-
- @Test
- public void keyedWithVarIntKeyShouldCreateKeyedBundle() throws Exception {
- createKeyedBundle(VarIntCoder.of(), 1234);
- }
-
- @Test
- public void keyedWithByteArrayKeyShouldCreateKeyedBundle() throws Exception {
- createKeyedBundle(ByteArrayCoder.of(), new byte[] {0, 2, 4, 99});
- }
-
- private <T> CommittedBundle<T> afterCommitGetElementsShouldHaveAddedElements(
- Iterable<WindowedValue<T>> elems) {
- UncommittedBundle<T> bundle = bundleFactory.createRootBundle();
- Collection<Matcher<? super WindowedValue<T>>> expectations = new ArrayList<>();
- Instant minElementTs = BoundedWindow.TIMESTAMP_MAX_VALUE;
- for (WindowedValue<T> elem : elems) {
- bundle.add(elem);
- expectations.add(equalTo(elem));
- if (elem.getTimestamp().isBefore(minElementTs)) {
- minElementTs = elem.getTimestamp();
- }
- }
- Matcher<Iterable<? extends WindowedValue<T>>> containsMatcher =
- Matchers.containsInAnyOrder(expectations);
- Instant commitTime = Instant.now();
- CommittedBundle<T> committed = bundle.commit(commitTime);
- assertThat(committed.getElements(), containsMatcher);
-
- // Sanity check that the test is meaningful.
- assertThat(minElementTs, not(equalTo(commitTime)));
- assertThat(committed.getMinimumTimestamp(), equalTo(minElementTs));
- assertThat(committed.getSynchronizedProcessingOutputWatermark(), equalTo(commitTime));
-
- return committed;
- }
-
- @Test
- public void getElementsBeforeAddShouldReturnEmptyIterable() {
- afterCommitGetElementsShouldHaveAddedElements(Collections.<WindowedValue<Integer>>emptyList());
- }
-
- @Test
- public void getElementsAfterAddShouldReturnAddedElements() {
- WindowedValue<Integer> firstValue = WindowedValue.valueInGlobalWindow(1);
- WindowedValue<Integer> secondValue =
- WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1000L));
-
- afterCommitGetElementsShouldHaveAddedElements(ImmutableList.of(firstValue, secondValue));
- }
-
- @Test
- public void addElementsAtEndOfTimeThrows() {
- Instant timestamp = BoundedWindow.TIMESTAMP_MAX_VALUE;
- WindowedValue<Integer> value = WindowedValue.timestampedValueInGlobalWindow(1, timestamp);
-
- UncommittedBundle<Integer> bundle = bundleFactory.createRootBundle();
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage(timestamp.toString());
- bundle.add(value);
- }
-
- @Test
- public void addElementsPastEndOfTimeThrows() {
- Instant timestamp = BoundedWindow.TIMESTAMP_MAX_VALUE.plus(Duration.standardMinutes(2));
- WindowedValue<Integer> value = WindowedValue.timestampedValueInGlobalWindow(1, timestamp);
-
- UncommittedBundle<Integer> bundle = bundleFactory.createRootBundle();
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage(timestamp.toString());
- bundle.add(value);
- }
-
- @SuppressWarnings("unchecked")
- @Test
- public void withElementsShouldReturnIndependentBundle() {
- WindowedValue<Integer> firstValue = WindowedValue.valueInGlobalWindow(1);
- WindowedValue<Integer> secondValue =
- WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1000L));
-
- CommittedBundle<Integer> committed =
- afterCommitGetElementsShouldHaveAddedElements(ImmutableList.of(firstValue, secondValue));
-
- WindowedValue<Integer> firstReplacement =
- WindowedValue.of(
- 9,
- new Instant(2048L),
- new IntervalWindow(new Instant(2044L), Instant.now()),
- PaneInfo.NO_FIRING);
- WindowedValue<Integer> secondReplacement =
- WindowedValue.timestampedValueInGlobalWindow(-1, Instant.now());
- CommittedBundle<Integer> withed =
- committed.withElements(ImmutableList.of(firstReplacement, secondReplacement));
-
- assertThat(withed.getElements(), containsInAnyOrder(firstReplacement, secondReplacement));
- assertThat(committed.getElements(), containsInAnyOrder(firstValue, secondValue));
- assertThat(withed.getKey(), Matchers.equalTo(committed.getKey()));
- assertThat(withed.getPCollection(), equalTo(committed.getPCollection()));
- assertThat(
- withed.getSynchronizedProcessingOutputWatermark(),
- equalTo(committed.getSynchronizedProcessingOutputWatermark()));
- assertThat(withed.getMinimumTimestamp(), equalTo(new Instant(2048L)));
- }
-
- @Test
- public void addAfterCommitShouldThrowException() {
- UncommittedBundle<Integer> bundle = bundleFactory.createRootBundle();
- bundle.add(WindowedValue.valueInGlobalWindow(1));
- CommittedBundle<Integer> firstCommit = bundle.commit(Instant.now());
- assertThat(firstCommit.getElements(), containsInAnyOrder(WindowedValue.valueInGlobalWindow(1)));
-
- thrown.expect(IllegalStateException.class);
- thrown.expectMessage("3");
- thrown.expectMessage("committed");
-
- bundle.add(WindowedValue.valueInGlobalWindow(3));
- }
-
- @Test
- public void commitAfterCommitShouldThrowException() {
- UncommittedBundle<Integer> bundle = bundleFactory.createRootBundle();
- bundle.add(WindowedValue.valueInGlobalWindow(1));
- CommittedBundle<Integer> firstCommit = bundle.commit(Instant.now());
- assertThat(firstCommit.getElements(), containsInAnyOrder(WindowedValue.valueInGlobalWindow(1)));
-
- thrown.expect(IllegalStateException.class);
- thrown.expectMessage("committed");
-
- bundle.commit(Instant.now());
- }
-
- @Test
- public void createKeyedBundleKeyed() {
- CommittedBundle<KV<String, Integer>> keyedBundle =
- bundleFactory
- .<String, KV<String, Integer>>createKeyedBundle(
- StructuralKey.of("foo", StringUtf8Coder.of()), downstream)
- .commit(Instant.now());
- assertThat(keyedBundle.getKey().getKey(), Matchers.equalTo("foo"));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactoryTest.java
deleted file mode 100644
index d7275d2..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactoryTest.java
+++ /dev/null
@@ -1,130 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.contains;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.hasSize;
-import static org.junit.Assert.assertArrayEquals;
-import static org.junit.Assert.assertThat;
-
-import java.util.Collection;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.ExecutableGraph;
-import org.apache.beam.runners.direct.portable.ImpulseEvaluatorFactory.ImpulseRootProvider;
-import org.apache.beam.runners.direct.portable.ImpulseEvaluatorFactory.ImpulseShard;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.MockitoAnnotations;
-
-/** Tests for {@link ImpulseEvaluatorFactory}. */
-@RunWith(JUnit4.class)
-public class ImpulseEvaluatorFactoryTest {
- private BundleFactory bundleFactory = ImmutableListBundleFactory.create();
-
- private PTransformNode impulseApplication =
- PipelineNode.pTransform(
- "impulse",
- PTransform.newBuilder()
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(PTransformTranslation.IMPULSE_TRANSFORM_URN)
- .build())
- .putOutputs("output", "impulse.out")
- .build());
- private PCollectionNode impulseOut =
- PipelineNode.pCollection(
- "impulse.out", RunnerApi.PCollection.newBuilder().setUniqueName("impulse.out").build());
- private ExecutableGraph<PTransformNode, PCollectionNode> graph =
- PortableGraph.forPipeline(
- RunnerApi.Pipeline.newBuilder()
- .addRootTransformIds("impulse")
- .setComponents(
- Components.newBuilder()
- .putTransforms("impulse", impulseApplication.getTransform())
- .putPcollections("impulse.out", impulseOut.getPCollection()))
- .build());
-
- @Before
- public void setup() {
- MockitoAnnotations.initMocks(this);
- }
-
- @Test
- public void testImpulse() throws Exception {
-
- ImpulseEvaluatorFactory factory = new ImpulseEvaluatorFactory(graph, bundleFactory);
-
- WindowedValue<ImpulseShard> inputShard = WindowedValue.valueInGlobalWindow(new ImpulseShard());
- CommittedBundle<ImpulseShard> inputShardBundle =
- bundleFactory.<ImpulseShard>createRootBundle().add(inputShard).commit(Instant.now());
-
- TransformEvaluator<ImpulseShard> evaluator =
- factory.forApplication(impulseApplication, inputShardBundle);
- evaluator.processElement(inputShard);
- TransformResult<ImpulseShard> result = evaluator.finishBundle();
- assertThat(
- "Exactly one output from a single ImpulseShard",
- Iterables.size(result.getOutputBundles()),
- equalTo(1));
- UncommittedBundle<?> outputBundle = result.getOutputBundles().iterator().next();
- CommittedBundle<?> committedOutputBundle = outputBundle.commit(Instant.now());
- assertThat(
- committedOutputBundle.getMinimumTimestamp(), equalTo(BoundedWindow.TIMESTAMP_MIN_VALUE));
- assertThat(committedOutputBundle.getPCollection(), equalTo(impulseOut));
- assertThat(
- "Should only be one impulse element",
- Iterables.size(committedOutputBundle.getElements()),
- equalTo(1));
- assertThat(
- committedOutputBundle.getElements().iterator().next().getWindows(),
- contains(GlobalWindow.INSTANCE));
- assertArrayEquals(
- "Output should be an empty byte array",
- new byte[0],
- (byte[]) committedOutputBundle.getElements().iterator().next().getValue());
- }
-
- @Test
- public void testRootProvider() {
- ImpulseRootProvider rootProvider = new ImpulseRootProvider(bundleFactory);
-
- Collection<? extends CommittedBundle<?>> inputs =
- rootProvider.getInitialInputs(impulseApplication, 100);
-
- assertThat("Only one impulse bundle per application", inputs, hasSize(1));
- assertThat(
- "Only one impulse shard per bundle",
- Iterables.size(inputs.iterator().next().getElements()),
- equalTo(1));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/MockClock.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/MockClock.java
deleted file mode 100644
index 181d8b5..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/MockClock.java
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-
-import org.apache.beam.runners.direct.Clock;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-
-/**
- * A clock that returns a constant value for now which can be set with calls to {@link
- * #set(Instant)}.
- *
- * <p>For uses of the {@link Clock} interface in unit tests.
- */
-class MockClock implements Clock {
-
- private Instant now;
-
- public static MockClock fromInstant(Instant initial) {
- return new MockClock(initial);
- }
-
- private MockClock(Instant initialNow) {
- this.now = initialNow;
- }
-
- public void set(Instant newNow) {
- checkArgument(
- !newNow.isBefore(now),
- "Cannot move MockClock backwards in time from %s to %s",
- now,
- newNow);
- this.now = newNow;
- }
-
- public void advance(Duration duration) {
- checkArgument(
- duration.getMillis() > 0,
- "Cannot move MockClock backwards in time by duration %s",
- duration);
- set(now.plus(duration));
- }
-
- @Override
- public Instant now() {
- return now;
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/PortableGraphTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/PortableGraphTest.java
deleted file mode 100644
index b9447eb..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/PortableGraphTest.java
+++ /dev/null
@@ -1,176 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.hasSize;
-import static org.hamcrest.Matchers.hasValue;
-import static org.junit.Assert.assertThat;
-
-import java.io.Serializable;
-import java.util.Collection;
-import java.util.List;
-import java.util.Objects;
-import java.util.stream.Collectors;
-import java.util.stream.Stream;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.PipelineTranslation;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.Impulse;
-import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.Values;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.sdk.values.TypeDescriptor;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link PortableGraph}. */
-@RunWith(JUnit4.class)
-public class PortableGraphTest implements Serializable {
- @Test
- public void getRootTransformsSucceeds() {
- Pipeline pipeline = Pipeline.create();
- pipeline.apply("impulse", Impulse.create());
- pipeline.apply("otherImpulse", Impulse.create());
-
- PortableGraph graph = PortableGraph.forPipeline(PipelineTranslation.toProto(pipeline));
- assertThat(graph.getRootTransforms(), hasSize(2));
-
- assertThat(
- graph.getRootTransforms().stream().map(PTransformNode::getId).collect(Collectors.toSet()),
- containsInAnyOrder("impulse", "otherImpulse"));
- }
-
- @Test
- public void getExecutablesReturnsTransforms() {
- Pipeline pipeline = Pipeline.create();
- pipeline
- .apply("Impulse", Impulse.create())
- .apply(
- "ParDo",
- ParDo.of(
- new DoFn<byte[], KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext ctxt) {
- ctxt.output(KV.of("foo", "bar"));
- }
- }))
- .apply(GroupByKey.create())
- .apply(Values.create());
-
- PortableGraph graph = PortableGraph.forPipeline(PipelineTranslation.toProto(pipeline));
- assertThat(graph.getExecutables(), hasSize(4));
- }
-
- @Test
- public void getExecutablesWithStages() {
- Pipeline pipeline = Pipeline.create();
- pipeline
- .apply("Impulse", Impulse.create())
- .apply(
- "ParDo",
- ParDo.of(
- new DoFn<byte[], KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext ctxt) {
- ctxt.output(KV.of("foo", "bar"));
- }
- }))
- .apply(
- MapElements.into(new TypeDescriptor<KV<String, Integer>>() {})
- .via(input -> KV.of(input.getKey(), input.getValue().hashCode())))
- .apply(GroupByKey.create())
- .apply(Values.create());
-
- RunnerApi.Pipeline proto = PipelineTranslation.toProto(pipeline);
- RunnerApi.Pipeline fused = GreedyPipelineFuser.fuse(proto).toPipeline();
- PortableGraph graph = PortableGraph.forPipeline(fused);
- assertThat(graph.getExecutables(), hasSize(4));
-
- Stream<FunctionSpec> specStream =
- graph.getExecutables().stream().map(PTransformNode::getTransform).map(PTransform::getSpec);
-
- List<String> urns = specStream.map(FunctionSpec::getUrn).collect(Collectors.toList());
- assertThat(
- urns,
- containsInAnyOrder(
- PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN,
- PTransformTranslation.IMPULSE_TRANSFORM_URN,
- ExecutableStage.URN,
- ExecutableStage.URN));
- }
-
- @Test
- public void getProducedAndGetProducerSucceed() {
- Pipeline pipeline = Pipeline.create();
- TupleTag<KV<String, String>> mainTag = new TupleTag<>();
- TupleTag<Long> otherTag = new TupleTag<Long>() {};
- pipeline
- .apply("Impulse", Impulse.create())
- .apply(
- "ParDo",
- ParDo.of(
- new DoFn<byte[], KV<String, String>>() {
- @ProcessElement
- public void processElement(ProcessContext ctxt) {
- ctxt.output(KV.of("foo", "bar"));
- }
- })
- .withOutputTags(mainTag, TupleTagList.of(otherTag)))
- .get(mainTag)
- .apply(
- MapElements.into(new TypeDescriptor<KV<String, Integer>>() {})
- .via(input -> KV.of(input.getKey(), Objects.hash(input.getValue()))))
- .apply("gbk", GroupByKey.create())
- .apply("vals", Values.create());
-
- RunnerApi.Pipeline proto = PipelineTranslation.toProto(pipeline);
- PortableGraph graph = PortableGraph.forPipeline(proto);
-
- PTransformNode gbkNode =
- PipelineNode.pTransform("gbk", proto.getComponents().getTransformsOrThrow("gbk"));
- Collection<PCollectionNode> gbkOutput = graph.getProduced(gbkNode);
- assertThat(gbkOutput, hasSize(1));
- assertThat(graph.getProducer(getOnlyElement(gbkOutput)), equalTo(gbkNode));
-
- PTransformNode parDoNode =
- PipelineNode.pTransform("ParDo", proto.getComponents().getTransformsOrThrow("ParDo"));
- Collection<PCollectionNode> parDoOutput = graph.getProduced(parDoNode);
- assertThat(parDoOutput, hasSize(2));
- for (PCollectionNode parDoOutputNode : parDoOutput) {
- assertThat(graph.getProducer(parDoOutputNode), equalTo(parDoNode));
- assertThat(parDoNode.getTransform().getOutputsMap(), hasValue(parDoOutputNode.getId()));
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ReferenceRunnerTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ReferenceRunnerTest.java
deleted file mode 100644
index 0645673..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ReferenceRunnerTest.java
+++ /dev/null
@@ -1,213 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.resume;
-import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.stop;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
-
-import java.io.Serializable;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.Set;
-import org.apache.beam.runners.core.construction.JavaReadViaImpulse;
-import org.apache.beam.runners.core.construction.PTransformMatchers;
-import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
-import org.apache.beam.runners.core.construction.PipelineTranslation;
-import org.apache.beam.runners.direct.ParDoMultiOverrideFactory;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.io.range.OffsetRange;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.runners.PTransformOverride;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.Reshuffle;
-import org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker;
-import org.apache.beam.sdk.transforms.windowing.FixedWindows;
-import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionTuple;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Ignore;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for the {@link ReferenceRunner}. */
-@RunWith(JUnit4.class)
-public class ReferenceRunnerTest implements Serializable {
- @Test
- public void pipelineExecution() throws Exception {
- Pipeline p = Pipeline.create();
- TupleTag<KV<String, Integer>> food = new TupleTag<>();
- TupleTag<Integer> originals = new TupleTag<Integer>() {};
- PCollectionTuple parDoOutputs =
- p.apply(Create.of(1, 2, 3))
- .apply(
- ParDo.of(
- new DoFn<Integer, KV<String, Integer>>() {
- @ProcessElement
- public void process(@Element Integer e, MultiOutputReceiver r) {
- for (int i = 0; i < e; i++) {
- r.get(food)
- .outputWithTimestamp(
- KV.of("foo", e),
- new Instant(0).plus(Duration.standardHours(i)));
- }
- r.get(originals).output(e);
- }
- })
- .withOutputTags(food, TupleTagList.of(originals)));
- FixedWindows windowFn = FixedWindows.of(Duration.standardMinutes(5L));
- PCollection<KV<String, Set<Integer>>> grouped =
- parDoOutputs
- .get(food)
- .apply(Window.into(windowFn))
- .apply(GroupByKey.create())
- .apply(
- ParDo.of(
- new DoFn<KV<String, Iterable<Integer>>, KV<String, Set<Integer>>>() {
- @ProcessElement
- public void process(
- @Element KV<String, Iterable<Integer>> e,
- OutputReceiver<KV<String, Set<Integer>>> r) {
- r.output(KV.of(e.getKey(), ImmutableSet.copyOf(e.getValue())));
- }
- }));
-
- PAssert.that(grouped)
- .containsInAnyOrder(
- KV.of("foo", ImmutableSet.of(1, 2, 3)),
- KV.of("foo", ImmutableSet.of(2, 3)),
- KV.of("foo", ImmutableSet.of(3)));
-
- p.replaceAll(Collections.singletonList(JavaReadViaImpulse.boundedOverride()));
-
- ReferenceRunner runner =
- ReferenceRunner.forInProcessPipeline(
- PipelineTranslation.toProto(p),
- PipelineOptionsTranslation.toProto(PipelineOptionsFactory.create()));
- runner.execute();
- }
-
- @Test
- public void testGBK() throws Exception {
- Pipeline p = Pipeline.create();
-
- PAssert.that(
- p.apply(Create.of(KV.of(42, 0), KV.of(42, 1), KV.of(42, 2)))
- // Will create one bundle for each value, since direct runner uses 1 bundle per key
- .apply(Reshuffle.viaRandomKey())
- // Multiple bundles will emit values onto the same key 42.
- // They must be processed sequentially rather than in parallel, since
- // the trigger firing code expects to receive values sequentially for a key.
- .apply(GroupByKey.create()))
- .satisfies(
- input -> {
- KV<Integer, Iterable<Integer>> kv = Iterables.getOnlyElement(input);
- assertEquals(42, kv.getKey().intValue());
- assertThat(kv.getValue(), containsInAnyOrder(0, 1, 2));
- return null;
- });
-
- p.replaceAll(Collections.singletonList(JavaReadViaImpulse.boundedOverride()));
-
- ReferenceRunner runner =
- ReferenceRunner.forInProcessPipeline(
- PipelineTranslation.toProto(p),
- PipelineOptionsTranslation.toProto(PipelineOptionsFactory.create()));
- runner.execute();
- }
-
- static class PairStringWithIndexToLength extends DoFn<String, KV<String, Integer>> {
- @ProcessElement
- public ProcessContinuation process(ProcessContext c, OffsetRangeTracker tracker) {
- for (long i = tracker.currentRestriction().getFrom(), numIterations = 0;
- tracker.tryClaim(i);
- ++i, ++numIterations) {
- c.output(KV.of(c.element(), (int) i));
- if (numIterations % 3 == 0) {
- return resume();
- }
- }
- return stop();
- }
-
- @GetInitialRestriction
- public OffsetRange getInitialRange(String element) {
- return new OffsetRange(0, element.length());
- }
-
- @SplitRestriction
- public void splitRange(
- String element, OffsetRange range, OutputReceiver<OffsetRange> receiver) {
- long middle = (range.getFrom() + range.getTo()) / 2;
- receiver.output(new OffsetRange(range.getFrom(), middle));
- receiver.output(new OffsetRange(middle, range.getTo()));
- }
- }
-
- @Test
- @Ignore("TODO: BEAM-3743")
- public void testSDF() throws Exception {
- Pipeline p = Pipeline.create();
-
- PCollection<KV<String, Integer>> res =
- p.apply(Create.of("a", "bb", "ccccc"))
- .apply(ParDo.of(new PairStringWithIndexToLength()))
- .setCoder(KvCoder.of(StringUtf8Coder.of(), BigEndianIntegerCoder.of()));
-
- PAssert.that(res)
- .containsInAnyOrder(
- Arrays.asList(
- KV.of("a", 0),
- KV.of("bb", 0),
- KV.of("bb", 1),
- KV.of("ccccc", 0),
- KV.of("ccccc", 1),
- KV.of("ccccc", 2),
- KV.of("ccccc", 3),
- KV.of("ccccc", 4)));
-
- p.replaceAll(
- Arrays.asList(
- JavaReadViaImpulse.boundedOverride(),
- PTransformOverride.of(
- PTransformMatchers.splittableParDo(), new ParDoMultiOverrideFactory())));
-
- ReferenceRunner runner =
- ReferenceRunner.forInProcessPipeline(
- PipelineTranslation.toProto(p),
- PipelineOptionsTranslation.toProto(PipelineOptionsFactory.create()));
- runner.execute();
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java
deleted file mode 100644
index fd55e85..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java
+++ /dev/null
@@ -1,238 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-
-import java.io.Serializable;
-import java.util.concurrent.BlockingQueue;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.LinkedBlockingQueue;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.runners.core.construction.PipelineTranslation;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
-import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
-import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
-import org.apache.beam.runners.fnexecution.control.SingleEnvironmentInstanceJobBundleFactory;
-import org.apache.beam.runners.fnexecution.data.GrpcDataService;
-import org.apache.beam.runners.fnexecution.environment.EmbeddedEnvironmentFactory;
-import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
-import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
-import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
-import org.apache.beam.runners.fnexecution.state.GrpcStateService;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.fn.IdGenerators;
-import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.Impulse;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Instant;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link RemoteStageEvaluatorFactory}. */
-@RunWith(JUnit4.class)
-public class RemoteStageEvaluatorFactoryTest implements Serializable {
-
- private transient RemoteStageEvaluatorFactory factory;
- private transient ExecutorService executor;
- private transient GrpcFnServer<GrpcDataService> dataServer;
- private transient GrpcFnServer<GrpcStateService> stateServer;
- private transient GrpcFnServer<FnApiControlClientPoolService> controlServer;
- private transient GrpcFnServer<GrpcLoggingService> loggingServer;
- private transient BundleFactory bundleFactory;
-
- @Before
- public void setup() throws Exception {
- InProcessServerFactory serverFactory = InProcessServerFactory.create();
-
- BlockingQueue<InstructionRequestHandler> clientPool = new LinkedBlockingQueue<>();
- controlServer =
- GrpcFnServer.allocatePortAndCreateFor(
- FnApiControlClientPoolService.offeringClientsToPool(
- (workerId, instructionHandler) -> clientPool.put(instructionHandler),
- GrpcContextHeaderAccessorProvider.getHeaderAccessor()),
- serverFactory);
- loggingServer =
- GrpcFnServer.allocatePortAndCreateFor(
- GrpcLoggingService.forWriter(Slf4jLogWriter.getDefault()), serverFactory);
-
- EnvironmentFactory environmentFactory =
- EmbeddedEnvironmentFactory.create(
- PipelineOptionsFactory.create(),
- loggingServer,
- controlServer,
- (workerId, timeout) -> clientPool.take());
- executor = Executors.newCachedThreadPool();
- dataServer =
- GrpcFnServer.allocatePortAndCreateFor(
- GrpcDataService.create(executor, OutboundObserverFactory.serverDirect()),
- serverFactory);
- stateServer = GrpcFnServer.allocatePortAndCreateFor(GrpcStateService.create(), serverFactory);
-
- bundleFactory = ImmutableListBundleFactory.create();
- JobBundleFactory jobBundleFactory =
- SingleEnvironmentInstanceJobBundleFactory.create(
- environmentFactory, dataServer, stateServer, IdGenerators.incrementingLongs());
- factory = new RemoteStageEvaluatorFactory(bundleFactory, jobBundleFactory);
- }
-
- @After
- public void teardown() throws Exception {
- try (AutoCloseable logging = loggingServer;
- AutoCloseable exec = executor::shutdownNow;
- AutoCloseable data = dataServer;
- AutoCloseable state = stateServer;
- AutoCloseable control = controlServer) {}
- }
-
- @Test
- public void executesRemoteStage() throws Exception {
- Pipeline p = Pipeline.create();
- p.apply("impulse", Impulse.create())
- .apply(
- "CreateInputs",
- ParDo.of(
- new DoFn<byte[], Integer>() {
- @ProcessElement
- public void create(ProcessContext ctxt) {
- ctxt.output(1);
- ctxt.output(2);
- ctxt.output(3);
- }
- }))
- .apply(
- "ParDo",
- ParDo.of(
- new DoFn<Integer, KV<String, Long>>() {
- @ProcessElement
- public void proc(ProcessContext ctxt) {
- ctxt.output(KV.of("foo", ctxt.element().longValue()));
- }
- }))
- .apply(GroupByKey.create());
-
- RunnerApi.Pipeline fusedPipeline =
- GreedyPipelineFuser.fuse(PipelineTranslation.toProto(p)).toPipeline();
- QueryablePipeline fusedQP = QueryablePipeline.forPipeline(fusedPipeline);
- PTransformNode impulseTransform = getOnlyElement(fusedQP.getRootTransforms());
- PCollectionNode impulseOutput = getOnlyElement(fusedQP.getOutputPCollections(impulseTransform));
- PTransformNode stage =
- fusedPipeline.getRootTransformIdsList().stream()
- .map(
- id ->
- PipelineNode.pTransform(
- id, fusedPipeline.getComponents().getTransformsOrThrow(id)))
- .filter(node -> node.getTransform().getSpec().getUrn().equals(ExecutableStage.URN))
- .findFirst()
- .orElseThrow(IllegalArgumentException::new);
-
- WindowedValue<byte[]> impulse = WindowedValue.valueInGlobalWindow(new byte[0]);
- CommittedBundle<byte[]> inputBundle =
- bundleFactory.<byte[]>createBundle(impulseOutput).add(impulse).commit(Instant.now());
- TransformEvaluator<byte[]> evaluator = factory.forApplication(stage, inputBundle);
- evaluator.processElement(impulse);
- TransformResult<byte[]> result = evaluator.finishBundle();
- assertThat(Iterables.size(result.getOutputBundles()), equalTo(1));
- CommittedBundle<?> outputs = getOnlyElement(result.getOutputBundles()).commit(Instant.now());
- assertThat(Iterables.size(outputs), equalTo(3));
- }
-
- @Test
- public void executesStageWithFlatten() throws Exception {
- ParDo.SingleOutput<byte[], KV<Integer, String>> parDo =
- ParDo.of(
- new DoFn<byte[], KV<Integer, String>>() {
- @ProcessElement
- public void process(ProcessContext ctxt) {
- ctxt.output(KV.of(1, "foo"));
- ctxt.output(KV.of(1, "bar"));
- ctxt.output(KV.of(2, "foo"));
- }
- });
- Pipeline p = Pipeline.create();
-
- PCollection<KV<Integer, String>> left = p.apply("left", Impulse.create()).apply(parDo);
- PCollection<KV<Integer, String>> right = p.apply("right", Impulse.create()).apply(parDo);
- PCollectionList.of(left).and(right).apply(Flatten.pCollections()).apply(GroupByKey.create());
-
- RunnerApi.Pipeline fusedPipeline =
- GreedyPipelineFuser.fuse(PipelineTranslation.toProto(p)).toPipeline();
- QueryablePipeline fusedQP = QueryablePipeline.forPipeline(fusedPipeline);
- PTransformNode leftRoot = null;
- PTransformNode rightRoot = null;
- for (PTransformNode root : fusedQP.getRootTransforms()) {
- if ("left".equals(root.getId())) {
- leftRoot = root;
- } else {
- rightRoot = root;
- }
- }
- checkState(leftRoot != null);
- checkState(rightRoot != null);
- PTransformNode stage =
- fusedPipeline.getRootTransformIdsList().stream()
- .map(
- id ->
- PipelineNode.pTransform(
- id, fusedPipeline.getComponents().getTransformsOrThrow(id)))
- .filter(node -> node.getTransform().getSpec().getUrn().equals(ExecutableStage.URN))
- .findFirst()
- .orElseThrow(IllegalArgumentException::new);
-
- WindowedValue<byte[]> impulse = WindowedValue.valueInGlobalWindow(new byte[0]);
- String inputId = getOnlyElement(stage.getTransform().getInputsMap().values());
- CommittedBundle<byte[]> inputBundle =
- bundleFactory
- .<byte[]>createBundle(
- PipelineNode.pCollection(
- inputId, fusedPipeline.getComponents().getPcollectionsOrThrow(inputId)))
- .add(impulse)
- .commit(Instant.now());
- TransformEvaluator<byte[]> evaluator = factory.forApplication(stage, inputBundle);
- evaluator.processElement(impulse);
- TransformResult<byte[]> result = evaluator.finishBundle();
- assertThat(Iterables.size(result.getOutputBundles()), equalTo(1));
- CommittedBundle<?> outputs = getOnlyElement(result.getOutputBundles()).commit(Instant.now());
- assertThat(Iterables.size(outputs), equalTo(3));
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/StepTransformResultTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/StepTransformResultTest.java
deleted file mode 100644
index e12ac01..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/StepTransformResultTest.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.emptyIterable;
-import static org.hamcrest.Matchers.hasItem;
-import static org.junit.Assert.assertThat;
-
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.runners.direct.portable.CommittedResult.OutputType;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.hamcrest.Matchers;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link StepTransformResult}. */
-@RunWith(JUnit4.class)
-public class StepTransformResultTest {
- private PTransformNode transform;
- private BundleFactory bundleFactory;
- private PCollectionNode pc;
-
- @Rule public TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
-
- @Before
- public void setup() {
- pc =
- PipelineNode.pCollection(
- "pc", RunnerApi.PCollection.newBuilder().setUniqueName("pc").build());
- transform =
- PipelineNode.pTransform("pt", PTransform.newBuilder().putOutputs("out", "pc").build());
-
- bundleFactory = ImmutableListBundleFactory.create();
- }
-
- @Test
- public void producedBundlesProducedOutputs() {
- UncommittedBundle<Integer> bundle = bundleFactory.createBundle(pc);
- TransformResult<Integer> result =
- StepTransformResult.<Integer>withoutHold(transform).addOutput(bundle).build();
-
- assertThat(result.getOutputBundles(), Matchers.containsInAnyOrder(bundle));
- }
-
- @Test
- public void withAdditionalOutputProducedOutputs() {
- TransformResult<Integer> result =
- StepTransformResult.<Integer>withoutHold(transform)
- .withAdditionalOutput(OutputType.PCOLLECTION_VIEW)
- .build();
-
- assertThat(result.getOutputTypes(), containsInAnyOrder(OutputType.PCOLLECTION_VIEW));
- }
-
- @Test
- public void producedBundlesAndAdditionalOutputProducedOutputs() {
- TransformResult<Integer> result =
- StepTransformResult.<Integer>withoutHold(transform)
- .addOutput(bundleFactory.createBundle(pc))
- .withAdditionalOutput(OutputType.PCOLLECTION_VIEW)
- .build();
-
- assertThat(result.getOutputTypes(), hasItem(OutputType.PCOLLECTION_VIEW));
- }
-
- @Test
- public void noBundlesNoAdditionalOutputProducedOutputsFalse() {
- TransformResult<Integer> result = StepTransformResult.<Integer>withoutHold(transform).build();
-
- assertThat(result.getOutputTypes(), emptyIterable());
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/TransformExecutorServicesTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/TransformExecutorServicesTest.java
deleted file mode 100644
index a761fa0..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/TransformExecutorServicesTest.java
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.never;
-import static org.mockito.Mockito.verify;
-
-import java.util.concurrent.ExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link TransformExecutorServices}. */
-@RunWith(JUnit4.class)
-public class TransformExecutorServicesTest {
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- private ExecutorService executorService;
-
- @Before
- public void setup() {
- executorService = MoreExecutors.newDirectExecutorService();
- }
-
- @Test
- public void parallelScheduleMultipleSchedulesBothImmediately() {
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> first = mock(DirectTransformExecutor.class);
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> second = mock(DirectTransformExecutor.class);
-
- TransformExecutorService parallel = TransformExecutorServices.parallel(executorService);
- parallel.schedule(first);
- parallel.schedule(second);
-
- verify(first).run();
- verify(second).run();
-
- parallel.complete(first);
- parallel.complete(second);
- }
-
- @Test
- public void parallelRejectedStillActiveThrows() {
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> first = mock(DirectTransformExecutor.class);
-
- TransformExecutorService parallel = TransformExecutorServices.parallel(executorService);
- executorService.shutdown();
- thrown.expect(IllegalStateException.class);
- thrown.expectMessage("still active");
- parallel.schedule(first);
- }
-
- @Test
- public void parallelRejectedShutdownSucceeds() {
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> first = mock(DirectTransformExecutor.class);
-
- TransformExecutorService parallel = TransformExecutorServices.parallel(executorService);
- executorService.shutdown();
- parallel.shutdown();
- parallel.schedule(first);
- }
-
- @Test
- public void serialScheduleTwoWaitsForFirstToComplete() {
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> first = mock(DirectTransformExecutor.class);
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> second = mock(DirectTransformExecutor.class);
-
- TransformExecutorService serial = TransformExecutorServices.serial(executorService);
- serial.schedule(first);
- verify(first).run();
-
- serial.schedule(second);
- verify(second, never()).run();
-
- serial.complete(first);
- verify(second).run();
-
- serial.complete(second);
- }
-
- @Test
- public void serialCompleteNotExecutingTaskThrows() {
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> first = mock(DirectTransformExecutor.class);
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> second = mock(DirectTransformExecutor.class);
-
- TransformExecutorService serial = TransformExecutorServices.serial(executorService);
- serial.schedule(first);
- thrown.expect(IllegalStateException.class);
- thrown.expectMessage("unexpected currently executing");
-
- serial.complete(second);
- }
-
- /**
- * Tests that a Serial {@link TransformExecutorService} does not schedule follow up work if the
- * executor is shut down when the initial work completes.
- */
- @Test
- public void serialShutdownCompleteActive() {
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> first = mock(DirectTransformExecutor.class);
- @SuppressWarnings("unchecked")
- DirectTransformExecutor<Object> second = mock(DirectTransformExecutor.class);
-
- TransformExecutorService serial = TransformExecutorServices.serial(executorService);
- serial.schedule(first);
- verify(first).run();
-
- serial.schedule(second);
- verify(second, never()).run();
-
- serial.shutdown();
- serial.complete(first);
- verify(second, never()).run();
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutorTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutorTest.java
deleted file mode 100644
index ac69875..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutorTest.java
+++ /dev/null
@@ -1,136 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.Executors;
-import java.util.concurrent.TimeUnit;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.FixedWindows;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link WatermarkCallbackExecutor}. */
-@RunWith(JUnit4.class)
-public class WatermarkCallbackExecutorTest {
- private WatermarkCallbackExecutor executor =
- WatermarkCallbackExecutor.create(Executors.newSingleThreadExecutor());
- private PTransformNode create;
- private PTransformNode sum;
-
- @Before
- public void setup() {
- create =
- PipelineNode.pTransform(
- "create",
- PTransform.newBuilder()
- .setUniqueName("create")
- .putInputs("in", "impulse.out")
- .putOutputs("out", "create.out")
- .build());
- sum =
- PipelineNode.pTransform(
- "sum",
- PTransform.newBuilder()
- .setUniqueName("sum")
- .putInputs("in", "create.in")
- .putOutputs("out", "sum.out")
- .build());
- }
-
- @Test
- public void onGuaranteedFiringFiresAfterTrigger() throws Exception {
- CountDownLatch latch = new CountDownLatch(1);
- executor.callOnGuaranteedFiring(
- create,
- GlobalWindow.INSTANCE,
- WindowingStrategy.globalDefault(),
- new CountDownLatchCallback(latch));
-
- executor.fireForWatermark(create, BoundedWindow.TIMESTAMP_MAX_VALUE);
- assertThat(latch.await(500, TimeUnit.MILLISECONDS), equalTo(true));
- }
-
- @Test
- public void multipleCallbacksShouldFireFires() throws Exception {
- CountDownLatch latch = new CountDownLatch(2);
- WindowFn<Object, IntervalWindow> windowFn = FixedWindows.of(Duration.standardMinutes(10));
- IntervalWindow window =
- new IntervalWindow(new Instant(0L), new Instant(0L).plus(Duration.standardMinutes(10)));
- executor.callOnGuaranteedFiring(
- create, window, WindowingStrategy.of(windowFn), new CountDownLatchCallback(latch));
- executor.callOnGuaranteedFiring(
- create, window, WindowingStrategy.of(windowFn), new CountDownLatchCallback(latch));
-
- executor.fireForWatermark(create, new Instant(0L).plus(Duration.standardMinutes(10)));
- assertThat(latch.await(500, TimeUnit.MILLISECONDS), equalTo(true));
- }
-
- @Test
- public void noCallbacksShouldFire() throws Exception {
- CountDownLatch latch = new CountDownLatch(1);
- WindowFn<Object, IntervalWindow> windowFn = FixedWindows.of(Duration.standardMinutes(10));
- IntervalWindow window =
- new IntervalWindow(new Instant(0L), new Instant(0L).plus(Duration.standardMinutes(10)));
- executor.callOnGuaranteedFiring(
- create, window, WindowingStrategy.of(windowFn), new CountDownLatchCallback(latch));
-
- executor.fireForWatermark(create, new Instant(0L).plus(Duration.standardMinutes(5)));
- assertThat(latch.await(500, TimeUnit.MILLISECONDS), equalTo(false));
- }
-
- @Test
- public void unrelatedStepShouldNotFire() throws Exception {
- CountDownLatch latch = new CountDownLatch(1);
- WindowFn<Object, IntervalWindow> windowFn = FixedWindows.of(Duration.standardMinutes(10));
- IntervalWindow window =
- new IntervalWindow(new Instant(0L), new Instant(0L).plus(Duration.standardMinutes(10)));
- executor.callOnGuaranteedFiring(
- sum, window, WindowingStrategy.of(windowFn), new CountDownLatchCallback(latch));
-
- executor.fireForWatermark(create, new Instant(0L).plus(Duration.standardMinutes(20)));
- assertThat(latch.await(500, TimeUnit.MILLISECONDS), equalTo(false));
- }
-
- private static class CountDownLatchCallback implements Runnable {
- private final CountDownLatch latch;
-
- public CountDownLatchCallback(CountDownLatch latch) {
- this.latch = latch;
- }
-
- @Override
- public void run() {
- latch.countDown();
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactoryTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactoryTest.java
deleted file mode 100644
index 2eb654b..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactoryTest.java
+++ /dev/null
@@ -1,314 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable;
-
-import static org.apache.beam.runners.core.WindowMatchers.isSingleWindowedValue;
-import static org.apache.beam.runners.core.WindowMatchers.isWindowedValue;
-import static org.apache.beam.sdk.transforms.windowing.PaneInfo.NO_FIRING;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.junit.Assert.assertThat;
-import static org.mockito.Mockito.when;
-
-import java.util.Collection;
-import java.util.Collections;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
-import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.IncompatibleWindowException;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.hamcrest.Matchers;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Ignore;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.MockitoAnnotations;
-
-/** Tests for {@link WindowEvaluatorFactory}. */
-@RunWith(JUnit4.class)
-@Ignore("TODO BEAM-4241 Not yet migrated")
-public class WindowEvaluatorFactoryTest {
- private static final Instant EPOCH = new Instant(0);
-
- private PCollectionNode input;
- private WindowEvaluatorFactory factory;
-
- @Mock private EvaluationContext evaluationContext;
-
- private BundleFactory bundleFactory;
-
- private WindowedValue<Long> valueInGlobalWindow =
- WindowedValue.timestampedValueInGlobalWindow(3L, new Instant(2L));
-
- private final PaneInfo intervalWindowPane = PaneInfo.createPane(false, false, Timing.LATE, 3, 2);
- private WindowedValue<Long> valueInIntervalWindow =
- WindowedValue.of(
- 2L, new Instant(-10L), new IntervalWindow(new Instant(-100), EPOCH), intervalWindowPane);
-
- private IntervalWindow intervalWindow1 =
- new IntervalWindow(EPOCH, BoundedWindow.TIMESTAMP_MAX_VALUE);
-
- private IntervalWindow intervalWindow2 =
- new IntervalWindow(
- EPOCH.plus(Duration.standardDays(3)), EPOCH.plus(Duration.standardDays(6)));
-
- private final PaneInfo multiWindowPane = PaneInfo.createPane(false, true, Timing.ON_TIME, 3, 0);
- private WindowedValue<Long> valueInGlobalAndTwoIntervalWindows =
- WindowedValue.of(
- 1L,
- EPOCH.plus(Duration.standardDays(3)),
- ImmutableList.of(GlobalWindow.INSTANCE, intervalWindow1, intervalWindow2),
- multiWindowPane);
-
- @Rule public TestPipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);
-
- @Before
- public void setup() {
- MockitoAnnotations.initMocks(this);
- input =
- PipelineNode.pCollection(
- "created", RunnerApi.PCollection.newBuilder().setUniqueName("created").build());
-
- bundleFactory = ImmutableListBundleFactory.create();
- factory = new WindowEvaluatorFactory(evaluationContext);
- }
-
- @Test
- public void singleWindowFnSucceeds() throws Exception {
- Duration windowDuration = Duration.standardDays(7);
- CommittedBundle<Long> inputBundle = createInputBundle();
-
- PCollectionNode windowed = null;
- UncommittedBundle<Long> outputBundle = createOutputBundle(windowed);
-
- BoundedWindow firstSecondWindow = new IntervalWindow(EPOCH, EPOCH.plus(windowDuration));
- BoundedWindow thirdWindow = new IntervalWindow(EPOCH.minus(windowDuration), EPOCH);
-
- TransformResult<Long> result = runEvaluator(inputBundle);
-
- assertThat(Iterables.getOnlyElement(result.getOutputBundles()), Matchers.equalTo(outputBundle));
- CommittedBundle<Long> committed = outputBundle.commit(Instant.now());
-
- assertThat(
- committed.getElements(),
- containsInAnyOrder(
- // value in global window
- isSingleWindowedValue(3L, new Instant(2L), firstSecondWindow, NO_FIRING),
-
- // value in just interval window
- isSingleWindowedValue(2L, new Instant(-10L), thirdWindow, intervalWindowPane),
-
- // value in global window and two interval windows
- isSingleWindowedValue(
- 1L, EPOCH.plus(Duration.standardDays(3)), firstSecondWindow, multiWindowPane),
- isSingleWindowedValue(
- 1L, EPOCH.plus(Duration.standardDays(3)), firstSecondWindow, multiWindowPane),
- isSingleWindowedValue(
- 1L, EPOCH.plus(Duration.standardDays(3)), firstSecondWindow, multiWindowPane)));
- }
-
- @Test
- public void multipleWindowsWindowFnSucceeds() throws Exception {
- Duration windowDuration = Duration.standardDays(6);
- Duration slidingBy = Duration.standardDays(3);
-
- CommittedBundle<Long> inputBundle = createInputBundle();
-
- PCollectionNode windowed = null;
- UncommittedBundle<Long> outputBundle = createOutputBundle(windowed);
-
- TransformResult<Long> result = runEvaluator(inputBundle);
-
- assertThat(Iterables.getOnlyElement(result.getOutputBundles()), Matchers.equalTo(outputBundle));
- CommittedBundle<Long> committed = outputBundle.commit(Instant.now());
-
- BoundedWindow w1 = new IntervalWindow(EPOCH, EPOCH.plus(windowDuration));
- BoundedWindow w2 =
- new IntervalWindow(EPOCH.plus(slidingBy), EPOCH.plus(slidingBy).plus(windowDuration));
- BoundedWindow wMinus1 = new IntervalWindow(EPOCH.minus(windowDuration), EPOCH);
- BoundedWindow wMinusSlide =
- new IntervalWindow(EPOCH.minus(windowDuration).plus(slidingBy), EPOCH.plus(slidingBy));
-
- assertThat(
- committed.getElements(),
- containsInAnyOrder(
- // Value in global window mapped to one windowed value in multiple windows
- isWindowedValue(
- valueInGlobalWindow.getValue(),
- valueInGlobalWindow.getTimestamp(),
- ImmutableSet.of(w1, wMinusSlide),
- NO_FIRING),
-
- // Value in interval window mapped to one windowed value in multiple windows
- isWindowedValue(
- valueInIntervalWindow.getValue(),
- valueInIntervalWindow.getTimestamp(),
- ImmutableSet.of(wMinus1, wMinusSlide),
- valueInIntervalWindow.getPane()),
-
- // Value in three windows mapped to three windowed values in the same multiple windows
- isWindowedValue(
- valueInGlobalAndTwoIntervalWindows.getValue(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- ImmutableSet.of(w1, w2),
- valueInGlobalAndTwoIntervalWindows.getPane()),
- isWindowedValue(
- valueInGlobalAndTwoIntervalWindows.getValue(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- ImmutableSet.of(w1, w2),
- valueInGlobalAndTwoIntervalWindows.getPane()),
- isWindowedValue(
- valueInGlobalAndTwoIntervalWindows.getValue(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- ImmutableSet.of(w1, w2),
- valueInGlobalAndTwoIntervalWindows.getPane())));
- }
-
- @Test
- public void referencesEarlierWindowsSucceeds() throws Exception {
- CommittedBundle<Long> inputBundle = createInputBundle();
-
- PCollectionNode windowed = null;
- UncommittedBundle<Long> outputBundle = createOutputBundle(windowed);
-
- TransformResult<Long> result = runEvaluator(inputBundle);
-
- assertThat(Iterables.getOnlyElement(result.getOutputBundles()), Matchers.equalTo(outputBundle));
- CommittedBundle<Long> committed = outputBundle.commit(Instant.now());
-
- assertThat(
- committed.getElements(),
- containsInAnyOrder(
- // Value in global window mapped to [timestamp, timestamp+1)
- isSingleWindowedValue(
- valueInGlobalWindow.getValue(),
- valueInGlobalWindow.getTimestamp(),
- new IntervalWindow(
- valueInGlobalWindow.getTimestamp(),
- valueInGlobalWindow.getTimestamp().plus(1L)),
- valueInGlobalWindow.getPane()),
-
- // Value in interval window mapped to the same window
- isWindowedValue(
- valueInIntervalWindow.getValue(),
- valueInIntervalWindow.getTimestamp(),
- valueInIntervalWindow.getWindows(),
- valueInIntervalWindow.getPane()),
-
- // Value in global window and two interval windows exploded and mapped in both ways
- isSingleWindowedValue(
- valueInGlobalAndTwoIntervalWindows.getValue(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- new IntervalWindow(
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp().plus(1L)),
- valueInGlobalAndTwoIntervalWindows.getPane()),
- isSingleWindowedValue(
- valueInGlobalAndTwoIntervalWindows.getValue(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- intervalWindow1,
- valueInGlobalAndTwoIntervalWindows.getPane()),
- isSingleWindowedValue(
- valueInGlobalAndTwoIntervalWindows.getValue(),
- valueInGlobalAndTwoIntervalWindows.getTimestamp(),
- intervalWindow2,
- valueInGlobalAndTwoIntervalWindows.getPane())));
- }
-
- private CommittedBundle<Long> createInputBundle() {
- CommittedBundle<Long> inputBundle =
- bundleFactory
- .<Long>createBundle(input)
- .add(valueInGlobalWindow)
- .add(valueInGlobalAndTwoIntervalWindows)
- .add(valueInIntervalWindow)
- .commit(Instant.now());
- return inputBundle;
- }
-
- private UncommittedBundle<Long> createOutputBundle(PCollectionNode output) {
- UncommittedBundle<Long> outputBundle = bundleFactory.createBundle(output);
- when(evaluationContext.<Long>createBundle(output)).thenReturn(outputBundle);
- throw new UnsupportedOperationException("Not yet migrated");
- }
-
- private TransformResult<Long> runEvaluator(CommittedBundle<Long> inputBundle) throws Exception {
- PTransformNode window = null;
- TransformEvaluator<Long> evaluator = factory.forApplication(window, inputBundle);
-
- evaluator.processElement(valueInGlobalWindow);
- evaluator.processElement(valueInGlobalAndTwoIntervalWindows);
- evaluator.processElement(valueInIntervalWindow);
- TransformResult<Long> result = evaluator.finishBundle();
- throw new UnsupportedOperationException("Not yet migrated");
- }
-
- private static class EvaluatorTestWindowFn extends NonMergingWindowFn<Long, BoundedWindow> {
- @Override
- public Collection<BoundedWindow> assignWindows(AssignContext c) throws Exception {
- if (c.window().equals(GlobalWindow.INSTANCE)) {
- return Collections.singleton(new IntervalWindow(c.timestamp(), c.timestamp().plus(1L)));
- }
- return Collections.singleton(c.window());
- }
-
- @Override
- public boolean isCompatible(WindowFn<?, ?> other) {
- return false;
- }
-
- @Override
- public void verifyCompatibility(WindowFn<?, ?> other) throws IncompatibleWindowException {
- throw new IncompatibleWindowException(
- other,
- String.format(
- "%s is not compatible with any other %s.",
- EvaluatorTestWindowFn.class.getSimpleName(), WindowFn.class.getSimpleName()));
- }
-
- @Override
- public Coder<BoundedWindow> windowCoder() {
- @SuppressWarnings({"unchecked", "rawtypes"})
- Coder coder = (Coder) GlobalWindow.Coder.INSTANCE;
- return coder;
- }
-
- @Override
- public WindowMappingFn<BoundedWindow> getDefaultWindowMappingFn() {
- throw new UnsupportedOperationException("Cannot be used as a side input");
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocationTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocationTest.java
deleted file mode 100644
index df2c100..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocationTest.java
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.is;
-import static org.junit.Assert.assertThat;
-
-import java.io.File;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.rules.TemporaryFolder;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link LocalArtifactStagingLocation}. */
-@RunWith(JUnit4.class)
-public class LocalArtifactStagingLocationTest {
- @Rule public TemporaryFolder tmp = new TemporaryFolder();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void createAtWithAbsentDirectory() throws Exception {
- File baseFolder = tmp.newFolder();
- File root = new File(baseFolder, "foo");
-
- checkState(!root.exists());
- LocalArtifactStagingLocation.createAt(root);
-
- assertThat(root.exists(), is(true));
- assertThat(root.listFiles().length, equalTo(1));
- }
-
- @Test
- public void createAtWithExistingDirectory() throws Exception {
- File root = tmp.newFolder();
- checkState(root.exists(), "root directory must exist");
-
- assertThat(root.exists(), is(true));
- assertThat(root.listFiles().length, equalTo(0));
- LocalArtifactStagingLocation.createAt(root);
-
- assertThat(root.exists(), is(true));
- assertThat(root.listFiles().length, equalTo(1));
- }
-
- @Test
- public void createAtWithUnwritableDirectory() throws Exception {
- File baseFolder = tmp.newFolder();
- File root = new File(baseFolder, "foo");
- checkState(root.mkdir(), "Must be able to create the root directory");
-
- assertThat(root.exists(), is(true));
- checkState(root.setWritable(false), "Must be able to set the root directory to unwritable");
-
- thrown.expect(IllegalStateException.class);
- LocalArtifactStagingLocation.createAt(root);
- }
-
- @Test
- public void testCreateAtThenForExisting() throws Exception {
- File baseFolder = tmp.newFolder();
- LocalArtifactStagingLocation newLocation = LocalArtifactStagingLocation.createAt(baseFolder);
- File newManifest = newLocation.getManifestFile();
- checkState(newManifest.createNewFile(), "Manifest creation failed");
- File newArtifact = newLocation.getArtifactFile("my_artifact");
- checkState(newArtifact.createNewFile(), "Artifact creation failed");
-
- LocalArtifactStagingLocation forExisting =
- LocalArtifactStagingLocation.forExistingDirectory(baseFolder);
- assertThat(forExisting.getManifestFile(), equalTo(newManifest));
- assertThat(forExisting.getArtifactFile("my_artifact"), equalTo(newArtifact));
- }
-
- @Test
- public void testForExistingWithoutRoot() throws Exception {
- File baseFolder = tmp.newFolder();
- File root = new File(baseFolder, "bar");
-
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("root");
- LocalArtifactStagingLocation.forExistingDirectory(root);
- }
-
- @Test
- public void testForExistingWithoutManifest() throws Exception {
- File baseFolder = tmp.newFolder();
- LocalArtifactStagingLocation newLocation = LocalArtifactStagingLocation.createAt(baseFolder);
- File newArtifact = newLocation.getArtifactFile("my_artifact");
- checkState(newArtifact.createNewFile(), "Artifact creation failed");
-
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("Manifest");
- LocalArtifactStagingLocation.forExistingDirectory(baseFolder);
- }
-
- @Test
- public void testForExistingWithoutArtifacts() throws Exception {
- File baseFolder = tmp.newFolder();
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("artifact directory");
-
- LocalArtifactStagingLocation.forExistingDirectory(baseFolder);
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalServiceTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalServiceTest.java
deleted file mode 100644
index 92d25ae..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalServiceTest.java
+++ /dev/null
@@ -1,240 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.containsString;
-import static org.hamcrest.Matchers.not;
-import static org.hamcrest.Matchers.nullValue;
-import static org.junit.Assert.assertArrayEquals;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.fail;
-
-import java.io.ByteArrayOutputStream;
-import java.io.File;
-import java.io.IOException;
-import java.nio.file.Files;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.atomic.AtomicReference;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetArtifactRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestResponse;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.Manifest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager.StagedFile;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.runners.fnexecution.ServerFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link LocalFileSystemArtifactRetrievalService}. */
-@RunWith(JUnit4.class)
-public class LocalFileSystemArtifactRetrievalServiceTest {
- @Rule public TemporaryFolder tmp = new TemporaryFolder();
-
- private File root;
- private ServerFactory serverFactory = InProcessServerFactory.create();
-
- private GrpcFnServer<LocalFileSystemArtifactStagerService> stagerServer;
-
- private GrpcFnServer<LocalFileSystemArtifactRetrievalService> retrievalServer;
- private ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub;
-
- @Before
- public void setup() throws Exception {
- root = tmp.newFolder();
- stagerServer =
- GrpcFnServer.allocatePortAndCreateFor(
- LocalFileSystemArtifactStagerService.forRootDirectory(root), serverFactory);
- }
-
- @After
- public void teardown() throws Exception {
- stagerServer.close();
- retrievalServer.close();
- }
-
- @Test
- public void retrieveManifest() throws Exception {
- Map<String, byte[]> artifacts = new HashMap<>();
- artifacts.put("foo", "bar, baz, quux".getBytes(UTF_8));
- artifacts.put("spam", new byte[] {127, -22, 5});
- stageAndCreateRetrievalService(artifacts);
-
- final AtomicReference<Manifest> returned = new AtomicReference<>();
- final CountDownLatch completed = new CountDownLatch(1);
- retrievalStub.getManifest(
- GetManifestRequest.getDefaultInstance(),
- new StreamObserver<GetManifestResponse>() {
- @Override
- public void onNext(GetManifestResponse value) {
- returned.set(value.getManifest());
- }
-
- @Override
- public void onError(Throwable t) {
- completed.countDown();
- }
-
- @Override
- public void onCompleted() {
- completed.countDown();
- }
- });
-
- completed.await();
- assertThat(returned.get(), not(nullValue()));
-
- List<String> manifestArtifacts = new ArrayList<>();
- for (ArtifactMetadata artifactMetadata : returned.get().getArtifactList()) {
- manifestArtifacts.add(artifactMetadata.getName());
- }
- assertThat(manifestArtifacts, containsInAnyOrder("foo", "spam"));
- }
-
- @Test
- public void retrieveArtifact() throws Exception {
- Map<String, byte[]> artifacts = new HashMap<>();
- byte[] fooContents = "bar, baz, quux".getBytes(UTF_8);
- artifacts.put("foo", fooContents);
- byte[] spamContents = {127, -22, 5};
- artifacts.put("spam", spamContents);
- stageAndCreateRetrievalService(artifacts);
-
- final CountDownLatch completed = new CountDownLatch(2);
- ByteArrayOutputStream returnedFooBytes = new ByteArrayOutputStream();
- retrievalStub.getArtifact(
- GetArtifactRequest.newBuilder().setName("foo").build(),
- new MultimapChunkAppender(returnedFooBytes, completed));
- ByteArrayOutputStream returnedSpamBytes = new ByteArrayOutputStream();
- retrievalStub.getArtifact(
- GetArtifactRequest.newBuilder().setName("spam").build(),
- new MultimapChunkAppender(returnedSpamBytes, completed));
-
- completed.await();
- assertArrayEquals(fooContents, returnedFooBytes.toByteArray());
- assertArrayEquals(spamContents, returnedSpamBytes.toByteArray());
- }
-
- @Test
- public void retrieveArtifactNotPresent() throws Exception {
- stageAndCreateRetrievalService(
- Collections.singletonMap("foo", "bar, baz, quux".getBytes(UTF_8)));
-
- final CountDownLatch completed = new CountDownLatch(1);
- final AtomicReference<Throwable> thrown = new AtomicReference<>();
- retrievalStub.getArtifact(
- GetArtifactRequest.newBuilder().setName("spam").build(),
- new StreamObserver<ArtifactChunk>() {
- @Override
- public void onNext(ArtifactChunk value) {
- fail(
- "Should never receive an "
- + ArtifactChunk.class.getSimpleName()
- + " for a nonexistent artifact");
- }
-
- @Override
- public void onError(Throwable t) {
- thrown.set(t);
- completed.countDown();
- }
-
- @Override
- public void onCompleted() {
- completed.countDown();
- }
- });
-
- completed.await();
- assertThat(thrown.get(), not(nullValue()));
- assertThat(thrown.get().getMessage(), containsString("No such artifact"));
- assertThat(thrown.get().getMessage(), containsString("spam"));
- }
-
- private void stageAndCreateRetrievalService(Map<String, byte[]> artifacts) throws Exception {
- List<StagedFile> artifactFiles = new ArrayList<>();
- for (Map.Entry<String, byte[]> artifact : artifacts.entrySet()) {
- File artifactFile = tmp.newFile(artifact.getKey());
- Files.write(artifactFile.toPath(), artifact.getValue());
- artifactFiles.add(StagedFile.of(artifactFile, artifactFile.getName()));
- }
- String stagingSessionToken = "token";
-
- ArtifactServiceStager stager =
- ArtifactServiceStager.overChannel(
- InProcessChannelBuilder.forName(stagerServer.getApiServiceDescriptor().getUrl())
- .build());
- stager.stage(stagingSessionToken, artifactFiles);
-
- retrievalServer =
- GrpcFnServer.allocatePortAndCreateFor(
- LocalFileSystemArtifactRetrievalService.forRootDirectory(root), serverFactory);
- retrievalStub =
- ArtifactRetrievalServiceGrpc.newStub(
- InProcessChannelBuilder.forName(retrievalServer.getApiServiceDescriptor().getUrl())
- .build());
- }
-
- private static class MultimapChunkAppender implements StreamObserver<ArtifactChunk> {
- private final ByteArrayOutputStream target;
- private final CountDownLatch completed;
-
- private MultimapChunkAppender(ByteArrayOutputStream target, CountDownLatch completed) {
- this.target = target;
- this.completed = completed;
- }
-
- @Override
- public void onNext(ArtifactChunk value) {
- try {
- target.write(value.getData().toByteArray());
- } catch (IOException e) {
- // This should never happen
- throw new AssertionError(e);
- }
- }
-
- @Override
- public void onError(Throwable t) {
- completed.countDown();
- }
-
- @Override
- public void onCompleted() {
- completed.countDown();
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerServiceTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerServiceTest.java
deleted file mode 100644
index 09e46b0..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerServiceTest.java
+++ /dev/null
@@ -1,313 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.hamcrest.Matchers.is;
-import static org.junit.Assert.assertThat;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.nio.ByteBuffer;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-import javax.annotation.Nullable;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
-import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
-import org.hamcrest.Matchers;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link LocalFileSystemArtifactStagerService}. */
-@RunWith(JUnit4.class)
-public class LocalFileSystemArtifactStagerServiceTest {
- @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
-
- private ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stub;
-
- private LocalFileSystemArtifactStagerService stager;
- private Server server;
-
- @Before
- public void setup() throws Exception {
- stager = LocalFileSystemArtifactStagerService.forRootDirectory(temporaryFolder.newFolder());
-
- server =
- InProcessServerBuilder.forName("fs_stager")
- .directExecutor()
- .addService(stager)
- .build()
- .start();
-
- stub =
- ArtifactStagingServiceGrpc.newStub(
- InProcessChannelBuilder.forName("fs_stager").usePlaintext().build());
- }
-
- @After
- public void teardown() {
- server.shutdownNow();
- }
-
- @Test
- public void singleDataPutArtifactSucceeds() throws Exception {
- byte[] data = "foo-bar-baz".getBytes(UTF_8);
- RecordingStreamObserver<ArtifactApi.PutArtifactResponse> responseObserver =
- new RecordingStreamObserver<>();
- StreamObserver<ArtifactApi.PutArtifactRequest> requestObserver =
- stub.putArtifact(responseObserver);
-
- String name = "my-artifact";
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setMetadata(
- ArtifactApi.PutArtifactMetadata.newBuilder()
- .setMetadata(ArtifactApi.ArtifactMetadata.newBuilder().setName(name).build())
- .setStagingSessionToken("token")
- .build())
- .build());
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(
- ArtifactApi.ArtifactChunk.newBuilder().setData(ByteString.copyFrom(data)).build())
- .build());
- requestObserver.onCompleted();
-
- responseObserver.awaitTerminalState();
-
- File staged = stager.getLocation().getArtifactFile(name);
- assertThat(staged.exists(), is(true));
- ByteBuffer buf = ByteBuffer.allocate(data.length);
- new FileInputStream(staged).getChannel().read(buf);
- Assert.assertArrayEquals(data, buf.array());
- }
-
- @Test
- public void multiPartPutArtifactSucceeds() throws Exception {
- byte[] partOne = "foo-".getBytes(UTF_8);
- byte[] partTwo = "bar-".getBytes(UTF_8);
- byte[] partThree = "baz".getBytes(UTF_8);
- RecordingStreamObserver<ArtifactApi.PutArtifactResponse> responseObserver =
- new RecordingStreamObserver<>();
- StreamObserver<ArtifactApi.PutArtifactRequest> requestObserver =
- stub.putArtifact(responseObserver);
-
- String name = "my-artifact";
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setMetadata(
- ArtifactApi.PutArtifactMetadata.newBuilder()
- .setMetadata(ArtifactApi.ArtifactMetadata.newBuilder().setName(name).build())
- .setStagingSessionToken("token")
- .build())
- .build());
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(
- ArtifactApi.ArtifactChunk.newBuilder()
- .setData(ByteString.copyFrom(partOne))
- .build())
- .build());
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(
- ArtifactApi.ArtifactChunk.newBuilder()
- .setData(ByteString.copyFrom(partTwo))
- .build())
- .build());
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(
- ArtifactApi.ArtifactChunk.newBuilder()
- .setData(ByteString.copyFrom(partThree))
- .build())
- .build());
- requestObserver.onCompleted();
-
- responseObserver.awaitTerminalState();
-
- File staged = stager.getLocation().getArtifactFile(name);
- assertThat(staged.exists(), is(true));
- ByteBuffer buf = ByteBuffer.allocate("foo-bar-baz".length());
- new FileInputStream(staged).getChannel().read(buf);
- Assert.assertArrayEquals("foo-bar-baz".getBytes(UTF_8), buf.array());
- }
-
- @Test
- public void putArtifactBeforeNameFails() {
- byte[] data = "foo-".getBytes(UTF_8);
- RecordingStreamObserver<ArtifactApi.PutArtifactResponse> responseObserver =
- new RecordingStreamObserver<>();
- StreamObserver<ArtifactApi.PutArtifactRequest> requestObserver =
- stub.putArtifact(responseObserver);
-
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(
- ArtifactApi.ArtifactChunk.newBuilder().setData(ByteString.copyFrom(data)).build())
- .build());
-
- responseObserver.awaitTerminalState();
-
- assertThat(responseObserver.error, Matchers.not(Matchers.nullValue()));
- }
-
- @Test
- public void putArtifactWithNoContentFails() {
- RecordingStreamObserver<ArtifactApi.PutArtifactResponse> responseObserver =
- new RecordingStreamObserver<>();
- StreamObserver<ArtifactApi.PutArtifactRequest> requestObserver =
- stub.putArtifact(responseObserver);
-
- requestObserver.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(ArtifactApi.ArtifactChunk.getDefaultInstance())
- .build());
-
- responseObserver.awaitTerminalState();
-
- assertThat(responseObserver.error, Matchers.not(Matchers.nullValue()));
- }
-
- @Test
- public void commitManifestWithAllArtifactsSucceeds() {
- ArtifactApi.ArtifactMetadata firstArtifact =
- stageBytes("first-artifact", "foo, bar, baz, quux".getBytes(UTF_8));
- ArtifactApi.ArtifactMetadata secondArtifact =
- stageBytes("second-artifact", "spam, ham, eggs".getBytes(UTF_8));
-
- ArtifactApi.Manifest manifest =
- ArtifactApi.Manifest.newBuilder()
- .addArtifact(firstArtifact)
- .addArtifact(secondArtifact)
- .build();
-
- RecordingStreamObserver<ArtifactApi.CommitManifestResponse> commitResponseObserver =
- new RecordingStreamObserver<>();
- stub.commitManifest(
- ArtifactApi.CommitManifestRequest.newBuilder().setManifest(manifest).build(),
- commitResponseObserver);
-
- commitResponseObserver.awaitTerminalState();
-
- assertThat(commitResponseObserver.completed, is(true));
- assertThat(commitResponseObserver.responses, Matchers.hasSize(1));
- ArtifactApi.CommitManifestResponse commitResponse = commitResponseObserver.responses.get(0);
- assertThat(commitResponse.getRetrievalToken(), Matchers.not(Matchers.nullValue()));
- }
-
- @Test
- public void commitManifestWithMissingArtifactFails() {
- ArtifactApi.ArtifactMetadata firstArtifact =
- stageBytes("first-artifact", "foo, bar, baz, quux".getBytes(UTF_8));
- ArtifactApi.ArtifactMetadata absentArtifact =
- ArtifactApi.ArtifactMetadata.newBuilder().setName("absent").build();
-
- ArtifactApi.Manifest manifest =
- ArtifactApi.Manifest.newBuilder()
- .addArtifact(firstArtifact)
- .addArtifact(absentArtifact)
- .build();
-
- RecordingStreamObserver<ArtifactApi.CommitManifestResponse> commitResponseObserver =
- new RecordingStreamObserver<>();
- stub.commitManifest(
- ArtifactApi.CommitManifestRequest.newBuilder().setManifest(manifest).build(),
- commitResponseObserver);
-
- commitResponseObserver.awaitTerminalState();
-
- assertThat(commitResponseObserver.error, Matchers.not(Matchers.nullValue()));
- }
-
- private ArtifactApi.ArtifactMetadata stageBytes(String name, byte[] bytes) {
- StreamObserver<ArtifactApi.PutArtifactRequest> requests =
- stub.putArtifact(new RecordingStreamObserver<>());
- requests.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setMetadata(
- ArtifactApi.PutArtifactMetadata.newBuilder()
- .setMetadata(ArtifactApi.ArtifactMetadata.newBuilder().setName(name).build())
- .setStagingSessionToken("token")
- .build())
- .build());
- requests.onNext(
- ArtifactApi.PutArtifactRequest.newBuilder()
- .setData(
- ArtifactApi.ArtifactChunk.newBuilder().setData(ByteString.copyFrom(bytes)).build())
- .build());
- requests.onCompleted();
- return ArtifactApi.ArtifactMetadata.newBuilder().setName(name).build();
- }
-
- private static class RecordingStreamObserver<T> implements StreamObserver<T> {
- private List<T> responses = new ArrayList<>();
- @Nullable private Throwable error = null;
- private boolean completed = false;
-
- @Override
- public void onNext(T value) {
- failIfTerminal();
- responses.add(value);
- }
-
- @Override
- public void onError(Throwable t) {
- failIfTerminal();
- error = t;
- }
-
- @Override
- public void onCompleted() {
- failIfTerminal();
- completed = true;
- }
-
- private boolean isTerminal() {
- return error != null || completed;
- }
-
- private void failIfTerminal() {
- if (isTerminal()) {
- Assert.fail(
- String.format(
- "Should have terminated after entering a terminal state: completed %s, error %s",
- completed, error));
- }
- }
-
- void awaitTerminalState() {
- while (!isTerminal()) {
- Uninterruptibles.sleepUninterruptibly(1, TimeUnit.MILLISECONDS);
- }
- }
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalServiceTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalServiceTest.java
deleted file mode 100644
index 068620f..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalServiceTest.java
+++ /dev/null
@@ -1,150 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.artifact;
-
-import java.util.Optional;
-import java.util.concurrent.SynchronousQueue;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetArtifactRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestResponse;
-import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
-import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link UnsupportedArtifactRetrievalService}. */
-@RunWith(JUnit4.class)
-public class UnsupportedArtifactRetrievalServiceTest {
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- private GrpcFnServer<ArtifactRetrievalService> server;
- private ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub stub;
-
- @Before
- public void setUp() throws Exception {
- server =
- GrpcFnServer.allocatePortAndCreateFor(
- UnsupportedArtifactRetrievalService.create(), InProcessServerFactory.create());
- stub =
- ArtifactRetrievalServiceGrpc.newStub(
- InProcessManagedChannelFactory.create()
- .forDescriptor(server.getApiServiceDescriptor()));
- }
-
- @Test
- public void getArtifactThrows() throws Exception {
- SynchronousQueue<Optional<Throwable>> thrown = new SynchronousQueue<>();
- stub.getArtifact(
- GetArtifactRequest.newBuilder().setName("foo").build(),
- new StreamObserver<ArtifactChunk>() {
- @Override
- public void onNext(ArtifactChunk value) {
- try {
- thrown.put(Optional.empty());
- } catch (InterruptedException e) {
- throw new AssertionError(e);
- }
- }
-
- @Override
- public void onError(Throwable t) {
- try {
- thrown.put(Optional.of(t));
- } catch (InterruptedException e) {
- throw new AssertionError(e);
- }
- }
-
- @Override
- public void onCompleted() {
- try {
- thrown.put(Optional.empty());
- } catch (InterruptedException e) {
- throw new AssertionError(e);
- }
- }
- });
- Throwable wasThrown =
- thrown
- .take()
- .orElseThrow(
- () ->
- new AssertionError(
- String.format(
- "The %s should respond to all calls with an error",
- UnsupportedArtifactRetrievalServiceTest.class.getSimpleName())));
- }
-
- @Test
- public void getManifestThrows() throws Exception {
- SynchronousQueue<Optional<Throwable>> thrown = new SynchronousQueue<>();
- stub.getManifest(
- GetManifestRequest.newBuilder().build(),
- new StreamObserver<GetManifestResponse>() {
- @Override
- public void onNext(GetManifestResponse value) {
- try {
- thrown.put(Optional.empty());
- } catch (InterruptedException e) {
- throw new AssertionError(e);
- }
- }
-
- @Override
- public void onError(Throwable t) {
- try {
- thrown.put(Optional.of(t));
- } catch (InterruptedException e) {
- throw new AssertionError(e);
- }
- }
-
- @Override
- public void onCompleted() {
- try {
- thrown.put(Optional.empty());
- } catch (InterruptedException e) {
- throw new AssertionError(e);
- }
- }
- });
- Throwable wasThrown =
- thrown
- .take()
- .orElseThrow(
- () ->
- new AssertionError(
- String.format(
- "The %s should respond to all calls with an error",
- UnsupportedArtifactRetrievalServiceTest.class.getSimpleName())));
- }
-
- @Test
- public void closeCompletes() throws Exception {
- server.getService().close();
- }
-}
diff --git a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServiceTest.java b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServiceTest.java
deleted file mode 100644
index 7671d45..0000000
--- a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServiceTest.java
+++ /dev/null
@@ -1,150 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct.portable.job;
-
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.hamcrest.Matchers.hasItems;
-import static org.junit.Assert.assertThat;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.nio.ByteBuffer;
-import java.nio.channels.FileChannel;
-import java.nio.file.Files;
-import java.nio.file.Paths;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
-import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
-import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
-import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager;
-import org.apache.beam.runners.core.construction.ArtifactServiceStager.StagedFile;
-import org.apache.beam.runners.fnexecution.GrpcFnServer;
-import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.hamcrest.Description;
-import org.hamcrest.Matcher;
-import org.hamcrest.TypeSafeMatcher;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link ReferenceRunnerJobService}. */
-@RunWith(JUnit4.class)
-public class ReferenceRunnerJobServiceTest {
- @Rule public TemporaryFolder runnerTemp = new TemporaryFolder();
- @Rule public TemporaryFolder clientTemp = new TemporaryFolder();
-
- private InProcessServerFactory serverFactory = InProcessServerFactory.create();
- private ReferenceRunnerJobService service;
- private GrpcFnServer<ReferenceRunnerJobService> server;
- private JobServiceBlockingStub stub;
-
- @Before
- public void setup() throws Exception {
- ReferenceRunnerJobService.Configuration configuration =
- new ReferenceRunnerJobService.Configuration();
- configuration.artifactStagingPath =
- Paths.get(runnerTemp.getRoot().toString(), "beam-artifact-staging").toString();
- service = ReferenceRunnerJobService.create(serverFactory, configuration);
- server = GrpcFnServer.allocatePortAndCreateFor(service, serverFactory);
- stub =
- JobServiceGrpc.newBlockingStub(
- InProcessChannelBuilder.forName(server.getApiServiceDescriptor().getUrl()).build());
- }
-
- @After
- public void teardown() throws Exception {
- server.close();
- }
-
- @Test
- public void testPrepareJob() throws Exception {
- PrepareJobResponse response =
- stub.prepare(
- PrepareJobRequest.newBuilder()
- .setPipelineOptions(Struct.getDefaultInstance())
- .setPipeline(Pipeline.getDefaultInstance())
- .setJobName("myJobName")
- .build());
-
- ApiServiceDescriptor stagingEndpoint = response.getArtifactStagingEndpoint();
- ArtifactServiceStager stager =
- ArtifactServiceStager.overChannel(
- InProcessChannelBuilder.forName(stagingEndpoint.getUrl()).build());
- String stagingSessionToken = response.getStagingSessionToken();
- File foo = writeTempFile("foo", "foo, bar, baz".getBytes(UTF_8));
- File bar = writeTempFile("spam", "spam, ham, eggs".getBytes(UTF_8));
- stager.stage(
- stagingSessionToken,
- ImmutableList.of(StagedFile.of(foo, foo.getName()), StagedFile.of(bar, bar.getName())));
- List<byte[]> tempDirFiles = readFlattenedFiles(runnerTemp.getRoot());
- assertThat(
- tempDirFiles,
- hasItems(
- arrayEquals(Files.readAllBytes(foo.toPath())),
- arrayEquals(Files.readAllBytes(bar.toPath()))));
- // TODO: 'run' the job with some sort of noop backend, to verify state is cleaned up.
- }
-
- private Matcher<byte[]> arrayEquals(final byte[] expected) {
- return new TypeSafeMatcher<byte[]>() {
- @Override
- protected boolean matchesSafely(byte[] actual) {
- return Arrays.equals(actual, expected);
- }
-
- @Override
- public void describeTo(Description description) {
- description.appendText("an array equal to ").appendValue(Arrays.toString(expected));
- }
- };
- }
-
- private List<byte[]> readFlattenedFiles(File root) throws Exception {
- if (root.isDirectory()) {
- List<byte[]> children = new ArrayList<>();
- for (File child : root.listFiles()) {
- children.addAll(readFlattenedFiles(child));
- }
- return children;
- } else {
- return Collections.singletonList(Files.readAllBytes(root.toPath()));
- }
- }
-
- private File writeTempFile(String fileName, byte[] contents) throws Exception {
- File file = clientTemp.newFile(fileName);
- try (FileOutputStream stream = new FileOutputStream(file);
- FileChannel channel = stream.getChannel()) {
- channel.write(ByteBuffer.wrap(contents));
- }
- return file;
- }
-}
diff --git a/runners/extensions-java/metrics/build.gradle b/runners/extensions-java/metrics/build.gradle
index 455e5be..022b15c 100644
--- a/runners/extensions-java/metrics/build.gradle
+++ b/runners/extensions-java/metrics/build.gradle
@@ -17,16 +17,16 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.runners.extensions.metrics')
description = "Apache Beam :: Runners :: Extensions Java :: Metrics"
ext.summary = "Beam Runners Extensions Metrics provides implementations of runners core metrics APIs."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.jackson_databind
- shadow library.java.jackson_datatype_joda
- shadowTest library.java.joda_time
- shadowTest library.java.junit
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.jackson_databind
+ compile library.java.jackson_datatype_joda
+ testCompile library.java.joda_time
+ testCompile library.java.junit
}
diff --git a/runners/flink/1.5/build.gradle b/runners/flink/1.5/build.gradle
deleted file mode 100644
index b063395..0000000
--- a/runners/flink/1.5/build.gradle
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '..'
-
-/* All properties required for loading the Flink build script. */
-project.ext {
- // Set the version of all Flink-related dependencies here.
- flink_version = '1.5.6'
- // Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink_2.11'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.5/job-server-container/build.gradle b/runners/flink/1.5/job-server-container/build.gradle
deleted file mode 100644
index afdb68a..0000000
--- a/runners/flink/1.5/job-server-container/build.gradle
+++ /dev/null
@@ -1,26 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server-container'
-
-project.ext {
- resource_path = basePath
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server_container.gradle"
diff --git a/runners/flink/1.5/job-server/build.gradle b/runners/flink/1.5/job-server/build.gradle
deleted file mode 100644
index fbba7a3..0000000
--- a/runners/flink/1.5/job-server/build.gradle
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server'
-
-project.ext {
- // Look for the source code in the parent module
- main_source_dirs = ["$basePath/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink_2.11-job-server'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java b/runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
deleted file mode 100644
index fbe88c0..0000000
--- a/runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
+++ /dev/null
@@ -1,208 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.types;
-
-import java.io.EOFException;
-import java.io.IOException;
-import java.util.Objects;
-import org.apache.beam.runners.flink.translation.wrappers.DataInputViewWrapper;
-import org.apache.beam.runners.flink.translation.wrappers.DataOutputViewWrapper;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.flink.api.common.typeutils.CompatibilityResult;
-import org.apache.flink.api.common.typeutils.TypeSerializer;
-import org.apache.flink.api.common.typeutils.TypeSerializerConfigSnapshot;
-import org.apache.flink.core.memory.DataInputView;
-import org.apache.flink.core.memory.DataOutputView;
-
-/**
- * Flink {@link org.apache.flink.api.common.typeutils.TypeSerializer} for Beam {@link
- * org.apache.beam.sdk.coders.Coder Coders}.
- */
-public class CoderTypeSerializer<T> extends TypeSerializer<T> {
-
- private Coder<T> coder;
-
- public CoderTypeSerializer(Coder<T> coder) {
- Preconditions.checkNotNull(coder);
- this.coder = coder;
- }
-
- @Override
- public boolean isImmutableType() {
- return false;
- }
-
- @Override
- public CoderTypeSerializer<T> duplicate() {
- return new CoderTypeSerializer<>(coder);
- }
-
- @Override
- public T createInstance() {
- return null;
- }
-
- @Override
- public T copy(T t) {
- try {
- return CoderUtils.clone(coder, t);
- } catch (CoderException e) {
- throw new RuntimeException("Could not clone.", e);
- }
- }
-
- @Override
- public T copy(T t, T reuse) {
- return copy(t);
- }
-
- @Override
- public int getLength() {
- return -1;
- }
-
- @Override
- public void serialize(T t, DataOutputView dataOutputView) throws IOException {
- DataOutputViewWrapper outputWrapper = new DataOutputViewWrapper(dataOutputView);
- coder.encode(t, outputWrapper);
- }
-
- @Override
- public T deserialize(DataInputView dataInputView) throws IOException {
- try {
- DataInputViewWrapper inputWrapper = new DataInputViewWrapper(dataInputView);
- return coder.decode(inputWrapper);
- } catch (CoderException e) {
- Throwable cause = e.getCause();
- if (cause instanceof EOFException) {
- throw (EOFException) cause;
- } else {
- throw e;
- }
- }
- }
-
- @Override
- public T deserialize(T t, DataInputView dataInputView) throws IOException {
- return deserialize(dataInputView);
- }
-
- @Override
- public void copy(DataInputView dataInputView, DataOutputView dataOutputView) throws IOException {
- serialize(deserialize(dataInputView), dataOutputView);
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (o == null || getClass() != o.getClass()) {
- return false;
- }
-
- CoderTypeSerializer that = (CoderTypeSerializer) o;
- return coder.equals(that.coder);
- }
-
- @Override
- public boolean canEqual(Object obj) {
- return obj instanceof CoderTypeSerializer;
- }
-
- @Override
- public int hashCode() {
- return coder.hashCode();
- }
-
- @Override
- public TypeSerializerConfigSnapshot snapshotConfiguration() {
- return new CoderTypeSerializerConfigSnapshot<>(coder);
- }
-
- @Override
- public CompatibilityResult<T> ensureCompatibility(TypeSerializerConfigSnapshot configSnapshot) {
- if (snapshotConfiguration().equals(configSnapshot)) {
- return CompatibilityResult.compatible();
- }
- return CompatibilityResult.requiresMigration();
- }
-
- /**
- * TypeSerializerConfigSnapshot of CoderTypeSerializer. This uses the class name of the {@link
- * Coder} to determine compatibility. This is a bit crude but better than using Java Serialization
- * to (de)serialize the {@link Coder}.
- */
- public static class CoderTypeSerializerConfigSnapshot<T> extends TypeSerializerConfigSnapshot {
-
- private static final int VERSION = 1;
- private String coderName;
-
- public CoderTypeSerializerConfigSnapshot() {
- // empty constructor for satisfying IOReadableWritable which is used for deserialization
- }
-
- public CoderTypeSerializerConfigSnapshot(Coder<T> coder) {
- this.coderName = coder.getClass().getName();
- }
-
- @Override
- public int getVersion() {
- return VERSION;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (o == null || getClass() != o.getClass()) {
- return false;
- }
-
- CoderTypeSerializerConfigSnapshot<?> that = (CoderTypeSerializerConfigSnapshot<?>) o;
-
- return coderName != null ? coderName.equals(that.coderName) : that.coderName == null;
- }
-
- @Override
- public void write(DataOutputView out) throws IOException {
- super.write(out);
- out.writeUTF(coderName);
- }
-
- @Override
- public void read(DataInputView in) throws IOException {
- super.read(in);
- this.coderName = in.readUTF();
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(coderName);
- }
- }
-
- @Override
- public String toString() {
- return "CoderTypeSerializer{" + "coder=" + coder + '}';
- }
-}
diff --git a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java b/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
deleted file mode 100644
index edc44f6..0000000
--- a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
+++ /dev/null
@@ -1,144 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.streaming;
-
-import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.core.Is.is;
-
-import java.nio.ByteBuffer;
-import java.util.UUID;
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.StateInternalsTest;
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.StateTag;
-import org.apache.beam.runners.core.StateTags;
-import org.apache.beam.runners.flink.translation.wrappers.streaming.state.FlinkStateInternals;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.state.WatermarkHoldState;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.flink.api.common.ExecutionConfig;
-import org.apache.flink.api.common.JobID;
-import org.apache.flink.api.java.typeutils.GenericTypeInfo;
-import org.apache.flink.runtime.jobgraph.JobVertexID;
-import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
-import org.apache.flink.runtime.query.KvStateRegistry;
-import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
-import org.apache.flink.runtime.state.KeyGroupRange;
-import org.apache.flink.runtime.state.KeyedStateBackend;
-import org.apache.flink.runtime.state.memory.MemoryStateBackend;
-import org.joda.time.Instant;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link FlinkStateInternals}. This is based on {@link StateInternalsTest}. */
-@RunWith(JUnit4.class)
-public class FlinkStateInternalsTest extends StateInternalsTest {
-
- @Override
- protected StateInternals createStateInternals() {
- try {
- KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
- return new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
-
- @Test
- public void testWatermarkHoldsPersistence() throws Exception {
- KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
- FlinkStateInternals stateInternals =
- new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
-
- StateTag<WatermarkHoldState> stateTag =
- StateTags.watermarkStateInternal("hold", TimestampCombiner.EARLIEST);
- WatermarkHoldState globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
- WatermarkHoldState fixedWindow =
- stateInternals.state(
- StateNamespaces.window(
- IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
- stateTag);
-
- Instant noHold = new Instant(Long.MAX_VALUE);
- assertThat(stateInternals.watermarkHold(), is(noHold));
-
- Instant high = new Instant(10);
- globalWindow.add(high);
- assertThat(stateInternals.watermarkHold(), is(high));
-
- Instant middle = new Instant(5);
- fixedWindow.add(middle);
- assertThat(stateInternals.watermarkHold(), is(middle));
-
- Instant low = new Instant(1);
- globalWindow.add(low);
- assertThat(stateInternals.watermarkHold(), is(low));
-
- // Try to overwrite with later hold (should not succeed)
- globalWindow.add(high);
- assertThat(stateInternals.watermarkHold(), is(low));
- fixedWindow.add(high);
- assertThat(stateInternals.watermarkHold(), is(low));
-
- changeKey(keyedStateBackend);
- // Discard watermark view and recover it
- stateInternals = new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
- globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
- fixedWindow =
- stateInternals.state(
- StateNamespaces.window(
- IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
- stateTag);
-
- assertThat(stateInternals.watermarkHold(), is(low));
-
- fixedWindow.clear();
- assertThat(stateInternals.watermarkHold(), is(low));
-
- globalWindow.clear();
- assertThat(stateInternals.watermarkHold(), is(noHold));
- }
-
- private KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
- MemoryStateBackend backend = new MemoryStateBackend();
-
- AbstractKeyedStateBackend<ByteBuffer> keyedStateBackend =
- backend.createKeyedStateBackend(
- new DummyEnvironment("test", 1, 0),
- new JobID(),
- "test_op",
- new GenericTypeInfo<>(ByteBuffer.class).createSerializer(new ExecutionConfig()),
- 2,
- new KeyGroupRange(0, 1),
- new KvStateRegistry().createTaskRegistry(new JobID(), new JobVertexID()));
-
- changeKey(keyedStateBackend);
-
- return keyedStateBackend;
- }
-
- private void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend) throws CoderException {
- keyedStateBackend.setCurrentKey(
- ByteBuffer.wrap(
- CoderUtils.encodeToByteArray(StringUtf8Coder.of(), UUID.randomUUID().toString())));
- }
-}
diff --git a/runners/flink/1.6/build.gradle b/runners/flink/1.6/build.gradle
deleted file mode 100644
index e8541cc..0000000
--- a/runners/flink/1.6/build.gradle
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '..'
-
-/* All properties required for loading the Flink build script */
-project.ext {
- // Set the version of all Flink-related dependencies here.
- flink_version = '1.6.4'
- // Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "../1.5/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "../1.5/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink-1.6'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.6/job-server-container/build.gradle b/runners/flink/1.6/job-server-container/build.gradle
deleted file mode 100644
index afdb68a..0000000
--- a/runners/flink/1.6/job-server-container/build.gradle
+++ /dev/null
@@ -1,26 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server-container'
-
-project.ext {
- resource_path = basePath
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server_container.gradle"
diff --git a/runners/flink/1.6/job-server/build.gradle b/runners/flink/1.6/job-server/build.gradle
deleted file mode 100644
index 39f1810..0000000
--- a/runners/flink/1.6/job-server/build.gradle
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-def basePath = '../../job-server'
-
-project.ext {
- // Look for the source code in the parent module
- main_source_dirs = ["$basePath/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java"]
- main_resources_dirs = ["$basePath/src/main/resources"]
- test_resources_dirs = ["$basePath/src/test/resources"]
- archives_base_name = 'beam-runners-flink-1.6-job-server'
-}
-
-// Load the main build script which contains all build logic.
-apply from: "$basePath/flink_job_server.gradle"
diff --git a/runners/flink/1.7/build.gradle b/runners/flink/1.7/build.gradle
index 9029153..4013247 100644
--- a/runners/flink/1.7/build.gradle
+++ b/runners/flink/1.7/build.gradle
@@ -23,8 +23,8 @@
// Set the version of all Flink-related dependencies here.
flink_version = '1.7.2'
// Main source directory and Flink version specific code.
- main_source_dirs = ["$basePath/src/main/java", "../1.5/src/main/java"]
- test_source_dirs = ["$basePath/src/test/java", "../1.5/src/test/java"]
+ main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
+ test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
main_resources_dirs = ["$basePath/src/main/resources"]
test_resources_dirs = ["$basePath/src/test/resources"]
archives_base_name = 'beam-runners-flink-1.7'
diff --git a/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
new file mode 100644
index 0000000..e29f97e
--- /dev/null
+++ b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.types;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.Objects;
+import org.apache.beam.runners.flink.translation.wrappers.DataInputViewWrapper;
+import org.apache.beam.runners.flink.translation.wrappers.DataOutputViewWrapper;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.flink.api.common.typeutils.CompatibilityResult;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerConfigSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+
+/**
+ * Flink {@link org.apache.flink.api.common.typeutils.TypeSerializer} for Beam {@link
+ * org.apache.beam.sdk.coders.Coder Coders}.
+ */
+public class CoderTypeSerializer<T> extends TypeSerializer<T> {
+
+ private final Coder<T> coder;
+
+ public CoderTypeSerializer(Coder<T> coder) {
+ Preconditions.checkNotNull(coder);
+ this.coder = coder;
+ }
+
+ @Override
+ public boolean isImmutableType() {
+ return false;
+ }
+
+ @Override
+ public CoderTypeSerializer<T> duplicate() {
+ return new CoderTypeSerializer<>(coder);
+ }
+
+ @Override
+ public T createInstance() {
+ return null;
+ }
+
+ @Override
+ public T copy(T t) {
+ try {
+ return CoderUtils.clone(coder, t);
+ } catch (CoderException e) {
+ throw new RuntimeException("Could not clone.", e);
+ }
+ }
+
+ @Override
+ public T copy(T t, T reuse) {
+ return copy(t);
+ }
+
+ @Override
+ public int getLength() {
+ return -1;
+ }
+
+ @Override
+ public void serialize(T t, DataOutputView dataOutputView) throws IOException {
+ DataOutputViewWrapper outputWrapper = new DataOutputViewWrapper(dataOutputView);
+ coder.encode(t, outputWrapper);
+ }
+
+ @Override
+ public T deserialize(DataInputView dataInputView) throws IOException {
+ try {
+ DataInputViewWrapper inputWrapper = new DataInputViewWrapper(dataInputView);
+ return coder.decode(inputWrapper);
+ } catch (CoderException e) {
+ Throwable cause = e.getCause();
+ if (cause instanceof EOFException) {
+ throw (EOFException) cause;
+ } else {
+ throw e;
+ }
+ }
+ }
+
+ @Override
+ public T deserialize(T t, DataInputView dataInputView) throws IOException {
+ return deserialize(dataInputView);
+ }
+
+ @Override
+ public void copy(DataInputView dataInputView, DataOutputView dataOutputView) throws IOException {
+ serialize(deserialize(dataInputView), dataOutputView);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ CoderTypeSerializer that = (CoderTypeSerializer) o;
+ return coder.equals(that.coder);
+ }
+
+ @Override
+ public boolean canEqual(Object obj) {
+ return obj instanceof CoderTypeSerializer;
+ }
+
+ @Override
+ public int hashCode() {
+ return coder.hashCode();
+ }
+
+ @Override
+ public TypeSerializerConfigSnapshot snapshotConfiguration() {
+ return new CoderTypeSerializerConfigSnapshot<>(coder);
+ }
+
+ @Override
+ public CompatibilityResult<T> ensureCompatibility(TypeSerializerConfigSnapshot configSnapshot) {
+ if (snapshotConfiguration().equals(configSnapshot)) {
+ return CompatibilityResult.compatible();
+ }
+ return CompatibilityResult.requiresMigration();
+ }
+
+ /**
+ * TypeSerializerConfigSnapshot of CoderTypeSerializer. This uses the class name of the {@link
+ * Coder} to determine compatibility. This is a bit crude but better than using Java Serialization
+ * to (de)serialize the {@link Coder}.
+ */
+ public static class CoderTypeSerializerConfigSnapshot<T> extends TypeSerializerConfigSnapshot {
+
+ private static final int VERSION = 1;
+ private String coderName;
+
+ public CoderTypeSerializerConfigSnapshot() {
+ // empty constructor for satisfying IOReadableWritable which is used for deserialization
+ }
+
+ public CoderTypeSerializerConfigSnapshot(Coder<T> coder) {
+ this.coderName = coder.getClass().getName();
+ }
+
+ @Override
+ public int getVersion() {
+ return VERSION;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ CoderTypeSerializerConfigSnapshot<?> that = (CoderTypeSerializerConfigSnapshot<?>) o;
+
+ return coderName != null ? coderName.equals(that.coderName) : that.coderName == null;
+ }
+
+ @Override
+ public void write(DataOutputView out) throws IOException {
+ super.write(out);
+ out.writeUTF(coderName);
+ }
+
+ @Override
+ public void read(DataInputView in) throws IOException {
+ super.read(in);
+ this.coderName = in.readUTF();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(coderName);
+ }
+ }
+
+ @Override
+ public String toString() {
+ return "CoderTypeSerializer{" + "coder=" + coder + '}';
+ }
+}
diff --git a/runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java b/runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
similarity index 100%
rename from runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
rename to runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
diff --git a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
similarity index 100%
rename from runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
rename to runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
diff --git a/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
new file mode 100644
index 0000000..a80a483
--- /dev/null
+++ b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.streaming;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+import java.nio.ByteBuffer;
+import java.util.UUID;
+import org.apache.beam.runners.core.StateInternals;
+import org.apache.beam.runners.core.StateInternalsTest;
+import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.StateTag;
+import org.apache.beam.runners.core.StateTags;
+import org.apache.beam.runners.flink.translation.wrappers.streaming.state.FlinkStateInternals;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.state.WatermarkHoldState;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.java.typeutils.GenericTypeInfo;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.operators.testutils.DummyEnvironment;
+import org.apache.flink.runtime.query.KvStateRegistry;
+import org.apache.flink.runtime.state.AbstractKeyedStateBackend;
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateBackend;
+import org.apache.flink.runtime.state.memory.MemoryStateBackend;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link FlinkStateInternals}. This is based on {@link StateInternalsTest}. */
+@RunWith(JUnit4.class)
+public class FlinkStateInternalsTest extends StateInternalsTest {
+
+ @Override
+ protected StateInternals createStateInternals() {
+ try {
+ KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
+ return new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void testWatermarkHoldsPersistence() throws Exception {
+ KeyedStateBackend<ByteBuffer> keyedStateBackend = createStateBackend();
+ FlinkStateInternals stateInternals =
+ new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
+
+ StateTag<WatermarkHoldState> stateTag =
+ StateTags.watermarkStateInternal("hold", TimestampCombiner.EARLIEST);
+ WatermarkHoldState globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
+ WatermarkHoldState fixedWindow =
+ stateInternals.state(
+ StateNamespaces.window(
+ IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
+ stateTag);
+
+ Instant noHold = new Instant(Long.MAX_VALUE);
+ assertThat(stateInternals.watermarkHold(), is(noHold));
+
+ Instant high = new Instant(10);
+ globalWindow.add(high);
+ assertThat(stateInternals.watermarkHold(), is(high));
+
+ Instant middle = new Instant(5);
+ fixedWindow.add(middle);
+ assertThat(stateInternals.watermarkHold(), is(middle));
+
+ Instant low = new Instant(1);
+ globalWindow.add(low);
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ // Try to overwrite with later hold (should not succeed)
+ globalWindow.add(high);
+ assertThat(stateInternals.watermarkHold(), is(low));
+ fixedWindow.add(high);
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ changeKey(keyedStateBackend);
+ // Discard watermark view and recover it
+ stateInternals = new FlinkStateInternals<>(keyedStateBackend, StringUtf8Coder.of());
+ globalWindow = stateInternals.state(StateNamespaces.global(), stateTag);
+ fixedWindow =
+ stateInternals.state(
+ StateNamespaces.window(
+ IntervalWindow.getCoder(), new IntervalWindow(new Instant(0), new Instant(10))),
+ stateTag);
+
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ fixedWindow.clear();
+ assertThat(stateInternals.watermarkHold(), is(low));
+
+ globalWindow.clear();
+ assertThat(stateInternals.watermarkHold(), is(noHold));
+ }
+
+ public static KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
+ MemoryStateBackend backend = new MemoryStateBackend();
+
+ AbstractKeyedStateBackend<ByteBuffer> keyedStateBackend =
+ backend.createKeyedStateBackend(
+ new DummyEnvironment("test", 1, 0),
+ new JobID(),
+ "test_op",
+ new GenericTypeInfo<>(ByteBuffer.class).createSerializer(new ExecutionConfig()),
+ 2,
+ new KeyGroupRange(0, 1),
+ new KvStateRegistry().createTaskRegistry(new JobID(), new JobVertexID()));
+
+ changeKey(keyedStateBackend);
+
+ return keyedStateBackend;
+ }
+
+ private static void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend)
+ throws CoderException {
+ keyedStateBackend.setCurrentKey(
+ ByteBuffer.wrap(
+ CoderUtils.encodeToByteArray(StringUtf8Coder.of(), UUID.randomUUID().toString())));
+ }
+}
diff --git a/runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java b/runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
similarity index 100%
rename from runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
rename to runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
diff --git a/runners/flink/1.8/build.gradle b/runners/flink/1.8/build.gradle
index 41ca8fc..d956493 100644
--- a/runners/flink/1.8/build.gradle
+++ b/runners/flink/1.8/build.gradle
@@ -21,7 +21,7 @@
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.8.0'
+ flink_version = '1.8.2'
// Main source directory and Flink version specific code.
main_source_dirs = ["$basePath/src/main/java", "./src/main/java"]
test_source_dirs = ["$basePath/src/test/java", "./src/test/java"]
diff --git a/runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java b/runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
index 2b0650f..2ff1cda 100644
--- a/runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
+++ b/runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.flink.api.common.typeutils.TypeSerializer;
import org.apache.flink.api.common.typeutils.TypeSerializerConfigSnapshot;
import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
@@ -39,7 +39,7 @@
*/
public class CoderTypeSerializer<T> extends TypeSerializer<T> {
- private Coder<T> coder;
+ private final Coder<T> coder;
public CoderTypeSerializer(Coder<T> coder) {
Preconditions.checkNotNull(coder);
diff --git a/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java b/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
index 82d2c91..ff4b220 100644
--- a/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
+++ b/runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
@@ -121,7 +121,7 @@
assertThat(stateInternals.watermarkHold(), is(noHold));
}
- private KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
+ public static KeyedStateBackend<ByteBuffer> createStateBackend() throws Exception {
MemoryStateBackend backend = new MemoryStateBackend();
AbstractKeyedStateBackend<ByteBuffer> keyedStateBackend =
@@ -143,7 +143,8 @@
return keyedStateBackend;
}
- private void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend) throws CoderException {
+ public static void changeKey(KeyedStateBackend<ByteBuffer> keyedStateBackend)
+ throws CoderException {
keyedStateBackend.setCurrentKey(
ByteBuffer.wrap(
CoderUtils.encodeToByteArray(StringUtf8Coder.of(), UUID.randomUUID().toString())));
diff --git a/runners/flink/flink_runner.gradle b/runners/flink/flink_runner.gradle
index 39fa777..893f153 100644
--- a/runners/flink/flink_runner.gradle
+++ b/runners/flink/flink_runner.gradle
@@ -26,8 +26,10 @@
import groovy.json.JsonOutput
apply plugin: 'org.apache.beam.module'
-archivesBaseName = project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName
-applyJavaNature()
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.flink',
+ archivesBaseName: (project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName)
+)
description = "Apache Beam :: Runners :: Flink $flink_version"
@@ -62,6 +64,17 @@
}
}
+/*
+ * By default, Spotless operates on the project files only.
+ * BeamModulePlugin sets a custom configuration which we override here in order
+ * for the Flink Runner's multiple source directories to be all scanned.
+ */
+spotless {
+ java {
+ target project.sourceSets.main.allJava + project.sourceSets.test.allJava
+ }
+}
+
test {
systemProperty "log4j.configuration", "log4j-test.properties"
//systemProperty "org.slf4j.simpleLogger.defaultLogLevel", "debug"
@@ -71,14 +84,7 @@
}
// TODO Running tests of all Flink versions in parallel can be too harsh on Jenkins memory
// Run them serially for now, to avoid "Exit code 137", i.e. Jenkins host killing the Gradle test process
- if (project.path == ":runners:flink:1.6") {
- mustRunAfter(":runners:flink:1.5:test")
- } else if (project.path == ":runners:flink:1.7") {
- mustRunAfter(":runners:flink:1.5:test")
- mustRunAfter(":runners:flink:1.6:test")
- } else if (project.path == ":runners:flink:1.8") {
- mustRunAfter(":runners:flink:1.5:test")
- mustRunAfter(":runners:flink:1.6:test")
+ if (project.path == ":runners:flink:1.8") {
mustRunAfter(":runners:flink:1.7:test")
}
}
@@ -88,43 +94,45 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:core-java", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":runners:java-fn-execution", configuration: "shadow")
- shadow project(path: ":sdks:java:build-tools", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.jackson_annotations
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.commons_compress
- shadow library.java.args4j
- shadow "org.apache.flink:flink-clients_2.11:$flink_version"
- shadow "org.apache.flink:flink-core:$flink_version"
- shadow "org.apache.flink:flink-metrics-core:$flink_version"
- shadow "org.apache.flink:flink-java:$flink_version"
- shadow "org.apache.flink:flink-runtime_2.11:$flink_version"
- shadow "org.apache.flink:flink-streaming-java_2.11:$flink_version"
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+ compileOnly project(":sdks:java:build-tools")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-java")
+ compile project(":runners:core-construction-java")
+ compile project(":runners:java-fn-execution")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile library.java.vendored_grpc_1_21_0
+ compile library.java.jackson_annotations
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile library.java.commons_compress
+ compile library.java.args4j
+ compile "org.apache.flink:flink-clients_2.11:$flink_version"
+ compile "org.apache.flink:flink-core:$flink_version"
+ compile "org.apache.flink:flink-metrics-core:$flink_version"
+ compile "org.apache.flink:flink-java:$flink_version"
+ compile "org.apache.flink:flink-runtime_2.11:$flink_version"
+ compile "org.apache.flink:flink-streaming-java_2.11:$flink_version"
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
// FlinkStateInternalsTest extends abstract StateInternalsTest
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest library.java.commons_lang3
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.google_api_services_bigquery
- shadowTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadowTest library.java.jackson_dataformat_yaml
- shadowTest "org.apache.flink:flink-core:$flink_version:tests"
- shadowTest "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
- shadowTest "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
- shadowTest "org.apache.flink:flink-test-utils_2.11:$flink_version"
- shadowTest project(":sdks:java:harness")
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ testCompile library.java.commons_lang3
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.powermock
+ testCompile library.java.google_api_services_bigquery
+ testCompile project(":sdks:java:io:google-cloud-platform")
+ testCompile library.java.jackson_dataformat_yaml
+ testCompile "org.apache.flink:flink-core:$flink_version:tests"
+ testCompile "org.apache.flink:flink-runtime_2.11:$flink_version:tests"
+ testCompile "org.apache.flink:flink-streaming-java_2.11:$flink_version:tests"
+ testCompile "org.apache.flink:flink-test-utils_2.11:$flink_version"
+ testCompile project(":sdks:java:harness")
testRuntimeOnly library.java.slf4j_simple
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesRunner project(project.path)
}
class ValidatesRunnerConfig {
@@ -160,6 +168,9 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
}
+ // TODO[BEAM-8205]: figure out if we can make the test work on Flink runner, or maybe create a
+ // new category tag and change the following line to: excludeCategories '<category tag>'.
+ exclude '**/AvroSchemaTest.class'
}
}
}
@@ -174,5 +185,5 @@
dependsOn validatesRunnerStreaming
}
-// Generates :runners:flink:1.5:runQuickstartJavaFlinkLocal
+// Generates :runners:flink:1.8:runQuickstartJavaFlinkLocal
createJavaExamplesArchetypeValidationTask(type: 'Quickstart', runner: 'FlinkLocal')
diff --git a/runners/flink/job-server/flink_job_server.gradle b/runners/flink/job-server/flink_job_server.gradle
index e9f19ca..ee856e7 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -28,9 +28,10 @@
apply plugin: 'application'
// we need to set mainClassName before applying shadow plugin
mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
-archivesBaseName = project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.flink.jobserver',
+ archivesBaseName: project.hasProperty('archives_base_name') ? archives_base_name : archivesBaseName,
validateShadowJar: false,
exportJavadoc: false,
shadowClosure: {
@@ -76,17 +77,18 @@
}
dependencies {
- compile project(path: flinkRunnerProject, configuration: "shadow")
- compile group: "org.slf4j", name: "jcl-over-slf4j", version: dependencies.create(project.library.java.slf4j_api).getVersion()
- validatesPortableRunner project(path: flinkRunnerProject, configuration: "shadowTest")
+ compile project(flinkRunnerProject)
+ runtime group: "org.slf4j", name: "jcl-over-slf4j", version: dependencies.create(project.library.java.slf4j_api).getVersion()
+ validatesPortableRunner project(path: flinkRunnerProject, configuration: "testRuntime")
validatesPortableRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:reference:java", configuration: "shadowTest")
- compile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- compile library.java.slf4j_simple
+ validatesPortableRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesPortableRunner project(path: ":runners:reference:java", configuration: "testRuntime")
+ runtime project(":sdks:java:extensions:google-cloud-platform-core")
+ runtime library.java.slf4j_simple
// TODO: Enable AWS and HDFS file system.
// For resolving external transform requests
- compile project(path: ":sdks:java:io:kafka", configuration: "shadow")
+ runtime project(":sdks:java:io:kafka")
+ runtime library.java.kafka_clients
}
// NOTE: runShadow must be used in order to run the job server. The standard run
@@ -104,6 +106,8 @@
args += ["--flink-master-url=${project.property('flinkMasterUrl')}"]
if (project.hasProperty('flinkConfDir'))
args += ["--flink-conf-dir=${project.property('flinkConfDir')}"]
+ if (project.hasProperty('sdkWorkerParallelism'))
+ args += ["--sdk-worker-parallelism=${project.property('sdkWorkerParallelism')}"]
// Enable remote debugging.
jvmArgs = ["-Xdebug", "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
@@ -123,7 +127,7 @@
createPortableValidatesRunnerTask(
name: "validatesPortableRunner${name}",
jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
- jobServerConfig: "--job-host=localhost,--job-port=0,--artifact-port=0",
+ jobServerConfig: "--job-host=localhost,--job-port=0,--artifact-port=0,--expansion-port=0",
testClasspathConfiguration: configurations.validatesPortableRunner,
numParallelTests: 1,
pipelineOpts: pipelineOptions,
@@ -164,23 +168,31 @@
dependsOn validatesPortableRunnerStreaming
}
-project.ext.validatesCrossLanguageTransforms =
- createPortableValidatesRunnerTask(
- name: "validatesCrossLanguageTransforms",
- jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
- jobServerConfig: "--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0,--expansion-port=0",
- testClasspathConfiguration: configurations.validatesPortableRunner,
- numParallelTests: 1,
- pipelineOpts: [
- // Limit resource consumption via parallelism
- "--parallelism=2",
- "--shutdownSourcesOnFinalWatermark",
- ],
- testCategories: {
- // Only include cross-language transform tests. Avoid to retest everything on Docker environment.
- includeCategories 'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
- },
- )
-project.evaluationDependsOn ':sdks:python'
-validatesCrossLanguageTransforms.dependsOn ':sdks:python:setupVirtualenv'
-validatesCrossLanguageTransforms.systemProperty "pythonTestExpansionCommand", ". ${project(':sdks:python').envdir}/bin/activate && pip install -e ${project(':sdks:python').projectDir}[test] && python -m apache_beam.runners.portability.expansion_service_test"
+project.ext.validatesCrossLanguageRunner = createCrossLanguageValidatesRunnerTask(
+ jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
+ jobServerConfig: "--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0",
+ testClasspathConfiguration: configurations.validatesPortableRunner,
+ numParallelTests: 1,
+ pipelineOpts: [
+ "--parallelism=2",
+ "--shutdownSourcesOnFinalWatermark",
+ ]
+)
+
+task testPipelineJar() {
+ dependsOn shadowJar
+ dependsOn ":sdks:python:container:py35:docker"
+ doLast{
+ exec {
+ executable "sh"
+ def options = [
+ "--flink_job_server_jar ${shadowJar.archivePath}",
+ "--env_dir ${project.rootProject.buildDir}/gradleenv/${project.path.hashCode()}",
+ "--python_root_dir ${project.rootDir}/sdks/python",
+ "--python_version 3.5",
+ "--python_container_image apachebeam/python3.5_sdk:${project['python_sdk_version']}",
+ ]
+ args "-c", "../../job-server/test_pipeline_jar.sh ${options.join(' ')}"
+ }
+ }
+}
diff --git a/runners/flink/job-server/test_pipeline_jar.sh b/runners/flink/job-server/test_pipeline_jar.sh
new file mode 100755
index 0000000..9db6b79
--- /dev/null
+++ b/runners/flink/job-server/test_pipeline_jar.sh
@@ -0,0 +1,124 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+set -v
+
+while [[ $# -gt 0 ]]
+do
+key="$1"
+case $key in
+ --flink_job_server_jar)
+ FLINK_JOB_SERVER_JAR="$2"
+ shift # past argument
+ shift # past value
+ ;;
+ --env_dir)
+ ENV_DIR="$2"
+ shift # past argument
+ shift # past value
+ ;;
+ --python_root_dir)
+ PYTHON_ROOT_DIR="$2"
+ shift # past argument
+ shift # past value
+ ;;
+ --python_version)
+ PYTHON_VERSION="$2"
+ shift # past argument
+ shift # past value
+ ;;
+ --python_container_image)
+ PYTHON_CONTAINER_IMAGE="$2"
+ shift # past argument
+ shift # past value
+ ;;
+ *) # unknown option
+ echo "Unknown option: $1"
+ exit 1
+ ;;
+esac
+done
+
+# Go to the root of the repository
+cd $(git rev-parse --show-toplevel)
+
+# Verify docker command exists
+command -v docker
+docker -v
+
+# Verify container has already been built
+docker images --format "{{.Repository}}:{{.Tag}}" | grep $PYTHON_CONTAINER_IMAGE
+
+# Set up Python environment
+virtualenv -p python$PYTHON_VERSION $ENV_DIR
+. $ENV_DIR/bin/activate
+pip install --retries 10 -e $PYTHON_ROOT_DIR
+
+PIPELINE_PY="
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms import Create
+from apache_beam.transforms import Map
+
+# To test that our main session is getting plumbed through artifact staging
+# correctly, create a global variable. If the main session is not plumbed
+# through properly, global_var will be undefined and the pipeline will fail.
+global_var = 1
+
+pipeline_options = PipelineOptions()
+pipeline_options.view_as(SetupOptions).save_main_session = True
+pipeline = beam.Pipeline(options=pipeline_options)
+pcoll = (pipeline
+ | Create([0, 1, 2])
+ | Map(lambda x: x + global_var))
+assert_that(pcoll, equal_to([1, 2, 3]))
+
+result = pipeline.run()
+result.wait_until_finish()
+"
+
+# Create the jar
+OUTPUT_JAR=flink-test-$(date +%Y%m%d-%H%M%S).jar
+(python -c "$PIPELINE_PY" \
+ --runner FlinkRunner \
+ --flink_job_server_jar $FLINK_JOB_SERVER_JAR \
+ --output_executable_path $OUTPUT_JAR \
+ --parallelism 1 \
+ --sdk_worker_parallelism 1 \
+ --environment_type DOCKER \
+ --environment_config=$PYTHON_CONTAINER_IMAGE \
+) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
+
+if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
+ # Execute the jar
+ java -jar $OUTPUT_JAR || TEST_EXIT_CODE=$?
+fi
+
+rm -rf $ENV_DIR
+rm -f $OUTPUT_JAR
+
+if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
+ echo ">>> SUCCESS"
+else
+ echo ">>> FAILURE"
+fi
+exit $TEST_EXIT_CODE
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/CreateStreamingFlinkView.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/CreateStreamingFlinkView.java
index f94c961..c5ef3b6 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/CreateStreamingFlinkView.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/CreateStreamingFlinkView.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Flink streaming overrides for various view (side input) transforms. */
class CreateStreamingFlinkView<ElemT, ViewT>
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
index 29ac2b0..ee40fb6 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
@@ -21,7 +21,7 @@
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.createOutputMap;
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.getWindowingStrategy;
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.instantiateCoder;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -41,7 +41,6 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
import org.apache.beam.runners.core.construction.NativeTransforms;
import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.ReadTranslation;
import org.apache.beam.runners.core.construction.RehydratedComponents;
import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
@@ -49,7 +48,7 @@
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
-import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContext;
+import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction;
import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStagePruningFunction;
import org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
@@ -57,7 +56,6 @@
import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
import org.apache.beam.runners.flink.translation.types.KvKeySelector;
import org.apache.beam.runners.flink.translation.wrappers.ImpulseInputFormat;
-import org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.wire.WireCoders;
import org.apache.beam.sdk.coders.ByteArrayCoder;
@@ -66,7 +64,6 @@
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.ListCoder;
import org.apache.beam.sdk.coders.VoidCoder;
-import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
@@ -78,14 +75,12 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.DataSet;
@@ -154,10 +149,6 @@
translatorMap.put(
PTransformTranslation.RESHUFFLE_URN,
FlinkBatchPortablePipelineTranslator::translateReshuffle);
- // Remove once Reads can be wrapped in SDFs
- translatorMap.put(
- PTransformTranslation.READ_TRANSFORM_URN,
- FlinkBatchPortablePipelineTranslator::translateRead);
return new FlinkBatchPortablePipelineTranslator(translatorMap.build());
}
@@ -245,12 +236,7 @@
@Override
public Set<String> knownUrns() {
- // Do not expose Read as a known URN because we only want to support Read
- // through the Java ExpansionService. We can't translate Reads for other
- // languages.
- return Sets.difference(
- urnToTransformTranslator.keySet(),
- ImmutableSet.of(PTransformTranslation.READ_TRANSFORM_URN));
+ return urnToTransformTranslator.keySet();
}
/** Predicate to determine whether a URN is a Flink native transform. */
@@ -341,10 +327,12 @@
final FlinkExecutableStageFunction<InputT> function =
new FlinkExecutableStageFunction<>(
+ transform.getTransform().getUniqueName(),
+ context.getPipelineOptions(),
stagePayload,
context.getJobInfo(),
outputMap,
- FlinkExecutableStageContext.factory(context.getPipelineOptions()),
+ FlinkExecutableStageContextFactory.getInstance(),
getWindowingStrategy(inputPCollectionId, components).getWindowFn().windowCoder());
final String operatorName = generateNameFromStagePayload(stagePayload);
@@ -551,43 +539,6 @@
Iterables.getOnlyElement(transform.getTransform().getOutputsMap().values()), dataSource);
}
- private static <T> void translateRead(
- PTransformNode transform, RunnerApi.Pipeline pipeline, BatchTranslationContext context) {
- String outputPCollectionId =
- Iterables.getOnlyElement(transform.getTransform().getOutputsMap().values());
- PCollectionNode collectionNode =
- PipelineNode.pCollection(
- outputPCollectionId,
- pipeline.getComponents().getPcollectionsOrThrow(outputPCollectionId));
-
- Coder<WindowedValue<T>> outputCoder;
- try {
- outputCoder = WireCoders.instantiateRunnerWireCoder(collectionNode, pipeline.getComponents());
- } catch (IOException e) {
- throw new RuntimeException("Failed to instantiate output code for source", e);
- }
-
- BoundedSource boundedSource;
- try {
- boundedSource =
- ReadTranslation.boundedSourceFromProto(
- RunnerApi.ReadPayload.parseFrom(transform.getTransform().getSpec().getPayload()));
- } catch (IOException e) {
- throw new RuntimeException("Failed to extract BoundedSource from ReadPayload.", e);
- }
-
- ExecutionEnvironment env = context.getExecutionEnvironment();
- String name = transform.getTransform().getUniqueName();
- DataSource<? extends WindowedValue<T>> dataSource =
- new DataSource<>(
- env,
- new SourceInputFormat<>(name, boundedSource, context.getPipelineOptions()),
- new CoderTypeInformation<>(outputCoder),
- name);
-
- context.addDataSet(outputPCollectionId, dataSource);
- }
-
/**
* Combiner that combines {@code T}s into a single {@code List<T>} containing all inputs.
*
@@ -652,7 +603,7 @@
String collectionId) {
TypeInformation<WindowedValue<?>> outputType = new CoderTypeInformation<>(outputCoder);
FlinkExecutableStagePruningFunction pruningFunction =
- new FlinkExecutableStagePruningFunction(unionTag);
+ new FlinkExecutableStagePruningFunction(unionTag, context.getPipelineOptions());
FlatMapOperator<RawUnionValue, WindowedValue<?>> pruningOperator =
new FlatMapOperator<>(
taggedDataset,
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java
index a9b707e..229eca5 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTransformTranslators.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.flink;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.ArrayList;
@@ -74,8 +74,8 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.flink.api.common.functions.RichGroupReduceFunction;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.DataSet;
@@ -502,12 +502,14 @@
TupleTag<?> mainOutputTag;
DoFnSchemaInformation doFnSchemaInformation;
+ Map<String, PCollectionView<?>> sideInputMapping;
try {
mainOutputTag = ParDoTranslation.getMainOutputTag(context.getCurrentTransform());
} catch (IOException e) {
throw new RuntimeException(e);
}
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
+ sideInputMapping = ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
Map<TupleTag<?>, Integer> outputMap = Maps.newHashMap();
// put the main output at index 0, FlinkMultiOutputDoFnFunction expects this
outputMap.put(mainOutputTag, 0);
@@ -590,7 +592,8 @@
mainOutputTag,
inputCoder,
outputCoderMap,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
// Based on the fact that the signature is stateful, DoFnSignatures ensures
// that it is also keyed.
@@ -611,7 +614,8 @@
mainOutputTag,
context.getInput(transform).getCoder(),
outputCoderMap,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
outputDataSet =
new MapPartitionOperator<>(inputDataSet, typeInformation, doFnWrapper, fullName);
@@ -636,7 +640,7 @@
TypeInformation<WindowedValue<T>> outputType = context.getTypeInfo(collection);
FlinkMultiOutputPruningFunction<T> pruningFunction =
- new FlinkMultiOutputPruningFunction<>(integerTag);
+ new FlinkMultiOutputPruningFunction<>(integerTag, context.getPipelineOptions());
FlatMapOperator<WindowedValue<RawUnionValue>, WindowedValue<T>> pruningOperator =
new FlatMapOperator<>(taggedDataSet, outputType, pruningFunction, collection.getName());
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTranslationContext.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTranslationContext.java
index 466069f..8e366cf 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTranslationContext.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchTranslationContext.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
index b540e9a..e2a8900 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
@@ -23,9 +23,11 @@
import java.util.Collections;
import java.util.List;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.sdk.util.InstanceBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.ExecutionMode;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.java.CollectionEnvironment;
import org.apache.flink.api.java.ExecutionEnvironment;
@@ -40,6 +42,7 @@
import org.apache.flink.configuration.RestOptions;
import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.environment.CheckpointConfig.ExternalizedCheckpointCleanup;
import org.apache.flink.streaming.api.environment.RemoteStreamEnvironment;
@@ -91,8 +94,10 @@
LOG.info("Using Flink Master URL {}:{}.", hostAndPort.getHost(), hostAndPort.getPort());
}
- // Set the execution more for data exchange.
- flinkBatchEnv.getConfig().setExecutionMode(options.getExecutionModeForBatch());
+ // Set the execution mode for data exchange.
+ flinkBatchEnv
+ .getConfig()
+ .setExecutionMode(ExecutionMode.valueOf(options.getExecutionModeForBatch()));
// set the correct parallelism.
if (options.getParallelism() != -1 && !(flinkBatchEnv instanceof CollectionEnvironment)) {
@@ -212,7 +217,8 @@
if (checkpointInterval < 1) {
throw new IllegalArgumentException("The checkpoint interval must be positive");
}
- flinkStreamEnv.enableCheckpointing(checkpointInterval, options.getCheckpointingMode());
+ flinkStreamEnv.enableCheckpointing(
+ checkpointInterval, CheckpointingMode.valueOf(options.getCheckpointingMode()));
if (options.getCheckpointTimeoutMillis() != -1) {
flinkStreamEnv
.getCheckpointConfig()
@@ -246,8 +252,12 @@
}
// State backend
- final StateBackend stateBackend = options.getStateBackend();
- if (stateBackend != null) {
+ if (options.getStateBackendFactory() != null) {
+ final StateBackend stateBackend =
+ InstanceBuilder.ofType(FlinkStateBackendFactory.class)
+ .fromClass(options.getStateBackendFactory())
+ .build()
+ .createStateBackend(options);
flinkStreamEnv.setStateBackend(stateBackend);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java
index 1db7806..24edb28 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java
@@ -20,17 +20,18 @@
import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
import java.io.IOException;
-import java.util.List;
import java.util.UUID;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation;
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineJarCreator;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -63,7 +64,6 @@
String invocationId =
String.format("%s_%s", flinkOptions.getJobName(), UUID.randomUUID().toString());
- LOG.info("Invoking job {}", invocationId);
if (FlinkPipelineOptions.AUTO.equals(flinkOptions.getFlinkMaster())) {
flinkOptions.setFlinkMaster(serverConfig.getFlinkMasterUrl());
@@ -74,16 +74,23 @@
portableOptions.setSdkWorkerParallelism(serverConfig.getSdkWorkerParallelism());
}
+ PortablePipelineRunner pipelineRunner;
+ if (portableOptions.getOutputExecutablePath() == null
+ || portableOptions.getOutputExecutablePath().isEmpty()) {
+ pipelineRunner =
+ new FlinkPipelineRunner(
+ flinkOptions,
+ serverConfig.getFlinkConfDir(),
+ detectClassPathResourcesToStage(FlinkJobInvoker.class.getClassLoader()));
+ } else {
+ pipelineRunner = new PortablePipelineJarCreator(FlinkPipelineRunner.class);
+ }
+
flinkOptions.setRunner(null);
+ LOG.info("Invoking job {} with pipeline runner {}", invocationId, pipelineRunner);
return createJobInvocation(
- invocationId,
- retrievalToken,
- executorService,
- pipeline,
- flinkOptions,
- serverConfig.getFlinkConfDir(),
- detectClassPathResourcesToStage(FlinkJobInvoker.class.getClassLoader()));
+ invocationId, retrievalToken, executorService, pipeline, flinkOptions, pipelineRunner);
}
static JobInvocation createJobInvocation(
@@ -92,16 +99,13 @@
ListeningExecutorService executorService,
RunnerApi.Pipeline pipeline,
FlinkPipelineOptions flinkOptions,
- @Nullable String confDir,
- List<String> filesToStage) {
+ PortablePipelineRunner pipelineRunner) {
JobInfo jobInfo =
JobInfo.create(
invocationId,
flinkOptions.getJobName(),
retrievalToken,
PipelineOptionsTranslation.toProto(flinkOptions));
- FlinkPipelineRunner pipelineRunner =
- new FlinkPipelineRunner(flinkOptions, confDir, filesToStage);
return new JobInvocation(jobInfo, executorService, pipeline, pipelineRunner);
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
index 2f3f981..0c283d8 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
@@ -21,7 +21,9 @@
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
import org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver;
+import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
@@ -59,8 +61,11 @@
public static void main(String[] args) throws Exception {
// TODO: Expose the fileSystem related options.
+ PipelineOptions options = PipelineOptionsFactory.create();
+ // Limiting gcs upload buffer to reduce memory usage while doing parallel artifact uploads.
+ options.as(GcsOptions.class).setGcsUploadBufferSizeBytes(1024 * 1024);
// Register standard file systems.
- FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+ FileSystems.setDefaultPipelineOptions(options);
fromParams(args).run();
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
index 7f0389f..0b23d43 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.runners.flink;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.runners.core.construction.PipelineResources;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.runtime.jobgraph.JobGraph;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
index ed34600..e9cb8dc 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
@@ -17,23 +17,25 @@
*/
package org.apache.beam.runners.flink;
-import com.fasterxml.jackson.annotation.JsonIgnore;
import java.util.List;
import org.apache.beam.sdk.options.ApplicationNameOptions;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.StreamingOptions;
-import org.apache.flink.api.common.ExecutionMode;
-import org.apache.flink.runtime.state.StateBackend;
-import org.apache.flink.streaming.api.CheckpointingMode;
-/** Options which can be used to configure a Flink PortablePipelineRunner. */
+/**
+ * Options which can be used to configure a Flink PortablePipelineRunner.
+ *
+ * <p>Avoid using `org.apache.flink.*` members below. This allows including the flink runner without
+ * requiring flink on the classpath (e.g. to use with the direct runner).
+ */
public interface FlinkPipelineOptions
extends PipelineOptions, ApplicationNameOptions, StreamingOptions {
String AUTO = "[auto]";
String PIPELINED = "PIPELINED";
+ String EXACTLY_ONCE = "EXACTLY_ONCE";
/**
* List of local files to make available to workers.
@@ -90,10 +92,10 @@
void setCheckpointingInterval(Long interval);
@Description("The checkpointing mode that defines consistency guarantee.")
- @Default.Enum("EXACTLY_ONCE")
- CheckpointingMode getCheckpointingMode();
+ @Default.String(EXACTLY_ONCE)
+ String getCheckpointingMode();
- void setCheckpointingMode(CheckpointingMode mode);
+ void setCheckpointingMode(String mode);
@Description(
"The maximum time in milliseconds that a checkpoint may take before being discarded.")
@@ -145,12 +147,11 @@
* streaming mode.
*/
@Description(
- "Sets the state backend to use in streaming mode. "
- + "Otherwise the default is read from the Flink config.")
- @JsonIgnore
- StateBackend getStateBackend();
+ "Sets the state backend factory to use in streaming mode. "
+ + "Defaults to the flink cluster's state.backend configuration.")
+ Class<? extends FlinkStateBackendFactory> getStateBackendFactory();
- void setStateBackend(StateBackend stateBackend);
+ void setStateBackendFactory(Class<? extends FlinkStateBackendFactory> stateBackendFactory);
@Description("Enable/disable Beam metrics in Flink Runner")
@Default.Boolean(true)
@@ -217,10 +218,10 @@
+ "Reference {@link org.apache.flink.api.common.ExecutionMode}. "
+ "Set this to BATCH_FORCED if pipelines get blocked, see "
+ "https://issues.apache.org/jira/browse/FLINK-10672")
- @Default.Enum(PIPELINED)
- ExecutionMode getExecutionModeForBatch();
+ @Default.String(PIPELINED)
+ String getExecutionModeForBatch();
- void setExecutionModeForBatch(ExecutionMode executionMode);
+ void setExecutionModeForBatch(String executionMode);
@Description(
"Savepoint restore path. If specified, restores the streaming pipeline from the provided path.")
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
index f3dc60f..212c75e 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
@@ -17,20 +17,38 @@
*/
package org.apache.beam.runners.flink;
+import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.hasUnboundedPCollections;
import java.util.List;
+import java.util.Map;
+import java.util.UUID;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
import org.apache.beam.runners.core.construction.graph.PipelineTrimmer;
+import org.apache.beam.runners.core.metrics.MetricsPusher;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineJarUtils;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
+import org.apache.beam.sdk.metrics.MetricsOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions.RetrievalServiceType;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.flink.api.common.JobExecutionResult;
+import org.apache.flink.client.program.DetachedEnvironment;
+import org.kohsuke.args4j.CmdLineException;
+import org.kohsuke.args4j.CmdLineParser;
+import org.kohsuke.args4j.Option;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -50,7 +68,7 @@
}
@Override
- public PipelineResult run(final Pipeline pipeline, JobInfo jobInfo) throws Exception {
+ public PortablePipelineResult run(final Pipeline pipeline, JobInfo jobInfo) throws Exception {
MetricsEnvironment.setMetricsSupported(false);
FlinkPortablePipelineTranslator<?> translator;
@@ -64,7 +82,7 @@
}
private <T extends FlinkPortablePipelineTranslator.TranslationContext>
- PipelineResult runPipelineWithTranslator(
+ PortablePipelineResult runPipelineWithTranslator(
final Pipeline pipeline, JobInfo jobInfo, FlinkPortablePipelineTranslator<T> translator)
throws Exception {
LOG.info("Translating pipeline to Flink program.");
@@ -86,6 +104,111 @@
fusedPipeline);
final JobExecutionResult result = executor.execute(pipelineOptions.getJobName());
- return FlinkRunner.createPipelineResult(result, pipelineOptions);
+ return createPortablePipelineResult(result, pipelineOptions);
+ }
+
+ private PortablePipelineResult createPortablePipelineResult(
+ JobExecutionResult result, PipelineOptions options) {
+ if (result instanceof DetachedEnvironment.DetachedJobExecutionResult) {
+ LOG.info("Pipeline submitted in Detached mode");
+ // no metricsPusher because metrics are not supported in detached mode
+ return new FlinkPortableRunnerResult.Detached();
+ } else {
+ LOG.info("Execution finished in {} msecs", result.getNetRuntime());
+ Map<String, Object> accumulators = result.getAllAccumulatorResults();
+ if (accumulators != null && !accumulators.isEmpty()) {
+ LOG.info("Final accumulator values:");
+ for (Map.Entry<String, Object> entry : result.getAllAccumulatorResults().entrySet()) {
+ LOG.info("{} : {}", entry.getKey(), entry.getValue());
+ }
+ }
+ FlinkPortableRunnerResult flinkRunnerResult =
+ new FlinkPortableRunnerResult(accumulators, result.getNetRuntime());
+ MetricsPusher metricsPusher =
+ new MetricsPusher(
+ flinkRunnerResult.getMetricsContainerStepMap(),
+ options.as(MetricsOptions.class),
+ flinkRunnerResult);
+ metricsPusher.start();
+ return flinkRunnerResult;
+ }
+ }
+
+ /**
+ * Main method to be called only as the entry point to an executable jar with structure as defined
+ * in {@link PortablePipelineJarUtils}.
+ */
+ public static void main(String[] args) throws Exception {
+ // Register standard file systems.
+ FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+
+ FlinkPipelineRunnerConfiguration configuration = parseArgs(args);
+ String baseJobName =
+ configuration.baseJobName == null
+ ? PortablePipelineJarUtils.getDefaultJobName()
+ : configuration.baseJobName;
+ Preconditions.checkArgument(
+ baseJobName != null,
+ "No default job name found. Job name must be set using --base-job-name.");
+ Pipeline pipeline = PortablePipelineJarUtils.getPipelineFromClasspath(baseJobName);
+ Struct originalOptions = PortablePipelineJarUtils.getPipelineOptionsFromClasspath(baseJobName);
+
+ // Flink pipeline jars distribute and retrieve artifacts via the classpath.
+ PortablePipelineOptions portablePipelineOptions =
+ PipelineOptionsTranslation.fromProto(originalOptions).as(PortablePipelineOptions.class);
+ portablePipelineOptions.setRetrievalServiceType(RetrievalServiceType.CLASSLOADER);
+ String retrievalToken = PortablePipelineJarUtils.getArtifactManifestUri(baseJobName);
+
+ FlinkPipelineOptions flinkOptions = portablePipelineOptions.as(FlinkPipelineOptions.class);
+ String invocationId =
+ String.format("%s_%s", flinkOptions.getJobName(), UUID.randomUUID().toString());
+
+ FlinkPipelineRunner runner =
+ new FlinkPipelineRunner(
+ flinkOptions,
+ configuration.flinkConfDir,
+ detectClassPathResourcesToStage(FlinkPipelineRunner.class.getClassLoader()));
+ JobInfo jobInfo =
+ JobInfo.create(
+ invocationId,
+ flinkOptions.getJobName(),
+ retrievalToken,
+ PipelineOptionsTranslation.toProto(flinkOptions));
+ try {
+ runner.run(pipeline, jobInfo);
+ } catch (Exception e) {
+ throw new RuntimeException(String.format("Job %s failed.", invocationId), e);
+ }
+ LOG.info("Job {} finished successfully.", invocationId);
+ }
+
+ private static class FlinkPipelineRunnerConfiguration {
+ @Option(
+ name = "--flink-conf-dir",
+ usage =
+ "Directory containing Flink YAML configuration files. "
+ + "These properties will be set to all jobs submitted to Flink and take precedence "
+ + "over configurations in FLINK_CONF_DIR.")
+ private String flinkConfDir = null;
+
+ @Option(
+ name = "--base-job-name",
+ usage =
+ "The job to run. This must correspond to a subdirectory of the jar's BEAM-PIPELINE "
+ + "directory. *Only needs to be specified if the jar contains multiple pipelines.*")
+ private String baseJobName = null;
+ }
+
+ private static FlinkPipelineRunnerConfiguration parseArgs(String[] args) {
+ FlinkPipelineRunnerConfiguration configuration = new FlinkPipelineRunnerConfiguration();
+ CmdLineParser parser = new CmdLineParser(configuration);
+ try {
+ parser.parseArgument(args);
+ } catch (CmdLineException e) {
+ LOG.error("Unable to parse command line arguments.", e);
+ parser.printUsage(System.err);
+ throw new IllegalArgumentException("Unable to parse command line arguments.", e);
+ }
+ return configuration;
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableRunnerResult.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableRunnerResult.java
new file mode 100644
index 0000000..91f5610
--- /dev/null
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableRunnerResult.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import java.util.Map;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.pipeline.v1.MetricsApi;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Result of executing a portable {@link org.apache.beam.sdk.Pipeline} with Flink. */
+public class FlinkPortableRunnerResult extends FlinkRunnerResult implements PortablePipelineResult {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FlinkPortableRunnerResult.class);
+
+ FlinkPortableRunnerResult(Map<String, Object> accumulators, long runtime) {
+ super(accumulators, runtime);
+ }
+
+ @Override
+ public JobApi.MetricResults portableMetrics() throws UnsupportedOperationException {
+ Iterable<MetricsApi.MonitoringInfo> monitoringInfos =
+ this.getMetricsContainerStepMap().getMonitoringInfos();
+
+ return JobApi.MetricResults.newBuilder().addAllAttempted(monitoringInfos).build();
+ }
+
+ static class Detached extends FlinkDetachedRunnerResult implements PortablePipelineResult {
+
+ @Override
+ public JobApi.MetricResults portableMetrics() throws UnsupportedOperationException {
+ LOG.warn(
+ "Collecting monitoring infos is not implemented yet in Flink portable runner (detached mode).");
+ return JobApi.MetricResults.newBuilder().build();
+ }
+ }
+}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
index 5d8840a..649b5ec 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.runners.TransformHierarchy;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.client.program.DetachedEnvironment;
import org.apache.flink.runtime.jobgraph.JobGraph;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerRegistrar.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerRegistrar.java
index c7dca49..49b1bb0 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerRegistrar.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* AutoService registrar - will register FlinkRunner and FlinkOptions as possible pipeline runner
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
index a97882e..b0d7faf 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
@@ -19,7 +19,6 @@
import static org.apache.beam.runners.core.metrics.MetricsContainerStepMap.asAttemptedOnlyMetricResults;
-import java.io.IOException;
import java.util.Collections;
import java.util.Map;
import org.apache.beam.runners.core.metrics.MetricsContainerStepMap;
@@ -57,8 +56,9 @@
}
@Override
- public State cancel() throws IOException {
- throw new UnsupportedOperationException("FlinkRunnerResult does not support cancel.");
+ public State cancel() {
+ // We can only be called here when we are done.
+ return State.DONE;
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStateBackendFactory.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStateBackendFactory.java
new file mode 100644
index 0000000..bb50d49
--- /dev/null
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStateBackendFactory.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import org.apache.flink.runtime.state.StateBackend;
+
+/** Constructs a StateBackend to use from flink pipeline options. */
+public interface FlinkStateBackendFactory {
+ StateBackend createStateBackend(FlinkPipelineOptions options);
+}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
index 6be0457..e5ff563 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslator.java
@@ -54,9 +54,9 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.apache.flink.runtime.state.KeyGroupRangeAssignment;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Preconditions;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
index f6e0c50..92b07a4 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
@@ -44,13 +44,13 @@
import org.apache.beam.runners.core.construction.ReadTranslation;
import org.apache.beam.runners.core.construction.RehydratedComponents;
import org.apache.beam.runners.core.construction.RunnerPCollectionView;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.TestStreamTranslation;
-import org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource;
import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.PipelineNode;
import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
-import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContext;
+import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.functions.ImpulseSourceFunction;
import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
import org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
@@ -69,8 +69,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.LengthPrefixCoder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.testing.TestStream;
@@ -86,22 +86,26 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PCollectionViews;
import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.common.functions.FlatMapFunction;
-import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
@@ -213,8 +217,8 @@
// TODO Legacy transforms which need to be removed
// Consider removing now that timers are supported
translatorMap.put(STREAMING_IMPULSE_TRANSFORM_URN, this::translateStreamingImpulse);
- // Remove once Reads can be wrapped in SDFs
- translatorMap.put(PTransformTranslation.READ_TRANSFORM_URN, this::translateRead);
+ // Remove once unbounded Reads can be wrapped in SDFs
+ translatorMap.put(PTransformTranslation.READ_TRANSFORM_URN, this::translateUnboundedRead);
// For testing only
translatorMap.put(PTransformTranslation.TEST_STREAM_TRANSFORM_URN, this::translateTestStream);
@@ -224,9 +228,11 @@
@Override
public Set<String> knownUrns() {
- // Do not expose Read as a known URN because we only want to support Read
- // through the Java ExpansionService. We can't translate Reads for other
- // languages.
+ // Do not expose Read as a known URN because PipelineTrimmer otherwise removes
+ // the subtransforms which are added in case of bounded reads. We only have a
+ // translator here for unbounded Reads which are native transforms which do not
+ // have subtransforms. Unbounded Reads are used by cross-language transforms, e.g.
+ // KafkaIO.
return Sets.difference(
urnToTransformTranslator.keySet(),
ImmutableSet.of(PTransformTranslation.READ_TRANSFORM_URN));
@@ -396,7 +402,9 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, V>>> workItemStream =
inputDataStream
- .flatMap(new FlinkStreamingTransformTranslators.ToKeyedWorkItem<>())
+ .flatMap(
+ new FlinkStreamingTransformTranslators.ToKeyedWorkItem<>(
+ context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -444,7 +452,7 @@
}
@SuppressWarnings("unchecked")
- private <T> void translateRead(
+ private <T> void translateUnboundedRead(
String id, RunnerApi.Pipeline pipeline, StreamingTranslationContext context) {
RunnerApi.PTransform transform = pipeline.getComponents().getTransformsOrThrow(id);
String outputCollectionId = Iterables.getOnlyElement(transform.getOutputsMap().values());
@@ -456,15 +464,12 @@
throw new RuntimeException("Failed to parse ReadPayload from transform", e);
}
- final ReadSourceTranslator<T> readTranslator;
- if (payload.getIsBounded() == RunnerApi.IsBounded.Enum.BOUNDED) {
- readTranslator = new ReadBoundedTranslator<>();
- } else {
- readTranslator = new ReadUnboundedTranslator<>();
- }
+ Preconditions.checkState(
+ payload.getIsBounded() != RunnerApi.IsBounded.Enum.BOUNDED,
+ "Bounded reads should run inside an environment instead of being translated by the Runner.");
DataStream<WindowedValue<T>> source =
- readTranslator.translateRead(
+ translateUnboundedSource(
transform.getUniqueName(),
outputCollectionId,
payload,
@@ -475,123 +480,61 @@
context.addDataStream(outputCollectionId, source);
}
- interface ReadSourceTranslator<T> {
- DataStream<WindowedValue<T>> translateRead(
- String transformName,
- String outputCollectionId,
- RunnerApi.ReadPayload payload,
- RunnerApi.Pipeline pipeline,
- PipelineOptions pipelineOptions,
- StreamExecutionEnvironment env);
- }
+ private static <T> DataStream<WindowedValue<T>> translateUnboundedSource(
+ String transformName,
+ String outputCollectionId,
+ RunnerApi.ReadPayload payload,
+ RunnerApi.Pipeline pipeline,
+ PipelineOptions pipelineOptions,
+ StreamExecutionEnvironment env) {
- private static class ReadBoundedTranslator<T> implements ReadSourceTranslator<T> {
+ final DataStream<WindowedValue<T>> source;
+ final DataStream<WindowedValue<ValueWithRecordId<T>>> nonDedupSource;
+ Coder<WindowedValue<T>> windowCoder =
+ instantiateCoder(outputCollectionId, pipeline.getComponents());
- @Override
- @SuppressWarnings("unchecked")
- public DataStream<WindowedValue<T>> translateRead(
- String transformName,
- String outputCollectionId,
- RunnerApi.ReadPayload payload,
- RunnerApi.Pipeline pipeline,
- PipelineOptions pipelineOptions,
- StreamExecutionEnvironment env) {
- Coder<WindowedValue<T>> windowCoder =
- instantiateCoder(outputCollectionId, pipeline.getComponents());
- TypeInformation<WindowedValue<T>> outputTypeInfo = new CoderTypeInformation<>(windowCoder);
+ TypeInformation<WindowedValue<T>> outputTypeInfo = new CoderTypeInformation<>(windowCoder);
- UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<?> convertedSource;
- try {
- convertedSource =
- new UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<>(
- ReadTranslation.boundedSourceFromProto(payload));
- } catch (IOException e) {
- throw new RuntimeException("Failed to extract UnboundedSource from payload", e);
+ WindowingStrategy windowStrategy =
+ getWindowingStrategy(outputCollectionId, pipeline.getComponents());
+ TypeInformation<WindowedValue<ValueWithRecordId<T>>> withIdTypeInfo =
+ new CoderTypeInformation<>(
+ WindowedValue.getFullCoder(
+ ValueWithRecordId.ValueWithRecordIdCoder.of(
+ ((WindowedValueCoder) windowCoder).getValueCoder()),
+ windowStrategy.getWindowFn().windowCoder()));
+
+ UnboundedSource unboundedSource = ReadTranslation.unboundedSourceFromProto(payload);
+
+ try {
+ int parallelism =
+ env.getMaxParallelism() > 0 ? env.getMaxParallelism() : env.getParallelism();
+ UnboundedSourceWrapper sourceWrapper =
+ new UnboundedSourceWrapper<>(
+ transformName, pipelineOptions, unboundedSource, parallelism);
+ nonDedupSource =
+ env.addSource(sourceWrapper)
+ .name(transformName)
+ .uid(transformName)
+ .returns(withIdTypeInfo);
+
+ if (unboundedSource.requiresDeduping()) {
+ source =
+ nonDedupSource
+ .keyBy(new FlinkStreamingTransformTranslators.ValueWithRecordIdKeySelector<>())
+ .transform("deduping", outputTypeInfo, new DedupingOperator<>(pipelineOptions))
+ .uid(format("%s/__deduplicated__", transformName));
+ } else {
+ source =
+ nonDedupSource
+ .flatMap(new FlinkStreamingTransformTranslators.StripIdsMap<>(pipelineOptions))
+ .returns(outputTypeInfo);
}
-
- try {
- int parallelism =
- env.getMaxParallelism() > 0 ? env.getMaxParallelism() : env.getParallelism();
- FlinkStreamingTransformTranslators.UnboundedSourceWrapperNoValueWithRecordId sourceWrapper =
- new FlinkStreamingTransformTranslators.UnboundedSourceWrapperNoValueWithRecordId<>(
- new UnboundedSourceWrapper<>(
- transformName, pipelineOptions, convertedSource, parallelism));
- return env.addSource(sourceWrapper)
- .name(transformName)
- .uid(transformName)
- .returns(outputTypeInfo);
- } catch (Exception e) {
- throw new RuntimeException("Error while translating BoundedSource: " + convertedSource, e);
- }
+ } catch (Exception e) {
+ throw new RuntimeException("Error while translating UnboundedSource: " + unboundedSource, e);
}
- }
- private static class ReadUnboundedTranslator<T> implements ReadSourceTranslator<T> {
-
- @Override
- @SuppressWarnings("unchecked")
- public DataStream<WindowedValue<T>> translateRead(
- String transformName,
- String outputCollectionId,
- RunnerApi.ReadPayload payload,
- RunnerApi.Pipeline pipeline,
- PipelineOptions pipelineOptions,
- StreamExecutionEnvironment env) {
-
- final DataStream<WindowedValue<T>> source;
- final DataStream<WindowedValue<ValueWithRecordId<T>>> nonDedupSource;
- Coder<WindowedValue<T>> windowCoder =
- instantiateCoder(outputCollectionId, pipeline.getComponents());
-
- TypeInformation<WindowedValue<T>> outputTypeInfo = new CoderTypeInformation<>(windowCoder);
-
- WindowingStrategy windowStrategy =
- getWindowingStrategy(outputCollectionId, pipeline.getComponents());
- TypeInformation<WindowedValue<ValueWithRecordId<T>>> withIdTypeInfo =
- new CoderTypeInformation<>(
- WindowedValue.getFullCoder(
- ValueWithRecordId.ValueWithRecordIdCoder.of(
- ((WindowedValueCoder) windowCoder).getValueCoder()),
- windowStrategy.getWindowFn().windowCoder()));
-
- UnboundedSource unboundedSource;
- try {
- unboundedSource = ReadTranslation.unboundedSourceFromProto(payload);
- } catch (IOException e) {
- throw new RuntimeException("Failed to extract UnboundedSource from payload", e);
- }
-
- try {
- int parallelism =
- env.getMaxParallelism() > 0 ? env.getMaxParallelism() : env.getParallelism();
- UnboundedSourceWrapper sourceWrapper =
- new UnboundedSourceWrapper<>(
- transformName, pipelineOptions, unboundedSource, parallelism);
- nonDedupSource =
- env.addSource(sourceWrapper)
- .name(transformName)
- .uid(transformName)
- .returns(withIdTypeInfo);
-
- if (unboundedSource.requiresDeduping()) {
- source =
- nonDedupSource
- .keyBy(new FlinkStreamingTransformTranslators.ValueWithRecordIdKeySelector<>())
- .transform("deduping", outputTypeInfo, new DedupingOperator<>())
- .uid(format("%s/__deduplicated__", transformName));
- } else {
- source =
- nonDedupSource
- .flatMap(new FlinkStreamingTransformTranslators.StripIdsMap<>())
- .returns(outputTypeInfo);
- }
- } catch (Exception e) {
- throw new RuntimeException(
- "Error while translating UnboundedSource: " + unboundedSource, e);
- }
-
- return source;
- }
+ return source;
}
private void translateImpulse(
@@ -737,11 +680,6 @@
valueCoder.getClass().getSimpleName()));
}
keyCoder = ((KvCoder) valueCoder).getKeyCoder();
- if (keyCoder instanceof LengthPrefixCoder) {
- // Remove any unnecessary length prefixes which add more payload
- // but also are not expected for state requests inside the operator.
- keyCoder = ((LengthPrefixCoder) keyCoder).getValueCoder();
- }
keySelector = new KvToByteBufferKeySelector(keyCoder);
inputDataStream = inputDataStream.keyBy(keySelector);
}
@@ -765,7 +703,7 @@
context.getPipelineOptions(),
stagePayload,
context.getJobInfo(),
- FlinkExecutableStageContext.factory(context.getPipelineOptions()),
+ FlinkExecutableStageContextFactory.getInstance(),
collectionIdToTupleTag,
getWindowingStrategy(inputPCollectionId, components),
keyCoder,
@@ -868,7 +806,11 @@
new LinkedHashMap<>();
// for PCollectionView compatibility, not used to transform materialization
ViewFn<Iterable<WindowedValue<?>>, ?> viewFn =
- (ViewFn) new PCollectionViews.MultimapViewFn<Iterable<WindowedValue<Void>>, Void>();
+ (ViewFn)
+ new PCollectionViews.MultimapViewFn<>(
+ (PCollectionViews.TypeDescriptorSupplier<Iterable<WindowedValue<Void>>>)
+ () -> TypeDescriptors.iterables(new TypeDescriptor<WindowedValue<Void>>() {}),
+ (PCollectionViews.TypeDescriptorSupplier<Void>) TypeDescriptors::voids);
for (RunnerApi.ExecutableStagePayload.SideInputId sideInputId :
stagePayload.getSideInputsList()) {
@@ -977,7 +919,7 @@
sideInput.getKey().getTransformId() + "-" + sideInput.getKey().getLocalName();
WindowedValueCoder<KV<Void, Object>> kvCoder = kvCoders.get(intTag);
DataStream<WindowedValue<KV<Void, Object>>> keyedSideInputStream =
- sideInputStream.map(new ToVoidKeyValue());
+ sideInputStream.map(new ToVoidKeyValue(context.getPipelineOptions()));
SingleOutputStreamOperator<WindowedValue<KV<Void, Iterable<Object>>>> viewStream =
addGBK(
@@ -991,7 +933,9 @@
DataStream<RawUnionValue> unionValueStream =
viewStream
- .map(new FlinkStreamingTransformTranslators.ToRawUnion<>(intTag))
+ .map(
+ new FlinkStreamingTransformTranslators.ToRawUnion<>(
+ intTag, context.getPipelineOptions()))
.returns(unionTypeInformation);
if (sideInputUnion == null) {
@@ -1017,7 +961,21 @@
}
private static class ToVoidKeyValue<T>
- implements MapFunction<WindowedValue<T>, WindowedValue<KV<Void, T>>> {
+ extends RichMapFunction<WindowedValue<T>, WindowedValue<KV<Void, T>>> {
+
+ private final SerializablePipelineOptions options;
+
+ public ToVoidKeyValue(PipelineOptions pipelineOptions) {
+ this.options = new SerializablePipelineOptions(pipelineOptions);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
@Override
public WindowedValue<KV<Void, T>> map(WindowedValue<T> value) {
return value.withValue(KV.of(null, value.getValue()));
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 2350779..785bf2b 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -36,6 +36,7 @@
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.ParDoTranslation;
import org.apache.beam.runners.core.construction.ReadTranslation;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.SplittableParDo;
import org.apache.beam.runners.core.construction.TransformPayloadTranslatorRegistrar;
import org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
@@ -57,7 +58,9 @@
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.Combine;
@@ -86,12 +89,12 @@
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.flink.api.common.functions.FlatMapFunction;
-import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.functions.RichMapFunction;
import org.apache.flink.api.common.functions.StoppableFunction;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.functions.KeySelector;
@@ -224,10 +227,16 @@
source =
nonDedupSource
.keyBy(new ValueWithRecordIdKeySelector<>())
- .transform("deduping", outputTypeInfo, new DedupingOperator<>())
+ .transform(
+ "deduping",
+ outputTypeInfo,
+ new DedupingOperator<>(context.getPipelineOptions()))
.uid(format("%s/__deduplicated__", fullName));
} else {
- source = nonDedupSource.flatMap(new StripIdsMap<>()).returns(outputTypeInfo);
+ source =
+ nonDedupSource
+ .flatMap(new StripIdsMap<>(context.getPipelineOptions()))
+ .returns(outputTypeInfo);
}
} catch (Exception e) {
throw new RuntimeException("Error while translating UnboundedSource: " + rawSource, e);
@@ -253,7 +262,20 @@
}
public static class StripIdsMap<T>
- implements FlatMapFunction<WindowedValue<ValueWithRecordId<T>>, WindowedValue<T>> {
+ extends RichFlatMapFunction<WindowedValue<ValueWithRecordId<T>>, WindowedValue<T>> {
+
+ private final SerializablePipelineOptions options;
+
+ StripIdsMap(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
@Override
public void flatMap(
@@ -332,11 +354,20 @@
}
/** Wraps each element in a {@link RawUnionValue} with the given tag id. */
- public static class ToRawUnion<T> implements MapFunction<T, RawUnionValue> {
+ public static class ToRawUnion<T> extends RichMapFunction<T, RawUnionValue> {
private final int intTag;
+ private final SerializablePipelineOptions options;
- public ToRawUnion(int intTag) {
+ ToRawUnion(int intTag, PipelineOptions pipelineOptions) {
this.intTag = intTag;
+ this.options = new SerializablePipelineOptions(pipelineOptions);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
}
@Override
@@ -385,7 +416,9 @@
final int intTag = tagToIntMapping.get(tag);
DataStream<Object> sideInputStream = context.getInputDataStream(sideInput);
DataStream<RawUnionValue> unionValueStream =
- sideInputStream.map(new ToRawUnion<>(intTag)).returns(unionTypeInformation);
+ sideInputStream
+ .map(new ToRawUnion<>(intTag, context.getPipelineOptions()))
+ .returns(unionTypeInformation);
if (sideInputUnion == null) {
sideInputUnion = unionValueStream;
@@ -425,7 +458,8 @@
Coder keyCoder,
KeySelector<WindowedValue<InputT>, ?> keySelector,
Map<Integer, PCollectionView<?>> transformedSideInputs,
- DoFnSchemaInformation doFnSchemaInformation);
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping);
}
static <InputT, OutputT> void translateParDo(
@@ -437,6 +471,7 @@
TupleTag<OutputT> mainOutputTag,
List<TupleTag<?>> additionalOutputTags,
DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping,
FlinkStreamingTranslationContext context,
DoFnOperatorFactory<InputT, OutputT> doFnOperatorFactory) {
@@ -516,7 +551,8 @@
keyCoder,
keySelector,
new HashMap<>() /* side-input mapping */,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
outputStream =
inputDataStream.transform(transformName, outputTypeInformation, doFnOperator);
@@ -543,7 +579,8 @@
keyCoder,
keySelector,
transformedSideInputs.f0,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
if (stateful) {
// we have to manually construct the two-input transform because we're not
@@ -621,6 +658,9 @@
throw new RuntimeException(e);
}
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
+
TupleTagList additionalOutputTags;
try {
additionalOutputTags =
@@ -641,6 +681,7 @@
mainOutputTag,
additionalOutputTags.getAll(),
doFnSchemaInformation,
+ sideInputMapping,
context,
(doFn1,
stepName,
@@ -658,7 +699,8 @@
keyCoder,
keySelector,
transformedSideInputs,
- doFnSchemaInformation1) ->
+ doFnSchemaInformation1,
+ sideInputMapping1) ->
new DoFnOperator<>(
doFn1,
stepName,
@@ -675,7 +717,8 @@
context1.getPipelineOptions(),
keyCoder,
keySelector,
- doFnSchemaInformation1));
+ doFnSchemaInformation1,
+ sideInputMapping1));
}
}
@@ -700,6 +743,7 @@
transform.getMainOutputTag(),
transform.getAdditionalOutputTags().getAll(),
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
context,
(doFn,
stepName,
@@ -717,7 +761,8 @@
keyCoder,
keySelector,
transformedSideInputs,
- doFnSchemaInformation) ->
+ doFnSchemaInformation,
+ sideInputMapping) ->
new SplittableDoFnOperator<>(
doFn,
stepName,
@@ -842,7 +887,7 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItem<>())
+ .flatMap(new ToKeyedWorkItem<>(context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -942,7 +987,7 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItem<>())
+ .flatMap(new ToKeyedWorkItem<>(context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -1077,7 +1122,7 @@
DataStream<WindowedValue<SingletonKeyedWorkItem<K, InputT>>> workItemStream =
inputDataStream
- .flatMap(new ToKeyedWorkItemInGlobalWindow<>())
+ .flatMap(new ToKeyedWorkItemInGlobalWindow<>(context.getPipelineOptions()))
.returns(workItemTypeInfo)
.name("ToKeyedWorkItem");
@@ -1093,6 +1138,19 @@
extends RichFlatMapFunction<
WindowedValue<KV<K, InputT>>, WindowedValue<SingletonKeyedWorkItem<K, InputT>>> {
+ private final SerializablePipelineOptions options;
+
+ ToKeyedWorkItemInGlobalWindow(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
@Override
public void flatMap(
WindowedValue<KV<K, InputT>> inWithMultipleWindows,
@@ -1187,6 +1245,19 @@
extends RichFlatMapFunction<
WindowedValue<KV<K, InputT>>, WindowedValue<SingletonKeyedWorkItem<K, InputT>>> {
+ private final SerializablePipelineOptions options;
+
+ ToKeyedWorkItem(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
@Override
public void flatMap(
WindowedValue<KV<K, InputT>> inWithMultipleWindows,
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTranslationContext.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTranslationContext.java
index 163645d..7fc7ce0 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTranslationContext.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTranslationContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.HashMap;
import java.util.Map;
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
@@ -49,9 +49,8 @@
private final PipelineOptions options;
/**
- * Keeps a mapping between the output value of the PTransform and the Flink Operator
- * that produced it, after the translation of the correspondinf PTransform to its Flink
- * equivalent.
+ * Keeps a mapping between the output value of the PTransform and the Flink Operator that produced
+ * it, after the translation of the correspondinf PTransform to its Flink equivalent.
*/
private final Map<PValue, DataStream<?>> dataStreams;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java
index 5e77d6e..61b8ba5 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkTransformOverrides.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.List;
import org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems;
@@ -27,7 +27,7 @@
import org.apache.beam.runners.core.construction.SplittableParDoNaiveBounded;
import org.apache.beam.sdk.runners.PTransformOverride;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link PTransform} overrides for Flink runner. */
class FlinkTransformOverrides {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
index 024fc76..9d853a2 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java
@@ -26,7 +26,6 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.flink.api.common.functions.RuntimeContext;
import org.joda.time.Instant;
/**
@@ -40,10 +39,10 @@
private final DoFnRunner<InputT, OutputT> delegate;
public DoFnRunnerWithMetricsUpdate(
- String stepName, DoFnRunner<InputT, OutputT> delegate, RuntimeContext runtimeContext) {
+ String stepName, DoFnRunner<InputT, OutputT> delegate, FlinkMetricContainer metricContainer) {
this.stepName = stepName;
this.delegate = delegate;
- container = new FlinkMetricContainer(runtimeContext);
+ this.container = metricContainer;
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
index 8a1781d..f8f26eb 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.flink.api.common.accumulators.Accumulator;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.configuration.GlobalConfiguration;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/AbstractFlinkCombineRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/AbstractFlinkCombineRunner.java
index b889649..133c944 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/AbstractFlinkCombineRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/AbstractFlinkCombineRunner.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.flink.util.Collector;
/**
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkAssignContext.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkAssignContext.java
index 5eeb996..137ff23 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkAssignContext.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkAssignContext.java
@@ -20,7 +20,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/** {@link org.apache.beam.sdk.transforms.windowing.WindowFn.AssignContext} for Flink functions. */
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java
deleted file mode 100644
index 1632a4a..0000000
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.functions;
-
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
-import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
-import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-
-/** Implementation of a {@link FlinkExecutableStageContext}. */
-class FlinkDefaultExecutableStageContext implements FlinkExecutableStageContext, AutoCloseable {
- private final JobBundleFactory jobBundleFactory;
-
- private static FlinkDefaultExecutableStageContext create(JobInfo jobInfo) {
- JobBundleFactory jobBundleFactory = DefaultJobBundleFactory.create(jobInfo);
- return new FlinkDefaultExecutableStageContext(jobBundleFactory);
- }
-
- private FlinkDefaultExecutableStageContext(JobBundleFactory jobBundleFactory) {
- this.jobBundleFactory = jobBundleFactory;
- }
-
- @Override
- public StageBundleFactory getStageBundleFactory(ExecutableStage executableStage) {
- return jobBundleFactory.forStage(executableStage);
- }
-
- @Override
- public void close() throws Exception {
- jobBundleFactory.close();
- }
-
- private static class JobFactoryState {
- private int index = 0;
- private final List<ReferenceCountingFlinkExecutableStageContextFactory> factories =
- new ArrayList<>();
- private final int maxFactories;
-
- private JobFactoryState(int maxFactories) {
- Preconditions.checkArgument(maxFactories >= 0, "sdk_worker_parallelism must be >= 0");
-
- if (maxFactories == 0) {
- // if this is 0, use the auto behavior of num_cores - 1 so that we leave some resources
- // available for the java process
- this.maxFactories = Math.max(Runtime.getRuntime().availableProcessors() - 1, 1);
- } else {
- this.maxFactories = maxFactories;
- }
- }
-
- private synchronized FlinkExecutableStageContext.Factory getFactory() {
- ReferenceCountingFlinkExecutableStageContextFactory factory;
- // If we haven't yet created maxFactories factories, create a new one. Otherwise use an
- // existing one from factories.
- if (factories.size() < maxFactories) {
- factory =
- ReferenceCountingFlinkExecutableStageContextFactory.create(
- FlinkDefaultExecutableStageContext::create);
- factories.add(factory);
- } else {
- factory = factories.get(index);
- }
-
- index = (index + 1) % maxFactories;
-
- return factory;
- }
- }
-
- enum MultiInstanceFactory implements Factory {
- MULTI_INSTANCE;
-
- // This map should only ever have a single element, as each job will have its own
- // classloader and therefore its own instance of MultiInstanceFactory.INSTANCE. This
- // code supports multiple JobInfos in order to provide a sensible implementation of
- // Factory.get(JobInfo), which in theory could be called with different JobInfos.
- private static final ConcurrentMap<String, JobFactoryState> jobFactories =
- new ConcurrentHashMap<>();
-
- @Override
- public FlinkExecutableStageContext get(JobInfo jobInfo) {
- JobFactoryState state =
- jobFactories.computeIfAbsent(
- jobInfo.jobId(),
- k -> {
- PortablePipelineOptions portableOptions =
- PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
- .as(PortablePipelineOptions.class);
-
- return new JobFactoryState(
- MoreObjects.firstNonNull(portableOptions.getSdkWorkerParallelism(), 1L)
- .intValue());
- });
-
- return state.getFactory().get(jobInfo);
- }
- }
-}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
index d125ee2..f5d2bd1 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunction.java
@@ -25,8 +25,10 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
@@ -37,7 +39,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.flink.api.common.functions.RichMapPartitionFunction;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.configuration.Configuration;
@@ -67,6 +69,7 @@
private final Coder<InputT> inputCoder;
private final Map<TupleTag<?>, Coder<?>> outputCoderMap;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
private transient DoFnInvoker<InputT, OutputT> doFnInvoker;
@@ -80,7 +83,8 @@
TupleTag<OutputT> mainOutputTag,
Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoderMap,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.doFn = doFn;
this.stepName = stepName;
@@ -92,6 +96,7 @@
this.inputCoder = inputCoder;
this.outputCoderMap = outputCoderMap;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
@Override
@@ -123,10 +128,13 @@
inputCoder,
outputCoderMap,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
if ((serializedOptions.get().as(FlinkPipelineOptions.class)).getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ doFnRunner =
+ new DoFnRunnerWithMetricsUpdate<>(
+ stepName, doFnRunner, new FlinkMetricContainer(getRuntimeContext()));
}
doFnRunner.startBundle();
@@ -139,7 +147,11 @@
}
@Override
- public void open(Configuration parameters) throws Exception {
+ public void open(Configuration parameters) {
+ // Note that the SerializablePipelineOptions already initialize FileSystems in the readObject()
+ // deserialization method. However, this is a hack, and we want to properly initialize the
+ // options where they are needed.
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
doFnInvoker = DoFnInvokers.tryInvokeSetupFor(doFn);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContext.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContext.java
deleted file mode 100644
index c3bc0ab..0000000
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContext.java
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.functions;
-
-import java.io.Serializable;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.flink.FlinkPipelineOptions;
-import org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.MultiInstanceFactory;
-import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
-import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-
-/** The Flink context required in order to execute {@link ExecutableStage stages}. */
-public interface FlinkExecutableStageContext extends AutoCloseable {
-
- /**
- * Creates {@link FlinkExecutableStageContext} instances. Serializable so that factories can be
- * defined at translation time and distributed to TaskManagers.
- */
- interface Factory extends Serializable {
-
- /** Get or create {@link FlinkExecutableStageContext} for given {@link JobInfo}. */
- FlinkExecutableStageContext get(JobInfo jobInfo);
- }
-
- static Factory factory(FlinkPipelineOptions options) {
- return MultiInstanceFactory.MULTI_INSTANCE;
- }
-
- StageBundleFactory getStageBundleFactory(ExecutableStage executableStage);
-}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContextFactory.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContextFactory.java
new file mode 100644
index 0000000..59e5e34
--- /dev/null
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContextFactory.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.translation.functions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.flink.api.java.ExecutionEnvironment;
+
+/** Singleton class that contains one {@link MultiInstanceFactory} per job. */
+public class FlinkExecutableStageContextFactory implements ExecutableStageContext.Factory {
+
+ private static final FlinkExecutableStageContextFactory instance =
+ new FlinkExecutableStageContextFactory();
+ // This map should only ever have a single element, as each job will have its own
+ // classloader and therefore its own instance of FlinkExecutableStageContextFactory. This
+ // code supports multiple JobInfos in order to provide a sensible implementation of
+ // Factory.get(JobInfo), which in theory could be called with different JobInfos.
+ private static final ConcurrentMap<String, MultiInstanceFactory> jobFactories =
+ new ConcurrentHashMap<>();
+
+ private FlinkExecutableStageContextFactory() {}
+
+ public static FlinkExecutableStageContextFactory getInstance() {
+ return instance;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+ MultiInstanceFactory jobFactory =
+ jobFactories.computeIfAbsent(
+ jobInfo.jobId(),
+ k -> {
+ PortablePipelineOptions portableOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
+ .as(PortablePipelineOptions.class);
+
+ return new MultiInstanceFactory(
+ MoreObjects.firstNonNull(portableOptions.getSdkWorkerParallelism(), 1L)
+ .intValue(),
+ // Clean up context immediately if its class is not loaded on Flink parent
+ // classloader.
+ (caller) ->
+ caller.getClass().getClassLoader()
+ != ExecutionEnvironment.class.getClassLoader());
+ });
+
+ return jobFactory.get(jobInfo);
+ }
+}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
index c02aa65..0b77a69 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunction.java
@@ -18,52 +18,39 @@
package org.apache.beam.runners.flink.translation.functions;
import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
import java.util.EnumMap;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
import java.util.Locale;
import java.util.Map;
-import java.util.function.BiConsumer;
import javax.annotation.Nullable;
import javax.annotation.concurrent.GuardedBy;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateKey;
import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.runners.core.InMemoryStateInternals;
import org.apache.beam.runners.core.InMemoryTimerInternals;
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.StateNamespace;
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.StateTag;
-import org.apache.beam.runners.core.StateTags;
import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.Timer;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.core.construction.graph.TimerReference;
import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
import org.apache.beam.runners.fnexecution.control.RemoteBundle;
import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.control.TimerReceiverFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.state.InMemoryBagUserStateFactory;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.runners.fnexecution.translation.BatchSideInputHandlerFactory;
+import org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.io.FileSystems;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.state.BagState;
-import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.flink.api.common.functions.AbstractRichFunction;
@@ -93,22 +80,24 @@
// Main constructor fields. All must be Serializable because Flink distributes Functions to
// task managers via java serialization.
+ // Pipeline options for initializing the FileSystems
+ private final SerializablePipelineOptions pipelineOptions;
// The executable stage this function will run.
private final RunnerApi.ExecutableStagePayload stagePayload;
// Pipeline options. Used for provisioning api.
private final JobInfo jobInfo;
// Map from PCollection id to the union tag used to represent this PCollection in the output.
private final Map<String, Integer> outputMap;
- private final FlinkExecutableStageContext.Factory contextFactory;
+ private final FlinkExecutableStageContextFactory contextFactory;
private final Coder windowCoder;
- // Unique name for namespacing metrics; currently just takes the input ID
- private final String stageName;
+ // Unique name for namespacing metrics
+ private final String stepName;
// Worker-local fields. These should only be constructed and consumed on Flink TaskManagers.
private transient RuntimeContext runtimeContext;
private transient FlinkMetricContainer container;
private transient StateRequestHandler stateRequestHandler;
- private transient FlinkExecutableStageContext stageContext;
+ private transient ExecutableStageContext stageContext;
private transient StageBundleFactory stageBundleFactory;
private transient BundleProgressHandler progressHandler;
// Only initialized when the ExecutableStage is stateful
@@ -118,24 +107,26 @@
private transient Object currentTimerKey;
public FlinkExecutableStageFunction(
+ String stepName,
+ PipelineOptions pipelineOptions,
RunnerApi.ExecutableStagePayload stagePayload,
JobInfo jobInfo,
Map<String, Integer> outputMap,
- FlinkExecutableStageContext.Factory contextFactory,
+ FlinkExecutableStageContextFactory contextFactory,
Coder windowCoder) {
+ this.stepName = stepName;
+ this.pipelineOptions = new SerializablePipelineOptions(pipelineOptions);
this.stagePayload = stagePayload;
this.jobInfo = jobInfo;
this.outputMap = outputMap;
this.contextFactory = contextFactory;
this.windowCoder = windowCoder;
- this.stageName = stagePayload.getInput();
}
@Override
public void open(Configuration parameters) throws Exception {
// Register standard file systems.
- // TODO Use actual pipeline options.
- FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+ FileSystems.setDefaultPipelineOptions(pipelineOptions.get());
executableStage = ExecutableStage.fromPayload(stagePayload);
runtimeContext = getRuntimeContext();
container = new FlinkMetricContainer(getRuntimeContext());
@@ -152,12 +143,12 @@
new BundleProgressHandler() {
@Override
public void onProgress(ProcessBundleProgressResponse progress) {
- container.updateMetrics(stageName, progress.getMonitoringInfosList());
+ container.updateMetrics(stepName, progress.getMonitoringInfosList());
}
@Override
public void onCompleted(ProcessBundleResponse response) {
- container.updateMetrics(stageName, response.getMonitoringInfosList());
+ container.updateMetrics(stepName, response.getMonitoringInfosList());
}
};
}
@@ -232,8 +223,6 @@
outputMap,
new TimerReceiverFactory(
stageBundleFactory,
- executableStage.getTimers(),
- stageBundleFactory.getProcessBundleDescriptor().getTimerSpecs(),
(WindowedValue timerElement, TimerInternals.TimerData timerData) -> {
currentTimerKey = ((KV) timerElement.getValue()).getKey();
timerInternals.setTimer(timerData);
@@ -256,7 +245,7 @@
try (RemoteBundle bundle =
stageBundleFactory.getBundle(receiverFactory, stateRequestHandler, progressHandler)) {
- fireEligibleTimers(
+ PipelineTranslatorUtils.fireEligibleTimers(
timerInternals,
(String timerId, WindowedValue timerValue) -> {
FnDataReceiver<WindowedValue<?>> fnTimerReceiver =
@@ -268,7 +257,8 @@
throw new RuntimeException(
String.format(Locale.ENGLISH, "Failed to process timer: %s", timerValue));
}
- });
+ },
+ currentTimerKey);
}
}
@@ -287,48 +277,6 @@
}
}
- /**
- * Fires all timers which are ready to be fired. This is done in a loop because timers may itself
- * schedule timers.
- */
- private void fireEligibleTimers(
- InMemoryTimerInternals timerInternals, BiConsumer<String, WindowedValue> timerConsumer) {
-
- boolean hasFired;
- do {
- hasFired = false;
- TimerInternals.TimerData timer;
-
- while ((timer = timerInternals.removeNextEventTimer()) != null) {
- hasFired = true;
- fireTimer(timer, timerConsumer);
- }
- while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
- hasFired = true;
- fireTimer(timer, timerConsumer);
- }
- while ((timer = timerInternals.removeNextSynchronizedProcessingTimer()) != null) {
- hasFired = true;
- fireTimer(timer, timerConsumer);
- }
- } while (hasFired);
- }
-
- private void fireTimer(
- TimerInternals.TimerData timer, BiConsumer<String, WindowedValue> timerConsumer) {
- StateNamespace namespace = timer.getNamespace();
- Preconditions.checkArgument(namespace instanceof StateNamespaces.WindowNamespace);
- BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
- Instant timestamp = timer.getTimestamp();
- WindowedValue<KV<Object, Timer>> timerValue =
- WindowedValue.of(
- KV.of(currentTimerKey, Timer.of(timestamp, new byte[0])),
- timestamp,
- Collections.singleton(window),
- PaneInfo.NO_FIRING);
- timerConsumer.accept(timer.getTimerId(), timerValue);
- }
-
@Override
public void close() throws Exception {
// close may be called multiple times when an exception is thrown
@@ -389,149 +337,4 @@
}
}
}
-
- private static class TimerReceiverFactory implements OutputReceiverFactory {
-
- private final StageBundleFactory stageBundleFactory;
- /** Timer PCollection id => TimerReference. */
- private final HashMap<String, ProcessBundleDescriptors.TimerSpec> timerOutputIdToSpecMap;
- /** Timer PCollection id => timer name => TimerSpec. */
- private final Map<String, Map<String, ProcessBundleDescriptors.TimerSpec>> timerSpecMap;
-
- private final BiConsumer<WindowedValue, TimerInternals.TimerData> timerDataConsumer;
- private final Coder windowCoder;
-
- TimerReceiverFactory(
- StageBundleFactory stageBundleFactory,
- Collection<TimerReference> timerReferenceCollection,
- Map<String, Map<String, ProcessBundleDescriptors.TimerSpec>> timerSpecMap,
- BiConsumer<WindowedValue, TimerInternals.TimerData> timerDataConsumer,
- Coder windowCoder) {
- this.stageBundleFactory = stageBundleFactory;
- this.timerOutputIdToSpecMap = new HashMap<>();
- // Gather all timers from all transforms by their output pCollectionId which is unique
- for (Map<String, ProcessBundleDescriptors.TimerSpec> transformTimerMap :
- stageBundleFactory.getProcessBundleDescriptor().getTimerSpecs().values()) {
- for (ProcessBundleDescriptors.TimerSpec timerSpec : transformTimerMap.values()) {
- timerOutputIdToSpecMap.put(timerSpec.outputCollectionId(), timerSpec);
- }
- }
- this.timerSpecMap = timerSpecMap;
- this.timerDataConsumer = timerDataConsumer;
- this.windowCoder = windowCoder;
- }
-
- @Override
- public <OutputT> FnDataReceiver<OutputT> create(String pCollectionId) {
- final ProcessBundleDescriptors.TimerSpec timerSpec =
- timerOutputIdToSpecMap.get(pCollectionId);
-
- return receivedElement -> {
- WindowedValue windowedValue = (WindowedValue) receivedElement;
- Timer timer =
- Preconditions.checkNotNull(
- (Timer) ((KV) windowedValue.getValue()).getValue(),
- "Received null Timer from SDK harness: %s",
- receivedElement);
- LOG.debug("Timer received: {} {}", pCollectionId, timer);
- for (Object window : windowedValue.getWindows()) {
- StateNamespace namespace = StateNamespaces.window(windowCoder, (BoundedWindow) window);
- TimeDomain timeDomain = timerSpec.getTimerSpec().getTimeDomain();
- String timerId = timerSpec.inputCollectionId();
- TimerInternals.TimerData timerData =
- TimerInternals.TimerData.of(timerId, namespace, timer.getTimestamp(), timeDomain);
- timerDataConsumer.accept(windowedValue, timerData);
- }
- };
- }
- }
-
- /**
- * Holds user state in memory if the ExecutableStage is stateful. Only one key is active at a time
- * due to the GroupReduceFunction being called once per key. Needs to be reset via {@code
- * resetForNewKey()} before processing a new key.
- */
- private static class InMemoryBagUserStateFactory
- implements StateRequestHandlers.BagUserStateHandlerFactory {
-
- private List<InMemorySingleKeyBagState> handlers;
-
- private InMemoryBagUserStateFactory() {
- handlers = new ArrayList<>();
- }
-
- @Override
- public <K, V, W extends BoundedWindow>
- StateRequestHandlers.BagUserStateHandler<K, V, W> forUserState(
- String pTransformId,
- String userStateId,
- Coder<K> keyCoder,
- Coder<V> valueCoder,
- Coder<W> windowCoder) {
-
- InMemorySingleKeyBagState<K, V, W> bagUserStateHandler =
- new InMemorySingleKeyBagState<>(userStateId, valueCoder, windowCoder);
- handlers.add(bagUserStateHandler);
-
- return bagUserStateHandler;
- }
-
- /** Prepares previous emitted state handlers for processing a new key. */
- void resetForNewKey() {
- for (InMemorySingleKeyBagState stateBags : handlers) {
- stateBags.reset();
- }
- }
-
- static class InMemorySingleKeyBagState<K, V, W extends BoundedWindow>
- implements StateRequestHandlers.BagUserStateHandler<K, V, W> {
-
- private final StateTag<BagState<V>> stateTag;
- private final Coder<W> windowCoder;
-
- /* Lazily initialized state internals upon first access */
- private volatile StateInternals stateInternals;
-
- InMemorySingleKeyBagState(String userStateId, Coder<V> valueCoder, Coder<W> windowCoder) {
- this.windowCoder = windowCoder;
- this.stateTag = StateTags.bag(userStateId, valueCoder);
- }
-
- @Override
- public Iterable<V> get(K key, W window) {
- initStateInternals(key);
- StateNamespace namespace = StateNamespaces.window(windowCoder, window);
- BagState<V> bagState = stateInternals.state(namespace, stateTag);
- return bagState.read();
- }
-
- @Override
- public void append(K key, W window, Iterator<V> values) {
- initStateInternals(key);
- StateNamespace namespace = StateNamespaces.window(windowCoder, window);
- BagState<V> bagState = stateInternals.state(namespace, stateTag);
- while (values.hasNext()) {
- bagState.add(values.next());
- }
- }
-
- @Override
- public void clear(K key, W window) {
- initStateInternals(key);
- StateNamespace namespace = StateNamespaces.window(windowCoder, window);
- BagState<V> bagState = stateInternals.state(namespace, stateTag);
- bagState.clear();
- }
-
- private void initStateInternals(K key) {
- if (stateInternals == null) {
- stateInternals = InMemoryStateInternals.forKey(key);
- }
- }
-
- void reset() {
- stateInternals = null;
- }
- }
- }
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java
index 12d5a51..639f297 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStagePruningFunction.java
@@ -17,23 +17,36 @@
*/
package org.apache.beam.runners.flink.translation.functions;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/** A Flink function that demultiplexes output from a {@link FlinkExecutableStageFunction}. */
public class FlinkExecutableStagePruningFunction
- implements FlatMapFunction<RawUnionValue, WindowedValue<?>> {
+ extends RichFlatMapFunction<RawUnionValue, WindowedValue<?>> {
private final int unionTag;
+ private final SerializablePipelineOptions options;
/**
* Creates a {@link FlinkExecutableStagePruningFunction} that extracts elements of the given union
* tag.
*/
- public FlinkExecutableStagePruningFunction(int unionTag) {
+ public FlinkExecutableStagePruningFunction(int unionTag, PipelineOptions pipelineOptions) {
this.unionTag = unionTag;
+ this.options = new SerializablePipelineOptions(pipelineOptions);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
index b9af5ad..b34649f 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
@@ -19,6 +19,7 @@
import java.util.Map;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -28,6 +29,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.flink.api.common.functions.RichGroupReduceFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -64,6 +66,13 @@
}
@Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
+ }
+
+ @Override
public void reduce(
Iterable<WindowedValue<KV<K, InputT>>> elements, Collector<WindowedValue<KV<K, OutputT>>> out)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java
index 27801e3..787b172 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMultiOutputPruningFunction.java
@@ -17,9 +17,14 @@
*/
package org.apache.beam.runners.flink.translation.functions;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -28,12 +33,21 @@
* FlinkDoFnFunction}.
*/
public class FlinkMultiOutputPruningFunction<T>
- implements FlatMapFunction<WindowedValue<RawUnionValue>, WindowedValue<T>> {
+ extends RichFlatMapFunction<WindowedValue<RawUnionValue>, WindowedValue<T>> {
private final int ourOutputTag;
+ private final SerializablePipelineOptions options;
- public FlinkMultiOutputPruningFunction(int ourOutputTag) {
+ public FlinkMultiOutputPruningFunction(int ourOutputTag, PipelineOptions options) {
this.ourOutputTag = ourOutputTag;
+ this.options = new SerializablePipelineOptions(options);
+ }
+
+ @Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
index 94f6778..b073304 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
@@ -19,6 +19,7 @@
import java.util.Map;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -28,6 +29,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.flink.api.common.functions.RichGroupCombineFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -63,6 +65,13 @@
}
@Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
+ }
+
+ @Override
public void combine(
Iterable<WindowedValue<KV<K, InputT>>> elements, Collector<WindowedValue<KV<K, AccumT>>> out)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
index 36cfd69..8ebf63c 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
@@ -19,6 +19,7 @@
import java.util.Map;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -28,6 +29,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.flink.api.common.functions.RichGroupReduceFunction;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.util.Collector;
/**
@@ -65,6 +67,13 @@
}
@Override
+ public void open(Configuration parameters) {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
+ }
+
+ @Override
public void reduce(
Iterable<WindowedValue<KV<K, AccumT>>> elements, Collector<WindowedValue<KV<K, OutputT>>> out)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
index 6759c0b..65d833d 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkSideInputReader.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.flink.translation.functions;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Collections;
import java.util.HashMap;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
index a4ce42c..eac0298 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
@@ -34,8 +34,10 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
@@ -48,7 +50,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.flink.api.common.functions.RichGroupReduceFunction;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.configuration.Configuration;
@@ -69,6 +71,7 @@
private final Coder<KV<K, V>> inputCoder;
private final Map<TupleTag<?>, Coder<?>> outputCoderMap;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
private transient DoFnInvoker doFnInvoker;
public FlinkStatefulDoFnFunction(
@@ -81,7 +84,8 @@
TupleTag<OutputT> mainOutputTag,
Coder<KV<K, V>> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoderMap,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.dofn = dofn;
this.stepName = stepName;
@@ -93,6 +97,7 @@
this.inputCoder = inputCoder;
this.outputCoderMap = outputCoderMap;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
@Override
@@ -149,10 +154,13 @@
inputCoder,
outputCoderMap,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
if ((serializedOptions.get().as(FlinkPipelineOptions.class)).getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ doFnRunner =
+ new DoFnRunnerWithMetricsUpdate<>(
+ stepName, doFnRunner, new FlinkMetricContainer(getRuntimeContext()));
}
doFnRunner.startBundle();
@@ -210,7 +218,11 @@
}
@Override
- public void open(Configuration parameters) throws Exception {
+ public void open(Configuration parameters) {
+ // Note that the SerializablePipelineOptions already initialize FileSystems in the readObject()
+ // deserialization method. However, this is a hack, and we want to properly initialize the
+ // options where they are needed.
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
doFnInvoker = DoFnInvokers.tryInvokeSetupFor(dofn);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java
index 03e5537..a4016e2 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStreamingSideInputHandlerFactory.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.flink.translation.functions;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* {@link StateRequestHandler} that uses {@link org.apache.beam.runners.core.SideInputHandler} to
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
index de13e52..6c518c7 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/HashingFlinkCombineRunner.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.util.Collector;
import org.joda.time.Instant;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunction.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunction.java
index c37dcb3..4e50ce1 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunction.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunction.java
@@ -18,6 +18,13 @@
package org.apache.beam.runners.flink.translation.functions;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
/**
@@ -25,7 +32,8 @@
* source alive although its work is already done. It will only shutdown when the streaming job is
* cancelled.
*/
-public class ImpulseSourceFunction implements SourceFunction<WindowedValue<byte[]>> {
+public class ImpulseSourceFunction
+ implements SourceFunction<WindowedValue<byte[]>>, CheckpointedFunction {
/** Keep source running even after it has done all the work. */
private final boolean keepSourceAlive;
@@ -33,6 +41,9 @@
/** Indicates the streaming job is running and the source can produce elements. */
private volatile boolean running;
+ /** Checkpointed state which indicates whether the impulse has finished. */
+ private transient ListState<Boolean> impulseEmitted;
+
public ImpulseSourceFunction(boolean keepSourceAlive) {
this.keepSourceAlive = keepSourceAlive;
this.running = true;
@@ -40,8 +51,13 @@
@Override
public void run(SourceContext<WindowedValue<byte[]>> sourceContext) throws Exception {
- // emit single impulse element
- sourceContext.collect(WindowedValue.valueInGlobalWindow(new byte[0]));
+ if (Iterables.isEmpty(impulseEmitted.get())) {
+ synchronized (sourceContext.getCheckpointLock()) {
+ // emit single impulse element
+ sourceContext.collect(WindowedValue.valueInGlobalWindow(new byte[0]));
+ impulseEmitted.add(true);
+ }
+ }
// Do nothing, but still look busy ...
// we can't return here since Flink requires that all operators stay up,
// otherwise checkpointing would not work correctly anymore
@@ -72,4 +88,15 @@
public void cancel() {
this.running = false;
}
+
+ @Override
+ public void snapshotState(FunctionSnapshotContext context) {}
+
+ @Override
+ public void initializeState(FunctionInitializationContext context) throws Exception {
+ impulseEmitted =
+ context
+ .getOperatorStateStore()
+ .getListState(new ListStateDescriptor<>("impulse-emitted", BooleanSerializer.INSTANCE));
+ }
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
deleted file mode 100644
index 487ef41..0000000
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
+++ /dev/null
@@ -1,248 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.functions;
-
-import java.io.Serializable;
-import java.util.Objects;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
-import org.apache.beam.runners.core.construction.graph.ExecutableStage;
-import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
-import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.sdk.function.ThrowingFunction;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
-import org.apache.flink.annotation.VisibleForTesting;
-import org.apache.flink.api.java.ExecutionEnvironment;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * {@link FlinkExecutableStageContext.Factory} which counts FlinkExecutableStageContext reference
- * for book keeping.
- */
-public class ReferenceCountingFlinkExecutableStageContextFactory
- implements FlinkExecutableStageContext.Factory {
- private static final Logger LOG =
- LoggerFactory.getLogger(ReferenceCountingFlinkExecutableStageContextFactory.class);
- private static final int MAX_RETRY = 3;
-
- private final Creator creator;
- private transient volatile ScheduledExecutorService executor;
- private transient volatile ConcurrentHashMap<String, WrappedContext> keyRegistry;
-
- public static ReferenceCountingFlinkExecutableStageContextFactory create(Creator creator) {
- return new ReferenceCountingFlinkExecutableStageContextFactory(creator);
- }
-
- private ReferenceCountingFlinkExecutableStageContextFactory(Creator creator) {
- this.creator = creator;
- }
-
- @Override
- public FlinkExecutableStageContext get(JobInfo jobInfo) {
- // Retry is needed in case where an existing wrapper is picked from the cache but by
- // the time we accessed wrapper.referenceCount, the wrapper was tombstoned by a pending
- // release task.
- // This race condition is highly unlikely to happen as there is no systematic coding
- // practice which can cause this error because of TTL. However, even in very unlikely case
- // when it happen we have the retry which get a valid context.
- // Note: There is no leak in this logic as the cleanup is only done in release.
- // In case of usage error where release is called before corresponding get finishes,
- // release might throw an error. If release did not throw an error than we can be sure that
- // the state of the system remains valid and appropriate cleanup will be done at TTL.
- for (int retry = 0; retry < MAX_RETRY; retry++) {
- // ConcurrentHashMap will handle the thread safety at the creation time.
- WrappedContext wrapper =
- getCache()
- .computeIfAbsent(
- jobInfo.jobId(),
- jobId -> {
- try {
- return new WrappedContext(jobInfo, creator.apply(jobInfo));
- } catch (Exception e) {
- throw new RuntimeException(
- "Unable to create context for job " + jobInfo.jobId(), e);
- }
- });
- // Take a lock on wrapper before modifying reference count.
- // Use null referenceCount == null as a tombstone for the wrapper.
- synchronized (wrapper) {
- if (wrapper.referenceCount != null) {
- // The wrapper is still valid.
- // Release has not yet got the lock and has not yet removed the wrapper.
- wrapper.referenceCount.incrementAndGet();
- return wrapper;
- }
- }
- }
-
- throw new RuntimeException(
- String.format(
- "Max retry %s exhausted while creating Context for job %s",
- MAX_RETRY, jobInfo.jobId()));
- }
-
- @SuppressWarnings("FutureReturnValueIgnored")
- private void scheduleRelease(JobInfo jobInfo) {
- WrappedContext wrapper = getCache().get(jobInfo.jobId());
- Preconditions.checkState(
- wrapper != null, "Releasing context for unknown job: " + jobInfo.jobId());
-
- PipelineOptions pipelineOptions =
- PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
- int environmentCacheTTLMillis =
- pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
- if (environmentCacheTTLMillis > 0) {
- // Do immediate cleanup if this class is not loaded on Flink parent classloader.
- if (this.getClass().getClassLoader() != ExecutionEnvironment.class.getClassLoader()) {
- LOG.warn(
- "{} is not loaded on parent Flink classloader. "
- + "Falling back to synchronous environment release for job {}.",
- this.getClass(),
- jobInfo.jobId());
- release(wrapper);
- } else {
- // Schedule task to clean the container later.
- // Ensure that this class is loaded in the parent Flink classloader.
- getExecutor()
- .schedule(() -> release(wrapper), environmentCacheTTLMillis, TimeUnit.MILLISECONDS);
- }
- } else {
- // Do not release this asynchronously, as the releasing could fail due to the classloader not
- // being available anymore after the tasks have been removed from the execution engine.
- release(wrapper);
- }
- }
-
- private ConcurrentHashMap<String, WrappedContext> getCache() {
- // Lazily initialize keyRegistry because serialization will set it to null.
- if (keyRegistry != null) {
- return keyRegistry;
- }
- synchronized (this) {
- if (keyRegistry == null) {
- keyRegistry = new ConcurrentHashMap<>();
- }
- return keyRegistry;
- }
- }
-
- private ScheduledExecutorService getExecutor() {
- // Lazily initialize executor because serialization will set it to null.
- if (executor != null) {
- return executor;
- }
- synchronized (this) {
- if (executor == null) {
- executor =
- Executors.newScheduledThreadPool(1, new ThreadFactoryBuilder().setDaemon(true).build());
- }
- return executor;
- }
- }
-
- @VisibleForTesting
- void release(FlinkExecutableStageContext context) {
- @SuppressWarnings({"unchecked", "Not exected to be called from outside."})
- WrappedContext wrapper = (WrappedContext) context;
- synchronized (wrapper) {
- if (wrapper.referenceCount.decrementAndGet() == 0) {
- // Tombstone wrapper.
- wrapper.referenceCount = null;
- if (getCache().remove(wrapper.jobInfo.jobId(), wrapper)) {
- try {
- wrapper.closeActual();
- } catch (Throwable t) {
- LOG.error("Unable to close FlinkExecutableStageContext.", t);
- }
- }
- }
- }
- }
-
- /**
- * {@link WrappedContext} does not expose equals of actual {@link FlinkExecutableStageContext}.
- */
- @VisibleForTesting
- class WrappedContext implements FlinkExecutableStageContext {
- private JobInfo jobInfo;
- private AtomicInteger referenceCount;
- @VisibleForTesting FlinkExecutableStageContext context;
-
- /** {@link WrappedContext#equals(Object)} is only based on {@link JobInfo#jobId()}. */
- WrappedContext(JobInfo jobInfo, FlinkExecutableStageContext context) {
- this.jobInfo = jobInfo;
- this.context = context;
- this.referenceCount = new AtomicInteger(0);
- }
-
- @Override
- public StageBundleFactory getStageBundleFactory(ExecutableStage executableStage) {
- return context.getStageBundleFactory(executableStage);
- }
-
- @Override
- public void close() {
- // Just schedule the context as we want to reuse it if possible.
- scheduleRelease(jobInfo);
- }
-
- private void closeActual() throws Exception {
- context.close();
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (o == null || getClass() != o.getClass()) {
- return false;
- }
- WrappedContext that = (WrappedContext) o;
- return Objects.equals(jobInfo.jobId(), that.jobInfo.jobId());
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(jobInfo);
- }
-
- @Override
- public String toString() {
- return "ContextWrapper{"
- + "jobId='"
- + jobInfo
- + '\''
- + ", referenceCount="
- + referenceCount
- + '}';
- }
- }
-
- /** Interface for creator which extends Serializable. */
- public interface Creator
- extends ThrowingFunction<JobInfo, FlinkExecutableStageContext>, Serializable {}
-}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SideInputInitializer.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SideInputInitializer.java
index e404298..e583e1a 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SideInputInitializer.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SideInputInitializer.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink.translation.functions;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.HashMap;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
index ae36351..283adc5 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/SortingFlinkCombineRunner.java
@@ -30,8 +30,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.flink.util.Collector;
import org.joda.time.Instant;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeInformation.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeInformation.java
index 5132445..c03bef9 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeInformation.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeInformation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink.translation.types;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.sdk.coders.Coder;
import org.apache.flink.api.common.ExecutionConfig;
@@ -61,8 +61,7 @@
@Override
@SuppressWarnings("unchecked")
public Class<T> getTypeClass() {
- // We don't have the Class, so we have to pass null here. What a shame...
- return (Class<T>) Object.class;
+ return (Class<T>) coder.getEncodedTypeDescriptor().getRawType();
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/InspectableByteArrayOutputStream.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/InspectableByteArrayOutputStream.java
deleted file mode 100644
index dec7495..0000000
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/InspectableByteArrayOutputStream.java
+++ /dev/null
@@ -1,32 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.types;
-
-import java.io.ByteArrayOutputStream;
-
-/**
- * Version of {@link java.io.ByteArrayOutputStream} that allows to retrieve the internal byte[]
- * buffer without incurring an array copy.
- */
-public class InspectableByteArrayOutputStream extends ByteArrayOutputStream {
-
- /** Get the underlying byte array. */
- public byte[] getBuffer() {
- return buf;
- }
-}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataInputViewWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataInputViewWrapper.java
index 4903e00..c69b3d1 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataInputViewWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataInputViewWrapper.java
@@ -24,21 +24,17 @@
/**
* Wrapper for {@link DataInputView}. We need this because Flink reads data using a {@link
- * org.apache.flink.core.memory.DataInputView} while Beam {@link
- * org.apache.beam.sdk.coders.Coder}s expect an {@link java.io.InputStream}.
+ * org.apache.flink.core.memory.DataInputView} while Beam {@link org.apache.beam.sdk.coders.Coder}s
+ * expect an {@link java.io.InputStream}.
*/
public class DataInputViewWrapper extends InputStream {
- private DataInputView inputView;
+ private final DataInputView inputView;
public DataInputViewWrapper(DataInputView inputView) {
this.inputView = inputView;
}
- public void setInputView(DataInputView inputView) {
- this.inputView = inputView;
- }
-
@Override
public int read() throws IOException {
try {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataOutputViewWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataOutputViewWrapper.java
index 061861e..6e8ae1d 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataOutputViewWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/DataOutputViewWrapper.java
@@ -28,16 +28,12 @@
*/
public class DataOutputViewWrapper extends OutputStream {
- private DataOutputView outputView;
+ private final DataOutputView outputView;
public DataOutputViewWrapper(DataOutputView outputView) {
this.outputView = outputView;
}
- public void setOutputView(DataOutputView outputView) {
- this.outputView = outputView;
- }
-
@Override
public void write(int b) throws IOException {
outputView.write(b);
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/ImpulseInputFormat.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/ImpulseInputFormat.java
index 11fbc62..8db4df8 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/ImpulseInputFormat.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/ImpulseInputFormat.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink.translation.wrappers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.flink.api.common.io.DefaultInputSplitAssigner;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 657822c..4f48287 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -23,6 +23,7 @@
import java.io.InputStream;
import java.io.OutputStream;
import java.io.Serializable;
+import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
@@ -55,6 +56,7 @@
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
+import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
import org.apache.beam.runners.flink.translation.utils.FlinkClassloading;
import org.apache.beam.runners.flink.translation.utils.NoopLock;
@@ -66,12 +68,14 @@
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.state.StateSpec;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
@@ -79,10 +83,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
@@ -119,7 +123,7 @@
public class DoFnOperator<InputT, OutputT> extends AbstractStreamOperator<WindowedValue<OutputT>>
implements OneInputStreamOperator<WindowedValue<InputT>, WindowedValue<OutputT>>,
TwoInputStreamOperator<WindowedValue<InputT>, RawUnionValue, WindowedValue<OutputT>>,
- Triggerable<Object, TimerData> {
+ Triggerable<ByteBuffer, TimerData> {
protected DoFn<InputT, OutputT> doFn;
@@ -176,6 +180,8 @@
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
+
/** If true, we must process elements only after a checkpoint is finished. */
private final boolean requiresStableInput;
@@ -187,6 +193,9 @@
private transient PushedBackElementsHandler<WindowedValue<InputT>> pushedBackElementsHandler;
+ /** Metrics container for reporting Beam metrics to Flink (null if metrics are disabled). */
+ @Nullable transient FlinkMetricContainer flinkMetricContainer;
+
/** Use an AtomicBoolean because we start/stop bundles by a timer thread (see below). */
private transient AtomicBoolean bundleStarted;
/** Number of processed elements in the current bundle. */
@@ -214,7 +223,8 @@
PipelineOptions options,
Coder<?> keyCoder,
KeySelector<WindowedValue<InputT>, ?> keySelector,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.doFn = doFn;
this.stepName = stepName;
this.windowedInputCoder = inputWindowedCoder;
@@ -243,6 +253,7 @@
this.maxBundleTimeMills = flinkOptions.getMaxBundleTimeMills();
Preconditions.checkArgument(maxBundleTimeMills > 0, "Bundle time must be at least 1");
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
this.requiresStableInput =
// WindowDoFnOperator does not use a DoFn
@@ -251,7 +262,8 @@
if (requiresStableInput) {
Preconditions.checkState(
- flinkOptions.getCheckpointingMode() == CheckpointingMode.EXACTLY_ONCE,
+ CheckpointingMode.valueOf(flinkOptions.getCheckpointingMode())
+ == CheckpointingMode.EXACTLY_ONCE,
"Checkpointing mode is not set to exactly once but @RequiresStableInput is used.");
Preconditions.checkState(
flinkOptions.getCheckpointingInterval() > 0,
@@ -304,8 +316,7 @@
Output<StreamRecord<WindowedValue<OutputT>>> output) {
// make sure that FileSystems is initialized correctly
- FlinkPipelineOptions options = serializedOptions.get().as(FlinkPipelineOptions.class);
- FileSystems.setDefaultPipelineOptions(options);
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
super.setup(containingTask, config, output);
}
@@ -356,6 +367,17 @@
keyedStateInternals =
new FlinkStateInternals<>((KeyedStateBackend) getKeyedStateBackend(), keyCoder);
+ if (doFn != null) {
+ DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
+ FlinkStateInternals.EarlyBinder earlyBinder =
+ new FlinkStateInternals.EarlyBinder(getKeyedStateBackend());
+ for (DoFnSignature.StateDeclaration value : signature.stateDeclarations().values()) {
+ StateSpec<?> spec =
+ (StateSpec<?>) signature.stateDeclarations().get(value.id()).field().get(doFn);
+ spec.bind(value.id(), earlyBinder);
+ }
+ }
+
if (timerService == null) {
timerService =
getInternalTimerService("beam-timer", new CoderTypeSerializer<>(timerCoder), this);
@@ -403,7 +425,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
if (requiresStableInput) {
// put this in front of the root FnRunner before any additional wrappers
@@ -420,7 +443,8 @@
doFnRunner = createWrappingDoFnRunner(doFnRunner);
if (options.getEnableMetrics()) {
- doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, getRuntimeContext());
+ flinkMetricContainer = new FlinkMetricContainer(getRuntimeContext());
+ doFnRunner = new DoFnRunnerWithMetricsUpdate<>(stepName, doFnRunner, flinkMetricContainer);
}
bundleStarted = new AtomicBoolean(false);
@@ -765,20 +789,20 @@
}
@Override
- public void onEventTime(InternalTimer<Object, TimerData> timer) throws Exception {
+ public void onEventTime(InternalTimer<ByteBuffer, TimerData> timer) throws Exception {
// We don't have to cal checkInvokeStartBundle() because it's already called in
// processWatermark*().
fireTimer(timer);
}
@Override
- public void onProcessingTime(InternalTimer<Object, TimerData> timer) throws Exception {
+ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) throws Exception {
checkInvokeStartBundle();
fireTimer(timer);
}
// allow overriding this in WindowDoFnOperator
- public void fireTimer(InternalTimer<?, TimerData> timer) {
+ protected void fireTimer(InternalTimer<ByteBuffer, TimerData> timer) {
TimerInternals.TimerData timerData = timer.getNamespace();
StateNamespace namespace = timerData.getNamespace();
// This is a user timer, so namespace must be WindowNamespace
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
index a29d49d..e3fe07c 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
@@ -31,6 +31,8 @@
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
@@ -38,6 +40,7 @@
import java.util.function.Consumer;
import java.util.function.Supplier;
import java.util.stream.Collectors;
+import javax.annotation.Nullable;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateKey.TypeCase;
@@ -54,10 +57,11 @@
import org.apache.beam.runners.core.construction.Timer;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.UserStateReference;
-import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
-import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContext;
+import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.functions.FlinkStreamingSideInputHandlerFactory;
+import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.TimerSpec;
@@ -68,6 +72,7 @@
import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.state.BagState;
@@ -82,7 +87,12 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf.ByteStringCoder;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.runtime.state.KeyedStateBackend;
import org.apache.flink.streaming.api.operators.InternalTimer;
@@ -106,19 +116,18 @@
private final RunnerApi.ExecutableStagePayload payload;
private final JobInfo jobInfo;
- private final FlinkExecutableStageContext.Factory contextFactory;
+ private final FlinkExecutableStageContextFactory contextFactory;
private final Map<String, TupleTag<?>> outputMap;
private final Map<RunnerApi.ExecutableStagePayload.SideInputId, PCollectionView<?>> sideInputIds;
/** A lock which has to be acquired when concurrently accessing state and timers. */
private final ReentrantLock stateBackendLock;
- private transient FlinkExecutableStageContext stageContext;
+ private transient ExecutableStageContext stageContext;
private transient StateRequestHandler stateRequestHandler;
private transient BundleProgressHandler progressHandler;
private transient StageBundleFactory stageBundleFactory;
private transient ExecutableStage executableStage;
private transient SdkHarnessDoFnRunner<InputT, OutputT> sdkHarnessRunner;
- private transient FlinkMetricContainer flinkMetricContainer;
private transient long backupWatermarkHold = Long.MIN_VALUE;
/** Constructor. */
@@ -136,7 +145,7 @@
PipelineOptions options,
RunnerApi.ExecutableStagePayload payload,
JobInfo jobInfo,
- FlinkExecutableStageContext.Factory contextFactory,
+ FlinkExecutableStageContextFactory contextFactory,
Map<String, TupleTag<?>> outputMap,
WindowingStrategy windowingStrategy,
Coder keyCoder,
@@ -156,7 +165,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
this.payload = payload;
this.jobInfo = jobInfo;
this.contextFactory = contextFactory;
@@ -173,13 +183,13 @@
@Override
public void open() throws Exception {
executableStage = ExecutableStage.fromPayload(payload);
+ initializeUserState(executableStage, getKeyedStateBackend());
// TODO: Wire this into the distributed cache and make it pluggable.
// TODO: Do we really want this layer of indirection when accessing the stage bundle factory?
// It's a little strange because this operator is responsible for the lifetime of the stage
// bundle "factory" (manager?) but not the job or Flink bundle factories. How do we make
// ownership of the higher level "factories" explicit? Do we care?
stageContext = contextFactory.get(jobInfo);
- flinkMetricContainer = new FlinkMetricContainer(getRuntimeContext());
stageBundleFactory = stageContext.getStageBundleFactory(executableStage);
stateRequestHandler = getStateRequestHandler(executableStage);
@@ -187,12 +197,16 @@
new BundleProgressHandler() {
@Override
public void onProgress(ProcessBundleProgressResponse progress) {
- flinkMetricContainer.updateMetrics(stepName, progress.getMonitoringInfosList());
+ if (flinkMetricContainer != null) {
+ flinkMetricContainer.updateMetrics(stepName, progress.getMonitoringInfosList());
+ }
}
@Override
public void onCompleted(ProcessBundleResponse response) {
- flinkMetricContainer.updateMetrics(stepName, response.getMonitoringInfosList());
+ if (flinkMetricContainer != null) {
+ flinkMetricContainer.updateMetrics(stepName, response.getMonitoringInfosList());
+ }
}
};
@@ -229,7 +243,10 @@
StateRequestHandlers.forBagUserStateHandlerFactory(
stageBundleFactory.getProcessBundleDescriptor(),
new BagUserStateFactory(
- keyedStateInternals, getKeyedStateBackend(), stateBackendLock));
+ () -> UUID.randomUUID().toString(),
+ keyedStateInternals,
+ getKeyedStateBackend(),
+ stateBackendLock));
} else {
userStateRequestHandler = StateRequestHandler.unsupported();
}
@@ -241,32 +258,37 @@
return StateRequestHandlers.delegateBasedUponType(handlerMap);
}
- private static class BagUserStateFactory
- implements StateRequestHandlers.BagUserStateHandlerFactory {
+ static class BagUserStateFactory<K extends ByteString, V, W extends BoundedWindow>
+ implements StateRequestHandlers.BagUserStateHandlerFactory<K, V, W> {
private final StateInternals stateInternals;
private final KeyedStateBackend<ByteBuffer> keyedStateBackend;
private final Lock stateBackendLock;
+ /** Holds the valid cache token for user state for this operator. */
+ private final ByteString cacheToken;
- private BagUserStateFactory(
+ BagUserStateFactory(
+ IdGenerator cacheTokenGenerator,
StateInternals stateInternals,
KeyedStateBackend<ByteBuffer> keyedStateBackend,
Lock stateBackendLock) {
-
this.stateInternals = stateInternals;
this.keyedStateBackend = keyedStateBackend;
this.stateBackendLock = stateBackendLock;
+ this.cacheToken = ByteString.copyFrom(cacheTokenGenerator.getId().getBytes(Charsets.UTF_8));
}
@Override
- public <K, V, W extends BoundedWindow>
- StateRequestHandlers.BagUserStateHandler<K, V, W> forUserState(
- String pTransformId,
- String userStateId,
- Coder<K> keyCoder,
- Coder<V> valueCoder,
- Coder<W> windowCoder) {
+ public StateRequestHandlers.BagUserStateHandler<K, V, W> forUserState(
+ // Transform id not used because multiple operators with state will not
+ // be fused together. See GreedyPCollectionFusers
+ String pTransformId,
+ String userStateId,
+ Coder<K> keyCoder,
+ Coder<V> valueCoder,
+ Coder<W> windowCoder) {
return new StateRequestHandlers.BagUserStateHandler<K, V, W>() {
+
@Override
public Iterable<V> get(K key, W window) {
try {
@@ -283,6 +305,7 @@
}
BagState<V> bagState =
stateInternals.state(namespace, StateTags.bag(userStateId, valueCoder));
+
return bagState.read();
} finally {
stateBackendLock.unlock();
@@ -335,11 +358,15 @@
}
}
+ @Override
+ public Optional<ByteString> getCacheToken() {
+ // Cache tokens remains valid for the life time of the operator
+ return Optional.of(cacheToken);
+ }
+
private void prepareStateBackend(K key) {
- // Key for state request is shipped already encoded as ByteString,
- // this is mostly a wrapping with ByteBuffer. We still follow the
- // usual key encoding procedure.
- final ByteBuffer encodedKey = FlinkKeyUtils.encodeKey(key, keyCoder);
+ // Key for state request is shipped encoded with NESTED context.
+ ByteBuffer encodedKey = FlinkKeyUtils.fromEncodedKey(key);
keyedStateBackend.setCurrentKey(encodedKey);
}
};
@@ -402,8 +429,8 @@
}
@Override
- public void fireTimer(InternalTimer<?, TimerInternals.TimerData> timer) {
- final ByteBuffer encodedKey = (ByteBuffer) timer.getKey();
+ protected void fireTimer(InternalTimer<ByteBuffer, TimerInternals.TimerData> timer) {
+ final ByteBuffer encodedKey = timer.getKey();
// We have to synchronize to ensure the state backend is not concurrently accessed by the state
// requests
try {
@@ -834,6 +861,26 @@
}
}
+ /**
+ * Eagerly create the user state to work around https://jira.apache.org/jira/browse/FLINK-12653.
+ */
+ private static void initializeUserState(
+ ExecutableStage executableStage, @Nullable KeyedStateBackend keyedStateBackend) {
+ executableStage
+ .getUserStates()
+ .forEach(
+ ref -> {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new ListStateDescriptor<>(
+ ref.localName(), new CoderTypeSerializer<>(ByteStringCoder.of())));
+ } catch (Exception e) {
+ throw new RuntimeException("Couldn't initialize user states.", e);
+ }
+ });
+ }
+
private static class NoOpDoFn<InputT, OutputT> extends DoFn<InputT, OutputT> {
@ProcessElement
public void doNothing(ProcessContext context) {}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java
index eac1e7c..ccd10d4 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtils.java
@@ -17,14 +17,22 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
import java.util.Locale;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
/**
* Utility functions for dealing with key encoding. Beam requires keys to be compared in binary
@@ -37,7 +45,7 @@
checkNotNull(keyCoder, "Provided coder must not be null");
final byte[] keyBytes;
try {
- keyBytes = CoderUtils.encodeToByteArray(keyCoder, key);
+ keyBytes = CoderUtils.encodeToByteArray(keyCoder, key, Coder.Context.NESTED);
} catch (Exception e) {
throw new RuntimeException(String.format(Locale.ENGLISH, "Failed to encode key: %s", key), e);
}
@@ -45,14 +53,14 @@
}
/** Decodes a key from a ByteBuffer containing a byte array. */
- static <K> K decodeKey(ByteBuffer byteBuffer, Coder<K> keyCoder) {
+ public static <K> K decodeKey(ByteBuffer byteBuffer, Coder<K> keyCoder) {
checkNotNull(byteBuffer, "Provided ByteBuffer must not be null");
checkNotNull(keyCoder, "Provided coder must not be null");
checkState(byteBuffer.hasArray(), "ByteBuffer key must contain an array.");
@SuppressWarnings("ByteBufferBackingArray")
final byte[] keyBytes = byteBuffer.array();
try {
- return CoderUtils.decodeFromByteArray(keyCoder, keyBytes);
+ return CoderUtils.decodeFromByteArray(keyCoder, keyBytes, Coder.Context.NESTED);
} catch (Exception e) {
throw new RuntimeException(
String.format(
@@ -60,4 +68,41 @@
e);
}
}
+
+ static ByteBuffer fromEncodedKey(ByteString encodedKey) {
+ return ByteBuffer.wrap(encodedKey.toByteArray());
+ }
+
+ /** The Coder for the Runner's encoded representation of a key. */
+ static class ByteBufferCoder extends StructuredCoder<ByteBuffer> {
+
+ public static ByteBufferCoder of() {
+ return INSTANCE;
+ }
+
+ private static final ByteBufferCoder INSTANCE = new ByteBufferCoder();
+
+ private ByteBufferCoder() {}
+
+ @Override
+ public void encode(ByteBuffer value, OutputStream outStream) throws IOException {
+ @SuppressWarnings("ByteBufferBackingArray")
+ byte[] array = value.array();
+ ByteArrayCoder.of().encode(array, outStream);
+ }
+
+ @Override
+ public ByteBuffer decode(InputStream inStream) throws IOException {
+ byte[] decode = ByteArrayCoder.of().decode(inStream);
+ return ByteBuffer.wrap(decode);
+ }
+
+ @Override
+ public List<? extends Coder<?>> getCoderArguments() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void verifyDeterministic() {}
+ }
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KeyedPushedBackElementsHandler.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KeyedPushedBackElementsHandler.java
index b5230b7..82229d9 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KeyedPushedBackElementsHandler.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KeyedPushedBackElementsHandler.java
@@ -38,37 +38,38 @@
static <K, T> KeyedPushedBackElementsHandler<K, T> create(
KeySelector<T, K> keySelector,
KeyedStateBackend<K> backend,
- ListStateDescriptor<T> stateDescriptor) {
+ ListStateDescriptor<T> stateDescriptor)
+ throws Exception {
return new KeyedPushedBackElementsHandler<>(keySelector, backend, stateDescriptor);
}
private final KeySelector<T, K> keySelector;
private final KeyedStateBackend<K> backend;
- private final ListStateDescriptor<T> stateDescriptor;
+ private final String stateName;
+ private final ListState<T> state;
private KeyedPushedBackElementsHandler(
KeySelector<T, K> keySelector,
KeyedStateBackend<K> backend,
- ListStateDescriptor<T> stateDescriptor) {
+ ListStateDescriptor<T> stateDescriptor)
+ throws Exception {
this.keySelector = keySelector;
this.backend = backend;
- this.stateDescriptor = stateDescriptor;
+ this.stateName = stateDescriptor.getName();
+ // Eagerly retrieve the state to work around https://jira.apache.org/jira/browse/FLINK-12653
+ this.state =
+ backend.getPartitionedState(
+ VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
}
@Override
public Stream<T> getElements() {
-
return backend
- .getKeys(stateDescriptor.getName(), VoidNamespace.INSTANCE)
+ .getKeys(stateName, VoidNamespace.INSTANCE)
.flatMap(
key -> {
try {
backend.setCurrentKey(key);
-
- ListState<T> state =
- backend.getPartitionedState(
- VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
-
return StreamSupport.stream(state.get().spliterator(), false);
} catch (Exception e) {
throw new RuntimeException("Error reading keyed state.", e);
@@ -80,29 +81,16 @@
public void clear() throws Exception {
// TODO we have to collect all keys because otherwise we get ConcurrentModificationExceptions
// from flink. We can change this once it's fixed in Flink
-
- List<K> keys =
- backend
- .getKeys(stateDescriptor.getName(), VoidNamespace.INSTANCE)
- .collect(Collectors.toList());
+ List<K> keys = backend.getKeys(stateName, VoidNamespace.INSTANCE).collect(Collectors.toList());
for (K key : keys) {
backend.setCurrentKey(key);
-
- ListState<T> state =
- backend.getPartitionedState(
- VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
-
state.clear();
}
}
@Override
public void pushBack(T element) throws Exception {
- ListState<T> state =
- backend.getPartitionedState(
- VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
-
backend.setCurrentKey(keySelector.getKey(element));
state.add(element);
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KvToByteBufferKeySelector.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KvToByteBufferKeySelector.java
index ea181d2..3e5e22c 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KvToByteBufferKeySelector.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/KvToByteBufferKeySelector.java
@@ -18,12 +18,12 @@
package org.apache.beam.runners.flink.translation.wrappers.streaming;
import java.nio.ByteBuffer;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.functions.KeySelector;
-import org.apache.flink.api.java.typeutils.GenericTypeInfo;
import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
/**
@@ -41,13 +41,13 @@
}
@Override
- public ByteBuffer getKey(WindowedValue<KV<K, V>> value) throws Exception {
+ public ByteBuffer getKey(WindowedValue<KV<K, V>> value) {
K key = value.getValue().getKey();
return FlinkKeyUtils.encodeKey(key, keyCoder);
}
@Override
public TypeInformation<ByteBuffer> getProducedType() {
- return new GenericTypeInfo<>(ByteBuffer.class);
+ return new CoderTypeInformation<>(FlinkKeyUtils.ByteBufferCoder.of());
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/NonKeyedPushedBackElementsHandler.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/NonKeyedPushedBackElementsHandler.java
index fbb1dc398..1686f50 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/NonKeyedPushedBackElementsHandler.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/NonKeyedPushedBackElementsHandler.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SingletonKeyedWorkItemCoder.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SingletonKeyedWorkItemCoder.java
index 7727932..23c67f7 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SingletonKeyedWorkItemCoder.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SingletonKeyedWorkItemCoder.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Singleton keyed work item coder. */
public class SingletonKeyedWorkItemCoder<K, ElemT>
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
index ff59500..4f8e0d7 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/SplittableDoFnOperator.java
@@ -17,8 +17,9 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
@@ -93,7 +94,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
@Override
@@ -156,7 +158,7 @@
}
@Override
- public void fireTimer(InternalTimer<?, TimerInternals.TimerData> timer) {
+ protected void fireTimer(InternalTimer<ByteBuffer, TimerInternals.TimerData> timer) {
timerInternals.cleanupPendingTimer(timer.getNamespace());
if (timer.getNamespace().getDomain().equals(TimeDomain.EVENT_TIME)) {
// ignore this, it can only be a state cleanup timers from StatefulDoFnRunner and ProcessFn
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
index 027a693..e2058bb 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperator.java
@@ -19,6 +19,7 @@
import static org.apache.beam.runners.core.TimerInternals.TimerData;
+import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
@@ -79,7 +80,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
this.systemReduceFn = systemReduceFn;
}
@@ -123,7 +125,7 @@
}
@Override
- public void fireTimer(InternalTimer<?, TimerData> timer) {
+ protected void fireTimer(InternalTimer<ByteBuffer, TimerData> timer) {
timerInternals.cleanupPendingTimer(timer.getNamespace());
doFnRunner.processElement(
WindowedValue.valueInGlobalWindow(
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java
index 60d937e..5677e1a 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/DedupingOperator.java
@@ -18,6 +18,9 @@
package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
import java.nio.ByteBuffer;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.flink.api.common.state.ValueState;
@@ -40,6 +43,7 @@
Triggerable<ByteBuffer, VoidNamespace> {
private static final long MAX_RETENTION_SINCE_ACCESS = Duration.standardMinutes(10L).getMillis();
+ private final SerializablePipelineOptions options;
// we keep the time when we last saw an element id for cleanup
private ValueStateDescriptor<Long> dedupingStateDescriptor =
@@ -47,6 +51,10 @@
private transient InternalTimerService<VoidNamespace> timerService;
+ public DedupingOperator(PipelineOptions options) {
+ this.options = new SerializablePipelineOptions(options);
+ }
+
@Override
public void initializeState(StateInitializationContext context) throws Exception {
super.initializeState(context);
@@ -56,6 +64,13 @@
}
@Override
+ public void open() {
+ // Initialize FileSystems for any coders which may want to use the FileSystem,
+ // see https://issues.apache.org/jira/browse/BEAM-8303
+ FileSystems.setDefaultPipelineOptions(options.get());
+ }
+
+ @Override
public void processElement(StreamRecord<WindowedValue<ValueWithRecordId<T>>> streamRecord)
throws Exception {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
index 2efd665..116cca3 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/StreamingImpulseSource.java
@@ -17,27 +17,29 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
-import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * A streaming source that periodically produces an empty byte array. This is mostly useful for
- * debugging, or for triggering periodic behavior in a portable pipeline.
+ * A streaming source that periodically produces a byte array. This is mostly useful for debugging,
+ * or for triggering periodic behavior in a portable pipeline.
*
* @deprecated Legacy non-portable source which can be replaced by a DoFn with timers.
+ * https://jira.apache.org/jira/browse/BEAM-8353
*/
@Deprecated
public class StreamingImpulseSource extends RichParallelSourceFunction<WindowedValue<byte[]>> {
private static final Logger LOG = LoggerFactory.getLogger(StreamingImpulseSource.class);
- private final AtomicBoolean cancelled = new AtomicBoolean(false);
- private long count = 0;
private final int intervalMillis;
private final int messageCount;
+ private volatile boolean running = true;
+ private long count;
+
public StreamingImpulseSource(int intervalMillis, int messageCount) {
this.intervalMillis = intervalMillis;
this.messageCount = messageCount;
@@ -55,9 +57,10 @@
subtaskCount++;
}
- while (!cancelled.get() && (messageCount == 0 || count < subtaskCount)) {
+ while (running && (messageCount == 0 || count < subtaskCount)) {
synchronized (ctx.getCheckpointLock()) {
- ctx.collect(WindowedValue.valueInGlobalWindow(new byte[] {}));
+ ctx.collect(
+ WindowedValue.valueInGlobalWindow(String.valueOf(count).getBytes(Charsets.UTF_8)));
count++;
}
@@ -73,6 +76,6 @@
@Override
public void cancel() {
- this.cancelled.set(true);
+ this.running = false;
}
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
index 9f171bc..28cc507 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
@@ -22,6 +22,7 @@
import java.util.LinkedHashMap;
import java.util.List;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.FlinkMetricContainer;
import org.apache.beam.runners.flink.metrics.ReaderInvocationUtil;
@@ -30,6 +31,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -39,7 +41,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.ValueWithRecordId;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.flink.api.common.ExecutionConfig;
import org.apache.flink.api.common.functions.StoppableFunction;
import org.apache.flink.api.common.state.ListState;
@@ -70,6 +72,13 @@
/** Keep the options so that we can initialize the localReaders. */
private final SerializablePipelineOptions serializedOptions;
+ /**
+ * We are processing bounded data and should read from the sources sequentially instead of reading
+ * round-robin from all the sources. In case of file sources this avoids having too many open
+ * files/connections at once.
+ */
+ private final boolean isConvertedBoundedSource;
+
/** For snapshot and restore. */
private final KvCoder<? extends UnboundedSource<OutputT, CheckpointMarkT>, CheckpointMarkT>
checkpointCoder;
@@ -134,6 +143,8 @@
throws Exception {
this.stepName = stepName;
this.serializedOptions = new SerializablePipelineOptions(pipelineOptions);
+ this.isConvertedBoundedSource =
+ source instanceof UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
if (source.requiresDeduping()) {
LOG.warn("Source {} requires deduping but Flink runner doesn't support this yet.", source);
@@ -162,6 +173,7 @@
/** Initialize and restore state before starting execution of the source. */
@Override
public void open(Configuration parameters) throws Exception {
+ FileSystems.setDefaultPipelineOptions(serializedOptions.get());
runtimeContext = (StreamingRuntimeContext) getRuntimeContext();
// figure out which split sources we're responsible for
@@ -217,34 +229,33 @@
// through to idle this executor.
LOG.info("Number of readers is 0 for this task executor, idle");
// Do nothing here but still execute the rest of the source logic
- } else if (localReaders.size() == 1) {
- // the easy case, we just read from one reader
- UnboundedSource.UnboundedReader<OutputT> reader = localReaders.get(0);
-
- synchronized (ctx.getCheckpointLock()) {
- boolean dataAvailable = readerInvoker.invokeStart(reader);
- if (dataAvailable) {
- emitElement(ctx, reader);
- }
- }
-
+ } else if (isConvertedBoundedSource) {
setNextWatermarkTimer(this.runtimeContext);
- while (isRunning) {
- boolean dataAvailable;
- synchronized (ctx.getCheckpointLock()) {
- dataAvailable = readerInvoker.invokeAdvance(reader);
+ // We read sequentially from all bounded sources
+ for (int i = 0; i < localReaders.size() && isRunning; i++) {
+ UnboundedSource.UnboundedReader<OutputT> reader = localReaders.get(i);
+ synchronized (ctx.getCheckpointLock()) {
+ boolean dataAvailable = readerInvoker.invokeStart(reader);
if (dataAvailable) {
emitElement(ctx, reader);
}
}
- if (!dataAvailable) {
- Thread.sleep(50);
- }
+
+ boolean dataAvailable;
+ do {
+ synchronized (ctx.getCheckpointLock()) {
+ dataAvailable = readerInvoker.invokeAdvance(reader);
+
+ if (dataAvailable) {
+ emitElement(ctx, reader);
+ }
+ }
+ } while (dataAvailable && isRunning);
}
} else {
- // a bit more complicated, we are responsible for several localReaders
+ // Read from multiple unbounded sources,
// loop through them and sleep if none of them had any data
int numReaders = localReaders.size();
@@ -329,7 +340,7 @@
timestamp,
GlobalWindow.INSTANCE,
PaneInfo.NO_FIRING);
- ctx.collectWithTimestamp(windowedValue, timestamp.getMillis());
+ ctx.collect(windowedValue);
}
@Override
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/KeyedBufferingElementsHandler.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/KeyedBufferingElementsHandler.java
index 5aba6a9..5dd010d 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/KeyedBufferingElementsHandler.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/KeyedBufferingElementsHandler.java
@@ -31,27 +31,29 @@
public class KeyedBufferingElementsHandler implements BufferingElementsHandler {
static KeyedBufferingElementsHandler create(
- KeyedStateBackend backend, ListStateDescriptor<BufferedElement> stateDescriptor) {
+ KeyedStateBackend backend, ListStateDescriptor<BufferedElement> stateDescriptor)
+ throws Exception {
return new KeyedBufferingElementsHandler(backend, stateDescriptor);
}
- private final KeyedStateBackend backend;
- private final ListStateDescriptor<BufferedElement> stateDescriptor;
+ private final KeyedStateBackend<Object> backend;
+ private final String stateName;
+ private final ListState<BufferedElement> state;
private KeyedBufferingElementsHandler(
- KeyedStateBackend backend, ListStateDescriptor<BufferedElement> stateDescriptor) {
+ KeyedStateBackend<Object> backend, ListStateDescriptor<BufferedElement> stateDescriptor)
+ throws Exception {
this.backend = backend;
- this.stateDescriptor = stateDescriptor;
+ this.stateName = stateDescriptor.getName();
+ // Eagerly retrieve the state to work around https://jira.apache.org/jira/browse/FLINK-12653
+ this.state =
+ backend.getPartitionedState(
+ VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
}
@Override
public void buffer(BufferedElement element) {
try {
- ListState<BufferedElement> state =
- (ListState<BufferedElement>)
- backend.getPartitionedState(
- VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
-
// assumes state backend is already keyed
state.add(element);
} catch (Exception e) {
@@ -62,19 +64,11 @@
@Override
public Stream<BufferedElement> getElements() {
return backend
- .getKeys(stateDescriptor.getName(), VoidNamespace.INSTANCE)
+ .getKeys(stateName, VoidNamespace.INSTANCE)
.flatMap(
key -> {
try {
backend.setCurrentKey(key);
-
- ListState<BufferedElement> state =
- (ListState<BufferedElement>)
- backend.getPartitionedState(
- VoidNamespace.INSTANCE,
- VoidNamespaceSerializer.INSTANCE,
- stateDescriptor);
-
return StreamSupport.stream(state.get().spliterator(), false);
} catch (Exception e) {
throw new RuntimeException(
@@ -85,21 +79,10 @@
@Override
public void clear() {
- List keys =
- (List)
- backend
- .getKeys(stateDescriptor.getName(), VoidNamespace.INSTANCE)
- .collect(Collectors.toList());
-
+ List keys = backend.getKeys(stateName, VoidNamespace.INSTANCE).collect(Collectors.toList());
try {
for (Object key : keys) {
backend.setCurrentKey(key);
-
- ListState<BufferedElement> state =
- (ListState<BufferedElement>)
- backend.getPartitionedState(
- VoidNamespace.INSTANCE, VoidNamespaceSerializer.INSTANCE, stateDescriptor);
-
state.clear();
}
} catch (Exception e) {
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/NonKeyedBufferingElementsHandler.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/NonKeyedBufferingElementsHandler.java
index c2f939c..f5f195e 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/NonKeyedBufferingElementsHandler.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/NonKeyedBufferingElementsHandler.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.flink.translation.wrappers.streaming.stableinput;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
index 4407d33..8d979fd 100644
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
+++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
@@ -28,8 +28,8 @@
import org.apache.beam.runners.core.StateNamespace;
import org.apache.beam.runners.core.StateTag;
import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
+import org.apache.beam.runners.flink.translation.wrappers.streaming.FlinkKeyUtils;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.InstantCoder;
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.state.BagState;
@@ -48,18 +48,18 @@
import org.apache.beam.sdk.transforms.CombineWithContext;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.CombineContextFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.flink.api.common.state.ListState;
import org.apache.flink.api.common.state.ListStateDescriptor;
import org.apache.flink.api.common.state.MapStateDescriptor;
import org.apache.flink.api.common.state.ValueStateDescriptor;
import org.apache.flink.api.common.typeutils.base.BooleanSerializer;
import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.api.common.typeutils.base.VoidSerializer;
import org.apache.flink.runtime.state.KeyedStateBackend;
import org.apache.flink.runtime.state.VoidNamespace;
import org.apache.flink.runtime.state.VoidNamespaceSerializer;
@@ -104,14 +104,7 @@
@Override
public K getKey() {
ByteBuffer keyBytes = flinkStateBackend.getCurrentKey();
- byte[] bytes = new byte[keyBytes.remaining()];
- keyBytes.get(bytes);
- keyBytes.position(keyBytes.position() - bytes.length);
- try {
- return CoderUtils.decodeFromByteArray(keyCoder, bytes);
- } catch (CoderException e) {
- throw new RuntimeException("Error decoding key.", e);
- }
+ return FlinkKeyUtils.decodeKey(keyBytes, keyCoder);
}
@Override
@@ -1233,4 +1226,121 @@
}
}
}
+
+ /** Eagerly create user state to work around https://jira.apache.org/jira/browse/FLINK-12653. */
+ public static class EarlyBinder implements StateBinder {
+
+ private final KeyedStateBackend keyedStateBackend;
+
+ public EarlyBinder(KeyedStateBackend keyedStateBackend) {
+ this.keyedStateBackend = keyedStateBackend;
+ }
+
+ @Override
+ public <T> ValueState<T> bindValue(String id, StateSpec<ValueState<T>> spec, Coder<T> coder) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new ValueStateDescriptor<>(id, new CoderTypeSerializer<>(coder)));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+
+ return null;
+ }
+
+ @Override
+ public <T> BagState<T> bindBag(String id, StateSpec<BagState<T>> spec, Coder<T> elemCoder) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new ListStateDescriptor<>(id, new CoderTypeSerializer<>(elemCoder)));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+
+ return null;
+ }
+
+ @Override
+ public <T> SetState<T> bindSet(String id, StateSpec<SetState<T>> spec, Coder<T> elemCoder) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new MapStateDescriptor<>(
+ id, new CoderTypeSerializer<>(elemCoder), VoidSerializer.INSTANCE));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ @Override
+ public <KeyT, ValueT> org.apache.beam.sdk.state.MapState<KeyT, ValueT> bindMap(
+ String id,
+ StateSpec<org.apache.beam.sdk.state.MapState<KeyT, ValueT>> spec,
+ Coder<KeyT> mapKeyCoder,
+ Coder<ValueT> mapValueCoder) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new MapStateDescriptor<>(
+ id,
+ new CoderTypeSerializer<>(mapKeyCoder),
+ new CoderTypeSerializer<>(mapValueCoder)));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ @Override
+ public <InputT, AccumT, OutputT> CombiningState<InputT, AccumT, OutputT> bindCombining(
+ String id,
+ StateSpec<CombiningState<InputT, AccumT, OutputT>> spec,
+ Coder<AccumT> accumCoder,
+ Combine.CombineFn<InputT, AccumT, OutputT> combineFn) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new ValueStateDescriptor<>(id, new CoderTypeSerializer<>(accumCoder)));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ @Override
+ public <InputT, AccumT, OutputT>
+ CombiningState<InputT, AccumT, OutputT> bindCombiningWithContext(
+ String id,
+ StateSpec<CombiningState<InputT, AccumT, OutputT>> spec,
+ Coder<AccumT> accumCoder,
+ CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT> combineFn) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ StringSerializer.INSTANCE,
+ new ValueStateDescriptor<>(id, new CoderTypeSerializer<>(accumCoder)));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ @Override
+ public WatermarkHoldState bindWatermark(
+ String id, StateSpec<WatermarkHoldState> spec, TimestampCombiner timestampCombiner) {
+ try {
+ keyedStateBackend.getOrCreateKeyedState(
+ VoidNamespaceSerializer.INSTANCE,
+ new MapStateDescriptor<>(
+ "watermark-holds",
+ StringSerializer.INSTANCE,
+ new CoderTypeSerializer<>(InstantCoder.of())));
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+ }
}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupCheckpointedOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupCheckpointedOperator.java
deleted file mode 100644
index 2a1ad9b..0000000
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupCheckpointedOperator.java
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.wrappers.streaming.state;
-
-import java.io.DataOutputStream;
-
-/** This interface is used to checkpoint key-groups state. */
-public interface KeyGroupCheckpointedOperator extends KeyGroupRestoringOperator {
- /**
- * Snapshots the state for a given {@code keyGroupIdx}.
- *
- * <p>AbstractStreamOperator would call this hook in AbstractStreamOperator.snapshotState() while
- * iterating over the key groups.
- *
- * @param keyGroupIndex the id of the key-group to be put in the snapshot.
- * @param out the stream to write to.
- */
- void snapshotKeyGroupState(int keyGroupIndex, DataOutputStream out) throws Exception;
-}
diff --git a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupRestoringOperator.java b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupRestoringOperator.java
deleted file mode 100644
index 079b85d..0000000
--- a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupRestoringOperator.java
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.wrappers.streaming.state;
-
-import java.io.DataInputStream;
-
-/** This interface is used to restore key-groups state. */
-public interface KeyGroupRestoringOperator {
- /**
- * Restore the state for a given {@code keyGroupIndex}.
- *
- * @param keyGroupIndex the id of the key-group to be put in the snapshot.
- * @param in the stream to read from.
- */
- void restoreKeyGroupState(int keyGroupIndex, DataInputStream in) throws Exception;
-}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
index be12cf4..af8c4ba 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
@@ -40,7 +40,7 @@
import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
-import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.reflect.Whitebox;
/** Tests for {@link FlinkExecutionEnvironments}. */
public class FlinkExecutionEnvironmentsTest {
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java
index 0071314..a90f7e6 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobServerDriverTest.java
@@ -24,7 +24,7 @@
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironmentTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironmentTest.java
index 74e979a..a2d2e0b 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironmentTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironmentTest.java
@@ -17,7 +17,6 @@
*/
package org.apache.beam.runners.flink;
-import static java.util.Arrays.asList;
import static org.apache.beam.sdk.testing.RegexMatcher.matches;
import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.instanceOf;
@@ -27,6 +26,7 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.core.Every.everyItem;
+import static org.junit.Assert.assertThrows;
import static org.junit.Assert.fail;
import java.io.ByteArrayOutputStream;
@@ -36,6 +36,7 @@
import java.io.Serializable;
import java.net.MalformedURLException;
import java.net.URL;
+import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import org.apache.beam.runners.core.construction.PTransformMatchers;
@@ -51,8 +52,8 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.RemoteEnvironment;
import org.apache.flink.streaming.api.environment.RemoteStreamEnvironment;
@@ -68,7 +69,7 @@
import org.junit.runners.JUnit4;
import org.mockito.ArgumentCaptor;
import org.mockito.Mockito;
-import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.reflect.Whitebox;
/** Tests for {@link FlinkPipelineExecutionEnvironment}. */
@RunWith(JUnit4.class)
@@ -106,17 +107,25 @@
@Test
public void shouldPrepareFilesToStageWhenFlinkMasterIsSetExplicitly() throws IOException {
- FlinkPipelineOptions options = testPreparingResourcesToStage("localhost:8081");
+ FlinkPipelineOptions options = testPreparingResourcesToStage("localhost:8081", false);
- assertThat(options.getFilesToStage().size(), is(1));
+ assertThat(options.getFilesToStage().size(), is(2));
assertThat(options.getFilesToStage().get(0), matches(".*\\.jar"));
}
@Test
+ public void shouldFailWhenFileDoesNotExistAndFlinkMasterIsSetExplicitly() {
+ assertThrows(
+ "To-be-staged file does not exist: ",
+ IllegalStateException.class,
+ () -> testPreparingResourcesToStage("localhost:8081", true));
+ }
+
+ @Test
public void shouldNotPrepareFilesToStageWhenFlinkMasterIsSetToAuto() throws IOException {
FlinkPipelineOptions options = testPreparingResourcesToStage("[auto]");
- assertThat(options.getFilesToStage().size(), is(2));
+ assertThat(options.getFilesToStage().size(), is(3));
assertThat(options.getFilesToStage(), everyItem(not(matches(".*\\.jar"))));
}
@@ -124,7 +133,7 @@
public void shouldNotPrepareFilesToStagewhenFlinkMasterIsSetToCollection() throws IOException {
FlinkPipelineOptions options = testPreparingResourcesToStage("[collection]");
- assertThat(options.getFilesToStage().size(), is(2));
+ assertThat(options.getFilesToStage().size(), is(3));
assertThat(options.getFilesToStage(), everyItem(not(matches(".*\\.jar"))));
}
@@ -132,7 +141,7 @@
public void shouldNotPrepareFilesToStageWhenFlinkMasterIsSetToLocal() throws IOException {
FlinkPipelineOptions options = testPreparingResourcesToStage("[local]");
- assertThat(options.getFilesToStage().size(), is(2));
+ assertThat(options.getFilesToStage().size(), is(3));
assertThat(options.getFilesToStage(), everyItem(not(matches(".*\\.jar"))));
}
@@ -360,16 +369,28 @@
private FlinkPipelineOptions testPreparingResourcesToStage(String flinkMaster)
throws IOException {
+ return testPreparingResourcesToStage(flinkMaster, true);
+ }
+
+ private FlinkPipelineOptions testPreparingResourcesToStage(
+ String flinkMaster, boolean includeNonExisting) throws IOException {
Pipeline pipeline = Pipeline.create();
String tempLocation = tmpFolder.newFolder().getAbsolutePath();
- File notEmptyDir = tmpFolder.newFolder();
- notEmptyDir.createNewFile();
- String notEmptyDirPath = notEmptyDir.getAbsolutePath();
- String notExistingPath = "/path/to/not/existing/dir";
+ List<String> filesToStage = new ArrayList<>();
- FlinkPipelineOptions options =
- setPipelineOptions(flinkMaster, tempLocation, asList(notEmptyDirPath, notExistingPath));
+ File stagingDir = tmpFolder.newFolder();
+ stagingDir.createNewFile();
+ filesToStage.add(stagingDir.getAbsolutePath());
+
+ File individualStagingFile = tmpFolder.newFile();
+ filesToStage.add(individualStagingFile.getAbsolutePath());
+
+ if (includeNonExisting) {
+ filesToStage.add("/path/to/not/existing/dir");
+ }
+
+ FlinkPipelineOptions options = setPipelineOptions(flinkMaster, tempLocation, filesToStage);
FlinkPipelineExecutionEnvironment flinkEnv = new FlinkPipelineExecutionEnvironment(options);
flinkEnv.translate(pipeline);
return options;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineOptionsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineOptionsTest.java
index d11d7ef..e5e297a 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineOptionsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineOptionsTest.java
@@ -77,7 +77,7 @@
assertThat(options.getLatencyTrackingInterval(), is(0L));
assertThat(options.isShutdownSourcesOnFinalWatermark(), is(false));
assertThat(options.getObjectReuse(), is(false));
- assertThat(options.getCheckpointingMode(), is(CheckpointingMode.EXACTLY_ONCE));
+ assertThat(options.getCheckpointingMode(), is(CheckpointingMode.EXACTLY_ONCE.name()));
assertThat(options.getMinPauseBetweenCheckpoints(), is(-1L));
assertThat(options.getCheckpointingInterval(), is(-1L));
assertThat(options.getCheckpointTimeoutMillis(), is(-1L));
@@ -85,10 +85,10 @@
assertThat(options.getNumberOfExecutionRetries(), is(-1));
assertThat(options.getExecutionRetryDelay(), is(-1L));
assertThat(options.getRetainExternalizedCheckpointsOnCancellation(), is(false));
- assertThat(options.getStateBackend(), is(nullValue()));
+ assertThat(options.getStateBackendFactory(), is(nullValue()));
assertThat(options.getMaxBundleSize(), is(1000L));
assertThat(options.getMaxBundleTimeMills(), is(1000L));
- assertThat(options.getExecutionModeForBatch(), is(ExecutionMode.PIPELINED));
+ assertThat(options.getExecutionModeForBatch(), is(ExecutionMode.PIPELINED.name()));
assertThat(options.getSavepointPath(), is(nullValue()));
assertThat(options.getAllowNonRestoredState(), is(false));
}
@@ -112,7 +112,8 @@
null,
null, /* key coder */
null /* key selector */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
/** Tests that PipelineOptions are present after serialization. */
@@ -138,7 +139,8 @@
options,
null, /* key coder */
null /* key selector */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
final byte[] serialized = SerializationUtils.serialize(doFnOperator);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkRunnerResultTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkRunnerResultTest.java
new file mode 100644
index 0000000..ba09816
--- /dev/null
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkRunnerResultTest.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.Is.is;
+
+import java.util.Collections;
+import org.apache.beam.sdk.PipelineResult;
+import org.joda.time.Duration;
+import org.junit.Test;
+
+/** Tests for {@link FlinkRunnerResult}. */
+public class FlinkRunnerResultTest {
+
+ @Test
+ public void testPipelineResultReturnsDone() {
+ FlinkRunnerResult result = new FlinkRunnerResult(Collections.emptyMap(), 100);
+ assertThat(result.getState(), is(PipelineResult.State.DONE));
+ }
+
+ @Test
+ public void testWaitUntilFinishReturnsDone() {
+ FlinkRunnerResult result = new FlinkRunnerResult(Collections.emptyMap(), 100);
+ assertThat(result.waitUntilFinish(), is(PipelineResult.State.DONE));
+ assertThat(result.waitUntilFinish(Duration.millis(100)), is(PipelineResult.State.DONE));
+ }
+
+ @Test
+ public void testCancelDoesNotThrowAnException() {
+ FlinkRunnerResult result = new FlinkRunnerResult(Collections.emptyMap(), 100);
+ result.cancel();
+ assertThat(result.getState(), is(PipelineResult.State.DONE));
+ }
+}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java
index ecc54c2..4315e62 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java
@@ -17,16 +17,18 @@
*/
package org.apache.beam.runners.flink;
-import static org.junit.Assert.assertNotNull;
+import static org.hamcrest.MatcherAssert.assertThat;
import java.io.Serializable;
import java.lang.reflect.Method;
import java.net.URI;
import java.util.Collections;
+import java.util.Objects;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.construction.Environments;
import org.apache.beam.runners.core.construction.PipelineTranslation;
@@ -50,9 +52,9 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.apache.flink.api.common.JobID;
import org.apache.flink.configuration.CheckpointingOptions;
import org.apache.flink.configuration.Configuration;
@@ -63,46 +65,55 @@
import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
import org.apache.flink.runtime.minicluster.MiniCluster;
import org.apache.flink.runtime.minicluster.MiniClusterConfiguration;
-import org.apache.flink.streaming.util.TestStreamEnvironment;
+import org.hamcrest.Matchers;
+import org.hamcrest.core.IsIterableContaining;
import org.joda.time.Duration;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
+import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-/** Tests that Flink's Savepoints work with the Flink Runner. */
+/**
+ * Tests that Flink's Savepoints work with the Flink Runner. This includes taking a savepoint of a
+ * running pipeline, shutting down the pipeline, and restarting the pipeline from the savepoint with
+ * a different parallelism.
+ */
public class FlinkSavepointTest implements Serializable {
private static final Logger LOG = LoggerFactory.getLogger(FlinkSavepointTest.class);
- /** Static for synchronization between the pipeline state and the test. */
- private static CountDownLatch oneShotLatch;
+ /** Flink cluster that runs over the lifespan of the tests. */
+ private static transient MiniCluster flinkCluster;
+ /** Static for synchronization between the pipeline state and the test. */
+ private static volatile CountDownLatch oneShotLatch;
+
+ /** Temporary folder for savepoints. */
@ClassRule public static transient TemporaryFolder tempFolder = new TemporaryFolder();
- private static transient MiniCluster flinkCluster;
+ /** Each test has a timeout of 60 seconds (for safety). */
+ @Rule public Timeout timeout = new Timeout(60, TimeUnit.SECONDS);
@BeforeClass
public static void beforeClass() throws Exception {
- final int parallelism = 4;
-
Configuration config = new Configuration();
// Avoid port collision in parallel tests
config.setInteger(RestOptions.PORT, 0);
config.setString(CheckpointingOptions.STATE_BACKEND, "filesystem");
+
+ String savepointPath = "file://" + tempFolder.getRoot().getAbsolutePath();
+ LOG.info("Savepoints will be written to {}", savepointPath);
// It is necessary to configure the checkpoint directory for the state backend,
// even though we only create savepoints in this test.
- config.setString(
- CheckpointingOptions.CHECKPOINTS_DIRECTORY,
- "file://" + tempFolder.getRoot().getAbsolutePath());
+ config.setString(CheckpointingOptions.CHECKPOINTS_DIRECTORY, savepointPath);
// Checkpoints will go into a subdirectory of this directory
- config.setString(
- CheckpointingOptions.SAVEPOINT_DIRECTORY,
- "file://" + tempFolder.getRoot().getAbsolutePath());
+ config.setString(CheckpointingOptions.SAVEPOINT_DIRECTORY, savepointPath);
MiniClusterConfiguration clusterConfig =
new MiniClusterConfiguration.Builder()
@@ -113,14 +124,12 @@
flinkCluster = new MiniCluster(clusterConfig);
flinkCluster.start();
-
- TestStreamEnvironment.setAsContext(flinkCluster, parallelism);
}
@AfterClass
public static void afterClass() throws Exception {
- TestStreamEnvironment.unsetAsContext();
flinkCluster.close();
+ flinkCluster = null;
}
@After
@@ -130,14 +139,18 @@
flinkCluster.cancelJob(jobStatusMessage.getJobId()).get();
}
}
+ while (!flinkCluster.listJobs().get().stream()
+ .allMatch(job -> job.getJobState().isTerminalState())) {
+ Thread.sleep(50);
+ }
}
- @Test(timeout = 60_000)
+ @Test
public void testSavepointRestoreLegacy() throws Exception {
runSavepointAndRestore(false);
}
- @Test(timeout = 60_000)
+ @Test
public void testSavepointRestorePortable() throws Exception {
runSavepointAndRestore(true);
}
@@ -145,8 +158,8 @@
private void runSavepointAndRestore(boolean isPortablePipeline) throws Exception {
FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
options.setStreaming(true);
- // savepoint assumes local file system
- options.setParallelism(1);
+ // Initial parallelism
+ options.setParallelism(2);
options.setRunner(FlinkRunner.class);
oneShotLatch = new CountDownLatch(1);
@@ -163,6 +176,8 @@
String savepointDir = takeSavepointAndCancelJob(jobID);
oneShotLatch = new CountDownLatch(1);
+ // Increase parallelism
+ options.setParallelism(4);
pipeline = Pipeline.create(options);
createStreamingJob(pipeline, true, isPortablePipeline);
@@ -185,15 +200,13 @@
.getOptions()
.as(PortablePipelineOptions.class)
.setDefaultEnvironmentType(Environments.ENVIRONMENT_EMBEDDED);
- pipeline
- .getOptions()
- .as(FlinkPipelineOptions.class)
- .setFlinkMaster(getFlinkMaster());
+ pipeline.getOptions().as(FlinkPipelineOptions.class).setFlinkMaster(getFlinkMaster());
RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline);
ListeningExecutorService executorService =
MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(1));
+ FlinkPipelineOptions pipelineOptions = pipeline.getOptions().as(FlinkPipelineOptions.class);
try {
JobInvocation jobInvocation =
FlinkJobInvoker.createJobInvocation(
@@ -201,9 +214,8 @@
"none",
executorService,
pipelineProto,
- pipeline.getOptions().as(FlinkPipelineOptions.class),
- null,
- Collections.emptyList());
+ pipelineOptions,
+ new FlinkPipelineRunner(pipelineOptions, null, Collections.emptyList()));
jobInvocation.start();
@@ -285,13 +297,16 @@
new InferableFunction<byte[], KV<String, Void>>() {
@Override
public KV<String, Void> apply(byte[] input) throws Exception {
+ // This only writes data to one of the two initial partitions.
+ // We want to test this due to
+ // https://jira.apache.org/jira/browse/BEAM-7144
return KV.of("key", null);
}
}))
.apply(
ParDo.of(
new DoFn<KV<String, Void>, KV<String, Long>>() {
- @StateId("valueState")
+ @StateId("nextInteger")
private final StateSpec<ValueState<Long>> valueStateSpec =
StateSpecs.value();
@@ -309,15 +324,15 @@
@OnTimer("timer")
public void onTimer(
OnTimerContext context,
- @StateId("valueState") ValueState<Long> intValueState,
+ @StateId("nextInteger") ValueState<Long> nextInteger,
@TimerId("timer") Timer timer) {
- Long current = intValueState.read();
+ Long current = nextInteger.read();
if (current == null) {
current = -1L;
}
long next = current + 1;
+ nextInteger.write(next);
context.output(KV.of("key", next));
- intValueState.write(next);
timer.offset(Duration.millis(100)).setRelative();
}
}));
@@ -350,12 +365,9 @@
ProcessContext context,
@StateId("valueState") ValueState<Integer> intValueState,
@StateId("bagState") BagState<Integer> intBagState) {
- Integer read = intValueState.read();
- assertNotNull(read);
- if (read == 42) {
- intValueState.write(0);
- oneShotLatch.countDown();
- }
+ assertThat(intValueState.read(), Matchers.is(42));
+ assertThat(intBagState.read(), IsIterableContaining.hasItems(40, 1, 1));
+ oneShotLatch.countDown();
}
}));
} else {
@@ -374,8 +386,7 @@
ProcessContext context,
@StateId("valueState") ValueState<Integer> intValueState,
@StateId("bagState") BagState<Integer> intBagState) {
- Long value = context.element().getValue();
- assertNotNull(value);
+ Long value = Objects.requireNonNull(context.element().getValue());
if (value == 0L) {
intValueState.write(42);
intBagState.add(40);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslatorTest.java
index 8eaf060..1b6ad7e 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslatorTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.ShardedKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.flink.runtime.state.KeyGroupRangeAssignment;
import org.junit.Test;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkTestPipeline.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkTestPipeline.java
index cfb4dde..293b49b 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkTestPipeline.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkTestPipeline.java
@@ -20,9 +20,7 @@
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.options.PipelineOptions;
-/**
- * {@link org.apache.beam.sdk.Pipeline} for testing Beam pipelines on the {@link FlinkRunner}.
- */
+/** {@link org.apache.beam.sdk.Pipeline} for testing Beam pipelines on the {@link FlinkRunner}. */
public class FlinkTestPipeline extends Pipeline {
/**
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
index 6214ff4..10db32c 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
@@ -43,9 +43,9 @@
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -148,8 +148,8 @@
flinkJobExecutor,
pipelineProto,
options.as(FlinkPipelineOptions.class),
- null,
- Collections.emptyList());
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
while (jobInvocation.getState() != Enum.DONE) {
Thread.sleep(1000);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
index c96ebb8..91f243b 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -202,8 +202,8 @@
flinkJobExecutor,
pipelineProto,
options.as(FlinkPipelineOptions.class),
- null,
- Collections.emptyList());
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java
index bffc903..9cddcd6 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableTimersExecutionTest.java
@@ -52,8 +52,8 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -189,8 +189,8 @@
flinkJobExecutor,
pipelineProto,
options.as(FlinkPipelineOptions.class),
- null,
- Collections.emptyList());
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
while (jobInvocation.getState() != Enum.DONE) {
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java
index fc25715..40d621f 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourcePortableTest.java
@@ -35,9 +35,9 @@
import org.apache.beam.sdk.testing.CrashingRunner;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -110,8 +110,8 @@
flinkJobExecutor,
pipelineProto,
options.as(FlinkPipelineOptions.class),
- null,
- Collections.emptyList());
+ new FlinkPipelineRunner(
+ options.as(FlinkPipelineOptions.class), null, Collections.emptyList()));
jobInvocation.start();
while (jobInvocation.getState() != Enum.DONE) {
Thread.sleep(100);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingTest.java
index a7acabf..5f2434d 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceStreamingTest.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.flink.test.util.AbstractTestBase;
import org.junit.After;
import org.junit.Before;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceTest.java
index 9c4b346..96d45dd 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/ReadSourceTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.flink.test.util.JavaProgramTestBase;
/** Reads from a bounded source in batch execution. */
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
index 237d176b..8a5f027 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainerTest.java
@@ -19,8 +19,8 @@
import static org.apache.beam.runners.flink.metrics.FlinkMetricContainer.getFlinkMetricNameString;
import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.anyObject;
import static org.mockito.Matchers.anyString;
import static org.mockito.Matchers.argThat;
@@ -51,7 +51,7 @@
import org.apache.beam.sdk.metrics.MetricKey;
import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.metrics.MetricGroup;
import org.apache.flink.metrics.SimpleCounter;
@@ -123,13 +123,15 @@
@Test
public void testMonitoringInfoUpdate() {
FlinkMetricContainer container = new FlinkMetricContainer(runtimeContext);
- MetricsContainer step = container.getMetricsContainer("step");
SimpleCounter userCounter = new SimpleCounter();
when(metricGroup.counter("ns1.metric1")).thenReturn(userCounter);
- SimpleCounter elemCounter = new SimpleCounter();
- when(metricGroup.counter("beam.metric:element_count:v1")).thenReturn(elemCounter);
+ SimpleCounter pCollectionCounter = new SimpleCounter();
+ when(metricGroup.counter("pcoll.metric:element_count:v1")).thenReturn(pCollectionCounter);
+
+ SimpleCounter pTransformCounter = new SimpleCounter();
+ when(metricGroup.counter("anyPTransform.myMetric")).thenReturn(pTransformCounter);
MonitoringInfo userCountMonitoringInfo =
new SimpleMonitoringInfoBuilder()
@@ -141,22 +143,32 @@
.build();
assertNotNull(userCountMonitoringInfo);
- MonitoringInfo elemCountMonitoringInfo =
+ MonitoringInfo pCollectionScoped =
new SimpleMonitoringInfoBuilder()
.setUrn(MonitoringInfoConstants.Urns.ELEMENT_COUNT)
.setInt64Value(222)
- .setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "step")
.setLabel(MonitoringInfoConstants.Labels.PCOLLECTION, "pcoll")
.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "anyPTransform")
.build();
- assertNotNull(elemCountMonitoringInfo);
+ assertNotNull(pCollectionScoped);
+
+ MonitoringInfo transformScoped =
+ new SimpleMonitoringInfoBuilder()
+ .setUrn(MonitoringInfoConstants.Urns.START_BUNDLE_MSECS)
+ .setInt64Value(333)
+ .setLabel(MonitoringInfoConstants.Labels.NAME, "myMetric")
+ .setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, "anyPTransform")
+ .build();
+ assertNotNull(transformScoped);
assertThat(userCounter.getCount(), is(0L));
- assertThat(elemCounter.getCount(), is(0L));
+ assertThat(pCollectionCounter.getCount(), is(0L));
+ assertThat(pTransformCounter.getCount(), is(0L));
container.updateMetrics(
- "step", ImmutableList.of(userCountMonitoringInfo, elemCountMonitoringInfo));
+ "step", ImmutableList.of(userCountMonitoringInfo, pCollectionScoped, transformScoped));
assertThat(userCounter.getCount(), is(111L));
- assertThat(elemCounter.getCount(), is(222L));
+ assertThat(pCollectionCounter.getCount(), is(222L));
+ assertThat(pTransformCounter.getCount(), is(333L));
}
@Test
@@ -248,7 +260,7 @@
argThat(
new ArgumentMatcher<FlinkDistributionGauge>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(FlinkDistributionGauge argument) {
DistributionResult actual = ((FlinkDistributionGauge) argument).getValue();
DistributionResult expected = DistributionResult.create(30, 10, 1, 5);
return actual.equals(expected);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByNullKeyTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByNullKeyTest.java
index 08dfeb9..6d08906 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByNullKeyTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByNullKeyTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.flink.test.util.AbstractTestBase;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TopWikipediaSessionsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TopWikipediaSessionsTest.java
index 9f05e81..63abfa5 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TopWikipediaSessionsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/TopWikipediaSessionsTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.flink.test.util.AbstractTestBase;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContextTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContextTest.java
deleted file mode 100644
index 5ffca28..0000000
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContextTest.java
+++ /dev/null
@@ -1,76 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.functions;
-
-import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
-import org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.MultiInstanceFactory;
-import org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory.WrappedContext;
-import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.options.PortablePipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link FlinkDefaultExecutableStageContext}. */
-@RunWith(JUnit4.class)
-public class FlinkDefaultExecutableStageContextTest {
- private static JobInfo constructJobInfo(String jobId, long parallelism) {
- PortablePipelineOptions portableOptions =
- PipelineOptionsFactory.as(PortablePipelineOptions.class);
- portableOptions.setSdkWorkerParallelism(parallelism);
-
- Struct pipelineOptions = PipelineOptionsTranslation.toProto(portableOptions);
- return JobInfo.create(jobId, "job-name", "retrieval-token", pipelineOptions);
- }
-
- @Test
- public void testMultiInstanceFactory() {
- JobInfo jobInfo = constructJobInfo("multi-instance-factory-test", 2);
-
- WrappedContext f1 = (WrappedContext) MultiInstanceFactory.MULTI_INSTANCE.get(jobInfo);
- WrappedContext f2 = (WrappedContext) MultiInstanceFactory.MULTI_INSTANCE.get(jobInfo);
- WrappedContext f3 = (WrappedContext) MultiInstanceFactory.MULTI_INSTANCE.get(jobInfo);
-
- Assert.assertNotEquals("We should create two different factories", f1.context, f2.context);
- Assert.assertEquals(
- "Future calls should be round-robbined to those two factories", f1.context, f3.context);
- }
-
- @Test
- public void testDefault() {
- JobInfo jobInfo = constructJobInfo("default-test", 0);
-
- int expectedParallelism = Math.max(1, Runtime.getRuntime().availableProcessors() - 1);
-
- WrappedContext f1 = (WrappedContext) MultiInstanceFactory.MULTI_INSTANCE.get(jobInfo);
- for (int i = 1; i < expectedParallelism; i++) {
- Assert.assertNotEquals(
- "We should create " + expectedParallelism + " different factories",
- f1.context,
- ((WrappedContext) MultiInstanceFactory.MULTI_INSTANCE.get(jobInfo)).context);
- }
-
- Assert.assertEquals(
- "Future calls should be round-robbined to those",
- f1.context,
- ((WrappedContext) MultiInstanceFactory.MULTI_INSTANCE.get(jobInfo)).context);
- }
-}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
index bb40556..93f7cd2 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageFunctionTest.java
@@ -33,6 +33,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
import org.apache.beam.runners.fnexecution.control.RemoteBundle;
@@ -40,10 +41,11 @@
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.flink.api.common.cache.DistributedCache;
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.configuration.Configuration;
@@ -57,7 +59,7 @@
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
-import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.reflect.Whitebox;
/** Tests for {@link FlinkExecutableStageFunction}. */
@RunWith(Parameterized.class)
@@ -75,7 +77,7 @@
@Mock private RuntimeContext runtimeContext;
@Mock private DistributedCache distributedCache;
@Mock private Collector<RawUnionValue> collector;
- @Mock private FlinkExecutableStageContext stageContext;
+ @Mock private ExecutableStageContext stageContext;
@Mock private StageBundleFactory stageBundleFactory;
@Mock private StateRequestHandler stateRequestHandler;
@Mock private ProcessBundleDescriptors.ExecutableProcessBundleDescriptor processBundleDescriptor;
@@ -186,7 +188,7 @@
}
@Override
- public Map<String, FnDataReceiver<WindowedValue<?>>> getInputReceivers() {
+ public Map<String, FnDataReceiver> getInputReceivers() {
return ImmutableMap.of(
"input",
input -> {
@@ -252,11 +254,18 @@
* behavior of the stage context itself is unchanged.
*/
private FlinkExecutableStageFunction<Integer> getFunction(Map<String, Integer> outputMap) {
- FlinkExecutableStageContext.Factory contextFactory =
- Mockito.mock(FlinkExecutableStageContext.Factory.class);
+ FlinkExecutableStageContextFactory contextFactory =
+ Mockito.mock(FlinkExecutableStageContextFactory.class);
when(contextFactory.get(any())).thenReturn(stageContext);
FlinkExecutableStageFunction<Integer> function =
- new FlinkExecutableStageFunction<>(stagePayload, jobInfo, outputMap, contextFactory, null);
+ new FlinkExecutableStageFunction<>(
+ "step",
+ PipelineOptionsFactory.create(),
+ stagePayload,
+ jobInfo,
+ outputMap,
+ contextFactory,
+ null);
function.setRuntimeContext(runtimeContext);
Whitebox.setInternalState(function, "stateRequestHandler", stateRequestHandler);
return function;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunctionTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunctionTest.java
index 61d9c24..85d3cfa 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunctionTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ImpulseSourceFunctionTest.java
@@ -17,19 +17,28 @@
*/
package org.apache.beam.runners.flink.translation.functions;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.core.Is.is;
import static org.hamcrest.core.IsInstanceOf.instanceOf;
-import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.argThat;
import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.OperatorStateStore;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TestName;
import org.mockito.ArgumentMatcher;
+import org.mockito.Matchers;
import org.mockito.Mockito;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -46,6 +55,7 @@
public ImpulseSourceFunctionTest() {
this.sourceContext = Mockito.mock(SourceFunction.SourceContext.class);
+ when(sourceContext.getCheckpointLock()).thenReturn(new Object());
}
@Test
@@ -55,16 +65,49 @@
}
@Test(timeout = 10_000)
- public void testImpulse() throws Exception {
+ public void testImpulseInitial() throws Exception {
ImpulseSourceFunction source = new ImpulseSourceFunction(false);
+ // No state available from previous runs
+ ListState<Object> mockListState = getMockListState(Collections.emptyList());
+ source.initializeState(getInitializationContext(mockListState));
+
+ // 1) Should finish
source.run(sourceContext);
- // should finish
+ // 2) Should use checkpoint lock
+ verify(sourceContext).getCheckpointLock();
+ // 3) Should emit impulse element
verify(sourceContext).collect(argThat(elementMatcher));
+ verifyNoMoreInteractions(sourceContext);
+ // 4) Should modify checkpoint state
+ verify(mockListState).get();
+ verify(mockListState).add(true);
+ verifyNoMoreInteractions(mockListState);
+ }
+
+ @Test(timeout = 10_000)
+ public void testImpulseRestored() throws Exception {
+ ImpulseSourceFunction source = new ImpulseSourceFunction(false);
+ // Previous state available
+ ListState<Object> mockListState = getMockListState(Collections.singletonList(true));
+ source.initializeState(getInitializationContext(mockListState));
+
+ // 1) Should finish
+ source.run(sourceContext);
+ // 2) Should _not_ emit impulse element
+ verifyNoMoreInteractions(sourceContext);
+ // 3) Should keep checkpoint state
+ verify(mockListState).get();
+ verifyNoMoreInteractions(mockListState);
}
@Test(timeout = 10_000)
public void testKeepAlive() throws Exception {
ImpulseSourceFunction source = new ImpulseSourceFunction(true);
+
+ // No previous state available (=impulse should be emitted)
+ ListState<Object> mockListState = getMockListState(Collections.emptyList());
+ source.initializeState(getInitializationContext(mockListState));
+
Thread sourceThread =
new Thread(
() -> {
@@ -85,11 +128,19 @@
sourceThread.join();
}
verify(sourceContext).collect(argThat(elementMatcher));
+ verify(mockListState).add(true);
+ verify(mockListState).get();
+ verifyNoMoreInteractions(mockListState);
}
@Test(timeout = 10_000)
public void testKeepAliveDuringInterrupt() throws Exception {
ImpulseSourceFunction source = new ImpulseSourceFunction(true);
+
+ // No previous state available (=impulse should not be emitted)
+ ListState<Object> mockListState = getMockListState(Collections.singletonList(true));
+ source.initializeState(getInitializationContext(mockListState));
+
Thread sourceThread =
new Thread(
() -> {
@@ -105,17 +156,41 @@
sourceThread.interrupt();
Thread.sleep(200);
assertThat(sourceThread.isAlive(), is(true));
+
// should quit
source.cancel();
sourceThread.interrupt();
sourceThread.join();
- verify(sourceContext).collect(argThat(elementMatcher));
+
+ // nothing should have been emitted because the impulse was emitted before restore
+ verifyNoMoreInteractions(sourceContext);
}
- private static class ImpulseElementMatcher extends ArgumentMatcher<WindowedValue<byte[]>> {
+ private static <T> FunctionInitializationContext getInitializationContext(ListState<T> listState)
+ throws Exception {
+ FunctionInitializationContext mock = Mockito.mock(FunctionInitializationContext.class);
+ OperatorStateStore mockOperatorState = getMockOperatorState(listState);
+ when(mock.getOperatorStateStore()).thenReturn(mockOperatorState);
+ return mock;
+ }
+
+ private static <T> OperatorStateStore getMockOperatorState(ListState<T> listState)
+ throws Exception {
+ OperatorStateStore mock = Mockito.mock(OperatorStateStore.class);
+ when(mock.getListState(Matchers.any(ListStateDescriptor.class))).thenReturn(listState);
+ return mock;
+ }
+
+ private static <T> ListState<T> getMockListState(List<T> initialState) throws Exception {
+ ListState mock = Mockito.mock(ListState.class);
+ when(mock.get()).thenReturn(initialState);
+ return mock;
+ }
+
+ private static class ImpulseElementMatcher implements ArgumentMatcher<WindowedValue<byte[]>> {
@Override
- public boolean matches(Object o) {
+ public boolean matches(WindowedValue<byte[]> o) {
return o instanceof WindowedValue
&& Arrays.equals((byte[]) ((WindowedValue) o).getValue(), new byte[] {});
}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactoryTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactoryTest.java
deleted file mode 100644
index 6882a47..0000000
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactoryTest.java
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.flink.translation.functions;
-
-import static org.hamcrest.core.Is.is;
-import static org.junit.Assert.assertThat;
-import static org.mockito.Matchers.any;
-import static org.mockito.Mockito.doThrow;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-import java.io.ByteArrayOutputStream;
-import java.io.PrintStream;
-import org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory.Creator;
-import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/** Tests for {@link ReferenceCountingFlinkExecutableStageContextFactory}. */
-@RunWith(JUnit4.class)
-public class ReferenceCountingFlinkExecutableStageContextFactoryTest {
-
- @Test
- public void testCreateReuseReleaseCreate() throws Exception {
-
- Creator creator = mock(Creator.class);
- FlinkExecutableStageContext c1 = mock(FlinkExecutableStageContext.class);
- FlinkExecutableStageContext c2 = mock(FlinkExecutableStageContext.class);
- FlinkExecutableStageContext c3 = mock(FlinkExecutableStageContext.class);
- FlinkExecutableStageContext c4 = mock(FlinkExecutableStageContext.class);
- when(creator.apply(any(JobInfo.class)))
- .thenReturn(c1)
- .thenReturn(c2)
- .thenReturn(c3)
- .thenReturn(c4);
- ReferenceCountingFlinkExecutableStageContextFactory factory =
- ReferenceCountingFlinkExecutableStageContextFactory.create(creator);
- JobInfo jobA = mock(JobInfo.class);
- when(jobA.jobId()).thenReturn("jobA");
- JobInfo jobB = mock(JobInfo.class);
- when(jobB.jobId()).thenReturn("jobB");
- FlinkExecutableStageContext ac1A = factory.get(jobA); // 1 open jobA
- FlinkExecutableStageContext ac2B = factory.get(jobB); // 1 open jobB
- Assert.assertSame(
- "Context should be cached and reused.", ac1A, factory.get(jobA)); // 2 open jobA
- Assert.assertSame(
- "Context should be cached and reused.", ac2B, factory.get(jobB)); // 2 open jobB
- factory.release(ac1A); // 1 open jobA
- Assert.assertSame(
- "Context should be cached and reused.", ac1A, factory.get(jobA)); // 2 open jobA
- factory.release(ac1A); // 1 open jobA
- factory.release(ac1A); // 0 open jobA
- FlinkExecutableStageContext ac3A = factory.get(jobA); // 1 open jobA
- Assert.assertNotSame("We should get a new instance.", ac1A, ac3A);
- Assert.assertSame(
- "Context should be cached and reused.", ac3A, factory.get(jobA)); // 2 open jobA
- factory.release(ac3A); // 1 open jobA
- factory.release(ac3A); // 0 open jobA
- Assert.assertSame(
- "Context should be cached and reused.", ac2B, factory.get(jobB)); // 3 open jobB
- factory.release(ac2B); // 2 open jobB
- factory.release(ac2B); // 1 open jobB
- factory.release(ac2B); // 0 open jobB
- FlinkExecutableStageContext ac4B = factory.get(jobB); // 1 open jobB
- Assert.assertNotSame("We should get a new instance.", ac2B, ac4B);
- factory.release(ac4B); // 0 open jobB
- }
-
- @Test
- public void testCatchThrowablesAndLogThem() throws Exception {
- PrintStream oldErr = System.err;
- oldErr.flush();
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- PrintStream newErr = new PrintStream(baos);
- try {
- System.setErr(newErr);
- Creator creator = mock(Creator.class);
- FlinkExecutableStageContext c1 = mock(FlinkExecutableStageContext.class);
- when(creator.apply(any(JobInfo.class))).thenReturn(c1);
- // throw an Throwable and ensure that it is caught and logged.
- doThrow(new NoClassDefFoundError()).when(c1).close();
- ReferenceCountingFlinkExecutableStageContextFactory factory =
- ReferenceCountingFlinkExecutableStageContextFactory.create(creator);
- JobInfo jobA = mock(JobInfo.class);
- when(jobA.jobId()).thenReturn("jobA");
- FlinkExecutableStageContext ac1A = factory.get(jobA);
- factory.release(ac1A);
- newErr.flush();
- String output = new String(baos.toByteArray(), Charsets.UTF_8);
- // Ensure that the error is logged
- assertThat(output.contains("Unable to close FlinkExecutableStageContext"), is(true));
- } finally {
- newErr.flush();
- System.setErr(oldErr);
- }
- }
-}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java
index 3a2c4a3..dbc6ea6 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DedupingOperatorTest.java
@@ -24,6 +24,7 @@
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.flink.api.common.typeinfo.TypeInformation;
@@ -100,7 +101,7 @@
private KeyedOneInputStreamOperatorTestHarness<
ByteBuffer, WindowedValue<ValueWithRecordId<String>>, WindowedValue<String>>
getDebupingHarness() throws Exception {
- DedupingOperator<String> operator = new DedupingOperator<>();
+ DedupingOperator<String> operator = new DedupingOperator<>(PipelineOptionsFactory.create());
return new KeyedOneInputStreamOperatorTestHarness<>(
operator,
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java
index 8afb218..57f7694 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperatorTest.java
@@ -71,11 +71,11 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.functions.KeySelector;
@@ -92,7 +92,7 @@
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.reflect.Whitebox;
/** Tests for {@link DoFnOperator}. */
@RunWith(JUnit4.class)
@@ -141,7 +141,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
new OneInputStreamOperatorTestHarness<>(doFnOperator);
@@ -202,7 +203,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
new OneInputStreamOperatorTestHarness<>(doFnOperator);
@@ -316,7 +318,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
VarIntCoder.of(), /* key coder */
WindowedValue::getValue,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<Integer>, WindowedValue<String>> testHarness =
new KeyedOneInputStreamOperatorTestHarness<>(
@@ -409,7 +412,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
VarIntCoder.of(), /* key coder */
WindowedValue::getValue,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<Integer>, WindowedValue<String>> testHarness =
new KeyedOneInputStreamOperatorTestHarness<>(
@@ -517,7 +521,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
StringUtf8Coder.of(), /* key coder */
kvWindowedValue -> kvWindowedValue.getValue().getKey(),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
KeyedOneInputStreamOperatorTestHarness<
String, WindowedValue<KV<String, Integer>>, WindowedValue<KV<String, Integer>>>
@@ -622,7 +627,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
keyCoder,
keyed ? WindowedValue::getValue : null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
TwoInputStreamOperatorTestHarness<WindowedValue<String>, RawUnionValue, WindowedValue<String>>
testHarness = new TwoInputStreamOperatorTestHarness<>(doFnOperator);
@@ -793,7 +799,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
return new TwoInputStreamOperatorTestHarness<>(doFnOperator);
});
@@ -831,7 +838,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
keyCoder,
WindowedValue::getValue,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
return new KeyedTwoInputStreamOperatorTestHarness<>(
doFnOperator,
@@ -928,7 +936,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
return new TwoInputStreamOperatorTestHarness<>(doFnOperator);
});
@@ -967,7 +976,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
keyCoder,
WindowedValue::getValue,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
return new KeyedTwoInputStreamOperatorTestHarness<>(
doFnOperator,
@@ -1142,7 +1152,8 @@
PipelineOptionsFactory.as(FlinkPipelineOptions.class),
VarIntCoder.of() /* key coder */,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
return new KeyedOneInputStreamOperatorTestHarness<>(doFnOperator, keySelector, keyCoderInfo);
}
@@ -1189,7 +1200,8 @@
options,
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
new OneInputStreamOperatorTestHarness<>(doFnOperator);
@@ -1238,7 +1250,8 @@
options,
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> newHarness =
new OneInputStreamOperatorTestHarness<>(newDoFnOperator);
@@ -1331,7 +1344,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<
WindowedValue<KV<String, String>>, WindowedValue<KV<String, String>>>
@@ -1387,7 +1401,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
testHarness =
new KeyedOneInputStreamOperatorTestHarness(
@@ -1470,7 +1485,8 @@
options,
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnOperator<String, String> doFnOperator = doFnOperatorSupplier.get();
OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
@@ -1585,7 +1601,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnOperator<KV<String, String>, KV<String, String>> doFnOperator = doFnOperatorSupplier.get();
OneInputStreamOperatorTestHarness<
@@ -1614,10 +1631,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key3", "finishBundle"))));
doFnOperator = doFnOperatorSupplier.get();
@@ -1635,10 +1652,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key3", "finishBundle"))));
// repeat to see if elements are evicted
@@ -1648,10 +1665,10 @@
assertThat(
stripStreamRecordFromWindowedValue(testHarness.getOutput()),
contains(
- WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
- WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key", "a")),
WindowedValue.valueInGlobalWindow(KV.of("key", "b")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "c")),
+ WindowedValue.valueInGlobalWindow(KV.of("key2", "d")),
WindowedValue.valueInGlobalWindow(KV.of("key3", "finishBundle"))));
}
@@ -1698,7 +1715,8 @@
options,
keyCoder,
keySelector,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
/**
@@ -1744,7 +1762,8 @@
options,
null,
null,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
OneInputStreamOperatorTestHarness<WindowedValue<String>, WindowedValue<String>> testHarness =
new OneInputStreamOperatorTestHarness<>(doFnOperator);
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java
index 16b35e0..8134b24 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperatorTest.java
@@ -22,6 +22,7 @@
import static org.hamcrest.Matchers.hasSize;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.iterableWithSize;
import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
@@ -29,6 +30,7 @@
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.verify;
@@ -37,6 +39,7 @@
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
@@ -45,6 +48,7 @@
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.Lock;
import javax.annotation.Nullable;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
@@ -58,20 +62,25 @@
import org.apache.beam.runners.core.TimerInternals;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate;
-import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContext;
+import org.apache.beam.runners.flink.streaming.FlinkStateInternalsTest;
+import org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageContextFactory;
import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.utils.NoopLock;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
import org.apache.beam.runners.fnexecution.control.RemoteBundle;
import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.state.BagState;
@@ -80,14 +89,18 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf.ByteStringCoder;
import org.apache.commons.lang3.SerializationUtils;
import org.apache.commons.lang3.mutable.MutableObject;
import org.apache.flink.api.common.cache.DistributedCache;
@@ -110,7 +123,7 @@
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
-import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.reflect.Whitebox;
/** Tests for {@link ExecutableStageDoFnOperator}. */
@RunWith(JUnit4.class)
@@ -119,7 +132,7 @@
@Mock private RuntimeContext runtimeContext;
@Mock private DistributedCache distributedCache;
- @Mock private FlinkExecutableStageContext stageContext;
+ @Mock private ExecutableStageContext stageContext;
@Mock private StageBundleFactory stageBundleFactory;
@Mock private StateRequestHandler stateRequestHandler;
@Mock private ProcessBundleDescriptors.ExecutableProcessBundleDescriptor processBundleDescriptor;
@@ -169,6 +182,7 @@
when(runtimeContext.getDistributedCache()).thenReturn(distributedCache);
when(stageContext.getStageBundleFactory(any())).thenReturn(stageBundleFactory);
when(processBundleDescriptor.getTimerSpecs()).thenReturn(Collections.emptyMap());
+ when(processBundleDescriptor.getBagUserStateSpecs()).thenReturn(Collections.emptyMap());
when(stageBundleFactory.getProcessBundleDescriptor()).thenReturn(processBundleDescriptor);
}
@@ -292,7 +306,7 @@
}
@Override
- public Map<String, FnDataReceiver<WindowedValue<?>>> getInputReceivers() {
+ public Map<String, FnDataReceiver> getInputReceivers() {
return ImmutableMap.of(
"input",
input -> {
@@ -639,6 +653,134 @@
}
@Test
+ public void testCacheTokenHandling() throws Exception {
+ InMemoryStateInternals test = InMemoryStateInternals.forKey("test");
+ KeyedStateBackend<ByteBuffer> stateBackend = FlinkStateInternalsTest.createStateBackend();
+
+ // User state the cache token is valid for the lifetime of the operator
+ for (String expectedToken : new String[] {"first token", "second token"}) {
+ final IdGenerator cacheTokenGenerator = () -> expectedToken;
+ ExecutableStageDoFnOperator.BagUserStateFactory<ByteString, Integer, GlobalWindow>
+ bagUserStateFactory =
+ new ExecutableStageDoFnOperator.BagUserStateFactory<>(
+ cacheTokenGenerator, test, stateBackend, NoopLock.get());
+
+ ByteString key1 = ByteString.copyFrom("key1", Charsets.UTF_8);
+ ByteString key2 = ByteString.copyFrom("key2", Charsets.UTF_8);
+
+ Map<String, Map<String, ProcessBundleDescriptors.BagUserStateSpec>> userStateMapMock =
+ Mockito.mock(Map.class);
+ Map<String, ProcessBundleDescriptors.BagUserStateSpec> transformMap = Mockito.mock(Map.class);
+
+ final String userState1 = "userstate1";
+ ProcessBundleDescriptors.BagUserStateSpec bagUserStateSpec1 = mockBagUserState(userState1);
+ when(transformMap.get(userState1)).thenReturn(bagUserStateSpec1);
+
+ final String userState2 = "userstate2";
+ ProcessBundleDescriptors.BagUserStateSpec bagUserStateSpec2 = mockBagUserState(userState2);
+ when(transformMap.get(userState2)).thenReturn(bagUserStateSpec2);
+
+ when(userStateMapMock.get(anyString())).thenReturn(transformMap);
+ when(processBundleDescriptor.getBagUserStateSpecs()).thenReturn(userStateMapMock);
+ StateRequestHandler stateRequestHandler =
+ StateRequestHandlers.forBagUserStateHandlerFactory(
+ processBundleDescriptor, bagUserStateFactory);
+
+ // There should be no cache token available before any requests have been made
+ assertThat(stateRequestHandler.getCacheTokens(), iterableWithSize(0));
+
+ // Make a request to generate initial cache token
+ stateRequestHandler.handle(getRequest(key1, userState1));
+ BeamFnApi.ProcessBundleRequest.CacheToken cacheTokenStruct =
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens());
+ assertThat(cacheTokenStruct.hasUserState(), is(true));
+ ByteString cacheToken = cacheTokenStruct.getToken();
+ final ByteString expectedCacheToken =
+ ByteString.copyFrom(expectedToken.getBytes(Charsets.UTF_8));
+ assertThat(cacheToken, is(expectedCacheToken));
+
+ List<RequestGenerator> generators =
+ Arrays.asList(
+ ExecutableStageDoFnOperatorTest::getRequest,
+ ExecutableStageDoFnOperatorTest::getAppend,
+ ExecutableStageDoFnOperatorTest::getClear);
+
+ for (RequestGenerator req : generators) {
+ // For every state read the tokens remains unchanged
+ stateRequestHandler.handle(req.makeRequest(key1, userState1));
+ assertThat(
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens()).getToken(),
+ is(expectedCacheToken));
+
+ // The token is still valid for another key in the same key range
+ stateRequestHandler.handle(req.makeRequest(key2, userState1));
+ assertThat(
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens()).getToken(),
+ is(expectedCacheToken));
+
+ // The token is still valid for another state cell in the same key range
+ stateRequestHandler.handle(req.makeRequest(key2, userState2));
+ assertThat(
+ Iterables.getOnlyElement(stateRequestHandler.getCacheTokens()).getToken(),
+ is(expectedCacheToken));
+ }
+ }
+ }
+
+ private interface RequestGenerator {
+ BeamFnApi.StateRequest makeRequest(ByteString key, String userStateId) throws Exception;
+ }
+
+ private static BeamFnApi.StateRequest getRequest(ByteString key, String userStateId)
+ throws Exception {
+ BeamFnApi.StateRequest.Builder builder = stateRequest(key, userStateId);
+ builder.setGet(BeamFnApi.StateGetRequest.newBuilder().build());
+ return builder.build();
+ }
+
+ private static BeamFnApi.StateRequest getAppend(ByteString key, String userStateId)
+ throws Exception {
+ BeamFnApi.StateRequest.Builder builder = stateRequest(key, userStateId);
+ builder.setAppend(BeamFnApi.StateAppendRequest.newBuilder().build());
+ return builder.build();
+ }
+
+ private static BeamFnApi.StateRequest getClear(ByteString key, String userStateId)
+ throws Exception {
+ BeamFnApi.StateRequest.Builder builder = stateRequest(key, userStateId);
+ builder.setClear(BeamFnApi.StateClearRequest.newBuilder().build());
+ return builder.build();
+ }
+
+ private static BeamFnApi.StateRequest.Builder stateRequest(ByteString key, String userStateId)
+ throws Exception {
+ return BeamFnApi.StateRequest.newBuilder()
+ .setStateKey(
+ BeamFnApi.StateKey.newBuilder()
+ .setBagUserState(
+ BeamFnApi.StateKey.BagUserState.newBuilder()
+ .setTransformId("transform")
+ .setKey(key)
+ .setUserStateId(userStateId)
+ .setWindow(
+ ByteString.copyFrom(
+ CoderUtils.encodeToByteArray(
+ GlobalWindow.Coder.INSTANCE, GlobalWindow.INSTANCE)))
+ .build()));
+ }
+
+ private static ProcessBundleDescriptors.BagUserStateSpec mockBagUserState(String userStateId) {
+ ProcessBundleDescriptors.BagUserStateSpec bagUserStateMock =
+ Mockito.mock(ProcessBundleDescriptors.BagUserStateSpec.class);
+ when(bagUserStateMock.keyCoder()).thenReturn(ByteStringCoder.of());
+ when(bagUserStateMock.valueCoder()).thenReturn(ByteStringCoder.of());
+ when(bagUserStateMock.transformId()).thenReturn("transformId");
+ when(bagUserStateMock.userStateId()).thenReturn(userStateId);
+ when(bagUserStateMock.windowCoder()).thenReturn(GlobalWindow.Coder.INSTANCE);
+ return bagUserStateMock;
+ }
+
+ @Test
public void testSerialization() {
WindowedValue.ValueOnlyWindowedValueCoder<Integer> coder =
WindowedValue.getValueOnlyCoder(VarIntCoder.of());
@@ -683,7 +825,7 @@
options,
stagePayload,
jobInfo,
- FlinkExecutableStageContext.factory(options),
+ FlinkExecutableStageContextFactory.getInstance(),
createOutputMap(mainOutput, ImmutableList.of(additionalOutput)),
WindowingStrategy.globalDefault(),
null,
@@ -720,8 +862,8 @@
@Nullable Coder keyCoder,
@Nullable Coder windowedInputCoder) {
- FlinkExecutableStageContext.Factory contextFactory =
- Mockito.mock(FlinkExecutableStageContext.Factory.class);
+ FlinkExecutableStageContextFactory contextFactory =
+ Mockito.mock(FlinkExecutableStageContextFactory.class);
when(contextFactory.get(any())).thenReturn(stageContext);
final ExecutableStagePayload stagePayload;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java
index 06b5d01..274b2bf 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/FlinkKeyUtilsTest.java
@@ -21,11 +21,13 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.core.Is.is;
-import com.google.protobuf.ByteString;
import java.nio.ByteBuffer;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VoidCoder;
-import org.apache.beam.sdk.extensions.protobuf.ByteStringCoder;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.junit.Test;
/** Tests for {@link FlinkKeyUtils}. */
@@ -52,12 +54,20 @@
@Test
@SuppressWarnings("ByteBufferBackingArray")
public void testCoderContext() throws Exception {
- byte[] bytes = {1, 1, 1};
- ByteString key = ByteString.copyFrom(bytes);
- ByteStringCoder coder = ByteStringCoder.of();
+ String input = "hello world";
+ Coder<String> coder = StringUtf8Coder.of();
- ByteBuffer encoded = FlinkKeyUtils.encodeKey(key, coder);
- // Ensure outer context is used where no length encoding is used.
- assertThat(encoded.array(), is(bytes));
+ ByteBuffer encoded = FlinkKeyUtils.encodeKey(input, coder);
+ // Ensure NESTED context is used
+ assertThat(
+ encoded.array(), is(CoderUtils.encodeToByteArray(coder, input, Coder.Context.NESTED)));
+ }
+
+ @Test
+ @SuppressWarnings("ByteBufferBackingArray")
+ public void testFromEncodedKey() {
+ ByteString input = ByteString.copyFrom("hello world".getBytes(Charsets.UTF_8));
+ ByteBuffer encodedKey = FlinkKeyUtils.fromEncodedKey(input);
+ assertThat(encodedKey.array(), is(input.toByteArray()));
}
}
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/StreamRecordStripper.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/StreamRecordStripper.java
index c8c7b24..f63825f 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/StreamRecordStripper.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/StreamRecordStripper.java
@@ -19,8 +19,8 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
class StreamRecordStripper {
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperatorTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperatorTest.java
index 1307b98..0738ccb 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperatorTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/WindowDoFnOperatorTest.java
@@ -50,7 +50,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.typeutils.GenericTypeInfo;
import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
index 6b86bdc..eb868ed 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
@@ -18,29 +18,36 @@
package org.apache.beam.runners.flink.translation.wrappers.streaming.io;
import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.contains;
import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
+import static org.mockito.Mockito.when;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashSet;
+import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.CountDownLatch;
+import java.util.stream.LongStream;
+import org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.CountingSource;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.ValueWithRecordId;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.flink.api.common.ExecutionConfig;
import org.apache.flink.configuration.Configuration;
+import org.apache.flink.metrics.groups.UnregisteredMetricsGroup;
import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
@@ -53,6 +60,7 @@
import org.apache.flink.streaming.runtime.streamstatus.StreamStatus;
import org.apache.flink.streaming.runtime.streamstatus.StreamStatusMaintainer;
import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService;
import org.apache.flink.streaming.util.AbstractStreamOperatorTestHarness;
import org.apache.flink.util.InstantiationUtil;
import org.apache.flink.util.OutputTag;
@@ -690,6 +698,78 @@
}
assertThat(thread.isAlive(), is(false));
}
+
+ @Test
+ public void testSequentialReadingFromBoundedSource() throws Exception {
+ UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<Long> source =
+ new UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<>(
+ CountingSource.upTo(1000));
+
+ FlinkPipelineOptions options = PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+ options.setShutdownSourcesOnFinalWatermark(true);
+
+ UnboundedSourceWrapper<
+ Long, UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter.Checkpoint<Long>>
+ sourceWrapper = new UnboundedSourceWrapper<>("sequentialRead", options, source, 4);
+
+ StreamingRuntimeContext runtimeContextMock = Mockito.mock(StreamingRuntimeContext.class);
+ Mockito.when(runtimeContextMock.getIndexOfThisSubtask()).thenReturn(0);
+ when(runtimeContextMock.getNumberOfParallelSubtasks()).thenReturn(2);
+ when(runtimeContextMock.getExecutionConfig()).thenReturn(new ExecutionConfig());
+
+ TestProcessingTimeService processingTimeService = new TestProcessingTimeService();
+ processingTimeService.setCurrentTime(0);
+ when(runtimeContextMock.getProcessingTimeService()).thenReturn(processingTimeService);
+
+ when(runtimeContextMock.getMetricGroup()).thenReturn(new UnregisteredMetricsGroup());
+
+ sourceWrapper.setRuntimeContext(runtimeContextMock);
+
+ sourceWrapper.open(new Configuration());
+ assertThat(sourceWrapper.getLocalReaders().size(), is(2));
+
+ List<Long> integers = new ArrayList<>();
+ sourceWrapper.run(
+ new SourceFunction.SourceContext<WindowedValue<ValueWithRecordId<Long>>>() {
+ private final Object checkpointLock = new Object();
+
+ @Override
+ public void collect(WindowedValue<ValueWithRecordId<Long>> element) {
+ integers.add(element.getValue().getValue());
+ }
+
+ @Override
+ public void collectWithTimestamp(
+ WindowedValue<ValueWithRecordId<Long>> element, long timestamp) {
+ throw new IllegalStateException("Should not collect with timestamp");
+ }
+
+ @Override
+ public void emitWatermark(Watermark mark) {}
+
+ @Override
+ public void markAsTemporarilyIdle() {}
+
+ @Override
+ public Object getCheckpointLock() {
+ return checkpointLock;
+ }
+
+ @Override
+ public void close() {}
+ });
+
+ // The source is effectively split into two parts: The initial splitting is performed with a
+ // parallelism of 4, but there are 2 parallel subtasks. This instances taskes 2 out of 4
+ // partitions.
+ assertThat(integers.size(), is(500));
+ assertThat(
+ integers,
+ contains(
+ LongStream.concat(LongStream.range(0, 250), LongStream.range(500, 750))
+ .boxed()
+ .toArray()));
+ }
}
private static final class TestStreamStatusMaintainer implements StreamStatusMaintainer {
diff --git a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferedElementsTest.java b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferedElementsTest.java
index 65d11ce..9ebdefc 100644
--- a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferedElementsTest.java
+++ b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferedElementsTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/gearpump/build.gradle b/runners/gearpump/build.gradle
index 8bae691..c1744f7 100644
--- a/runners/gearpump/build.gradle
+++ b/runners/gearpump/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.gearpump')
description = "Apache Beam :: Runners :: Gearpump"
@@ -37,26 +37,26 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
+ compile library.java.vendored_guava_26_0_jre
compileOnly "com.typesafe:config:1.3.0"
compileOnly "org.scala-lang:scala-library:2.12.7"
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:core-java", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow "io.github.gearpump:gearpump-core_2.12:$gearpump_version:assembly"
- shadow "io.github.gearpump:gearpump-streaming_2.12:$gearpump_version:assembly"
- shadow library.java.joda_time
- shadow library.java.jackson_annotations
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest library.java.junit
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.jackson_databind
- shadowTest library.java.jackson_dataformat_yaml
- shadowTest library.java.mockito_core
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-java")
+ compile project(":runners:core-construction-java")
+ compile "io.github.gearpump:gearpump-core_2.12:$gearpump_version:assembly"
+ compile "io.github.gearpump:gearpump-streaming_2.12:$gearpump_version:assembly"
+ compile library.java.joda_time
+ compile library.java.jackson_annotations
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.jackson_databind
+ testCompile library.java.jackson_dataformat_yaml
+ testCompile library.java.mockito_core
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesRunner project(project.path)
}
task validatesRunnerStreaming(type: Test) {
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineOptions.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineOptions.java
index 28ff8a4..31ca0f7 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineOptions.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineOptions.java
@@ -48,7 +48,7 @@
"Whether the pipeline will be run on a remote cluster. If false, it will be run on a EmbeddedCluster")
void setRemote(Boolean remote);
- @Default.Boolean(true)
+ @Default.Boolean(false)
Boolean getRemote();
void setClientContext(ClientContext clientContext);
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrar.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrar.java
index 3e0fd86..2ba08bb 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrar.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
index 3bf6dc5..e8bc656 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/CreateStreamingGearpumpView.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Gearpump streaming overrides for various view (side input) transforms. */
class CreateStreamingGearpumpView<ElemT, ViewT>
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslator.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslator.java
index 12f1766..5c8253f 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslator.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslator.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** Flatten.FlattenPCollectionList is translated to Gearpump merge function. */
public class FlattenPCollectionsTranslator<T>
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GearpumpPipelineTranslator.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GearpumpPipelineTranslator.java
index 8ef4b53..249ced7 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GearpumpPipelineTranslator.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GearpumpPipelineTranslator.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
index 741f420..cc4ade9 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
@@ -43,8 +43,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
/** {@link GroupByKey} is translated to Gearpump groupBy function. */
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoMultiOutputTranslator.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoMultiOutputTranslator.java
index 4ee52dc..fb5202b 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoMultiOutputTranslator.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/ParDoMultiOutputTranslator.java
@@ -52,7 +52,7 @@
public void translate(ParDo.MultiOutput<InputT, OutputT> transform, TranslationContext context) {
PCollection<InputT> inputT = (PCollection<InputT>) context.getInput();
JavaStream<WindowedValue<InputT>> inputStream = context.getInputStream(inputT);
- Collection<PCollectionView<?>> sideInputs = transform.getSideInputs();
+ Collection<PCollectionView<?>> sideInputs = transform.getSideInputs().values();
Map<String, PCollectionView<?>> tagsToSideInputs =
TranslatorUtils.getTagsToSideInputs(sideInputs);
@@ -77,6 +77,9 @@
DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
+
JavaStream<TranslatorUtils.RawUnionValue> outputStream =
TranslatorUtils.toList(unionStream)
.flatMap(
@@ -89,7 +92,8 @@
mainOutput,
outputCoders,
sideOutputs,
- doFnSchemaInformation),
+ doFnSchemaInformation,
+ sideInputMapping),
transform.getName());
for (Map.Entry<TupleTag<?>, PValue> output : outputs.entrySet()) {
JavaStream<WindowedValue<OutputT>> taggedStream =
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
index a403fb4..c18ef94 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/TranslationContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.gearpump.translators;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import io.gearpump.cluster.UserConfig;
import io.gearpump.streaming.dsl.javaapi.JavaStream;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.runners.TransformHierarchy;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Maintains context data for {@link TransformTranslator}s. */
@SuppressWarnings({"rawtypes", "unchecked"})
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslator.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslator.java
index d6f826b..2f7e03e 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslator.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslator.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/** {@link Window.Assign} is translated to Gearpump flatMap function. */
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
index 4f7bb19..b55af69 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/functions/DoFnFunction.java
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Gearpump {@link FlatMapFunction} wrapper over Beam {@link DoFn}. */
@SuppressWarnings("unchecked")
@@ -74,7 +74,8 @@
TupleTag<OutputT> mainOutput,
Map<TupleTag<?>, Coder<?>> outputCoders,
List<TupleTag<?>> sideOutputs,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.doFn = doFn;
this.outputManager = new DoFnOutputManager();
this.doFnRunnerFactory =
@@ -88,7 +89,8 @@
new NoOpStepContext(),
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
this.sideInputs = sideInputs;
this.tagsToSideInputs = sideInputTagMapping;
this.mainOutput = mainOutput;
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
index 83c4b08..7a6dab8 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/DoFnRunnerFactory.java
@@ -50,6 +50,7 @@
private final List<TupleTag<?>> sideOutputTags;
private final StepContext stepContext;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
Map<TupleTag<?>, Coder<?>> outputCoders;
private final WindowingStrategy<?, ?> windowingStrategy;
@@ -63,7 +64,8 @@
StepContext stepContext,
Map<TupleTag<?>, Coder<?>> outputCoders,
WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.fn = doFn;
this.serializedOptions = new SerializablePipelineOptions(pipelineOptions);
this.sideInputs = sideInputs;
@@ -74,6 +76,7 @@
this.outputCoders = outputCoders;
this.windowingStrategy = windowingStrategy;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
public PushbackSideInputDoFnRunner<InputT, OutputT> createRunner(
@@ -91,7 +94,8 @@
null,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
return SimplePushbackSideInputDoFnRunner.create(underlying, sideInputs, sideInputReader);
}
}
diff --git a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
index 98b2043..6fc9897 100644
--- a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
+++ b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtils.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** Utility methods for translators. */
public class TranslatorUtils {
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrarTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrarTest.java
index f45d8e8..a605c6d 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrarTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/GearpumpRunnerRegistrarTest.java
@@ -21,7 +21,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
/** Tests for {@link GearpumpRunnerRegistrar}. */
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/PipelineOptionsTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/PipelineOptionsTest.java
index cf630d3..f756e46 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/PipelineOptionsTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/PipelineOptionsTest.java
@@ -28,7 +28,7 @@
import java.util.Map;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.junit.Test;
/** Tests for {@link GearpumpPipelineOptions}. */
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslatorTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslatorTest.java
index 13b8544..3eceb20 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslatorTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionsTranslatorTest.java
@@ -47,9 +47,9 @@
private FlattenPCollectionsTranslator translator = new FlattenPCollectionsTranslator();
private Flatten.PCollections transform = mock(Flatten.PCollections.class);
- private static class UnboundedSourceWrapperMatcher extends ArgumentMatcher<DataSource> {
+ private static class UnboundedSourceWrapperMatcher implements ArgumentMatcher<DataSource> {
@Override
- public boolean matches(Object o) {
+ public boolean matches(DataSource o) {
return o instanceof UnboundedSourceWrapper;
}
}
@@ -148,6 +148,6 @@
translator.translate(transform, translationContext);
verify(javaStream1).map(any(MapFunction.class), eq("dummy"));
- verify(javaStream1).merge(any(JavaStream.class), eq(1), eq(transformName));
+ verify(javaStream1).merge(eq(null), eq(1), eq(transformName));
}
}
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslatorTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslatorTest.java
index e749eaf..feaab42 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslatorTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslatorTest.java
@@ -35,8 +35,8 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadBoundedTranslatorTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadBoundedTranslatorTest.java
index b09d096..d1fb6cf 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadBoundedTranslatorTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadBoundedTranslatorTest.java
@@ -37,9 +37,9 @@
/** Tests for {@link ReadBoundedTranslator}. */
public class ReadBoundedTranslatorTest {
- private static class BoundedSourceWrapperMatcher extends ArgumentMatcher<DataSource> {
+ private static class BoundedSourceWrapperMatcher implements ArgumentMatcher<DataSource> {
@Override
- public boolean matches(Object o) {
+ public boolean matches(DataSource o) {
return o instanceof BoundedSourceWrapper;
}
}
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadUnboundedTranslatorTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadUnboundedTranslatorTest.java
index b828f9c..d82fdd0 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadUnboundedTranslatorTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/ReadUnboundedTranslatorTest.java
@@ -37,9 +37,9 @@
/** Tests for {@link ReadUnboundedTranslator}. */
public class ReadUnboundedTranslatorTest {
- private static class UnboundedSourceWrapperMatcher extends ArgumentMatcher<DataSource> {
+ private static class UnboundedSourceWrapperMatcher implements ArgumentMatcher<DataSource> {
@Override
- public boolean matches(Object o) {
+ public boolean matches(DataSource o) {
return o instanceof UnboundedSourceWrapper;
}
}
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslatorTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslatorTest.java
index bfe5c01..ee7e347 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslatorTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/WindowAssignTranslatorTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSourceTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSourceTest.java
index 7f54fa9..a30a109 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSourceTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSourceTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/ValueSoureTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/ValueSoureTest.java
index bd71838..ddb6f50 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/ValueSoureTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/io/ValueSoureTest.java
@@ -33,8 +33,8 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtilsTest.java b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtilsTest.java
index 24cc966..b2721de 100644
--- a/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtilsTest.java
+++ b/runners/gearpump/src/test/java/org/apache/beam/runners/gearpump/translators/utils/TranslatorUtilsTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
/** Tests for {@link TranslatorUtils}. */
diff --git a/runners/google-cloud-dataflow-java/build.gradle b/runners/google-cloud-dataflow-java/build.gradle
index 47d91fa..1569d29 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.dataflow')
description = "Apache Beam :: Runners :: Google Cloud Dataflow"
@@ -39,7 +39,7 @@
filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
'dataflow.legacy_environment_major_version' : '7',
'dataflow.fnapi_environment_major_version' : '7',
- 'dataflow.container_version' : 'beam-master-20190415'
+ 'dataflow.container_version' : 'beam-master-20190829'
]
}
@@ -47,6 +47,9 @@
test {
systemProperty "beamTestPipelineOptions", ""
systemProperty "beamUseDummyRunner", "true"
+ useJUnit {
+ excludeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ }
}
configurations {
@@ -57,58 +60,55 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.google_api_client
- shadow library.java.google_http_client
- shadow library.java.google_http_client_jackson2
- shadow library.java.google_api_services_dataflow
- shadow library.java.google_api_services_clouddebugger
- shadow library.java.google_api_services_storage
- shadow library.java.google_auth_library_credentials
- shadow library.java.google_auth_library_oauth2_http
- shadow library.java.bigdataoss_util
- shadow library.java.avro
- shadow library.java.joda_time
- shadow library.java.jackson_core
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
- shadow library.java.slf4j_api
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadowTest")
- shadowTest library.java.guava_testlib
- shadowTest library.java.slf4j_jdk14
- shadowTest library.java.mockito_core
- shadowTest library.java.google_cloud_dataflow_java_proto_library_all
- shadowTest library.java.datastore_v1_protos
- shadowTest library.java.jackson_dataformat_yaml
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:io:google-cloud-platform")
+ compile project(":runners:core-construction-java")
+ compile library.java.avro
+ compile library.java.bigdataoss_util
+ compile library.java.google_api_client
+ compile library.java.google_api_services_clouddebugger
+ compile library.java.google_api_services_dataflow
+ compile library.java.google_api_services_storage
+ compile library.java.google_auth_library_credentials
+ compile library.java.google_auth_library_oauth2_http
+ compile library.java.google_http_client
+ compile library.java.google_http_client_jackson2
+ compile library.java.jackson_annotations
+ compile library.java.jackson_core
+ compile library.java.jackson_databind
+ compile library.java.joda_time
+ compile library.java.slf4j_api
+ compile library.java.vendored_grpc_1_21_0
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile project(path: ":sdks:java:io:google-cloud-platform", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntime")
+ testCompile library.java.google_cloud_dataflow_java_proto_library_all
+ testCompile library.java.jackson_dataformat_yaml
+ testCompile library.java.mockito_core
+ testCompile library.java.proto_google_cloud_datastore_v1
+ testCompile library.java.slf4j_jdk14
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
+ validatesRunner project(project.path)
+ validatesRunner project(path: project.path, configuration: "testRuntime")
validatesRunner library.java.hamcrest_core
validatesRunner library.java.hamcrest_library
- coreSDKJavaIntegrationTest project(path: project.path, configuration: "shadow")
+ coreSDKJavaIntegrationTest project(project.path)
coreSDKJavaIntegrationTest project(path: ":sdks:java:core", configuration: "shadowTest")
- examplesJavaIntegrationTest project(path: project.path, configuration: "shadow")
- examplesJavaIntegrationTest project(path: ":examples:java", configuration: "shadowTest")
- googleCloudPlatformIntegrationTest project(path: project.path, configuration: "shadow")
- googleCloudPlatformIntegrationTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadowTest")
-}
-
-test {
- systemProperties = [ "beamUseDummyRunner" : "true" ]
+ examplesJavaIntegrationTest project(project.path)
+ examplesJavaIntegrationTest project(path: ":examples:java", configuration: "testRuntime")
+ googleCloudPlatformIntegrationTest project(project.path)
+ googleCloudPlatformIntegrationTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "testRuntime")
}
def dataflowProject = project.findProperty('dataflowProject') ?: 'apache-beam-testing'
def dataflowValidatesTempRoot = project.findProperty('dataflowTempRoot') ?: 'gs://temp-storage-for-validates-runner-tests'
def dataflowPostCommitTempRoot = project.findProperty('dataflowTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests'
+def dataflowPostCommitTempRootKms = project.findProperty('dataflowTempRootKms') ?: 'gs://temp-storage-for-end-to-end-tests-cmek'
def dataflowUploadTemp = project.findProperty('dataflowTempRoot') ?: 'gs://temp-storage-for-upload-tests'
def testFilesToStage = project.findProperty('filesToStage') ?: 'test.txt'
def dataflowLegacyWorkerJar = project.findProperty('dataflowWorkerJar') ?: project(":runners:google-cloud-dataflow-java:worker:legacy-worker").shadowJar.archivePath
@@ -170,7 +170,8 @@
// to the number of CPU cores, but can be increased by setting --max-workers=N.
maxParallelForks Integer.MAX_VALUE
classpath = configurations.validatesRunner
- testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs) +
+ files(project(project.path).sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
commonExcludeCategories.each {
@@ -187,6 +188,7 @@
"--project=${dataflowProject}",
"--tempRoot=${dataflowValidatesTempRoot}",
"--dataflowWorkerJar=${dataflowLegacyWorkerJar}",
+ "--workerHarnessContainerImage=",
])
systemProperty "java.specification.version", "11"
@@ -195,7 +197,8 @@
// to the number of CPU cores, but can be increased by setting --max-workers=N.
maxParallelForks Integer.MAX_VALUE
classpath = configurations.validatesRunner
- testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs) +
+ files(project(project.path).sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
commonExcludeCategories.each {
@@ -211,7 +214,9 @@
task buildAndPushDockerContainer() {
dependsOn ":sdks:java:container:docker"
finalizedBy 'cleanUpDockerImages'
- def defaultDockerImageName = containerImageName(name: "java")
+ def defaultDockerImageName = containerImageName(
+ name: "java_sdk",
+ root: "apachebeam")
doLast {
exec {
commandLine "docker", "tag", "${defaultDockerImageName}", "${dockerImageName}"
@@ -258,7 +263,8 @@
// to the number of CPU cores, but can be increased by setting --max-workers=N.
maxParallelForks Integer.MAX_VALUE
classpath = configurations.validatesRunner
- testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs) +
+ files(project(project.path).sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
commonExcludeCategories.each {
@@ -289,7 +295,8 @@
// to the number of CPU cores, but can be increased by setting --max-workers=N.
maxParallelForks Integer.MAX_VALUE
classpath = configurations.validatesRunner
- testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs) +
+ files(project(project.path).sourceSets.test.output.classesDirs)
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
commonExcludeCategories.each {
@@ -330,11 +337,12 @@
exclude '**/BigQueryIOReadIT.class'
exclude '**/BigQueryIOStorageReadTableRowIT.class'
exclude '**/PubsubReadIT.class'
- exclude '**/*KmsKeyIT.class'
maxParallelForks 4
classpath = configurations.googleCloudPlatformIntegrationTest
testClassesDirs = files(project(":sdks:java:io:google-cloud-platform").sourceSets.test.output.classesDirs)
- useJUnit { }
+ useJUnit {
+ excludeCategories "org.apache.beam.sdk.testing.UsesKms"
+ }
}
task googleCloudPlatformLegacyWorkerKmsIntegrationTest(type: Test) {
@@ -343,18 +351,20 @@
systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
"--runner=TestDataflowRunner",
"--project=${dataflowProject}",
- "--tempRoot=${dataflowPostCommitTempRoot}",
+ "--tempRoot=${dataflowPostCommitTempRootKms}",
"--dataflowWorkerJar=${dataflowLegacyWorkerJar}",
"--workerHarnessContainerImage=",
"--dataflowKmsKey=${dataflowKmsKey}",
])
- include '**/*KmsKeyIT.class'
+ include '**/*IT.class'
exclude '**/BigQueryKmsKeyIT.class' // Only needs to run on direct runner.
maxParallelForks 4
classpath = configurations.googleCloudPlatformIntegrationTest
testClassesDirs = files(project(":sdks:java:io:google-cloud-platform").sourceSets.test.output.classesDirs)
- useJUnit { }
+ useJUnit {
+ includeCategories "org.apache.beam.sdk.testing.UsesKms"
+ }
}
task googleCloudPlatformFnApiWorkerIntegrationTest(type: Test) {
diff --git a/runners/google-cloud-dataflow-java/examples-streaming/build.gradle b/runners/google-cloud-dataflow-java/examples-streaming/build.gradle
index 7d67036..c32481f 100644
--- a/runners/google-cloud-dataflow-java/examples-streaming/build.gradle
+++ b/runners/google-cloud-dataflow-java/examples-streaming/build.gradle
@@ -28,9 +28,9 @@
configurations { dataflowStreamingRunnerPreCommit }
dependencies {
- testRuntimeOnly project(path: ":examples:java", configuration: "shadow")
- testRuntimeOnly project(path: ":examples:java", configuration: "shadowTest")
- testRuntimeOnly project(path: ":runners:google-cloud-dataflow-java", configuration: "shadow")
+ testRuntimeOnly project(":examples:java")
+ testRuntimeOnly project(path: ":examples:java", configuration: "testRuntime")
+ testRuntimeOnly project(":runners:google-cloud-dataflow-java")
}
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests/'
diff --git a/runners/google-cloud-dataflow-java/examples/build.gradle b/runners/google-cloud-dataflow-java/examples/build.gradle
index f73c75a..8054488 100644
--- a/runners/google-cloud-dataflow-java/examples/build.gradle
+++ b/runners/google-cloud-dataflow-java/examples/build.gradle
@@ -31,9 +31,9 @@
configurations { dataflowRunnerPreCommit }
dependencies {
- testRuntimeOnly project(path: ":examples:java", configuration: "shadow")
- testRuntimeOnly project(path: ":examples:java", configuration: "shadowTest")
- testRuntimeOnly project(path: ":runners:google-cloud-dataflow-java", configuration: "shadow")
+ testRuntimeOnly project(":examples:java")
+ testRuntimeOnly project(path: ":examples:java", configuration: "testRuntime")
+ testRuntimeOnly project(":runners:google-cloud-dataflow-java")
}
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverrides.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverrides.java
index ff0590b..82cf7538 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverrides.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverrides.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.List;
import java.util.Map;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
index fd90b66..daa5dba 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/BatchViewOverrides.java
@@ -17,8 +17,9 @@
*/
package org.apache.beam.runners.dataflow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
@@ -31,6 +32,7 @@
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import javax.annotation.Nonnull;
import org.apache.beam.runners.dataflow.internal.IsmFormat;
import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
@@ -73,15 +75,15 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ForwardingMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ForwardingMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.joda.time.Instant;
/**
@@ -1154,8 +1156,11 @@
return (WindowedValueToValue) INSTANCE;
}
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public V apply(WindowedValue<V> input) {
+ public V apply(@Nonnull WindowedValue<V> input) {
return input.getValue();
}
}
@@ -1173,8 +1178,11 @@
return (IterableWithWindowedValuesToIterable) INSTANCE;
}
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public Iterable<V> apply(Iterable<WindowedValue<V>> input) {
+ public Iterable<V> apply(@Nonnull Iterable<WindowedValue<V>> input) {
return Iterables.transform(input, WindowedValueToValue.of());
}
}
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
index 31c3a37..abbda69 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.dataflow.Dataflow;
import com.google.api.services.dataflow.Dataflow.Projects.Locations.Jobs;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
index 57b92cb..a7177e2 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowMetrics.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import com.google.api.client.util.ArrayMap;
import com.google.api.services.dataflow.model.JobMetrics;
@@ -37,8 +37,8 @@
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
index b4eeab3..a4a4700 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.toStringHelper;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.toStringHelper;
import java.util.ArrayDeque;
import org.apache.beam.sdk.Pipeline;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index 049a904..8d67799 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow;
import static org.apache.beam.runners.dataflow.util.TimeUtil.fromCloudTime;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.util.BackOff;
@@ -38,21 +38,23 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import org.apache.beam.runners.dataflow.util.MonitoringUtil;
+import org.apache.beam.runners.dataflow.util.MonitoringUtil.JobMessagesHandler;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
import org.apache.beam.sdk.metrics.MetricResults;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/** A DataflowPipelineJob represents a job submitted to Dataflow using {@link DataflowRunner}. */
public class DataflowPipelineJob implements PipelineResult {
+
private static final Logger LOG = LoggerFactory.getLogger(DataflowPipelineJob.class);
/** The id for the job. */
@@ -92,6 +94,8 @@
static final Duration STATUS_POLLING_INTERVAL = Duration.standardSeconds(2);
+ static final Duration DEFAULT_MAX_BACKOFF = Duration.standardMinutes(2);
+
static final double DEFAULT_BACKOFF_EXPONENT = 1.5;
/** The amount of polling retries for job status and messages information. */
@@ -103,7 +107,9 @@
FluentBackoff.DEFAULT
.withInitialBackoff(MESSAGES_POLLING_INTERVAL)
.withMaxRetries(MESSAGES_POLLING_RETRIES)
- .withExponent(DEFAULT_BACKOFF_EXPONENT);
+ .withExponent(DEFAULT_BACKOFF_EXPONENT)
+ .withMaxBackoff(DEFAULT_MAX_BACKOFF);
+
protected static final FluentBackoff STATUS_BACKOFF_FACTORY =
FluentBackoff.DEFAULT
.withInitialBackoff(STATUS_POLLING_INTERVAL)
@@ -238,6 +244,16 @@
duration, messageHandler, sleeper, nanoClock, new MonitoringUtil(dataflowClient));
}
+ private static BackOff getMessagesBackoff(Duration duration) {
+ FluentBackoff factory = MESSAGES_BACKOFF_FACTORY;
+
+ if (!duration.isShorterThan(Duration.ZERO)) {
+ factory = factory.withMaxCumulativeBackoff(duration);
+ }
+
+ return BackOffAdapter.toGcpBackOff(factory.backoff());
+ }
+
/**
* Waits until the pipeline finishes and returns the final status.
*
@@ -261,96 +277,128 @@
MonitoringUtil monitor)
throws IOException, InterruptedException {
- BackOff backoff;
- if (!duration.isLongerThan(Duration.ZERO)) {
- backoff = BackOffAdapter.toGcpBackOff(MESSAGES_BACKOFF_FACTORY.backoff());
- } else {
- backoff =
- BackOffAdapter.toGcpBackOff(
- MESSAGES_BACKOFF_FACTORY.withMaxCumulativeBackoff(duration).backoff());
- }
+ BackOff backoff = getMessagesBackoff(duration);
// This function tracks the cumulative time from the *first request* to enforce the wall-clock
// limit. Any backoff instance could, at best, track the the time since the first attempt at a
// given request. Thus, we need to track the cumulative time ourselves.
long startNanos = nanoClock.nanoTime();
- State state;
+ State state = State.UNKNOWN;
+ Exception exception;
do {
- // Get the state of the job before listing messages. This ensures we always fetch job
- // messages after the job finishes to ensure we have all them.
- state =
- getStateWithRetries(
- BackOffAdapter.toGcpBackOff(STATUS_BACKOFF_FACTORY.withMaxRetries(0).backoff()),
- sleeper);
- boolean hasError = state == State.UNKNOWN;
-
- if (messageHandler != null && !hasError) {
- // Process all the job messages that have accumulated so far.
- try {
- List<JobMessage> allMessages = monitor.getJobMessages(getJobId(), lastTimestamp);
-
- if (!allMessages.isEmpty()) {
- lastTimestamp =
- fromCloudTime(allMessages.get(allMessages.size() - 1).getTime()).getMillis();
- messageHandler.process(allMessages);
- }
- } catch (GoogleJsonResponseException | SocketTimeoutException e) {
- hasError = true;
- LOG.warn("There were problems getting current job messages: {}.", e.getMessage());
- LOG.debug("Exception information:", e);
- }
+ exception = null;
+ try {
+ // Get the state of the job before listing messages. This ensures we always fetch job
+ // messages after the job finishes to ensure we have all them.
+ state =
+ getStateWithRetries(
+ BackOffAdapter.toGcpBackOff(STATUS_BACKOFF_FACTORY.withMaxRetries(0).backoff()),
+ sleeper);
+ } catch (IOException e) {
+ exception = e;
+ LOG.warn("Failed to get job state: {}", e.getMessage());
+ LOG.debug("Failed to get job state: {}", e);
+ continue;
}
- if (!hasError) {
- // We can stop if the job is done.
- if (state.isTerminal()) {
- switch (state) {
- case DONE:
- case CANCELLED:
- LOG.info("Job {} finished with status {}.", getJobId(), state);
- break;
- case UPDATED:
- LOG.info(
- "Job {} has been updated and is running as the new job with id {}. "
- + "To access the updated job on the Dataflow monitoring console, "
- + "please navigate to {}",
- getJobId(),
- getReplacedByJob().getJobId(),
- MonitoringUtil.getJobMonitoringPageURL(
- getReplacedByJob().getProjectId(),
- getRegion(),
- getReplacedByJob().getJobId()));
- break;
- default:
- LOG.info("Job {} failed with status {}.", getJobId(), state);
- }
- return state;
- }
+ exception = processJobMessages(messageHandler, monitor);
- // The job is not done, so we must keep polling.
- backoff.reset();
-
- // If a total duration for all backoff has been set, update the new cumulative sleep time to
- // be the remaining total backoff duration, stopping if we have already exceeded the
- // allotted time.
- if (duration.isLongerThan(Duration.ZERO)) {
- long nanosConsumed = nanoClock.nanoTime() - startNanos;
- Duration consumed = Duration.millis((nanosConsumed + 999999) / 1000000);
- Duration remaining = duration.minus(consumed);
- if (remaining.isLongerThan(Duration.ZERO)) {
- backoff =
- BackOffAdapter.toGcpBackOff(
- MESSAGES_BACKOFF_FACTORY.withMaxCumulativeBackoff(remaining).backoff());
- } else {
- // If there is no time remaining, don't bother backing off.
- backoff = BackOff.STOP_BACKOFF;
- }
- }
+ if (exception != null) {
+ continue;
}
+
+ // We can stop if the job is done.
+ if (state.isTerminal()) {
+ logTerminalState(state);
+ return state;
+ }
+
+ // Reset attempts count and update cumulative wait time.
+ backoff = resetBackoff(duration, nanoClock, startNanos);
} while (BackOffUtils.next(sleeper, backoff));
- LOG.warn("No terminal state was returned. State value {}", state);
- return null; // Timed out.
+
+ // At this point Backoff decided that we retried enough times.
+ // This can be either due to exceeding allowed timeout for job to complete, or receiving
+ // error multiple times in a row.
+
+ if (exception == null) {
+ LOG.warn("No terminal state was returned within allotted timeout. State value {}", state);
+ } else {
+ LOG.error("Failed to fetch job metadata with error: {}", exception);
+ }
+
+ return null;
+ }
+
+ private void logTerminalState(State state) {
+ switch (state) {
+ case DONE:
+ case CANCELLED:
+ LOG.info("Job {} finished with status {}.", getJobId(), state);
+ break;
+ case UPDATED:
+ LOG.info(
+ "Job {} has been updated and is running as the new job with id {}. "
+ + "To access the updated job on the Dataflow monitoring console, "
+ + "please navigate to {}",
+ getJobId(),
+ getReplacedByJob().getJobId(),
+ MonitoringUtil.getJobMonitoringPageURL(
+ getReplacedByJob().getProjectId(), getRegion(), getReplacedByJob().getJobId()));
+ break;
+ default:
+ LOG.info("Job {} failed with status {}.", getJobId(), state);
+ }
+ }
+
+ /**
+ * Reset backoff. If duration is limited, calculate time remaining, otherwise just reset retry
+ * count.
+ *
+ * <p>If a total duration for all backoff has been set, update the new cumulative sleep time to be
+ * the remaining total backoff duration, stopping if we have already exceeded the allotted time.
+ */
+ private static BackOff resetBackoff(Duration duration, NanoClock nanoClock, long startNanos) {
+ BackOff backoff;
+ if (duration.isLongerThan(Duration.ZERO)) {
+ long nanosConsumed = nanoClock.nanoTime() - startNanos;
+ Duration consumed = Duration.millis((nanosConsumed + 999999) / 1000000);
+ Duration remaining = duration.minus(consumed);
+ if (remaining.isLongerThan(Duration.ZERO)) {
+ backoff = getMessagesBackoff(remaining);
+ } else {
+ backoff = BackOff.STOP_BACKOFF;
+ }
+ } else {
+ backoff = getMessagesBackoff(duration);
+ }
+ return backoff;
+ }
+
+ /**
+ * Process all the job messages that have accumulated so far.
+ *
+ * @return Exception that caused failure to process messages or null.
+ */
+ private Exception processJobMessages(
+ @Nullable JobMessagesHandler messageHandler, MonitoringUtil monitor) throws IOException {
+ if (messageHandler != null) {
+ try {
+ List<JobMessage> allMessages = monitor.getJobMessages(getJobId(), lastTimestamp);
+
+ if (!allMessages.isEmpty()) {
+ lastTimestamp =
+ fromCloudTime(allMessages.get(allMessages.size() - 1).getTime()).getMillis();
+ messageHandler.process(allMessages);
+ }
+ } catch (GoogleJsonResponseException | SocketTimeoutException e) {
+ LOG.warn("Failed to get job messages: {}", e.getMessage());
+ LOG.debug("Failed to get job messages: {}", e);
+ return e;
+ }
+ }
+ return null;
}
private AtomicReference<FutureTask<State>> cancelState = new AtomicReference<>();
@@ -372,10 +420,11 @@
() -> {
Job content = new Job();
content.setProjectId(getProjectId());
- content.setId(jobId);
+ String currentJobId = getJobId();
+ content.setId(currentJobId);
content.setRequestedState("JOB_STATE_CANCELLED");
try {
- Job job = dataflowClient.updateJob(getJobId(), content);
+ Job job = dataflowClient.updateJob(currentJobId, content);
return MonitoringUtil.toState(job.getCurrentState());
} catch (IOException e) {
State state = getState();
@@ -426,7 +475,7 @@
return terminalState;
}
- return getStateWithRetries(
+ return getStateWithRetriesOrUnknownOnException(
BackOffAdapter.toGcpBackOff(STATUS_BACKOFF_FACTORY.backoff()), Sleeper.DEFAULT);
}
@@ -439,13 +488,9 @@
* @return The state of the job or State.UNKNOWN in case of failure.
*/
@VisibleForTesting
- State getStateWithRetries(BackOff attempts, Sleeper sleeper) {
- if (terminalState != null) {
- return terminalState;
- }
+ State getStateWithRetriesOrUnknownOnException(BackOff attempts, Sleeper sleeper) {
try {
- Job job = getJobWithRetries(attempts, sleeper);
- return MonitoringUtil.toState(job.getCurrentState());
+ return getStateWithRetries(attempts, sleeper);
} catch (IOException exn) {
// The only IOException that getJobWithRetries is permitted to throw is the final IOException
// that caused the failure of retry. Other exceptions are wrapped in an unchecked exceptions
@@ -454,6 +499,14 @@
}
}
+ State getStateWithRetries(BackOff attempts, Sleeper sleeper) throws IOException {
+ if (terminalState != null) {
+ return terminalState;
+ }
+ Job job = getJobWithRetries(attempts, sleeper);
+ return MonitoringUtil.toState(job.getCurrentState());
+ }
+
/**
* Attempts to get the underlying {@link Job}. Uses exponential backoff on failure up to the
* maximum number of passed in attempts.
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.java
index b9b4dca..f4baa5c 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrar.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Contains the {@link PipelineOptionsRegistrar} and {@link PipelineRunnerRegistrar} for the {@link
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 93e6453..41e5cbb 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -27,10 +27,10 @@
import static org.apache.beam.sdk.options.ExperimentalOptions.hasExperiment;
import static org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray;
import static org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings.isNullOrEmpty;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings.isNullOrEmpty;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
@@ -74,6 +74,7 @@
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.StreamingOptions;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.runners.TransformHierarchy;
@@ -103,10 +104,10 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -121,10 +122,17 @@
private static final Logger LOG = LoggerFactory.getLogger(DataflowPipelineTranslator.class);
private static final ObjectMapper MAPPER = new ObjectMapper();
- private static byte[] serializeWindowingStrategy(WindowingStrategy<?, ?> windowingStrategy) {
+ private static byte[] serializeWindowingStrategy(
+ WindowingStrategy<?, ?> windowingStrategy, PipelineOptions options) {
try {
SdkComponents sdkComponents = SdkComponents.create();
- sdkComponents.registerEnvironment(Environments.JAVA_SDK_HARNESS_ENVIRONMENT);
+
+ String workerHarnessContainerImageURL =
+ DataflowRunner.getContainerImageForJob(options.as(DataflowPipelineOptions.class));
+ RunnerApi.Environment defaultEnvironmentForDataflow =
+ Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+ sdkComponents.registerEnvironment(defaultEnvironmentForDataflow);
+
return WindowingStrategyTranslation.toMessageProto(windowingStrategy, sdkComponents)
.toByteArray();
} catch (Exception e) {
@@ -164,7 +172,13 @@
// Capture the sdkComponents for look up during step translations
SdkComponents sdkComponents = SdkComponents.create();
- sdkComponents.registerEnvironment(Environments.JAVA_SDK_HARNESS_ENVIRONMENT);
+
+ String workerHarnessContainerImageURL =
+ DataflowRunner.getContainerImageForJob(options.as(DataflowPipelineOptions.class));
+ RunnerApi.Environment defaultEnvironmentForDataflow =
+ Environments.createDockerEnvironment(workerHarnessContainerImageURL);
+ sdkComponents.registerEnvironment(defaultEnvironmentForDataflow);
+
RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline, sdkComponents, true);
LOG.debug("Portable pipeline proto:\n{}", TextFormat.printToString(pipelineProto));
@@ -329,6 +343,8 @@
List<String> experiments = options.getExperiments();
if (experiments == null) {
experiments = new ArrayList<String>();
+ } else {
+ experiments = new ArrayList<String>(experiments);
}
if (!experiments.contains(GcpOptions.STREAMING_ENGINE_EXPERIMENT)) {
experiments.add(GcpOptions.STREAMING_ENGINE_EXPERIMENT);
@@ -752,7 +768,8 @@
WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
stepContext.addInput(
PropertyNames.WINDOWING_STRATEGY,
- byteArrayToJsonString(serializeWindowingStrategy(windowingStrategy)));
+ byteArrayToJsonString(
+ serializeWindowingStrategy(windowingStrategy, context.getPipelineOptions())));
stepContext.addInput(
PropertyNames.IS_MERGING_WINDOW_FN,
!windowingStrategy.getWindowFn().isNonMerging());
@@ -896,7 +913,8 @@
stepContext.addInput(PropertyNames.DISALLOW_COMBINER_LIFTING, !allowCombinerLifting);
stepContext.addInput(
PropertyNames.SERIALIZED_FN,
- byteArrayToJsonString(serializeWindowingStrategy(windowingStrategy)));
+ byteArrayToJsonString(
+ serializeWindowingStrategy(windowingStrategy, context.getPipelineOptions())));
stepContext.addInput(
PropertyNames.IS_MERGING_WINDOW_FN,
!windowingStrategy.getWindowFn().isNonMerging());
@@ -917,14 +935,18 @@
DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation =
ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
-
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
Map<TupleTag<?>, Coder<?>> outputCoders =
context.getOutputs(transform).entrySet().stream()
.collect(
Collectors.toMap(
Map.Entry::getKey, e -> ((PCollection) e.getValue()).getCoder()));
translateInputs(
- stepContext, context.getInput(transform), transform.getSideInputs(), context);
+ stepContext,
+ context.getInput(transform),
+ transform.getSideInputs().values(),
+ context);
translateOutputs(context.getOutputs(transform), stepContext);
String ptransformId =
context.getSdkComponents().getPTransformIdOrThrow(context.getCurrentTransform());
@@ -933,12 +955,13 @@
ptransformId,
transform.getFn(),
context.getInput(transform).getWindowingStrategy(),
- transform.getSideInputs(),
+ transform.getSideInputs().values(),
context.getInput(transform).getCoder(),
context,
transform.getMainOutputTag(),
outputCoders,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
// TODO: Move this logic into translateFn once the legacy ProcessKeyedElements is
// removed.
@@ -970,7 +993,8 @@
DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation =
ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
-
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
StepTranslationContext stepContext = context.addStep(transform, "ParallelDo");
Map<TupleTag<?>, Coder<?>> outputCoders =
context.getOutputs(transform).entrySet().stream()
@@ -979,7 +1003,10 @@
Map.Entry::getKey, e -> ((PCollection) e.getValue()).getCoder()));
translateInputs(
- stepContext, context.getInput(transform), transform.getSideInputs(), context);
+ stepContext,
+ context.getInput(transform),
+ transform.getSideInputs().values(),
+ context);
stepContext.addOutput(
transform.getMainOutputTag().getId(), context.getOutput(transform));
String ptransformId =
@@ -989,12 +1016,13 @@
ptransformId,
transform.getFn(),
context.getInput(transform).getWindowingStrategy(),
- transform.getSideInputs(),
+ transform.getSideInputs().values(),
context.getInput(transform).getCoder(),
context,
transform.getMainOutputTag(),
outputCoders,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
// TODO: Move this logic into translateFn once the legacy ProcessKeyedElements is
// removed.
@@ -1027,7 +1055,8 @@
stepContext.addOutput(PropertyNames.OUTPUT, context.getOutput(transform));
WindowingStrategy<?, ?> strategy = context.getOutput(transform).getWindowingStrategy();
- byte[] serializedBytes = serializeWindowingStrategy(strategy);
+ byte[] serializedBytes =
+ serializeWindowingStrategy(strategy, context.getPipelineOptions());
String serializedJson = byteArrayToJsonString(serializedBytes);
stepContext.addInput(PropertyNames.SERIALIZED_FN, serializedJson);
}
@@ -1056,7 +1085,8 @@
DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation =
ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
-
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
StepTranslationContext stepContext =
context.addStep(transform, "SplittableProcessKeyed");
Map<TupleTag<?>, Coder<?>> outputCoders =
@@ -1079,7 +1109,8 @@
context,
transform.getMainOutputTag(),
outputCoders,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
stepContext.addInput(
PropertyNames.RESTRICTION_CODER,
@@ -1091,7 +1122,7 @@
private static void translateInputs(
StepTranslationContext stepContext,
PCollection<?> input,
- List<PCollectionView<?>> sideInputs,
+ Iterable<PCollectionView<?>> sideInputs,
TranslationContext context) {
stepContext.addInput(PropertyNames.PARALLEL_INPUT, input);
translateSideInputs(stepContext, sideInputs, context);
@@ -1100,7 +1131,7 @@
// Used for ParDo
private static void translateSideInputs(
StepTranslationContext stepContext,
- List<PCollectionView<?>> sideInputs,
+ Iterable<PCollectionView<?>> sideInputs,
TranslationContext context) {
Map<String, Object> nonParInputs = new HashMap<>();
@@ -1123,7 +1154,8 @@
TranslationContext context,
TupleTag<?> mainOutput,
Map<TupleTag<?>, Coder<?>> outputCoders,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass());
if (signature.usesState() || signature.usesTimers()) {
@@ -1149,7 +1181,8 @@
inputCoder,
outputCoders,
mainOutput,
- doFnSchemaInformation))));
+ doFnSchemaInformation,
+ sideInputMapping))));
}
// Setting USES_KEYED_STATE will cause an ungrouped shuffle, which works
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index d1c3edf..a13d87b 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -22,10 +22,10 @@
import static org.apache.beam.sdk.util.CoderUtils.encodeToByteArray;
import static org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray;
import static org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings.isNullOrEmpty;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings.isNullOrEmpty;
import com.fasterxml.jackson.databind.Module;
import com.fasterxml.jackson.databind.ObjectMapper;
@@ -48,6 +48,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.HashSet;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Random;
@@ -99,6 +100,7 @@
import org.apache.beam.sdk.io.WriteFilesResult;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessageWithAttributesAndMessageIdCoder;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessageWithAttributesCoder;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource;
@@ -118,6 +120,10 @@
import org.apache.beam.sdk.transforms.Combine.GroupedValues;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.ProcessContext;
+import org.apache.beam.sdk.transforms.DoFn.ProcessElement;
+import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.GroupIntoBatches;
import org.apache.beam.sdk.transforms.Impulse;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
@@ -150,12 +156,13 @@
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Utf8;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Utf8;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.joda.time.DateTimeUtils;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
@@ -238,6 +245,14 @@
"Missing required values: " + Joiner.on(',').join(missing));
}
+ if (dataflowOptions.getRegion() == null) {
+ dataflowOptions.setRegion("us-central1");
+ LOG.warn(
+ "--region not set; will default to us-central1. Future releases of Beam will "
+ + "require the user to set the region explicitly. "
+ + "https://cloud.google.com/compute/docs/regions-zones/regions-zones");
+ }
+
PathValidator validator = dataflowOptions.getPathValidator();
String gcpTempLocation;
try {
@@ -415,6 +430,14 @@
// natively in the Dataflow service.
} else {
overridesBuilder
+ // Replace GroupIntoBatches before the state/timer replacements below since
+ // GroupIntoBatches internally uses a stateful DoFn.
+ .add(
+ PTransformOverride.of(
+ PTransformMatchers.classEqualTo(GroupIntoBatches.class),
+ new BatchGroupIntoBatchesOverrideFactory()));
+
+ overridesBuilder
// State and timer pardos are implemented by expansion to GBK-then-ParDo
.add(
PTransformOverride.of(
@@ -774,7 +797,7 @@
checkState(
!"${pom.version}".equals(version),
"Unable to submit a job to the Dataflow service with unset version ${pom.version}");
- System.out.println("Dataflow SDK version: " + version);
+ LOG.info("Dataflow SDK version: {}", version);
newJob.getEnvironment().setUserAgent((Map) dataflowRunnerInfo.getProperties());
// The Dataflow Service may write to the temporary directory directly, so
@@ -830,6 +853,19 @@
hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
}
+ // Upload the job to GCS and remove the graph object from the API call. The graph
+ // will be downloaded from GCS by the service.
+ if (hasExperiment(options, "upload_graph")) {
+ DataflowPackage stagedGraph =
+ options
+ .getStager()
+ .stageToFile(
+ DataflowPipelineTranslator.jobToString(newJob).getBytes(UTF_8),
+ DATAFLOW_GRAPH_FILE_NAME);
+ newJob.getSteps().clear();
+ newJob.setStepsLocation(stagedGraph.getLocation());
+ }
+
if (!isNullOrEmpty(options.getDataflowJobFile())
|| !isNullOrEmpty(options.getTemplateLocation())) {
boolean isTemplate = !isNullOrEmpty(options.getTemplateLocation());
@@ -878,19 +914,6 @@
newJob.setCreatedFromSnapshotId(options.getCreateFromSnapshot());
}
- // Upload the job to GCS and remove the graph object from the API call. The graph
- // will be downloaded from GCS by the service.
- if (hasExperiment(options, "upload_graph")) {
- DataflowPackage stagedGraph =
- options
- .getStager()
- .stageToFile(
- DataflowPipelineTranslator.jobToString(newJob).getBytes(UTF_8),
- DATAFLOW_GRAPH_FILE_NAME);
- newJob.getSteps().clear();
- newJob.setStepsLocation(stagedGraph.getLocation());
- }
-
Job jobResult;
try {
jobResult = dataflowClient.createJob(newJob);
@@ -951,7 +974,7 @@
"To access the Dataflow monitoring console, please navigate to {}",
MonitoringUtil.getJobMonitoringPageURL(
options.getProject(), options.getRegion(), jobResult.getId()));
- System.out.println("Submitted job: " + jobResult.getId());
+ LOG.info("Submitted job: {}", jobResult.getId());
LOG.info(
"To cancel the job using the 'gcloud' tool, run:\n> {}",
@@ -1139,11 +1162,12 @@
@Override
public PCollection<PubsubMessage> expand(PBegin input) {
+ Coder coder =
+ transform.getNeedsMessageId()
+ ? new PubsubMessageWithAttributesAndMessageIdCoder()
+ : new PubsubMessageWithAttributesCoder();
return PCollection.createPrimitiveOutputInternal(
- input.getPipeline(),
- WindowingStrategy.globalDefault(),
- IsBounded.UNBOUNDED,
- new PubsubMessageWithAttributesCoder());
+ input.getPipeline(), WindowingStrategy.globalDefault(), IsBounded.UNBOUNDED, coder);
}
@Override
@@ -1422,6 +1446,61 @@
DataflowPipelineTranslator.registerTransformTranslator(Impulse.class, new ImpulseTranslator());
}
+ private static class BatchGroupIntoBatchesOverrideFactory<K, V>
+ implements PTransformOverrideFactory<
+ PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>, GroupIntoBatches<K, V>> {
+
+ @Override
+ public PTransformReplacement<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>>
+ getReplacementTransform(
+ AppliedPTransform<
+ PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>, GroupIntoBatches<K, V>>
+ transform) {
+ return PTransformReplacement.of(
+ PTransformReplacements.getSingletonMainInput(transform),
+ new BatchGroupIntoBatches(transform.getTransform().getBatchSize()));
+ }
+
+ @Override
+ public Map<PValue, ReplacementOutput> mapOutputs(
+ Map<TupleTag<?>, PValue> outputs, PCollection<KV<K, Iterable<V>>> newOutput) {
+ return ReplacementOutputs.singleton(outputs, newOutput);
+ }
+ }
+
+ /** Specialized implementation of {@link GroupIntoBatches} for bounded Dataflow pipelines. */
+ static class BatchGroupIntoBatches<K, V>
+ extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Iterable<V>>>> {
+ private final long batchSize;
+
+ private BatchGroupIntoBatches(long batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ @Override
+ public PCollection<KV<K, Iterable<V>>> expand(PCollection<KV<K, V>> input) {
+ return input
+ .apply("GroupAll", GroupByKey.create())
+ .apply(
+ "SplitIntoBatches",
+ ParDo.of(
+ new DoFn<KV<K, Iterable<V>>, KV<K, Iterable<V>>>() {
+ @ProcessElement
+ public void process(ProcessContext c) {
+ // Iterators.partition lazily creates the partitions as they are accessed
+ // allowing it to partition very large iterators.
+ Iterator<List<V>> iterator =
+ Iterators.partition(c.element().getValue().iterator(), (int) batchSize);
+
+ // Note that GroupIntoBatches only outputs when the batch is non-empty.
+ while (iterator.hasNext()) {
+ c.output(KV.of(c.element().getKey(), iterator.next()));
+ }
+ }
+ }));
+ }
+ }
+
private static class StreamingUnboundedReadOverrideFactory<T>
implements PTransformOverrideFactory<PBegin, PCollection<T>, Read.Unbounded<T>> {
@Override
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
index b393e51..6aa49d4 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerInfo.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.runners.dataflow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
import java.util.Map;
import java.util.Properties;
import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactory.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactory.java
index 19a01d8..390f0f0 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactory.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactory.java
@@ -22,7 +22,7 @@
import static org.apache.beam.sdk.options.ExperimentalOptions.hasExperiment;
import static org.apache.beam.sdk.transforms.reflect.DoFnSignatures.getStateSpecOrThrow;
import static org.apache.beam.sdk.transforms.reflect.DoFnSignatures.getTimerSpecOrThrow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -57,8 +57,8 @@
import org.apache.beam.sdk.values.PCollectionViews;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A {@link PTransformOverrideFactory} that produces {@link ParDoSingle} instances from {@link
@@ -118,13 +118,13 @@
return onlyOutputTag;
}
- public List<PCollectionView<?>> getSideInputs() {
+ public Map<String, PCollectionView<?>> getSideInputs() {
return original.getSideInputs();
}
@Override
public Map<TupleTag<?>, PValue> getAdditionalInputs() {
- return PCollectionViews.toAdditionalInputs(getSideInputs());
+ return PCollectionViews.toAdditionalInputs(getSideInputs().values());
}
@Override
@@ -178,7 +178,7 @@
Set<String> allInputs =
transform.getInputs().keySet().stream().map(TupleTag::getId).collect(Collectors.toSet());
Set<String> sideInputs =
- parDo.getSideInputs().stream()
+ parDo.getSideInputs().values().stream()
.map(s -> s.getTagInternal().getId())
.collect(Collectors.toSet());
Set<String> timerInputs = signature.timerDeclarations().keySet();
@@ -195,7 +195,11 @@
@Override
public RunnerApi.SdkFunctionSpec translateDoFn(SdkComponents newComponents) {
return ParDoTranslation.translateDoFn(
- parDo.getFn(), parDo.getMainOutputTag(), doFnSchemaInformation, newComponents);
+ parDo.getFn(),
+ parDo.getMainOutputTag(),
+ parDo.getSideInputs(),
+ doFnSchemaInformation,
+ newComponents);
}
@Override
@@ -206,7 +210,8 @@
@Override
public Map<String, RunnerApi.SideInput> translateSideInputs(SdkComponents components) {
- return ParDoTranslation.translateSideInputs(parDo.getSideInputs(), components);
+ return ParDoTranslation.translateSideInputs(
+ parDo.getSideInputs().values().stream().collect(Collectors.toList()), components);
}
@Override
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TestDataflowRunner.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TestDataflowRunner.java
index a7e608e..5975b91 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TestDataflowRunner.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TestDataflowRunner.java
@@ -38,10 +38,10 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java
index 40a05f3..ff0a1a0 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java
@@ -21,7 +21,7 @@
import static org.apache.beam.runners.dataflow.util.Structs.addString;
import static org.apache.beam.runners.dataflow.util.Structs.addStringList;
import static org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.SourceMetadata;
import java.util.ArrayList;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.io.Source;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
index b6f5466..1a51974 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/IsmFormat.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.dataflow.internal;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.DataInputStream;
@@ -44,9 +44,9 @@
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
/**
* An Ism file is a prefix encoded composite key value file broken into shards. Each composite key
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java
index 35df563..c035839 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java
@@ -126,8 +126,8 @@
@Description(
"The Google Compute Engine region for creating Dataflow jobs. See "
+ "https://cloud.google.com/compute/docs/regions-zones/regions-zones for a list of valid "
- + "options. Default is up to the Dataflow service.")
- @Default.String("us-central1")
+ + "options. Currently defaults to us-central1, but future releases of Beam will "
+ + "require the user to set the region explicitly.")
String getRegion();
void setRegion(String region);
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
index da0c6f5..eb313e6 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java
@@ -22,6 +22,7 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.DataflowRunnerInfo;
import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.DefaultValueFactory;
import org.apache.beam.sdk.options.Description;
@@ -30,7 +31,7 @@
/** Options that are used to configure the Dataflow pipeline worker pool. */
@Description("Options that are used to configure the Dataflow pipeline worker pool.")
-public interface DataflowPipelineWorkerPoolOptions extends PipelineOptions {
+public interface DataflowPipelineWorkerPoolOptions extends GcpOptions {
/**
* Number of workers to use when executing the Dataflow job. Note that selection of an autoscaling
* algorithm other then {@code NONE} will affect the size of the worker pool. If left unspecified,
@@ -167,20 +168,6 @@
void setSubnetwork(String value);
/**
- * GCE <a href="https://developers.google.com/compute/docs/zones" >availability zone</a> for
- * launching workers.
- *
- * <p>Default is up to the Dataflow service.
- */
- @Description(
- "GCE availability zone for launching workers. See "
- + "https://developers.google.com/compute/docs/zones for a list of valid options. "
- + "Default is up to the Dataflow service.")
- String getZone();
-
- void setZone(String value);
-
- /**
* Machine type to create Dataflow worker VMs as.
*
* <p>See <a href="https://cloud.google.com/compute/docs/machine-types">GCE machine types</a> for
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.java
index 0c4afc0..c2fde48 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptions.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.annotation.JsonCreator;
import java.util.Arrays;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudKnownType.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudKnownType.java
index bd9f443..ca5ca6a 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudKnownType.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudKnownType.java
@@ -21,7 +21,7 @@
import java.util.HashMap;
import java.util.Map;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A utility for manipulating well-known cloud types. */
@SuppressWarnings("ImmutableEnumChecker")
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObject.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObject.java
index 4a54473..e341004 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObject.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObject.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.json.GenericJson;
import com.google.api.client.util.Key;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
index ba72975..7834784 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectTranslators.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Collections;
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.util.StringUtils;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Utilities for creating {@link CloudObjectTranslator} instances for {@link Coder Coders}. */
class CloudObjectTranslators {
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjects.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjects.java
index ee78f6f..ec51351 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjects.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjects.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Map;
import java.util.ServiceLoader;
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/** Utilities for converting an object to a {@link CloudObject}. */
public class CloudObjects {
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTemplateJob.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTemplateJob.java
index 879386e..6b94920 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTemplateJob.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTemplateJob.java
@@ -21,7 +21,7 @@
import com.google.api.client.util.Sleeper;
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.DataflowPipelineJob;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
/** A {@link DataflowPipelineJob} that is returned when {@code --templateRunner} is set. */
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
index c7b1be8..3503b95 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
@@ -31,7 +31,7 @@
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import org.apache.beam.sdk.extensions.gcp.auth.NullCredentialInitializer;
import org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Helpers for cloud communication. */
public class DataflowTransport {
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java
index 237d92d..d6bac2a 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java
@@ -43,12 +43,13 @@
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.io.gcp.bigquery.TableDestinationCoderV2;
+import org.apache.beam.sdk.io.gcp.bigquery.TableDestinationCoderV3;
import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap.Builder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap.Builder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/**
* The {@link CoderCloudObjectTranslatorRegistrar} containing the default collection of {@link
@@ -99,6 +100,7 @@
RandomAccessDataCoder.class,
StringUtf8Coder.class,
TableDestinationCoderV2.class,
+ TableDestinationCoderV3.class,
TableRowJsonCoder.class,
TextualIntegerCoder.class,
VarIntCoder.class,
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
index cfad582..5c2ee5b 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.dataflow.model.DataflowPackage;
import java.util.List;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
index 9b2bafc..1010e09 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
@@ -35,9 +35,9 @@
import org.apache.beam.runners.dataflow.DataflowClient;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -219,6 +219,6 @@
}
public static State toState(String stateName) {
- return MoreObjects.firstNonNull(DATAFLOW_STATE_TO_JOB_STATE.get(stateName), State.UNKNOWN);
+ return MoreObjects.firstNonNull(DATAFLOW_STATE_TO_JOB_STATE.get(stateName), State.UNRECOGNIZED);
}
}
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java
index 1e56cde..f8b7784 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/OutputReference.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.json.GenericJson;
import com.google.api.client.util.Key;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
index 1517e28..69c6422 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.fasterxml.jackson.core.Base64Variants;
import com.google.api.client.util.BackOff;
@@ -56,13 +56,13 @@
import org.apache.beam.sdk.util.MimeTypes;
import org.apache.beam.sdk.util.MoreFutures;
import org.apache.beam.sdk.util.ZipFiles;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnels;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnels;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.Seconds;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
index bc0dd1a..e413856 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RandomAccessData.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.ByteArrayInputStream;
import java.io.IOException;
@@ -34,9 +34,9 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.util.VarInt;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
/**
* An elastic-sized byte array which allows you to manipulate it as a stream, or access it directly.
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java
index 2395f12..2ff8c77 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SchemaCoderCloudObjectTranslator.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.util;
import java.io.IOException;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
import org.apache.beam.runners.core.construction.SchemaTranslation;
import org.apache.beam.runners.core.construction.SdkComponents;
import org.apache.beam.sdk.schemas.Schema;
@@ -52,7 +52,7 @@
base,
SCHEMA,
StringUtils.byteArrayToJsonString(
- SchemaTranslation.toProto(target.getSchema()).toByteArray()));
+ SchemaTranslation.schemaToProto(target.getSchema()).toByteArray()));
return base;
}
@@ -72,8 +72,8 @@
StringUtils.jsonStringToByteArray(
Structs.getString(cloudObject, FROM_ROW_FUNCTION)),
"fromRowFunction");
- RunnerApi.Schema protoSchema =
- RunnerApi.Schema.parseFrom(
+ SchemaApi.Schema protoSchema =
+ SchemaApi.Schema.parseFrom(
StringUtils.jsonStringToByteArray(Structs.getString(cloudObject, SCHEMA)));
Schema schema = SchemaTranslation.fromProto(protoSchema);
return SchemaCoder.of(schema, toRowFunction, fromRowFunction);
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SerializableCoderCloudObjectTranslator.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SerializableCoderCloudObjectTranslator.java
index 5a958c0..53ad45a 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SerializableCoderCloudObjectTranslator.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/SerializableCoderCloudObjectTranslator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import org.apache.beam.runners.core.construction.SdkComponents;
diff --git a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
index 0d17c7d..8f2c621 100644
--- a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
+++ b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/TimeUtil.java
@@ -20,7 +20,7 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverridesTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverridesTest.java
index a13791c..936b972 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverridesTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchStatefulParDoOverridesTest.java
@@ -49,7 +49,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
index e698e02..65de4bb 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/BatchViewOverridesTest.java
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
index 124ae86..7ed285d 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowMetricsTest.java
@@ -47,9 +47,9 @@
import org.apache.beam.sdk.metrics.MetricQueryResults;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
index 48faf23..0d2907d 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
@@ -59,9 +59,9 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
@@ -225,13 +225,27 @@
}
/**
+ * Tests that the {@link DataflowPipelineJob} understands that the {@link State#UPDATED UPDATED}
+ * state is terminal.
+ */
+ @Test
+ public void testWaitToFinishLogsError() throws Exception {
+ assertEquals(State.UPDATED, mockWaitToFinishInState(State.UPDATED));
+ expectedLogs.verifyInfo(
+ String.format(
+ "Job %s has been updated and is running as the new job with id %s.",
+ JOB_ID, REPLACEMENT_JOB_ID));
+ }
+
+ /**
* Tests that the {@link DataflowPipelineJob} understands that the {@link State#UNKNOWN UNKNOWN}
* state is terminal.
*/
@Test
public void testWaitToFinishUnknown() throws Exception {
assertEquals(null, mockWaitToFinishInState(State.UNKNOWN));
- expectedLogs.verifyWarn("No terminal state was returned. State value UNKNOWN");
+ expectedLogs.verifyWarn(
+ "No terminal state was returned within allotted timeout. State value UNKNOWN");
}
@Test
@@ -311,13 +325,31 @@
assertEquals(
State.RUNNING,
- job.getStateWithRetries(
+ job.getStateWithRetriesOrUnknownOnException(
BackOffAdapter.toGcpBackOff(DataflowPipelineJob.STATUS_BACKOFF_FACTORY.backoff()),
fastClock));
}
@Test
- public void testGetStateWithExceptionReturnsUnknown() throws Exception {
+ public void testGetStateWithRetriesPassesExceptionThrough() throws Exception {
+ Dataflow.Projects.Locations.Jobs.Get statusRequest =
+ mock(Dataflow.Projects.Locations.Jobs.Get.class);
+
+ when(mockJobs.get(eq(PROJECT_ID), eq(REGION_ID), eq(JOB_ID))).thenReturn(statusRequest);
+ when(statusRequest.execute()).thenThrow(IOException.class);
+
+ DataflowPipelineJob job =
+ new DataflowPipelineJob(DataflowClient.create(options), JOB_ID, options, ImmutableMap.of());
+
+ long startTime = fastClock.nanoTime();
+ thrown.expect(IOException.class);
+ job.getStateWithRetries(
+ BackOffAdapter.toGcpBackOff(DataflowPipelineJob.STATUS_BACKOFF_FACTORY.backoff()),
+ fastClock);
+ }
+
+ @Test
+ public void testGetStateNoThrowWithExceptionReturnsUnknown() throws Exception {
Dataflow.Projects.Locations.Jobs.Get statusRequest =
mock(Dataflow.Projects.Locations.Jobs.Get.class);
@@ -330,7 +362,7 @@
long startTime = fastClock.nanoTime();
assertEquals(
State.UNKNOWN,
- job.getStateWithRetries(
+ job.getStateWithRetriesOrUnknownOnException(
BackOffAdapter.toGcpBackOff(DataflowPipelineJob.STATUS_BACKOFF_FACTORY.backoff()),
fastClock));
long timeDiff = TimeUnit.NANOSECONDS.toMillis(fastClock.nanoTime() - startTime);
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrarTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrarTest.java
index b8e2f16..ac80ec6 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrarTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRegistrarTest.java
@@ -24,8 +24,8 @@
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
index e027bae..85b1e22 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslatorTest.java
@@ -50,6 +50,8 @@
import java.util.Set;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
+import org.apache.beam.model.pipeline.v1.RunnerApi.DockerPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.model.pipeline.v1.RunnerApi.ParDoPayload;
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.ParDoTranslation;
@@ -99,10 +101,10 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.junit.Assert;
@@ -120,9 +122,9 @@
// A Custom Mockito matcher for an initial Job that checks that all
// expected fields are set.
- private static class IsValidCreateRequest extends ArgumentMatcher<Job> {
+ private static class IsValidCreateRequest implements ArgumentMatcher<Job> {
@Override
- public boolean matches(Object o) {
+ public boolean matches(Job o) {
Job job = (Job) o;
return job.getId() == null
&& job.getProjectId() == null
@@ -956,6 +958,34 @@
assertEquals(expectedFn2DisplayData, ImmutableSet.copyOf(fn2displayData));
}
+ /**
+ * Tests that when {@link DataflowPipelineOptions#setWorkerHarnessContainerImage(String)} pipeline
+ * option is set, {@link DataflowRunner} sets that value as the {@link
+ * DockerPayload#getContainerImage()} of the default {@link Environment} used when generating the
+ * model pipeline proto.
+ */
+ @Test
+ public void testSetWorkerHarnessContainerImageInPipelineProto() throws Exception {
+ DataflowPipelineOptions options = buildPipelineOptions();
+ String containerImage = "gcr.io/IMAGE/foo";
+ options.as(DataflowPipelineOptions.class).setWorkerHarnessContainerImage(containerImage);
+
+ JobSpecification specification =
+ DataflowPipelineTranslator.fromOptions(options)
+ .translate(
+ Pipeline.create(options),
+ DataflowRunner.fromOptions(options),
+ Collections.emptyList());
+ RunnerApi.Pipeline pipelineProto = specification.getPipelineProto();
+
+ assertEquals(1, pipelineProto.getComponents().getEnvironmentsCount());
+ Environment defaultEnvironment =
+ Iterables.getOnlyElement(pipelineProto.getComponents().getEnvironmentsMap().values());
+
+ DockerPayload payload = DockerPayload.parseFrom(defaultEnvironment.getPayload());
+ assertEquals(DataflowRunner.getContainerImageForJob(options), payload.getContainerImage());
+ }
+
private static void assertAllStepOutputsHaveUniqueIds(Job job) throws Exception {
List<String> outputIds = new ArrayList<>();
for (Step step : job.getSteps()) {
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
index 9214ecf..f46d034 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
@@ -18,21 +18,25 @@
package org.apache.beam.runners.dataflow;
import static org.apache.beam.runners.dataflow.DataflowRunner.getContainerImageForJob;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.both;
+import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasEntry;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.hasKey;
import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.startsWith;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
+import static org.junit.Assume.assumeFalse;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyInt;
import static org.mockito.Matchers.anyListOf;
@@ -47,8 +51,10 @@
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
@@ -69,11 +75,13 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.regex.Pattern;
import javax.annotation.Nullable;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchGroupIntoBatches;
import org.apache.beam.runners.dataflow.DataflowRunner.StreamingShardedWriteFactory;
import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
@@ -111,11 +119,15 @@
import org.apache.beam.sdk.state.StateSpecs;
import org.apache.beam.sdk.state.ValueState;
import org.apache.beam.sdk.testing.ExpectedLogs;
+import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupIntoBatches;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
@@ -126,8 +138,9 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Description;
import org.hamcrest.Matchers;
import org.hamcrest.TypeSafeMatcher;
@@ -136,6 +149,7 @@
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
@@ -163,6 +177,7 @@
@Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
@Rule public transient ExpectedException thrown = ExpectedException.none();
@Rule public transient ExpectedLogs expectedLogs = ExpectedLogs.none(DataflowRunner.class);
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
private transient Dataflow.Projects.Locations.Jobs mockJobs;
private transient GcsUtil mockGcsUtil;
@@ -262,7 +277,7 @@
when(mockLocations.jobs()).thenReturn(mockJobs);
when(mockJobs.create(eq(PROJECT_ID), eq(REGION_ID), isA(Job.class))).thenReturn(mockRequest);
when(mockJobs.list(eq(PROJECT_ID), eq(REGION_ID))).thenReturn(mockList);
- when(mockList.setPageToken(anyString())).thenReturn(mockList);
+ when(mockList.setPageToken(any())).thenReturn(mockList);
when(mockList.execute())
.thenReturn(
new ListJobsResponse()
@@ -765,7 +780,7 @@
DataflowPipelineOptions options = buildPipelineOptions();
options.setFilesToStage(null);
DataflowRunner.fromOptions(options);
- assertTrue(!options.getFilesToStage().isEmpty());
+ assertFalse(options.getFilesToStage().isEmpty());
}
@Test
@@ -1305,6 +1320,33 @@
}
/**
+ * Tests that the {@link DataflowRunner} with {@code --templateLocation} returns normally when the
+ * runner is successfully run with upload_graph experiment turned on. The result template should
+ * not contain raw steps and stepsLocation file should be set.
+ */
+ @Test
+ public void testTemplateRunnerWithUploadGraph() throws Exception {
+ File existingFile = tmpFolder.newFile();
+ DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+ options.setExperiments(Arrays.asList("upload_graph"));
+ options.setJobName("TestJobName");
+ options.setGcpCredential(new TestCredential());
+ options.setPathValidatorClass(NoopPathValidator.class);
+ options.setProject("test-project");
+ options.setRunner(DataflowRunner.class);
+ options.setTemplateLocation(existingFile.getPath());
+ options.setTempLocation(tmpFolder.getRoot().getPath());
+ Pipeline p = Pipeline.create(options);
+ p.apply(Create.of(ImmutableList.of(1)));
+ p.run();
+ expectedLogs.verifyInfo("Template successfully created");
+ ObjectMapper objectMapper = new ObjectMapper();
+ JsonNode node = objectMapper.readTree(existingFile);
+ assertEquals(0, node.get("steps").size());
+ assertNotNull(node.get("stepsLocation"));
+ }
+
+ /**
* Tests that the {@link DataflowRunner} with {@code --templateLocation} throws the appropriate
* exception when an output file is not writable.
*/
@@ -1402,6 +1444,55 @@
verifyMergingStatefulParDoRejected(options);
}
+ @Test
+ @Category(ValidatesRunner.class)
+ public void testBatchGroupIntoBatchesOverride() {
+ // Ignore this test for streaming pipelines.
+ assumeFalse(pipeline.getOptions().as(StreamingOptions.class).isStreaming());
+
+ final int batchSize = 2;
+ List<KV<String, Integer>> testValues =
+ Arrays.asList(KV.of("A", 1), KV.of("B", 0), KV.of("A", 2), KV.of("A", 4), KV.of("A", 8));
+ PCollection<KV<String, Iterable<Integer>>> output =
+ pipeline.apply(Create.of(testValues)).apply(GroupIntoBatches.ofSize(batchSize));
+ PAssert.thatMultimap(output)
+ .satisfies(
+ new SerializableFunction<Map<String, Iterable<Iterable<Integer>>>, Void>() {
+ @Override
+ public Void apply(Map<String, Iterable<Iterable<Integer>>> input) {
+ assertEquals(2, input.size());
+ assertThat(input.keySet(), containsInAnyOrder("A", "B"));
+ Map<String, Integer> sums = new HashMap<>();
+ for (Map.Entry<String, Iterable<Iterable<Integer>>> entry : input.entrySet()) {
+ for (Iterable<Integer> batch : entry.getValue()) {
+ assertThat(Iterables.size(batch), lessThanOrEqualTo(batchSize));
+ for (Integer value : batch) {
+ sums.put(entry.getKey(), value + sums.getOrDefault(entry.getKey(), 0));
+ }
+ }
+ }
+ assertEquals(15, (int) sums.get("A"));
+ assertEquals(0, (int) sums.get("B"));
+ return null;
+ }
+ });
+ pipeline.run();
+
+ AtomicBoolean sawGroupIntoBatchesOverride = new AtomicBoolean(false);
+ pipeline.traverseTopologically(
+ new PipelineVisitor.Defaults() {
+
+ @Override
+ public CompositeBehavior enterCompositeTransform(Node node) {
+ if (node.getTransform() instanceof BatchGroupIntoBatches) {
+ sawGroupIntoBatchesOverride.set(true);
+ }
+ return CompositeBehavior.ENTER_TRANSFORM;
+ }
+ });
+ assertTrue(sawGroupIntoBatchesOverride.get());
+ }
+
private void testStreamingWriteOverride(PipelineOptions options, int expectedNumShards) {
TestPipeline p = TestPipeline.fromOptions(options);
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactoryTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactoryTest.java
index e9d97f0..c55bffc 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/PrimitiveParDoSingleFactoryTest.java
@@ -37,7 +37,7 @@
import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -115,7 +115,7 @@
factory.getReplacementTransform(application);
ParDoSingle<Integer, Long> parDoSingle =
(ParDoSingle<Integer, Long>) replacementTransform.getTransform();
- assertThat(parDoSingle.getSideInputs(), containsInAnyOrder(sideStrings, sideLong));
+ assertThat(parDoSingle.getSideInputs().values(), containsInAnyOrder(sideStrings, sideLong));
}
@Test
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
index 24a5fe0..75d4166 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java
@@ -56,9 +56,9 @@
import org.apache.beam.sdk.testing.TestPipelineOptions;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.joda.time.Duration;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
index c751871..754f061 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.ResetDateTimeProvider;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptionsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptionsTest.java
index f37c34b..1fa5b14 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptionsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowWorkerLoggingOptionsTest.java
@@ -23,7 +23,7 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions.WorkerLogLevelOverrides;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
index 4be760b..7f10d92 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectsTest.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.dataflow.util;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.emptyIterable;
import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.instanceOf;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import java.io.IOException;
@@ -31,8 +32,8 @@
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
import java.util.Set;
-import org.apache.beam.runners.core.construction.ModelCoderRegistrar;
import org.apache.beam.runners.core.construction.SdkComponents;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.ByteArrayCoder;
@@ -50,8 +51,11 @@
import org.apache.beam.sdk.coders.SetCoder;
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.LogicalTypes;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.join.CoGbkResult.CoGbkResultCoder;
import org.apache.beam.sdk.transforms.join.CoGbkResultSchema;
import org.apache.beam.sdk.transforms.join.UnionCoder;
@@ -59,10 +63,13 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList.Builder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList.Builder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.junit.runners.Parameterized;
@@ -70,7 +77,22 @@
import org.junit.runners.Parameterized.Parameters;
/** Tests for {@link CloudObjects}. */
+@RunWith(Enclosed.class)
public class CloudObjectsTest {
+ private static final Schema TEST_SCHEMA =
+ Schema.builder()
+ .addBooleanField("bool")
+ .addByteField("int8")
+ .addInt16Field("int16")
+ .addInt32Field("int32")
+ .addInt64Field("int64")
+ .addFloatField("float")
+ .addDoubleField("double")
+ .addStringField("string")
+ .addArrayField("list_int32", FieldType.INT32)
+ .addLogicalTypeField("fixed_bytes", LogicalTypes.FixedBytes.of(4))
+ .build();
+
/** Tests that all of the Default Coders are tested. */
@RunWith(JUnit4.class)
public static class DefaultsPresentTest {
@@ -143,7 +165,8 @@
CoGbkResultSchema.of(
ImmutableList.of(new TupleTag<Long>(), new TupleTag<byte[]>())),
UnionCoder.of(ImmutableList.of(VarLongCoder.of(), ByteArrayCoder.of()))))
- .add(SchemaCoder.of(Schema.builder().build()));
+ .add(SchemaCoder.of(Schema.builder().build(), new RowIdentity(), new RowIdentity()))
+ .add(SchemaCoder.of(TEST_SCHEMA, new RowIdentity(), new RowIdentity()));
for (Class<? extends Coder> atomicCoder :
DefaultCoderCloudObjectTranslatorRegistrar.KNOWN_ATOMIC_CODERS) {
dataBuilder.add(InstanceBuilder.ofType(atomicCoder).fromFactoryMethod("of").build());
@@ -177,21 +200,33 @@
private static void checkPipelineProtoCoderIds(
Coder<?> coder, CloudObject cloudObject, SdkComponents sdkComponents) throws Exception {
- if (ModelCoderRegistrar.isKnownCoder(coder)) {
+ if (CloudObjects.DATAFLOW_KNOWN_CODERS.contains(coder.getClass())) {
assertFalse(cloudObject.containsKey(PropertyNames.PIPELINE_PROTO_CODER_ID));
} else {
assertTrue(cloudObject.containsKey(PropertyNames.PIPELINE_PROTO_CODER_ID));
assertEquals(
sdkComponents.registerCoder(coder),
- cloudObject.get(PropertyNames.PIPELINE_PROTO_CODER_ID));
+ ((CloudObject) cloudObject.get(PropertyNames.PIPELINE_PROTO_CODER_ID))
+ .get(PropertyNames.VALUE));
}
- List<? extends Coder<?>> coderArguments = coder.getCoderArguments();
+ List<? extends Coder<?>> expectedComponents;
+ if (coder instanceof StructuredCoder) {
+ expectedComponents = ((StructuredCoder) coder).getComponents();
+ } else {
+ expectedComponents = coder.getCoderArguments();
+ }
Object cloudComponentsObject = cloudObject.get(PropertyNames.COMPONENT_ENCODINGS);
- assertTrue(cloudComponentsObject instanceof List);
- List<CloudObject> cloudComponents = (List<CloudObject>) cloudComponentsObject;
- assertEquals(coderArguments.size(), cloudComponents.size());
- for (int i = 0; i < coderArguments.size(); i++) {
- checkPipelineProtoCoderIds(coderArguments.get(i), cloudComponents.get(i), sdkComponents);
+ List<CloudObject> cloudComponents;
+ if (cloudComponentsObject == null) {
+ cloudComponents = Lists.newArrayList();
+ } else {
+ assertThat(cloudComponentsObject, instanceOf(List.class));
+ cloudComponents = (List<CloudObject>) cloudComponentsObject;
+ }
+ assertEquals(expectedComponents.size(), cloudComponents.size());
+ for (int i = 0; i < expectedComponents.size(); i++) {
+ checkPipelineProtoCoderIds(
+ expectedComponents.get(i), cloudComponents.get(i), sdkComponents);
}
}
}
@@ -236,4 +271,25 @@
@Override
public void verifyDeterministic() throws NonDeterministicException {}
}
+
+ /** Hack to satisfy SchemaCoder.equals until BEAM-8146 is fixed. */
+ private static class RowIdentity implements SerializableFunction<Row, Row> {
+ @Override
+ public Row apply(Row input) {
+ return input;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(getClass());
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ return o != null && getClass() == o.getClass();
+ }
+ }
}
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
index 77ee87f..df169b6 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
@@ -104,13 +104,13 @@
}
@Test
- public void testToStateWithNullReturnsUnknown() {
- assertEquals(State.UNKNOWN, MonitoringUtil.toState(null));
+ public void testToStateWithNullReturnsUnrecognized() {
+ assertEquals(State.UNRECOGNIZED, MonitoringUtil.toState(null));
}
@Test
public void testToStateWithOtherValueReturnsUnknown() {
- assertEquals(State.UNKNOWN, MonitoringUtil.toState("FOO_BAR_BAZ"));
+ assertEquals(State.UNRECOGNIZED, MonitoringUtil.toState("FOO_BAR_BAZ"));
}
@Test
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
index e21a5ab..4013d18 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/PackageUtilTest.java
@@ -80,12 +80,12 @@
import org.apache.beam.sdk.testing.ExpectedLogs;
import org.apache.beam.sdk.testing.RegexMatcher;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.LineReader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.hamcrest.Matchers;
import org.junit.After;
import org.junit.Before;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/RandomAccessDataTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/RandomAccessDataTest.java
index 2a216c5..cf08076 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/RandomAccessDataTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/RandomAccessDataTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.coders.Coder.Context;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.testing.CoderProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/StructsTest.java b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/StructsTest.java
index 8c88a43..91804ab 100644
--- a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/StructsTest.java
+++ b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/StructsTest.java
@@ -126,8 +126,8 @@
public void testGetBooleanParameter() throws Exception {
Map<String, Object> o = makeCloudDictionary();
- Assert.assertEquals(true, getBoolean(o, "singletonBooleanKey", false));
- Assert.assertEquals(false, getBoolean(o, "missingKey", false));
+ Assert.assertTrue(getBoolean(o, "singletonBooleanKey", false));
+ Assert.assertFalse(getBoolean(o, "missingKey", false));
try {
getBoolean(o, "emptyKey", false);
diff --git a/runners/google-cloud-dataflow-java/worker/build.gradle b/runners/google-cloud-dataflow-java/worker/build.gradle
index 5ba415a..4aba6eb 100644
--- a/runners/google-cloud-dataflow-java/worker/build.gradle
+++ b/runners/google-cloud-dataflow-java/worker/build.gradle
@@ -18,8 +18,6 @@
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-runners-google-cloud-dataflow-java-fn-api-worker'
-
// Set a specific version of 'com.google.apis:google-api-services-dataflow'
// by adding -Pdataflow.version=<version> in Gradle command. Otherwise,
// 'google_clients_version' defined in BeamModulePlugin will be used as default.
@@ -29,6 +27,7 @@
def google_api_services_dataflow = project.hasProperty(DATAFLOW_VERSION) ? "com.google.apis:google-api-services-dataflow:" + getProperty(DATAFLOW_VERSION) : library.java.google_api_services_dataflow
applyJavaNature(
+ archivesBaseName: 'beam-runners-google-cloud-dataflow-java-fn-api-worker',
publish: false,
exportJavadoc: false,
enableSpotbugs: true,
@@ -66,18 +65,18 @@
//
// All main sourceset dependencies here should be listed as compile scope so that the dependencies
// are all packaged into a single uber jar allowing the jar to serve as an application.
- compile project(path: ":runners:google-cloud-dataflow-java", configuration: "shadow")
+ compile project(":runners:google-cloud-dataflow-java")
compile project(path: ":sdks:java:core", configuration: "shadow")
- compile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- compile project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:io:google-cloud-platform")
compile project(path: ":model:fn-execution", configuration: "shadow")
compile project(path: ":model:pipeline", configuration: "shadow")
- compile project(path: ":runners:core-construction-java", configuration: "shadow")
- compile project(path: ":runners:core-java", configuration: "shadow")
- compile project(path: ":runners:java-fn-execution", configuration: "shadow")
- compile project(path: ":sdks:java:fn-execution", configuration: "shadow")
+ compile project(":runners:core-construction-java")
+ compile project(":runners:core-java")
+ compile project(":runners:java-fn-execution")
+ compile project(":sdks:java:fn-execution")
compile project(path: ":runners:google-cloud-dataflow-java:worker:windmill", configuration: "shadow")
- compile library.java.vendored_grpc_1_13_1
+ compile library.java.vendored_grpc_1_21_0
compile google_api_services_dataflow
compile library.java.avro
compile library.java.google_api_client
@@ -87,7 +86,7 @@
compile library.java.jackson_core
compile library.java.jackson_databind
compile library.java.joda_time
- shadow library.java.vendored_guava_20_0
+ shadow library.java.vendored_guava_26_0_jre
compile library.java.slf4j_api
compile "javax.servlet:javax.servlet-api:3.1.0"
compile "org.conscrypt:conscrypt-openjdk:1.1.3:linux-x86_64"
@@ -98,12 +97,11 @@
// All test sourceset dependencies can be marked as shadowTest since we create an uber jar without
// relocating any code.
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
+ shadowTest project(path: ":runners:core-java", configuration: "testRuntime")
shadowTest project(path: ":sdks:java:harness", configuration: "shadowTest")
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadowTest")
+ shadowTest project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntime")
shadowTest project(path: ":runners:direct-java", configuration: "shadow")
- shadowTest library.java.guava_testlib
shadowTest library.java.hamcrest_core
shadowTest library.java.hamcrest_library
shadowTest library.java.junit
diff --git a/runners/google-cloud-dataflow-java/worker/legacy-worker/build.gradle b/runners/google-cloud-dataflow-java/worker/legacy-worker/build.gradle
index 3594a75..6dc99f2 100644
--- a/runners/google-cloud-dataflow-java/worker/legacy-worker/build.gradle
+++ b/runners/google-cloud-dataflow-java/worker/legacy-worker/build.gradle
@@ -25,8 +25,6 @@
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-runners-google-cloud-dataflow-java-legacy-worker'
-
// Set a specific version of 'com.google.apis:google-api-services-dataflow'
// by adding -Pdataflow.version=<version> in Gradle command. Otherwise,
// 'google_clients_version' defined in BeamModulePlugin will be used as default.
@@ -55,13 +53,15 @@
library.java.jackson_databind,
library.java.joda_time,
library.java.slf4j_api,
- library.java.vendored_grpc_1_13_1,
+ library.java.vendored_grpc_1_21_0,
]
-def sdk_provided_project_dependencies = [
+def sdk_provided_shaded_project_dependencies = [
":model:pipeline",
- ":runners:google-cloud-dataflow-java",
":sdks:java:core",
+]
+def sdk_provided_project_dependencies = [
+ ":runners:google-cloud-dataflow-java",
":sdks:java:extensions:google-cloud-platform-core",
":sdks:java:io:google-cloud-platform",
]
@@ -81,6 +81,7 @@
]
applyJavaNature(
+ archivesBaseName: 'beam-runners-google-cloud-dataflow-java-legacy-worker',
publish: false,
exportJavadoc: false,
enableSpotbugs: false /* TODO(BEAM-5658): enable spotbugs */,
@@ -91,8 +92,10 @@
"com/google/cloud/dataflow/worker/DataflowRunnerHarness.class",
// TODO(BEAM-6136): Enable relocation for conscrypt
"org/conscrypt/**",
+ // Allow slf4j implementation worker for logging during pipeline execution
+ "org/slf4j/impl/**"
],
- shadowClosure: DEFAULT_SHADOW_CLOSURE << {
+ shadowClosure: {
// Each included dependency must also include all of its necessary transitive dependencies
// or have them provided by the users pipeline during job submission. Typically a users
// pipeline includes :runners:google-cloud-dataflow-java and its transitive dependencies
@@ -104,26 +107,30 @@
// that the shaded jar is correctly built.
dependencies {
+ include(dependency(library.java.slf4j_jdk14))
+ }
+
+ dependencies {
include(project(path: ":model:fn-execution", configuration: "shadow"))
}
relocate("org.apache.beam.model.fnexecution.v1", getWorkerRelocatedPath("org.apache.beam.model.fnexecution.v1"))
dependencies {
- include(project(path: ":runners:core-construction-java", configuration: "shadow"))
- include(project(path: ":runners:core-java", configuration: "shadow"))
+ include(project(":runners:core-construction-java"))
+ include(project(":runners:core-java"))
}
relocate("org.apache.beam.runners.core", getWorkerRelocatedPath("org.apache.beam.runners.core"))
relocate("org.apache.beam.repackaged.beam_runners_core_construction_java", getWorkerRelocatedPath("org.apache.beam.repackaged.beam_runners_core_construction_java"))
relocate("org.apache.beam.repackaged.beam_runners_core_java", getWorkerRelocatedPath("org.apache.beam.repackaged.beam_runners_core_java"))
dependencies {
- include(project(path: ":runners:java-fn-execution", configuration: "shadow"))
+ include(project(":runners:java-fn-execution"))
}
relocate("org.apache.beam.runners.fnexecution", getWorkerRelocatedPath("org.apache.beam.runners.fnexecution"))
relocate("org.apache.beam.repackaged.beam_runners_java_fn_execution", getWorkerRelocatedPath("org.apache.beam.repackaged.beam_runners_java_fn_execution"))
dependencies {
- include(project(path: ":sdks:java:fn-execution", configuration: "shadow"))
+ include(project(":sdks:java:fn-execution"))
}
relocate("org.apache.beam.sdk.fn", getWorkerRelocatedPath("org.apache.beam.sdk.fn"))
relocate("org.apache.beam.repackaged.beam_sdks_java_fn_execution", getWorkerRelocatedPath("org.apache.beam.repackaged.beam_sdks_java_fn_execution"))
@@ -187,17 +194,20 @@
sdk_provided_dependencies.each {
provided(it)
}
- sdk_provided_project_dependencies.each {
+ sdk_provided_shaded_project_dependencies.each {
provided project(path: it, configuration: "shadow")
}
+ sdk_provided_project_dependencies.each {
+ provided project(it)
+ }
compile project(path: ":model:fn-execution", configuration: "shadow")
- compile project(path: ":runners:core-construction-java", configuration: "shadow")
- compile project(path: ":runners:core-java", configuration: "shadow")
- compile project(path: ":runners:java-fn-execution", configuration: "shadow")
- compile project(path: ":sdks:java:fn-execution", configuration: "shadow")
+ compile project(":runners:core-construction-java")
+ compile project(":runners:core-java")
+ compile project(":runners:java-fn-execution")
+ compile project(":sdks:java:fn-execution")
compile project(path: ":runners:google-cloud-dataflow-java:worker:windmill", configuration: "shadow")
- shadow library.java.vendored_guava_20_0
+ shadow library.java.vendored_guava_26_0_jre
compile "org.conscrypt:conscrypt-openjdk:1.1.3:linux-x86_64"
compile "org.eclipse.jetty:jetty-server:9.2.10.v20150310"
compile "org.eclipse.jetty:jetty-servlet:9.2.10.v20150310"
@@ -207,9 +217,8 @@
// Any test dependency which intersects with our relocation rules above needs to be relocated
// as well and placed within the testCompile configuration. Otherwise we can place it within
// the shadowTest configuration.
- testCompile project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest library.java.guava_testlib
- shadowTest project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadowTest")
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ shadowTest project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntime")
shadowTest project(path: ":runners:direct-java", configuration: "shadow")
shadowTest project(path: ":sdks:java:harness", configuration: "shadowTest")
shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
@@ -219,6 +228,25 @@
shadowTest library.java.mockito_core
}
+project.task('validateShadedJarContainsSlf4jJdk14', dependsOn: 'shadowJar') {
+ ext.outFile = project.file("${project.reportsDir}/${name}.out")
+ inputs.files project.configurations.shadow.artifacts.files
+ outputs.files outFile
+ doLast {
+ project.configurations.shadow.artifacts.files.each {
+ FileTree slf4jImpl = project.zipTree(it).matching {
+ include "org/slf4j/impl/JDK14LoggerAdapter.class"
+ }
+ outFile.text = slf4jImpl.files
+ if (slf4jImpl.files.isEmpty()) {
+ throw new GradleException("Did not find slf4j-jdk14 in Dataflow Worker uber jar")
+ }
+ }
+ }
+}
+
+tasks.check.dependsOn project.tasks.validateShadedJarContainsSlf4jJdk14
+
//TODO(BEAM-5657): checktyle task should be enabled in the future.
checkstyleMain.enabled = false
checkstyleTest.enabled = false
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ApplianceShuffleEntryReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ApplianceShuffleEntryReader.java
index ac157ff..71228c5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ApplianceShuffleEntryReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ApplianceShuffleEntryReader.java
@@ -25,7 +25,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShuffleEntryReader;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShufflePosition;
import org.apache.beam.sdk.util.common.Reiterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/** An implementation of ShuffleEntryReader that uses ApplianceShuffleReader. */
public class ApplianceShuffleEntryReader implements ShuffleEntryReader {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AssignWindowsParDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AssignWindowsParDoFnFactory.java
index a12451b..1c9f50b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AssignWindowsParDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AssignWindowsParDoFnFactory.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import static org.apache.beam.runners.dataflow.util.Structs.getBytes;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.Collection;
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReader.java
index f816f21..bfeb3ca 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReader.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.ApproximateReportedProgress;
import com.google.api.services.dataflow.model.ApproximateSplitRequest;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReaderFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReaderFactory.java
index 5870e65..160a814 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReaderFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteReaderFactory.java
@@ -29,7 +29,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates an {@link AvroByteReader} from a CloudObject spec. */
public class AvroByteReaderFactory implements ReaderFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSink.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSink.java
index f96f59d..9e0a2e9 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSink.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSink.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A sink that writes Avro files. Records are written to the Avro file as a series of byte arrays.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSinkFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSinkFactory.java
index 929ff2a..508cf71 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSinkFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/AvroByteSinkFactory.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates an {@link AvroByteSink} from a {@link CloudObject} spec. */
public final class AvroByteSinkFactory implements SinkFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
index 42b6c45..a5fb660 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorker.java
@@ -52,11 +52,11 @@
import org.apache.beam.sdk.fn.IdGenerators;
import org.apache.beam.sdk.util.Weighted;
import org.apache.beam.sdk.util.WeightedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -115,7 +115,7 @@
*/
private final Cache<?, ?> sideInputWeakReferenceCache;
- private static final int DEFAULT_STATUS_PORT = 18081;
+ private static final int DEFAULT_STATUS_PORT = 8081;
/** Status pages returning health of worker. */
private WorkerStatusPages statusPages;
@@ -209,7 +209,9 @@
.build();
this.memoryMonitor = MemoryMonitor.fromOptions(options);
- this.statusPages = WorkerStatusPages.create(DEFAULT_STATUS_PORT, this.memoryMonitor);
+ this.statusPages =
+ WorkerStatusPages.create(
+ DEFAULT_STATUS_PORT, this.memoryMonitor, sdkHarnessRegistry::sdkHarnessesAreHealthy);
if (!DataflowRunner.hasExperiment(options, "disable_debug_capture")) {
this.debugCaptureManager =
@@ -278,15 +280,12 @@
return result;
}
- private Node createPortNode(String predecessorId, String successorId) {
+ private Node createPortNode() {
return RemoteGrpcPortNode.create(
RemoteGrpcPort.newBuilder()
.setApiServiceDescriptor(sdkHarnessRegistry.beamFnDataApiServiceDescriptor())
.build(),
- idGenerator.getId(),
- idGenerator.getId(),
- predecessorId,
- successorId);
+ idGenerator.getId());
}
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
index 7933002..54a03c59 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
@@ -44,11 +44,11 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WeightedValue;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/** {@link DataflowExecutionContext} for use in batch mode. */
@@ -63,9 +63,11 @@
private final MetricsContainerRegistry<MetricsContainerImpl> containerRegistry;
- // TODO: Move throttle time Metric to a dedicated namespace.
+ // TODO(BEAM-7863): Move throttle time Metric to a dedicated namespace.
protected static final String DATASTORE_THROTTLE_TIME_NAMESPACE =
"org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$DatastoreWriterFn";
+ protected static final String HTTP_CLIENT_API_THROTTLE_TIME_NAMESPACE =
+ "org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer$LoggingHttpBackOffHandler";
private BatchModeExecutionContext(
CounterFactory counterFactory,
@@ -498,15 +500,25 @@
public Long extractThrottleTime() {
Long totalThrottleTime = 0L;
for (MetricsContainerImpl container : containerRegistry.getContainers()) {
- // TODO: Update Datastore to use generic throttling-msecs metric.
- CounterCell throttleTime =
+ // TODO(BEAM-7863): Update throttling counters to use generic throttling-msecs metric.
+ CounterCell dataStoreThrottlingTime =
container.tryGetCounter(
MetricName.named(
BatchModeExecutionContext.DATASTORE_THROTTLE_TIME_NAMESPACE,
"cumulativeThrottlingSeconds"));
- if (throttleTime != null) {
- totalThrottleTime += throttleTime.getCumulative();
+ if (dataStoreThrottlingTime != null) {
+ totalThrottleTime += dataStoreThrottlingTime.getCumulative();
}
+
+ CounterCell httpClientApiThrottlingTime =
+ container.tryGetCounter(
+ MetricName.named(
+ BatchModeExecutionContext.HTTP_CLIENT_API_THROTTLE_TIME_NAMESPACE,
+ "cumulativeThrottlingSeconds"));
+ if (httpClientApiThrottlingTime != null) {
+ totalThrottleTime += httpClientApiThrottlingTime.getCumulative();
+ }
+
CounterCell throttlingMsecs =
container.tryGetCounter(DataflowSystemMetrics.THROTTLING_MSECS_METRIC_NAME);
if (throttlingMsecs != null) {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BeamFnMapTaskExecutorFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BeamFnMapTaskExecutorFactory.java
index 10e833f..d620b8a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BeamFnMapTaskExecutorFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BeamFnMapTaskExecutorFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.client.util.Throwables;
import com.google.api.services.dataflow.model.InstructionOutput;
@@ -36,7 +36,6 @@
import java.util.List;
import java.util.Map;
import java.util.function.Function;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.ElementByteSizeObservable;
@@ -102,12 +101,12 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -289,11 +288,6 @@
Iterables.filter(network.successors(input), OutputReceiverNode.class);
Operation operation;
if (outputReceiverNodes.iterator().hasNext()) {
- Target target =
- Target.newBuilder()
- .setPrimitiveTransformReference(input.getPrimitiveTransformId())
- .setName(input.getOutputId())
- .build();
OutputReceiver[] outputReceivers =
new OutputReceiver[] {
Iterables.getOnlyElement(outputReceiverNodes).getOutputReceiver()
@@ -302,22 +296,16 @@
operation =
new RemoteGrpcPortReadOperation<>(
beamFnDataService,
- target,
+ input.getPrimitiveTransformId(),
registerFnOperation::getProcessBundleInstructionId,
(Coder) coder,
outputReceivers,
context);
} else {
- Target target =
- Target.newBuilder()
- .setPrimitiveTransformReference(input.getPrimitiveTransformId())
- .setName(input.getInputId())
- .build();
-
operation =
new RemoteGrpcPortWriteOperation<>(
beamFnDataService,
- target,
+ input.getPrimitiveTransformId(),
registerFnOperation::getProcessBundleInstructionId,
(Coder) coder,
context);
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ByteStringCoder.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ByteStringCoder.java
index 67187c8..3d9a2c4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ByteStringCoder.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ByteStringCoder.java
@@ -23,8 +23,8 @@
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.util.VarInt;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A simplified {@link Coder} for {@link ByteString}, to avoid a dependency on
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ChunkingShuffleBatchReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ChunkingShuffleBatchReader.java
index ceb39f0..7f25c81 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ChunkingShuffleBatchReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ChunkingShuffleBatchReader.java
@@ -29,7 +29,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShuffleBatchReader;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShuffleEntry;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShufflePosition;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/** ChunkingShuffleBatchReader reads data from a shuffle dataset using a ShuffleReader. */
final class ChunkingShuffleBatchReader implements ShuffleBatchReader {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactory.java
index 680780f..5b865de 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactory.java
@@ -46,7 +46,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/**
* A {@link ParDoFnFactory} to create instances of user {@link CombineFn} according to
@@ -92,6 +92,7 @@
executionContext.getStepContext(operationContext),
operationContext,
doFnInfo.getDoFnSchemaInformation(),
+ doFnInfo.getSideInputMapping(),
SimpleDoFnRunnerFactory.INSTANCE);
}
@@ -141,7 +142,8 @@
inputCoder,
Collections.emptyMap(), // Not needed here.
new TupleTag<>(PropertyNames.OUTPUT),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
private final GlobalCombineFnRunner<InputT, ?, OutputT> combineFnRunner;
@@ -206,7 +208,8 @@
inputCoder,
Collections.emptyMap(), // Not needed here.
new TupleTag<>(PropertyNames.OUTPUT),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
private final GlobalCombineFnRunner<InputT, AccumT, ?> combineFnRunner;
@@ -265,7 +268,8 @@
inputCoder,
Collections.emptyMap(), // Not needed here.
new TupleTag<>(PropertyNames.OUTPUT),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
private final GlobalCombineFnRunner<?, AccumT, ?> combineFnRunner;
@@ -314,7 +318,8 @@
inputCoder,
Collections.emptyMap(), // Not needed here.
new TupleTag<>(PropertyNames.OUTPUT),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
}
private final GlobalCombineFnRunner<?, AccumT, OutputT> combineFnRunner;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ConcatReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ConcatReader.java
index 3d2d398..b4e1081 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ConcatReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ConcatReader.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.dataflow.model.ApproximateReportedProgress;
import com.google.api.services.dataflow.model.ApproximateSplitRequest;
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.range.OffsetRangeTracker;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ContextActivationObserverRegistry.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ContextActivationObserverRegistry.java
index e57da1e..b7558af 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ContextActivationObserverRegistry.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ContextActivationObserverRegistry.java
@@ -23,9 +23,9 @@
import java.util.Set;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.util.common.ReflectHelpers.ObjectsClassComparator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.LoggerFactory;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CounterShortIdCache.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CounterShortIdCache.java
index 44cf34f..dfa3133 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CounterShortIdCache.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CounterShortIdCache.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.dataflow.model.CounterStructuredName;
import com.google.api.services.dataflow.model.CounterUpdate;
@@ -29,7 +29,7 @@
import com.google.api.services.dataflow.model.WorkItemStatus;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactory.java
index d03627a..3042bc5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.List;
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link ParDoFnFactory} that creates a system {@link ParDoFn} responsible for limiting the users
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowApiUtils.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowApiUtils.java
index 18bebd6..1900a51 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowApiUtils.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowApiUtils.java
@@ -23,8 +23,8 @@
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
/** A utility class for generic interactions with the Google Cloud Dataflow API. */
public final class DataflowApiUtils {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowBatchWorkerHarness.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowBatchWorkerHarness.java
index c47d8ec..51955f0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowBatchWorkerHarness.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowBatchWorkerHarness.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.ArrayList;
@@ -25,12 +25,13 @@
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import org.apache.beam.sdk.fn.JvmInitializers;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.util.BackOff;
import org.apache.beam.sdk.util.BackOffUtils;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.Sleeper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -53,12 +54,16 @@
/** Creates the worker harness and then runs it. */
public static void main(String[] args) throws Exception {
+ // Call user-defined initialization immediately upon starting, as is guaranteed in
+ // JvmInitializer
+ JvmInitializers.runOnStartup();
DataflowWorkerHarnessHelper.initializeLogging(DataflowBatchWorkerHarness.class);
DataflowWorkerHarnessOptions pipelineOptions =
DataflowWorkerHarnessHelper.initializeGlobalStateAndPipelineOptions(
DataflowBatchWorkerHarness.class);
DataflowBatchWorkerHarness batchHarness = new DataflowBatchWorkerHarness(pipelineOptions);
DataflowWorkerHarnessHelper.configureLogging(pipelineOptions);
+ JvmInitializers.runBeforeProcessing(pipelineOptions);
batchHarness.run();
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTracker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTracker.java
index 70016bc..492bd4e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTracker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTracker.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.time.Duration;
import java.util.ArrayDeque;
@@ -36,10 +36,10 @@
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ElementExecutionTracker;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.PeekingIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.PeekingIterator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
index 2ce168ce..181e33e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.io.Closeable;
@@ -44,8 +44,8 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Closer;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Closer;
import org.joda.time.Instant;
/** Execution context for the Dataflow worker. */
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateKey.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateKey.java
index f3f3743..17a6c84 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateKey.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateKey.java
@@ -20,8 +20,8 @@
import com.google.auto.value.AutoValue;
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ComparisonChain;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ComparisonChain;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
/**
* Execution states are uniquely identified by the step name, the state name, and for states
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateRegistry.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateRegistry.java
index 133e2ad..d636d76 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateRegistry.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateRegistry.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates.notNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates.notNull;
import com.google.api.services.dataflow.model.CounterUpdate;
import java.util.Map;
@@ -27,7 +27,7 @@
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfileScope;
import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
/** Manages the instances of {@link ExecutionState} */
public abstract class DataflowExecutionStateRegistry {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowMapTaskExecutorFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowMapTaskExecutorFactory.java
index 9d9ed22..820bfd3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowMapTaskExecutorFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowMapTaskExecutorFactory.java
@@ -28,7 +28,7 @@
import org.apache.beam.runners.fnexecution.state.GrpcStateService;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/** Creates a {@link DataflowMapTaskExecutor} from a {@link MapTask} definition. */
public interface DataflowMapTaskExecutorFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java
index 23e302e..18ce7c5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContext.java
@@ -30,6 +30,7 @@
import org.apache.beam.runners.core.SimpleDoFnRunner;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingInitializer;
@@ -37,9 +38,9 @@
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfileScope;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OperationContext;
import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.format.PeriodFormatter;
import org.joda.time.format.PeriodFormatterBuilder;
@@ -209,6 +210,19 @@
return profileScope;
}
+ @Override
+ public String getDescription() {
+ StringBuilder description = new StringBuilder();
+ description.append(getStepName().stageName());
+ description.append("-");
+ if (getStepName().originalName() != null) {
+ description.append(getStepName().originalName());
+ description.append("-");
+ }
+ description.append(getStateName());
+ return description.toString();
+ }
+
private static final ImmutableSet<String> FRAMEWORK_CLASSES =
ImmutableSet.of(SimpleDoFnRunner.class.getName(), DoFnInstanceManagers.class.getName());
@@ -282,7 +296,7 @@
.setStructuredNameAndMetadata(
new CounterStructuredNameAndMetadata()
.setName(name)
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(isCumulative)
.setInteger(longToSplitInt(value));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOutputCounter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOutputCounter.java
index 4e9e0ac..ea6b941 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOutputCounter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowOutputCounter.java
@@ -25,7 +25,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ElementCounter;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OutputObjectAndByteCounter;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A Dataflow-specific version of {@link ElementCounter}, which specifies the object counter name
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowPortabilityPCollectionView.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowPortabilityPCollectionView.java
index 9a3740e..ed95c9c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowPortabilityPCollectionView.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowPortabilityPCollectionView.java
@@ -38,6 +38,7 @@
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.WindowingStrategy;
/**
@@ -105,6 +106,11 @@
public MultimapView<K, V> apply(MultimapView<K, V> o) {
return o;
}
+
+ @Override
+ public TypeDescriptor<MultimapView<K, V>> getTypeDescriptor() {
+ throw new UnsupportedOperationException();
+ }
};
@Override
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowProcessFnRunner.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowProcessFnRunner.java
index 0c23f40..2903cc0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowProcessFnRunner.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowProcessFnRunner.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import static org.apache.beam.runners.core.SplittableParDoViaKeyedWorkItems.ProcessFn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import org.apache.beam.runners.core.DoFnRunner;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java
index 14f3e1c..eda6b03 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowRunnerHarness.java
@@ -38,8 +38,8 @@
import org.apache.beam.runners.fnexecution.control.FnApiControlClient;
import org.apache.beam.runners.fnexecution.state.GrpcStateService;
import org.apache.beam.sdk.io.FileSystems;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSideInputReadCounter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSideInputReadCounter.java
index 8b71029..fb9f844 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSideInputReadCounter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSideInputReadCounter.java
@@ -25,7 +25,7 @@
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* This class tracks time and bytes spent when consuming side inputs.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSystemMetrics.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSystemMetrics.java
index e048272..0dae356 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSystemMetrics.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowSystemMetrics.java
@@ -20,7 +20,7 @@
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/** This holds system metrics related constants used in Batch and Streaming. */
public class DataflowSystemMetrics {
@@ -39,7 +39,8 @@
JAVA_HARNESS_USED_MEMORY("dataflow_java_harness_used_memory"),
JAVA_HARNESS_MAX_MEMORY("dataflow_java_harness_max_memory"),
JAVA_HARNESS_RESTARTS("dataflow_java_harness_restarts"),
- WINDMILL_QUOTA_THROTTLING("dataflow_streaming_engine_throttled_msecs");
+ WINDMILL_QUOTA_THROTTLING("dataflow_streaming_engine_throttled_msecs"),
+ MEMORY_THRASHING("dataflow_streaming_engine_user_worker_thrashing");
private final String name;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdater.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdater.java
index f662992..e99780e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdater.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdater.java
@@ -22,13 +22,15 @@
import com.google.api.client.util.Clock;
import com.google.api.services.dataflow.model.ApproximateSplitRequest;
+import com.google.api.services.dataflow.model.HotKeyDetection;
import com.google.api.services.dataflow.model.WorkItem;
import com.google.api.services.dataflow.model.WorkItemServiceState;
import java.util.concurrent.ScheduledExecutorService;
import javax.annotation.concurrent.NotThreadSafe;
+import org.apache.beam.runners.dataflow.util.TimeUtil;
import org.apache.beam.runners.dataflow.worker.util.common.worker.WorkExecutor;
import org.apache.beam.runners.dataflow.worker.util.common.worker.WorkProgressUpdater;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -39,6 +41,7 @@
*/
@NotThreadSafe
public class DataflowWorkProgressUpdater extends WorkProgressUpdater {
+
private static final Logger LOG = LoggerFactory.getLogger(DataflowWorkProgressUpdater.class);
private final WorkItemStatusClient workItemStatusClient;
@@ -46,11 +49,14 @@
/** The WorkItem for which work progress updates are sent. */
private final WorkItem workItem;
+ private HotKeyLogger hotKeyLogger;
+
public DataflowWorkProgressUpdater(
WorkItemStatusClient workItemStatusClient, WorkItem workItem, WorkExecutor worker) {
super(worker, Integer.MAX_VALUE);
this.workItemStatusClient = workItemStatusClient;
this.workItem = workItem;
+ this.hotKeyLogger = new HotKeyLogger();
}
/**
@@ -64,10 +70,12 @@
WorkItem workItem,
WorkExecutor worker,
ScheduledExecutorService executor,
- Clock clock) {
+ Clock clock,
+ HotKeyLogger hotKeyLogger) {
super(worker, Integer.MAX_VALUE, executor, clock);
this.workItemStatusClient = workItemStatusClient;
this.workItem = workItem;
+ this.hotKeyLogger = hotKeyLogger;
}
@Override
@@ -90,7 +98,16 @@
WorkItemServiceState result =
workItemStatusClient.reportUpdate(
dynamicSplitResultToReport, Duration.millis(requestedLeaseDurationMs));
+
if (result != null) {
+ if (result.getHotKeyDetection() != null
+ && result.getHotKeyDetection().getUserStepName() != null) {
+ HotKeyDetection hotKeyDetection = result.getHotKeyDetection();
+ hotKeyLogger.logHotKeyDetection(
+ hotKeyDetection.getUserStepName(),
+ TimeUtil.fromCloudDuration(hotKeyDetection.getHotKeyAge()));
+ }
+
// Resets state after a successful progress report.
dynamicSplitResultToReport = null;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClient.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClient.java
index b383b06..5c4a9c0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClient.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClient.java
@@ -24,7 +24,7 @@
import static org.apache.beam.runners.dataflow.worker.util.WorkerPropertyNames.WORK_ITEM_TYPE_REMOTE_SOURCE_TASK;
import static org.apache.beam.runners.dataflow.worker.util.WorkerPropertyNames.WORK_ITEM_TYPE_SEQ_MAP_TASK;
import static org.apache.beam.runners.dataflow.worker.util.WorkerPropertyNames.WORK_ITEM_TYPE_STREAMING_CONFIG_TASK;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import com.google.api.services.dataflow.Dataflow;
import com.google.api.services.dataflow.model.LeaseWorkItemRequest;
@@ -43,9 +43,9 @@
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingMDC;
import org.apache.beam.runners.dataflow.worker.util.common.worker.WorkProgressUpdater;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.Interval;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelper.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelper.java
index bac5fd7..a990c38 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelper.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelper.java
@@ -30,7 +30,7 @@
import org.apache.beam.runners.dataflow.worker.ExperimentContext.Experiment;
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingInitializer;
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingMDC;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
import org.conscrypt.OpenSSLProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactory.java
index 85fda36..9dc8b1e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactory.java
@@ -24,7 +24,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoFn;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A factory that dispatches to all known factories in the Dataflow SDK based on the value of {@link
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DoFnRunnerFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DoFnRunnerFactory.java
index 004a49b..93eccca 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DoFnRunnerFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DoFnRunnerFactory.java
@@ -44,5 +44,6 @@
DataflowExecutionContext.DataflowStepContext stepContext,
DataflowExecutionContext.DataflowStepContext userStepContext,
OutputManager outputManager,
- DoFnSchemaInformation doFnSchemaInformation);
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping);
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ExperimentContext.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ExperimentContext.java
index 080514e..6749dd7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ExperimentContext.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ExperimentContext.java
@@ -21,8 +21,8 @@
import java.util.Set;
import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A convenient class to provide fast lookup of enabled experiments in the worker code.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FetchAndFilterStreamingSideInputsOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FetchAndFilterStreamingSideInputsOperation.java
index 91a779d..6c1ddde 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FetchAndFilterStreamingSideInputsOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FetchAndFilterStreamingSideInputsOperation.java
@@ -50,7 +50,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* This {@link ReceivingOperation} is responsible for fetching any ready side inputs and also
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Filepatterns.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Filepatterns.java
index 3e3aeee..3c6c356 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Filepatterns.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Filepatterns.java
@@ -19,7 +19,7 @@
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Utilities for handling filepatterns. */
public class Filepatterns {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java
index edd0151..7e298e7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFn.java
@@ -17,14 +17,13 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.IOException;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutionException;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor;
@@ -57,9 +56,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -229,7 +228,7 @@
.setInstructionId(processRequestInstructionId)
.setProcessBundle(
ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference(registerIfRequired()))
+ .setProcessBundleDescriptorId(registerIfRequired()))
.build();
ConcurrentLinkedQueue<WindowedValue<KV<byte[], TargetWindowT>>> outputValue =
@@ -238,12 +237,7 @@
// Open the inbound consumer
InboundDataClient waitForInboundTermination =
beamFnDataService.receive(
- LogicalEndpoint.of(
- processRequestInstructionId,
- BeamFnApi.Target.newBuilder()
- .setName("out")
- .setPrimitiveTransformReference("write")
- .build()),
+ LogicalEndpoint.of(processRequestInstructionId, "write"),
inboundCoder,
outputValue::add);
@@ -253,13 +247,7 @@
// Open the outbound consumer
try (CloseableFnDataReceiver<WindowedValue<KV<byte[], BoundedWindow>>> outboundConsumer =
beamFnDataService.send(
- LogicalEndpoint.of(
- processRequestInstructionId,
- BeamFnApi.Target.newBuilder()
- .setName("in")
- .setPrimitiveTransformReference("read")
- .build()),
- outboundCoder)) {
+ LogicalEndpoint.of(processRequestInstructionId, "read"), outboundCoder)) {
outboundConsumer.accept(WindowedValue.valueInGlobalWindow(KV.of(EMPTY_ARRAY, mainWindow)));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactory.java
index 5e90250..b12f889 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactory.java
@@ -20,7 +20,7 @@
import static org.apache.beam.runners.dataflow.util.Structs.getBytes;
import static org.apache.beam.runners.dataflow.util.Structs.getObject;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.ArrayList;
@@ -58,8 +58,8 @@
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowsParDoFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowsParDoFn.java
index 80d4246..89bccb6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowsParDoFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowsParDoFn.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.List;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
index 676a9d5..36df60d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
@@ -21,7 +21,7 @@
import static org.apache.beam.runners.dataflow.worker.SourceTranslationUtils.cloudPositionToReaderPosition;
import static org.apache.beam.runners.dataflow.worker.SourceTranslationUtils.cloudProgressToReaderProgress;
import static org.apache.beam.runners.dataflow.worker.SourceTranslationUtils.splitRequestToApproximateSplitRequest;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.ApproximateReportedProgress;
import com.google.api.services.dataflow.model.ApproximateSplitRequest;
@@ -56,7 +56,7 @@
import org.apache.beam.sdk.util.common.Reiterable;
import org.apache.beam.sdk.util.common.Reiterator;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderFactory.java
index a6098ee..c2bfe23 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderFactory.java
@@ -20,7 +20,7 @@
import static com.google.api.client.util.Base64.decodeBase64;
import static org.apache.beam.runners.dataflow.util.Structs.getBoolean;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.util.Map;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates a GroupingShuffleReader from a CloudObject spec. */
public class GroupingShuffleReaderFactory implements ReaderFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderWithFaultyBytesReadCounter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderWithFaultyBytesReadCounter.java
index 4bd3501..25da6ca 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderWithFaultyBytesReadCounter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderWithFaultyBytesReadCounter.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A source that reads from a shuffled dataset and yields key-grouped data. Like {@link
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/HotKeyLogger.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/HotKeyLogger.java
new file mode 100644
index 0000000..212f231
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/HotKeyLogger.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker;
+
+import com.google.api.client.util.Clock;
+import java.text.MessageFormat;
+import org.apache.beam.runners.dataflow.util.TimeUtil;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class HotKeyLogger {
+ Logger LOG = LoggerFactory.getLogger(HotKeyLogger.class);
+
+ /** Clock used to either provide real system time or mocked to virtualize time for testing. */
+ private Clock clock = Clock.SYSTEM;
+
+ /**
+ * The previous time the HotKeyDetection was logged. This is used to throttle logging to every 5
+ * minutes.
+ */
+ private long prevHotKeyDetectionLogMs = 0;
+
+ /** Throttles logging the detection to every loggingPeriod */
+ private final Duration loggingPeriod = Duration.standardMinutes(5);
+
+ HotKeyLogger() {}
+
+ HotKeyLogger(Clock clock) {
+ this.clock = clock;
+ }
+
+ /** Logs a detection of the hot key every 5 minutes. */
+ public void logHotKeyDetection(String userStepName, Duration hotKeyAge) {
+ if (isThrottled()) {
+ return;
+ }
+ LOG.warn(getHotKeyMessage(userStepName, TimeUtil.toCloudDuration(hotKeyAge)));
+ }
+
+ /**
+ * Returns true if the class should log the HotKeyMessage. This method throttles logging to every
+ * 5 minutes.
+ */
+ protected boolean isThrottled() {
+ // Throttle logging the HotKeyDetection to every 5 minutes.
+ long nowMs = clock.currentTimeMillis();
+ if (nowMs - prevHotKeyDetectionLogMs < loggingPeriod.getMillis()) {
+ return true;
+ }
+ prevHotKeyDetectionLogMs = nowMs;
+ return false;
+ }
+
+ protected String getHotKeyMessage(String userStepName, String hotKeyAge) {
+ return MessageFormat.format(
+ "A hot key was detected in step ''{0}'' with age of ''{1}''. This is"
+ + " a symptom of key distribution being skewed. To fix, please inspect your data and "
+ + "pipeline to ensure that elements are evenly distributed across your key space.",
+ userStepName, hotKeyAge);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReader.java
index 38c3edc..8e72413 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReader.java
@@ -18,8 +18,8 @@
package org.apache.beam.runners.dataflow.worker;
import static com.google.api.client.util.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.ApproximateReportedProgress;
import java.io.IOException;
@@ -33,8 +33,8 @@
import org.apache.beam.sdk.io.range.OffsetRangeTracker;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.StringUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReaderFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReaderFactory.java
index 64ffb2f..dae4668 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReaderFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/InMemoryReaderFactory.java
@@ -29,7 +29,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates an InMemoryReader from a CloudObject spec. */
public class InMemoryReaderFactory implements ReaderFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactory.java
index 5b17cbc..b827d6c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.InstructionOutput;
import com.google.api.services.dataflow.model.MapTask;
@@ -75,10 +75,10 @@
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReader.java
index 040a8dc..28b9d0e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReader.java
@@ -26,7 +26,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A {@link NativeReader} that reads Ism files.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactory.java
index 9f3acd1..08cc59e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactory.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.util.Map;
@@ -38,8 +38,8 @@
import org.apache.beam.sdk.util.WeightedValue;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Creates an {@link IsmReader} from a {@link CloudObject} spec. Note that it is invalid to use a
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderImpl.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderImpl.java
index a2be8f0..8f0a1f6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderImpl.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmReaderImpl.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.Closeable;
@@ -62,18 +62,18 @@
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.util.WeightedValue;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSortedMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Longs;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Longs;
/**
* A {@link NativeReader} that reads Ism files.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReader.java
index 24bfaf7..b1ea344 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReader.java
@@ -19,12 +19,13 @@
import static org.apache.beam.runners.dataflow.util.Structs.addString;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import com.google.api.services.dataflow.model.Source;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.IOException;
import java.util.AbstractList;
import java.util.AbstractMap;
@@ -44,6 +45,7 @@
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
+import javax.annotation.Nonnull;
import org.apache.beam.runners.core.SideInputReader;
import org.apache.beam.runners.dataflow.internal.IsmFormat;
import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
@@ -72,16 +74,16 @@
import org.apache.beam.sdk.values.PCollectionViews.MultimapViewFn;
import org.apache.beam.sdk.values.PCollectionViews.SingletonViewFn;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
/**
* A side input reader over a set of {@link IsmFormat} files constructed by Dataflow. This reader
@@ -1012,8 +1014,11 @@
private static class MapToValue<K, V>
implements Function<KV<K, IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator>, V> {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public V apply(KV<K, IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator> input) {
+ public V apply(@Nonnull KV<K, IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator> input) {
IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator startedReader = input.getValue();
WindowedValue<IsmRecord<WindowedValue<V>>> value = startedReader.getCurrent();
return value.getValue().getValue().getValue();
@@ -1026,8 +1031,13 @@
*/
private class MapToIterable<K, V>
implements Function<KV<K, IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator>, Iterable<V>> {
+
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public Iterable<V> apply(KV<K, IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator> input) {
+ public Iterable<V> apply(
+ @Nonnull KV<K, IsmReader<WindowedValue<V>>.IsmPrefixReaderIterator> input) {
try {
return Iterables.unmodifiableIterable(
new ListOverReaderIterators<>(
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSink.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSink.java
index 8b3017d..88eb96b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSink.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSink.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.nio.channels.Channels;
@@ -46,9 +46,9 @@
import org.apache.beam.sdk.util.MimeTypes;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
/**
* A {@link Sink} that writes Ism files.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSinkFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSinkFactory.java
index 7fc71ac..5d2bec5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSinkFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/IsmSinkFactory.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.math.RoundingMode;
@@ -35,8 +35,8 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.math.DoubleMath;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.DoubleMath;
/**
* Creates an {@link IsmSink} from a {@link CloudObject} spec. Note that it is invalid to use a non
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReader.java
index 8f0ae34..92245f9b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReader.java
@@ -25,8 +25,8 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
/**
* A {@link SideInputReader} which initializes on first {@link #get(PCollectionView, BoundedWindow)}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricTrackingWindmillServerStub.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricTrackingWindmillServerStub.java
index 6c88d58..734f49e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricTrackingWindmillServerStub.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricTrackingWindmillServerStub.java
@@ -30,8 +30,8 @@
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.KeyedGetDataRequest;
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub;
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.GetDataStream;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.SettableFuture;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture;
import org.joda.time.Duration;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsContainerRegistry.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsContainerRegistry.java
index 214ff89..8120788 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsContainerRegistry.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsContainerRegistry.java
@@ -20,7 +20,7 @@
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.ConcurrentSkipListMap;
import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
/**
* Manages the instances of {@link MetricsContainer} that have been created for a specific context.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsEnvironmentContextActivationObserverRegistration.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsEnvironmentContextActivationObserverRegistration.java
index b9499af..9500170 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsEnvironmentContextActivationObserverRegistration.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsEnvironmentContextActivationObserverRegistration.java
@@ -22,7 +22,7 @@
import java.io.IOException;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Registration of a {@link ContextActivationObserver} for the {@link MetricsEnvironment} with the
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java
index 398c2ec..710fef1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToCounterUpdateConverter.java
@@ -52,6 +52,7 @@
/** Well-defined {@code kind} strings for use in {@link CounterUpdate} protos. */
public enum Kind {
DISTRIBUTION("DISTRIBUTION"),
+ MEAN("MEAN"),
SUM("SUM");
private final String kind;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/OrderedCode.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/OrderedCode.java
index 2d87fa7..84036e8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/OrderedCode.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/OrderedCode.java
@@ -20,8 +20,8 @@
import java.math.RoundingMode;
import java.util.ArrayList;
import java.util.Arrays;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.math.LongMath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Longs;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Longs;
/**
* This module provides routines for encoding a sequence of typed entities into a byte array. The
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactory.java
index a73b584..8a6ab60 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.List;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFns.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFns.java
index b9930c7..ffb6170 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFns.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFns.java
@@ -47,9 +47,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
import org.joda.time.Instant;
/** A factory class that creates {@link ParDoFn} for {@link PartialGroupByKeyInstruction}. */
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReader.java
index 7bbbdd2..b58747c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReader.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A source that reads from a key-sharded dataset, and returns KVs without any values grouping.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderFactory.java
index e11ba29..d0d6e9c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderFactory.java
@@ -19,7 +19,7 @@
import static com.google.api.client.util.Base64.decodeBase64;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.util.Map;
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates a PartitioningShuffleReader from a CloudObject spec. */
public class PartitioningShuffleReaderFactory implements ReaderFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubReader.java
index 81edab2..5d3dd91 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubReader.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import static org.apache.beam.runners.dataflow.util.Structs.getBytes;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -37,7 +37,7 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** A Reader that receives elements from Pubsub, via a Windmill server. */
class PubsubReader<T> extends NativeReader<WindowedValue<T>> {
@@ -122,7 +122,9 @@
value =
parseFn.apply(
new PubsubMessage(
- pubsubMessage.getData().toByteArray(), pubsubMessage.getAttributes()));
+ pubsubMessage.getData().toByteArray(),
+ pubsubMessage.getAttributesMap(),
+ pubsubMessage.getMessageId()));
} else {
value = coder.decode(data, Coder.Context.OUTER);
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubSink.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubSink.java
index 00e6321..147fd76 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubSink.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/PubsubSink.java
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A sink that writes to Pubsub, via a Windmill server.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderCache.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderCache.java
index fa9ed8d..6b00560 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderCache.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderCache.java
@@ -22,12 +22,12 @@
import javax.annotation.concurrent.ThreadSafe;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalCause;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalNotification;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalCause;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalNotification;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderRegistry.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderRegistry.java
index 682cfab..3535fec 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderRegistry.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReaderRegistry.java
@@ -30,10 +30,10 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* An immutable registry from {@link String} identifiers (provided to the worker by the Dataflow
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactory.java
index 2ae1c98..b7e2288 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.List;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrar.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrar.java
index 98dad29..fb22bbd 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrar.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrar.java
@@ -37,7 +37,7 @@
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.util.InstanceBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A registrar for {@link CloudObjectTranslator}s for the Dataflow runner harness.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistries.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistries.java
index 2f22cbd..11be787 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistries.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistries.java
@@ -29,7 +29,7 @@
import org.apache.beam.runners.fnexecution.control.FnApiControlClient;
import org.apache.beam.runners.fnexecution.data.GrpcDataService;
import org.apache.beam.runners.fnexecution.state.GrpcStateService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -59,6 +59,7 @@
private final BeamFnDataGrpcService beamFnDataGrpcService;
private final ConcurrentHashMap<FnApiControlClient, WorkCountingSdkWorkerHarness> workerMap =
new ConcurrentHashMap<>();
+ private final AtomicBoolean sdkHarnessesAreHealthy = new AtomicBoolean(true);
private final PriorityBlockingQueue<WorkCountingSdkWorkerHarness> workers =
new PriorityBlockingQueue<>(
@@ -107,6 +108,17 @@
workers.remove(worker);
}
LOG.info("Unregistered Control client {}", worker != null ? worker.getWorkerId() : null);
+
+ // unregisterWorkerClient() will be called only when the connection between SDK harness and
+ // runner harness is broken or SDK harness respond to runner harness with an error. In either
+ // case, the SDK should be marked as unhealthy.
+ sdkHarnessesAreHealthy.set(false);
+ LOG.info("SDK harness {} became unhealthy", worker != null ? worker.getWorkerId() : null);
+ }
+
+ @Override
+ public boolean sdkHarnessesAreHealthy() {
+ return sdkHarnessesAreHealthy.get();
}
/* Any modification to workers has race condition with unregisterWorkerClient. To resolve this
@@ -254,6 +266,11 @@
}
@Override
+ public boolean sdkHarnessesAreHealthy() {
+ return true;
+ }
+
+ @Override
public SdkWorkerHarness getAvailableWorkerAndAssignWork() {
return sdkWorkerHarness;
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistry.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistry.java
index e25c87b..7caf7fc 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistry.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SdkHarnessRegistry.java
@@ -40,6 +40,9 @@
*/
void unregisterWorkerClient(FnApiControlClient controlClient);
+ /** Returns true if all of the registered SDK harnesses are healthy. */
+ boolean sdkHarnessesAreHealthy();
+
/** Find the available worker and assign work to it or wait till a worker becomes available */
SdkWorkerHarness getAvailableWorkerAndAssignWork();
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSink.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSink.java
index d2f2e3c..2ad66b5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSink.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSink.java
@@ -37,8 +37,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
/**
* A sink that writes to a shuffle dataset.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkFactory.java
index 49f6992..2e955b7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkFactory.java
@@ -20,7 +20,7 @@
import static com.google.api.client.util.Base64.decodeBase64;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
import static org.apache.beam.runners.dataflow.worker.ShuffleSink.parseShuffleKind;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.util.Map;
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates a {@link ShuffleSink} from a {@link CloudObject} spec. */
public class ShuffleSinkFactory implements SinkFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleDoFnRunnerFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleDoFnRunnerFactory.java
index 2601cea..0804a32 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleDoFnRunnerFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleDoFnRunnerFactory.java
@@ -49,7 +49,8 @@
DataflowExecutionContext.DataflowStepContext stepContext,
DataflowExecutionContext.DataflowStepContext userStepContext,
OutputManager outputManager,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
DoFnRunner<InputT, OutputT> fnRunner =
DoFnRunners.simpleRunner(
options,
@@ -62,7 +63,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
boolean hasStreamingSideInput =
options.as(StreamingOptions.class).isStreaming() && !sideInputReader.isEmpty();
if (hasStreamingSideInput) {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
index 6951e8b..712a017 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Closeable;
import java.util.Collection;
@@ -54,10 +54,11 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.DoFnInfo;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -93,6 +94,7 @@
private final boolean hasStreamingSideInput;
private final OutputsPerElementTracker outputsPerElementTracker;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
// Various DoFn helpers, null between bundles
@Nullable private DoFnRunner<InputT, OutputT> fnRunner;
@@ -113,6 +115,7 @@
DataflowExecutionContext.DataflowStepContext stepContext,
DataflowOperationContext operationContext,
DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping,
DoFnRunnerFactory runnerFactory) {
this.options = options;
this.doFnInstanceManager = doFnInstanceManager;
@@ -143,6 +146,7 @@
options.as(StreamingOptions.class).isStreaming() && !sideInputReader.isEmpty();
this.outputsPerElementTracker = createOutputsPerElementTracker();
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
private OutputsPerElementTracker createOutputsPerElementTracker() {
@@ -302,7 +306,8 @@
stepContext,
userStepContext,
outputManager,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
fnRunner.startBundle();
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SinkRegistry.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SinkRegistry.java
index 4caaec6..5962568 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SinkRegistry.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SinkRegistry.java
@@ -27,9 +27,9 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* An immutable registry from {@link String} identifiers (provided to the worker by the Dataflow
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SizeReportingSinkWrapper.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SizeReportingSinkWrapper.java
index d32d019..4005202 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SizeReportingSinkWrapper.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SizeReportingSinkWrapper.java
@@ -19,7 +19,7 @@
import java.io.IOException;
import org.apache.beam.runners.dataflow.worker.util.common.worker.Sink;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A wrapper for Sink that reports bytes buffered (or written) to {@link DataflowExecutionContext}.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SourceOperationExecutorFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SourceOperationExecutorFactory.java
index 9b7c47c..32c9dee 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SourceOperationExecutorFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SourceOperationExecutorFactory.java
@@ -23,7 +23,7 @@
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/** Creates a SourceOperationExecutor from a SourceOperation. */
public class SourceOperationExecutorFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SplittableProcessFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SplittableProcessFnFactory.java
index b90c49b..b840c68 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SplittableProcessFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SplittableProcessFnFactory.java
@@ -96,7 +96,8 @@
doFnInfo.getWindowingStrategy().getWindowFn().windowCoder()),
doFnInfo.getOutputCoders(),
doFnInfo.getMainOutput(),
- doFnInfo.getDoFnSchemaInformation());
+ doFnInfo.getDoFnSchemaInformation(),
+ doFnInfo.getSideInputMapping());
}
}
@@ -121,7 +122,8 @@
DataflowExecutionContext.DataflowStepContext stepContext,
DataflowExecutionContext.DataflowStepContext userStepContext,
OutputManager outputManager,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
ProcessFn<InputT, OutputT, RestrictionT, TrackerT> processFn =
(ProcessFn<InputT, OutputT, RestrictionT, TrackerT>) fn;
processFn.setStateInternalsFactory(key -> (StateInternals) stepContext.stateInternals());
@@ -170,7 +172,8 @@
inputCoder,
outputCoders,
processFn.getInputWindowingStrategy(),
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
DoFnRunner<KeyedWorkItem<byte[], KV<InputT, RestrictionT>>, OutputT> fnRunner =
new DataflowProcessFnRunner<>(simpleRunner);
boolean hasStreamingSideInput =
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StateFetcher.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StateFetcher.java
index 9f8bbf2..3c804db 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StateFetcher.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StateFetcher.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Closeable;
import java.util.Collections;
@@ -38,12 +38,12 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Weigher;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Weigher;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
index a2bb479..a603d16 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
@@ -20,7 +20,7 @@
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.beam.runners.dataflow.DataflowRunner.hasExperiment;
import static org.apache.beam.runners.dataflow.worker.DataflowSystemMetrics.THROTTLING_MSECS_METRIC_NAME;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.CounterStructuredName;
import com.google.api.services.dataflow.model.CounterUpdate;
@@ -78,6 +78,7 @@
import org.apache.beam.runners.dataflow.worker.apiary.FixMultiOutputInfosOnParDoInstructions;
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
+import org.apache.beam.runners.dataflow.worker.counters.CounterUpdateAggregators;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.graph.CloneAmbiguousFlattensFunction;
@@ -118,6 +119,7 @@
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.JvmInitializers;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.util.BackOff;
import org.apache.beam.sdk.util.BackOffUtils;
@@ -125,19 +127,21 @@
import org.apache.beam.sdk.util.Sleeper;
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.EvictingQueue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultimapBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.EvictingQueue;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.MultimapBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -196,6 +200,8 @@
/** Maximum number of failure stacktraces to report in each update sent to backend. */
private static final int MAX_FAILURES_TO_REPORT_IN_UPDATE = 1000;
+ private final AtomicLong counterAggregationErrorCount = new AtomicLong();
+
/** Returns whether an exception was caused by a {@link OutOfMemoryError}. */
private static boolean isOutOfMemoryError(Throwable t) {
while (t != null) {
@@ -241,6 +247,8 @@
}
public static void main(String[] args) throws Exception {
+ JvmInitializers.runOnStartup();
+
DataflowWorkerHarnessHelper.initializeLogging(StreamingDataflowWorker.class);
DataflowWorkerHarnessOptions options =
DataflowWorkerHarnessHelper.initializeGlobalStateAndPipelineOptions(
@@ -261,6 +269,7 @@
StreamingDataflowWorker worker =
StreamingDataflowWorker.fromDataflowWorkerHarnessOptions(options, sdkHarnessRegistry);
+ JvmInitializers.runBeforeProcessing(options);
worker.startStatusPages();
worker.start();
}
@@ -395,9 +404,13 @@
private final StreamingDataflowWorkerOptions options;
private final boolean windmillServiceEnabled;
private final long clientId;
+
private final MetricTrackingWindmillServerStub metricTrackingWindmillServer;
private final CounterSet pendingDeltaCounters = new CounterSet();
private final CounterSet pendingCumulativeCounters = new CounterSet();
+ private final java.util.concurrent.ConcurrentLinkedQueue<CounterUpdate> pendingMonitoringInfos =
+ new ConcurrentLinkedQueue<>();
+
// Map from stage name to StageInfo containing metrics container registry and per stage counters.
private final ConcurrentMap<String, StageInfo> stageInfoMap = new ConcurrentHashMap();
@@ -410,6 +423,7 @@
private final Counter<Long, Long> javaHarnessUsedMemory;
private final Counter<Long, Long> javaHarnessMaxMemory;
private final Counter<Integer, Integer> windmillMaxObservedWorkItemCommitBytes;
+ private final Counter<Integer, Integer> memoryThrashing;
private Timer refreshActiveWorkTimer;
private Timer statusPageTimer;
@@ -454,6 +468,8 @@
private final ReaderRegistry readerRegistry = ReaderRegistry.defaultRegistry();
private final SinkRegistry sinkRegistry = SinkRegistry.defaultRegistry();
+ private HotKeyLogger hotKeyLogger;
+
/** Contains a few of the stage specific fields. E.g. metrics container registry, counters etc. */
private static class StageInfo {
@@ -527,7 +543,8 @@
options.as(StreamingDataflowWorkerOptions.class),
pipeline,
sdkHarnessRegistry,
- true);
+ true,
+ new HotKeyLogger());
}
public static StreamingDataflowWorker fromDataflowWorkerHarnessOptions(
@@ -541,7 +558,8 @@
options.as(StreamingDataflowWorkerOptions.class),
null,
sdkHarnessRegistry,
- true);
+ true,
+ new HotKeyLogger());
}
@VisibleForTesting
@@ -552,15 +570,19 @@
StreamingDataflowWorkerOptions options,
@Nullable RunnerApi.Pipeline pipeline,
SdkHarnessRegistry sdkHarnessRegistry,
- boolean publishCounters)
+ boolean publishCounters,
+ HotKeyLogger hotKeyLogger)
throws IOException {
this.mapTaskExecutorFactory = mapTaskExecutorFactory;
this.workUnitClient = workUnitClient;
this.options = options;
this.sdkHarnessRegistry = sdkHarnessRegistry;
+ this.hotKeyLogger = hotKeyLogger;
this.windmillServiceEnabled = options.isEnableStreamingEngine();
this.memoryMonitor = MemoryMonitor.fromOptions(options);
- this.statusPages = WorkerStatusPages.create(DEFAULT_STATUS_PORT, memoryMonitor);
+ this.statusPages =
+ WorkerStatusPages.create(
+ DEFAULT_STATUS_PORT, memoryMonitor, sdkHarnessRegistry::sdkHarnessesAreHealthy);
if (windmillServiceEnabled) {
this.debugCaptureManager =
new DebugCapture.Manager(options, statusPages.getDebugCapturePages());
@@ -586,6 +608,9 @@
this.windmillMaxObservedWorkItemCommitBytes =
pendingCumulativeCounters.intMax(
StreamingSystemCounterNames.WINDMILL_MAX_WORK_ITEM_COMMIT_BYTES.counterName());
+ this.memoryThrashing =
+ pendingCumulativeCounters.intSum(
+ StreamingSystemCounterNames.MEMORY_THRASHING.counterName());
this.isDoneFuture = new CompletableFuture<>();
this.threadFactory =
@@ -698,15 +723,12 @@
LOG.debug("maxWorkItemCommitBytes: {}", maxWorkItemCommitBytes);
}
- private Node createPortNode(String predecessorId, String successorId) {
+ private Node createPortNode() {
return RemoteGrpcPortNode.create(
RemoteGrpcPort.newBuilder()
.setApiServiceDescriptor(sdkHarnessRegistry.beamFnDataApiServiceDescriptor())
.build(),
- idGenerator.getId(),
- idGenerator.getId(),
- predecessorId,
- successorId);
+ idGenerator.getId());
}
private int chooseMaximumNumberOfThreads() {
@@ -1012,6 +1034,17 @@
Preconditions.checkState(
outputDataWatermark == null || !outputDataWatermark.isAfter(inputDataWatermark));
SdkWorkerHarness worker = sdkHarnessRegistry.getAvailableWorkerAndAssignWork();
+
+ if (workItem.hasHotKeyInfo()) {
+ Windmill.HotKeyInfo hotKeyInfo = workItem.getHotKeyInfo();
+ Duration hotKeyAge = Duration.millis(hotKeyInfo.getHotKeyAgeUsec() / 1000);
+
+ // The MapTask instruction is ordered by dependencies, such that the first element is
+ // always going to be the shuffle task.
+ String stepName = computationState.getMapTask().getInstructions().get(0).getName();
+ hotKeyLogger.logHotKeyDetection(stepName, hotKeyAge);
+ }
+
Work work =
new Work(workItem) {
@Override
@@ -1030,7 +1063,11 @@
}
}
};
- computationState.activateWork(workItem.getKey(), work);
+ if (!computationState.activateWork(workItem.getKey(), work)) {
+ // Free worker if the work was not activated.
+ // This can happen if it's duplicate work or some other reason.
+ sdkHarnessRegistry.completeWork(worker);
+ }
}
abstract static class Work implements Runnable {
@@ -1123,7 +1160,7 @@
final ByteString key = workItem.getKey();
work.setState(State.PROCESSING);
DataflowWorkerLoggingMDC.setWorkId(
- key.toStringUtf8() + "-" + Long.toString(workItem.getWorkToken()));
+ TextFormat.escapeBytes(key) + "-" + Long.toString(workItem.getWorkToken()));
DataflowWorkerLoggingMDC.setStageName(computationId);
LOG.debug("Starting processing for {}:\n{}", computationId, work);
@@ -1282,6 +1319,9 @@
// Blocks while executing work.
executionState.getWorkExecutor().execute();
+ Iterables.addAll(
+ this.pendingMonitoringInfos, executionState.getWorkExecutor().extractMetricUpdates());
+
commitCallbacks.putAll(executionState.getContext().flushState());
// Release the execution state for another thread to use.
@@ -1840,9 +1880,11 @@
/** Sends counter updates to Dataflow backend. */
private void sendWorkerUpdatesToDataflowService(
CounterSet deltaCounters, CounterSet cumulativeCounters) throws IOException {
-
// Throttle time is tracked by the windmillServer but is reported to DFE here.
windmillQuotaThrottling.addValue(windmillServer.getAndResetThrottleTime());
+ if (memoryMonitor.isThrashing()) {
+ memoryThrashing.addValue(1);
+ }
List<CounterUpdate> counterUpdates = new ArrayList<>(128);
@@ -1852,6 +1894,63 @@
cumulativeCounters.extractUpdates(false, DataflowCounterUpdateExtractor.INSTANCE));
counterUpdates.addAll(
deltaCounters.extractModifiedDeltaUpdates(DataflowCounterUpdateExtractor.INSTANCE));
+ if (hasExperiment(options, "beam_fn_api")) {
+ Map<Object, List<CounterUpdate>> fnApiCounters = new HashMap<>();
+
+ while (!this.pendingMonitoringInfos.isEmpty()) {
+ final CounterUpdate item = this.pendingMonitoringInfos.poll();
+
+ // This change will treat counter as delta.
+ // This is required because we receive cumulative results from FnAPI harness,
+ // while streaming job is expected to receive delta updates to counters on same
+ // WorkItem.
+ if (item.getCumulative()) {
+ item.setCumulative(false);
+ // Group counterUpdates by counterUpdateKey so they can be aggregated before sending to
+ // dataflow service.
+ fnApiCounters
+ .computeIfAbsent(getCounterUpdateKey(item), k -> new ArrayList<>())
+ .add(item);
+ } else {
+ // In current world all counters coming from FnAPI are cumulative.
+ // This is a safety check in case new counter type appears in FnAPI.
+ throw new UnsupportedOperationException(
+ "FnApi counters are expected to provide cumulative values."
+ + " Please, update conversion to delta logic"
+ + " if non-cumulative counter type is required.");
+ }
+ }
+
+ // Aggregates counterUpdates with same counterUpdateKey to single CounterUpdate if possible
+ // so we can avoid excessive I/Os for reporting to dataflow service.
+ for (List<CounterUpdate> counterUpdateList : fnApiCounters.values()) {
+ if (counterUpdateList.isEmpty()) {
+ continue;
+ }
+ List<CounterUpdate> aggregatedCounterUpdateList =
+ CounterUpdateAggregators.aggregate(counterUpdateList);
+
+ // Log a warning message if encountered enough non-aggregatable counter updates since this
+ // can lead to a severe performance penalty if dataflow service can not handle the
+ // updates.
+ if (aggregatedCounterUpdateList.size() > 10) {
+ CounterUpdate head = aggregatedCounterUpdateList.get(0);
+ this.counterAggregationErrorCount.getAndIncrement();
+ // log warning message only when error count is the power of 2 to avoid spamming.
+ if (this.counterAggregationErrorCount.get() > 10
+ && Long.bitCount(this.counterAggregationErrorCount.get()) == 1) {
+ LOG.warn(
+ "Found non-aggregated counter updates of size {} with kind {}, this will likely "
+ + "cause performance degradation and excessive GC if size is large.",
+ counterUpdateList.size(),
+ MoreObjects.firstNonNull(
+ head.getNameAndKind(), head.getStructuredNameAndMetadata()));
+ }
+ }
+
+ counterUpdates.addAll(aggregatedCounterUpdateList);
+ }
+ }
}
// Handle duplicate counters from different stages. Store all the counters in a multi-map and
@@ -1911,6 +2010,7 @@
.setWorkItemId(WINDMILL_COUNTER_UPDATE_WORK_ID)
.setErrors(errors)
.setCounterUpdates(counterUpdates);
+
workUnitClient.reportWorkItemStatus(workItemStatus);
// Send any counters appearing more than once in subsequent RPCs:
@@ -1994,7 +2094,7 @@
}
/** Mark the given key and work as active. */
- public void activateWork(ByteString key, Work work) {
+ public boolean activateWork(ByteString key, Work work) {
synchronized (activeWork) {
Queue<Work> queue = activeWork.get(key);
if (queue == null) {
@@ -2004,12 +2104,16 @@
// Fall through to execute without the lock held.
} else {
if (queue.peek().getWorkItem().getWorkToken() != work.getWorkItem().getWorkToken()) {
+ // Queue the work for later processing.
queue.add(work);
+ return true;
}
- return;
+ // Skip the work if duplicate
+ return false;
}
}
executor.execute(work);
+ return true;
}
/** Marks the work for a the given key as complete. Schedules queued work for the key if any. */
@@ -2019,17 +2123,22 @@
Queue<Work> queue = activeWork.get(key);
Preconditions.checkNotNull(queue);
Work completedWork = queue.poll();
- Preconditions.checkNotNull(
- completedWork,
- "No active state for key %s, expected token %s",
- key.toStringUtf8(),
- workToken);
- Preconditions.checkState(
- completedWork.getWorkItem().getWorkToken() == workToken,
- "Token mismatch for key %s: %s and %s",
- key.toStringUtf8(),
- completedWork.getWorkItem().getWorkToken(),
- workToken);
+ // avoid Preconditions.checkNotNull and checkState here to prevent eagerly evaluating the
+ // format string parameters for the error message.
+ if (completedWork == null) {
+ throw new NullPointerException(
+ String.format(
+ "No active state for key %s, expected token %s",
+ TextFormat.escapeBytes(key), workToken));
+ }
+ if (completedWork.getWorkItem().getWorkToken() != workToken) {
+ throw new IllegalStateException(
+ String.format(
+ "Token mismatch for key %s: %s and %s",
+ TextFormat.escapeBytes(key),
+ completedWork.getWorkItem().getWorkToken(),
+ workToken));
+ }
if (queue.peek() == null) {
activeWork.remove(key);
return;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsDoFns.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsDoFns.java
index 9b8d19a..dc098e7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsDoFns.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsDoFns.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.util.AppliedCombineFn;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/** {@link GroupAlsoByWindowFn}'s that merge windows and groups elements in those windows. */
public abstract class StreamingGroupAlsoByWindowsDoFns {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunner.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunner.java
index 22a345f..bf5efee 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunner.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunner.java
@@ -35,9 +35,9 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
index bd1cc85..a114b6f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.CounterUpdate;
import com.google.api.services.dataflow.model.SideInputInfo;
@@ -54,13 +54,13 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -511,7 +511,8 @@
stateFamily,
stateReader,
work.getIsNewKey(),
- stateCache.forKey(getSerializedKey(), stateFamily, getWork().getCacheToken()),
+ stateCache.forKey(
+ getSerializedKey(), stateFamily, getWork().getCacheToken(), getWorkToken()),
scopedReadStateSupplier);
this.systemTimerInternals =
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactory.java
index c870f7a..238d6d6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactory.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.List;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterParDoFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterParDoFn.java
index 852a920..44c2d95 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterParDoFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterParDoFn.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoFn;
import org.apache.beam.runners.dataflow.worker.util.common.worker.Receiver;
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* A {@link ParDoFn} that writes side input data using {@link
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcher.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcher.java
index 2d22650..7c5babf 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcher.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcher.java
@@ -47,11 +47,11 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Parser;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Parser;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** A class that handles streaming side inputs in a {@link DoFnRunner}. */
public class StreamingSideInputFetcher<InputT, W extends BoundedWindow> {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainer.java
index fbf635d..2c38e08 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainer.java
@@ -18,8 +18,10 @@
package org.apache.beam.runners.dataflow.worker;
import com.google.api.services.dataflow.model.CounterUpdate;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.util.Map;
import java.util.Map.Entry;
+import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.metrics.DistributionData;
import org.apache.beam.runners.core.metrics.GaugeCell;
@@ -30,9 +32,9 @@
import org.apache.beam.sdk.metrics.MetricKey;
import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
/**
* For Dataflow Streaming, we want to efficiently support many threads report metric updates, and a
@@ -86,9 +88,13 @@
return FluentIterable.from(counters.entries())
.transform(
new Function<Entry<MetricName, DeltaCounterCell>, CounterUpdate>() {
+
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
@Nullable
- public CounterUpdate apply(Map.Entry<MetricName, DeltaCounterCell> entry) {
+ public CounterUpdate apply(@Nonnull Map.Entry<MetricName, DeltaCounterCell> entry) {
long value = entry.getValue().getSumAndReset();
if (value == 0) {
return null;
@@ -105,9 +111,13 @@
return FluentIterable.from(distributions.entries())
.transform(
new Function<Entry<MetricName, DeltaDistributionCell>, CounterUpdate>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
@Nullable
- public CounterUpdate apply(Map.Entry<MetricName, DeltaDistributionCell> entry) {
+ public CounterUpdate apply(
+ @Nonnull Map.Entry<MetricName, DeltaDistributionCell> entry) {
DistributionData value = entry.getValue().getAndReset();
if (value.count() == 0) {
return null;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactory.java
index 8993ace..6de9336 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.Iterator;
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link ParDoFnFactory} that creates a system {@link ParDoFn} responsible for transforming
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
index 77d5ae1..b1aec29 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A source that reads from a shuffled dataset, without any key grouping. Returns just the values.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderFactory.java
index f034e38..95e570b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderFactory.java
@@ -28,7 +28,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Creates an UngroupedShuffleReader from a CloudObject spec. */
public class UngroupedShuffleReaderFactory implements ReaderFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java
index 115ad67..6c6b2f4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedWindmillReader.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactory.java
index 55dfa48..84d4712 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactory.java
@@ -19,7 +19,7 @@
import static org.apache.beam.runners.dataflow.DataflowRunner.StreamingPCollectionViewWriterFn;
import static org.apache.beam.runners.dataflow.util.Structs.getBytes;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.List;
@@ -39,8 +39,8 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
/**
* A {@link ParDoFnFactory} to create instances of user {@link GroupAlsoByWindowFn} according to
@@ -120,6 +120,7 @@
stepContext,
operationContext,
doFnInfo.getDoFnSchemaInformation(),
+ doFnInfo.getSideInputMapping(),
runnerFactory));
} else if (doFnInfo.getDoFn() instanceof StreamingPCollectionViewWriterFn) {
@@ -147,6 +148,7 @@
stepContext,
operationContext,
doFnInfo.getDoFnSchemaInformation(),
+ doFnInfo.getSideInputMapping(),
runnerFactory);
}
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactory.java
index e251fe69..db5eeb8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.SideInputInfo;
import java.util.List;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Weighers.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Weighers.java
index 1aac36d..d7c3847 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Weighers.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/Weighers.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import org.apache.beam.sdk.util.Weighted;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Weigher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Weigher;
/**
* A {@code Weigher}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItem.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItem.java
index b764bfb..d8d14c6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItem.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItem.java
@@ -36,10 +36,10 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillNamespacePrefix.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillNamespacePrefix.java
index 17ad555..eba5c5d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillNamespacePrefix.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillNamespacePrefix.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
/**
* A prefix for a Windmill state or timer tag to separate user state and timers from system state
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBase.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBase.java
index 04d4239..79d8b06 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBase.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBase.java
@@ -22,7 +22,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
/**
* Base class for iterators that decode messages from bundles inside a {@link Windmill.WorkItem}.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillSink.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillSink.java
index 1e6a265..60ddce5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillSink.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillSink.java
@@ -40,8 +40,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.sdk.values.ValueWithRecordId.ValueWithRecordIdCoder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
class WindmillSink<T> extends Sink<WindowedValue<T>> {
private WindmillStreamWriter writer;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateCache.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateCache.java
index 408fe9b..d74b0db 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateCache.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateCache.java
@@ -27,16 +27,18 @@
import javax.servlet.http.HttpServletResponse;
import org.apache.beam.runners.core.StateNamespace;
import org.apache.beam.runners.core.StateTag;
+import org.apache.beam.runners.core.StateTags;
import org.apache.beam.runners.dataflow.worker.status.BaseStatusServlet;
import org.apache.beam.runners.dataflow.worker.status.StatusDataProvider;
import org.apache.beam.sdk.state.State;
import org.apache.beam.sdk.util.Weighted;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalCause;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Weigher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalCause;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Weigher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
/**
* Process-wide cache of per-key state.
@@ -54,9 +56,9 @@
private static final int INITIAL_HASH_MAP_CAPACITY = 4;
// Overhead of each hash map entry.
private static final int HASH_MAP_ENTRY_OVERHEAD = 16;
- // Overhead of each cache entry. Two longs, plus a hash table.
+ // Overhead of each cache entry. Three longs, plus a hash table.
private static final int PER_CACHE_ENTRY_OVERHEAD =
- 16 + HASH_MAP_ENTRY_OVERHEAD * INITIAL_HASH_MAP_CAPACITY;
+ 24 + HASH_MAP_ENTRY_OVERHEAD * INITIAL_HASH_MAP_CAPACITY;
private Cache<StateId, StateCacheEntry> stateCache;
private HashMultimap<ComputationKey, StateId> keyIndex =
@@ -112,8 +114,8 @@
}
/** Returns a per-computation, per-key view of the state cache. */
- public ForKey forKey(ByteString key, String stateFamily, long cacheToken) {
- return new ForKey(computation, key, stateFamily, cacheToken);
+ public ForKey forKey(ByteString key, String stateFamily, long cacheToken, long workToken) {
+ return new ForKey(computation, key, stateFamily, cacheToken, workToken);
}
}
@@ -122,24 +124,36 @@
private final String computation;
private final ByteString key;
private final String stateFamily;
+ // Cache token must be consistent for the key for the cache to be valid.
private final long cacheToken;
- private ForKey(String computation, ByteString key, String stateFamily, long cacheToken) {
+ // The work token for processing must be greater than the last work token. As work items are
+ // increasing for a key, a less-than or equal to work token indicates that the current token is
+ // for stale processing. We don't use the cache so that fetches performed will fail with a no
+ // longer valid work token.
+ private final long workToken;
+
+ private ForKey(
+ String computation, ByteString key, String stateFamily, long cacheToken, long workToken) {
this.computation = computation;
this.key = key;
this.stateFamily = stateFamily;
this.cacheToken = cacheToken;
+ this.workToken = workToken;
}
public <T extends State> T get(StateNamespace namespace, StateTag<T> address) {
return WindmillStateCache.this.get(
- computation, key, stateFamily, cacheToken, namespace, address);
+ computation, key, stateFamily, cacheToken, workToken, namespace, address);
}
+ // Note that once a value has been put for a given workToken, get calls with that same workToken
+ // will fail. This is ok as we only put entries when we are building the commit and will no
+ // longer be performing gets for the same work token.
public <T extends State> void put(
StateNamespace namespace, StateTag<T> address, T value, long weight) {
WindmillStateCache.this.put(
- computation, key, stateFamily, cacheToken, namespace, address, value, weight);
+ computation, key, stateFamily, cacheToken, workToken, namespace, address, value, weight);
}
}
@@ -152,7 +166,8 @@
String computation,
ByteString processingKey,
String stateFamily,
- long token,
+ long cacheToken,
+ long workToken,
StateNamespace namespace,
StateTag<T> address) {
StateId id = new StateId(computation, processingKey, stateFamily, namespace);
@@ -160,10 +175,14 @@
if (entry == null) {
return null;
}
- if (entry.getToken() != token) {
+ if (entry.getCacheToken() != cacheToken) {
stateCache.invalidate(id);
return null;
}
+ if (workToken <= entry.getLastWorkToken()) {
+ // We don't used the cached item but we don't invalidate it.
+ return null;
+ }
return entry.get(namespace, address);
}
@@ -171,7 +190,8 @@
String computation,
ByteString processingKey,
String stateFamily,
- long token,
+ long cacheToken,
+ long workToken,
StateNamespace namespace,
StateTag<T> address,
T value,
@@ -183,11 +203,12 @@
keyIndex.put(id.getComputationKey(), id);
}
}
- if (entry == null || entry.getToken() != token) {
- entry = new StateCacheEntry(token);
+ if (entry == null || entry.getCacheToken() != cacheToken) {
+ entry = new StateCacheEntry(cacheToken);
this.displayedWeight += (int) id.getWeight();
this.displayedWeight += (int) entry.getWeight();
}
+ entry.setLastWorkToken(workToken);
this.displayedWeight += (int) entry.put(namespace, address, value, weight);
// Always add back to the cache to update the weight.
stateCache.put(id, entry);
@@ -264,20 +285,26 @@
}
/**
- * Entry in the state cache that stores a map of values and a token representing the validity of
- * the values.
+ * Entry in the state cache that stores a map of values, a cache token representing the validity
+ * of the values, and a work token that is increasing to ensure sequential processing.
*/
private static class StateCacheEntry implements Weighted {
- private final long token;
+ private final long cacheToken;
+ private long lastWorkToken;
private final Map<NamespacedTag<?>, WeightedValue<?>> values;
private long weight;
- public StateCacheEntry(long token) {
+ public StateCacheEntry(long cacheToken) {
this.values = new HashMap<>(INITIAL_HASH_MAP_CAPACITY);
- this.token = token;
+ this.cacheToken = cacheToken;
+ this.lastWorkToken = Long.MIN_VALUE;
this.weight = 0;
}
+ public void setLastWorkToken(long workToken) {
+ this.lastWorkToken = workToken;
+ }
+
@SuppressWarnings("unchecked")
public <T extends State> T get(StateNamespace namespace, StateTag<T> tag) {
WeightedValue<T> weightedValue =
@@ -310,17 +337,21 @@
return weight + PER_CACHE_ENTRY_OVERHEAD;
}
- public long getToken() {
- return token;
+ public long getCacheToken() {
+ return cacheToken;
+ }
+
+ public long getLastWorkToken() {
+ return lastWorkToken;
}
private static class NamespacedTag<T extends State> {
private final StateNamespace namespace;
- private final StateTag<T> tag;
+ private final Equivalence.Wrapper<StateTag> tag;
NamespacedTag(StateNamespace namespace, StateTag<T> tag) {
this.namespace = namespace;
- this.tag = tag;
+ this.tag = StateTags.ID_EQUIVALENCE.wrap((StateTag) tag);
}
@Override
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java
index a0d5f4e..9c9779e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java
@@ -55,13 +55,13 @@
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CombineFnUtil;
import org.apache.beam.sdk.util.Weighted;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
import org.joda.time.Instant;
/** Implementation of {@link StateInternals} using Windmill to manage the underlying data. */
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateReader.java
index 62e0322..0050602 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateReader.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
@@ -31,6 +32,7 @@
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.TagBag;
@@ -38,16 +40,16 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.Weighted;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.AbstractIterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ForwardingList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ForwardingFuture;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.SettableFuture;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.AbstractIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ForwardingList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ForwardingFuture;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture;
import org.joda.time.Instant;
/**
@@ -354,8 +356,11 @@
this.elemCoder = elemCoder;
}
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public Iterable<T> apply(ValuesAndContPosition<T> valuesAndContPosition) {
+ public Iterable<T> apply(@Nonnull ValuesAndContPosition<T> valuesAndContPosition) {
if (valuesAndContPosition.continuationPosition == null) {
// Number of values is small enough Windmill sent us the entire bag in one response.
reader = null;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimeUtils.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimeUtils.java
index 018dd9e..9505bb4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimeUtils.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimeUtils.java
@@ -19,7 +19,7 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.joda.time.Instant;
/** Some timestamp conversion helpers for working with Windmill. */
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternals.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternals.java
index ba7cdc4..c2deb2f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternals.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternals.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import javax.annotation.Nullable;
import org.apache.beam.runners.core.StateNamespace;
@@ -29,11 +29,11 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table.Cell;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table.Cell;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindowingWindmillReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindowingWindmillReader.java
index c93951c..ddd0be0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindowingWindmillReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindowingWindmillReader.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -35,7 +35,8 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* A Reader that receives input data from a Windmill server, and returns a singleton iterable
@@ -116,31 +117,54 @@
final WorkItem workItem = context.getWork();
KeyedWorkItem<K, T> keyedWorkItem =
new WindmillKeyedWorkItem<>(key, workItem, windowCoder, windowsCoder, valueCoder);
+ final boolean isEmptyWorkItem =
+ (Iterables.isEmpty(keyedWorkItem.timersIterable())
+ && Iterables.isEmpty(keyedWorkItem.elementsIterable()));
final WindowedValue<KeyedWorkItem<K, T>> value = new ValueInEmptyWindows<>(keyedWorkItem);
- return new NativeReaderIterator<WindowedValue<KeyedWorkItem<K, T>>>() {
- private WindowedValue<KeyedWorkItem<K, T>> current;
+ // Return a noop iterator when current workitem is an empty workitem.
+ if (isEmptyWorkItem) {
+ return new NativeReaderIterator<WindowedValue<KeyedWorkItem<K, T>>>() {
+ @Override
+ public boolean start() throws IOException {
+ return false;
+ }
- @Override
- public boolean start() throws IOException {
- current = value;
- return true;
- }
+ @Override
+ public boolean advance() throws IOException {
+ return false;
+ }
- @Override
- public boolean advance() throws IOException {
- current = null;
- return false;
- }
-
- @Override
- public WindowedValue<KeyedWorkItem<K, T>> getCurrent() {
- if (current == null) {
+ @Override
+ public WindowedValue<KeyedWorkItem<K, T>> getCurrent() {
throw new NoSuchElementException();
}
- return value;
- }
- };
+ };
+ } else {
+ return new NativeReaderIterator<WindowedValue<KeyedWorkItem<K, T>>>() {
+ private WindowedValue<KeyedWorkItem<K, T>> current;
+
+ @Override
+ public boolean start() throws IOException {
+ current = value;
+ return true;
+ }
+
+ @Override
+ public boolean advance() throws IOException {
+ current = null;
+ return false;
+ }
+
+ @Override
+ public WindowedValue<KeyedWorkItem<K, T>> getCurrent() {
+ if (current == null) {
+ throw new NoSuchElementException();
+ }
+ return value;
+ }
+ };
+ }
}
@Override
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
index 20bdee4..5f89dc6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClient.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.dataflow.model.CounterUpdate;
import com.google.api.services.dataflow.model.MetricStructuredName;
@@ -49,8 +49,8 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader.DynamicSplitResult;
import org.apache.beam.runners.dataflow.worker.util.common.worker.NativeReader.Progress;
import org.apache.beam.sdk.util.UserCodeException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkUnitClient.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkUnitClient.java
index a0d63aa..12d8506 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkUnitClient.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkUnitClient.java
@@ -21,7 +21,7 @@
import com.google.api.services.dataflow.model.WorkItemServiceState;
import com.google.api.services.dataflow.model.WorkItemStatus;
import java.io.IOException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
/** Abstract base class describing a client for WorkItem work units. */
interface WorkUnitClient {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSources.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSources.java
index 3958497..4964a89 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSources.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSources.java
@@ -24,7 +24,7 @@
import static org.apache.beam.runners.dataflow.util.Structs.getStrings;
import static org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray;
import static org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.client.util.Base64;
import com.google.api.services.dataflow.model.ApproximateReportedProgress;
@@ -61,11 +61,11 @@
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.ValueWithRecordId;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerUncaughtExceptionHandler.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerUncaughtExceptionHandler.java
index 2cbb5de..fa3910c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerUncaughtExceptionHandler.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerUncaughtExceptionHandler.java
@@ -21,7 +21,7 @@
import java.lang.Thread.UncaughtExceptionHandler;
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingInitializer;
import org.apache.beam.runners.dataflow.worker.util.common.worker.JvmRuntime;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/Apiary.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/Apiary.java
index 1cad9cc..4d0dfe2 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/Apiary.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/Apiary.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker.apiary;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Static convenience methods to work around default encodings done by Apiary for default fields.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructions.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructions.java
index 87e4e33..e7e0c18 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructions.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructions.java
@@ -24,7 +24,7 @@
import java.util.List;
import java.util.function.Function;
import org.apache.beam.sdk.fn.IdGenerator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* {@link ParDoInstruction}s are meant to always have {@link MultiOutputInfo}s which give names to
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/Counter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/Counter.java
index 7ab6c35..2a86674 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/Counter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/Counter.java
@@ -22,7 +22,7 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory.CounterDistribution;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory.CounterMean;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A Counter enables the aggregation of a stream of values over time. The cumulative aggregate value
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactory.java
index bfca7dc..0091674 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactory.java
@@ -27,11 +27,11 @@
import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.runners.dataflow.worker.counters.Counter.AtomicCounterValue;
import org.apache.beam.runners.dataflow.worker.counters.Counter.CounterUpdateExtractor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.math.LongMath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AtomicDouble;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AtomicDouble;
/** Factory interface for creating counters. */
public class CounterFactory {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterName.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterName.java
index 5a6c051..ded29e4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterName.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterName.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.runners.dataflow.worker.counters;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.ToStringHelper;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.ToStringHelper;
/**
* The name of a counter identifies the user-specified name, as well as the origin, the step the
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterSet.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterSet.java
index e434475..793560d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterSet.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterSet.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.counters;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Collections;
@@ -25,7 +25,7 @@
import java.util.concurrent.ConcurrentHashMap;
import org.apache.beam.runners.dataflow.worker.counters.Counter.AtomicCounterValue;
import org.apache.beam.runners.dataflow.worker.counters.Counter.CounterUpdateExtractor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java
new file mode 100644
index 0000000..c6e393e
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.List;
+
+/**
+ * CounterUpdateAggregator performs aggregation over a list of CounterUpdate and return combined
+ * result.
+ */
+interface CounterUpdateAggregator {
+
+ /**
+ * Implementation of aggregate function should provide logic to take the list of CounterUpdates
+ * and return single combined CounterUpdate object. Reporting the aggregated result to Dataflow
+ * should have same effect as reporting the elements in list individually to Dataflow.
+ *
+ * @param counterUpdates CounterUpdates to aggregate.
+ * @return Aggregated CounterUpdate.
+ */
+ CounterUpdate aggregate(List<CounterUpdate> counterUpdates);
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java
new file mode 100644
index 0000000..32f99e7
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.common.collect.ImmutableMap;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+
+public class CounterUpdateAggregators {
+
+ private static final Map<String, CounterUpdateAggregator> aggregators =
+ ImmutableMap.of(
+ Kind.SUM.toString(), new SumCounterUpdateAggregator(),
+ Kind.MEAN.toString(), new MeanCounterUpdateAggregator(),
+ Kind.DISTRIBUTION.toString(), new DistributionCounterUpdateAggregator());
+
+ private static String getCounterUpdateKind(CounterUpdate counterUpdate) {
+ if (counterUpdate.getStructuredNameAndMetadata() != null
+ && counterUpdate.getStructuredNameAndMetadata().getMetadata() != null) {
+ return counterUpdate.getStructuredNameAndMetadata().getMetadata().getKind();
+ }
+ if (counterUpdate.getNameAndKind() != null) {
+ return counterUpdate.getNameAndKind().getKind();
+ }
+ throw new IllegalArgumentException(
+ "CounterUpdate must have either StructuredNameAndMetadata or NameAndKind.");
+ }
+
+ /**
+ * Try to aggregate a List of CounterUpdates. The first CounterUpdate entry of the List will be
+ * examined to identify the CounterUpdate kind with {@link #getCounterUpdateKind(CounterUpdate)}
+ * and find the suitable {@link CounterUpdateAggregator}, if there is no suitable aggregator the
+ * original list will be returned.
+ *
+ * <p>Note that this method assumes the CounterUpdate elements in this list has the same {@link
+ * com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata
+ * StructruredNameAndMetadata} or {@link com.google.api.services.dataflow.model.NameAndKind
+ * NameAndKind}, also the value type should be the same across all the elements.
+ *
+ * @param counterUpdates List of CounterUpdate to be aggregated.
+ * @return A singleton list of combined CounterUpdate if it is possible to aggregate the elements,
+ * other wise return the original list.
+ */
+ public static List<CounterUpdate> aggregate(List<CounterUpdate> counterUpdates) {
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return counterUpdates;
+ }
+ CounterUpdate first = counterUpdates.get(0);
+ String kind = getCounterUpdateKind(first);
+ if (aggregators.containsKey(kind)) {
+ // Return list containing combined CounterUpdate
+ return Collections.singletonList(aggregators.get(kind).aggregate(counterUpdates));
+ }
+ // not able to aggregate the counter updates.
+ return counterUpdates;
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DataflowCounterUpdateExtractor.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DataflowCounterUpdateExtractor.java
index abf1d2b..b4cc8b4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DataflowCounterUpdateExtractor.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DataflowCounterUpdateExtractor.java
@@ -29,7 +29,7 @@
import com.google.api.services.dataflow.model.SplitInt64;
import org.apache.beam.runners.dataflow.worker.counters.Counter.CounterUpdateExtractor;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory.CounterMean;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Factory methods for extracting {@link CounterUpdate} updates from counters. */
public class DataflowCounterUpdateExtractor implements CounterUpdateExtractor<CounterUpdate> {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java
new file mode 100644
index 0000000..b850864
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.DistributionUpdate;
+import java.util.List;
+
+public class DistributionCounterUpdateAggregator implements CounterUpdateAggregator {
+
+ @Override
+ public CounterUpdate aggregate(List<CounterUpdate> counterUpdates) {
+
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return null;
+ }
+ if (counterUpdates.stream().anyMatch(c -> c.getDistribution() == null)) {
+ throw new UnsupportedOperationException(
+ "Aggregating DISTRIBUTION counter updates over non-distribution type is not implemented.");
+ }
+ CounterUpdate initial = counterUpdates.remove(0);
+ return counterUpdates.stream()
+ .reduce(
+ initial,
+ (first, second) ->
+ first.setDistribution(
+ new DistributionUpdate()
+ .setCount(
+ longToSplitInt(
+ splitIntToLong(first.getDistribution().getCount())
+ + splitIntToLong(second.getDistribution().getCount())))
+ .setMax(
+ longToSplitInt(
+ Math.max(
+ splitIntToLong(first.getDistribution().getMax()),
+ splitIntToLong(second.getDistribution().getMax()))))
+ .setMin(
+ longToSplitInt(
+ Math.min(
+ splitIntToLong(first.getDistribution().getMin()),
+ splitIntToLong(second.getDistribution().getMin()))))
+ .setSum(
+ longToSplitInt(
+ splitIntToLong(first.getDistribution().getSum())
+ + splitIntToLong(second.getDistribution().getSum())))));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java
new file mode 100644
index 0000000..4cc2a46
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.IntegerMean;
+import java.util.List;
+
+public class MeanCounterUpdateAggregator implements CounterUpdateAggregator {
+
+ @Override
+ public CounterUpdate aggregate(List<CounterUpdate> counterUpdates) {
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return null;
+ }
+ if (counterUpdates.stream().anyMatch(c -> c.getIntegerMean() == null)) {
+ throw new UnsupportedOperationException(
+ "Aggregating MEAN counter updates over non-integerMean type is not implemented.");
+ }
+
+ CounterUpdate initial = counterUpdates.remove(0);
+ return counterUpdates.stream()
+ .reduce(
+ initial,
+ (first, second) ->
+ first.setIntegerMean(
+ new IntegerMean()
+ .setCount(
+ longToSplitInt(
+ splitIntToLong(first.getIntegerMean().getCount())
+ + splitIntToLong(second.getIntegerMean().getCount())))
+ .setSum(
+ longToSplitInt(
+ splitIntToLong(first.getIntegerMean().getSum())
+ + splitIntToLong(second.getIntegerMean().getSum())))));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java
new file mode 100644
index 0000000..bff2489
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.List;
+
+public class SumCounterUpdateAggregator implements CounterUpdateAggregator {
+
+ @Override
+ public CounterUpdate aggregate(List<CounterUpdate> counterUpdates) {
+ if (counterUpdates == null || counterUpdates.isEmpty()) {
+ return null;
+ }
+ if (counterUpdates.stream().anyMatch(c -> c.getInteger() == null)) {
+ throw new UnsupportedOperationException(
+ "Aggregating SUM counter updates over non-integer type is not implemented.");
+ }
+
+ CounterUpdate initial = counterUpdates.remove(0);
+ return counterUpdates.stream()
+ .reduce(
+ initial,
+ (first, second) ->
+ first.setInteger(
+ longToSplitInt(
+ splitIntToLong(first.getInteger()) + splitIntToLong(second.getInteger()))));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlService.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlService.java
index 2fd3aca..d701083 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlService.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlService.java
@@ -26,7 +26,7 @@
import org.apache.beam.runners.dataflow.worker.fn.grpc.BeamFnService;
import org.apache.beam.runners.fnexecution.HeaderAccessor;
import org.apache.beam.runners.fnexecution.control.FnApiControlClient;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
index f88da99..ee41190 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java
@@ -63,9 +63,9 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.WorkExecutor;
import org.apache.beam.sdk.metrics.MetricKey;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -86,7 +86,7 @@
ExecutionStateTracker executionStateTracker) {
super(operations, counters, executionStateTracker);
this.progressTracker = createProgressTracker();
- LOG.info("Creating BeamFnMapTaskExecutor");
+ LOG.debug("Creating BeamFnMapTaskExecutor");
}
/**
@@ -345,16 +345,25 @@
// is deprecated.
ProcessBundleProgressResponse processBundleProgressResponse =
MoreFutures.get(bundleProcessOperation.getProcessBundleProgress());
- updateMetrics(processBundleProgressResponse.getMonitoringInfosList());
+
+ final List<MonitoringInfo> monitoringInfosList =
+ processBundleProgressResponse.getMonitoringInfosList();
// Supporting deprecated metrics until all supported runners are migrated to using
// MonitoringInfos
Metrics metrics = processBundleProgressResponse.getMetrics();
+ double elementsConsumed =
+ bundleProcessOperation.getInputElementsConsumed(monitoringInfosList);
+
+ if (elementsConsumed == 0) {
+ elementsConsumed = bundleProcessOperation.getInputElementsConsumed(metrics);
+ }
+
+ updateMetrics(monitoringInfosList);
updateMetricsDeprecated(metrics);
// todo(migryz): utilize monitoringInfos here.
// Requires Element Count metrics to be implemented.
- double elementsConsumed = bundleProcessOperation.getInputElementsConsumed(metrics);
grpcWriteOperationElementsProcessed.accept((int) elementsConsumed);
progressInterpolator.addPoint(
@@ -365,19 +374,22 @@
if (!isTransientProgressError(exn.getMessage())) {
grpcWriteOperationElementsProcessed.accept(-1); // Not supported.
progressErrors++;
- // Only log verbosely every power of two to avoid spamming the logs.
- if (Integer.bitCount(progressErrors) == 1) {
- LOG.warn(
- String.format(
- "Progress updating failed %s times. Following exception safely handled.",
- progressErrors),
- exn);
- } else {
- LOG.debug(
- String.format(
- "Progress updating failed %s times. Following exception safely handled.",
- progressErrors),
- exn);
+ // JRH schedules progress report every 5 sec. So wait for 5 mins to show the log.
+ if (progressErrors == 60) {
+ String bundleId = bundleProcessOperation.getCurrentProcessBundleInstructionId();
+ if (bundleId == null) {
+ LOG.info(
+ String.format(
+ "Runner failed to get progress from SDK because current bundle has been "
+ + "finished or not start in SDK yet"));
+ } else {
+ LOG.warn(
+ String.format(
+ "Runner has failed to get progress from SDK for %s times for bundle %s. "
+ + "Possibly caused by SDK doesn't support progress report.",
+ progressErrors, bundleId),
+ exn);
+ }
}
}
@@ -400,13 +412,19 @@
* @param monitoringInfos Usually received from FnApi.
*/
private void updateMetrics(List<MonitoringInfo> monitoringInfos) {
+ List<MonitoringInfo> monitoringInfosCopy = new ArrayList<>(monitoringInfos);
+
+ List<MonitoringInfo> misToFilter =
+ bundleProcessOperation.findIOPCollectionMonitoringInfos(monitoringInfos);
+ monitoringInfosCopy.removeAll(misToFilter);
+
final MonitoringInfoToCounterUpdateTransformer monitoringInfoToCounterUpdateTransformer =
new FnApiMonitoringInfoToCounterUpdateTransformer(
this.bundleProcessOperation.getPtransformIdToUserStepContext(),
this.bundleProcessOperation.getPCollectionIdToNameContext());
counterUpdates =
- monitoringInfos.stream()
+ monitoringInfosCopy.stream()
.map(monitoringInfoToCounterUpdateTransformer::transform)
.filter(Objects::nonNull)
.collect(Collectors.toList());
@@ -481,7 +499,7 @@
@Override
public void start() {
- LOG.info("Starting BeamFnMapTaskExecutor, launching progress thread");
+ LOG.debug("Starting BeamFnMapTaskExecutor, launching progress thread");
progressErrors = 0;
nextProgressFuture =
scheduler.scheduleAtFixedRate(
@@ -493,7 +511,7 @@
@Override
public void stop() {
- LOG.info("Stopping BeamFnMapTaskExecutor, grabbing final metric updates");
+ LOG.debug("Stopping BeamFnMapTaskExecutor, grabbing final metric updates");
nextProgressFuture.cancel(true);
try {
nextProgressFuture.get();
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java
index df02c46..34dc4b1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker.fn.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.ByteArrayInputStream;
import java.io.IOException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java
index 86325a5..6dfc20a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/ElementCountMonitoringInfoToCounterUpdateTransformer.java
@@ -25,6 +25,7 @@
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.slf4j.Logger;
@@ -102,7 +103,7 @@
String counterName = pcollectionName + "-ElementCount";
NameAndKind name = new NameAndKind();
- name.setName(counterName).setKind("SUM");
+ name.setName(counterName).setKind(Kind.SUM.toString());
return new CounterUpdate()
.setNameAndKind(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformer.java
index b8f3bf2..9be62fa 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformer.java
@@ -25,7 +25,7 @@
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Generic MonitoringInfo to CounterUpdate transformer for FnApi.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
index 023735a..feaf1b4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
@@ -29,8 +29,9 @@
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -134,7 +135,7 @@
.setName(counterName)
.setOriginalStepName(stepContext.getNameContext().originalName())
.setExecutionStepName(stepContext.getNameContext().stageName()))
- .setMetadata(new CounterMetadata().setKind("SUM"));
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()));
return new CounterUpdate()
.setStructuredNameAndMetadata(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java
index 6a35654..20eeaba 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MeanByteCountMonitoringInfoToCounterUpdateTransformer.java
@@ -29,6 +29,7 @@
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,7 +44,7 @@
private final Map<String, NameContext> pcollectionIdToNameContext;
// TODO(BEAM-6945): utilize value from metrics.proto once it gets in.
- private static final String SUPPORTED_URN = "beam:metric:sampled_byte_size:v1";
+ private static final String SUPPORTED_URN = MonitoringInfoConstants.Urns.SAMPLED_BYTE_SIZE;
/**
* @param specValidator SpecMonitoringInfoValidator to utilize for default validation.
@@ -74,7 +75,6 @@
throw new RuntimeException(String.format("Received unexpected counter urn: %s", urn));
}
- // TODO(migryz): extract and utilize pcollection label from beam_fn_api.proto
if (!pcollectionIdToNameContext.containsKey(
monitoringInfo.getLabelsMap().get(MonitoringInfoConstants.Labels.PCOLLECTION))) {
return Optional.of(
@@ -109,7 +109,7 @@
String counterName = pcollectionName + "-MeanByteCount";
NameAndKind name = new NameAndKind();
- name.setName(counterName).setKind("MEAN");
+ name.setName(counterName).setKind(Kind.MEAN.toString());
return new CounterUpdate()
.setNameAndKind(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java
index 7799e6f..73bbf4b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperation.java
@@ -17,13 +17,16 @@
*/
package org.apache.beam.runners.dataflow.worker.fn.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Closeable;
import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.ConcurrentHashMap;
@@ -45,9 +48,11 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
import org.apache.beam.runners.core.SideInputReader;
import org.apache.beam.runners.core.StateNamespaces;
import org.apache.beam.runners.core.StateTags;
+import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.dataflow.worker.ByteStringCoder;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
import org.apache.beam.runners.dataflow.worker.DataflowOperationContext;
@@ -61,19 +66,20 @@
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.data.RemoteGrpcPortRead;
+import org.apache.beam.sdk.fn.data.RemoteGrpcPortWrite;
import org.apache.beam.sdk.state.BagState;
import org.apache.beam.sdk.transforms.Materializations;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.MoreFutures;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -110,6 +116,8 @@
private @Nullable String grpcReadTransformId = null;
private String grpcReadTransformOutputName = null;
+ private String grpcReadTransformOutputPCollectionName = null;
+ private final Set<String> grpcReadTransformReadWritePCollectionNames;
public RegisterAndProcessBundleOperation(
IdGenerator idGenerator,
@@ -145,6 +153,7 @@
LOG.debug(
"Process bundle descriptor {}", toDot(registerRequest.getProcessBundleDescriptor(0)));
}
+
for (Map.Entry<String, RunnerApi.PTransform> pTransform :
registerRequest.getProcessBundleDescriptor(0).getTransformsMap().entrySet()) {
if (pTransform.getValue().getSpec().getUrn().equals(RemoteGrpcPortRead.URN)) {
@@ -152,13 +161,45 @@
// TODO: Handle the case of more than one input.
grpcReadTransformId = null;
grpcReadTransformOutputName = null;
+ grpcReadTransformOutputPCollectionName = null;
break;
}
grpcReadTransformId = pTransform.getKey();
grpcReadTransformOutputName =
Iterables.getOnlyElement(pTransform.getValue().getOutputsMap().keySet());
+ grpcReadTransformOutputPCollectionName =
+ pTransform.getValue().getOutputsMap().get(grpcReadTransformOutputName);
}
}
+
+ grpcReadTransformReadWritePCollectionNames =
+ extractCrossBoundaryGrpcPCollectionNames(
+ registerRequest.getProcessBundleDescriptor(0).getTransformsMap().entrySet());
+ }
+
+ private Set<String> extractCrossBoundaryGrpcPCollectionNames(
+ final Set<Entry<String, PTransform>> ptransforms) {
+ Set<String> result = new HashSet<>();
+
+ // GRPC Read/Write expected to only have one Output/Input respectively.
+ for (Map.Entry<String, RunnerApi.PTransform> pTransform : ptransforms) {
+ if (pTransform.getValue().getSpec().getUrn().equals(RemoteGrpcPortRead.URN)) {
+ String grpcReadTransformOutputName =
+ Iterables.getOnlyElement(pTransform.getValue().getOutputsMap().keySet());
+ String pcollectionName =
+ pTransform.getValue().getOutputsMap().get(grpcReadTransformOutputName);
+ result.add(pcollectionName);
+ }
+
+ if (pTransform.getValue().getSpec().getUrn().equals(RemoteGrpcPortWrite.URN)) {
+ String grpcTransformInputName =
+ Iterables.getOnlyElement(pTransform.getValue().getInputsMap().keySet());
+ String pcollectionName = pTransform.getValue().getInputsMap().get(grpcTransformInputName);
+ result.add(pcollectionName);
+ }
+ }
+
+ return result;
}
/** Generates a dot description of the process bundle descriptor. */
@@ -239,6 +280,10 @@
return processBundleId;
}
+ public String getCurrentProcessBundleInstructionId() {
+ return processBundleId;
+ }
+
@Override
public void start() throws Exception {
try (Closeable scope = context.enterStart()) {
@@ -263,7 +308,7 @@
.setInstructionId(getProcessBundleInstructionId())
.setProcessBundle(
ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference(
+ .setProcessBundleDescriptorId(
registerRequest.getProcessBundleDescriptor(0).getId()))
.build();
@@ -341,7 +386,7 @@
InstructionRequest.newBuilder()
.setInstructionId(idGenerator.getId())
.setProcessBundleProgress(
- ProcessBundleProgressRequest.newBuilder().setInstructionReference(processBundleId))
+ ProcessBundleProgressRequest.newBuilder().setInstructionId(processBundleId))
.build();
return instructionRequestHandler
@@ -375,6 +420,48 @@
}
}
+ /*
+ * Returns a subset of monitoring infos that refer to grpc IO.
+ */
+ public List<MonitoringInfo> findIOPCollectionMonitoringInfos(
+ Iterable<MonitoringInfo> monitoringInfos) {
+ List<MonitoringInfo> result = new ArrayList<MonitoringInfo>();
+ if (grpcReadTransformReadWritePCollectionNames.isEmpty()) {
+ return result;
+ }
+
+ for (MonitoringInfo mi : monitoringInfos) {
+ if (mi.getUrn().equals(MonitoringInfoConstants.Urns.ELEMENT_COUNT)) {
+ String pcollection =
+ mi.getLabelsOrDefault(MonitoringInfoConstants.Labels.PCOLLECTION, null);
+ if ((pcollection != null)
+ && (grpcReadTransformReadWritePCollectionNames.contains(pcollection))) {
+ result.add(mi);
+ }
+ }
+ }
+
+ return result;
+ }
+
+ long getInputElementsConsumed(final Iterable<MonitoringInfo> monitoringInfos) {
+ if (grpcReadTransformId == null) {
+ return 0;
+ }
+
+ for (MonitoringInfo mi : monitoringInfos) {
+ if (mi.getUrn().equals(MonitoringInfoConstants.Urns.ELEMENT_COUNT)) {
+ String pcollection =
+ mi.getLabelsOrDefault(MonitoringInfoConstants.Labels.PCOLLECTION, null);
+ if (pcollection != null && pcollection.equals(grpcReadTransformOutputPCollectionName)) {
+ return mi.getMetric().getCounterData().getInt64Value();
+ }
+ }
+ }
+
+ return 0;
+ }
+
/** Returns the number of input elements consumed by the gRPC read, if known, otherwise 0. */
double getInputElementsConsumed(BeamFnApi.Metrics metrics) {
return metrics
@@ -412,36 +499,36 @@
stateRequest.getStateKey().getMultimapSideInput();
SideInputReader sideInputReader =
- ptransformIdToSideInputReader.get(multimapSideInputStateKey.getPtransformId());
+ ptransformIdToSideInputReader.get(multimapSideInputStateKey.getTransformId());
checkState(
sideInputReader != null,
- String.format("Unknown PTransform '%s'", multimapSideInputStateKey.getPtransformId()));
+ String.format("Unknown PTransform '%s'", multimapSideInputStateKey.getTransformId()));
PCollectionView<Materializations.MultimapView<Object, Object>> view =
(PCollectionView<Materializations.MultimapView<Object, Object>>)
ptransformIdToSideInputIdToPCollectionView.get(
- multimapSideInputStateKey.getPtransformId(),
+ multimapSideInputStateKey.getTransformId(),
multimapSideInputStateKey.getSideInputId());
checkState(
view != null,
String.format(
"Unknown side input '%s' on PTransform '%s'",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()));
+ multimapSideInputStateKey.getTransformId()));
checkState(
Materializations.MULTIMAP_MATERIALIZATION_URN.equals(
view.getViewFn().getMaterialization().getUrn()),
String.format(
"Unknown materialization for side input '%s' on PTransform '%s' with urn '%s'",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId(),
+ multimapSideInputStateKey.getTransformId(),
view.getViewFn().getMaterialization().getUrn()));
checkState(
view.getCoderInternal() instanceof KvCoder,
String.format(
"Materialization of side input '%s' on PTransform '%s' expects %s but received %s.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId(),
+ multimapSideInputStateKey.getTransformId(),
KvCoder.class.getSimpleName(),
view.getCoderInternal().getClass().getSimpleName()));
Coder<Object> keyCoder = ((KvCoder) view.getCoderInternal()).getKeyCoder();
@@ -460,7 +547,7 @@
String.format(
"Unable to decode window for side input '%s' on PTransform '%s'.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()),
+ multimapSideInputStateKey.getTransformId()),
e);
}
@@ -473,7 +560,7 @@
String.format(
"Unable to decode user key for side input '%s' on PTransform '%s'.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()),
+ multimapSideInputStateKey.getTransformId()),
e);
}
@@ -491,7 +578,7 @@
String.format(
"Unable to encode values for side input '%s' on PTransform '%s'.",
multimapSideInputStateKey.getSideInputId(),
- multimapSideInputStateKey.getPtransformId()),
+ multimapSideInputStateKey.getTransformId()),
e);
}
}
@@ -500,10 +587,10 @@
StateRequest stateRequest) {
StateKey.BagUserState bagUserStateKey = stateRequest.getStateKey().getBagUserState();
DataflowStepContext userStepContext =
- ptransformIdToUserStepContext.get(bagUserStateKey.getPtransformId());
+ ptransformIdToUserStepContext.get(bagUserStateKey.getTransformId());
checkState(
userStepContext != null,
- String.format("Unknown PTransform id '%s'", bagUserStateKey.getPtransformId()));
+ String.format("Unknown PTransform id '%s'", bagUserStateKey.getTransformId()));
// TODO: We should not be required to hold onto a pointer to the bag states for the
// user. InMemoryStateInternals assumes that the Java garbage collector does the clean-up work
// but instead StateInternals should hold its own references and write out any data and
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java
index cf98d50..abf5a87 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserDistributionMonitoringInfoToCounterUpdateTransformer.java
@@ -30,6 +30,7 @@
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.slf4j.Logger;
@@ -113,7 +114,7 @@
.setName(counterName)
.setOriginalStepName(stepContext.getNameContext().originalName())
.setOriginNamespace(counterNamespace))
- .setMetadata(new CounterMetadata().setKind("DISTRIBUTION"));
+ .setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString()));
return new CounterUpdate()
.setStructuredNameAndMetadata(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
index 470a1fa..5babdd3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
@@ -28,6 +28,7 @@
import org.apache.beam.runners.core.metrics.MonitoringInfoConstants;
import org.apache.beam.runners.core.metrics.SpecMonitoringInfoValidator;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Origin;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.slf4j.Logger;
@@ -109,7 +110,7 @@
.setName(counterName)
.setOriginalStepName(stepContext.getNameContext().originalName())
.setOriginNamespace(counterNamespace))
- .setMetadata(new CounterMetadata().setKind("SUM"));
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()));
return new CounterUpdate()
.setStructuredNameAndMetadata(name)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcService.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcService.java
index 2fceb94..7c69bf1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcService.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcService.java
@@ -43,10 +43,9 @@
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -117,8 +116,8 @@
private DeferredInboundDataClient(
String clientId,
LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> consumer) {
+ Coder<T> coder,
+ FnDataReceiver<T> consumer) {
this.future =
getClientFuture(clientId)
.thenCompose(
@@ -209,17 +208,14 @@
return new GrpcDataService() {
@Override
public <T> InboundDataClient receive(
- LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> consumer) {
+ LogicalEndpoint inputLocation, Coder<T> coder, FnDataReceiver<T> consumer) {
LOG.debug("Registering consumer for {}", inputLocation);
return new DeferredInboundDataClient(clientId, inputLocation, coder, consumer);
}
@Override
- public <T> CloseableFnDataReceiver<WindowedValue<T>> send(
- LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder) {
+ public <T> CloseableFnDataReceiver<T> send(LogicalEndpoint outputLocation, Coder<T> coder) {
LOG.debug("Creating output consumer for {}", outputLocation);
try {
if (outboundBufferLimit.isPresent()) {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperation.java
index 6dcb942..dbb74bf 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperation.java
@@ -18,7 +18,6 @@
package org.apache.beam.runners.dataflow.worker.fn.data;
import java.io.Closeable;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.runners.dataflow.worker.util.common.worker.Operation;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OperationContext;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver;
@@ -28,7 +27,7 @@
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -49,12 +48,12 @@
// Should only be set and cleared once per start/finish cycle in the start method and
// finish method respectively.
private String bundleId;
- private final Target target;
+ private final String ptransformId;
private InboundDataClient inboundDataClient;
public RemoteGrpcPortReadOperation(
FnDataService beamFnDataService,
- Target target,
+ String ptransformId,
IdGenerator bundleIdSupplier,
Coder<WindowedValue<T>> coder,
OutputReceiver[] receivers,
@@ -63,7 +62,7 @@
this.coder = coder;
this.beamFnDataService = beamFnDataService;
this.bundleIdSupplier = bundleIdSupplier;
- this.target = target;
+ this.ptransformId = ptransformId;
}
@Override
@@ -73,14 +72,14 @@
super.start();
inboundDataClient =
beamFnDataService.receive(
- LogicalEndpoint.of(bundleId, target), coder, this::consumeOutput);
+ LogicalEndpoint.of(bundleId, ptransformId), coder, this::consumeOutput);
}
}
@Override
public void finish() throws Exception {
try (Closeable scope = context.enterFinish()) {
- LOG.debug("Waiting for instruction {} and target {} to close.", bundleId, target);
+ LOG.debug("Waiting for instruction {} and transform {} to close.", bundleId, ptransformId);
inboundDataClient.awaitCompletion();
bundleId = null;
super.finish();
@@ -108,7 +107,7 @@
@Override
public String toString() {
return MoreObjects.toStringHelper(this)
- .add("target", target)
+ .add("ptransformId", ptransformId)
.add("coder", coder)
.add("bundleId", bundleId)
.toString();
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperation.java
index 3a31058..505e490 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperation.java
@@ -24,7 +24,6 @@
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Consumer;
import java.util.function.Supplier;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.runners.dataflow.worker.util.common.worker.Operation;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OperationContext;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver;
@@ -35,8 +34,8 @@
import org.apache.beam.sdk.fn.data.CloseableFnDataReceiver;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -55,7 +54,7 @@
// Should only be set and cleared once per start/finish cycle in the start method and
// finish method respectively.
private String bundleId;
- private final Target target;
+ private final String ptransformId;
private CloseableFnDataReceiver<WindowedValue<T>> receiver;
private final AtomicInteger elementsSent = new AtomicInteger();
@@ -72,16 +71,22 @@
public RemoteGrpcPortWriteOperation(
FnDataService beamFnDataService,
- Target target,
+ String ptransformId,
IdGenerator bundleIdSupplier,
Coder<WindowedValue<T>> coder,
OperationContext context) {
- this(beamFnDataService, target, bundleIdSupplier, coder, context, System::currentTimeMillis);
+ this(
+ beamFnDataService,
+ ptransformId,
+ bundleIdSupplier,
+ coder,
+ context,
+ System::currentTimeMillis);
}
public RemoteGrpcPortWriteOperation(
FnDataService beamFnDataService,
- Target target,
+ String ptransformId,
IdGenerator bundleIdSupplier,
Coder<WindowedValue<T>> coder,
OperationContext context,
@@ -90,7 +95,7 @@
this.coder = coder;
this.beamFnDataService = beamFnDataService;
this.bundleIdSupplier = bundleIdSupplier;
- this.target = target;
+ this.ptransformId = ptransformId;
this.currentTimeMillis = currentTimeMillis;
}
@@ -103,7 +108,7 @@
elementsFlushed = 0;
super.start();
bundleId = bundleIdSupplier.getId();
- receiver = beamFnDataService.send(LogicalEndpoint.of(bundleId, target), coder);
+ receiver = beamFnDataService.send(LogicalEndpoint.of(bundleId, ptransformId), coder);
}
}
@@ -239,7 +244,7 @@
@Override
public String toString() {
return MoreObjects.toStringHelper(this)
- .add("target", target)
+ .add("ptransformId", ptransformId)
.add("coder", coder)
.add("bundleId", bundleId)
.toString();
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingService.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingService.java
index 888a31f..d1b62d1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingService.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingService.java
@@ -28,9 +28,9 @@
import org.apache.beam.runners.dataflow.worker.fn.grpc.BeamFnService;
import org.apache.beam.runners.dataflow.worker.logging.DataflowWorkerLoggingMDC;
import org.apache.beam.runners.fnexecution.HeaderAccessor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactory.java
index 7e85eb3..1fabd6d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactory.java
@@ -27,9 +27,9 @@
import org.apache.beam.sdk.fn.stream.ForwardingClientResponseObserver;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A {@link StreamObserver} factory that wraps provided {@link CallStreamObserver}s making them flow
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunction.java
index acdd034..73bbf52 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunction.java
@@ -25,9 +25,9 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.InstructionOutputNode;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* A function which optimises the execution of {@link FlattenInstruction}s with ambiguous {@link
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
index a96ceac..fc42534 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
@@ -20,7 +20,7 @@
import static org.apache.beam.runners.dataflow.util.Structs.getBytes;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
import static org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCoders.forSideInputInfos;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.api.services.dataflow.model.InstructionOutput;
@@ -78,13 +78,13 @@
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
import org.joda.time.Duration;
/**
@@ -94,10 +94,10 @@
*/
public class CreateExecutableStageNodeFunction
implements Function<MutableNetwork<Node, Edge>, Node> {
- private static final String DATA_INPUT_URN = "urn:org.apache.beam:source:runner:0.1";
+ private static final String DATA_INPUT_URN = "beam:source:runner:0.1";
- private static final String DATA_OUTPUT_URN = "urn:org.apache.beam:sink:runner:0.1";
- private static final String JAVA_SOURCE_URN = "urn:org.apache.beam:source:java:0.1";
+ private static final String DATA_OUTPUT_URN = "beam:sink:runner:0.1";
+ private static final String JAVA_SOURCE_URN = "beam:source:java:0.1";
public static final String COMBINE_PER_KEY_URN =
BeamUrns.getUrn(StandardPTransforms.Composites.COMBINE_PER_KEY);
@@ -249,11 +249,7 @@
componentsBuilder.putCoders(
coderId,
RunnerApi.Coder.newBuilder()
- .setSpec(
- RunnerApi.SdkFunctionSpec.newBuilder()
- .setSpec(
- RunnerApi.FunctionSpec.newBuilder()
- .setPayload(output.toByteString())))
+ .setSpec(RunnerApi.FunctionSpec.newBuilder().setPayload(output.toByteString()))
.build());
// For non-java coder, hope it's GlobalWindows by default.
// TODO(BEAM-6231): Actually discover the right windowing strategy.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunction.java
index 5f40d98..c4b38e2 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunction.java
@@ -23,7 +23,6 @@
import java.util.HashSet;
import java.util.List;
import java.util.Set;
-import java.util.function.BiFunction;
import java.util.function.Function;
import java.util.function.Supplier;
import org.apache.beam.runners.dataflow.worker.graph.Edges.Edge;
@@ -33,13 +32,13 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
import org.apache.beam.sdk.fn.IdGenerator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ImmutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Graphs;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ImmutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* Splits the instruction graph into SDK and runner harness portions replacing the SDK sub-graphs
@@ -75,7 +74,7 @@
public class CreateRegisterFnOperationFunction
implements Function<MutableNetwork<Node, Edge>, MutableNetwork<Node, Edge>> {
private final IdGenerator idGenerator;
- private final BiFunction<String, String, Node> portSupplier;
+ private final Supplier<Node> portSupplier;
private final Function<MutableNetwork<Node, Edge>, Node> registerFnOperationFunction;
private final boolean useExecutableStageBundleExecution;
@@ -92,7 +91,7 @@
*/
public CreateRegisterFnOperationFunction(
IdGenerator idGenerator,
- BiFunction<String, String, Node> portSupplier,
+ Supplier<Node> portSupplier,
Function<MutableNetwork<Node, Edge>, Node> registerFnOperationFunction,
boolean useExecutableStageBundleExecution) {
this.idGenerator = idGenerator;
@@ -266,9 +265,7 @@
InstructionOutputNode portOutputNode =
InstructionOutputNode.create(
outputNode.getInstructionOutput(), outputNode.getPcollectionId());
- String predecessorPortEdgeId = idGenerator.getId();
- String successorPortEdgeId = idGenerator.getId();
- Node portNode = portSupplier.apply(predecessorPortEdgeId, successorPortEdgeId);
+ Node portNode = portSupplier.get();
network.addNode(newPredecessorOutputNode);
network.addNode(portNode);
for (Node predecessor : predecessors) {
@@ -292,11 +289,11 @@
network.addEdge(
newPredecessorOutputNode,
portNode,
- MultiOutputInfoEdge.create(new MultiOutputInfo().setTag(predecessorPortEdgeId)));
+ MultiOutputInfoEdge.create(new MultiOutputInfo().setTag(idGenerator.getId())));
network.addEdge(
portNode,
portOutputNode,
- MultiOutputInfoEdge.create(new MultiOutputInfo().setTag(successorPortEdgeId)));
+ MultiOutputInfoEdge.create(new MultiOutputInfo().setTag(idGenerator.getId())));
for (Node successor : successors) {
for (Edge edge : ImmutableList.copyOf(network.edgesConnecting(outputNode, successor))) {
network.addEdge(portOutputNode, successor, edge.clone());
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunction.java
index 388326e..84173a1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunction.java
@@ -25,11 +25,11 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ExecutionLocation;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* A function which sets the location for {@link FlattenInstruction}s by looking at the locations of
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunction.java
index ab09437..6e9f21f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunction.java
@@ -28,8 +28,8 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
import org.apache.beam.runners.dataflow.worker.util.CloudSourceUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* A function which sets the location for {@link ParallelInstructionNode}s in the network, with the
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Edges.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Edges.java
index ee75358..e88aa33 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Edges.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Edges.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.dataflow.model.MultiOutputInfo;
import com.google.auto.value.AutoValue;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodes.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodes.java
index eaba46c..6ba81d1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodes.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodes.java
@@ -37,12 +37,12 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* Inserts a {@link ParDoFn} that handles filtering blocked side inputs and fetching ready side
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCoders.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCoders.java
index 1537288..20142d6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCoders.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCoders.java
@@ -43,10 +43,10 @@
import org.apache.beam.runners.dataflow.worker.util.WorkerPropertyNames;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.LengthPrefixCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/** Utility for replacing or wrapping unknown coders with {@link LengthPrefixCoder}. */
public class LengthPrefixUnknownCoders {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunction.java
index e08240a..fd98bac 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunction.java
@@ -35,9 +35,9 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.fn.IdGenerator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
/**
* Creates a directed bipartite network of {@link ParallelInstructionNode}s and {@link
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Networks.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Networks.java
index e57d9af..03d365b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Networks.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Networks.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.ArrayDeque;
import java.util.ArrayList;
@@ -32,12 +32,12 @@
import java.util.Set;
import java.util.function.Function;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.EndpointPair;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.EndpointPair;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
/** Static utility methods for {@link Network} instances that are directed. */
public class Networks {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Nodes.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Nodes.java
index e65b107..853af69 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Nodes.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/Nodes.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.graph;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.json.GenericJson;
import com.google.api.client.json.JsonFactory;
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Container class for different types of network nodes. All nodes only have reference equality. */
public class Nodes {
@@ -261,25 +261,14 @@
@AutoValue
public abstract static class RemoteGrpcPortNode extends Node {
public static RemoteGrpcPortNode create(
- BeamFnApi.RemoteGrpcPort port,
- String primitiveTransformId,
- String functionSpecId,
- String inputId,
- String outputId) {
+ BeamFnApi.RemoteGrpcPort port, String primitiveTransformId) {
checkNotNull(port);
- return new AutoValue_Nodes_RemoteGrpcPortNode(
- port, primitiveTransformId, functionSpecId, inputId, outputId);
+ return new AutoValue_Nodes_RemoteGrpcPortNode(port, primitiveTransformId);
}
public abstract BeamFnApi.RemoteGrpcPort getRemoteGrpcPort();
public abstract String getPrimitiveTransformId();
-
- public abstract String getFunctionSpecId();
-
- public abstract String getInputId();
-
- public abstract String getOutputId();
}
/** A node that stores {@link org.apache.beam.model.fnexecution.v1.BeamFnApi.RegisterRequest}s. */
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RegisterNodeFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RegisterNodeFunction.java
index c5e7d13..0c33ee8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RegisterNodeFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RegisterNodeFunction.java
@@ -20,7 +20,7 @@
import static org.apache.beam.runners.dataflow.util.Structs.getBytes;
import static org.apache.beam.runners.dataflow.util.Structs.getString;
import static org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCoders.forSideInputInfos;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.api.services.dataflow.model.InstructionOutput;
@@ -80,13 +80,13 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
/**
* Converts a {@link Network} representation of {@link MapTask} destined for the SDK harness into an
@@ -96,10 +96,10 @@
*/
public class RegisterNodeFunction implements Function<MutableNetwork<Node, Edge>, Node> {
/** Must match declared fields within {@code ProcessBundleHandler}. */
- private static final String DATA_INPUT_URN = "urn:org.apache.beam:source:runner:0.1";
+ private static final String DATA_INPUT_URN = "beam:source:runner:0.1";
- private static final String DATA_OUTPUT_URN = "urn:org.apache.beam:sink:runner:0.1";
- private static final String JAVA_SOURCE_URN = "urn:org.apache.beam:source:java:0.1";
+ private static final String DATA_OUTPUT_URN = "beam:sink:runner:0.1";
+ private static final String JAVA_SOURCE_URN = "beam:source:java:0.1";
public static final String COMBINE_PER_KEY_URN =
BeamUrns.getUrn(StandardPTransforms.Composites.COMBINE_PER_KEY);
@@ -248,11 +248,7 @@
processBundleDescriptor.putCoders(
coderId,
RunnerApi.Coder.newBuilder()
- .setSpec(
- RunnerApi.SdkFunctionSpec.newBuilder()
- .setSpec(
- RunnerApi.FunctionSpec.newBuilder()
- .setPayload(output.toByteString())))
+ .setSpec(RunnerApi.FunctionSpec.newBuilder().setPayload(output.toByteString()))
.build());
}
} catch (IOException e) {
@@ -417,7 +413,8 @@
Set<Node> successors = input.successors(node);
if (predecessors.isEmpty() && !successors.isEmpty()) {
pTransform.putOutputs(
- node.getInputId(), nodesToPCollections.get(Iterables.getOnlyElement(successors)));
+ "generatedOutput" + idGenerator.getId(),
+ nodesToPCollections.get(Iterables.getOnlyElement(successors)));
pTransform.setSpec(
RunnerApi.FunctionSpec.newBuilder()
.setUrn(DATA_INPUT_URN)
@@ -425,7 +422,8 @@
.build());
} else if (!predecessors.isEmpty() && successors.isEmpty()) {
pTransform.putInputs(
- node.getOutputId(), nodesToPCollections.get(Iterables.getOnlyElement(predecessors)));
+ "generatedInput" + idGenerator.getId(),
+ nodesToPCollections.get(Iterables.getOnlyElement(predecessors)));
pTransform.setSpec(
RunnerApi.FunctionSpec.newBuilder()
.setUrn(DATA_OUTPUT_URN)
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunction.java
index 7f3eeae..7b51fef 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunction.java
@@ -23,10 +23,10 @@
import org.apache.beam.runners.dataflow.worker.graph.Edges.Edge;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* A function which removes {@link FlattenInstruction}s from the network representation of a {@link
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunction.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunction.java
index cbc4275..8f506ee 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunction.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunction.java
@@ -30,7 +30,7 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
import org.apache.beam.runners.dataflow.worker.util.WorkerPropertyNames;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
/**
* A function that replaces PartialGroupByKey nodes with ParDo nodes that can be translated by the
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java
index 0fdb4ae..1af00f1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java
@@ -44,9 +44,9 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.dataflow.worker.DataflowOperationContext.DataflowExecutionState;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
/**
* Formats {@link LogRecord} into JSON format for Cloud Logging. Any exception is represented using
@@ -196,13 +196,13 @@
writeIfNotEmpty("thread", logEntry.getThread());
writeIfNotEmpty("job", DataflowWorkerLoggingMDC.getJobId());
// TODO: Write the stage execution information by translating the currently execution
- // instruction reference to a stage.
+ // instruction id to a stage.
// writeIfNotNull("stage", ...);
- writeIfNotEmpty("step", logEntry.getPrimitiveTransformReference());
+ writeIfNotEmpty("step", logEntry.getTransformId());
writeIfNotEmpty("worker", DataflowWorkerLoggingMDC.getWorkerId());
// Id should match to id in //depot/google3/third_party/cloud/dataflow/worker/agent/sdk.go
writeIfNotEmpty("portability_worker_id", DataflowWorkerLoggingMDC.getSdkHarnessId());
- writeIfNotEmpty("work", logEntry.getInstructionReference());
+ writeIfNotEmpty("work", logEntry.getInstructionId());
writeIfNotEmpty("logger", logEntry.getLogLocation());
// TODO: Figure out a way to get exceptions transported across Beam Fn Logging API
writeIfNotEmpty("exception", logEntry.getTrace());
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializer.java
index 3200bc8..5668847 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializer.java
@@ -33,9 +33,9 @@
import java.util.logging.LogManager;
import java.util.logging.Logger;
import org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* Sets up {@link java.util.logging} configuration on the Dataflow worker with a rotating file
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactory.java
index 8525f99..2c5136c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactory.java
@@ -28,7 +28,7 @@
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link PrintStream} factory that creates {@link PrintStream}s which output to the specified JUL
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfiler.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfiler.java
index 0d4ac47..784f6aa 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfiler.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfiler.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.profiler;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/BaseStatusServlet.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/BaseStatusServlet.java
index 7e117a4..41580fe5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/BaseStatusServlet.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/BaseStatusServlet.java
@@ -22,7 +22,7 @@
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
/** Base class for status servlets. */
public abstract class BaseStatusServlet extends HttpServlet {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/DebugCapture.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/DebugCapture.java
index 7024274..f02ddda 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/DebugCapture.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/DebugCapture.java
@@ -35,7 +35,7 @@
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.DateTime;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HealthzServlet.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HealthzServlet.java
index 68744ec..9d00b1a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HealthzServlet.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HealthzServlet.java
@@ -18,20 +18,29 @@
package org.apache.beam.runners.dataflow.worker.status;
import java.io.IOException;
+import java.util.function.BooleanSupplier;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
-/** Respond to /healthz with "ok". */
+/** Respond to /healthz with health information. */
public class HealthzServlet extends BaseStatusServlet {
- public HealthzServlet() {
+ private final BooleanSupplier healthyIndicator;
+
+ public HealthzServlet(BooleanSupplier healthyIndicator) {
super("healthz");
+ this.healthyIndicator = healthyIndicator;
}
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
response.setContentType("text/html;charset=utf-8");
- response.setStatus(HttpServletResponse.SC_OK);
- response.getWriter().println("ok");
+ if (healthyIndicator.getAsBoolean()) {
+ response.setStatus(HttpServletResponse.SC_OK);
+ response.getWriter().println("ok");
+ } else {
+ response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+ response.getWriter().println("internal server error");
+ }
}
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HeapzServlet.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HeapzServlet.java
index c6ff540..9b2c613 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HeapzServlet.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/HeapzServlet.java
@@ -28,7 +28,7 @@
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.beam.runners.dataflow.worker.util.MemoryMonitor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
/**
* Respond to /heapz with a page allowing downloading of the heap dumps.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServlet.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServlet.java
index 9ad594e..be9a103 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServlet.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServlet.java
@@ -29,7 +29,7 @@
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** Respond to /threadz with the stack traces of all running threads. */
class ThreadzServlet extends BaseStatusServlet implements Capturable {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java
index 25efaf7..cd3b9de 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPages.java
@@ -20,12 +20,13 @@
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
+import java.util.function.BooleanSupplier;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.apache.beam.runners.dataflow.worker.status.DebugCapture.Capturable;
import org.apache.beam.runners.dataflow.worker.util.MemoryMonitor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletHandler;
import org.eclipse.jetty.servlet.ServletHolder;
@@ -43,13 +44,13 @@
private final ServletHandler servletHandler = new ServletHandler();
@VisibleForTesting
- WorkerStatusPages(Server server, MemoryMonitor memoryMonitor) {
+ WorkerStatusPages(Server server, MemoryMonitor memoryMonitor, BooleanSupplier healthyIndicator) {
this.statusServer = server;
this.statusServer.setHandler(servletHandler);
// Install the default servlets (threadz, healthz, heapz, statusz)
addServlet(threadzServlet);
- addServlet(new HealthzServlet());
+ addServlet(new HealthzServlet(healthyIndicator));
addServlet(new HeapzServlet(memoryMonitor));
addServlet(statuszServlet);
@@ -57,12 +58,13 @@
addStatusDataProvider("resources", "Resources", memoryMonitor);
}
- public static WorkerStatusPages create(int defaultStatusPort, MemoryMonitor memoryMonitor) {
+ public static WorkerStatusPages create(
+ int defaultStatusPort, MemoryMonitor memoryMonitor, BooleanSupplier healthyIndicator) {
int statusPort = defaultStatusPort;
if (System.getProperties().containsKey("status_port")) {
statusPort = Integer.parseInt(System.getProperty("status_port"));
}
- return new WorkerStatusPages(new Server(statusPort), memoryMonitor);
+ return new WorkerStatusPages(new Server(statusPort), memoryMonitor, healthyIndicator);
}
/** Start the server. */
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowAndCombineFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowAndCombineFn.java
index 3ab91ad..37b1136 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowAndCombineFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowAndCombineFn.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Arrays;
import java.util.Collection;
@@ -40,8 +40,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowViaIteratorsFn.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowViaIteratorsFn.java
index 34d3ce6..43e49dd 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowViaIteratorsFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowViaIteratorsFn.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Arrays;
@@ -37,10 +37,10 @@
import org.apache.beam.sdk.util.common.Reiterator;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowsDoFns.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowsDoFns.java
index a81dad9..d7b2dca 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowsDoFns.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/BatchGroupAlsoByWindowsDoFns.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.runners.core.StateInternalsFactory;
import org.apache.beam.runners.core.SystemReduceFn;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java
index 85d4012..5e6c35d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java
@@ -47,11 +47,11 @@
import org.apache.beam.sdk.io.fs.CreateOptions.StandardCreateOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AtomicDouble;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AtomicDouble;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -339,6 +339,10 @@
}
}
+ public boolean isThrashing() {
+ return isThrashing.get();
+ }
+
/**
* Check if we've observed high gc workload in sufficient sample periods to justify classifying
* the server as in gc thrashing.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ScalableBloomFilter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ScalableBloomFilter.java
index 2e7d5b1..eabcb48 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ScalableBloomFilter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ScalableBloomFilter.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.InputStream;
@@ -31,12 +31,12 @@
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.BloomFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnel;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.PrimitiveSink;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.math.DoubleMath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.math.LongMath;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.BloomFilter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnel;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.PrimitiveSink;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.DoubleMath;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
/**
* A Bloom filter implementation that maintains an expected false probability of {@code 0.000001}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/TimerOrElement.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/TimerOrElement.java
index dd4b89b..6065b68 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/TimerOrElement.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/TimerOrElement.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;
@@ -34,7 +34,7 @@
import org.apache.beam.runners.dataflow.util.Structs;
import org.apache.beam.runners.dataflow.worker.WindmillKeyedWorkItem.FakeKeyedWorkItemCoder;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Empty class which exists because the back end will sometimes insert uses of {@code
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ValueInEmptyWindows.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ValueInEmptyWindows.java
index 7c0cd58..66f2159 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ValueInEmptyWindows.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ValueInEmptyWindows.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Instant;
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/ForwardingReiterator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/ForwardingReiterator.java
index 875e4d2..2ffa849 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/ForwardingReiterator.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/ForwardingReiterator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util.common;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.sdk.util.common.Reiterator;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/BatchingShuffleEntryReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/BatchingShuffleEntryReader.java
index 74d7a3b..4f65112 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/BatchingShuffleEntryReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/BatchingShuffleEntryReader.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker.util.common.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ListIterator;
import java.util.NoSuchElementException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
index 718a3e9..3f6ab3c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
@@ -22,9 +22,9 @@
import java.util.Arrays;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Bytes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
/**
* Represents a position of a {@code GroupingShuffleReader} as an opaque array of bytes, encoded in
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/CachingShuffleBatchReader.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/CachingShuffleBatchReader.java
index 0cfe9b9..fc87bc4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/CachingShuffleBatchReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/CachingShuffleBatchReader.java
@@ -21,12 +21,12 @@
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
/** A {@link ShuffleBatchReader} that caches batches as they're read. */
public class CachingShuffleBatchReader implements ShuffleBatchReader {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/FlattenOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/FlattenOperation.java
index 91903a0..fd839e8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/FlattenOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/FlattenOperation.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker.util.common.worker;
import java.io.Closeable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** A flatten operation. */
public class FlattenOperation extends ReceivingOperation {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIterator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIterator.java
index 58f9f17..155e89c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIterator.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIterator.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker.util.common.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Arrays;
import java.util.NoSuchElementException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleRangeTracker.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleRangeTracker.java
index fb3fd46..a0a9e6b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleRangeTracker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleRangeTracker.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.runners.dataflow.worker.util.common.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import javax.annotation.Nullable;
import org.apache.beam.sdk.io.range.RangeTracker;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingTables.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingTables.java
index 534d7a0..d1ab378 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingTables.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingTables.java
@@ -23,7 +23,7 @@
import java.util.List;
import java.util.Map;
import java.util.Random;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** Static utility methods that provide {@link GroupingTable} implementations. */
public class GroupingTables {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutor.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutor.java
index 07d23ef..6365bf8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutor.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutor.java
@@ -23,8 +23,8 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoOperation.java
index 4412a60..4b6b2b3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ParDoOperation.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker.util.common.worker;
import java.io.Closeable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** A ParDo mapping function. */
public class ParDoOperation extends ReceivingOperation {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ProgressTrackingReiterator.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ProgressTrackingReiterator.java
index 334880a..3130d24 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ProgressTrackingReiterator.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ProgressTrackingReiterator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util.common.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.runners.dataflow.worker.util.common.ForwardingReiterator;
import org.apache.beam.sdk.util.common.Reiterator;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperation.java
index 2c2ffce..0b979be 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperation.java
@@ -31,9 +31,9 @@
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ShuffleReadCounter.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ShuffleReadCounter.java
index 8351a84..306e7b8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ShuffleReadCounter.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ShuffleReadCounter.java
@@ -23,7 +23,7 @@
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** Counts the Bytes and MSECS spent within a shuffle read. */
public class ShuffleReadCounter {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkExecutor.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkExecutor.java
index 08a0371..b549c0f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkExecutor.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkExecutor.java
@@ -20,7 +20,7 @@
import java.util.List;
import javax.annotation.Nullable;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** Abstract executor for WorkItem tasks. */
public interface WorkExecutor extends AutoCloseable {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdater.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdater.java
index 733d431..c52c1c9 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdater.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdater.java
@@ -24,8 +24,8 @@
import java.util.concurrent.TimeUnit;
import javax.annotation.concurrent.GuardedBy;
import javax.annotation.concurrent.NotThreadSafe;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -82,7 +82,7 @@
private final ScheduledExecutorService executor;
/** Clock used to either provide real system time or mocked to virtualize time for testing. */
- private final Clock clock;
+ protected final Clock clock;
/** The lease duration to request from the external worker service. */
protected long requestedLeaseDurationMs;
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WriteOperation.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WriteOperation.java
index 0d3e517..a22b243 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WriteOperation.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WriteOperation.java
@@ -20,7 +20,7 @@
import java.io.Closeable;
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** A write operation. */
public class WriteOperation extends ReceivingOperation {
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/DirectStreamObserver.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/DirectStreamObserver.java
index b491b6d..a54733d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/DirectStreamObserver.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/DirectStreamObserver.java
@@ -19,8 +19,8 @@
import java.util.concurrent.Phaser;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A {@link StreamObserver} which uses synchronization on the underlying {@link CallStreamObserver}
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/ForwardingClientResponseObserver.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/ForwardingClientResponseObserver.java
index cecbd1b..74d8e4d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/ForwardingClientResponseObserver.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/ForwardingClientResponseObserver.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.runners.dataflow.worker.windmill;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientResponseObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientResponseObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A {@link ClientResponseObserver} which delegates all {@link StreamObserver} calls.
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
index 299b085..27187fa 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
@@ -84,23 +84,23 @@
import org.apache.beam.sdk.util.BackOffUtils;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.Sleeper;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.CallCredentials;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Channel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.auth.MoreCallCredentials;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.netty.GrpcSslContexts;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.netty.NegotiationType;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.netty.NettyChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.CallCredentials;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.auth.MoreCallCredentials;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.GrpcSslContexts;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NegotiationType;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -122,10 +122,10 @@
private static final Duration MIN_BACKOFF = Duration.millis(1);
private static final Duration MAX_BACKOFF = Duration.standardSeconds(30);
- // Internal gRPC batch size is 64KB, so pick something slightly smaller to account for other
- // fields in the commit.
- private static final int COMMIT_STREAM_CHUNK_SIZE = 63 * 1024;
- private static final int GET_DATA_STREAM_CHUNK_SIZE = 63 * 1024;
+ // Default gRPC streams to 2MB chunks, which has shown to be a large enough chunk size to reduce
+ // per-chunk overhead, and small enough that we can still granularly flow-control.
+ private static final int COMMIT_STREAM_CHUNK_SIZE = 2 << 20;
+ private static final int GET_DATA_STREAM_CHUNK_SIZE = 2 << 20;
private static final AtomicLong nextId = new AtomicLong(0);
@@ -233,11 +233,11 @@
*/
private static class VendoredRequestMetadataCallbackAdapter
implements com.google.auth.RequestMetadataCallback {
- private final org.apache.beam.vendor.grpc.v1p13p1.com.google.auth.RequestMetadataCallback
+ private final org.apache.beam.vendor.grpc.v1p21p0.com.google.auth.RequestMetadataCallback
callback;
private VendoredRequestMetadataCallbackAdapter(
- org.apache.beam.vendor.grpc.v1p13p1.com.google.auth.RequestMetadataCallback callback) {
+ org.apache.beam.vendor.grpc.v1p21p0.com.google.auth.RequestMetadataCallback callback) {
this.callback = callback;
}
@@ -261,7 +261,7 @@
* delegate to reduce maintenance burden.
*/
private static class VendoredCredentialsAdapter
- extends org.apache.beam.vendor.grpc.v1p13p1.com.google.auth.Credentials {
+ extends org.apache.beam.vendor.grpc.v1p21p0.com.google.auth.Credentials {
private final com.google.auth.Credentials credentials;
private VendoredCredentialsAdapter(com.google.auth.Credentials credentials) {
@@ -282,7 +282,7 @@
public void getRequestMetadata(
final URI uri,
Executor executor,
- final org.apache.beam.vendor.grpc.v1p13p1.com.google.auth.RequestMetadataCallback
+ final org.apache.beam.vendor.grpc.v1p21p0.com.google.auth.RequestMetadataCallback
callback) {
credentials.getRequestMetadata(
uri, executor, new VendoredRequestMetadataCallbackAdapter(callback));
@@ -316,7 +316,7 @@
this.syncStubList.clear();
this.endpoints = ImmutableSet.<HostAndPort>copyOf(endpoints);
for (HostAndPort endpoint : this.endpoints) {
- if ("localhost".equals(endpoint.getHostText())) {
+ if ("localhost".equals(endpoint.getHost())) {
initializeLocalHost(endpoint.getPort());
} else {
CallCredentials creds =
@@ -348,7 +348,7 @@
}
private Channel remoteChannel(HostAndPort endpoint) throws IOException {
- return NettyChannelBuilder.forAddress(endpoint.getHostText(), endpoint.getPort())
+ return NettyChannelBuilder.forAddress(endpoint.getHost(), endpoint.getPort())
.maxInboundMessageSize(java.lang.Integer.MAX_VALUE)
.negotiationType(NegotiationType.TLS)
// Set ciphers(null) to not use GCM, which is disabled for Dataflow
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/StreamObserverFactory.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/StreamObserverFactory.java
index c8fd3de..6731951 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/StreamObserverFactory.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/StreamObserverFactory.java
@@ -20,8 +20,8 @@
import java.util.function.Function;
import org.apache.beam.sdk.fn.stream.AdvancingPhaser;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* Uses {@link PipelineOptions} to configure which underlying {@link StreamObserver} implementation
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerBase.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerBase.java
index 151e280..86b673e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerBase.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerBase.java
@@ -19,7 +19,7 @@
import java.io.IOException;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
/**
* Implementation of a WindmillServerStub which communcates with an actual windmill server at the
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerStub.java b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerStub.java
index 230cf80..2b5453b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerStub.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillServerStub.java
@@ -34,7 +34,7 @@
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.CommitStatus;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.KeyedGetDataRequest;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.KeyedGetDataResponse;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestExecutors.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestExecutors.java
index 5f88811..57c53a3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestExecutors.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestExecutors.java
@@ -20,7 +20,7 @@
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ForwardingExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ForwardingExecutorService;
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestStreams.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestStreams.java
index e9276e8..07ccdb1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestStreams.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/harness/test/TestStreams.java
@@ -19,8 +19,8 @@
import java.util.function.Consumer;
import java.util.function.Supplier;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/** Utility methods which enable testing of {@link StreamObserver}s. */
public class TestStreams {
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorkerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorkerTest.java
index 4fc485f..8bbe47e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorkerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchDataflowWorkerTest.java
@@ -23,7 +23,6 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
-import static org.mockito.Matchers.argThat;
import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
@@ -42,8 +41,8 @@
import org.apache.beam.runners.dataflow.util.TimeUtil;
import org.apache.beam.sdk.extensions.gcp.util.FastNanoClockAndSleeper;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
import org.joda.time.Duration;
@@ -56,6 +55,7 @@
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;
+import org.mockito.hamcrest.MockitoHamcrest;
/** Unit tests for {@link BatchDataflowWorker}. */
@RunWith(JUnit4.class)
@@ -108,7 +108,7 @@
assertTrue(worker.getAndPerformWork());
verify(mockWorkUnitClient)
.reportWorkItemStatus(
- argThat(
+ MockitoHamcrest.argThat(
new TypeSafeMatcher<WorkItemStatus>() {
@Override
public void describeTo(Description description) {}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java
index 6d58f4d..898396d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContextTest.java
@@ -34,6 +34,7 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.dataflow.worker.BatchModeExecutionContext.BatchModeExecutionState;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.NoopProfileScope;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfileScope;
@@ -78,7 +79,7 @@
.setOriginNamespace("namespace")
.setName("some-counter")
.setOriginalStepName("originalName"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(42));
@@ -102,7 +103,7 @@
.setOriginNamespace("namespace")
.setName("uncommitted-counter")
.setOriginalStepName("originalName"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(64));
@@ -145,7 +146,7 @@
.setOriginNamespace("namespace")
.setName("some-distribution")
.setOriginalStepName("originalName"))
- .setMetadata(new CounterMetadata().setKind("DISTRIBUTION")))
+ .setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString())))
.setCumulative(true)
.setDistribution(
new DistributionUpdate()
@@ -249,7 +250,7 @@
.setOrigin("SYSTEM")
.setName(counterName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(value));
}
@@ -265,7 +266,7 @@
.setName(counterName)
.setOriginalStepName(originalStepName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(longToSplitInt(value));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactoryTest.java
index 3608862..fb3e7f5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CombineValuesFnFactoryTest.java
@@ -56,9 +56,9 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java
index afcd328..85873ee 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java
@@ -31,8 +31,8 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoFn;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTrackerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTrackerTest.java
index db1ce6f..10d7c80 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTrackerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowElementExecutionTrackerTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ElementExecutionTracker;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateTrackerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateTrackerTest.java
index f1f8aa3..30ccedf 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateTrackerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateTrackerTest.java
@@ -38,7 +38,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ElementExecutionTracker;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.DateTimeUtils.MillisProvider;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowMatchers.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowMatchers.java
index 1044c7c..469add3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowMatchers.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowMatchers.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.dataflow.worker;
import java.io.Serializable;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContextTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContextTest.java
index 014d3cb..3412de3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContextTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowOperationContextTest.java
@@ -46,7 +46,7 @@
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfileScope;
import org.apache.beam.sdk.metrics.MetricsContainer;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.junit.After;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdaterTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdaterTest.java
index 648c578..abbde25 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdaterTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkProgressUpdaterTest.java
@@ -32,6 +32,7 @@
import static org.mockito.Mockito.when;
import com.google.api.client.testing.http.FixedClock;
+import com.google.api.services.dataflow.model.HotKeyDetection;
import com.google.api.services.dataflow.model.Position;
import com.google.api.services.dataflow.model.WorkItem;
import com.google.api.services.dataflow.model.WorkItemServiceState;
@@ -64,6 +65,8 @@
private static final String PROJECT_ID = "TEST_PROJECT_ID";
private static final String JOB_ID = "TEST_JOB_ID";
private static final Long WORK_ID = 1234567890L;
+ private static final String STEP_ID = "TEST_STEP_ID";
+ private static final Duration HOT_KEY_AGE = Duration.standardSeconds(1);
@Rule public final ExpectedException thrown = ExpectedException.none();
@@ -74,6 +77,7 @@
private FixedClock clock;
@Mock private WorkItemStatusClient workItemStatusClient;
@Mock private DataflowWorkExecutor worker;
+ @Mock private HotKeyLogger hotKeyLogger;
@Captor private ArgumentCaptor<DynamicSplitResult> splitResultCaptor;
@Before
@@ -93,7 +97,8 @@
progressUpdater =
new DataflowWorkProgressUpdater(
- workItemStatusClient, workItem, worker, executor.getExecutor(), clock) {
+ workItemStatusClient, workItem, worker, executor.getExecutor(), clock, hotKeyLogger) {
+
// Shorten reporting interval boundaries for faster testing.
@Override
protected long getMinReportingInterval() {
@@ -123,6 +128,18 @@
progressUpdater.stopReportingProgress();
}
+ @Test
+ public void workProgressLogsHotKeyDetection() throws Exception {
+ when(workItemStatusClient.reportUpdate(isNull(DynamicSplitResult.class), isA(Duration.class)))
+ .thenReturn(generateServiceState(null, 1000));
+ progressUpdater.startReportingProgress();
+ executor.runNextRunnable();
+
+ verify(hotKeyLogger, atLeastOnce()).logHotKeyDetection(STEP_ID, HOT_KEY_AGE);
+
+ progressUpdater.stopReportingProgress();
+ }
+
/** Verifies that the update after a split is requested acknowledeges it. */
@Test
public void workProgressSendsSplitResults() throws Exception {
@@ -247,6 +264,11 @@
ReaderTestUtils.approximateSplitRequestAtPosition(suggestedStopPosition));
}
+ HotKeyDetection hotKeyDetection = new HotKeyDetection();
+ hotKeyDetection.setUserStepName(STEP_ID);
+ hotKeyDetection.setHotKeyAge(toCloudDuration(HOT_KEY_AGE));
+ responseState.setHotKeyDetection(hotKeyDetection);
+
return responseState;
}
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClientTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClientTest.java
index 44c6319..947d290 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClientTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkUnitClientTest.java
@@ -42,9 +42,9 @@
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -86,6 +86,7 @@
pipelineOptions.setWorkerId(WORKER_ID);
pipelineOptions.setGcpCredential(new TestCredential());
pipelineOptions.setDataflowClient(service);
+ pipelineOptions.setRegion("us-central1");
}
@Test
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelperTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelperTest.java
index 9152051..72c894a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelperTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowWorkerHarnessHelperTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.runners.dataflow.worker.testing.RestoreDataflowLoggingMDC;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactoryTest.java
index 7646fa1..a82878d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DefaultParDoFnFactoryTest.java
@@ -23,6 +23,7 @@
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
+import java.util.Collections;
import org.apache.beam.runners.dataflow.util.CloudObject;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver;
@@ -39,7 +40,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -91,7 +92,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>("output") /* main output */,
- DoFnSchemaInformation.create())));
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap())));
CloudObject cloudUserFn = CloudObject.forClassName("DoFn");
addString(cloudUserFn, "serialized_fn", serializedFn);
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DoFnInstanceManagersTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DoFnInstanceManagersTest.java
index 31b7ff6..904b462 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DoFnInstanceManagersTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DoFnInstanceManagersTest.java
@@ -23,6 +23,7 @@
import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
+import java.util.Collections;
import org.apache.beam.runners.dataflow.util.PropertyNames;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
@@ -67,7 +68,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnInstanceManager mgr = DoFnInstanceManagers.singleInstance(info);
assertThat(mgr.peek(), Matchers.<DoFnInfo<?, ?>>theInstance(info));
@@ -88,7 +90,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnInstanceManager mgr = DoFnInstanceManagers.singleInstance(info);
mgr.abort(mgr.get());
@@ -108,7 +111,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnInstanceManager mgr = DoFnInstanceManagers.cloningPool(info);
DoFnInfo<?, ?> retrievedInfo = mgr.get();
@@ -130,7 +134,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnInstanceManager mgr = DoFnInstanceManagers.cloningPool(info);
DoFnInfo<?, ?> retrievedInfo = mgr.get();
@@ -154,7 +159,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
DoFnInstanceManager mgr = DoFnInstanceManagers.cloningPool(info);
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ExperimentContextTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ExperimentContextTest.java
index 9682d19..56a6575 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ExperimentContextTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ExperimentContextTest.java
@@ -26,7 +26,7 @@
import org.apache.beam.runners.dataflow.worker.ExperimentContext.Experiment;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FakeWindmillServer.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FakeWindmillServer.java
index 66c9cf0..2c27606 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FakeWindmillServer.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FakeWindmillServer.java
@@ -42,8 +42,8 @@
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.KeyedGetDataRequest;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill.WorkItemCommitRequest;
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Instant;
import org.junit.rules.ErrorCollector;
import org.slf4j.Logger;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java
index 67a5d0b..81d4a67 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/FnApiWindowMappingFnTest.java
@@ -50,7 +50,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
@@ -117,22 +117,21 @@
@Override
public <T> InboundDataClient receive(
- LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> consumer) {
+ LogicalEndpoint inputLocation, Coder<T> coder, FnDataReceiver<T> consumer) {
this.inboundReceiver = (FnDataReceiver) consumer;
this.inboundDataClient = CompletableFutureInboundDataClient.create();
return inboundDataClient;
}
@Override
- public <T> CloseableFnDataReceiver<WindowedValue<T>> send(
- LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder) {
- return new CloseableFnDataReceiver<WindowedValue<T>>() {
+ public <T> CloseableFnDataReceiver<T> send(LogicalEndpoint outputLocation, Coder<T> coder) {
+ return new CloseableFnDataReceiver<T>() {
@Override
- public void accept(WindowedValue<T> windowedValue) throws Exception {
+ public void accept(T value) throws Exception {
+ WindowedValue<KV<Object, Object>> windowedValue =
+ (WindowedValue<KV<Object, Object>>) value;
inputValues.add(windowedValue);
- KV<Object, Object> kv = (KV) windowedValue.getValue();
+ KV<Object, Object> kv = windowedValue.getValue();
inboundReceiver.accept(windowedValue.withValue(KV.of(kv.getKey(), outputValue)));
inboundDataClient.complete();
}
@@ -160,8 +159,7 @@
.build());
} else if (RequestCase.PROCESS_BUNDLE.equals(request.getRequestCase())) {
assertEquals(
- processBundleDescriptorId,
- request.getProcessBundle().getProcessBundleDescriptorReference());
+ processBundleDescriptorId, request.getProcessBundle().getProcessBundleDescriptorId());
return CompletableFuture.completedFuture(
InstructionResponse.newBuilder()
.setInstructionId(request.getInstructionId())
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactoryTest.java
index b8dd45a..f259722 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupAlsoByWindowParDoFnFactoryTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderTest.java
index ff53cb4..ed7bbc0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReaderTest.java
@@ -80,7 +80,7 @@
import org.apache.beam.sdk.util.common.Reiterable;
import org.apache.beam.sdk.util.common.Reiterator;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.After;
import org.junit.Before;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/HotKeyLoggerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/HotKeyLoggerTest.java
new file mode 100644
index 0000000..467034a
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/HotKeyLoggerTest.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.verify;
+import static org.powermock.api.mockito.PowerMockito.mock;
+import static org.powermock.api.mockito.PowerMockito.mockStatic;
+import static org.powermock.api.mockito.PowerMockito.when;
+
+import com.google.api.client.testing.http.FixedClock;
+import com.google.api.client.util.Clock;
+import org.joda.time.Duration;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mockito;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+@RunWith(PowerMockRunner.class)
+@PrepareForTest({HotKeyLoggerTest.class, LoggerFactory.class})
+public class HotKeyLoggerTest {
+ private FixedClock clock;
+
+ @Before
+ public void SetUp() {
+ clock = new FixedClock(Clock.SYSTEM.currentTimeMillis());
+ }
+
+ @Test
+ public void correctHotKeyMessage() {
+ HotKeyLogger hotKeyLogger = new HotKeyLogger(clock);
+
+ assertFalse(hotKeyLogger.isThrottled());
+ String m = hotKeyLogger.getHotKeyMessage("TEST_STEP_ID", "1s");
+ assertTrue(hotKeyLogger.isThrottled());
+
+ assertEquals(
+ "A hot key was detected in step 'TEST_STEP_ID' with age of '1s'. This is a "
+ + "symptom of key distribution being skewed. To fix, please inspect your data and "
+ + "pipeline to ensure that elements are evenly distributed across your key space.",
+ m);
+ }
+
+ @Test
+ public void throttlesLoggingHotKeyMessage() {
+ HotKeyLogger hotKeyLogger = new HotKeyLogger(clock);
+
+ clock.setTime(Clock.SYSTEM.currentTimeMillis());
+ assertFalse(hotKeyLogger.isThrottled());
+ assertTrue(hotKeyLogger.isThrottled());
+
+ // The class throttles every 5 minutes, so the first time it is called is true. The second time
+ // is throttled and returns false.
+ clock.setTime(clock.currentTimeMillis() + Duration.standardMinutes(5L).getMillis());
+ assertFalse(hotKeyLogger.isThrottled());
+ assertTrue(hotKeyLogger.isThrottled());
+
+ // Test that the state variable is set and can log again in 5 minutes.
+ clock.setTime(clock.currentTimeMillis() + Duration.standardMinutes(5L).getMillis());
+ assertFalse(hotKeyLogger.isThrottled());
+ assertTrue(hotKeyLogger.isThrottled());
+ }
+
+ @Test
+ public void logsHotKeyMessage() {
+ mockStatic(LoggerFactory.class);
+ Logger logger = mock(Logger.class);
+ when(LoggerFactory.getLogger(any(Class.class))).thenReturn(logger);
+
+ HotKeyLogger hotKeyLogger = new HotKeyLogger(clock);
+
+ // Logs because not throttled.
+ hotKeyLogger.logHotKeyDetection("TEST_STEP_ID", Duration.standardHours(1));
+
+ // Does not log because throttled.
+ hotKeyLogger.logHotKeyDetection("TEST_STEP_ID", Duration.standardHours(1));
+
+ // Only logs once because of throttling.
+ verify(logger, Mockito.times(1)).warn(anyString());
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactoryTest.java
index cad7595..a4178f4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorFactoryTest.java
@@ -51,6 +51,7 @@
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.List;
import java.util.function.Function;
import javax.annotation.Nullable;
@@ -99,11 +100,11 @@
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -484,7 +485,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create())));
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap())));
CloudObject cloudUserFn = CloudObject.forClassName("DoFn");
addString(cloudUserFn, PropertyNames.SERIALIZED_FN, serializedFn);
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorTest.java
index 67d3670..5ad42db 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IntrinsicMapTaskExecutorTest.java
@@ -69,7 +69,7 @@
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmFormatTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmFormatTest.java
index d041b08..e974124 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmFormatTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmFormatTest.java
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.testing.CoderPropertiesTest.NonDeterministicCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactoryTest.java
index cae5fb1..4bf9353 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderFactoryTest.java
@@ -39,10 +39,10 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderTest.java
index c9f79c8..92e3065 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmReaderTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
@@ -71,15 +71,15 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WeightedValue;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java
index 24de638..db981a9 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSideInputReaderTest.java
@@ -19,8 +19,8 @@
import static org.apache.beam.runners.dataflow.util.Structs.getString;
import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.concat;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.concat;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.empty;
import static org.hamcrest.Matchers.hasItem;
@@ -71,6 +71,7 @@
import org.apache.beam.runners.dataflow.util.RandomAccessData;
import org.apache.beam.runners.dataflow.worker.DataflowOperationContext.DataflowExecutionState;
import org.apache.beam.runners.dataflow.worker.ExperimentContext.Experiment;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.counters.Counter;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
@@ -106,20 +107,20 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TreeMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Closer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Closer;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.hamcrest.Matcher;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -1331,7 +1332,7 @@
new CounterUpdate()
.setStructuredNameAndMetadata(
new CounterStructuredNameAndMetadata()
- .setMetadata(new CounterMetadata().setKind("SUM"))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString()))
.setName(
new CounterStructuredName()
.setOrigin("SYSTEM")
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSinkTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSinkTest.java
index a60cfab..f64e8b8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSinkTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/IsmSinkTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.testing.CoderPropertiesTest.NonDeterministicCoder;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReaderTest.java
index 53430c4..213871f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReaderTest.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogRecordMatcherTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogRecordMatcherTest.java
index 208950e..1c1e0d0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogRecordMatcherTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogRecordMatcherTest.java
@@ -24,7 +24,7 @@
import java.util.Set;
import java.util.logging.Level;
import java.util.logging.LogRecord;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogSaverTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogSaverTest.java
index 88c074b..8b7068a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogSaverTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/LogSaverTest.java
@@ -25,7 +25,7 @@
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/OrderedCodeTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/OrderedCodeTest.java
index 465da45..311e008 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/OrderedCodeTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/OrderedCodeTest.java
@@ -23,8 +23,8 @@
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Bytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactoryTest.java
index 603458e..ebb32fa 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PairWithConstantKeyDoFnFactoryTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.StringUtils;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFnsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFnsTest.java
index ac693f4..966710d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFnsTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartialGroupByKeyParDoFnsTest.java
@@ -71,9 +71,9 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.hamcrest.collection.IsIterableContainingInAnyOrder;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderTest.java
index c757c23..cf784e1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PartitioningShuffleReaderTest.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubReaderTest.java
index f5ca0bc..fddfdc5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubReaderTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubSinkTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubSinkTest.java
index e7b06aa..9f45286 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubSinkTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/PubsubSinkTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReaderCacheTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReaderCacheTest.java
index 5e61ac3..51f20a4 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReaderCacheTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReaderCacheTest.java
@@ -26,8 +26,8 @@
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.apache.beam.sdk.io.UnboundedSource;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
import org.joda.time.Duration;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactoryTest.java
index 310264d..1aa8673 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ReifyTimestampAndWindowsParDoFnFactoryTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.equalTo;
import static org.junit.Assert.assertThat;
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrarTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrarTest.java
index ecc08c4..81f2646 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrarTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/RunnerHarnessCoderCloudObjectTranslatorRegistrarTest.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VoidCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkTest.java
index d6f8d04..1a87b3c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ShuffleSinkTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFnTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFnTest.java
index c079824..2933bf5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFnTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFnTest.java
@@ -59,9 +59,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -205,7 +205,8 @@
null /* side input views */,
null /* input coder */,
MAIN_OUTPUT,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
TestReceiver receiver = new TestReceiver();
TestReceiver receiver1 = new TestReceiver();
TestReceiver receiver2 = new TestReceiver();
@@ -230,6 +231,7 @@
.getStepContext(operationContext),
operationContext,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
SimpleDoFnRunnerFactory.INSTANCE);
userParDoFn.startBundle(receiver, receiver1, receiver2, receiver3);
@@ -284,7 +286,8 @@
null /* side input views */,
null /* input coder */,
MAIN_OUTPUT,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
TestReceiver receiver = new TestReceiver();
ParDoFn userParDoFn =
@@ -298,6 +301,7 @@
.getStepContext(operationContext),
operationContext,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
SimpleDoFnRunnerFactory.INSTANCE);
try {
@@ -333,7 +337,8 @@
null /* side input views */,
null /* input coder */,
MAIN_OUTPUT,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
TestReceiver receiver = new TestReceiver();
ParDoFn userParDoFn =
@@ -347,6 +352,7 @@
.getStepContext(operationContext),
operationContext,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
SimpleDoFnRunnerFactory.INSTANCE);
try {
@@ -424,7 +430,8 @@
null /* side input views */,
null /* input coder */,
MAIN_OUTPUT,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
CounterSet counters = new CounterSet();
TestOperationContext operationContext = TestOperationContext.create(counters);
ParDoFn userParDoFn =
@@ -438,6 +445,7 @@
.getStepContext(operationContext),
operationContext,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
SimpleDoFnRunnerFactory.INSTANCE);
userParDoFn.startBundle(new TestReceiver(), new TestReceiver());
@@ -484,7 +492,8 @@
null /* side input views */,
null /* input coder */,
MAIN_OUTPUT,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
ParDoFn userParDoFn =
new SimpleParDoFn<>(
@@ -498,6 +507,7 @@
.getStepContext(operationContext),
operationContext,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
SimpleDoFnRunnerFactory.INSTANCE);
// This test ensures proper behavior of the state sampling even with lazy initialization.
@@ -575,7 +585,8 @@
null /* side input views */,
null /* input coder */,
MAIN_OUTPUT,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
ParDoFn parDoFn =
new SimpleParDoFn<>(
@@ -587,6 +598,7 @@
stepContext,
operationContext,
DoFnSchemaInformation.create(),
+ Collections.emptyMap(),
SimpleDoFnRunnerFactory.INSTANCE);
parDoFn.startBundle(new TestReceiver());
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StateFetcherTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StateFetcherTest.java
index 790c924..5d90a85 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StateFetcherTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StateFetcherTest.java
@@ -43,10 +43,10 @@
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java
index 3cff21a..356dddb 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorkerTest.java
@@ -30,7 +30,10 @@
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
+import static org.mockito.ArgumentMatchers.nullable;
+import static org.mockito.Matchers.any;
import static org.mockito.Mockito.atLeast;
+import static org.mockito.Mockito.atLeastOnce;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
@@ -134,15 +137,15 @@
import org.apache.beam.sdk.values.ValueWithRecordId;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString.Output;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedLong;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString.Output;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedLong;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
@@ -219,6 +222,7 @@
@Rule public ErrorCollector errorCollector = new ErrorCollector();
WorkUnitClient mockWorkUnitClient = mock(WorkUnitClient.class);
+ HotKeyLogger hotKeyLogger = mock(HotKeyLogger.class);
private final Supplier<Long> idGenerator =
new Supplier<Long>() {
@@ -315,7 +319,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create()))));
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap()))));
return new ParallelInstruction()
.setSystemName(DEFAULT_PARDO_SYSTEM_NAME)
.setName(DEFAULT_PARDO_USER_NAME)
@@ -413,6 +418,7 @@
messageBuilder.setMetadata(addPaneTag(PaneInfo.NO_FIRING, metadata));
}
}
+
return builder.build();
}
@@ -498,6 +504,9 @@
+ " work_token: "
+ index
+ " cache_token: 3"
+ + " hot_key_info {"
+ + " hot_key_age_usec: 1000000"
+ + " }"
+ " message_bundles {"
+ " source_computation_id: \""
+ DEFAULT_SOURCE_COMPUTATION_ID
@@ -634,7 +643,8 @@
options,
null /* pipeline */,
SdkHarnessRegistries.emptySdkHarnessRegistry(),
- publishCounters);
+ publishCounters,
+ hotKeyLogger);
worker.addStateNameMappings(
ImmutableMap.of(DEFAULT_PARDO_USER_NAME, DEFAULT_PARDO_STATE_FAMILY));
return worker;
@@ -665,6 +675,8 @@
assertEquals(
makeExpectedOutput(i, TimeUnit.MILLISECONDS.toMicros(i)).build(), result.get((long) i));
}
+
+ verify(hotKeyLogger, atLeastOnce()).logHotKeyDetection(nullable(String.class), any());
}
@Test
@@ -702,6 +714,8 @@
assertEquals(
makeExpectedOutput(i, TimeUnit.MILLISECONDS.toMicros(i)).build(), result.get((long) i));
}
+
+ verify(hotKeyLogger, atLeastOnce()).logHotKeyDetection(nullable(String.class), any());
}
static class BlockingFn extends DoFn<String, String> implements TestRule {
@@ -2093,22 +2107,22 @@
ByteString key2 = ByteString.copyFromUtf8("key2");
MockWork m1 = new MockWork(1);
- computationState.activateWork(key1, m1);
+ assertTrue(computationState.activateWork(key1, m1));
Mockito.verify(mockExecutor).execute(m1);
computationState.completeWork(key1, 1);
Mockito.verifyNoMoreInteractions(mockExecutor);
// Verify work queues.
MockWork m2 = new MockWork(2);
- computationState.activateWork(key1, m2);
+ assertTrue(computationState.activateWork(key1, m2));
Mockito.verify(mockExecutor).execute(m2);
MockWork m3 = new MockWork(3);
- computationState.activateWork(key1, m3);
+ assertTrue(computationState.activateWork(key1, m3));
Mockito.verifyNoMoreInteractions(mockExecutor);
// Verify another key is a separate queue.
MockWork m4 = new MockWork(4);
- computationState.activateWork(key2, m4);
+ assertTrue(computationState.activateWork(key2, m4));
Mockito.verify(mockExecutor).execute(m4);
computationState.completeWork(key2, 4);
Mockito.verifyNoMoreInteractions(mockExecutor);
@@ -2118,9 +2132,12 @@
computationState.completeWork(key1, 3);
Mockito.verifyNoMoreInteractions(mockExecutor);
+ // Verify duplicate work dropped.
MockWork m5 = new MockWork(5);
computationState.activateWork(key1, m5);
Mockito.verify(mockExecutor).execute(m5);
+ assertFalse(computationState.activateWork(key1, m5));
+ Mockito.verifyNoMoreInteractions(mockExecutor);
computationState.completeWork(key1, 5);
Mockito.verifyNoMoreInteractions(mockExecutor);
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowFnsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowFnsTest.java
index f6a63f2..0acb1dc 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowFnsTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowFnsTest.java
@@ -75,7 +75,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsReshuffleDoFnTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsReshuffleDoFnTest.java
index c92eddf..11322a9 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsReshuffleDoFnTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowsReshuffleDoFnTest.java
@@ -48,7 +48,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunnerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunnerTest.java
index 8f0b9f0..790a7d5 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunnerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingKeyedWorkItemSideInputDoFnRunnerTest.java
@@ -54,8 +54,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java
index 6ebabc4..03d4376 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContextTest.java
@@ -47,6 +47,7 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker;
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowExecutionStateTracker;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.StreamingModeExecutionState;
import org.apache.beam.runners.dataflow.worker.StreamingModeExecutionContext.StreamingModeExecutionStateRegistry;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
@@ -64,8 +65,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
@@ -278,7 +279,7 @@
.setName(counterName)
.setOriginalStepName(originalStepName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(false)
.setInteger(longToSplitInt(value));
}
@@ -292,7 +293,7 @@
.setOrigin("SYSTEM")
.setName(counterName)
.setExecutionStepName(stageName))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(false)
.setInteger(longToSplitInt(value));
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactoryTest.java
index a502b52..a776cdb 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingPCollectionViewWriterDoFnFactoryTest.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunnerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunnerTest.java
index 6c845a1..24d17ff 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunnerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputDoFnRunnerTest.java
@@ -63,8 +63,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -408,7 +408,8 @@
null,
Collections.emptyMap(),
WindowingStrategy.of(windowFn),
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
return new StreamingSideInputDoFnRunner<>(simpleDoFnRunner, sideInputFetcher);
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcherTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcherTest.java
index 73368f3..d70d7f6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcherTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcherTest.java
@@ -49,9 +49,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java
index 8217cc6..c743478 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/StreamingStepMetricsContainerTest.java
@@ -98,7 +98,7 @@
.setOriginNamespace("ns")
.setName("name2")
.setOriginalStepName("s2"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(false)
.setInteger(longToSplitInt(12))));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/TestShuffleReader.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/TestShuffleReader.java
index 7d1bc67..99fdedb 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/TestShuffleReader.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/TestShuffleReader.java
@@ -31,7 +31,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShuffleEntryReader;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShufflePosition;
import org.apache.beam.sdk.util.common.Reiterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
/** A fake implementation of a ShuffleEntryReader, for testing. */
public class TestShuffleReader implements ShuffleEntryReader {
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactoryTest.java
index a2ed615..df53228 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ToIsmRecordForMultimapDoFnFactoryTest.java
@@ -30,8 +30,8 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoFn;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderTest.java
index e1df67e..31e3c45 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReaderTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Assert;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactoryTest.java
index 7c33572..5e688f3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/UserParDoFnFactoryTest.java
@@ -68,7 +68,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
@@ -311,7 +311,8 @@
null /* side input views */,
null /* input coder */,
new TupleTag<>(PropertyNames.OUTPUT) /* main output id */,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
object.set(
PropertyNames.SERIALIZED_FN,
StringUtils.byteArrayToJsonString(SerializableUtils.serializeToByteArray(info)));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactoryTest.java
index 42dce5b..f1078dd 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/ValuesDoFnFactoryTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.runners.dataflow.util.PropertyNames;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoFn;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItemTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItemTest.java
index a84113e..4441f35 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItemTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillKeyedWorkItemTest.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBaseTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBaseTest.java
index 79ccccc..bff116f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBaseTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillReaderIteratorBaseTest.java
@@ -26,7 +26,7 @@
import java.util.List;
import org.apache.beam.runners.dataflow.worker.windmill.Windmill;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateCacheTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateCacheTest.java
index 0c2db3e..64f425a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateCacheTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateCacheTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.state.State;
import org.apache.beam.sdk.state.StateSpec;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Test;
@@ -131,26 +131,27 @@
@Before
public void setUp() {
cache = new WindmillStateCache();
- keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L);
assertEquals(0, cache.getWeight());
}
@Test
public void testBasic() throws Exception {
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 1L);
assertNull(keyCache.get(StateNamespaces.global(), new TestStateTag("tag1")));
assertNull(keyCache.get(windowNamespace(0), new TestStateTag("tag2")));
assertNull(keyCache.get(triggerNamespace(0, 0), new TestStateTag("tag3")));
assertNull(keyCache.get(triggerNamespace(0, 0), new TestStateTag("tag2")));
keyCache.put(StateNamespaces.global(), new TestStateTag("tag1"), new TestState("g1"), 2);
- assertEquals(121, cache.getWeight());
+ assertEquals(129, cache.getWeight());
keyCache.put(windowNamespace(0), new TestStateTag("tag2"), new TestState("w2"), 2);
- assertEquals(242, cache.getWeight());
+ assertEquals(258, cache.getWeight());
keyCache.put(triggerNamespace(0, 0), new TestStateTag("tag3"), new TestState("t3"), 2);
- assertEquals(260, cache.getWeight());
+ assertEquals(276, cache.getWeight());
keyCache.put(triggerNamespace(0, 0), new TestStateTag("tag2"), new TestState("t2"), 2);
- assertEquals(278, cache.getWeight());
+ assertEquals(294, cache.getWeight());
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 2L);
assertEquals(
new TestState("g1"), keyCache.get(StateNamespaces.global(), new TestStateTag("tag1")));
assertEquals(new TestState("w2"), keyCache.get(windowNamespace(0), new TestStateTag("tag2")));
@@ -163,14 +164,17 @@
/** Verifies that values are cached in the appropriate namespaces. */
@Test
public void testInvalidation() throws Exception {
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 1L);
assertNull(keyCache.get(StateNamespaces.global(), new TestStateTag("tag1")));
keyCache.put(StateNamespaces.global(), new TestStateTag("tag1"), new TestState("g1"), 2);
- assertEquals(121, cache.getWeight());
+
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 2L);
+ assertEquals(129, cache.getWeight());
assertEquals(
new TestState("g1"), keyCache.get(StateNamespaces.global(), new TestStateTag("tag1")));
- keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 1L);
- assertEquals(121, cache.getWeight());
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 1L, 3L);
+ assertEquals(129, cache.getWeight());
assertNull(keyCache.get(StateNamespaces.global(), new TestStateTag("tag1")));
assertEquals(0, cache.getWeight());
}
@@ -178,45 +182,102 @@
/** Verifies that the cache is invalidated when the cache token changes. */
@Test
public void testEviction() throws Exception {
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 1L);
keyCache.put(windowNamespace(0), new TestStateTag("tag2"), new TestState("w2"), 2);
- assertEquals(121, cache.getWeight());
+ assertEquals(129, cache.getWeight());
keyCache.put(triggerNamespace(0, 0), new TestStateTag("tag3"), new TestState("t3"), 2000000000);
assertEquals(0, cache.getWeight());
// Eviction is atomic across the whole window.
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 2L);
assertNull(keyCache.get(windowNamespace(0), new TestStateTag("tag2")));
assertNull(keyCache.get(triggerNamespace(0, 0), new TestStateTag("tag3")));
}
+ /** Verifies that the cache does not vend for stale work tokens. */
+ @Test
+ public void testStaleWorkItem() throws Exception {
+ TestStateTag tag = new TestStateTag("tag2");
+
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 2L);
+ keyCache.put(windowNamespace(0), tag, new TestState("w2"), 2);
+ assertEquals(129, cache.getWeight());
+ // Same cache.
+ assertNull(keyCache.get(windowNamespace(0), tag));
+
+ // Previous work token.
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 1L);
+ assertNull(keyCache.get(windowNamespace(0), tag));
+
+ // Retry of work token that inserted.
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 2L);
+ assertNull(keyCache.get(windowNamespace(0), tag));
+
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 10L);
+ assertEquals(new TestState("w2"), keyCache.get(windowNamespace(0), tag));
+ keyCache.put(windowNamespace(0), tag, new TestState("w3"), 2);
+
+ // Ensure that second put updated work token.
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 5L);
+ assertNull(keyCache.get(windowNamespace(0), tag));
+
+ keyCache = cache.forComputation(COMPUTATION).forKey(KEY, STATE_FAMILY, 0L, 15L);
+ assertEquals(new TestState("w3"), keyCache.get(windowNamespace(0), tag));
+ }
+
/** Verifies that caches are kept independently per-key. */
@Test
public void testMultipleKeys() throws Exception {
- WindmillStateCache.ForKey keyCache1 =
- cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L);
- WindmillStateCache.ForKey keyCache2 =
- cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key2"), STATE_FAMILY, 0L);
- WindmillStateCache.ForKey keyCache3 =
- cache.forComputation("comp2").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L);
+ TestStateTag tag = new TestStateTag("tag1");
- keyCache1.put(StateNamespaces.global(), new TestStateTag("tag1"), new TestState("g1"), 2);
- assertEquals(
- new TestState("g1"), keyCache1.get(StateNamespaces.global(), new TestStateTag("tag1")));
- assertNull(keyCache2.get(StateNamespaces.global(), new TestStateTag("tag1")));
- assertNull(keyCache3.get(StateNamespaces.global(), new TestStateTag("tag1")));
+ WindmillStateCache.ForKey keyCache1 =
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 0L);
+ WindmillStateCache.ForKey keyCache2 =
+ cache
+ .forComputation("comp1")
+ .forKey(ByteString.copyFromUtf8("key2"), STATE_FAMILY, 0L, 10L);
+ WindmillStateCache.ForKey keyCache3 =
+ cache.forComputation("comp2").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 0L);
+
+ TestState state1 = new TestState("g1");
+ keyCache1.put(StateNamespaces.global(), tag, state1, 2);
+ assertNull(keyCache1.get(StateNamespaces.global(), tag));
+ keyCache1 =
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 1L);
+ assertEquals(state1, keyCache1.get(StateNamespaces.global(), tag));
+ assertNull(keyCache2.get(StateNamespaces.global(), tag));
+ assertNull(keyCache3.get(StateNamespaces.global(), tag));
+
+ TestState state2 = new TestState("g2");
+ keyCache2.put(StateNamespaces.global(), tag, state2, 2);
+ assertNull(keyCache2.get(StateNamespaces.global(), tag));
+ keyCache2 =
+ cache
+ .forComputation("comp1")
+ .forKey(ByteString.copyFromUtf8("key2"), STATE_FAMILY, 0L, 20L);
+ assertEquals(state2, keyCache2.get(StateNamespaces.global(), tag));
+ assertEquals(state1, keyCache1.get(StateNamespaces.global(), tag));
+ assertNull(keyCache3.get(StateNamespaces.global(), tag));
}
/** Verifies explicit invalidation does indeed invalidate the correct entries. */
@Test
public void testExplicitInvalidation() throws Exception {
WindmillStateCache.ForKey keyCache1 =
- cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L);
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 0L);
WindmillStateCache.ForKey keyCache2 =
- cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key2"), STATE_FAMILY, 0L);
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key2"), STATE_FAMILY, 0L, 0L);
WindmillStateCache.ForKey keyCache3 =
- cache.forComputation("comp2").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L);
+ cache.forComputation("comp2").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 0L);
keyCache1.put(StateNamespaces.global(), new TestStateTag("tag1"), new TestState("g1"), 1);
keyCache2.put(StateNamespaces.global(), new TestStateTag("tag2"), new TestState("g2"), 2);
keyCache3.put(StateNamespaces.global(), new TestStateTag("tag3"), new TestState("g3"), 3);
+ keyCache1 =
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 1L);
+ keyCache2 =
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key2"), STATE_FAMILY, 0L, 1L);
+ keyCache3 =
+ cache.forComputation("comp2").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 1L);
assertEquals(
new TestState("g1"), keyCache1.get(StateNamespaces.global(), new TestStateTag("tag1")));
assertEquals(
@@ -232,4 +293,40 @@
assertEquals(
new TestState("g3"), keyCache3.get(StateNamespaces.global(), new TestStateTag("tag3")));
}
+
+ private static class TestStateTagWithBadEquality extends TestStateTag {
+ public TestStateTagWithBadEquality(String id) {
+ super(id);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return this == other;
+ }
+
+ @Override
+ public int hashCode() {
+ return System.identityHashCode(this);
+ }
+ }
+
+ /**
+ * Verifies that caching works properly even when the StateTag does not properly implement
+ * equals() and hashCode()
+ */
+ @Test
+ public void testBadCoderEquality() throws Exception {
+ WindmillStateCache.ForKey keyCache1 =
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 0L);
+
+ StateTag<TestState> tag = new TestStateTagWithBadEquality("tag1");
+ keyCache1.put(StateNamespaces.global(), tag, new TestState("g1"), 1);
+
+ keyCache1 =
+ cache.forComputation("comp1").forKey(ByteString.copyFromUtf8("key1"), STATE_FAMILY, 0L, 1L);
+ assertEquals(new TestState("g1"), keyCache1.get(StateNamespaces.global(), tag));
+ assertEquals(
+ new TestState("g1"),
+ keyCache1.get(StateNamespaces.global(), new TestStateTagWithBadEquality("tag1")));
+ }
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternalsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternalsTest.java
index 30777b1..f708500 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternalsTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternalsTest.java
@@ -50,11 +50,11 @@
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.SettableFuture;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.After;
@@ -78,6 +78,7 @@
private static final ByteString COMBINING_KEY = key(NAMESPACE, "combining");
private final Coder<int[]> accumCoder =
Sum.ofIntegers().getAccumulatorCoder(null, VarIntCoder.of());
+ private long workToken = 0;
@Mock private WindmillStateReader mockReader;
@@ -99,21 +100,26 @@
public void setUp() {
MockitoAnnotations.initMocks(this);
cache = new WindmillStateCache();
+ resetUnderTest();
+ }
+
+ public void resetUnderTest() {
+ workToken++;
underTest =
new WindmillStateInternals<>(
"dummyKey",
STATE_FAMILY,
mockReader,
false,
- cache.forComputation("comp").forKey(ByteString.EMPTY, STATE_FAMILY, 17L),
+ cache.forComputation("comp").forKey(ByteString.EMPTY, STATE_FAMILY, 17L, workToken),
readStateSupplier);
underTestNewKey =
new WindmillStateInternals<String>(
- "dummyKey",
+ "dummyNewKey",
STATE_FAMILY,
mockReader,
true,
- cache.forComputation("comp").forKey(ByteString.EMPTY, STATE_FAMILY, 17L),
+ cache.forComputation("comp").forKey(ByteString.EMPTY, STATE_FAMILY, 17L, workToken),
readStateSupplier);
}
@@ -401,12 +407,12 @@
public void testCombiningAddPersistWithCompact() throws Exception {
forceCompactOnWrite();
- Mockito.stub(
+ Mockito.when(
mockReader.bagFuture(
org.mockito.Matchers.<ByteString>any(),
org.mockito.Matchers.<String>any(),
org.mockito.Matchers.<Coder<int[]>>any()))
- .toReturn(
+ .thenReturn(
Futures.<Iterable<int[]>>immediateFuture(
ImmutableList.of(new int[] {40}, new int[] {60})));
@@ -848,15 +854,17 @@
value.write("Hi");
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(118, cache.getWeight());
+ assertEquals(126, cache.getWeight());
+ resetUnderTest();
value = underTest.state(NAMESPACE, addr);
assertEquals("Hi", value.read());
value.clear();
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(116, cache.getWeight());
+ assertEquals(124, cache.getWeight());
+ resetUnderTest();
value = underTest.state(NAMESPACE, addr);
assertEquals(null, value.read());
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
@@ -886,8 +894,9 @@
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(126, cache.getWeight());
+ assertEquals(134, cache.getWeight());
+ resetUnderTest();
bag = underTest.state(NAMESPACE, addr);
bag.add("goodbye");
@@ -905,8 +914,9 @@
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(119, cache.getWeight());
+ assertEquals(127, cache.getWeight());
+ resetUnderTest();
bag = underTest.state(NAMESPACE, addr);
bag.add("new2");
assertThat(bag.read(), Matchers.containsInAnyOrder("new", "new2"));
@@ -915,8 +925,9 @@
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(120, cache.getWeight());
+ assertEquals(128, cache.getWeight());
+ resetUnderTest();
bag = underTest.state(NAMESPACE, addr);
assertThat(bag.read(), Matchers.containsInAnyOrder("new3"));
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
@@ -945,16 +956,18 @@
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(124, cache.getWeight());
+ assertEquals(132, cache.getWeight());
+ resetUnderTest();
hold = underTest.state(NAMESPACE, addr);
assertThat(hold.read(), Matchers.equalTo(new Instant(2000)));
hold.clear();
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(124, cache.getWeight());
+ assertEquals(132, cache.getWeight());
+ resetUnderTest();
hold = underTest.state(NAMESPACE, addr);
assertEquals(null, hold.read());
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
@@ -983,8 +996,9 @@
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(117, cache.getWeight());
+ assertEquals(125, cache.getWeight());
+ resetUnderTest();
value = underTest.state(NAMESPACE, COMBINING_ADDR);
assertThat(value.read(), Matchers.equalTo(3));
value.add(3);
@@ -993,8 +1007,9 @@
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
- assertEquals(116, cache.getWeight());
+ assertEquals(124, cache.getWeight());
+ resetUnderTest();
value = underTest.state(NAMESPACE, COMBINING_ADDR);
assertThat(value.read(), Matchers.equalTo(0));
underTest.persist(Windmill.WorkItemCommitRequest.newBuilder());
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateReaderTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateReaderTest.java
index aba4a99..ef529df 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateReaderTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillStateReaderTest.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString.Output;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString.Output;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Before;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternalsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternalsTest.java
index ec52508..d0f0f8a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternalsTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternalsTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java
index 9fcdb14..26f8502 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkItemStatusClientTest.java
@@ -53,6 +53,7 @@
import org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
import org.apache.beam.runners.dataflow.worker.SourceTranslationUtils.DataflowReaderPosition;
import org.apache.beam.runners.dataflow.worker.WorkerCustomSources.BoundedSourceSplit;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
@@ -66,7 +67,7 @@
import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Before;
import org.junit.Rule;
@@ -346,7 +347,7 @@
public void populateCounterUpdatesWithOutputCounters() throws Exception {
final CounterUpdate counter =
new CounterUpdate()
- .setNameAndKind(new NameAndKind().setName("some-counter").setKind("SUM"))
+ .setNameAndKind(new NameAndKind().setName("some-counter").setKind(Kind.SUM.toString()))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
@@ -368,7 +369,7 @@
public void populateCounterUpdatesWithMetricsAndCounters() throws Exception {
final CounterUpdate expectedCounter =
new CounterUpdate()
- .setNameAndKind(new NameAndKind().setName("some-counter").setKind("SUM"))
+ .setNameAndKind(new NameAndKind().setName("some-counter").setKind(Kind.SUM.toString()))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
@@ -385,7 +386,7 @@
.setOriginNamespace("namespace")
.setName("some-counter")
.setOriginalStepName("step"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
@@ -422,7 +423,7 @@
.setOrigin("SYSTEM")
.setName("start-msecs")
.setOriginalStepName("step"))
- .setMetadata(new CounterMetadata().setKind("SUM")))
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
.setCumulative(true)
.setInteger(DataflowCounterUpdateExtractor.longToSplitInt(42));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesSplitOnlySourceTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesSplitOnlySourceTest.java
index 7ec5443..f03719d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesSplitOnlySourceTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesSplitOnlySourceTest.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.io.CountingSource;
import org.apache.beam.sdk.io.OffsetBasedSource;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesTest.java
index 4205e85..181183d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSourcesTest.java
@@ -30,8 +30,8 @@
import static org.apache.beam.sdk.util.CoderUtils.encodeToByteArray;
import static org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray;
import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.contains;
@@ -102,10 +102,10 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.ValueWithRecordId;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerPipelineOptionsFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerPipelineOptionsFactoryTest.java
index bbbf1ff..9c789dc 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerPipelineOptionsFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/WorkerPipelineOptionsFactoryTest.java
@@ -25,7 +25,7 @@
import java.nio.file.Paths;
import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructionsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructionsTest.java
index fdfec07..233009c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructionsTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/apiary/FixMultiOutputInfosOnParDoInstructionsTest.java
@@ -26,7 +26,7 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.fn.IdGenerators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactoryTest.java
index d8c286b..6a13fb2 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterFactoryTest.java
@@ -21,7 +21,7 @@
import java.util.List;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory.CounterDistribution;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterTest.java
index 64b2a7a..fcd44f8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterTest.java
@@ -26,7 +26,7 @@
import org.apache.beam.runners.dataflow.worker.counters.Counter.CommitState;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory.CounterDistribution;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory.CounterMean;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -335,51 +335,51 @@
@Test
public void testBoolAnd() {
Counter<Boolean, Boolean> c = counters.booleanAnd(name);
- assertEquals(true, c.getAggregate());
+ assertTrue(c.getAggregate());
c.addValue(true);
- assertEquals(true, c.getAggregate());
+ assertTrue(c.getAggregate());
c.addValue(false);
- assertEquals(false, c.getAggregate());
+ assertFalse(c.getAggregate());
c.getAndReset();
c.addValue(true).addValue(true);
- assertEquals(true, c.getAggregate());
+ assertTrue(c.getAggregate());
c.addValue(false);
- assertEquals(false, c.getAggregate());
+ assertFalse(c.getAggregate());
- assertEquals(false, c.getAndReset());
- assertEquals(true, c.getAggregate());
+ assertFalse(c.getAndReset());
+ assertTrue(c.getAggregate());
c.addValue(false);
- assertEquals(false, c.getAggregate());
+ assertFalse(c.getAggregate());
}
@Test
public void testBoolOr() {
Counter<Boolean, Boolean> c = counters.booleanOr(name);
- assertEquals(false, c.getAggregate());
+ assertFalse(c.getAggregate());
c.addValue(false);
- assertEquals(false, c.getAggregate());
+ assertFalse(c.getAggregate());
c.addValue(true);
- assertEquals(true, c.getAggregate());
+ assertTrue(c.getAggregate());
c.getAndReset();
c.addValue(false).addValue(false);
- assertEquals(false, c.getAggregate());
+ assertFalse(c.getAggregate());
c.addValue(true);
- assertEquals(true, c.getAggregate());
+ assertTrue(c.getAggregate());
- assertEquals(true, c.getAndReset());
- assertEquals(false, c.getAggregate());
+ assertTrue(c.getAndReset());
+ assertFalse(c.getAggregate());
c.addValue(true);
- assertEquals(true, c.getAggregate());
+ assertTrue(c.getAggregate());
}
@Test
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java
new file mode 100644
index 0000000..17e6aa0
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.DistributionUpdate;
+import com.google.api.services.dataflow.model.IntegerMean;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Test;
+
+public class CounterUpdateAggregatorsTest {
+
+ @Test
+ public void testAggregateSum() {
+ List<CounterUpdate> sumUpdates = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ sumUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
+ .setInteger(longToSplitInt((long) i)));
+ }
+ List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(sumUpdates);
+ assertEquals(1, aggregated.size());
+ CounterUpdate combined = aggregated.get(0);
+ assertEquals(45L, splitIntToLong(combined.getInteger()));
+ }
+
+ @Test
+ public void testAggregateMean() {
+ List<CounterUpdate> meanUpdates = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ meanUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.MEAN.toString())))
+ .setIntegerMean(
+ new IntegerMean().setSum(longToSplitInt((long) i)).setCount(longToSplitInt(1L))));
+ }
+ List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(meanUpdates);
+ assertEquals(1, aggregated.size());
+ CounterUpdate combined = aggregated.get(0);
+ assertEquals(45L, splitIntToLong(combined.getIntegerMean().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getIntegerMean().getCount()));
+ }
+
+ @Test
+ public void testAggregateDistribution() {
+ List<CounterUpdate> distributionUpdates = new ArrayList<>();
+ for (int i = 0; i < 10; i++) {
+ distributionUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.DISTRIBUTION.toString())))
+ .setDistribution(
+ new DistributionUpdate()
+ .setSum(longToSplitInt((long) i))
+ .setMax(longToSplitInt((long) i))
+ .setMin(longToSplitInt((long) i))
+ .setCount(longToSplitInt((long) 1))));
+ }
+ List<CounterUpdate> aggregated = CounterUpdateAggregators.aggregate(distributionUpdates);
+ assertEquals(1, aggregated.size());
+ CounterUpdate combined = aggregated.get(0);
+ assertEquals(45L, splitIntToLong(combined.getDistribution().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getDistribution().getCount()));
+ assertEquals(9L, splitIntToLong(combined.getDistribution().getMax()));
+ assertEquals(0L, splitIntToLong(combined.getDistribution().getMin()));
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java
new file mode 100644
index 0000000..8dac8a7
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.DistributionUpdate;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Before;
+import org.junit.Test;
+
+public class DistributionCounterUpdateAggregatorTest {
+
+ private List<CounterUpdate> counterUpdates;
+ private DistributionCounterUpdateAggregator aggregator;
+
+ @Before
+ public void setUp() {
+ counterUpdates = new ArrayList<>();
+ aggregator = new DistributionCounterUpdateAggregator();
+ for (int i = 0; i < 10; i++) {
+ counterUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.MEAN.toString())))
+ .setDistribution(
+ new DistributionUpdate()
+ .setSum(longToSplitInt((long) i))
+ .setMax(longToSplitInt((long) i))
+ .setMin(longToSplitInt((long) i))
+ .setCount(longToSplitInt((long) 1))));
+ }
+ }
+
+ @Test
+ public void testAggregate() {
+ CounterUpdate combined = aggregator.aggregate(counterUpdates);
+ assertEquals(45L, splitIntToLong(combined.getDistribution().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getDistribution().getCount()));
+ assertEquals(9L, splitIntToLong(combined.getDistribution().getMax()));
+ assertEquals(0L, splitIntToLong(combined.getDistribution().getMin()));
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testAggregateWithNullIntegerDistribution() {
+ counterUpdates.get(0).setDistribution(null);
+ aggregator.aggregate(counterUpdates);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java
new file mode 100644
index 0000000..9ea7a31
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import com.google.api.services.dataflow.model.IntegerMean;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Before;
+import org.junit.Test;
+
+public class MeanCounterUpdateAggregatorTest {
+
+ private List<CounterUpdate> counterUpdates;
+ private MeanCounterUpdateAggregator aggregator;
+
+ @Before
+ public void setUp() {
+ counterUpdates = new ArrayList<>();
+ aggregator = new MeanCounterUpdateAggregator();
+ for (int i = 0; i < 10; i++) {
+ counterUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.MEAN.toString())))
+ .setIntegerMean(
+ new IntegerMean().setSum(longToSplitInt((long) i)).setCount(longToSplitInt(1L))));
+ }
+ }
+
+ @Test
+ public void testAggregate() {
+ CounterUpdate combined = aggregator.aggregate(counterUpdates);
+ assertEquals(45L, splitIntToLong(combined.getIntegerMean().getSum()));
+ assertEquals(10L, splitIntToLong(combined.getIntegerMean().getCount()));
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testAggregateWithNullIntegerMean() {
+ counterUpdates.get(0).setIntegerMean(null);
+ aggregator.aggregate(counterUpdates);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java
new file mode 100644
index 0000000..e30354f
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.counters;
+
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.longToSplitInt;
+import static org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor.splitIntToLong;
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.services.dataflow.model.CounterMetadata;
+import com.google.api.services.dataflow.model.CounterStructuredNameAndMetadata;
+import com.google.api.services.dataflow.model.CounterUpdate;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.dataflow.worker.MetricsToCounterUpdateConverter.Kind;
+import org.junit.Before;
+import org.junit.Test;
+
+public class SumCounterUpdateAggregatorTest {
+ private List<CounterUpdate> counterUpdates;
+ private SumCounterUpdateAggregator aggregator;
+
+ @Before
+ public void setUp() {
+ counterUpdates = new ArrayList<>();
+ aggregator = new SumCounterUpdateAggregator();
+ for (int i = 0; i < 10; i++) {
+ counterUpdates.add(
+ new CounterUpdate()
+ .setStructuredNameAndMetadata(
+ new CounterStructuredNameAndMetadata()
+ .setMetadata(new CounterMetadata().setKind(Kind.SUM.toString())))
+ .setInteger(longToSplitInt((long) i)));
+ }
+ }
+
+ @Test
+ public void testAggregate() {
+ CounterUpdate combined = aggregator.aggregate(counterUpdates);
+ assertEquals(45L, splitIntToLong(combined.getInteger()));
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testAggregateWithNullInteger() {
+ counterUpdates.get(0).setInteger(null);
+ aggregator.aggregate(counterUpdates);
+ }
+}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlServiceTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlServiceTest.java
index 86e2c3e..0cac04e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlServiceTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/BeamFnControlServiceTest.java
@@ -18,11 +18,13 @@
package org.apache.beam.runners.dataflow.worker.fn;
import static org.junit.Assert.assertNotNull;
+import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import java.net.InetAddress;
import java.net.ServerSocket;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnControlGrpc;
@@ -33,11 +35,11 @@
import org.apache.beam.runners.fnexecution.control.FnApiControlClient;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -67,6 +69,15 @@
@Test
public void testClientConnecting() throws Exception {
+ CountDownLatch requestCompleted = new CountDownLatch(1);
+ doAnswer(
+ invocation -> {
+ requestCompleted.countDown();
+ return null;
+ })
+ .when(requestObserver)
+ .onCompleted();
+
PipelineOptions options = PipelineOptionsFactory.create();
Endpoints.ApiServiceDescriptor descriptor = findOpenPort();
BeamFnControlService service =
@@ -87,7 +98,8 @@
server.shutdown();
server.awaitTermination(1, TimeUnit.SECONDS);
server.shutdownNow();
- Thread.sleep(1000); // Wait for stub to close stream.
+
+ requestCompleted.await(5, TimeUnit.SECONDS); // Wait until request streams have been closed.
verify(requestObserver).onCompleted();
verifyNoMoreInteractions(requestObserver);
@@ -95,6 +107,22 @@
@Test
public void testMultipleClientsConnecting() throws Exception {
+ CountDownLatch requestCompleted = new CountDownLatch(2);
+ doAnswer(
+ invocation -> {
+ requestCompleted.countDown();
+ return null;
+ })
+ .when(requestObserver)
+ .onCompleted();
+ doAnswer(
+ invocation -> {
+ requestCompleted.countDown();
+ return null;
+ })
+ .when(anotherRequestObserver)
+ .onCompleted();
+
PipelineOptions options = PipelineOptionsFactory.create();
Endpoints.ApiServiceDescriptor descriptor = findOpenPort();
BeamFnControlService service =
@@ -126,7 +154,8 @@
server.shutdown();
server.awaitTermination(1, TimeUnit.SECONDS);
server.shutdownNow();
- Thread.sleep(1000); // Wait for stub to close stream.
+
+ requestCompleted.await(5, TimeUnit.SECONDS); // Wait until request streams have been closed.
verify(requestObserver).onCompleted();
verifyNoMoreInteractions(requestObserver);
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
index 6263610..89d4595 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutorTest.java
@@ -60,10 +60,10 @@
import org.apache.beam.sdk.fn.data.RemoteGrpcPortRead;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactoryTest.java
index 77942b5..3e9bfe7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactoryTest.java
@@ -44,8 +44,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java
index 71c36df..321a236 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/RegisterAndProcessBundleOperationTest.java
@@ -80,10 +80,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.ValueInSingleWindow.Coder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableTable;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -233,8 +233,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("778")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("555"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("555"))
.build());
operation.finish();
@@ -245,8 +244,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("779")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("555"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("555"))
.build());
operation.finish();
}
@@ -516,8 +514,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("778")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("555"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("555"))
.build());
}
@@ -549,7 +546,7 @@
StateKey.newBuilder()
.setBagUserState(
StateKey.BagUserState.newBuilder()
- .setPtransformId("testPTransformId")
+ .setTransformId("testPTransformId")
.setWindow(ByteString.EMPTY)
.setUserStateId("testUserStateId")))
.buildPartial();
@@ -657,7 +654,7 @@
StateKey.newBuilder()
.setMultimapSideInput(
StateKey.MultimapSideInput.newBuilder()
- .setPtransformId("testPTransformId")
+ .setTransformId("testPTransformId")
.setSideInputId("testSideInputId")
.setWindow(
ByteString.copyFrom(
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/TimerReceiverTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/TimerReceiverTest.java
index 94b6e29..6ea4380 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/TimerReceiverTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/TimerReceiverTest.java
@@ -17,7 +17,8 @@
*/
package org.apache.beam.runners.dataflow.worker.fn.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.junit.Assert.assertTrue;
import java.io.Serializable;
import java.util.HashMap;
@@ -29,7 +30,6 @@
import java.util.concurrent.ThreadFactory;
import javax.annotation.Nullable;
import org.apache.beam.fn.harness.FnHarness;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.InMemoryTimerInternals;
import org.apache.beam.runners.core.StateInternals;
@@ -70,11 +70,11 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.After;
@@ -234,7 +234,7 @@
Object windowedTimer = WindowedValue.valueInGlobalWindow(timer);
// Simulate the SDK Harness sending a timer element to the Runner Harness.
- org.junit.Assert.assertTrue(timerReceiver.receive(timerOutputPCollection, windowedTimer));
+ assertTrue(timerReceiver.receive(timerOutputPCollection, windowedTimer));
// Expect that we get a timer element when we finish.
Object expected =
@@ -352,9 +352,9 @@
Object windowedTimer2 = WindowedValue.valueInGlobalWindow(timer2);
// Simulate the SDK Harness sending a timer element to the Runner Harness.
- org.junit.Assert.assertTrue(
+ assertTrue(
timerReceiver.receive(timerSpecMap.get(timerId1).outputCollectionId(), windowedTimer1));
- org.junit.Assert.assertTrue(
+ assertTrue(
timerReceiver.receive(timerSpecMap.get(timerId2).outputCollectionId(), windowedTimer2));
// Expect that we get a timer element when we finish.
@@ -419,22 +419,22 @@
StateRequestHandler stateRequestHandler,
BundleProgressHandler progressHandler)
throws Exception {
- ImmutableMap.Builder<BeamFnApi.Target, RemoteOutputReceiver<?>> outputReceivers =
+ ImmutableMap.Builder<String, RemoteOutputReceiver<?>> outputReceivers =
ImmutableMap.builder();
- for (Map.Entry<BeamFnApi.Target, Coder<WindowedValue<?>>> targetCoder :
- processBundleDescriptor.getOutputTargetCoders().entrySet()) {
- BeamFnApi.Target target = targetCoder.getKey();
- Coder<WindowedValue<?>> coder = targetCoder.getValue();
+ for (Map.Entry<String, Coder> remoteOutputCoder :
+ processBundleDescriptor.getRemoteOutputCoders().entrySet()) {
String bundleOutputPCollection =
Iterables.getOnlyElement(
processBundleDescriptor
.getProcessBundleDescriptor()
- .getTransformsOrThrow(target.getPrimitiveTransformReference())
+ .getTransformsOrThrow(remoteOutputCoder.getKey())
.getInputsMap()
.values());
FnDataReceiver<WindowedValue<?>> outputReceiver =
outputReceiverFactory.create(bundleOutputPCollection);
- outputReceivers.put(target, RemoteOutputReceiver.of(coder, outputReceiver));
+ outputReceivers.put(
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of(remoteOutputCoder.getValue(), outputReceiver));
}
return processor.newBundle(outputReceivers.build(), stateRequestHandler, progressHandler);
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java
index 01ddeea..9c2b57a 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/BeamFnDataGrpcServiceTest.java
@@ -51,22 +51,22 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.BindableService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.CallOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Channel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ClientCall;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ClientInterceptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCall.SimpleForwardingClientCall;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata.Key;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.MethodDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerInterceptors;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.BindableService;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.CallOptions;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ClientCall;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ClientInterceptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingClientCall.SimpleForwardingClientCall;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata.Key;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.MethodDescriptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerInterceptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
@@ -77,8 +77,7 @@
@RunWith(JUnit4.class)
@SuppressWarnings("FutureReturnValueIgnored")
public class BeamFnDataGrpcServiceTest {
- private static final BeamFnApi.Target TARGET =
- BeamFnApi.Target.newBuilder().setPrimitiveTransformReference("888").setName("test").build();
+ private static final String TRANSFORM_ID = "888";
private static final Coder<WindowedValue<String>> CODER =
LengthPrefixCoder.of(WindowedValue.getValueOnlyCoder(StringUtf8Coder.of()));
private static final String DEFAULT_CLIENT = "";
@@ -130,7 +129,7 @@
CloseableFnDataReceiver<WindowedValue<String>> consumer =
service
.getDataService(DEFAULT_CLIENT)
- .send(LogicalEndpoint.of(Integer.toString(i), TARGET), CODER);
+ .send(LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID), CODER);
consumer.accept(valueInGlobalWindow("A" + i));
consumer.accept(valueInGlobalWindow("B" + i));
@@ -203,7 +202,7 @@
CloseableFnDataReceiver<WindowedValue<String>> consumer =
service
.getDataService(Integer.toString(client))
- .send(LogicalEndpoint.of(instructionId, TARGET), CODER);
+ .send(LogicalEndpoint.of(instructionId, TRANSFORM_ID), CODER);
consumer.accept(valueInGlobalWindow("A" + instructionId));
consumer.accept(valueInGlobalWindow("B" + instructionId));
@@ -236,7 +235,7 @@
CountDownLatch waitForInboundElements = new CountDownLatch(1);
for (int i = 0; i < 3; ++i) {
- String instructionReference = Integer.toString(i);
+ String instructionId = Integer.toString(i);
executorService.submit(
() -> {
ManagedChannel channel =
@@ -244,7 +243,7 @@
StreamObserver<BeamFnApi.Elements> outboundObserver =
BeamFnDataGrpc.newStub(channel)
.data(TestStreams.withOnNext(clientInboundElements::add).build());
- outboundObserver.onNext(elementsWithData(instructionReference));
+ outboundObserver.onNext(elementsWithData(instructionId));
waitForInboundElements.await();
outboundObserver.onCompleted();
return null;
@@ -260,7 +259,9 @@
service
.getDataService(DEFAULT_CLIENT)
.receive(
- LogicalEndpoint.of(Integer.toString(i), TARGET), CODER, serverInboundValue::add));
+ LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID),
+ CODER,
+ serverInboundValue::add));
}
// Waiting for the client provides the necessary synchronization for the elements to arrive.
@@ -283,8 +284,8 @@
return BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(id)
- .setTarget(TARGET)
+ .setInstructionId(id)
+ .setTransformId(TRANSFORM_ID)
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("A" + id)))
.concat(
@@ -293,7 +294,8 @@
.concat(
ByteString.copyFrom(
encodeToByteArray(CODER, valueInGlobalWindow("C" + id))))))
- .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionReference(id).setTarget(TARGET))
+ .addData(
+ BeamFnApi.Elements.Data.newBuilder().setInstructionId(id).setTransformId(TRANSFORM_ID))
.build();
}
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperationTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperationTest.java
index d739801..793599c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperationTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortReadOperationTest.java
@@ -30,7 +30,6 @@
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.runners.dataflow.worker.NameContextsForTests;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OperationContext;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver;
@@ -60,8 +59,7 @@
public class RemoteGrpcPortReadOperationTest {
private static final Coder<WindowedValue<String>> CODER =
WindowedValue.getFullCoder(StringUtf8Coder.of(), GlobalWindow.Coder.INSTANCE);
- private static final BeamFnApi.Target TARGET =
- BeamFnApi.Target.newBuilder().setPrimitiveTransformReference("1").setName("name").build();
+ private static final String TRANSFORM_ID = "1";
private static final String BUNDLE_ID = "999";
private static final String BUNDLE_ID_2 = "222";
@@ -79,7 +77,7 @@
operation =
new RemoteGrpcPortReadOperation<>(
beamFnDataService,
- TARGET,
+ TRANSFORM_ID,
bundleIdSupplier,
CODER,
new OutputReceiver[] {testReceiver},
@@ -100,7 +98,8 @@
operation.start();
verify(beamFnDataService)
- .receive(eq(LogicalEndpoint.of(BUNDLE_ID, TARGET)), eq(CODER), consumerCaptor.capture());
+ .receive(
+ eq(LogicalEndpoint.of(BUNDLE_ID, TRANSFORM_ID)), eq(CODER), consumerCaptor.capture());
Future<Void> operationFinish =
Executors.newSingleThreadExecutor()
@@ -132,7 +131,8 @@
when(bundleIdSupplier.getId()).thenReturn(BUNDLE_ID_2);
operation.start();
verify(beamFnDataService)
- .receive(eq(LogicalEndpoint.of(BUNDLE_ID_2, TARGET)), eq(CODER), consumerCaptor.capture());
+ .receive(
+ eq(LogicalEndpoint.of(BUNDLE_ID_2, TRANSFORM_ID)), eq(CODER), consumerCaptor.capture());
}
@Test
@@ -144,7 +144,8 @@
operation.start();
verify(beamFnDataService)
- .receive(eq(LogicalEndpoint.of(BUNDLE_ID, TARGET)), eq(CODER), consumerCaptor.capture());
+ .receive(
+ eq(LogicalEndpoint.of(BUNDLE_ID, TRANSFORM_ID)), eq(CODER), consumerCaptor.capture());
assertFalse(inboundDataClient.isDone());
operation.abort();
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperationTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperationTest.java
index 404e718..df31a89 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperationTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/data/RemoteGrpcPortWriteOperationTest.java
@@ -31,7 +31,6 @@
import java.util.ArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.runners.dataflow.worker.util.common.worker.OperationContext;
import org.apache.beam.runners.fnexecution.data.FnDataService;
import org.apache.beam.sdk.coders.Coder;
@@ -54,8 +53,7 @@
public class RemoteGrpcPortWriteOperationTest {
private static final Coder<WindowedValue<String>> CODER =
WindowedValue.getFullCoder(StringUtf8Coder.of(), GlobalWindow.Coder.INSTANCE);
- private static final BeamFnApi.Target TARGET =
- BeamFnApi.Target.newBuilder().setPrimitiveTransformReference("1").setName("name").build();
+ private static final String TRANSFORM_ID = "1";
private static final String BUNDLE_ID = "999";
private static final String BUNDLE_ID_2 = "222";
@@ -69,7 +67,7 @@
MockitoAnnotations.initMocks(this);
operation =
new RemoteGrpcPortWriteOperation<>(
- beamFnDataService, TARGET, bundleIdSupplier, CODER, operationContext);
+ beamFnDataService, TRANSFORM_ID, bundleIdSupplier, CODER, operationContext);
}
@Test
@@ -84,7 +82,7 @@
.thenReturn(recordingConsumer);
when(bundleIdSupplier.getId()).thenReturn(BUNDLE_ID);
operation.start();
- verify(beamFnDataService).send(LogicalEndpoint.of(BUNDLE_ID, TARGET), CODER);
+ verify(beamFnDataService).send(LogicalEndpoint.of(BUNDLE_ID, TRANSFORM_ID), CODER);
assertFalse(recordingConsumer.closed);
operation.process(valueInGlobalWindow("ABC"));
@@ -106,7 +104,7 @@
when(beamFnDataService.send(any(), Matchers.<Coder<WindowedValue<String>>>any()))
.thenReturn(recordingConsumer);
operation.start();
- verify(beamFnDataService).send(LogicalEndpoint.of(BUNDLE_ID_2, TARGET), CODER);
+ verify(beamFnDataService).send(LogicalEndpoint.of(BUNDLE_ID_2, TRANSFORM_ID), CODER);
verifyNoMoreInteractions(beamFnDataService);
}
@@ -118,7 +116,7 @@
.thenReturn(recordingConsumer);
when(bundleIdSupplier.getId()).thenReturn(BUNDLE_ID);
operation.start();
- verify(beamFnDataService).send(LogicalEndpoint.of(BUNDLE_ID, TARGET), CODER);
+ verify(beamFnDataService).send(LogicalEndpoint.of(BUNDLE_ID, TRANSFORM_ID), CODER);
assertFalse(recordingConsumer.closed);
operation.process(valueInGlobalWindow("ABC"));
@@ -147,7 +145,7 @@
operation =
new RemoteGrpcPortWriteOperation<>(
beamFnDataService,
- TARGET,
+ TRANSFORM_ID,
bundleIdSupplier,
CODER,
operationContext,
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java
index 5e80335..55b81e0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/logging/BeamFnLoggingServiceTest.java
@@ -38,13 +38,13 @@
import org.apache.beam.runners.dataflow.worker.fn.stream.ServerStreamObserverFactory;
import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.BindableService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.BindableService;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
import org.junit.After;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -82,7 +82,7 @@
Collection<Callable<Void>> tasks = new ArrayList<>();
for (int i = 1; i <= 3; ++i) {
- int instructionReference = i;
+ int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForServerHangup = new CountDownLatch(1);
@@ -95,8 +95,7 @@
TestStreams.withOnNext(BeamFnLoggingServiceTest::discardMessage)
.withOnCompleted(waitForServerHangup::countDown)
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
outboundObserver.onCompleted();
waitForServerHangup.await();
return null;
@@ -128,7 +127,7 @@
GrpcContextHeaderAccessorProvider.getHeaderAccessor())) {
server = createServer(service, service.getApiServiceDescriptor());
for (int i = 1; i <= 3; ++i) {
- int instructionReference = i;
+ int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForTermination = new CountDownLatch(1);
@@ -141,9 +140,8 @@
TestStreams.withOnNext(BeamFnLoggingServiceTest::discardMessage)
.withOnError(waitForTermination::countDown)
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
- outboundObserver.onError(new RuntimeException("Client " + instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
+ outboundObserver.onError(new RuntimeException("Client " + instructionId));
waitForTermination.await();
return null;
});
@@ -167,7 +165,7 @@
server = createServer(service, service.getApiServiceDescriptor());
for (int i = 1; i <= 3; ++i) {
- long instructionReference = i;
+ long instructionId = i;
futures.add(
executorService.submit(
() -> {
@@ -181,7 +179,7 @@
TestStreams.withOnNext(BeamFnLoggingServiceTest::discardMessage)
.withOnCompleted(waitForServerHangup::countDown)
.build());
- outboundObserver.onNext(createLogsWithIds(instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId));
waitForServerHangup.await();
return null;
}));
@@ -210,7 +208,7 @@
}
private BeamFnApi.LogEntry createLogWithId(long id) {
- return BeamFnApi.LogEntry.newBuilder().setInstructionReference(Long.toString(id)).build();
+ return BeamFnApi.LogEntry.newBuilder().setInstructionId(Long.toString(id)).build();
}
private Server createServer(
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactoryTest.java
index ad8f0e3..43d6975 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/stream/ServerStreamObserverFactoryTest.java
@@ -24,8 +24,8 @@
import org.apache.beam.sdk.fn.stream.BufferingStreamObserver;
import org.apache.beam.sdk.fn.stream.DirectStreamObserver;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunctionTest.java
index c270dee..9175cd2 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CloneAmbiguousFlattensFunctionTest.java
@@ -34,10 +34,9 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.InstructionOutputNode;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -51,10 +50,8 @@
@Test
public void testEmptyNetwork() throws Exception {
- assertTrue(
- Graphs.equivalent(
- createEmptyNetwork(),
- new CloneAmbiguousFlattensFunction().apply(createEmptyNetwork())));
+ assertEquals(
+ createEmptyNetwork(), new CloneAmbiguousFlattensFunction().apply(createEmptyNetwork()));
}
/**
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunctionTest.java
index d9a5efe..9da1af1 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/CreateRegisterFnOperationFunctionTest.java
@@ -21,9 +21,9 @@
import static org.hamcrest.Matchers.everyItem;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.not;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
-import static org.mockito.Matchers.anyString;
import static org.mockito.Mockito.when;
import com.google.api.services.dataflow.model.InstructionOutput;
@@ -32,8 +32,8 @@
import com.google.api.services.dataflow.model.ReadInstruction;
import com.google.auto.value.AutoValue;
import java.util.List;
-import java.util.function.BiFunction;
import java.util.function.Function;
+import java.util.function.Supplier;
import org.apache.beam.runners.dataflow.worker.graph.Edges.DefaultEdge;
import org.apache.beam.runners.dataflow.worker.graph.Edges.Edge;
import org.apache.beam.runners.dataflow.worker.graph.Edges.HappensBeforeEdge;
@@ -41,12 +41,12 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
import org.apache.beam.sdk.fn.IdGenerators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Graphs;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.hamcrest.Matchers;
import org.junit.Before;
import org.junit.Test;
@@ -60,7 +60,7 @@
@RunWith(JUnit4.class)
public class CreateRegisterFnOperationFunctionTest {
- @Mock private BiFunction<String, String, Node> portSupplier;
+ @Mock private Supplier<Node> portSupplier;
@Mock private Function<MutableNetwork<Node, Edge>, Node> registerFnOperationFunction;
private Function<MutableNetwork<Node, Edge>, MutableNetwork<Node, Edge>>
createRegisterFnOperation;
@@ -80,9 +80,10 @@
MutableNetwork<Node, Edge> expectedNetwork = createEmptyNetwork();
assertNetworkMaintainsBipartiteStructure(appliedNetwork);
- assertTrue(
+ assertEquals(
String.format("Expected network %s but got network %s", expectedNetwork, appliedNetwork),
- Graphs.equivalent(expectedNetwork, appliedNetwork));
+ expectedNetwork,
+ appliedNetwork);
}
@Test
@@ -109,9 +110,10 @@
createRegisterFnOperation.apply(Graphs.copyOf(expectedNetwork));
assertNetworkMaintainsBipartiteStructure(appliedNetwork);
- assertTrue(
+ assertEquals(
String.format("Expected network %s but got network %s", expectedNetwork, appliedNetwork),
- Graphs.equivalent(expectedNetwork, appliedNetwork));
+ expectedNetwork,
+ appliedNetwork);
}
@Test
@@ -149,12 +151,14 @@
assertNetworkMaintainsBipartiteStructure(appliedNetwork);
assertNetworkMaintainsBipartiteStructure(networkCapture.getValue());
- assertTrue(
+ assertEquals(
String.format("Expected network %s but got network %s", expectedNetwork, appliedNetwork),
- Graphs.equivalent(expectedNetwork, appliedNetwork));
- assertTrue(
+ expectedNetwork,
+ appliedNetwork);
+ assertEquals(
String.format("Expected network %s but got network %s", network, networkCapture.getValue()),
- Graphs.equivalent(network, networkCapture.getValue()));
+ network,
+ networkCapture.getValue());
}
@Test
@@ -167,7 +171,7 @@
Node firstPort = TestNode.create("FirstPort");
Node secondPort = TestNode.create("SecondPort");
- when(portSupplier.apply(anyString(), anyString())).thenReturn(firstPort, secondPort);
+ when(portSupplier.get()).thenReturn(firstPort, secondPort);
Node readNode = createReadNode("Read", Nodes.ExecutionLocation.RUNNER_HARNESS);
Edge readNodeEdge = DefaultEdge.create();
@@ -257,7 +261,7 @@
Node firstPort = TestNode.create("FirstPort");
Node secondPort = TestNode.create("SecondPort");
- when(portSupplier.apply(anyString(), anyString())).thenReturn(firstPort, secondPort);
+ when(portSupplier.get()).thenReturn(firstPort, secondPort);
Node readNode = createReadNode("Read", Nodes.ExecutionLocation.SDK_HARNESS);
Edge readNodeEdge = DefaultEdge.create();
@@ -372,7 +376,7 @@
Node firstPort = TestNode.create("FirstPort");
Node secondPort = TestNode.create("SecondPort");
- when(portSupplier.apply(anyString(), anyString())).thenReturn(firstPort, secondPort);
+ when(portSupplier.get()).thenReturn(firstPort, secondPort);
Node runnerReadNode = createReadNode("RunnerRead", Nodes.ExecutionLocation.RUNNER_HARNESS);
Edge runnerReadNodeEdge = DefaultEdge.create();
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunctionTest.java
index 26bdc07..9ade1f2 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceFlattenLocationsFunctionTest.java
@@ -18,7 +18,6 @@
package org.apache.beam.runners.dataflow.worker.graph;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
import com.google.api.services.dataflow.model.FlattenInstruction;
import com.google.api.services.dataflow.model.InstructionOutput;
@@ -29,11 +28,10 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.InstructionOutputNode;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -47,10 +45,8 @@
@Test
public void testEmptyNetwork() throws Exception {
- assertTrue(
- Graphs.equivalent(
- createEmptyNetwork(),
- new DeduceFlattenLocationsFunction().apply(createEmptyNetwork())));
+ assertEquals(
+ createEmptyNetwork(), new DeduceFlattenLocationsFunction().apply(createEmptyNetwork()));
}
/*
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunctionTest.java
index 63c971b..a1824d6 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/DeduceNodeLocationsFunctionTest.java
@@ -38,13 +38,12 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.InstructionOutputNode;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Equivalence;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ImmutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ImmutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -82,9 +81,8 @@
@Test
public void testEmptyNetwork() {
- assertTrue(
- Graphs.equivalent(
- createEmptyNetwork(), new DeduceNodeLocationsFunction().apply(createEmptyNetwork())));
+ assertEquals(
+ createEmptyNetwork(), new DeduceNodeLocationsFunction().apply(createEmptyNetwork()));
}
@Test
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodesTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodesTest.java
index bba8cbb..5cf303c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodesTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/InsertFetchAndFilterStreamingSideInputNodesTest.java
@@ -54,15 +54,15 @@
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Equivalence;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Equivalence.Wrapper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ImmutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Equivalence.Wrapper;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ImmutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java
index 679a292..ebe0d4e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java
@@ -57,10 +57,10 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunctionTest.java
index b3ced60..09d2205 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/MapTaskToNetworkFunctionTest.java
@@ -46,9 +46,9 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.fn.IdGenerators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NetworksTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NetworksTest.java
index 79d92ba..e3c8c7b 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NetworksTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NetworksTest.java
@@ -35,11 +35,11 @@
import java.util.function.Function;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NodesTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NodesTest.java
index 5a71d6d..3888103 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NodesTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/NodesTest.java
@@ -48,8 +48,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunctionTest.java
index c309b46..a5724ae 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/RemoveFlattenInstructionsFunctionTest.java
@@ -18,9 +18,9 @@
package org.apache.beam.runners.dataflow.worker.graph;
import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertThat;
-import static org.junit.Assert.assertTrue;
import com.google.api.services.dataflow.model.FlattenInstruction;
import com.google.api.services.dataflow.model.InstructionOutput;
@@ -34,11 +34,10 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.InstructionOutputNode;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Graphs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ImmutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ImmutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -50,10 +49,8 @@
@Test
public void testEmptyNetwork() {
- assertTrue(
- Graphs.equivalent(
- createEmptyNetwork(),
- new RemoveFlattenInstructionsFunction().apply(createEmptyNetwork())));
+ assertEquals(
+ createEmptyNetwork(), new RemoveFlattenInstructionsFunction().apply(createEmptyNetwork()));
}
@Test
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunctionTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunctionTest.java
index 81b568c..37733c2 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunctionTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/ReplacePgbkWithPrecombineFunctionTest.java
@@ -36,10 +36,10 @@
import org.apache.beam.runners.dataflow.worker.graph.Nodes.InstructionOutputNode;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.Node;
import org.apache.beam.runners.dataflow.worker.graph.Nodes.ParallelInstructionNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.ImmutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.MutableNetwork;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.Network;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.graph.NetworkBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.ImmutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.MutableNetwork;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.NetworkBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java
index e48291b..568fcff 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandlerTest.java
@@ -35,8 +35,8 @@
import org.apache.beam.runners.dataflow.worker.NameContextsForTests;
import org.apache.beam.runners.dataflow.worker.TestOperationContext.TestDataflowExecutionState;
import org.apache.beam.runners.dataflow.worker.testing.RestoreDataflowLoggingMDC;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Timestamp;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Timestamp;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
@@ -348,7 +348,7 @@
.setLogLocation("LoggerName")
.setSeverity(BeamFnApi.LogEntry.Severity.Enum.INFO)
.setMessage(message)
- .setInstructionReference("1")
+ .setInstructionId("1")
.setThread("2")
.setTimestamp(Timestamp.newBuilder().setSeconds(0).setNanos(1 * 1000000))
.build();
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializerTest.java
index 8b6164f..4d04277 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingInitializerTest.java
@@ -47,7 +47,7 @@
import org.apache.beam.runners.dataflow.options.DataflowWorkerLoggingOptions.WorkerLogLevelOverrides;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactoryTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactoryTest.java
index 349c6af..65f5128 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactoryTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/logging/JulHandlerPrintStreamAdapterFactoryTest.java
@@ -27,7 +27,7 @@
import java.util.logging.Level;
import java.util.logging.LogRecord;
import org.apache.beam.runners.dataflow.worker.LogSaver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfilerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfilerTest.java
index 7e46e27..c2aefc7 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfilerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/profiler/ScopedProfilerTest.java
@@ -23,7 +23,7 @@
import java.util.HashMap;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfileScope;
import org.apache.beam.runners.dataflow.worker.profiler.ScopedProfiler.ProfilerWrapper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServletTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServletTest.java
index e5546d6..5b8de01 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServletTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/ThreadzServletTest.java
@@ -24,7 +24,7 @@
import java.util.Arrays;
import java.util.List;
import java.util.Map;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPagesTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPagesTest.java
index 97becef..8373bea 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPagesTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/status/WorkerStatusPagesTest.java
@@ -20,6 +20,7 @@
import static org.hamcrest.Matchers.containsString;
import static org.junit.Assert.assertThat;
+import java.util.function.BooleanSupplier;
import org.apache.beam.runners.dataflow.worker.util.MemoryMonitor;
import org.eclipse.jetty.server.LocalConnector;
import org.eclipse.jetty.server.Server;
@@ -38,12 +39,13 @@
private final Server server = new Server();
private final LocalConnector connector = new LocalConnector(server);
@Mock private MemoryMonitor mockMemoryMonitor;
+ private final BooleanSupplier mockHealthyIndicator = () -> true;
private WorkerStatusPages wsp;
@Before
public void setUp() throws Exception {
MockitoAnnotations.initMocks(this);
- wsp = new WorkerStatusPages(server, mockMemoryMonitor);
+ wsp = new WorkerStatusPages(server, mockMemoryMonitor, mockHealthyIndicator);
server.addConnector(connector);
wsp.start();
}
@@ -64,13 +66,25 @@
}
@Test
- public void testHealthz() throws Exception {
- String response = getPage("/threadz");
+ public void testHealthzHealthy() throws Exception {
+ String response = getPage("/healthz");
assertThat(response, containsString("HTTP/1.1 200 OK"));
assertThat(response, containsString("ok"));
}
@Test
+ public void testHealthzUnhealthy() throws Exception {
+ // set up WorkerStatusPages that respond unhealthy status on "healthz"
+ wsp.stop();
+ wsp = new WorkerStatusPages(server, mockMemoryMonitor, () -> false);
+ wsp.start();
+
+ String response = getPage("/healthz");
+ assertThat(response, containsString("HTTP/1.1 500 Server Error"));
+ assertThat(response, containsString("internal server error"));
+ }
+
+ @Test
public void testUnknownHandler() throws Exception {
String response = getPage("/missinghandlerz");
assertThat(response, containsString("HTTP/1.1 302 Found"));
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/CounterHamcrestMatchers.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/CounterHamcrestMatchers.java
index f0dda05..4da654f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/CounterHamcrestMatchers.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/CounterHamcrestMatchers.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.dataflow.worker.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.dataflow.model.CounterMetadata;
import com.google.api.services.dataflow.model.CounterStructuredName;
@@ -31,7 +31,7 @@
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory;
import org.apache.beam.runners.dataflow.worker.counters.CounterName;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/DataflowCounterUpdateExtractorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/DataflowCounterUpdateExtractorTest.java
index e56b74a..fb1f641 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/DataflowCounterUpdateExtractorTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/DataflowCounterUpdateExtractorTest.java
@@ -44,7 +44,7 @@
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
import org.apache.beam.runners.dataflow.worker.counters.DataflowCounterUpdateExtractor;
import org.apache.beam.runners.dataflow.worker.util.common.worker.ShuffleReadCounter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/GroupAlsoByWindowProperties.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/GroupAlsoByWindowProperties.java
index 2114ffe..9badb6d 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/GroupAlsoByWindowProperties.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/GroupAlsoByWindowProperties.java
@@ -48,11 +48,11 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/ListOutputManager.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/ListOutputManager.java
index b094c17..4d4188c 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/ListOutputManager.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/ListOutputManager.java
@@ -23,8 +23,8 @@
import org.apache.beam.runners.core.DoFnRunners.OutputManager;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* An implementation of {@code OutputManager} using simple lists, for testing and in-memory contexts
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutorTestUtils.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutorTestUtils.java
index 071cb25..bc9a8b8 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutorTestUtils.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutorTestUtils.java
@@ -29,7 +29,7 @@
import org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.ElementByteSizeObservableCoder;
import org.apache.beam.runners.dataflow.worker.counters.CounterSet;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Utilities for tests. */
@SuppressWarnings({"rawtypes", "unchecked"})
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIteratorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIteratorTest.java
index c193733..4591ca0 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIteratorTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/GroupingShuffleEntryIteratorTest.java
@@ -43,8 +43,8 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.common.Reiterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutorTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutorTest.java
index ad0b862..a40296e 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutorTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/MapTaskExecutorTest.java
@@ -62,7 +62,7 @@
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperationTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperationTest.java
index d7c9567..db360d3 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperationTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ReadOperationTest.java
@@ -65,7 +65,7 @@
import org.apache.beam.runners.dataflow.worker.util.common.worker.ExecutorTestUtils.TestReader;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.range.OffsetRangeTracker;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -125,7 +125,7 @@
executor = mock(ScheduledExecutorService.class);
when(executor.scheduleAtFixedRate(
any(Runnable.class), anyLong(), anyLong(), any(TimeUnit.class)))
- .then(invocation -> schedule(invocation.getArgumentAt(0, Runnable.class)));
+ .then(invocation -> schedule(invocation.getArgument(0, Runnable.class)));
}
public ScheduledExecutorService getExecutor() {
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/TestOutputReceiver.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/TestOutputReceiver.java
index dea8393..afd86aa 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/TestOutputReceiver.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/TestOutputReceiver.java
@@ -27,7 +27,7 @@
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** An OutputReceiver that allows the output elements to be retrieved. */
public class TestOutputReceiver extends OutputReceiver {
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdaterTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdaterTest.java
index c7d4afd..47b661f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdaterTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/common/worker/WorkProgressUpdaterTest.java
@@ -165,8 +165,7 @@
// Set the initial lease expiration to 20s so that the first update occurs at 10s, ie before
// the periodic checkpoint.
initialLeaseExpirationMs = clock.currentTimeMillis() + 20 * 1000L;
- when(progressHelper.reportProgress(any(NativeReader.DynamicSplitResult.class)))
- .thenReturn(4 * 1000L); // Next update at 14s.
+ when(progressHelper.reportProgress(null)).thenReturn(4 * 1000L); // Next update at 14s.
progressUpdater.startReportingProgress();
executor.runNextRunnable();
@@ -177,8 +176,7 @@
verify(progressHelper, times(1)).reportProgress(null);
verify(progressHelper, never()).reportProgress(checkpointPos);
- when(progressHelper.reportProgress(any(NativeReader.DynamicSplitResult.class)))
- .thenReturn(4 * 1000L); // Next update at 18s.
+ when(progressHelper.reportProgress(null)).thenReturn(4 * 1000L); // Next update at 18s.
executor.runNextRunnable();
@@ -188,8 +186,7 @@
verify(progressHelper, times(2)).reportProgress(null);
verify(progressHelper, never()).reportProgress(checkpointPos);
- when(progressHelper.reportProgress(any(NativeReader.DynamicSplitResult.class)))
- .thenReturn(4 * 1000L); // Next update at 22s.
+ when(progressHelper.reportProgress(null)).thenReturn(4 * 1000L); // Next update at 22s.
executor.runNextRunnable();
@@ -253,8 +250,7 @@
// the periodic checkpoint.
// Do one update.
initialLeaseExpirationMs = clock.currentTimeMillis() + 20 * 1000L;
- when(progressHelper.reportProgress(any(NativeReader.DynamicSplitResult.class)))
- .thenReturn(4 * 1000L); // Next update at 14s.
+ when(progressHelper.reportProgress(null)).thenReturn(4 * 1000L); // Next update at 14s.
progressUpdater.startReportingProgress();
executor.runNextRunnable();
diff --git a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServerTest.java b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServerTest.java
index a8a2022..9adce9f 100644
--- a/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServerTest.java
+++ b/runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServerTest.java
@@ -59,13 +59,13 @@
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.CommitWorkStream;
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.GetDataStream;
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.GetWorkStream;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.util.MutableHandlerRegistry;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.util.MutableHandlerRegistry;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.After;
@@ -87,7 +87,7 @@
@Rule public ErrorCollector errorCollector = new ErrorCollector();
private Server server;
private GrpcWindmillServer client;
- private static final int STREAM_CHUNK_SIZE = 63 * 1024;
+ private static final int STREAM_CHUNK_SIZE = 2 << 20;
private int remainingErrors = 20;
@Before
diff --git a/runners/google-cloud-dataflow-java/worker/windmill/build.gradle b/runners/google-cloud-dataflow-java/worker/windmill/build.gradle
index da0319e..53a2882 100644
--- a/runners/google-cloud-dataflow-java/worker/windmill/build.gradle
+++ b/runners/google-cloud-dataflow-java/worker/windmill/build.gradle
@@ -17,8 +17,11 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-runners-google-cloud-dataflow-java-windmill'
-applyPortabilityNature(shadowJarValidationExcludes: ["org/apache/beam/runners/dataflow/worker/windmill/**"])
+applyPortabilityNature(
+ publish: false,
+ shadowJarValidationExcludes: ["org/apache/beam/runners/dataflow/worker/windmill/**"],
+ archivesBaseName: 'beam-runners-google-cloud-dataflow-java-windmill'
+)
description = "Apache Beam :: Runners :: Google Cloud Dataflow Java :: Windmill"
ext.summary = "Windmill proto specifications"
diff --git a/runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto b/runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto
index 29c664d..5310902 100644
--- a/runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto
+++ b/runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto
@@ -134,6 +134,12 @@
optional string state_family = 4;
}
+// Proto describing a hot key detected on a given WorkItem.
+message HotKeyInfo {
+ // The age of the hot key measured from when it was first detected.
+ optional int64 hot_key_age_usec = 1;
+}
+
message WorkItem {
required bytes key = 1;
required fixed64 work_token = 2;
@@ -149,6 +155,11 @@
// Indicates that this is a new key with no data associated. This allows
// the harness to optimize data fetching.
optional bool is_new_key = 10;
+
+ // A hot key is a symptom of poor data distribution in which there are enough
+ // elements mapped to a single key to impact pipeline performance. When
+ // present, this field includes metadata associated with any hot key.
+ optional HotKeyInfo hot_key_info = 11;
}
message ComputationWorkItems {
diff --git a/runners/java-fn-execution/build.gradle b/runners/java-fn-execution/build.gradle
index 1d1ccba..f032d8f 100644
--- a/runners/java-fn-execution/build.gradle
+++ b/runners/java-fn-execution/build.gradle
@@ -16,26 +16,26 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.fnexecution')
description = "Apache Beam :: Runners :: Java Fn Execution"
dependencies {
- shadow library.java.vendored_guava_20_0
- compile project(path: ":runners:core-construction-java", configuration: "shadow")
- provided project(path: ":sdks:java:harness")
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":model:fn-execution", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:fn-execution", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":vendor:sdks-java-extensions-protobuf", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.slf4j_api
- shadow library.java.args4j
+ compile library.java.vendored_guava_26_0_jre
+ compile project(":runners:core-construction-java")
+ provided project(":sdks:java:harness")
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":model:fn-execution", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:fn-execution")
+ compile project(":runners:core-construction-java")
+ compile project(path: ":vendor:sdks-java-extensions-protobuf", configuration: "shadow")
+ compile library.java.vendored_grpc_1_21_0
+ compile library.java.slf4j_api
+ compile library.java.args4j
testCompile project(":sdks:java:harness")
- testCompile project(path: ":runners:core-construction-java", configuration: "shadow")
- testCompile project(path: ":runners:core-java", configuration: "shadowTest")
+ testCompile project(":runners:core-construction-java")
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.junit
testCompile library.java.hamcrest_core
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/FnService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/FnService.java
index 393df84..3055b0b 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/FnService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/FnService.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.BindableService;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.BindableService;
/** An interface sharing common behavior with services used during execution of user Fns. */
public interface FnService extends AutoCloseable, BindableService {
@@ -26,8 +26,8 @@
*
* <p>There should be no more calls to any service method by the time a call to {@link #close()}
* begins. Specifically, this means that a {@link
- * org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server} that this service is bound to should have
- * completed a call to the {@link org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server#shutdown()}
+ * org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server} that this service is bound to should have
+ * completed a call to the {@link org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server#shutdown()}
* method, and all future incoming calls will be rejected.
*/
@Override
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProvider.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProvider.java
index 6014c06..5d758a2 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProvider.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProvider.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.runners.fnexecution;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Context;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata.Key;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerCall;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerCall.Listener;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerCallHandler;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerInterceptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Context;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata.Key;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerCall;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerCall.Listener;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerCallHandler;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerInterceptor;
/**
* A HeaderAccessorProvider which intercept the header in a GRPC request and expose the relevant
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcFnServer.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcFnServer.java
index fb7fcd6..f7a4a4b 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcFnServer.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/GrpcFnServer.java
@@ -20,8 +20,8 @@
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link Server gRPC Server} which manages a single {@link FnService}. The lifetime of the
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/InProcessServerFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/InProcessServerFactory.java
index 967f8fc..a899cb2 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/InProcessServerFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/InProcessServerFactory.java
@@ -21,10 +21,10 @@
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.BindableService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerInterceptors;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.BindableService;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerInterceptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
/**
* A {@link ServerFactory} which creates {@link Server servers} with the {@link
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
index 298f054..ff0d5b4 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.File;
import java.io.IOException;
@@ -29,17 +29,17 @@
import java.util.function.Supplier;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.sdk.fn.channel.SocketAddressFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.BindableService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerInterceptors;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.netty.NettyServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.EpollEventLoopGroup;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.EpollServerDomainSocketChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.EpollServerSocketChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.unix.DomainSocketAddress;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.util.internal.ThreadLocalRandom;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.BindableService;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerInterceptors;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NettyServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.EpollEventLoopGroup;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.EpollServerDomainSocketChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.EpollServerSocketChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.unix.DomainSocketAddress;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.util.internal.ThreadLocalRandom;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
/** A {@link Server gRPC server} factory. */
public abstract class ServerFactory {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java
new file mode 100644
index 0000000..93ae657
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An {@link ArtifactRetrievalService} that handles everything aside from actually opening the
+ * backing resources.
+ */
+public abstract class AbstractArtifactRetrievalService
+ extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+ implements ArtifactRetrievalService {
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractArtifactRetrievalService.class);
+
+ private static final int ARTIFACT_CHUNK_SIZE_BYTES = 2 << 20; // 2MB
+
+ public AbstractArtifactRetrievalService() {
+ this(
+ CacheBuilder.newBuilder()
+ .expireAfterAccess(1, TimeUnit.HOURS /* arbitrary */)
+ .maximumSize(100 /* arbitrary */)
+ .build());
+ }
+
+ public AbstractArtifactRetrievalService(Cache<String, ArtifactApi.ProxyManifest> manifestCache) {
+ this.manifestCache = manifestCache;
+ }
+
+ public abstract InputStream openManifest(String retrievalToken) throws IOException;
+
+ public abstract InputStream openUri(String retrievalToken, String uri) throws IOException;
+
+ private final Cache<String, ArtifactApi.ProxyManifest> manifestCache;
+
+ public ArtifactApi.ProxyManifest getManifestProxy(String retrievalToken)
+ throws IOException, ExecutionException {
+ return manifestCache.get(
+ retrievalToken,
+ () -> {
+ try (InputStream stream = openManifest(retrievalToken)) {
+ return loadManifest(stream, retrievalToken);
+ }
+ });
+ }
+
+ @Override
+ public void getManifest(
+ ArtifactApi.GetManifestRequest request,
+ StreamObserver<ArtifactApi.GetManifestResponse> responseObserver) {
+ final String token = request.getRetrievalToken();
+ if (Strings.isNullOrEmpty(token)) {
+ throw new StatusRuntimeException(
+ Status.INVALID_ARGUMENT.withDescription("Empty artifact token"));
+ }
+
+ LOG.info("GetManifest for {}", token);
+ try {
+ ArtifactApi.ProxyManifest proxyManifest = getManifestProxy(token);
+ ArtifactApi.GetManifestResponse response =
+ ArtifactApi.GetManifestResponse.newBuilder()
+ .setManifest(proxyManifest.getManifest())
+ .build();
+ LOG.info(
+ "GetManifest for {} -> {} artifacts",
+ token,
+ proxyManifest.getManifest().getArtifactCount());
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ } catch (Exception e) {
+ LOG.warn("GetManifest for {} failed.", token, e);
+ responseObserver.onError(e);
+ }
+ }
+
+ @Override
+ public void getArtifact(
+ ArtifactApi.GetArtifactRequest request,
+ StreamObserver<ArtifactApi.ArtifactChunk> responseObserver) {
+ LOG.debug("GetArtifact {}", request);
+ String name = request.getName();
+ try {
+ ArtifactApi.ProxyManifest proxyManifest = getManifestProxy(request.getRetrievalToken());
+ // look for file at URI specified by proxy manifest location
+ ArtifactApi.ProxyManifest.Location location =
+ proxyManifest.getLocationList().stream()
+ .filter(loc -> loc.getName().equals(name))
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new StatusRuntimeException(
+ Status.NOT_FOUND.withDescription(
+ String.format("Artifact location not found in manifest: %s", name))));
+
+ List<ArtifactMetadata> existingArtifacts = proxyManifest.getManifest().getArtifactList();
+ ArtifactMetadata metadata =
+ existingArtifacts.stream()
+ .filter(meta -> meta.getName().equals(name))
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new StatusRuntimeException(
+ Status.NOT_FOUND.withDescription(
+ String.format("Artifact metadata not found in manifest: %s", name))));
+
+ Hasher hasher = Hashing.sha256().newHasher();
+ byte[] data = new byte[ARTIFACT_CHUNK_SIZE_BYTES];
+ try (InputStream stream = openUri(request.getRetrievalToken(), location.getUri())) {
+ int len;
+ while ((len = stream.read(data)) != -1) {
+ hasher.putBytes(data, 0, len);
+ responseObserver.onNext(
+ ArtifactApi.ArtifactChunk.newBuilder()
+ .setData(ByteString.copyFrom(data, 0, len))
+ .build());
+ }
+ }
+ if (metadata.getSha256() != null && !metadata.getSha256().isEmpty()) {
+ String expected = metadata.getSha256();
+ String actual = hasher.hash().toString();
+ if (!actual.equals(expected)) {
+ throw new StatusRuntimeException(
+ Status.DATA_LOSS.withDescription(
+ String.format(
+ "Artifact %s is corrupt: expected sha256 %s, actual %s",
+ name, expected, actual)));
+ }
+ }
+ responseObserver.onCompleted();
+ } catch (IOException | ExecutionException e) {
+ LOG.info("GetArtifact {} failed", request, e);
+ responseObserver.onError(e);
+ }
+ }
+
+ @Override
+ public void close() throws Exception {}
+
+ static ProxyManifest loadManifest(InputStream stream, String manifestName) throws IOException {
+ ProxyManifest.Builder manifestBuilder = ProxyManifest.newBuilder();
+ String contents = new String(ByteStreams.toByteArray(stream), StandardCharsets.UTF_8);
+ JsonFormat.parser().merge(contents, manifestBuilder);
+ ProxyManifest proxyManifest = manifestBuilder.build();
+ checkArgument(
+ proxyManifest.hasManifest(),
+ String.format("Invalid ProxyManifest at %s: doesn't have a Manifest", manifestName));
+ checkArgument(
+ proxyManifest.getLocationCount() == proxyManifest.getManifest().getArtifactCount(),
+ String.format(
+ "Invalid ProxyManifestat %s: %d locations but %d artifacts",
+ manifestName,
+ proxyManifest.getLocationCount(),
+ proxyManifest.getManifest().getArtifactCount()));
+ LOG.info(
+ "Manifest at {} has {} artifact locations",
+ manifestName,
+ proxyManifest.getManifest().getArtifactCount());
+ return proxyManifest;
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java
new file mode 100644
index 0000000..25f09a3
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import java.io.IOException;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hasher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * An {@link ArtifactStagingServiceImplBase} that handles everything aside from actually opening the
+ * backing resources.
+ */
+public abstract class AbstractArtifactStagingService extends ArtifactStagingServiceImplBase
+ implements FnService {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AbstractArtifactStagingService.class);
+
+ private static final Charset CHARSET = StandardCharsets.UTF_8;
+
+ public abstract String getArtifactUri(String stagingSessionToken, String encodedFileName)
+ throws Exception;
+
+ public abstract WritableByteChannel openUri(String uri) throws IOException;
+
+ public abstract void removeUri(String uri) throws IOException;
+
+ public abstract WritableByteChannel openManifest(String stagingSessionToken) throws Exception;
+
+ public abstract void removeArtifacts(String stagingSessionToken) throws Exception;
+
+ public abstract String getRetrievalToken(String stagingSessionToken) throws Exception;
+
+ @Override
+ public StreamObserver<PutArtifactRequest> putArtifact(
+ StreamObserver<PutArtifactResponse> responseObserver) {
+ return new PutArtifactStreamObserver(responseObserver);
+ }
+
+ @Override
+ public void commitManifest(
+ CommitManifestRequest request, StreamObserver<CommitManifestResponse> responseObserver) {
+ try {
+ String stagingSessionToken = request.getStagingSessionToken();
+ ProxyManifest.Builder proxyManifestBuilder =
+ ProxyManifest.newBuilder().setManifest(request.getManifest());
+ for (ArtifactMetadata artifactMetadata : request.getManifest().getArtifactList()) {
+ proxyManifestBuilder.addLocation(
+ Location.newBuilder()
+ .setName(artifactMetadata.getName())
+ .setUri(getArtifactUri(stagingSessionToken, encodedFileName(artifactMetadata)))
+ .build());
+ }
+ try (WritableByteChannel manifestWritableByteChannel = openManifest(stagingSessionToken)) {
+ manifestWritableByteChannel.write(
+ CHARSET.encode(JsonFormat.printer().print(proxyManifestBuilder.build())));
+ }
+ // TODO: Validate integrity of staged files.
+ responseObserver.onNext(
+ CommitManifestResponse.newBuilder()
+ .setRetrievalToken(getRetrievalToken(stagingSessionToken))
+ .build());
+ responseObserver.onCompleted();
+ } catch (Exception e) {
+ // TODO: Cleanup all the artifacts.
+ LOG.error("Unable to commit manifest.", e);
+ responseObserver.onError(e);
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ // Nothing to close here.
+ }
+
+ private String encodedFileName(ArtifactMetadata artifactMetadata) {
+ return "artifact_"
+ + Hashing.sha256().hashString(artifactMetadata.getName(), CHARSET).toString();
+ }
+
+ private class PutArtifactStreamObserver implements StreamObserver<PutArtifactRequest> {
+
+ private final StreamObserver<PutArtifactResponse> outboundObserver;
+ private PutArtifactMetadata metadata;
+ private String artifactId;
+ private WritableByteChannel artifactWritableByteChannel;
+ private Hasher hasher;
+
+ PutArtifactStreamObserver(StreamObserver<PutArtifactResponse> outboundObserver) {
+ this.outboundObserver = outboundObserver;
+ }
+
+ @Override
+ public void onNext(PutArtifactRequest putArtifactRequest) {
+ // Create the directory structure for storing artifacts in the first call.
+ if (metadata == null) {
+ checkNotNull(putArtifactRequest);
+ checkNotNull(putArtifactRequest.getMetadata());
+ metadata = putArtifactRequest.getMetadata();
+ LOG.debug("stored metadata: {}", metadata);
+ // Check the base path exists or create the base path
+ try {
+ artifactId =
+ getArtifactUri(
+ putArtifactRequest.getMetadata().getStagingSessionToken(),
+ encodedFileName(metadata.getMetadata()));
+ LOG.debug(
+ "Going to stage artifact {} to {}.", metadata.getMetadata().getName(), artifactId);
+ artifactWritableByteChannel = openUri(artifactId);
+ hasher = Hashing.sha256().newHasher();
+ } catch (Exception e) {
+ String message =
+ String.format(
+ "Failed to begin staging artifact %s", metadata.getMetadata().getName());
+ LOG.error(message, e);
+ outboundObserver.onError(
+ new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
+ }
+ } else {
+ try {
+ ByteString data = putArtifactRequest.getData().getData();
+ artifactWritableByteChannel.write(data.asReadOnlyByteBuffer());
+ hasher.putBytes(data.toByteArray());
+ } catch (IOException e) {
+ String message =
+ String.format(
+ "Failed to write chunk of artifact %s to %s",
+ metadata.getMetadata().getName(), artifactId);
+ LOG.error(message, e);
+ outboundObserver.onError(
+ new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
+ }
+ }
+ }
+
+ @Override
+ public void onError(Throwable throwable) {
+ // Delete the artifact.
+ LOG.error("Staging artifact failed for " + artifactId, throwable);
+ try {
+ if (artifactWritableByteChannel != null) {
+ artifactWritableByteChannel.close();
+ }
+ if (artifactId != null) {
+ removeUri(artifactId);
+ }
+
+ } catch (IOException e) {
+ outboundObserver.onError(
+ new StatusRuntimeException(
+ Status.DATA_LOSS.withDescription(
+ String.format("Failed to clean up artifact file %s", artifactId))));
+ return;
+ }
+ outboundObserver.onError(
+ new StatusRuntimeException(
+ Status.DATA_LOSS
+ .withDescription(String.format("Failed to stage artifact %s", artifactId))
+ .withCause(throwable)));
+ }
+
+ @Override
+ public void onCompleted() {
+ // Close the stream.
+ LOG.debug("Staging artifact completed for " + artifactId);
+ if (artifactWritableByteChannel != null) {
+ try {
+ artifactWritableByteChannel.close();
+ } catch (IOException e) {
+ onError(e);
+ return;
+ }
+ }
+ String expectedSha256 = metadata.getMetadata().getSha256();
+ if (expectedSha256 != null && !expectedSha256.isEmpty()) {
+ String actualSha256 = hasher.hash().toString();
+ if (!actualSha256.equals(expectedSha256)) {
+ outboundObserver.onError(
+ new StatusRuntimeException(
+ Status.INVALID_ARGUMENT.withDescription(
+ String.format(
+ "Artifact %s is corrupt: expected sah256 %s, but has sha256 %s",
+ metadata.getMetadata().getName(), expectedSha256, actualSha256))));
+ return;
+ }
+ }
+ outboundObserver.onNext(PutArtifactResponse.newBuilder().build());
+ outboundObserver.onCompleted();
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
index 0d83727..14a123e 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
@@ -17,34 +17,17 @@
*/
package org.apache.beam.runners.fnexecution.artifact;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-
import java.io.IOException;
import java.io.InputStream;
import java.nio.channels.Channels;
-import java.nio.charset.StandardCharsets;
-import java.util.List;
-import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.JsonFormat;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -53,125 +36,45 @@
* the artifact layout and retrieval token format produced by {@link
* BeamFileSystemArtifactStagingService}.
*/
-public class BeamFileSystemArtifactRetrievalService
- extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
- implements ArtifactRetrievalService {
+public class BeamFileSystemArtifactRetrievalService extends AbstractArtifactRetrievalService {
private static final Logger LOG =
LoggerFactory.getLogger(BeamFileSystemArtifactRetrievalService.class);
- private static final int ARTIFACT_CHUNK_SIZE_BYTES = 2 << 20; // 2MB
+ private static final Cache<String, ArtifactApi.ProxyManifest> MANIFEST_CACHE =
+ CacheBuilder.newBuilder()
+ .expireAfterAccess(1, TimeUnit.HOURS /* arbitrary */)
+ .maximumSize(100 /* arbitrary */)
+ .build();
+
+ public BeamFileSystemArtifactRetrievalService() {
+ super(MANIFEST_CACHE);
+ }
public static BeamFileSystemArtifactRetrievalService create() {
return new BeamFileSystemArtifactRetrievalService();
}
@Override
- public void getManifest(
- ArtifactApi.GetManifestRequest request,
- StreamObserver<ArtifactApi.GetManifestResponse> responseObserver) {
- final String token = request.getRetrievalToken();
- if (Strings.isNullOrEmpty(token)) {
- throw new StatusRuntimeException(
- Status.INVALID_ARGUMENT.withDescription("Empty artifact token"));
- }
-
- LOG.info("GetManifest for {}", token);
- try {
- ArtifactApi.ProxyManifest proxyManifest = MANIFEST_CACHE.get(token);
- ArtifactApi.GetManifestResponse response =
- ArtifactApi.GetManifestResponse.newBuilder()
- .setManifest(proxyManifest.getManifest())
- .build();
- LOG.info(
- "GetManifest for {} -> {} artifacts",
- token,
- proxyManifest.getManifest().getArtifactCount());
- responseObserver.onNext(response);
- responseObserver.onCompleted();
- } catch (Exception e) {
- LOG.info("GetManifest for {} failed", token, e);
- responseObserver.onError(e);
- }
+ public InputStream openUri(String retrievalToken, String uri) throws IOException {
+ ResourceId artifactResourceId = FileSystems.matchNewResource(uri, false /* is directory */);
+ return Channels.newInputStream(FileSystems.open(artifactResourceId));
}
@Override
- public void getArtifact(
- ArtifactApi.GetArtifactRequest request,
- StreamObserver<ArtifactApi.ArtifactChunk> responseObserver) {
- LOG.debug("GetArtifact {}", request);
- String name = request.getName();
+ public InputStream openManifest(String retrievalToken) throws IOException {
+ ResourceId manifestResourceId = getManifestLocationFromToken(retrievalToken);
try {
- ArtifactApi.ProxyManifest proxyManifest = MANIFEST_CACHE.get(request.getRetrievalToken());
- // look for file at URI specified by proxy manifest location
- ArtifactApi.ProxyManifest.Location location =
- proxyManifest.getLocationList().stream()
- .filter(loc -> loc.getName().equals(name))
- .findFirst()
- .orElseThrow(
- () ->
- new StatusRuntimeException(
- Status.NOT_FOUND.withDescription(
- String.format("Artifact location not found in manifest: %s", name))));
-
- List<ArtifactMetadata> existingArtifacts = proxyManifest.getManifest().getArtifactList();
- ArtifactMetadata metadata =
- existingArtifacts.stream()
- .filter(meta -> meta.getName().equals(name))
- .findFirst()
- .orElseThrow(
- () ->
- new StatusRuntimeException(
- Status.NOT_FOUND.withDescription(
- String.format("Artifact metadata not found in manifest: %s", name))));
-
- ResourceId artifactResourceId =
- FileSystems.matchNewResource(location.getUri(), false /* is directory */);
- LOG.debug("Artifact {} located in {}", name, artifactResourceId);
- Hasher hasher = Hashing.sha256().newHasher();
- byte[] data = new byte[ARTIFACT_CHUNK_SIZE_BYTES];
- try (InputStream stream = Channels.newInputStream(FileSystems.open(artifactResourceId))) {
- int len;
- while ((len = stream.read(data)) != -1) {
- hasher.putBytes(data, 0, len);
- responseObserver.onNext(
- ArtifactApi.ArtifactChunk.newBuilder()
- .setData(ByteString.copyFrom(data, 0, len))
- .build());
- }
- }
- if (metadata.getSha256() != null && !metadata.getSha256().isEmpty()) {
- String expected = metadata.getSha256();
- String actual = hasher.hash().toString();
- if (!actual.equals(expected)) {
- throw new StatusRuntimeException(
- Status.DATA_LOSS.withDescription(
- String.format(
- "Artifact %s is corrupt: expected sha256 %s, actual %s",
- name, expected, actual)));
- }
- }
- responseObserver.onCompleted();
- } catch (IOException | ExecutionException e) {
- LOG.info("GetArtifact {} failed", request, e);
- responseObserver.onError(e);
+ return Channels.newInputStream(FileSystems.open(manifestResourceId));
+ } catch (IOException e) {
+ LOG.warn(
+ "GetManifest for {} failed. Make sure the artifact staging directory (configurable "
+ + "via --artifacts-dir argument to the job server) is accessible to workers.",
+ retrievalToken,
+ e);
+ throw e;
}
}
- @Override
- public void close() throws Exception {}
-
- private static final LoadingCache<String, ArtifactApi.ProxyManifest> MANIFEST_CACHE =
- CacheBuilder.newBuilder()
- .expireAfterAccess(1, TimeUnit.HOURS /* arbitrary */)
- .maximumSize(100 /* arbitrary */)
- .build(
- new CacheLoader<String, ProxyManifest>() {
- @Override
- public ProxyManifest load(String retrievalToken) throws Exception {
- return loadManifest(retrievalToken);
- }
- });
-
@VisibleForTesting
static ProxyManifest loadManifest(String retrievalToken) throws IOException {
LOG.info("Loading manifest for retrieval token {}", retrievalToken);
@@ -181,27 +84,9 @@
}
static ProxyManifest loadManifest(ResourceId manifestResourceId) throws IOException {
- ProxyManifest.Builder manifestBuilder = ProxyManifest.newBuilder();
- try (InputStream stream = Channels.newInputStream(FileSystems.open(manifestResourceId))) {
- String contents = new String(ByteStreams.toByteArray(stream), StandardCharsets.UTF_8);
- JsonFormat.parser().merge(contents, manifestBuilder);
- }
- ProxyManifest proxyManifest = manifestBuilder.build();
- checkArgument(
- proxyManifest.hasManifest(),
- String.format("Invalid ProxyManifest at %s: doesn't have a Manifest", manifestResourceId));
- checkArgument(
- proxyManifest.getLocationCount() == proxyManifest.getManifest().getArtifactCount(),
- String.format(
- "Invalid ProxyManifestat %s: %d locations but %d artifacts",
- manifestResourceId,
- proxyManifest.getLocationCount(),
- proxyManifest.getManifest().getArtifactCount()));
- LOG.info(
- "Manifest at {} has {} artifact locations",
- manifestResourceId,
- proxyManifest.getManifest().getArtifactCount());
- return proxyManifest;
+ return loadManifest(
+ Channels.newInputStream(FileSystems.open(manifestResourceId)),
+ manifestResourceId.toString());
}
private static ResourceId getManifestLocationFromToken(String retrievalToken) {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
index 5f7fe8c..c9baa17 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
@@ -17,38 +17,22 @@
*/
package org.apache.beam.runners.fnexecution.artifact;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.io.Serializable;
import java.nio.channels.WritableByteChannel;
-import java.nio.charset.Charset;
-import java.nio.charset.StandardCharsets;
import java.util.Collections;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.CommitManifestResponse;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactMetadata;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactRequest;
-import org.apache.beam.model.jobmanagement.v1.ArtifactApi.PutArtifactResponse;
import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase;
-import org.apache.beam.runners.fnexecution.FnService;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.MoveOptions.StandardMoveOptions;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.JsonFormat;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hasher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -66,86 +50,37 @@
*
* <p>The manifest file is encoded in {@link ProxyManifest}.
*/
-public class BeamFileSystemArtifactStagingService extends ArtifactStagingServiceImplBase
- implements FnService {
+public class BeamFileSystemArtifactStagingService extends AbstractArtifactStagingService {
private static final Logger LOG =
LoggerFactory.getLogger(BeamFileSystemArtifactStagingService.class);
private static final ObjectMapper MAPPER = new ObjectMapper();
// Use UTF8 for all text encoding.
- private static final Charset CHARSET = StandardCharsets.UTF_8;
public static final String MANIFEST = "MANIFEST";
public static final String ARTIFACTS = "artifacts";
@Override
- public StreamObserver<PutArtifactRequest> putArtifact(
- StreamObserver<PutArtifactResponse> responseObserver) {
- return new PutArtifactStreamObserver(responseObserver);
+ public String getArtifactUri(String stagingSession, String encodedFileName) throws Exception {
+ StagingSessionToken stagingSessionToken = StagingSessionToken.decode(stagingSession);
+ ResourceId artifactDirResourceId = getArtifactDirResourceId(stagingSessionToken);
+ return artifactDirResourceId
+ .resolve(encodedFileName, StandardResolveOptions.RESOLVE_FILE)
+ .toString();
}
@Override
- public void commitManifest(
- CommitManifestRequest request, StreamObserver<CommitManifestResponse> responseObserver) {
- try {
- StagingSessionToken stagingSessionToken =
- StagingSessionToken.decode(request.getStagingSessionToken());
- ResourceId manifestResourceId = getManifestFileResourceId(stagingSessionToken);
- ResourceId artifactDirResourceId = getArtifactDirResourceId(stagingSessionToken);
- ProxyManifest.Builder proxyManifestBuilder =
- ProxyManifest.newBuilder().setManifest(request.getManifest());
- for (ArtifactMetadata artifactMetadata : request.getManifest().getArtifactList()) {
- proxyManifestBuilder.addLocation(
- Location.newBuilder()
- .setName(artifactMetadata.getName())
- .setUri(
- artifactDirResourceId
- .resolve(
- encodedFileName(artifactMetadata), StandardResolveOptions.RESOLVE_FILE)
- .toString())
- .build());
- }
- try (WritableByteChannel manifestWritableByteChannel =
- FileSystems.create(manifestResourceId, MimeTypes.TEXT)) {
- manifestWritableByteChannel.write(
- CHARSET.encode(JsonFormat.printer().print(proxyManifestBuilder.build())));
- }
- // TODO: Validate integrity of staged files.
- responseObserver.onNext(
- CommitManifestResponse.newBuilder()
- .setRetrievalToken(manifestResourceId.toString())
- .build());
- responseObserver.onCompleted();
- } catch (Exception e) {
- // TODO: Cleanup all the artifacts.
- LOG.error("Unable to commit manifest.", e);
- responseObserver.onError(e);
- }
+ public WritableByteChannel openUri(String uri) throws IOException {
+ return FileSystems.create(FileSystems.matchNewResource(uri, false), MimeTypes.BINARY);
}
@Override
- public void close() throws Exception {
- // Nothing to close here.
+ public void removeUri(String uri) throws IOException {
+ FileSystems.delete(
+ Collections.singletonList(FileSystems.matchNewResource(uri, false)),
+ StandardMoveOptions.IGNORE_MISSING_FILES);
}
- /**
- * Generate a stagingSessionToken compatible with {@link BeamFileSystemArtifactStagingService}.
- *
- * @param sessionId Unique sessionId for artifact staging.
- * @param basePath Base path to upload artifacts.
- * @return Encoded stagingSessionToken.
- */
- public static String generateStagingSessionToken(String sessionId, String basePath) {
- StagingSessionToken stagingSessionToken = new StagingSessionToken();
- stagingSessionToken.setSessionId(sessionId);
- stagingSessionToken.setBasePath(basePath);
- return stagingSessionToken.encode();
- }
-
- private String encodedFileName(ArtifactMetadata artifactMetadata) {
- return "artifact_"
- + Hashing.sha256().hashString(artifactMetadata.getName(), CHARSET).toString();
- }
-
+ @Override
public void removeArtifacts(String stagingSessionToken) throws Exception {
StagingSessionToken parsedToken = StagingSessionToken.decode(stagingSessionToken);
ResourceId dir = getJobDirResourceId(parsedToken);
@@ -176,6 +111,19 @@
LOG.info("Removed dir {}", dir);
}
+ @Override
+ public WritableByteChannel openManifest(String stagingSession) throws Exception {
+ return FileSystems.create(
+ getManifestFileResourceId(StagingSessionToken.decode(stagingSession)), MimeTypes.TEXT);
+ }
+
+ @Override
+ public String getRetrievalToken(String stagingSession) throws Exception {
+ StagingSessionToken stagingSessionToken = StagingSessionToken.decode(stagingSession);
+ ResourceId manifestResourceId = getManifestFileResourceId(stagingSessionToken);
+ return manifestResourceId.toString();
+ }
+
private ResourceId getJobDirResourceId(StagingSessionToken stagingSessionToken) {
ResourceId baseResourceId;
// Get or Create the base path
@@ -196,124 +144,25 @@
.resolve(ARTIFACTS, StandardResolveOptions.RESOLVE_DIRECTORY);
}
- private class PutArtifactStreamObserver implements StreamObserver<PutArtifactRequest> {
-
- private final StreamObserver<PutArtifactResponse> outboundObserver;
- private PutArtifactMetadata metadata;
- private ResourceId artifactId;
- private WritableByteChannel artifactWritableByteChannel;
- private Hasher hasher;
-
- PutArtifactStreamObserver(StreamObserver<PutArtifactResponse> outboundObserver) {
- this.outboundObserver = outboundObserver;
- }
-
- @Override
- public void onNext(PutArtifactRequest putArtifactRequest) {
- // Create the directory structure for storing artifacts in the first call.
- if (metadata == null) {
- checkNotNull(putArtifactRequest);
- checkNotNull(putArtifactRequest.getMetadata());
- metadata = putArtifactRequest.getMetadata();
- LOG.debug("stored metadata: {}", metadata);
- // Check the base path exists or create the base path
- try {
- ResourceId artifactsDirId =
- getArtifactDirResourceId(
- StagingSessionToken.decode(
- putArtifactRequest.getMetadata().getStagingSessionToken()));
- artifactId =
- artifactsDirId.resolve(
- encodedFileName(metadata.getMetadata()), StandardResolveOptions.RESOLVE_FILE);
- LOG.debug(
- "Going to stage artifact {} to {}.", metadata.getMetadata().getName(), artifactId);
- artifactWritableByteChannel = FileSystems.create(artifactId, MimeTypes.BINARY);
- hasher = Hashing.sha256().newHasher();
- } catch (Exception e) {
- String message =
- String.format(
- "Failed to begin staging artifact %s", metadata.getMetadata().getName());
- outboundObserver.onError(
- new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
- }
- } else {
- try {
- ByteString data = putArtifactRequest.getData().getData();
- artifactWritableByteChannel.write(data.asReadOnlyByteBuffer());
- hasher.putBytes(data.toByteArray());
- } catch (IOException e) {
- String message =
- String.format(
- "Failed to write chunk of artifact %s to %s",
- metadata.getMetadata().getName(), artifactId);
- outboundObserver.onError(
- new StatusRuntimeException(Status.DATA_LOSS.withDescription(message).withCause(e)));
- }
- }
- }
-
- @Override
- public void onError(Throwable throwable) {
- // Delete the artifact.
- LOG.error("Staging artifact failed for " + artifactId, throwable);
- try {
- if (artifactWritableByteChannel != null) {
- artifactWritableByteChannel.close();
- }
- if (artifactId != null) {
- FileSystems.delete(
- Collections.singletonList(artifactId), StandardMoveOptions.IGNORE_MISSING_FILES);
- }
-
- } catch (IOException e) {
- outboundObserver.onError(
- new StatusRuntimeException(
- Status.DATA_LOSS.withDescription(
- String.format("Failed to clean up artifact file %s", artifactId))));
- return;
- }
- outboundObserver.onError(
- new StatusRuntimeException(
- Status.DATA_LOSS
- .withDescription(String.format("Failed to stage artifact %s", artifactId))
- .withCause(throwable)));
- }
-
- @Override
- public void onCompleted() {
- // Close the stream.
- LOG.debug("Staging artifact completed for " + artifactId);
- if (artifactWritableByteChannel != null) {
- try {
- artifactWritableByteChannel.close();
- } catch (IOException e) {
- onError(e);
- return;
- }
- }
- String expectedSha256 = metadata.getMetadata().getSha256();
- if (expectedSha256 != null && !expectedSha256.isEmpty()) {
- String actualSha256 = hasher.hash().toString();
- if (!actualSha256.equals(expectedSha256)) {
- outboundObserver.onError(
- new StatusRuntimeException(
- Status.INVALID_ARGUMENT.withDescription(
- String.format(
- "Artifact %s is corrupt: expected sah256 %s, but has sha256 %s",
- metadata.getMetadata().getName(), expectedSha256, actualSha256))));
- return;
- }
- }
- outboundObserver.onNext(PutArtifactResponse.newBuilder().build());
- outboundObserver.onCompleted();
- }
+ /**
+ * Generate a stagingSessionToken compatible with {@link BeamFileSystemArtifactStagingService}.
+ *
+ * @param sessionId Unique sessionId for artifact staging.
+ * @param basePath Base path to upload artifacts.
+ * @return Encoded stagingSessionToken.
+ */
+ public static String generateStagingSessionToken(String sessionId, String basePath) {
+ StagingSessionToken stagingSessionToken = new StagingSessionToken();
+ stagingSessionToken.setSessionId(sessionId);
+ stagingSessionToken.setBasePath(basePath);
+ return stagingSessionToken.encode();
}
/**
* Serializable StagingSessionToken used to stage files with {@link
* BeamFileSystemArtifactStagingService}.
*/
- private static class StagingSessionToken implements Serializable {
+ protected static class StagingSessionToken implements Serializable {
private String sessionId;
private String basePath;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java
new file mode 100644
index 0000000..5f4b90b
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * An {@link ArtifactRetrievalService} that loads artifacts as {@link ClassLoader} resources.
+ *
+ * <p>The retrieval token should be a path to a JSON-formatted ProxyManifest accessible via {@link
+ * ClassLoader#getResource(String)} whose resource locations also point to paths loadable via {@link
+ * ClassLoader#getResource(String)}.
+ */
+public class ClassLoaderArtifactRetrievalService extends AbstractArtifactRetrievalService {
+
+ private final ClassLoader classLoader;
+
+ public ClassLoaderArtifactRetrievalService() {
+ this(ClassLoaderArtifactRetrievalService.class.getClassLoader());
+ }
+
+ public ClassLoaderArtifactRetrievalService(ClassLoader classLoader) {
+ this.classLoader = classLoader;
+ }
+
+ @Override
+ public InputStream openManifest(String retrievalToken) throws IOException {
+ return openUri(retrievalToken, retrievalToken);
+ }
+
+ @Override
+ public InputStream openUri(String retrievalToken, String uri) throws IOException {
+ if (uri.charAt(0) == '/') {
+ uri = uri.substring(1);
+ }
+ InputStream result = classLoader.getResourceAsStream(uri);
+ if (result == null) {
+ throw new IOException("Unable to load " + uri + " with " + classLoader);
+ }
+ return result;
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java
new file mode 100644
index 0000000..774efaf
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.file.FileSystem;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Comparator;
+import java.util.stream.Stream;
+import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc;
+
+/**
+ * An {@link ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase} that loads artifacts into a
+ * Java {@link FileSystem}.
+ */
+public class JavaFilesystemArtifactStagingService extends AbstractArtifactStagingService {
+
+ public static final String MANIFEST = "MANIFEST.json";
+ public static final String ARTIFACTS = "ARTIFACTS";
+
+ private final FileSystem fileSystem;
+ private final Path artifactRootDir;
+
+ public JavaFilesystemArtifactStagingService(FileSystem fileSystem, String artifactRootDir) {
+ this.fileSystem = fileSystem;
+ this.artifactRootDir = fileSystem.getPath(artifactRootDir);
+ }
+
+ @Override
+ public String getArtifactUri(String stagingSessionToken, String encodedFileName)
+ throws Exception {
+ return artifactRootDir
+ .resolve(stagingSessionToken)
+ .resolve(ARTIFACTS)
+ .resolve(encodedFileName)
+ .toString();
+ }
+
+ @Override
+ public WritableByteChannel openUri(String uri) throws IOException {
+ Path parent = fileSystem.getPath(uri).getParent();
+ if (parent == null) {
+ throw new RuntimeException("Provided URI did not have a parent: " + uri);
+ }
+ Files.createDirectories(parent);
+ return Channels.newChannel(Files.newOutputStream(fileSystem.getPath(uri)));
+ }
+
+ @Override
+ public void removeUri(String uri) throws IOException {
+ Files.deleteIfExists(fileSystem.getPath(uri));
+ }
+
+ @Override
+ public WritableByteChannel openManifest(String stagingSessionToken) throws Exception {
+ return openUri(getManifestUri(stagingSessionToken));
+ }
+
+ @Override
+ public void removeArtifacts(String stagingSessionToken) throws Exception {
+ try (Stream<Path> paths = Files.walk(artifactRootDir.resolve(stagingSessionToken))) {
+ paths.sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);
+ }
+ }
+
+ @Override
+ public String getRetrievalToken(String stagingSessionToken) throws Exception {
+ return getManifestUri(stagingSessionToken);
+ }
+
+ private String getManifestUri(String stagingSessionToken) {
+ return artifactRootDir.resolve(stagingSessionToken).resolve(MANIFEST).toString();
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
new file mode 100644
index 0000000..caf5867
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+
+/** Implementation of a {@link ExecutableStageContext}. */
+public class DefaultExecutableStageContext implements ExecutableStageContext, AutoCloseable {
+ private final JobBundleFactory jobBundleFactory;
+
+ private static DefaultExecutableStageContext create(JobInfo jobInfo) {
+ JobBundleFactory jobBundleFactory = DefaultJobBundleFactory.create(jobInfo);
+ return new DefaultExecutableStageContext(jobBundleFactory);
+ }
+
+ private DefaultExecutableStageContext(JobBundleFactory jobBundleFactory) {
+ this.jobBundleFactory = jobBundleFactory;
+ }
+
+ @Override
+ public StageBundleFactory getStageBundleFactory(ExecutableStage executableStage) {
+ return jobBundleFactory.forStage(executableStage);
+ }
+
+ @Override
+ public void close() throws Exception {
+ jobBundleFactory.close();
+ }
+
+ /**
+ * {@link ExecutableStageContext.Factory} that creates and round-robins between a number of child
+ * {@link ExecutableStageContext.Factory} instances.
+ */
+ public static class MultiInstanceFactory implements ExecutableStageContext.Factory {
+
+ private int index = 0;
+ private final List<ReferenceCountingExecutableStageContextFactory> factories =
+ new ArrayList<>();
+ private final int maxFactories;
+ private final SerializableFunction<Object, Boolean> isReleaseSynchronous;
+
+ public MultiInstanceFactory(
+ int maxFactories, SerializableFunction<Object, Boolean> isReleaseSynchronous) {
+ this.isReleaseSynchronous = isReleaseSynchronous;
+ Preconditions.checkArgument(maxFactories >= 0, "sdk_worker_parallelism must be >= 0");
+
+ if (maxFactories == 0) {
+ // if this is 0, use the auto behavior of num_cores - 1 so that we leave some resources
+ // available for the java process
+ this.maxFactories = Math.max(Runtime.getRuntime().availableProcessors() - 1, 1);
+ } else {
+ this.maxFactories = maxFactories;
+ }
+ }
+
+ private synchronized ExecutableStageContext.Factory getFactory() {
+ ReferenceCountingExecutableStageContextFactory factory;
+ // If we haven't yet created maxFactories factories, create a new one. Otherwise use an
+ // existing one from factories.
+ if (factories.size() < maxFactories) {
+ factory =
+ ReferenceCountingExecutableStageContextFactory.create(
+ DefaultExecutableStageContext::create, isReleaseSynchronous);
+ factories.add(factory);
+ } else {
+ factory = factories.get(index);
+ }
+
+ index = (index + 1) % maxFactories;
+
+ return factory;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+ return getFactory().get(jobInfo);
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
index 5e9ae9d..cc9e482 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
@@ -18,13 +18,13 @@
package org.apache.beam.runners.fnexecution.control;
import com.google.auto.value.AutoValue;
-import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.IOException;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardEnvironments;
import org.apache.beam.runners.core.construction.BeamUrns;
@@ -36,6 +36,7 @@
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ClassLoaderArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
import org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
import org.apache.beam.runners.fnexecution.data.GrpcDataService;
@@ -57,15 +58,17 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.function.ThrowingFunction;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalNotification;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions.RetrievalServiceType;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalNotification;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -79,26 +82,29 @@
@ThreadSafe
public class DefaultJobBundleFactory implements JobBundleFactory {
private static final Logger LOG = LoggerFactory.getLogger(DefaultJobBundleFactory.class);
+ private static final IdGenerator factoryIdGenerator = IdGenerators.incrementingLongs();
+ private final String factoryId = factoryIdGenerator.getId();
private final LoadingCache<Environment, WrappedSdkHarnessClient> environmentCache;
private final Map<String, EnvironmentFactory.Provider> environmentFactoryProviderMap;
private final ExecutorService executor;
private final MapControlClientPool clientPool;
private final IdGenerator stageIdGenerator;
+ private final int environmentExpirationMillis;
public static DefaultJobBundleFactory create(JobInfo jobInfo) {
+ PipelineOptions pipelineOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
Map<String, EnvironmentFactory.Provider> environmentFactoryProviderMap =
ImmutableMap.of(
BeamUrns.getUrn(StandardEnvironments.Environments.DOCKER),
- new DockerEnvironmentFactory.Provider(
- PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())),
+ new DockerEnvironmentFactory.Provider(pipelineOptions),
BeamUrns.getUrn(StandardEnvironments.Environments.PROCESS),
- new ProcessEnvironmentFactory.Provider(),
+ new ProcessEnvironmentFactory.Provider(pipelineOptions),
BeamUrns.getUrn(StandardEnvironments.Environments.EXTERNAL),
new ExternalEnvironmentFactory.Provider(),
Environments.ENVIRONMENT_EMBEDDED, // Non Public urn for testing.
- new EmbeddedEnvironmentFactory.Provider(
- PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())));
+ new EmbeddedEnvironmentFactory.Provider(pipelineOptions));
return new DefaultJobBundleFactory(jobInfo, environmentFactoryProviderMap);
}
@@ -109,94 +115,77 @@
DefaultJobBundleFactory(
JobInfo jobInfo, Map<String, EnvironmentFactory.Provider> environmentFactoryMap) {
- IdGenerator stageIdGenerator = IdGenerators.incrementingLongs();
+ IdGenerator stageIdSuffixGenerator = IdGenerators.incrementingLongs();
this.environmentFactoryProviderMap = environmentFactoryMap;
this.executor = Executors.newCachedThreadPool();
this.clientPool = MapControlClientPool.create();
- this.stageIdGenerator = stageIdGenerator;
+ this.stageIdGenerator = () -> factoryId + "-" + stageIdSuffixGenerator.getId();
+ this.environmentExpirationMillis = getEnvironmentExpirationMillis(jobInfo);
this.environmentCache =
createEnvironmentCache(serverFactory -> createServerInfo(jobInfo, serverFactory));
}
@VisibleForTesting
DefaultJobBundleFactory(
+ JobInfo jobInfo,
Map<String, EnvironmentFactory.Provider> environmentFactoryMap,
IdGenerator stageIdGenerator,
- GrpcFnServer<FnApiControlClientPoolService> controlServer,
- GrpcFnServer<GrpcLoggingService> loggingServer,
- GrpcFnServer<ArtifactRetrievalService> retrievalServer,
- GrpcFnServer<StaticGrpcProvisionService> provisioningServer,
- GrpcFnServer<GrpcDataService> dataServer,
- GrpcFnServer<GrpcStateService> stateServer) {
+ ServerInfo serverInfo) {
this.environmentFactoryProviderMap = environmentFactoryMap;
this.executor = Executors.newCachedThreadPool();
this.clientPool = MapControlClientPool.create();
this.stageIdGenerator = stageIdGenerator;
- ServerInfo serverInfo =
- new AutoValue_DefaultJobBundleFactory_ServerInfo.Builder()
- .setControlServer(controlServer)
- .setLoggingServer(loggingServer)
- .setRetrievalServer(retrievalServer)
- .setProvisioningServer(provisioningServer)
- .setDataServer(dataServer)
- .setStateServer(stateServer)
- .build();
+ this.environmentExpirationMillis = getEnvironmentExpirationMillis(jobInfo);
this.environmentCache = createEnvironmentCache(serverFactory -> serverInfo);
}
+ private static int getEnvironmentExpirationMillis(JobInfo jobInfo) {
+ PipelineOptions pipelineOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+ return pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentExpirationMillis();
+ }
+
private LoadingCache<Environment, WrappedSdkHarnessClient> createEnvironmentCache(
ThrowingFunction<ServerFactory, ServerInfo> serverInfoCreator) {
- return CacheBuilder.newBuilder()
- .removalListener(
- (RemovalNotification<Environment, WrappedSdkHarnessClient> notification) -> {
- LOG.debug("Cleaning up for environment {}", notification.getKey().getUrn());
- try {
- notification.getValue().close();
- } catch (Exception e) {
- LOG.warn(
- String.format("Error cleaning up environment %s", notification.getKey()), e);
- }
- })
- .build(
- new CacheLoader<Environment, WrappedSdkHarnessClient>() {
- @Override
- public WrappedSdkHarnessClient load(Environment environment) throws Exception {
- EnvironmentFactory.Provider environmentFactoryProvider =
- environmentFactoryProviderMap.get(environment.getUrn());
- ServerFactory serverFactory = environmentFactoryProvider.getServerFactory();
- ServerInfo serverInfo = serverInfoCreator.apply(serverFactory);
+ CacheBuilder builder =
+ CacheBuilder.newBuilder()
+ .removalListener(
+ (RemovalNotification<Environment, WrappedSdkHarnessClient> notification) -> {
+ int refCount = notification.getValue().unref();
+ LOG.debug(
+ "Removed environment {} with {} remaining bundle references.",
+ notification.getKey(),
+ refCount);
+ });
- EnvironmentFactory environmentFactory =
- environmentFactoryProvider.createEnvironmentFactory(
- serverInfo.getControlServer(),
- serverInfo.getLoggingServer(),
- serverInfo.getRetrievalServer(),
- serverInfo.getProvisioningServer(),
- clientPool,
- stageIdGenerator);
- return WrappedSdkHarnessClient.wrapping(
- environmentFactory.createEnvironment(environment), serverInfo);
- }
- });
+ if (environmentExpirationMillis > 0) {
+ builder = builder.expireAfterWrite(environmentExpirationMillis, TimeUnit.MILLISECONDS);
+ }
+ return builder.build(
+ new CacheLoader<Environment, WrappedSdkHarnessClient>() {
+ @Override
+ public WrappedSdkHarnessClient load(Environment environment) throws Exception {
+ EnvironmentFactory.Provider environmentFactoryProvider =
+ environmentFactoryProviderMap.get(environment.getUrn());
+ ServerFactory serverFactory = environmentFactoryProvider.getServerFactory();
+ ServerInfo serverInfo = serverInfoCreator.apply(serverFactory);
+ EnvironmentFactory environmentFactory =
+ environmentFactoryProvider.createEnvironmentFactory(
+ serverInfo.getControlServer(),
+ serverInfo.getLoggingServer(),
+ serverInfo.getRetrievalServer(),
+ serverInfo.getProvisioningServer(),
+ clientPool,
+ stageIdGenerator);
+ return WrappedSdkHarnessClient.wrapping(
+ environmentFactory.createEnvironment(environment), serverInfo);
+ }
+ });
}
@Override
public StageBundleFactory forStage(ExecutableStage executableStage) {
- WrappedSdkHarnessClient wrappedClient =
- environmentCache.getUnchecked(executableStage.getEnvironment());
- ExecutableProcessBundleDescriptor processBundleDescriptor;
- try {
- processBundleDescriptor =
- ProcessBundleDescriptors.fromExecutableStage(
- stageIdGenerator.getId(),
- executableStage,
- wrappedClient.getServerInfo().getDataServer().getApiServiceDescriptor(),
- wrappedClient.getServerInfo().getStateServer().getApiServiceDescriptor());
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- return SimpleStageBundleFactory.create(
- wrappedClient, processBundleDescriptor, wrappedClient.getServerInfo().getStateServer());
+ return new SimpleStageBundleFactory(executableStage);
}
@Override
@@ -209,37 +198,42 @@
executor.shutdown();
}
- /** A simple stage bundle factory for remotely processing bundles. */
- protected static class SimpleStageBundleFactory implements StageBundleFactory {
+ /**
+ * A {@link StageBundleFactory} for remotely processing bundles that supports environment
+ * expiration.
+ */
+ private class SimpleStageBundleFactory implements StageBundleFactory {
- private final BundleProcessor processor;
- private final ExecutableProcessBundleDescriptor processBundleDescriptor;
+ private final ExecutableStage executableStage;
+ private BundleProcessor processor;
+ private ExecutableProcessBundleDescriptor processBundleDescriptor;
+ private WrappedSdkHarnessClient wrappedClient;
- // Store the wrapped client in order to keep a live reference into the cache.
- @SuppressFBWarnings private WrappedSdkHarnessClient wrappedClient;
+ private SimpleStageBundleFactory(ExecutableStage executableStage) {
+ this.executableStage = executableStage;
+ prepare(environmentCache.getUnchecked(executableStage.getEnvironment()));
+ }
- static SimpleStageBundleFactory create(
- WrappedSdkHarnessClient wrappedClient,
- ExecutableProcessBundleDescriptor processBundleDescriptor,
- GrpcFnServer<GrpcStateService> stateServer) {
- @SuppressWarnings("unchecked")
- BundleProcessor processor =
+ private void prepare(WrappedSdkHarnessClient wrappedClient) {
+ try {
+ this.wrappedClient = wrappedClient;
+ this.processBundleDescriptor =
+ ProcessBundleDescriptors.fromExecutableStage(
+ stageIdGenerator.getId(),
+ executableStage,
+ wrappedClient.getServerInfo().getDataServer().getApiServiceDescriptor(),
+ wrappedClient.getServerInfo().getStateServer().getApiServiceDescriptor());
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to create ProcessBundleDescriptor.", e);
+ }
+
+ this.processor =
wrappedClient
.getClient()
.getProcessor(
processBundleDescriptor.getProcessBundleDescriptor(),
processBundleDescriptor.getRemoteInputDestinations(),
- stateServer.getService());
- return new SimpleStageBundleFactory(processBundleDescriptor, processor, wrappedClient);
- }
-
- SimpleStageBundleFactory(
- ExecutableProcessBundleDescriptor processBundleDescriptor,
- BundleProcessor processor,
- WrappedSdkHarnessClient wrappedClient) {
- this.processBundleDescriptor = processBundleDescriptor;
- this.processor = processor;
- this.wrappedClient = wrappedClient;
+ wrappedClient.getServerInfo().getStateServer().getService());
}
@Override
@@ -250,24 +244,55 @@
throws Exception {
// TODO: Consider having BundleProcessor#newBundle take in an OutputReceiverFactory rather
// than constructing the receiver map here. Every bundle factory will need this.
- ImmutableMap.Builder<Target, RemoteOutputReceiver<?>> outputReceivers =
+ ImmutableMap.Builder<String, RemoteOutputReceiver<?>> outputReceivers =
ImmutableMap.builder();
- for (Map.Entry<Target, Coder<WindowedValue<?>>> targetCoder :
- processBundleDescriptor.getOutputTargetCoders().entrySet()) {
- Target target = targetCoder.getKey();
- Coder<WindowedValue<?>> coder = targetCoder.getValue();
+ for (Map.Entry<String, Coder> remoteOutputCoder :
+ processBundleDescriptor.getRemoteOutputCoders().entrySet()) {
+ String outputTransform = remoteOutputCoder.getKey();
+ Coder coder = remoteOutputCoder.getValue();
String bundleOutputPCollection =
Iterables.getOnlyElement(
processBundleDescriptor
.getProcessBundleDescriptor()
- .getTransformsOrThrow(target.getPrimitiveTransformReference())
+ .getTransformsOrThrow(outputTransform)
.getInputsMap()
.values());
- FnDataReceiver<WindowedValue<?>> outputReceiver =
- outputReceiverFactory.create(bundleOutputPCollection);
- outputReceivers.put(target, RemoteOutputReceiver.of(coder, outputReceiver));
+ FnDataReceiver outputReceiver = outputReceiverFactory.create(bundleOutputPCollection);
+ outputReceivers.put(outputTransform, RemoteOutputReceiver.of(coder, outputReceiver));
}
- return processor.newBundle(outputReceivers.build(), stateRequestHandler, progressHandler);
+
+ if (environmentExpirationMillis == 0) {
+ return processor.newBundle(outputReceivers.build(), stateRequestHandler, progressHandler);
+ }
+
+ final WrappedSdkHarnessClient client =
+ environmentCache.getUnchecked(executableStage.getEnvironment());
+ client.ref();
+
+ if (client != wrappedClient) {
+ // reset after environment expired
+ prepare(client);
+ }
+
+ final RemoteBundle bundle =
+ processor.newBundle(outputReceivers.build(), stateRequestHandler, progressHandler);
+ return new RemoteBundle() {
+ @Override
+ public String getId() {
+ return bundle.getId();
+ }
+
+ @Override
+ public Map<String, FnDataReceiver> getInputReceivers() {
+ return bundle.getInputReceivers();
+ }
+
+ @Override
+ public void close() throws Exception {
+ bundle.close();
+ client.unref();
+ }
+ };
}
@Override
@@ -292,6 +317,7 @@
private final RemoteEnvironment environment;
private final SdkHarnessClient client;
private final ServerInfo serverInfo;
+ private final AtomicInteger bundleRefCount = new AtomicInteger();
static WrappedSdkHarnessClient wrapping(RemoteEnvironment environment, ServerInfo serverInfo) {
SdkHarnessClient client =
@@ -305,6 +331,7 @@
this.environment = environment;
this.client = client;
this.serverInfo = serverInfo;
+ ref();
}
SdkHarnessClient getClient() {
@@ -328,12 +355,41 @@
AutoCloseable provisioningServer = serverInfo.getProvisioningServer()) {}
// TODO: Wait for executor shutdown?
}
+
+ private int ref() {
+ return bundleRefCount.incrementAndGet();
+ }
+
+ private int unref() {
+ int count = bundleRefCount.decrementAndGet();
+ if (count == 0) {
+ // Close environment after it was removed from cache and all bundles finished.
+ LOG.info("Closing environment {}", environment.getEnvironment());
+ try {
+ close();
+ } catch (Exception e) {
+ LOG.warn("Error cleaning up environment {}", environment.getEnvironment(), e);
+ }
+ }
+ return count;
+ }
}
private ServerInfo createServerInfo(JobInfo jobInfo, ServerFactory serverFactory)
throws IOException {
Preconditions.checkNotNull(serverFactory, "serverFactory can not be null");
+ PortablePipelineOptions portableOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
+ .as(PortablePipelineOptions.class);
+ ArtifactRetrievalService artifactRetrievalService;
+
+ if (portableOptions.getRetrievalServiceType() == RetrievalServiceType.CLASSLOADER) {
+ artifactRetrievalService = new ClassLoaderArtifactRetrievalService();
+ } else {
+ artifactRetrievalService = BeamFileSystemArtifactRetrievalService.create();
+ }
+
GrpcFnServer<FnApiControlClientPoolService> controlServer =
GrpcFnServer.allocatePortAndCreateFor(
FnApiControlClientPoolService.offeringClientsToPool(
@@ -343,8 +399,7 @@
GrpcFnServer.allocatePortAndCreateFor(
GrpcLoggingService.forWriter(Slf4jLogWriter.getDefault()), serverFactory);
GrpcFnServer<ArtifactRetrievalService> retrievalServer =
- GrpcFnServer.allocatePortAndCreateFor(
- BeamFileSystemArtifactRetrievalService.create(), serverFactory);
+ GrpcFnServer.allocatePortAndCreateFor(artifactRetrievalService, serverFactory);
GrpcFnServer<StaticGrpcProvisionService> provisioningServer =
GrpcFnServer.allocatePortAndCreateFor(
StaticGrpcProvisionService.create(jobInfo.toProvisionInfo()), serverFactory);
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ExecutableStageContext.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ExecutableStageContext.java
new file mode 100644
index 0000000..662a2a7
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ExecutableStageContext.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import java.io.Serializable;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+
+/** The context required in order to execute {@link ExecutableStage stages}. */
+public interface ExecutableStageContext extends AutoCloseable {
+
+ /**
+ * Creates {@link ExecutableStageContext} instances. Serializable so that factories can be defined
+ * at translation time and distributed to TaskManagers.
+ */
+ interface Factory extends Serializable {
+
+ /** Get or create {@link ExecutableStageContext} for given {@link JobInfo}. */
+ ExecutableStageContext get(JobInfo jobInfo);
+ }
+
+ StageBundleFactory getStageBundleFactory(ExecutableStage executableStage);
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
index 58ec5dd..f56e7f2 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClient.java
@@ -29,9 +29,9 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
index b2db72c..83b5a1b 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolService.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collection;
@@ -26,8 +26,8 @@
import org.apache.beam.model.fnexecution.v1.BeamFnControlGrpc;
import org.apache.beam.runners.fnexecution.FnService;
import org.apache.beam.runners.fnexecution.HeaderAccessor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/MapControlClientPool.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/MapControlClientPool.java
index f932edb..420d7fd 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/MapControlClientPool.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/MapControlClientPool.java
@@ -23,7 +23,7 @@
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* A {@link ControlClientPool} backed by a client map. It is expected that a given client id will be
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java
index 04a883d..b324cb1 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptors.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.fnexecution.control;
import static org.apache.beam.runners.core.construction.SyntheticComponents.uniqueId;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -28,10 +28,8 @@
import java.util.Map;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.RemoteGrpcPort;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
@@ -58,11 +56,11 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf.ByteStringCoder;
/** Utility methods for creating {@link ProcessBundleDescriptor} instances. */
@@ -116,17 +114,16 @@
Components.Builder components =
stage.getComponents().toBuilder().clearTransforms().putAllTransforms(stageTransforms);
- ImmutableMap.Builder<String, RemoteInputDestination<WindowedValue<?>>>
- inputDestinationsBuilder = ImmutableMap.builder();
- ImmutableMap.Builder<Target, Coder<WindowedValue<?>>> outputTargetCodersBuilder =
+ ImmutableMap.Builder<String, RemoteInputDestination> inputDestinationsBuilder =
ImmutableMap.builder();
+ ImmutableMap.Builder<String, Coder> remoteOutputCodersBuilder = ImmutableMap.builder();
// The order of these does not matter.
inputDestinationsBuilder.put(
stage.getInputPCollection().getId(),
addStageInput(dataEndpoint, stage.getInputPCollection(), components));
- outputTargetCodersBuilder.putAll(
+ remoteOutputCodersBuilder.putAll(
addStageOutputs(dataEndpoint, stage.getOutputPCollections(), components));
Map<String, Map<String, SideInputSpec>> sideInputSpecs = addSideInputs(stage, components);
@@ -136,7 +133,7 @@
Map<String, Map<String, TimerSpec>> timerSpecs =
forTimerSpecs(
- dataEndpoint, stage, components, inputDestinationsBuilder, outputTargetCodersBuilder);
+ dataEndpoint, stage, components, inputDestinationsBuilder, remoteOutputCodersBuilder);
// Copy data from components to ProcessBundleDescriptor.
ProcessBundleDescriptor.Builder bundleDescriptorBuilder =
@@ -155,23 +152,23 @@
return ExecutableProcessBundleDescriptor.of(
bundleDescriptorBuilder.build(),
inputDestinationsBuilder.build(),
- outputTargetCodersBuilder.build(),
+ remoteOutputCodersBuilder.build(),
sideInputSpecs,
bagUserStateSpecs,
timerSpecs);
}
- private static Map<Target, Coder<WindowedValue<?>>> addStageOutputs(
+ private static Map<String, Coder<WindowedValue<?>>> addStageOutputs(
ApiServiceDescriptor dataEndpoint,
Collection<PCollectionNode> outputPCollections,
Components.Builder components)
throws IOException {
- Map<Target, Coder<WindowedValue<?>>> outputTargetCoders = new LinkedHashMap<>();
+ Map<String, Coder<WindowedValue<?>>> remoteOutputCoders = new LinkedHashMap<>();
for (PCollectionNode outputPCollection : outputPCollections) {
- TargetEncoding targetEncoding = addStageOutput(dataEndpoint, components, outputPCollection);
- outputTargetCoders.put(targetEncoding.getTarget(), targetEncoding.getCoder());
+ OutputEncoding outputEncoding = addStageOutput(dataEndpoint, components, outputPCollection);
+ remoteOutputCoders.put(outputEncoding.getPTransformId(), outputEncoding.getCoder());
}
- return outputTargetCoders;
+ return remoteOutputCoders;
}
private static RemoteInputDestination<WindowedValue<?>> addStageInput(
@@ -195,15 +192,10 @@
PTransform inputTransform =
RemoteGrpcPortRead.readFromPort(inputPort, inputPCollection.getId()).toPTransform();
components.putTransforms(inputId, inputTransform);
- return RemoteInputDestination.of(
- wireCoder,
- Target.newBuilder()
- .setPrimitiveTransformReference(inputId)
- .setName(Iterables.getOnlyElement(inputTransform.getOutputsMap().keySet()))
- .build());
+ return RemoteInputDestination.of(wireCoder, inputId);
}
- private static TargetEncoding addStageOutput(
+ private static OutputEncoding addStageOutput(
ApiServiceDescriptor dataEndpoint,
Components.Builder components,
PCollectionNode outputPCollection)
@@ -225,12 +217,7 @@
components::containsTransforms);
PTransform outputTransform = outputWrite.toPTransform();
components.putTransforms(outputId, outputTransform);
- return new AutoValue_ProcessBundleDescriptors_TargetEncoding(
- Target.newBuilder()
- .setPrimitiveTransformReference(outputId)
- .setName(Iterables.getOnlyElement(outputTransform.getInputsMap().keySet()))
- .build(),
- wireCoder);
+ return new AutoValue_ProcessBundleDescriptors_OutputEncoding(outputId, wireCoder);
}
public static Map<String, Map<String, SideInputSpec>> getSideInputs(ExecutableStage stage)
@@ -308,8 +295,8 @@
ApiServiceDescriptor dataEndpoint,
ExecutableStage stage,
Components.Builder components,
- ImmutableMap.Builder<String, RemoteInputDestination<WindowedValue<?>>> remoteInputsBuilder,
- ImmutableMap.Builder<Target, Coder<WindowedValue<?>>> outputTargetCodersBuilder)
+ ImmutableMap.Builder<String, RemoteInputDestination> remoteInputsBuilder,
+ ImmutableMap.Builder<String, Coder> outputTransformCodersBuilder)
throws IOException {
ImmutableTable.Builder<String, String, TimerSpec> idsToSpec = ImmutableTable.builder();
for (TimerReference timerReference : stage.getTimers()) {
@@ -378,12 +365,12 @@
timerReference.transform().getId(), timerReference.localName()),
components.getPcollectionsMap()::containsKey);
components.putPcollections(outputTimerPCollectionId, timerCollectionSpec);
- TargetEncoding targetEncoding =
+ OutputEncoding outputEncoding =
addStageOutput(
dataEndpoint,
components,
PipelineNode.pCollection(outputTimerPCollectionId, timerCollectionSpec));
- outputTargetCodersBuilder.put(targetEncoding.getTarget(), targetEncoding.getCoder());
+ outputTransformCodersBuilder.put(outputEncoding.getPTransformId(), outputEncoding.getCoder());
components.putTransforms(
timerReference.transform().getId(),
// Since a transform can have more then one timer, update the transform inside components
@@ -403,7 +390,7 @@
timerReference.localName(),
inputTimerPCollectionId,
outputTimerPCollectionId,
- targetEncoding.getTarget(),
+ outputEncoding.getPTransformId(),
spec));
}
return idsToSpec.build().rowMap();
@@ -419,9 +406,7 @@
components.putCoders(
id,
RunnerApi.Coder.newBuilder()
- .setSpec(
- RunnerApi.SdkFunctionSpec.newBuilder()
- .setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn(ModelCoders.KV_CODER_URN)))
+ .setSpec(RunnerApi.FunctionSpec.newBuilder().setUrn(ModelCoders.KV_CODER_URN))
.addComponentCoderIds(keyCoderId)
.addComponentCoderIds(valueCoderId)
.build());
@@ -429,8 +414,8 @@
}
@AutoValue
- abstract static class TargetEncoding {
- abstract BeamFnApi.Target getTarget();
+ abstract static class OutputEncoding {
+ abstract String getPTransformId();
abstract Coder<WindowedValue<?>> getCoder();
}
@@ -500,10 +485,15 @@
String timerId,
String inputCollectionId,
String outputCollectionId,
- Target outputTarget,
+ String outputTransformId,
org.apache.beam.sdk.state.TimerSpec timerSpec) {
return new AutoValue_ProcessBundleDescriptors_TimerSpec(
- transformId, timerId, inputCollectionId, outputCollectionId, outputTarget, timerSpec);
+ transformId,
+ timerId,
+ inputCollectionId,
+ outputCollectionId,
+ outputTransformId,
+ timerSpec);
}
public abstract String transformId();
@@ -514,7 +504,7 @@
public abstract String outputCollectionId();
- public abstract Target outputTarget();
+ public abstract String outputTransformId();
public abstract org.apache.beam.sdk.state.TimerSpec getTimerSpec();
}
@@ -524,8 +514,8 @@
public abstract static class ExecutableProcessBundleDescriptor {
public static ExecutableProcessBundleDescriptor of(
ProcessBundleDescriptor descriptor,
- Map<String, RemoteInputDestination<WindowedValue<?>>> inputDestinations,
- Map<BeamFnApi.Target, Coder<WindowedValue<?>>> outputTargetCoders,
+ Map<String, RemoteInputDestination> inputDestinations,
+ Map<String, Coder> outputTransformCoders,
Map<String, Map<String, SideInputSpec>> sideInputSpecs,
Map<String, Map<String, BagUserStateSpec>> bagUserStateSpecs,
Map<String, Map<String, TimerSpec>> timerSpecs) {
@@ -550,7 +540,7 @@
return new AutoValue_ProcessBundleDescriptors_ExecutableProcessBundleDescriptor(
descriptor,
inputDestinations,
- Collections.unmodifiableMap(outputTargetCoders),
+ Collections.unmodifiableMap(outputTransformCoders),
copyOfSideInputSpecs.build().rowMap(),
copyOfBagUserStateSpecs.build().rowMap(),
copyOfTimerSpecs.build().rowMap());
@@ -562,14 +552,13 @@
* Get {@link RemoteInputDestination}s that input data/timers are sent to the {@link
* ProcessBundleDescriptor} over.
*/
- public abstract Map<String, RemoteInputDestination<WindowedValue<?>>>
- getRemoteInputDestinations();
+ public abstract Map<String, RemoteInputDestination> getRemoteInputDestinations();
/**
- * Get all of the targets materialized by this {@link ExecutableProcessBundleDescriptor} and the
- * java {@link Coder} for the wire format of that {@link BeamFnApi.Target}.
+ * Get all of the transforms materialized by this {@link ExecutableProcessBundleDescriptor} and
+ * the Java {@link Coder} for the wire format of that transform.
*/
- public abstract Map<BeamFnApi.Target, Coder<WindowedValue<?>>> getOutputTargetCoders();
+ public abstract Map<String, Coder> getRemoteOutputCoders();
/**
* Get a mapping from PTransform id to side input id to {@link SideInputSpec side inputs} that
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactory.java
new file mode 100644
index 0000000..3e48be1
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactory.java
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import java.io.Serializable;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.function.ThrowingFunction;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link ExecutableStageContext.Factory} which counts ExecutableStageContext reference for book
+ * keeping.
+ */
+public class ReferenceCountingExecutableStageContextFactory
+ implements ExecutableStageContext.Factory {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReferenceCountingExecutableStageContextFactory.class);
+ private static final int MAX_RETRY = 3;
+
+ private final Creator creator;
+ private transient volatile ScheduledExecutorService executor;
+ private transient volatile ConcurrentHashMap<String, WrappedContext> keyRegistry;
+ private final SerializableFunction<Object, Boolean> isReleaseSynchronous;
+
+ public static ReferenceCountingExecutableStageContextFactory create(
+ Creator creator, SerializableFunction<Object, Boolean> isReleaseSynchronous) {
+ return new ReferenceCountingExecutableStageContextFactory(creator, isReleaseSynchronous);
+ }
+
+ private ReferenceCountingExecutableStageContextFactory(
+ Creator creator, SerializableFunction<Object, Boolean> isReleaseSynchronous) {
+ this.creator = creator;
+ this.isReleaseSynchronous = isReleaseSynchronous;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+ // Retry is needed in case where an existing wrapper is picked from the cache but by
+ // the time we accessed wrapper.referenceCount, the wrapper was tombstoned by a pending
+ // release task.
+ // This race condition is highly unlikely to happen as there is no systematic coding
+ // practice which can cause this error because of TTL. However, even in very unlikely case
+ // when it happen we have the retry which get a valid context.
+ // Note: There is no leak in this logic as the cleanup is only done in release.
+ // In case of usage error where release is called before corresponding get finishes,
+ // release might throw an error. If release did not throw an error than we can be sure that
+ // the state of the system remains valid and appropriate cleanup will be done at TTL.
+ for (int retry = 0; retry < MAX_RETRY; retry++) {
+ // ConcurrentHashMap will handle the thread safety at the creation time.
+ WrappedContext wrapper =
+ getCache()
+ .computeIfAbsent(
+ jobInfo.jobId(),
+ jobId -> {
+ try {
+ return new WrappedContext(jobInfo, creator.apply(jobInfo));
+ } catch (Exception e) {
+ throw new RuntimeException(
+ "Unable to create context for job " + jobInfo.jobId(), e);
+ }
+ });
+ // Take a lock on wrapper before modifying reference count.
+ // Use null referenceCount == null as a tombstone for the wrapper.
+ synchronized (wrapper) {
+ if (wrapper.referenceCount != null) {
+ // The wrapper is still valid.
+ // Release has not yet got the lock and has not yet removed the wrapper.
+ wrapper.referenceCount.incrementAndGet();
+ return wrapper;
+ }
+ }
+ }
+
+ throw new RuntimeException(
+ String.format(
+ "Max retry %s exhausted while creating Context for job %s",
+ MAX_RETRY, jobInfo.jobId()));
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void scheduleRelease(JobInfo jobInfo) {
+ WrappedContext wrapper = getCache().get(jobInfo.jobId());
+ Preconditions.checkState(
+ wrapper != null, "Releasing context for unknown job: " + jobInfo.jobId());
+
+ PipelineOptions pipelineOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+ int environmentCacheTTLMillis =
+ pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+ if (environmentCacheTTLMillis > 0) {
+ if (isReleaseSynchronous.apply(this)) {
+ // Do immediate cleanup
+ release(wrapper);
+ } else {
+ // Schedule task to clean the container later.
+ getExecutor()
+ .schedule(() -> release(wrapper), environmentCacheTTLMillis, TimeUnit.MILLISECONDS);
+ }
+ } else {
+ // Do not release this asynchronously, as the releasing could fail due to the classloader not
+ // being available anymore after the tasks have been removed from the execution engine.
+ release(wrapper);
+ }
+ }
+
+ private ConcurrentHashMap<String, WrappedContext> getCache() {
+ // Lazily initialize keyRegistry because serialization will set it to null.
+ if (keyRegistry != null) {
+ return keyRegistry;
+ }
+ synchronized (this) {
+ if (keyRegistry == null) {
+ keyRegistry = new ConcurrentHashMap<>();
+ }
+ return keyRegistry;
+ }
+ }
+
+ private ScheduledExecutorService getExecutor() {
+ // Lazily initialize executor because serialization will set it to null.
+ if (executor != null) {
+ return executor;
+ }
+ synchronized (this) {
+ if (executor == null) {
+ executor =
+ Executors.newScheduledThreadPool(1, new ThreadFactoryBuilder().setDaemon(true).build());
+ }
+ return executor;
+ }
+ }
+
+ @VisibleForTesting
+ void release(ExecutableStageContext context) {
+ @SuppressWarnings({"unchecked", "Not exected to be called from outside."})
+ WrappedContext wrapper = (WrappedContext) context;
+ synchronized (wrapper) {
+ if (wrapper.referenceCount.decrementAndGet() == 0) {
+ // Tombstone wrapper.
+ wrapper.referenceCount = null;
+ if (getCache().remove(wrapper.jobInfo.jobId(), wrapper)) {
+ try {
+ wrapper.closeActual();
+ } catch (Throwable t) {
+ LOG.error("Unable to close ExecutableStageContext.", t);
+ }
+ }
+ }
+ }
+ }
+
+ /** {@link WrappedContext} does not expose equals of actual {@link ExecutableStageContext}. */
+ @VisibleForTesting
+ class WrappedContext implements ExecutableStageContext {
+ private JobInfo jobInfo;
+ private AtomicInteger referenceCount;
+ @VisibleForTesting ExecutableStageContext context;
+
+ /** {@link WrappedContext#equals(Object)} is only based on {@link JobInfo#jobId()}. */
+ WrappedContext(JobInfo jobInfo, ExecutableStageContext context) {
+ this.jobInfo = jobInfo;
+ this.context = context;
+ this.referenceCount = new AtomicInteger(0);
+ }
+
+ @Override
+ public StageBundleFactory getStageBundleFactory(ExecutableStage executableStage) {
+ return context.getStageBundleFactory(executableStage);
+ }
+
+ @Override
+ public void close() {
+ // Just schedule the context as we want to reuse it if possible.
+ scheduleRelease(jobInfo);
+ }
+
+ private void closeActual() throws Exception {
+ context.close();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ WrappedContext that = (WrappedContext) o;
+ return Objects.equals(jobInfo.jobId(), that.jobInfo.jobId());
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(jobInfo);
+ }
+
+ @Override
+ public String toString() {
+ return "ContextWrapper{"
+ + "jobId='"
+ + jobInfo
+ + '\''
+ + ", referenceCount="
+ + referenceCount
+ + '}';
+ }
+ }
+
+ /** Interface for creator which extends Serializable. */
+ public interface Creator
+ extends ThrowingFunction<JobInfo, ExecutableStageContext>, Serializable {}
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/RemoteBundle.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/RemoteBundle.java
index 069c854..30fea85 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/RemoteBundle.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/RemoteBundle.java
@@ -19,7 +19,6 @@
import java.util.Map;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.sdk.util.WindowedValue;
/**
* A bundle capable of handling input data elements for a {@link
@@ -37,7 +36,7 @@
* Get a map of PCollection ids to {@link FnDataReceiver receiver}s which consume input elements,
* forwarding them to the remote environment.
*/
- Map<String, FnDataReceiver<WindowedValue<?>>> getInputReceivers();
+ Map<String, FnDataReceiver> getInputReceivers();
/**
* Closes this bundle. This causes the input {@link FnDataReceiver} to be closed (future calls to
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
index d19a7a8..b3f1c2e 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.fnexecution.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.HashMap;
import java.util.Map;
@@ -30,12 +30,10 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.RegisterResponse;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.runners.fnexecution.data.FnDataService;
import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
import org.apache.beam.runners.fnexecution.state.StateDelegator;
-import org.apache.beam.runners.fnexecution.state.StateDelegator.Registration;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.IdGenerator;
@@ -45,8 +43,7 @@
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -65,13 +62,13 @@
public class BundleProcessor {
private final ProcessBundleDescriptor processBundleDescriptor;
private final CompletionStage<RegisterResponse> registrationFuture;
- private final Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputs;
+ private final Map<String, RemoteInputDestination> remoteInputs;
private final StateDelegator stateDelegator;
private BundleProcessor(
ProcessBundleDescriptor processBundleDescriptor,
CompletionStage<RegisterResponse> registrationFuture,
- Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputs,
+ Map<String, RemoteInputDestination> remoteInputs,
StateDelegator stateDelegator) {
this.processBundleDescriptor = processBundleDescriptor;
this.registrationFuture = registrationFuture;
@@ -100,7 +97,7 @@
* }</pre>
*/
public ActiveBundle newBundle(
- Map<BeamFnApi.Target, RemoteOutputReceiver<?>> outputReceivers,
+ Map<String, RemoteOutputReceiver<?>> outputReceivers,
BundleProgressHandler progressHandler) {
return newBundle(
outputReceivers,
@@ -130,7 +127,7 @@
* }</pre>
*/
public ActiveBundle newBundle(
- Map<BeamFnApi.Target, RemoteOutputReceiver<?>> outputReceivers,
+ Map<String, RemoteOutputReceiver<?>> outputReceivers,
StateRequestHandler stateRequestHandler,
BundleProgressHandler progressHandler) {
String bundleId = idGenerator.getId();
@@ -141,7 +138,8 @@
.setInstructionId(bundleId)
.setProcessBundle(
BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference(processBundleDescriptor.getId()))
+ .setProcessBundleDescriptorId(processBundleDescriptor.getId())
+ .addAllCacheTokens(stateRequestHandler.getCacheTokens()))
.build());
LOG.debug(
"Sent {} with ID {} for {} with ID {}",
@@ -152,25 +150,20 @@
CompletionStage<BeamFnApi.ProcessBundleResponse> specificResponse =
genericResponse.thenApply(InstructionResponse::getProcessBundle);
- Map<BeamFnApi.Target, InboundDataClient> outputClients = new HashMap<>();
- for (Map.Entry<BeamFnApi.Target, RemoteOutputReceiver<?>> targetReceiver :
- outputReceivers.entrySet()) {
+ Map<String, InboundDataClient> outputClients = new HashMap<>();
+ for (Map.Entry<String, RemoteOutputReceiver<?>> receiver : outputReceivers.entrySet()) {
InboundDataClient outputClient =
- attachReceiver(
- bundleId,
- targetReceiver.getKey(),
- (RemoteOutputReceiver) targetReceiver.getValue());
- outputClients.put(targetReceiver.getKey(), outputClient);
+ attachReceiver(bundleId, receiver.getKey(), (RemoteOutputReceiver) receiver.getValue());
+ outputClients.put(receiver.getKey(), outputClient);
}
- ImmutableMap.Builder<String, CloseableFnDataReceiver<WindowedValue<?>>> dataReceiversBuilder =
+ ImmutableMap.Builder<String, CloseableFnDataReceiver> dataReceiversBuilder =
ImmutableMap.builder();
- for (Map.Entry<String, RemoteInputDestination<WindowedValue<?>>> remoteInput :
- remoteInputs.entrySet()) {
+ for (Map.Entry<String, RemoteInputDestination> remoteInput : remoteInputs.entrySet()) {
dataReceiversBuilder.put(
remoteInput.getKey(),
fnApiDataService.send(
- LogicalEndpoint.of(bundleId, remoteInput.getValue().getTarget()),
+ LogicalEndpoint.of(bundleId, remoteInput.getValue().getPTransformId()),
(Coder) remoteInput.getValue().getCoder()));
}
@@ -184,11 +177,9 @@
}
private <OutputT> InboundDataClient attachReceiver(
- String bundleId,
- BeamFnApi.Target target,
- RemoteOutputReceiver<WindowedValue<OutputT>> receiver) {
+ String bundleId, String ptransformId, RemoteOutputReceiver<OutputT> receiver) {
return fnApiDataService.receive(
- LogicalEndpoint.of(bundleId, target), receiver.getCoder(), receiver.getReceiver());
+ LogicalEndpoint.of(bundleId, ptransformId), receiver.getCoder(), receiver.getReceiver());
}
}
@@ -196,16 +187,16 @@
public static class ActiveBundle implements RemoteBundle {
private final String bundleId;
private final CompletionStage<BeamFnApi.ProcessBundleResponse> response;
- private final Map<String, CloseableFnDataReceiver<WindowedValue<?>>> inputReceivers;
- private final Map<BeamFnApi.Target, InboundDataClient> outputClients;
+ private final Map<String, CloseableFnDataReceiver> inputReceivers;
+ private final Map<String, InboundDataClient> outputClients;
private final StateDelegator.Registration stateRegistration;
private final BundleProgressHandler progressHandler;
private ActiveBundle(
String bundleId,
CompletionStage<ProcessBundleResponse> response,
- Map<String, CloseableFnDataReceiver<WindowedValue<?>>> inputReceivers,
- Map<Target, InboundDataClient> outputClients,
+ Map<String, CloseableFnDataReceiver> inputReceivers,
+ Map<String, InboundDataClient> outputClients,
StateDelegator.Registration stateRegistration,
BundleProgressHandler progressHandler) {
this.bundleId = bundleId;
@@ -227,7 +218,7 @@
* elements, forwarding them to the remote environment.
*/
@Override
- public Map<String, FnDataReceiver<WindowedValue<?>>> getInputReceivers() {
+ public Map<String, FnDataReceiver> getInputReceivers() {
return (Map) inputReceivers;
}
@@ -360,7 +351,7 @@
*/
public BundleProcessor getProcessor(
BeamFnApi.ProcessBundleDescriptor descriptor,
- Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputDesinations) {
+ Map<String, RemoteInputDestination> remoteInputDesinations) {
checkState(
!descriptor.hasStateApiServiceDescriptor(),
"The %s cannot support a %s containing a state %s.",
@@ -386,12 +377,12 @@
*/
public BundleProcessor getProcessor(
BeamFnApi.ProcessBundleDescriptor descriptor,
- Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputDesinations,
+ Map<String, RemoteInputDestination> remoteInputDestinations,
StateDelegator stateDelegator) {
@SuppressWarnings("unchecked")
BundleProcessor bundleProcessor =
clientProcessors.computeIfAbsent(
- descriptor.getId(), s -> create(descriptor, remoteInputDesinations, stateDelegator));
+ descriptor.getId(), s -> create(descriptor, remoteInputDestinations, stateDelegator));
checkArgument(
bundleProcessor.processBundleDescriptor.equals(descriptor),
"The provided %s with id %s collides with an existing %s with the same id but "
@@ -430,7 +421,7 @@
/** Registers a {@link BeamFnApi.ProcessBundleDescriptor} for future processing. */
private BundleProcessor create(
BeamFnApi.ProcessBundleDescriptor processBundleDescriptor,
- Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputDestinations,
+ Map<String, RemoteInputDestination> remoteInputDestinations,
StateDelegator stateDelegator) {
LOG.debug("Registering {}", processBundleDescriptor);
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactory.java
index 38fcc93..c54764e 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactory.java
@@ -22,7 +22,6 @@
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
@@ -35,8 +34,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* A {@link JobBundleFactory} which can manage a single instance of an {@link Environment}.
@@ -153,21 +151,20 @@
OutputReceiverFactory outputReceiverFactory,
StateRequestHandler stateRequestHandler,
BundleProgressHandler progressHandler) {
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Map.Entry<Target, Coder<WindowedValue<?>>> targetCoders :
- descriptor.getOutputTargetCoders().entrySet()) {
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Map.Entry<String, Coder> remoteOutputCoder :
+ descriptor.getRemoteOutputCoders().entrySet()) {
String bundleOutputPCollection =
Iterables.getOnlyElement(
descriptor
.getProcessBundleDescriptor()
- .getTransformsOrThrow(targetCoders.getKey().getPrimitiveTransformReference())
+ .getTransformsOrThrow(remoteOutputCoder.getKey())
.getInputsMap()
.values());
- FnDataReceiver<WindowedValue<?>> outputReceiver =
- outputReceiverFactory.create(bundleOutputPCollection);
+ FnDataReceiver<?> outputReceiver = outputReceiverFactory.create(bundleOutputPCollection);
outputReceivers.put(
- targetCoders.getKey(),
- RemoteOutputReceiver.of(targetCoders.getValue(), outputReceiver));
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of((Coder) remoteOutputCoder.getValue(), outputReceiver));
}
return processor.newBundle(outputReceivers, stateRequestHandler, progressHandler);
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/TimerReceiverFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/TimerReceiverFactory.java
new file mode 100644
index 0000000..a277b9e
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/TimerReceiverFactory.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.BiConsumer;
+import org.apache.beam.runners.core.StateNamespace;
+import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.TimerInternals;
+import org.apache.beam.runners.core.TimerInternals.TimerData;
+import org.apache.beam.runners.core.construction.Timer;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.TimerSpec;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.state.TimeDomain;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link OutputReceiverFactory} that passes outputs to {@link
+ * TimerReceiverFactory#timerDataConsumer}.
+ */
+public class TimerReceiverFactory implements OutputReceiverFactory {
+ private static final Logger LOG = LoggerFactory.getLogger(TimerReceiverFactory.class);
+
+ /** Timer PCollection id => TimerReference. */
+ private final HashMap<String, TimerSpec> timerOutputIdToSpecMap;
+
+ private final BiConsumer<WindowedValue, TimerData> timerDataConsumer;
+ private final Coder windowCoder;
+
+ public TimerReceiverFactory(
+ StageBundleFactory stageBundleFactory,
+ BiConsumer<WindowedValue, TimerInternals.TimerData> timerDataConsumer,
+ Coder windowCoder) {
+ this.timerOutputIdToSpecMap = new HashMap<>();
+ // Gather all timers from all transforms by their output pCollectionId which is unique
+ for (Map<String, ProcessBundleDescriptors.TimerSpec> transformTimerMap :
+ stageBundleFactory.getProcessBundleDescriptor().getTimerSpecs().values()) {
+ for (ProcessBundleDescriptors.TimerSpec timerSpec : transformTimerMap.values()) {
+ timerOutputIdToSpecMap.put(timerSpec.outputCollectionId(), timerSpec);
+ }
+ }
+ this.timerDataConsumer = timerDataConsumer;
+ this.windowCoder = windowCoder;
+ }
+
+ @Override
+ public <OutputT> FnDataReceiver<OutputT> create(String pCollectionId) {
+ final ProcessBundleDescriptors.TimerSpec timerSpec = timerOutputIdToSpecMap.get(pCollectionId);
+
+ return receivedElement -> {
+ WindowedValue windowedValue = (WindowedValue) receivedElement;
+ Timer timer =
+ Preconditions.checkNotNull(
+ (Timer) ((KV) windowedValue.getValue()).getValue(),
+ "Received null Timer from SDK harness: %s",
+ receivedElement);
+ LOG.debug("Timer received: {} {}", pCollectionId, timer);
+ for (Object window : windowedValue.getWindows()) {
+ StateNamespace namespace = StateNamespaces.window(windowCoder, (BoundedWindow) window);
+ TimeDomain timeDomain = timerSpec.getTimerSpec().getTimeDomain();
+ String timerId = timerSpec.inputCollectionId();
+ TimerInternals.TimerData timerData =
+ TimerInternals.TimerData.of(timerId, namespace, timer.getTimestamp(), timeDomain);
+ timerDataConsumer.accept(windowedValue, timerData);
+ }
+ };
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/FnDataService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/FnDataService.java
index 8d72d1a..9c81883 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/FnDataService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/FnDataService.java
@@ -22,7 +22,6 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
-import org.apache.beam.sdk.util.WindowedValue;
/**
* The {@link FnDataService} is able to forward inbound elements to a consumer and is also a
@@ -44,9 +43,7 @@
* <p>The provided receiver is not required to be thread safe.
*/
<T> InboundDataClient receive(
- LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> listener);
+ LogicalEndpoint inputLocation, Coder<T> coder, FnDataReceiver<T> listener);
/**
* Creates a receiver to which you can write data values and have them sent over this data plane
@@ -58,6 +55,5 @@
*
* <p>The returned receiver is not thread safe.
*/
- <T> CloseableFnDataReceiver<WindowedValue<T>> send(
- LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder);
+ <T> CloseableFnDataReceiver<T> send(LogicalEndpoint outputLocation, Coder<T> coder);
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
index 8cebea0..69d378f 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
@@ -35,9 +35,8 @@
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.SettableFuture;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -129,13 +128,11 @@
@Override
@SuppressWarnings("FutureReturnValueIgnored")
public <T> InboundDataClient receive(
- final LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> listener) {
+ final LogicalEndpoint inputLocation, Coder<T> coder, FnDataReceiver<T> listener) {
LOG.debug(
- "Registering receiver for instruction {} and target {}",
+ "Registering receiver for instruction {} and transform {}",
inputLocation.getInstructionId(),
- inputLocation.getTarget());
+ inputLocation.getTransformId());
final BeamFnDataInboundObserver<T> observer =
BeamFnDataInboundObserver.forConsumer(coder, listener);
if (connectedClient.isDone()) {
@@ -164,12 +161,11 @@
}
@Override
- public <T> CloseableFnDataReceiver<WindowedValue<T>> send(
- LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder) {
+ public <T> CloseableFnDataReceiver<T> send(LogicalEndpoint outputLocation, Coder<T> coder) {
LOG.debug(
- "Creating sender for instruction {} and target {}",
+ "Creating sender for instruction {} and transform {}",
outputLocation.getInstructionId(),
- outputLocation.getTarget());
+ outputLocation.getTransformId());
try {
return BeamFnDataBufferingOutboundObserver.forLocation(
outputLocation, coder, connectedClient.get(3, TimeUnit.MINUTES).getOutboundObserver());
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestination.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestination.java
index 36a4be9..e1030ae 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestination.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestination.java
@@ -27,11 +27,11 @@
*/
@AutoValue
public abstract class RemoteInputDestination<T> {
- public static <T> RemoteInputDestination<T> of(Coder<T> coder, BeamFnApi.Target target) {
- return new AutoValue_RemoteInputDestination<>(coder, target);
+ public static <T> RemoteInputDestination<T> of(Coder<T> coder, String ptransformId) {
+ return new AutoValue_RemoteInputDestination<>(coder, ptransformId);
}
public abstract Coder<T> getCoder();
- public abstract BeamFnApi.Target getTarget();
+ public abstract String getPTransformId();
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
index cbd65be..fd4b66d 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerCommand.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.environment;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.BufferedReader;
@@ -33,7 +33,7 @@
import java.util.concurrent.TimeoutException;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -121,6 +121,20 @@
}
/**
+ * Returns logs for the container specified by {@code containerId}.
+ *
+ * @throws IOException if an IOException occurs or if the given container id does not exist
+ */
+ public String getContainerLogs(String containerId)
+ throws IOException, TimeoutException, InterruptedException {
+ checkArgument(containerId != null);
+ checkArgument(
+ CONTAINER_ID_PATTERN.matcher(containerId).matches(),
+ "Container ID must be a 64-character hexadecimal string");
+ return runShortCommand(Arrays.asList(dockerExecutable, "logs", containerId), true, "\n");
+ }
+
+ /**
* Kills a docker container by container id.
*
* @throws IOException if an IOException occurs or if the given container id does not exist
@@ -134,10 +148,41 @@
runShortCommand(Arrays.asList(dockerExecutable, "kill", containerId));
}
- /** Run the given command invocation and return stdout as a String. */
+ /**
+ * Removes docker container with container id.
+ *
+ * @throws IOException if an IOException occurs, or if the given container id either does not
+ * exist or is still running
+ */
+ public void removeContainer(String containerId)
+ throws IOException, TimeoutException, InterruptedException {
+ checkArgument(containerId != null);
+ checkArgument(
+ CONTAINER_ID_PATTERN.matcher(containerId).matches(),
+ "Container ID must be a 64-character hexadecimal string");
+ runShortCommand(Arrays.asList(dockerExecutable, "rm", containerId));
+ }
+
private String runShortCommand(List<String> invocation)
throws IOException, TimeoutException, InterruptedException {
+ return runShortCommand(invocation, false, "");
+ }
+
+ /**
+ * Runs a command, blocks until {@link DockerCommand#commandTimeout} has elapsed, then returns the
+ * command's output.
+ *
+ * @param invocation command and arguments to be run
+ * @param redirectErrorStream if true, include the process's stderr in the return value
+ * @param delimiter used for separating output lines
+ * @return stdout of the command, including stderr if {@code redirectErrorStream} is true
+ * @throws TimeoutException if command has not finished by {@link DockerCommand#commandTimeout}
+ */
+ private String runShortCommand(
+ List<String> invocation, boolean redirectErrorStream, CharSequence delimiter)
+ throws IOException, TimeoutException, InterruptedException {
ProcessBuilder pb = new ProcessBuilder(invocation);
+ pb.redirectErrorStream(redirectErrorStream);
Process process = pb.start();
// TODO: Consider supplying executor service here.
CompletableFuture<String> resultString =
@@ -147,17 +192,26 @@
BufferedReader reader =
new BufferedReader(
new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8));
- return reader.lines().collect(Collectors.joining());
+ return reader.lines().collect(Collectors.joining(delimiter));
});
- // NOTE: We only consume the error string in the case of an error.
- CompletableFuture<String> errorFuture =
- CompletableFuture.supplyAsync(
- () -> {
- BufferedReader reader =
- new BufferedReader(
- new InputStreamReader(process.getErrorStream(), StandardCharsets.UTF_8));
- return reader.lines().collect(Collectors.joining());
- });
+ CompletableFuture<String> errorFuture;
+ String errorStringName;
+ if (redirectErrorStream) {
+ // The standard output and standard error are combined into one stream.
+ errorStringName = "stdout and stderr";
+ errorFuture = resultString;
+ } else {
+ // The error stream is separate, and we only consume it in the case of an error.
+ errorStringName = "stderr";
+ errorFuture =
+ CompletableFuture.supplyAsync(
+ () -> {
+ BufferedReader reader =
+ new BufferedReader(
+ new InputStreamReader(process.getErrorStream(), StandardCharsets.UTF_8));
+ return reader.lines().collect(Collectors.joining(delimiter));
+ });
+ }
// TODO: Retry on interrupt?
boolean processDone = process.waitFor(commandTimeout.toMillis(), TimeUnit.MILLISECONDS);
if (!processDone) {
@@ -173,12 +227,16 @@
try {
errorString = errorFuture.get(commandTimeout.toMillis(), TimeUnit.MILLISECONDS);
} catch (Exception stderrEx) {
- errorString = String.format("Error capturing stderr: %s", stderrEx.getMessage());
+ errorString =
+ String.format("Error capturing %s: %s", errorStringName, stderrEx.getMessage());
}
throw new IOException(
String.format(
- "Received exit code %d for command '%s'. stderr: %s",
- exitCode, invocation.stream().collect(Collectors.joining(" ")), errorString));
+ "Received exit code %d for command '%s'. %s: %s",
+ exitCode,
+ invocation.stream().collect(Collectors.joining(" ")),
+ errorStringName,
+ errorString));
}
try {
// TODO: Consider a stricter timeout.
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerContainerEnvironment.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerContainerEnvironment.java
index a4e33a1..2cffd93 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerContainerEnvironment.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerContainerEnvironment.java
@@ -20,6 +20,8 @@
import javax.annotation.concurrent.ThreadSafe;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* A {@link RemoteEnvironment} that wraps a running Docker container.
@@ -30,12 +32,16 @@
@ThreadSafe
class DockerContainerEnvironment implements RemoteEnvironment {
+ private static final Logger LOG = LoggerFactory.getLogger(DockerContainerEnvironment.class);
+
static DockerContainerEnvironment create(
DockerCommand docker,
Environment environment,
String containerId,
- InstructionRequestHandler instructionHandler) {
- return new DockerContainerEnvironment(docker, environment, containerId, instructionHandler);
+ InstructionRequestHandler instructionHandler,
+ boolean retainDockerContainer) {
+ return new DockerContainerEnvironment(
+ docker, environment, containerId, instructionHandler, retainDockerContainer);
}
private final Object lock = new Object();
@@ -43,6 +49,7 @@
private final Environment environment;
private final String containerId;
private final InstructionRequestHandler instructionHandler;
+ private final boolean retainDockerContainer;
private boolean isClosed = false;
@@ -50,11 +57,13 @@
DockerCommand docker,
Environment environment,
String containerId,
- InstructionRequestHandler instructionHandler) {
+ InstructionRequestHandler instructionHandler,
+ boolean retainDockerContainer) {
this.docker = docker;
this.environment = environment;
this.containerId = containerId;
this.instructionHandler = instructionHandler;
+ this.retainDockerContainer = retainDockerContainer;
}
@Override
@@ -79,7 +88,12 @@
if (!isClosed) {
isClosed = true;
instructionHandler.close();
+ String containerLogs = docker.getContainerLogs(containerId);
+ LOG.info("Closing Docker container {}. Logs:\n{}", containerId, containerLogs);
docker.killContainer(containerId);
+ if (!retainDockerContainer) {
+ docker.removeContainer(containerId);
+ }
}
}
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
index 94834fc..154aaec 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.environment;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import java.nio.file.Files;
import java.nio.file.Paths;
@@ -39,9 +39,10 @@
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.options.ManualDockerEnvironmentOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -62,7 +63,7 @@
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
ControlClientPool.Source clientSource,
IdGenerator idGenerator,
- boolean retainDockerContainer) {
+ PipelineOptions pipelineOptions) {
return new DockerEnvironmentFactory(
docker,
controlServiceServer,
@@ -71,7 +72,7 @@
provisioningServiceServer,
idGenerator,
clientSource,
- retainDockerContainer);
+ pipelineOptions);
}
private final DockerCommand docker;
@@ -81,7 +82,7 @@
private final GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer;
private final IdGenerator idGenerator;
private final ControlClientPool.Source clientSource;
- private final boolean retainDockerContainer;
+ private final PipelineOptions pipelineOptions;
private DockerEnvironmentFactory(
DockerCommand docker,
@@ -91,7 +92,7 @@
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
IdGenerator idGenerator,
ControlClientPool.Source clientSource,
- boolean retainDockerContainer) {
+ PipelineOptions pipelineOptions) {
this.docker = docker;
this.controlServiceServer = controlServiceServer;
this.loggingServiceServer = loggingServiceServer;
@@ -99,7 +100,7 @@
this.provisioningServiceServer = provisioningServiceServer;
this.idGenerator = idGenerator;
this.clientSource = clientSource;
- this.retainDockerContainer = retainDockerContainer;
+ this.pipelineOptions = pipelineOptions;
}
/** Creates a new, active {@link RemoteEnvironment} backed by a local Docker container. */
@@ -123,7 +124,7 @@
String provisionEndpoint = provisioningServiceServer.getApiServiceDescriptor().getUrl();
String controlEndpoint = controlServiceServer.getApiServiceDescriptor().getUrl();
- ImmutableList.Builder<String> dockerArgsBuilder =
+ ImmutableList.Builder<String> dockerOptsBuilder =
ImmutableList.<String>builder()
.addAll(gcsCredentialArgs())
// NOTE: Host networking does not work on Mac, but the command line flag is accepted.
@@ -132,46 +133,60 @@
// host networking on Mac)
.add("--env=DOCKER_MAC_CONTAINER=" + System.getenv("DOCKER_MAC_CONTAINER"));
+ Boolean retainDockerContainer =
+ pipelineOptions.as(ManualDockerEnvironmentOptions.class).getRetainDockerContainers();
if (!retainDockerContainer) {
- dockerArgsBuilder.add("--rm");
+ dockerOptsBuilder.add("--rm");
}
- List<String> args =
- ImmutableList.of(
- String.format("--id=%s", workerId),
- String.format("--logging_endpoint=%s", loggingEndpoint),
- String.format("--artifact_endpoint=%s", artifactEndpoint),
- String.format("--provision_endpoint=%s", provisionEndpoint),
- String.format("--control_endpoint=%s", controlEndpoint));
+ String semiPersistDir = pipelineOptions.as(RemoteEnvironmentOptions.class).getSemiPersistDir();
+ ImmutableList.Builder<String> argsBuilder =
+ ImmutableList.<String>builder()
+ .add(String.format("--id=%s", workerId))
+ .add(String.format("--logging_endpoint=%s", loggingEndpoint))
+ .add(String.format("--artifact_endpoint=%s", artifactEndpoint))
+ .add(String.format("--provision_endpoint=%s", provisionEndpoint))
+ .add(String.format("--control_endpoint=%s", controlEndpoint));
+ if (semiPersistDir != null) {
+ argsBuilder.add(String.format("--semi_persist_dir=%s", semiPersistDir));
+ }
LOG.debug("Creating Docker Container with ID {}", workerId);
// Wrap the blocking call to clientSource.get in case an exception is thrown.
String containerId = null;
InstructionRequestHandler instructionHandler = null;
try {
- containerId = docker.runImage(containerImage, dockerArgsBuilder.build(), args);
+ containerId = docker.runImage(containerImage, dockerOptsBuilder.build(), argsBuilder.build());
LOG.debug("Created Docker Container with Container ID {}", containerId);
// Wait on a client from the gRPC server.
- while (instructionHandler == null) {
+ try {
+ instructionHandler = clientSource.take(workerId, Duration.ofMinutes(1));
+ } catch (TimeoutException timeoutEx) {
+ RuntimeException runtimeException =
+ new RuntimeException(
+ String.format(
+ "Docker container %s failed to start up successfully within 1 minute.",
+ containerImage),
+ timeoutEx);
try {
- instructionHandler = clientSource.take(workerId, Duration.ofMinutes(1));
- } catch (TimeoutException timeoutEx) {
- Preconditions.checkArgument(
- docker.isContainerRunning(containerId), "No container running for id " + containerId);
- LOG.info(
- "Still waiting for startup of environment {} for worker id {}",
- dockerPayload.getContainerImage(),
- workerId);
- } catch (InterruptedException interruptEx) {
- Thread.currentThread().interrupt();
- throw new RuntimeException(interruptEx);
+ String containerLogs = docker.getContainerLogs(containerId);
+ LOG.error("Docker container {} logs:\n{}", containerId, containerLogs);
+ } catch (Exception getLogsException) {
+ runtimeException.addSuppressed(getLogsException);
}
+ throw runtimeException;
+ } catch (InterruptedException interruptEx) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(interruptEx);
}
} catch (Exception e) {
if (containerId != null) {
// Kill the launched docker container if we can't retrieve a client for it.
try {
docker.killContainer(containerId);
+ if (!retainDockerContainer) {
+ docker.removeContainer(containerId);
+ }
} catch (Exception dockerException) {
e.addSuppressed(dockerException);
}
@@ -179,7 +194,8 @@
throw e;
}
- return DockerContainerEnvironment.create(docker, environment, containerId, instructionHandler);
+ return DockerContainerEnvironment.create(
+ docker, environment, containerId, instructionHandler, retainDockerContainer);
}
private List<String> gcsCredentialArgs() {
@@ -206,7 +222,7 @@
* hostname has historically changed between versions, so this is subject to breakages and will
* likely only support the latest version at any time.
*/
- private static class DockerOnMac {
+ static class DockerOnMac {
// TODO: This host name seems to change with every other Docker release. Do we attempt to keep
// up
// or attempt to document the supported Docker version(s)?
@@ -220,7 +236,7 @@
private static final int MAC_PORT_END = 8200;
private static final AtomicInteger MAC_PORT = new AtomicInteger(MAC_PORT_START);
- private static ServerFactory getServerFactory() {
+ static ServerFactory getServerFactory() {
ServerFactory.UrlFactory dockerUrlFactory =
(host, port) -> HostAndPort.fromParts(DOCKER_FOR_MAC_HOST, port).toString();
if (RUNNING_INSIDE_DOCKER_ON_MAC) {
@@ -239,11 +255,10 @@
/** Provider for DockerEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
- private final boolean retainDockerContainer;
+ private final PipelineOptions pipelineOptions;
public Provider(PipelineOptions options) {
- this.retainDockerContainer =
- options.as(ManualDockerEnvironmentOptions.class).getRetainDockerContainers();
+ this.pipelineOptions = options;
}
@Override
@@ -262,7 +277,7 @@
provisioningServiceServer,
clientPool.getSource(),
idGenerator,
- retainDockerContainer);
+ pipelineOptions);
}
@Override
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
index e2e6382..a696770 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EmbeddedEnvironmentFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.environment;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java
index 9c633ac..822e0c3 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ExternalEnvironmentFactory.java
@@ -25,6 +25,7 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.runners.core.construction.BeamUrns;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.control.ControlClientPool;
import org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
@@ -33,7 +34,7 @@
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -41,6 +42,9 @@
public class ExternalEnvironmentFactory implements EnvironmentFactory {
private static final Logger LOG = LoggerFactory.getLogger(ExternalEnvironmentFactory.class);
+ // setting the environment variable allows to connect to worker pool running in Docker on Mac
+ private static final boolean IS_WORKER_POOL_IN_DOCKER_VM =
+ System.getenv().containsKey("BEAM_WORKER_POOL_IN_DOCKER_VM");
public static ExternalEnvironmentFactory create(
GrpcFnServer<FnApiControlClientPoolService> controlServiceServer,
@@ -92,8 +96,8 @@
RunnerApi.ExternalPayload.parseFrom(environment.getPayload());
final String workerId = idGenerator.getId();
- BeamFnApi.NotifyRunnerAvailableRequest notifyRunnerAvailableRequest =
- BeamFnApi.NotifyRunnerAvailableRequest.newBuilder()
+ BeamFnApi.StartWorkerRequest startWorkerRequest =
+ BeamFnApi.StartWorkerRequest.newBuilder()
.setWorkerId(workerId)
.setControlEndpoint(controlServiceServer.getApiServiceDescriptor())
.setLoggingEndpoint(loggingServiceServer.getApiServiceDescriptor())
@@ -103,12 +107,12 @@
.build();
LOG.debug("Requesting worker ID {}", workerId);
- BeamFnApi.NotifyRunnerAvailableResponse notifyRunnerAvailableResponse =
+ BeamFnApi.StartWorkerResponse startWorkerResponse =
BeamFnExternalWorkerPoolGrpc.newBlockingStub(
ManagedChannelFactory.createDefault().forDescriptor(externalPayload.getEndpoint()))
- .notifyRunnerAvailable(notifyRunnerAvailableRequest);
- if (!notifyRunnerAvailableResponse.getError().isEmpty()) {
- throw new RuntimeException(notifyRunnerAvailableResponse.getError());
+ .startWorker(startWorkerRequest);
+ if (!startWorkerResponse.getError().isEmpty()) {
+ throw new RuntimeException(startWorkerResponse.getError());
}
// Wait on a client from the gRPC server.
@@ -138,6 +142,22 @@
public InstructionRequestHandler getInstructionRequestHandler() {
return finalInstructionHandler;
}
+
+ @Override
+ public void close() throws Exception {
+ finalInstructionHandler.close();
+ BeamFnApi.StopWorkerRequest stopWorkerRequest =
+ BeamFnApi.StopWorkerRequest.newBuilder().setWorkerId(workerId).build();
+ LOG.debug("Closing worker ID {}", workerId);
+ BeamFnApi.StopWorkerResponse stopWorkerResponse =
+ BeamFnExternalWorkerPoolGrpc.newBlockingStub(
+ ManagedChannelFactory.createDefault()
+ .forDescriptor(externalPayload.getEndpoint()))
+ .stopWorker(stopWorkerRequest);
+ if (!stopWorkerResponse.getError().isEmpty()) {
+ throw new RuntimeException(stopWorkerResponse.getError());
+ }
+ }
};
}
@@ -159,5 +179,13 @@
clientPool.getSource(),
idGenerator);
}
+
+ @Override
+ public ServerFactory getServerFactory() {
+ if (IS_WORKER_POOL_IN_DOCKER_VM) {
+ return DockerEnvironmentFactory.DockerOnMac.getServerFactory();
+ }
+ return ServerFactory.createDefault();
+ }
}
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
index bb3de8c..a90a245 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactory.java
@@ -30,8 +30,10 @@
import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -50,7 +52,8 @@
GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer,
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
ControlClientPool.Source clientSource,
- IdGenerator idGenerator) {
+ IdGenerator idGenerator,
+ PipelineOptions pipelineOptions) {
return new ProcessEnvironmentFactory(
processManager,
controlServiceServer,
@@ -58,7 +61,8 @@
retrievalServiceServer,
provisioningServiceServer,
idGenerator,
- clientSource);
+ clientSource,
+ pipelineOptions);
}
private final ProcessManager processManager;
@@ -68,6 +72,7 @@
private final GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer;
private final IdGenerator idGenerator;
private final ControlClientPool.Source clientSource;
+ private final PipelineOptions pipelineOptions;
private ProcessEnvironmentFactory(
ProcessManager processManager,
@@ -76,7 +81,8 @@
GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer,
GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer,
IdGenerator idGenerator,
- ControlClientPool.Source clientSource) {
+ ControlClientPool.Source clientSource,
+ PipelineOptions pipelineOptions) {
this.processManager = processManager;
this.controlServiceServer = controlServiceServer;
this.loggingServiceServer = loggingServiceServer;
@@ -84,6 +90,7 @@
this.provisioningServiceServer = provisioningServiceServer;
this.idGenerator = idGenerator;
this.clientSource = clientSource;
+ this.pipelineOptions = pipelineOptions;
}
/** Creates a new, active {@link RemoteEnvironment} backed by a forked process. */
@@ -104,20 +111,25 @@
String provisionEndpoint = provisioningServiceServer.getApiServiceDescriptor().getUrl();
String controlEndpoint = controlServiceServer.getApiServiceDescriptor().getUrl();
- ImmutableList<String> args =
- ImmutableList.of(
- String.format("--id=%s", workerId),
- String.format("--logging_endpoint=%s", loggingEndpoint),
- String.format("--artifact_endpoint=%s", artifactEndpoint),
- String.format("--provision_endpoint=%s", provisionEndpoint),
- String.format("--control_endpoint=%s", controlEndpoint));
+ String semiPersistDir = pipelineOptions.as(RemoteEnvironmentOptions.class).getSemiPersistDir();
+ ImmutableList.Builder<String> argsBuilder =
+ ImmutableList.<String>builder()
+ .add(String.format("--id=%s", workerId))
+ .add(String.format("--logging_endpoint=%s", loggingEndpoint))
+ .add(String.format("--artifact_endpoint=%s", artifactEndpoint))
+ .add(String.format("--provision_endpoint=%s", provisionEndpoint))
+ .add(String.format("--control_endpoint=%s", controlEndpoint));
+ if (semiPersistDir != null) {
+ argsBuilder.add(String.format("--semi_persist_dir=%s", semiPersistDir));
+ }
LOG.debug("Creating Process for worker ID {}", workerId);
// Wrap the blocking call to clientSource.get in case an exception is thrown.
InstructionRequestHandler instructionHandler = null;
try {
ProcessManager.RunningProcess process =
- processManager.startProcess(workerId, executable, args, processPayload.getEnvMap());
+ processManager.startProcess(
+ workerId, executable, argsBuilder.build(), processPayload.getEnvMap());
// Wait on a client from the gRPC server.
while (instructionHandler == null) {
try {
@@ -148,6 +160,12 @@
/** Provider of ProcessEnvironmentFactory. */
public static class Provider implements EnvironmentFactory.Provider {
+ private final PipelineOptions pipelineOptions;
+
+ public Provider(PipelineOptions options) {
+ this.pipelineOptions = options;
+ }
+
@Override
public EnvironmentFactory createEnvironmentFactory(
GrpcFnServer<FnApiControlClientPoolService> controlServiceServer,
@@ -163,7 +181,8 @@
retrievalServiceServer,
provisioningServiceServer,
clientPool.getSource(),
- idGenerator);
+ idGenerator,
+ pipelineOptions);
}
}
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
index b22aa2e..e5864f9 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.environment;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.File;
@@ -28,8 +28,8 @@
import java.util.List;
import java.util.Map;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
index 642d3bd..9ef3f06 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
@@ -17,17 +17,25 @@
*/
package org.apache.beam.runners.fnexecution.jobsubmission;
+import java.util.ArrayList;
+import java.util.List;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.function.Consumer;
import java.util.function.Function;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
import org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsRequest;
import org.apache.beam.model.jobmanagement.v1.JobApi.DescribePipelineOptionsResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobPipelineResponse;
import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobsResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobInfo;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
@@ -38,16 +46,17 @@
import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.construction.graph.PipelineValidator;
import org.apache.beam.runners.fnexecution.FnService;
import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
import org.apache.beam.sdk.function.ThrowingConsumer;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -216,6 +225,24 @@
}
@Override
+ public void getJobs(GetJobsRequest request, StreamObserver<GetJobsResponse> responseObserver) {
+ LOG.trace("{} {}", GetJobsRequest.class.getSimpleName(), request);
+
+ try {
+ List<JobInfo> result = new ArrayList<>();
+ for (JobInvocation invocation : invocations.values()) {
+ result.add(invocation.toProto());
+ }
+ GetJobsResponse response = GetJobsResponse.newBuilder().addAllJobInfo(result).build();
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ } catch (Exception e) {
+ LOG.error("Encountered Unexpected Exception", e);
+ responseObserver.onError(Status.INTERNAL.withCause(e).asException());
+ }
+ }
+
+ @Override
public void getState(
GetJobStateRequest request, StreamObserver<GetJobStateResponse> responseObserver) {
LOG.trace("{} {}", GetJobStateRequest.class.getSimpleName(), request);
@@ -226,6 +253,30 @@
GetJobStateResponse response = GetJobStateResponse.newBuilder().setState(state).build();
responseObserver.onNext(response);
responseObserver.onCompleted();
+ } catch (StatusRuntimeException | StatusException e) {
+ responseObserver.onError(e);
+ } catch (Exception e) {
+ String errMessage =
+ String.format("Encountered Unexpected Exception for Invocation %s", invocationId);
+ LOG.error(errMessage, e);
+ responseObserver.onError(Status.INTERNAL.withCause(e).asException());
+ }
+ }
+
+ @Override
+ public void getPipeline(
+ GetJobPipelineRequest request, StreamObserver<GetJobPipelineResponse> responseObserver) {
+ LOG.trace("{} {}", GetJobPipelineRequest.class.getSimpleName(), request);
+ String invocationId = request.getJobId();
+ try {
+ JobInvocation invocation = getInvocation(invocationId);
+ RunnerApi.Pipeline pipeline = invocation.getPipeline();
+ GetJobPipelineResponse response =
+ GetJobPipelineResponse.newBuilder().setPipeline(pipeline).build();
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ } catch (StatusRuntimeException | StatusException e) {
+ responseObserver.onError(e);
} catch (Exception e) {
String errMessage =
String.format("Encountered Unexpected Exception for Invocation %s", invocationId);
@@ -245,6 +296,8 @@
CancelJobResponse response = CancelJobResponse.newBuilder().setState(state).build();
responseObserver.onNext(response);
responseObserver.onCompleted();
+ } catch (StatusRuntimeException | StatusException e) {
+ responseObserver.onError(e);
} catch (Exception e) {
String errMessage =
String.format("Encountered Unexpected Exception for Invocation %s", invocationId);
@@ -268,6 +321,8 @@
}
};
invocation.addStateListener(stateListener);
+ } catch (StatusRuntimeException | StatusException e) {
+ responseObserver.onError(e);
} catch (Exception e) {
String errMessage =
String.format("Encountered Unexpected Exception for Invocation %s", invocationId);
@@ -303,6 +358,8 @@
invocation.addStateListener(stateListener);
invocation.addMessageListener(messageListener);
+ } catch (StatusRuntimeException | StatusException e) {
+ responseObserver.onError(e);
} catch (Exception e) {
String errMessage =
String.format("Encountered Unexpected Exception for Invocation %s", invocationId);
@@ -312,6 +369,31 @@
}
@Override
+ public void getJobMetrics(
+ JobApi.GetJobMetricsRequest request,
+ StreamObserver<JobApi.GetJobMetricsResponse> responseObserver) {
+
+ String invocationId = request.getJobId();
+ LOG.info("Getting job metrics for {}", invocationId);
+
+ try {
+ JobInvocation invocation = getInvocation(invocationId);
+ JobApi.MetricResults metrics = invocation.getMetrics();
+ JobApi.GetJobMetricsResponse response =
+ JobApi.GetJobMetricsResponse.newBuilder().setMetrics(metrics).build();
+
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+ } catch (StatusRuntimeException | StatusException e) {
+ responseObserver.onError(e);
+ } catch (Exception e) {
+ LOG.error(String.format("Encountered exception for job invocation %s", invocationId), e);
+ responseObserver.onError(Status.INTERNAL.withCause(e).asException());
+ }
+ LOG.info("Finished getting job metrics for {}", invocationId);
+ }
+
+ @Override
public void describePipelineOptions(
DescribePipelineOptionsRequest request,
StreamObserver<DescribePipelineOptionsResponse> responseObserver) {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocation.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocation.java
index d32aef4..12acfa1 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocation.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocation.java
@@ -17,15 +17,17 @@
*/
package org.apache.beam.runners.fnexecution.jobsubmission;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getRootCause;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getRootCause;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.CancellationException;
import java.util.function.Consumer;
+import javax.annotation.Nonnull;
import javax.annotation.Nullable;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobState.Enum;
@@ -33,10 +35,10 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.FutureCallback;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListenableFuture;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.FutureCallback;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListenableFuture;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -52,7 +54,9 @@
private List<Consumer<Enum>> stateObservers;
private List<Consumer<JobMessage>> messageObservers;
private JobState.Enum jobState;
- @Nullable private ListenableFuture<PipelineResult> invocationFuture;
+ private JobApi.MetricResults metrics;
+ private PortablePipelineResult resultHandle;
+ @Nullable private ListenableFuture<PortablePipelineResult> invocationFuture;
public JobInvocation(
JobInfo jobInfo,
@@ -67,9 +71,10 @@
this.messageObservers = new ArrayList<>();
this.invocationFuture = null;
this.jobState = JobState.Enum.STOPPED;
+ this.metrics = JobApi.MetricResults.newBuilder().build();
}
- private PipelineResult runPipeline() throws Exception {
+ private PortablePipelineResult runPipeline() throws Exception {
return pipelineRunner.run(pipeline, jobInfo);
}
@@ -85,21 +90,46 @@
setState(JobState.Enum.RUNNING);
Futures.addCallback(
invocationFuture,
- new FutureCallback<PipelineResult>() {
+ new FutureCallback<PortablePipelineResult>() {
@Override
- public void onSuccess(@Nullable PipelineResult pipelineResult) {
+ public void onSuccess(PortablePipelineResult pipelineResult) {
if (pipelineResult != null) {
- checkArgument(
- pipelineResult.getState() == PipelineResult.State.DONE,
- "Success on non-Done state: " + pipelineResult.getState());
- setState(JobState.Enum.DONE);
+ PipelineResult.State state = pipelineResult.getState();
+
+ if (state.isTerminal()) {
+ metrics = pipelineResult.portableMetrics();
+ } else {
+ resultHandle = pipelineResult;
+ }
+
+ switch (state) {
+ case DONE:
+ setState(Enum.DONE);
+ break;
+ case RUNNING:
+ setState(Enum.RUNNING);
+ break;
+ case CANCELLED:
+ setState(Enum.CANCELLED);
+ break;
+ case FAILED:
+ setState(Enum.FAILED);
+ break;
+ default:
+ setState(JobState.Enum.UNSPECIFIED);
+ }
} else {
setState(JobState.Enum.UNSPECIFIED);
}
}
@Override
- public void onFailure(Throwable throwable) {
+ public void onFailure(@Nonnull Throwable throwable) {
+ if (throwable instanceof CancellationException) {
+ // We have canceled execution, just update the job state
+ setState(JobState.Enum.CANCELLED);
+ return;
+ }
String message = String.format("Error during job invocation %s.", getId());
LOG.error(message, throwable);
sendMessage(
@@ -130,12 +160,15 @@
this.invocationFuture.cancel(true /* mayInterruptIfRunning */);
Futures.addCallback(
invocationFuture,
- new FutureCallback<PipelineResult>() {
+ new FutureCallback<PortablePipelineResult>() {
@Override
- public void onSuccess(@Nullable PipelineResult pipelineResult) {
- if (pipelineResult != null) {
+ public void onSuccess(PortablePipelineResult pipelineResult) {
+ // Do not cancel when we are already done.
+ if (pipelineResult != null
+ && pipelineResult.getState() != PipelineResult.State.DONE) {
try {
pipelineResult.cancel();
+ setState(JobState.Enum.CANCELLED);
} catch (IOException exn) {
throw new RuntimeException(exn);
}
@@ -149,11 +182,23 @@
}
}
+ public JobApi.MetricResults getMetrics() {
+ if (resultHandle != null) {
+ metrics = resultHandle.portableMetrics();
+ }
+ return metrics;
+ }
+
/** Retrieve the job's current state. */
public JobState.Enum getState() {
return this.jobState;
}
+ /** Retrieve the job's pipeline. */
+ public RunnerApi.Pipeline getPipeline() {
+ return this.pipeline;
+ }
+
/** Listen for job state changes with a {@link Consumer}. */
public synchronized void addStateListener(Consumer<JobState.Enum> stateStreamObserver) {
stateStreamObserver.accept(getState());
@@ -165,6 +210,16 @@
messageObservers.add(messageStreamObserver);
}
+ /** Convert to {@link JobApi.JobInfo}. */
+ public JobApi.JobInfo toProto() {
+ return JobApi.JobInfo.newBuilder()
+ .setJobId(jobInfo.jobId())
+ .setJobName(jobInfo.jobName())
+ .setPipelineOptions(jobInfo.pipelineOptions())
+ .setState(getState())
+ .build();
+ }
+
private synchronized void setState(JobState.Enum state) {
this.jobState = state;
for (Consumer<JobState.Enum> observer : stateObservers) {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvoker.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvoker.java
index 1ed74ad..7612d8b 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvoker.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvoker.java
@@ -22,10 +22,10 @@
import java.util.concurrent.ThreadFactory;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
/** Factory to create {@link JobInvocation} instances. */
public abstract class JobInvoker {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobPreparation.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobPreparation.java
index c18988c..a304093 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobPreparation.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobPreparation.java
@@ -19,7 +19,7 @@
import com.google.auto.value.AutoValue;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
/** A job that has been prepared, but not invoked. */
@AutoValue
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
index 53f4ab8..f8977ff 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
@@ -25,7 +25,7 @@
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.kohsuke.args4j.Option;
import org.kohsuke.args4j.spi.ExplicitBooleanOptionHandler;
import org.slf4j.Logger;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java
new file mode 100644
index 0000000..951a8cb
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java
@@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.StandardCharsets;
+import java.util.Enumeration;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import java.util.jar.Attributes;
+import java.util.jar.Attributes.Name;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+import java.util.jar.JarOutputStream;
+import java.util.jar.Manifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactChunk;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetArtifactRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestRequest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.PipelineResources;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.InProcessServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.MessageOrBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.commons.compress.utils.IOUtils;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PortablePipelineRunner} that bundles the input pipeline along with all dependencies,
+ * artifacts, etc. required to run the pipeline into a jar that can be executed later.
+ *
+ * <p>Each {@link PortablePipelineJarCreator} instance is not threadsafe; a new instance is expected
+ * to be constructed and {@link #run} once per job.
+ */
+public class PortablePipelineJarCreator implements PortablePipelineRunner {
+ private static final Logger LOG = LoggerFactory.getLogger(PortablePipelineJarCreator.class);
+
+ private final Class mainClass;
+
+ @VisibleForTesting JarOutputStream outputStream;
+ /** Wrapper over {@link #outputStream}. */
+ @VisibleForTesting WritableByteChannel outputChannel;
+
+ public PortablePipelineJarCreator(Class mainClass) {
+ this.mainClass = mainClass;
+ }
+
+ /**
+ * <em>Does not actually run the pipeline.</em> Instead bundles the input pipeline along with all
+ * dependencies, artifacts, etc. required to run the pipeline into a jar that can be executed
+ * later.
+ */
+ @Override
+ public PortablePipelineResult run(Pipeline pipeline, JobInfo jobInfo) throws Exception {
+ PortablePipelineOptions pipelineOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
+ .as(PortablePipelineOptions.class);
+
+ final String jobName = jobInfo.jobName();
+ File outputFile = new File(pipelineOptions.getOutputExecutablePath());
+ LOG.info("Creating jar {} for job {}", outputFile.getAbsolutePath(), jobName);
+ outputStream =
+ new JarOutputStream(new FileOutputStream(outputFile), createManifest(mainClass, jobName));
+ outputChannel = Channels.newChannel(outputStream);
+ PortablePipelineJarUtils.writeDefaultJobName(outputStream, jobName);
+ writeClassPathResources(mainClass.getClassLoader());
+ writeAsJson(pipeline, PortablePipelineJarUtils.getPipelineUri(jobName));
+ writeAsJson(
+ PipelineOptionsTranslation.toProto(pipelineOptions),
+ PortablePipelineJarUtils.getPipelineOptionsUri(jobName));
+ writeArtifacts(jobInfo.retrievalToken(), jobName);
+ // Closing the channel also closes the underlying stream.
+ outputChannel.close();
+
+ LOG.info("Jar {} created successfully.", outputFile.getAbsolutePath());
+ return new JarCreatorPipelineResult();
+ }
+
+ @VisibleForTesting
+ Manifest createManifest(Class mainClass, String defaultJobName) {
+ Manifest manifest = new Manifest();
+ manifest.getMainAttributes().put(Attributes.Name.MANIFEST_VERSION, "1.0");
+ boolean classHasMainMethod = false;
+ try {
+ Class returnType = mainClass.getMethod("main", String[].class).getReturnType();
+ if (returnType == Void.TYPE) {
+ classHasMainMethod = true;
+ } else {
+ LOG.warn(
+ "No Main-Class will be set in jar because main method in {} returns {}, expected void",
+ mainClass,
+ returnType);
+ }
+ } catch (NoSuchMethodException e) {
+ LOG.warn("No Main-Class will be set in jar because {} lacks a main method.", mainClass);
+ }
+ if (classHasMainMethod) {
+ manifest.getMainAttributes().put(Name.MAIN_CLASS, mainClass.getName());
+ }
+ return manifest;
+ }
+
+ /** Copy resources from {@code classLoader} to {@link #outputStream}. */
+ private void writeClassPathResources(ClassLoader classLoader) throws IOException {
+ List<String> classPathResources =
+ PipelineResources.detectClassPathResourcesToStage(classLoader);
+ Preconditions.checkArgument(
+ classPathResources.size() == 1, "Expected exactly one jar on " + classLoader.toString());
+ copyResourcesFromJar(new JarFile(classPathResources.get(0)));
+ }
+
+ /** Copy resources from {@code inputJar} to {@link #outputStream}. */
+ @VisibleForTesting
+ protected void copyResourcesFromJar(JarFile inputJar) throws IOException {
+ Enumeration<JarEntry> inputJarEntries = inputJar.entries();
+ // The zip spec allows multiple files with the same name; the Java zip libraries do not.
+ // Keep track of the files we've already written to filter out duplicates.
+ // Also, ignore the old manifest; we want to write our own.
+ Set<String> previousEntryNames = new HashSet<>(ImmutableList.of(JarFile.MANIFEST_NAME));
+ while (inputJarEntries.hasMoreElements()) {
+ JarEntry inputJarEntry = inputJarEntries.nextElement();
+ InputStream inputStream = inputJar.getInputStream(inputJarEntry);
+ String entryName = inputJarEntry.getName();
+ if (previousEntryNames.contains(entryName)) {
+ LOG.debug("Skipping duplicated file {}", entryName);
+ } else {
+ JarEntry outputJarEntry = new JarEntry(inputJarEntry);
+ outputStream.putNextEntry(outputJarEntry);
+ LOG.trace("Copying jar entry {}", inputJarEntry);
+ IOUtils.copy(inputStream, outputStream);
+ previousEntryNames.add(entryName);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ interface ArtifactRetriever {
+ GetManifestResponse getManifest(GetManifestRequest request);
+
+ Iterator<ArtifactChunk> getArtifact(GetArtifactRequest request);
+ }
+
+ /**
+ * Copy all artifacts retrievable via the {@link ArtifactRetrievalServiceBlockingStub} to the
+ * {@code outputStream}.
+ *
+ * @return A {@link ProxyManifest} pointing to the artifacts' location in the output jar.
+ */
+ @VisibleForTesting
+ ProxyManifest copyStagedArtifacts(
+ String retrievalToken, ArtifactRetriever retrievalServiceStub, String jobName)
+ throws IOException {
+ GetManifestRequest manifestRequest =
+ GetManifestRequest.newBuilder().setRetrievalToken(retrievalToken).build();
+ ArtifactApi.Manifest manifest = retrievalServiceStub.getManifest(manifestRequest).getManifest();
+ // Create a new proxy manifest to locate artifacts at jar runtime.
+ ProxyManifest.Builder proxyManifestBuilder = ProxyManifest.newBuilder().setManifest(manifest);
+ for (ArtifactMetadata artifact : manifest.getArtifactList()) {
+ String outputPath =
+ PortablePipelineJarUtils.getArtifactUri(jobName, UUID.randomUUID().toString());
+ LOG.trace("Copying artifact {} to {}", artifact.getName(), outputPath);
+ proxyManifestBuilder.addLocation(
+ Location.newBuilder().setName(artifact.getName()).setUri("/" + outputPath).build());
+ outputStream.putNextEntry(new JarEntry(outputPath));
+ GetArtifactRequest artifactRequest =
+ GetArtifactRequest.newBuilder()
+ .setRetrievalToken(retrievalToken)
+ .setName(artifact.getName())
+ .build();
+ Iterator<ArtifactChunk> artifactResponse = retrievalServiceStub.getArtifact(artifactRequest);
+ while (artifactResponse.hasNext()) {
+ artifactResponse.next().getData().writeTo(outputStream);
+ }
+ }
+ return proxyManifestBuilder.build();
+ }
+
+ /**
+ * Uses {@link BeamFileSystemArtifactRetrievalService} to fetch artifacts, then writes the
+ * artifacts to {@code outputStream}. Include a {@link ProxyManifest} to locate artifacts later.
+ */
+ private void writeArtifacts(String retrievalToken, String jobName) throws Exception {
+ try (GrpcFnServer artifactServer =
+ GrpcFnServer.allocatePortAndCreateFor(
+ BeamFileSystemArtifactRetrievalService.create(), InProcessServerFactory.create())) {
+ ManagedChannel grpcChannel =
+ InProcessManagedChannelFactory.create()
+ .forDescriptor(artifactServer.getApiServiceDescriptor());
+ ArtifactRetrievalServiceBlockingStub retrievalServiceStub =
+ ArtifactRetrievalServiceGrpc.newBlockingStub(grpcChannel);
+ ProxyManifest proxyManifest =
+ copyStagedArtifacts(
+ retrievalToken,
+ new ArtifactRetriever() {
+ @Override
+ public GetManifestResponse getManifest(GetManifestRequest request) {
+ return retrievalServiceStub.getManifest(request);
+ }
+
+ @Override
+ public Iterator<ArtifactChunk> getArtifact(GetArtifactRequest request) {
+ return retrievalServiceStub.getArtifact(request);
+ }
+ },
+ jobName);
+ writeAsJson(proxyManifest, PortablePipelineJarUtils.getArtifactManifestUri(jobName));
+ grpcChannel.shutdown();
+ }
+ }
+
+ /** Helper method for writing {@code message} in UTF-8 JSON format. */
+ private void writeAsJson(MessageOrBuilder message, String outputPath) throws IOException {
+ outputStream.putNextEntry(new JarEntry(outputPath));
+ outputChannel.write(StandardCharsets.UTF_8.encode(JsonFormat.printer().print(message)));
+ }
+
+ private static class JarCreatorPipelineResult implements PortablePipelineResult {
+
+ @Override
+ public State getState() {
+ return State.DONE;
+ }
+
+ @Override
+ public State cancel() {
+ return State.DONE;
+ }
+
+ @Override
+ public State waitUntilFinish(Duration duration) {
+ return State.DONE;
+ }
+
+ @Override
+ public State waitUntilFinish() {
+ return State.DONE;
+ }
+
+ @Override
+ public MetricResults metrics() {
+ throw new UnsupportedOperationException("Jar creation does not yield metrics.");
+ }
+
+ @Override
+ public JobApi.MetricResults portableMetrics() throws UnsupportedOperationException {
+ return JobApi.MetricResults.getDefaultInstance();
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java
new file mode 100644
index 0000000..291605a
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import com.fasterxml.jackson.core.JsonFactory;
+import com.fasterxml.jackson.core.JsonGenerator.Feature;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.jar.JarEntry;
+import java.util.jar.JarOutputStream;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message.Builder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.JsonFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Contains common code for writing and reading portable pipeline jars.
+ *
+ * <p>Jar layout:
+ *
+ * <ul>
+ * <li>META-INF/
+ * <ul>
+ * <li>MANIFEST.MF
+ * </ul>
+ * <li>BEAM-PIPELINE/
+ * <ul>
+ * <li>pipeline-manifest.json
+ * <li>[1st pipeline (default)]
+ * <ul>
+ * <li>pipeline.json
+ * <li>pipeline-options.json
+ * <li>artifact-manifest.json
+ * <li>artifacts/
+ * <ul>
+ * <li>...artifact files...
+ * </ul>
+ * </ul>
+ * <li>[nth pipeline]
+ * <ul>
+ * Same as above
+ * </ul>
+ * </ul>
+ * <li>...Java classes...
+ * </ul>
+ */
+public abstract class PortablePipelineJarUtils {
+ private static final String ARTIFACT_FOLDER = "artifacts";
+ private static final String PIPELINE_FOLDER = "BEAM-PIPELINE";
+ private static final String ARTIFACT_MANIFEST = "artifact-manifest.json";
+ private static final String PIPELINE = "pipeline.json";
+ private static final String PIPELINE_OPTIONS = "pipeline-options.json";
+ private static final String PIPELINE_MANIFEST = PIPELINE_FOLDER + "/pipeline-manifest.json";
+
+ private static final Logger LOG = LoggerFactory.getLogger(PortablePipelineJarUtils.class);
+ private static final ObjectMapper OBJECT_MAPPER =
+ new ObjectMapper(new JsonFactory().configure(Feature.AUTO_CLOSE_TARGET, false));
+
+ private static class PipelineManifest {
+ public String defaultJobName;
+ }
+
+ private static InputStream getResourceFromClassPath(String resourcePath) throws IOException {
+ InputStream inputStream =
+ PortablePipelineJarUtils.class.getClassLoader().getResourceAsStream(resourcePath);
+ if (inputStream == null) {
+ throw new FileNotFoundException(
+ String.format("Resource %s not found on classpath.", resourcePath));
+ }
+ return inputStream;
+ }
+
+ /** Populates {@code builder} using the JSON resource specified by {@code resourcePath}. */
+ private static void parseJsonResource(String resourcePath, Builder builder) throws IOException {
+ try (InputStream inputStream = getResourceFromClassPath(resourcePath)) {
+ String contents = new String(ByteStreams.toByteArray(inputStream), StandardCharsets.UTF_8);
+ JsonFormat.parser().merge(contents, builder);
+ }
+ }
+
+ public static Pipeline getPipelineFromClasspath(String jobName) throws IOException {
+ Pipeline.Builder builder = Pipeline.newBuilder();
+ parseJsonResource(getPipelineUri(jobName), builder);
+ return builder.build();
+ }
+
+ public static Struct getPipelineOptionsFromClasspath(String jobName) throws IOException {
+ Struct.Builder builder = Struct.newBuilder();
+ parseJsonResource(getPipelineOptionsUri(jobName), builder);
+ return builder.build();
+ }
+
+ public static String getArtifactManifestUri(String jobName) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + ARTIFACT_MANIFEST;
+ }
+
+ static String getPipelineUri(String jobName) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + PIPELINE;
+ }
+
+ static String getPipelineOptionsUri(String jobName) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + PIPELINE_OPTIONS;
+ }
+
+ static String getArtifactUri(String jobName, String artifactId) {
+ return PIPELINE_FOLDER + "/" + jobName + "/" + ARTIFACT_FOLDER + "/" + artifactId;
+ }
+
+ public static String getDefaultJobName() throws IOException {
+ try (InputStream inputStream = getResourceFromClassPath(PIPELINE_MANIFEST)) {
+ PipelineManifest pipelineManifest =
+ OBJECT_MAPPER.readValue(inputStream, PipelineManifest.class);
+ return pipelineManifest.defaultJobName;
+ }
+ }
+
+ public static void writeDefaultJobName(JarOutputStream outputStream, String jobName)
+ throws IOException {
+ outputStream.putNextEntry(new JarEntry(PIPELINE_MANIFEST));
+ PipelineManifest pipelineManifest = new PipelineManifest();
+ pipelineManifest.defaultJobName = jobName;
+ OBJECT_MAPPER.writeValue(outputStream, pipelineManifest);
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineResult.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineResult.java
new file mode 100644
index 0000000..845e40e
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineResult.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.annotations.Experimental;
+
+/** Result of a portable {@link PortablePipelineRunner#run(RunnerApi.Pipeline, JobInfo)}. */
+public interface PortablePipelineResult extends PipelineResult {
+
+ /**
+ * Returns the object to access monitoring infos from the pipeline.
+ *
+ * @throws UnsupportedOperationException if the runner doesn't support retrieving metrics.
+ */
+ @Experimental(Experimental.Kind.METRICS)
+ JobApi.MetricResults portableMetrics() throws UnsupportedOperationException;
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineRunner.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineRunner.java
index 08adee6..01a58f6 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineRunner.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineRunner.java
@@ -19,9 +19,8 @@
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.sdk.PipelineResult;
/** Runs a portable Beam pipeline on some execution engine. */
public interface PortablePipelineRunner {
- PipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo) throws Exception;
+ PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo) throws Exception;
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingService.java
index 2d2b189..a37a2f3 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingService.java
@@ -24,8 +24,8 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.LogControl;
import org.apache.beam.model.fnexecution.v1.BeamFnLoggingGrpc;
import org.apache.beam.runners.fnexecution.FnService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/JobInfo.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/JobInfo.java
index 7226a89..aea6bb3 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/JobInfo.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/JobInfo.java
@@ -20,7 +20,7 @@
import com.google.auto.value.AutoValue;
import java.io.Serializable;
import org.apache.beam.model.fnexecution.v1.ProvisionApi;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
/**
* A subset of {@link org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo} that
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionService.java
index fff8bd7..4fec80c 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionService.java
@@ -24,7 +24,7 @@
import org.apache.beam.model.fnexecution.v1.ProvisionServiceGrpc;
import org.apache.beam.model.fnexecution.v1.ProvisionServiceGrpc.ProvisionServiceImplBase;
import org.apache.beam.runners.fnexecution.FnService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A {@link ProvisionServiceImplBase provision service} that returns a static response to all calls.
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java
index c249012..38a885f 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.fnexecution.splittabledofn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.List;
@@ -43,9 +43,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.Timestamps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.Timestamps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
index 71d8c6a..9c72d81 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.state;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
@@ -28,8 +28,8 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
import org.apache.beam.runners.fnexecution.FnService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/** An implementation of the Beam Fn State service. */
public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase
@@ -125,7 +125,7 @@
@Override
public void onNext(StateRequest request) {
StateRequestHandler handler =
- requestHandlers.getOrDefault(request.getInstructionReference(), this::handlerNotFound);
+ requestHandlers.getOrDefault(request.getInstructionId(), this::handlerNotFound);
try {
CompletionStage<StateResponse.Builder> result = handler.handle(request);
result.whenComplete(
@@ -156,8 +156,7 @@
StateResponse.newBuilder()
.setError(
String.format(
- "Unknown process bundle instruction id '%s'",
- request.getInstructionReference())));
+ "Unknown process bundle instruction id '%s'", request.getInstructionId())));
return result;
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java
new file mode 100644
index 0000000..f840864
--- /dev/null
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.UUID;
+import org.apache.beam.runners.core.InMemoryStateInternals;
+import org.apache.beam.runners.core.StateInternals;
+import org.apache.beam.runners.core.StateNamespace;
+import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.StateTag;
+import org.apache.beam.runners.core.StateTags;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.state.BagState;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+
+/**
+ * Holds user state in memory. Only one key is active at a time due to the GroupReduceFunction being
+ * called once per key. Needs to be reset via {@code resetForNewKey()} before processing a new key.
+ */
+public class InMemoryBagUserStateFactory<K, V, W extends BoundedWindow>
+ implements StateRequestHandlers.BagUserStateHandlerFactory<K, V, W> {
+
+ private List<InMemorySingleKeyBagState> handlers;
+
+ public InMemoryBagUserStateFactory() {
+ handlers = new ArrayList<>();
+ }
+
+ @Override
+ public StateRequestHandlers.BagUserStateHandler<K, V, W> forUserState(
+ String pTransformId,
+ String userStateId,
+ Coder<K> keyCoder,
+ Coder<V> valueCoder,
+ Coder<W> windowCoder) {
+
+ InMemorySingleKeyBagState<K, V, W> bagUserStateHandler =
+ new InMemorySingleKeyBagState<>(userStateId, valueCoder, windowCoder);
+ handlers.add(bagUserStateHandler);
+
+ return bagUserStateHandler;
+ }
+
+ /** Prepares previous emitted state handlers for processing a new key. */
+ public void resetForNewKey() {
+ for (InMemorySingleKeyBagState stateBags : handlers) {
+ stateBags.reset();
+ }
+ }
+
+ static class InMemorySingleKeyBagState<K, V, W extends BoundedWindow>
+ implements StateRequestHandlers.BagUserStateHandler<K, V, W> {
+
+ private final StateTag<BagState<V>> stateTag;
+ private final Coder<W> windowCoder;
+ private final ByteString cacheToken;
+
+ /* Lazily initialized state internals upon first access */
+ private volatile StateInternals stateInternals;
+
+ InMemorySingleKeyBagState(String userStateId, Coder<V> valueCoder, Coder<W> windowCoder) {
+ this.windowCoder = windowCoder;
+ this.stateTag = StateTags.bag(userStateId, valueCoder);
+ this.cacheToken = ByteString.copyFrom(UUID.randomUUID().toString().getBytes(Charsets.UTF_8));
+ }
+
+ @Override
+ public Iterable<V> get(K key, W window) {
+ initStateInternals(key);
+ StateNamespace namespace = StateNamespaces.window(windowCoder, window);
+ BagState<V> bagState = stateInternals.state(namespace, stateTag);
+ return bagState.read();
+ }
+
+ @Override
+ public void append(K key, W window, Iterator<V> values) {
+ initStateInternals(key);
+ StateNamespace namespace = StateNamespaces.window(windowCoder, window);
+ BagState<V> bagState = stateInternals.state(namespace, stateTag);
+ while (values.hasNext()) {
+ bagState.add(values.next());
+ }
+ }
+
+ @Override
+ public void clear(K key, W window) {
+ initStateInternals(key);
+ StateNamespace namespace = StateNamespaces.window(windowCoder, window);
+ BagState<V> bagState = stateInternals.state(namespace, stateTag);
+ bagState.clear();
+ }
+
+ @Override
+ public Optional<ByteString> getCacheToken() {
+ return Optional.of(cacheToken);
+ }
+
+ private void initStateInternals(K key) {
+ if (stateInternals == null) {
+ stateInternals = InMemoryStateInternals.forKey(key);
+ }
+ }
+
+ void reset() {
+ stateInternals = null;
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
index d085893..1ca0313 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandler.java
@@ -17,7 +17,9 @@
*/
package org.apache.beam.runners.fnexecution.state;
+import java.util.Collections;
import java.util.concurrent.CompletionStage;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
@@ -37,6 +39,11 @@
*/
CompletionStage<StateResponse.Builder> handle(StateRequest request) throws Exception;
+ /** Retrieves a list of valid cache tokens. */
+ default Iterable<BeamFnApi.ProcessBundleRequest.CacheToken> getCacheTokens() {
+ return Collections.emptyList();
+ }
+
static StateRequestHandler unsupported() {
return request -> {
throw new UnsupportedOperationException(
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java
index 0656e30..4d8f816 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateRequestHandlers.java
@@ -17,17 +17,21 @@
*/
package org.apache.beam.runners.fnexecution.state;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.EnumMap;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.ConcurrentHashMap;
import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateAppendResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateClearResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateGetResponse;
@@ -44,8 +48,8 @@
import org.apache.beam.sdk.fn.stream.DataStreams.ElementDelimitedOutputStream;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.common.Reiterable;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf.ByteStringCoder;
/**
@@ -137,6 +141,11 @@
/** Clears the bag user state for the given key and window. */
void clear(K key, W window);
+
+ /** Returns the currently valid cache token. */
+ default Optional<ByteString> getCacheToken() {
+ return Optional.empty();
+ }
}
/**
@@ -145,8 +154,8 @@
* <p>Note that this factory should be thread safe.
*/
@ThreadSafe
- public interface BagUserStateHandlerFactory {
- <K, V, W extends BoundedWindow> BagUserStateHandler<K, V, W> forUserState(
+ public interface BagUserStateHandlerFactory<K, V, W extends BoundedWindow> {
+ BagUserStateHandler<K, V, W> forUserState(
String pTransformId,
String userStateId,
Coder<K> keyCoder,
@@ -154,21 +163,13 @@
Coder<W> windowCoder);
/** Throws a {@link UnsupportedOperationException} on the first access. */
- static BagUserStateHandlerFactory unsupported() {
- return new BagUserStateHandlerFactory() {
- @Override
- public <K, V, W extends BoundedWindow> BagUserStateHandler<K, V, W> forUserState(
- String pTransformId,
- String userStateId,
- Coder<K> keyCoder,
- Coder<V> valueCoder,
- Coder<W> windowCoder) {
- throw new UnsupportedOperationException(
- String.format(
- "The %s does not support handling sides inputs for PTransform %s with user state "
- + "id %s.",
- BagUserStateHandler.class.getSimpleName(), pTransformId, userStateId));
- }
+ static <K, V, W extends BoundedWindow> BagUserStateHandlerFactory<K, V, W> unsupported() {
+ return (pTransformId, userStateId, keyCoder, valueCoder, windowCoder) -> {
+ throw new UnsupportedOperationException(
+ String.format(
+ "The %s does not support handling sides inputs for PTransform %s with user state "
+ + "id %s.",
+ BagUserStateHandler.class.getSimpleName(), pTransformId, userStateId));
};
}
}
@@ -205,6 +206,19 @@
.handle(request);
}
+ @Override
+ public Iterable<BeamFnApi.ProcessBundleRequest.CacheToken> getCacheTokens() {
+ // Use loops here due to the horrible performance of Java Streams:
+ // https://medium.com/@milan.mimica/slow-like-a-stream-fast-like-a-loop-524f70391182
+ Set<BeamFnApi.ProcessBundleRequest.CacheToken> cacheTokens = new HashSet<>();
+ for (StateRequestHandler handler : handlers.values()) {
+ for (BeamFnApi.ProcessBundleRequest.CacheToken cacheToken : handler.getCacheTokens()) {
+ cacheTokens.add(cacheToken);
+ }
+ }
+ return cacheTokens;
+ }
+
private CompletionStage<StateResponse.Builder> handlerNotFound(StateRequest request) {
CompletableFuture<StateResponse.Builder> rval = new CompletableFuture<>();
rval.completeExceptionally(new IllegalStateException());
@@ -236,14 +250,14 @@
private final Map<String, Map<String, SideInputSpec>> sideInputSpecs;
private final SideInputHandlerFactory sideInputHandlerFactory;
- private final ConcurrentHashMap<SideInputSpec, SideInputHandler> cache;
+ private final ConcurrentHashMap<SideInputSpec, SideInputHandler> handlerCache;
StateRequestHandlerToSideInputHandlerFactoryAdapter(
Map<String, Map<String, SideInputSpec>> sideInputSpecs,
SideInputHandlerFactory sideInputHandlerFactory) {
this.sideInputSpecs = sideInputSpecs;
this.sideInputHandlerFactory = sideInputHandlerFactory;
- this.cache = new ConcurrentHashMap<>();
+ this.handlerCache = new ConcurrentHashMap<>();
}
@Override
@@ -258,8 +272,9 @@
StateKey.MultimapSideInput stateKey = request.getStateKey().getMultimapSideInput();
SideInputSpec<?, ?, ?> referenceSpec =
- sideInputSpecs.get(stateKey.getPtransformId()).get(stateKey.getSideInputId());
- SideInputHandler<?, ?> handler = cache.computeIfAbsent(referenceSpec, this::createHandler);
+ sideInputSpecs.get(stateKey.getTransformId()).get(stateKey.getSideInputId());
+ SideInputHandler<?, ?> handler =
+ handlerCache.computeIfAbsent(referenceSpec, this::createHandler);
switch (request.getRequestCase()) {
case GET:
@@ -289,7 +304,7 @@
StateKey.MultimapSideInput stateKey = request.getStateKey().getMultimapSideInput();
SideInputSpec<K, V, W> sideInputReferenceSpec =
- sideInputSpecs.get(stateKey.getPtransformId()).get(stateKey.getSideInputId());
+ sideInputSpecs.get(stateKey.getTransformId()).get(stateKey.getSideInputId());
W window = sideInputReferenceSpec.windowCoder().decode(stateKey.getWindow().newInput());
@@ -347,14 +362,14 @@
private final ExecutableProcessBundleDescriptor processBundleDescriptor;
private final BagUserStateHandlerFactory handlerFactory;
- private final ConcurrentHashMap<BagUserStateSpec, BagUserStateHandler> cache;
+ private final ConcurrentHashMap<BagUserStateSpec, BagUserStateHandler> handlerCache;
ByteStringStateRequestHandlerToBagUserStateHandlerFactoryAdapter(
ExecutableProcessBundleDescriptor processBundleDescriptor,
BagUserStateHandlerFactory handlerFactory) {
this.processBundleDescriptor = processBundleDescriptor;
this.handlerFactory = handlerFactory;
- this.cache = new ConcurrentHashMap<>();
+ this.handlerCache = new ConcurrentHashMap<>();
}
@Override
@@ -371,7 +386,7 @@
BagUserStateSpec<Object, Object, BoundedWindow> referenceSpec =
processBundleDescriptor
.getBagUserStateSpecs()
- .get(stateKey.getPtransformId())
+ .get(stateKey.getTransformId())
.get(stateKey.getUserStateId());
// Note that by using the ByteStringCoder, we simplify the issue of encoding/decoding the
@@ -390,7 +405,7 @@
ByteStringCoder.class.getSimpleName());
BagUserStateHandler<ByteString, ByteString, BoundedWindow> handler =
- cache.computeIfAbsent(referenceSpec, this::createHandler);
+ handlerCache.computeIfAbsent(referenceSpec, this::createHandler);
ByteString key = stateKey.getKey();
BoundedWindow window = referenceSpec.windowCoder().decode(stateKey.getWindow().newInput());
@@ -414,6 +429,24 @@
}
}
+ @Override
+ public Iterable<BeamFnApi.ProcessBundleRequest.CacheToken> getCacheTokens() {
+ // Use a loop here due to the horrible performance of Java Streams:
+ // https://medium.com/@milan.mimica/slow-like-a-stream-fast-like-a-loop-524f70391182
+ Set<BeamFnApi.ProcessBundleRequest.CacheToken> cacheTokens = new HashSet<>();
+ for (BagUserStateHandler handler : handlerCache.values()) {
+ if (handler.getCacheToken().isPresent()) {
+ cacheTokens.add(
+ BeamFnApi.ProcessBundleRequest.CacheToken.newBuilder()
+ .setUserState(
+ BeamFnApi.ProcessBundleRequest.CacheToken.UserState.getDefaultInstance())
+ .setToken((ByteString) handler.getCacheToken().get())
+ .build());
+ }
+ }
+ return cacheTokens;
+ }
+
private static <W extends BoundedWindow>
CompletionStage<StateResponse.Builder> handleGetRequest(
StateRequest request,
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/BatchSideInputHandlerFactory.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/BatchSideInputHandlerFactory.java
index 8ad6bbf..1f5a6c65 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/BatchSideInputHandlerFactory.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/BatchSideInputHandlerFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.ByteArrayInputStream;
@@ -39,9 +39,9 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
/** {@link StateRequestHandler} that uses a {@link SideInputGetter} to access side inputs. */
public class BatchSideInputHandlerFactory implements SideInputHandlerFactory {
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtils.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtils.java
index a0ea37e..6e6c80a 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtils.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtils.java
@@ -17,23 +17,35 @@
*/
package org.apache.beam.runners.fnexecution.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.Collection;
+import java.util.Collections;
+import java.util.function.BiConsumer;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.runners.core.InMemoryTimerInternals;
+import org.apache.beam.runners.core.StateNamespace;
+import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.TimerInternals;
import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.Timer;
import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
import org.apache.beam.runners.core.construction.graph.PipelineNode;
import org.apache.beam.runners.fnexecution.wire.WireCoders;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.joda.time.Instant;
/** Utilities for pipeline translation. */
public final class PipelineTranslatorUtils {
@@ -89,4 +101,50 @@
return pCollecctions.stream()
.anyMatch(pc -> pc.getIsBounded() == RunnerApi.IsBounded.Enum.UNBOUNDED);
}
+
+ /**
+ * Fires all timers which are ready to be fired. This is done in a loop because timers may itself
+ * schedule timers.
+ */
+ public static void fireEligibleTimers(
+ InMemoryTimerInternals timerInternals,
+ BiConsumer<String, WindowedValue> timerConsumer,
+ Object currentTimerKey) {
+
+ boolean hasFired;
+ do {
+ hasFired = false;
+ TimerInternals.TimerData timer;
+
+ while ((timer = timerInternals.removeNextEventTimer()) != null) {
+ hasFired = true;
+ fireTimer(timer, timerConsumer, currentTimerKey);
+ }
+ while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
+ hasFired = true;
+ fireTimer(timer, timerConsumer, currentTimerKey);
+ }
+ while ((timer = timerInternals.removeNextSynchronizedProcessingTimer()) != null) {
+ hasFired = true;
+ fireTimer(timer, timerConsumer, currentTimerKey);
+ }
+ } while (hasFired);
+ }
+
+ private static void fireTimer(
+ TimerInternals.TimerData timer,
+ BiConsumer<String, WindowedValue> timerConsumer,
+ Object currentTimerKey) {
+ StateNamespace namespace = timer.getNamespace();
+ Preconditions.checkArgument(namespace instanceof StateNamespaces.WindowNamespace);
+ BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
+ Instant timestamp = timer.getTimestamp();
+ WindowedValue<KV<Object, Timer>> timerValue =
+ WindowedValue.of(
+ KV.of(currentTimerKey, Timer.of(timestamp, new byte[0])),
+ timestamp,
+ Collections.singleton(window),
+ PaneInfo.NO_FIRING);
+ timerConsumer.accept(timer.getTimerId(), timerValue);
+ }
}
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCoders.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCoders.java
index 2190a8f..1e0d74d 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCoders.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCoders.java
@@ -48,7 +48,7 @@
public static String addLengthPrefixedCoder(
String coderId, RunnerApi.Components.Builder components, boolean replaceWithByteArrayCoder) {
String lengthPrefixedByteArrayCoderId = addLengthPrefixByteArrayCoder(components);
- String urn = components.getCodersOrThrow(coderId).getSpec().getSpec().getUrn();
+ String urn = components.getCodersOrThrow(coderId).getSpec().getUrn();
// We handle three cases:
// 1) the requested coder is already a length prefix coder. In this case we just honor the
@@ -84,11 +84,7 @@
private static String addWrappedWithLengthPrefixCoder(
String coderId, RunnerApi.Components.Builder components) {
Coder.Builder lengthPrefixed = Coder.newBuilder().addComponentCoderIds(coderId);
- lengthPrefixed
- .getSpecBuilder()
- .getSpecBuilder()
- .setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN)
- .build();
+ lengthPrefixed.getSpecBuilder().setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN).build();
return addCoder(lengthPrefixed.build(), components, coderId + "-length_prefix");
}
@@ -96,7 +92,7 @@
private static String addLengthPrefixByteArrayCoder(RunnerApi.Components.Builder components) {
// Add byte array coder
Coder.Builder byteArrayCoder = Coder.newBuilder();
- byteArrayCoder.getSpecBuilder().getSpecBuilder().setUrn(ModelCoders.BYTES_CODER_URN);
+ byteArrayCoder.getSpecBuilder().setUrn(ModelCoders.BYTES_CODER_URN);
String byteArrayCoderId = addCoder(byteArrayCoder.build(), components, "byte_array");
// Wrap it into length-prefixed coder
@@ -104,7 +100,6 @@
lengthPrefixByteArrayCoder
.addComponentCoderIds(byteArrayCoderId)
.getSpecBuilder()
- .getSpecBuilder()
.setUrn(ModelCoders.LENGTH_PREFIX_CODER_URN);
return addCoder(lengthPrefixByteArrayCoder.build(), components, "length_prefix_byte_array");
diff --git a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/WireCoders.java b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/WireCoders.java
index 433fe80..46d894a 100644
--- a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/WireCoders.java
+++ b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/WireCoders.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.wire;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import org.apache.beam.model.pipeline.v1.RunnerApi;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/EmbeddedSdkHarness.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/EmbeddedSdkHarness.java
index e7108f5..b7dd034 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/EmbeddedSdkHarness.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/EmbeddedSdkHarness.java
@@ -33,7 +33,7 @@
import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.junit.rules.ExternalResource;
import org.junit.rules.TestRule;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProviderTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProviderTest.java
index ad5f537..532e904 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProviderTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/GrpcContextHeaderAccessorProviderTest.java
@@ -23,17 +23,17 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements;
import org.apache.beam.model.fnexecution.v1.BeamFnDataGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.CallOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Channel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ClientCall;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ClientInterceptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingClientCall.SimpleForwardingClientCall;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.MethodDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.CallOptions;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Channel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ClientCall;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ClientInterceptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingClientCall.SimpleForwardingClientCall;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.MethodDescriptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
index a6d5311..0972d7b 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/ServerFactoryTest.java
@@ -42,14 +42,14 @@
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.Epoll;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.Epoll;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Test;
/** Tests for {@link ServerFactory}. */
@@ -57,11 +57,11 @@
private static final BeamFnApi.Elements CLIENT_DATA =
BeamFnApi.Elements.newBuilder()
- .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionReference("1"))
+ .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionId("1"))
.build();
private static final BeamFnApi.Elements SERVER_DATA =
BeamFnApi.Elements.newBuilder()
- .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionReference("1"))
+ .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionId("1"))
.build();
@Test
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactServicesTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactServicesTest.java
index 05dd45b..40807ca 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactServicesTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactServicesTest.java
@@ -58,15 +58,15 @@
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.InProcessServerFactory;
import org.apache.beam.sdk.io.FileSystems;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java
new file mode 100644
index 0000000..65d54a9
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java
@@ -0,0 +1,406 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.artifact;
+
+import java.io.ByteArrayOutputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.nio.charset.Charset;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipOutputStream;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.InProcessServerFactory;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link ClassLoaderArtifactRetrievalService} and {@link
+ * JavaFilesystemArtifactStagingService}.
+ */
+@RunWith(JUnit4.class)
+public class ClassLoaderArtifactServiceTest {
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private static final int ARTIFACT_CHUNK_SIZE = 100;
+
+ private static final Charset BIJECTIVE_CHARSET = Charsets.ISO_8859_1;
+
+ public interface ArtifactServicePair extends AutoCloseable {
+
+ String getStagingToken(String nonce);
+
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub createStagingStub() throws Exception;
+
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub createStagingBlockingStub()
+ throws Exception;
+
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub createRetrievalStub()
+ throws Exception;
+
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub createRetrievalBlockingStub()
+ throws Exception;
+ }
+
+ /**
+ * An ArtifactServicePair that loads artifacts into a jar file and then serves them up via a
+ * ClassLoader using out of that jar.
+ */
+ private ArtifactServicePair classLoaderService() throws IOException {
+ return new ArtifactServicePair() {
+
+ Path jarPath = Paths.get(tempFolder.newFile("jar.jar").getPath());
+
+ // These are initialized when the staging service is requested.
+ FileSystem jarFilesystem;
+ JavaFilesystemArtifactStagingService stagingService;
+ GrpcFnServer<JavaFilesystemArtifactStagingService> stagingServer;
+ ClassLoaderArtifactRetrievalService retrievalService;
+ GrpcFnServer<ClassLoaderArtifactRetrievalService> retrievalServer;
+
+ // These are initialized when the retrieval service is requested, closing the jar file
+ // created above.
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingStub;
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub stagingBlockingStub;
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub;
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub retrievalBlockingStub;
+
+ @Override
+ public void close() throws Exception {
+ if (stagingServer != null) {
+ stagingServer.close();
+ }
+ if (stagingService != null) {
+ stagingService.close();
+ }
+ if (retrievalServer != null) {
+ retrievalServer.close();
+ }
+ if (retrievalService != null) {
+ retrievalService.close();
+ }
+ }
+
+ @Override
+ public String getStagingToken(String nonce) {
+ return "/path/to/subdir" + nonce.hashCode();
+ }
+
+ private void startStagingService() throws Exception {
+ try (FileOutputStream fileOut = new FileOutputStream(jarPath.toString())) {
+ try (ZipOutputStream zipOut = new ZipOutputStream(fileOut)) {
+ ZipEntry zipEntry = new ZipEntry("someFile");
+ zipOut.putNextEntry(zipEntry);
+ zipOut.write(new byte[] {'s', 't', 'u', 'f', 'f'});
+ zipOut.closeEntry();
+ }
+ }
+ jarFilesystem =
+ FileSystems.newFileSystem(
+ URI.create("jar:file:" + jarPath.toString()), ImmutableMap.of());
+ JavaFilesystemArtifactStagingService stagingService =
+ new JavaFilesystemArtifactStagingService(jarFilesystem, "/path/to/root");
+ GrpcFnServer<JavaFilesystemArtifactStagingService> stagingServer =
+ GrpcFnServer.allocatePortAndCreateFor(stagingService, InProcessServerFactory.create());
+ ManagedChannel stagingChannel =
+ InProcessChannelBuilder.forName(stagingServer.getApiServiceDescriptor().getUrl())
+ .build();
+ stagingStub = ArtifactStagingServiceGrpc.newStub(stagingChannel);
+ stagingBlockingStub = ArtifactStagingServiceGrpc.newBlockingStub(stagingChannel);
+ }
+
+ @Override
+ public ArtifactStagingServiceGrpc.ArtifactStagingServiceStub createStagingStub()
+ throws Exception {
+ if (stagingStub == null) {
+ startStagingService();
+ }
+ return stagingStub;
+ }
+
+ @Override
+ public ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub
+ createStagingBlockingStub() throws Exception {
+ if (stagingBlockingStub == null) {
+ startStagingService();
+ }
+ return stagingBlockingStub;
+ }
+
+ public void startupRetrievalService() throws Exception {
+ jarFilesystem.close();
+ retrievalService =
+ new ClassLoaderArtifactRetrievalService(
+ new URLClassLoader(new URL[] {jarPath.toUri().toURL()}));
+ retrievalServer =
+ GrpcFnServer.allocatePortAndCreateFor(
+ retrievalService, InProcessServerFactory.create());
+ ManagedChannel retrievalChannel =
+ InProcessChannelBuilder.forName(retrievalServer.getApiServiceDescriptor().getUrl())
+ .build();
+ retrievalStub = ArtifactRetrievalServiceGrpc.newStub(retrievalChannel);
+ retrievalBlockingStub = ArtifactRetrievalServiceGrpc.newBlockingStub(retrievalChannel);
+ }
+
+ @Override
+ public ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub createRetrievalStub()
+ throws Exception {
+ if (retrievalStub == null) {
+ startupRetrievalService();
+ }
+ return retrievalStub;
+ }
+
+ @Override
+ public ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub
+ createRetrievalBlockingStub() throws Exception {
+ if (retrievalBlockingStub == null) {
+ startupRetrievalService();
+ }
+ return retrievalBlockingStub;
+ }
+ };
+ }
+
+ private ArtifactApi.ArtifactMetadata putArtifact(
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingStub,
+ String stagingSessionToken,
+ String name,
+ String contents)
+ throws InterruptedException, ExecutionException, TimeoutException {
+ ArtifactApi.ArtifactMetadata metadata =
+ ArtifactApi.ArtifactMetadata.newBuilder().setName(name).build();
+ CompletableFuture<Void> complete = new CompletableFuture<>();
+ StreamObserver<ArtifactApi.PutArtifactRequest> outputStreamObserver =
+ stagingStub.putArtifact(
+ new StreamObserver<ArtifactApi.PutArtifactResponse>() {
+
+ @Override
+ public void onNext(ArtifactApi.PutArtifactResponse putArtifactResponse) {
+ // Do nothing.
+ }
+
+ @Override
+ public void onError(Throwable th) {
+ complete.completeExceptionally(th);
+ }
+
+ @Override
+ public void onCompleted() {
+ complete.complete(null);
+ }
+ });
+ outputStreamObserver.onNext(
+ ArtifactApi.PutArtifactRequest.newBuilder()
+ .setMetadata(
+ ArtifactApi.PutArtifactMetadata.newBuilder()
+ .setMetadata(metadata)
+ .setStagingSessionToken(stagingSessionToken))
+ .build());
+
+ byte[] byteContents = contents.getBytes(BIJECTIVE_CHARSET);
+ for (int start = 0; start < byteContents.length; start += ARTIFACT_CHUNK_SIZE) {
+ outputStreamObserver.onNext(
+ ArtifactApi.PutArtifactRequest.newBuilder()
+ .setData(
+ ArtifactApi.ArtifactChunk.newBuilder()
+ .setData(
+ ByteString.copyFrom(
+ byteContents,
+ start,
+ Math.min(byteContents.length - start, ARTIFACT_CHUNK_SIZE)))
+ .build())
+ .build());
+ }
+ outputStreamObserver.onCompleted();
+ complete.get(10, TimeUnit.SECONDS);
+ return metadata;
+ }
+
+ private String commitManifest(
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub stagingStub,
+ String stagingToken,
+ List<ArtifactApi.ArtifactMetadata> artifacts) {
+ return stagingStub
+ .commitManifest(
+ ArtifactApi.CommitManifestRequest.newBuilder()
+ .setStagingSessionToken(stagingToken)
+ .setManifest(ArtifactApi.Manifest.newBuilder().addAllArtifact(artifacts))
+ .build())
+ .getRetrievalToken();
+ }
+
+ private String getArtifact(
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub,
+ String retrievalToken,
+ String name)
+ throws ExecutionException, InterruptedException {
+ CompletableFuture<String> result = new CompletableFuture<>();
+ retrievalStub.getArtifact(
+ ArtifactApi.GetArtifactRequest.newBuilder()
+ .setRetrievalToken(retrievalToken)
+ .setName(name)
+ .build(),
+ new StreamObserver<ArtifactApi.ArtifactChunk>() {
+
+ private ByteArrayOutputStream all = new ByteArrayOutputStream();
+
+ @Override
+ public void onNext(ArtifactApi.ArtifactChunk artifactChunk) {
+ try {
+ all.write(artifactChunk.getData().toByteArray());
+ } catch (IOException exn) {
+ Assert.fail("ByteArrayOutputStream threw exception: " + exn);
+ }
+ }
+
+ @Override
+ public void onError(Throwable th) {
+ result.completeExceptionally(th);
+ }
+
+ @Override
+ public void onCompleted() {
+ result.complete(new String(all.toByteArray(), BIJECTIVE_CHARSET));
+ }
+ });
+ return result.get();
+ }
+
+ private String stageArtifacts(
+ ArtifactServicePair service, String stagingToken, Map<String, String> artifacts)
+ throws Exception {
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingStub = service.createStagingStub();
+ ArtifactStagingServiceGrpc.ArtifactStagingServiceBlockingStub stagingBlockingStub =
+ service.createStagingBlockingStub();
+ List<ArtifactApi.ArtifactMetadata> artifactMetadatas = new ArrayList<>();
+ for (Map.Entry<String, String> entry : artifacts.entrySet()) {
+ artifactMetadatas.add(
+ putArtifact(stagingStub, stagingToken, entry.getKey(), entry.getValue()));
+ }
+ return commitManifest(stagingBlockingStub, stagingToken, artifactMetadatas);
+ }
+
+ private void checkArtifacts(
+ ArtifactServicePair service, String retrievalToken, Map<String, String> artifacts)
+ throws Exception {
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceStub retrievalStub =
+ service.createRetrievalStub();
+ ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceBlockingStub retrievalBlockingStub =
+ service.createRetrievalBlockingStub();
+ ArtifactApi.Manifest manifest =
+ retrievalBlockingStub
+ .getManifest(
+ ArtifactApi.GetManifestRequest.newBuilder()
+ .setRetrievalToken(retrievalToken)
+ .build())
+ .getManifest();
+ Assert.assertEquals(manifest.getArtifactCount(), artifacts.size());
+ for (ArtifactApi.ArtifactMetadata artifact : manifest.getArtifactList()) {
+ String contents = getArtifact(retrievalStub, retrievalToken, artifact.getName());
+ Assert.assertEquals(artifacts.get(artifact.getName()), contents);
+ }
+ }
+
+ private void runTest(ArtifactServicePair service, Map<String, String> artifacts)
+ throws Exception {
+ checkArtifacts(
+ service, stageArtifacts(service, service.getStagingToken("nonce"), artifacts), artifacts);
+ }
+
+ private Map<String, String> identityMap(String... keys) {
+ ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();
+ for (String key : keys) {
+ builder.put(key, key);
+ }
+ return builder.build();
+ }
+
+ @Test
+ public void testBasic() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ runTest(service, ImmutableMap.of("a", "Aa", "b", "Bbb", "c", "C"));
+ }
+ }
+
+ @Test
+ public void testOddFilenames() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ runTest(
+ service,
+ identityMap(
+ "some whitespace\n\t",
+ "some whitespace\n",
+ "nullTerminated\0",
+ "nullTerminated\0\0",
+ "../../../../../../../slashes",
+ "..\\..\\..\\..\\..\\..\\..\\backslashes",
+ "/private"));
+ }
+ }
+
+ @Test
+ public void testMultipleChunks() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ byte[] contents = new byte[ARTIFACT_CHUNK_SIZE * 9 / 2];
+ for (int i = 0; i < contents.length; i++) {
+ contents[i] = (byte) (i * i + Integer.MAX_VALUE / (i + 1));
+ }
+ runTest(service, ImmutableMap.of("filename", new String(contents, BIJECTIVE_CHARSET)));
+ }
+ }
+
+ @Test
+ public void testMultipleTokens() throws Exception {
+ try (ArtifactServicePair service = classLoaderService()) {
+ Map<String, String> artifacts1 = ImmutableMap.of("a", "a1", "b", "b");
+ Map<String, String> artifacts2 = ImmutableMap.of("a", "a2", "c", "c");
+ String token1 = stageArtifacts(service, service.getStagingToken("1"), artifacts1);
+ String token2 = stageArtifacts(service, service.getStagingToken("2"), artifacts2);
+ checkArtifacts(service, token1, artifacts1);
+ checkArtifacts(service, token2, artifacts2);
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java
new file mode 100644
index 0000000..3a1ed86
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory.WrappedContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link DefaultExecutableStageContext}. */
+@RunWith(JUnit4.class)
+public class DefaultExecutableStageContextTest {
+
+ @Test
+ public void testMultiInstanceFactory() {
+ JobInfo jobInfo =
+ JobInfo.create(
+ "multi-instance-factory-test",
+ "job-name",
+ "retrieval-token",
+ Struct.getDefaultInstance());
+ MultiInstanceFactory multiInstanceFactory = new MultiInstanceFactory(2, (x) -> true);
+
+ WrappedContext f1 = (WrappedContext) multiInstanceFactory.get(jobInfo);
+ WrappedContext f2 = (WrappedContext) multiInstanceFactory.get(jobInfo);
+ WrappedContext f3 = (WrappedContext) multiInstanceFactory.get(jobInfo);
+
+ Assert.assertNotEquals("We should create two different factories", f1.context, f2.context);
+ Assert.assertEquals(
+ "Future calls should be round-robbined to those two factories", f1.context, f3.context);
+ }
+
+ @Test
+ public void testDefault() {
+ JobInfo jobInfo =
+ JobInfo.create("default-test", "job-name", "retrieval-token", Struct.getDefaultInstance());
+ MultiInstanceFactory multiInstanceFactory = new MultiInstanceFactory(0, (x) -> true);
+ int expectedParallelism = Math.max(1, Runtime.getRuntime().availableProcessors() - 1);
+
+ WrappedContext f1 = (WrappedContext) multiInstanceFactory.get(jobInfo);
+ for (int i = 1; i < expectedParallelism; i++) {
+ Assert.assertNotEquals(
+ "We should create " + expectedParallelism + " different factories",
+ f1.context,
+ ((WrappedContext) multiInstanceFactory.get(jobInfo)).context);
+ }
+
+ Assert.assertEquals(
+ "Future calls should be round-robbined to those",
+ f1.context,
+ ((WrappedContext) multiInstanceFactory.get(jobInfo)).context);
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java
index 20c7578..4db1b64 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java
@@ -23,6 +23,7 @@
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;
+import java.util.Collections;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse;
@@ -33,9 +34,9 @@
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload;
import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
-import org.apache.beam.model.pipeline.v1.RunnerApi.SdkFunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.WindowingStrategy;
import org.apache.beam.runners.core.construction.ModelCoders;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.ServerFactory;
@@ -48,11 +49,16 @@
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateDelegator;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.fn.data.CloseableFnDataReceiver;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -90,6 +96,7 @@
IdGenerator idGenerator) -> envFactory;
private final Map<String, EnvironmentFactory.Provider> envFactoryProviderMap =
ImmutableMap.of(environment.getUrn(), envFactoryProvider);
+ private DefaultJobBundleFactory.ServerInfo serverInfo;
@Before
public void setUpMocks() throws Exception {
@@ -100,22 +107,30 @@
.thenReturn(CompletableFuture.completedFuture(instructionResponse));
when(dataServer.getApiServiceDescriptor())
.thenReturn(ApiServiceDescriptor.getDefaultInstance());
+ GrpcDataService dataService = mock(GrpcDataService.class);
+ when(dataService.send(any(), any())).thenReturn(mock(CloseableFnDataReceiver.class));
+ when(dataServer.getService()).thenReturn(dataService);
when(stateServer.getApiServiceDescriptor())
.thenReturn(ApiServiceDescriptor.getDefaultInstance());
+ GrpcStateService stateService = mock(GrpcStateService.class);
+ when(stateService.registerForProcessBundleInstructionId(any(), any()))
+ .thenReturn(mock(StateDelegator.Registration.class));
+ when(stateServer.getService()).thenReturn(stateService);
+ serverInfo =
+ new AutoValue_DefaultJobBundleFactory_ServerInfo.Builder()
+ .setControlServer(controlServer)
+ .setLoggingServer(loggingServer)
+ .setRetrievalServer(retrievalServer)
+ .setProvisioningServer(provisioningServer)
+ .setDataServer(dataServer)
+ .setStateServer(stateServer)
+ .build();
}
@Test
public void createsCorrectEnvironment() throws Exception {
try (DefaultJobBundleFactory bundleFactory =
- new DefaultJobBundleFactory(
- envFactoryProviderMap,
- stageIdGenerator,
- controlServer,
- loggingServer,
- retrievalServer,
- provisioningServer,
- dataServer,
- stateServer)) {
+ createDefaultJobBundleFactory(envFactoryProviderMap)) {
bundleFactory.forStage(getExecutableStage(environment));
verify(envFactory).createEnvironment(environment);
}
@@ -160,9 +175,7 @@
environmentA.getUrn(), environmentProviderFactoryA,
environmentB.getUrn(), environmentProviderFactoryB);
try (DefaultJobBundleFactory bundleFactory =
- DefaultJobBundleFactory.create(
- JobInfo.create("testJob", "testJob", "token", Struct.getDefaultInstance()),
- environmentFactoryProviderMap)) {
+ createDefaultJobBundleFactory(environmentFactoryProviderMap)) {
bundleFactory.forStage(getExecutableStage(environmentA));
verify(environmentProviderFactoryA, Mockito.times(1))
.createEnvironmentFactory(any(), any(), any(), any(), any(), any());
@@ -221,18 +234,50 @@
}
@Test
- public void closesEnvironmentOnCleanup() throws Exception {
- DefaultJobBundleFactory bundleFactory =
+ public void expiresEnvironment() throws Exception {
+ ServerFactory serverFactory = ServerFactory.createDefault();
+
+ Environment environmentA = Environment.newBuilder().setUrn("env:urn:a").build();
+ EnvironmentFactory envFactoryA = mock(EnvironmentFactory.class);
+ when(envFactoryA.createEnvironment(environmentA)).thenReturn(remoteEnvironment);
+ EnvironmentFactory.Provider environmentProviderFactoryA =
+ mock(EnvironmentFactory.Provider.class);
+ when(environmentProviderFactoryA.createEnvironmentFactory(
+ any(), any(), any(), any(), any(), any()))
+ .thenReturn(envFactoryA);
+ when(environmentProviderFactoryA.getServerFactory()).thenReturn(serverFactory);
+
+ Map<String, Provider> environmentFactoryProviderMap =
+ ImmutableMap.of(environmentA.getUrn(), environmentProviderFactoryA);
+
+ PortablePipelineOptions portableOptions =
+ PipelineOptionsFactory.as(PortablePipelineOptions.class);
+ portableOptions.setEnvironmentExpirationMillis(1);
+ Struct pipelineOptions = PipelineOptionsTranslation.toProto(portableOptions);
+
+ try (DefaultJobBundleFactory bundleFactory =
new DefaultJobBundleFactory(
- envFactoryProviderMap,
+ JobInfo.create("testJob", "testJob", "token", pipelineOptions),
+ environmentFactoryProviderMap,
stageIdGenerator,
- controlServer,
- loggingServer,
- retrievalServer,
- provisioningServer,
- dataServer,
- stateServer);
- try (AutoCloseable unused = bundleFactory) {
+ serverInfo)) {
+ OutputReceiverFactory orf = mock(OutputReceiverFactory.class);
+ StateRequestHandler srh = mock(StateRequestHandler.class);
+ when(srh.getCacheTokens()).thenReturn(Collections.emptyList());
+ StageBundleFactory sbf = bundleFactory.forStage(getExecutableStage(environmentA));
+ Thread.sleep(10); // allow environment to expire
+ sbf.getBundle(orf, srh, BundleProgressHandler.ignored()).close();
+ Thread.sleep(10); // allow environment to expire
+ sbf.getBundle(orf, srh, BundleProgressHandler.ignored()).close();
+ }
+ verify(envFactoryA, Mockito.times(3)).createEnvironment(environmentA);
+ verify(remoteEnvironment, Mockito.times(3)).close();
+ }
+
+ @Test
+ public void closesEnvironmentOnCleanup() throws Exception {
+ try (DefaultJobBundleFactory bundleFactory =
+ createDefaultJobBundleFactory(envFactoryProviderMap)) {
bundleFactory.forStage(getExecutableStage(environment));
}
verify(remoteEnvironment).close();
@@ -241,15 +286,7 @@
@Test
public void cachesEnvironment() throws Exception {
try (DefaultJobBundleFactory bundleFactory =
- new DefaultJobBundleFactory(
- envFactoryProviderMap,
- stageIdGenerator,
- controlServer,
- loggingServer,
- retrievalServer,
- provisioningServer,
- dataServer,
- stateServer)) {
+ createDefaultJobBundleFactory(envFactoryProviderMap)) {
StageBundleFactory bf1 = bundleFactory.forStage(getExecutableStage(environment));
StageBundleFactory bf2 = bundleFactory.forStage(getExecutableStage(environment));
// NOTE: We hang on to stage bundle references to ensure their underlying environments are not
@@ -277,15 +314,7 @@
.thenReturn(CompletableFuture.completedFuture(instructionResponse));
try (DefaultJobBundleFactory bundleFactory =
- new DefaultJobBundleFactory(
- envFactoryProviderMapFoo,
- stageIdGenerator,
- controlServer,
- loggingServer,
- retrievalServer,
- provisioningServer,
- dataServer,
- stateServer)) {
+ createDefaultJobBundleFactory(envFactoryProviderMapFoo)) {
bundleFactory.forStage(getExecutableStage(environment));
bundleFactory.forStage(getExecutableStage(envFoo));
verify(envFactory).createEnvironment(environment);
@@ -294,6 +323,15 @@
}
}
+ private DefaultJobBundleFactory createDefaultJobBundleFactory(
+ Map<String, EnvironmentFactory.Provider> envFactoryProviderMap) {
+ return new DefaultJobBundleFactory(
+ JobInfo.create("testJob", "testJob", "token", Struct.getDefaultInstance()),
+ envFactoryProviderMap,
+ stageIdGenerator,
+ serverInfo);
+ }
+
private static ExecutableStage getExecutableStage(Environment environment) {
return ExecutableStage.fromPayload(
ExecutableStagePayload.newBuilder()
@@ -314,11 +352,8 @@
"coder-id",
Coder.newBuilder()
.setSpec(
- SdkFunctionSpec.newBuilder()
- .setSpec(
- FunctionSpec.newBuilder()
- .setUrn(ModelCoders.INTERVAL_WINDOW_CODER_URN)
- .build())
+ FunctionSpec.newBuilder()
+ .setUrn(ModelCoders.INTERVAL_WINDOW_CODER_URN)
.build())
.build())
.build())
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolServiceTest.java
index 6bdbaff..cb65a0e 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientPoolServiceTest.java
@@ -35,8 +35,8 @@
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.InProcessServerFactory;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientTest.java
index 0777ad3..341b53c 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/FnApiControlClientTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactoryTest.java
new file mode 100644
index 0000000..3d407bc
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactoryTest.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.ByteArrayOutputStream;
+import java.io.PrintStream;
+import org.apache.beam.runners.fnexecution.control.ReferenceCountingExecutableStageContextFactory.Creator;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.junit.Assert;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link ReferenceCountingExecutableStageContextFactory}. */
+@RunWith(JUnit4.class)
+public class ReferenceCountingExecutableStageContextFactoryTest {
+
+ @Test
+ public void testCreateReuseReleaseCreate() throws Exception {
+
+ Creator creator = mock(Creator.class);
+ ExecutableStageContext c1 = mock(ExecutableStageContext.class);
+ ExecutableStageContext c2 = mock(ExecutableStageContext.class);
+ ExecutableStageContext c3 = mock(ExecutableStageContext.class);
+ ExecutableStageContext c4 = mock(ExecutableStageContext.class);
+ when(creator.apply(any(JobInfo.class)))
+ .thenReturn(c1)
+ .thenReturn(c2)
+ .thenReturn(c3)
+ .thenReturn(c4);
+ ReferenceCountingExecutableStageContextFactory factory =
+ ReferenceCountingExecutableStageContextFactory.create(creator, (x) -> true);
+ JobInfo jobA = mock(JobInfo.class);
+ when(jobA.jobId()).thenReturn("jobA");
+ JobInfo jobB = mock(JobInfo.class);
+ when(jobB.jobId()).thenReturn("jobB");
+ ExecutableStageContext ac1A = factory.get(jobA); // 1 open jobA
+ ExecutableStageContext ac2B = factory.get(jobB); // 1 open jobB
+ Assert.assertSame(
+ "Context should be cached and reused.", ac1A, factory.get(jobA)); // 2 open jobA
+ Assert.assertSame(
+ "Context should be cached and reused.", ac2B, factory.get(jobB)); // 2 open jobB
+ factory.release(ac1A); // 1 open jobA
+ Assert.assertSame(
+ "Context should be cached and reused.", ac1A, factory.get(jobA)); // 2 open jobA
+ factory.release(ac1A); // 1 open jobA
+ factory.release(ac1A); // 0 open jobA
+ ExecutableStageContext ac3A = factory.get(jobA); // 1 open jobA
+ Assert.assertNotSame("We should get a new instance.", ac1A, ac3A);
+ Assert.assertSame(
+ "Context should be cached and reused.", ac3A, factory.get(jobA)); // 2 open jobA
+ factory.release(ac3A); // 1 open jobA
+ factory.release(ac3A); // 0 open jobA
+ Assert.assertSame(
+ "Context should be cached and reused.", ac2B, factory.get(jobB)); // 3 open jobB
+ factory.release(ac2B); // 2 open jobB
+ factory.release(ac2B); // 1 open jobB
+ factory.release(ac2B); // 0 open jobB
+ ExecutableStageContext ac4B = factory.get(jobB); // 1 open jobB
+ Assert.assertNotSame("We should get a new instance.", ac2B, ac4B);
+ factory.release(ac4B); // 0 open jobB
+ }
+
+ @Test
+ public void testCatchThrowablesAndLogThem() throws Exception {
+ PrintStream oldErr = System.err;
+ oldErr.flush();
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ PrintStream newErr = new PrintStream(baos);
+ try {
+ System.setErr(newErr);
+ Creator creator = mock(Creator.class);
+ ExecutableStageContext c1 = mock(ExecutableStageContext.class);
+ when(creator.apply(any(JobInfo.class))).thenReturn(c1);
+ // throw an Throwable and ensure that it is caught and logged.
+ doThrow(new NoClassDefFoundError()).when(c1).close();
+ ReferenceCountingExecutableStageContextFactory factory =
+ ReferenceCountingExecutableStageContextFactory.create(creator, (x) -> true);
+ JobInfo jobA = mock(JobInfo.class);
+ when(jobA.jobId()).thenReturn("jobA");
+ ExecutableStageContext ac1A = factory.get(jobA);
+ factory.release(ac1A);
+ newErr.flush();
+ String output = new String(baos.toByteArray(), Charsets.UTF_8);
+ // Ensure that the error is logged
+ assertTrue(output.contains("Unable to close ExecutableStageContext"));
+ } finally {
+ newErr.flush();
+ System.setErr(oldErr);
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index 3aea2b9..0925767 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.fnexecution.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
@@ -45,7 +45,6 @@
import org.apache.beam.fn.harness.FnHarness;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.MetricsApi.MonitoringInfo;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.construction.PipelineTranslation;
@@ -112,15 +111,15 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Collections2;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Collections2;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.hamcrest.collection.IsEmptyIterable;
@@ -266,17 +265,19 @@
BundleProcessor processor =
controlClient.getProcessor(
descriptor.getProcessBundleDescriptor(), descriptor.getRemoteInputDestinations());
- Map<Target, ? super Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, Collection<? super WindowedValue<?>>> outputValues = new HashMap<>();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, ? super Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, ? super Coder<WindowedValue<?>>> remoteOutputCoders =
+ descriptor.getRemoteOutputCoders();
+ Map<String, Collection<? super WindowedValue<?>>> outputValues = new HashMap<>();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, ? super Coder<WindowedValue<?>>> remoteOutputCoder :
+ remoteOutputCoders.entrySet()) {
List<? super WindowedValue<?>> outputContents =
Collections.synchronizedList(new ArrayList<>());
- outputValues.put(targetCoder.getKey(), outputContents);
+ outputValues.put(remoteOutputCoder.getKey(), outputContents);
outputReceivers.put(
- targetCoder.getKey(),
+ remoteOutputCoder.getKey(),
RemoteOutputReceiver.of(
- (Coder) targetCoder.getValue(),
+ (Coder) remoteOutputCoder.getValue(),
(FnDataReceiver<? super WindowedValue<?>>) outputContents::add));
}
// The impulse example
@@ -329,17 +330,19 @@
BundleProcessor processor =
controlClient.getProcessor(
descriptor.getProcessBundleDescriptor(), descriptor.getRemoteInputDestinations());
- Map<Target, ? super Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, Collection<? super WindowedValue<?>>> outputValues = new HashMap<>();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, ? super Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, ? super Coder<WindowedValue<?>>> remoteOutputCoders =
+ descriptor.getRemoteOutputCoders();
+ Map<String, Collection<? super WindowedValue<?>>> outputValues = new HashMap<>();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, ? super Coder<WindowedValue<?>>> remoteOutputCoder :
+ remoteOutputCoders.entrySet()) {
List<? super WindowedValue<?>> outputContents =
Collections.synchronizedList(new ArrayList<>());
- outputValues.put(targetCoder.getKey(), outputContents);
+ outputValues.put(remoteOutputCoder.getKey(), outputContents);
outputReceivers.put(
- targetCoder.getKey(),
+ remoteOutputCoder.getKey(),
RemoteOutputReceiver.of(
- (Coder) targetCoder.getValue(),
+ (Coder) remoteOutputCoder.getValue(),
(FnDataReceiver<? super WindowedValue<?>>) outputContents::add));
}
@@ -438,15 +441,16 @@
descriptor.getProcessBundleDescriptor(),
descriptor.getRemoteInputDestinations(),
stateDelegator);
- Map<Target, Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, Coder> remoteOutputCoders = descriptor.getRemoteOutputCoders();
+ Map<String, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, Coder> remoteOutputCoder : remoteOutputCoders.entrySet()) {
List<WindowedValue<?>> outputContents = Collections.synchronizedList(new ArrayList<>());
- outputValues.put(targetCoder.getKey(), outputContents);
+ outputValues.put(remoteOutputCoder.getKey(), outputContents);
outputReceivers.put(
- targetCoder.getKey(),
- RemoteOutputReceiver.of(targetCoder.getValue(), outputContents::add));
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of(
+ (Coder<WindowedValue<?>>) remoteOutputCoder.getValue(), outputContents::add));
}
Iterable<String> sideInputData = Arrays.asList("A", "B", "C");
@@ -587,15 +591,16 @@
descriptor.getRemoteInputDestinations(),
stateDelegator);
- Map<Target, Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, Coder> remoteOutputCoders = descriptor.getRemoteOutputCoders();
+ Map<String, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, Coder> remoteOutputCoder : remoteOutputCoders.entrySet()) {
List<WindowedValue<?>> outputContents = Collections.synchronizedList(new ArrayList<>());
- outputValues.put(targetCoder.getKey(), outputContents);
+ outputValues.put(remoteOutputCoder.getKey(), outputContents);
outputReceivers.put(
- targetCoder.getKey(),
- RemoteOutputReceiver.of(targetCoder.getValue(), outputContents::add));
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of(
+ (Coder<WindowedValue<?>>) remoteOutputCoder.getValue(), outputContents::add));
}
Iterable<String> sideInputData = Arrays.asList("A", "B", "C");
@@ -847,15 +852,16 @@
descriptor.getProcessBundleDescriptor(),
descriptor.getRemoteInputDestinations(),
stateDelegator);
- Map<Target, Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, Coder> remoteOutputCoders = descriptor.getRemoteOutputCoders();
+ Map<String, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, Coder> remoteOutputCoder : remoteOutputCoders.entrySet()) {
List<WindowedValue<?>> outputContents = Collections.synchronizedList(new ArrayList<>());
- outputValues.put(targetCoder.getKey(), outputContents);
+ outputValues.put(remoteOutputCoder.getKey(), outputContents);
outputReceivers.put(
- targetCoder.getKey(),
- RemoteOutputReceiver.of(targetCoder.getValue(), outputContents::add));
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of(
+ (Coder<WindowedValue<?>>) remoteOutputCoder.getValue(), outputContents::add));
}
Map<String, List<ByteString>> userStateData =
@@ -881,27 +887,28 @@
StateRequestHandler stateRequestHandler =
StateRequestHandlers.forBagUserStateHandlerFactory(
descriptor,
- new BagUserStateHandlerFactory() {
+ new BagUserStateHandlerFactory<ByteString, Object, BoundedWindow>() {
@Override
- public <K, V, W extends BoundedWindow> BagUserStateHandler<K, V, W> forUserState(
+ public BagUserStateHandler<ByteString, Object, BoundedWindow> forUserState(
String pTransformId,
String userStateId,
- Coder<K> keyCoder,
- Coder<V> valueCoder,
- Coder<W> windowCoder) {
- return new BagUserStateHandler<K, V, W>() {
+ Coder<ByteString> keyCoder,
+ Coder<Object> valueCoder,
+ Coder<BoundedWindow> windowCoder) {
+ return new BagUserStateHandler<ByteString, Object, BoundedWindow>() {
@Override
- public Iterable<V> get(K key, W window) {
+ public Iterable<Object> get(ByteString key, BoundedWindow window) {
return (Iterable) userStateData.get(userStateId);
}
@Override
- public void append(K key, W window, Iterator<V> values) {
+ public void append(
+ ByteString key, BoundedWindow window, Iterator<Object> values) {
Iterators.addAll(userStateData.get(userStateId), (Iterator) values);
}
@Override
- public void clear(K key, W window) {
+ public void clear(ByteString key, BoundedWindow window) {
userStateData.get(userStateId).clear();
}
};
@@ -1018,30 +1025,31 @@
descriptor.getProcessBundleDescriptor(),
descriptor.getRemoteInputDestinations(),
stateDelegator);
- Map<Target, Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, Coder> remoteOutputCoders = descriptor.getRemoteOutputCoders();
+ Map<String, Collection<WindowedValue<?>>> outputValues = new HashMap<>();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, Coder> remoteOutputCoder : remoteOutputCoders.entrySet()) {
List<WindowedValue<?>> outputContents = Collections.synchronizedList(new ArrayList<>());
- outputValues.put(targetCoder.getKey(), outputContents);
+ outputValues.put(remoteOutputCoder.getKey(), outputContents);
outputReceivers.put(
- targetCoder.getKey(),
- RemoteOutputReceiver.of(targetCoder.getValue(), outputContents::add));
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of(
+ (Coder<WindowedValue<?>>) remoteOutputCoder.getValue(), outputContents::add));
}
String eventTimeInputPCollectionId = null;
- Target eventTimeOutputTarget = null;
+ String eventTimeOutputTransformId = null;
String processingTimeInputPCollectionId = null;
- Target processingTimeOutputTarget = null;
+ String processingTimeOutputTransformId = null;
for (Map<String, ProcessBundleDescriptors.TimerSpec> timerSpecs :
descriptor.getTimerSpecs().values()) {
for (ProcessBundleDescriptors.TimerSpec timerSpec : timerSpecs.values()) {
if (TimeDomain.EVENT_TIME.equals(timerSpec.getTimerSpec().getTimeDomain())) {
eventTimeInputPCollectionId = timerSpec.inputCollectionId();
- eventTimeOutputTarget = timerSpec.outputTarget();
+ eventTimeOutputTransformId = timerSpec.outputTransformId();
} else if (TimeDomain.PROCESSING_TIME.equals(timerSpec.getTimerSpec().getTimeDomain())) {
processingTimeInputPCollectionId = timerSpec.inputCollectionId();
- processingTimeOutputTarget = timerSpec.outputTarget();
+ processingTimeOutputTransformId = timerSpec.outputTransformId();
} else {
fail(String.format("Unknown timer specification %s", timerSpec));
}
@@ -1068,25 +1076,25 @@
.get(processingTimeInputPCollectionId)
.accept(WindowedValue.valueInGlobalWindow(timerBytes("Z", 200L)));
}
- Set<Target> timerOutputTargets =
- ImmutableSet.of(eventTimeOutputTarget, processingTimeOutputTarget);
- Target mainOutputTarget =
+ Set<String> timerOutputCoders =
+ ImmutableSet.of(eventTimeOutputTransformId, processingTimeOutputTransformId);
+ String mainOutputTransform =
Iterables.getOnlyElement(
- Sets.difference(descriptor.getOutputTargetCoders().keySet(), timerOutputTargets));
+ Sets.difference(descriptor.getRemoteOutputCoders().keySet(), timerOutputCoders));
assertThat(
- outputValues.get(mainOutputTarget),
+ outputValues.get(mainOutputTransform),
containsInAnyOrder(
WindowedValue.valueInGlobalWindow(KV.of("mainX", "")),
WindowedValue.valueInGlobalWindow(KV.of("event", "")),
WindowedValue.valueInGlobalWindow(KV.of("processing", ""))));
assertThat(
- timerStructuralValues(outputValues.get(eventTimeOutputTarget)),
+ timerStructuralValues(outputValues.get(eventTimeOutputTransformId)),
containsInAnyOrder(
timerStructuralValue(WindowedValue.valueInGlobalWindow(timerBytes("X", 1L))),
timerStructuralValue(WindowedValue.valueInGlobalWindow(timerBytes("Y", 11L))),
timerStructuralValue(WindowedValue.valueInGlobalWindow(timerBytes("Z", 21L)))));
assertThat(
- timerStructuralValues(outputValues.get(processingTimeOutputTarget)),
+ timerStructuralValues(outputValues.get(processingTimeOutputTransformId)),
containsInAnyOrder(
timerStructuralValue(WindowedValue.valueInGlobalWindow(timerBytes("X", 2L))),
timerStructuralValue(WindowedValue.valueInGlobalWindow(timerBytes("Y", 12L))),
@@ -1163,12 +1171,13 @@
descriptor.getProcessBundleDescriptor(),
descriptor.getRemoteInputDestinations(),
stateDelegator);
- Map<Target, Coder<WindowedValue<?>>> outputTargets = descriptor.getOutputTargetCoders();
- Map<Target, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
- for (Entry<Target, Coder<WindowedValue<?>>> targetCoder : outputTargets.entrySet()) {
+ Map<String, Coder> remoteOutputCoders = descriptor.getRemoteOutputCoders();
+ Map<String, RemoteOutputReceiver<?>> outputReceivers = new HashMap<>();
+ for (Entry<String, Coder> remoteOutputCoder : remoteOutputCoders.entrySet()) {
outputReceivers.putIfAbsent(
- targetCoder.getKey(),
- RemoteOutputReceiver.of(targetCoder.getValue(), outputValues::add));
+ remoteOutputCoder.getKey(),
+ RemoteOutputReceiver.of(
+ (Coder<WindowedValue<?>>) remoteOutputCoder.getValue(), outputValues::add));
}
try (ActiveBundle bundle =
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java
index 850c406..5bc0d1c 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClientTest.java
@@ -17,12 +17,13 @@
*/
package org.apache.beam.runners.fnexecution.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotSame;
import static org.junit.Assert.assertSame;
-import static org.junit.Assert.assertThat;
import static org.junit.Assert.fail;
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.eq;
@@ -35,6 +36,7 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
+import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
@@ -73,15 +75,17 @@
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
+import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
/** Unit tests for {@link SdkHarnessClient}. */
@@ -98,8 +102,8 @@
private SdkHarnessClient sdkHarnessClient;
private ProcessBundleDescriptor descriptor;
private String inputPCollection;
- private BeamFnApi.Target sdkGrpcReadTarget;
- private BeamFnApi.Target sdkGrpcWriteTarget;
+ private static final String SDK_GRPC_READ_TRANSFORM = "read";
+ private static final String SDK_GRPC_WRITE_TRANSFORM = "write";
@Before
public void setup() throws Exception {
@@ -145,24 +149,12 @@
}
pbdBuilder
.putTransforms("proc", targetProcessor)
- .putTransforms("read", readNode.toPTransform())
- .putTransforms("write", writeNode.toPTransform());
+ .putTransforms(SDK_GRPC_READ_TRANSFORM, readNode.toPTransform())
+ .putTransforms(SDK_GRPC_WRITE_TRANSFORM, writeNode.toPTransform());
descriptor = pbdBuilder.build();
inputPCollection =
getOnlyElement(descriptor.getTransformsOrThrow("read").getOutputsMap().values());
- sdkGrpcReadTarget =
- BeamFnApi.Target.newBuilder()
- .setName(
- getOnlyElement(descriptor.getTransformsOrThrow("read").getOutputsMap().keySet()))
- .setPrimitiveTransformReference("read")
- .build();
- sdkGrpcWriteTarget =
- BeamFnApi.Target.newBuilder()
- .setName(
- getOnlyElement(descriptor.getTransformsOrThrow("write").getInputsMap().keySet()))
- .setPrimitiveTransformReference("write")
- .build();
}
@Test
@@ -176,13 +168,13 @@
ProcessBundleDescriptor descriptor2 =
ProcessBundleDescriptor.newBuilder().setId("descriptor2").build();
- Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputs =
+ Map<String, RemoteInputDestination> remoteInputs =
Collections.singletonMap(
"inputPC",
RemoteInputDestination.of(
(FullWindowedValueCoder)
FullWindowedValueCoder.of(VarIntCoder.of(), GlobalWindow.Coder.INSTANCE),
- sdkGrpcReadTarget));
+ SDK_GRPC_READ_TRANSFORM));
BundleProcessor processor1 = sdkHarnessClient.getProcessor(descriptor1, remoteInputs);
BundleProcessor processor2 = sdkHarnessClient.getProcessor(descriptor2, remoteInputs);
@@ -205,13 +197,13 @@
.setStateApiServiceDescriptor(ApiServiceDescriptor.newBuilder().setUrl("foo"))
.build();
- Map<String, RemoteInputDestination<WindowedValue<?>>> remoteInputs =
+ Map<String, RemoteInputDestination> remoteInputs =
Collections.singletonMap(
"inputPC",
RemoteInputDestination.of(
(FullWindowedValueCoder)
FullWindowedValueCoder.of(VarIntCoder.of(), GlobalWindow.Coder.INSTANCE),
- sdkGrpcReadTarget));
+ SDK_GRPC_READ_TRANSFORM));
thrown.expect(IllegalStateException.class);
thrown.expectMessage("containing a state");
@@ -232,7 +224,8 @@
descriptor,
Collections.singletonMap(
"inputPC",
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)));
+ RemoteInputDestination.of(
+ (FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)));
when(dataService.send(any(), eq(coder))).thenReturn(mock(CloseableFnDataReceiver.class));
try (ActiveBundle activeBundle =
@@ -260,13 +253,13 @@
RemoteInputDestination.of(
(FullWindowedValueCoder)
FullWindowedValueCoder.of(StringUtf8Coder.of(), Coder.INSTANCE),
- sdkGrpcReadTarget)));
+ SDK_GRPC_READ_TRANSFORM)));
Collection<WindowedValue<String>> outputs = new ArrayList<>();
try (ActiveBundle activeBundle =
processor.newBundle(
Collections.singletonMap(
- sdkGrpcWriteTarget,
+ SDK_GRPC_WRITE_TRANSFORM,
RemoteOutputReceiver.of(
FullWindowedValueCoder.of(
LengthPrefixCoder.of(StringUtf8Coder.of()), Coder.INSTANCE),
@@ -307,7 +300,8 @@
descriptor,
Collections.singletonMap(
"inputPC",
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)));
+ RemoteInputDestination.of(
+ (FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)));
when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
@@ -319,7 +313,8 @@
try {
try (ActiveBundle activeBundle =
processor.newBundle(
- ImmutableMap.of(sdkGrpcWriteTarget, mockRemoteOutputReceiver), mockProgressHandler)) {
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
+ mockProgressHandler)) {
// We shouldn't be required to complete the process bundle response future.
}
fail("Exception expected");
@@ -343,6 +338,7 @@
when(mockStateDelegator.registerForProcessBundleInstructionId(any(), any()))
.thenReturn(mockStateRegistration);
StateRequestHandler mockStateHandler = mock(StateRequestHandler.class);
+ when(mockStateHandler.getCacheTokens()).thenReturn(Collections.emptyList());
BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
@@ -357,7 +353,7 @@
descriptor,
Collections.singletonMap(
inputPCollection,
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)),
+ RemoteInputDestination.of((FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)),
mockStateDelegator);
when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
@@ -369,7 +365,7 @@
try {
try (ActiveBundle activeBundle =
processor.newBundle(
- ImmutableMap.of(sdkGrpcWriteTarget, mockRemoteOutputReceiver),
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
mockStateHandler,
mockProgressHandler)) {
// We shouldn't be required to complete the process bundle response future.
@@ -403,7 +399,8 @@
descriptor,
Collections.singletonMap(
"inputPC",
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)));
+ RemoteInputDestination.of(
+ (FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)));
when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
@@ -413,7 +410,8 @@
try {
try (ActiveBundle activeBundle =
processor.newBundle(
- ImmutableMap.of(sdkGrpcWriteTarget, mockRemoteOutputReceiver), mockProgressHandler)) {
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
+ mockProgressHandler)) {
processBundleResponseFuture.completeExceptionally(testException);
}
fail("Exception expected");
@@ -436,6 +434,7 @@
when(mockStateDelegator.registerForProcessBundleInstructionId(any(), any()))
.thenReturn(mockStateRegistration);
StateRequestHandler mockStateHandler = mock(StateRequestHandler.class);
+ when(mockStateHandler.getCacheTokens()).thenReturn(Collections.emptyList());
BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
@@ -450,7 +449,7 @@
descriptor,
Collections.singletonMap(
inputPCollection,
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)),
+ RemoteInputDestination.of((FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)),
mockStateDelegator);
when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
@@ -460,7 +459,7 @@
try {
try (ActiveBundle activeBundle =
processor.newBundle(
- ImmutableMap.of(sdkGrpcWriteTarget, mockRemoteOutputReceiver),
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
mockStateHandler,
mockProgressHandler)) {
processBundleResponseFuture.completeExceptionally(testException);
@@ -494,7 +493,8 @@
descriptor,
Collections.singletonMap(
"inputPC",
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)));
+ RemoteInputDestination.of(
+ (FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)));
when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
doThrow(testException).when(mockOutputReceiver).awaitCompletion();
@@ -505,7 +505,8 @@
try {
try (ActiveBundle activeBundle =
processor.newBundle(
- ImmutableMap.of(sdkGrpcWriteTarget, mockRemoteOutputReceiver), mockProgressHandler)) {
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
+ mockProgressHandler)) {
// Correlating the ProcessBundleRequest and ProcessBundleResponse is owned by the underlying
// FnApiControlClient. The SdkHarnessClient owns just wrapping the request and unwrapping
// the response.
@@ -534,6 +535,7 @@
when(mockStateDelegator.registerForProcessBundleInstructionId(any(), any()))
.thenReturn(mockStateRegistration);
StateRequestHandler mockStateHandler = mock(StateRequestHandler.class);
+ when(mockStateHandler.getCacheTokens()).thenReturn(Collections.emptyList());
BundleProgressHandler mockProgressHandler = mock(BundleProgressHandler.class);
CompletableFuture<InstructionResponse> processBundleResponseFuture = new CompletableFuture<>();
@@ -548,7 +550,7 @@
descriptor,
Collections.singletonMap(
inputPCollection,
- RemoteInputDestination.of((FullWindowedValueCoder) coder, sdkGrpcReadTarget)),
+ RemoteInputDestination.of((FullWindowedValueCoder) coder, SDK_GRPC_READ_TRANSFORM)),
mockStateDelegator);
when(dataService.receive(any(), any(), any())).thenReturn(mockOutputReceiver);
when(dataService.send(any(), eq(coder))).thenReturn(mockInputSender);
@@ -559,7 +561,7 @@
try {
try (ActiveBundle activeBundle =
processor.newBundle(
- ImmutableMap.of(sdkGrpcWriteTarget, mockRemoteOutputReceiver),
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mockRemoteOutputReceiver),
mockStateHandler,
mockProgressHandler)) {
// Correlating the ProcessBundleRequest and ProcessBundleResponse is owned by the underlying
@@ -579,6 +581,51 @@
}
}
+ @Test
+ public void verifyCacheTokensAreUsedInNewBundleRequest() {
+ CompletableFuture<InstructionResponse> registerResponseFuture = new CompletableFuture<>();
+ when(fnApiControlClient.handle(any(BeamFnApi.InstructionRequest.class)))
+ .thenReturn(registerResponseFuture);
+
+ ProcessBundleDescriptor descriptor1 =
+ ProcessBundleDescriptor.newBuilder().setId("descriptor1").build();
+
+ Map<String, RemoteInputDestination> remoteInputs =
+ Collections.singletonMap(
+ "inputPC",
+ RemoteInputDestination.of(
+ FullWindowedValueCoder.of(VarIntCoder.of(), GlobalWindow.Coder.INSTANCE),
+ SDK_GRPC_READ_TRANSFORM));
+
+ BundleProcessor processor1 = sdkHarnessClient.getProcessor(descriptor1, remoteInputs);
+ when(dataService.send(any(), any())).thenReturn(mock(CloseableFnDataReceiver.class));
+
+ StateRequestHandler stateRequestHandler = Mockito.mock(StateRequestHandler.class);
+ List<BeamFnApi.ProcessBundleRequest.CacheToken> cacheTokens =
+ Collections.singletonList(
+ BeamFnApi.ProcessBundleRequest.CacheToken.newBuilder().getDefaultInstanceForType());
+ when(stateRequestHandler.getCacheTokens()).thenReturn(cacheTokens);
+
+ processor1.newBundle(
+ ImmutableMap.of(SDK_GRPC_WRITE_TRANSFORM, mock(RemoteOutputReceiver.class)),
+ stateRequestHandler,
+ BundleProgressHandler.ignored());
+
+ // Retrieve the requests made to the FnApiControlClient
+ ArgumentCaptor<BeamFnApi.InstructionRequest> reqCaptor =
+ ArgumentCaptor.forClass(BeamFnApi.InstructionRequest.class);
+ Mockito.verify(fnApiControlClient, Mockito.times(2)).handle(reqCaptor.capture());
+ List<BeamFnApi.InstructionRequest> requests = reqCaptor.getAllValues();
+
+ // Verify that the cache tokens are included in the ProcessBundleRequest
+ assertThat(
+ requests.get(0).getRequestCase(), is(BeamFnApi.InstructionRequest.RequestCase.REGISTER));
+ assertThat(
+ requests.get(1).getRequestCase(),
+ is(BeamFnApi.InstructionRequest.RequestCase.PROCESS_BUNDLE));
+ assertThat(requests.get(1).getProcessBundle().getCacheTokensList(), is(cacheTokens));
+ }
+
private static class TestFn extends DoFn<String, String> {
@ProcessElement
public void processElement(ProcessContext context) {
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactoryTest.java
index 75d2bb6..93a90bb 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/SingleEnvironmentInstanceJobBundleFactoryTest.java
@@ -49,7 +49,7 @@
import org.apache.beam.sdk.fn.IdGenerators;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
index 4d93629..a4458c0 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
@@ -46,10 +46,10 @@
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.fn.test.TestStreams;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -57,8 +57,7 @@
/** Tests for {@link GrpcDataService}. */
@RunWith(JUnit4.class)
public class GrpcDataServiceTest {
- private static final BeamFnApi.Target TARGET =
- BeamFnApi.Target.newBuilder().setPrimitiveTransformReference("888").setName("test").build();
+ private static final String TRANSFORM_ID = "888";
private static final Coder<WindowedValue<String>> CODER =
LengthPrefixCoder.of(WindowedValue.getValueOnlyCoder(StringUtf8Coder.of()));
@@ -92,7 +91,7 @@
for (int i = 0; i < 3; ++i) {
CloseableFnDataReceiver<WindowedValue<String>> consumer =
- service.send(LogicalEndpoint.of(Integer.toString(i), TARGET), CODER);
+ service.send(LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID), CODER);
consumer.accept(WindowedValue.valueInGlobalWindow("A" + i));
consumer.accept(WindowedValue.valueInGlobalWindow("B" + i));
@@ -122,7 +121,7 @@
GrpcFnServer.allocatePortAndCreateFor(service, InProcessServerFactory.create())) {
Collection<Future<Void>> clientFutures = new ArrayList<>();
for (int i = 0; i < 3; ++i) {
- final String instructionReference = Integer.toString(i);
+ final String instructionId = Integer.toString(i);
clientFutures.add(
executorService.submit(
() -> {
@@ -132,7 +131,7 @@
StreamObserver<Elements> outboundObserver =
BeamFnDataGrpc.newStub(channel)
.data(TestStreams.withOnNext(clientInboundElements::add).build());
- outboundObserver.onNext(elementsWithData(instructionReference));
+ outboundObserver.onNext(elementsWithData(instructionId));
waitForInboundElements.await();
outboundObserver.onCompleted();
return null;
@@ -146,7 +145,9 @@
serverInboundValues.add(serverInboundValue);
readFutures.add(
service.receive(
- LogicalEndpoint.of(Integer.toString(i), TARGET), CODER, serverInboundValue::add));
+ LogicalEndpoint.of(Integer.toString(i), TRANSFORM_ID),
+ CODER,
+ serverInboundValue::add));
}
for (InboundDataClient readFuture : readFutures) {
readFuture.awaitCompletion();
@@ -171,8 +172,8 @@
return BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(id)
- .setTarget(TARGET)
+ .setInstructionId(id)
+ .setTransformId(TRANSFORM_ID)
.setData(
ByteString.copyFrom(
encodeToByteArray(CODER, WindowedValue.valueInGlobalWindow("A" + id)))
@@ -184,7 +185,8 @@
ByteString.copyFrom(
encodeToByteArray(
CODER, WindowedValue.valueInGlobalWindow("C" + id))))))
- .addData(BeamFnApi.Elements.Data.newBuilder().setInstructionReference(id).setTarget(TARGET))
+ .addData(
+ BeamFnApi.Elements.Data.newBuilder().setInstructionId(id).setTransformId(TRANSFORM_ID))
.build();
}
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestinationTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestinationTest.java
index b9f3392..9b2f919 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestinationTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestinationTest.java
@@ -17,10 +17,9 @@
*/
package org.apache.beam.runners.fnexecution.data;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
@@ -37,17 +36,14 @@
@Test
public void testConstruction() {
- BeamFnApi.Target target =
- BeamFnApi.Target.newBuilder()
- .setName("my_name")
- .setPrimitiveTransformReference("my_target_pt")
- .build();
+ String transformId = "my_target_pt";
+
KvCoder<byte[], Iterable<Long>> coder =
KvCoder.of(LengthPrefixCoder.of(ByteArrayCoder.of()), IterableCoder.of(VarLongCoder.of()));
RemoteInputDestination<KV<byte[], Iterable<Long>>> destination =
- RemoteInputDestination.of(coder, target);
+ RemoteInputDestination.of(coder, transformId);
assertThat(destination.getCoder(), equalTo(coder));
- assertThat(destination.getTarget(), equalTo(target));
+ assertThat(destination.getPTransformId(), equalTo(transformId));
}
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerCommandTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerCommandTest.java
index 4b47d6d..6c27975 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerCommandTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerCommandTest.java
@@ -26,8 +26,8 @@
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.apache.beam.runners.fnexecution.environment.testing.NeedsDocker;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java
index fa67aab..a199cb2 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/DockerEnvironmentFactoryTest.java
@@ -21,30 +21,40 @@
import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.TimeoutException;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import org.apache.beam.runners.core.construction.Environments;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.control.ControlClientPool;
import org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
import org.apache.beam.runners.fnexecution.control.InstructionRequestHandler;
import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.options.ManualDockerEnvironmentOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
import org.junit.Before;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
/** Tests for {@link DockerEnvironmentFactory}. */
-@RunWith(JUnit4.class)
public class DockerEnvironmentFactoryTest {
private static final ApiServiceDescriptor SERVICE_DESCRIPTOR =
@@ -56,15 +66,14 @@
private static final IdGenerator ID_GENERATOR = IdGenerators.incrementingLongs();
- @Mock private DockerCommand docker;
+ @Mock DockerCommand docker;
- @Mock private GrpcFnServer<FnApiControlClientPoolService> controlServiceServer;
- @Mock private GrpcFnServer<GrpcLoggingService> loggingServiceServer;
- @Mock private GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer;
- @Mock private GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer;
+ @Mock GrpcFnServer<FnApiControlClientPoolService> controlServiceServer;
+ @Mock GrpcFnServer<GrpcLoggingService> loggingServiceServer;
+ @Mock GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer;
+ @Mock GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer;
- @Mock private InstructionRequestHandler client;
- private DockerEnvironmentFactory factory;
+ @Mock InstructionRequestHandler client;
@Before
public void initMocks() {
@@ -74,49 +83,133 @@
when(loggingServiceServer.getApiServiceDescriptor()).thenReturn(SERVICE_DESCRIPTOR);
when(retrievalServiceServer.getApiServiceDescriptor()).thenReturn(SERVICE_DESCRIPTOR);
when(provisioningServiceServer.getApiServiceDescriptor()).thenReturn(SERVICE_DESCRIPTOR);
- factory =
- DockerEnvironmentFactory.forServicesWithDocker(
- docker,
- controlServiceServer,
- loggingServiceServer,
- retrievalServiceServer,
- provisioningServiceServer,
- (workerId, timeout) -> client,
- ID_GENERATOR,
- false);
}
- @Test
- public void createsCorrectEnvironment() throws Exception {
- when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
- .thenReturn(CONTAINER_ID);
- when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ @RunWith(Parameterized.class)
+ public static class ParameterizedTest extends DockerEnvironmentFactoryTest {
+ @Parameter(0)
+ public boolean throwsException;
- RemoteEnvironment handle = factory.createEnvironment(ENVIRONMENT);
- assertThat(handle.getInstructionRequestHandler(), is(client));
- assertThat(handle.getEnvironment(), equalTo(ENVIRONMENT));
+ @Parameter(1)
+ public boolean retainDockerContainer;
+
+ @Parameter(2)
+ public int removeContainerTimes;
+
+ @Rule public ExpectedException expectedException = ExpectedException.none();
+
+ private final ControlClientPool.Source normalClientSource = (workerId, timeout) -> client;
+ private final ControlClientPool.Source exceptionClientSource =
+ (workerId, timeout) -> {
+ throw new Exception();
+ };
+
+ @Parameterized.Parameters(
+ name =
+ "{index}: Test with throwsException={0}, retainDockerContainer={1} should remove container {2} time(s)")
+ public static Collection<Object[]> data() {
+ Object[][] data =
+ new Object[][] {{false, false, 1}, {false, true, 0}, {true, false, 1}, {true, true, 0}};
+ return Arrays.asList(data);
+ }
+
+ @Test
+ public void cleansUpContainerCorrectly() throws Exception {
+ when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
+ .thenReturn(CONTAINER_ID);
+ when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ ManualDockerEnvironmentOptions pipelineOptions =
+ PipelineOptionsFactory.as(ManualDockerEnvironmentOptions.class);
+ pipelineOptions.setRetainDockerContainers(retainDockerContainer);
+ DockerEnvironmentFactory factory =
+ DockerEnvironmentFactory.forServicesWithDocker(
+ docker,
+ controlServiceServer,
+ loggingServiceServer,
+ retrievalServiceServer,
+ provisioningServiceServer,
+ throwsException ? exceptionClientSource : normalClientSource,
+ ID_GENERATOR,
+ pipelineOptions);
+ if (throwsException) {
+ expectedException.expect(Exception.class);
+ }
+
+ RemoteEnvironment handle = factory.createEnvironment(ENVIRONMENT);
+ handle.close();
+
+ verify(docker).killContainer(CONTAINER_ID);
+ verify(docker, times(removeContainerTimes)).removeContainer(CONTAINER_ID);
+ }
}
- @Test
- public void destroysCorrectContainer() throws Exception {
- when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
- .thenReturn(CONTAINER_ID);
- when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ public static class NonParameterizedTest extends DockerEnvironmentFactoryTest {
+ @Test
+ public void createsCorrectEnvironment() throws Exception {
+ when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
+ .thenReturn(CONTAINER_ID);
+ when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ DockerEnvironmentFactory factory = getFactory((workerId, timeout) -> client);
- RemoteEnvironment handle = factory.createEnvironment(ENVIRONMENT);
- handle.close();
- verify(docker).killContainer(CONTAINER_ID);
- }
+ RemoteEnvironment handle = factory.createEnvironment(ENVIRONMENT);
- @Test
- public void createsMultipleEnvironments() throws Exception {
- when(docker.isContainerRunning(anyString())).thenReturn(true);
- Environment fooEnv = Environments.createDockerEnvironment("foo");
- RemoteEnvironment fooHandle = factory.createEnvironment(fooEnv);
- assertThat(fooHandle.getEnvironment(), is(equalTo(fooEnv)));
+ assertThat(handle.getInstructionRequestHandler(), is(client));
+ assertThat(handle.getEnvironment(), equalTo(ENVIRONMENT));
+ }
- Environment barEnv = Environments.createDockerEnvironment("bar");
- RemoteEnvironment barHandle = factory.createEnvironment(barEnv);
- assertThat(barHandle.getEnvironment(), is(equalTo(barEnv)));
+ @Test(expected = RuntimeException.class)
+ public void logsDockerOutputOnTimeoutException() throws Exception {
+ when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
+ .thenReturn(CONTAINER_ID);
+ when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ DockerEnvironmentFactory factory =
+ getFactory(
+ (workerId, timeout) -> {
+ throw new TimeoutException();
+ });
+
+ factory.createEnvironment(ENVIRONMENT);
+
+ verify(docker).getContainerLogs(CONTAINER_ID);
+ }
+
+ @Test
+ public void logsDockerOutputOnClose() throws Exception {
+ when(docker.runImage(Mockito.eq(IMAGE_NAME), Mockito.any(), Mockito.any()))
+ .thenReturn(CONTAINER_ID);
+ when(docker.isContainerRunning(Mockito.eq(CONTAINER_ID))).thenReturn(true);
+ DockerEnvironmentFactory factory = getFactory((workerId, timeout) -> client);
+
+ RemoteEnvironment handle = factory.createEnvironment(ENVIRONMENT);
+ handle.close();
+
+ verify(docker).getContainerLogs(CONTAINER_ID);
+ }
+
+ @Test
+ public void createsMultipleEnvironments() throws Exception {
+ when(docker.isContainerRunning(anyString())).thenReturn(true);
+ DockerEnvironmentFactory factory = getFactory((workerId, timeout) -> client);
+
+ Environment fooEnv = Environments.createDockerEnvironment("foo");
+ RemoteEnvironment fooHandle = factory.createEnvironment(fooEnv);
+ assertThat(fooHandle.getEnvironment(), is(equalTo(fooEnv)));
+
+ Environment barEnv = Environments.createDockerEnvironment("bar");
+ RemoteEnvironment barHandle = factory.createEnvironment(barEnv);
+ assertThat(barHandle.getEnvironment(), is(equalTo(barEnv)));
+ }
+
+ private DockerEnvironmentFactory getFactory(ControlClientPool.Source clientSource) {
+ return DockerEnvironmentFactory.forServicesWithDocker(
+ docker,
+ controlServiceServer,
+ loggingServiceServer,
+ retrievalServiceServer,
+ provisioningServiceServer,
+ clientSource,
+ ID_GENERATOR,
+ PipelineOptionsFactory.as(RemoteEnvironmentOptions.class));
+ }
}
}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java
index 80e1810..a38b1b5 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentFactoryTest.java
@@ -40,6 +40,8 @@
import org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.RemoteEnvironmentOptions;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -88,7 +90,8 @@
retrievalServiceServer,
provisioningServiceServer,
(workerId, timeout) -> client,
- ID_GENERATOR);
+ ID_GENERATOR,
+ PipelineOptionsFactory.as(RemoteEnvironmentOptions.class));
}
@Test
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobServiceTest.java
index 961016a..e7b01af 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobServiceTest.java
@@ -19,6 +19,7 @@
import static org.hamcrest.collection.IsCollectionWithSize.hasSize;
import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.Is.isA;
import static org.hamcrest.core.IsNull.notNullValue;
import static org.junit.Assert.assertThat;
import static org.mockito.Matchers.any;
@@ -27,11 +28,13 @@
import static org.mockito.Mockito.when;
import java.util.ArrayList;
+import java.util.List;
import org.apache.beam.model.jobmanagement.v1.JobApi;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -47,6 +50,12 @@
private static final String TEST_RETRIEVAL_TOKEN = "test-staging-token";
private static final RunnerApi.Pipeline TEST_PIPELINE = RunnerApi.Pipeline.getDefaultInstance();
private static final Struct TEST_OPTIONS = Struct.getDefaultInstance();
+ private static final JobApi.JobInfo TEST_JOB_INFO =
+ JobApi.JobInfo.newBuilder()
+ .setJobId(TEST_JOB_ID)
+ .setJobName(TEST_JOB_NAME)
+ .setPipelineOptions(TEST_OPTIONS)
+ .build();
Endpoints.ApiServiceDescriptor stagingServiceDescriptor;
@Mock JobInvoker invoker;
@@ -62,6 +71,36 @@
InMemoryJobService.create(stagingServiceDescriptor, session -> "token", null, invoker);
when(invoker.invoke(TEST_PIPELINE, TEST_OPTIONS, TEST_RETRIEVAL_TOKEN)).thenReturn(invocation);
when(invocation.getId()).thenReturn(TEST_JOB_ID);
+ when(invocation.getPipeline()).thenReturn(TEST_PIPELINE);
+ when(invocation.toProto()).thenReturn(TEST_JOB_INFO);
+ }
+
+ private JobApi.PrepareJobResponse prepareJob() {
+ JobApi.PrepareJobRequest request =
+ JobApi.PrepareJobRequest.newBuilder()
+ .setJobName(TEST_JOB_NAME)
+ .setPipeline(RunnerApi.Pipeline.getDefaultInstance())
+ .setPipelineOptions(Struct.getDefaultInstance())
+ .build();
+ RecordingObserver<JobApi.PrepareJobResponse> recorder = new RecordingObserver<>();
+ service.prepare(request, recorder);
+ return recorder.values.get(0);
+ }
+
+ private JobApi.RunJobResponse runJob(String preparationId) {
+ JobApi.RunJobRequest runRequest =
+ JobApi.RunJobRequest.newBuilder()
+ .setPreparationId(preparationId)
+ .setRetrievalToken(TEST_RETRIEVAL_TOKEN)
+ .build();
+ RecordingObserver<JobApi.RunJobResponse> recorder = new RecordingObserver<>();
+ service.run(runRequest, recorder);
+ return recorder.values.get(0);
+ }
+
+ private JobApi.RunJobResponse prepareAndRunJob() {
+ JobApi.PrepareJobResponse prepareResponse = prepareJob();
+ return runJob(prepareResponse.getPreparationId());
}
@Test
@@ -82,17 +121,53 @@
}
@Test
+ public void testGetJobsIsSuccessful() throws Exception {
+ prepareAndRunJob();
+
+ JobApi.GetJobsRequest request = JobApi.GetJobsRequest.newBuilder().build();
+ RecordingObserver<JobApi.GetJobsResponse> recorder = new RecordingObserver<>();
+ service.getJobs(request, recorder);
+ assertThat(recorder.isSuccessful(), is(true));
+ assertThat(recorder.values, hasSize(1));
+ JobApi.GetJobsResponse response = recorder.values.get(0);
+ List<JobApi.JobInfo> jobs = response.getJobInfoList();
+ assertThat(jobs, hasSize(1));
+ JobApi.JobInfo job = jobs.get(0);
+ assertThat(job.getJobId(), is(TEST_JOB_ID));
+ assertThat(job.getJobName(), is(TEST_JOB_NAME));
+ }
+
+ @Test
+ public void testGetPipelineFailure() {
+ prepareJob();
+
+ JobApi.GetJobPipelineRequest request =
+ JobApi.GetJobPipelineRequest.newBuilder().setJobId(TEST_JOB_ID).build();
+ RecordingObserver<JobApi.GetJobPipelineResponse> recorder = new RecordingObserver<>();
+ service.getPipeline(request, recorder);
+ // job has not been run yet
+ assertThat(recorder.isSuccessful(), is(false));
+ assertThat(recorder.error, isA(StatusException.class));
+ }
+
+ @Test
+ public void testGetPipelineIsSuccessful() throws Exception {
+ prepareAndRunJob();
+
+ JobApi.GetJobPipelineRequest request =
+ JobApi.GetJobPipelineRequest.newBuilder().setJobId(TEST_JOB_ID).build();
+ RecordingObserver<JobApi.GetJobPipelineResponse> recorder = new RecordingObserver<>();
+ service.getPipeline(request, recorder);
+ assertThat(recorder.isSuccessful(), is(true));
+ assertThat(recorder.values, hasSize(1));
+ JobApi.GetJobPipelineResponse response = recorder.values.get(0);
+ assertThat(response.getPipeline(), is(TEST_PIPELINE));
+ }
+
+ @Test
public void testJobSubmissionUsesJobInvokerAndIsSuccess() throws Exception {
- // prepare job
- JobApi.PrepareJobRequest prepareRequest =
- JobApi.PrepareJobRequest.newBuilder()
- .setJobName(TEST_JOB_NAME)
- .setPipeline(RunnerApi.Pipeline.getDefaultInstance())
- .setPipelineOptions(Struct.getDefaultInstance())
- .build();
- RecordingObserver<JobApi.PrepareJobResponse> prepareRecorder = new RecordingObserver<>();
- service.prepare(prepareRequest, prepareRecorder);
- JobApi.PrepareJobResponse prepareResponse = prepareRecorder.values.get(0);
+ JobApi.PrepareJobResponse prepareResponse = prepareJob();
+
// run job
JobApi.RunJobRequest runRequest =
JobApi.RunJobRequest.newBuilder()
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocationTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocationTest.java
new file mode 100644
index 0000000..30e34d4
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocationTest.java
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.allOf;
+import static org.hamcrest.Matchers.notNullValue;
+import static org.hamcrest.core.Is.is;
+import static org.hamcrest.core.IsSame.sameInstance;
+
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.PipelineTranslation;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+/** Tests for {@link JobInvocation}. */
+public class JobInvocationTest {
+
+ private static ExecutorService executorService;
+
+ private JobInvocation jobInvocation;
+ private ControllablePipelineRunner runner;
+
+ @Before
+ public void setup() {
+ executorService = Executors.newFixedThreadPool(1);
+ JobInfo jobInfo =
+ JobInfo.create("jobid", "jobName", "retrievalToken", Struct.getDefaultInstance());
+ ListeningExecutorService listeningExecutorService =
+ MoreExecutors.listeningDecorator(executorService);
+ Pipeline pipeline = Pipeline.create();
+ runner = new ControllablePipelineRunner();
+ jobInvocation =
+ new JobInvocation(
+ jobInfo, listeningExecutorService, PipelineTranslation.toProto(pipeline), runner);
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ executorService.shutdownNow();
+ executorService = null;
+ }
+
+ @Test(timeout = 10_000)
+ public void testStateAfterCompletion() throws Exception {
+ jobInvocation.start();
+ assertThat(jobInvocation.getState(), is(JobApi.JobState.Enum.RUNNING));
+
+ TestPipelineResult pipelineResult = new TestPipelineResult(PipelineResult.State.DONE);
+ runner.setResult(pipelineResult);
+
+ awaitJobState(jobInvocation, JobApi.JobState.Enum.DONE);
+ }
+
+ @Test(timeout = 10_000)
+ public void testStateAfterCompletionWithoutResult() throws Exception {
+ jobInvocation.start();
+ assertThat(jobInvocation.getState(), is(JobApi.JobState.Enum.RUNNING));
+
+ // Let pipeline finish without a result.
+ runner.setResult(null);
+
+ awaitJobState(jobInvocation, JobApi.JobState.Enum.UNSPECIFIED);
+ }
+
+ @Test(timeout = 10_000)
+ public void testStateAfterCancellation() throws Exception {
+ jobInvocation.start();
+ assertThat(jobInvocation.getState(), is(JobApi.JobState.Enum.RUNNING));
+
+ jobInvocation.cancel();
+ awaitJobState(jobInvocation, JobApi.JobState.Enum.CANCELLED);
+ }
+
+ @Test(timeout = 10_000)
+ public void testStateAfterCancellationWithPipelineResult() throws Exception {
+ jobInvocation.start();
+ assertThat(jobInvocation.getState(), is(JobApi.JobState.Enum.RUNNING));
+
+ TestPipelineResult pipelineResult = new TestPipelineResult(PipelineResult.State.FAILED);
+ runner.setResult(pipelineResult);
+ awaitJobState(jobInvocation, JobApi.JobState.Enum.FAILED);
+
+ jobInvocation.cancel();
+ pipelineResult.cancelLatch.await();
+ }
+
+ @Test(timeout = 10_000)
+ public void testNoCancellationWhenDone() throws Exception {
+ jobInvocation.start();
+ assertThat(jobInvocation.getState(), is(JobApi.JobState.Enum.RUNNING));
+
+ TestPipelineResult pipelineResult = new TestPipelineResult(PipelineResult.State.DONE);
+ runner.setResult(pipelineResult);
+ awaitJobState(jobInvocation, JobApi.JobState.Enum.DONE);
+
+ jobInvocation.cancel();
+ assertThat(jobInvocation.getState(), is(JobApi.JobState.Enum.DONE));
+ // Ensure that cancel has not been called
+ assertThat(pipelineResult.cancelLatch.getCount(), is(1L));
+ }
+
+ @Test(timeout = 10_000)
+ public void testReturnsMetricsFromJobInvocationAfterSuccess() throws Exception {
+ JobApi.MetricResults expectedMonitoringInfos = JobApi.MetricResults.newBuilder().build();
+ TestPipelineResult result =
+ new TestPipelineResult(PipelineResult.State.DONE, expectedMonitoringInfos);
+
+ jobInvocation.start();
+ runner.setResult(result);
+
+ awaitJobState(jobInvocation, JobApi.JobState.Enum.DONE);
+
+ assertThat(
+ jobInvocation.getMetrics(),
+ allOf(is(notNullValue()), is(sameInstance(result.portableMetrics()))));
+ }
+
+ private static void awaitJobState(JobInvocation jobInvocation, JobApi.JobState.Enum jobState)
+ throws Exception {
+ while (jobInvocation.getState() != jobState) {
+ Thread.sleep(50);
+ }
+ }
+
+ private static class ControllablePipelineRunner implements PortablePipelineRunner {
+
+ private final CountDownLatch latch = new CountDownLatch(1);
+ private volatile PortablePipelineResult result;
+
+ @Override
+ public PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo)
+ throws Exception {
+ latch.await();
+ return result;
+ }
+
+ void setResult(PortablePipelineResult pipelineResult) {
+ result = pipelineResult;
+ latch.countDown();
+ }
+ }
+
+ private static class TestPipelineResult implements PortablePipelineResult {
+
+ private final State state;
+ private final CountDownLatch cancelLatch = new CountDownLatch(1);
+ private JobApi.MetricResults monitoringInfos;
+
+ private TestPipelineResult(State state, JobApi.MetricResults monitoringInfos) {
+ this.state = state;
+ this.monitoringInfos = monitoringInfos;
+ }
+
+ private TestPipelineResult(State state) {
+ this(state, JobApi.MetricResults.newBuilder().build());
+ }
+
+ @Override
+ public State getState() {
+ return state;
+ }
+
+ @Override
+ public State cancel() {
+ cancelLatch.countDown();
+ return State.CANCELLED;
+ }
+
+ @Override
+ public State waitUntilFinish(Duration duration) {
+ return null;
+ }
+
+ @Override
+ public State waitUntilFinish() {
+ return null;
+ }
+
+ @Override
+ public MetricResults metrics() {
+ return null;
+ }
+
+ @Override
+ public JobApi.MetricResults portableMetrics() {
+ return monitoringInfos;
+ }
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java
new file mode 100644
index 0000000..3b4a71d
--- /dev/null
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+import static org.mockito.MockitoAnnotations.initMocks;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.channels.WritableByteChannel;
+import java.util.Collections;
+import java.util.List;
+import java.util.jar.Attributes.Name;
+import java.util.jar.JarEntry;
+import java.util.jar.JarFile;
+import java.util.jar.JarInputStream;
+import java.util.jar.JarOutputStream;
+import java.util.jar.Manifest;
+import java.util.stream.Collectors;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactMetadata;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.GetManifestResponse;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi.ProxyManifest.Location;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineJarCreator.ArtifactRetriever;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.Mock;
+
+/** Unit tests for {@link PortablePipelineJarCreator}. */
+@RunWith(JUnit4.class)
+public class PortablePipelineJarCreatorTest implements Serializable {
+
+ @Mock private JarFile inputJar;
+ @Mock private JarOutputStream outputStream;
+ @Mock private WritableByteChannel outputChannel;
+ @Mock private ArtifactRetriever retrievalServiceStub;
+ private PortablePipelineJarCreator jarCreator;
+
+ @Before
+ public void setup() throws IOException {
+ initMocks(this);
+ JarInputStream emptyInputStream = new JarInputStream(new ByteArrayInputStream(new byte[0]));
+ when(inputJar.getInputStream(any())).thenReturn(emptyInputStream);
+ when(retrievalServiceStub.getArtifact(any())).thenReturn(Collections.emptyIterator());
+ jarCreator = new PortablePipelineJarCreator(null);
+ jarCreator.outputStream = outputStream;
+ jarCreator.outputChannel = outputChannel;
+ }
+
+ @Test
+ public void testCopyResourcesFromJar_copiesResources() throws IOException {
+ List<JarEntry> entries =
+ ImmutableList.of(new JarEntry("foo"), new JarEntry("bar"), new JarEntry("baz"));
+ when(inputJar.entries()).thenReturn(Collections.enumeration(entries));
+ jarCreator.copyResourcesFromJar(inputJar);
+ verify(outputStream, times(3)).putNextEntry(any());
+ }
+
+ @Test
+ public void testCopyResourcesFromJar_ignoresManifest() throws IOException {
+ List<JarEntry> manifestEntry = ImmutableList.of(new JarEntry(JarFile.MANIFEST_NAME));
+ when(inputJar.entries()).thenReturn(Collections.enumeration(manifestEntry));
+ jarCreator.copyResourcesFromJar(inputJar);
+ verify(outputStream, never()).putNextEntry(any());
+ }
+
+ @Test
+ public void testCopyResourcesFromJar_ignoresDuplicates() throws IOException {
+ List<JarEntry> duplicateEntries = ImmutableList.of(new JarEntry("foo"), new JarEntry("foo"));
+ when(inputJar.entries()).thenReturn(Collections.enumeration(duplicateEntries));
+ jarCreator.copyResourcesFromJar(inputJar);
+ verify(outputStream, times(1)).putNextEntry(any());
+ }
+
+ @Test
+ public void testCopyStagedArtifacts_returnsProxyManifest() throws IOException {
+ ArtifactMetadata artifact1 = ArtifactMetadata.newBuilder().setName("foo").build();
+ ArtifactMetadata artifact2 = ArtifactMetadata.newBuilder().setName("bar").build();
+ List<ArtifactMetadata> artifacts = ImmutableList.of(artifact1, artifact2);
+ ArtifactApi.Manifest manifest =
+ ArtifactApi.Manifest.newBuilder().addAllArtifact(artifacts).build();
+ when(retrievalServiceStub.getManifest(any()))
+ .thenReturn(GetManifestResponse.newBuilder().setManifest(manifest).build());
+
+ ProxyManifest proxyManifest =
+ jarCreator.copyStagedArtifacts("retrievalToken", retrievalServiceStub, "job");
+
+ assertEquals(manifest, proxyManifest.getManifest());
+ List<String> outputArtifactNames =
+ proxyManifest.getLocationList().stream()
+ .map(Location::getName)
+ .collect(Collectors.toList());
+ assertThat(outputArtifactNames, containsInAnyOrder("foo", "bar"));
+ }
+
+ @Test
+ public void testCopyStagedArtifacts_copiesArtifacts() throws IOException {
+ ArtifactMetadata artifact1 = ArtifactMetadata.newBuilder().setName("foo").build();
+ ArtifactMetadata artifact2 = ArtifactMetadata.newBuilder().setName("bar").build();
+ List<ArtifactMetadata> artifacts = ImmutableList.of(artifact1, artifact2);
+ ArtifactApi.Manifest manifest =
+ ArtifactApi.Manifest.newBuilder().addAllArtifact(artifacts).build();
+ when(retrievalServiceStub.getManifest(any()))
+ .thenReturn(GetManifestResponse.newBuilder().setManifest(manifest).build());
+
+ jarCreator.copyStagedArtifacts("retrievalToken", retrievalServiceStub, "job");
+
+ verify(outputStream, times(2)).putNextEntry(any());
+ }
+
+ private static class FakePipelineRunnner {
+ public static void main(String[] args) {
+ System.out.println("Hello world");
+ }
+ }
+
+ @Test
+ public void testCreateManifest_withMainMethod() {
+ Manifest manifest = jarCreator.createManifest(FakePipelineRunnner.class, "job");
+ assertEquals(
+ FakePipelineRunnner.class.getName(),
+ manifest.getMainAttributes().getValue(Name.MAIN_CLASS));
+ }
+
+ private static class EmptyPipelineRunner {}
+
+ @Test
+ public void testCreateManifest_withoutMainMethod() {
+ Manifest manifest = jarCreator.createManifest(EmptyPipelineRunner.class, "job");
+ assertNull(manifest.getMainAttributes().getValue(Name.MAIN_CLASS));
+ }
+
+ private static class EvilPipelineRunner {
+ public static int main(String[] args) {
+ return 0;
+ }
+ }
+
+ @Test
+ public void testCreateManifest_withInvalidMainMethod() {
+ Manifest manifest = jarCreator.createManifest(EvilPipelineRunner.class, "job");
+ assertNull(manifest.getMainAttributes().getValue(Name.MAIN_CLASS));
+ }
+}
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java
index 16397fc..3bfda79 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/logging/GrpcLoggingServiceTest.java
@@ -37,9 +37,9 @@
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.InProcessServerFactory;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -62,7 +62,7 @@
Collection<Callable<Void>> tasks = new ArrayList<>();
for (int i = 1; i <= 3; ++i) {
- final int instructionReference = i;
+ final int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForServerHangup = new CountDownLatch(1);
@@ -74,8 +74,7 @@
TestStreams.withOnNext(messageDiscarder)
.withOnCompleted(new CountDown(waitForServerHangup))
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
outboundObserver.onCompleted();
waitForServerHangup.await();
return null;
@@ -105,7 +104,7 @@
Collection<Callable<Void>> tasks = new ArrayList<>();
for (int i = 1; i <= 3; ++i) {
- final int instructionReference = i;
+ final int instructionId = i;
tasks.add(
() -> {
CountDownLatch waitForTermination = new CountDownLatch(1);
@@ -118,9 +117,8 @@
TestStreams.withOnNext(messageDiscarder)
.withOnError(new CountDown(waitForTermination))
.build());
- outboundObserver.onNext(
- createLogsWithIds(instructionReference, -instructionReference));
- outboundObserver.onError(new RuntimeException("Client " + instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId, -instructionId));
+ outboundObserver.onError(new RuntimeException("Client " + instructionId));
waitForTermination.await();
return null;
});
@@ -141,7 +139,7 @@
GrpcFnServer.allocatePortAndCreateFor(service, InProcessServerFactory.create())) {
for (int i = 1; i <= 3; ++i) {
- final long instructionReference = i;
+ final long instructionId = i;
futures.add(
executorService.submit(
() -> {
@@ -156,7 +154,7 @@
TestStreams.withOnNext(messageDiscarder)
.withOnCompleted(new CountDown(waitForServerHangup))
.build());
- outboundObserver.onNext(createLogsWithIds(instructionReference));
+ outboundObserver.onNext(createLogsWithIds(instructionId));
waitForServerHangup.await();
return null;
}
@@ -181,7 +179,7 @@
}
private BeamFnApi.LogEntry createLogWithId(long id) {
- return BeamFnApi.LogEntry.newBuilder().setInstructionReference(Long.toString(id)).build();
+ return BeamFnApi.LogEntry.newBuilder().setInstructionId(Long.toString(id)).build();
}
private static class CollectionAppendingLogWriter implements LogWriter {
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionServiceTest.java
index b36a53e..850a070 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/provisioning/StaticGrpcProvisionServiceTest.java
@@ -31,11 +31,11 @@
import org.apache.beam.model.fnexecution.v1.ProvisionServiceGrpc.ProvisionServiceBlockingStub;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.InProcessServerFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ListValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.NullValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Value;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ListValue;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.NullValue;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Value;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
index 34bb512..f8b3f29 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
@@ -31,8 +31,8 @@
import java.util.concurrent.TimeUnit;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -75,7 +75,7 @@
// send state request
BeamFnApi.StateRequest request =
- BeamFnApi.StateRequest.newBuilder().setInstructionReference(bundleInstructionId).build();
+ BeamFnApi.StateRequest.newBuilder().setInstructionId(bundleInstructionId).build();
requestObserver.onNext(request);
// assert behavior
@@ -113,7 +113,7 @@
// send state request
BeamFnApi.StateRequest request =
- BeamFnApi.StateRequest.newBuilder().setInstructionReference(bundleInstructionId).build();
+ BeamFnApi.StateRequest.newBuilder().setInstructionId(bundleInstructionId).build();
requestObserver.onNext(request);
// wait for response
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtilsTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtilsTest.java
index a37b12b..83b983d 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtilsTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtilsTest.java
@@ -23,8 +23,8 @@
import java.util.Arrays;
import java.util.List;
import java.util.Map;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
/** Tests for {@link PipelineTranslatorUtils}. */
diff --git a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCodersTest.java b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCodersTest.java
index 8bb2fab..303ca80 100644
--- a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCodersTest.java
+++ b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/wire/LengthPrefixUnknownCodersTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.coders.LengthPrefixCoder;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/runners/jet-experimental/build.gradle b/runners/jet-experimental/build.gradle
deleted file mode 100644
index 4ac9867..0000000
--- a/runners/jet-experimental/build.gradle
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import groovy.json.JsonOutput
-
-plugins { id 'org.apache.beam.module' }
-applyJavaNature()
-
-description = "Apache Beam :: Runners :: Hazelcast Jet"
-
-evaluationDependsOn(":sdks:java:core")
-evaluationDependsOn(":runners:core-java")
-
-project.ext {
- jet_version = '3.0'
- hazelcast_version = '3.12'
-}
-
-configurations {
- validatesRunner
-}
-
-dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:core-java", configuration: "shadow")
- shadow "com.hazelcast.jet:hazelcast-jet:$jet_version"
-
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
- shadowTest "com.hazelcast:hazelcast:$hazelcast_version:tests"
- shadowTest "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
-
- validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadowTest")
-}
-
-task validatesRunnerBatch(type: Test) {
- group = "Verification"
- systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
- "--runner=TestJetRunner",
- "--jetGroupName=jet",
- "--jetLocalParallelism=2"
- ])
-
- classpath = configurations.validatesRunner
- testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
- useJUnit {
- includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
- excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse doesn't cooperate properly with Jet when multiple cluster members are used
- exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality not yet in the runner
- }
-
- maxHeapSize = '4g'
-}
-
-task validatesRunner {
- group = "Verification"
- description "Validates Jet runner"
- dependsOn validatesRunnerBatch
-}
-
-spotless {
- java {
- paddedCell()
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
deleted file mode 100644
index 3886e85..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
+++ /dev/null
@@ -1,220 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.hazelcast.jet.core.DAG;
-import com.hazelcast.jet.core.Edge;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.ProcessorMetaSupplier;
-import com.hazelcast.jet.core.Vertex;
-import com.hazelcast.jet.function.FunctionEx;
-import com.hazelcast.jet.function.SupplierEx;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-
-/** Utility class for wiring up Jet DAGs based on Beam pipelines. */
-public class DAGBuilder {
-
- private final DAG dag = new DAG();
- private final int localParallelism;
-
- private final Map<String, Vertex> edgeStartPoints = new HashMap<>();
- private final Map<String, List<Vertex>> edgeEndPoints = new HashMap<>();
- private final Map<String, Coder> edgeCoders = new HashMap<>();
- private final Map<String, String> pCollsOfEdges = new HashMap<>();
-
- private final List<WiringListener> listeners = new ArrayList<>();
-
- private int vertexId = 0;
-
- DAGBuilder(JetPipelineOptions options) {
- this.localParallelism = options.getJetLocalParallelism();
- }
-
- DAG getDag() {
- wireUp();
- return dag;
- }
-
- void registerConstructionListeners(WiringListener listener) {
- listeners.add(listener);
- }
-
- String newVertexId(String transformName) {
- return vertexId++ + " (" + transformName + ")";
- }
-
- void registerCollectionOfEdge(String edgeId, String pCollId) {
- String prevPCollId = pCollsOfEdges.put(edgeId, pCollId);
- if (prevPCollId != null) {
- throw new RuntimeException("Oops!");
- }
- }
-
- void registerEdgeStartPoint(String edgeId, Vertex vertex, Coder coder) {
- Objects.requireNonNull(edgeId);
- Objects.requireNonNull(vertex);
- Objects.requireNonNull(coder);
-
- Vertex prevVertex = edgeStartPoints.put(edgeId, vertex);
- if (prevVertex != null) {
- throw new RuntimeException("Oops!");
- }
-
- Coder prevCoder = edgeCoders.put(edgeId, coder);
- if (prevCoder != null) {
- throw new RuntimeException("Oops!");
- }
- }
-
- void registerEdgeEndPoint(String edgeId, Vertex vertex) {
- edgeEndPoints.computeIfAbsent(edgeId, x -> new ArrayList<>()).add(vertex);
- }
-
- Vertex addVertex(String id, ProcessorMetaSupplier processorMetaSupplier) {
- return dag.newVertex(id, processorMetaSupplier);
- }
-
- Vertex addVertex(String id, SupplierEx<Processor> processor) {
- return dag.newVertex(id, processor).localParallelism(localParallelism);
- }
-
- private void wireUp() {
- new WiringInstaller().wireUp();
- }
-
- /**
- * Listener that can be registered with a {@link DAGBuilder} in order to be notified when edges
- * are being registered.
- */
- public interface WiringListener {
-
- void isOutboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId);
-
- void isInboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId);
- }
-
- private class WiringInstaller {
-
- private final Map<Vertex, Integer> inboundOrdinals = new HashMap<>();
- private final Map<Vertex, Integer> outboundOrdinals = new HashMap<>();
-
- void wireUp() {
- Collection<String> edgeIds = new HashSet<>();
- edgeIds.addAll(edgeStartPoints.keySet());
- edgeIds.addAll(edgeEndPoints.keySet());
-
- for (String edgeId : edgeIds) {
- String pCollId = pCollsOfEdges.get(edgeId);
- if (pCollId == null) {
- throw new RuntimeException("Oops!");
- }
-
- Vertex sourceVertex = edgeStartPoints.get(edgeId);
- if (sourceVertex == null) {
- throw new RuntimeException("Oops!");
- }
-
- Coder edgeCoder = edgeCoders.get(edgeId);
- if (edgeCoder == null) {
- throw new RuntimeException("Oops!");
- }
-
- List<Vertex> destinationVertices =
- edgeEndPoints.getOrDefault(edgeId, Collections.emptyList());
- boolean sideInputEdge = edgeId.contains("PCollectionView"); // todo: this is a hack!
- for (Vertex destinationVertex : destinationVertices) {
- addEdge(sourceVertex, destinationVertex, edgeCoder, edgeId, pCollId, sideInputEdge);
- }
- }
- }
-
- private void addEdge(
- Vertex sourceVertex,
- Vertex destinationVertex,
- Coder coder,
- String edgeId,
- String pCollId,
- boolean sideInputEdge) {
- // todo: set up the edges properly, including other aspects too, like parallelism
-
- try {
- Edge edge =
- Edge.from(sourceVertex, getNextFreeOrdinal(sourceVertex, false))
- .to(destinationVertex, getNextFreeOrdinal(destinationVertex, true))
- .distributed(); // todo: why is it always distributed?
- if (sideInputEdge) {
- edge = edge.broadcast();
- } else {
- edge =
- edge.partitioned(
- new PartitionedKeyExtractor(
- coder)); // todo: we likely don't need to partition everything
- }
- dag.edge(edge);
-
- String sourceVertexName = sourceVertex.getName();
- String destinationVertexName = destinationVertex.getName();
- for (WiringListener listener : listeners) {
- listener.isInboundEdgeOfVertex(edge, edgeId, pCollId, destinationVertexName);
- listener.isOutboundEdgeOfVertex(edge, edgeId, pCollId, sourceVertexName);
- }
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
-
- private int getNextFreeOrdinal(Vertex vertex, boolean inbound) {
- Map<Vertex, Integer> ordinals = inbound ? inboundOrdinals : outboundOrdinals;
- int nextOrdinal = 1 + ordinals.getOrDefault(vertex, -1);
- ordinals.put(vertex, nextOrdinal);
- return nextOrdinal;
- }
- }
-
- private static class PartitionedKeyExtractor implements FunctionEx<byte[], Object> {
- private final Coder coder;
-
- PartitionedKeyExtractor(Coder coder) {
- this.coder = coder;
- }
-
- @Override
- public Object applyEx(byte[] b) throws Exception {
- Object t = CoderUtils.decodeFromByteArray(coder, b); // todo: decoding twice....
- Object key = null;
- if (t instanceof WindowedValue) {
- t = ((WindowedValue) t).getValue();
- }
- if (t instanceof KV) {
- key = ((KV) t).getKey();
- }
- return key == null ? "all" : key; // todo: why "all"?
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
deleted file mode 100644
index c4a3d1c..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.hazelcast.jet.core.DAG;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.sdk.runners.TransformHierarchy;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.values.PValue;
-
-/** Logic that specifies how to apply translations when traversing the nodes of a Beam pipeline. */
-class JetGraphVisitor extends Pipeline.PipelineVisitor.Defaults {
-
- private final JetTranslationContext translationContext;
-
- private boolean finalized = false;
-
- JetGraphVisitor(JetPipelineOptions options) {
- this.translationContext = new JetTranslationContext(options);
- }
-
- @Override
- public CompositeBehavior enterCompositeTransform(TransformHierarchy.Node node) {
- if (finalized) {
- throw new IllegalStateException("Attempting to traverse an already finalized pipeline!");
- }
-
- PTransform<?, ?> transform = node.getTransform();
- if (transform != null) {
- JetTransformTranslator<?> translator = JetTransformTranslators.getTranslator(transform);
- if (translator != null) {
- translate(node, translator);
- return CompositeBehavior.DO_NOT_ENTER_TRANSFORM;
- }
- }
- return CompositeBehavior.ENTER_TRANSFORM;
- }
-
- @Override
- public void leaveCompositeTransform(TransformHierarchy.Node node) {
- if (finalized) {
- throw new IllegalStateException("Attempting to traverse an already finalized pipeline!");
- }
- if (node.isRootNode()) {
- finalized = true;
- }
- }
-
- @Override
- public void visitPrimitiveTransform(TransformHierarchy.Node node) {
- PTransform<?, ?> transform = node.getTransform();
- JetTransformTranslator<?> translator = JetTransformTranslators.getTranslator(transform);
- if (translator == null) {
- String transformUrn = PTransformTranslation.urnForTransform(transform);
- throw new UnsupportedOperationException(
- "The transform " + transformUrn + " is currently not supported.");
- }
- translate(node, translator);
- }
-
- @Override
- public void visitValue(PValue value, TransformHierarchy.Node producer) {
- // do nothing here
- }
-
- DAG getDAG() {
- return translationContext.getDagBuilder().getDag();
- }
-
- private <T extends PTransform<?, ?>> void translate(
- TransformHierarchy.Node node, JetTransformTranslator<?> translator) {
- @SuppressWarnings("unchecked")
- JetTransformTranslator<T> typedTranslator = (JetTransformTranslator<T>) translator;
- Pipeline pipeline = getPipeline();
- AppliedPTransform<?, ?, ?> appliedTransform = node.toAppliedPTransform(pipeline);
- typedTranslator.translate(pipeline, appliedTransform, node, translationContext);
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
deleted file mode 100644
index 5b5a30e..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import org.apache.beam.sdk.options.Default;
-import org.apache.beam.sdk.options.Description;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.Validation;
-
-/** Pipeline options specific to the Jet runner. */
-public interface JetPipelineOptions extends PipelineOptions {
-
- @Description("Name of Jet group")
- @Validation.Required
- String getJetGroupName();
-
- void setJetGroupName(String jetGroupName);
-
- @Description("Local parallelism of Jet nodes")
- @Validation.Required
- @Default.Integer(-1)
- Integer getJetLocalParallelism();
-
- void setJetLocalParallelism(Integer localParallelism);
-
- @Description(
- "Specifies if the Runner should start its own Jet cluster") // todo: this is a hack, we will
- // need to use a real, stand-alone
- // cluster and submit the runner
- // code in a Jar to it + connect
- // via network
- @Validation.Required
- @Default.Boolean(true)
- Boolean getJetStartOwnCluster();
-
- void setJetStartOwnCluster(Boolean startOwnCluser);
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
deleted file mode 100644
index 10cf7a1..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
+++ /dev/null
@@ -1,119 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.hazelcast.jet.IMapJet;
-import com.hazelcast.jet.Job;
-import com.hazelcast.jet.core.JobStatus;
-import java.util.Objects;
-import javax.annotation.concurrent.GuardedBy;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.jet.metrics.JetMetricResults;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.metrics.MetricResults;
-import org.joda.time.Duration;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** Jet specific implementation of {@link PipelineResult}. */
-public class JetPipelineResult implements PipelineResult {
-
- private static final Logger LOG = LoggerFactory.getLogger(JetRunner.class);
-
- private final IMapJet<String, MetricUpdates> metricsAccumulator;
- private final JetMetricResults metricResults = new JetMetricResults();
-
- @GuardedBy("this")
- private Job job = null;
-
- @GuardedBy("this")
- private State state = State.UNKNOWN;
-
- JetPipelineResult(IMapJet<String, MetricUpdates> metricsAccumulator) {
- this.metricsAccumulator = Objects.requireNonNull(metricsAccumulator);
- this.metricsAccumulator.addEntryListener(metricResults, true);
- }
-
- private static State getState(Job job) {
- JobStatus status = job.getStatus();
- switch (status) {
- case COMPLETED:
- return State.DONE;
- case COMPLETING:
- case RUNNING:
- case STARTING:
- return State.RUNNING;
- case FAILED:
- return State.FAILED;
- case NOT_RUNNING:
- case SUSPENDED:
- return State.STOPPED;
- default:
- LOG.warn("Unhandled " + JobStatus.class.getSimpleName() + ": " + status.name() + "!");
- return State.UNKNOWN;
- }
- }
-
- synchronized void setJob(Job job) {
- Job nonNullJob = job == null ? this.job : job;
- this.state = getState(nonNullJob);
- this.job = job;
- }
-
- @Override
- public synchronized State getState() {
- if (job != null) {
- state = getState(job);
- }
- return state;
- }
-
- @Override
- public synchronized State cancel() {
- if (job != null) {
- job.cancel();
- job = null;
- state = State.STOPPED;
- }
- return state;
- }
-
- @Override
- public State waitUntilFinish(Duration duration) {
- return waitUntilFinish(); // todo: how to time out?
- }
-
- @Override
- public synchronized State waitUntilFinish() {
- if (job != null) {
- try {
- job.join();
- } catch (Exception e) {
- e.printStackTrace(); // todo: what to do?
- return State.FAILED;
- }
- state = getState(job);
- }
- return state;
- }
-
- @Override
- public MetricResults metrics() {
- return metricResults;
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunner.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunner.java
deleted file mode 100644
index 562f6a4..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunner.java
+++ /dev/null
@@ -1,152 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.hazelcast.client.config.ClientConfig;
-import com.hazelcast.jet.IMapJet;
-import com.hazelcast.jet.Jet;
-import com.hazelcast.jet.JetInstance;
-import com.hazelcast.jet.Job;
-import com.hazelcast.jet.core.DAG;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.function.Function;
-import org.apache.beam.runners.core.construction.UnconsumedReads;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.jet.metrics.JetMetricsContainer;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.runners.PTransformOverride;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/** Jet specific implementation of Beam's {@link PipelineRunner}. */
-public class JetRunner extends PipelineRunner<PipelineResult> {
-
- private static final Logger LOG = LoggerFactory.getLogger(JetRunner.class);
- private final JetPipelineOptions options;
- private final Function<ClientConfig, JetInstance> jetClientSupplier;
-
- private JetRunner(
- PipelineOptions options, Function<ClientConfig, JetInstance> jetClientSupplier) {
- this.options = validate(options.as(JetPipelineOptions.class));
- this.jetClientSupplier = jetClientSupplier;
- }
-
- public static JetRunner fromOptions(PipelineOptions options) {
- return fromOptions(options, Jet::newJetClient);
- }
-
- public static JetRunner fromOptions(
- PipelineOptions options, Function<ClientConfig, JetInstance> jetClientSupplier) {
- return new JetRunner(options, jetClientSupplier);
- }
-
- private static List<PTransformOverride> getDefaultOverrides() {
- // return Collections.singletonList(JavaReadViaImpulse.boundedOverride()); //todo: needed
- // once we start using GreedyPipelineFuser
- return Collections.emptyList();
- }
-
- private static JetPipelineOptions validate(JetPipelineOptions options) {
- if (options.getJetGroupName() == null) {
- throw new IllegalArgumentException("Jet group name not set in options");
- }
-
- Integer localParallelism = options.getJetLocalParallelism();
- if (localParallelism == null) {
- throw new IllegalArgumentException("Jet node local parallelism must be specified");
- }
- if (localParallelism != -1 && localParallelism < 1) {
- throw new IllegalArgumentException("Jet node local parallelism must be >1 or -1");
- }
-
- return options;
- }
-
- @Override
- public PipelineResult run(Pipeline pipeline) {
- Boolean startOwnCluster = options.getJetStartOwnCluster();
- if (startOwnCluster) {
- Collection<JetInstance> jetInstances =
- Arrays.asList(Jet.newJetInstance(), Jet.newJetInstance());
- LOG.info("Started " + jetInstances.size() + " Jet cluster members");
- }
- try {
- return runInternal(pipeline);
- } finally {
- if (startOwnCluster) {
- Jet.shutdownAll();
- LOG.info("Stopped all Jet cluster members");
- }
- }
- }
-
- private PipelineResult runInternal(Pipeline pipeline) {
- try {
- normalize(pipeline);
- DAG dag = translate(pipeline);
- return run(dag);
- } catch (UnsupportedOperationException uoe) {
- LOG.error("Failed running pipeline!", uoe);
- return new FailedRunningPipelineResults(uoe);
- }
- }
-
- private void normalize(Pipeline pipeline) {
- pipeline.replaceAll(getDefaultOverrides());
- UnconsumedReads.ensureAllReadsConsumed(pipeline);
- }
-
- private DAG translate(Pipeline pipeline) {
- JetGraphVisitor graphVisitor = new JetGraphVisitor(options);
- pipeline.traverseTopologically(graphVisitor);
- return graphVisitor.getDAG();
- }
-
- private JetPipelineResult run(DAG dag) {
- JetInstance jet = getJetInstance(options);
-
- IMapJet<String, MetricUpdates> metricsAccumulator =
- jet.getMap(JetMetricsContainer.METRICS_ACCUMULATOR_NAME);
- metricsAccumulator.clear();
- Job job = jet.newJob(dag);
-
- JetPipelineResult result = new JetPipelineResult(metricsAccumulator);
- result.setJob(job);
-
- job.join();
- result.setJob(null);
- job.cancel();
- jet.shutdown();
-
- return result;
- }
-
- private JetInstance getJetInstance(JetPipelineOptions options) {
- String jetGroupName = options.getJetGroupName();
-
- ClientConfig clientConfig = new ClientConfig();
- clientConfig.getGroupConfig().setName(jetGroupName);
- return jetClientSupplier.apply(clientConfig);
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
deleted file mode 100644
index 7929266..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-
-/**
- * Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
- * JetRunner}.
- *
- * <p>{@link AutoService} will register Apex's implementations of the {@link PipelineRunner} and
- * {@link PipelineOptions} as available pipeline runner services.
- */
-public final class JetRunnerRegistrar {
- private JetRunnerRegistrar() {}
-
- /** Registers the {@link JetRunner}. */
- @AutoService(PipelineRunnerRegistrar.class)
- public static class Runner implements PipelineRunnerRegistrar {
- @Override
- public Iterable<Class<? extends PipelineRunner<?>>> getPipelineRunners() {
- return ImmutableList.of(JetRunner.class);
- }
- }
-
- /** Registers the {@link JetPipelineOptions}. */
- @AutoService(PipelineOptionsRegistrar.class)
- public static class Options implements PipelineOptionsRegistrar {
- @Override
- public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
- return ImmutableList.of(JetPipelineOptions.class);
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
deleted file mode 100644
index c672eaf..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
+++ /dev/null
@@ -1,462 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.ProcessorMetaSupplier;
-import com.hazelcast.jet.core.Vertex;
-import com.hazelcast.jet.function.SupplierEx;
-import com.hazelcast.jet.impl.util.ExceptionUtil;
-import java.io.IOException;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-import org.apache.beam.runners.core.construction.CreatePCollectionViewTranslation;
-import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.ParDoTranslation;
-import org.apache.beam.runners.core.construction.ReadTranslation;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.jet.processors.AssignWindowP;
-import org.apache.beam.runners.jet.processors.BoundedSourceP;
-import org.apache.beam.runners.jet.processors.FlattenP;
-import org.apache.beam.runners.jet.processors.ImpulseP;
-import org.apache.beam.runners.jet.processors.ParDoP;
-import org.apache.beam.runners.jet.processors.StatefulParDoP;
-import org.apache.beam.runners.jet.processors.TestStreamP;
-import org.apache.beam.runners.jet.processors.ViewP;
-import org.apache.beam.runners.jet.processors.WindowGroupP;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.io.BoundedSource;
-import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.sdk.runners.TransformHierarchy.Node;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.sdk.values.PCollectionTuple;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-
-@SuppressWarnings("unchecked")
-class JetTransformTranslators {
-
- /** A map from a Transform URN to the translator. */
- private static final Map<String, JetTransformTranslator> TRANSLATORS = new HashMap<>();
-
- static {
- TRANSLATORS.put(PTransformTranslation.READ_TRANSFORM_URN, new ReadSourceTranslator());
- TRANSLATORS.put(PTransformTranslation.CREATE_VIEW_TRANSFORM_URN, new CreateViewTranslator());
- TRANSLATORS.put(PTransformTranslation.PAR_DO_TRANSFORM_URN, new ParDoTranslator());
- TRANSLATORS.put(PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN, new GroupByKeyTranslator());
- TRANSLATORS.put(PTransformTranslation.FLATTEN_TRANSFORM_URN, new FlattenTranslator());
- TRANSLATORS.put(PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN, new WindowTranslator());
- TRANSLATORS.put(PTransformTranslation.IMPULSE_TRANSFORM_URN, new ImpulseTranslator());
- TRANSLATORS.put(PTransformTranslation.TEST_STREAM_TRANSFORM_URN, new TestStreamTranslator());
- }
-
- static JetTransformTranslator<?> getTranslator(PTransform<?, ?> transform) {
- String urn = PTransformTranslation.urnForTransformOrNull(transform);
- return urn == null ? null : TRANSLATORS.get(urn);
- }
-
- private static class ReadSourceTranslator<T>
- implements JetTransformTranslator<PTransform<PBegin, PCollection<T>>> {
-
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- if (!Utils.isBounded(appliedTransform)) {
- throw new UnsupportedOperationException(); // todo
- }
-
- BoundedSource<T> source;
- try {
- source =
- ReadTranslation.boundedSourceFromTransform(
- (AppliedPTransform<PBegin, PCollection<T>, PTransform<PBegin, PCollection<T>>>)
- appliedTransform);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
-
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder =
- Utils.getCoder((PCollection) Utils.getOutput(appliedTransform).getValue());
-
- String transformName = appliedTransform.getFullName();
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(transformName);
- SerializablePipelineOptions pipelineOptions = context.getOptions();
- ProcessorMetaSupplier processorSupplier =
- BoundedSourceP.supplier(source, pipelineOptions, outputCoder, vertexId);
-
- Vertex vertex = dagBuilder.addVertex(vertexId, processorSupplier);
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
- }
-
- private static class ParDoTranslator
- implements JetTransformTranslator<PTransform<PCollection, PCollectionTuple>> {
-
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- boolean usesStateOrTimers = Utils.usesStateOrTimers(appliedTransform);
- DoFn<?, ?> doFn = Utils.getDoFn(appliedTransform);
-
- Map<TupleTag<?>, PValue> outputs = Utils.getOutputs(appliedTransform);
-
- TupleTag<?> mainOutputTag;
- try {
- mainOutputTag = ParDoTranslation.getMainOutputTag(appliedTransform);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- Map<TupleTag<?>, Integer> outputMap = new HashMap<>();
- int count = 1;
- for (TupleTag<?> tag : outputs.keySet()) {
- if (!outputMap.containsKey(tag)) {
- outputMap.put(tag, count++);
- }
- }
- final WindowingStrategy<?, ?> windowingStrategy =
- Utils.getWindowingStrategy(appliedTransform);
-
- Map<TupleTag<?>, Coder<?>> outputValueCoders = Utils.getOutputValueCoders(appliedTransform);
- Map<TupleTag<?>, Coder> outputCoders =
- Utils.getCoders(Utils.getOutputs(appliedTransform), Map.Entry::getKey);
-
- String transformName = appliedTransform.getFullName();
- DAGBuilder dagBuilder = context.getDagBuilder();
- String stepId =
- transformName.contains("/")
- ? transformName.substring(0, transformName.indexOf('/'))
- : transformName;
- String vertexId =
- dagBuilder.newVertexId(transformName) + (usesStateOrTimers ? " - STATEFUL" : "");
- SerializablePipelineOptions pipelineOptions = context.getOptions();
- Coder inputValueCoder = Utils.getInput(appliedTransform).getCoder();
- Coder inputCoder = Utils.getCoder(Utils.getInput(appliedTransform));
- List<PCollectionView<?>> sideInputs = Utils.getSideInputs(appliedTransform);
- Map<? extends PCollectionView<?>, Coder> sideInputCoders =
- sideInputs.stream()
- .collect(Collectors.toMap(si -> si, si -> Utils.getCoder(si.getPCollection())));
- DoFnSchemaInformation doFnSchemaInformation =
- ParDoTranslation.getSchemaInformation(appliedTransform);
- SupplierEx<Processor> processorSupplier =
- usesStateOrTimers
- ? new StatefulParDoP.Supplier(
- stepId,
- vertexId,
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- pipelineOptions,
- mainOutputTag,
- outputMap.keySet(),
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- sideInputs)
- : new ParDoP.Supplier(
- stepId,
- vertexId,
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- pipelineOptions,
- mainOutputTag,
- outputMap.keySet(),
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- sideInputs);
-
- Vertex vertex = dagBuilder.addVertex(vertexId, processorSupplier);
- dagBuilder.registerConstructionListeners((DAGBuilder.WiringListener) processorSupplier);
-
- PValue mainInput = Utils.getMainInput(pipeline, node);
- dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(mainInput), vertex);
-
- Map<TupleTag<?>, PValue> additionalInputs = Utils.getAdditionalInputs(node);
- if (additionalInputs != null && !additionalInputs.isEmpty()) {
- for (TupleTag<?> tupleTag : additionalInputs.keySet()) {
- dagBuilder.registerEdgeEndPoint(tupleTag.getId(), vertex);
- }
- }
-
- for (Map.Entry<TupleTag<?>, PValue> entry : outputs.entrySet()) {
- TupleTag<?> pCollId = entry.getKey();
- String edgeId = Utils.getTupleTagId(entry.getValue());
- dagBuilder.registerCollectionOfEdge(edgeId, pCollId.getId());
- dagBuilder.registerEdgeStartPoint(edgeId, vertex, outputCoders.get(pCollId));
- }
-
- return vertex;
- }
- }
-
- private static class GroupByKeyTranslator<K, InputT>
- implements JetTransformTranslator<
- PTransform<PCollection<KV<K, InputT>>, PCollection<KV<K, Iterable<InputT>>>>> {
-
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- String transformName = appliedTransform.getFullName();
-
- PCollection<KV<K, InputT>> input = Utils.getInput(appliedTransform);
- Coder inputCoder = Utils.getCoder(input);
- Coder inputValueCoder = Utils.getInput(appliedTransform).getCoder();
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
-
- WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
-
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(transformName);
- Vertex vertex =
- dagBuilder.addVertex(
- vertexId,
- WindowGroupP.supplier(
- context.getOptions(),
- inputValueCoder,
- inputCoder,
- outputCoder,
- windowingStrategy,
- vertexId));
-
- dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
- }
-
- private static class CreateViewTranslator<T>
- implements JetTransformTranslator<PTransform<PCollection<T>, PCollection<T>>> {
-
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- PCollectionView<T> view;
- try {
- view =
- CreatePCollectionViewTranslation.getView(
- (AppliedPTransform<
- PCollection<T>, PCollection<T>, PTransform<PCollection<T>, PCollection<T>>>)
- appliedTransform);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
-
- String transformName = appliedTransform.getFullName();
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(transformName);
- PCollection<T> input = Utils.getInput(appliedTransform);
- Coder inputCoder = Utils.getCoder(input);
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
-
- Vertex vertex =
- dagBuilder.addVertex(
- vertexId,
- ViewP.supplier(inputCoder, outputCoder, input.getWindowingStrategy(), vertexId));
-
- dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
-
- String viewTag = Utils.getTupleTagId(view);
- dagBuilder.registerCollectionOfEdge(viewTag, view.getTagInternal().getId());
- dagBuilder.registerEdgeStartPoint(viewTag, vertex, outputCoder);
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
- }
-
- private static class FlattenTranslator<T>
- implements JetTransformTranslator<PTransform<PCollectionList<T>, PCollection<T>>> {
-
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- Collection<PValue> mainInputs = Utils.getMainInputs(pipeline, node);
- Map<String, Coder> inputCoders =
- Utils.getCoders(
- Utils.getInputs(appliedTransform), e -> Utils.getTupleTagId(e.getValue()));
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
-
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(appliedTransform.getFullName());
- FlattenP.Supplier processorSupplier =
- new FlattenP.Supplier(inputCoders, outputCoder, vertexId);
- Vertex vertex = dagBuilder.addVertex(vertexId, processorSupplier);
- dagBuilder.registerConstructionListeners(processorSupplier);
-
- for (PValue value : mainInputs) {
- PCollection<T> input = (PCollection<T>) value;
- dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
- }
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
- }
-
- private static class WindowTranslator<T>
- implements JetTransformTranslator<PTransform<PCollection<T>, PCollection<T>>> {
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- WindowingStrategy<T, BoundedWindow> windowingStrategy =
- (WindowingStrategy<T, BoundedWindow>)
- ((PCollection) Utils.getOutput(appliedTransform).getValue()).getWindowingStrategy();
-
- PCollection<WindowedValue> input = Utils.getInput(appliedTransform);
- Coder inputCoder = Utils.getCoder(input);
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder =
- Utils.getCoder((PCollection) Utils.getOutput(appliedTransform).getValue());
-
- String transformName = appliedTransform.getFullName();
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(transformName);
-
- Vertex vertex =
- dagBuilder.addVertex(
- vertexId,
- AssignWindowP.supplier(inputCoder, outputCoder, windowingStrategy, vertexId));
- dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
- }
-
- private static class ImpulseTranslator
- implements JetTransformTranslator<PTransform<PBegin, PCollection<byte[]>>> {
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- String transformName = appliedTransform.getFullName();
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(transformName);
-
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
- Vertex vertex = dagBuilder.addVertex(vertexId, ImpulseP.supplier(vertexId));
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
- }
-
- private static class TestStreamTranslator<T>
- implements JetTransformTranslator<PTransform<PBegin, PCollection<T>>> {
- @Override
- public Vertex translate(
- Pipeline pipeline,
- AppliedPTransform<?, ?, ?> appliedTransform,
- Node node,
- JetTranslationContext context) {
- String transformName = appliedTransform.getFullName();
- DAGBuilder dagBuilder = context.getDagBuilder();
- String vertexId = dagBuilder.newVertexId(transformName);
-
- TestStream<T> testStream = (TestStream<T>) appliedTransform.getTransform();
-
- // events in the transform are not serializable, we have to translate them. We'll also flatten
- // the collection.
- Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
- Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
- TestStream.TestStreamCoder<T> payloadCoder =
- TestStream.TestStreamCoder.of(testStream.getValueCoder());
- byte[] encodedPayload = getEncodedPayload(testStream, payloadCoder);
- Vertex vertex =
- dagBuilder.addVertex(
- vertexId, TestStreamP.supplier(encodedPayload, payloadCoder, outputCoder));
-
- String outputEdgeId = Utils.getTupleTagId(output.getValue());
- dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
- dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
- return vertex;
- }
-
- private static <T> byte[] getEncodedPayload(
- TestStream<T> testStream, TestStream.TestStreamCoder<T> coder) {
- try {
- return CoderUtils.encodeToByteArray(coder, testStream);
- } catch (CoderException e) {
- throw ExceptionUtil.rethrow(e);
- }
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java
deleted file mode 100644
index bc7da80..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java
+++ /dev/null
@@ -1,249 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import static com.hazelcast.jet.impl.util.ExceptionUtil.rethrow;
-import static java.util.stream.Collectors.toList;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.io.ObjectInputStream;
-import java.io.ObjectOutputStream;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.function.Function;
-import java.util.stream.Collectors;
-import java.util.stream.IntStream;
-import org.apache.beam.runners.core.construction.ParDoTranslation;
-import org.apache.beam.runners.core.construction.TransformInputs;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.ListCoder;
-import org.apache.beam.sdk.runners.AppliedPTransform;
-import org.apache.beam.sdk.runners.TransformHierarchy;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-
-/** Various common methods used by the Jet based runner. */
-public class Utils {
-
- public static String getTupleTagId(PValue value) {
- Map<TupleTag<?>, PValue> expansion = value.expand();
- return Iterables.getOnlyElement(expansion.keySet()).getId();
- }
-
- static PValue getMainInput(Pipeline pipeline, TransformHierarchy.Node node) {
- Collection<PValue> mainInputs = getMainInputs(pipeline, node);
- return mainInputs == null ? null : Iterables.getOnlyElement(mainInputs);
- }
-
- static Collection<PValue> getMainInputs(Pipeline pipeline, TransformHierarchy.Node node) {
- if (node.getTransform() == null) {
- return null;
- }
- return TransformInputs.nonAdditionalInputs(node.toAppliedPTransform(pipeline));
- }
-
- static Map<TupleTag<?>, PValue> getInputs(AppliedPTransform<?, ?, ?> appliedTransform) {
- return appliedTransform.getInputs();
- }
-
- static Map<TupleTag<?>, PValue> getAdditionalInputs(TransformHierarchy.Node node) {
- return node.getTransform() != null ? node.getTransform().getAdditionalInputs() : null;
- }
-
- static PCollection getInput(AppliedPTransform<?, ?, ?> appliedTransform) {
- if (appliedTransform.getTransform() == null) {
- return null;
- }
- return (PCollection)
- Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(appliedTransform));
- }
-
- static Map<TupleTag<?>, PValue> getOutputs(AppliedPTransform<?, ?, ?> appliedTransform) {
- if (appliedTransform.getTransform() == null) {
- return null;
- }
- return appliedTransform.getOutputs();
- }
-
- static Map.Entry<TupleTag<?>, PValue> getOutput(AppliedPTransform<?, ?, ?> appliedTransform) {
- return Iterables.getOnlyElement(getOutputs(appliedTransform).entrySet());
- }
-
- static <T> boolean isBounded(AppliedPTransform<?, ?, ?> appliedTransform) {
- return ((PCollection) getOutput(appliedTransform).getValue())
- .isBounded()
- .equals(PCollection.IsBounded.BOUNDED);
- }
-
- static Coder getCoder(PCollection pCollection) {
- if (pCollection == null) {
- return null;
- }
-
- if (pCollection.getWindowingStrategy() == null) {
- return pCollection.getCoder();
- } else {
- return WindowedValue.FullWindowedValueCoder.of(
- pCollection.getCoder(), pCollection.getWindowingStrategy().getWindowFn().windowCoder());
- }
- }
-
- static <T> Map<T, Coder> getCoders(
- Map<TupleTag<?>, PValue> pCollections,
- Function<Map.Entry<TupleTag<?>, PValue>, T> tupleTagExtractor) {
- return pCollections.entrySet().stream()
- .collect(Collectors.toMap(tupleTagExtractor, e -> getCoder((PCollection) e.getValue())));
- }
-
- static Map<TupleTag<?>, Coder<?>> getOutputValueCoders(
- AppliedPTransform<?, ?, ?> appliedTransform) {
- return appliedTransform.getOutputs().entrySet().stream()
- .filter(e -> e.getValue() instanceof PCollection)
- .collect(Collectors.toMap(Map.Entry::getKey, e -> ((PCollection) e.getValue()).getCoder()));
- }
-
- static List<PCollectionView<?>> getSideInputs(AppliedPTransform<?, ?, ?> appliedTransform) {
- PTransform<?, ?> transform = appliedTransform.getTransform();
- if (transform instanceof ParDo.MultiOutput) {
- ParDo.MultiOutput multiParDo = (ParDo.MultiOutput) transform;
- return multiParDo.getSideInputs();
- } else if (transform instanceof ParDo.SingleOutput) {
- ParDo.SingleOutput singleParDo = (ParDo.SingleOutput) transform;
- return singleParDo.getSideInputs();
- }
- return Collections.emptyList();
- }
-
- static boolean usesStateOrTimers(AppliedPTransform<?, ?, ?> appliedTransform) {
- try {
- return ParDoTranslation.usesStateOrTimers(appliedTransform);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- static DoFn<?, ?> getDoFn(AppliedPTransform<?, ?, ?> appliedTransform) {
- try {
- DoFn<?, ?> doFn = ParDoTranslation.getDoFn(appliedTransform);
- if (DoFnSignatures.signatureForDoFn(doFn).processElement().isSplittable()) {
- throw new IllegalStateException(
- "Not expected to directly translate splittable DoFn, should have been overridden: "
- + doFn); // todo
- }
- return doFn;
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- static WindowingStrategy<?, ?> getWindowingStrategy(AppliedPTransform<?, ?, ?> appliedTransform) {
- // assume that the windowing strategy is the same for all outputs
-
- Map<TupleTag<?>, PValue> outputs = getOutputs(appliedTransform);
-
- if (outputs == null || outputs.isEmpty()) {
- throw new IllegalStateException("No outputs defined.");
- }
-
- PValue taggedValue = outputs.values().iterator().next();
- checkState(
- taggedValue instanceof PCollection,
- "Within ParDo, got a non-PCollection output %s of type %s",
- taggedValue,
- taggedValue.getClass().getSimpleName());
- PCollection<?> coll = (PCollection<?>) taggedValue;
- return coll.getWindowingStrategy();
- }
-
- /**
- * Assigns the {@code list} to {@code count} sublists in a round-robin fashion. One call returns
- * the {@code index}-th sublist.
- *
- * <p>For example, for a 7-element list where {@code count == 3}, it would respectively return for
- * indices 0..2:
- *
- * <pre>
- * 0, 3, 6
- * 1, 4
- * 2, 5
- * </pre>
- */
- public static <T> List<T> roundRobinSubList(List<T> list, int index, int count) {
- if (index < 0 || index >= count) {
- throw new IllegalArgumentException("index=" + index + ", count=" + count);
- }
- return IntStream.range(0, list.size())
- .filter(i -> i % count == index)
- .mapToObj(list::get)
- .collect(toList());
- }
-
- /** Returns a deep clone of an object by serializing and deserializing it (ser-de). */
- @SuppressWarnings("unchecked")
- public static <T> T serde(T object) {
- try {
- ByteArrayOutputStream baos = new ByteArrayOutputStream();
- ObjectOutputStream oos = new ObjectOutputStream(baos);
- oos.writeObject(object);
- oos.close();
- byte[] byteData = baos.toByteArray();
- ByteArrayInputStream bais = new ByteArrayInputStream(byteData);
- return (T) new ObjectInputStream(bais).readObject();
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
-
- public static <T> byte[] encodeWindowedValue(WindowedValue<T> windowedValue, Coder coder) {
- try {
- return CoderUtils.encodeToByteArray(coder, windowedValue);
- } catch (IOException e) {
- throw rethrow(e);
- }
- }
-
- public static <T> WindowedValue<T> decodeWindowedValue(byte[] item, Coder coder) {
- try {
- return (WindowedValue<T>) CoderUtils.decodeFromByteArray(coder, item);
- } catch (IOException e) {
- throw rethrow(e);
- }
- }
-
- public static WindowedValue.FullWindowedValueCoder deriveIterableValueCoder(
- WindowedValue.FullWindowedValueCoder elementCoder) {
- return WindowedValue.FullWindowedValueCoder.of(
- ListCoder.of(elementCoder.getValueCoder()), elementCoder.getWindowCoder());
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
deleted file mode 100644
index 519e612..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
+++ /dev/null
@@ -1,159 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.metrics;
-
-import com.hazelcast.core.EntryEvent;
-import com.hazelcast.core.MapEvent;
-import com.hazelcast.map.listener.EntryAddedListener;
-import com.hazelcast.map.listener.MapClearedListener;
-import java.util.HashMap;
-import java.util.Map;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.metrics.DistributionData;
-import org.apache.beam.runners.core.metrics.GaugeData;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
-import org.apache.beam.sdk.metrics.DistributionResult;
-import org.apache.beam.sdk.metrics.GaugeResult;
-import org.apache.beam.sdk.metrics.MetricFiltering;
-import org.apache.beam.sdk.metrics.MetricKey;
-import org.apache.beam.sdk.metrics.MetricQueryResults;
-import org.apache.beam.sdk.metrics.MetricResult;
-import org.apache.beam.sdk.metrics.MetricResults;
-import org.apache.beam.sdk.metrics.MetricsFilter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-
-/** Jet specific {@link MetricResults}. */
-public class JetMetricResults extends MetricResults
- implements EntryAddedListener<String, MetricUpdates>, MapClearedListener {
-
- private final Map<MetricKey, Long> counters = new HashMap<>();
- private final Map<MetricKey, DistributionData> distributions = new HashMap<>();
- private final Map<MetricKey, GaugeData> gauges = new HashMap<>();
-
- private static MetricKey normalizeStepName(MetricKey key) {
- return MetricKey.create(
- JetMetricsContainer.ownerIdFromStepName(key.stepName()), key.metricName());
- }
-
- @Override
- public void entryAdded(EntryEvent<String, MetricUpdates> event) {
- merge(event.getValue());
- }
-
- @Override
- public void mapCleared(MapEvent mapEvent) {
- counters.clear();
- distributions.clear();
- gauges.clear();
- }
-
- private void merge(MetricUpdates metricUpdates) {
- mergeCounters(metricUpdates.counterUpdates());
- mergeDistributions(metricUpdates.distributionUpdates());
- mergeGauges(metricUpdates.gaugeUpdates());
- }
-
- private void mergeGauges(Iterable<MetricUpdate<GaugeData>> updates) {
- for (MetricUpdate<GaugeData> update : updates) {
- MetricKey key = normalizeStepName(update.getKey());
- GaugeData oldGauge = gauges.getOrDefault(key, GaugeData.empty());
- GaugeData updatedGauge = update.getUpdate().combine(oldGauge);
- gauges.put(key, updatedGauge);
- }
- }
-
- private void mergeDistributions(Iterable<MetricUpdate<DistributionData>> updates) {
- for (MetricUpdate<DistributionData> update : updates) {
- MetricKey key = normalizeStepName(update.getKey());
- DistributionData oldDistribution = distributions.getOrDefault(key, DistributionData.EMPTY);
- DistributionData updatedDistribution = update.getUpdate().combine(oldDistribution);
- distributions.put(key, updatedDistribution);
- }
- }
-
- private void mergeCounters(Iterable<MetricUpdate<Long>> updates) {
- for (MetricUpdate<Long> update : updates) {
- MetricKey key = normalizeStepName(update.getKey());
- Long oldValue = counters.getOrDefault(key, 0L);
- Long updatedValue = oldValue + update.getUpdate();
- counters.put(key, updatedValue);
- }
- }
-
- @Override
- public MetricQueryResults queryMetrics(@Nullable MetricsFilter filter) {
- return new QueryResults(filter);
- }
-
- private class QueryResults extends MetricQueryResults {
- private final MetricsFilter filter;
-
- private QueryResults(MetricsFilter filter) {
- this.filter = filter;
- }
-
- @Override
- public Iterable<MetricResult<Long>> getCounters() {
- return FluentIterable.from(counters.entrySet())
- .filter(matchesFilter(filter))
- .transform(this::counterUpdateToResult)
- .toList();
- }
-
- private MetricResult<Long> counterUpdateToResult(Map.Entry<MetricKey, Long> entry) {
- MetricKey key = entry.getKey();
- Long counter = entry.getValue();
- return MetricResult.create(key, counter, counter);
- }
-
- @Override
- public Iterable<MetricResult<DistributionResult>> getDistributions() {
- return FluentIterable.from(distributions.entrySet())
- .filter(matchesFilter(filter))
- .transform(this::distributionUpdateToResult)
- .toList();
- }
-
- private MetricResult<DistributionResult> distributionUpdateToResult(
- Map.Entry<MetricKey, DistributionData> entry) {
- MetricKey key = entry.getKey();
- DistributionResult distributionResult = entry.getValue().extractResult();
- return MetricResult.create(key, distributionResult, distributionResult);
- }
-
- @Override
- public Iterable<MetricResult<GaugeResult>> getGauges() {
- return FluentIterable.from(gauges.entrySet())
- .filter(matchesFilter(filter))
- .transform(this::gaugeUpdateToResult)
- .toList();
- }
-
- private MetricResult<GaugeResult> gaugeUpdateToResult(Map.Entry<MetricKey, GaugeData> entry) {
- MetricKey key = entry.getKey();
- GaugeResult gaugeResult = entry.getValue().extractResult();
- return MetricResult.create(key, gaugeResult, gaugeResult);
- }
-
- private Predicate<Map.Entry<MetricKey, ?>> matchesFilter(final MetricsFilter filter) {
- return entry -> MetricFiltering.matches(filter, entry.getKey());
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
deleted file mode 100644
index aba5f1d..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
+++ /dev/null
@@ -1,125 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.metrics;
-
-import com.hazelcast.jet.IMapJet;
-import com.hazelcast.jet.core.Processor;
-import java.io.Serializable;
-import java.util.HashMap;
-import java.util.Map;
-import org.apache.beam.runners.core.metrics.DistributionData;
-import org.apache.beam.runners.core.metrics.GaugeData;
-import org.apache.beam.runners.core.metrics.MetricUpdates;
-import org.apache.beam.sdk.metrics.Counter;
-import org.apache.beam.sdk.metrics.Distribution;
-import org.apache.beam.sdk.metrics.Gauge;
-import org.apache.beam.sdk.metrics.MetricKey;
-import org.apache.beam.sdk.metrics.MetricName;
-import org.apache.beam.sdk.metrics.MetricsContainer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-
-/** Jet specific implementation of {@link MetricsContainer}. */
-public class JetMetricsContainer implements MetricsContainer {
-
- public static final String METRICS_ACCUMULATOR_NAME =
- "metrics"; // todo: should be unique for the current pipeline, I guess
- private final String stepName;
- private final Map<MetricName, CounterImpl> counters = new HashMap<>();
- private final Map<MetricName, DistributionImpl> distributions = new HashMap<>();
- private final Map<MetricName, GaugeImpl> gauges = new HashMap<>();
- private final IMapJet<String, MetricUpdates> accumulator;
-
- public JetMetricsContainer(String stepName, Processor.Context context) {
- this.stepName = stepName + "/" + context.globalProcessorIndex();
- this.accumulator = context.jetInstance().getMap(METRICS_ACCUMULATOR_NAME);
- }
-
- public static String ownerIdFromStepName(String stepName) {
- return stepName.substring(0, stepName.indexOf('/'));
- }
-
- @Override
- public Counter getCounter(MetricName metricName) {
- return counters.computeIfAbsent(metricName, CounterImpl::new);
- }
-
- @Override
- public Distribution getDistribution(MetricName metricName) {
- return distributions.computeIfAbsent(metricName, DistributionImpl::new);
- }
-
- @Override
- public Gauge getGauge(MetricName metricName) {
- return gauges.computeIfAbsent(metricName, GaugeImpl::new);
- }
-
- public void flush() {
- ImmutableList<MetricUpdates.MetricUpdate<Long>> counters = extractUpdates(this.counters);
- ImmutableList<MetricUpdates.MetricUpdate<DistributionData>> distributions =
- extractUpdates(this.distributions);
- ImmutableList<MetricUpdates.MetricUpdate<GaugeData>> gauges = extractUpdates(this.gauges);
- MetricUpdates updates = new MetricUpdatesImpl(counters, distributions, gauges);
- accumulator.put(stepName, updates);
- }
-
- private <UpdateT, CellT extends AbstractMetric<UpdateT>>
- ImmutableList<MetricUpdates.MetricUpdate<UpdateT>> extractUpdates(
- Map<MetricName, CellT> cells) {
- ImmutableList.Builder<MetricUpdates.MetricUpdate<UpdateT>> updates = ImmutableList.builder();
- for (CellT cell : cells.values()) {
- UpdateT value = cell.getValue();
- if (value != null) {
- MetricKey key = MetricKey.create(stepName, cell.getName());
- MetricUpdates.MetricUpdate<UpdateT> update = MetricUpdates.MetricUpdate.create(key, value);
- updates.add(update);
- }
- }
- return updates.build();
- }
-
- private static class MetricUpdatesImpl extends MetricUpdates implements Serializable {
-
- private final Iterable<MetricUpdate<Long>> counters;
- private final Iterable<MetricUpdate<DistributionData>> distributions;
- private final Iterable<MetricUpdate<GaugeData>> gauges;
-
- public MetricUpdatesImpl(
- Iterable<MetricUpdate<Long>> counters,
- Iterable<MetricUpdate<DistributionData>> distributions,
- Iterable<MetricUpdate<GaugeData>> gauges) {
- this.counters = counters;
- this.distributions = distributions;
- this.gauges = gauges;
- }
-
- @Override
- public Iterable<MetricUpdate<Long>> counterUpdates() {
- return counters;
- }
-
- @Override
- public Iterable<MetricUpdate<DistributionData>> distributionUpdates() {
- return distributions;
- }
-
- @Override
- public Iterable<MetricUpdate<GaugeData>> gaugeUpdates() {
- return gauges;
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
deleted file mode 100644
index 7041729..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
+++ /dev/null
@@ -1,492 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.Edge;
-import com.hazelcast.jet.core.Inbox;
-import com.hazelcast.jet.core.Outbox;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.Watermark;
-import com.hazelcast.jet.function.SupplierEx;
-import java.util.ArrayDeque;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.Deque;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.function.Function;
-import java.util.stream.Collectors;
-import java.util.stream.IntStream;
-import javax.annotation.CheckReturnValue;
-import javax.annotation.Nonnull;
-import org.apache.beam.runners.core.DoFnRunner;
-import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.core.InMemoryStateInternals;
-import org.apache.beam.runners.core.NullSideInputReader;
-import org.apache.beam.runners.core.SideInputHandler;
-import org.apache.beam.runners.core.SideInputReader;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.jet.DAGBuilder;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.runners.jet.metrics.JetMetricsContainer;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.metrics.MetricsEnvironment;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
-import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
-import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-
-abstract class AbstractParDoP<InputT, OutputT> implements Processor {
-
- private final SerializablePipelineOptions pipelineOptions;
- private final DoFn<InputT, OutputT> doFn;
- private final WindowingStrategy<?, ?> windowingStrategy;
- private final DoFnSchemaInformation doFnSchemaInformation;
- private final Map<TupleTag<?>, int[]> outputCollToOrdinals;
- private final TupleTag<OutputT> mainOutputTag;
- private final Coder<InputT> inputCoder;
- private final Map<PCollectionView<?>, Coder<?>> sideInputCoders;
- private final Map<TupleTag<?>, Coder<?>> outputCoders;
- private final Coder<InputT> inputValueCoder;
- private final Map<TupleTag<?>, Coder<?>> outputValueCoders;
- private final Map<Integer, PCollectionView<?>> ordinalToSideInput;
- private final String ownerId; // do not remove, useful for debugging
- private final String stepId;
-
- DoFnRunner<InputT, OutputT> doFnRunner;
- JetOutputManager outputManager;
- private DoFnInvoker<InputT, OutputT> doFnInvoker;
- private SideInputHandler sideInputHandler;
- private JetMetricsContainer metricsContainer;
- private SimpleInbox bufferedItems;
- private Set<Integer> completedSideInputs = new HashSet<>();
- private SideInputReader sideInputReader;
- private Outbox outbox;
-
- AbstractParDoP(
- DoFn<InputT, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- Map<TupleTag<?>, int[]> outputCollToOrdinals,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Coder<InputT> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- Map<Integer, PCollectionView<?>> ordinalToSideInput,
- String ownerId,
- String stepId) {
- this.pipelineOptions = pipelineOptions;
- this.doFn = Utils.serde(doFn);
- this.windowingStrategy = windowingStrategy;
- this.doFnSchemaInformation = doFnSchemaInformation;
- this.outputCollToOrdinals = outputCollToOrdinals;
- this.mainOutputTag = mainOutputTag;
- this.inputCoder = inputCoder;
- this.sideInputCoders =
- sideInputCoders.entrySet().stream()
- .collect(
- Collectors.toMap(
- Map.Entry::getKey,
- e ->
- Utils.deriveIterableValueCoder(
- (WindowedValue.FullWindowedValueCoder) e.getValue())));
- this.outputCoders = outputCoders;
- this.inputValueCoder = inputValueCoder;
- this.outputValueCoders = outputValueCoders;
- this.ordinalToSideInput = ordinalToSideInput;
- this.ownerId = ownerId;
- this.stepId = stepId;
- }
-
- @Override
- public void init(@Nonnull Outbox outbox, @Nonnull Context context) {
- this.outbox = outbox;
- metricsContainer = new JetMetricsContainer(stepId, context);
- MetricsEnvironment.setCurrentContainer(
- metricsContainer); // todo: this is correct only as long as the processor is non-cooperative
-
- doFnInvoker = DoFnInvokers.invokerFor(doFn);
- doFnInvoker.invokeSetup();
-
- if (ordinalToSideInput.isEmpty()) {
- sideInputReader = NullSideInputReader.of(Collections.emptyList());
- } else {
- bufferedItems = new SimpleInbox();
- sideInputHandler =
- new SideInputHandler(ordinalToSideInput.values(), InMemoryStateInternals.forKey(null));
- sideInputReader = sideInputHandler;
- }
-
- outputManager = new JetOutputManager(outbox, outputCoders, outputCollToOrdinals);
-
- doFnRunner =
- getDoFnRunner(
- pipelineOptions.get(),
- doFn,
- sideInputReader,
- outputManager,
- mainOutputTag,
- Lists.newArrayList(outputCollToOrdinals.keySet()),
- inputValueCoder,
- outputValueCoders,
- windowingStrategy,
- doFnSchemaInformation);
- }
-
- protected abstract DoFnRunner<InputT, OutputT> getDoFnRunner(
- PipelineOptions pipelineOptions,
- DoFn<InputT, OutputT> doFn,
- SideInputReader sideInputReader,
- JetOutputManager outputManager,
- TupleTag<OutputT> mainOutputTag,
- List<TupleTag<?>> additionalOutputTags,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation);
-
- @Override
- public boolean isCooperative() {
- return false; // todo: re-examine later, we should be non-cooperative for doFns that do I/O, can
- // be cooperative for others
- }
-
- @Override
- public void close() {
- doFnInvoker.invokeTeardown();
- }
-
- @Override
- public void process(int ordinal, @Nonnull Inbox inbox) {
- if (!outputManager.tryFlush()) {
- // don't process more items until outputManager is empty
- return;
- }
- PCollectionView<?> sideInputView = ordinalToSideInput.get(ordinal);
- if (sideInputView != null) {
- processSideInput(sideInputView, inbox);
- } else {
- if (bufferedItems != null) {
- processBufferedRegularItems(inbox);
- } else {
- processNonBufferedRegularItems(inbox);
- }
- }
- }
-
- private void processSideInput(PCollectionView<?> sideInputView, Inbox inbox) {
- for (byte[] value; (value = (byte[]) inbox.poll()) != null; ) {
- Coder<?> sideInputCoder = sideInputCoders.get(sideInputView);
- WindowedValue<Iterable<?>> windowedValue = Utils.decodeWindowedValue(value, sideInputCoder);
- sideInputHandler.addSideInputValue(sideInputView, windowedValue);
- }
- }
-
- private void processNonBufferedRegularItems(Inbox inbox) {
- startRunnerBundle(doFnRunner);
- for (byte[] value; (value = (byte[]) inbox.poll()) != null; ) {
- WindowedValue<InputT> windowedValue = Utils.decodeWindowedValue(value, inputCoder);
- processElementWithRunner(doFnRunner, windowedValue);
- if (!outputManager.tryFlush()) {
- break;
- }
- }
- finishRunnerBundle(doFnRunner);
- // finishBundle can also add items to outputManager, they will be flushed in tryProcess() or
- // complete()
- }
-
- protected void startRunnerBundle(DoFnRunner<InputT, OutputT> runner) {
- runner.startBundle();
- }
-
- protected void processElementWithRunner(
- DoFnRunner<InputT, OutputT> runner, WindowedValue<InputT> windowedValue) {
- runner.processElement(windowedValue);
- }
-
- protected void finishRunnerBundle(DoFnRunner<InputT, OutputT> runner) {
- runner.finishBundle();
- }
-
- private void processBufferedRegularItems(Inbox inbox) {
- for (byte[] value; (value = (byte[]) inbox.poll()) != null; ) {
- bufferedItems.add(value);
- }
- }
-
- @Override
- public boolean tryProcess() {
- return outputManager.tryFlush();
- }
-
- @Override
- public boolean tryProcessWatermark(@Nonnull Watermark watermark) {
- return outbox.offer(watermark);
- }
-
- @Override
- public boolean completeEdge(int ordinal) {
- if (ordinalToSideInput.get(ordinal) == null) {
- return true; // ignore non-side-input edges
- }
- completedSideInputs.add(ordinal);
- if (completedSideInputs.size() != ordinalToSideInput.size()) {
- // there are more side inputs to complete
- return true;
- }
- processNonBufferedRegularItems(bufferedItems);
- if (bufferedItems.isEmpty()) {
- bufferedItems = null;
- return true;
- }
- return false;
- }
-
- @Override
- public boolean complete() {
- boolean successful = outputManager.tryFlush();
- if (successful) {
- metricsContainer.flush();
- MetricsEnvironment.setCurrentContainer(
- null); // todo: this is correct only as long as the processor is non-cooperative
- }
- return successful;
- }
-
- /**
- * An output manager that stores the output in an ArrayList, one for each output ordinal, and a
- * way to drain to outbox ({@link #tryFlush()}).
- */
- static class JetOutputManager implements DoFnRunners.OutputManager {
-
- private final Outbox outbox;
- private final Map<TupleTag<?>, Coder<?>> outputCoders;
- private final Map<TupleTag<?>, int[]> outputCollToOrdinals;
- private final List<Object>[] outputBuckets;
-
- // the flush position to continue flushing to outbox
- private int currentBucket, currentItem;
-
- @SuppressWarnings("unchecked")
- JetOutputManager(
- Outbox outbox,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Map<TupleTag<?>, int[]> outputCollToOrdinals) {
- this.outbox = outbox;
- this.outputCoders = outputCoders;
- this.outputCollToOrdinals = outputCollToOrdinals;
- assert !outputCollToOrdinals.isEmpty();
- int maxOrdinal =
- outputCollToOrdinals.values().stream().flatMapToInt(IntStream::of).max().orElse(-1);
- outputBuckets = new List[maxOrdinal + 1];
- Arrays.setAll(outputBuckets, i -> new ArrayList<>());
- }
-
- @Override
- public <T> void output(TupleTag<T> tag, WindowedValue<T> outputValue) {
- assert currentBucket == 0 && currentItem == 0 : "adding output while flushing";
- Coder<?> coder = outputCoders.get(tag);
- byte[] output = Utils.encodeWindowedValue(outputValue, coder);
- for (int ordinal : outputCollToOrdinals.get(tag)) {
- outputBuckets[ordinal].add(output);
- }
- }
-
- @CheckReturnValue
- boolean tryFlush() {
- for (; currentBucket < outputBuckets.length; currentBucket++) {
- List<Object> bucket = outputBuckets[currentBucket];
- for (; currentItem < bucket.size(); currentItem++) {
- if (!outbox.offer(currentBucket, bucket.get(currentItem))) {
- return false;
- }
- }
- bucket.clear();
- currentItem = 0;
- }
- currentBucket = 0;
- int sum = 0;
- for (List<Object> outputBucket : outputBuckets) {
- sum += outputBucket.size();
- }
- return sum == 0;
- }
- }
-
- abstract static class AbstractSupplier<InputT, OutputT>
- implements SupplierEx<Processor>, DAGBuilder.WiringListener {
-
- final String ownerId;
- private final String stepId;
-
- private final SerializablePipelineOptions pipelineOptions;
- private final DoFn<InputT, OutputT> doFn;
- private final WindowingStrategy<?, ?> windowingStrategy;
- private final DoFnSchemaInformation doFnSchemaInformation;
- private final TupleTag<OutputT> mainOutputTag;
- private final Map<TupleTag<?>, List<Integer>> outputCollToOrdinals;
- private final Coder<InputT> inputCoder;
- private final Map<PCollectionView<?>, Coder<?>> sideInputCoders;
- private final Map<TupleTag<?>, Coder<?>> outputCoders;
- private final Coder<InputT> inputValueCoder;
- private final Map<TupleTag<?>, Coder<?>> outputValueCoders;
- private final List<PCollectionView<?>> sideInputs;
-
- private final Map<Integer, PCollectionView<?>> ordinalToSideInput = new HashMap<>();
-
- AbstractSupplier(
- String stepId,
- String ownerId,
- DoFn<InputT, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Set<TupleTag<OutputT>> allOutputTags,
- Coder<InputT> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- List<PCollectionView<?>> sideInputs) {
- this.stepId = stepId;
- this.ownerId = ownerId;
- this.pipelineOptions = pipelineOptions;
- this.doFn = doFn;
- this.windowingStrategy = windowingStrategy;
- this.doFnSchemaInformation = doFnSchemaInformation;
- this.outputCollToOrdinals =
- allOutputTags.stream()
- .collect(Collectors.toMap(Function.identity(), t -> new ArrayList<>()));
- this.mainOutputTag = mainOutputTag;
- this.inputCoder = inputCoder;
- this.sideInputCoders = sideInputCoders;
- this.outputCoders = outputCoders;
- this.inputValueCoder = inputValueCoder;
- this.outputValueCoders = outputValueCoders;
- this.sideInputs = sideInputs;
- }
-
- @Override
- public Processor getEx() {
- if (ordinalToSideInput.size() != sideInputs.size()) {
- throw new RuntimeException("Oops");
- }
- return getEx(
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- outputCollToOrdinals.entrySet().stream()
- .collect(
- Collectors.toMap(
- Map.Entry::getKey, e -> e.getValue().stream().mapToInt(i -> i).toArray())),
- pipelineOptions,
- mainOutputTag,
- inputCoder,
- Collections.unmodifiableMap(sideInputCoders),
- Collections.unmodifiableMap(outputCoders),
- inputValueCoder,
- Collections.unmodifiableMap(outputValueCoders),
- Collections.unmodifiableMap(ordinalToSideInput),
- ownerId,
- stepId);
- }
-
- abstract Processor getEx(
- DoFn<InputT, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- Map<TupleTag<?>, int[]> outputCollToOrdinals,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Coder<InputT> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- Map<Integer, PCollectionView<?>> ordinalToSideInput,
- String ownerId,
- String stepId);
-
- @Override
- public void isOutboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
- if (ownerId.equals(vertexId)) {
- List<Integer> ordinals = outputCollToOrdinals.get(new TupleTag<>(pCollId));
- if (ordinals == null) {
- throw new RuntimeException("Oops"); // todo
- }
-
- ordinals.add(edge.getSourceOrdinal());
- }
- }
-
- @Override
- public void isInboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
- if (ownerId.equals(vertexId)) {
- for (PCollectionView<?> pCollectionView : sideInputs) {
- if (edgeId.equals(Utils.getTupleTagId(pCollectionView))) {
- ordinalToSideInput.put(edge.getDestOrdinal(), pCollectionView);
- break;
- }
- }
- }
- }
- }
-
- private static class SimpleInbox implements Inbox {
- private Deque<Object> items = new ArrayDeque<>();
-
- public void add(Object item) {
- items.add(item);
- }
-
- @Override
- public boolean isEmpty() {
- return items.isEmpty();
- }
-
- @Override
- public Object peek() {
- return items.peek();
- }
-
- @Override
- public Object poll() {
- return items.poll();
- }
-
- @Override
- public void remove() {
- items.remove();
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
deleted file mode 100644
index d7fab09..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
+++ /dev/null
@@ -1,129 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.ResettableSingletonTraverser;
-import com.hazelcast.jet.function.SupplierEx;
-import java.util.Collection;
-import javax.annotation.Nonnull;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.Instant;
-
-/**
- * /** * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's Windowing primitive.
- *
- * @param <T> type of element being windowed
- */
-public class AssignWindowP<T> extends AbstractProcessor {
-
- @SuppressWarnings("FieldCanBeLocal")
- private final String ownerId; // do not remove, useful for debugging
-
- private final ResettableSingletonTraverser<byte[]> traverser =
- new ResettableSingletonTraverser<>();
- private final FlatMapper<byte[], byte[]> flatMapper;
- private final WindowAssignContext<T> windowAssignContext;
-
- private AssignWindowP(
- Coder inputCoder,
- Coder outputCoder,
- WindowingStrategy<T, BoundedWindow> windowingStrategy,
- String ownerId) {
- this.ownerId = ownerId;
-
- windowAssignContext = new WindowAssignContext<>(windowingStrategy.getWindowFn());
-
- flatMapper =
- flatMapper(
- item -> {
- Collection<BoundedWindow> windows;
- WindowedValue<T> inputValue = Utils.decodeWindowedValue(item, inputCoder);
- windowAssignContext.setValue(inputValue);
- try {
- windows = windowingStrategy.getWindowFn().assignWindows(windowAssignContext);
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- WindowedValue<T> outputValue =
- WindowedValue.of(
- inputValue.getValue(),
- inputValue.getTimestamp(),
- windows,
- inputValue.getPane());
- traverser.accept(Utils.encodeWindowedValue(outputValue, outputCoder));
- return traverser;
- });
- }
-
- public static <InputT> SupplierEx<Processor> supplier(
- Coder inputCoder,
- Coder outputCoder,
- WindowingStrategy<InputT, BoundedWindow> windowingStrategy,
- String ownerId) {
- return () -> new AssignWindowP<>(inputCoder, outputCoder, windowingStrategy, ownerId);
- }
-
- @Override
- protected boolean tryProcess(int ordinal, @Nonnull Object item) {
- return flatMapper.tryProcess((byte[]) item);
- }
-
- private static class WindowAssignContext<InputT>
- extends WindowFn<InputT, BoundedWindow>.AssignContext {
- private WindowedValue<InputT> value;
-
- WindowAssignContext(WindowFn<InputT, BoundedWindow> fn) {
- fn.super();
- }
-
- public void setValue(WindowedValue<InputT> value) {
- if (Iterables.size(value.getWindows()) != 1) {
- throw new IllegalArgumentException(
- String.format(
- "%s passed to window assignment must be in a single window, but it was in %s: %s",
- WindowedValue.class.getSimpleName(),
- Iterables.size(value.getWindows()),
- value.getWindows()));
- }
- this.value = value;
- }
-
- @Override
- public InputT element() {
- return value.getValue();
- }
-
- @Override
- public Instant timestamp() {
- return value.getTimestamp();
- }
-
- @Override
- public BoundedWindow window() {
- return Iterables.getOnlyElement(value.getWindows());
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
deleted file mode 100644
index 5e52a4f..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
+++ /dev/null
@@ -1,217 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import static com.hazelcast.jet.Traversers.traverseIterable;
-import static com.hazelcast.jet.impl.util.ExceptionUtil.rethrow;
-import static org.apache.beam.runners.jet.Utils.roundRobinSubList;
-
-import com.hazelcast.jet.Traverser;
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.ProcessorMetaSupplier;
-import com.hazelcast.jet.core.ProcessorSupplier;
-import com.hazelcast.nio.Address;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.List;
-import java.util.function.Function;
-import javax.annotation.Nonnull;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.io.BoundedSource;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.util.WindowedValue;
-
-/**
- * Jet {@link com.hazelcast.jet.core.Processor} implementation for reading from a bounded Beam
- * source.
- */
-public class BoundedSourceP<T> extends AbstractProcessor implements Traverser {
-
- private final Traverser<BoundedSource<T>> shardsTraverser;
- private final PipelineOptions options;
- private final Coder outputCoder;
-
- @SuppressWarnings({"FieldCanBeLocal", "unused"})
- private final String ownerId; // do not remove it, very useful for debugging
-
- private BoundedSource.BoundedReader currentReader;
-
- BoundedSourceP(
- List<BoundedSource<T>> shards, PipelineOptions options, Coder outputCoder, String ownerId) {
- this.shardsTraverser = traverseIterable(shards);
- this.options = options;
- this.outputCoder = outputCoder;
- this.ownerId = ownerId;
- }
-
- public static <T> ProcessorMetaSupplier supplier(
- BoundedSource<T> boundedSource,
- SerializablePipelineOptions options,
- Coder outputCoder,
- String ownerId) {
- return new BoundedSourceMetaProcessorSupplier<>(boundedSource, options, outputCoder, ownerId);
- }
-
- @Override
- protected void init(@Nonnull Processor.Context context) throws Exception {
- nextShard();
- }
-
- @Override
- public Object next() {
- if (currentReader == null) {
- return null;
- }
- try {
- Object item = currentReader.getCurrent();
- WindowedValue<Object> res =
- WindowedValue.timestampedValueInGlobalWindow(item, currentReader.getCurrentTimestamp());
- if (!currentReader.advance()) {
- nextShard();
- }
- return outputCoder == null
- ? res
- : Utils.encodeWindowedValue(
- res, outputCoder); // todo: this is not nice, have done this only as a quick fix for
- // BoundedSourcePTest
- } catch (IOException e) {
- throw rethrow(e);
- }
- }
-
- /**
- * Called when currentReader is null or drained. At the end it will contain a started reader of
- * the next shard or null.
- */
- private void nextShard() throws IOException {
- for (; ; ) {
- if (currentReader != null) {
- currentReader.close();
- currentReader = null;
- }
- BoundedSource<T> shard = shardsTraverser.next();
- if (shard == null) {
- break; // all shards done
- }
- currentReader = shard.createReader(options);
- if (currentReader.start()) {
- break;
- }
- }
- }
-
- @Override
- public boolean complete() {
- return emitFromTraverser(this);
- }
-
- @Override
- public boolean isCooperative() {
- return false;
- }
-
- @Override
- public void close() throws Exception {
- if (currentReader != null) {
- currentReader.close();
- }
- }
-
- private static class BoundedSourceMetaProcessorSupplier<T> implements ProcessorMetaSupplier {
-
- private final BoundedSource<T> boundedSource;
- private final SerializablePipelineOptions options;
- private final Coder outputCoder;
- private final String ownerId;
-
- private transient List<? extends BoundedSource<T>> shards;
-
- private BoundedSourceMetaProcessorSupplier(
- BoundedSource<T> boundedSource,
- SerializablePipelineOptions options,
- Coder outputCoder,
- String ownerId) {
- this.boundedSource = boundedSource;
- this.options = options;
- this.outputCoder = outputCoder;
- this.ownerId = ownerId;
- }
-
- @Override
- public void init(@Nonnull ProcessorMetaSupplier.Context context) throws Exception {
- long desiredSizeBytes =
- Math.max(
- 1, boundedSource.getEstimatedSizeBytes(options.get()) / context.totalParallelism());
- shards = boundedSource.split(desiredSizeBytes, options.get());
- }
-
- @SuppressWarnings("unchecked")
- @Nonnull
- @Override
- public Function<? super Address, ? extends ProcessorSupplier> get(
- @Nonnull List<Address> addresses) {
- return address ->
- new BoundedSourceProcessorSupplier(
- roundRobinSubList(shards, addresses.indexOf(address), addresses.size()),
- options,
- outputCoder,
- ownerId);
- }
- }
-
- private static class BoundedSourceProcessorSupplier<T> implements ProcessorSupplier {
- private final List<BoundedSource<T>> shards;
- private final SerializablePipelineOptions options;
- private final Coder outputCoder;
- private final String ownerId;
- private transient ProcessorSupplier.Context context;
-
- private BoundedSourceProcessorSupplier(
- List<BoundedSource<T>> shards,
- SerializablePipelineOptions options,
- Coder outputCoder,
- String ownerId) {
- this.shards = shards;
- this.options = options;
- this.outputCoder = outputCoder;
- this.ownerId = ownerId;
- }
-
- @Override
- public void init(@Nonnull Context context) {
- this.context = context;
- }
-
- @Nonnull
- @Override
- public Collection<? extends Processor> get(int count) {
- int indexBase = context.memberIndex() * context.localParallelism();
- List<Processor> res = new ArrayList<>(count);
- for (int i = 0; i < count; i++, indexBase++) {
- res.add(
- new BoundedSourceP<>(
- roundRobinSubList(shards, i, count), options.get(), outputCoder, ownerId));
- }
- return res;
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
deleted file mode 100644
index 220a608..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.Edge;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.function.SupplierEx;
-import java.util.HashMap;
-import java.util.Map;
-import javax.annotation.Nonnull;
-import org.apache.beam.runners.jet.DAGBuilder;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.util.WindowedValue;
-
-/** Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's Flatten primitive. */
-public class FlattenP extends AbstractProcessor {
-
- private final Map<Integer, Coder> inputOrdinalCoders;
- private final Coder outputCoder;
-
- @SuppressWarnings("FieldCanBeLocal") // do not remove, useful for debugging
- private final String ownerId;
-
- private FlattenP(Map<Integer, Coder> inputOrdinalCoders, Coder outputCoder, String ownerId) {
- this.inputOrdinalCoders = inputOrdinalCoders;
- this.outputCoder = outputCoder;
- this.ownerId = ownerId;
- }
-
- @Override
- protected boolean tryProcess(int ordinal, @Nonnull Object item) {
- Coder inputCoder = inputOrdinalCoders.get(ordinal);
- WindowedValue<Object> windowedValue = Utils.decodeWindowedValue((byte[]) item, inputCoder);
- return tryEmit(Utils.encodeWindowedValue(windowedValue, outputCoder));
- }
-
- /** Jet {@link Processor} supplier that will provide instances of {@link FlattenP}. */
- public static final class Supplier implements SupplierEx<Processor>, DAGBuilder.WiringListener {
-
- private final Map<String, Coder> inputCollectionCoders;
- private final Coder outputCoder;
- private final String ownerId;
- private final Map<Integer, Coder> inputOrdinalCoders;
-
- public Supplier(Map<String, Coder> inputCoders, Coder outputCoder, String ownerId) {
- this.inputCollectionCoders = inputCoders;
- this.outputCoder = outputCoder;
- this.ownerId = ownerId;
- this.inputOrdinalCoders = new HashMap<>();
- }
-
- @Override
- public Processor getEx() {
- return new FlattenP(inputOrdinalCoders, outputCoder, ownerId);
- }
-
- @Override
- public void isOutboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
- // do nothing
- }
-
- @Override
- public void isInboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
- if (ownerId.equals(vertexId)) {
- Coder coder = inputCollectionCoders.get(edgeId);
- inputOrdinalCoders.put(edge.getDestOrdinal(), coder);
- }
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
deleted file mode 100644
index ffab6c9..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.function.SupplierEx;
-import org.apache.beam.sdk.util.WindowedValue;
-
-/**
- * /** * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's Impulse primitive.
- */
-public class ImpulseP extends AbstractProcessor {
-
- private final String ownerId; // do not remove it, very useful for debugging
-
- private ImpulseP(String ownerId) {
- this.ownerId = ownerId;
- }
-
- public static SupplierEx<Processor> supplier(String ownerId) {
- return () -> new ImpulseP(ownerId);
- }
-
- @Override
- public boolean complete() {
- return tryEmit(
- WindowedValue.valueInGlobalWindow(
- new byte[0])); // todo: should EACH processor emit this byte[] or just a SINGLE one?
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
deleted file mode 100644
index aa54f8e..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
+++ /dev/null
@@ -1,191 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.Processor;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import org.apache.beam.runners.core.DoFnRunner;
-import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.core.SideInputReader;
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.StepContext;
-import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-
-/**
- * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's ParDo primitive (when no
- * user-state is being used).
- */
-public class ParDoP<InputT, OutputT>
- extends AbstractParDoP<InputT, OutputT> { // todo: unify with StatefulParDoP?
-
- private ParDoP(
- DoFn<InputT, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- Map<TupleTag<?>, int[]> outputCollToOrdinals,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Coder<InputT> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- Map<Integer, PCollectionView<?>> ordinalToSideInput,
- String ownerId,
- String stepId) {
- super(
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- outputCollToOrdinals,
- pipelineOptions,
- mainOutputTag,
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- ordinalToSideInput,
- ownerId,
- stepId);
- }
-
- @Override
- protected DoFnRunner<InputT, OutputT> getDoFnRunner(
- PipelineOptions pipelineOptions,
- DoFn<InputT, OutputT> doFn,
- SideInputReader sideInputReader,
- JetOutputManager outputManager,
- TupleTag<OutputT> mainOutputTag,
- List<TupleTag<?>> additionalOutputTags,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation) {
- return DoFnRunners.simpleRunner(
- pipelineOptions,
- doFn,
- sideInputReader,
- outputManager,
- mainOutputTag,
- additionalOutputTags,
- new NotImplementedStepContext(),
- inputValueCoder,
- outputValueCoders,
- windowingStrategy,
- doFnSchemaInformation);
- }
-
- /**
- * Jet {@link Processor} supplier that will provide instances of {@link ParDoP}.
- *
- * @param <OutputT> the type of main output elements of the DoFn being used
- */
- public static class Supplier<InputT, OutputT> extends AbstractSupplier<InputT, OutputT> {
-
- public Supplier(
- String stepId,
- String ownerId,
- DoFn<InputT, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Set<TupleTag<OutputT>> allOutputTags,
- Coder<InputT> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- List<PCollectionView<?>> sideInputs) {
- super(
- stepId,
- ownerId,
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- pipelineOptions,
- mainOutputTag,
- allOutputTags,
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- sideInputs);
- }
-
- @Override
- Processor getEx(
- DoFn<InputT, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- Map<TupleTag<?>, int[]> outputCollToOrdinals,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Coder<InputT> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<InputT> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- Map<Integer, PCollectionView<?>> ordinalToSideInput,
- String ownerId,
- String stepId) {
- return new ParDoP<>(
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- outputCollToOrdinals,
- pipelineOptions,
- mainOutputTag,
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- ordinalToSideInput,
- ownerId,
- stepId);
- }
- }
-
- private static class NotImplementedStepContext implements StepContext {
-
- // not needed when not handling state & timers
-
- @Override
- public StateInternals stateInternals() {
- throw new UnsupportedOperationException("stateInternals is not supported");
- }
-
- @Override
- public TimerInternals timerInternals() {
- throw new UnsupportedOperationException("timerInternals is not supported");
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
deleted file mode 100644
index 0b9c5aa..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
+++ /dev/null
@@ -1,302 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.Watermark;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import javax.annotation.Nonnull;
-import org.apache.beam.runners.core.DoFnRunner;
-import org.apache.beam.runners.core.DoFnRunners;
-import org.apache.beam.runners.core.InMemoryStateInternals;
-import org.apache.beam.runners.core.InMemoryTimerInternals;
-import org.apache.beam.runners.core.SideInputReader;
-import org.apache.beam.runners.core.StateInternals;
-import org.apache.beam.runners.core.StateNamespace;
-import org.apache.beam.runners.core.StateNamespaces;
-import org.apache.beam.runners.core.StepContext;
-import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.joda.time.Instant;
-
-/**
- * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's stateful ParDo primitive.
- */
-public class StatefulParDoP<OutputT>
- extends AbstractParDoP<KV<?, ?>, OutputT> { // todo: unify with ParDoP?
-
- private KeyedStepContext keyedStepContext;
- private InMemoryTimerInternals timerInternals;
-
- private StatefulParDoP(
- DoFn<KV<?, ?>, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- Map<TupleTag<?>, int[]> outputCollToOrdinals,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Coder<KV<?, ?>> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<KV<?, ?>> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- Map<Integer, PCollectionView<?>> ordinalToSideInput,
- String ownerId,
- String stepId) {
- super(
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- outputCollToOrdinals,
- pipelineOptions,
- mainOutputTag,
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- ordinalToSideInput,
- ownerId,
- stepId);
- }
-
- private static void fireTimer(
- TimerInternals.TimerData timer, DoFnRunner<KV<?, ?>, ?> doFnRunner) {
- StateNamespace namespace = timer.getNamespace();
- BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
- doFnRunner.onTimer(timer.getTimerId(), window, timer.getTimestamp(), timer.getDomain());
- }
-
- @Override
- protected DoFnRunner<KV<?, ?>, OutputT> getDoFnRunner(
- PipelineOptions pipelineOptions,
- DoFn<KV<?, ?>, OutputT> doFn,
- SideInputReader sideInputReader,
- JetOutputManager outputManager,
- TupleTag<OutputT> mainOutputTag,
- List<TupleTag<?>> additionalOutputTags,
- Coder<KV<?, ?>> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation) {
- timerInternals = new InMemoryTimerInternals();
- keyedStepContext = new KeyedStepContext(timerInternals);
- return DoFnRunners.simpleRunner(
- pipelineOptions,
- doFn,
- sideInputReader,
- outputManager,
- mainOutputTag,
- additionalOutputTags,
- keyedStepContext,
- inputValueCoder,
- outputValueCoders,
- windowingStrategy,
- doFnSchemaInformation);
- }
-
- @Override
- protected void startRunnerBundle(DoFnRunner<KV<?, ?>, OutputT> runner) {
- try {
- Instant now = Instant.now();
- timerInternals.advanceProcessingTime(now);
- timerInternals.advanceSynchronizedProcessingTime(now);
- } catch (Exception e) {
- throw new RuntimeException("Failed advancing time!");
- }
-
- super.startRunnerBundle(runner);
- }
-
- @Override
- protected void processElementWithRunner(
- DoFnRunner<KV<?, ?>, OutputT> runner, WindowedValue<KV<?, ?>> windowedValue) {
- KV<?, ?> kv = windowedValue.getValue();
- Object key = kv.getKey();
- keyedStepContext.setKey(key);
-
- super.processElementWithRunner(runner, windowedValue);
- }
-
- @Override
- public boolean tryProcessWatermark(@Nonnull Watermark watermark) {
- return flushTimers(watermark.timestamp()) && super.tryProcessWatermark(watermark);
- }
-
- @Override
- public boolean complete() {
- return flushTimers(BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) && super.complete();
- }
-
- private boolean flushTimers(long watermark) {
- if (timerInternals.currentInputWatermarkTime().isBefore(watermark)) {
- try {
- Instant watermarkInstant = new Instant(watermark);
- timerInternals.advanceInputWatermark(watermarkInstant);
- if (watermarkInstant.equals(BoundedWindow.TIMESTAMP_MAX_VALUE)) {
- timerInternals.advanceProcessingTime(watermarkInstant);
- timerInternals.advanceSynchronizedProcessingTime(watermarkInstant);
- }
- fireEligibleTimers(timerInternals);
- } catch (Exception e) {
- throw new RuntimeException("Failed advancing processing time", e);
- }
- }
- return outputManager.tryFlush();
- }
-
- private void fireEligibleTimers(InMemoryTimerInternals timerInternals) {
- while (true) {
- TimerInternals.TimerData timer;
- boolean hasFired = false;
-
- while ((timer = timerInternals.removeNextEventTimer()) != null) {
- hasFired = true;
- fireTimer(timer, doFnRunner);
- }
-
- while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
- hasFired = true;
- fireTimer(timer, doFnRunner);
- }
-
- while ((timer = timerInternals.removeNextSynchronizedProcessingTimer()) != null) {
- hasFired = true;
- fireTimer(timer, doFnRunner);
- }
-
- if (!hasFired) {
- break;
- }
- }
- }
-
- /**
- * Jet {@link Processor} supplier that will provide instances of {@link StatefulParDoP}.
- *
- * @param <OutputT> the type of main output elements of the DoFn being used
- */
- public static class Supplier<OutputT> extends AbstractSupplier<KV<?, ?>, OutputT> {
-
- public Supplier(
- String stepId,
- String ownerId,
- DoFn<KV<?, ?>, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Set<TupleTag<OutputT>> allOutputTags,
- Coder<KV<?, ?>> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<KV<?, ?>> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- List<PCollectionView<?>> sideInputs) {
- super(
- stepId,
- ownerId,
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- pipelineOptions,
- mainOutputTag,
- allOutputTags,
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- sideInputs);
- }
-
- @Override
- Processor getEx(
- DoFn<KV<?, ?>, OutputT> doFn,
- WindowingStrategy<?, ?> windowingStrategy,
- DoFnSchemaInformation doFnSchemaInformation,
- Map<TupleTag<?>, int[]> outputCollToOrdinals,
- SerializablePipelineOptions pipelineOptions,
- TupleTag<OutputT> mainOutputTag,
- Coder<KV<?, ?>> inputCoder,
- Map<PCollectionView<?>, Coder<?>> sideInputCoders,
- Map<TupleTag<?>, Coder<?>> outputCoders,
- Coder<KV<?, ?>> inputValueCoder,
- Map<TupleTag<?>, Coder<?>> outputValueCoders,
- Map<Integer, PCollectionView<?>> ordinalToSideInput,
- String ownerId,
- String stepId) {
- return new StatefulParDoP<>(
- doFn,
- windowingStrategy,
- doFnSchemaInformation,
- outputCollToOrdinals,
- pipelineOptions,
- mainOutputTag,
- inputCoder,
- sideInputCoders,
- outputCoders,
- inputValueCoder,
- outputValueCoders,
- ordinalToSideInput,
- ownerId,
- stepId);
- }
- }
-
- private static class KeyedStepContext implements StepContext {
-
- private final Map<Object, InMemoryStateInternals> stateInternalsOfKeys;
- private final InMemoryTimerInternals timerInternals;
-
- private InMemoryStateInternals currentStateInternals;
-
- KeyedStepContext(InMemoryTimerInternals timerInternals) {
- this.stateInternalsOfKeys = new HashMap<>();
- this.timerInternals = timerInternals;
- }
-
- void setKey(Object key) {
- currentStateInternals =
- stateInternalsOfKeys.computeIfAbsent(key, InMemoryStateInternals::forKey);
- }
-
- @Override
- public StateInternals stateInternals() {
- return currentStateInternals;
- }
-
- @Override
- public TimerInternals timerInternals() {
- return timerInternals;
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/TestStreamP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/TestStreamP.java
deleted file mode 100644
index cf491d5..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/TestStreamP.java
+++ /dev/null
@@ -1,102 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.Traverser;
-import com.hazelcast.jet.Traversers;
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.ProcessorMetaSupplier;
-import com.hazelcast.jet.core.ProcessorSupplier;
-import com.hazelcast.jet.core.Watermark;
-import com.hazelcast.jet.impl.util.ExceptionUtil;
-import java.util.List;
-import java.util.stream.Stream;
-import java.util.stream.StreamSupport;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.joda.time.Instant;
-
-/**
- * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's {@link TestStream}
- * transform.
- */
-public class TestStreamP extends AbstractProcessor {
-
- private final Traverser traverser;
-
- @SuppressWarnings("unchecked")
- private TestStreamP(byte[] payload, TestStream.TestStreamCoder payloadCoder, Coder outputCoder) {
- List events = decodePayload(payload, payloadCoder).getEvents();
- traverser =
- Traversers.traverseStream(
- events.stream()
- .flatMap(
- event -> {
- if (event instanceof TestStream.WatermarkEvent) {
- Instant watermark = ((TestStream.WatermarkEvent) event).getWatermark();
- if (BoundedWindow.TIMESTAMP_MAX_VALUE.equals(watermark)) {
- // this is an element added by advanceWatermarkToInfinity(), we ignore it,
- // it's always at the end
- return null;
- }
- return Stream.of(new Watermark(watermark.getMillis()));
- } else if (event instanceof TestStream.ElementEvent) {
- return StreamSupport.stream(
- ((TestStream.ElementEvent<?>) event).getElements().spliterator(),
- false)
- .map(
- tv ->
- WindowedValue.timestampedValueInGlobalWindow(
- tv.getValue(), tv.getTimestamp()))
- .map(wV -> Utils.encodeWindowedValue(wV, outputCoder));
- } else {
- throw new UnsupportedOperationException(
- "Event type not supported in TestStream: "
- + event.getClass()
- + ", event: "
- + event);
- }
- }));
- }
-
- public static <T> ProcessorMetaSupplier supplier(
- byte[] payload, TestStream.TestStreamCoder payloadCoder, Coder outputCoder) {
- return ProcessorMetaSupplier.forceTotalParallelismOne(
- ProcessorSupplier.of(() -> new TestStreamP(payload, payloadCoder, outputCoder)));
- }
-
- private static TestStream decodePayload(byte[] payload, TestStream.TestStreamCoder coder) {
- try {
- return (TestStream) CoderUtils.decodeFromByteArray(coder, payload);
- } catch (CoderException e) {
- throw ExceptionUtil.rethrow(e);
- }
- }
-
- @Override
- public boolean complete() {
- // todo: TestStream says it should cease emitting, but not stop after the items.
- // But I don't know how they end the job otherwise...
- return emitFromTraverser(traverser);
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
deleted file mode 100644
index da95226..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
+++ /dev/null
@@ -1,130 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.Traverser;
-import com.hazelcast.jet.Traversers;
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.function.SupplierEx;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import javax.annotation.Nonnull;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.joda.time.Instant;
-
-/**
- * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's side input producing
- * primitives. Collects all input {@link WindowedValue}s, groups them by windows and keys and when
- * input is complete emits them.
- */
-public class ViewP extends AbstractProcessor {
-
- private final TimestampCombiner timestampCombiner;
- private final Coder inputCoder;
- private final Coder outputCoder;
-
- @SuppressWarnings({"FieldCanBeLocal", "unused"})
- private final String ownerId; // do not remove, useful for debugging
-
- private Map<BoundedWindow, TimestampAndValues> values = new HashMap<>();
- private Traverser<byte[]> resultTraverser;
-
- private ViewP(
- Coder inputCoder, Coder outputCoder, WindowingStrategy windowingStrategy, String ownerId) {
- this.timestampCombiner = windowingStrategy.getTimestampCombiner();
- this.inputCoder = inputCoder;
- this.outputCoder =
- Utils.deriveIterableValueCoder((WindowedValue.FullWindowedValueCoder) outputCoder);
- this.ownerId = ownerId;
- }
-
- @Override
- protected boolean tryProcess(int ordinal, @Nonnull Object item) {
- WindowedValue<?> windowedValue = Utils.decodeWindowedValue((byte[]) item, inputCoder);
- for (BoundedWindow window : windowedValue.getWindows()) {
- values.merge(
- window,
- new TimestampAndValues(
- windowedValue.getPane(), windowedValue.getTimestamp(), windowedValue.getValue()),
- (o, n) -> o.merge(timestampCombiner, n));
- }
-
- return true;
- }
-
- @Override
- public boolean complete() {
- if (resultTraverser == null) {
- resultTraverser =
- Traversers.traverseStream(
- values.entrySet().stream()
- .map(
- e -> {
- WindowedValue<?> outputValue =
- WindowedValue.of(
- e.getValue().values,
- e.getValue().timestamp,
- Collections.singleton(e.getKey()),
- e.getValue().pane);
- return Utils.encodeWindowedValue(outputValue, outputCoder);
- }));
- }
- return emitFromTraverser(resultTraverser);
- }
-
- public static SupplierEx<Processor> supplier(
- Coder inputCoder,
- Coder outputCoder,
- WindowingStrategy<?, ?> windowingStrategy,
- String ownerId) {
- return () -> new ViewP(inputCoder, outputCoder, windowingStrategy, ownerId);
- }
-
- private static class TimestampAndValues {
- private final List<Object> values = new ArrayList<>();
- private Instant timestamp;
- private PaneInfo pane;
-
- TimestampAndValues(PaneInfo pane, Instant timestamp, Object value) {
- this.pane = pane;
- this.timestamp = timestamp;
- this.values.add(value);
- }
-
- public Iterable<Object> getValues() {
- return values;
- }
-
- TimestampAndValues merge(TimestampCombiner timestampCombiner, TimestampAndValues other) {
- pane = other.pane;
- timestamp = timestampCombiner.combine(timestamp, other.timestamp);
- values.addAll(other.values);
- return this;
- }
- }
-}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java b/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
deleted file mode 100644
index 29c4977..0000000
--- a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
+++ /dev/null
@@ -1,296 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet.processors;
-
-import com.hazelcast.jet.core.AbstractProcessor;
-import com.hazelcast.jet.core.AppendableTraverser;
-import com.hazelcast.jet.core.Processor;
-import com.hazelcast.jet.core.Watermark;
-import com.hazelcast.jet.function.SupplierEx;
-import com.hazelcast.jet.impl.util.ExceptionUtil;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import javax.annotation.Nonnull;
-import javax.annotation.Nullable;
-import org.apache.beam.runners.core.InMemoryStateInternals;
-import org.apache.beam.runners.core.InMemoryTimerInternals;
-import org.apache.beam.runners.core.LateDataUtils;
-import org.apache.beam.runners.core.NullSideInputReader;
-import org.apache.beam.runners.core.OutputWindowedValue;
-import org.apache.beam.runners.core.ReduceFnRunner;
-import org.apache.beam.runners.core.SystemReduceFn;
-import org.apache.beam.runners.core.TimerInternals;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.core.construction.TriggerTranslation;
-import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
-import org.apache.beam.runners.core.triggers.TriggerStateMachines;
-import org.apache.beam.runners.jet.Utils;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.state.State;
-import org.apache.beam.sdk.state.WatermarkHoldState;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowTracing;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.joda.time.Instant;
-
-/**
- * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's GroupByKeyOnly +
- * GroupAlsoByWindow primitives.
- *
- * @param <K> key type of {@link KV} values from the output of this primitive
- * @param <V> type of elements being windowed
- */
-public class WindowGroupP<K, V> extends AbstractProcessor {
-
- private static final byte[] EMPTY_BYTES = new byte[0];
- private final SerializablePipelineOptions pipelineOptions;
- private final Coder<V> inputValueValueCoder;
- private final Coder outputCoder;
- private final WindowingStrategy<V, BoundedWindow> windowingStrategy;
- private final Map<K, KeyManager> keyManagers = new HashMap<>();
- private final AppendableTraverser<byte[]> appendableTraverser =
- new AppendableTraverser<>(128); // todo: right capacity?
- private final FlatMapper<byte[], byte[]> flatMapper;
-
- @SuppressWarnings({"FieldCanBeLocal", "unused"})
- private final String ownerId; // do not remove, useful for debugging
-
- private Instant latestWatermark = BoundedWindow.TIMESTAMP_MIN_VALUE;
-
- private WindowGroupP(
- SerializablePipelineOptions pipelineOptions,
- Coder inputCoder,
- Coder inputValueCoder,
- Coder outputCoder,
- WindowingStrategy<V, BoundedWindow> windowingStrategy,
- String ownerId) {
- this.pipelineOptions = pipelineOptions;
- this.inputValueValueCoder = ((KvCoder<K, V>) inputValueCoder).getValueCoder();
- this.outputCoder = outputCoder;
- this.windowingStrategy = windowingStrategy;
- this.ownerId = ownerId;
-
- this.flatMapper =
- flatMapper(
- item -> {
- if (item.length > 0) {
- WindowedValue<KV<K, V>> windowedValue = Utils.decodeWindowedValue(item, inputCoder);
- KV<K, V> kv = windowedValue.getValue();
- K key = kv.getKey();
- V value = kv.getValue();
- WindowedValue<V> updatedWindowedValue =
- WindowedValue.of(
- value,
- windowedValue.getTimestamp(),
- windowedValue.getWindows(),
- windowedValue.getPane());
- KeyManager keyManager =
- keyManagers.computeIfAbsent(key, k -> new KeyManager(k, latestWatermark));
- keyManager.processElement(updatedWindowedValue);
- }
- return appendableTraverser;
- });
- }
-
- @SuppressWarnings("unchecked")
- public static SupplierEx<Processor> supplier(
- SerializablePipelineOptions pipelineOptions,
- Coder inputValueCoder,
- Coder inputCoder,
- Coder outputCoder,
- WindowingStrategy windowingStrategy,
- String ownerId) {
- return () ->
- new WindowGroupP<>(
- pipelineOptions, inputCoder, inputValueCoder, outputCoder, windowingStrategy, ownerId);
- }
-
- @Override
- protected boolean tryProcess(int ordinal, @Nonnull Object item) {
- return flatMapper.tryProcess((byte[]) item);
- }
-
- @Override
- public boolean tryProcessWatermark(@Nonnull Watermark watermark) {
- advanceWatermark(watermark.timestamp());
- return flatMapper.tryProcess(EMPTY_BYTES);
- }
-
- @Override
- public boolean complete() {
- advanceWatermark(BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis());
- return flatMapper.tryProcess(EMPTY_BYTES);
- }
-
- private void advanceWatermark(long millis) {
- this.latestWatermark = new Instant(millis);
- this.keyManagers.values().forEach(m -> m.advanceWatermark(latestWatermark));
- }
-
- private static class InMemoryStateInternalsImpl extends InMemoryStateInternals {
-
- InMemoryStateInternalsImpl(@Nullable Object key) {
- super(key);
- }
-
- Instant earliestWatermarkHold() {
- Instant minimum = null;
- for (State storage : inMemoryState.values()) {
- if (storage instanceof WatermarkHoldState) {
- Instant hold = ((WatermarkHoldState) storage).read();
- if (minimum == null || (hold != null && hold.isBefore(minimum))) {
- minimum = hold;
- }
- }
- }
- return minimum;
- }
- }
-
- /** Helper class that holds all the per key based data structures needed. */
- private class KeyManager {
-
- private final InMemoryTimerInternals timerInternals;
- private final InMemoryStateInternalsImpl stateInternals;
- private final ReduceFnRunner<K, V, Iterable<V>, BoundedWindow> reduceFnRunner;
-
- KeyManager(K key, Instant currentWaterMark) {
- this.timerInternals = new InMemoryTimerInternals();
- this.stateInternals = new InMemoryStateInternalsImpl(key);
- this.reduceFnRunner =
- new ReduceFnRunner<>(
- key,
- windowingStrategy,
- ExecutableTriggerStateMachine.create(
- TriggerStateMachines.stateMachineForTrigger(
- TriggerTranslation.toProto(windowingStrategy.getTrigger()))),
- stateInternals,
- timerInternals,
- new OutputWindowedValue<KV<K, Iterable<V>>>() {
- @Override
- public void outputWindowedValue(
- KV<K, Iterable<V>> output,
- Instant timestamp,
- Collection<? extends BoundedWindow> windows,
- PaneInfo pane) {
- WindowedValue<KV<K, Iterable<V>>> windowedValue =
- WindowedValue.of(output, timestamp, windows, pane);
- byte[] encodedValue = Utils.encodeWindowedValue(windowedValue, outputCoder);
- //noinspection ResultOfMethodCallIgnored
- appendableTraverser.append(encodedValue);
- }
-
- @Override
- public <AdditionalOutputT> void outputWindowedValue(
- TupleTag<AdditionalOutputT> tag,
- AdditionalOutputT output,
- Instant timestamp,
- Collection<? extends BoundedWindow> windows,
- PaneInfo pane) {
- throw new UnsupportedOperationException("Grouping should not use side outputs");
- }
- },
- NullSideInputReader.empty(),
- SystemReduceFn.buffering(inputValueValueCoder),
- pipelineOptions.get());
- advanceWatermark(currentWaterMark);
- }
-
- void advanceWatermark(Instant watermark) {
- try {
- timerInternals.advanceProcessingTime(Instant.now());
- advanceInputWatermark(watermark);
- Instant hold = stateInternals.earliestWatermarkHold();
- if (hold == null) {
- WindowTracing.trace(
- "TestInMemoryTimerInternals.advanceInputWatermark: no holds, "
- + "so output watermark = input watermark");
- hold = timerInternals.currentInputWatermarkTime();
- }
- advanceOutputWatermark(hold);
- reduceFnRunner.persist();
- } catch (Exception e) {
- throw ExceptionUtil.rethrow(e);
- }
- }
-
- private void advanceInputWatermark(Instant watermark) throws Exception {
- timerInternals.advanceInputWatermark(watermark);
- while (true) {
- TimerInternals.TimerData timer;
- List<TimerInternals.TimerData> timers = new ArrayList<>();
- while ((timer = timerInternals.removeNextEventTimer()) != null) {
- timers.add(timer);
- }
- if (timers.isEmpty()) {
- break;
- }
- reduceFnRunner.onTimers(timers);
- }
- }
-
- private void advanceOutputWatermark(Instant watermark) {
- Objects.requireNonNull(watermark);
- timerInternals.advanceOutputWatermark(watermark);
- }
-
- void processElement(WindowedValue<V> windowedValue) {
- Collection<? extends BoundedWindow> windows = dropLateWindows(windowedValue.getWindows());
- if (!windows.isEmpty()) {
- try {
- reduceFnRunner.processElements(
- Collections.singletonList(
- windowedValue)); // todo: try to process more than one element at a time...
- reduceFnRunner.persist();
- } catch (Exception e) {
- throw ExceptionUtil.rethrow(e);
- }
- }
- }
-
- private Collection<? extends BoundedWindow> dropLateWindows(
- Collection<? extends BoundedWindow> windows) {
- List<BoundedWindow> filteredWindows =
- new ArrayList<>(
- windows
- .size()); // todo: reduce garbage, most of the time it will be one window only and
- // there won't be expired windows
- for (BoundedWindow window : windows) {
- if (!isExpiredWindow(window)) {
- filteredWindows.add(window);
- }
- }
- return filteredWindows;
- }
-
- private boolean isExpiredWindow(BoundedWindow window) {
- Instant inputWM = timerInternals.currentInputWatermarkTime();
- return LateDataUtils.garbageCollectionTime(window, windowingStrategy).isBefore(inputWM);
- }
- }
-}
diff --git a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java b/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
deleted file mode 100644
index e7b4a98..0000000
--- a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-
-/**
- * Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
- * TestJetRunner}.
- *
- * <p>{@link AutoService} will register Apex's implementations of the {@link PipelineRunner} and
- * {@link PipelineOptions} as available pipeline runner services.
- */
-public final class JetTestRunnerRegistrar {
- private JetTestRunnerRegistrar() {}
-
- /** Registers the {@link JetRunner}. */
- @AutoService(PipelineRunnerRegistrar.class)
- public static class Runner implements PipelineRunnerRegistrar {
- @Override
- public Iterable<Class<? extends PipelineRunner<?>>> getPipelineRunners() {
- return ImmutableList.of(TestJetRunner.class);
- }
- }
-
- /** Registers the {@link JetPipelineOptions}. */
- @AutoService(PipelineOptionsRegistrar.class)
- public static class Options implements PipelineOptionsRegistrar {
- @Override
- public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
- return ImmutableList.of(JetPipelineOptions.class);
- }
- }
-}
diff --git a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java b/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
deleted file mode 100644
index adcb4b0..0000000
--- a/runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.jet;
-
-import com.hazelcast.config.EventJournalConfig;
-import com.hazelcast.jet.JetInstance;
-import com.hazelcast.jet.JetTestInstanceFactory;
-import com.hazelcast.jet.config.JetConfig;
-import java.util.Arrays;
-import java.util.Collection;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.PipelineRunner;
-import org.apache.beam.sdk.options.PipelineOptions;
-
-/** Slightly altered version of the Jet based runner, used in unit-tests. */
-public class TestJetRunner extends PipelineRunner<PipelineResult> {
-
- private final JetTestInstanceFactory factory = new JetTestInstanceFactory();
-
- public static TestJetRunner fromOptions(PipelineOptions options) {
- return new TestJetRunner(options);
- }
-
- private final JetRunner delegate;
-
- private TestJetRunner(PipelineOptions options) {
- JetPipelineOptions jetPipelineOptions = options.as(JetPipelineOptions.class);
- jetPipelineOptions.setJetStartOwnCluster(false);
-
- this.delegate = JetRunner.fromOptions(options, factory::newClient);
- }
-
- @Override
- public PipelineResult run(Pipeline pipeline) {
- Collection<JetInstance> instances = initInstances(factory);
- System.out.println("Created " + instances.size() + " instances.");
- try {
- PipelineResult result = delegate.run(pipeline);
- if (result instanceof FailedRunningPipelineResults) {
- RuntimeException failureCause = ((FailedRunningPipelineResults) result).getCause();
- throw failureCause;
- }
- return result;
- } finally {
- System.out.println("Shutting down " + instances.size() + " instances...");
- factory.shutdownAll();
- }
- }
-
- private Collection<JetInstance> initInstances(JetTestInstanceFactory factory) {
- JetConfig config = new JetConfig();
- config.getHazelcastConfig().addEventJournalConfig(new EventJournalConfig().setMapName("map"));
-
- return Arrays.asList(factory.newMember(config), factory.newMember(config));
- }
-}
diff --git a/runners/jet/build.gradle b/runners/jet/build.gradle
new file mode 100644
index 0000000..b97b016
--- /dev/null
+++ b/runners/jet/build.gradle
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.jet')
+
+description = "Apache Beam :: Runners :: Hazelcast Jet"
+
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":runners:core-java")
+evaluationDependsOn(":runners:core-construction-java")
+
+project.ext {
+ jet_version = '3.0'
+ hazelcast_version = '3.12'
+}
+
+configurations {
+ needsRunner
+ validatesRunner
+}
+
+dependencies {
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-java")
+ compile "com.hazelcast.jet:hazelcast-jet:$jet_version"
+
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ testCompile project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile "com.hazelcast.jet:hazelcast-jet-core:$jet_version:tests"
+ testCompile "com.hazelcast:hazelcast:$hazelcast_version:tests"
+ testCompile "com.hazelcast:hazelcast-client:$hazelcast_version:tests"
+
+ needsRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+ needsRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ needsRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ needsRunner project(path: project.path, configuration: "testRuntime")
+
+ validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesRunner project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ validatesRunner project(path: project.path, configuration: "testRuntime")
+}
+
+task validatesRunnerBatch(type: Test) {
+ group = "Verification"
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(["--runner=TestJetRunner"])
+
+ classpath = configurations.validatesRunner
+ testClassesDirs = files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ useJUnit {
+ includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
+ excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse' //impulse doesn't cooperate properly with Jet when multiple cluster members are used
+ exclude '**/SplittableDoFnTest.class' //Splittable DoFn functionality not yet in the runner
+ }
+
+ maxHeapSize = '4g'
+}
+
+task validatesRunner {
+ group = "Verification"
+ description "Validates Jet Runner"
+ dependsOn validatesRunnerBatch
+}
+
+task needsRunnerTests(type: Test) {
+ group = "Verification"
+ description = "Runs tests that require a runner to validate that piplines/transforms work correctly"
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(["--runner=TestJetRunner"])
+
+ classpath = configurations.needsRunner
+ testClassesDirs += files(project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
+ testClassesDirs += files(project(":runners:core-java").sourceSets.test.output.classesDirs)
+ testClassesDirs += files(project(":sdks:java:core").sourceSets.test.output.classesDirs)
+ useJUnit {
+ includeCategories "org.apache.beam.sdk.testing.NeedsRunner"
+ excludeCategories "org.apache.beam.sdk.testing.LargeKeys\$Above100MB"
+ }
+}
+
+task needsRunner {
+ group = "Verification"
+ description "Runs lower level tests with the Jet Runner"
+ dependsOn needsRunnerTests
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
new file mode 100644
index 0000000..86edb80
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.jet.core.DAG;
+import com.hazelcast.jet.core.Edge;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.Vertex;
+import com.hazelcast.jet.function.FunctionEx;
+import com.hazelcast.jet.function.SupplierEx;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
+
+/** Utility class for wiring up Jet DAGs based on Beam pipelines. */
+public class DAGBuilder {
+
+ private final DAG dag = new DAG();
+ private final int localParallelism;
+
+ private final Map<String, Vertex> edgeStartPoints = new HashMap<>();
+ private final Map<String, List<Vertex>> edgeEndPoints = new HashMap<>();
+ private final Map<String, Coder> edgeCoders = new HashMap<>();
+ private final Map<String, String> pCollsOfEdges = new HashMap<>();
+ private final Set<String> sideInputCollections = new HashSet<>();
+
+ private final List<WiringListener> listeners = new ArrayList<>();
+
+ private int vertexId = 0;
+
+ DAGBuilder(JetPipelineOptions options) {
+ this.localParallelism = options.getJetDefaultParallelism();
+ }
+
+ DAG getDag() {
+ wireUp();
+ return dag;
+ }
+
+ void registerConstructionListeners(WiringListener listener) {
+ listeners.add(listener);
+ }
+
+ String newVertexId(String transformName) {
+ return vertexId++ + " (" + transformName + ")";
+ }
+
+ void registerCollectionOfEdge(String edgeId, String pCollId) {
+ String prevPCollId = pCollsOfEdges.put(edgeId, pCollId);
+ if (prevPCollId != null) {
+ throw new RuntimeException("Oops!");
+ }
+ }
+
+ void registerEdgeStartPoint(String edgeId, Vertex vertex, Coder coder) {
+ Objects.requireNonNull(edgeId);
+ Objects.requireNonNull(vertex);
+ Objects.requireNonNull(coder);
+
+ Vertex prevVertex = edgeStartPoints.put(edgeId, vertex);
+ if (prevVertex != null) {
+ throw new RuntimeException("Oops!");
+ }
+
+ Coder prevCoder = edgeCoders.put(edgeId, coder);
+ if (prevCoder != null) {
+ throw new RuntimeException("Oops!");
+ }
+ }
+
+ void registerEdgeEndPoint(String edgeId, Vertex vertex) {
+ edgeEndPoints.computeIfAbsent(edgeId, x -> new ArrayList<>()).add(vertex);
+ }
+
+ void registerSideInput(PCollectionView<?> view) {
+ sideInputCollections.add(view.getTagInternal().getId());
+ }
+
+ Vertex addVertex(String id, ProcessorMetaSupplier processorMetaSupplier) {
+ return dag.newVertex(id, processorMetaSupplier);
+ }
+
+ Vertex addVertex(String id, SupplierEx<Processor> processor) {
+ return dag.newVertex(id, processor).localParallelism(localParallelism);
+ }
+
+ private void wireUp() {
+ new WiringInstaller().wireUp();
+ }
+
+ /**
+ * Listener that can be registered with a {@link DAGBuilder} in order to be notified when edges
+ * are being registered.
+ */
+ public interface WiringListener {
+
+ void isOutboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId);
+
+ void isInboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId);
+ }
+
+ private class WiringInstaller {
+
+ private final Map<Vertex, Integer> inboundOrdinals = new HashMap<>();
+ private final Map<Vertex, Integer> outboundOrdinals = new HashMap<>();
+
+ void wireUp() {
+ Collection<String> edgeIds = new HashSet<>();
+ edgeIds.addAll(edgeStartPoints.keySet());
+ edgeIds.addAll(edgeEndPoints.keySet());
+
+ for (String edgeId : edgeIds) {
+ String pCollId = pCollsOfEdges.get(edgeId);
+ if (pCollId == null) {
+ throw new RuntimeException("Oops!");
+ }
+
+ Vertex sourceVertex = edgeStartPoints.get(edgeId);
+ if (sourceVertex == null) {
+ throw new RuntimeException("Oops!");
+ }
+
+ Coder edgeCoder = edgeCoders.get(edgeId);
+ if (edgeCoder == null) {
+ throw new RuntimeException("Oops!");
+ }
+
+ List<Vertex> destinationVertices =
+ edgeEndPoints.getOrDefault(edgeId, Collections.emptyList());
+ boolean sideInputEdge = sideInputCollections.contains(pCollId);
+ for (Vertex destinationVertex : destinationVertices) {
+ addEdge(sourceVertex, destinationVertex, edgeCoder, edgeId, pCollId, sideInputEdge);
+ }
+ }
+ }
+
+ private void addEdge(
+ Vertex sourceVertex,
+ Vertex destinationVertex,
+ Coder coder,
+ String edgeId,
+ String pCollId,
+ boolean sideInputEdge) {
+ try {
+ Edge edge =
+ Edge.from(sourceVertex, getNextFreeOrdinal(sourceVertex, false))
+ .to(destinationVertex, getNextFreeOrdinal(destinationVertex, true));
+ edge = edge.distributed();
+ if (sideInputEdge) {
+ edge = edge.broadcast();
+ } else {
+ edge = edge.partitioned(new PartitionedKeyExtractor(coder));
+ }
+ dag.edge(edge);
+
+ String sourceVertexName = sourceVertex.getName();
+ String destinationVertexName = destinationVertex.getName();
+ for (WiringListener listener : listeners) {
+ listener.isInboundEdgeOfVertex(edge, edgeId, pCollId, destinationVertexName);
+ listener.isOutboundEdgeOfVertex(edge, edgeId, pCollId, sourceVertexName);
+ }
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private int getNextFreeOrdinal(Vertex vertex, boolean inbound) {
+ Map<Vertex, Integer> ordinals = inbound ? inboundOrdinals : outboundOrdinals;
+ int nextOrdinal = 1 + ordinals.getOrDefault(vertex, -1);
+ ordinals.put(vertex, nextOrdinal);
+ return nextOrdinal;
+ }
+ }
+
+ private static class PartitionedKeyExtractor<K, V> implements FunctionEx<byte[], Object> {
+ private final WindowedValue.WindowedValueCoder<KV<K, V>> coder;
+
+ PartitionedKeyExtractor(Coder coder) {
+ this.coder =
+ Utils.isKeyedValueCoder(coder)
+ ? (WindowedValue.WindowedValueCoder<KV<K, V>>) coder
+ : null;
+ }
+
+ @Override
+ public Object applyEx(byte[] b) throws Exception {
+ if (coder == null) {
+ return "ALL";
+ } else {
+ WindowedValue<KV<K, V>> windowedValue =
+ CoderUtils.decodeFromByteArray(coder, b); // todo: decoding twice....
+ KvCoder<K, V> kvCoder = (KvCoder<K, V>) coder.getValueCoder();
+ return CoderUtils.encodeToByteArray(
+ kvCoder.getKeyCoder(), windowedValue.getValue().getKey());
+ }
+ }
+ }
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
new file mode 100644
index 0000000..59e1db4
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.jet.core.DAG;
+import java.util.function.Function;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.runners.TransformHierarchy;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PValue;
+
+/** Logic that specifies how to apply translations when traversing the nodes of a Beam pipeline. */
+class JetGraphVisitor extends Pipeline.PipelineVisitor.Defaults {
+
+ private final JetTranslationContext translationContext;
+ private final Function<PTransform<?, ?>, JetTransformTranslator<?>> translatorProvider;
+
+ private boolean finalized = false;
+
+ JetGraphVisitor(
+ JetPipelineOptions options,
+ Function<PTransform<?, ?>, JetTransformTranslator<?>> translatorProvider) {
+ this.translationContext = new JetTranslationContext(options);
+ this.translatorProvider = translatorProvider;
+ }
+
+ @Override
+ public CompositeBehavior enterCompositeTransform(TransformHierarchy.Node node) {
+ if (finalized) {
+ throw new IllegalStateException("Attempting to traverse an already finalized pipeline!");
+ }
+
+ PTransform<?, ?> transform = node.getTransform();
+ if (transform != null) {
+ JetTransformTranslator<?> translator = translatorProvider.apply(transform);
+ if (translator != null) {
+ translate(node, translator);
+ return CompositeBehavior.DO_NOT_ENTER_TRANSFORM;
+ }
+ }
+ return CompositeBehavior.ENTER_TRANSFORM;
+ }
+
+ @Override
+ public void leaveCompositeTransform(TransformHierarchy.Node node) {
+ if (finalized) {
+ throw new IllegalStateException("Attempting to traverse an already finalized pipeline!");
+ }
+ if (node.isRootNode()) {
+ finalized = true;
+ }
+ }
+
+ @Override
+ public void visitPrimitiveTransform(TransformHierarchy.Node node) {
+ PTransform<?, ?> transform = node.getTransform();
+ JetTransformTranslator<?> translator = translatorProvider.apply(transform);
+ if (translator == null) {
+ String transformUrn = PTransformTranslation.urnForTransform(transform);
+ throw new UnsupportedOperationException(
+ "The transform " + transformUrn + " is currently not supported.");
+ }
+ translate(node, translator);
+ }
+
+ @Override
+ public void visitValue(PValue value, TransformHierarchy.Node producer) {
+ // do nothing here
+ }
+
+ DAG getDAG() {
+ return translationContext.getDagBuilder().getDag();
+ }
+
+ private <T extends PTransform<?, ?>> void translate(
+ TransformHierarchy.Node node, JetTransformTranslator<?> translator) {
+ @SuppressWarnings("unchecked")
+ JetTransformTranslator<T> typedTranslator = (JetTransformTranslator<T>) translator;
+ Pipeline pipeline = getPipeline();
+ AppliedPTransform<?, ?, ?> appliedTransform = node.toAppliedPTransform(pipeline);
+ typedTranslator.translate(pipeline, appliedTransform, node, translationContext);
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
new file mode 100644
index 0000000..b12bdf5
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.Validation;
+
+/** Pipeline options specific to the Jet runner. */
+public interface JetPipelineOptions extends PipelineOptions {
+
+ @Description("Name of Jet group")
+ @Validation.Required
+ @Default.String("jet")
+ String getJetGroupName();
+
+ void setJetGroupName(String jetGroupName);
+
+ @Description("Specifies the addresses of the Jet cluster; needed only with external clusters")
+ @Validation.Required
+ @Default.String("127.0.0.1:5701")
+ String getJetServers();
+
+ void setJetServers(String jetServers);
+
+ @Description(
+ "Specifies where the fat-jar containing all the code is located; needed only with external clusters")
+ String getCodeJarPathname();
+
+ void setCodeJarPathname(String codeJarPathname);
+
+ @Description("Local parallelism of Jet nodes")
+ @Validation.Required
+ @Default.Integer(2)
+ Integer getJetDefaultParallelism();
+
+ void setJetDefaultParallelism(Integer localParallelism);
+
+ @Description("Number of locally started Jet Cluster Members")
+ @Validation.Required
+ @Default.Integer(0)
+ Integer getJetLocalMode();
+
+ void setJetLocalMode(Integer noOfLocalClusterMembers);
+
+ @Description("Weather Jet Processors for DoFns should use green threads or not")
+ @Validation.Required
+ @Default.Boolean(false)
+ Boolean getJetProcessorsCooperative();
+
+ void setJetProcessorsCooperative(Boolean cooperative);
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
new file mode 100644
index 0000000..17f69c5
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.jet.IMapJet;
+import com.hazelcast.jet.Job;
+import com.hazelcast.jet.core.JobStatus;
+import java.io.IOException;
+import java.util.Objects;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.core.metrics.MetricUpdates;
+import org.apache.beam.runners.jet.metrics.JetMetricResults;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Jet specific implementation of {@link PipelineResult}. */
+public class JetPipelineResult implements PipelineResult {
+
+ private static final Logger LOG = LoggerFactory.getLogger(JetRunner.class);
+
+ private final Job job;
+ private final JetMetricResults metricResults;
+ private volatile State terminalState;
+
+ private CompletableFuture<Void> completionFuture;
+
+ JetPipelineResult(@Nonnull Job job, @Nonnull IMapJet<String, MetricUpdates> metricsAccumulator) {
+ this.job = Objects.requireNonNull(job);
+ // save the terminal state when the job completes because the `job` instance will become invalid
+ // afterwards
+ metricResults = new JetMetricResults(metricsAccumulator);
+ }
+
+ void setCompletionFuture(CompletableFuture<Void> completionFuture) {
+ this.completionFuture = completionFuture;
+ }
+
+ void freeze(Throwable throwable) {
+ metricResults.freeze();
+ terminalState = throwable != null ? State.FAILED : State.DONE;
+ }
+
+ @Override
+ public State getState() {
+ if (terminalState != null) {
+ return terminalState;
+ }
+ JobStatus status = job.getStatus();
+ switch (status) {
+ case COMPLETED:
+ return State.DONE;
+ case COMPLETING:
+ case RUNNING:
+ case STARTING:
+ return State.RUNNING;
+ case FAILED:
+ return State.FAILED;
+ case NOT_RUNNING:
+ case SUSPENDED:
+ case SUSPENDED_EXPORTING_SNAPSHOT:
+ return State.STOPPED;
+ default:
+ LOG.warn("Unhandled " + JobStatus.class.getSimpleName() + ": " + status.name() + "!");
+ return State.UNKNOWN;
+ }
+ }
+
+ @Override
+ public State cancel() throws IOException {
+ if (terminalState != null) {
+ throw new IllegalStateException("Job already completed");
+ }
+ try {
+ job.cancel();
+ job.join();
+ } catch (CancellationException ignored) {
+ } catch (Exception e) {
+ throw new IOException("Failed to cancel the job: " + e, e);
+ }
+ return State.FAILED;
+ }
+
+ @Override
+ public State waitUntilFinish(Duration duration) {
+ if (terminalState != null) {
+ return terminalState;
+ }
+
+ try {
+ completionFuture.get(duration.getMillis(), TimeUnit.MILLISECONDS);
+ return State.DONE;
+ } catch (InterruptedException | TimeoutException e) {
+ return getState(); // job should be RUNNING or STOPPED
+ } catch (ExecutionException e) {
+ throw new CompletionException(e.getCause());
+ }
+ }
+
+ @Override
+ public State waitUntilFinish() {
+ return waitUntilFinish(new Duration(Long.MAX_VALUE));
+ }
+
+ @Override
+ public MetricResults metrics() {
+ return metricResults;
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunner.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunner.java
new file mode 100644
index 0000000..ccccb1f
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunner.java
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.client.config.ClientConfig;
+import com.hazelcast.jet.IMapJet;
+import com.hazelcast.jet.Jet;
+import com.hazelcast.jet.JetInstance;
+import com.hazelcast.jet.Job;
+import com.hazelcast.jet.config.JobConfig;
+import com.hazelcast.jet.core.DAG;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import org.apache.beam.runners.core.construction.UnconsumedReads;
+import org.apache.beam.runners.core.metrics.MetricUpdates;
+import org.apache.beam.runners.jet.metrics.JetMetricsContainer;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Jet specific implementation of Beam's {@link PipelineRunner}. */
+public class JetRunner extends PipelineRunner<PipelineResult> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(JetRunner.class);
+
+ public static JetRunner fromOptions(PipelineOptions options) {
+ return fromOptions(options, Jet::newJetClient);
+ }
+
+ public static JetRunner fromOptions(
+ PipelineOptions options, Function<ClientConfig, JetInstance> jetClientSupplier) {
+ return new JetRunner(options, jetClientSupplier);
+ }
+
+ private final JetPipelineOptions options;
+ private final Function<ClientConfig, JetInstance> jetClientSupplier;
+
+ private Function<PTransform<?, ?>, JetTransformTranslator<?>> translatorProvider;
+
+ private JetRunner(
+ PipelineOptions options, Function<ClientConfig, JetInstance> jetClientSupplier) {
+ this.options = validate(options.as(JetPipelineOptions.class));
+ this.jetClientSupplier = jetClientSupplier;
+ this.translatorProvider = JetTransformTranslators::getTranslator;
+ }
+
+ @Override
+ public PipelineResult run(Pipeline pipeline) {
+ try {
+ normalize(pipeline);
+ DAG dag = translate(pipeline);
+ return run(dag);
+ } catch (UnsupportedOperationException uoe) {
+ LOG.error("Failed running pipeline!", uoe);
+ return new FailedRunningPipelineResults(uoe);
+ }
+ }
+
+ void addExtraTranslators(
+ Function<PTransform<?, ?>, JetTransformTranslator<?>> extraTranslatorProvider) {
+ Function<PTransform<?, ?>, JetTransformTranslator<?>> initialTranslatorProvider =
+ this.translatorProvider;
+ this.translatorProvider =
+ transform -> {
+ JetTransformTranslator<?> translator = initialTranslatorProvider.apply(transform);
+ if (translator == null) {
+ translator = extraTranslatorProvider.apply(transform);
+ }
+ return translator;
+ };
+ }
+
+ private void normalize(Pipeline pipeline) {
+ pipeline.replaceAll(getDefaultOverrides());
+ UnconsumedReads.ensureAllReadsConsumed(pipeline);
+ }
+
+ private DAG translate(Pipeline pipeline) {
+ JetGraphVisitor graphVisitor = new JetGraphVisitor(options, translatorProvider);
+ pipeline.traverseTopologically(graphVisitor);
+ return graphVisitor.getDAG();
+ }
+
+ private JetPipelineResult run(DAG dag) {
+ startClusterIfNeeded(options);
+
+ JetInstance jet =
+ getJetInstance(
+ options); // todo: we use single client for each job, it might be better to have a
+ // shared client with refcount
+
+ Job job = jet.newJob(dag, getJobConfig(options));
+ IMapJet<String, MetricUpdates> metricsAccumulator =
+ jet.getMap(JetMetricsContainer.getMetricsMapName(job.getId()));
+ JetPipelineResult pipelineResult = new JetPipelineResult(job, metricsAccumulator);
+ CompletableFuture<Void> completionFuture =
+ job.getFuture()
+ .whenCompleteAsync(
+ (r, f) -> {
+ pipelineResult.freeze(f);
+ metricsAccumulator.destroy();
+ jet.shutdown();
+
+ stopClusterIfNeeded(options);
+ });
+ pipelineResult.setCompletionFuture(completionFuture);
+
+ return pipelineResult;
+ }
+
+ private void startClusterIfNeeded(JetPipelineOptions options) {
+ Integer noOfLocalMembers = options.getJetLocalMode();
+ if (noOfLocalMembers > 0) {
+ Collection<JetInstance> jetInstances = new ArrayList<>();
+ for (int i = 0; i < noOfLocalMembers; i++) {
+ jetInstances.add(Jet.newJetInstance());
+ }
+ LOG.info("Started " + jetInstances.size() + " Jet cluster members");
+ }
+ }
+
+ private void stopClusterIfNeeded(JetPipelineOptions options) {
+ Integer noOfLocalMembers = options.getJetLocalMode();
+ if (noOfLocalMembers > 0) {
+ Jet.shutdownAll();
+ LOG.info("Stopped all Jet cluster members");
+ }
+ }
+
+ private JobConfig getJobConfig(JetPipelineOptions options) {
+ JobConfig jobConfig = new JobConfig();
+
+ String jobName = options.getJobName();
+ if (jobName != null) {
+ jobConfig.setName(jobName);
+ }
+
+ boolean hasNoLocalMembers = options.getJetLocalMode() <= 0;
+ if (hasNoLocalMembers) {
+ String codeJarPathname = options.getCodeJarPathname();
+ if (codeJarPathname != null && !codeJarPathname.isEmpty()) {
+ jobConfig.addJar(codeJarPathname);
+ }
+ }
+
+ return jobConfig;
+ }
+
+ private JetInstance getJetInstance(JetPipelineOptions options) {
+ String jetGroupName = options.getJetGroupName();
+
+ ClientConfig clientConfig = new ClientConfig();
+ clientConfig.getGroupConfig().setName(jetGroupName);
+ boolean hasNoLocalMembers = options.getJetLocalMode() <= 0;
+ if (hasNoLocalMembers) {
+ clientConfig
+ .getNetworkConfig()
+ .setAddresses(Arrays.asList(options.getJetServers().split(",")));
+ }
+ return jetClientSupplier.apply(clientConfig);
+ }
+
+ private static List<PTransformOverride> getDefaultOverrides() {
+ return Collections.emptyList();
+ }
+
+ private static JetPipelineOptions validate(JetPipelineOptions options) {
+ if (options.getJetGroupName() == null) {
+ throw new IllegalArgumentException("Jet group name not set in options");
+ }
+
+ Integer localParallelism = options.getJetDefaultParallelism();
+ if (localParallelism == null) {
+ throw new IllegalArgumentException("Jet node local parallelism must be specified");
+ }
+ if (localParallelism != -1 && localParallelism < 1) {
+ throw new IllegalArgumentException("Jet node local parallelism must be >1 or -1");
+ }
+
+ return options;
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
new file mode 100644
index 0000000..7026930
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
+import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
+ * JetRunner}.
+ *
+ * <p>{@link AutoService} will register Apex's implementations of the {@link PipelineRunner} and
+ * {@link PipelineOptions} as available pipeline runner services.
+ */
+public final class JetRunnerRegistrar {
+ private JetRunnerRegistrar() {}
+
+ /** Registers the {@link JetRunner}. */
+ @AutoService(PipelineRunnerRegistrar.class)
+ public static class Runner implements PipelineRunnerRegistrar {
+ @Override
+ public Iterable<Class<? extends PipelineRunner<?>>> getPipelineRunners() {
+ return ImmutableList.of(JetRunner.class);
+ }
+ }
+
+ /** Registers the {@link JetPipelineOptions}. */
+ @AutoService(PipelineOptionsRegistrar.class)
+ public static class Options implements PipelineOptionsRegistrar {
+ @Override
+ public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
+ return ImmutableList.of(JetPipelineOptions.class);
+ }
+ }
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
new file mode 100644
index 0000000..460ae14
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
@@ -0,0 +1,431 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.Vertex;
+import com.hazelcast.jet.function.SupplierEx;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.runners.core.construction.CreatePCollectionViewTranslation;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.ParDoTranslation;
+import org.apache.beam.runners.core.construction.ReadTranslation;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.jet.processors.AssignWindowP;
+import org.apache.beam.runners.jet.processors.BoundedSourceP;
+import org.apache.beam.runners.jet.processors.FlattenP;
+import org.apache.beam.runners.jet.processors.ImpulseP;
+import org.apache.beam.runners.jet.processors.ParDoP;
+import org.apache.beam.runners.jet.processors.StatefulParDoP;
+import org.apache.beam.runners.jet.processors.UnboundedSourceP;
+import org.apache.beam.runners.jet.processors.ViewP;
+import org.apache.beam.runners.jet.processors.WindowGroupP;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.runners.TransformHierarchy.Node;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+
+@SuppressWarnings("unchecked")
+class JetTransformTranslators {
+
+ /** A map from a Transform URN to the translator. */
+ private static final Map<String, JetTransformTranslator> TRANSLATORS = new HashMap<>();
+
+ static {
+ TRANSLATORS.put(PTransformTranslation.READ_TRANSFORM_URN, new ReadSourceTranslator());
+ TRANSLATORS.put(PTransformTranslation.CREATE_VIEW_TRANSFORM_URN, new CreateViewTranslator());
+ TRANSLATORS.put(PTransformTranslation.PAR_DO_TRANSFORM_URN, new ParDoTranslator());
+ TRANSLATORS.put(PTransformTranslation.GROUP_BY_KEY_TRANSFORM_URN, new GroupByKeyTranslator());
+ TRANSLATORS.put(PTransformTranslation.FLATTEN_TRANSFORM_URN, new FlattenTranslator());
+ TRANSLATORS.put(PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN, new WindowTranslator());
+ TRANSLATORS.put(PTransformTranslation.IMPULSE_TRANSFORM_URN, new ImpulseTranslator());
+ }
+
+ static JetTransformTranslator<?> getTranslator(PTransform<?, ?> transform) {
+ String urn = PTransformTranslation.urnForTransformOrNull(transform);
+ return urn == null ? null : TRANSLATORS.get(urn);
+ }
+
+ private static class ReadSourceTranslator<T>
+ implements JetTransformTranslator<PTransform<PBegin, PCollection<T>>> {
+
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder =
+ Utils.getCoder((PCollection) Utils.getOutput(appliedTransform).getValue());
+
+ String transformName = appliedTransform.getFullName();
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(transformName);
+ SerializablePipelineOptions pipelineOptions = context.getOptions();
+ ProcessorMetaSupplier processorSupplier =
+ getProcessorSupplier(
+ (AppliedPTransform<PBegin, PCollection<T>, PTransform<PBegin, PCollection<T>>>)
+ appliedTransform,
+ outputCoder,
+ vertexId,
+ pipelineOptions);
+
+ Vertex vertex = dagBuilder.addVertex(vertexId, processorSupplier);
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+
+ private ProcessorMetaSupplier getProcessorSupplier(
+ AppliedPTransform<PBegin, PCollection<T>, PTransform<PBegin, PCollection<T>>>
+ appliedTransform,
+ Coder outputCoder,
+ String vertexId,
+ SerializablePipelineOptions pipelineOptions) {
+ try {
+ if (Utils.isBounded(appliedTransform)) {
+ BoundedSource<T> source = ReadTranslation.boundedSourceFromTransform(appliedTransform);
+ return BoundedSourceP.supplier(source, pipelineOptions, outputCoder, vertexId);
+ } else {
+ UnboundedSource<T, ?> source =
+ ReadTranslation.unboundedSourceFromTransform(appliedTransform);
+ if (source.requiresDeduping()) {
+ throw new UnsupportedOperationException(
+ "Sources requiring deduping not supported!"); // todo
+ }
+ return UnboundedSourceP.supplier(source, pipelineOptions, outputCoder, vertexId);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ private static class ParDoTranslator
+ implements JetTransformTranslator<PTransform<PCollection, PCollectionTuple>> {
+
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ boolean usesStateOrTimers = Utils.usesStateOrTimers(appliedTransform);
+ DoFn<?, ?> doFn = Utils.getDoFn(appliedTransform);
+
+ Map<TupleTag<?>, PValue> outputs = Utils.getOutputs(appliedTransform);
+
+ TupleTag<?> mainOutputTag;
+ try {
+ mainOutputTag = ParDoTranslation.getMainOutputTag(appliedTransform);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ Map<TupleTag<?>, Integer> outputMap = new HashMap<>();
+ int count = 1;
+ for (TupleTag<?> tag : outputs.keySet()) {
+ if (!outputMap.containsKey(tag)) {
+ outputMap.put(tag, count++);
+ }
+ }
+ final WindowingStrategy<?, ?> windowingStrategy =
+ Utils.getWindowingStrategy(appliedTransform);
+
+ Map<TupleTag<?>, Coder<?>> outputValueCoders = Utils.getOutputValueCoders(appliedTransform);
+ Map<TupleTag<?>, Coder> outputCoders =
+ Utils.getCoders(Utils.getOutputs(appliedTransform), Map.Entry::getKey);
+
+ String transformName = appliedTransform.getFullName();
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String stepId =
+ transformName.contains("/")
+ ? transformName.substring(0, transformName.indexOf('/'))
+ : transformName;
+ String vertexId =
+ dagBuilder.newVertexId(transformName) + (usesStateOrTimers ? " - STATEFUL" : "");
+ SerializablePipelineOptions pipelineOptions = context.getOptions();
+ Coder inputValueCoder = ((PCollection) Utils.getInput(appliedTransform)).getCoder();
+ Coder inputCoder = Utils.getCoder((PCollection) Utils.getInput(appliedTransform));
+ List<PCollectionView<?>> sideInputs = Utils.getSideInputs(appliedTransform);
+ Map<? extends PCollectionView<?>, Coder> sideInputCoders =
+ sideInputs.stream()
+ .collect(Collectors.toMap(si -> si, si -> Utils.getCoder(si.getPCollection())));
+ DoFnSchemaInformation doFnSchemaInformation =
+ ParDoTranslation.getSchemaInformation(appliedTransform);
+ SupplierEx<Processor> processorSupplier =
+ usesStateOrTimers
+ ? new StatefulParDoP.Supplier(
+ stepId,
+ vertexId,
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ pipelineOptions,
+ mainOutputTag,
+ outputMap.keySet(),
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ sideInputs)
+ : new ParDoP.Supplier(
+ stepId,
+ vertexId,
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ pipelineOptions,
+ mainOutputTag,
+ outputMap.keySet(),
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ sideInputs);
+
+ Vertex vertex = dagBuilder.addVertex(vertexId, processorSupplier);
+ dagBuilder.registerConstructionListeners((DAGBuilder.WiringListener) processorSupplier);
+
+ PValue mainInput = Utils.getMainInput(pipeline, node);
+ dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(mainInput), vertex);
+
+ Map<TupleTag<?>, PValue> additionalInputs = Utils.getAdditionalInputs(node);
+ if (additionalInputs != null && !additionalInputs.isEmpty()) {
+ for (TupleTag<?> tupleTag : additionalInputs.keySet()) {
+ dagBuilder.registerEdgeEndPoint(tupleTag.getId(), vertex);
+ }
+ }
+
+ for (Map.Entry<TupleTag<?>, PValue> entry : outputs.entrySet()) {
+ TupleTag<?> pCollId = entry.getKey();
+ String edgeId = Utils.getTupleTagId(entry.getValue());
+ dagBuilder.registerCollectionOfEdge(edgeId, pCollId.getId());
+ dagBuilder.registerEdgeStartPoint(edgeId, vertex, outputCoders.get(pCollId));
+ }
+
+ return vertex;
+ }
+ }
+
+ private static class GroupByKeyTranslator<K, InputT>
+ implements JetTransformTranslator<
+ PTransform<PCollection<KV<K, InputT>>, PCollection<KV<K, Iterable<InputT>>>>> {
+
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ String transformName = appliedTransform.getFullName();
+
+ PCollection<KV<K, InputT>> input =
+ (PCollection<KV<K, InputT>>) Utils.getInput(appliedTransform);
+ WindowedValue.WindowedValueCoder<KV<K, InputT>> inputCoder =
+ Utils.getWindowedValueCoder(input);
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
+
+ WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
+
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(transformName);
+ Vertex vertex =
+ dagBuilder.addVertex(
+ vertexId,
+ WindowGroupP.supplier(
+ context.getOptions(), inputCoder, outputCoder, windowingStrategy, vertexId));
+
+ dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+ }
+
+ private static class CreateViewTranslator<T>
+ implements JetTransformTranslator<PTransform<PCollection<T>, PCollection<T>>> {
+
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ PCollectionView<T> view;
+ try {
+ view =
+ CreatePCollectionViewTranslation.getView(
+ (AppliedPTransform<
+ PCollection<T>, PCollection<T>, PTransform<PCollection<T>, PCollection<T>>>)
+ appliedTransform);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ String transformName = appliedTransform.getFullName();
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(transformName);
+ PCollection<T> input = (PCollection<T>) Utils.getInput(appliedTransform);
+ Coder inputCoder = Utils.getCoder(input);
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
+
+ Vertex vertex =
+ dagBuilder.addVertex(
+ vertexId,
+ ViewP.supplier(inputCoder, outputCoder, input.getWindowingStrategy(), vertexId));
+
+ dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
+
+ String viewTag = Utils.getTupleTagId(view);
+ dagBuilder.registerSideInput(view);
+ dagBuilder.registerCollectionOfEdge(viewTag, view.getTagInternal().getId());
+ dagBuilder.registerEdgeStartPoint(viewTag, vertex, outputCoder);
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+ }
+
+ private static class FlattenTranslator<T>
+ implements JetTransformTranslator<PTransform<PCollectionList<T>, PCollection<T>>> {
+
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ Collection<PValue> mainInputs = Utils.getMainInputs(pipeline, node);
+ Map<String, Coder> inputCoders =
+ Utils.getCoders(
+ Utils.getInputs(appliedTransform), e -> Utils.getTupleTagId(e.getValue()));
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
+
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(appliedTransform.getFullName());
+ FlattenP.Supplier processorSupplier =
+ new FlattenP.Supplier(inputCoders, outputCoder, vertexId);
+ Vertex vertex = dagBuilder.addVertex(vertexId, processorSupplier);
+ dagBuilder.registerConstructionListeners(processorSupplier);
+
+ for (PValue value : mainInputs) {
+ PCollection<T> input = (PCollection<T>) value;
+ dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
+ }
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+ }
+
+ private static class WindowTranslator<T>
+ implements JetTransformTranslator<PTransform<PCollection<T>, PCollection<T>>> {
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ WindowingStrategy<T, BoundedWindow> windowingStrategy =
+ (WindowingStrategy<T, BoundedWindow>)
+ ((PCollection) Utils.getOutput(appliedTransform).getValue()).getWindowingStrategy();
+
+ PCollection<WindowedValue> input =
+ (PCollection<WindowedValue>) Utils.getInput(appliedTransform);
+ Coder inputCoder = Utils.getCoder(input);
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder =
+ Utils.getCoder((PCollection) Utils.getOutput(appliedTransform).getValue());
+
+ String transformName = appliedTransform.getFullName();
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(transformName);
+
+ Vertex vertex =
+ dagBuilder.addVertex(
+ vertexId,
+ AssignWindowP.supplier(inputCoder, outputCoder, windowingStrategy, vertexId));
+ dagBuilder.registerEdgeEndPoint(Utils.getTupleTagId(input), vertex);
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+ }
+
+ private static class ImpulseTranslator
+ implements JetTransformTranslator<PTransform<PBegin, PCollection<byte[]>>> {
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ Node node,
+ JetTranslationContext context) {
+ String transformName = appliedTransform.getFullName();
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(transformName);
+
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder =
+ Utils.getCoder((PCollection) Utils.getOutput(appliedTransform).getValue());
+ Vertex vertex = dagBuilder.addVertex(vertexId, ImpulseP.supplier(outputCoder, vertexId));
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+ }
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/Utils.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/Utils.java
new file mode 100644
index 0000000..7489bea
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/Utils.java
@@ -0,0 +1,292 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import static com.hazelcast.jet.impl.util.ExceptionUtil.rethrow;
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.core.construction.ParDoTranslation;
+import org.apache.beam.runners.core.construction.TransformInputs;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.runners.TransformHierarchy;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/** Various common methods used by the Jet based runner. */
+public class Utils {
+
+ public static String getTupleTagId(PValue value) {
+ Map<TupleTag<?>, PValue> expansion = value.expand();
+ return Iterables.getOnlyElement(expansion.keySet()).getId();
+ }
+
+ static PValue getMainInput(Pipeline pipeline, TransformHierarchy.Node node) {
+ Collection<PValue> mainInputs = getMainInputs(pipeline, node);
+ return mainInputs == null ? null : Iterables.getOnlyElement(mainInputs);
+ }
+
+ static Collection<PValue> getMainInputs(Pipeline pipeline, TransformHierarchy.Node node) {
+ if (node.getTransform() == null) {
+ return null;
+ }
+ return TransformInputs.nonAdditionalInputs(node.toAppliedPTransform(pipeline));
+ }
+
+ static Map<TupleTag<?>, PValue> getInputs(AppliedPTransform<?, ?, ?> appliedTransform) {
+ return appliedTransform.getInputs();
+ }
+
+ static Map<TupleTag<?>, PValue> getAdditionalInputs(TransformHierarchy.Node node) {
+ return node.getTransform() != null ? node.getTransform().getAdditionalInputs() : null;
+ }
+
+ @SuppressWarnings("unchecked")
+ static PValue getInput(AppliedPTransform<?, ?, ?> appliedTransform) {
+ if (appliedTransform.getTransform() == null) {
+ return null;
+ }
+ return Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(appliedTransform));
+ }
+
+ static Map<TupleTag<?>, PValue> getOutputs(AppliedPTransform<?, ?, ?> appliedTransform) {
+ if (appliedTransform.getTransform() == null) {
+ return null;
+ }
+ return appliedTransform.getOutputs();
+ }
+
+ static Map.Entry<TupleTag<?>, PValue> getOutput(AppliedPTransform<?, ?, ?> appliedTransform) {
+ return Iterables.getOnlyElement(getOutputs(appliedTransform).entrySet());
+ }
+
+ static <T> boolean isBounded(AppliedPTransform<?, ?, ?> appliedTransform) {
+ return ((PCollection) getOutput(appliedTransform).getValue())
+ .isBounded()
+ .equals(PCollection.IsBounded.BOUNDED);
+ }
+
+ static boolean isKeyedValueCoder(Coder coder) {
+ if (coder instanceof KvCoder) {
+ return true;
+ } else if (coder instanceof WindowedValue.WindowedValueCoder) {
+ return ((WindowedValue.WindowedValueCoder) coder).getValueCoder() instanceof KvCoder;
+ }
+ return false;
+ }
+
+ static Coder getCoder(PCollection pCollection) {
+ if (pCollection.getWindowingStrategy() == null) {
+ return pCollection.getCoder();
+ } else {
+ return getWindowedValueCoder(pCollection);
+ }
+ }
+
+ static <T> WindowedValue.WindowedValueCoder<T> getWindowedValueCoder(PCollection<T> pCollection) {
+ return WindowedValue.FullWindowedValueCoder.of(
+ pCollection.getCoder(), pCollection.getWindowingStrategy().getWindowFn().windowCoder());
+ }
+
+ static <T> Map<T, Coder> getCoders(
+ Map<TupleTag<?>, PValue> pCollections,
+ Function<Map.Entry<TupleTag<?>, PValue>, T> tupleTagExtractor) {
+ return pCollections.entrySet().stream()
+ .collect(Collectors.toMap(tupleTagExtractor, e -> getCoder((PCollection) e.getValue())));
+ }
+
+ static Map<TupleTag<?>, Coder<?>> getOutputValueCoders(
+ AppliedPTransform<?, ?, ?> appliedTransform) {
+ return appliedTransform.getOutputs().entrySet().stream()
+ .filter(e -> e.getValue() instanceof PCollection)
+ .collect(Collectors.toMap(Map.Entry::getKey, e -> ((PCollection) e.getValue()).getCoder()));
+ }
+
+ static List<PCollectionView<?>> getSideInputs(AppliedPTransform<?, ?, ?> appliedTransform) {
+ PTransform<?, ?> transform = appliedTransform.getTransform();
+ if (transform instanceof ParDo.MultiOutput) {
+ ParDo.MultiOutput multiParDo = (ParDo.MultiOutput) transform;
+ return (List) multiParDo.getSideInputs().values().stream().collect(Collectors.toList());
+ } else if (transform instanceof ParDo.SingleOutput) {
+ ParDo.SingleOutput singleParDo = (ParDo.SingleOutput) transform;
+ return (List) singleParDo.getSideInputs().values().stream().collect(Collectors.toList());
+ }
+ return Collections.emptyList();
+ }
+
+ static boolean usesStateOrTimers(AppliedPTransform<?, ?, ?> appliedTransform) {
+ try {
+ return ParDoTranslation.usesStateOrTimers(appliedTransform);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ static DoFn<?, ?> getDoFn(AppliedPTransform<?, ?, ?> appliedTransform) {
+ try {
+ DoFn<?, ?> doFn = ParDoTranslation.getDoFn(appliedTransform);
+ if (DoFnSignatures.signatureForDoFn(doFn).processElement().isSplittable()) {
+ throw new IllegalStateException(
+ "Not expected to directly translate splittable DoFn, should have been overridden: "
+ + doFn); // todo
+ }
+ return doFn;
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ static WindowingStrategy<?, ?> getWindowingStrategy(AppliedPTransform<?, ?, ?> appliedTransform) {
+ // assume that the windowing strategy is the same for all outputs
+
+ Map<TupleTag<?>, PValue> outputs = getOutputs(appliedTransform);
+
+ if (outputs == null || outputs.isEmpty()) {
+ throw new IllegalStateException("No outputs defined.");
+ }
+
+ PValue taggedValue = outputs.values().iterator().next();
+ checkState(
+ taggedValue instanceof PCollection,
+ "Within ParDo, got a non-PCollection output %s of type %s",
+ taggedValue,
+ taggedValue.getClass().getSimpleName());
+ PCollection<?> coll = (PCollection<?>) taggedValue;
+ return coll.getWindowingStrategy();
+ }
+
+ /**
+ * Assigns the {@code list} to {@code count} sublists in a round-robin fashion. One call returns
+ * the {@code index}-th sublist.
+ *
+ * <p>For example, for a 7-element list where {@code count == 3}, it would respectively return for
+ * indices 0..2:
+ *
+ * <pre>
+ * 0, 3, 6
+ * 1, 4
+ * 2, 5
+ * </pre>
+ */
+ @Nonnull
+ public static <T> List<T> roundRobinSubList(@Nonnull List<T> list, int index, int count) {
+ if (index < 0 || index >= count) {
+ throw new IllegalArgumentException("index=" + index + ", count=" + count);
+ }
+ return IntStream.range(0, list.size())
+ .filter(i -> i % count == index)
+ .mapToObj(list::get)
+ .collect(toList());
+ }
+
+ /** Returns a deep clone of an object by serializing and deserializing it (ser-de). */
+ @SuppressWarnings("unchecked")
+ public static <T> T serde(T object) {
+ try {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos);
+ oos.writeObject(object);
+ oos.close();
+ byte[] byteData = baos.toByteArray();
+ ByteArrayInputStream bais = new ByteArrayInputStream(byteData);
+ return (T) new ObjectInputStream(bais).readObject();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static <T> byte[] encode(T value, Coder<T> coder) {
+ try {
+ return CoderUtils.encodeToByteArray(coder, value);
+ } catch (IOException e) {
+ throw rethrow(e);
+ }
+ }
+
+ public static <T> WindowedValue<T> decodeWindowedValue(byte[] item, Coder coder) {
+ try {
+ return (WindowedValue<T>) CoderUtils.decodeFromByteArray(coder, item);
+ } catch (IOException e) {
+ throw rethrow(e);
+ }
+ }
+
+ public static WindowedValue.FullWindowedValueCoder deriveIterableValueCoder(
+ WindowedValue.FullWindowedValueCoder elementCoder) {
+ return WindowedValue.FullWindowedValueCoder.of(
+ ListCoder.of(elementCoder.getValueCoder()), elementCoder.getWindowCoder());
+ }
+
+ /** A wrapper of {@code byte[]} that can be used as a hash-map key. */
+ public static class ByteArrayKey {
+ private final byte[] value;
+ private int hash;
+
+ public ByteArrayKey(@Nonnull byte[] value) {
+ this.value = value;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ ByteArrayKey that = (ByteArrayKey) o;
+ return Arrays.equals(value, that.value);
+ }
+
+ @Override
+ public int hashCode() {
+ if (hash == 0) {
+ hash = Arrays.hashCode(value);
+ }
+ return hash;
+ }
+ }
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
new file mode 100644
index 0000000..f9db657
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.metrics;
+
+import com.hazelcast.jet.IMapJet;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+import org.apache.beam.runners.core.metrics.DistributionData;
+import org.apache.beam.runners.core.metrics.GaugeData;
+import org.apache.beam.runners.core.metrics.MetricUpdates;
+import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
+import org.apache.beam.sdk.metrics.DistributionResult;
+import org.apache.beam.sdk.metrics.GaugeResult;
+import org.apache.beam.sdk.metrics.MetricFiltering;
+import org.apache.beam.sdk.metrics.MetricKey;
+import org.apache.beam.sdk.metrics.MetricQueryResults;
+import org.apache.beam.sdk.metrics.MetricResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.apache.beam.sdk.metrics.MetricsFilter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+
+/** Jet specific {@link MetricResults}. */
+public class JetMetricResults extends MetricResults {
+
+ @GuardedBy("this")
+ private final Counters counters = new Counters();
+
+ @GuardedBy("this")
+ private final Distributions distributions = new Distributions();
+
+ @GuardedBy("this")
+ private final Gauges gauges = new Gauges();
+
+ @GuardedBy("this")
+ private IMapJet<String, MetricUpdates> metricsAccumulator;
+
+ public JetMetricResults(IMapJet<String, MetricUpdates> metricsAccumulator) {
+ this.metricsAccumulator = metricsAccumulator;
+ }
+
+ public synchronized void freeze() {
+ updateLocalMetrics(metricsAccumulator);
+ this.metricsAccumulator = null;
+ }
+
+ @Override
+ public synchronized MetricQueryResults queryMetrics(@Nullable MetricsFilter filter) {
+ if (metricsAccumulator != null) {
+ updateLocalMetrics(metricsAccumulator);
+ }
+ return new QueryResults(
+ counters.filter(filter), distributions.filter(filter), gauges.filter(filter));
+ }
+
+ private synchronized void updateLocalMetrics(IMapJet<String, MetricUpdates> metricsAccumulator) {
+ counters.clear();
+ distributions.clear();
+ gauges.clear();
+
+ for (MetricUpdates metricUpdates : metricsAccumulator.values()) {
+ counters.merge(metricUpdates.counterUpdates());
+ distributions.merge(metricUpdates.distributionUpdates());
+ gauges.merge(metricUpdates.gaugeUpdates());
+ }
+ }
+
+ private static Predicate<Map.Entry<MetricKey, ?>> matchesFilter(final MetricsFilter filter) {
+ return entry -> MetricFiltering.matches(filter, entry.getKey());
+ }
+
+ private static class QueryResults extends MetricQueryResults {
+ private final Iterable<MetricResult<Long>> counters;
+ private final Iterable<MetricResult<DistributionResult>> distributions;
+ private final Iterable<MetricResult<GaugeResult>> gauges;
+
+ private QueryResults(
+ Iterable<MetricResult<Long>> counters,
+ Iterable<MetricResult<DistributionResult>> distributions,
+ Iterable<MetricResult<GaugeResult>> gauges) {
+ this.counters = counters;
+ this.distributions = distributions;
+ this.gauges = gauges;
+ }
+
+ @Override
+ public Iterable<MetricResult<Long>> getCounters() {
+ return counters;
+ }
+
+ @Override
+ public Iterable<MetricResult<DistributionResult>> getDistributions() {
+ return distributions;
+ }
+
+ @Override
+ public Iterable<MetricResult<GaugeResult>> getGauges() {
+ return gauges;
+ }
+ }
+
+ private static class Counters {
+
+ private final Map<MetricKey, Long> counters = new HashMap<>();
+
+ void merge(Iterable<MetricUpdate<Long>> updates) {
+ for (MetricUpdate<Long> update : updates) {
+ MetricKey key = update.getKey();
+ Long oldValue = counters.getOrDefault(key, 0L);
+ Long updatedValue = oldValue + update.getUpdate();
+ counters.put(key, updatedValue);
+ }
+ }
+
+ void clear() {
+ counters.clear();
+ }
+
+ Iterable<MetricResult<Long>> filter(MetricsFilter filter) {
+ return FluentIterable.from(counters.entrySet())
+ .filter(matchesFilter(filter))
+ .transform(this::toUpdateResult)
+ .toList();
+ }
+
+ private MetricResult<Long> toUpdateResult(Map.Entry<MetricKey, Long> entry) {
+ MetricKey key = entry.getKey();
+ Long counter = entry.getValue();
+ return MetricResult.create(key, counter, counter);
+ }
+ }
+
+ private static class Distributions {
+
+ private final Map<MetricKey, DistributionData> distributions = new HashMap<>();
+
+ void merge(Iterable<MetricUpdate<DistributionData>> updates) {
+ for (MetricUpdate<DistributionData> update : updates) {
+ MetricKey key = update.getKey();
+ DistributionData oldDistribution = distributions.getOrDefault(key, DistributionData.EMPTY);
+ DistributionData updatedDistribution = update.getUpdate().combine(oldDistribution);
+ distributions.put(key, updatedDistribution);
+ }
+ }
+
+ void clear() {
+ distributions.clear();
+ }
+
+ Iterable<MetricResult<DistributionResult>> filter(MetricsFilter filter) {
+ return FluentIterable.from(distributions.entrySet())
+ .filter(matchesFilter(filter))
+ .transform(this::toUpdateResult)
+ .toList();
+ }
+
+ private MetricResult<DistributionResult> toUpdateResult(
+ Map.Entry<MetricKey, DistributionData> entry) {
+ MetricKey key = entry.getKey();
+ DistributionResult distributionResult = entry.getValue().extractResult();
+ return MetricResult.create(key, distributionResult, distributionResult);
+ }
+ }
+
+ private static class Gauges {
+
+ private final Map<MetricKey, GaugeData> gauges = new HashMap<>();
+
+ void merge(Iterable<MetricUpdate<GaugeData>> updates) {
+ for (MetricUpdate<GaugeData> update : updates) {
+ MetricKey key = update.getKey();
+ GaugeData oldGauge = gauges.getOrDefault(key, GaugeData.empty());
+ GaugeData updatedGauge = update.getUpdate().combine(oldGauge);
+ gauges.put(key, updatedGauge);
+ }
+ }
+
+ void clear() {
+ gauges.clear();
+ }
+
+ Iterable<MetricResult<GaugeResult>> filter(MetricsFilter filter) {
+ return FluentIterable.from(gauges.entrySet())
+ .filter(matchesFilter(filter))
+ .transform(this::toUpdateResult)
+ .toList();
+ }
+
+ private MetricResult<GaugeResult> toUpdateResult(Map.Entry<MetricKey, GaugeData> entry) {
+ MetricKey key = entry.getKey();
+ GaugeResult gaugeResult = entry.getValue().extractResult();
+ return MetricResult.create(key, gaugeResult, gaugeResult);
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
new file mode 100644
index 0000000..a300d4b
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.metrics;
+
+import com.hazelcast.jet.IMapJet;
+import com.hazelcast.jet.Util;
+import com.hazelcast.jet.core.Processor;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.runners.core.metrics.DistributionData;
+import org.apache.beam.runners.core.metrics.GaugeData;
+import org.apache.beam.runners.core.metrics.MetricUpdates;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Distribution;
+import org.apache.beam.sdk.metrics.Gauge;
+import org.apache.beam.sdk.metrics.MetricKey;
+import org.apache.beam.sdk.metrics.MetricName;
+import org.apache.beam.sdk.metrics.MetricsContainer;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Jet specific implementation of {@link MetricsContainer}. */
+public class JetMetricsContainer implements MetricsContainer {
+
+ public static String getMetricsMapName(long jobId) {
+ return Util.idToString(jobId) + "_METRICS";
+ }
+
+ private final String stepName;
+ private final String metricsKey;
+
+ private final Map<MetricName, CounterImpl> counters = new HashMap<>();
+ private final Map<MetricName, DistributionImpl> distributions = new HashMap<>();
+ private final Map<MetricName, GaugeImpl> gauges = new HashMap<>();
+
+ private final IMapJet<String, MetricUpdates> accumulator;
+
+ public JetMetricsContainer(String stepName, String ownerId, Processor.Context context) {
+ this.metricsKey = context.globalProcessorIndex() + "/" + stepName + "/" + ownerId;
+ this.stepName = stepName;
+ this.accumulator = context.jetInstance().getMap(getMetricsMapName(context.jobId()));
+ }
+
+ @Override
+ public Counter getCounter(MetricName metricName) {
+ return counters.computeIfAbsent(metricName, CounterImpl::new);
+ }
+
+ @Override
+ public Distribution getDistribution(MetricName metricName) {
+ return distributions.computeIfAbsent(metricName, DistributionImpl::new);
+ }
+
+ @Override
+ public Gauge getGauge(MetricName metricName) {
+ return gauges.computeIfAbsent(metricName, GaugeImpl::new);
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ public void flush(boolean async) {
+ if (counters.isEmpty() && distributions.isEmpty() && gauges.isEmpty()) {
+ return;
+ }
+
+ ImmutableList<MetricUpdates.MetricUpdate<Long>> counters = extractUpdates(this.counters);
+ ImmutableList<MetricUpdates.MetricUpdate<DistributionData>> distributions =
+ extractUpdates(this.distributions);
+ ImmutableList<MetricUpdates.MetricUpdate<GaugeData>> gauges = extractUpdates(this.gauges);
+ MetricUpdates updates = new MetricUpdatesImpl(counters, distributions, gauges);
+
+ if (async) {
+ accumulator.setAsync(metricsKey, updates);
+ } else {
+ accumulator.set(metricsKey, updates);
+ }
+ }
+
+ private <UpdateT, CellT extends AbstractMetric<UpdateT>>
+ ImmutableList<MetricUpdates.MetricUpdate<UpdateT>> extractUpdates(
+ Map<MetricName, CellT> cells) {
+ ImmutableList.Builder<MetricUpdates.MetricUpdate<UpdateT>> updates = ImmutableList.builder();
+ for (CellT cell : cells.values()) {
+ UpdateT value = cell.getValue();
+ if (value != null) {
+ MetricKey key = MetricKey.create(stepName, cell.getName());
+ MetricUpdates.MetricUpdate<UpdateT> update = MetricUpdates.MetricUpdate.create(key, value);
+ updates.add(update);
+ }
+ }
+ return updates.build();
+ }
+
+ private static class MetricUpdatesImpl extends MetricUpdates implements Serializable {
+
+ private final Iterable<MetricUpdate<Long>> counters;
+ private final Iterable<MetricUpdate<DistributionData>> distributions;
+ private final Iterable<MetricUpdate<GaugeData>> gauges;
+
+ MetricUpdatesImpl(
+ Iterable<MetricUpdate<Long>> counters,
+ Iterable<MetricUpdate<DistributionData>> distributions,
+ Iterable<MetricUpdate<GaugeData>> gauges) {
+ this.counters = counters;
+ this.distributions = distributions;
+ this.gauges = gauges;
+ }
+
+ @Override
+ public Iterable<MetricUpdate<Long>> counterUpdates() {
+ return counters;
+ }
+
+ @Override
+ public Iterable<MetricUpdate<DistributionData>> distributionUpdates() {
+ return distributions;
+ }
+
+ @Override
+ public Iterable<MetricUpdate<GaugeData>> gaugeUpdates() {
+ return gauges;
+ }
+ }
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/package-info.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/package-info.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/package-info.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/package-info.java
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
new file mode 100644
index 0000000..5287149
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
@@ -0,0 +1,524 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.Edge;
+import com.hazelcast.jet.core.Inbox;
+import com.hazelcast.jet.core.Outbox;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.Watermark;
+import com.hazelcast.jet.function.SupplierEx;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import javax.annotation.CheckReturnValue;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.core.DoFnRunner;
+import org.apache.beam.runners.core.DoFnRunners;
+import org.apache.beam.runners.core.InMemoryStateInternals;
+import org.apache.beam.runners.core.NullSideInputReader;
+import org.apache.beam.runners.core.SideInputHandler;
+import org.apache.beam.runners.core.SideInputReader;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.jet.DAGBuilder;
+import org.apache.beam.runners.jet.JetPipelineOptions;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.runners.jet.metrics.JetMetricsContainer;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.metrics.MetricsEnvironment;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
+import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
+import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+abstract class AbstractParDoP<InputT, OutputT> implements Processor {
+
+ private final SerializablePipelineOptions pipelineOptions;
+ private final DoFn<InputT, OutputT> doFn;
+ private final WindowingStrategy<?, ?> windowingStrategy;
+ private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<TupleTag<?>, int[]> outputCollToOrdinals;
+ private final TupleTag<OutputT> mainOutputTag;
+ private final Coder<InputT> inputCoder;
+ private final Map<PCollectionView<?>, Coder<?>> sideInputCoders;
+ private final Map<TupleTag<?>, Coder<?>> outputCoders;
+ private final Coder<InputT> inputValueCoder;
+ private final Map<TupleTag<?>, Coder<?>> outputValueCoders;
+ private final Map<Integer, PCollectionView<?>> ordinalToSideInput;
+ private final String ownerId;
+ private final String stepId;
+ private final boolean cooperative;
+ private final long metricsFlushPeriod =
+ TimeUnit.SECONDS.toMillis(1) + ThreadLocalRandom.current().nextLong(500);
+
+ DoFnRunner<InputT, OutputT> doFnRunner;
+ JetOutputManager outputManager;
+
+ private DoFnInvoker<InputT, OutputT> doFnInvoker;
+ private SideInputHandler sideInputHandler;
+ private JetMetricsContainer metricsContainer;
+ private SimpleInbox bufferedItems;
+ private Set<Integer> completedSideInputs = new HashSet<>();
+ private SideInputReader sideInputReader;
+ private Outbox outbox;
+ private long lastMetricsFlushTime = System.currentTimeMillis();
+ private Map<String, PCollectionView<?>> sideInputMapping;
+
+ AbstractParDoP(
+ DoFn<InputT, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Coder<InputT> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ Map<Integer, PCollectionView<?>> ordinalToSideInput,
+ String ownerId,
+ String stepId) {
+ this.pipelineOptions = pipelineOptions;
+ this.doFn = Utils.serde(doFn);
+ this.windowingStrategy = windowingStrategy;
+ this.doFnSchemaInformation = doFnSchemaInformation;
+ this.outputCollToOrdinals = outputCollToOrdinals;
+ this.mainOutputTag = mainOutputTag;
+ this.inputCoder = inputCoder;
+ this.sideInputCoders =
+ sideInputCoders.entrySet().stream()
+ .collect(
+ Collectors.toMap(
+ Map.Entry::getKey,
+ e ->
+ Utils.deriveIterableValueCoder(
+ (WindowedValue.FullWindowedValueCoder) e.getValue())));
+ this.outputCoders = outputCoders;
+ this.inputValueCoder = inputValueCoder;
+ this.outputValueCoders = outputValueCoders;
+ this.ordinalToSideInput = ordinalToSideInput;
+ this.ownerId = ownerId;
+ this.stepId = stepId;
+ this.cooperative = isCooperativenessAllowed(pipelineOptions) && hasOutput();
+ }
+
+ @Override
+ public void init(@Nonnull Outbox outbox, @Nonnull Context context) {
+ this.outbox = outbox;
+ this.metricsContainer = new JetMetricsContainer(stepId, ownerId, context);
+
+ doFnInvoker = DoFnInvokers.invokerFor(doFn);
+ doFnInvoker.invokeSetup();
+
+ if (ordinalToSideInput.isEmpty()) {
+ sideInputReader = NullSideInputReader.of(Collections.emptyList());
+ } else {
+ bufferedItems = new SimpleInbox();
+ sideInputHandler =
+ new SideInputHandler(ordinalToSideInput.values(), InMemoryStateInternals.forKey(null));
+ sideInputReader = sideInputHandler;
+ }
+
+ outputManager = new JetOutputManager(outbox, outputCoders, outputCollToOrdinals);
+
+ doFnRunner =
+ getDoFnRunner(
+ pipelineOptions.get(),
+ doFn,
+ sideInputReader,
+ outputManager,
+ mainOutputTag,
+ Lists.newArrayList(outputCollToOrdinals.keySet()),
+ inputValueCoder,
+ outputValueCoders,
+ windowingStrategy,
+ doFnSchemaInformation,
+ sideInputMapping);
+ }
+
+ protected abstract DoFnRunner<InputT, OutputT> getDoFnRunner(
+ PipelineOptions pipelineOptions,
+ DoFn<InputT, OutputT> doFn,
+ SideInputReader sideInputReader,
+ JetOutputManager outputManager,
+ TupleTag<OutputT> mainOutputTag,
+ List<TupleTag<?>> additionalOutputTags,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping);
+
+ @Override
+ public boolean isCooperative() {
+ return cooperative;
+ }
+
+ @Override
+ public void close() {
+ doFnInvoker.invokeTeardown();
+ }
+
+ @Override
+ public void process(int ordinal, @Nonnull Inbox inbox) {
+ MetricsEnvironment.setCurrentContainer(metricsContainer);
+
+ if (!outputManager.tryFlush()) {
+ // don't process more items until outputManager is empty
+ return;
+ }
+ PCollectionView<?> sideInputView = ordinalToSideInput.get(ordinal);
+ if (sideInputView != null) {
+ processSideInput(sideInputView, inbox);
+ } else {
+ if (bufferedItems != null) {
+ processBufferedRegularItems(inbox);
+ } else {
+ processNonBufferedRegularItems(inbox);
+ }
+ }
+
+ MetricsEnvironment.setCurrentContainer(null);
+ }
+
+ private void processSideInput(PCollectionView<?> sideInputView, Inbox inbox) {
+ for (byte[] value; (value = (byte[]) inbox.poll()) != null; ) {
+ Coder<?> sideInputCoder = sideInputCoders.get(sideInputView);
+ WindowedValue<Iterable<?>> windowedValue = Utils.decodeWindowedValue(value, sideInputCoder);
+ sideInputHandler.addSideInputValue(sideInputView, windowedValue);
+ }
+ }
+
+ private void processNonBufferedRegularItems(Inbox inbox) {
+ startRunnerBundle(doFnRunner);
+ for (byte[] value; (value = (byte[]) inbox.poll()) != null; ) {
+ WindowedValue<InputT> windowedValue = Utils.decodeWindowedValue(value, inputCoder);
+ processElementWithRunner(doFnRunner, windowedValue);
+ if (!outputManager.tryFlush()) {
+ break;
+ }
+ }
+ finishRunnerBundle(doFnRunner);
+ // finishBundle can also add items to outputManager, they will be flushed in tryProcess() or
+ // complete()
+ }
+
+ protected void startRunnerBundle(DoFnRunner<InputT, OutputT> runner) {
+ runner.startBundle();
+ }
+
+ protected void processElementWithRunner(
+ DoFnRunner<InputT, OutputT> runner, WindowedValue<InputT> windowedValue) {
+ runner.processElement(windowedValue);
+ }
+
+ protected void finishRunnerBundle(DoFnRunner<InputT, OutputT> runner) {
+ runner.finishBundle();
+ }
+
+ private void processBufferedRegularItems(Inbox inbox) {
+ for (byte[] value; (value = (byte[]) inbox.poll()) != null; ) {
+ bufferedItems.add(value);
+ }
+ }
+
+ @Override
+ public boolean tryProcess() {
+ boolean successful = outputManager.tryFlush();
+ if (successful && System.currentTimeMillis() > lastMetricsFlushTime + metricsFlushPeriod) {
+ metricsContainer.flush(true);
+ lastMetricsFlushTime = System.currentTimeMillis();
+ }
+ return successful;
+ }
+
+ @Override
+ public boolean tryProcessWatermark(@Nonnull Watermark watermark) {
+ return outbox.offer(watermark);
+ }
+
+ @Override
+ public boolean completeEdge(int ordinal) {
+ if (ordinalToSideInput.get(ordinal) == null) {
+ return true; // ignore non-side-input edges
+ }
+ completedSideInputs.add(ordinal);
+ if (completedSideInputs.size() != ordinalToSideInput.size()) {
+ // there are more side inputs to complete
+ return true;
+ }
+ processNonBufferedRegularItems(bufferedItems);
+ if (bufferedItems.isEmpty()) {
+ bufferedItems = null;
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public boolean complete() {
+ boolean successful = outputManager.tryFlush();
+ if (successful) {
+ metricsContainer.flush(false);
+ }
+ return successful;
+ }
+
+ private boolean hasOutput() {
+ for (int[] value : outputCollToOrdinals.values()) {
+ if (value.length > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static Boolean isCooperativenessAllowed(
+ SerializablePipelineOptions serializablePipelineOptions) {
+ PipelineOptions pipelineOptions = serializablePipelineOptions.get();
+ JetPipelineOptions jetPipelineOptions = pipelineOptions.as(JetPipelineOptions.class);
+ return jetPipelineOptions.getJetProcessorsCooperative();
+ }
+
+ /**
+ * An output manager that stores the output in an ArrayList, one for each output ordinal, and a
+ * way to drain to outbox ({@link #tryFlush()}).
+ */
+ static class JetOutputManager implements DoFnRunners.OutputManager {
+
+ private final Outbox outbox;
+ private final Map<TupleTag<?>, Coder<?>> outputCoders;
+ private final Map<TupleTag<?>, int[]> outputCollToOrdinals;
+ private final List<Object>[] outputBuckets;
+
+ // the flush position to continue flushing to outbox
+ private int currentBucket, currentItem;
+
+ @SuppressWarnings("unchecked")
+ JetOutputManager(
+ Outbox outbox,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals) {
+ this.outbox = outbox;
+ this.outputCoders = outputCoders;
+ this.outputCollToOrdinals = outputCollToOrdinals;
+ assert !outputCollToOrdinals.isEmpty();
+ int maxOrdinal =
+ outputCollToOrdinals.values().stream().flatMapToInt(IntStream::of).max().orElse(-1);
+ outputBuckets = new List[maxOrdinal + 1];
+ Arrays.setAll(outputBuckets, i -> new ArrayList<>());
+ }
+
+ @Override
+ public <T> void output(TupleTag<T> tag, WindowedValue<T> outputValue) {
+ assert currentBucket == 0 && currentItem == 0 : "adding output while flushing";
+ Coder coder = outputCoders.get(tag);
+ byte[] output = Utils.encode(outputValue, coder);
+ for (int ordinal : outputCollToOrdinals.get(tag)) {
+ outputBuckets[ordinal].add(output);
+ }
+ }
+
+ @CheckReturnValue
+ boolean tryFlush() {
+ for (; currentBucket < outputBuckets.length; currentBucket++) {
+ List<Object> bucket = outputBuckets[currentBucket];
+ for (; currentItem < bucket.size(); currentItem++) {
+ if (!outbox.offer(currentBucket, bucket.get(currentItem))) {
+ return false;
+ }
+ }
+ bucket.clear();
+ currentItem = 0;
+ }
+ currentBucket = 0;
+ int sum = 0;
+ for (List<Object> outputBucket : outputBuckets) {
+ sum += outputBucket.size();
+ }
+ return sum == 0;
+ }
+ }
+
+ abstract static class AbstractSupplier<InputT, OutputT>
+ implements SupplierEx<Processor>, DAGBuilder.WiringListener {
+
+ protected final String ownerId;
+ private final String stepId;
+
+ private final SerializablePipelineOptions pipelineOptions;
+ private final DoFn<InputT, OutputT> doFn;
+ private final WindowingStrategy<?, ?> windowingStrategy;
+ private final DoFnSchemaInformation doFnSchemaInformation;
+ private final TupleTag<OutputT> mainOutputTag;
+ private final Map<TupleTag<?>, List<Integer>> outputCollToOrdinals;
+ private final Coder<InputT> inputCoder;
+ private final Map<PCollectionView<?>, Coder<?>> sideInputCoders;
+ private final Map<TupleTag<?>, Coder<?>> outputCoders;
+ private final Coder<InputT> inputValueCoder;
+ private final Map<TupleTag<?>, Coder<?>> outputValueCoders;
+ private final List<PCollectionView<?>> sideInputs;
+
+ private final Map<Integer, PCollectionView<?>> ordinalToSideInput = new HashMap<>();
+
+ AbstractSupplier(
+ String stepId,
+ String ownerId,
+ DoFn<InputT, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Set<TupleTag<OutputT>> allOutputTags,
+ Coder<InputT> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ List<PCollectionView<?>> sideInputs) {
+ this.stepId = stepId;
+ this.ownerId = ownerId;
+ this.pipelineOptions = pipelineOptions;
+ this.doFn = doFn;
+ this.windowingStrategy = windowingStrategy;
+ this.doFnSchemaInformation = doFnSchemaInformation;
+ this.outputCollToOrdinals =
+ allOutputTags.stream()
+ .collect(Collectors.toMap(Function.identity(), t -> new ArrayList<>()));
+ this.mainOutputTag = mainOutputTag;
+ this.inputCoder = inputCoder;
+ this.sideInputCoders = sideInputCoders;
+ this.outputCoders = outputCoders;
+ this.inputValueCoder = inputValueCoder;
+ this.outputValueCoders = outputValueCoders;
+ this.sideInputs = sideInputs;
+ }
+
+ @Override
+ public Processor getEx() {
+ if (ordinalToSideInput.size() != sideInputs.size()) {
+ throw new RuntimeException("Oops");
+ }
+ return getEx(
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ outputCollToOrdinals.entrySet().stream()
+ .collect(
+ Collectors.toMap(
+ Map.Entry::getKey, e -> e.getValue().stream().mapToInt(i -> i).toArray())),
+ pipelineOptions,
+ mainOutputTag,
+ inputCoder,
+ Collections.unmodifiableMap(sideInputCoders),
+ Collections.unmodifiableMap(outputCoders),
+ inputValueCoder,
+ Collections.unmodifiableMap(outputValueCoders),
+ Collections.unmodifiableMap(ordinalToSideInput),
+ ownerId,
+ stepId);
+ }
+
+ abstract Processor getEx(
+ DoFn<InputT, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Coder<InputT> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ Map<Integer, PCollectionView<?>> ordinalToSideInput,
+ String ownerId,
+ String stepId);
+
+ @Override
+ public void isOutboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
+ if (ownerId.equals(vertexId)) {
+ List<Integer> ordinals = outputCollToOrdinals.get(new TupleTag<>(pCollId));
+ if (ordinals == null) {
+ throw new RuntimeException("Oops"); // todo
+ }
+
+ ordinals.add(edge.getSourceOrdinal());
+ }
+ }
+
+ @Override
+ public void isInboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
+ if (ownerId.equals(vertexId)) {
+ for (PCollectionView<?> pCollectionView : sideInputs) {
+ if (edgeId.equals(Utils.getTupleTagId(pCollectionView))) {
+ ordinalToSideInput.put(edge.getDestOrdinal(), pCollectionView);
+ break;
+ }
+ }
+ }
+ }
+ }
+
+ private static class SimpleInbox implements Inbox {
+ private Deque<Object> items = new ArrayDeque<>();
+
+ void add(Object item) {
+ items.add(item);
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return items.isEmpty();
+ }
+
+ @Override
+ public Object peek() {
+ return items.peek();
+ }
+
+ @Override
+ public Object poll() {
+ return items.poll();
+ }
+
+ @Override
+ public void remove() {
+ items.remove();
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
new file mode 100644
index 0000000..08e42be
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.ResettableSingletonTraverser;
+import com.hazelcast.jet.function.SupplierEx;
+import java.util.Collection;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.joda.time.Instant;
+
+/**
+ * /** * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's Windowing primitive.
+ *
+ * @param <T> type of element being windowed
+ */
+public class AssignWindowP<T> extends AbstractProcessor {
+
+ @SuppressWarnings({"FieldCanBeLocal", "unused"})
+ private final String ownerId; // do not remove, useful for debugging
+
+ private final ResettableSingletonTraverser<byte[]> traverser =
+ new ResettableSingletonTraverser<>();
+ private final FlatMapper<byte[], byte[]> flatMapper;
+ private final WindowAssignContext<T> windowAssignContext;
+
+ private AssignWindowP(
+ Coder inputCoder,
+ Coder outputCoder,
+ WindowingStrategy<T, BoundedWindow> windowingStrategy,
+ String ownerId) {
+ this.ownerId = ownerId;
+
+ windowAssignContext = new WindowAssignContext<>(windowingStrategy.getWindowFn());
+
+ flatMapper =
+ flatMapper(
+ item -> {
+ Collection<BoundedWindow> windows;
+ WindowedValue<T> inputValue = Utils.decodeWindowedValue(item, inputCoder);
+ windowAssignContext.setValue(inputValue);
+ try {
+ windows = windowingStrategy.getWindowFn().assignWindows(windowAssignContext);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ WindowedValue<T> outputValue =
+ WindowedValue.of(
+ inputValue.getValue(),
+ inputValue.getTimestamp(),
+ windows,
+ inputValue.getPane());
+ traverser.accept(Utils.encode(outputValue, outputCoder));
+ return traverser;
+ });
+ }
+
+ public static <T> SupplierEx<Processor> supplier(
+ Coder inputCoder,
+ Coder outputCoder,
+ WindowingStrategy<T, BoundedWindow> windowingStrategy,
+ String ownerId) {
+ return () -> new AssignWindowP<>(inputCoder, outputCoder, windowingStrategy, ownerId);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Override
+ protected boolean tryProcess(int ordinal, @Nonnull Object item) {
+ return flatMapper.tryProcess((byte[]) item);
+ }
+
+ private static class WindowAssignContext<InputT>
+ extends WindowFn<InputT, BoundedWindow>.AssignContext {
+ private WindowedValue<InputT> value;
+
+ WindowAssignContext(WindowFn<InputT, BoundedWindow> fn) {
+ fn.super();
+ }
+
+ public void setValue(WindowedValue<InputT> value) {
+ if (Iterables.size(value.getWindows()) != 1) {
+ throw new IllegalArgumentException(
+ String.format(
+ "%s passed to window assignment must be in a single window, but it was in %s: %s",
+ WindowedValue.class.getSimpleName(),
+ Iterables.size(value.getWindows()),
+ value.getWindows()));
+ }
+ this.value = value;
+ }
+
+ @Override
+ public InputT element() {
+ return value.getValue();
+ }
+
+ @Override
+ public Instant timestamp() {
+ return value.getTimestamp();
+ }
+
+ @Override
+ public BoundedWindow window() {
+ return Iterables.getOnlyElement(value.getWindows());
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
new file mode 100644
index 0000000..5b3417630
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.Traversers;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.jet.impl.util.ExceptionUtil;
+import com.hazelcast.nio.Address;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for reading from a bounded Beam
+ * source.
+ */
+public class BoundedSourceP<T> extends AbstractProcessor implements Traverser {
+
+ private final Traverser<BoundedSource<T>> shardsTraverser;
+ private final PipelineOptions options;
+ private final Coder outputCoder;
+
+ @SuppressWarnings({"FieldCanBeLocal", "unused"})
+ private final String ownerId; // do not remove it, very useful for debugging
+
+ private BoundedSource.BoundedReader currentReader;
+
+ BoundedSourceP(
+ List<BoundedSource<T>> shards, PipelineOptions options, Coder outputCoder, String ownerId) {
+ this.shardsTraverser = Traversers.traverseIterable(shards);
+ this.options = options;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ protected void init(@Nonnull Processor.Context context) throws Exception {
+ nextShard();
+ }
+
+ @Override
+ public Object next() {
+ if (currentReader == null) {
+ return null;
+ }
+ try {
+ Object item = currentReader.getCurrent();
+ WindowedValue<Object> res =
+ WindowedValue.timestampedValueInGlobalWindow(item, currentReader.getCurrentTimestamp());
+ if (!currentReader.advance()) {
+ nextShard();
+ }
+ return outputCoder == null
+ ? res
+ : Utils.encode(
+ res, outputCoder); // todo: this is not nice, have done this only as a quick fix for
+ // BoundedSourcePTest
+ } catch (IOException e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+
+ /**
+ * Called when currentReader is null or drained. At the end it will contain a started reader of
+ * the next shard or null.
+ */
+ private void nextShard() throws IOException {
+ for (; ; ) {
+ if (currentReader != null) {
+ currentReader.close();
+ currentReader = null;
+ }
+ BoundedSource<T> shard = shardsTraverser.next();
+ if (shard == null) {
+ break; // all shards done
+ }
+ currentReader = shard.createReader(options);
+ if (currentReader.start()) {
+ break;
+ }
+ }
+ }
+
+ @Override
+ public boolean complete() {
+ return emitFromTraverser(this);
+ }
+
+ @Override
+ public boolean isCooperative() {
+ return false;
+ }
+
+ @Override
+ public void close() throws Exception {
+ if (currentReader != null) {
+ currentReader.close();
+ }
+ }
+
+ public static <T> ProcessorMetaSupplier supplier(
+ BoundedSource<T> boundedSource,
+ SerializablePipelineOptions options,
+ Coder outputCoder,
+ String ownerId) {
+ return new BoundedSourceMetaProcessorSupplier<>(boundedSource, options, outputCoder, ownerId);
+ }
+
+ private static class BoundedSourceMetaProcessorSupplier<T> implements ProcessorMetaSupplier {
+
+ private final BoundedSource<T> boundedSource;
+ private final SerializablePipelineOptions options;
+ private final Coder outputCoder;
+ private final String ownerId;
+
+ private transient List<? extends BoundedSource<T>> shards;
+
+ private BoundedSourceMetaProcessorSupplier(
+ BoundedSource<T> boundedSource,
+ SerializablePipelineOptions options,
+ Coder outputCoder,
+ String ownerId) {
+ this.boundedSource = boundedSource;
+ this.options = options;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ public void init(@Nonnull ProcessorMetaSupplier.Context context) throws Exception {
+ long desiredSizeBytes =
+ Math.max(
+ 1, boundedSource.getEstimatedSizeBytes(options.get()) / context.totalParallelism());
+ shards = boundedSource.split(desiredSizeBytes, options.get());
+ }
+
+ @SuppressWarnings("unchecked")
+ @Nonnull
+ @Override
+ public Function<? super Address, ? extends ProcessorSupplier> get(
+ @Nonnull List<Address> addresses) {
+ return address ->
+ new BoundedSourceProcessorSupplier(
+ Utils.roundRobinSubList(shards, addresses.indexOf(address), addresses.size()),
+ options,
+ outputCoder,
+ ownerId);
+ }
+ }
+
+ private static class BoundedSourceProcessorSupplier<T> implements ProcessorSupplier {
+ private final List<BoundedSource<T>> shards;
+ private final SerializablePipelineOptions options;
+ private final Coder outputCoder;
+ private final String ownerId;
+ private transient ProcessorSupplier.Context context;
+
+ private BoundedSourceProcessorSupplier(
+ List<BoundedSource<T>> shards,
+ SerializablePipelineOptions options,
+ Coder outputCoder,
+ String ownerId) {
+ this.shards = shards;
+ this.options = options;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ public void init(@Nonnull Context context) {
+ this.context = context;
+ }
+
+ @Nonnull
+ @Override
+ public Collection<? extends Processor> get(int count) {
+ int indexBase = context.memberIndex() * context.localParallelism();
+ List<Processor> res = new ArrayList<>(count);
+ for (int i = 0; i < count; i++, indexBase++) {
+ res.add(
+ new BoundedSourceP<>(
+ Utils.roundRobinSubList(shards, i, count), options.get(), outputCoder, ownerId));
+ }
+ return res;
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
new file mode 100644
index 0000000..c57b939
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.Edge;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.function.SupplierEx;
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.jet.DAGBuilder;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's Flatten primitive. */
+public class FlattenP extends AbstractProcessor {
+
+ private final Map<Integer, Coder> inputOrdinalCoders;
+ private final Coder outputCoder;
+
+ @SuppressWarnings("FieldCanBeLocal") // do not remove, useful for debugging
+ private final String ownerId;
+
+ private FlattenP(Map<Integer, Coder> inputOrdinalCoders, Coder outputCoder, String ownerId) {
+ this.inputOrdinalCoders = inputOrdinalCoders;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ protected boolean tryProcess(int ordinal, @Nonnull Object item) {
+ Coder inputCoder = inputOrdinalCoders.get(ordinal);
+ WindowedValue<Object> windowedValue = Utils.decodeWindowedValue((byte[]) item, inputCoder);
+ return tryEmit(Utils.encode(windowedValue, outputCoder));
+ }
+
+ /** Jet {@link Processor} supplier that will provide instances of {@link FlattenP}. */
+ public static final class Supplier implements SupplierEx<Processor>, DAGBuilder.WiringListener {
+
+ private final Map<String, Coder> inputCollectionCoders;
+ private final Coder outputCoder;
+ private final String ownerId;
+ private final Map<Integer, Coder> inputOrdinalCoders;
+
+ public Supplier(Map<String, Coder> inputCoders, Coder outputCoder, String ownerId) {
+ this.inputCollectionCoders = inputCoders;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ this.inputOrdinalCoders = new HashMap<>();
+ }
+
+ @Override
+ public Processor getEx() {
+ return new FlattenP(inputOrdinalCoders, outputCoder, ownerId);
+ }
+
+ @Override
+ public void isOutboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
+ // do nothing
+ }
+
+ @Override
+ public void isInboundEdgeOfVertex(Edge edge, String edgeId, String pCollId, String vertexId) {
+ if (ownerId.equals(vertexId)) {
+ Coder coder = inputCollectionCoders.get(edgeId);
+ inputOrdinalCoders.put(edge.getDestOrdinal(), coder);
+ }
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
new file mode 100644
index 0000000..70dd480
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.nio.Address;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/**
+ * /** * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's Impulse primitive.
+ */
+public class ImpulseP extends AbstractProcessor {
+
+ private final boolean active;
+ private final Coder outputCoder;
+ private final String ownerId; // do not remove it, very useful for debugging
+
+ private ImpulseP(boolean active, Coder outputCoder, String ownerId) {
+ this.active = active;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ public boolean complete() {
+ if (active) {
+ return tryEmit(Utils.encode(WindowedValue.valueInGlobalWindow(new byte[0]), outputCoder));
+ } else {
+ return true;
+ }
+ }
+
+ public static ProcessorMetaSupplier supplier(Coder outputCoder, String ownerId) {
+ return new ImpulseMetaProcessorSupplier(outputCoder, ownerId);
+ }
+
+ private static class ImpulseMetaProcessorSupplier implements ProcessorMetaSupplier {
+
+ private final Coder outputCoder;
+ private final String ownerId;
+
+ private ImpulseMetaProcessorSupplier(Coder outputCoder, String ownerId) {
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @SuppressWarnings("unchecked")
+ @Nonnull
+ @Override
+ public Function<? super Address, ? extends ProcessorSupplier> get(
+ @Nonnull List<Address> addresses) {
+ return address -> new ImpulseProcessorSupplier(outputCoder, ownerId);
+ }
+ }
+
+ private static class ImpulseProcessorSupplier<T> implements ProcessorSupplier {
+ private final Coder outputCoder;
+ private final String ownerId;
+ private transient ProcessorSupplier.Context context;
+
+ private ImpulseProcessorSupplier(Coder outputCoder, String ownerId) {
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ public void init(@Nonnull Context context) {
+ this.context = context;
+ }
+
+ @Nonnull
+ @Override
+ public Collection<? extends Processor> get(int count) {
+ int indexBase = context.memberIndex() * context.localParallelism();
+ List<Processor> res = new ArrayList<>(count);
+ for (int i = 0; i < count; i++, indexBase++) {
+ res.add(new ImpulseP(indexBase == 0, outputCoder, ownerId));
+ }
+ return res;
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
new file mode 100644
index 0000000..38f07c9
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.Processor;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.beam.runners.core.DoFnRunner;
+import org.apache.beam.runners.core.DoFnRunners;
+import org.apache.beam.runners.core.SideInputReader;
+import org.apache.beam.runners.core.StateInternals;
+import org.apache.beam.runners.core.StepContext;
+import org.apache.beam.runners.core.TimerInternals;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's ParDo primitive (when no
+ * user-state is being used).
+ */
+public class ParDoP<InputT, OutputT>
+ extends AbstractParDoP<InputT, OutputT> { // todo: unify with StatefulParDoP?
+
+ private ParDoP(
+ DoFn<InputT, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Coder<InputT> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ Map<Integer, PCollectionView<?>> ordinalToSideInput,
+ String ownerId,
+ String stepId) {
+ super(
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ outputCollToOrdinals,
+ pipelineOptions,
+ mainOutputTag,
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ ordinalToSideInput,
+ ownerId,
+ stepId);
+ }
+
+ @Override
+ protected DoFnRunner<InputT, OutputT> getDoFnRunner(
+ PipelineOptions pipelineOptions,
+ DoFn<InputT, OutputT> doFn,
+ SideInputReader sideInputReader,
+ JetOutputManager outputManager,
+ TupleTag<OutputT> mainOutputTag,
+ List<TupleTag<?>> additionalOutputTags,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
+ return DoFnRunners.simpleRunner(
+ pipelineOptions,
+ doFn,
+ sideInputReader,
+ outputManager,
+ mainOutputTag,
+ additionalOutputTags,
+ new NotImplementedStepContext(),
+ inputValueCoder,
+ outputValueCoders,
+ windowingStrategy,
+ doFnSchemaInformation,
+ sideInputMapping);
+ }
+
+ /**
+ * Jet {@link Processor} supplier that will provide instances of {@link ParDoP}.
+ *
+ * @param <OutputT> the type of main output elements of the DoFn being used
+ */
+ public static class Supplier<InputT, OutputT> extends AbstractSupplier<InputT, OutputT> {
+
+ public Supplier(
+ String stepId,
+ String ownerId,
+ DoFn<InputT, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Set<TupleTag<OutputT>> allOutputTags,
+ Coder<InputT> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ List<PCollectionView<?>> sideInputs) {
+ super(
+ stepId,
+ ownerId,
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ pipelineOptions,
+ mainOutputTag,
+ allOutputTags,
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ sideInputs);
+ }
+
+ @Override
+ Processor getEx(
+ DoFn<InputT, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Coder<InputT> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<InputT> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ Map<Integer, PCollectionView<?>> ordinalToSideInput,
+ String ownerId,
+ String stepId) {
+ return new ParDoP<>(
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ outputCollToOrdinals,
+ pipelineOptions,
+ mainOutputTag,
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ ordinalToSideInput,
+ ownerId,
+ stepId);
+ }
+ }
+
+ private static class NotImplementedStepContext implements StepContext {
+
+ // not needed when not handling state & timers
+
+ @Override
+ public StateInternals stateInternals() {
+ throw new UnsupportedOperationException("stateInternals is not supported");
+ }
+
+ @Override
+ public TimerInternals timerInternals() {
+ throw new UnsupportedOperationException("timerInternals is not supported");
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
new file mode 100644
index 0000000..e291117
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.Watermark;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.core.DoFnRunner;
+import org.apache.beam.runners.core.DoFnRunners;
+import org.apache.beam.runners.core.InMemoryStateInternals;
+import org.apache.beam.runners.core.InMemoryTimerInternals;
+import org.apache.beam.runners.core.SideInputReader;
+import org.apache.beam.runners.core.StateInternals;
+import org.apache.beam.runners.core.StateNamespace;
+import org.apache.beam.runners.core.StateNamespaces;
+import org.apache.beam.runners.core.StepContext;
+import org.apache.beam.runners.core.TimerInternals;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.joda.time.Instant;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's stateful ParDo primitive.
+ */
+public class StatefulParDoP<OutputT>
+ extends AbstractParDoP<KV<?, ?>, OutputT> { // todo: unify with ParDoP?
+
+ private KeyedStepContext keyedStepContext;
+ private InMemoryTimerInternals timerInternals;
+
+ private StatefulParDoP(
+ DoFn<KV<?, ?>, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Coder<KV<?, ?>> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<KV<?, ?>> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ Map<Integer, PCollectionView<?>> ordinalToSideInput,
+ String ownerId,
+ String stepId) {
+ super(
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ outputCollToOrdinals,
+ pipelineOptions,
+ mainOutputTag,
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ ordinalToSideInput,
+ ownerId,
+ stepId);
+ }
+
+ private static void fireTimer(
+ TimerInternals.TimerData timer, DoFnRunner<KV<?, ?>, ?> doFnRunner) {
+ StateNamespace namespace = timer.getNamespace();
+ BoundedWindow window = ((StateNamespaces.WindowNamespace) namespace).getWindow();
+ doFnRunner.onTimer(timer.getTimerId(), window, timer.getTimestamp(), timer.getDomain());
+ }
+
+ @Override
+ protected DoFnRunner<KV<?, ?>, OutputT> getDoFnRunner(
+ PipelineOptions pipelineOptions,
+ DoFn<KV<?, ?>, OutputT> doFn,
+ SideInputReader sideInputReader,
+ JetOutputManager outputManager,
+ TupleTag<OutputT> mainOutputTag,
+ List<TupleTag<?>> additionalOutputTags,
+ Coder<KV<?, ?>> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
+ timerInternals = new InMemoryTimerInternals();
+ keyedStepContext = new KeyedStepContext(timerInternals);
+ return DoFnRunners.simpleRunner(
+ pipelineOptions,
+ doFn,
+ sideInputReader,
+ outputManager,
+ mainOutputTag,
+ additionalOutputTags,
+ keyedStepContext,
+ inputValueCoder,
+ outputValueCoders,
+ windowingStrategy,
+ doFnSchemaInformation,
+ sideInputMapping);
+ }
+
+ @Override
+ protected void startRunnerBundle(DoFnRunner<KV<?, ?>, OutputT> runner) {
+ try {
+ Instant now = Instant.now();
+ timerInternals.advanceProcessingTime(now);
+ timerInternals.advanceSynchronizedProcessingTime(now);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed advancing time!");
+ }
+
+ super.startRunnerBundle(runner);
+ }
+
+ @Override
+ protected void processElementWithRunner(
+ DoFnRunner<KV<?, ?>, OutputT> runner, WindowedValue<KV<?, ?>> windowedValue) {
+ KV<?, ?> kv = windowedValue.getValue();
+ Object key = kv.getKey();
+ keyedStepContext.setKey(key);
+
+ super.processElementWithRunner(runner, windowedValue);
+ }
+
+ @Override
+ public boolean tryProcessWatermark(@Nonnull Watermark watermark) {
+ return flushTimers(watermark.timestamp()) && super.tryProcessWatermark(watermark);
+ }
+
+ @Override
+ public boolean complete() {
+ return flushTimers(BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis()) && super.complete();
+ }
+
+ private boolean flushTimers(long watermark) {
+ if (timerInternals.currentInputWatermarkTime().isBefore(watermark)) {
+ try {
+ Instant watermarkInstant = new Instant(watermark);
+ timerInternals.advanceInputWatermark(watermarkInstant);
+ if (watermarkInstant.equals(BoundedWindow.TIMESTAMP_MAX_VALUE)) {
+ timerInternals.advanceProcessingTime(watermarkInstant);
+ timerInternals.advanceSynchronizedProcessingTime(watermarkInstant);
+ }
+ fireEligibleTimers(timerInternals);
+ } catch (Exception e) {
+ throw new RuntimeException("Failed advancing processing time", e);
+ }
+ }
+ return outputManager.tryFlush();
+ }
+
+ private void fireEligibleTimers(InMemoryTimerInternals timerInternals) {
+ while (true) {
+ TimerInternals.TimerData timer;
+ boolean hasFired = false;
+
+ while ((timer = timerInternals.removeNextEventTimer()) != null) {
+ hasFired = true;
+ fireTimer(timer, doFnRunner);
+ }
+
+ while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
+ hasFired = true;
+ fireTimer(timer, doFnRunner);
+ }
+
+ while ((timer = timerInternals.removeNextSynchronizedProcessingTimer()) != null) {
+ hasFired = true;
+ fireTimer(timer, doFnRunner);
+ }
+
+ if (!hasFired) {
+ break;
+ }
+ }
+ }
+
+ /**
+ * Jet {@link Processor} supplier that will provide instances of {@link StatefulParDoP}.
+ *
+ * @param <OutputT> the type of main output elements of the DoFn being used
+ */
+ public static class Supplier<OutputT> extends AbstractSupplier<KV<?, ?>, OutputT> {
+
+ public Supplier(
+ String stepId,
+ String ownerId,
+ DoFn<KV<?, ?>, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Set<TupleTag<OutputT>> allOutputTags,
+ Coder<KV<?, ?>> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<KV<?, ?>> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ List<PCollectionView<?>> sideInputs) {
+ super(
+ stepId,
+ ownerId,
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ pipelineOptions,
+ mainOutputTag,
+ allOutputTags,
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ sideInputs);
+ }
+
+ @Override
+ Processor getEx(
+ DoFn<KV<?, ?>, OutputT> doFn,
+ WindowingStrategy<?, ?> windowingStrategy,
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<TupleTag<?>, int[]> outputCollToOrdinals,
+ SerializablePipelineOptions pipelineOptions,
+ TupleTag<OutputT> mainOutputTag,
+ Coder<KV<?, ?>> inputCoder,
+ Map<PCollectionView<?>, Coder<?>> sideInputCoders,
+ Map<TupleTag<?>, Coder<?>> outputCoders,
+ Coder<KV<?, ?>> inputValueCoder,
+ Map<TupleTag<?>, Coder<?>> outputValueCoders,
+ Map<Integer, PCollectionView<?>> ordinalToSideInput,
+ String ownerId,
+ String stepId) {
+ return new StatefulParDoP<>(
+ doFn,
+ windowingStrategy,
+ doFnSchemaInformation,
+ outputCollToOrdinals,
+ pipelineOptions,
+ mainOutputTag,
+ inputCoder,
+ sideInputCoders,
+ outputCoders,
+ inputValueCoder,
+ outputValueCoders,
+ ordinalToSideInput,
+ ownerId,
+ stepId);
+ }
+ }
+
+ private static class KeyedStepContext implements StepContext {
+
+ private final Map<Object, InMemoryStateInternals> stateInternalsOfKeys;
+ private final InMemoryTimerInternals timerInternals;
+
+ private InMemoryStateInternals currentStateInternals;
+
+ KeyedStepContext(InMemoryTimerInternals timerInternals) {
+ this.stateInternalsOfKeys = new HashMap<>();
+ this.timerInternals = timerInternals;
+ }
+
+ void setKey(Object key) {
+ currentStateInternals =
+ stateInternalsOfKeys.computeIfAbsent(key, InMemoryStateInternals::forKey);
+ }
+
+ @Override
+ public StateInternals stateInternals() {
+ return currentStateInternals;
+ }
+
+ @Override
+ public TimerInternals timerInternals() {
+ return timerInternals;
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java
new file mode 100644
index 0000000..a387fad
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.Traversers;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.jet.core.Watermark;
+import com.hazelcast.jet.impl.util.ExceptionUtil;
+import com.hazelcast.nio.Address;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.function.Function;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for reading from an unbounded Beam
+ * source.
+ */
+public class UnboundedSourceP<T, CmT extends UnboundedSource.CheckpointMark>
+ extends AbstractProcessor {
+
+ private UnboundedSource.UnboundedReader<T>[] readers;
+ private final List<? extends UnboundedSource<T, CmT>> allShards;
+ private final PipelineOptions options;
+ private final Coder outputCoder;
+
+ @SuppressWarnings({"FieldCanBeLocal", "unused"})
+ private final String ownerId; // do not remove it, very useful for debugging
+
+ private Traverser<Object> traverser;
+
+ private UnboundedSourceP(
+ List<? extends UnboundedSource<T, CmT>> allShards,
+ PipelineOptions options,
+ Coder outputCoder,
+ String ownerId) {
+ this.allShards = allShards;
+ this.options = options;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ protected void init(@Nonnull Processor.Context context) throws IOException {
+ List<? extends UnboundedSource<T, CmT>> myShards =
+ Utils.roundRobinSubList(
+ allShards, context.globalProcessorIndex(), context.totalParallelism());
+ this.readers = createReaders(myShards, options);
+
+ Function<UnboundedSource.UnboundedReader<T>, byte[]> mapFn =
+ (reader) ->
+ Utils.encode(
+ WindowedValue.timestampedValueInGlobalWindow(
+ reader.getCurrent(), reader.getCurrentTimestamp()),
+ outputCoder);
+
+ if (myShards.size() == 0) {
+ traverser = Traversers.empty();
+ } else if (myShards.size() == 1) {
+ traverser = new SingleReaderTraverser<>(readers[0], mapFn);
+ } else {
+ traverser = new CoalescingTraverser<>(readers, mapFn);
+ }
+
+ for (UnboundedSource.UnboundedReader<T> reader : readers) {
+ reader.start();
+ }
+ }
+
+ @Override
+ public boolean complete() {
+ emitFromTraverser(traverser);
+ return readers.length == 0;
+ }
+
+ @Override
+ public boolean isCooperative() {
+ return false;
+ }
+
+ @Override
+ public void close() {
+ Arrays.stream(readers).forEach(UnboundedSourceP::stopReader);
+ Arrays.fill(readers, null);
+ }
+
+ @SuppressWarnings("unchecked")
+ private static <T, CmT extends UnboundedSource.CheckpointMark>
+ UnboundedSource.UnboundedReader<T>[] createReaders(
+ List<? extends UnboundedSource<T, CmT>> shards, PipelineOptions options) {
+ return shards.stream()
+ .map(shard -> createReader(options, shard))
+ .toArray(UnboundedSource.UnboundedReader[]::new);
+ }
+
+ private static long[] initWatermarks(int size) {
+ long[] watermarks = new long[size];
+ Arrays.fill(watermarks, Long.MIN_VALUE);
+ return watermarks;
+ }
+
+ private static <T> UnboundedSource.UnboundedReader<T> createReader(
+ PipelineOptions options, UnboundedSource<T, ?> shard) {
+ try {
+ return shard.createReader(options, null);
+ } catch (IOException e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+
+ private static void stopReader(UnboundedSource.UnboundedReader<?> reader) {
+ try {
+ reader.close();
+ } catch (IOException e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+
+ private static long getMin(long[] instants) {
+ long min = instants[0];
+ for (int i = 1; i < instants.length; i++) {
+ if (instants[i] < min) {
+ min = instants[i];
+ }
+ }
+ return min;
+ }
+
+ public static <T, CmT extends UnboundedSource.CheckpointMark> ProcessorMetaSupplier supplier(
+ UnboundedSource<T, CmT> unboundedSource,
+ SerializablePipelineOptions options,
+ Coder outputCoder,
+ String ownerId) {
+ return new UnboundedSourceProcessorMetaSupplier<>(
+ unboundedSource, options, outputCoder, ownerId);
+ }
+
+ private static class UnboundedSourceProcessorMetaSupplier<
+ T, CmT extends UnboundedSource.CheckpointMark>
+ implements ProcessorMetaSupplier {
+
+ private final UnboundedSource<T, CmT> unboundedSource;
+ private final SerializablePipelineOptions options;
+ private final Coder outputCoder;
+ private final String ownerId;
+
+ private List<? extends UnboundedSource<T, CmT>> shards;
+
+ private UnboundedSourceProcessorMetaSupplier(
+ UnboundedSource<T, CmT> unboundedSource,
+ SerializablePipelineOptions options,
+ Coder outputCoder,
+ String ownerId) {
+ this.unboundedSource = unboundedSource;
+ this.options = options;
+ this.outputCoder = outputCoder;
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ public void init(@Nonnull ProcessorMetaSupplier.Context context) throws Exception {
+ shards = unboundedSource.split(context.totalParallelism(), options.get());
+ }
+
+ @Nonnull
+ @Override
+ public Function<? super Address, ? extends ProcessorSupplier> get(
+ @Nonnull List<Address> addresses) {
+ return address ->
+ ProcessorSupplier.of(
+ () -> new UnboundedSourceP<>(shards, options.get(), outputCoder, ownerId));
+ }
+ }
+
+ private static class SingleReaderTraverser<InputT> implements Traverser<Object> {
+ private final UnboundedSource.UnboundedReader<InputT> reader;
+ private final Function<UnboundedSource.UnboundedReader<InputT>, byte[]> mapFn;
+ private long lastWatermark = Long.MIN_VALUE;
+
+ SingleReaderTraverser(
+ UnboundedSource.UnboundedReader<InputT> reader,
+ Function<UnboundedSource.UnboundedReader<InputT>, byte[]> mapFn) {
+ this.reader = reader;
+ this.mapFn = mapFn;
+ }
+
+ @Override
+ public Object next() {
+ long wm = reader.getWatermark().getMillis();
+ if (wm > lastWatermark) {
+ lastWatermark = wm;
+ return new Watermark(wm);
+ }
+ try {
+ return reader.advance() ? mapFn.apply(reader) : null;
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ private static class CoalescingTraverser<InputT> implements Traverser<Object> {
+ private final UnboundedSource.UnboundedReader<InputT>[] readers;
+ private final Function<UnboundedSource.UnboundedReader<InputT>, byte[]> mapFn;
+
+ private int currentReaderIndex;
+ private long minWatermark = Long.MIN_VALUE;
+ private long lastSentWatermark = Long.MIN_VALUE;
+ private long[] watermarks;
+
+ CoalescingTraverser(
+ UnboundedSource.UnboundedReader<InputT>[] readers,
+ Function<UnboundedSource.UnboundedReader<InputT>, byte[]> mapFn) {
+ this.readers = readers;
+ watermarks = initWatermarks(readers.length);
+ this.mapFn = mapFn;
+ }
+
+ @Override
+ public Object next() {
+ if (minWatermark > lastSentWatermark) {
+ lastSentWatermark = minWatermark;
+ return new Watermark(lastSentWatermark);
+ }
+
+ try {
+ // trying to fetch a value from the next reader
+ for (int i = 0; i < readers.length; i++) {
+ currentReaderIndex++;
+ if (currentReaderIndex >= readers.length) {
+ currentReaderIndex = 0;
+ }
+ UnboundedSource.UnboundedReader<InputT> currentReader = readers[currentReaderIndex];
+ if (currentReader.advance()) {
+ long currentWatermark = currentReader.getWatermark().getMillis();
+ long origWatermark = watermarks[currentReaderIndex];
+ if (currentWatermark > origWatermark) {
+ watermarks[currentReaderIndex] =
+ currentWatermark; // todo: we should probably do this only on a timer...
+ if (origWatermark == minWatermark) {
+ minWatermark = getMin(watermarks);
+ }
+ }
+ return mapFn.apply(currentReader);
+ }
+ }
+
+ // all advances have failed
+ return null;
+ } catch (IOException e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
new file mode 100644
index 0000000..bb31e9e
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.Traversers;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.function.SupplierEx;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nonnull;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.joda.time.Instant;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's side input producing
+ * primitives. Collects all input {@link WindowedValue}s, groups them by windows and keys and when
+ * input is complete emits them.
+ */
+public class ViewP extends AbstractProcessor {
+
+ private final TimestampCombiner timestampCombiner;
+ private final Coder inputCoder;
+ private final Coder outputCoder;
+
+ @SuppressWarnings({"FieldCanBeLocal", "unused"})
+ private final String ownerId; // do not remove, useful for debugging
+
+ private Map<BoundedWindow, TimestampAndValues> values = new HashMap<>();
+ private Traverser<byte[]> resultTraverser;
+
+ private ViewP(
+ Coder inputCoder, Coder outputCoder, WindowingStrategy windowingStrategy, String ownerId) {
+ this.timestampCombiner = windowingStrategy.getTimestampCombiner();
+ this.inputCoder = inputCoder;
+ this.outputCoder =
+ Utils.deriveIterableValueCoder((WindowedValue.FullWindowedValueCoder) outputCoder);
+ this.ownerId = ownerId;
+ }
+
+ @Override
+ protected boolean tryProcess(int ordinal, @Nonnull Object item) {
+ WindowedValue<?> windowedValue = Utils.decodeWindowedValue((byte[]) item, inputCoder);
+ for (BoundedWindow window : windowedValue.getWindows()) {
+ values.merge(
+ window,
+ new TimestampAndValues(
+ windowedValue.getPane(), windowedValue.getTimestamp(), windowedValue.getValue()),
+ (o, n) -> o.merge(timestampCombiner, n));
+ }
+
+ return true;
+ }
+
+ @Override
+ public boolean complete() {
+ if (resultTraverser == null) {
+ resultTraverser =
+ Traversers.traverseStream(
+ values.entrySet().stream()
+ .map(
+ e -> {
+ WindowedValue<?> outputValue =
+ WindowedValue.of(
+ e.getValue().values,
+ e.getValue().timestamp,
+ Collections.singleton(e.getKey()),
+ e.getValue().pane);
+ return Utils.encode(outputValue, outputCoder);
+ }));
+ }
+ return emitFromTraverser(resultTraverser);
+ }
+
+ public static SupplierEx<Processor> supplier(
+ Coder inputCoder,
+ Coder outputCoder,
+ WindowingStrategy<?, ?> windowingStrategy,
+ String ownerId) {
+ return () -> new ViewP(inputCoder, outputCoder, windowingStrategy, ownerId);
+ }
+
+ private static class TimestampAndValues {
+ private final List<Object> values = new ArrayList<>();
+ private Instant timestamp;
+ private PaneInfo pane;
+
+ TimestampAndValues(PaneInfo pane, Instant timestamp, Object value) {
+ this.pane = pane;
+ this.timestamp = timestamp;
+ this.values.add(value);
+ }
+
+ public Iterable<Object> getValues() {
+ return values;
+ }
+
+ TimestampAndValues merge(TimestampCombiner timestampCombiner, TimestampAndValues other) {
+ pane = other.pane;
+ timestamp = timestampCombiner.combine(timestamp, other.timestamp);
+ values.addAll(other.values);
+ return this;
+ }
+ }
+}
diff --git a/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
new file mode 100644
index 0000000..93eebb1
--- /dev/null
+++ b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
@@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet.processors;
+
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.AppendableTraverser;
+import com.hazelcast.jet.core.Processor;
+import com.hazelcast.jet.core.Watermark;
+import com.hazelcast.jet.function.SupplierEx;
+import com.hazelcast.jet.impl.util.ExceptionUtil;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.beam.runners.core.InMemoryStateInternals;
+import org.apache.beam.runners.core.InMemoryTimerInternals;
+import org.apache.beam.runners.core.LateDataUtils;
+import org.apache.beam.runners.core.NullSideInputReader;
+import org.apache.beam.runners.core.OutputWindowedValue;
+import org.apache.beam.runners.core.ReduceFnRunner;
+import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.TimerInternals;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.core.construction.TriggerTranslation;
+import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
+import org.apache.beam.runners.core.triggers.TriggerStateMachines;
+import org.apache.beam.runners.jet.Utils;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.state.State;
+import org.apache.beam.sdk.state.WatermarkHoldState;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.WindowTracing;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.joda.time.Instant;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's GroupByKeyOnly +
+ * GroupAlsoByWindow primitives.
+ *
+ * @param <K> key type of {@link KV} values from the output of this primitive
+ * @param <V> type of elements being windowed
+ */
+public class WindowGroupP<K, V> extends AbstractProcessor {
+
+ private static final int PROCESSING_TIME_MIN_INCREMENT = 100;
+
+ private static final Object COMPLETE_MARKER = new Object();
+ private static final Object TRY_PROCESS_MARKER = new Object();
+
+ private final SerializablePipelineOptions pipelineOptions;
+ private final Coder<V> inputValueValueCoder;
+ private final Coder outputCoder;
+ private final WindowingStrategy<V, BoundedWindow> windowingStrategy;
+ private final Map<Utils.ByteArrayKey, KeyManager> keyManagers = new HashMap<>();
+ private final AppendableTraverser<Object> appendableTraverser =
+ new AppendableTraverser<>(128); // todo: right capacity?
+ private final FlatMapper<Object, Object> flatMapper;
+
+ @SuppressWarnings({"FieldCanBeLocal", "unused"})
+ private final String ownerId; // do not remove, useful for debugging
+
+ private Instant latestWatermark = BoundedWindow.TIMESTAMP_MIN_VALUE;
+ private long lastProcessingTime = System.currentTimeMillis();
+
+ private WindowGroupP(
+ SerializablePipelineOptions pipelineOptions,
+ WindowedValue.WindowedValueCoder<KV<K, V>> inputCoder,
+ Coder outputCoder,
+ WindowingStrategy<V, BoundedWindow> windowingStrategy,
+ String ownerId) {
+ this.pipelineOptions = pipelineOptions;
+ KvCoder<K, V> inputValueCoder = (KvCoder<K, V>) inputCoder.getValueCoder();
+ this.inputValueValueCoder = inputValueCoder.getValueCoder();
+ this.outputCoder = outputCoder;
+ this.windowingStrategy = windowingStrategy;
+ this.ownerId = ownerId;
+
+ this.flatMapper =
+ flatMapper(
+ item -> {
+ if (COMPLETE_MARKER == item) {
+ long millis = BoundedWindow.TIMESTAMP_MAX_VALUE.getMillis();
+ advanceWatermark(millis);
+ } else if (TRY_PROCESS_MARKER == item) {
+ Instant now = Instant.now();
+ if (now.getMillis() - lastProcessingTime > PROCESSING_TIME_MIN_INCREMENT) {
+ lastProcessingTime = now.getMillis();
+ advanceProcessingTime(now);
+ }
+ } else if (item instanceof Watermark) {
+ advanceWatermark(((Watermark) item).timestamp());
+ appendableTraverser.append(item);
+ } else {
+ WindowedValue<KV<K, V>> windowedValue =
+ Utils.decodeWindowedValue((byte[]) item, inputCoder);
+ KV<K, V> kv = windowedValue.getValue();
+ K key = kv.getKey();
+ V value = kv.getValue();
+ Utils.ByteArrayKey keyBytes =
+ new Utils.ByteArrayKey(Utils.encode(key, inputValueCoder.getKeyCoder()));
+ WindowedValue<V> updatedWindowedValue =
+ WindowedValue.of(
+ value,
+ windowedValue.getTimestamp(),
+ windowedValue.getWindows(),
+ windowedValue.getPane());
+ keyManagers
+ .computeIfAbsent(keyBytes, x -> new KeyManager(key))
+ .processElement(updatedWindowedValue);
+ }
+ return appendableTraverser;
+ });
+ }
+
+ @SuppressWarnings("unchecked")
+ public static <K, V> SupplierEx<Processor> supplier(
+ SerializablePipelineOptions pipelineOptions,
+ WindowedValue.WindowedValueCoder<KV<K, V>> inputCoder,
+ Coder outputCoder,
+ WindowingStrategy windowingStrategy,
+ String ownerId) {
+ return () ->
+ new WindowGroupP<>(pipelineOptions, inputCoder, outputCoder, windowingStrategy, ownerId);
+ }
+
+ @Override
+ public boolean tryProcess() {
+ return flatMapper.tryProcess(TRY_PROCESS_MARKER);
+ }
+
+ @Override
+ protected boolean tryProcess(int ordinal, @Nonnull Object item) {
+ return flatMapper.tryProcess(item);
+ }
+
+ @Override
+ public boolean tryProcessWatermark(@Nonnull Watermark watermark) {
+ return flatMapper.tryProcess(watermark);
+ }
+
+ @Override
+ public boolean complete() {
+ return flatMapper.tryProcess(COMPLETE_MARKER);
+ }
+
+ private void advanceWatermark(long millis) {
+ this.latestWatermark = new Instant(millis);
+ Instant now = Instant.now();
+ for (KeyManager m : keyManagers.values()) {
+ m.advanceWatermark(latestWatermark, now);
+ }
+ }
+
+ private void advanceProcessingTime(Instant now) {
+ for (KeyManager m : keyManagers.values()) {
+ m.advanceProcessingTime(now);
+ }
+ }
+
+ private static class InMemoryStateInternalsImpl extends InMemoryStateInternals {
+
+ InMemoryStateInternalsImpl(@Nullable Object key) {
+ super(key);
+ }
+
+ Instant earliestWatermarkHold() {
+ Instant minimum = null;
+ for (State storage : inMemoryState.values()) {
+ if (storage instanceof WatermarkHoldState) {
+ Instant hold = ((WatermarkHoldState) storage).read();
+ if (minimum == null || (hold != null && hold.isBefore(minimum))) {
+ minimum = hold;
+ }
+ }
+ }
+ return minimum;
+ }
+ }
+
+ private class KeyManager {
+
+ private final InMemoryTimerInternals timerInternals;
+ private final InMemoryStateInternalsImpl stateInternals;
+ private final ReduceFnRunner<K, V, Iterable<V>, BoundedWindow> reduceFnRunner;
+
+ KeyManager(K key) {
+ this.timerInternals = new InMemoryTimerInternals();
+ this.stateInternals = new InMemoryStateInternalsImpl(key);
+ this.reduceFnRunner =
+ new ReduceFnRunner<>(
+ key,
+ windowingStrategy,
+ ExecutableTriggerStateMachine.create(
+ TriggerStateMachines.stateMachineForTrigger(
+ TriggerTranslation.toProto(windowingStrategy.getTrigger()))),
+ stateInternals,
+ timerInternals,
+ new OutputWindowedValue<KV<K, Iterable<V>>>() {
+ @Override
+ public void outputWindowedValue(
+ KV<K, Iterable<V>> output,
+ Instant timestamp,
+ Collection<? extends BoundedWindow> windows,
+ PaneInfo pane) {
+ WindowedValue<KV<K, Iterable<V>>> windowedValue =
+ WindowedValue.of(output, timestamp, windows, pane);
+ byte[] encodedValue = Utils.encode(windowedValue, outputCoder);
+ //noinspection ResultOfMethodCallIgnored
+ appendableTraverser.append(encodedValue);
+ }
+
+ @Override
+ public <AdditionalOutputT> void outputWindowedValue(
+ TupleTag<AdditionalOutputT> tag,
+ AdditionalOutputT output,
+ Instant timestamp,
+ Collection<? extends BoundedWindow> windows,
+ PaneInfo pane) {
+ throw new UnsupportedOperationException("Grouping should not use side outputs");
+ }
+ },
+ NullSideInputReader.empty(),
+ SystemReduceFn.buffering(inputValueValueCoder),
+ pipelineOptions.get());
+ advanceWatermark(latestWatermark, Instant.now());
+ }
+
+ void advanceWatermark(Instant watermark, Instant now) {
+ try {
+ timerInternals.advanceProcessingTime(now);
+ advanceInputWatermark(watermark);
+ Instant hold = stateInternals.earliestWatermarkHold();
+ if (hold == null) {
+ WindowTracing.trace(
+ "TestInMemoryTimerInternals.advanceInputWatermark: no holds, "
+ + "so output watermark = input watermark");
+ hold = timerInternals.currentInputWatermarkTime();
+ }
+ advanceOutputWatermark(hold);
+ reduceFnRunner.persist();
+ } catch (Exception e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+
+ void advanceProcessingTime(Instant now) {
+ try {
+ timerInternals.advanceProcessingTime(now);
+ reduceFnRunner.persist();
+ } catch (Exception e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+
+ private void advanceInputWatermark(Instant watermark) throws Exception {
+ timerInternals.advanceInputWatermark(watermark);
+ while (true) {
+ TimerInternals.TimerData timer;
+ List<TimerInternals.TimerData> timers = new ArrayList<>();
+ while ((timer = timerInternals.removeNextEventTimer()) != null) {
+ timers.add(timer);
+ }
+ if (timers.isEmpty()) {
+ break;
+ }
+ reduceFnRunner.onTimers(timers);
+ }
+ }
+
+ private void advanceOutputWatermark(Instant watermark) {
+ Objects.requireNonNull(watermark);
+ timerInternals.advanceOutputWatermark(watermark);
+ }
+
+ public void processElement(WindowedValue<V> windowedValue) {
+ Collection<? extends BoundedWindow> windows = dropLateWindows(windowedValue.getWindows());
+ if (!windows.isEmpty()) {
+ try {
+ reduceFnRunner.processElements(
+ Collections.singletonList(
+ windowedValue)); // todo: try to process more than one element at a time...
+ reduceFnRunner.persist();
+ } catch (Exception e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+ }
+
+ private Collection<? extends BoundedWindow> dropLateWindows(
+ Collection<? extends BoundedWindow> windows) {
+ boolean hasExpired = false;
+ for (Iterator<? extends BoundedWindow> iterator = windows.iterator();
+ !hasExpired && iterator.hasNext(); ) {
+ if (isExpiredWindow(iterator.next())) {
+ hasExpired = true;
+ }
+ }
+ if (!hasExpired) {
+ return windows;
+ }
+ // if there are expired items, return a filtered collection
+ return windows.stream()
+ .filter(window -> !isExpiredWindow(window))
+ .collect(Collectors.toList());
+ }
+
+ private boolean isExpiredWindow(BoundedWindow window) {
+ Instant inputWM = timerInternals.currentInputWatermarkTime();
+ return LateDataUtils.garbageCollectionTime(window, windowingStrategy).isBefore(inputWM);
+ }
+ }
+}
diff --git a/runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/package-info.java b/runners/jet/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
similarity index 100%
rename from runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
rename to runners/jet/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
diff --git a/runners/jet/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java b/runners/jet/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
new file mode 100644
index 0000000..b8d3e7b
--- /dev/null
+++ b/runners/jet/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
+import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
+ * TestJetRunner}.
+ *
+ * <p>{@link AutoService} will register Apex's implementations of the {@link PipelineRunner} and
+ * {@link PipelineOptions} as available pipeline runner services.
+ */
+public final class JetTestRunnerRegistrar {
+ private JetTestRunnerRegistrar() {}
+
+ /** Registers the {@link JetRunner}. */
+ @AutoService(PipelineRunnerRegistrar.class)
+ public static class Runner implements PipelineRunnerRegistrar {
+ @Override
+ public Iterable<Class<? extends PipelineRunner<?>>> getPipelineRunners() {
+ return ImmutableList.of(TestJetRunner.class);
+ }
+ }
+
+ /** Registers the {@link JetPipelineOptions}. */
+ @AutoService(PipelineOptionsRegistrar.class)
+ public static class Options implements PipelineOptionsRegistrar {
+ @Override
+ public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
+ return ImmutableList.of(JetPipelineOptions.class);
+ }
+ }
+}
diff --git a/runners/jet/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java b/runners/jet/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
new file mode 100644
index 0000000..c5c0b00
--- /dev/null
+++ b/runners/jet/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.config.EventJournalConfig;
+import com.hazelcast.jet.JetInstance;
+import com.hazelcast.jet.JetTestInstanceFactory;
+import com.hazelcast.jet.config.JetConfig;
+import com.hazelcast.jet.core.Vertex;
+import com.hazelcast.jet.impl.util.ExceptionUtil;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.runners.TransformHierarchy;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TupleTag;
+
+/** Slightly altered version of the Jet based runner, used in unit-tests. */
+public class TestJetRunner extends PipelineRunner<PipelineResult> {
+
+ /** A map from a Transform URN to the translator. */
+ private static final Map<String, JetTransformTranslator> TRANSLATORS = new HashMap<>();
+
+ static {
+ TRANSLATORS.put(PTransformTranslation.TEST_STREAM_TRANSFORM_URN, new TestStreamTranslator());
+ }
+
+ private final JetTestInstanceFactory factory;
+ private final JetRunner delegate;
+
+ private TestJetRunner(PipelineOptions options) {
+ this.factory = new JetTestInstanceFactory();
+
+ this.delegate = JetRunner.fromOptions(options, factory::newClient);
+ this.delegate.addExtraTranslators(TestJetRunner::getTranslator);
+ }
+
+ public static TestJetRunner fromOptions(PipelineOptions options) {
+ return new TestJetRunner(options);
+ }
+
+ @Override
+ public PipelineResult run(Pipeline pipeline) {
+ Collection<JetInstance> instances = initMemberInstances(factory);
+ try {
+ PipelineResult result = delegate.run(pipeline);
+ if (result instanceof FailedRunningPipelineResults) {
+ throw ((FailedRunningPipelineResults) result).getCause();
+ }
+ result.waitUntilFinish();
+ return result;
+ } finally {
+ killMemberInstances(instances, factory);
+ }
+ }
+
+ private static Collection<JetInstance> initMemberInstances(
+ JetTestInstanceFactory internalFactory) {
+ JetConfig config = new JetConfig();
+ config.getHazelcastConfig().addEventJournalConfig(new EventJournalConfig().setMapName("map"));
+
+ return Arrays.asList(internalFactory.newMember(config), internalFactory.newMember(config));
+ }
+
+ private static void killMemberInstances(
+ Collection<JetInstance> instances, JetTestInstanceFactory internalFactory) {
+ if (!instances.isEmpty()) {
+ // there are own member instances to kill
+ internalFactory.shutdownAll();
+ }
+ }
+
+ private static JetTransformTranslator<?> getTranslator(PTransform<?, ?> transform) {
+ String urn = PTransformTranslation.urnForTransformOrNull(transform);
+ return urn == null ? null : TRANSLATORS.get(urn);
+ }
+
+ private static class TestStreamTranslator<T>
+ implements JetTransformTranslator<PTransform<PBegin, PCollection<T>>> {
+ @Override
+ public Vertex translate(
+ Pipeline pipeline,
+ AppliedPTransform<?, ?, ?> appliedTransform,
+ TransformHierarchy.Node node,
+ JetTranslationContext context) {
+ String transformName = appliedTransform.getFullName();
+ DAGBuilder dagBuilder = context.getDagBuilder();
+ String vertexId = dagBuilder.newVertexId(transformName);
+
+ TestStream<T> testStream = (TestStream<T>) appliedTransform.getTransform();
+
+ // events in the transform are not serializable, we have to translate them. We'll also flatten
+ // the collection.
+ Map.Entry<TupleTag<?>, PValue> output = Utils.getOutput(appliedTransform);
+ Coder outputCoder = Utils.getCoder((PCollection) output.getValue());
+ TestStream.TestStreamCoder<T> payloadCoder =
+ TestStream.TestStreamCoder.of(testStream.getValueCoder());
+ byte[] encodedPayload = getEncodedPayload(testStream, payloadCoder);
+ Vertex vertex =
+ dagBuilder.addVertex(
+ vertexId, TestStreamP.supplier(encodedPayload, payloadCoder, outputCoder));
+
+ String outputEdgeId = Utils.getTupleTagId(output.getValue());
+ dagBuilder.registerCollectionOfEdge(outputEdgeId, output.getKey().getId());
+ dagBuilder.registerEdgeStartPoint(outputEdgeId, vertex, outputCoder);
+ return vertex;
+ }
+
+ private static <T> byte[] getEncodedPayload(
+ TestStream<T> testStream, TestStream.TestStreamCoder<T> coder) {
+ try {
+ return CoderUtils.encodeToByteArray(coder, testStream);
+ } catch (CoderException e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+ }
+}
diff --git a/runners/jet/src/test/java/org/apache/beam/runners/jet/TestStreamP.java b/runners/jet/src/test/java/org/apache/beam/runners/jet/TestStreamP.java
new file mode 100644
index 0000000..84174f8
--- /dev/null
+++ b/runners/jet/src/test/java/org/apache/beam/runners/jet/TestStreamP.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.jet;
+
+import com.hazelcast.jet.Traverser;
+import com.hazelcast.jet.Traversers;
+import com.hazelcast.jet.core.AbstractProcessor;
+import com.hazelcast.jet.core.ProcessorMetaSupplier;
+import com.hazelcast.jet.core.ProcessorSupplier;
+import com.hazelcast.jet.core.Watermark;
+import com.hazelcast.jet.impl.util.ExceptionUtil;
+import com.hazelcast.jet.impl.util.ThrottleWrappedP;
+import java.util.List;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.testing.TestStream;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.joda.time.Instant;
+
+/**
+ * Jet {@link com.hazelcast.jet.core.Processor} implementation for Beam's {@link TestStream}
+ * transform.
+ */
+public class TestStreamP extends AbstractProcessor {
+
+ private final Traverser traverser;
+
+ @SuppressWarnings("unchecked")
+ private TestStreamP(byte[] payload, TestStream.TestStreamCoder payloadCoder, Coder outputCoder) {
+ List events = decodePayload(payload, payloadCoder).getEvents();
+ traverser =
+ Traversers.traverseStream(
+ events.stream()
+ .flatMap(
+ event -> {
+ if (event instanceof TestStream.WatermarkEvent) {
+ Instant watermark = ((TestStream.WatermarkEvent) event).getWatermark();
+ if (BoundedWindow.TIMESTAMP_MAX_VALUE.equals(watermark)) {
+ // this is an element added by advanceWatermarkToInfinity(), we ignore it,
+ // it's always at the end
+ return null;
+ }
+ return Stream.of(new Watermark(watermark.getMillis()));
+ } else if (event instanceof TestStream.ElementEvent) {
+ return StreamSupport.stream(
+ ((TestStream.ElementEvent<?>) event).getElements().spliterator(),
+ false)
+ .map(
+ tv ->
+ WindowedValue.timestampedValueInGlobalWindow(
+ tv.getValue(), tv.getTimestamp()))
+ .map(wV -> Utils.encode(wV, outputCoder));
+ } else {
+ throw new UnsupportedOperationException(
+ "Event type not supported in TestStream: "
+ + event.getClass()
+ + ", event: "
+ + event);
+ }
+ }));
+ }
+
+ public static <T> ProcessorMetaSupplier supplier(
+ byte[] payload, TestStream.TestStreamCoder payloadCoder, Coder outputCoder) {
+ return ProcessorMetaSupplier.forceTotalParallelismOne(
+ ProcessorSupplier.of(
+ () -> new ThrottleWrappedP(new TestStreamP(payload, payloadCoder, outputCoder), 4)));
+ }
+
+ private static TestStream decodePayload(byte[] payload, TestStream.TestStreamCoder coder) {
+ try {
+ return (TestStream) CoderUtils.decodeFromByteArray(coder, payload);
+ } catch (CoderException e) {
+ throw ExceptionUtil.rethrow(e);
+ }
+ }
+
+ @Override
+ public boolean complete() {
+ return emitFromTraverser(traverser);
+ }
+}
diff --git a/runners/local-java/build.gradle b/runners/local-java/build.gradle
index fd57a74..343327a 100644
--- a/runners/local-java/build.gradle
+++ b/runners/local-java/build.gradle
@@ -18,9 +18,10 @@
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-runners-local-java-core'
-
-applyJavaNature()
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.local',
+ archivesBaseName: 'beam-runners-local-java-core'
+)
description = "Apache Beam :: Runners :: Local Java Core"
@@ -31,9 +32,9 @@
* but should not be used within this library to execute any UDFs.
* TODO: Add an APISurfaceTest to force this to be the case, if possible.
*/
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.joda_time
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.junit
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.joda_time
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.junit
}
diff --git a/runners/reference/java/build.gradle b/runners/reference/java/build.gradle
index 59d659c..35d4dff 100644
--- a/runners/reference/java/build.gradle
+++ b/runners/reference/java/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.reference')
description = "Apache Beam :: Runners :: Reference :: Java"
ext.summary = """A Java implementation of the Beam Model which utilizes the portability
@@ -29,19 +29,15 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
+ compile library.java.vendored_guava_26_0_jre
compile library.java.hamcrest_library
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":runners:java-fn-execution", configuration: "shadow")
- shadow project(path: ":sdks:java:fn-execution", configuration: "shadow")
- shadow project(path: ":sdks:java:harness", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.slf4j_api
- shadowTest project(path: ":runners:core-construction-java", configuration: "shadowTest")
- shadowTest library.java.guava
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.slf4j_jdk14
+ compile project(":runners:java-fn-execution")
+ compile project(path: ":sdks:java:harness", configuration: "shadow")
+ compile library.java.vendored_grpc_1_21_0
+ compile library.java.slf4j_api
+ testCompile project(path: ":runners:core-construction-java", configuration: "testRuntime")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.slf4j_jdk14
}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
index 96ba294..ba533ea 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/CloseableResource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.reference;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
index 337eb4b..c23f727 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/ExternalWorkerService.java
@@ -18,14 +18,14 @@
package org.apache.beam.runners.reference;
import org.apache.beam.fn.harness.FnHarness;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.NotifyRunnerAvailableRequest;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.NotifyRunnerAvailableResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StartWorkerResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc.BeamFnExternalWorkerPoolImplBase;
import org.apache.beam.runners.fnexecution.FnService;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -39,14 +39,13 @@
private final PipelineOptions options;
private final ServerFactory serverFactory = ServerFactory.createDefault();
- public ExternalWorkerService(PipelineOptions options) throws Exception {
+ public ExternalWorkerService(PipelineOptions options) {
this.options = options;
}
@Override
- public void notifyRunnerAvailable(
- NotifyRunnerAvailableRequest request,
- StreamObserver<NotifyRunnerAvailableResponse> responseObserver) {
+ public void startWorker(
+ StartWorkerRequest request, StreamObserver<StartWorkerResponse> responseObserver) {
LOG.info(
"Starting worker {} pointing at {}.",
request.getWorkerId(),
@@ -70,7 +69,7 @@
th.setDaemon(true);
th.start();
- responseObserver.onNext(NotifyRunnerAvailableResponse.newBuilder().build());
+ responseObserver.onNext(StartWorkerResponse.newBuilder().build());
responseObserver.onCompleted();
}
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
index fe7701a..2164e2b 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/JobServicePipelineResult.java
@@ -30,7 +30,7 @@
import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceBlockingStub;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.metrics.MetricResults;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,7 +44,7 @@
private final ByteString jobId;
private final CloseableResource<JobServiceBlockingStub> jobService;
@Nullable private State terminationState;
- @Nullable private Runnable cleanup;
+ @Nullable private final Runnable cleanup;
JobServicePipelineResult(
ByteString jobId, CloseableResource<JobServiceBlockingStub> jobService, Runnable cleanup) {
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
index e71cf1b..240decc 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunner.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.reference;
import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.File;
import java.io.FileOutputStream;
@@ -49,11 +49,11 @@
import org.apache.beam.sdk.options.PipelineOptionsValidator;
import org.apache.beam.sdk.options.PortablePipelineOptions;
import org.apache.beam.sdk.util.ZipFiles;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java
index 60784c5..f989090 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/PortableRunnerRegistrar.java
@@ -20,9 +20,9 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-/** Registrar for the poratble runner. */
+/** Registrar for the portable runner. */
@AutoService(PipelineRunnerRegistrar.class)
public class PortableRunnerRegistrar implements PipelineRunnerRegistrar {
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
index 34b026f..e3b22e6 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestJobService.java
@@ -26,7 +26,7 @@
import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc.JobServiceImplBase;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A JobService for tests.
diff --git a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
index 8323df7..33ba8b1 100644
--- a/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
+++ b/runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/TestPortablePipelineOptions.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.options.PortablePipelineOptions;
import org.apache.beam.sdk.options.Validation.Required;
import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Options for {@link TestPortableRunner}. */
public interface TestPortablePipelineOptions extends TestPipelineOptions, PortablePipelineOptions {
diff --git a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java b/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
index ac8aa5d..d9f91cd 100644
--- a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
+++ b/runners/reference/java/src/test/java/org/apache/beam/runners/reference/CloseableResourceTest.java
@@ -46,11 +46,7 @@
public void callsCloser() throws Exception {
AtomicBoolean closed = new AtomicBoolean(false);
try (CloseableResource<Foo> ignored =
- CloseableResource.of(
- new Foo(),
- foo -> {
- closed.set(true);
- })) {
+ CloseableResource.of(new Foo(), foo -> closed.set(true))) {
// Do nothing.
}
assertThat(closed.get(), is(true));
diff --git a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java b/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
index 0c5832e..da43c54 100644
--- a/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
+++ b/runners/reference/java/src/test/java/org/apache/beam/runners/reference/PortableRunnerTest.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.PortablePipelineOptions;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -47,7 +47,7 @@
private static final ApiServiceDescriptor ENDPOINT_DESCRIPTOR =
ApiServiceDescriptor.newBuilder().setUrl(ENDPOINT_URL).build();
- private PipelineOptions options = createPipelineOptions();
+ private final PipelineOptions options = createPipelineOptions();
@Rule public transient TestPipeline p = TestPipeline.fromOptions(options);
diff --git a/runners/reference/job-server/build.gradle b/runners/reference/job-server/build.gradle
deleted file mode 100644
index 53459a8..0000000
--- a/runners/reference/job-server/build.gradle
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-plugins {
- id 'org.apache.beam.module'
- id 'application'
-}
-// we need to set mainClassName before applying shadow plugin
-mainClassName = "org.apache.beam.runners.direct.portable.job.ReferenceRunnerJobServer"
-
-applyJavaNature(
- exportJavadoc: false,
- validateShadowJar: false,
- shadowClosure: {
- }
-)
-
-description = "Apache Beam :: Runners :: Reference :: Job Server"
-
-dependencies {
- compile project(path: ":runners:direct-java", configuration: "shadow")
- compile project(path: ":runners:java-fn-execution", configuration: "shadow")
- compile library.java.slf4j_simple
-}
-
-run {
- args = []
- if (project.hasProperty('port'))
- args += ["--port=${project.property('port')}"]
-
- // Enable remote debugging.
- jvmArgs = ["-Xdebug", "-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
- if (project.hasProperty("logLevel"))
- jvmArgs += ["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
- if (project.hasProperty("vendorLogLevel")) {
- jvmArgs += ["-Dorg.slf4j.simpleLogger.log.org.apache.beam.vendor=${project.property('vendorLogLevel')}"]
- } else {
- jvmArgs += ["-Dorg.slf4j.simpleLogger.log.org.apache.beam.vendor=info"]
- }
-}
diff --git a/runners/samza/build.gradle b/runners/samza/build.gradle
index 5b95105..209db64 100644
--- a/runners/samza/build.gradle
+++ b/runners/samza/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.runners.samza')
description = "Apache Beam :: Runners :: Samza"
@@ -37,35 +37,35 @@
def samza_version = "1.1.0"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:core-java", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":runners:java-fn-execution", configuration: "shadow")
- shadow library.java.jackson_annotations
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.commons_compress
- shadow library.java.commons_io_2x
- shadow library.java.args4j
- shadow "org.apache.samza:samza-api:$samza_version"
- shadow "org.apache.samza:samza-core_2.11:$samza_version"
- shadow "org.apache.samza:samza-kafka_2.11:$samza_version"
- shadow "org.apache.samza:samza-kv_2.11:$samza_version"
- shadow "org.apache.samza:samza-kv-rocksdb_2.11:$samza_version"
- shadow "org.apache.samza:samza-kv-inmemory_2.11:$samza_version"
- shadow "org.apache.samza:samza-yarn_2.11:$samza_version"
- shadow "org.apache.kafka:kafka-clients:0.11.0.2"
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest library.java.commons_lang3
- shadowTest library.java.hamcrest_core
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.jackson_dataformat_yaml
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-java")
+ compile project(":runners:core-construction-java")
+ compile project(":runners:java-fn-execution")
+ compile library.java.jackson_annotations
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile library.java.commons_compress
+ compile library.java.commons_io_2x
+ compile library.java.args4j
+ compile "org.apache.samza:samza-api:$samza_version"
+ compile "org.apache.samza:samza-core_2.11:$samza_version"
+ compile "org.apache.samza:samza-kafka_2.11:$samza_version"
+ compile "org.apache.samza:samza-kv_2.11:$samza_version"
+ compile "org.apache.samza:samza-kv-rocksdb_2.11:$samza_version"
+ compile "org.apache.samza:samza-kv-inmemory_2.11:$samza_version"
+ compile "org.apache.samza:samza-yarn_2.11:$samza_version"
+ compile "org.apache.kafka:kafka-clients:0.11.0.2"
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ testCompile library.java.commons_lang3
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.jackson_dataformat_yaml
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
+ validatesRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesRunner project(project.path)
}
task validatesRunner(type: Test) {
diff --git a/runners/samza/job-server/build.gradle b/runners/samza/job-server/build.gradle
index d4bc76c..34c177f 100644
--- a/runners/samza/job-server/build.gradle
+++ b/runners/samza/job-server/build.gradle
@@ -24,6 +24,7 @@
mainClassName = "org.apache.beam.runners.samza.SamzaJobServerDriver"
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.samza.jobserver',
validateShadowJar: false,
exportJavadoc: false,
shadowClosure: {
@@ -32,12 +33,11 @@
)
dependencies {
- compile project(path: ":runners:samza", configuration: "shadow")
- compile group: "org.slf4j", name: "jcl-over-slf4j", version: dependencies.create(project.library.java.slf4j_api).getVersion()
- compile library.java.slf4j_simple
- shadow library.java.guava
+ compile project(":runners:samza")
+ runtime group: "org.slf4j", name: "jcl-over-slf4j", version: dependencies.create(project.library.java.slf4j_api).getVersion()
+ runtime library.java.slf4j_simple
}
runShadow {
args = []
-}
\ No newline at end of file
+}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionContext.java
index 0867e51..6518da3 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobServerDriver.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobServerDriver.java
index 7f0aabc..f21d666 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobServerDriver.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobServerDriver.java
@@ -32,8 +32,8 @@
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
index 5ef07cf..ed4437f 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
@@ -100,12 +100,6 @@
void setStateDurable(Boolean stateDurable);
- @Description("The maximum number of event-time timers buffered in memory for a transform.")
- @Default.Integer(50000)
- int getTimerBufferSize();
-
- void setTimerBufferSize(int timerBufferSize);
-
@JsonIgnore
@Description("The metrics reporters that will be used to emit metrics.")
List<MetricsReporter> getMetricsReporters();
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
index 49ed1d4..24ed330 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsValidator;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
index c6aebfa..85bd576 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
@@ -20,10 +20,10 @@
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.runners.samza.util.PortablePipelineDotRenderer;
-import org.apache.beam.sdk.PipelineResult;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -35,11 +35,11 @@
private final SamzaPipelineOptions options;
@Override
- public PipelineResult run(final Pipeline pipeline, JobInfo jobInfo) {
+ public PortablePipelineResult run(final Pipeline pipeline, JobInfo jobInfo) {
// Fused pipeline proto.
final RunnerApi.Pipeline fusedPipeline = GreedyPipelineFuser.fuse(pipeline).toPipeline();
LOG.info("Portable pipeline to run:");
- LOG.info(PortablePipelineDotRenderer.toDotString(fusedPipeline));
+ LOG.info(PipelineDotRenderer.toDotString(fusedPipeline));
// the pipeline option coming from sdk will set the sdk specific runner which will break
// serialization
// so we need to reset the runner here to a valid Java runner
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineResult.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineResult.java
new file mode 100644
index 0000000..11c8b55
--- /dev/null
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineResult.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza;
+
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
+import org.apache.samza.application.StreamApplication;
+import org.apache.samza.config.Config;
+import org.apache.samza.runtime.ApplicationRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** The result from executing a Samza Portable Pipeline. */
+public class SamzaPortablePipelineResult extends SamzaPipelineResult
+ implements PortablePipelineResult {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SamzaPortablePipelineResult.class);
+
+ SamzaPortablePipelineResult(
+ StreamApplication app,
+ ApplicationRunner runner,
+ SamzaExecutionContext executionContext,
+ SamzaPipelineLifeCycleListener listener,
+ Config config) {
+ super(app, runner, executionContext, listener, config);
+ }
+
+ @Override
+ public JobApi.MetricResults portableMetrics() throws UnsupportedOperationException {
+ LOG.warn("Collecting monitoring infos is not implemented yet in Samza portable runner.");
+ return JobApi.MetricResults.newBuilder().build();
+ }
+}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
index 0827b8d..0eb50c4 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
@@ -23,6 +23,8 @@
import java.util.Map;
import java.util.ServiceLoader;
import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.samza.translation.ConfigBuilder;
import org.apache.beam.runners.samza.translation.PViewToIdMapper;
import org.apache.beam.runners.samza.translation.PortableTranslationContext;
@@ -30,13 +32,12 @@
import org.apache.beam.runners.samza.translation.SamzaPortablePipelineTranslator;
import org.apache.beam.runners.samza.translation.SamzaTransformOverrides;
import org.apache.beam.runners.samza.translation.TranslationContext;
-import org.apache.beam.runners.samza.util.PipelineDotRenderer;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.apache.samza.application.StreamApplication;
import org.apache.samza.config.Config;
import org.apache.samza.context.ExternalContext;
@@ -70,7 +71,7 @@
listenerReg.hasNext() ? Iterators.getOnlyElement(listenerReg).getLifeCycleListener() : null;
}
- public SamzaPipelineResult runPortablePipeline(RunnerApi.Pipeline pipeline) {
+ public PortablePipelineResult runPortablePipeline(RunnerApi.Pipeline pipeline) {
final ConfigBuilder configBuilder = new ConfigBuilder(options);
SamzaPortablePipelineTranslator.createConfig(pipeline, configBuilder, options);
@@ -92,7 +93,8 @@
pipeline, new PortableTranslationContext(appDescriptor, options));
};
- return runSamzaApp(app, config, executionContext);
+ ApplicationRunner runner = runSamzaApp(app, config);
+ return new SamzaPortablePipelineResult(app, runner, executionContext, listener, config);
}
@Override
@@ -131,7 +133,8 @@
pipeline, new TranslationContext(appDescriptor, idMap, options));
};
- return runSamzaApp(app, config, executionContext);
+ ApplicationRunner runner = runSamzaApp(app, config);
+ return new SamzaPipelineResult(app, runner, executionContext, listener, config);
}
private Map<String, MetricsReporterFactory> getMetricsReporters() {
@@ -150,12 +153,9 @@
}
}
- private SamzaPipelineResult runSamzaApp(
- StreamApplication app, Config config, SamzaExecutionContext executionContext) {
+ private ApplicationRunner runSamzaApp(StreamApplication app, Config config) {
final ApplicationRunner runner = ApplicationRunners.getApplicationRunner(app, config);
- final SamzaPipelineResult result =
- new SamzaPipelineResult(app, runner, executionContext, listener, config);
ExternalContext externalContext = null;
if (listener != null) {
@@ -169,6 +169,6 @@
listener.onSubmit();
}
- return result;
+ return runner;
}
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunnerRegistrar.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunnerRegistrar.java
index 6359dd9..dd44b86 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunnerRegistrar.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunnerRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* AutoService registrar - will register SamzaRunner and SamzaOptions as possible pipeline runner
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
index 1d776b8..d3f50c9 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.samza.Partition;
import org.apache.samza.SamzaException;
import org.apache.samza.config.Config;
@@ -101,7 +101,8 @@
@Override
public Map<SystemStreamPartition, String> getOffsetsAfter(
Map<SystemStreamPartition, String> offsets) {
- return offsets.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, null));
+ // BEAM checkpoints the next offset so here we just need to return the map itself
+ return offsets;
}
@Override
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/UnboundedSourceSystem.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/UnboundedSourceSystem.java
index f5066cd..b118a3c 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/UnboundedSourceSystem.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/UnboundedSourceSystem.java
@@ -46,8 +46,8 @@
import org.apache.beam.sdk.io.UnboundedSource.UnboundedReader;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.commons.lang3.StringUtils;
import org.apache.samza.Partition;
import org.apache.samza.SamzaException;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
index d986b15..86e2ee6 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
@@ -53,7 +53,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.apache.samza.config.Config;
import org.apache.samza.context.Context;
import org.apache.samza.operators.Scheduler;
@@ -74,8 +74,8 @@
private final OutputManagerFactory<OutT> outputManagerFactory;
// NOTE: we use HashMap here to guarantee Serializability
private final HashMap<String, PCollectionView<?>> idToViewMap;
- private final String stepName;
- private final String stepId;
+ private final String transformFullName;
+ private final String transformId;
private final Coder<InT> inputCoder;
private final HashMap<TupleTag<?>, Coder<?>> outputCoders;
private final PCollection.IsBounded isBounded;
@@ -104,6 +104,7 @@
private transient List<WindowedValue<InT>> pushbackValues;
private transient StageBundleFactory stageBundleFactory;
private DoFnSchemaInformation doFnSchemaInformation;
+ private Map<String, PCollectionView<?>> sideInputMapping;
public DoFnOp(
TupleTag<FnOutT> mainOutputTag,
@@ -116,13 +117,14 @@
WindowingStrategy windowingStrategy,
Map<String, PCollectionView<?>> idToViewMap,
OutputManagerFactory<OutT> outputManagerFactory,
- String stepName,
- String stepId,
+ String transformFullName,
+ String transformId,
PCollection.IsBounded isBounded,
boolean isPortable,
RunnerApi.ExecutableStagePayload stagePayload,
Map<String, TupleTag<?>> idToTupleTagMap,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.mainOutputTag = mainOutputTag;
this.doFn = doFn;
this.sideInputs = sideInputs;
@@ -132,14 +134,15 @@
this.windowingStrategy = windowingStrategy;
this.idToViewMap = new HashMap<>(idToViewMap);
this.outputManagerFactory = outputManagerFactory;
- this.stepName = stepName;
- this.stepId = stepId;
+ this.transformFullName = transformFullName;
+ this.transformId = transformId;
this.keyCoder = keyCoder;
this.isBounded = isBounded;
this.isPortable = isPortable;
this.stagePayload = stagePayload;
this.idToTupleTagMap = new HashMap<>(idToTupleTagMap);
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
@Override
@@ -159,10 +162,9 @@
.get()
.as(SamzaPipelineOptions.class);
- final String stateId = "pardo-" + stepId;
final SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stateId, null, context.getTaskContext(), pipelineOptions, signature);
+ transformId, null, context.getTaskContext(), pipelineOptions, signature);
this.timerInternalsFactory =
SamzaTimerInternalsFactory.createTimerInternalFactory(
@@ -189,15 +191,15 @@
mainOutputTag,
idToTupleTagMap,
context,
- stepName);
+ transformFullName);
} else {
this.fnRunner =
SamzaDoFnRunners.create(
pipelineOptions,
doFn,
windowingStrategy,
- stepName,
- stateId,
+ transformFullName,
+ transformId,
context,
mainOutputTag,
sideInputHandler,
@@ -207,7 +209,8 @@
inputCoder,
sideOutputTags,
outputCoders,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
}
this.pushbackFnRunner =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnRunnerWithKeyedInternals.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnRunnerWithKeyedInternals.java
index 595e7a9..6fb2bd3 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnRunnerWithKeyedInternals.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnRunnerWithKeyedInternals.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza.runtime;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.runners.core.DoFnRunner;
import org.apache.beam.runners.core.KeyedWorkItem;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
index 4748ebb..387ca9a 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
@@ -65,8 +65,8 @@
private final OutputManagerFactory<KV<K, OutputT>> outputManagerFactory;
private final Coder<K> keyCoder;
private final SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn;
- private final String stepName;
- private final String stepId;
+ private final String transformFullName;
+ private final String transformId;
private final IsBounded isBounded;
private transient StateInternalsFactory<K> stateInternalsFactory;
@@ -80,14 +80,14 @@
SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn,
WindowingStrategy<?, BoundedWindow> windowingStrategy,
OutputManagerFactory<KV<K, OutputT>> outputManagerFactory,
- String stepName,
- String stepId,
+ String transformFullName,
+ String transformId,
IsBounded isBounded) {
this.mainOutputTag = mainOutputTag;
this.windowingStrategy = windowingStrategy;
this.outputManagerFactory = outputManagerFactory;
- this.stepName = stepName;
- this.stepId = stepId;
+ this.transformFullName = transformFullName;
+ this.transformId = transformId;
this.isBounded = isBounded;
if (!(inputCoder instanceof KeyedWorkItemCoder)) {
@@ -115,13 +115,13 @@
final SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stepId, null, context.getTaskContext(), pipelineOptions, null);
+ transformId, null, context.getTaskContext(), pipelineOptions, null);
final DoFnRunners.OutputManager outputManager = outputManagerFactory.create(emitter);
this.stateInternalsFactory =
new SamzaStoreStateInternals.Factory<>(
- stepId,
+ transformId,
Collections.singletonMap(
SamzaStoreStateInternals.BEAM_STORE,
SamzaStoreStateInternals.getBeamStore(context.getTaskContext())),
@@ -176,12 +176,14 @@
null,
Collections.emptyMap(),
windowingStrategy,
- DoFnSchemaInformation.create());
+ DoFnSchemaInformation.create(),
+ Collections.emptyMap());
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
this.fnRunner =
- DoFnRunnerWithMetrics.wrap(doFnRunner, executionContext.getMetricsContainer(), stepName);
+ DoFnRunnerWithMetrics.wrap(
+ doFnRunner, executionContext.getMetricsContainer(), transformFullName);
}
@Override
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KeyedInternals.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KeyedInternals.java
index 4d0b4b6..d504929 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KeyedInternals.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KeyedInternals.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza.runtime;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.List;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaAssignContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaAssignContext.java
index 0364447..c0b8456 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaAssignContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaAssignContext.java
@@ -20,7 +20,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
class SamzaAssignContext<InT, W extends BoundedWindow> extends WindowFn<InT, W>.AssignContext {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
index 0aa95a8..49b4a28 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
@@ -45,10 +45,11 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.samza.context.Context;
import org.joda.time.Instant;
@@ -60,8 +61,8 @@
SamzaPipelineOptions pipelineOptions,
DoFn<InT, FnOutT> doFn,
WindowingStrategy<?, ?> windowingStrategy,
- String stepName,
- String stateId,
+ String transformFullName,
+ String transformId,
Context context,
TupleTag<FnOutT> mainOutputTag,
SideInputHandler sideInputHandler,
@@ -71,14 +72,15 @@
Coder<InT> inputCoder,
List<TupleTag<?>> sideOutputTags,
Map<TupleTag<?>, Coder<?>> outputCoders,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
final KeyedInternals keyedInternals;
final TimerInternals timerInternals;
final StateInternals stateInternals;
final DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
final SamzaStoreStateInternals.Factory<?> stateInternalsFactory =
SamzaStoreStateInternals.createStateInternalFactory(
- stateId, keyCoder, context.getTaskContext(), pipelineOptions, signature);
+ transformId, keyCoder, context.getTaskContext(), pipelineOptions, signature);
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
@@ -104,12 +106,13 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
final DoFnRunner<InT, FnOutT> doFnRunnerWithMetrics =
pipelineOptions.getEnableMetrics()
? DoFnRunnerWithMetrics.wrap(
- underlyingRunner, executionContext.getMetricsContainer(), stepName)
+ underlyingRunner, executionContext.getMetricsContainer(), transformFullName)
: underlyingRunner;
if (keyedInternals != null) {
@@ -165,14 +168,14 @@
TupleTag<FnOutT> mainOutputTag,
Map<String, TupleTag<?>> idToTupleTagMap,
Context context,
- String stepName) {
+ String transformFullName) {
final SamzaExecutionContext executionContext =
(SamzaExecutionContext) context.getApplicationContainerContext();
final DoFnRunner<InT, FnOutT> sdkHarnessDoFnRunner =
new SdkHarnessDoFnRunner<>(
outputManager, stageBundleFactory, mainOutputTag, idToTupleTagMap);
return DoFnRunnerWithMetrics.wrap(
- sdkHarnessDoFnRunner, executionContext.getMetricsContainer(), stepName);
+ sdkHarnessDoFnRunner, executionContext.getMetricsContainer(), transformFullName);
}
private static class SdkHarnessDoFnRunner<InT, FnOutT> implements DoFnRunner<InT, FnOutT> {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternals.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternals.java
index 5b266bf..3bd9834 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternals.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternals.java
@@ -61,9 +61,9 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
import org.apache.samza.context.TaskContext;
import org.apache.samza.storage.kv.Entry;
import org.apache.samza.storage.kv.KeyValueIterator;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
index 4394675..676129d 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
@@ -19,9 +19,14 @@
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
import java.util.NavigableSet;
import java.util.TreeSet;
import javax.annotation.Nullable;
@@ -32,8 +37,12 @@
import org.apache.beam.runners.core.TimerInternalsFactory;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.SamzaRunner;
-import org.apache.beam.runners.samza.state.SamzaSetState;
+import org.apache.beam.runners.samza.state.SamzaMapState;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.StructuredCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
@@ -55,7 +64,6 @@
private final NavigableSet<KeyedTimerData<K>> eventTimeTimers;
private final Coder<K> keyCoder;
private final Scheduler<KeyedTimerData<K>> timerRegistry;
- private final int timerBufferSize;
private final SamzaTimerState state;
private final IsBounded isBounded;
@@ -65,14 +73,12 @@
private SamzaTimerInternalsFactory(
Coder<K> keyCoder,
Scheduler<KeyedTimerData<K>> timerRegistry,
- int timerBufferSize,
String timerStateId,
SamzaStoreStateInternals.Factory<?> nonKeyedStateInternalsFactory,
Coder<BoundedWindow> windowCoder,
IsBounded isBounded) {
this.keyCoder = keyCoder;
this.timerRegistry = timerRegistry;
- this.timerBufferSize = timerBufferSize;
this.eventTimeTimers = new TreeSet<>();
this.state = new SamzaTimerState(timerStateId, nonKeyedStateInternalsFactory, windowCoder);
this.isBounded = isBounded;
@@ -92,7 +98,6 @@
return new SamzaTimerInternalsFactory<>(
keyCoder,
timerRegistry,
- pipelineOptions.getTimerBufferSize(),
timerStateId,
nonKeyedStateInternalsFactory,
windowCoder,
@@ -151,11 +156,6 @@
final KeyedTimerData<K> keyedTimerData = eventTimeTimers.pollFirst();
readyTimers.add(keyedTimerData);
state.deletePersisted(keyedTimerData);
-
- // if all the buffered timers are processed, load the next batch from state
- if (eventTimeTimers.isEmpty()) {
- state.loadEventTimeTimers();
- }
}
return readyTimers;
@@ -198,26 +198,41 @@
}
final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ if (eventTimeTimers.contains(keyedTimerData)) {
+ return;
+ }
- // persist it first
- state.persist(keyedTimerData);
+ final Long lastTimestamp = state.get(keyedTimerData);
+ final Long newTimestamp = timerData.getTimestamp().getMillis();
- switch (timerData.getDomain()) {
- case EVENT_TIME:
- eventTimeTimers.add(keyedTimerData);
- while (eventTimeTimers.size() > timerBufferSize) {
- eventTimeTimers.pollLast();
- }
- break;
+ if (!newTimestamp.equals(lastTimestamp)) {
+ if (lastTimestamp != null) {
+ final TimerData lastTimerData =
+ TimerData.of(
+ timerData.getTimerId(),
+ timerData.getNamespace(),
+ new Instant(lastTimestamp),
+ timerData.getDomain());
+ deleteTimer(lastTimerData, false);
+ }
- case PROCESSING_TIME:
- timerRegistry.schedule(keyedTimerData, timerData.getTimestamp().getMillis());
- break;
+ // persist it first
+ state.persist(keyedTimerData);
- default:
- throw new UnsupportedOperationException(
- String.format(
- "%s currently only supports even time or processing time", SamzaRunner.class));
+ switch (timerData.getDomain()) {
+ case EVENT_TIME:
+ eventTimeTimers.add(keyedTimerData);
+ break;
+
+ case PROCESSING_TIME:
+ timerRegistry.schedule(keyedTimerData, timerData.getTimestamp().getMillis());
+ break;
+
+ default:
+ throw new UnsupportedOperationException(
+ String.format(
+ "%s currently only supports even time or processing time", SamzaRunner.class));
+ }
}
}
@@ -233,9 +248,14 @@
@Override
public void deleteTimer(TimerData timerData) {
- final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ deleteTimer(timerData, true);
+ }
- state.deletePersisted(keyedTimerData);
+ private void deleteTimer(TimerData timerData, boolean updateState) {
+ final KeyedTimerData<K> keyedTimerData = new KeyedTimerData<>(keyBytes, key, timerData);
+ if (updateState) {
+ state.deletePersisted(keyedTimerData);
+ }
switch (timerData.getDomain()) {
case EVENT_TIME:
@@ -276,8 +296,8 @@
}
private class SamzaTimerState {
- private final SamzaSetState<KeyedTimerData<K>> eventTimerTimerState;
- private final SamzaSetState<KeyedTimerData<K>> processingTimerTimerState;
+ private final SamzaMapState<TimerKey<K>, Long> eventTimerTimerState;
+ private final SamzaMapState<TimerKey<K>, Long> processingTimerTimerState;
SamzaTimerState(
String timerStateId,
@@ -285,38 +305,56 @@
Coder<BoundedWindow> windowCoder) {
this.eventTimerTimerState =
- (SamzaSetState<KeyedTimerData<K>>)
+ (SamzaMapState<TimerKey<K>, Long>)
nonKeyedStateInternalsFactory
.stateInternalsForKey(null)
.state(
StateNamespaces.global(),
- StateTags.set(
+ StateTags.map(
timerStateId + "-et",
- new KeyedTimerData.KeyedTimerDataCoder<>(keyCoder, windowCoder)));
+ new TimerKeyCoder<>(keyCoder, windowCoder),
+ VarLongCoder.of()));
this.processingTimerTimerState =
- (SamzaSetState<KeyedTimerData<K>>)
+ (SamzaMapState<TimerKey<K>, Long>)
nonKeyedStateInternalsFactory
.stateInternalsForKey(null)
.state(
StateNamespaces.global(),
- StateTags.set(
+ StateTags.map(
timerStateId + "-pt",
- new KeyedTimerData.KeyedTimerDataCoder<>(keyCoder, windowCoder)));
+ new TimerKeyCoder<>(keyCoder, windowCoder),
+ VarLongCoder.of()));
restore();
}
- void persist(KeyedTimerData<K> keyedTimerData) {
+ Long get(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
switch (keyedTimerData.getTimerData().getDomain()) {
case EVENT_TIME:
- if (!eventTimeTimers.contains(keyedTimerData)) {
- eventTimerTimerState.add(keyedTimerData);
- }
+ return eventTimerTimerState.get(timerKey).read();
+
+ case PROCESSING_TIME:
+ return processingTimerTimerState.get(timerKey).read();
+
+ default:
+ throw new UnsupportedOperationException(
+ String.format("%s currently only supports event time", SamzaRunner.class));
+ }
+ }
+
+ void persist(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
+ switch (keyedTimerData.getTimerData().getDomain()) {
+ case EVENT_TIME:
+ eventTimerTimerState.put(
+ timerKey, keyedTimerData.getTimerData().getTimestamp().getMillis());
break;
case PROCESSING_TIME:
- processingTimerTimerState.add(keyedTimerData);
+ processingTimerTimerState.put(
+ timerKey, keyedTimerData.getTimerData().getTimestamp().getMillis());
break;
default:
@@ -326,13 +364,14 @@
}
void deletePersisted(KeyedTimerData<K> keyedTimerData) {
+ final TimerKey<K> timerKey = TimerKey.of(keyedTimerData);
switch (keyedTimerData.getTimerData().getDomain()) {
case EVENT_TIME:
- eventTimerTimerState.remove(keyedTimerData);
+ eventTimerTimerState.remove(timerKey);
break;
case PROCESSING_TIME:
- processingTimerTimerState.remove(keyedTimerData);
+ processingTimerTimerState.remove(timerKey);
break;
default:
@@ -342,37 +381,38 @@
}
private void loadEventTimeTimers() {
- if (!eventTimerTimerState.isEmpty().read()) {
- final Iterator<KeyedTimerData<K>> iter = eventTimerTimerState.readIterator().read();
- int i = 0;
- for (; i < timerBufferSize && iter.hasNext(); i++) {
- eventTimeTimers.add(iter.next());
- }
+ final Iterator<Map.Entry<TimerKey<K>, Long>> iter =
+ eventTimerTimerState.readIterator().read();
+ // since the iterator will reach to the end, it will be closed automatically
+ while (iter.hasNext()) {
+ final Map.Entry<TimerKey<K>, Long> entry = iter.next();
+ final KeyedTimerData keyedTimerData =
+ TimerKey.toKeyedTimerData(
+ entry.getKey(), entry.getValue(), TimeDomain.EVENT_TIME, keyCoder);
- LOG.info("Loaded {} event time timers in memory", i);
-
- // manually close the iterator here
- final SamzaStoreStateInternals.KeyValueIteratorState iteratorState =
- (SamzaStoreStateInternals.KeyValueIteratorState) eventTimerTimerState;
-
- iteratorState.closeIterators();
+ eventTimeTimers.add(keyedTimerData);
}
+
+ LOG.info("Loaded {} event time timers in memory", eventTimeTimers.size());
}
private void loadProcessingTimeTimers() {
- if (!processingTimerTimerState.isEmpty().read()) {
- final Iterator<KeyedTimerData<K>> iter = processingTimerTimerState.readIterator().read();
- // since the iterator will reach to the end, it will be closed automatically
- int count = 0;
- while (iter.hasNext()) {
- final KeyedTimerData<K> keyedTimerData = iter.next();
- timerRegistry.schedule(
- keyedTimerData, keyedTimerData.getTimerData().getTimestamp().getMillis());
- ++count;
- }
+ final Iterator<Map.Entry<TimerKey<K>, Long>> iter =
+ processingTimerTimerState.readIterator().read();
+ // since the iterator will reach to the end, it will be closed automatically
+ int count = 0;
+ while (iter.hasNext()) {
+ final Map.Entry<TimerKey<K>, Long> entry = iter.next();
+ final KeyedTimerData keyedTimerData =
+ TimerKey.toKeyedTimerData(
+ entry.getKey(), entry.getValue(), TimeDomain.PROCESSING_TIME, keyCoder);
- LOG.info("Loaded {} processing time timers in memory", count);
+ timerRegistry.schedule(
+ keyedTimerData, keyedTimerData.getTimerData().getTimestamp().getMillis());
+ ++count;
}
+
+ LOG.info("Loaded {} processing time timers in memory", count);
}
private void restore() {
@@ -380,4 +420,146 @@
loadProcessingTimeTimers();
}
}
+
+ private static class TimerKey<K> {
+ private final K key;
+ private final StateNamespace stateNamespace;
+ private final String timerId;
+
+ static <K> TimerKey<K> of(KeyedTimerData<K> keyedTimerData) {
+ final TimerInternals.TimerData timerData = keyedTimerData.getTimerData();
+ return new TimerKey<>(
+ keyedTimerData.getKey(), timerData.getNamespace(), timerData.getTimerId());
+ }
+
+ static <K> KeyedTimerData<K> toKeyedTimerData(
+ TimerKey<K> timerKey, long timestamp, TimeDomain domain, Coder<K> keyCoder) {
+ byte[] keyBytes = null;
+ if (keyCoder != null && timerKey.key != null) {
+ final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ try {
+ keyCoder.encode(timerKey.key, baos);
+ } catch (IOException e) {
+ throw new RuntimeException("Could not encode key: " + timerKey.key, e);
+ }
+ keyBytes = baos.toByteArray();
+ }
+
+ return new KeyedTimerData<K>(
+ keyBytes,
+ timerKey.key,
+ TimerInternals.TimerData.of(
+ timerKey.timerId, timerKey.stateNamespace, new Instant(timestamp), domain));
+ }
+
+ private TimerKey(K key, StateNamespace stateNamespace, String timerId) {
+ this.key = key;
+ this.stateNamespace = stateNamespace;
+ this.timerId = timerId;
+ }
+
+ public K getKey() {
+ return key;
+ }
+
+ public StateNamespace getStateNamespace() {
+ return stateNamespace;
+ }
+
+ public String getTimerId() {
+ return timerId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ TimerKey<?> timerKey = (TimerKey<?>) o;
+
+ if (key != null ? !key.equals(timerKey.key) : timerKey.key != null) {
+ return false;
+ }
+ if (!stateNamespace.equals(timerKey.stateNamespace)) {
+ return false;
+ }
+
+ return timerId.equals(timerKey.timerId);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = key != null ? key.hashCode() : 0;
+ result = 31 * result + stateNamespace.hashCode();
+ result = 31 * result + timerId.hashCode();
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "TimerKey{"
+ + "key="
+ + key
+ + ", stateNamespace="
+ + stateNamespace
+ + ", timerId='"
+ + timerId
+ + '\''
+ + '}';
+ }
+ }
+
+ /** Coder for {@link TimerKey}. */
+ public static class TimerKeyCoder<K> extends StructuredCoder<TimerKey<K>> {
+ private static final StringUtf8Coder STRING_CODER = StringUtf8Coder.of();
+
+ private final Coder<K> keyCoder;
+ private final Coder<? extends BoundedWindow> windowCoder;
+
+ TimerKeyCoder(Coder<K> keyCoder, Coder<? extends BoundedWindow> windowCoder) {
+ this.keyCoder = keyCoder;
+ this.windowCoder = windowCoder;
+ }
+
+ @Override
+ public void encode(TimerKey<K> value, OutputStream outStream)
+ throws CoderException, IOException {
+
+ // encode the timestamp first
+ STRING_CODER.encode(value.timerId, outStream);
+ STRING_CODER.encode(value.stateNamespace.stringKey(), outStream);
+
+ if (keyCoder != null) {
+ keyCoder.encode(value.key, outStream);
+ }
+ }
+
+ @Override
+ public TimerKey<K> decode(InputStream inStream) throws CoderException, IOException {
+ // decode the timestamp first
+ final String timerId = STRING_CODER.decode(inStream);
+ // The namespace needs two-phase deserialization:
+ // first from bytes into a string, then from string to namespace object using windowCoder.
+ final StateNamespace namespace =
+ StateNamespaces.fromString(STRING_CODER.decode(inStream), windowCoder);
+ K key = null;
+ if (keyCoder != null) {
+ key = keyCoder.decode(inStream);
+ }
+
+ return new TimerKey<>(key, namespace, timerId);
+ }
+
+ @Override
+ public List<? extends Coder<?>> getCoderArguments() {
+ return Arrays.asList(keyCoder, windowCoder);
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {}
+ }
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigBuilder.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigBuilder.java
index 975baa2..accdd12 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigBuilder.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigBuilder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.File;
import java.net.URI;
@@ -29,8 +29,8 @@
import org.apache.beam.runners.samza.SamzaExecutionEnvironment;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.container.BeamContainerRunner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.commons.lang3.StringUtils;
import org.apache.samza.config.ApplicationConfig;
import org.apache.samza.config.Config;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigContext.java
index effcaf8..ee9d9ea 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigContext.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.runners.TransformHierarchy;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Helper that provides context data such as output for config generation. */
public class ConfigContext {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/FlattenPCollectionsTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/FlattenPCollectionsTranslator.java
index e72e601..9e21614 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/FlattenPCollectionsTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/FlattenPCollectionsTranslator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.HashSet;
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.samza.operators.MessageStream;
/**
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
index 0af0df9..a14ac4c 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.runners.samza.translation;
+import static org.apache.beam.runners.samza.util.SamzaPipelineTranslatorUtils.escape;
+
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.KeyedWorkItem;
import org.apache.beam.runners.core.KeyedWorkItemCoder;
@@ -46,7 +48,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.samza.operators.MessageStream;
import org.apache.samza.serializers.KVSerde;
@@ -92,8 +94,8 @@
windowingStrategy,
kvInputCoder,
elementCoder,
- ctx.getCurrentTopologicalId(),
- node.getFullName(),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
outputTag,
input.isBounded());
@@ -126,8 +128,6 @@
final Coder<WindowedValue<KV<K, InputT>>> elementCoder =
WindowedValue.FullWindowedValueCoder.of(kvInputCoder, windowCoder);
- final int topologyId = ctx.getCurrentTopologicalId();
- final String nodeFullname = transform.getTransform().getUniqueName();
final TupleTag<KV<K, OutputT>> outputTag =
new TupleTag<>(Iterables.getOnlyElement(transform.getTransform().getOutputsMap().keySet()));
@@ -147,8 +147,8 @@
windowingStrategy,
kvInputCoder,
elementCoder,
- topologyId,
- nodeFullname,
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
outputTag,
isBounded);
ctx.registerMessageStream(ctx.getOutputId(transform), outputStream);
@@ -161,8 +161,8 @@
WindowingStrategy<?, BoundedWindow> windowingStrategy,
KvCoder<K, InputT> kvInputCoder,
Coder<WindowedValue<KV<K, InputT>>> elementCoder,
- int topologyId,
- String nodeFullname,
+ String transformFullName,
+ String transformId,
TupleTag<KV<K, OutputT>> outputTag,
PCollection.IsBounded isBounded) {
final MessageStream<OpMessage<KV<K, InputT>>> filteredInputStream =
@@ -180,8 +180,7 @@
KVSerde.of(
SamzaCoders.toSerde(kvInputCoder.getKeyCoder()),
SamzaCoders.toSerde(elementCoder)),
- // TODO: infer a fixed id from the name
- "gbk-" + topologyId)
+ "gbk-" + escape(transformId))
.map(kv -> OpMessage.ofElement(kv.getValue()));
}
@@ -202,9 +201,8 @@
reduceFn,
windowingStrategy,
new DoFnOp.SingleOutputManagerFactory<>(),
- nodeFullname,
- // TODO: infer a fixed id from the name
- outputTag.getId(),
+ transformFullName,
+ transformId,
isBounded)));
return outputStream;
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
index ad699af..f00c34b 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
@@ -54,7 +54,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.apache.samza.operators.MessageStream;
import org.apache.samza.operators.functions.FlatMapFunction;
import org.apache.samza.operators.functions.WatermarkFunction;
@@ -106,7 +106,7 @@
final MessageStream<OpMessage<InT>> inputStream = ctx.getMessageStream(input);
final List<MessageStream<OpMessage<InT>>> sideInputStreams =
- transform.getSideInputs().stream()
+ transform.getSideInputs().values().stream()
.map(ctx::<InT>getViewStream)
.collect(Collectors.toList());
final ArrayList<Map.Entry<TupleTag<?>, PValue>> outputs =
@@ -128,13 +128,16 @@
}
final HashMap<String, PCollectionView<?>> idToPValueMap = new HashMap<>();
- for (PCollectionView<?> view : transform.getSideInputs()) {
+ for (PCollectionView<?> view : transform.getSideInputs().values()) {
idToPValueMap.put(ctx.getViewId(view), view);
}
DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(ctx.getCurrentTransform());
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(ctx.getCurrentTransform());
+
final DoFnOp<InT, OutT, RawUnionValue> op =
new DoFnOp<>(
transform.getMainOutputTag(),
@@ -142,19 +145,19 @@
keyCoder,
(Coder<InT>) input.getCoder(),
outputCoders,
- transform.getSideInputs(),
+ transform.getSideInputs().values(),
transform.getAdditionalOutputTags().getAll(),
input.getWindowingStrategy(),
idToPValueMap,
new DoFnOp.MultiOutputManagerFactory(tagToIndexMap),
- node.getFullName(),
- // TODO: infer a fixed id from the name
- String.valueOf(ctx.getCurrentTopologicalId()),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
input.isBounded(),
false,
null,
Collections.emptyMap(),
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
final MessageStream<OpMessage<InT>> mergedStreams;
if (sideInputStreams.isEmpty()) {
@@ -234,12 +237,14 @@
});
WindowedValue.WindowedValueCoder<InT> windowedInputCoder =
- SamzaPipelineTranslatorUtils.instantiateCoder(inputId, pipeline.getComponents());
- final String nodeFullname = transform.getTransform().getUniqueName();
+ ctx.instantiateCoder(inputId, pipeline.getComponents());
final DoFnSchemaInformation doFnSchemaInformation;
doFnSchemaInformation = ParDoTranslation.getSchemaInformation(transform.getTransform());
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(transform.getTransform());
+
final RunnerApi.PCollection input = pipeline.getComponents().getPcollectionsOrThrow(inputId);
final PCollection.IsBounded isBounded = SamzaPipelineTranslatorUtils.isBounded(input);
@@ -255,14 +260,14 @@
SamzaPipelineTranslatorUtils.getPortableWindowStrategy(transform, pipeline),
Collections.emptyMap(), // idToViewMap not in use until side input support
new DoFnOp.MultiOutputManagerFactory(tagToIndexMap),
- nodeFullname,
- // TODO: infer a fixed id from the name
- String.valueOf(ctx.getCurrentTopologicalId()),
+ ctx.getTransformFullName(),
+ ctx.getTransformId(),
isBounded,
true,
stagePayload,
idToTupleTagMap,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
final MessageStream<OpMessage<InT>> mergedStreams;
if (sideInputStreams.isEmpty()) {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
index 93a28b7..c40913d 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
@@ -33,16 +33,19 @@
import org.apache.beam.runners.fnexecution.wire.WireCoders;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.runtime.OpMessage;
+import org.apache.beam.runners.samza.util.HashIdGenerator;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.samza.application.descriptors.StreamApplicationDescriptor;
import org.apache.samza.operators.KV;
import org.apache.samza.operators.MessageStream;
import org.apache.samza.operators.OutputStream;
import org.apache.samza.system.descriptors.InputDescriptor;
import org.apache.samza.system.descriptors.OutputDescriptor;
+import org.apache.samza.table.Table;
+import org.apache.samza.table.descriptors.TableDescriptor;
/**
* Helper that keeps the mapping from BEAM PCollection id to Samza {@link MessageStream}. It also
@@ -53,8 +56,11 @@
private final Map<String, MessageStream<?>> messsageStreams = new HashMap<>();
private final StreamApplicationDescriptor appDescriptor;
private final SamzaPipelineOptions options;
- private int topologicalId;
private final Set<String> registeredInputStreams = new HashSet<>();
+ private final Map<String, Table> registeredTables = new HashMap<>();
+ private final HashIdGenerator idGenerator = new HashIdGenerator();
+
+ private PipelineNode.PTransformNode currentTransform;
public PortableTranslationContext(
StreamApplicationDescriptor appDescriptor, SamzaPipelineOptions options) {
@@ -66,14 +72,6 @@
return this.options;
}
- public void setCurrentTopologicalId(int id) {
- this.topologicalId = id;
- }
-
- public int getCurrentTopologicalId() {
- return this.topologicalId;
- }
-
public <T> List<MessageStream<OpMessage<T>>> getAllInputMessageStreams(
PipelineNode.PTransformNode transform) {
final Collection<String> inputStreamIds = transform.getTransform().getInputsMap().values();
@@ -166,4 +164,26 @@
(WindowingStrategy<?, BoundedWindow>) windowingStrategy;
return ret;
}
+
+ @SuppressWarnings("unchecked")
+ public <K, V> Table<KV<K, V>> getTable(TableDescriptor<K, V, ?> tableDesc) {
+ return registeredTables.computeIfAbsent(
+ tableDesc.getTableId(), id -> appDescriptor.getTable(tableDesc));
+ }
+
+ public void setCurrentTransform(PipelineNode.PTransformNode currentTransform) {
+ this.currentTransform = currentTransform;
+ }
+
+ public void clearCurrentTransform() {
+ this.currentTransform = null;
+ }
+
+ public String getTransformFullName() {
+ return currentTransform.getTransform().getUniqueName();
+ }
+
+ public String getTransformId() {
+ return idGenerator.getId(currentTransform.getTransform().getUniqueName());
+ }
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ReadTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ReadTranslator.java
index c5dcd9d..3c1dbb6 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ReadTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ReadTranslator.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.samza.operators.KV;
import org.apache.samza.serializers.KVSerde;
import org.apache.samza.serializers.NoOpSerde;
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
index f2fe9fa..c30b181 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.samza.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.util.HashMap;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -55,7 +55,6 @@
public static void translate(Pipeline pipeline, TranslationContext ctx) {
final TransformVisitorFn translateFn =
new TransformVisitorFn() {
- private int topologicalId = 0;
@Override
public <T extends PTransform<?, ?>> void apply(
@@ -64,7 +63,6 @@
Pipeline pipeline,
TransformTranslator<T> translator) {
ctx.setCurrentTransform(node.toAppliedPTransform(pipeline));
- ctx.setCurrentTopologicalId(topologicalId++);
translator.translate(transform, node, ctx);
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
index 5974cb6..372b362 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
@@ -24,7 +24,7 @@
import org.apache.beam.runners.core.construction.graph.PipelineNode;
import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -52,14 +52,17 @@
QueryablePipeline queryablePipeline =
QueryablePipeline.forTransforms(
pipeline.getRootTransformIdsList(), pipeline.getComponents());
- int topologicalId = 0;
+
for (PipelineNode.PTransformNode transform :
queryablePipeline.getTopologicallyOrderedTransforms()) {
- ctx.setCurrentTopologicalId(topologicalId++);
+ ctx.setCurrentTransform(transform);
+
LOG.info("Translating transform urn: {}", transform.getTransform().getSpec().getUrn());
TRANSLATORS
.get(transform.getTransform().getSpec().getUrn())
.translatePortable(transform, queryablePipeline, ctx);
+
+ ctx.clearCurrentTransform();
}
}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTransformOverride.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTransformOverride.java
index 5181588..8644e73 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTransformOverride.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTransformOverride.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Samza override for {@link View} (side input) transforms. */
class SamzaPublishViewTransformOverride<ElemT, ViewT>
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
index 76612e8..308be26 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
@@ -58,7 +58,7 @@
ctx.getPipelineOptions().getMaxSourceParallelism() == 1
? elementStream
: elementStream.broadcast(
- SamzaCoders.toSerde(elementCoder), "view-" + ctx.getCurrentTopologicalId());
+ SamzaCoders.toSerde(elementCoder), "view-" + ctx.getTransformId());
final String viewId = ctx.getViewId(transform.getView());
final MessageStream<OpMessage<Iterable<ElemT>>> outputStream =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTransformOverrides.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTransformOverrides.java
index 944c4c2..b91f7b3 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTransformOverrides.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTransformOverrides.java
@@ -24,7 +24,7 @@
import org.apache.beam.runners.core.construction.SplittableParDoNaiveBounded;
import org.apache.beam.runners.core.construction.UnsupportedOverrideFactory;
import org.apache.beam.sdk.runners.PTransformOverride;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link org.apache.beam.sdk.transforms.PTransform} overrides for Samza runner. */
public class SamzaTransformOverrides {
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
index 4120ef0..a633382 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
@@ -24,6 +24,7 @@
import org.apache.beam.runners.core.construction.TransformInputs;
import org.apache.beam.runners.samza.SamzaPipelineOptions;
import org.apache.beam.runners.samza.runtime.OpMessage;
+import org.apache.beam.runners.samza.util.HashIdGenerator;
import org.apache.beam.sdk.runners.AppliedPTransform;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -32,7 +33,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.samza.application.descriptors.StreamApplicationDescriptor;
import org.apache.samza.config.Config;
import org.apache.samza.config.MapConfig;
@@ -73,9 +74,9 @@
private final Map<String, MessageStream> registeredInputStreams = new HashMap<>();
private final Map<String, Table> registeredTables = new HashMap<>();
private final SamzaPipelineOptions options;
+ private final HashIdGenerator idGenerator = new HashIdGenerator();
private AppliedPTransform<?, ?, ?> currentTransform;
- private int topologicalId;
public TranslationContext(
StreamApplicationDescriptor appDescriptor,
@@ -168,20 +169,6 @@
return currentTransform;
}
- /**
- * Uniquely identify a node when doing a topological traversal of the BEAM {@link
- * org.apache.beam.sdk.Pipeline}. It's changed on a per-node basis.
- *
- * @param id id for the node.
- */
- public void setCurrentTopologicalId(int id) {
- this.topologicalId = id;
- }
-
- public int getCurrentTopologicalId() {
- return this.topologicalId;
- }
-
@SuppressWarnings("unchecked")
public <InT extends PValue> InT getInput(PTransform<InT, ?> transform) {
return (InT)
@@ -225,6 +212,14 @@
return id;
}
+ public String getTransformFullName() {
+ return currentTransform.getFullName();
+ }
+
+ public String getTransformId() {
+ return idGenerator.getId(currentTransform.getFullName());
+ }
+
/** The dummy stream created will only be used in Beam tests. */
private static InputDescriptor<OpMessage<String>, ?> createDummyStreamDescriptor(String id) {
final GenericSystemDescriptor dummySystem =
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/WindowAssignTranslator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/WindowAssignTranslator.java
index f884047..95c7328 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/WindowAssignTranslator.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/translation/WindowAssignTranslator.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
import org.apache.samza.operators.MessageStream;
/**
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
new file mode 100644
index 0000000..ecf2bc6
--- /dev/null
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza.util;
+
+import java.util.HashSet;
+import java.util.Set;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class generates hash-based unique ids from String. The id length is the hash length and the
+ * suffix length combined. Ids generated are guaranteed to be unique, such that same names will be
+ * hashed to different ids.
+ */
+public class HashIdGenerator {
+ private static final Logger LOG = LoggerFactory.getLogger(HashIdGenerator.class);
+
+ private static final int DEFAULT_MAX_HASH_LENGTH = 5;
+ private final int maxHashLength;
+ private final Set<String> usedIds = new HashSet<>();
+
+ public HashIdGenerator(int maxHashLength) {
+ this.maxHashLength = maxHashLength;
+ }
+
+ public HashIdGenerator() {
+ this(DEFAULT_MAX_HASH_LENGTH);
+ }
+
+ public String getId(String name) {
+ // Use the id directly if it is unique and the length is less than max
+ if (name.length() <= maxHashLength && usedIds.add(name)) {
+ return name;
+ }
+
+ // Pick the last bytes of hashcode and use hex format
+ final String hexString = Integer.toHexString(name.hashCode());
+ final String origId =
+ hexString.length() <= maxHashLength
+ ? hexString
+ : hexString.substring(Math.max(0, hexString.length() - maxHashLength));
+ String id = origId;
+ int suffixNum = 2;
+ while (!usedIds.add(id)) {
+ // A duplicate! Retry.
+ id = origId + "-" + suffixNum++;
+ }
+ LOG.info("Name {} is mapped to id {}", name, id);
+ return id;
+ }
+}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/PipelineDotRenderer.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/PipelineDotRenderer.java
deleted file mode 100644
index 25cd44a..0000000
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/PipelineDotRenderer.java
+++ /dev/null
@@ -1,128 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.samza.util;
-
-import java.util.HashMap;
-import java.util.Map;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.runners.TransformHierarchy;
-import org.apache.beam.sdk.values.PValue;
-
-/** A DOT renderer for BEAM {@link Pipeline} DAG. */
-public class PipelineDotRenderer implements Pipeline.PipelineVisitor {
- public static String toDotString(Pipeline pipeline) {
- final PipelineDotRenderer visitor = new PipelineDotRenderer();
- visitor.begin();
- pipeline.traverseTopologically(visitor);
- visitor.end();
- return visitor.dotBuilder.toString();
- }
-
- private final StringBuilder dotBuilder = new StringBuilder();
- private final Map<TransformHierarchy.Node, Integer> nodeToId = new HashMap<>();
- private final Map<PValue, Integer> valueToProducerNodeId = new HashMap<>();
-
- private int indent;
- private int nextNodeId;
-
- private PipelineDotRenderer() {}
-
- @Override
- public void enterPipeline(Pipeline p) {}
-
- @Override
- public void leavePipeline(Pipeline pipeline) {}
-
- @Override
- public CompositeBehavior enterCompositeTransform(TransformHierarchy.Node node) {
- writeLine("subgraph cluster_%d {", nextNodeId++);
- enterBlock();
- writeLine("label = \"%s\"", escapeString(node.getFullName()));
- return CompositeBehavior.ENTER_TRANSFORM;
- }
-
- @Override
- public void leaveCompositeTransform(TransformHierarchy.Node node) {
- exitBlock();
- writeLine("}");
- }
-
- @Override
- public void visitPrimitiveTransform(TransformHierarchy.Node node) {
- final int nodeId = nextNodeId++;
- writeLine("%d [label=\"%s\"]", nodeId, escapeString(node.getTransform().getName()));
-
- node.getOutputs()
- .values()
- .forEach(
- x -> {
- valueToProducerNodeId.put(x, nodeId);
- });
-
- node.getInputs()
- .forEach(
- (key, value) -> {
- final int producerId = valueToProducerNodeId.get(value);
- String style = "solid";
- if (node.getTransform().getAdditionalInputs().containsKey(key)) {
- style = "dashed";
- }
- writeLine(
- "%d -> %d [style=%s label=\"%s\"]",
- producerId, nodeId, style, escapeString(shortenTag(key.getId())));
- });
- }
-
- @Override
- public void visitValue(PValue value, TransformHierarchy.Node producer) {}
-
- private void begin() {
- writeLine("digraph {");
- enterBlock();
- writeLine("rankdir=LR");
- }
-
- private void end() {
- exitBlock();
- writeLine("}");
- }
-
- private void enterBlock() {
- indent += 4;
- }
-
- private void exitBlock() {
- indent -= 4;
- }
-
- private void writeLine(String format, Object... args) {
- if (indent != 0) {
- dotBuilder.append(String.format("%-" + indent + "s", ""));
- }
- dotBuilder.append(String.format(format, args));
- dotBuilder.append("\n");
- }
-
- private static String escapeString(String x) {
- return x.replace("\"", "\\\"");
- }
-
- private static String shortenTag(String tag) {
- return tag.replaceFirst(".*:([a-zA-Z#0-9]+).*", "$1");
- }
-}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/PortablePipelineDotRenderer.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/PortablePipelineDotRenderer.java
deleted file mode 100644
index 6146796..0000000
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/PortablePipelineDotRenderer.java
+++ /dev/null
@@ -1,124 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.samza.util;
-
-import java.util.HashMap;
-import java.util.Map;
-import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.runners.core.construction.graph.PipelineNode;
-import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
-
-/**
- * A DOT renderer for BEAM portable {@link org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline}.
- */
-public class PortablePipelineDotRenderer {
- private final StringBuilder dotBuilder = new StringBuilder();
- private final Map<String, Integer> valueToProducerNodeId = new HashMap<>();
- private int indent;
- private int nextNodeId;
-
- public static String toDotString(RunnerApi.Pipeline pipeline) {
- final PortablePipelineDotRenderer renderer = new PortablePipelineDotRenderer();
- return renderer.toDot(pipeline);
- }
-
- private PortablePipelineDotRenderer() {}
-
- private String toDot(RunnerApi.Pipeline pipeline) {
- final QueryablePipeline p =
- QueryablePipeline.forTransforms(
- pipeline.getRootTransformIdsList(), pipeline.getComponents());
-
- begin();
-
- for (PipelineNode.PTransformNode transform : p.getTopologicallyOrderedTransforms()) {
- visitTransform(transform);
- }
-
- end();
-
- return dotBuilder.toString();
- }
-
- private void visitTransform(PipelineNode.PTransformNode node) {
- final int nodeId = nextNodeId++;
- final RunnerApi.PTransform transform = node.getTransform();
- writeLine(
- "%d [label=\"%s\\n%s\"]",
- nodeId,
- escapeString(transform.getUniqueName()),
- escapeString(transform.getSpec().getUrn()));
-
- transform
- .getOutputs()
- .values()
- .forEach(
- x -> {
- valueToProducerNodeId.put(x, nodeId);
- });
-
- transform
- .getInputs()
- .forEach(
- (key, value) -> {
- final int producerId = valueToProducerNodeId.get(value);
- String style = "solid";
- writeLine(
- "%d -> %d [style=%s label=\"%s\"]",
- producerId,
- nodeId,
- style,
- escapeString(value.substring(value.lastIndexOf('_') + 1)));
- });
- }
-
- private void begin() {
- writeLine("digraph {");
- enterBlock();
- writeLine("rankdir=LR");
- }
-
- private void end() {
- exitBlock();
- writeLine("}");
- }
-
- private void enterBlock() {
- indent += 4;
- }
-
- private void exitBlock() {
- indent -= 4;
- }
-
- private void writeLine(String format, Object... args) {
- if (indent != 0) {
- dotBuilder.append(String.format("%-" + indent + "s", ""));
- }
- dotBuilder.append(String.format(format, args));
- dotBuilder.append("\n");
- }
-
- private static String escapeString(String x) {
- return x.replace("\"", "\\\"");
- }
-
- private static String shortenTag(String tag) {
- return tag.replaceFirst(".*:([a-zA-Z#0-9]+).*", "$1");
- }
-}
diff --git a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaPipelineTranslatorUtils.java b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaPipelineTranslatorUtils.java
index 50cccfe..758515a4 100644
--- a/runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaPipelineTranslatorUtils.java
+++ b/runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaPipelineTranslatorUtils.java
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Utilities for pipeline translation. */
public final class SamzaPipelineTranslatorUtils {
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystemTest.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystemTest.java
index 964f0fe..ca31f86 100644
--- a/runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystemTest.java
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystemTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.samza.Partition;
import org.apache.samza.metrics.MetricsRegistryMap;
import org.apache.samza.system.IncomingMessageEnvelope;
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternalsTest.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternalsTest.java
index f75a758..97ae705 100644
--- a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternalsTest.java
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternalsTest.java
@@ -53,9 +53,9 @@
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.samza.context.ContainerContext;
import org.apache.samza.context.JobContext;
import org.apache.samza.metrics.MetricsRegistry;
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
index 9328676..27d8ba8 100644
--- a/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
@@ -27,6 +27,7 @@
import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.apache.beam.runners.core.StateNamespace;
@@ -119,7 +120,6 @@
public void testEventTimeTimers() {
final SamzaPipelineOptions pipelineOptions =
PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
- pipelineOptions.setTimerBufferSize(1);
final RocksDbKeyValueStore store = createStore("store1");
final SamzaTimerInternalsFactory<String> timerInternalsFactory =
@@ -153,17 +153,17 @@
}
@Test
- public void testRestore() {
+ public void testRestore() throws Exception {
final SamzaPipelineOptions pipelineOptions =
PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
- pipelineOptions.setTimerBufferSize(1);
RocksDbKeyValueStore store = createStore("store2");
final SamzaTimerInternalsFactory<String> timerInternalsFactory =
createTimerInternalsFactory(null, "timer", pipelineOptions, store);
+ final String key = "testKey";
final StateNamespace nameSpace = StateNamespaces.global();
- final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey("testKey");
+ final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey(key);
final TimerInternals.TimerData timer1 =
TimerInternals.TimerData.of("timer1", nameSpace, new Instant(10), TimeDomain.EVENT_TIME);
timerInternals.setTimer(timer1);
@@ -183,6 +183,15 @@
Collection<KeyedTimerData<String>> readyTimers = restoredFactory.removeReadyTimers();
assertEquals(2, readyTimers.size());
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ StringUtf8Coder.of().encode(key, baos);
+ byte[] keyBytes = baos.toByteArray();
+ assertEquals(
+ readyTimers,
+ Arrays.asList(
+ new KeyedTimerData<>(keyBytes, key, timer1),
+ new KeyedTimerData<>(keyBytes, key, timer2)));
+
store.close();
}
@@ -229,4 +238,44 @@
store.close();
}
+
+ @Test
+ public void testOverride() {
+ final SamzaPipelineOptions pipelineOptions =
+ PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
+
+ RocksDbKeyValueStore store = createStore("store4");
+ final SamzaTimerInternalsFactory<String> timerInternalsFactory =
+ createTimerInternalsFactory(null, "timer", pipelineOptions, store);
+
+ final StateNamespace nameSpace = StateNamespaces.global();
+ final TimerInternals timerInternals = timerInternalsFactory.timerInternalsForKey("testKey");
+ final TimerInternals.TimerData timer1 =
+ TimerInternals.TimerData.of("timerId", nameSpace, new Instant(10), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer1);
+
+ // this timer should override the first timer
+ final TimerInternals.TimerData timer2 =
+ TimerInternals.TimerData.of("timerId", nameSpace, new Instant(100), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer2);
+
+ final TimerInternals.TimerData timer3 =
+ TimerInternals.TimerData.of("timerId2", nameSpace, new Instant(200), TimeDomain.EVENT_TIME);
+ timerInternals.setTimer(timer3);
+
+ // this timer shouldn't override since it has a different id
+ timerInternalsFactory.setInputWatermark(new Instant(50));
+ Collection<KeyedTimerData<String>> readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(0, readyTimers.size());
+
+ timerInternalsFactory.setInputWatermark(new Instant(150));
+ readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(1, readyTimers.size());
+
+ timerInternalsFactory.setInputWatermark(new Instant(250));
+ readyTimers = timerInternalsFactory.removeReadyTimers();
+ assertEquals(1, readyTimers.size());
+
+ store.close();
+ }
}
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/translation/ConfigGeneratorTest.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/translation/ConfigGeneratorTest.java
index daee3b1..f71585d 100644
--- a/runners/samza/src/test/java/org/apache/beam/runners/samza/translation/ConfigGeneratorTest.java
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/translation/ConfigGeneratorTest.java
@@ -37,8 +37,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.samza.config.Config;
import org.apache.samza.config.JobCoordinatorConfig;
import org.apache.samza.config.ZkConfig;
diff --git a/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java b/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
new file mode 100644
index 0000000..881ce7f
--- /dev/null
+++ b/runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.samza.util;
+
+import static org.mockito.Mockito.mock;
+
+import java.util.Set;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.Max;
+import org.apache.beam.sdk.transforms.Min;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.junit.Assert;
+import org.junit.Test;
+
+/** Test class for {@link HashIdGenerator}. */
+public class TestHashIdGenerator {
+
+ @Test
+ public void testGetId() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ final Set<String> ids =
+ ImmutableSet.of(
+ idGenerator.getId(Count.perKey().getName()),
+ idGenerator.getId(MapElements.into(null).getName()),
+ idGenerator.getId(Count.globally().getName()),
+ idGenerator.getId(Combine.perKey(mock(SerializableFunction.class)).getName()),
+ idGenerator.getId(Min.perKey().getName()),
+ idGenerator.getId(Max.globally().getName()));
+ Assert.assertEquals(6, ids.size());
+ }
+
+ @Test
+ public void testGetShortId() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id = idGenerator.getId("abcd");
+ Assert.assertEquals("abcd", id);
+ }
+
+ @Test
+ public void testSameNames() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id1 = idGenerator.getId(Count.perKey().getName());
+ String id2 = idGenerator.getId(Count.perKey().getName());
+ Assert.assertNotEquals(id1, id2);
+ }
+
+ @Test
+ public void testSameShortNames() {
+ final HashIdGenerator idGenerator = new HashIdGenerator();
+ String id = idGenerator.getId("abcd");
+ Assert.assertEquals("abcd", id);
+ String id2 = idGenerator.getId("abcd");
+ Assert.assertNotEquals("abcd", id2);
+ }
+
+ @Test
+ public void testLongHash() {
+ final HashIdGenerator idGenerator = new HashIdGenerator(10);
+ String id1 = idGenerator.getId(Count.perKey().getName());
+ String id2 = idGenerator.getId(Count.perKey().getName());
+ String id3 = idGenerator.getId(Count.perKey().getName());
+ String id4 = idGenerator.getId(Count.perKey().getName());
+ Assert.assertNotEquals(id1, id2);
+ Assert.assertNotEquals(id3, id2);
+ Assert.assertNotEquals(id3, id4);
+ }
+}
diff --git a/runners/spark/build.gradle b/runners/spark/build.gradle
index b11259c..9b8ff6e 100644
--- a/runners/spark/build.gradle
+++ b/runners/spark/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.runners.spark')
description = "Apache Beam :: Runners :: Spark"
@@ -54,47 +54,45 @@
}
dependencies {
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:core-construction-java", configuration: "shadow")
- shadow project(path: ":runners:core-java", configuration: "shadow")
- shadow project(path: ":runners:java-fn-execution", configuration: "shadow")
- shadow library.java.guava
- shadow library.java.jackson_annotations
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.args4j
- shadow "io.dropwizard.metrics:metrics-core:3.1.2"
- shadow library.java.jackson_module_scala
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":runners:core-construction-java")
+ compile project(":runners:core-java")
+ compile project(":runners:java-fn-execution")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile library.java.jackson_annotations
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile library.java.args4j
provided library.java.spark_core
provided library.java.spark_streaming
provided library.java.spark_network_common
provided library.java.hadoop_common
- provided library.java.hadoop_mapreduce_client_core
- provided library.java.commons_compress
provided library.java.commons_lang3
provided library.java.commons_io_2x
provided library.java.hamcrest_core
provided library.java.hamcrest_library
- provided "org.apache.zookeeper:zookeeper:3.4.11"
- provided "org.scala-lang:scala-library:2.11.8"
provided "com.esotericsoftware.kryo:kryo:2.21"
- shadowTest project(path: ":sdks:java:io:kafka", configuration: "shadow")
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
+ runtimeOnly library.java.jackson_module_scala
+ runtimeOnly "org.scala-lang:scala-library:2.11.8"
+ testCompile project(":sdks:java:io:kafka")
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
// SparkStateInternalsTest extends abstract StateInternalsTest
- shadowTest project(path: ":runners:core-java", configuration: "shadowTest")
- shadowTest project(":sdks:java:harness")
- shadowTest library.java.avro
- shadowTest library.java.kafka_clients
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.jackson_dataformat_yaml
- shadowTest "org.apache.kafka:kafka_2.11:0.11.0.1"
+ testCompile project(path: ":runners:core-java", configuration: "testRuntime")
+ testCompile project(":sdks:java:harness")
+ testCompile library.java.avro
+ testCompile library.java.kafka
+ testCompile library.java.kafka_clients
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.jackson_dataformat_yaml
+ testCompile "org.apache.zookeeper:zookeeper:3.4.11"
validatesRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesRunner project(path: ":sdks:java:io:hadoop-format", configuration: "shadowTest")
- validatesRunner project(path: ":examples:java", configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadowTest")
- validatesRunner project(path: project.path, configuration: "shadow")
+ validatesRunner project(":sdks:java:io:hadoop-format")
+ validatesRunner project(":sdks:java:io:hadoop-format").sourceSets.test.output
+ validatesRunner project(path: ":examples:java", configuration: "testRuntime")
+ validatesRunner project(path: project.path, configuration: "testRuntime")
+ validatesRunner project(project.path)
validatesRunner project(path: project.path, configuration: "provided")
}
@@ -142,6 +140,7 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesImpulse'
excludeCategories 'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
}
+ jvmArgs '-Xmx3g'
}
task validatesRunnerStreaming(type: Test) {
diff --git a/runners/spark/job-server/build.gradle b/runners/spark/job-server/build.gradle
index 4930e00..4c7ee60 100644
--- a/runners/spark/job-server/build.gradle
+++ b/runners/spark/job-server/build.gradle
@@ -28,6 +28,7 @@
mainClassName = "org.apache.beam.runners.spark.SparkJobServerDriver"
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.spark.jobserver',
validateShadowJar: false,
exportJavadoc: false,
shadowClosure: {
@@ -48,14 +49,14 @@
}
dependencies {
- compile project(path: sparkRunnerProject, configuration: "shadow")
+ compile project(sparkRunnerProject)
compile project(path: sparkRunnerProject, configuration: "provided")
- validatesPortableRunner project(path: sparkRunnerProject, configuration: "shadowTest")
+ validatesPortableRunner project(path: sparkRunnerProject, configuration: "testRuntime")
validatesPortableRunner project(path: sparkRunnerProject, configuration: "provided")
validatesPortableRunner project(path: ":sdks:java:core", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:core-java", configuration: "shadowTest")
- validatesPortableRunner project(path: ":runners:reference:java", configuration: "shadowTest")
- compile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
+ validatesPortableRunner project(path: ":runners:core-java", configuration: "testRuntime")
+ validatesPortableRunner project(path: ":runners:reference:java", configuration: "testRuntime")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
// TODO: Enable AWS and HDFS file system.
}
@@ -85,7 +86,7 @@
createPortableValidatesRunnerTask(
name: "validatesPortableRunner${name}",
jobServerDriver: "org.apache.beam.runners.spark.SparkJobServerDriver",
- jobServerConfig: "--job-host=localhost,--job-port=0,--artifact-port=0",
+ jobServerConfig: "--job-host=localhost,--job-port=0,--artifact-port=0,--expansion-port=0",
testClasspathConfiguration: configurations.validatesPortableRunner,
numParallelTests: 1,
environment: BeamModulePlugin.PortableValidatesRunnerConfiguration.Environment.EMBEDDED,
@@ -109,8 +110,6 @@
excludeCategories 'org.apache.beam.sdk.testing.UsesMapState'
excludeCategories 'org.apache.beam.sdk.testing.UsesSetState'
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
- // TODO re-enable when state is supported
- excludeCategories 'org.apache.beam.sdk.testing.UsesStatefulParDo'
//SplitableDoFnTests
excludeCategories 'org.apache.beam.sdk.testing.UsesBoundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesSplittableParDoWithWindowedSideInputs'
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobInvoker.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobInvoker.java
index da35ae2..3e01f6d 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobInvoker.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobInvoker.java
@@ -24,8 +24,8 @@
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation;
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Struct;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Struct;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java
index 0589045..f0302f1 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java
@@ -20,7 +20,9 @@
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.jobsubmission.JobInvoker;
import org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver;
+import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
@@ -51,7 +53,11 @@
}
public static void main(String[] args) {
- FileSystems.setDefaultPipelineOptions(PipelineOptionsFactory.create());
+ PipelineOptions options = PipelineOptionsFactory.create();
+ // Limiting gcs upload buffer to reduce memory usage while doing parallel artifact uploads.
+ options.as(GcsOptions.class).setGcsUploadBufferSizeBytes(1024 * 1024);
+ // Register standard file systems.
+ FileSystems.setDefaultPipelineOptions(options);
fromParams(args).run();
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkNativePipelineVisitor.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkNativePipelineVisitor.java
index 43c4e5d..b85a71e 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkNativePipelineVisitor.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkNativePipelineVisitor.java
@@ -30,8 +30,8 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.commons.lang3.StringUtils;
/**
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index 2fd8042..267f666 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
@@ -17,7 +17,9 @@
*/
package org.apache.beam.runners.spark;
+import java.io.File;
import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.runners.core.construction.PipelineResources;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.options.ApplicationNameOptions;
@@ -26,7 +28,7 @@
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.StreamingOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* Spark runner {@link PipelineOptions} handles Spark execution-related configurations, such as the
@@ -150,17 +152,24 @@
void setCacheDisabled(boolean value);
/**
- * Local configurations work in the same JVM and have no problems with improperly formatted files
- * on classpath (eg. directories with .class files or empty directories). Prepare files for
- * staging only when using remote cluster (passing the master address explicitly).
+ * Classpath contains non jar files (eg. directories with .class files or empty directories) will
+ * cause exception in running log. Though the {@link org.apache.spark.SparkContext} can handle
+ * this when running in local master, it's better not to include non-jars files in classpath.
*/
- static void prepareFilesToStageForRemoteClusterExecution(SparkPipelineOptions options) {
- if (!options.getSparkMaster().matches("local\\[?\\d*\\]?")) {
- options.setFilesToStage(
- PipelineResources.prepareFilesForStaging(
- options.getFilesToStage(),
- MoreObjects.firstNonNull(
- options.getTempLocation(), System.getProperty("java.io.tmpdir"))));
- }
+ static void prepareFilesToStage(SparkPipelineOptions options) {
+ List<String> filesToStage =
+ options.getFilesToStage().stream()
+ .map(File::new)
+ .filter(File::exists)
+ .map(
+ file -> {
+ return file.getAbsolutePath();
+ })
+ .collect(Collectors.toList());
+ options.setFilesToStage(
+ PipelineResources.prepareFilesForStaging(
+ filesToStage,
+ MoreObjects.firstNonNull(
+ options.getTempLocation(), System.getProperty("java.io.tmpdir"))));
}
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
index 66e2470..fd3fbcf 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineResult.java
@@ -25,6 +25,8 @@
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.jobmanagement.v1.JobApi;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.spark.metrics.MetricsAccumulator;
import org.apache.beam.runners.spark.translation.SparkContextFactory;
import org.apache.beam.sdk.Pipeline;
@@ -35,6 +37,8 @@
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** Represents a Spark pipeline execution result. */
public abstract class SparkPipelineResult implements PipelineResult {
@@ -83,7 +87,7 @@
@Override
public PipelineResult.State waitUntilFinish() {
- return waitUntilFinish(Duration.millis(Long.MAX_VALUE));
+ return waitUntilFinish(Duration.millis(-1));
}
@Override
@@ -133,11 +137,30 @@
@Override
protected State awaitTermination(final Duration duration)
throws TimeoutException, ExecutionException, InterruptedException {
- pipelineExecution.get(duration.getMillis(), TimeUnit.MILLISECONDS);
+ if (duration.getMillis() > 0) {
+ pipelineExecution.get(duration.getMillis(), TimeUnit.MILLISECONDS);
+ } else {
+ pipelineExecution.get();
+ }
return PipelineResult.State.DONE;
}
}
+ static class PortableBatchMode extends BatchMode implements PortablePipelineResult {
+
+ private static final Logger LOG = LoggerFactory.getLogger(BatchMode.class);
+
+ PortableBatchMode(Future<?> pipelineExecution, JavaSparkContext javaSparkContext) {
+ super(pipelineExecution, javaSparkContext);
+ }
+
+ @Override
+ public JobApi.MetricResults portableMetrics() throws UnsupportedOperationException {
+ LOG.warn("Collecting monitoring infos is not implemented yet in Spark portable runner.");
+ return JobApi.MetricResults.newBuilder().build();
+ }
+ }
+
/** Represents a streaming Spark pipeline result. */
static class StreamingMode extends SparkPipelineResult {
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
index b0a1063..725df75 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java
@@ -18,7 +18,7 @@
package org.apache.beam.runners.spark;
import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
-import static org.apache.beam.runners.spark.SparkPipelineOptions.prepareFilesToStageForRemoteClusterExecution;
+import static org.apache.beam.runners.spark.SparkPipelineOptions.prepareFilesToStage;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
@@ -29,6 +29,7 @@
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
import org.apache.beam.runners.core.construction.graph.PipelineTrimmer;
import org.apache.beam.runners.core.metrics.MetricsPusher;
+import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineResult;
import org.apache.beam.runners.fnexecution.jobsubmission.PortablePipelineRunner;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.spark.aggregators.AggregatorsAccumulator;
@@ -54,7 +55,7 @@
}
@Override
- public SparkPipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo) {
+ public PortablePipelineResult run(RunnerApi.Pipeline pipeline, JobInfo jobInfo) {
SparkBatchPortablePipelineTranslator translator = new SparkBatchPortablePipelineTranslator();
// Don't let the fuser fuse any subcomponents of native transforms.
@@ -74,7 +75,7 @@
LOG.info(
"PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath");
}
- prepareFilesToStageForRemoteClusterExecution(pipelineOptions);
+ prepareFilesToStage(pipelineOptions);
LOG.info(
"Will stage {} files. (Enable logging at DEBUG level to see which files will be staged.)",
pipelineOptions.getFilesToStage().size());
@@ -102,7 +103,8 @@
LOG.info(String.format("Job %s finished.", jobInfo.jobId()));
});
- SparkPipelineResult result = new SparkPipelineResult.BatchMode(submissionFuture, jsc);
+ PortablePipelineResult result =
+ new SparkPipelineResult.PortableBatchMode(submissionFuture, jsc);
MetricsPusher metricsPusher =
new MetricsPusher(
MetricsAccumulator.getInstance().value(),
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
index 2c7cb74..c4f17f8 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
@@ -18,9 +18,10 @@
package org.apache.beam.runners.spark;
import static org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage;
-import static org.apache.beam.runners.spark.SparkPipelineOptions.prepareFilesToStageForRemoteClusterExecution;
+import static org.apache.beam.runners.spark.SparkPipelineOptions.prepareFilesToStage;
import java.util.Collection;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ExecutorService;
@@ -60,7 +61,7 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.spark.SparkEnv$;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.metrics.MetricsSystem;
@@ -165,7 +166,7 @@
pipeline.replaceAll(SparkTransformOverrides.getDefaultOverrides(mOptions.isStreaming()));
- prepareFilesToStageForRemoteClusterExecution(mOptions);
+ prepareFilesToStage(mOptions);
if (mOptions.isStreaming()) {
CheckpointDir checkpointDir = new CheckpointDir(mOptions.getCheckpointDir());
@@ -334,8 +335,12 @@
// we populate cache candidates by updating the map with inputs of each node.
// The goal is to detect the PCollections accessed more than one time, and so enable cache
// on the underlying RDDs or DStreams.
+ Map<TupleTag<?>, PValue> inputs = new HashMap<>(node.getInputs());
+ for (TupleTag<?> tupleTag : node.getTransform().getAdditionalInputs().keySet()) {
+ inputs.remove(tupleTag);
+ }
- for (PValue value : node.getInputs().values()) {
+ for (PValue value : inputs.values()) {
if (value instanceof PCollection) {
long count = 1L;
if (ctxt.getCacheCandidates().get(value) != null) {
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
index 0c40c5b..3e64e76 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Contains the {@link PipelineRunnerRegistrar} and {@link PipelineOptionsRegistrar} for the {@link
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkTransformOverrides.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkTransformOverrides.java
index 8ddb307..f96905a 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkTransformOverrides.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkTransformOverrides.java
@@ -25,9 +25,9 @@
import org.apache.beam.runners.core.construction.UnsupportedOverrideFactory;
import org.apache.beam.sdk.runners.PTransformOverride;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
-/** {@link PTransform} overrides for Flink runner. */
+/** {@link PTransform} overrides for Spark runner. */
class SparkTransformOverrides {
public static List<PTransformOverride> getDefaultOverrides(boolean streaming) {
ImmutableList.Builder<PTransformOverride> builder = ImmutableList.builder();
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
index ac9c307..1907fab 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.isOneOf;
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsValidator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.apache.commons.io.FileUtils;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/AggregatorsAccumulator.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/AggregatorsAccumulator.java
index 4722b43..89ea552 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/AggregatorsAccumulator.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/AggregatorsAccumulator.java
@@ -21,8 +21,8 @@
import org.apache.beam.runners.spark.SparkPipelineOptions;
import org.apache.beam.runners.spark.translation.streaming.Checkpoint;
import org.apache.beam.runners.spark.translation.streaming.Checkpoint.CheckpointDir;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.spark.api.java.JavaSparkContext;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/NamedAggregators.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/NamedAggregators.java
index 3d59f64..450788a 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/NamedAggregators.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/NamedAggregators.java
@@ -21,8 +21,8 @@
import java.util.Map;
import java.util.TreeMap;
import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* This class wraps a map of named aggregators. Spark expects that all accumulators be declared
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/CoderHelpers.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/CoderHelpers.java
index 9f4e43a..55d8c3f 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/CoderHelpers.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/CoderHelpers.java
@@ -17,8 +17,9 @@
*/
package org.apache.beam.runners.spark.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
@@ -27,6 +28,7 @@
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;
+import javax.annotation.Nonnull;
import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.sdk.coders.Coder;
import org.apache.spark.api.java.function.Function;
@@ -142,19 +144,40 @@
}
/**
- * A function wrapper for converting a byte array pair to a key-value pair.
+ * A function for converting a byte array pair to a key-value pair.
*
- * @param keyCoder Coder to deserialize keys.
- * @param valueCoder Coder to deserialize values.
* @param <K> The type of the key being deserialized.
* @param <V> The type of the value being deserialized.
- * @return A function that accepts a pair of byte arrays and returns a key-value pair.
*/
- public static <K, V> PairFunction<Tuple2<ByteArray, byte[]>, K, V> fromByteFunction(
- final Coder<K> keyCoder, final Coder<V> valueCoder) {
- return tuple ->
- new Tuple2<>(
- fromByteArray(tuple._1().getValue(), keyCoder), fromByteArray(tuple._2(), valueCoder));
+ public static class FromByteFunction<K, V>
+ implements PairFunction<Tuple2<ByteArray, byte[]>, K, V>,
+ org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function<
+ Tuple2<ByteArray, byte[]>, Tuple2<K, V>> {
+ private final Coder<K> keyCoder;
+ private final Coder<V> valueCoder;
+
+ /**
+ * @param keyCoder Coder to deserialize keys.
+ * @param valueCoder Coder to deserialize values.
+ */
+ public FromByteFunction(final Coder<K> keyCoder, final Coder<V> valueCoder) {
+ this.keyCoder = keyCoder;
+ this.valueCoder = valueCoder;
+ }
+
+ @Override
+ public Tuple2<K, V> call(Tuple2<ByteArray, byte[]> tuple) {
+ return new Tuple2<>(
+ fromByteArray(tuple._1().getValue(), keyCoder), fromByteArray(tuple._2(), valueCoder));
+ }
+
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
+ @Override
+ public Tuple2<K, V> apply(@Nonnull Tuple2<ByteArray, byte[]> tuple) {
+ return call(tuple);
+ }
}
/**
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistrator.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistrator.java
index 8043005..681ca43 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistrator.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/SparkRunnerKryoRegistrator.java
@@ -22,14 +22,13 @@
import java.util.LinkedHashMap;
import org.apache.beam.runners.spark.io.MicrobatchSource;
import org.apache.beam.runners.spark.stateful.SparkGroupAlsoByWindowViaWindowSet.StateAndTimers;
-import org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions.WindowedKey;
import org.apache.beam.runners.spark.translation.ValueAndCoderKryoSerializer;
import org.apache.beam.runners.spark.translation.ValueAndCoderLazySerializable;
import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
import org.apache.spark.serializer.KryoRegistrator;
import scala.collection.mutable.WrappedArray;
@@ -59,7 +58,6 @@
kryo.register(PaneInfo.class);
kryo.register(StateAndTimers.class);
kryo.register(TupleTag.class);
- kryo.register(WindowedKey.class);
kryo.register(WrappedArray.ofRef.class);
try {
@@ -67,7 +65,7 @@
Class.forName("org.apache.beam.sdk.util.WindowedValue$TimestampedValueInGlobalWindow"));
kryo.register(
Class.forName(
- "org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable$Factory"));
+ "org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable$Factory"));
} catch (ClassNotFoundException e) {
throw new IllegalStateException("Unable to register classes with kryo.", e);
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/CreateStream.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/CreateStream.java
index ce460a3..708bf89 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/CreateStream.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/CreateStream.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayDeque;
import java.util.Arrays;
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
index e013c81..8ac32e2 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
@@ -30,12 +30,12 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.BackOff;
import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalListener;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalNotification;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalListener;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalNotification;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
index abc8244..8c5de3c 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.spark.SparkPipelineOptions;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceRDD.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceRDD.java
index 007e80b..0295ac3 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceRDD.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceRDD.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.spark.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Closeable;
import java.io.IOException;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.metrics.MetricsContainer;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.spark.Dependency;
import org.apache.spark.HashPartitioner;
import org.apache.spark.InterruptibleIterator;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
index 8235ee6..9103b2d 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext$;
import org.apache.spark.api.java.function.FlatMapFunction;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
index f21b1f6..a18b92d 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
@@ -22,8 +22,8 @@
import org.apache.beam.runners.spark.SparkPipelineOptions;
import org.apache.beam.runners.spark.translation.streaming.Checkpoint;
import org.apache.beam.runners.spark.translation.streaming.Checkpoint.CheckpointDir;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.spark.api.java.JavaSparkContext;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/SparkBeamMetric.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/SparkBeamMetric.java
index 11b4e04..c43bb09 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/SparkBeamMetric.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/SparkBeamMetric.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.metrics.MetricQueryResults;
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* An adapter between the {@link MetricsContainerStepMap} and Codahale's {@link Metric} interface.
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/WithMetricsSupport.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/WithMetricsSupport.java
index 3bcc262..9450986 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/WithMetricsSupport.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/WithMetricsSupport.java
@@ -29,14 +29,14 @@
import java.util.Map;
import java.util.SortedMap;
import org.apache.beam.runners.spark.aggregators.NamedAggregators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSortedMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
index 275c951..5cb0bec 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkGroupAlsoByWindowViaWindowSet.java
@@ -38,13 +38,13 @@
import org.apache.beam.runners.core.triggers.TriggerStateMachines;
import org.apache.beam.runners.spark.SparkPipelineOptions;
import org.apache.beam.runners.spark.coders.CoderHelpers;
+import org.apache.beam.runners.spark.translation.ReifyTimestampsAndWindowsFunction;
import org.apache.beam.runners.spark.translation.TranslationUtils;
import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.runners.spark.util.GlobalWatermarkHolder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -54,12 +54,12 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.AbstractIterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.AbstractIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
import org.apache.spark.api.java.JavaSparkContext$;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.streaming.Duration;
@@ -73,8 +73,8 @@
import scala.Option;
import scala.Tuple2;
import scala.Tuple3;
-import scala.collection.GenTraversable;
import scala.collection.Iterator;
+import scala.collection.JavaConversions;
import scala.collection.Seq;
import scala.runtime.AbstractFunction1;
@@ -151,7 +151,7 @@
Iterator<
Tuple3<
/*K*/ ByteArray,
- Seq</*Itr<WV<I>>*/ byte[]>,
+ Seq</*WV<I>*/ byte[]>,
Option<Tuple2<StateAndTimers, /*WV<KV<K, Itr<I>>>*/ List<byte[]>>>>>,
Iterator<
Tuple2</*K*/ ByteArray, Tuple2<StateAndTimers, /*WV<KV<K, Itr<I>>>*/ List<byte[]>>>>>
@@ -159,9 +159,7 @@
private class UpdateStateByKeyOutputIterator
extends AbstractIterator<
- Tuple2<
- /*K*/ ByteArray,
- Tuple2<StateAndTimers, /*WV<KV<K, KV<Long(Time),Itr<I>>>>*/ List<byte[]>>>> {
+ Tuple2</*K*/ ByteArray, Tuple2<StateAndTimers, /*WV<KV<K, Itr<I>>>*/ List<byte[]>>>> {
private final Iterator<
Tuple3<ByteArray, Seq<byte[]>, Option<Tuple2<StateAndTimers, List<byte[]>>>>>
@@ -236,7 +234,7 @@
input.next();
final ByteArray encodedKey = next._1();
- final Seq<byte[]> encodedKeyedElements = next._2();
+ final Seq<byte[]> encodedElements = next._2();
final Option<Tuple2<StateAndTimers, List<byte[]>>> prevStateAndTimersOpt = next._3();
final K key = CoderHelpers.fromByteArray(encodedKey.getValue(), keyCoder);
@@ -270,23 +268,12 @@
reduceFn,
options.get());
- if (!encodedKeyedElements.isEmpty()) {
+ if (!encodedElements.isEmpty()) {
// new input for key.
try {
- // cast to GenTraversable to avoid a ambiguous call to head() which can come from
- // multiple super interfacesof Seq<byte[]>
- byte[] headBytes = ((GenTraversable<byte[]>) encodedKeyedElements).head();
- final KV<Long, Iterable<WindowedValue<InputT>>> keyedElements =
- CoderHelpers.fromByteArray(headBytes, KvCoder.of(VarLongCoder.of(), itrWvCoder));
-
- final Long rddTimestamp = keyedElements.getKey();
-
- LOG.debug(
- logPrefix + ": processing RDD with timestamp: {}, watermarks: {}",
- rddTimestamp,
- watermarks);
-
- final Iterable<WindowedValue<InputT>> elements = keyedElements.getValue();
+ final Iterable<WindowedValue<InputT>> elements =
+ FluentIterable.from(JavaConversions.asJavaIterable(encodedElements))
+ .transform(bytes -> CoderHelpers.fromByteArray(bytes, wvCoder));
LOG.trace(logPrefix + ": input elements: {}", elements);
@@ -413,7 +400,7 @@
final Iterator<
Tuple3<
/*K*/ ByteArray,
- Seq</*Itr<WV<I>>*/ byte[]>,
+ Seq</*WV<I>*/ byte[]>,
Option<Tuple2<StateAndTimers, /*WV<KV<K, Itr<I>>>*/ List<byte[]>>>>>
input) {
// --- ACTUAL STATEFUL OPERATION:
@@ -495,10 +482,6 @@
final Coder<K> keyCoder,
final FullWindowedValueCoder<InputT> wvCoder) {
- /*K*/
- /*WV<KV<K, Itr<I>>>*/
- /*K*/
- /*WV<KV<K, Itr<I>>>*/
return JavaPairDStream.fromPairDStream(
firedStream,
JavaSparkContext$.MODULE$.fakeClassTag(),
@@ -532,7 +515,7 @@
}
private static <K, InputT> PairDStreamFunctions<ByteArray, byte[]> buildPairDStream(
- final JavaDStream<WindowedValue<KV<K, Iterable<WindowedValue<InputT>>>>> inputDStream,
+ final JavaDStream<WindowedValue<KV<K, InputT>>> inputDStream,
final Coder<K> keyCoder,
final Coder<WindowedValue<InputT>> wvCoder) {
@@ -547,27 +530,11 @@
// ---- Iterable: Itr
// ---- AccumT: A
// ---- InputT: I
- // we use mapPartitions with the RDD API because its the only available API
- // that allows to preserve partitioning.
final DStream<Tuple2<ByteArray, byte[]>> tupleDStream =
inputDStream
- .transformToPair(
- (rdd, time) ->
- rdd.mapPartitions(
- TranslationUtils.functionToFlatMapFunction(WindowedValue::getValue),
- true)
- .mapPartitionsToPair(TranslationUtils.toPairFlatMapFunction(), true)
- .mapValues(
- // add the batch timestamp for visibility (e.g., debugging)
- values -> KV.of(time.milliseconds(), values))
- // move to bytes representation and use coders for deserialization
- // because of checkpointing.
- .mapPartitionsToPair(
- TranslationUtils.pairFunctionToPairFlatMapFunction(
- CoderHelpers.toByteFunction(
- keyCoder,
- KvCoder.of(VarLongCoder.of(), IterableCoder.of(wvCoder)))),
- true))
+ .map(new ReifyTimestampsAndWindowsFunction<>())
+ .mapToPair(TranslationUtils.toPairFunction())
+ .mapToPair(CoderHelpers.toByteFunction(keyCoder, wvCoder))
.dstream();
return DStream.toPairDStreamFunctions(
@@ -578,8 +545,8 @@
}
public static <K, InputT, W extends BoundedWindow>
- JavaDStream<WindowedValue<KV<K, Iterable<InputT>>>> groupAlsoByWindow(
- final JavaDStream<WindowedValue<KV<K, Iterable<WindowedValue<InputT>>>>> inputDStream,
+ JavaDStream<WindowedValue<KV<K, Iterable<InputT>>>> groupByKeyAndWindow(
+ final JavaDStream<WindowedValue<KV<K, InputT>>> inputDStream,
final Coder<K> keyCoder,
final Coder<WindowedValue<InputT>> wvCoder,
final WindowingStrategy<?, W> windowingStrategy,
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkStateInternals.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkStateInternals.java
index 2ad71ca..5273a3a 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkStateInternals.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkStateInternals.java
@@ -41,8 +41,8 @@
import org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.CombineFnUtil;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
import org.joda.time.Instant;
/** An implementation of {@link StateInternals} for the SparkRunner. */
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java
index 66d7501..02305a5 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/SparkTimerInternals.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.stateful;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Collections;
@@ -32,8 +32,8 @@
import org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Instant;
/** An implementation of {@link TimerInternals} for the SparkRunner. */
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
index ad60420..1398f8e 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
@@ -38,10 +38,10 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.spark.streaming.State;
import org.apache.spark.streaming.StateSpec;
import org.joda.time.Instant;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
index 3a1586d..37b2e22 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/BoundedDataset.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaRDDLike;
import org.apache.spark.api.java.JavaSparkContext;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
index cf71efb..9ecbb97 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.HashMap;
import java.util.LinkedHashMap;
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
index 3055340..9a10354 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
@@ -17,20 +17,17 @@
*/
package org.apache.beam.runners.spark.translation;
-import java.util.Collections;
import javax.annotation.Nullable;
import org.apache.beam.runners.spark.coders.CoderHelpers;
import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.spark.Partitioner;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
@@ -45,7 +42,7 @@
* An implementation of {@link
* org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly} for the Spark runner.
*/
- public static <K, V> JavaRDD<WindowedValue<KV<K, Iterable<WindowedValue<V>>>>> groupByKeyOnly(
+ public static <K, V> JavaRDD<KV<K, Iterable<WindowedValue<V>>>> groupByKeyOnly(
JavaRDD<WindowedValue<KV<K, V>>> rdd,
Coder<K> keyCoder,
WindowedValueCoder<V> wvCoder,
@@ -54,7 +51,6 @@
// can be transferred over the network for the shuffle.
JavaPairRDD<ByteArray, byte[]> pairRDD =
rdd.map(new ReifyTimestampsAndWindowsFunction<>())
- .map(WindowedValue::getValue)
.mapToPair(TranslationUtils.toPairFunction())
.mapToPair(CoderHelpers.toByteFunction(keyCoder, wvCoder));
@@ -62,48 +58,67 @@
JavaPairRDD<ByteArray, Iterable<byte[]>> groupedRDD =
(partitioner != null) ? pairRDD.groupByKey(partitioner) : pairRDD.groupByKey();
- // using mapPartitions allows to preserve the partitioner
- // and avoid unnecessary shuffle downstream.
return groupedRDD
- .mapPartitionsToPair(
- TranslationUtils.pairFunctionToPairFlatMapFunction(
- CoderHelpers.fromByteFunctionIterable(keyCoder, wvCoder)),
- true)
- .mapPartitions(TranslationUtils.fromPairFlatMapFunction(), true)
- .mapPartitions(
- TranslationUtils.functionToFlatMapFunction(WindowedValue::valueInGlobalWindow), true);
+ .mapToPair(CoderHelpers.fromByteFunctionIterable(keyCoder, wvCoder))
+ .map(new TranslationUtils.FromPairFunction<>());
+ }
+
+ /**
+ * Spark-level group by key operation that keeps original Beam {@link KV} pairs unchanged.
+ *
+ * @returns {@link JavaPairRDD} where the first value in the pair is the serialized key, and the
+ * second is an iterable of the {@link KV} pairs with that key.
+ */
+ static <K, V> JavaPairRDD<ByteArray, Iterable<WindowedValue<KV<K, V>>>> groupByKeyPair(
+ JavaRDD<WindowedValue<KV<K, V>>> rdd, Coder<K> keyCoder, WindowedValueCoder<V> wvCoder) {
+ // we use coders to convert objects in the PCollection to byte arrays, so they
+ // can be transferred over the network for the shuffle.
+ JavaPairRDD<ByteArray, byte[]> pairRDD =
+ rdd.map(new ReifyTimestampsAndWindowsFunction<>())
+ .mapToPair(TranslationUtils.toPairFunction())
+ .mapToPair(CoderHelpers.toByteFunction(keyCoder, wvCoder));
+
+ JavaPairRDD<ByteArray, Iterable<Tuple2<ByteArray, byte[]>>> groupedRDD =
+ pairRDD.groupBy((value) -> value._1);
+
+ return groupedRDD
+ .mapValues(
+ it -> Iterables.transform(it, new CoderHelpers.FromByteFunction<>(keyCoder, wvCoder)))
+ .mapValues(it -> Iterables.transform(it, new TranslationUtils.FromPairFunction()))
+ .mapValues(
+ it -> Iterables.transform(it, new TranslationUtils.ToKVByWindowInValueFunction<>()));
}
/** Apply a composite {@link org.apache.beam.sdk.transforms.Combine.Globally} transformation. */
- public static <InputT, AccumT> Optional<Iterable<WindowedValue<AccumT>>> combineGlobally(
- JavaRDD<WindowedValue<InputT>> rdd,
- final SparkGlobalCombineFn<InputT, AccumT, ?> sparkCombineFn,
- final Coder<AccumT> aCoder,
- final WindowingStrategy<?, ?> windowingStrategy) {
+ public static <InputT, OutputT, AccumT>
+ SparkCombineFn.WindowedAccumulator<InputT, InputT, AccumT, ?> combineGlobally(
+ JavaRDD<WindowedValue<InputT>> rdd,
+ final SparkCombineFn<InputT, InputT, AccumT, OutputT> sparkCombineFn,
+ final Coder<AccumT> aCoder,
+ final WindowingStrategy<?, ?> windowingStrategy) {
- final WindowedValue.FullWindowedValueCoder<AccumT> wvaCoder =
- WindowedValue.FullWindowedValueCoder.of(
- aCoder, windowingStrategy.getWindowFn().windowCoder());
- final IterableCoder<WindowedValue<AccumT>> iterAccumCoder = IterableCoder.of(wvaCoder);
+ @SuppressWarnings("unchecked")
+ final Coder<BoundedWindow> windowCoder = (Coder) windowingStrategy.getWindowFn().windowCoder();
+ final SparkCombineFn.WindowedAccumulatorCoder<InputT, InputT, AccumT> waCoder =
+ sparkCombineFn.accumulatorCoder(windowCoder, aCoder, windowingStrategy);
- ValueAndCoderLazySerializable<Iterable<WindowedValue<AccumT>>> accumulatedResult =
- rdd.aggregate(
- ValueAndCoderLazySerializable.of(Collections.emptyList(), iterAccumCoder),
- (ab, ib) -> {
- Iterable<WindowedValue<AccumT>> merged =
- sparkCombineFn.seqOp(ab.getOrDecode(iterAccumCoder), ib);
- return ValueAndCoderLazySerializable.of(merged, iterAccumCoder);
- },
- (a1b, a2b) -> {
- Iterable<WindowedValue<AccumT>> merged =
- sparkCombineFn.combOp(
- a1b.getOrDecode(iterAccumCoder), a2b.getOrDecode(iterAccumCoder));
- return ValueAndCoderLazySerializable.of(merged, iterAccumCoder);
- });
+ ValueAndCoderLazySerializable<SparkCombineFn.WindowedAccumulator<InputT, InputT, AccumT, ?>>
+ accumulatedResult =
+ rdd.aggregate(
+ ValueAndCoderLazySerializable.of(sparkCombineFn.createCombiner(), waCoder),
+ (ab, ib) -> {
+ SparkCombineFn.WindowedAccumulator<InputT, InputT, AccumT, ?> merged =
+ sparkCombineFn.mergeValue(ab.getOrDecode(waCoder), ib);
+ return ValueAndCoderLazySerializable.of(merged, waCoder);
+ },
+ (a1b, a2b) -> {
+ SparkCombineFn.WindowedAccumulator<InputT, InputT, AccumT, ?> merged =
+ sparkCombineFn.mergeCombiners(
+ a1b.getOrDecode(waCoder), a2b.getOrDecode(waCoder));
+ return ValueAndCoderLazySerializable.of(merged, waCoder);
+ });
- final Iterable<WindowedValue<AccumT>> result = accumulatedResult.getOrDecode(iterAccumCoder);
-
- return Iterables.isEmpty(result) ? Optional.absent() : Optional.of(result);
+ return accumulatedResult.getOrDecode(waCoder);
}
/**
@@ -114,18 +129,20 @@
* streaming, this will be called from within a serialized context (DStream's transform callback),
* so passed arguments need to be Serializable.
*/
- public static <K, InputT, AccumT>
- JavaPairRDD<K, Iterable<WindowedValue<KV<K, AccumT>>>> combinePerKey(
- JavaRDD<WindowedValue<KV<K, InputT>>> rdd,
- final SparkKeyedCombineFn<K, InputT, AccumT, ?> sparkCombineFn,
+ public static <K, V, AccumT>
+ JavaPairRDD<K, SparkCombineFn.WindowedAccumulator<KV<K, V>, V, AccumT, ?>> combinePerKey(
+ JavaRDD<WindowedValue<KV<K, V>>> rdd,
+ final SparkCombineFn<KV<K, V>, V, AccumT, ?> sparkCombineFn,
final Coder<K> keyCoder,
+ final Coder<V> valueCoder,
final Coder<AccumT> aCoder,
final WindowingStrategy<?, ?> windowingStrategy) {
- final WindowedValue.FullWindowedValueCoder<KV<K, AccumT>> wkvaCoder =
- WindowedValue.FullWindowedValueCoder.of(
- KvCoder.of(keyCoder, aCoder), windowingStrategy.getWindowFn().windowCoder());
- final IterableCoder<WindowedValue<KV<K, AccumT>>> iterAccumCoder = IterableCoder.of(wkvaCoder);
+ boolean mustBringWindowToKey = sparkCombineFn.mustBringWindowToKey();
+ @SuppressWarnings("unchecked")
+ Coder<BoundedWindow> windowCoder = (Coder) windowingStrategy.getWindowFn().windowCoder();
+ final SparkCombineFn.WindowedAccumulatorCoder<KV<K, V>, V, AccumT> waCoder =
+ sparkCombineFn.accumulatorCoder(windowCoder, aCoder, windowingStrategy);
// We need to duplicate K as both the key of the JavaPairRDD as well as inside the value,
// since the functions passed to combineByKey don't receive the associated key of each
@@ -134,45 +151,44 @@
// Once Spark provides a way to include keys in the arguments of combine/merge functions,
// we won't need to duplicate the keys anymore.
// Key has to bw windowed in order to group by window as well.
- JavaPairRDD<ByteArray, WindowedValue<KV<K, InputT>>> inRddDuplicatedKeyPair =
- rdd.mapToPair(TranslationUtils.toPairByKeyInWindowedValue(keyCoder));
+ final JavaPairRDD<ByteArray, WindowedValue<KV<K, V>>> inRddDuplicatedKeyPair;
+ if (!mustBringWindowToKey) {
+ inRddDuplicatedKeyPair = rdd.mapToPair(TranslationUtils.toPairByKeyInWindowedValue(keyCoder));
+ } else {
+ inRddDuplicatedKeyPair =
+ GroupNonMergingWindowsFunctions.bringWindowToKey(rdd, keyCoder, windowCoder);
+ }
- JavaPairRDD<ByteArray, ValueAndCoderLazySerializable<Iterable<WindowedValue<KV<K, AccumT>>>>>
+ JavaPairRDD<
+ ByteArray,
+ ValueAndCoderLazySerializable<
+ SparkCombineFn.WindowedAccumulator<KV<K, V>, V, AccumT, ?>>>
accumulatedResult =
inRddDuplicatedKeyPair.combineByKey(
input ->
- ValueAndCoderLazySerializable.of(
- sparkCombineFn.createCombiner(input), iterAccumCoder),
+ ValueAndCoderLazySerializable.of(sparkCombineFn.createCombiner(input), waCoder),
(acc, input) ->
ValueAndCoderLazySerializable.of(
- sparkCombineFn.mergeValue(input, acc.getOrDecode(iterAccumCoder)),
- iterAccumCoder),
+ sparkCombineFn.mergeValue(acc.getOrDecode(waCoder), input), waCoder),
(acc1, acc2) ->
ValueAndCoderLazySerializable.of(
sparkCombineFn.mergeCombiners(
- acc1.getOrDecode(iterAccumCoder), acc2.getOrDecode(iterAccumCoder)),
- iterAccumCoder));
+ acc1.getOrDecode(waCoder), acc2.getOrDecode(waCoder)),
+ waCoder));
return accumulatedResult.mapToPair(
i ->
new Tuple2<>(
- CoderHelpers.fromByteArray(i._1.getValue(), keyCoder),
- i._2.getOrDecode(iterAccumCoder)));
+ CoderHelpers.fromByteArray(i._1.getValue(), keyCoder), i._2.getOrDecode(waCoder)));
}
/** An implementation of {@link Reshuffle} for the Spark runner. */
- public static <K, V> JavaRDD<WindowedValue<KV<K, V>>> reshuffle(
- JavaRDD<WindowedValue<KV<K, V>>> rdd, Coder<K> keyCoder, WindowedValueCoder<V> wvCoder) {
-
+ public static <T> JavaRDD<WindowedValue<T>> reshuffle(
+ JavaRDD<WindowedValue<T>> rdd, WindowedValueCoder<T> wvCoder) {
// Use coders to convert objects in the PCollection to byte arrays, so they
// can be transferred over the network for the shuffle.
- return rdd.map(new ReifyTimestampsAndWindowsFunction<>())
- .map(WindowedValue::getValue)
- .mapToPair(TranslationUtils.toPairFunction())
- .mapToPair(CoderHelpers.toByteFunction(keyCoder, wvCoder))
+ return rdd.map(CoderHelpers.toByteFunction(wvCoder))
.repartition(rdd.getNumPartitions())
- .mapToPair(CoderHelpers.fromByteFunction(keyCoder, wvCoder))
- .map(TranslationUtils.fromPairFunction())
- .map(TranslationUtils.toKVByWindowInValue());
+ .map(CoderHelpers.fromByteFunction(wvCoder));
}
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java
index 2b35d2f..42cd72c 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctions.java
@@ -17,31 +17,52 @@
*/
package org.apache.beam.runners.spark.translation;
-import java.io.Serializable;
-import java.util.Arrays;
import java.util.Iterator;
import java.util.Objects;
import org.apache.beam.runners.spark.coders.CoderHelpers;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.AbstractIterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.PeekingIterator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.AbstractIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.PeekingIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
import org.apache.spark.HashPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.joda.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import scala.Tuple2;
/** Functions for GroupByKey with Non-Merging windows translations to Spark. */
public class GroupNonMergingWindowsFunctions {
+ private static final Logger LOG = LoggerFactory.getLogger(GroupNonMergingWindowsFunctions.class);
+
+ /**
+ * Verify if given windowing strategy and coders are suitable for group by key and window
+ * optimization.
+ *
+ * @param windowingStrategy the windowing strategy
+ * @return {@code true} if group by key and window can be used
+ */
+ static boolean isEligibleForGroupByWindow(WindowingStrategy<?, ?> windowingStrategy) {
+ return windowingStrategy.getWindowFn().isNonMerging()
+ && windowingStrategy.getTimestampCombiner() == TimestampCombiner.END_OF_WINDOW
+ && windowingStrategy.getWindowFn().windowCoder().consistentWithEquals();
+ }
+
/**
* Creates composite key of K and W and group all values for that composite key with Spark's
* repartitionAndSortWithinPartitions. Stream of sorted by composite key's is transformed to key
@@ -55,40 +76,78 @@
JavaRDD<WindowedValue<KV<K, V>>> rdd,
Coder<K> keyCoder,
Coder<V> valueCoder,
- WindowingStrategy<?, W> windowingStrategy) {
+ WindowingStrategy<?, W> windowingStrategy,
+ Partitioner partitioner) {
final Coder<W> windowCoder = windowingStrategy.getWindowFn().windowCoder();
- final WindowedValue.FullWindowedValueCoder<byte[]> windowedValueCoder =
- WindowedValue.getFullCoder(ByteArrayCoder.of(), windowCoder);
- return rdd.flatMapToPair(
- (WindowedValue<KV<K, V>> windowedValue) -> {
- final byte[] keyBytes =
- CoderHelpers.toByteArray(windowedValue.getValue().getKey(), keyCoder);
- final byte[] valueBytes =
- CoderHelpers.toByteArray(windowedValue.getValue().getValue(), valueCoder);
- return Iterators.transform(
- windowedValue.explodeWindows().iterator(),
- item -> {
- Objects.requireNonNull(item, "Exploded window can not be null.");
- @SuppressWarnings("unchecked")
- final W window = (W) Iterables.getOnlyElement(item.getWindows());
- final byte[] windowBytes = CoderHelpers.toByteArray(window, windowCoder);
- final byte[] windowValueBytes =
- CoderHelpers.toByteArray(
- WindowedValue.of(
- valueBytes, item.getTimestamp(), window, item.getPane()),
- windowedValueCoder);
- final WindowedKey windowedKey = new WindowedKey(keyBytes, windowBytes);
- return new Tuple2<>(windowedKey, windowValueBytes);
- });
- })
- .repartitionAndSortWithinPartitions(new HashPartitioner(rdd.getNumPartitions()))
+ FullWindowedValueCoder<KV<K, V>> windowedKvCoder =
+ WindowedValue.FullWindowedValueCoder.of(KvCoder.of(keyCoder, valueCoder), windowCoder);
+ JavaPairRDD<ByteArray, byte[]> windowInKey =
+ bringWindowToKey(
+ rdd, keyCoder, windowCoder, wv -> CoderHelpers.toByteArray(wv, windowedKvCoder));
+ return windowInKey
+ .repartitionAndSortWithinPartitions(getPartitioner(partitioner, rdd))
.mapPartitions(
- it ->
- new GroupByKeyIterator<>(
- it, keyCoder, valueCoder, windowingStrategy, windowedValueCoder))
+ it -> new GroupByKeyIterator<>(it, keyCoder, windowingStrategy, windowedKvCoder))
.filter(Objects::nonNull); // filter last null element from GroupByKeyIterator
}
+ static <K, V, W extends BoundedWindow>
+ JavaPairRDD<ByteArray, WindowedValue<KV<K, V>>> bringWindowToKey(
+ JavaRDD<WindowedValue<KV<K, V>>> rdd, Coder<K> keyCoder, Coder<W> windowCoder) {
+ return bringWindowToKey(rdd, keyCoder, windowCoder, e -> e);
+ }
+
+ /** Creates pair RDD with key being a composite of original key and window. */
+ static <K, V, OutputT, W extends BoundedWindow> JavaPairRDD<ByteArray, OutputT> bringWindowToKey(
+ JavaRDD<WindowedValue<KV<K, V>>> rdd,
+ Coder<K> keyCoder,
+ Coder<W> windowCoder,
+ SerializableFunction<WindowedValue<KV<K, V>>, OutputT> mappingFn) {
+
+ if (!isKeyAndWindowCoderConsistentWithEquals(keyCoder, windowCoder)) {
+ LOG.warn(
+ "Either coder {} or {} is not consistent with equals. "
+ + "That might cause issues on some runners.",
+ keyCoder,
+ windowCoder);
+ }
+
+ return rdd.flatMapToPair(
+ (WindowedValue<KV<K, V>> windowedValue) -> {
+ final byte[] keyBytes =
+ CoderHelpers.toByteArray(windowedValue.getValue().getKey(), keyCoder);
+ return Iterators.transform(
+ windowedValue.explodeWindows().iterator(),
+ item -> {
+ Objects.requireNonNull(item, "Exploded window can not be null.");
+ @SuppressWarnings("unchecked")
+ final W window = (W) Iterables.getOnlyElement(item.getWindows());
+ final byte[] windowBytes = CoderHelpers.toByteArray(window, windowCoder);
+ WindowedValue<KV<K, V>> valueOut =
+ WindowedValue.of(item.getValue(), item.getTimestamp(), window, item.getPane());
+ final ByteArray windowedKey = new ByteArray(Bytes.concat(keyBytes, windowBytes));
+ return new Tuple2<>(windowedKey, mappingFn.apply(valueOut));
+ });
+ });
+ }
+
+ private static boolean isKeyAndWindowCoderConsistentWithEquals(
+ Coder<?> keyCoder, Coder<?> windowCoder) {
+ try {
+ keyCoder.verifyDeterministic();
+ windowCoder.verifyDeterministic();
+ return keyCoder.consistentWithEquals() && windowCoder.consistentWithEquals();
+ } catch (Coder.NonDeterministicException ex) {
+ throw new IllegalArgumentException(
+ "Coder for both key " + keyCoder + " and " + windowCoder + " must be deterministic", ex);
+ }
+ }
+
+ private static <K, V> Partitioner getPartitioner(
+ Partitioner partitioner, JavaRDD<WindowedValue<KV<K, V>>> rdd) {
+ return partitioner == null ? new HashPartitioner(rdd.getNumPartitions()) : partitioner;
+ }
+
/**
* Transform stream of sorted key values into stream of value iterators for each key. This
* iterator can be iterated only once!
@@ -101,24 +160,23 @@
static class GroupByKeyIterator<K, V, W extends BoundedWindow>
implements Iterator<WindowedValue<KV<K, Iterable<V>>>> {
- private final PeekingIterator<Tuple2<WindowedKey, byte[]>> inner;
+ private final PeekingIterator<Tuple2<ByteArray, byte[]>> inner;
private final Coder<K> keyCoder;
- private final Coder<V> valueCoder;
private final WindowingStrategy<?, W> windowingStrategy;
- private final FullWindowedValueCoder<byte[]> windowedValueCoder;
+ private final FullWindowedValueCoder<KV<K, V>> windowedValueCoder;
private boolean hasNext = true;
- private WindowedKey currentKey = null;
+ private ByteArray currentKey = null;
GroupByKeyIterator(
- Iterator<Tuple2<WindowedKey, byte[]>> inner,
+ Iterator<Tuple2<ByteArray, byte[]>> inner,
Coder<K> keyCoder,
- Coder<V> valueCoder,
WindowingStrategy<?, W> windowingStrategy,
- WindowedValue.FullWindowedValueCoder<byte[]> windowedValueCoder) {
+ WindowedValue.FullWindowedValueCoder<KV<K, V>> windowedValueCoder)
+ throws Coder.NonDeterministicException {
+
this.inner = Iterators.peekingIterator(inner);
this.keyCoder = keyCoder;
- this.valueCoder = valueCoder;
this.windowingStrategy = windowingStrategy;
this.windowedValueCoder = windowedValueCoder;
}
@@ -131,7 +189,7 @@
@Override
public WindowedValue<KV<K, Iterable<V>>> next() {
while (inner.hasNext()) {
- final WindowedKey nextKey = inner.peek()._1;
+ final ByteArray nextKey = inner.peek()._1;
if (nextKey.equals(currentKey)) {
// we still did not see all values for a given key
inner.next();
@@ -148,23 +206,23 @@
class ValueIterator implements Iterable<V> {
- boolean usedAsIterable = false;
- private final PeekingIterator<Tuple2<WindowedKey, byte[]>> inner;
- private final WindowedKey currentKey;
+ boolean consumed = false;
+ private final PeekingIterator<Tuple2<ByteArray, byte[]>> inner;
+ private final ByteArray currentKey;
- ValueIterator(PeekingIterator<Tuple2<WindowedKey, byte[]>> inner, WindowedKey currentKey) {
+ ValueIterator(PeekingIterator<Tuple2<ByteArray, byte[]>> inner, ByteArray currentKey) {
this.inner = inner;
this.currentKey = currentKey;
}
@Override
public Iterator<V> iterator() {
- if (usedAsIterable) {
+ if (consumed) {
throw new IllegalStateException(
"ValueIterator can't be iterated more than once,"
+ "otherwise there could be data lost");
}
- usedAsIterable = true;
+ consumed = true;
return new AbstractIterator<V>() {
@Override
protected V computeNext() {
@@ -178,16 +236,16 @@
}
private V decodeValue(byte[] windowedValueBytes) {
- final WindowedValue<byte[]> windowedValue =
+ final WindowedValue<KV<K, V>> windowedValue =
CoderHelpers.fromByteArray(windowedValueBytes, windowedValueCoder);
- return CoderHelpers.fromByteArray(windowedValue.getValue(), valueCoder);
+ return windowedValue.getValue().getValue();
}
- private WindowedValue<KV<K, V>> decodeItem(Tuple2<WindowedKey, byte[]> item) {
- final K key = CoderHelpers.fromByteArray(item._1.getKey(), keyCoder);
- final WindowedValue<byte[]> windowedValue =
+ private WindowedValue<KV<K, V>> decodeItem(Tuple2<ByteArray, byte[]> item) {
+ final K key = CoderHelpers.fromByteArray(item._1.getValue(), keyCoder);
+ final WindowedValue<KV<K, V>> windowedValue =
CoderHelpers.fromByteArray(item._2, windowedValueCoder);
- final V value = CoderHelpers.fromByteArray(windowedValue.getValue(), valueCoder);
+ final V value = windowedValue.getValue().getValue();
@SuppressWarnings("unchecked")
final W window = (W) Iterables.getOnlyElement(windowedValue.getWindows());
final Instant timestamp =
@@ -198,55 +256,9 @@
windowingStrategy
.getWindowFn()
.getOutputTime(windowedValue.getTimestamp(), window));
- return WindowedValue.of(KV.of(key, value), timestamp, window, windowedValue.getPane());
- }
- }
-
- /** Composite key of key and window for groupByKey transformation. */
- public static class WindowedKey implements Comparable<WindowedKey>, Serializable {
-
- private final byte[] key;
- private final byte[] window;
-
- WindowedKey(byte[] key, byte[] window) {
- this.key = key;
- this.window = window;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (o == null || getClass() != o.getClass()) {
- return false;
- }
- WindowedKey that = (WindowedKey) o;
- return Arrays.equals(key, that.key) && Arrays.equals(window, that.window);
- }
-
- @Override
- public int hashCode() {
- int result = Arrays.hashCode(key);
- result = 31 * result + Arrays.hashCode(window);
- return result;
- }
-
- byte[] getKey() {
- return key;
- }
-
- byte[] getWindow() {
- return window;
- }
-
- @Override
- public int compareTo(WindowedKey o) {
- int keyCompare = UnsignedBytes.lexicographicalComparator().compare(this.getKey(), o.getKey());
- if (keyCompare == 0) {
- return UnsignedBytes.lexicographicalComparator().compare(this.getWindow(), o.getWindow());
- }
- return keyCompare;
+ // BEAM-7341: Elements produced by GbK are always ON_TIME and ONLY_FIRING
+ return WindowedValue.of(
+ KV.of(key, value), timestamp, window, PaneInfo.ON_TIME_AND_ONLY_FIRING);
}
}
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java
index ea8d732..ee2a581 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java
@@ -42,12 +42,13 @@
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.LinkedListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.LinkedListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.apache.spark.api.java.function.PairFlatMapFunction;
import org.apache.spark.util.AccumulatorV2;
import scala.Tuple2;
@@ -75,6 +76,7 @@
private final WindowingStrategy<?, ?> windowingStrategy;
private final boolean stateful;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
/**
* @param metricsAccum The Spark {@link AccumulatorV2} that backs the Beam metrics.
@@ -100,7 +102,8 @@
Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
WindowingStrategy<?, ?> windowingStrategy,
boolean stateful,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.metricsAccum = metricsAccum;
this.stepName = stepName;
this.doFn = SerializableUtils.clone(doFn);
@@ -113,6 +116,7 @@
this.windowingStrategy = windowingStrategy;
this.stateful = stateful;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
@Override
@@ -166,7 +170,8 @@
inputCoder,
outputCoders,
windowingStrategy,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
DoFnRunnerWithMetrics<InputT, OutputT> doFnRunnerWithMetrics =
new DoFnRunnerWithMetrics<>(stepName, doFnRunner, metricsAccum);
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ReifyTimestampsAndWindowsFunction.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ReifyTimestampsAndWindowsFunction.java
index 6eff26a..dfb58c5 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ReifyTimestampsAndWindowsFunction.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ReifyTimestampsAndWindowsFunction.java
@@ -26,21 +26,13 @@
* Simple {@link Function} to bring the windowing information into the value from the implicit
* background representation of the {@link PCollection}.
*/
-class ReifyTimestampsAndWindowsFunction<K, V>
- implements Function<WindowedValue<KV<K, V>>, WindowedValue<KV<K, WindowedValue<V>>>> {
+public class ReifyTimestampsAndWindowsFunction<K, V>
+ implements Function<WindowedValue<KV<K, V>>, KV<K, WindowedValue<V>>> {
@Override
- public WindowedValue<KV<K, WindowedValue<V>>> call(WindowedValue<KV<K, V>> elem)
- throws Exception {
- return WindowedValue.of(
- KV.of(
- elem.getValue().getKey(),
- WindowedValue.of(
- elem.getValue().getValue(),
- elem.getTimestamp(),
- elem.getWindows(),
- elem.getPane())),
- elem.getTimestamp(),
- elem.getWindows(),
- elem.getPane());
+ public KV<K, WindowedValue<V>> call(WindowedValue<KV<K, V>> elem) throws Exception {
+ return KV.of(
+ elem.getValue().getKey(),
+ WindowedValue.of(
+ elem.getValue().getValue(), elem.getTimestamp(), elem.getWindows(), elem.getPane()));
}
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAbstractCombineFn.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAbstractCombineFn.java
deleted file mode 100644
index ee68198..0000000
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAbstractCombineFn.java
+++ /dev/null
@@ -1,124 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.spark.translation;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
-import java.io.Serializable;
-import java.util.Collection;
-import java.util.Comparator;
-import java.util.List;
-import java.util.Map;
-import org.apache.beam.runners.core.SideInputReader;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.spark.util.SideInputBroadcast;
-import org.apache.beam.runners.spark.util.SparkSideInputReader;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.transforms.CombineWithContext;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-
-/**
- * An abstract for the SparkRunner implementation of {@link
- * org.apache.beam.sdk.transforms.Combine.CombineFn}.
- */
-class SparkAbstractCombineFn implements Serializable {
- private final SerializablePipelineOptions options;
- private final Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs;
- final WindowingStrategy<?, BoundedWindow> windowingStrategy;
-
- SparkAbstractCombineFn(
- SerializablePipelineOptions options,
- Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
- WindowingStrategy<?, ?> windowingStrategy) {
- this.options = options;
- this.sideInputs = sideInputs;
- this.windowingStrategy = (WindowingStrategy<?, BoundedWindow>) windowingStrategy;
- }
-
- // each Spark task should get it's own copy of this SparkKeyedCombineFn, and since Spark tasks
- // are single-threaded, it is safe to reuse the context.
- // the combine context is not Serializable so we'll use lazy initialization.
- // ** DO NOT attempt to turn this into a Singleton as Spark may run multiple tasks in parallel
- // in the same JVM (Executor). **
- // ** DO NOT use combineContext directly inside this class, use ctxtForInput instead. **
- private transient SparkCombineContext combineContext;
-
- SparkCombineContext ctxtForInput(WindowedValue<?> input) {
- if (combineContext == null) {
- combineContext = new SparkCombineContext(options.get(), new SparkSideInputReader(sideInputs));
- }
- return combineContext.forInput(input);
- }
-
- static <T> Iterable<WindowedValue<T>> sortByWindows(Iterable<WindowedValue<T>> iter) {
- List<WindowedValue<T>> sorted = Lists.newArrayList(iter);
- sorted.sort(Comparator.comparing(o -> Iterables.getOnlyElement(o.getWindows()).maxTimestamp()));
- return sorted;
- }
-
- static boolean isIntersecting(IntervalWindow union, IntervalWindow window) {
- return union == null || union.intersects(window);
- }
-
- static IntervalWindow merge(IntervalWindow union, IntervalWindow window) {
- return union == null ? window : union.span(window);
- }
-
- /** An implementation of {@link CombineWithContext.Context} for the SparkRunner. */
- private static class SparkCombineContext extends CombineWithContext.Context {
- private final PipelineOptions pipelineOptions;
- private final SideInputReader sideInputReader;
-
- SparkCombineContext(PipelineOptions pipelineOptions, SideInputReader sideInputReader) {
- this.pipelineOptions = pipelineOptions;
- this.sideInputReader = sideInputReader;
- }
-
- private WindowedValue<?> input = null;
-
- SparkCombineContext forInput(WindowedValue<?> input) {
- this.input = input;
- return this;
- }
-
- @Override
- public PipelineOptions getPipelineOptions() {
- return pipelineOptions;
- }
-
- @Override
- public <T> T sideInput(PCollectionView<T> view) {
- checkNotNull(input, "Input in SparkCombineContext must not be null!");
- // validate element window.
- final Collection<? extends BoundedWindow> elementWindows = input.getWindows();
- checkState(
- elementWindows.size() == 1,
- "sideInput can only be called when the main " + "input element is in exactly one window");
- return sideInputReader.get(view, elementWindows.iterator().next());
- }
- }
-}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAssignWindowFn.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAssignWindowFn.java
index 46b5bc2..5a6ce6f 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAssignWindowFn.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAssignWindowFn.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.transforms.windowing.Window.Assign;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.spark.api.java.function.Function;
import org.joda.time.Instant;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkBatchPortablePipelineTranslator.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkBatchPortablePipelineTranslator.java
index 8e7796f..7402c00 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkBatchPortablePipelineTranslator.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkBatchPortablePipelineTranslator.java
@@ -19,19 +19,23 @@
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.createOutputMap;
import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.getWindowingStrategy;
+import static org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils.instantiateCoder;
+import com.google.auto.service.AutoService;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
import org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.construction.NativeTransforms;
import org.apache.beam.runners.core.construction.PTransformTranslation;
-import org.apache.beam.runners.core.construction.ReadTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.PipelineNode;
import org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
@@ -39,36 +43,37 @@
import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
import org.apache.beam.runners.fnexecution.wire.WireCoders;
import org.apache.beam.runners.spark.SparkPipelineOptions;
-import org.apache.beam.runners.spark.aggregators.AggregatorsAccumulator;
-import org.apache.beam.runners.spark.io.SourceRDD;
import org.apache.beam.runners.spark.metrics.MetricsAccumulator;
+import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.spark.HashPartitioner;
import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
-import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.broadcast.Broadcast;
+import org.apache.spark.storage.StorageLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import scala.Tuple2;
/** Translates a bounded portable pipeline into a Spark job. */
public class SparkBatchPortablePipelineTranslator {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(SparkBatchPortablePipelineTranslator.class);
+
private final ImmutableMap<String, PTransformTranslator> urnToTransformTranslator;
interface PTransformTranslator {
@@ -79,12 +84,7 @@
}
public Set<String> knownUrns() {
- // Do not expose Read as a known URN because we only want to support Read
- // through the Java ExpansionService. We can't translate Reads for other
- // languages.
- return Sets.difference(
- urnToTransformTranslator.keySet(),
- ImmutableSet.of(PTransformTranslation.READ_TRANSFORM_URN));
+ return urnToTransformTranslator.keySet();
}
public SparkBatchPortablePipelineTranslator() {
@@ -101,8 +101,8 @@
PTransformTranslation.FLATTEN_TRANSFORM_URN,
SparkBatchPortablePipelineTranslator::translateFlatten);
translatorMap.put(
- PTransformTranslation.READ_TRANSFORM_URN,
- SparkBatchPortablePipelineTranslator::translateRead);
+ PTransformTranslation.RESHUFFLE_URN,
+ SparkBatchPortablePipelineTranslator::translateReshuffle);
this.urnToTransformTranslator = translatorMap.build();
}
@@ -112,6 +112,24 @@
QueryablePipeline.forTransforms(
pipeline.getRootTransformIdsList(), pipeline.getComponents());
for (PipelineNode.PTransformNode transformNode : p.getTopologicallyOrderedTransforms()) {
+ // Pre-scan pipeline to count which pCollections are consumed as inputs more than once so
+ // their corresponding RDDs can later be cached.
+ for (String inputId : transformNode.getTransform().getInputsMap().values()) {
+ context.incrementConsumptionCountBy(inputId, 1);
+ }
+ // Executable stage consists of two parts: computation and extraction. This means the result
+ // of computation is an intermediate RDD, which we might also need to cache.
+ if (transformNode.getTransform().getSpec().getUrn().equals(ExecutableStage.URN)) {
+ context.incrementConsumptionCountBy(
+ getExecutableStageIntermediateId(transformNode),
+ transformNode.getTransform().getOutputsMap().size());
+ }
+ for (String outputId : transformNode.getTransform().getOutputsMap().values()) {
+ WindowedValueCoder outputCoder = getWindowedValueCoder(outputId, pipeline.getComponents());
+ context.putCoder(outputId, outputCoder);
+ }
+ }
+ for (PipelineNode.PTransformNode transformNode : p.getTopologicallyOrderedTransforms()) {
urnToTransformTranslator
.getOrDefault(
transformNode.getTransform().getSpec().getUrn(),
@@ -141,18 +159,9 @@
RunnerApi.Components components = pipeline.getComponents();
String inputId = getInputId(transformNode);
- PCollection inputPCollection = components.getPcollectionsOrThrow(inputId);
Dataset inputDataset = context.popDataset(inputId);
JavaRDD<WindowedValue<KV<K, V>>> inputRdd = ((BoundedDataset<KV<K, V>>) inputDataset).getRDD();
- PCollectionNode inputPCollectionNode = PipelineNode.pCollection(inputId, inputPCollection);
- WindowedValueCoder<KV<K, V>> inputCoder;
- try {
- inputCoder =
- (WindowedValueCoder)
- WireCoders.instantiateRunnerWireCoder(inputPCollectionNode, components);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ WindowedValueCoder<KV<K, V>> inputCoder = getWindowedValueCoder(inputId, components);
KvCoder<K, V> inputKvCoder = (KvCoder<K, V>) inputCoder.getValueCoder();
Coder<K> inputKeyCoder = inputKvCoder.getKeyCoder();
Coder<V> inputValueCoder = inputKvCoder.getValueCoder();
@@ -162,15 +171,14 @@
WindowedValue.FullWindowedValueCoder.of(inputValueCoder, windowFn.windowCoder());
JavaRDD<WindowedValue<KV<K, Iterable<V>>>> groupedByKeyAndWindow;
- if (windowingStrategy.getWindowFn().isNonMerging()
- && windowingStrategy.getTimestampCombiner() == TimestampCombiner.END_OF_WINDOW) {
+ Partitioner partitioner = getPartitioner(context);
+ if (GroupNonMergingWindowsFunctions.isEligibleForGroupByWindow(windowingStrategy)) {
// we can have a memory sensitive translation for non-merging windows
groupedByKeyAndWindow =
GroupNonMergingWindowsFunctions.groupByKeyAndWindow(
- inputRdd, inputKeyCoder, inputValueCoder, windowingStrategy);
+ inputRdd, inputKeyCoder, inputValueCoder, windowingStrategy, partitioner);
} else {
- Partitioner partitioner = getPartitioner(context);
- JavaRDD<WindowedValue<KV<K, Iterable<WindowedValue<V>>>>> groupedByKeyOnly =
+ JavaRDD<KV<K, Iterable<WindowedValue<V>>>> groupedByKeyOnly =
GroupCombineFunctions.groupByKeyOnly(inputRdd, inputKeyCoder, wvCoder, partitioner);
// for batch, GroupAlsoByWindow uses an in-memory StateInternals.
groupedByKeyAndWindow =
@@ -179,8 +187,7 @@
windowingStrategy,
new TranslationUtils.InMemoryStateInternalsFactory<>(),
SystemReduceFn.buffering(inputValueCoder),
- context.serializablePipelineOptions,
- AggregatorsAccumulator.getInstance()));
+ context.serializablePipelineOptions));
}
context.pushDataset(getOutputId(transformNode), new BoundedDataset<>(groupedByKeyAndWindow));
}
@@ -198,35 +205,101 @@
}
String inputPCollectionId = stagePayload.getInput();
Dataset inputDataset = context.popDataset(inputPCollectionId);
- JavaRDD<WindowedValue<InputT>> inputRdd = ((BoundedDataset<InputT>) inputDataset).getRDD();
Map<String, String> outputs = transformNode.getTransform().getOutputsMap();
- BiMap<String, Integer> outputMap = createOutputMap(outputs.values());
+ BiMap<String, Integer> outputExtractionMap = createOutputMap(outputs.values());
+ Components components = pipeline.getComponents();
+ Coder windowCoder =
+ getWindowingStrategy(inputPCollectionId, components).getWindowFn().windowCoder();
ImmutableMap.Builder<String, Tuple2<Broadcast<List<byte[]>>, WindowedValueCoder<SideInputT>>>
broadcastVariablesBuilder = ImmutableMap.builder();
for (SideInputId sideInputId : stagePayload.getSideInputsList()) {
- RunnerApi.Components components = stagePayload.getComponents();
+ RunnerApi.Components stagePayloadComponents = stagePayload.getComponents();
String collectionId =
- components
+ stagePayloadComponents
.getTransformsOrThrow(sideInputId.getTransformId())
.getInputsOrThrow(sideInputId.getLocalName());
Tuple2<Broadcast<List<byte[]>>, WindowedValueCoder<SideInputT>> tuple2 =
- broadcastSideInput(collectionId, components, context);
+ broadcastSideInput(collectionId, stagePayloadComponents, context);
broadcastVariablesBuilder.put(collectionId, tuple2);
}
- SparkExecutableStageFunction<InputT, SideInputT> function =
- new SparkExecutableStageFunction<>(
- stagePayload,
- context.jobInfo,
- outputMap,
- broadcastVariablesBuilder.build(),
- MetricsAccumulator.getInstance());
- JavaRDD<RawUnionValue> staged = inputRdd.mapPartitions(function);
+ JavaRDD<RawUnionValue> staged;
+ if (stagePayload.getUserStatesCount() > 0 || stagePayload.getTimersCount() > 0) {
+ Coder<WindowedValue<InputT>> windowedInputCoder =
+ instantiateCoder(inputPCollectionId, components);
+ Coder valueCoder =
+ ((WindowedValue.FullWindowedValueCoder) windowedInputCoder).getValueCoder();
+ // Stateful stages are only allowed of KV input to be able to group on the key
+ if (!(valueCoder instanceof KvCoder)) {
+ throw new IllegalStateException(
+ String.format(
+ Locale.ENGLISH,
+ "The element coder for stateful DoFn '%s' must be KvCoder but is: %s",
+ inputPCollectionId,
+ valueCoder.getClass().getSimpleName()));
+ }
+ Coder keyCoder = ((KvCoder) valueCoder).getKeyCoder();
+ Coder innerValueCoder = ((KvCoder) valueCoder).getValueCoder();
+ WindowingStrategy windowingStrategy = getWindowingStrategy(inputPCollectionId, components);
+ WindowFn<Object, BoundedWindow> windowFn = windowingStrategy.getWindowFn();
+ WindowedValue.WindowedValueCoder wvCoder =
+ WindowedValue.FullWindowedValueCoder.of(innerValueCoder, windowFn.windowCoder());
+
+ JavaPairRDD<ByteArray, Iterable<WindowedValue<KV>>> groupedByKey =
+ groupByKeyPair(inputDataset, keyCoder, wvCoder);
+ SparkExecutableStageFunction<KV, SideInputT> function =
+ new SparkExecutableStageFunction<>(
+ stagePayload,
+ context.jobInfo,
+ outputExtractionMap,
+ SparkExecutableStageContextFactory.getInstance(),
+ broadcastVariablesBuilder.build(),
+ MetricsAccumulator.getInstance(),
+ windowCoder);
+ staged = groupedByKey.flatMap(function.forPair());
+ } else {
+ JavaRDD<WindowedValue<InputT>> inputRdd2 = ((BoundedDataset<InputT>) inputDataset).getRDD();
+ SparkExecutableStageFunction<InputT, SideInputT> function2 =
+ new SparkExecutableStageFunction<>(
+ stagePayload,
+ context.jobInfo,
+ outputExtractionMap,
+ SparkExecutableStageContextFactory.getInstance(),
+ broadcastVariablesBuilder.build(),
+ MetricsAccumulator.getInstance(),
+ windowCoder);
+ staged = inputRdd2.mapPartitions(function2);
+ }
+
+ String intermediateId = getExecutableStageIntermediateId(transformNode);
+ context.pushDataset(
+ intermediateId,
+ new Dataset() {
+ @Override
+ public void cache(String storageLevel, Coder<?> coder) {
+ StorageLevel level = StorageLevel.fromString(storageLevel);
+ staged.persist(level);
+ }
+
+ @Override
+ public void action() {
+ // Empty function to force computation of RDD.
+ staged.foreach(TranslationUtils.emptyVoidFunction());
+ }
+
+ @Override
+ public void setName(String name) {
+ staged.setName(name);
+ }
+ });
+ // pop dataset to mark RDD as used
+ context.popDataset(intermediateId);
for (String outputId : outputs.values()) {
JavaRDD<WindowedValue<OutputT>> outputRdd =
- staged.flatMap(new SparkExecutableStageExtractionFunction<>(outputMap.get(outputId)));
+ staged.flatMap(
+ new SparkExecutableStageExtractionFunction<>(outputExtractionMap.get(outputId)));
context.pushDataset(outputId, new BoundedDataset<>(outputRdd));
}
if (outputs.isEmpty()) {
@@ -242,6 +315,13 @@
}
}
+ /** Wrapper to help with type inference for {@link GroupCombineFunctions#groupByKeyPair}. */
+ private static <K, V> JavaPairRDD<ByteArray, Iterable<WindowedValue<KV<K, V>>>> groupByKeyPair(
+ Dataset dataset, Coder<K> keyCoder, WindowedValueCoder<V> wvCoder) {
+ JavaRDD<WindowedValue<KV<K, V>>> inputRdd = ((BoundedDataset<KV<K, V>>) dataset).getRDD();
+ return GroupCombineFunctions.groupByKeyPair(inputRdd, keyCoder, wvCoder);
+ }
+
/**
* Collect and serialize the data and then broadcast the result. *This can be expensive.*
*
@@ -249,17 +329,9 @@
*/
private static <T> Tuple2<Broadcast<List<byte[]>>, WindowedValueCoder<T>> broadcastSideInput(
String collectionId, RunnerApi.Components components, SparkTranslationContext context) {
- PCollection collection = components.getPcollectionsOrThrow(collectionId);
@SuppressWarnings("unchecked")
BoundedDataset<T> dataset = (BoundedDataset<T>) context.popDataset(collectionId);
- PCollectionNode collectionNode = PipelineNode.pCollection(collectionId, collection);
- WindowedValueCoder<T> coder;
- try {
- coder =
- (WindowedValueCoder<T>) WireCoders.instantiateRunnerWireCoder(collectionNode, components);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
+ WindowedValueCoder<T> coder = getWindowedValueCoder(collectionId, components);
List<byte[]> bytes = dataset.getBytes(coder);
Broadcast<List<byte[]>> broadcast = context.getSparkContext().broadcast(bytes);
return new Tuple2<>(broadcast, coder);
@@ -285,27 +357,13 @@
context.pushDataset(getOutputId(transformNode), new BoundedDataset<>(unionRDD));
}
- private static <T> void translateRead(
+ private static <T> void translateReshuffle(
PTransformNode transformNode, RunnerApi.Pipeline pipeline, SparkTranslationContext context) {
- String stepName = transformNode.getTransform().getUniqueName();
- final JavaSparkContext jsc = context.getSparkContext();
-
- BoundedSource boundedSource;
- try {
- boundedSource =
- ReadTranslation.boundedSourceFromProto(
- RunnerApi.ReadPayload.parseFrom(transformNode.getTransform().getSpec().getPayload()));
- } catch (IOException e) {
- throw new RuntimeException("Failed to extract BoundedSource from ReadPayload.", e);
- }
-
- // create an RDD from a BoundedSource.
- JavaRDD<WindowedValue<T>> input =
- new SourceRDD.Bounded<>(
- jsc.sc(), boundedSource, context.serializablePipelineOptions, stepName)
- .toJavaRDD();
-
- context.pushDataset(getOutputId(transformNode), new BoundedDataset<>(input));
+ String inputId = getInputId(transformNode);
+ WindowedValueCoder<T> coder = getWindowedValueCoder(inputId, pipeline.getComponents());
+ JavaRDD<WindowedValue<T>> inRDD = ((BoundedDataset<T>) context.popDataset(inputId)).getRDD();
+ JavaRDD<WindowedValue<T>> reshuffled = GroupCombineFunctions.reshuffle(inRDD, coder);
+ context.pushDataset(getOutputId(transformNode), new BoundedDataset<>(reshuffled));
}
@Nullable
@@ -324,4 +382,32 @@
private static String getOutputId(PTransformNode transformNode) {
return Iterables.getOnlyElement(transformNode.getTransform().getOutputsMap().values());
}
+
+ private static <T> WindowedValueCoder<T> getWindowedValueCoder(
+ String pCollectionId, RunnerApi.Components components) {
+ PCollection pCollection = components.getPcollectionsOrThrow(pCollectionId);
+ PCollectionNode pCollectionNode = PipelineNode.pCollection(pCollectionId, pCollection);
+ WindowedValueCoder<T> coder;
+ try {
+ coder =
+ (WindowedValueCoder) WireCoders.instantiateRunnerWireCoder(pCollectionNode, components);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ return coder;
+ }
+
+ private static String getExecutableStageIntermediateId(PTransformNode transformNode) {
+ return transformNode.getId();
+ }
+
+ /** Predicate to determine whether a URN is a Spark native transform. */
+ @AutoService(NativeTransforms.IsNativeTransform.class)
+ public static class IsSparkNativeTransform implements NativeTransforms.IsNativeTransform {
+ @Override
+ public boolean test(RunnerApi.PTransform pTransform) {
+ return PTransformTranslation.RESHUFFLE_URN.equals(
+ PTransformTranslation.urnForTransformOrNull(pTransform));
+ }
+ }
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkCombineFn.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkCombineFn.java
new file mode 100644
index 0000000..49ebd6b
--- /dev/null
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkCombineFn.java
@@ -0,0 +1,833 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.translation;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.function.BiConsumer;
+import java.util.function.BiFunction;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+import javax.annotation.Nullable;
+import org.apache.beam.runners.core.SideInputReader;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.runners.spark.util.SideInputBroadcast;
+import org.apache.beam.runners.spark.util.SparkSideInputReader;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.CombineWithContext;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.UserCodeException;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.spark.api.java.function.Function;
+import org.joda.time.Instant;
+
+/**
+ * A {@link org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn} with a {@link
+ * org.apache.beam.sdk.transforms.CombineWithContext.Context} for the SparkRunner.
+ */
+public class SparkCombineFn<InputT, ValueT, AccumT, OutputT> implements Serializable {
+
+ /** Accumulator of WindowedValues holding values for different windows. */
+ public interface WindowedAccumulator<
+ InputT, ValueT, AccumT, ImplT extends WindowedAccumulator<InputT, ValueT, AccumT, ImplT>> {
+
+ /**
+ * Type of the accumulator. The type depends (mostly) on {@link WindowingStrategy}, and more
+ * specialized versions enable more optimizations.
+ */
+ enum Type {
+ MERGING,
+ NON_MERGING,
+ EXPLODE_WINDOWS,
+ SINGLE_WINDOW;
+
+ boolean isMapBased() {
+ return this == NON_MERGING || this == MERGING;
+ }
+ }
+
+ /**
+ * Check if this accumulator is empty.
+ *
+ * @return {@code true} if this accumulator is empty
+ */
+ boolean isEmpty();
+
+ /** Add value with unexploded windows into the accumulator. */
+ void add(WindowedValue<InputT> value, SparkCombineFn<InputT, ValueT, AccumT, ?> context)
+ throws Exception;
+
+ /**
+ * Merge other acccumulator into this one.
+ *
+ * @param other the other accumulator to merge
+ */
+ void merge(ImplT other, SparkCombineFn<?, ?, AccumT, ?> context) throws Exception;
+
+ /** Extract output. */
+ Collection<WindowedValue<AccumT>> extractOutput();
+
+ /** Create concrete accumulator for given type. */
+ static <InputT, ValueT, AccumT> WindowedAccumulator<InputT, ValueT, AccumT, ?> create(
+ SparkCombineFn<InputT, ValueT, AccumT, ?> context,
+ Function<InputT, ValueT> toValue,
+ WindowingStrategy<?, ?> windowingStrategy,
+ Comparator<BoundedWindow> windowComparator) {
+ return create(toValue, context.getType(windowingStrategy), windowComparator);
+ }
+
+ static <InputT, ValueT, AccumT> WindowedAccumulator<InputT, ValueT, AccumT, ?> create(
+ Function<InputT, ValueT> toValue, Type type, Comparator<BoundedWindow> windowComparator) {
+ switch (type) {
+ case MERGING:
+ return MergingWindowedAccumulator.create(toValue, windowComparator);
+ case NON_MERGING:
+ return NonMergingWindowedAccumulator.create(toValue);
+ case SINGLE_WINDOW:
+ case EXPLODE_WINDOWS:
+ return SingleWindowWindowedAccumulator.create(toValue);
+ default:
+ throw new IllegalArgumentException("Unknown type: " + type);
+ }
+ }
+
+ /** Create concrete accumulator for given type. */
+ static <InputT, ValueT, AccumT> WindowedAccumulator<InputT, ValueT, AccumT, ?> create(
+ Function<InputT, ValueT> toValue,
+ Type type,
+ Iterable<WindowedValue<AccumT>> values,
+ Comparator<BoundedWindow> windowComparator) {
+ switch (type) {
+ case MERGING:
+ return MergingWindowedAccumulator.from(toValue, values, windowComparator);
+ case NON_MERGING:
+ return NonMergingWindowedAccumulator.from(toValue, values);
+ case SINGLE_WINDOW:
+ case EXPLODE_WINDOWS:
+ Iterator<WindowedValue<AccumT>> iter = values.iterator();
+ if (iter.hasNext()) {
+ return SingleWindowWindowedAccumulator.create(toValue, iter.next());
+ }
+ return SingleWindowWindowedAccumulator.create(toValue);
+ default:
+ throw new IllegalArgumentException("Unknown type: " + type);
+ }
+ }
+ }
+
+ /**
+ * Accumulator for {@link WindowFn WindowFns} which create only single window per key, and
+ * therefore don't need any map to hold different windows per key.
+ *
+ * @param <AccumT> type of accumulator
+ */
+ static class SingleWindowWindowedAccumulator<InputT, ValueT, AccumT>
+ implements WindowedAccumulator<
+ InputT, ValueT, AccumT, SingleWindowWindowedAccumulator<InputT, ValueT, AccumT>> {
+
+ static <InputT, ValueT, AccumT> SingleWindowWindowedAccumulator<InputT, ValueT, AccumT> create(
+ Function<InputT, ValueT> toValue) {
+ return new SingleWindowWindowedAccumulator<>(toValue);
+ }
+
+ static <InputT, ValueT, AccumT> WindowedAccumulator<InputT, ValueT, AccumT, ?> create(
+ Function<InputT, ValueT> toValue, WindowedValue<AccumT> accumulator) {
+ return new SingleWindowWindowedAccumulator<>(toValue, accumulator);
+ }
+
+ final Function<InputT, ValueT> toValue;
+ AccumT windowAccumulator = null;
+ Instant accTimestamp = null;
+ BoundedWindow accWindow = null;
+
+ SingleWindowWindowedAccumulator(Function<InputT, ValueT> toValue) {
+ this.toValue = toValue;
+ }
+
+ SingleWindowWindowedAccumulator(
+ Function<InputT, ValueT> toValue, WindowedValue<AccumT> accumulator) {
+ this.toValue = toValue;
+ this.windowAccumulator = accumulator.getValue();
+ this.accTimestamp =
+ accumulator.getTimestamp().equals(BoundedWindow.TIMESTAMP_MIN_VALUE)
+ ? null
+ : accumulator.getTimestamp();
+ this.accWindow = getWindow(accumulator);
+ }
+
+ @Override
+ public void add(WindowedValue<InputT> value, SparkCombineFn<InputT, ValueT, AccumT, ?> context)
+ throws Exception {
+ BoundedWindow window = getWindow(value);
+ SparkCombineContext ctx = context.ctxtForValue(value);
+ TimestampCombiner combiner = context.windowingStrategy.getTimestampCombiner();
+ Instant windowTimestamp =
+ combiner.assign(
+ window,
+ context.windowingStrategy.getWindowFn().getOutputTime(value.getTimestamp(), window));
+ final AccumT acc;
+ final Instant timestamp;
+ if (windowAccumulator == null) {
+ acc = context.combineFn.createAccumulator(ctx);
+ timestamp = windowTimestamp;
+ } else {
+ acc = windowAccumulator;
+ timestamp = accTimestamp;
+ }
+ AccumT result = context.combineFn.addInput(acc, toValue(value), ctx);
+ Instant timestampCombined = combiner.combine(windowTimestamp, timestamp);
+ windowAccumulator = result;
+ accTimestamp = timestampCombined;
+ accWindow = window;
+ }
+
+ @Override
+ public void merge(
+ SingleWindowWindowedAccumulator<InputT, ValueT, AccumT> other,
+ SparkCombineFn<?, ?, AccumT, ?> context) {
+ if (windowAccumulator != null && other.windowAccumulator != null) {
+ List<AccumT> accumulators = Arrays.asList(windowAccumulator, other.windowAccumulator);
+ AccumT merged =
+ context.combineFn.mergeAccumulators(
+ accumulators, context.ctxtForWindows(Arrays.asList(accWindow)));
+ Instant combined =
+ context
+ .windowingStrategy
+ .getTimestampCombiner()
+ .combine(accTimestamp, other.accTimestamp);
+ windowAccumulator = merged;
+ accTimestamp = combined;
+ } else if (windowAccumulator == null) {
+ windowAccumulator = other.windowAccumulator;
+ accTimestamp = other.accTimestamp;
+ accWindow = other.accWindow;
+ }
+ }
+
+ @Override
+ public Collection<WindowedValue<AccumT>> extractOutput() {
+ if (windowAccumulator != null) {
+ return Arrays.asList(
+ WindowedValue.of(
+ windowAccumulator, accTimestamp, accWindow, PaneInfo.ON_TIME_AND_ONLY_FIRING));
+ }
+ return Collections.emptyList();
+ }
+
+ private ValueT toValue(WindowedValue<InputT> input) {
+ try {
+ return toValue.call(input.getValue());
+ } catch (Exception ex) {
+ throw UserCodeException.wrap(ex);
+ }
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return windowAccumulator == null;
+ }
+ }
+
+ /**
+ * Abstract base class for {@link WindowedAccumulator WindowedAccumulators} which hold
+ * accumulators in map.
+ *
+ * @param <AccumT> type of accumulator
+ * @param <ImplT> type of final subclass
+ */
+ abstract static class MapBasedWindowedAccumulator<
+ InputT,
+ ValueT,
+ AccumT,
+ ImplT extends MapBasedWindowedAccumulator<InputT, ValueT, AccumT, ImplT>>
+ implements WindowedAccumulator<InputT, ValueT, AccumT, ImplT> {
+
+ final Function<InputT, ValueT> toValue;
+ final Map<BoundedWindow, WindowedValue<AccumT>> map;
+
+ MapBasedWindowedAccumulator(
+ Function<InputT, ValueT> toValue, Map<BoundedWindow, WindowedValue<AccumT>> map) {
+ this.toValue = toValue;
+ this.map = map;
+ }
+
+ @Override
+ public void add(WindowedValue<InputT> value, SparkCombineFn<InputT, ValueT, AccumT, ?> context)
+ throws Exception {
+ for (WindowedValue<InputT> v : value.explodeWindows()) {
+ SparkCombineContext ctx = context.ctxtForValue(v);
+ BoundedWindow window = getWindow(v);
+ TimestampCombiner combiner = context.windowingStrategy.getTimestampCombiner();
+ Instant windowTimestamp =
+ combiner.assign(
+ window,
+ context.windowingStrategy.getWindowFn().getOutputTime(v.getTimestamp(), window));
+ map.compute(
+ window,
+ (w, windowAccumulator) -> {
+ final AccumT acc;
+ final Instant timestamp;
+ if (windowAccumulator == null) {
+ acc = context.combineFn.createAccumulator(ctx);
+ timestamp = windowTimestamp;
+ } else {
+ acc = windowAccumulator.getValue();
+ timestamp = windowAccumulator.getTimestamp();
+ }
+ AccumT result = context.combineFn.addInput(acc, toValue(v), ctx);
+ Instant timestampCombined = combiner.combine(windowTimestamp, timestamp);
+ return WindowedValue.of(result, timestampCombined, window, PaneInfo.NO_FIRING);
+ });
+ }
+ mergeWindows(context);
+ }
+
+ @Override
+ public void merge(ImplT other, SparkCombineFn<?, ?, AccumT, ?> context) throws Exception {
+ other.map.forEach(
+ (window, acc) -> {
+ WindowedValue<AccumT> thisAcc = this.map.get(window);
+ if (thisAcc == null) {
+ // just copy
+ this.map.put(window, acc);
+ } else {
+ // merge
+ this.map.put(
+ window,
+ WindowedValue.of(
+ context.combineFn.mergeAccumulators(
+ Lists.newArrayList(thisAcc.getValue(), acc.getValue()),
+ context.ctxtForValue(acc)),
+ context
+ .windowingStrategy
+ .getTimestampCombiner()
+ .combine(acc.getTimestamp(), thisAcc.getTimestamp()),
+ window,
+ PaneInfo.NO_FIRING));
+ }
+ });
+ mergeWindows(context);
+ }
+
+ @Override
+ public Collection<WindowedValue<AccumT>> extractOutput() {
+ return map.values();
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return map.isEmpty();
+ }
+
+ void mergeWindows(SparkCombineFn<?, ?, AccumT, ?> fn) throws Exception {}
+
+ private ValueT toValue(WindowedValue<InputT> value) {
+ try {
+ return toValue.call(value.getValue());
+ } catch (Exception ex) {
+ throw UserCodeException.wrap(ex);
+ }
+ }
+ }
+
+ /** Accumulator for non-merging windows. */
+ static class NonMergingWindowedAccumulator<InputT, ValueT, AccumT>
+ extends MapBasedWindowedAccumulator<
+ InputT, ValueT, AccumT, NonMergingWindowedAccumulator<InputT, ValueT, AccumT>> {
+
+ static <InputT, ValueT, AccumT> NonMergingWindowedAccumulator<InputT, ValueT, AccumT> create(
+ Function<InputT, ValueT> toValue) {
+ return new NonMergingWindowedAccumulator<>(toValue);
+ }
+
+ static <InputT, ValueT, AccumT> NonMergingWindowedAccumulator<InputT, ValueT, AccumT> from(
+ Function<InputT, ValueT> toValue, Iterable<WindowedValue<AccumT>> values) {
+ return new NonMergingWindowedAccumulator<>(toValue, values);
+ }
+
+ @SuppressWarnings("unchecked")
+ private NonMergingWindowedAccumulator(Function<InputT, ValueT> toValue) {
+ super(toValue, new HashMap<>());
+ }
+
+ private NonMergingWindowedAccumulator(
+ Function<InputT, ValueT> toValue, Iterable<WindowedValue<AccumT>> values) {
+ super(toValue, asMap(values, new HashMap<>()));
+ }
+ }
+
+ /** Accumulator for merging windows. */
+ static class MergingWindowedAccumulator<InputT, ValueT, AccumT>
+ extends MapBasedWindowedAccumulator<
+ InputT, ValueT, AccumT, MergingWindowedAccumulator<InputT, ValueT, AccumT>> {
+
+ static <InputT, ValueT, AccumT> MergingWindowedAccumulator<InputT, ValueT, AccumT> create(
+ Function<InputT, ValueT> toValue, Comparator<BoundedWindow> windowComparator) {
+ return new MergingWindowedAccumulator<>(toValue, windowComparator);
+ }
+
+ static <InputT, ValueT, AccumT> MergingWindowedAccumulator<InputT, ValueT, AccumT> from(
+ Function<InputT, ValueT> toValue,
+ Iterable<WindowedValue<AccumT>> values,
+ Comparator<BoundedWindow> windowComparator) {
+ return new MergingWindowedAccumulator<>(toValue, values, windowComparator);
+ }
+
+ @SuppressWarnings("unchecked")
+ private MergingWindowedAccumulator(
+ Function<InputT, ValueT> toValue, Comparator<BoundedWindow> windowComparator) {
+ super(toValue, new TreeMap<>(windowComparator));
+ }
+
+ private MergingWindowedAccumulator(
+ Function<InputT, ValueT> toValue,
+ Iterable<WindowedValue<AccumT>> values,
+ Comparator<BoundedWindow> windowComparator) {
+ super(toValue, asMap(values, new TreeMap<>(windowComparator)));
+ }
+
+ @Override
+ void mergeWindows(SparkCombineFn<?, ?, AccumT, ?> fn) throws Exception {
+
+ SparkCombineContext ctx = fn.ctxtForWindows(this.map.keySet());
+
+ @SuppressWarnings("unchecked")
+ WindowFn<Object, BoundedWindow> windowFn = (WindowFn) fn.windowingStrategy.getWindowFn();
+ windowFn.mergeWindows(
+ asMergeContext(
+ windowFn,
+ (a, b) -> fn.combineFn.mergeAccumulators(Lists.newArrayList(a, b), ctx),
+ (toBeMerged, mergeResult) -> {
+ Instant mergedInstant =
+ fn.windowingStrategy
+ .getTimestampCombiner()
+ .merge(
+ mergeResult.getKey(),
+ toBeMerged.stream()
+ .map(w -> map.get(w).getTimestamp())
+ .collect(Collectors.toList()));
+ toBeMerged.forEach(this.map::remove);
+ this.map.put(
+ mergeResult.getKey(),
+ WindowedValue.of(
+ mergeResult.getValue(),
+ mergedInstant,
+ mergeResult.getKey(),
+ PaneInfo.NO_FIRING));
+ },
+ map));
+ }
+
+ private WindowFn<Object, BoundedWindow>.MergeContext asMergeContext(
+ WindowFn<Object, BoundedWindow> windowFn,
+ BiFunction<AccumT, AccumT, AccumT> mergeFn,
+ BiConsumer<Collection<BoundedWindow>, KV<BoundedWindow, AccumT>> afterMerge,
+ Map<BoundedWindow, WindowedValue<AccumT>> map) {
+
+ return windowFn.new MergeContext() {
+
+ @Override
+ public Collection<BoundedWindow> windows() {
+ return map.keySet();
+ }
+
+ @Override
+ public void merge(Collection<BoundedWindow> toBeMerged, BoundedWindow mergeResult) {
+ AccumT accumulator = null;
+ for (BoundedWindow w : toBeMerged) {
+ WindowedValue<AccumT> windowAccumulator = Objects.requireNonNull(map.get(w));
+ if (accumulator == null) {
+ accumulator = windowAccumulator.getValue();
+ } else {
+ accumulator = mergeFn.apply(accumulator, windowAccumulator.getValue());
+ }
+ }
+ afterMerge.accept(toBeMerged, KV.of(mergeResult, accumulator));
+ }
+ };
+ }
+
+ @Override
+ public String toString() {
+ return "MergingWindowedAccumulator(" + this.map + ")";
+ }
+ }
+
+ static class WindowedAccumulatorCoder<InputT, ValueT, AccumT>
+ extends Coder<WindowedAccumulator<InputT, ValueT, AccumT, ?>> {
+
+ private final Function<InputT, ValueT> toValue;
+ private final IterableCoder<WindowedValue<AccumT>> wrap;
+ private final Coder<WindowedValue<AccumT>> accumCoder;
+ private final Comparator<BoundedWindow> windowComparator;
+ private final WindowedAccumulator.Type type;
+
+ @SuppressWarnings("unchecked")
+ WindowedAccumulatorCoder(
+ Function<InputT, ValueT> toValue,
+ Coder<BoundedWindow> windowCoder,
+ Comparator<BoundedWindow> windowComparator,
+ Coder<AccumT> accumCoder,
+ WindowedAccumulator.Type type) {
+
+ this.toValue = toValue;
+ this.accumCoder = WindowedValue.FullWindowedValueCoder.of(accumCoder, windowCoder);
+ this.windowComparator = windowComparator;
+ this.wrap = IterableCoder.of(this.accumCoder);
+ this.type = type;
+ }
+
+ @Override
+ public void encode(WindowedAccumulator<InputT, ValueT, AccumT, ?> value, OutputStream outStream)
+ throws CoderException, IOException {
+ if (type.isMapBased()) {
+ wrap.encode(((MapBasedWindowedAccumulator<?, ?, AccumT, ?>) value).map.values(), outStream);
+ } else {
+ SingleWindowWindowedAccumulator<?, ?, AccumT> swwa =
+ (SingleWindowWindowedAccumulator<?, ?, AccumT>) value;
+ if (swwa.isEmpty()) {
+ outStream.write(0);
+ } else {
+ outStream.write(1);
+ accumCoder.encode(
+ WindowedValue.of(
+ swwa.windowAccumulator, swwa.accTimestamp, swwa.accWindow, PaneInfo.NO_FIRING),
+ outStream);
+ }
+ }
+ }
+
+ @Override
+ public WindowedAccumulator<InputT, ValueT, AccumT, ?> decode(InputStream inStream)
+ throws CoderException, IOException {
+ if (type.isMapBased()) {
+ return WindowedAccumulator.create(toValue, type, wrap.decode(inStream), windowComparator);
+ }
+ boolean empty = inStream.read() == 0;
+ if (empty) {
+ return WindowedAccumulator.create(toValue, type, windowComparator);
+ }
+ return WindowedAccumulator.create(
+ toValue, type, Arrays.asList(accumCoder.decode(inStream)), windowComparator);
+ }
+
+ @Override
+ public List<? extends Coder<?>> getCoderArguments() {
+ return wrap.getComponents();
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {}
+ }
+
+ /** An implementation of {@link CombineWithContext.Context} for the SparkRunner. */
+ static class SparkCombineContext extends CombineWithContext.Context {
+ private final PipelineOptions pipelineOptions;
+ private final SideInputReader sideInputReader;
+
+ SparkCombineContext(PipelineOptions pipelineOptions, SideInputReader sideInputReader) {
+ this.pipelineOptions = pipelineOptions;
+ this.sideInputReader = sideInputReader;
+ }
+
+ Collection<? extends BoundedWindow> windows = null;
+
+ SparkCombineContext forInput(final Collection<? extends BoundedWindow> windows) {
+ this.windows = Objects.requireNonNull(windows);
+ return this;
+ }
+
+ @Override
+ public PipelineOptions getPipelineOptions() {
+ return pipelineOptions;
+ }
+
+ @Override
+ public <T> T sideInput(PCollectionView<T> view) {
+ // validate element window.
+ Preconditions.checkState(
+ windows.size() == 1,
+ "sideInput can only be called when the main " + "input element is in exactly one window");
+ return sideInputReader.get(view, windows.iterator().next());
+ }
+ }
+
+ @VisibleForTesting
+ static <K, V, AccumT, OutputT> SparkCombineFn<KV<K, V>, V, AccumT, OutputT> keyed(
+ CombineWithContext.CombineFnWithContext<V, AccumT, OutputT> combineFn,
+ SerializablePipelineOptions options,
+ Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
+ WindowingStrategy<?, ?> windowingStrategy,
+ WindowedAccumulator.Type nonMergingStrategy) {
+ return new SparkCombineFn<>(
+ false, KV::getValue, combineFn, options, sideInputs, windowingStrategy, nonMergingStrategy);
+ }
+
+ public static <K, V, AccumT, OutputT> SparkCombineFn<KV<K, V>, V, AccumT, OutputT> keyed(
+ CombineWithContext.CombineFnWithContext<V, AccumT, OutputT> combineFn,
+ SerializablePipelineOptions options,
+ Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
+ WindowingStrategy<?, ?> windowingStrategy) {
+ return new SparkCombineFn<>(
+ false, KV::getValue, combineFn, options, sideInputs, windowingStrategy);
+ }
+
+ public static <InputT, AccumT, OutputT> SparkCombineFn<InputT, InputT, AccumT, OutputT> globally(
+ CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT> combineFn,
+ SerializablePipelineOptions options,
+ Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
+ WindowingStrategy<?, ?> windowingStrategy) {
+ return new SparkCombineFn<>(true, e -> e, combineFn, options, sideInputs, windowingStrategy);
+ }
+
+ private static <T> Map<BoundedWindow, WindowedValue<T>> asMap(
+ Iterable<WindowedValue<T>> values, Map<BoundedWindow, WindowedValue<T>> res) {
+ for (WindowedValue<T> v : values) {
+ res.put(getWindow(v), v);
+ }
+ return res;
+ }
+
+ /**
+ * Create comparator for given type descriptor. Note that the returned {@link Comparator} has to
+ * be {@link Serializable}.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private static Comparator<BoundedWindow> asWindowComparator(
+ @Nullable TypeDescriptor<?> windowType) {
+ final Comparator<BoundedWindow> comparator;
+ if (windowType != null
+ && StreamSupport.stream(windowType.getInterfaces().spliterator(), false)
+ .anyMatch(t -> t.isSubtypeOf(TypeDescriptor.of(Comparable.class)))) {
+ comparator = (Comparator) Comparator.naturalOrder();
+ } else {
+ java.util.function.Function<BoundedWindow, Instant> keyExtractor =
+ (java.util.function.Function<BoundedWindow, Instant> & Serializable)
+ BoundedWindow::maxTimestamp;
+ comparator = Comparator.comparing(keyExtractor);
+ }
+ return comparator;
+ }
+
+ private final boolean globalCombine;
+ private final SerializablePipelineOptions options;
+ private final Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs;
+ final WindowingStrategy<?, BoundedWindow> windowingStrategy;
+
+ private final Function<InputT, ValueT> toValue;
+ private final WindowedAccumulator.Type defaultNonMergingCombineStrategy;
+ private final CombineWithContext.CombineFnWithContext<ValueT, AccumT, OutputT> combineFn;
+ private final Comparator<BoundedWindow> windowComparator;
+
+ // each Spark task should get it's own copy of this SparkCombineFn, and since Spark tasks
+ // are single-threaded, it is safe to reuse the context.
+ // the combine context is not Serializable so we'll use lazy initialization.
+ // ** DO NOT attempt to turn this into a Singleton as Spark may run multiple tasks in parallel
+ // in the same JVM (Executor). **
+ // ** DO NOT use combineContext directly inside this class, use ctxtForValue instead. **
+ private transient SparkCombineContext combineContext;
+
+ SparkCombineFn(
+ boolean global,
+ Function<InputT, ValueT> toValue,
+ CombineWithContext.CombineFnWithContext<ValueT, AccumT, OutputT> combineFn,
+ SerializablePipelineOptions options,
+ Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
+ WindowingStrategy<?, ?> windowingStrategy) {
+ this(
+ global,
+ toValue,
+ combineFn,
+ options,
+ sideInputs,
+ windowingStrategy,
+ WindowedAccumulator.Type.EXPLODE_WINDOWS);
+ }
+
+ @VisibleForTesting
+ SparkCombineFn(
+ boolean global,
+ Function<InputT, ValueT> toValue,
+ CombineWithContext.CombineFnWithContext<ValueT, AccumT, OutputT> combineFn,
+ SerializablePipelineOptions options,
+ Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
+ WindowingStrategy<?, ?> windowingStrategy,
+ WindowedAccumulator.Type defaultNonMergingCombineStrategy) {
+
+ this.globalCombine = global;
+ this.options = options;
+ this.sideInputs = sideInputs;
+ @SuppressWarnings("unchecked")
+ WindowingStrategy<?, BoundedWindow> castStrategy = (WindowingStrategy) windowingStrategy;
+ this.windowingStrategy = castStrategy;
+ this.toValue = toValue;
+ this.defaultNonMergingCombineStrategy = defaultNonMergingCombineStrategy;
+ this.combineFn = combineFn;
+ @SuppressWarnings("unchecked")
+ TypeDescriptor<BoundedWindow> untyped =
+ (TypeDescriptor<BoundedWindow>) windowingStrategy.getWindowFn().getWindowTypeDescriptor();
+ this.windowComparator = asWindowComparator(untyped);
+ }
+
+ /** Create empty combiner. Implements Spark's zeroValue for aggregateFn. */
+ WindowedAccumulator<InputT, ValueT, AccumT, ?> createCombiner() {
+ return WindowedAccumulator.create(this, toValue, windowingStrategy, windowComparator);
+ }
+
+ /**
+ * Implements Spark's createCombiner function in:
+ *
+ * <p>{@link org.apache.spark.rdd.PairRDDFunctions#combineByKey}.
+ */
+ WindowedAccumulator<InputT, ValueT, AccumT, ?> createCombiner(WindowedValue<InputT> value) {
+ try {
+ WindowedAccumulator<InputT, ValueT, AccumT, ?> accumulator =
+ WindowedAccumulator.create(this, toValue, windowingStrategy, windowComparator);
+ accumulator.add(value, this);
+ return accumulator;
+ } catch (Exception ex) {
+ throw new IllegalStateException(ex);
+ }
+ }
+
+ /**
+ * Implements Spark's mergeValue function in:
+ *
+ * <p>{@link org.apache.spark.rdd.PairRDDFunctions#combineByKey}.
+ */
+ WindowedAccumulator<InputT, ValueT, AccumT, ?> mergeValue(
+ WindowedAccumulator<InputT, ValueT, AccumT, ?> accumulator, WindowedValue<InputT> value) {
+ try {
+ accumulator.add(value, this);
+ return accumulator;
+ } catch (Exception ex) {
+ throw new IllegalStateException(ex);
+ }
+ }
+
+ /**
+ * Implements Spark's mergeCombiners function in:
+ *
+ * <p>{@link org.apache.spark.rdd.PairRDDFunctions#combineByKey}.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ WindowedAccumulator<InputT, ValueT, AccumT, ?> mergeCombiners(
+ WindowedAccumulator ac1, WindowedAccumulator ac2) {
+ try {
+ ac1.merge(ac2, this);
+ return ac1;
+ } catch (Exception ex) {
+ throw new IllegalStateException(ex);
+ }
+ }
+
+ Iterable<WindowedValue<OutputT>> extractOutput(WindowedAccumulator<?, ?, AccumT, ?> accumulator) {
+ return extractOutputStream(accumulator).collect(Collectors.toList());
+ }
+
+ /** Extracts the stream of accumulated values. */
+ @Internal
+ public Stream<WindowedValue<OutputT>> extractOutputStream(
+ WindowedAccumulator<?, ?, AccumT, ?> accumulator) {
+ return accumulator.extractOutput().stream()
+ .filter(Objects::nonNull)
+ .map(
+ windowAcc ->
+ windowAcc.withValue(
+ combineFn.extractOutput(windowAcc.getValue(), ctxtForValue(windowAcc))));
+ }
+
+ WindowedAccumulatorCoder<InputT, ValueT, AccumT> accumulatorCoder(
+ Coder<BoundedWindow> windowCoder,
+ Coder<AccumT> accumulatorCoder,
+ WindowingStrategy<?, ?> windowingStrategy) {
+ return new WindowedAccumulatorCoder<>(
+ toValue, windowCoder, windowComparator, accumulatorCoder, getType(windowingStrategy));
+ }
+
+ CombineWithContext.CombineFnWithContext<ValueT, AccumT, OutputT> getCombineFn() {
+ return combineFn;
+ }
+
+ boolean mustBringWindowToKey() {
+ return !getType(windowingStrategy).isMapBased();
+ }
+
+ private WindowedAccumulator.Type getType(WindowingStrategy<?, ?> windowingStrategy) {
+ if (windowingStrategy.getWindowFn().isNonMerging()) {
+ if (globalCombine) {
+ /* global combine must use map-based accumulator to incorporate multiple windows */
+ return WindowedAccumulator.Type.NON_MERGING;
+ }
+ if (windowingStrategy.getWindowFn().assignsToOneWindow()
+ && GroupNonMergingWindowsFunctions.isEligibleForGroupByWindow(windowingStrategy)) {
+ return WindowedAccumulator.Type.SINGLE_WINDOW;
+ }
+ return defaultNonMergingCombineStrategy;
+ }
+ return WindowedAccumulator.Type.MERGING;
+ }
+
+ private static BoundedWindow getWindow(WindowedValue<?> value) {
+ if (value.isSingleWindowedValue()) {
+ return ((WindowedValue.SingleWindowedValue) value).getWindow();
+ }
+ return Iterables.getOnlyElement(value.getWindows());
+ }
+
+ @SuppressWarnings("unchecked")
+ SparkCombineContext ctxtForValue(WindowedValue<?> input) {
+ return ctxtForWindows((Collection) input.getWindows());
+ }
+
+ SparkCombineContext ctxtForWindows(Collection<BoundedWindow> windows) {
+ if (combineContext == null) {
+ combineContext = new SparkCombineContext(options.get(), new SparkSideInputReader(sideInputs));
+ }
+ return combineContext.forInput(windows);
+ }
+}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageContextFactory.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageContextFactory.java
new file mode 100644
index 0000000..f059c28
--- /dev/null
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageContextFactory.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.translation;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.fnexecution.control.DefaultExecutableStageContext.MultiInstanceFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+
+/**
+ * Singleton class that contains one {@link MultiInstanceFactory} per job. Assumes it is safe to
+ * release the backing environment asynchronously.
+ */
+public class SparkExecutableStageContextFactory implements ExecutableStageContext.Factory {
+
+ private static final SparkExecutableStageContextFactory instance =
+ new SparkExecutableStageContextFactory();
+ // This map should only ever have a single element, as each job will have its own
+ // classloader and therefore its own instance of SparkExecutableStageContextFactory. This
+ // code supports multiple JobInfos in order to provide a sensible implementation of
+ // Factory.get(JobInfo), which in theory could be called with different JobInfos.
+ private static final ConcurrentMap<String, MultiInstanceFactory> jobFactories =
+ new ConcurrentHashMap<>();
+
+ private SparkExecutableStageContextFactory() {}
+
+ public static SparkExecutableStageContextFactory getInstance() {
+ return instance;
+ }
+
+ @Override
+ public ExecutableStageContext get(JobInfo jobInfo) {
+ MultiInstanceFactory jobFactory =
+ jobFactories.computeIfAbsent(
+ jobInfo.jobId(),
+ k -> {
+ PortablePipelineOptions portableOptions =
+ PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions())
+ .as(PortablePipelineOptions.class);
+
+ return new MultiInstanceFactory(
+ MoreObjects.firstNonNull(portableOptions.getSdkWorkerParallelism(), 1L)
+ .intValue(),
+ // Always release environment asynchronously.
+ (caller) -> false);
+ });
+
+ return jobFactory.get(jobInfo);
+ }
+}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java
index d49300ea..2d1585b 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java
@@ -26,32 +26,44 @@
import java.util.Map;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.stream.Collectors;
+import javax.annotation.Nullable;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateKey;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateKey.TypeCase;
import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.InMemoryTimerInternals;
+import org.apache.beam.runners.core.TimerInternals;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
-import org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
import org.apache.beam.runners.fnexecution.control.RemoteBundle;
import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.control.TimerReceiverFactory;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.state.InMemoryBagUserStateFactory;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.runners.fnexecution.translation.BatchSideInputHandlerFactory;
+import org.apache.beam.runners.fnexecution.translation.PipelineTranslatorUtils;
import org.apache.beam.runners.spark.coders.CoderHelpers;
import org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator;
+import org.apache.beam.runners.spark.util.ByteArray;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.broadcast.Broadcast;
+import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import scala.Tuple2;
@@ -72,80 +84,149 @@
private final RunnerApi.ExecutableStagePayload stagePayload;
private final Map<String, Integer> outputMap;
- private final JobBundleFactoryCreator jobBundleFactoryCreator;
+ private final SparkExecutableStageContextFactory contextFactory;
// map from pCollection id to tuple of serialized bytes and coder to decode the bytes
private final Map<String, Tuple2<Broadcast<List<byte[]>>, WindowedValueCoder<SideInputT>>>
sideInputs;
private final MetricsContainerStepMapAccumulator metricsAccumulator;
+ private final Coder windowCoder;
+ private final JobInfo jobInfo;
+
+ private transient InMemoryBagUserStateFactory bagUserStateHandlerFactory;
+ private transient Object currentTimerKey;
SparkExecutableStageFunction(
RunnerApi.ExecutableStagePayload stagePayload,
JobInfo jobInfo,
Map<String, Integer> outputMap,
+ SparkExecutableStageContextFactory contextFactory,
Map<String, Tuple2<Broadcast<List<byte[]>>, WindowedValueCoder<SideInputT>>> sideInputs,
- MetricsContainerStepMapAccumulator metricsAccumulator) {
- this(
- stagePayload,
- outputMap,
- () -> DefaultJobBundleFactory.create(jobInfo),
- sideInputs,
- metricsAccumulator);
- }
-
- SparkExecutableStageFunction(
- RunnerApi.ExecutableStagePayload stagePayload,
- Map<String, Integer> outputMap,
- JobBundleFactoryCreator jobBundleFactoryCreator,
- Map<String, Tuple2<Broadcast<List<byte[]>>, WindowedValueCoder<SideInputT>>> sideInputs,
- MetricsContainerStepMapAccumulator metricsAccumulator) {
+ MetricsContainerStepMapAccumulator metricsAccumulator,
+ Coder windowCoder) {
this.stagePayload = stagePayload;
+ this.jobInfo = jobInfo;
this.outputMap = outputMap;
- this.jobBundleFactoryCreator = jobBundleFactoryCreator;
+ this.contextFactory = contextFactory;
this.sideInputs = sideInputs;
this.metricsAccumulator = metricsAccumulator;
+ this.windowCoder = windowCoder;
+ }
+
+ /** Call the executable stage function on the values of a PairRDD, ignoring the key. */
+ FlatMapFunction<Tuple2<ByteArray, Iterable<WindowedValue<InputT>>>, RawUnionValue> forPair() {
+ return (input) -> call(input._2.iterator());
}
@Override
public Iterator<RawUnionValue> call(Iterator<WindowedValue<InputT>> inputs) throws Exception {
- JobBundleFactory jobBundleFactory = jobBundleFactoryCreator.create();
- ExecutableStage executableStage = ExecutableStage.fromPayload(stagePayload);
- try (StageBundleFactory stageBundleFactory = jobBundleFactory.forStage(executableStage)) {
- ConcurrentLinkedQueue<RawUnionValue> collector = new ConcurrentLinkedQueue<>();
- ReceiverFactory receiverFactory = new ReceiverFactory(collector, outputMap);
- StateRequestHandler stateRequestHandler =
- getStateRequestHandler(executableStage, stageBundleFactory.getProcessBundleDescriptor());
- String stageName = stagePayload.getInput();
- MetricsContainerImpl container = metricsAccumulator.value().getContainer(stageName);
- BundleProgressHandler bundleProgressHandler =
- new BundleProgressHandler() {
- @Override
- public void onProgress(ProcessBundleProgressResponse progress) {
- container.update(progress.getMonitoringInfosList());
- }
+ try (ExecutableStageContext stageContext = contextFactory.get(jobInfo)) {
+ ExecutableStage executableStage = ExecutableStage.fromPayload(stagePayload);
+ try (StageBundleFactory stageBundleFactory =
+ stageContext.getStageBundleFactory(executableStage)) {
+ ConcurrentLinkedQueue<RawUnionValue> collector = new ConcurrentLinkedQueue<>();
+ StateRequestHandler stateRequestHandler =
+ getStateRequestHandler(
+ executableStage, stageBundleFactory.getProcessBundleDescriptor());
+ if (executableStage.getTimers().size() > 0) {
+ // Used with Batch, we know that all the data is available for this key. We can't use the
+ // timer manager from the context because it doesn't exist. So we create one and advance
+ // time to the end after processing all elements.
+ final InMemoryTimerInternals timerInternals = new InMemoryTimerInternals();
+ timerInternals.advanceProcessingTime(Instant.now());
+ timerInternals.advanceSynchronizedProcessingTime(Instant.now());
- @Override
- public void onCompleted(ProcessBundleResponse response) {
- container.update(response.getMonitoringInfosList());
- }
- };
- try (RemoteBundle bundle =
- stageBundleFactory.getBundle(
- receiverFactory, stateRequestHandler, bundleProgressHandler)) {
- String inputPCollectionId = executableStage.getInputPCollection().getId();
- FnDataReceiver<WindowedValue<?>> mainReceiver =
- bundle.getInputReceivers().get(inputPCollectionId);
- while (inputs.hasNext()) {
- WindowedValue<InputT> input = inputs.next();
- mainReceiver.accept(input);
+ ReceiverFactory receiverFactory =
+ new ReceiverFactory(
+ collector,
+ outputMap,
+ new TimerReceiverFactory(
+ stageBundleFactory,
+ (WindowedValue timerElement, TimerInternals.TimerData timerData) -> {
+ currentTimerKey = ((KV) timerElement.getValue()).getKey();
+ timerInternals.setTimer(timerData);
+ },
+ windowCoder));
+
+ // Process inputs.
+ processElements(
+ executableStage, stateRequestHandler, receiverFactory, stageBundleFactory, inputs);
+
+ // Finish any pending windows by advancing the input watermark to infinity.
+ timerInternals.advanceInputWatermark(BoundedWindow.TIMESTAMP_MAX_VALUE);
+ // Finally, advance the processing time to infinity to fire any timers.
+ timerInternals.advanceProcessingTime(BoundedWindow.TIMESTAMP_MAX_VALUE);
+ timerInternals.advanceSynchronizedProcessingTime(BoundedWindow.TIMESTAMP_MAX_VALUE);
+
+ // Now we fire the timers and process elements generated by timers (which may be timers
+ // itself)
+ try (RemoteBundle bundle =
+ stageBundleFactory.getBundle(
+ receiverFactory, stateRequestHandler, getBundleProgressHandler())) {
+
+ PipelineTranslatorUtils.fireEligibleTimers(
+ timerInternals,
+ (String timerId, WindowedValue timerValue) -> {
+ FnDataReceiver<WindowedValue<?>> fnTimerReceiver =
+ bundle.getInputReceivers().get(timerId);
+ Preconditions.checkNotNull(
+ fnTimerReceiver, "No FnDataReceiver found for %s", timerId);
+ try {
+ fnTimerReceiver.accept(timerValue);
+ } catch (Exception e) {
+ throw new RuntimeException(
+ String.format(Locale.ENGLISH, "Failed to process timer: %s", timerValue));
+ }
+ },
+ currentTimerKey);
+ }
+ } else {
+ ReceiverFactory receiverFactory = new ReceiverFactory(collector, outputMap);
+ processElements(
+ executableStage, stateRequestHandler, receiverFactory, stageBundleFactory, inputs);
}
+ return collector.iterator();
}
- return collector.iterator();
- } catch (Exception e) {
- LOG.error("Spark executable stage fn terminated with exception: ", e);
- throw e;
}
}
+ // Processes the inputs of the executable stage. Output is returned via side effects on the
+ // receiver.
+ private void processElements(
+ ExecutableStage executableStage,
+ StateRequestHandler stateRequestHandler,
+ ReceiverFactory receiverFactory,
+ StageBundleFactory stageBundleFactory,
+ Iterator<WindowedValue<InputT>> inputs)
+ throws Exception {
+ try (RemoteBundle bundle =
+ stageBundleFactory.getBundle(
+ receiverFactory, stateRequestHandler, getBundleProgressHandler())) {
+ String inputPCollectionId = executableStage.getInputPCollection().getId();
+ FnDataReceiver<WindowedValue<?>> mainReceiver =
+ bundle.getInputReceivers().get(inputPCollectionId);
+ while (inputs.hasNext()) {
+ WindowedValue<InputT> input = inputs.next();
+ mainReceiver.accept(input);
+ }
+ }
+ }
+
+ private BundleProgressHandler getBundleProgressHandler() {
+ String stageName = stagePayload.getInput();
+ MetricsContainerImpl container = metricsAccumulator.value().getContainer(stageName);
+ return new BundleProgressHandler() {
+ @Override
+ public void onProgress(ProcessBundleProgressResponse progress) {
+ container.update(progress.getMonitoringInfosList());
+ }
+
+ @Override
+ public void onCompleted(ProcessBundleResponse response) {
+ container.update(response.getMonitoringInfosList());
+ }
+ };
+ }
+
private StateRequestHandler getStateRequestHandler(
ExecutableStage executableStage,
ProcessBundleDescriptors.ExecutableProcessBundleDescriptor processBundleDescriptor) {
@@ -174,7 +255,24 @@
} catch (IOException e) {
throw new RuntimeException("Failed to setup state handler", e);
}
+
+ if (bagUserStateHandlerFactory == null) {
+ bagUserStateHandlerFactory = new InMemoryBagUserStateFactory();
+ }
+
+ final StateRequestHandler userStateHandler;
+ if (executableStage.getUserStates().size() > 0) {
+ // Need to discard the old key's state
+ bagUserStateHandlerFactory.resetForNewKey();
+ userStateHandler =
+ StateRequestHandlers.forBagUserStateHandlerFactory(
+ processBundleDescriptor, bagUserStateHandlerFactory);
+ } else {
+ userStateHandler = StateRequestHandler.unsupported();
+ }
+
handlerMap.put(StateKey.TypeCase.MULTIMAP_SIDE_INPUT, sideInputHandler);
+ handlerMap.put(StateKey.TypeCase.BAG_USER_STATE, userStateHandler);
return StateRequestHandlers.delegateBasedUponType(handlerMap);
}
@@ -190,22 +288,35 @@
private final ConcurrentLinkedQueue<RawUnionValue> collector;
private final Map<String, Integer> outputMap;
+ @Nullable private final TimerReceiverFactory timerReceiverFactory;
ReceiverFactory(
ConcurrentLinkedQueue<RawUnionValue> collector, Map<String, Integer> outputMap) {
+ this(collector, outputMap, null);
+ }
+
+ ReceiverFactory(
+ ConcurrentLinkedQueue<RawUnionValue> collector,
+ Map<String, Integer> outputMap,
+ @Nullable TimerReceiverFactory timerReceiverFactory) {
this.collector = collector;
this.outputMap = outputMap;
+ this.timerReceiverFactory = timerReceiverFactory;
}
@Override
public <OutputT> FnDataReceiver<OutputT> create(String pCollectionId) {
Integer unionTag = outputMap.get(pCollectionId);
- if (unionTag == null) {
+ if (unionTag != null) {
+ int tagInt = unionTag;
+ return receivedElement -> collector.add(new RawUnionValue(tagInt, receivedElement));
+ } else if (timerReceiverFactory != null) {
+ // Delegate to TimerReceiverFactory
+ return timerReceiverFactory.create(pCollectionId);
+ } else {
throw new IllegalStateException(
String.format(Locale.ENGLISH, "Unknown PCollectionId %s", pCollectionId));
}
- int tagInt = unionTag;
- return receivedElement -> collector.add(new RawUnionValue(tagInt, receivedElement));
}
}
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java
deleted file mode 100644
index 80c7d3d..0000000
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java
+++ /dev/null
@@ -1,261 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.spark.translation;
-
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-import java.util.stream.StreamSupport;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.spark.util.SideInputBroadcast;
-import org.apache.beam.sdk.transforms.CombineWithContext;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.joda.time.Instant;
-
-/**
- * A {@link org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn} with a {@link
- * CombineWithContext.Context} for the SparkRunner.
- */
-class SparkGlobalCombineFn<InputT, AccumT, OutputT> extends SparkAbstractCombineFn {
- private final CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT> combineFn;
-
- public SparkGlobalCombineFn(
- CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT> combineFn,
- SerializablePipelineOptions options,
- Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
- WindowingStrategy<?, ?> windowingStrategy) {
- super(options, sideInputs, windowingStrategy);
- this.combineFn = combineFn;
- }
-
- /**
- * Implements Spark's zeroValue function in:
- *
- * <p>{@link org.apache.spark.api.java.JavaRDD#aggregate}.
- */
- Iterable<WindowedValue<AccumT>> zeroValue() {
- return Lists.newArrayList();
- }
-
- private Iterable<WindowedValue<AccumT>> createAccumulator(WindowedValue<InputT> input) {
-
- // sort exploded inputs.
- Iterable<WindowedValue<InputT>> sortedInputs = sortByWindows(input.explodeWindows());
-
- TimestampCombiner timestampCombiner = windowingStrategy.getTimestampCombiner();
- WindowFn<?, BoundedWindow> windowFn = windowingStrategy.getWindowFn();
-
- // --- inputs iterator, by window order.
- final Iterator<WindowedValue<InputT>> iterator = sortedInputs.iterator();
- WindowedValue<InputT> currentInput = iterator.next();
- BoundedWindow currentWindow = Iterables.getFirst(currentInput.getWindows(), null);
-
- // first create the accumulator and accumulate first input.
- AccumT accumulator = combineFn.createAccumulator(ctxtForInput(currentInput));
- accumulator =
- combineFn.addInput(accumulator, currentInput.getValue(), ctxtForInput(currentInput));
-
- // keep track of the timestamps assigned by the TimestampCombiner.
- Instant windowTimestamp =
- timestampCombiner.assign(
- currentWindow,
- windowingStrategy
- .getWindowFn()
- .getOutputTime(currentInput.getTimestamp(), currentWindow));
-
- // accumulate the next windows, or output.
- List<WindowedValue<AccumT>> output = Lists.newArrayList();
-
- // if merging, merge overlapping windows, e.g. Sessions.
- final boolean merging = !windowingStrategy.getWindowFn().isNonMerging();
-
- while (iterator.hasNext()) {
- WindowedValue<InputT> nextValue = iterator.next();
- BoundedWindow nextWindow = Iterables.getOnlyElement(nextValue.getWindows());
-
- boolean mergingAndIntersecting =
- merging && isIntersecting((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
-
- if (mergingAndIntersecting || nextWindow.equals(currentWindow)) {
- if (mergingAndIntersecting) {
- // merge intersecting windows.
- currentWindow = merge((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
- }
- // keep accumulating and carry on ;-)
- accumulator =
- combineFn.addInput(accumulator, nextValue.getValue(), ctxtForInput(nextValue));
- windowTimestamp =
- timestampCombiner.merge(
- currentWindow,
- windowTimestamp,
- windowingStrategy
- .getWindowFn()
- .getOutputTime(nextValue.getTimestamp(), currentWindow));
- } else {
- // moving to the next window, first add the current accumulation to output
- // and initialize the accumulator.
- output.add(
- WindowedValue.of(accumulator, windowTimestamp, currentWindow, PaneInfo.NO_FIRING));
- // re-init accumulator, window and timestamp.
- accumulator = combineFn.createAccumulator(ctxtForInput(nextValue));
- accumulator =
- combineFn.addInput(accumulator, nextValue.getValue(), ctxtForInput(nextValue));
- currentWindow = nextWindow;
- windowTimestamp =
- timestampCombiner.assign(
- currentWindow, windowFn.getOutputTime(nextValue.getTimestamp(), currentWindow));
- }
- }
-
- // add last accumulator to the output.
- output.add(WindowedValue.of(accumulator, windowTimestamp, currentWindow, PaneInfo.NO_FIRING));
-
- return output;
- }
-
- /**
- * Implement Spark's seqOp function in:
- *
- * <p>{@link org.apache.spark.api.java.JavaRDD#aggregate}.
- */
- Iterable<WindowedValue<AccumT>> seqOp(
- Iterable<WindowedValue<AccumT>> accum, WindowedValue<InputT> input) {
- return combOp(accum, createAccumulator(input));
- }
-
- /**
- * Implement Spark's combOp function in:
- *
- * <p>{@link org.apache.spark.api.java.JavaRDD#aggregate}.
- */
- Iterable<WindowedValue<AccumT>> combOp(
- Iterable<WindowedValue<AccumT>> a1, Iterable<WindowedValue<AccumT>> a2) {
-
- // concatenate accumulators.
- Iterable<WindowedValue<AccumT>> accumulators = Iterables.concat(a1, a2);
- // if empty, return an empty accumulators iterable.
- if (!accumulators.iterator().hasNext()) {
- return Lists.newArrayList();
- }
-
- // sort accumulators, no need to explode since inputs were exploded.
- Iterable<WindowedValue<AccumT>> sortedAccumulators = sortByWindows(accumulators);
-
- TimestampCombiner timestampCombiner = windowingStrategy.getTimestampCombiner();
-
- // --- accumulators iterator, by window order.
- final Iterator<WindowedValue<AccumT>> iterator = sortedAccumulators.iterator();
-
- // get the first accumulator and assign it to the current window's accumulators.
- WindowedValue<AccumT> currentValue = iterator.next();
- BoundedWindow currentWindow = Iterables.getFirst(currentValue.getWindows(), null);
- List<AccumT> currentWindowAccumulators = Lists.newArrayList();
- currentWindowAccumulators.add(currentValue.getValue());
-
- // keep track of the timestamps assigned by the TimestampCombiner,
- // in createCombiner we already merge the timestamps assigned
- // to individual elements, here we will just merge them.
- List<Instant> windowTimestamps = Lists.newArrayList();
- windowTimestamps.add(currentValue.getTimestamp());
-
- // accumulate the next windows, or output.
- List<WindowedValue<AccumT>> output = Lists.newArrayList();
-
- // if merging, merge overlapping windows, e.g. Sessions.
- final boolean merging = !windowingStrategy.getWindowFn().isNonMerging();
-
- while (iterator.hasNext()) {
- WindowedValue<AccumT> nextValue = iterator.next();
- BoundedWindow nextWindow = Iterables.getOnlyElement(nextValue.getWindows());
-
- boolean mergingAndIntersecting =
- merging && isIntersecting((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
-
- if (mergingAndIntersecting || nextWindow.equals(currentWindow)) {
- if (mergingAndIntersecting) {
- // merge intersecting windows.
- currentWindow = merge((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
- }
- // add to window accumulators.
- currentWindowAccumulators.add(nextValue.getValue());
- windowTimestamps.add(nextValue.getTimestamp());
- } else {
- // before moving to the next window,
- // add the current accumulation to the output and initialize the accumulation.
-
- // merge the timestamps of all accumulators to merge.
- Instant mergedTimestamp = timestampCombiner.merge(currentWindow, windowTimestamps);
-
- // merge accumulators.
- // transforming a KV<K, Iterable<AccumT>> into a KV<K, Iterable<AccumT>>.
- // for the (possibly merged) window.
- Iterable<AccumT> accumsToMerge = Iterables.unmodifiableIterable(currentWindowAccumulators);
- WindowedValue<Iterable<AccumT>> preMergeWindowedValue =
- WindowedValue.of(accumsToMerge, mergedTimestamp, currentWindow, PaneInfo.NO_FIRING);
- // applying the actual combiner onto the accumulators.
- AccumT accumulated =
- combineFn.mergeAccumulators(accumsToMerge, ctxtForInput(preMergeWindowedValue));
- WindowedValue<AccumT> postMergeWindowedValue = preMergeWindowedValue.withValue(accumulated);
- // emit the accumulated output.
- output.add(postMergeWindowedValue);
-
- // re-init accumulator, window and timestamps.
- currentWindowAccumulators.clear();
- currentWindowAccumulators.add(nextValue.getValue());
- currentWindow = nextWindow;
- windowTimestamps.clear();
- windowTimestamps.add(nextValue.getTimestamp());
- }
- }
-
- // merge the last chunk of accumulators.
- Instant mergedTimestamp = timestampCombiner.merge(currentWindow, windowTimestamps);
- Iterable<AccumT> accumsToMerge = Iterables.unmodifiableIterable(currentWindowAccumulators);
- WindowedValue<Iterable<AccumT>> preMergeWindowedValue =
- WindowedValue.of(accumsToMerge, mergedTimestamp, currentWindow, PaneInfo.NO_FIRING);
- AccumT accumulated =
- combineFn.mergeAccumulators(accumsToMerge, ctxtForInput(preMergeWindowedValue));
- WindowedValue<AccumT> postMergeWindowedValue = preMergeWindowedValue.withValue(accumulated);
- output.add(postMergeWindowedValue);
-
- return output;
- }
-
- Iterable<WindowedValue<OutputT>> extractOutput(Iterable<WindowedValue<AccumT>> wvas) {
- return StreamSupport.stream(wvas.spliterator(), false)
- .map(
- wva -> {
- if (wva == null) {
- return null;
- }
- return wva.withValue(combineFn.extractOutput(wva.getValue(), ctxtForInput(wva)));
- })
- .collect(Collectors.toList());
- }
-}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowViaOutputBufferFn.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowViaOutputBufferFn.java
index f3b2600..5b6704b 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowViaOutputBufferFn.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowViaOutputBufferFn.java
@@ -34,7 +34,6 @@
import org.apache.beam.runners.core.construction.TriggerTranslation;
import org.apache.beam.runners.core.triggers.ExecutableTriggerStateMachine;
import org.apache.beam.runners.core.triggers.TriggerStateMachines;
-import org.apache.beam.runners.spark.aggregators.NamedAggregatorsAccumulator;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
@@ -47,8 +46,7 @@
/** An implementation of {@link GroupAlsoByWindow} for the Spark runner. */
class SparkGroupAlsoByWindowViaOutputBufferFn<K, InputT, W extends BoundedWindow>
implements FlatMapFunction<
- WindowedValue<KV<K, Iterable<WindowedValue<InputT>>>>,
- WindowedValue<KV<K, Iterable<InputT>>>> {
+ KV<K, Iterable<WindowedValue<InputT>>>, WindowedValue<KV<K, Iterable<InputT>>>> {
private final WindowingStrategy<?, W> windowingStrategy;
private final StateInternalsFactory<K> stateInternalsFactory;
@@ -59,8 +57,7 @@
WindowingStrategy<?, W> windowingStrategy,
StateInternalsFactory<K> stateInternalsFactory,
SystemReduceFn<K, InputT, Iterable<InputT>, Iterable<InputT>, W> reduceFn,
- SerializablePipelineOptions options,
- NamedAggregatorsAccumulator accumulator) {
+ SerializablePipelineOptions options) {
this.windowingStrategy = windowingStrategy;
this.stateInternalsFactory = stateInternalsFactory;
this.reduceFn = reduceFn;
@@ -69,9 +66,9 @@
@Override
public Iterator<WindowedValue<KV<K, Iterable<InputT>>>> call(
- WindowedValue<KV<K, Iterable<WindowedValue<InputT>>>> windowedValue) throws Exception {
- K key = windowedValue.getValue().getKey();
- Iterable<WindowedValue<InputT>> values = windowedValue.getValue().getValue();
+ KV<K, Iterable<WindowedValue<InputT>>> kv) throws Exception {
+ K key = kv.getKey();
+ Iterable<WindowedValue<InputT>> values = kv.getValue();
// ------ based on GroupAlsoByWindowsViaOutputBufferDoFn ------//
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java
deleted file mode 100644
index 9d10ec3..0000000
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java
+++ /dev/null
@@ -1,278 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.spark.translation;
-
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-import java.util.stream.StreamSupport;
-import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
-import org.apache.beam.runners.spark.util.SideInputBroadcast;
-import org.apache.beam.sdk.transforms.CombineWithContext;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.joda.time.Instant;
-
-/**
- * A {@link org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn} with a {@link
- * org.apache.beam.sdk.transforms.CombineWithContext.Context} for the SparkRunner.
- */
-public class SparkKeyedCombineFn<K, InputT, AccumT, OutputT> extends SparkAbstractCombineFn {
- private final CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT> combineFn;
-
- public SparkKeyedCombineFn(
- CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT> combineFn,
- SerializablePipelineOptions options,
- Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs,
- WindowingStrategy<?, ?> windowingStrategy) {
- super(options, sideInputs, windowingStrategy);
- this.combineFn = combineFn;
- }
-
- /** Applying the combine function directly on a key's grouped values - post grouping. */
- public OutputT apply(WindowedValue<KV<K, Iterable<InputT>>> windowedKv) {
- // apply combine function on grouped values.
- return combineFn.apply(windowedKv.getValue().getValue(), ctxtForInput(windowedKv));
- }
-
- /**
- * Implements Spark's createCombiner function in:
- *
- * <p>{@link org.apache.spark.rdd.PairRDDFunctions#combineByKey}.
- */
- Iterable<WindowedValue<KV<K, AccumT>>> createCombiner(WindowedValue<KV<K, InputT>> wkvi) {
- // sort exploded inputs.
- Iterable<WindowedValue<KV<K, InputT>>> sortedInputs = sortByWindows(wkvi.explodeWindows());
-
- TimestampCombiner timestampCombiner = windowingStrategy.getTimestampCombiner();
- WindowFn<?, BoundedWindow> windowFn = windowingStrategy.getWindowFn();
-
- // --- inputs iterator, by window order.
- final Iterator<WindowedValue<KV<K, InputT>>> iterator = sortedInputs.iterator();
- WindowedValue<KV<K, InputT>> currentInput = iterator.next();
- BoundedWindow currentWindow = Iterables.getFirst(currentInput.getWindows(), null);
-
- // first create the accumulator and accumulate first input.
- K key = currentInput.getValue().getKey();
- AccumT accumulator = combineFn.createAccumulator(ctxtForInput(currentInput));
- accumulator =
- combineFn.addInput(
- accumulator, currentInput.getValue().getValue(), ctxtForInput(currentInput));
-
- // keep track of the timestamps assigned by the TimestampCombiner.
- Instant windowTimestamp =
- timestampCombiner.assign(
- currentWindow,
- windowingStrategy
- .getWindowFn()
- .getOutputTime(currentInput.getTimestamp(), currentWindow));
-
- // accumulate the next windows, or output.
- List<WindowedValue<KV<K, AccumT>>> output = Lists.newArrayList();
-
- // if merging, merge overlapping windows, e.g. Sessions.
- final boolean merging = !windowingStrategy.getWindowFn().isNonMerging();
-
- while (iterator.hasNext()) {
- WindowedValue<KV<K, InputT>> nextValue = iterator.next();
- BoundedWindow nextWindow = Iterables.getOnlyElement(nextValue.getWindows());
-
- boolean mergingAndIntersecting =
- merging && isIntersecting((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
-
- if (mergingAndIntersecting || nextWindow.equals(currentWindow)) {
- if (mergingAndIntersecting) {
- // merge intersecting windows.
- currentWindow = merge((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
- }
- // keep accumulating and carry on ;-)
- accumulator =
- combineFn.addInput(
- accumulator, nextValue.getValue().getValue(), ctxtForInput(nextValue));
- windowTimestamp =
- timestampCombiner.combine(
- windowTimestamp,
- timestampCombiner.assign(
- currentWindow,
- windowFn.getOutputTime(nextValue.getTimestamp(), currentWindow)));
- } else {
- // moving to the next window, first add the current accumulation to output
- // and initialize the accumulator.
- output.add(
- WindowedValue.of(
- KV.of(key, accumulator), windowTimestamp, currentWindow, PaneInfo.NO_FIRING));
- // re-init accumulator, window and timestamp.
- accumulator = combineFn.createAccumulator(ctxtForInput(nextValue));
- accumulator =
- combineFn.addInput(
- accumulator, nextValue.getValue().getValue(), ctxtForInput(nextValue));
- currentWindow = nextWindow;
- windowTimestamp =
- timestampCombiner.assign(
- currentWindow, windowFn.getOutputTime(nextValue.getTimestamp(), currentWindow));
- }
- }
-
- // add last accumulator to the output.
- output.add(
- WindowedValue.of(
- KV.of(key, accumulator), windowTimestamp, currentWindow, PaneInfo.NO_FIRING));
-
- return output;
- }
-
- /**
- * Implements Spark's mergeValue function in:
- *
- * <p>{@link org.apache.spark.rdd.PairRDDFunctions#combineByKey}.
- */
- Iterable<WindowedValue<KV<K, AccumT>>> mergeValue(
- WindowedValue<KV<K, InputT>> wkvi, Iterable<WindowedValue<KV<K, AccumT>>> wkvas) {
- // by calling createCombiner on the inputs and afterwards merging the accumulators,we avoid
- // an explode&accumulate for the input that will result in poor O(n^2) performance:
- // first sort the exploded input - O(nlogn).
- // follow with an accumulators sort = O(mlogm).
- // now for each (exploded) input, find a matching accumulator (if exists) to merge into, or
- // create a new one - O(n*m).
- // this results in - O(nlogn) + O(mlogm) + O(n*m) ~> O(n^2)
- // instead, calling createCombiner will create accumulators from the input - O(nlogn) + O(n).
- // now, calling mergeCombiners will finally result in - O((n+m)log(n+m)) + O(n+m) ~> O(nlogn).
- return mergeCombiners(createCombiner(wkvi), wkvas);
- }
-
- /**
- * Implements Spark's mergeCombiners function in:
- *
- * <p>{@link org.apache.spark.rdd.PairRDDFunctions#combineByKey}.
- */
- Iterable<WindowedValue<KV<K, AccumT>>> mergeCombiners(
- Iterable<WindowedValue<KV<K, AccumT>>> a1, Iterable<WindowedValue<KV<K, AccumT>>> a2) {
- // concatenate accumulators.
- Iterable<WindowedValue<KV<K, AccumT>>> accumulators = Iterables.concat(a1, a2);
-
- // sort accumulators, no need to explode since inputs were exploded.
- Iterable<WindowedValue<KV<K, AccumT>>> sortedAccumulators = sortByWindows(accumulators);
-
- TimestampCombiner timestampCombiner = windowingStrategy.getTimestampCombiner();
-
- // --- accumulators iterator, by window order.
- final Iterator<WindowedValue<KV<K, AccumT>>> iterator = sortedAccumulators.iterator();
-
- // get the first accumulator and assign it to the current window's accumulators.
- WindowedValue<KV<K, AccumT>> currentValue = iterator.next();
- K key = currentValue.getValue().getKey();
- BoundedWindow currentWindow = Iterables.getFirst(currentValue.getWindows(), null);
- List<AccumT> currentWindowAccumulators = Lists.newArrayList();
- currentWindowAccumulators.add(currentValue.getValue().getValue());
-
- // keep track of the timestamps assigned by the TimestampCombiner,
- // in createCombiner we already merge the timestamps assigned
- // to individual elements, here we will just merge them.
- List<Instant> windowTimestamps = Lists.newArrayList();
- windowTimestamps.add(currentValue.getTimestamp());
-
- // accumulate the next windows, or output.
- List<WindowedValue<KV<K, AccumT>>> output = Lists.newArrayList();
-
- // if merging, merge overlapping windows, e.g. Sessions.
- final boolean merging = !windowingStrategy.getWindowFn().isNonMerging();
-
- while (iterator.hasNext()) {
- WindowedValue<KV<K, AccumT>> nextValue = iterator.next();
- BoundedWindow nextWindow = Iterables.getOnlyElement(nextValue.getWindows());
-
- boolean mergingAndIntersecting =
- merging && isIntersecting((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
-
- if (mergingAndIntersecting || nextWindow.equals(currentWindow)) {
- if (mergingAndIntersecting) {
- // merge intersecting windows.
- currentWindow = merge((IntervalWindow) currentWindow, (IntervalWindow) nextWindow);
- }
- // add to window accumulators.
- currentWindowAccumulators.add(nextValue.getValue().getValue());
- windowTimestamps.add(nextValue.getTimestamp());
- } else {
- // before moving to the next window,
- // add the current accumulation to the output and initialize the accumulation.
-
- // merge the timestamps of all accumulators to merge.
- Instant mergedTimestamp = timestampCombiner.merge(currentWindow, windowTimestamps);
-
- // merge accumulators.
- // transforming a KV<K, Iterable<AccumT>> into a KV<K, Iterable<AccumT>>.
- // for the (possibly merged) window.
- Iterable<AccumT> accumsToMerge = Iterables.unmodifiableIterable(currentWindowAccumulators);
- WindowedValue<KV<K, Iterable<AccumT>>> preMergeWindowedValue =
- WindowedValue.of(
- KV.of(key, accumsToMerge), mergedTimestamp, currentWindow, PaneInfo.NO_FIRING);
- // applying the actual combiner onto the accumulators.
- AccumT accumulated =
- combineFn.mergeAccumulators(accumsToMerge, ctxtForInput(preMergeWindowedValue));
- WindowedValue<KV<K, AccumT>> postMergeWindowedValue =
- preMergeWindowedValue.withValue(KV.of(key, accumulated));
- // emit the accumulated output.
- output.add(postMergeWindowedValue);
-
- // re-init accumulator, window and timestamps.
- currentWindowAccumulators.clear();
- currentWindowAccumulators.add(nextValue.getValue().getValue());
- currentWindow = nextWindow;
- windowTimestamps.clear();
- windowTimestamps.add(nextValue.getTimestamp());
- }
- }
-
- // merge the last chunk of accumulators.
- Instant mergedTimestamp = timestampCombiner.merge(currentWindow, windowTimestamps);
- Iterable<AccumT> accumsToMerge = Iterables.unmodifiableIterable(currentWindowAccumulators);
- WindowedValue<KV<K, Iterable<AccumT>>> preMergeWindowedValue =
- WindowedValue.of(
- KV.of(key, accumsToMerge), mergedTimestamp, currentWindow, PaneInfo.NO_FIRING);
- AccumT accumulated =
- combineFn.mergeAccumulators(accumsToMerge, ctxtForInput(preMergeWindowedValue));
- WindowedValue<KV<K, AccumT>> postMergeWindowedValue =
- preMergeWindowedValue.withValue(KV.of(key, accumulated));
- output.add(postMergeWindowedValue);
-
- return output;
- }
-
- Iterable<WindowedValue<OutputT>> extractOutput(Iterable<WindowedValue<KV<K, AccumT>>> wkvas) {
- return StreamSupport.stream(wkvas.spliterator(), false)
- .map(
- wkva -> {
- if (wkva == null) {
- return null;
- }
- AccumT accumulator = wkva.getValue().getValue();
- return wkva.withValue(combineFn.extractOutput(accumulator, ctxtForInput(wkva)));
- })
- .collect(Collectors.toList());
- }
-}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
index 8df4f27..e978f46 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.translation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Iterator;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.AbstractIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.AbstractIterator;
/** Spark runner process context processes Spark partitions using Beam's {@link DoFnRunner}. */
class SparkProcessContext<FnInputT, FnOutputT, OutputT> {
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkTranslationContext.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkTranslationContext.java
index 772e0d2..8c2cee8 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkTranslationContext.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkTranslationContext.java
@@ -17,12 +17,16 @@
*/
package org.apache.beam.runners.spark.translation;
+import com.sun.istack.Nullable;
+import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Set;
import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.spark.SparkPipelineOptions;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.spark.api.java.JavaSparkContext;
@@ -33,6 +37,9 @@
public class SparkTranslationContext {
private final JavaSparkContext jsc;
final JobInfo jobInfo;
+ // Map pCollection IDs to the number of times they are consumed as inputs.
+ private final Map<String, Integer> consumptionCount = new HashMap<>();
+ private final Map<String, Coder> coderMap = new HashMap<>();
private final Map<String, Dataset> datasets = new LinkedHashMap<>();
private final Set<Dataset> leaves = new LinkedHashSet<>();
final SerializablePipelineOptions serializablePipelineOptions;
@@ -51,7 +58,13 @@
/** Add output of transform to context. */
public void pushDataset(String pCollectionId, Dataset dataset) {
dataset.setName(pCollectionId);
- // TODO cache
+ SparkPipelineOptions sparkOptions =
+ serializablePipelineOptions.get().as(SparkPipelineOptions.class);
+ if (!sparkOptions.isCacheDisabled() && consumptionCount.getOrDefault(pCollectionId, 0) > 1) {
+ String storageLevel = sparkOptions.getStorageLevel();
+ @Nullable Coder coder = coderMap.get(pCollectionId);
+ dataset.cache(storageLevel, coder);
+ }
datasets.put(pCollectionId, dataset);
leaves.add(dataset);
}
@@ -70,6 +83,15 @@
}
}
+ void incrementConsumptionCountBy(String pCollectionId, int addend) {
+ int count = consumptionCount.getOrDefault(pCollectionId, 0);
+ consumptionCount.put(pCollectionId, count + addend);
+ }
+
+ void putCoder(String pCollectionId, Coder coder) {
+ coderMap.put(pCollectionId, coder);
+ }
+
/** Generate a unique pCollection id number to identify runner-generated sinks. */
public int nextSinkId() {
return sinkId++;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index 8cb79ad..4494b5d 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -18,8 +18,8 @@
package org.apache.beam.runners.spark.translation;
import static org.apache.beam.runners.spark.translation.TranslationUtils.canAvoidRddSerialization;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Collection;
import java.util.HashMap;
@@ -29,13 +29,12 @@
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.ParDoTranslation;
import org.apache.beam.runners.spark.SparkPipelineOptions;
-import org.apache.beam.runners.spark.aggregators.AggregatorsAccumulator;
-import org.apache.beam.runners.spark.aggregators.NamedAggregatorsAccumulator;
import org.apache.beam.runners.spark.coders.CoderHelpers;
import org.apache.beam.runners.spark.io.SourceRDD;
import org.apache.beam.runners.spark.metrics.MetricsAccumulator;
import org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator;
import org.apache.beam.runners.spark.util.SideInputBroadcast;
+import org.apache.beam.runners.spark.util.SparkCompat;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
@@ -54,7 +53,6 @@
import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.CombineFnUtil;
@@ -65,9 +63,8 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.spark.HashPartitioner;
import org.apache.spark.Partitioner;
import org.apache.spark.api.java.JavaPairRDD;
@@ -121,7 +118,6 @@
JavaRDD<WindowedValue<KV<K, V>>> inRDD =
((BoundedDataset<KV<K, V>>) context.borrowDataset(transform)).getRDD();
final KvCoder<K, V> coder = (KvCoder<K, V>) context.getInput(transform).getCoder();
- final NamedAggregatorsAccumulator accum = AggregatorsAccumulator.getInstance();
@SuppressWarnings("unchecked")
final WindowingStrategy<?, W> windowingStrategy =
(WindowingStrategy<?, W>) context.getInput(transform).getWindowingStrategy();
@@ -134,17 +130,15 @@
WindowedValue.FullWindowedValueCoder.of(coder.getValueCoder(), windowFn.windowCoder());
JavaRDD<WindowedValue<KV<K, Iterable<V>>>> groupedByKey;
- if (windowingStrategy.getWindowFn().isNonMerging()
- && windowingStrategy.getTimestampCombiner() == TimestampCombiner.END_OF_WINDOW) {
+ Partitioner partitioner = getPartitioner(context);
+ if (GroupNonMergingWindowsFunctions.isEligibleForGroupByWindow(windowingStrategy)) {
// we can have a memory sensitive translation for non-merging windows
groupedByKey =
GroupNonMergingWindowsFunctions.groupByKeyAndWindow(
- inRDD, keyCoder, coder.getValueCoder(), windowingStrategy);
+ inRDD, keyCoder, coder.getValueCoder(), windowingStrategy, partitioner);
} else {
-
// --- group by key only.
- Partitioner partitioner = getPartitioner(context);
- JavaRDD<WindowedValue<KV<K, Iterable<WindowedValue<V>>>>> groupedByKeyOnly =
+ JavaRDD<KV<K, Iterable<WindowedValue<V>>>> groupedByKeyOnly =
GroupCombineFunctions.groupByKeyOnly(inRDD, keyCoder, wvCoder, partitioner);
// --- now group also by window.
@@ -155,8 +149,7 @@
windowingStrategy,
new TranslationUtils.InMemoryStateInternalsFactory<>(),
SystemReduceFn.buffering(coder.getValueCoder()),
- context.getSerializableOptions(),
- accum));
+ context.getSerializableOptions()));
}
context.putDataset(transform, new BoundedDataset<>(groupedByKey));
}
@@ -169,17 +162,18 @@
}
private static <K, InputT, OutputT>
- TransformEvaluator<Combine.GroupedValues<K, InputT, OutputT>> combineGrouped() {
- return new TransformEvaluator<Combine.GroupedValues<K, InputT, OutputT>>() {
+ TransformEvaluator<Combine.GroupedValues<KV<K, InputT>, InputT, OutputT>> combineGrouped() {
+ return new TransformEvaluator<Combine.GroupedValues<KV<K, InputT>, InputT, OutputT>>() {
@Override
public void evaluate(
- Combine.GroupedValues<K, InputT, OutputT> transform, EvaluationContext context) {
+ Combine.GroupedValues<KV<K, InputT>, InputT, OutputT> transform,
+ EvaluationContext context) {
@SuppressWarnings("unchecked")
CombineWithContext.CombineFnWithContext<InputT, ?, OutputT> combineFn =
(CombineWithContext.CombineFnWithContext<InputT, ?, OutputT>)
CombineFnUtil.toFnWithContext(transform.getFn());
- final SparkKeyedCombineFn<K, InputT, ?, OutputT> sparkCombineFn =
- new SparkKeyedCombineFn<>(
+ final SparkCombineFn<KV<K, InputT>, InputT, ?, OutputT> sparkCombineFn =
+ SparkCombineFn.keyed(
combineFn,
context.getSerializableOptions(),
TranslationUtils.getSideInputs(transform.getSideInputs(), context),
@@ -189,11 +183,15 @@
JavaRDD<WindowedValue<KV<K, Iterable<InputT>>>> inRDD =
((BoundedDataset<KV<K, Iterable<InputT>>>) context.borrowDataset(transform)).getRDD();
+ @SuppressWarnings("unchecked")
JavaRDD<WindowedValue<KV<K, OutputT>>> outRDD =
inRDD.map(
in ->
WindowedValue.of(
- KV.of(in.getValue().getKey(), sparkCombineFn.apply(in)),
+ KV.of(
+ in.getValue().getKey(),
+ combineFn.apply(
+ in.getValue().getValue(), sparkCombineFn.ctxtForValue(in))),
in.getTimestamp(),
in.getWindows(),
in.getPane()));
@@ -226,8 +224,8 @@
oCoder, windowingStrategy.getWindowFn().windowCoder());
final boolean hasDefault = transform.isInsertDefault();
- final SparkGlobalCombineFn<InputT, AccumT, OutputT> sparkCombineFn =
- new SparkGlobalCombineFn<>(
+ final SparkCombineFn<InputT, InputT, AccumT, OutputT> sparkCombineFn =
+ SparkCombineFn.globally(
combineFn,
context.getSerializableOptions(),
TranslationUtils.getSideInputs(transform.getSideInputs(), context),
@@ -245,12 +243,11 @@
JavaRDD<WindowedValue<OutputT>> outRdd;
- Optional<Iterable<WindowedValue<AccumT>>> maybeAccumulated =
+ SparkCombineFn.WindowedAccumulator<InputT, InputT, AccumT, ?> accumulated =
GroupCombineFunctions.combineGlobally(inRdd, sparkCombineFn, aCoder, windowingStrategy);
- if (maybeAccumulated.isPresent()) {
- Iterable<WindowedValue<OutputT>> output =
- sparkCombineFn.extractOutput(maybeAccumulated.get());
+ if (!accumulated.isEmpty()) {
+ Iterable<WindowedValue<OutputT>> output = sparkCombineFn.extractOutput(accumulated);
outRdd =
context
.getSparkContext()
@@ -298,8 +295,8 @@
final WindowingStrategy<?, ?> windowingStrategy = input.getWindowingStrategy();
final Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> sideInputs =
TranslationUtils.getSideInputs(transform.getSideInputs(), context);
- final SparkKeyedCombineFn<K, InputT, AccumT, OutputT> sparkCombineFn =
- new SparkKeyedCombineFn<>(
+ final SparkCombineFn<KV<K, InputT>, InputT, AccumT, OutputT> sparkCombineFn =
+ SparkCombineFn.keyed(
combineFn, context.getSerializableOptions(), sideInputs, windowingStrategy);
final Coder<AccumT> vaCoder;
try {
@@ -314,15 +311,22 @@
JavaRDD<WindowedValue<KV<K, InputT>>> inRdd =
((BoundedDataset<KV<K, InputT>>) context.borrowDataset(transform)).getRDD();
- JavaPairRDD<K, Iterable<WindowedValue<KV<K, AccumT>>>> accumulatePerKey =
+ JavaPairRDD<K, SparkCombineFn.WindowedAccumulator<KV<K, InputT>, InputT, AccumT, ?>>
+ accumulatePerKey;
+ accumulatePerKey =
GroupCombineFunctions.combinePerKey(
- inRdd, sparkCombineFn, inputCoder.getKeyCoder(), vaCoder, windowingStrategy);
+ inRdd,
+ sparkCombineFn,
+ inputCoder.getKeyCoder(),
+ inputCoder.getValueCoder(),
+ vaCoder,
+ windowingStrategy);
+ JavaPairRDD<K, WindowedValue<OutputT>> kwvs =
+ SparkCompat.extractOutput(accumulatePerKey, sparkCombineFn);
JavaRDD<WindowedValue<KV<K, OutputT>>> outRdd =
- accumulatePerKey
- .flatMapValues(sparkCombineFn::extractOutput)
- .map(TranslationUtils.fromPairFunction())
- .map(TranslationUtils.toKVByWindowInValue());
+ kwvs.map(new TranslationUtils.FromPairFunction())
+ .map(new TranslationUtils.ToKVByWindowInValueFunction<>());
context.putDataset(transform, new BoundedDataset<>(outRdd));
}
@@ -363,6 +367,9 @@
doFnSchemaInformation =
ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
+ Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
+
MultiDoFnFunction<InputT, OutputT> multiDoFnFunction =
new MultiDoFnFunction<>(
metricsAccum,
@@ -373,10 +380,11 @@
transform.getAdditionalOutputTags().getAll(),
inputCoder,
outputCoders,
- TranslationUtils.getSideInputs(transform.getSideInputs(), context),
+ TranslationUtils.getSideInputs(transform.getSideInputs().values(), context),
windowingStrategy,
stateful,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
if (stateful) {
// Based on the fact that the signature is stateful, DoFnSignatures ensures
@@ -437,14 +445,14 @@
final WindowedValue.WindowedValueCoder<V> wvCoder =
WindowedValue.FullWindowedValueCoder.of(kvCoder.getValueCoder(), windowCoder);
- JavaRDD<WindowedValue<KV<K, Iterable<WindowedValue<V>>>>> groupRDD =
+ JavaRDD<KV<K, Iterable<WindowedValue<V>>>> groupRDD =
GroupCombineFunctions.groupByKeyOnly(kvInRDD, keyCoder, wvCoder, partitioner);
return groupRDD
.map(
input -> {
- final K key = input.getValue().getKey();
- Iterable<WindowedValue<V>> value = input.getValue().getValue();
+ final K key = input.getKey();
+ Iterable<WindowedValue<V>> value = input.getValue();
return FluentIterable.from(value)
.transform(
windowedValue ->
@@ -543,12 +551,11 @@
@SuppressWarnings("unchecked")
final WindowFn<Object, W> windowFn = (WindowFn<Object, W>) windowingStrategy.getWindowFn();
- final Coder<K> keyCoder = coder.getKeyCoder();
- final WindowedValue.WindowedValueCoder<V> wvCoder =
- WindowedValue.FullWindowedValueCoder.of(coder.getValueCoder(), windowFn.windowCoder());
+ final WindowedValue.WindowedValueCoder<KV<K, V>> wvCoder =
+ WindowedValue.FullWindowedValueCoder.of(coder, windowFn.windowCoder());
JavaRDD<WindowedValue<KV<K, V>>> reshuffled =
- GroupCombineFunctions.reshuffle(inRDD, keyCoder, wvCoder);
+ GroupCombineFunctions.reshuffle(inRDD, wvCoder);
context.putDataset(transform, new BoundedDataset<>(reshuffled));
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
index 8764923..16a4ca9 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.runners.spark.translation;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.Serializable;
import java.util.HashMap;
import java.util.Iterator;
-import java.util.List;
import java.util.Map;
+import javax.annotation.Nonnull;
import org.apache.beam.runners.core.InMemoryStateInternals;
import org.apache.beam.runners.core.StateInternals;
import org.apache.beam.runners.core.StateInternalsFactory;
@@ -44,9 +45,9 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function;
@@ -76,7 +77,7 @@
}
/**
- * A SparkKeyedCombineFn function applied to grouped KVs.
+ * A SparkCombineFn function applied to grouped KVs.
*
* @param <K> Grouped key type.
* @param <InputT> Grouped values type.
@@ -84,17 +85,21 @@
*/
public static class CombineGroupedValues<K, InputT, OutputT>
implements Function<WindowedValue<KV<K, Iterable<InputT>>>, WindowedValue<KV<K, OutputT>>> {
- private final SparkKeyedCombineFn<K, InputT, ?, OutputT> fn;
+ private final SparkCombineFn<KV<K, InputT>, InputT, ?, OutputT> fn;
- public CombineGroupedValues(SparkKeyedCombineFn<K, InputT, ?, OutputT> fn) {
+ public CombineGroupedValues(SparkCombineFn<KV<K, InputT>, InputT, ?, OutputT> fn) {
this.fn = fn;
}
+ @SuppressWarnings("unchecked")
@Override
public WindowedValue<KV<K, OutputT>> call(WindowedValue<KV<K, Iterable<InputT>>> windowedKv)
throws Exception {
return WindowedValue.of(
- KV.of(windowedKv.getValue().getKey(), fn.apply(windowedKv)),
+ KV.of(
+ windowedKv.getValue().getKey(),
+ fn.getCombineFn()
+ .apply(windowedKv.getValue().getValue(), fn.ctxtForValue(windowedKv))),
windowedKv.getTimestamp(),
windowedKv.getWindows(),
windowedKv.getPane());
@@ -140,8 +145,22 @@
}
/** A pair to {@link KV} function . */
- static <K, V> Function<Tuple2<K, V>, KV<K, V>> fromPairFunction() {
- return t2 -> KV.of(t2._1(), t2._2());
+ static class FromPairFunction<K, V>
+ implements Function<Tuple2<K, V>, KV<K, V>>,
+ org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function<
+ Tuple2<K, V>, KV<K, V>> {
+ @Override
+ public KV<K, V> call(Tuple2<K, V> t2) {
+ return KV.of(t2._1(), t2._2());
+ }
+
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
+ @Override
+ public KV<K, V> apply(@Nonnull Tuple2<K, V> t2) {
+ return call(t2);
+ }
}
/** A pair to {@link KV} flatmap function . */
@@ -160,11 +179,24 @@
}
/** Extract window from a {@link KV} with {@link WindowedValue} value. */
- static <K, V> Function<KV<K, WindowedValue<V>>, WindowedValue<KV<K, V>>> toKVByWindowInValue() {
- return kv -> {
+ static class ToKVByWindowInValueFunction<K, V>
+ implements Function<KV<K, WindowedValue<V>>, WindowedValue<KV<K, V>>>,
+ org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function<
+ KV<K, WindowedValue<V>>, WindowedValue<KV<K, V>>> {
+
+ @Override
+ public WindowedValue<KV<K, V>> call(KV<K, WindowedValue<V>> kv) {
WindowedValue<V> wv = kv.getValue();
return wv.withValue(KV.of(kv.getKey(), wv.getValue()));
- };
+ }
+
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
+ @Override
+ public WindowedValue<KV<K, V>> apply(@Nonnull KV<K, WindowedValue<V>> kv) {
+ return call(kv);
+ }
}
/**
@@ -195,7 +227,7 @@
* @return a map of tagged {@link SideInputBroadcast}s and their {@link WindowingStrategy}.
*/
static Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> getSideInputs(
- List<PCollectionView<?>> views, EvaluationContext context) {
+ Iterable<PCollectionView<?>> views, EvaluationContext context) {
return getSideInputs(views, context.getSparkContext(), context.getPViews());
}
@@ -208,7 +240,7 @@
* @return a map of tagged {@link SideInputBroadcast}s and their {@link WindowingStrategy}.
*/
public static Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>> getSideInputs(
- List<PCollectionView<?>> views, JavaSparkContext context, SparkPCollectionView pviews) {
+ Iterable<PCollectionView<?>> views, JavaSparkContext context, SparkPCollectionView pviews) {
if (views == null) {
return ImmutableMap.of();
} else {
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ValueAndCoderLazySerializable.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ValueAndCoderLazySerializable.java
index 193a999..5b67c93 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ValueAndCoderLazySerializable.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ValueAndCoderLazySerializable.java
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A holder object that lets you serialize an element with a Coder with minimal wasted space.
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
index e654aeb..aaeb27a 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/SparkRunnerStreamingContextFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.translation.streaming;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import org.apache.beam.runners.spark.SparkPipelineOptions;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
index 1e6c50c..194dbd6 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
@@ -18,8 +18,8 @@
package org.apache.beam.runners.spark.translation.streaming;
import static org.apache.beam.runners.spark.translation.TranslationUtils.rejectStateAndTimers;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.service.AutoService;
import java.util.ArrayList;
@@ -49,13 +49,14 @@
import org.apache.beam.runners.spark.translation.GroupCombineFunctions;
import org.apache.beam.runners.spark.translation.MultiDoFnFunction;
import org.apache.beam.runners.spark.translation.SparkAssignWindowFn;
-import org.apache.beam.runners.spark.translation.SparkKeyedCombineFn;
+import org.apache.beam.runners.spark.translation.SparkCombineFn;
import org.apache.beam.runners.spark.translation.SparkPCollectionView;
import org.apache.beam.runners.spark.translation.SparkPipelineTranslator;
import org.apache.beam.runners.spark.translation.TransformEvaluator;
import org.apache.beam.runners.spark.translation.TranslationUtils;
import org.apache.beam.runners.spark.util.GlobalWatermarkHolder;
import org.apache.beam.runners.spark.util.SideInputBroadcast;
+import org.apache.beam.runners.spark.util.SparkCompat;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.io.Read;
@@ -78,12 +79,12 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.spark.HashPartitioner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.JavaSparkContext$;
@@ -243,7 +244,7 @@
}
// start by unifying streams into a single stream.
JavaDStream<WindowedValue<T>> unifiedStreams =
- context.getStreamingContext().union(dStreams.remove(0), dStreams);
+ SparkCompat.joinStreams(context.getStreamingContext(), dStreams);
context.putDataset(transform, new UnboundedDataset<>(unifiedStreams, streamingSources));
}
@@ -301,20 +302,9 @@
final WindowedValue.WindowedValueCoder<V> wvCoder =
WindowedValue.FullWindowedValueCoder.of(coder.getValueCoder(), windowFn.windowCoder());
- // --- group by key only.
- JavaDStream<WindowedValue<KV<K, Iterable<WindowedValue<V>>>>> groupedByKeyStream =
- dStream.transform(
- rdd ->
- GroupCombineFunctions.groupByKeyOnly(
- rdd,
- coder.getKeyCoder(),
- wvCoder,
- new HashPartitioner(rdd.rdd().sparkContext().defaultParallelism())));
-
- // --- now group also by window.
JavaDStream<WindowedValue<KV<K, Iterable<V>>>> outStream =
- SparkGroupAlsoByWindowViaWindowSet.groupAlsoByWindow(
- groupedByKeyStream,
+ SparkGroupAlsoByWindowViaWindowSet.groupByKeyAndWindow(
+ dStream,
coder.getKeyCoder(),
wvCoder,
windowingStrategy,
@@ -358,8 +348,8 @@
JavaDStream<WindowedValue<KV<K, OutputT>>> outStream =
dStream.transform(
rdd -> {
- SparkKeyedCombineFn<K, InputT, ?, OutputT> combineFnWithContext =
- new SparkKeyedCombineFn<>(
+ SparkCombineFn<KV<K, InputT>, InputT, ?, OutputT> combineFnWithContext =
+ SparkCombineFn.keyed(
fn,
options,
TranslationUtils.getSideInputs(
@@ -407,6 +397,9 @@
final DoFnSchemaInformation doFnSchemaInformation =
ParDoTranslation.getSchemaInformation(context.getCurrentTransform());
+ final Map<String, PCollectionView<?>> sideInputMapping =
+ ParDoTranslation.getSideInputMapping(context.getCurrentTransform());
+
final String stepName = context.getCurrentTransform().getFullName();
JavaPairDStream<TupleTag<?>, WindowedValue<?>> all =
dStream.transformToPair(
@@ -416,7 +409,7 @@
final Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>>
sideInputs =
TranslationUtils.getSideInputs(
- transform.getSideInputs(),
+ transform.getSideInputs().values(),
JavaSparkContext.fromSparkContext(rdd.context()),
pviews);
@@ -433,7 +426,8 @@
sideInputs,
windowingStrategy,
false,
- doFnSchemaInformation));
+ doFnSchemaInformation,
+ sideInputMapping));
});
Map<TupleTag<?>, PValue> outputs = context.getOutputs(transform);
@@ -486,12 +480,11 @@
@SuppressWarnings("unchecked")
final WindowFn<Object, W> windowFn = (WindowFn<Object, W>) windowingStrategy.getWindowFn();
- final WindowedValue.WindowedValueCoder<V> wvCoder =
- WindowedValue.FullWindowedValueCoder.of(coder.getValueCoder(), windowFn.windowCoder());
+ final WindowedValue.WindowedValueCoder<KV<K, V>> wvCoder =
+ WindowedValue.FullWindowedValueCoder.of(coder, windowFn.windowCoder());
JavaDStream<WindowedValue<KV<K, V>>> reshuffledStream =
- dStream.transform(
- rdd -> GroupCombineFunctions.reshuffle(rdd, coder.getKeyCoder(), wvCoder));
+ dStream.transform(rdd -> GroupCombineFunctions.reshuffle(rdd, wvCoder));
context.putDataset(transform, new UnboundedDataset<>(reshuffledStream, streamSources));
}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/WatermarkSyncedDStream.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/WatermarkSyncedDStream.java
index a7c01b1..85dac37 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/WatermarkSyncedDStream.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/WatermarkSyncedDStream.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.runners.spark.translation.streaming;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Queue;
import java.util.concurrent.TimeUnit;
import org.apache.beam.runners.spark.util.GlobalWatermarkHolder;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext$;
import org.apache.spark.rdd.RDD;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/ByteArray.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/ByteArray.java
index 2b770d3..c53483a 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/ByteArray.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/ByteArray.java
@@ -19,7 +19,7 @@
import java.io.Serializable;
import java.util.Arrays;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
/** Serializable byte array. */
public class ByteArray implements Serializable, Comparable<ByteArray> {
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/CachedSideInputReader.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/CachedSideInputReader.java
index 5d2e521..625c5db 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/CachedSideInputReader.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/CachedSideInputReader.java
@@ -24,7 +24,7 @@
import org.apache.beam.runners.spark.util.SideInputStorage.Value;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
import org.apache.spark.util.SizeEstimator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/GlobalWatermarkHolder.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/GlobalWatermarkHolder.java
index f941114..5abc432 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/GlobalWatermarkHolder.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/GlobalWatermarkHolder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Serializable;
import java.util.HashMap;
@@ -28,11 +28,11 @@
import java.util.concurrent.TimeUnit;
import javax.annotation.Nonnull;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.spark.SparkEnv;
import org.apache.spark.broadcast.Broadcast;
import org.apache.spark.storage.BlockId;
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SideInputStorage.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SideInputStorage.java
index 482702f..8ed0b4e 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SideInputStorage.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SideInputStorage.java
@@ -21,8 +21,8 @@
import java.util.concurrent.TimeUnit;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
/**
* Cache deserialized side inputs for executor so every task doesn't need to deserialize them again.
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCompat.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCompat.java
new file mode 100644
index 0000000..ed0f286
--- /dev/null
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCompat.java
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.util;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.beam.runners.spark.translation.SparkCombineFn;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.spark.api.java.function.Function;
+import org.apache.spark.streaming.api.java.JavaDStream;
+import org.apache.spark.streaming.api.java.JavaStreamingContext;
+
+/** A set of functions to provide API compatibility between Spark 2 and Spark 3. */
+@Internal
+public class SparkCompat {
+ /**
+ * Union of dStreams in the given StreamingContext.
+ *
+ * <p>This is required because the API to join (union) DStreams is different among Spark versions.
+ * See https://issues.apache.org/jira/browse/SPARK-25737
+ */
+ public static <T> JavaDStream<WindowedValue<T>> joinStreams(
+ JavaStreamingContext streamingContext, List<JavaDStream<WindowedValue<T>>> dStreams) {
+ try {
+ if (streamingContext.sparkContext().version().startsWith("3")) {
+ // This invokes by reflection the equivalent of:
+ // return streamingContext.union(
+ // JavaConverters.asScalaIteratorConverter(dStreams.iterator()).asScala().toSeq());
+ Method method = streamingContext.getClass().getDeclaredMethod("union", JavaDStream[].class);
+ Object result =
+ method.invoke(streamingContext, new Object[] {dStreams.toArray(new JavaDStream[0])});
+ return (JavaDStream<WindowedValue<T>>) result;
+ }
+ // This invokes by reflection the equivalent of:
+ // return streamingContext.union(dStreams.remove(0), dStreams);
+ Method method =
+ streamingContext.getClass().getDeclaredMethod("union", JavaDStream.class, List.class);
+ Object result = method.invoke(streamingContext, dStreams.remove(0), dStreams);
+ return (JavaDStream<WindowedValue<T>>) result;
+ } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
+ throw new RuntimeException("Error invoking Spark union", e);
+ }
+ }
+
+ /**
+ * Extracts the output for a given collection of WindowedAccumulators.
+ *
+ * <p>This is required because the API of JavaPairRDD.flatMapValues is different among Spark
+ * versions. See https://issues.apache.org/jira/browse/SPARK-19287
+ */
+ public static <K, InputT, AccumT, OutputT> JavaPairRDD<K, WindowedValue<OutputT>> extractOutput(
+ JavaPairRDD<K, SparkCombineFn.WindowedAccumulator<KV<K, InputT>, InputT, AccumT, ?>>
+ accumulatePerKey,
+ SparkCombineFn<KV<K, InputT>, InputT, AccumT, OutputT> sparkCombineFn) {
+ try {
+ if (accumulatePerKey.context().version().startsWith("3")) {
+ FlatMapFunction<
+ SparkCombineFn.WindowedAccumulator<KV<K, InputT>, InputT, AccumT, ?>,
+ WindowedValue<OutputT>>
+ flatMapFunction =
+ (FlatMapFunction<
+ SparkCombineFn.WindowedAccumulator<KV<K, InputT>, InputT, AccumT, ?>,
+ WindowedValue<OutputT>>)
+ windowedAccumulator ->
+ sparkCombineFn.extractOutputStream(windowedAccumulator).iterator();
+ // This invokes by reflection the equivalent of:
+ // return accumulatePerKey.flatMapValues(flatMapFunction);
+ Method method =
+ accumulatePerKey.getClass().getDeclaredMethod("flatMapValues", FlatMapFunction.class);
+ Object result = method.invoke(accumulatePerKey, flatMapFunction);
+ return (JavaPairRDD<K, WindowedValue<OutputT>>) result;
+ }
+
+ Function<
+ SparkCombineFn.WindowedAccumulator<KV<K, InputT>, InputT, AccumT, ?>,
+ Iterable<WindowedValue<OutputT>>>
+ flatMapFunction =
+ windowedAccumulator ->
+ sparkCombineFn
+ .extractOutputStream(windowedAccumulator)
+ .collect(Collectors.toList());
+ // This invokes by reflection the equivalent of:
+ // return accumulatePerKey.flatMapValues(flatMapFunction);
+ Method method =
+ accumulatePerKey.getClass().getDeclaredMethod("flatMapValues", Function.class);
+ Object result = method.invoke(accumulatePerKey, flatMapFunction);
+ return (JavaPairRDD<K, WindowedValue<OutputT>>) result;
+ } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
+ throw new RuntimeException("Error invoking Spark flatMapValues", e);
+ }
+ }
+}
diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkSideInputReader.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkSideInputReader.java
index 844c825..3103357 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkSideInputReader.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkSideInputReader.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Map;
import java.util.stream.Collectors;
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** A {@link SideInputReader} for the SparkRunner. */
public class SparkSideInputReader implements SideInputReader {
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/CacheTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/CacheTest.java
index 880a466..eb9dcd9 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/CacheTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/CacheTest.java
@@ -22,6 +22,7 @@
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.mock;
+import java.util.List;
import org.apache.beam.runners.spark.translation.Dataset;
import org.apache.beam.runners.spark.translation.EvaluationContext;
import org.apache.beam.runners.spark.translation.SparkContextFactory;
@@ -32,8 +33,12 @@
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.Create.Values;
+import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
import org.apache.spark.api.java.JavaSparkContext;
import org.junit.Test;
@@ -49,12 +54,28 @@
SparkPipelineOptions options = createOptions();
Pipeline pipeline = Pipeline.create(options);
PCollection<String> pCollection = pipeline.apply(Create.of("foo", "bar"));
- // first read
+
+ // First use of pCollection.
pCollection.apply(Count.globally());
- // second read
- // as we access the same PCollection two times, the Spark runner does optimization and so
- // will cache the RDD representing this PCollection
- pCollection.apply(Count.globally());
+ // Second use of pCollection.
+ PCollectionView<List<String>> view = pCollection.apply(View.asList());
+
+ // Internally View.asList() creates a PCollection that underlies the PCollectionView, that
+ // PCollection should not be cached as the SparkRunner does not access that PCollection to
+ // access the PCollectionView.
+ pipeline
+ .apply(Create.of("foo", "baz"))
+ .apply(
+ ParDo.of(
+ new DoFn<String, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext processContext) {
+ if (processContext.sideInput(view).contains(processContext.element())) {
+ processContext.output(processContext.element());
+ }
+ }
+ })
+ .withSideInputs(view));
JavaSparkContext jsc = SparkContextFactory.getSparkContext(options);
EvaluationContext ctxt = new EvaluationContext(jsc, pipeline, options);
@@ -62,6 +83,7 @@
new SparkRunner.CacheVisitor(new TransformTranslator.Translator(), ctxt);
pipeline.traverseTopologically(cacheVisitor);
assertEquals(2L, (long) ctxt.getCacheCandidates().get(pCollection));
+ assertEquals(1L, ctxt.getCacheCandidates().values().stream().filter(l -> l > 1).count());
}
@Test
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
index 6357b64..4a57ade 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/ProvidedSparkContextTest.java
@@ -30,8 +30,8 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.apache.spark.api.java.JavaSparkContext;
import org.junit.Test;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkPortableExecutionTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkPortableExecutionTest.java
index d7d3428..7df4ec6 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkPortableExecutionTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkPortableExecutionTest.java
@@ -17,9 +17,11 @@
*/
package org.apache.beam.runners.spark;
+import java.io.File;
import java.io.Serializable;
+import java.nio.file.FileSystems;
import java.util.Collections;
-import java.util.concurrent.Executors;
+import java.util.UUID;
import java.util.concurrent.TimeUnit;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobState.Enum;
import org.apache.beam.model.pipeline.v1.RunnerApi;
@@ -42,12 +44,15 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.junit.AfterClass;
+import org.junit.Assert;
import org.junit.BeforeClass;
+import org.junit.ClassRule;
import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -56,15 +61,13 @@
*/
public class SparkPortableExecutionTest implements Serializable {
+ @ClassRule public static TemporaryFolder temporaryFolder = new TemporaryFolder();
private static final Logger LOG = LoggerFactory.getLogger(SparkPortableExecutionTest.class);
-
private static ListeningExecutorService sparkJobExecutor;
@BeforeClass
public static void setup() {
- // Restrict this to only one thread to avoid multiple Spark clusters up at the same time
- // which is not suitable for memory-constraint environments, i.e. Jenkins.
- sparkJobExecutor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(1));
+ sparkJobExecutor = MoreExecutors.newDirectExecutorService();
}
@AfterClass
@@ -159,8 +162,117 @@
pipelineProto,
options.as(SparkPipelineOptions.class));
jobInvocation.start();
- while (jobInvocation.getState() != Enum.DONE) {
- Thread.sleep(1000);
+ Assert.assertEquals(Enum.DONE, jobInvocation.getState());
+ }
+
+ /**
+ * Verifies that each executable stage runs exactly once, even if that executable stage has
+ * multiple immediate outputs. While re-computation may be necessary in the event of failure,
+ * re-computation of a whole executable stage is expensive and can cause unexpected behavior when
+ * the executable stage has side effects (BEAM-7131).
+ *
+ * <pre>
+ * |-> B -> GBK
+ * A -|
+ * |-> C -> GBK
+ * </pre>
+ */
+ @Test(timeout = 120_000)
+ public void testExecStageWithMultipleOutputs() throws Exception {
+ PipelineOptions options = PipelineOptionsFactory.create();
+ options.setRunner(CrashingRunner.class);
+ options
+ .as(PortablePipelineOptions.class)
+ .setDefaultEnvironmentType(Environments.ENVIRONMENT_EMBEDDED);
+ Pipeline pipeline = Pipeline.create(options);
+ PCollection<KV<String, String>> a =
+ pipeline
+ .apply("impulse", Impulse.create())
+ .apply("A", ParDo.of(new DoFnWithSideEffect<>("A")));
+ PCollection<KV<String, String>> b = a.apply("B", ParDo.of(new DoFnWithSideEffect<>("B")));
+ PCollection<KV<String, String>> c = a.apply("C", ParDo.of(new DoFnWithSideEffect<>("C")));
+ // Use GBKs to force re-computation of executable stage unless cached.
+ b.apply(GroupByKey.create());
+ c.apply(GroupByKey.create());
+ RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline);
+ JobInvocation jobInvocation =
+ SparkJobInvoker.createJobInvocation(
+ "testExecStageWithMultipleOutputs",
+ "testExecStageWithMultipleOutputsRetrievalToken",
+ sparkJobExecutor,
+ pipelineProto,
+ options.as(SparkPipelineOptions.class));
+ jobInvocation.start();
+ Assert.assertEquals(Enum.DONE, jobInvocation.getState());
+ }
+
+ /**
+ * Verifies that each executable stage runs exactly once, even if that executable stage has
+ * multiple downstream consumers. While re-computation may be necessary in the event of failure,
+ * re-computation of a whole executable stage is expensive and can cause unexpected behavior when
+ * the executable stage has side effects (BEAM-7131).
+ *
+ * <pre>
+ * |-> G
+ * F -> GBK -|
+ * |-> H
+ * </pre>
+ */
+ @Test(timeout = 120_000)
+ public void testExecStageWithMultipleConsumers() throws Exception {
+ PipelineOptions options = PipelineOptionsFactory.create();
+ options.setRunner(CrashingRunner.class);
+ options
+ .as(PortablePipelineOptions.class)
+ .setDefaultEnvironmentType(Environments.ENVIRONMENT_EMBEDDED);
+ Pipeline pipeline = Pipeline.create(options);
+ PCollection<KV<String, Iterable<String>>> f =
+ pipeline
+ .apply("impulse", Impulse.create())
+ .apply("F", ParDo.of(new DoFnWithSideEffect<>("F")))
+ // use GBK to prevent fusion of F, G, and H
+ .apply(GroupByKey.create());
+ f.apply("G", ParDo.of(new DoFnWithSideEffect<>("G")));
+ f.apply("H", ParDo.of(new DoFnWithSideEffect<>("H")));
+ RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline);
+ JobInvocation jobInvocation =
+ SparkJobInvoker.createJobInvocation(
+ "testExecStageWithMultipleConsumers",
+ "testExecStageWithMultipleConsumersRetrievalToken",
+ sparkJobExecutor,
+ pipelineProto,
+ options.as(SparkPipelineOptions.class));
+ jobInvocation.start();
+ Assert.assertEquals(Enum.DONE, jobInvocation.getState());
+ }
+
+ /** A non-idempotent DoFn that cannot be run more than once without error. */
+ private class DoFnWithSideEffect<InputT> extends DoFn<InputT, KV<String, String>> {
+
+ private final String name;
+ private final File file;
+
+ DoFnWithSideEffect(String name) {
+ this.name = name;
+ String path =
+ FileSystems.getDefault()
+ .getPath(
+ temporaryFolder.getRoot().getAbsolutePath(),
+ String.format("%s-%s", this.name, UUID.randomUUID().toString()))
+ .toString();
+ file = new File(path);
+ }
+
+ @ProcessElement
+ public void process(ProcessContext context) throws Exception {
+ context.output(KV.of(name, name));
+ // Verify this DoFn has not run more than once by enacting a side effect via the local file
+ // system.
+ Assert.assertTrue(
+ String.format(
+ "Create file %s failed (DoFn %s should only have been run once).",
+ file.getAbsolutePath(), name),
+ file.createNewFile());
}
}
}
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
index f84945f..2d12cd8 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
@@ -84,7 +84,7 @@
+ "_.mapPartitions(new org.apache.beam.sdk.transforms.Contextful())\n"
+ "_.combineByKey(..., new org.apache.beam.sdk.transforms.Count$CountFn(), ...)\n"
+ "_.groupByKey()\n"
- + "_.mapPartitions(new org.apache.beam.sdk.transforms.Combine$GroupedValues$1())\n"
+ + "_.map(new org.apache.beam.sdk.transforms.Sum$SumLongFn())\n"
+ "_.mapPartitions(new org.apache.beam.sdk.transforms.Contextful())\n"
+ "sparkContext.union(...)\n"
+ "_.mapPartitions("
@@ -141,7 +141,7 @@
+ "SparkRunnerDebuggerTest$FormatKVFn())\n"
+ "_.mapPartitions(new org.apache.beam.sdk.transforms.Contextful())\n"
+ "_.groupByKey()\n"
- + "_.mapPartitions(new org.apache.beam.sdk.transforms.Combine$GroupedValues$1())\n"
+ + "_.map(new org.apache.beam.sdk.transforms.Combine$IterableCombineFn())\n"
+ "_.mapPartitions(new org.apache.beam.sdk.transforms.Distinct$3())\n"
+ "_.mapPartitions(new org.apache.beam.sdk.transforms.Contextful())\n"
+ "_.<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write>";
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerRegistrarTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerRegistrarTest.java
index 5046eeb..03852a7 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerRegistrarTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerRegistrarTest.java
@@ -23,8 +23,8 @@
import java.util.ServiceLoader;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
import org.apache.beam.sdk.runners.PipelineRunnerRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/TestSparkPipelineOptionsRegistrar.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/TestSparkPipelineOptionsRegistrar.java
index 7044b72..2199e1d 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/TestSparkPipelineOptionsRegistrar.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/TestSparkPipelineOptionsRegistrar.java
@@ -20,7 +20,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A registrar for {@link TestSparkPipelineOptions} to temporarily work around some complexities in
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/InMemoryMetrics.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/InMemoryMetrics.java
index 7b72b54..a4b3e54 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/InMemoryMetrics.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/InMemoryMetrics.java
@@ -21,7 +21,7 @@
import com.codahale.metrics.MetricRegistry;
import java.util.Properties;
import org.apache.beam.runners.spark.metrics.WithMetricsSupport;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
import org.apache.spark.metrics.sink.Sink;
/** An in-memory {@link Sink} implementation for tests. */
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/SparkMetricsSinkTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/SparkMetricsSinkTest.java
index bb76a5d..4050ec1 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/SparkMetricsSinkTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/aggregators/metrics/sink/SparkMetricsSinkTest.java
@@ -35,8 +35,8 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/io/AvroPipelineTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/io/AvroPipelineTest.java
index 51e7132..fa49a9f 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/io/AvroPipelineTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/io/AvroPipelineTest.java
@@ -33,8 +33,8 @@
import org.apache.beam.sdk.io.AvroIO;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Resources;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Resources;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/io/NumShardsTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/io/NumShardsTest.java
index 5f080fb..777d1d4 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/io/NumShardsTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/io/NumShardsTest.java
@@ -33,8 +33,8 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -82,6 +82,6 @@
}
}
assertEquals(3, count);
- assertTrue(expected.isEmpty());
+ assertTrue(expected.toString(), expected.isEmpty());
}
}
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/io/ReaderToIteratorAdapterTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/io/ReaderToIteratorAdapterTest.java
index e0f5901..019cbe6 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/io/ReaderToIteratorAdapterTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/io/ReaderToIteratorAdapterTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.runners.spark.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertThat;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctionsTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctionsTest.java
index 6d348a1..ddc1c53 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctionsTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/GroupNonMergingWindowsFunctionsTest.java
@@ -24,18 +24,22 @@
import java.util.List;
import org.apache.beam.runners.spark.coders.CoderHelpers;
import org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions.GroupByKeyIterator;
-import org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions.WindowedKey;
-import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.runners.spark.util.ByteArray;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
+import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Assert;
import org.junit.Test;
@@ -45,7 +49,7 @@
public class GroupNonMergingWindowsFunctionsTest {
@Test
- public void testGroupByKeyIterator() {
+ public void testGroupByKeyIterator() throws Coder.NonDeterministicException {
GroupByKeyIterator<String, Integer, GlobalWindow> iteratorUnderTest = createGbkIterator();
Assert.assertTrue(iteratorUnderTest.hasNext());
@@ -67,8 +71,36 @@
assertEquals(3L, valuesIteratorForK2.next().longValue());
}
+ @Test
+ public void testGroupByKeyIteratorOnNonGlobalWindows() throws Coder.NonDeterministicException {
+ Instant now = Instant.now();
+ GroupByKeyIterator<String, Integer, IntervalWindow> iteratorUnderTest =
+ createGbkIterator(
+ new IntervalWindow(now, now.plus(1)),
+ IntervalWindow.getCoder(),
+ WindowingStrategy.of(FixedWindows.of(Duration.millis(1))));
+
+ Assert.assertTrue(iteratorUnderTest.hasNext());
+ WindowedValue<KV<String, Iterable<Integer>>> k1Win = iteratorUnderTest.next();
+ // testing that calling 2x hasNext doesn't move to next key iterator
+ Assert.assertTrue(iteratorUnderTest.hasNext());
+ Assert.assertTrue(iteratorUnderTest.hasNext());
+
+ Iterator<Integer> valuesIteratorForK1 = k1Win.getValue().getValue().iterator();
+
+ Assert.assertTrue("Now we expect first value for K1 to pop up.", valuesIteratorForK1.hasNext());
+ assertEquals(1L, valuesIteratorForK1.next().longValue());
+ Assert.assertTrue(valuesIteratorForK1.hasNext());
+ Assert.assertTrue(valuesIteratorForK1.hasNext());
+ assertEquals(2L, valuesIteratorForK1.next().longValue());
+
+ WindowedValue<KV<String, Iterable<Integer>>> k2Win = iteratorUnderTest.next();
+ Iterator<Integer> valuesIteratorForK2 = k2Win.getValue().getValue().iterator();
+ assertEquals(3L, valuesIteratorForK2.next().longValue());
+ }
+
@Test(expected = IllegalStateException.class)
- public void testGbkIteratorValuesCannotBeReiterated() {
+ public void testGbkIteratorValuesCannotBeReiterated() throws Coder.NonDeterministicException {
GroupByKeyIterator<String, Integer, GlobalWindow> iteratorUnderTest = createGbkIterator();
WindowedValue<KV<String, Iterable<Integer>>> firstEl = iteratorUnderTest.next();
Iterable<Integer> value = firstEl.getValue().getValue();
@@ -80,54 +112,76 @@
}
}
- private GroupByKeyIterator<String, Integer, GlobalWindow> createGbkIterator() {
+ private GroupByKeyIterator<String, Integer, GlobalWindow> createGbkIterator()
+ throws Coder.NonDeterministicException {
+ return createGbkIterator(
+ GlobalWindow.INSTANCE, GlobalWindow.Coder.INSTANCE, WindowingStrategy.globalDefault());
+ }
+
+ private <W extends BoundedWindow> GroupByKeyIterator<String, Integer, W> createGbkIterator(
+ W window, Coder<W> winCoder, WindowingStrategy<Object, W> winStrategy)
+ throws Coder.NonDeterministicException {
+
StringUtf8Coder keyCoder = StringUtf8Coder.of();
- BigEndianIntegerCoder valueCoder = BigEndianIntegerCoder.of();
- WindowingStrategy<Object, GlobalWindow> winStrategy = WindowingStrategy.of(new GlobalWindows());
+ final WindowedValue.FullWindowedValueCoder<KV<String, Integer>> winValCoder =
+ WindowedValue.getFullCoder(
+ KvCoder.of(StringUtf8Coder.of(), VarIntCoder.of()),
+ winStrategy.getWindowFn().windowCoder());
- final WindowedValue.FullWindowedValueCoder<byte[]> winValCoder =
- WindowedValue.getFullCoder(ByteArrayCoder.of(), winStrategy.getWindowFn().windowCoder());
-
- ItemFactory<String, Integer> factory =
- new ItemFactory<>(
- keyCoder, valueCoder, winValCoder, winStrategy.getWindowFn().windowCoder());
- List<Tuple2<WindowedKey, byte[]>> items =
+ ItemFactory<String, Integer, W> factory =
+ ItemFactory.forWindow(keyCoder, winValCoder, winCoder, window);
+ List<Tuple2<ByteArray, byte[]>> items =
Arrays.asList(
factory.create("k1", 1),
factory.create("k1", 2),
factory.create("k2", 3),
factory.create("k2", 4),
factory.create("k2", 5));
- return new GroupByKeyIterator<>(
- items.iterator(), keyCoder, valueCoder, winStrategy, winValCoder);
+ return new GroupByKeyIterator<>(items.iterator(), keyCoder, winStrategy, winValCoder);
}
- private static class ItemFactory<K, V> {
+ private static class ItemFactory<K, V, W extends BoundedWindow> {
+
+ static <K, V> ItemFactory<K, V, GlobalWindow> forGlogalWindow(
+ Coder<K> keyCoder, FullWindowedValueCoder<KV<K, V>> winValCoder) {
+ return new ItemFactory<>(
+ keyCoder, winValCoder, GlobalWindow.Coder.INSTANCE, GlobalWindow.INSTANCE);
+ }
+
+ static <K, V, W extends BoundedWindow> ItemFactory<K, V, W> forWindow(
+ Coder<K> keyCoder,
+ FullWindowedValueCoder<KV<K, V>> winValCoder,
+ Coder<W> winCoder,
+ W window) {
+ return new ItemFactory<>(keyCoder, winValCoder, winCoder, window);
+ }
+
private final Coder<K> keyCoder;
- private final Coder<V> valueCoder;
- private final WindowedValue.FullWindowedValueCoder<byte[]> winValCoder;
- private final byte[] globalWindow;
+ private final WindowedValue.FullWindowedValueCoder<KV<K, V>> winValCoder;
+ private final byte[] windowBytes;
+ private final W window;
ItemFactory(
Coder<K> keyCoder,
- Coder<V> valueCoder,
- FullWindowedValueCoder<byte[]> winValCoder,
- Coder<GlobalWindow> winCoder) {
+ FullWindowedValueCoder<KV<K, V>> winValCoder,
+ Coder<W> winCoder,
+ W window) {
this.keyCoder = keyCoder;
- this.valueCoder = valueCoder;
this.winValCoder = winValCoder;
- this.globalWindow = CoderHelpers.toByteArray(GlobalWindow.INSTANCE, winCoder);
+ this.windowBytes = CoderHelpers.toByteArray(window, winCoder);
+ this.window = window;
}
- private Tuple2<WindowedKey, byte[]> create(K key, V value) {
- WindowedKey kaw = new WindowedKey(CoderHelpers.toByteArray(key, keyCoder), globalWindow);
+ private Tuple2<ByteArray, byte[]> create(K key, V value) {
+ ByteArray kaw =
+ new ByteArray(Bytes.concat(CoderHelpers.toByteArray(key, keyCoder), windowBytes));
- byte[] valueInbytes = CoderHelpers.toByteArray(value, valueCoder);
-
- WindowedValue<byte[]> windowedValue =
- WindowedValue.of(
- valueInbytes, Instant.now(), GlobalWindow.INSTANCE, PaneInfo.ON_TIME_AND_ONLY_FIRING);
- return new Tuple2<>(kaw, CoderHelpers.toByteArray(windowedValue, winValCoder));
+ byte[] windowedValue =
+ CoderHelpers.toByteArray(
+ WindowedValue.of(
+ KV.of(key, value), Instant.now(), window, PaneInfo.ON_TIME_AND_ONLY_FIRING),
+ winValCoder);
+ return new Tuple2<>(kaw, windowedValue);
}
}
}
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkCombineFnTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkCombineFnTest.java
new file mode 100644
index 0000000..d8d4ae7
--- /dev/null
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkCombineFnTest.java
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.translation;
+
+import static org.junit.Assert.assertEquals;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+import org.apache.beam.runners.core.construction.SerializablePipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.CombineWithContext;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.CombineFnUtil;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Test;
+
+/** * Test suite for {@link SparkCombineFn}. */
+public class SparkCombineFnTest {
+
+ final SerializablePipelineOptions opts =
+ new SerializablePipelineOptions(PipelineOptionsFactory.create());
+ CombineWithContext.CombineFnWithContext<Integer, Long, Long> combineFn;
+
+ @Before
+ public void setUp() {
+ combineFn =
+ (CombineWithContext.CombineFnWithContext<Integer, Long, Long>)
+ CombineFnUtil.toFnWithContext(getSumFn());
+ }
+
+ @Test
+ public void testGlobalWindowCombineFn() throws Exception {
+ SparkCombineFn<KV<String, Integer>, Integer, Long, Long> sparkCombineFn =
+ SparkCombineFn.keyed(
+ combineFn, opts, Collections.emptyMap(), WindowingStrategy.globalDefault());
+
+ WindowedValue<KV<String, Integer>> first = input("key", 1, Instant.now());
+ WindowedValue<KV<String, Integer>> second = input("key", 2, Instant.now());
+ WindowedValue<KV<String, Integer>> third = input("key", 3, Instant.now());
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c1 =
+ sparkCombineFn.createCombiner(first);
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c2 =
+ sparkCombineFn.createCombiner(third);
+ sparkCombineFn.mergeValue(c1, second);
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c3 =
+ sparkCombineFn.mergeCombiners(c1, c2);
+ assertEquals(6, (long) Iterables.getOnlyElement(sparkCombineFn.extractOutput(c3)).getValue());
+ }
+
+ @Test
+ public void testGlobalCombineFn() throws Exception {
+ SparkCombineFn<Integer, Integer, Long, Long> sparkCombineFn =
+ SparkCombineFn.globally(
+ combineFn, opts, Collections.emptyMap(), WindowingStrategy.globalDefault());
+
+ WindowedValue<Integer> first = inputValue(1, Instant.now());
+ WindowedValue<Integer> second = inputValue(2, Instant.now());
+ WindowedValue<Integer> third = inputValue(3, Instant.now());
+ SparkCombineFn.WindowedAccumulator<Integer, Integer, Long, ?> c1 =
+ sparkCombineFn.createCombiner(first);
+ SparkCombineFn.WindowedAccumulator<Integer, Integer, Long, ?> c2 =
+ sparkCombineFn.createCombiner(third);
+ sparkCombineFn.mergeValue(c1, second);
+ SparkCombineFn.WindowedAccumulator<Integer, Integer, Long, ?> c3 =
+ sparkCombineFn.mergeCombiners(c1, c2);
+ assertEquals(6, (long) Iterables.getOnlyElement(sparkCombineFn.extractOutput(c3)).getValue());
+ }
+
+ @Test
+ public void testSessionCombineFn() throws Exception {
+ WindowingStrategy<Object, IntervalWindow> strategy =
+ WindowingStrategy.of(Sessions.withGapDuration(Duration.millis(1000)));
+
+ SparkCombineFn<KV<String, Integer>, Integer, Long, Long> sparkCombineFn =
+ SparkCombineFn.keyed(combineFn, opts, Collections.emptyMap(), strategy);
+
+ Instant now = Instant.ofEpochMilli(0);
+ WindowedValue<KV<String, Integer>> first =
+ input("key", 1, now.plus(5000), strategy.getWindowFn());
+ WindowedValue<KV<String, Integer>> second =
+ input("key", 2, now.plus(1000), strategy.getWindowFn());
+ WindowedValue<KV<String, Integer>> third =
+ input("key", 3, now.plus(500), strategy.getWindowFn());
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c1 =
+ sparkCombineFn.createCombiner(first);
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c2 =
+ sparkCombineFn.createCombiner(third);
+ sparkCombineFn.mergeValue(c1, second);
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c3 =
+ sparkCombineFn.mergeCombiners(c1, c2);
+ Iterable<WindowedValue<Long>> output = sparkCombineFn.extractOutput(c3);
+ assertEquals(2, Iterables.size(output));
+ List<String> format =
+ StreamSupport.stream(output.spliterator(), false)
+ .map(val -> val.getValue() + ":" + val.getTimestamp().getMillis())
+ .collect(Collectors.toList());
+ assertEquals(Lists.newArrayList("5:1999", "1:5999"), format);
+ }
+
+ @Test
+ public void testSlidingCombineFnNonMerging() throws Exception {
+ WindowingStrategy<Object, IntervalWindow> strategy =
+ WindowingStrategy.of(SlidingWindows.of(Duration.millis(3000)).every(Duration.millis(1000)));
+
+ SparkCombineFn<KV<String, Integer>, Integer, Long, Long> sparkCombineFn =
+ SparkCombineFn.keyed(
+ combineFn,
+ opts,
+ Collections.emptyMap(),
+ strategy,
+ SparkCombineFn.WindowedAccumulator.Type.NON_MERGING);
+
+ Instant now = Instant.ofEpochMilli(0);
+ WindowedValue<KV<String, Integer>> first =
+ input("key", 1, now.plus(5000), strategy.getWindowFn());
+ WindowedValue<KV<String, Integer>> second =
+ input("key", 2, now.plus(1500), strategy.getWindowFn());
+ WindowedValue<KV<String, Integer>> third =
+ input("key", 3, now.plus(500), strategy.getWindowFn());
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c1 =
+ sparkCombineFn.createCombiner(first);
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c2 =
+ sparkCombineFn.createCombiner(third);
+ sparkCombineFn.mergeValue(c1, second);
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> c3 =
+ sparkCombineFn.mergeCombiners(c1, c2);
+ Iterable<WindowedValue<Long>> output = sparkCombineFn.extractOutput(c3);
+ assertEquals(7, Iterables.size(output));
+ List<String> format =
+ StreamSupport.stream(output.spliterator(), false)
+ .map(val -> val.getValue() + ":" + val.getTimestamp().getMillis())
+ .collect(Collectors.toList());
+ assertUnorderedEquals(
+ Lists.newArrayList("3:999", "5:1999", "5:2999", "2:3999", "1:5999", "1:6999", "1:7999"),
+ format);
+ }
+
+ @Test
+ public void testSlidingCombineFnExplode() throws Exception {
+ WindowingStrategy<Object, IntervalWindow> strategy =
+ WindowingStrategy.of(SlidingWindows.of(Duration.millis(3000)).every(Duration.millis(1000)));
+
+ SparkCombineFn<KV<String, Integer>, Integer, Long, Long> sparkCombineFn =
+ SparkCombineFn.keyed(
+ combineFn,
+ opts,
+ Collections.emptyMap(),
+ strategy,
+ SparkCombineFn.WindowedAccumulator.Type.EXPLODE_WINDOWS);
+
+ Instant now = Instant.ofEpochMilli(0);
+ WindowedValue<KV<String, Integer>> first =
+ input("key", 1, now.plus(5000), strategy.getWindowFn());
+ WindowedValue<KV<String, Integer>> second =
+ input("key", 2, now.plus(1500), strategy.getWindowFn());
+ WindowedValue<KV<String, Integer>> third =
+ input("key", 3, now.plus(500), strategy.getWindowFn());
+
+ Map<KV<String, BoundedWindow>, List<WindowedValue<KV<String, Integer>>>> groupByKeyAndWindow;
+ groupByKeyAndWindow =
+ Stream.of(first, second, third)
+ .flatMap(e -> StreamSupport.stream(e.explodeWindows().spliterator(), false))
+ .collect(
+ Collectors.groupingBy(
+ e -> KV.of(e.getValue().getKey(), Iterables.getOnlyElement(e.getWindows()))));
+
+ List<String> result = new ArrayList<>();
+ for (Map.Entry<KV<String, BoundedWindow>, List<WindowedValue<KV<String, Integer>>>> e :
+ groupByKeyAndWindow.entrySet()) {
+
+ SparkCombineFn.WindowedAccumulator<KV<String, Integer>, Integer, Long, ?> combiner = null;
+ for (WindowedValue<KV<String, Integer>> v : e.getValue()) {
+ if (combiner == null) {
+ combiner = sparkCombineFn.createCombiner(v);
+ } else {
+ combiner.add(v, sparkCombineFn);
+ }
+ }
+ WindowedValue<Long> combined = Iterables.getOnlyElement(combiner.extractOutput());
+ result.add(combined.getValue() + ":" + combined.getTimestamp().getMillis());
+ }
+
+ assertUnorderedEquals(
+ Lists.newArrayList("3:999", "5:1999", "5:2999", "2:3999", "1:5999", "1:6999", "1:7999"),
+ result);
+ }
+
+ private static Combine.CombineFn<Integer, Long, Long> getSumFn() {
+ return new Combine.CombineFn<Integer, Long, Long>() {
+
+ @Override
+ public Long createAccumulator() {
+ return 0L;
+ }
+
+ @Override
+ public Long addInput(Long mutableAccumulator, Integer input) {
+ return mutableAccumulator + input;
+ }
+
+ @Override
+ public Long mergeAccumulators(Iterable<Long> accumulators) {
+ return StreamSupport.stream(accumulators.spliterator(), false).mapToLong(e -> e).sum();
+ }
+
+ @Override
+ public Long extractOutput(Long accumulator) {
+ return accumulator;
+ }
+ };
+ }
+
+ private <K, V> WindowedValue<KV<K, V>> input(K key, V value, Instant timestamp) throws Exception {
+
+ return input(key, value, timestamp, WindowingStrategy.globalDefault().getWindowFn());
+ }
+
+ private <K, V> WindowedValue<KV<K, V>> input(
+ K key, V value, Instant timestamp, WindowFn<?, ?> windowFn) throws Exception {
+
+ return inputValue(KV.of(key, value), timestamp, windowFn);
+ }
+
+ private <V> WindowedValue<V> inputValue(V value, Instant timestamp) throws Exception {
+ return inputValue(value, timestamp, WindowingStrategy.globalDefault().getWindowFn());
+ }
+
+ private <V> WindowedValue<V> inputValue(V value, Instant timestamp, WindowFn<?, ?> windowFn)
+ throws Exception {
+
+ @SuppressWarnings("unchecked")
+ WindowFn<V, BoundedWindow> cast = (WindowFn<V, BoundedWindow>) windowFn;
+ return WindowedValue.of(
+ value,
+ timestamp,
+ cast.assignWindows(assignContext(cast, value, timestamp)),
+ PaneInfo.NO_FIRING);
+ }
+
+ <V> WindowFn<V, BoundedWindow>.AssignContext assignContext(
+ WindowFn<V, BoundedWindow> windowFn, V value, Instant timestamp) {
+ return windowFn.new AssignContext() {
+ @Override
+ public V element() {
+ return value;
+ }
+
+ @Override
+ public Instant timestamp() {
+ return timestamp;
+ }
+
+ @Override
+ public BoundedWindow window() {
+ return GlobalWindow.INSTANCE;
+ }
+ };
+ }
+
+ private <T> void assertUnorderedEquals(List<T> actual, List<T> expected) {
+ assertEquals(
+ actual.stream().collect(Collectors.toSet()), expected.stream().collect(Collectors.toSet()));
+ }
+}
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunctionTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunctionTest.java
index 64657ae..5a59fdd 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunctionTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunctionTest.java
@@ -37,18 +37,17 @@
import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
import org.apache.beam.runners.core.metrics.MetricsContainerStepMap;
import org.apache.beam.runners.fnexecution.control.BundleProgressHandler;
-import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.ExecutableStageContext;
import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
import org.apache.beam.runners.fnexecution.control.RemoteBundle;
import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator;
-import org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.JobBundleFactoryCreator;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Before;
import org.junit.Test;
import org.mockito.Mock;
@@ -57,8 +56,8 @@
/** Unit tests for {@link SparkExecutableStageFunction}. */
public class SparkExecutableStageFunctionTest {
- @Mock private JobBundleFactoryCreator jobBundleFactoryCreator;
- @Mock private JobBundleFactory jobBundleFactory;
+ @Mock private SparkExecutableStageContextFactory contextFactory;
+ @Mock private ExecutableStageContext stageContext;
@Mock private StageBundleFactory stageBundleFactory;
@Mock private RemoteBundle remoteBundle;
@Mock private MetricsContainerStepMapAccumulator metricsAccumulator;
@@ -84,11 +83,11 @@
@Before
public void setUpMocks() throws Exception {
MockitoAnnotations.initMocks(this);
- when(jobBundleFactoryCreator.create()).thenReturn(jobBundleFactory);
- when(jobBundleFactory.forStage(any())).thenReturn(stageBundleFactory);
+ when(contextFactory.get(any())).thenReturn(stageContext);
+ when(stageContext.getStageBundleFactory(any())).thenReturn(stageBundleFactory);
when(stageBundleFactory.getBundle(any(), any(), any())).thenReturn(remoteBundle);
@SuppressWarnings("unchecked")
- ImmutableMap<String, FnDataReceiver<WindowedValue<?>>> inputReceiver =
+ ImmutableMap<String, FnDataReceiver> inputReceiver =
ImmutableMap.of("input", Mockito.mock(FnDataReceiver.class));
when(remoteBundle.getInputReceivers()).thenReturn(inputReceiver);
when(metricsAccumulator.value()).thenReturn(stepMap);
@@ -153,7 +152,7 @@
}
@Override
- public Map<String, FnDataReceiver<WindowedValue<?>>> getInputReceivers() {
+ public Map<String, FnDataReceiver> getInputReceivers() {
return ImmutableMap.of(
"input",
input -> {
@@ -184,7 +183,7 @@
@Override
public void close() {}
};
- when(jobBundleFactory.forStage(any())).thenReturn(stageBundleFactory);
+ when(stageContext.getStageBundleFactory(any())).thenReturn(stageBundleFactory);
SparkExecutableStageFunction<Integer, ?> function = getFunction(outputTagMap);
Iterator<RawUnionValue> iterator = function.call(Collections.emptyIterator());
@@ -210,9 +209,11 @@
Map<String, Integer> outputMap) {
return new SparkExecutableStageFunction<>(
stagePayload,
+ null,
outputMap,
- jobBundleFactoryCreator,
+ contextFactory,
Collections.emptyMap(),
- metricsAccumulator);
+ metricsAccumulator,
+ null);
}
}
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
index 9dc76ca..bc85b75 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
@@ -69,10 +69,10 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.Serializer;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SparkCoGroupByKeyStreamingTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SparkCoGroupByKeyStreamingTest.java
index 49b1a92..3c140c9 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SparkCoGroupByKeyStreamingTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/SparkCoGroupByKeyStreamingTest.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/StreamingSourceMetricsTest.java b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/StreamingSourceMetricsTest.java
index 60f7fa9..d706c79 100644
--- a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/StreamingSourceMetricsTest.java
+++ b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/StreamingSourceMetricsTest.java
@@ -17,8 +17,10 @@
*/
package org.apache.beam.runners.spark.translation.streaming;
-import static org.apache.beam.sdk.metrics.MetricResultsMatchers.attemptedMetricsResult;
+import static org.apache.beam.sdk.metrics.MetricResultsMatchers.metricsResult;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertThat;
import java.io.Serializable;
@@ -32,7 +34,10 @@
import org.apache.beam.sdk.metrics.MetricsFilter;
import org.apache.beam.sdk.metrics.SourceMetrics;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.PCollection;
import org.joda.time.Duration;
+import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -47,11 +52,18 @@
@Test
@Category(StreamingTest.class)
public void testUnboundedSourceMetrics() {
- final long numElements = 1000;
+ final long minElements = 1000;
- pipeline.apply(
- // Use maxReadTime to force unbounded mode.
- GenerateSequence.from(0).to(numElements).withMaxReadTime(Duration.standardDays(1)));
+ // Use a GenerateSequence for the UnboundedSequence, but push the watermark to infinity at
+ // minElements to let the test pipeline cleanly shut it down. Shutdown will occur shortly
+ // afterwards, but at least minElements will be reported in the metrics.
+ PCollection<Long> pc =
+ pipeline.apply(
+ GenerateSequence.from(1)
+ .withRate(minElements / 10, Duration.millis(500L))
+ .withTimestampFn(
+ t -> t < minElements ? Instant.now() : BoundedWindow.TIMESTAMP_MAX_VALUE));
+ assertThat(pc.isBounded(), is(PCollection.IsBounded.UNBOUNDED));
PipelineResult pipelineResult = pipeline.run();
@@ -68,10 +80,11 @@
assertThat(
metrics.getCounters(),
hasItem(
- attemptedMetricsResult(
+ metricsResult(
ELEMENTS_READ.getNamespace(),
ELEMENTS_READ.getName(),
- "Read(UnboundedCountingSource)",
- 1000L)));
+ "GenerateSequence/Read(UnboundedCountingSource)",
+ greaterThanOrEqualTo(minElements),
+ false)));
}
}
diff --git a/sdks/CONTAINERS.md b/sdks/CONTAINERS.md
deleted file mode 100644
index e5461c1..0000000
--- a/sdks/CONTAINERS.md
+++ /dev/null
@@ -1,191 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Docker containers
-
-The Beam [portability effort](https://s.apache.org/beam-fn-api) aims to make it possible
-for any SDK to work with any runner. One aspect of the effort is the isolation of the SDK
-and user code execution environment from the runner execution environment using
-[docker](https://www.docker.com/), as defined in the portability
-[container contract](https://s.apache.org/beam-fn-api-container-contract).
-
-This document describes how to build and push container images to that end. The push
-step generally requires an account with a public docker registry, such
-as [bintray.io](bintray.io) or
-[Google Container Registry](https://cloud.google.com/container-registry). These
-instructions assume familiarity with docker and a bintray account under the
-current username with a docker repository named "apache".
-
-## How to build container images
-
-**Prerequisites**: install [docker](https://www.docker.com/) on your
-platform. You can verify that it works by running `docker images` or any other
-docker command.
-
-Run Gradle with the `docker` target:
-
-```
-$ pwd
-[...]/beam
-$ ./gradlew docker
-[...]
-> Task :sdks:python:container:docker
-a571bb44bc32: Verifying Checksum
-a571bb44bc32: Download complete
-aa6d783919f6: Verifying Checksum
-aa6d783919f6: Download complete
-f2b6b4884fc8: Verifying Checksum
-f2b6b4884fc8: Download complete
-f2b6b4884fc8: Pull complete
-74eaa8be7221: Pull complete
-2d6e98fe4040: Pull complete
-414666f7554d: Pull complete
-bb0bcc8d7f6a: Pull complete
-a571bb44bc32: Pull complete
-aa6d783919f6: Pull complete
-Digest: sha256:d9455be2cc68ded908084ec5b63a5cbb87f12ec0915c2f146751bd50b9aef01a
-Status: Downloaded newer image for python:2
- ---> 2863c80c418c
-Step 2/6 : MAINTAINER "Apache Beam <dev@beam.apache.org>"
- ---> Running in c787617f4af1
-Removing intermediate container c787617f4af1
- ---> b4ffbbf94717
-[...]
- ---> a77003ead1a1
-Step 5/6 : ADD target/linux_amd64/boot /opt/apache/beam/
- ---> 4998013b3d63
-Step 6/6 : ENTRYPOINT ["/opt/apache/beam/boot"]
- ---> Running in 30079dc4204b
-Removing intermediate container 30079dc4204b
- ---> 4ea515403a1a
-Successfully built 4ea515403a1a
-Successfully tagged herohde-docker-apache.bintray.io/beam/python:latest
-[...]
-```
-
-Note that the container images include built content, including the Go boot
-code. Some images, notably python, take a while to build, so building just
-the specific images needed can be a lot faster:
-
-```
-$ ./gradlew -p sdks/java/container docker
-$ ./gradlew -p sdks/python/container docker
-$ ./gradlew -p sdks/go/container docker
-```
-
-**(Optional)** When built, you can see, inspect and run them locally:
-
-```
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED SIZE
-herohde-docker-apache.bintray.io/beam/python latest 4ea515403a1a 3 minutes ago 1.27GB
-herohde-docker-apache.bintray.io/beam/java latest 0103512f1d8f 34 minutes ago 780MB
-herohde-docker-apache.bintray.io/beam/go latest ce055985808a 35 minutes ago 121MB
-[...]
-```
-
-Despite the names, these container images live only on your local machine.
-While we will re-use the same tag "latest" for each build, the
-images IDs will change.
-
-**(Optional)**: the default setting for `docker-repository-root` specifies the above bintray
-location. You can override it by adding:
-
-```
--Pdocker-repository-root=<location>
-```
-
-Similarly, if you want to specify a specific tag instead of "latest", such as a "2.3.0"
-version, you can do so by adding:
-
-```
--Pdocker-tag=<tag>
-```
-
-### Adding dependencies, and making Python go vroom vroom
-
-Not all dependencies are like insurance on used Vespa, if you don't have them some job's just won't run at all and you can't sweet talk your way out of a tensorflow dependency. On the other hand, for Python users dependencies can be automatically installed at run time on each container, which is a great way to find out what your systems timeout limits are. Regardless as to if you have dependency which isn't being installed for you and you need, or you just don't want to install tensorflow 1.6.0 every time you start a new worker this can help.
-
-For Python we have a sample Dockerfile which will take the user specified requirements and install them on top of your base image. If your building from source follow the directions above, otherwise you can set the environment variable BASE_PYTHON_CONTAINER_IMAGE to the desired released version.
-
-```
-USER_REQUIREMENTS=~/my_req.txt ./sdks/python/scripts/add_requirements.sh
-```
-
-Once your custom container is built, remember to upload it to the registry of your choice.
-
-If you build a custom container when you run your job you will need to specify instead of the default latest container, so for example Holden would specify:
-
-```
---worker_harness_container_image=holden-docker-apache.bintray.io/beam/python-with-requirements
-```
-
-## How to push container images
-
-**Preprequisites**: obtain a docker registry account and ensure docker can push images to it,
-usually by doing `docker login` with the appropriate information. The image you want
-to push must also be present in the local docker image repository.
-
-For the Python SDK harness container image, run:
-
-```
-$ docker push $USER-docker-apache.bintray.io/beam/python:latest
-The push refers to repository [herohde-docker-apache.bintray.io/beam/python]
-723b66d57e21: Pushed
-12d5806e6806: Pushed
-b394bd077c6e: Pushed
-ca82a2274c57: Pushed
-de2fbb43bd2a: Pushed
-4e32c2de91a6: Pushed
-6e1b48dc2ccc: Pushed
-ff57bdb79ac8: Pushed
-6e5e20cbf4a7: Pushed
-86985c679800: Pushed
-8fad67424c4e: Pushed
-latest: digest: sha256:86ad57055324457c3ea950f914721c596c7fa261c216efb881d0ca0bb8457535 size: 2646
-```
-
-Similarly for the Java and Go SDK harness container images. If you want to push the same image
-to multiple registries, you can retag the image using `docker tag` and push.
-
-**(Optional)** On any machine, you can now pull the pushed container image:
-
-```
-$ docker pull $USER-docker-apache.bintray.io/beam/python:latest
-latest: Pulling from beam/python
-f2b6b4884fc8: Pull complete
-4fb899b4df21: Pull complete
-74eaa8be7221: Pull complete
-2d6e98fe4040: Pull complete
-414666f7554d: Pull complete
-bb0bcc8d7f6a: Pull complete
-a571bb44bc32: Pull complete
-aa6d783919f6: Pull complete
-7255d71dee8f: Pull complete
-08274803455d: Pull complete
-ef79fab5686a: Pull complete
-Digest: sha256:86ad57055324457c3ea950f914721c596c7fa261c216efb881d0ca0bb8457535
-Status: Downloaded newer image for herohde-docker-apache.bintray.io/beam/python:latest
-$ docker images
-REPOSITORY TAG IMAGE ID CREATED SIZE
-herohde-docker-apache.bintray.io/beam/python latest 4ea515403a1a 35 minutes ago 1.27 GB
-[...]
-```
-
-Note that the image IDs and digests match their local counterparts.
diff --git a/sdks/go/cmd/starcgen/starcgen.go b/sdks/go/cmd/starcgen/starcgen.go
index e3b627b..601ec13 100644
--- a/sdks/go/cmd/starcgen/starcgen.go
+++ b/sdks/go/cmd/starcgen/starcgen.go
@@ -172,7 +172,11 @@
if err != nil {
log.Fatalf("error opening %q: %v", *output, err)
}
- if err := Generate(f, *output, pkg, strings.Split(*ids, ","), fset, fs); err != nil {
+ splitIds := make([]string, 0) // If no ids are specified, we should pass an empty slice.
+ if len(*ids) > 0 {
+ splitIds = strings.Split(*ids, ",")
+ }
+ if err := Generate(f, *output, pkg, splitIds, fset, fs); err != nil {
log.Fatal(err)
}
}
diff --git a/sdks/go/container/build.gradle b/sdks/go/container/build.gradle
index c367d39..44782f9 100644
--- a/sdks/go/container/build.gradle
+++ b/sdks/go/container/build.gradle
@@ -46,7 +46,10 @@
}
docker {
- name containerImageName(name: "go")
+ name containerImageName(
+ name: "go_sdk",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] : "apachebeam")
files "./build/"
}
// Ensure that making the docker image builds any required artifacts
diff --git a/sdks/go/gogradle.lock b/sdks/go/gogradle.lock
index 9ab419c..3713503 100644
--- a/sdks/go/gogradle.lock
+++ b/sdks/go/gogradle.lock
@@ -1,6 +1,6 @@
# This file is generated by gogradle automatically, you should NEVER modify it manually.
---
-apiVersion: "0.8.1"
+apiVersion: "0.11.4"
dependencies:
build:
- vcs: "git"
@@ -362,6 +362,13 @@
name: "github.com/mitchellh/mapstructure"
commit: "a4e142e9c047c904fa2f1e144d9a84e6133024bc"
transitive: false
+ - urls:
+ - "https://github.com/nightlyone/lockfile"
+ - "git@github.com:nightlyone/lockfile.git"
+ vcs: "git"
+ name: "github.com/nightlyone/lockfile"
+ commit: "0ad87eef1443f64d3d8c50da647e2b1552851124"
+ transitive: false
- name: "github.com/olekukonko/tablewriter"
host:
name: "github.com/coreos/etcd"
diff --git a/sdks/go/pkg/beam/combine.go b/sdks/go/pkg/beam/combine.go
index ecee08b..126dbc4 100644
--- a/sdks/go/pkg/beam/combine.go
+++ b/sdks/go/pkg/beam/combine.go
@@ -23,22 +23,24 @@
// Combine inserts a global Combine transform into the pipeline. It
// expects a PCollection<T> as input where T is a concrete type.
-func Combine(s Scope, combinefn interface{}, col PCollection) PCollection {
- return Must(TryCombine(s, combinefn, col))
+// Combine supports TypeDefinition options for binding generic types in combinefn.
+func Combine(s Scope, combinefn interface{}, col PCollection, opts ...Option) PCollection {
+ return Must(TryCombine(s, combinefn, col, opts...))
}
// CombinePerKey inserts a GBK and per-key Combine transform into the pipeline. It
// expects a PCollection<KV<K,T>>. The CombineFn may optionally take a key parameter.
-func CombinePerKey(s Scope, combinefn interface{}, col PCollection) PCollection {
- return Must(TryCombinePerKey(s, combinefn, col))
+// CombinePerKey supports TypeDefinition options for binding generic types in combinefn.
+func CombinePerKey(s Scope, combinefn interface{}, col PCollection, opts ...Option) PCollection {
+ return Must(TryCombinePerKey(s, combinefn, col, opts...))
}
// TryCombine attempts to insert a global Combine transform into the pipeline. It may fail
// for multiple reasons, notably that the combinefn is not valid or cannot be bound
// -- due to type mismatch, say -- to the incoming PCollections.
-func TryCombine(s Scope, combinefn interface{}, col PCollection) (PCollection, error) {
+func TryCombine(s Scope, combinefn interface{}, col PCollection, opts ...Option) (PCollection, error) {
pre := AddFixedKey(s, col)
- post, err := TryCombinePerKey(s, combinefn, pre)
+ post, err := TryCombinePerKey(s, combinefn, pre, opts...)
if err != nil {
return PCollection{}, err
}
@@ -52,10 +54,18 @@
// TryCombinePerKey attempts to insert a per-key Combine transform into the pipeline. It may fail
// for multiple reasons, notably that the combinefn is not valid or cannot be bound
// -- due to type mismatch, say -- to the incoming PCollection.
-func TryCombinePerKey(s Scope, combinefn interface{}, col PCollection) (PCollection, error) {
+func TryCombinePerKey(s Scope, combinefn interface{}, col PCollection, opts ...Option) (PCollection, error) {
s = s.Scope(graph.CombinePerKeyScope)
ValidateKVType(col)
- col, err := TryGroupByKey(s, col)
+ side, typedefs, err := validate(s, col, opts)
+ if err != nil {
+ return PCollection{}, addCombinePerKeyCtx(err, s)
+ }
+ if len(side) > 0 {
+ return PCollection{}, addCombinePerKeyCtx(errors.New("combine does not support side inputs"), s)
+ }
+
+ col, err = TryGroupByKey(s, col)
if err != nil {
return PCollection{}, addCombinePerKeyCtx(err, s)
}
@@ -74,7 +84,7 @@
return PCollection{}, addCombinePerKeyCtx(wrapped, s)
}
- edge, err := graph.NewCombine(s.real, s.scope, fn, col.n, accumCoder)
+ edge, err := graph.NewCombine(s.real, s.scope, fn, col.n, accumCoder, typedefs)
if err != nil {
return PCollection{}, addCombinePerKeyCtx(err, s)
}
diff --git a/sdks/go/pkg/beam/combine_test.go b/sdks/go/pkg/beam/combine_test.go
new file mode 100644
index 0000000..14b0127
--- /dev/null
+++ b/sdks/go/pkg/beam/combine_test.go
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package beam_test
+
+import (
+ "reflect"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+// foolFn is a no-op CombineFn.
+type foolFn struct {
+ OutputType beam.EncodedType
+}
+
+type foolAccum struct{}
+
+func (f *foolFn) CreateAccumulator() *foolAccum {
+ return &foolAccum{}
+}
+
+func (f *foolFn) AddInput(a *foolAccum, v beam.U) *foolAccum {
+ return a
+}
+
+func (f *foolFn) MergeAccumulators(a *foolAccum, b *foolAccum) *foolAccum {
+ return a
+}
+
+func (f *foolFn) ExtractOutput(a *foolAccum) beam.V {
+ return reflect.New(f.OutputType.T).Elem().Interface()
+}
+
+func TestCombineWithTypeDefinition(t *testing.T) {
+ _, s := beam.NewPipelineWithRoot()
+ in := beam.Create(s, 1, 2, 3)
+ strType := reflect.TypeOf("")
+ combineFn := &foolFn{OutputType: beam.EncodedType{T: strType}}
+ output := beam.Combine(s, combineFn, in, beam.TypeDefinition{Var: beam.VType, T: strType})
+ if output.Type().Type() != strType {
+ t.Errorf("expect combine output type to be %v, got %v", strType, output.Type().Type())
+ }
+}
diff --git a/sdks/go/pkg/beam/core/funcx/fn.go b/sdks/go/pkg/beam/core/funcx/fn.go
index c924782..bba6c56 100644
--- a/sdks/go/pkg/beam/core/funcx/fn.go
+++ b/sdks/go/pkg/beam/core/funcx/fn.go
@@ -147,6 +147,55 @@
return -1, false
}
+// Emits returns (index, num, true) iff the function expects one or more
+// emitters. The index returned is the index of the first emitter param in the
+// signature. The num return value is the number of emitters in the signature.
+// When there are multiple emitters in the signature, they will all be located
+// contiguously, so the range of emitter params is [index, index+num).
+func (u *Fn) Emits() (pos int, num int, exists bool) {
+ pos = -1
+ exists = false
+ for i, p := range u.Param {
+ if !exists && p.Kind == FnEmit {
+ // This should execute when hitting the first emitter.
+ pos = i
+ num = 1
+ exists = true
+ } else if exists && p.Kind == FnEmit {
+ // Subsequent emitters after the first.
+ num++
+ } else if exists {
+ // Breaks out when no emitters are left.
+ break
+ }
+ }
+ return pos, num, exists
+}
+
+// Inputs returns (index, num, true) iff the function expects one or more
+// inputs, consisting of the main input followed by any number of side inputs.
+// The index returned is the index of the first input, which is always the main
+// input. The num return value is the number of total inputs in the signature.
+// The main input and all side inputs are located contiguously
+func (u *Fn) Inputs() (pos int, num int, exists bool) {
+ pos = -1
+ exists = false
+ for i, p := range u.Param {
+ if !exists && (p.Kind == FnValue || p.Kind == FnIter || p.Kind == FnReIter) {
+ // This executes on hitting the first input.
+ pos = i
+ num = 1
+ exists = true
+ } else if exists && (p.Kind == FnValue || p.Kind == FnIter || p.Kind == FnReIter) {
+ // Subsequent inputs after the first.
+ num++
+ } else if exists {
+ break
+ }
+ }
+ return pos, num, exists
+}
+
// Type returns (index, true) iff the function expects a reflect.FullType.
func (u *Fn) Type() (pos int, exists bool) {
for i, p := range u.Param {
@@ -304,7 +353,7 @@
}
// The order of present parameters and return values must be as follows:
-// func(FnContext?, FnWindow?, FnEventTime?, FnType?, (FnValue, SideInput*)?, FnEmit*) (RetEventTime?, RetEventTime?, RetError?)
+// func(FnContext?, FnWindow?, FnEventTime?, FnType?, (FnValue, SideInput*)?, FnEmit*) (RetEventTime?, RetOutput?, RetError?)
// where ? indicates 0 or 1, and * indicates any number.
// and a SideInput is one of FnValue or FnIter or FnReIter
// Note: Fns with inputs must have at least one FnValue as the main input.
@@ -332,7 +381,6 @@
errWindowParamPrecedence = errors.New("may only have a single Window parameter and it must precede the EventTime and main input parameter")
errEventTimeParamPrecedence = errors.New("may only have a single beam.EventTime parameter and it must precede the main input parameter")
errReflectTypePrecedence = errors.New("may only have a single reflect.Type parameter and it must precede the main input parameter")
- errSideInputPrecedence = errors.New("side input parameters must follow main input parameter")
errInputPrecedence = errors.New("inputs parameters must precede emit function parameters")
)
@@ -405,10 +453,8 @@
return -1, errEventTimeParamPrecedence
case FnType:
return -1, errReflectTypePrecedence
- case FnValue:
+ case FnIter, FnReIter, FnValue:
return psInput, nil
- case FnIter, FnReIter:
- return -1, errSideInputPrecedence
case FnEmit:
return psOutput, nil
default:
diff --git a/sdks/go/pkg/beam/core/funcx/fn_test.go b/sdks/go/pkg/beam/core/funcx/fn_test.go
index 209a8a7..410fb59 100644
--- a/sdks/go/pkg/beam/core/funcx/fn_test.go
+++ b/sdks/go/pkg/beam/core/funcx/fn_test.go
@@ -148,16 +148,6 @@
Err: errReflectTypePrecedence,
},
{
- Name: "errSideInputPrecedence- Iter before main input",
- Fn: func(func(*int) bool, func(*int, *string) bool, int) {},
- Err: errSideInputPrecedence,
- },
- {
- Name: "errSideInputPrecedence- ReIter before main input",
- Fn: func(func() func(*int) bool, int) {},
- Err: errSideInputPrecedence,
- },
- {
Name: "errInputPrecedence- Iter before after output",
Fn: func(int, func(int), func(*int) bool, func(*int, *string) bool) {},
Err: errInputPrecedence,
@@ -224,6 +214,157 @@
}
}
+func TestEmits(t *testing.T) {
+ tests := []struct {
+ Name string
+ Params []FnParamKind
+ Pos int
+ Num int
+ Exists bool
+ }{
+ {
+ Name: "no params",
+ Params: []FnParamKind{},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "no emits",
+ Params: []FnParamKind{FnContext, FnEventTime, FnType, FnValue},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "single emit",
+ Params: []FnParamKind{FnValue, FnEmit},
+ Pos: 1,
+ Num: 1,
+ Exists: true,
+ },
+ {
+ Name: "multiple emits",
+ Params: []FnParamKind{FnValue, FnEmit, FnEmit, FnEmit},
+ Pos: 1,
+ Num: 3,
+ Exists: true,
+ },
+ {
+ Name: "multiple emits 2",
+ Params: []FnParamKind{FnValue, FnEmit, FnEmit, FnEmit, FnValue},
+ Pos: 1,
+ Num: 3,
+ Exists: true,
+ },
+ }
+
+ for _, test := range tests {
+ test := test
+ t.Run(test.Name, func(t *testing.T) {
+ // Create a Fn with a filled params list.
+ params := make([]FnParam, len(test.Params))
+ for i, kind := range test.Params {
+ params[i].Kind = kind
+ params[i].T = nil
+ }
+ fn := new(Fn)
+ fn.Param = params
+
+ // Validate we get expected results for Emits function.
+ pos, num, exists := fn.Emits()
+ if exists != test.Exists {
+ t.Errorf("Emits() exists: got %v, want %v", exists, test.Exists)
+ }
+ if num != test.Num {
+ t.Errorf("Emits() num: got %v, want %v", num, test.Num)
+ }
+ if pos != test.Pos {
+ t.Errorf("Emits() pos: got %v, want %v", pos, test.Pos)
+ }
+ })
+ }
+}
+
+func TestInputs(t *testing.T) {
+ tests := []struct {
+ Name string
+ Params []FnParamKind
+ Pos int
+ Num int
+ Exists bool
+ }{
+ {
+ Name: "no params",
+ Params: []FnParamKind{},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "no inputs",
+ Params: []FnParamKind{FnContext, FnEventTime, FnType, FnEmit},
+ Pos: -1,
+ Num: 0,
+ Exists: false,
+ },
+ {
+ Name: "no main input",
+ Params: []FnParamKind{FnContext, FnIter, FnReIter, FnEmit},
+ Pos: 1,
+ Num: 2,
+ Exists: true,
+ },
+ {
+ Name: "single input",
+ Params: []FnParamKind{FnContext, FnValue},
+ Pos: 1,
+ Num: 1,
+ Exists: true,
+ },
+ {
+ Name: "multiple inputs",
+ Params: []FnParamKind{FnContext, FnValue, FnIter, FnReIter},
+ Pos: 1,
+ Num: 3,
+ Exists: true,
+ },
+ {
+ Name: "multiple inputs 2",
+ Params: []FnParamKind{FnContext, FnValue, FnIter, FnValue, FnReIter, FnEmit},
+ Pos: 1,
+ Num: 4,
+ Exists: true,
+ },
+ }
+
+ for _, test := range tests {
+ test := test
+ t.Run(test.Name, func(t *testing.T) {
+ // Create a Fn with a filled params list.
+ params := make([]FnParam, len(test.Params))
+ for i, kind := range test.Params {
+ params[i].Kind = kind
+ params[i].T = nil
+ }
+ fn := new(Fn)
+ fn.Param = params
+
+ // Validate we get expected results for Inputs function.
+ pos, num, exists := fn.Inputs()
+ if exists != test.Exists {
+ t.Errorf("Inputs(%v) - exists: got %v, want %v", params, exists, test.Exists)
+ }
+ if num != test.Num {
+ t.Errorf("Inputs(%v) - num: got %v, want %v", params, num, test.Num)
+ }
+ if pos != test.Pos {
+ t.Errorf("Inputs(%v) - pos: got %v, want %v", params, pos, test.Pos)
+ }
+ })
+ }
+}
+
func projectParamKind(u *Fn) []FnParamKind {
var ret []FnParamKind
for _, p := range u.Param {
diff --git a/sdks/go/pkg/beam/core/graph/bind.go b/sdks/go/pkg/beam/core/graph/bind.go
index 77a6d69..f5cc7f8 100644
--- a/sdks/go/pkg/beam/core/graph/bind.go
+++ b/sdks/go/pkg/beam/core/graph/bind.go
@@ -60,30 +60,34 @@
//
// Here, the inbound shape and output types are different from before.
func Bind(fn *funcx.Fn, typedefs map[string]reflect.Type, in ...typex.FullType) ([]typex.FullType, []InputKind, []typex.FullType, []typex.FullType, error) {
+ addContext := func(err error, fn *funcx.Fn) error {
+ return errors.WithContextf(err, "binding fn %v", fn.Fn.Name())
+ }
+
inbound, kinds, err := findInbound(fn, in...)
if err != nil {
- return nil, nil, nil, nil, errors.WithContextf(err, "binding fn %v", fn.Fn.Name())
+ return nil, nil, nil, nil, addContext(err, fn)
}
outbound, err := findOutbound(fn)
if err != nil {
- return nil, nil, nil, nil, errors.WithContextf(err, "binding fn %v", fn.Fn.Name())
+ return nil, nil, nil, nil, addContext(err, fn)
}
subst, err := typex.Bind(inbound, in)
if err != nil {
- return nil, nil, nil, nil, errors.WithContextf(err, "binding fn %v", fn.Fn.Name())
+ return nil, nil, nil, nil, addContext(err, fn)
}
for k, v := range typedefs {
if substK, exists := subst[k]; exists {
err := errors.Errorf("cannot substitute type %v with %v, already defined as %v", k, v, substK)
- return nil, nil, nil, nil, errors.WithContextf(err, "binding fn %v", fn.Fn.Name())
+ return nil, nil, nil, nil, addContext(err, fn)
}
subst[k] = v
}
out, err := typex.Substitute(outbound, subst)
if err != nil {
- return nil, nil, nil, nil, errors.WithContextf(err, "binding fn %v", fn.Fn.Name())
+ return nil, nil, nil, nil, addContext(err, fn)
}
return inbound, kinds, outbound, out, nil
}
@@ -128,6 +132,9 @@
func findInbound(fn *funcx.Fn, in ...typex.FullType) ([]typex.FullType, []InputKind, error) {
// log.Printf("Bind inbound: %v %v", fn, in)
+ addContext := func(err error, p []funcx.FnParam, in interface{}) error {
+ return errors.WithContextf(err, "binding params %v to input %v", p, in)
+ }
var inbound []typex.FullType
var kinds []InputKind
@@ -136,29 +143,26 @@
for _, input := range in {
arity, err := inboundArity(input, index == 0)
if err != nil {
- return nil, nil, errors.WithContextf(err, "binding params %v to input %v", params, input)
+ return nil, nil, addContext(err, params, input)
}
if len(params)-index < arity {
- err := errors.New("too few params")
- return nil, nil, errors.WithContextf(err, "binding params %v to input %v", params[index:], input)
+ return nil, nil, addContext(errors.New("too few params"), params[index:], input)
}
paramsToBind := params[index : index+arity]
elm, kind, err := tryBindInbound(input, paramsToBind, index == 0)
if err != nil {
- return nil, nil, errors.WithContextf(err, "binding params %v to input %v", paramsToBind, input)
+ return nil, nil, addContext(err, paramsToBind, input)
}
inbound = append(inbound, elm)
kinds = append(kinds, kind)
index += arity
}
if index < len(params) {
- err := errors.New("too few inputs: forgot an input or to annotate options?")
- return nil, nil, errors.WithContextf(err, "binding params %v to inputs %v:", params, in)
+ return nil, nil, addContext(errors.New("too few inputs: forgot an input or to annotate options?"), params, in)
}
if index > len(params) {
- err := errors.New("too many inputs")
- return nil, nil, errors.WithContextf(err, "binding params %v to inputs %v:", params, in)
+ return nil, nil, addContext(errors.New("too many inputs"), params, in)
}
return inbound, kinds, nil
}
diff --git a/sdks/go/pkg/beam/core/graph/edge.go b/sdks/go/pkg/beam/core/graph/edge.go
index 846dceb..a37ecb7 100644
--- a/sdks/go/pkg/beam/core/graph/edge.go
+++ b/sdks/go/pkg/beam/core/graph/edge.go
@@ -186,14 +186,15 @@
// NewCoGBK inserts a new CoGBK edge into the graph.
func NewCoGBK(g *Graph, s *Scope, ns []*Node) (*MultiEdge, error) {
+ addContext := func(err error, s *Scope) error {
+ return errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ }
+
if len(ns) == 0 {
- // TODO(BEAM-7086) Reduce the repetition in the context of all the errors in this file.
- err := errors.New("needs at least 1 input")
- return nil, errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ return nil, addContext(errors.New("needs at least 1 input"), s)
}
if !typex.IsKV(ns[0].Type()) {
- err := errors.Errorf("input type must be KV: %v", ns[0])
- return nil, errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ return nil, addContext(errors.Errorf("input type must be KV: %v", ns[0]), s)
}
// (1) Create CoGBK result type: KV<T,U>, .., KV<T,Z> -> CoGBK<T,U,..,Z>.
@@ -206,20 +207,16 @@
for i := 1; i < len(ns); i++ {
n := ns[i]
if !typex.IsKV(n.Type()) {
- err := errors.Errorf("input type must be KV: %v", n)
- return nil, errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ return nil, addContext(errors.Errorf("input type must be KV: %v", n), s)
}
if !n.Coder.Components[0].Equals(c) {
- err := errors.Errorf("key coder for %v is %v, want %v", n, n.Coder.Components[0], c)
- return nil, errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ return nil, addContext(errors.Errorf("key coder for %v is %v, want %v", n, n.Coder.Components[0], c), s)
}
if !w.Equals(n.WindowingStrategy()) {
- err := errors.Errorf("mismatched CoGBK windowing strategies: %v, want %v", n.WindowingStrategy(), w)
- return nil, errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ return nil, addContext(errors.Errorf("mismatched CoGBK windowing strategies: %v, want %v", n.WindowingStrategy(), w), s)
}
if bounded != n.Bounded() {
- err := errors.Errorf("unmatched CoGBK boundedness: %v, want %v", n.Bounded(), bounded)
- return nil, errors.WithContextf(err, "creating new CoGBK in scope %v", s)
+ return nil, addContext(errors.Errorf("unmatched CoGBK boundedness: %v, want %v", n.Bounded(), bounded), s)
}
comp = append(comp, n.Type().Components()[1])
@@ -242,9 +239,12 @@
// NewFlatten inserts a new Flatten edge in the graph. Flatten output type is
// the shared input type.
func NewFlatten(g *Graph, s *Scope, in []*Node) (*MultiEdge, error) {
+ addContext := func(err error, s *Scope) error {
+ return errors.WithContextf(err, "creating new Flatten in scope %v", s)
+ }
+
if len(in) < 2 {
- err := errors.Errorf("Flatten needs at least 2 input, got %v", len(in))
- return nil, errors.WithContextf(err, "creating new Flatten in scope %v", s)
+ return nil, addContext(errors.Errorf("Flatten needs at least 2 input, got %v", len(in)), s)
}
t := in[0].Type()
w := inputWindow(in)
@@ -260,17 +260,14 @@
}
for _, n := range in {
if !typex.IsEqual(t, n.Type()) {
- err := errors.Errorf("mismatched Flatten input types: %v, want %v", n.Type(), t)
- return nil, errors.WithContextf(err, "creating new Flatten in scope %v", s)
+ return nil, addContext(errors.Errorf("mismatched Flatten input types: %v, want %v", n.Type(), t), s)
}
if !w.Equals(n.WindowingStrategy()) {
- err := errors.Errorf("mismatched Flatten window types: %v, want %v", n.WindowingStrategy(), w)
- return nil, errors.WithContextf(err, "creating new Flatten in scope %v", s)
+ return nil, addContext(errors.Errorf("mismatched Flatten window types: %v, want %v", n.WindowingStrategy(), w), s)
}
}
if typex.IsCoGBK(t) {
- err := errors.Errorf("Flatten input type cannot be CoGBK: %v", t)
- return nil, errors.WithContextf(err, "creating new Flatten in scope %v", s)
+ return nil, addContext(errors.Errorf("Flatten input type cannot be CoGBK: %v", t), s)
}
edge := g.NewEdge(s)
@@ -337,15 +334,17 @@
// NewCombine inserts a new Combine edge into the graph. Combines cannot have side
// input.
-func NewCombine(g *Graph, s *Scope, u *CombineFn, in *Node, ac *coder.Coder) (*MultiEdge, error) {
+func NewCombine(g *Graph, s *Scope, u *CombineFn, in *Node, ac *coder.Coder, typedefs map[string]reflect.Type) (*MultiEdge, error) {
+ addContext := func(err error, s *Scope) error {
+ return errors.WithContextf(err, "creating new Combine in scope %v", s)
+ }
+
inT := in.Type()
if !typex.IsCoGBK(inT) {
- err := errors.Errorf("Combine requires CoGBK type: %v", inT)
- return nil, errors.WithContextf(err, "creating new Combine in scope %v", s)
+ return nil, addContext(errors.Errorf("Combine requires CoGBK type: %v", inT), s)
}
if len(inT.Components()) > 2 {
- err := errors.Errorf("Combine cannot follow multi-input CoGBK: %v", inT)
- return nil, errors.WithContextf(err, "creating new Combine in scope %v", s)
+ return nil, addContext(errors.Errorf("Combine cannot follow multi-input CoGBK: %v", inT), s)
}
// Create a synthetic function for binding purposes. It takes main input
@@ -392,9 +391,9 @@
key := in.Type().Components()[0]
synth.Ret = append([]funcx.ReturnParam{{Kind: funcx.RetValue, T: key.Type()}}, synth.Ret...)
- inbound, kinds, outbound, out, err := Bind(synth, nil, inT)
+ inbound, kinds, outbound, out, err := Bind(synth, typedefs, inT)
if err != nil {
- return nil, errors.WithContextf(err, "creating new Combine in scope %v", s)
+ return nil, addContext(err, s)
}
edge := g.NewEdge(s)
diff --git a/sdks/go/pkg/beam/core/graph/fn.go b/sdks/go/pkg/beam/core/graph/fn.go
index d174c4f..cada9d0 100644
--- a/sdks/go/pkg/beam/core/graph/fn.go
+++ b/sdks/go/pkg/beam/core/graph/fn.go
@@ -209,6 +209,10 @@
// AsDoFn converts a Fn to a DoFn, if possible.
func AsDoFn(fn *Fn) (*DoFn, error) {
+ addContext := func(err error, fn *Fn) error {
+ return errors.WithContextf(err, "graph.AsDoFn: for Fn named %v", fn.Name())
+ }
+
if fn.methods == nil {
fn.methods = make(map[string]*funcx.Fn)
}
@@ -220,14 +224,190 @@
}
if _, ok := fn.methods[processElementName]; !ok {
- return nil, errors.Errorf("graph.AsDoFn: failed to find %v method: %v", processElementName, fn)
+ err := errors.Errorf("failed to find %v method", processElementName)
+ return nil, addContext(err, fn)
}
- // TODO(herohde) 5/18/2017: validate the signatures, incl. consistency.
+ // Start validating DoFn. First, check that ProcessElement has a main input.
+ processFn := fn.methods[processElementName]
+ pos, num, ok := processFn.Inputs()
+ if ok {
+ first := processFn.Param[pos].Kind
+ if first != funcx.FnValue {
+ err := errors.New("side input parameters must follow main input parameter")
+ err = errors.SetTopLevelMsgf(err,
+ "Method %v of DoFns should always have a main input before side inputs, "+
+ "but it has side inputs (as Iters or ReIters) first in DoFn %v.",
+ processElementName, fn.Name())
+ err = errors.WithContextf(err, "method %v", processElementName)
+ return nil, addContext(err, fn)
+ }
+ }
+
+ // If the ProcessElement function includes side inputs or emit functions those must also be
+ // present in the signatures of startBundle and finishBundle.
+ if ok && num > 1 {
+ if startFn, ok := fn.methods[startBundleName]; ok {
+ processFnInputs := processFn.Param[pos : pos+num]
+ if err := validateMethodInputs(processFnInputs, startFn, startBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ if finishFn, ok := fn.methods[finishBundleName]; ok {
+ processFnInputs := processFn.Param[pos : pos+num]
+ if err := validateMethodInputs(processFnInputs, finishFn, finishBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ }
+
+ pos, num, ok = processFn.Emits()
+ if ok {
+ if startFn, ok := fn.methods[startBundleName]; ok {
+ processFnEmits := processFn.Param[pos : pos+num]
+ if err := validateMethodEmits(processFnEmits, startFn, startBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ if finishFn, ok := fn.methods[finishBundleName]; ok {
+ processFnEmits := processFn.Param[pos : pos+num]
+ if err := validateMethodEmits(processFnEmits, finishFn, finishBundleName); err != nil {
+ return nil, addContext(err, fn)
+ }
+ }
+ }
+
+ // Check that Setup and Teardown have no parameters other than Context.
+ for _, name := range []string{setupName, teardownName} {
+ if method, ok := fn.methods[name]; ok {
+ params := method.Param
+ if len(params) > 1 || (len(params) == 1 && params[0].Kind != funcx.FnContext) {
+ err := errors.Errorf(
+ "method %v has invalid parameters, "+
+ "only allowed an optional context.Context", name)
+ err = errors.SetTopLevelMsgf(err,
+ "Method %v of DoFns should have no parameters other than "+
+ "an optional context.Context, but invalid parameters are "+
+ "present in DoFn %v.",
+ name, fn.Name())
+ return nil, addContext(err, fn)
+ }
+ }
+ }
+
+ // Check that none of the methods (except ProcessElement) have any return
+ // values other than error.
+ for _, name := range []string{setupName, startBundleName, finishBundleName, teardownName} {
+ if method, ok := fn.methods[name]; ok {
+ returns := method.Ret
+ if len(returns) > 1 || (len(returns) == 1 && returns[0].Kind != funcx.RetError) {
+ err := errors.Errorf(
+ "method %v has invalid return values, "+
+ "only allowed an optional error", name)
+ err = errors.SetTopLevelMsgf(err,
+ "Method %v of DoFns should have no return values other "+
+ "than an optional error, but invalid return values are present "+
+ "in DoFn %v.",
+ name, fn.Name())
+ return nil, addContext(err, fn)
+ }
+ }
+ }
return (*DoFn)(fn), nil
}
+// validateMethodEmits compares the emits found in a DoFn method signature with the emits found in
+// the signature for ProcessElement, and performs validation that those match. This function
+// should only be used to validate methods that are expected to have the same emit parameters as
+// ProcessElement.
+func validateMethodEmits(processFnEmits []funcx.FnParam, method *funcx.Fn, methodName string) error {
+ methodPos, methodNum, ok := method.Emits()
+ if !ok {
+ err := errors.Errorf("emit parameters expected in method %v", methodName)
+ return errors.SetTopLevelMsgf(err,
+ "Missing emit parameters in the %v method of a DoFn. "+
+ "If emit parameters are present in %v those parameters must also be present in %v.",
+ methodName, processElementName, methodName)
+ }
+
+ processFnNum := len(processFnEmits)
+ if methodNum != processFnNum {
+ err := errors.Errorf("number of emits in method %v does not match method %v: got %d, expected %d",
+ methodName, processElementName, methodNum, processFnNum)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect number of emit parameters in the %v method of a DoFn. "+
+ "The emit parameters should match those of the %v method.",
+ methodName, processElementName)
+ }
+
+ methodEmits := method.Param[methodPos : methodPos+methodNum]
+ for i := 0; i < processFnNum; i++ {
+ if processFnEmits[i].T != methodEmits[i].T {
+ var err error = &funcx.TypeMismatchError{Got: methodEmits[i].T, Want: processFnEmits[i].T}
+ err = errors.Wrapf(err, "emit parameter in method %v does not match emit parameter in %v",
+ methodName, processElementName)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect emit parameters in the %v method of a DoFn. "+
+ "The emit parameters should match those of the %v method.",
+ methodName, processElementName)
+ }
+ }
+
+ return nil
+}
+
+// validateMethodInputs compares the inputs found in a DoFn method signature with the inputs found
+// in the signature for ProcessElement, and performs validation to check that the side inputs
+// match. This function should only be used to validate methods that are expected to have matching
+// side inputs to ProcessElement.
+func validateMethodInputs(processFnInputs []funcx.FnParam, method *funcx.Fn, methodName string) error {
+ methodPos, methodNum, ok := method.Inputs()
+
+ // Note: The second input to ProcessElements is not guaranteed to be a side input (it could be
+ // the Value part of a KV main input). Since we can't know whether to interpret it as a main or
+ // side input, some of these checks have to work around it in specific ways.
+ if !ok {
+ if len(processFnInputs) <= 2 {
+ return nil // This case is fine, since both ProcessElement inputs may be main inputs.
+ }
+ err := errors.Errorf("side inputs expected in method %v", methodName)
+ return errors.SetTopLevelMsgf(err,
+ "Missing side inputs in the %v method of a DoFn. "+
+ "If side inputs are present in %v those side inputs must also be present in %v.",
+ methodName, processElementName, methodName)
+ }
+
+ processFnNum := len(processFnInputs)
+ // The number of side inputs is the number of inputs minus 1 or 2 depending on whether the second
+ // input is a main or side input, so that's what we expect in the method's parameters.
+ // Ex. if ProcessElement has 7 inputs, method must have either 5 or 6 inputs.
+ if (methodNum != processFnNum-1) && (methodNum != processFnNum-2) {
+ err := errors.Errorf("number of side inputs in method %v does not match method %v: got %d, expected either %d or %d",
+ methodName, processElementName, methodNum, processFnNum-1, processFnNum-2)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect number of side inputs in the %v method of a DoFn. "+
+ "The side inputs should match those of the %v method.",
+ methodName, processElementName)
+ }
+
+ methodInputs := method.Param[methodPos : methodPos+methodNum]
+ offset := processFnNum - methodNum // We need an offset to skip the main inputs in ProcessFnInputs
+ for i := 0; i < methodNum; i++ {
+ if processFnInputs[i+offset].T != methodInputs[i].T {
+ var err error = &funcx.TypeMismatchError{Got: methodInputs[i].T, Want: processFnInputs[i+offset].T}
+ err = errors.Wrapf(err, "side input in method %v does not match side input in %v",
+ methodName, processElementName)
+ return errors.SetTopLevelMsgf(err,
+ "Incorrect side inputs in the %v method of a DoFn. "+
+ "The side inputs should match those of the %v method.",
+ methodName, processElementName)
+ }
+ }
+
+ return nil
+}
+
// CombineFn represents a CombineFn.
type CombineFn Fn
diff --git a/sdks/go/pkg/beam/core/graph/fn_test.go b/sdks/go/pkg/beam/core/graph/fn_test.go
index d6b8a67..6cdf9eb 100644
--- a/sdks/go/pkg/beam/core/graph/fn_test.go
+++ b/sdks/go/pkg/beam/core/graph/fn_test.go
@@ -19,8 +19,73 @@
"context"
"reflect"
"testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
)
+func TestNewDoFn(t *testing.T) {
+ t.Run("valid", func(t *testing.T) {
+ tests := []struct {
+ dfn interface{}
+ }{
+ {dfn: func() int { return 0 }},
+ {dfn: func(string, int) int { return 0 }},
+ {dfn: func(context.Context, typex.Window, typex.EventTime, reflect.Type, string, int, func(*int) bool, func() func(*int) bool, func(int)) (typex.EventTime, int, error) {
+ return 0, 0, nil
+ }},
+ {dfn: &GoodDoFn{}},
+ {dfn: &GoodDoFnOmittedMethods{}},
+ {dfn: &GoodDoFnEmits{}},
+ {dfn: &GoodDoFnSideInputs{}},
+ {dfn: &GoodDoFnKvSideInputs{}},
+ {dfn: &GoodDoFnKvNoSideInputs{}},
+ {dfn: &GoodDoFnAllExtras{}},
+ {dfn: &GoodDoFnUnexportedExtraMethod{}},
+ }
+
+ for _, test := range tests {
+ t.Run(reflect.TypeOf(test.dfn).String(), func(t *testing.T) {
+ if _, err := NewDoFn(test.dfn); err != nil {
+ t.Fatalf("NewDoFn failed: %v", err)
+ }
+ })
+ }
+ })
+ t.Run("invalid", func(t *testing.T) {
+ tests := []struct {
+ dfn interface{}
+ }{
+ // Validate emit parameters.
+ {dfn: &BadDoFnNoEmitsStartBundle{}},
+ {dfn: &BadDoFnMissingEmitsStartBundle{}},
+ {dfn: &BadDoFnMismatchedEmitsStartBundle{}},
+ {dfn: &BadDoFnNoEmitsFinishBundle{}},
+ // Validate side inputs.
+ {dfn: &BadDoFnNoSideInputsStartBundle{}},
+ {dfn: &BadDoFnMissingSideInputsStartBundle{}},
+ {dfn: &BadDoFnMismatchedSideInputsStartBundle{}},
+ {dfn: &BadDoFnNoSideInputsFinishBundle{}},
+ // Validate setup/teardown.
+ {dfn: &BadDoFnParamsInSetup{}},
+ {dfn: &BadDoFnParamsInTeardown{}},
+ // Validate return values.
+ {dfn: &BadDoFnReturnValuesInStartBundle{}},
+ {dfn: &BadDoFnReturnValuesInFinishBundle{}},
+ {dfn: &BadDoFnReturnValuesInSetup{}},
+ {dfn: &BadDoFnReturnValuesInTeardown{}},
+ }
+ for _, test := range tests {
+ t.Run(reflect.TypeOf(test.dfn).String(), func(t *testing.T) {
+ if cfn, err := NewDoFn(test.dfn); err != nil {
+ t.Logf("NewDoFn failed as expected:\n%v", err)
+ } else {
+ t.Errorf("NewDoFn(%v) = %v, want failure", cfn.Name(), cfn)
+ }
+ })
+ }
+ })
+}
+
func TestNewCombineFn(t *testing.T) {
t.Run("valid", func(t *testing.T) {
tests := []struct {
@@ -96,10 +161,236 @@
// Do not copy. The following types are for testing signatures only.
// They are not working examples.
// Keep all test functions Above this point.
-type MyAccum struct{}
+
+// Examples of correct DoFn signatures
+
+type GoodDoFn struct{}
+
+func (fn *GoodDoFn) ProcessElement(int) int {
+ return 0
+}
+
+func (fn *GoodDoFn) StartBundle() {
+}
+
+func (fn *GoodDoFn) FinishBundle() {
+}
+
+func (fn *GoodDoFn) Setup() {
+}
+
+func (fn *GoodDoFn) Teardown() {
+}
+
+type GoodDoFnOmittedMethods struct{}
+
+func (fn *GoodDoFnOmittedMethods) ProcessElement(int) int {
+ return 0
+}
+
+type GoodDoFnEmits struct{}
+
+func (fn *GoodDoFnEmits) ProcessElement(int, func(int), func(string)) int {
+ return 0
+}
+
+func (fn *GoodDoFnEmits) StartBundle(func(int), func(string)) {
+}
+
+func (fn *GoodDoFnEmits) FinishBundle(func(int), func(string)) {
+}
+
+type GoodDoFnSideInputs struct{}
+
+func (fn *GoodDoFnSideInputs) ProcessElement(int, func(*int) bool, string, func() func(*int) bool) int {
+ return 0
+}
+
+func (fn *GoodDoFnSideInputs) StartBundle(func(*int) bool, string, func() func(*int) bool) {
+}
+
+func (fn *GoodDoFnSideInputs) FinishBundle(func(*int) bool, string, func() func(*int) bool) {
+}
+
+type GoodDoFnKvSideInputs struct{}
+
+func (fn *GoodDoFnKvSideInputs) ProcessElement(int, int, string, func(*int) bool, func() func(*int) bool) int {
+ return 0
+}
+
+func (fn *GoodDoFnKvSideInputs) StartBundle(string, func(*int) bool, func() func(*int) bool) {
+}
+
+func (fn *GoodDoFnKvSideInputs) FinishBundle(string, func(*int) bool, func() func(*int) bool) {
+}
+
+type GoodDoFnKvNoSideInputs struct{}
+
+func (fn *GoodDoFnKvNoSideInputs) ProcessElement(int, int) int {
+ return 0
+}
+
+func (fn *GoodDoFnKvNoSideInputs) StartBundle() {
+}
+
+func (fn *GoodDoFnKvNoSideInputs) FinishBundle() {
+}
+
+type GoodDoFnAllExtras struct{}
+
+func (fn *GoodDoFnAllExtras) ProcessElement(context.Context, typex.Window, typex.EventTime, reflect.Type, string, int, func(*int) bool, func() func(*int) bool, func(int)) (typex.EventTime, int, error) {
+ return 0, 0, nil
+}
+
+func (fn *GoodDoFnAllExtras) StartBundle(context.Context, func(*int) bool, func() func(*int) bool, func(int)) {
+}
+
+func (fn *GoodDoFnAllExtras) FinishBundle(context.Context, func(*int) bool, func() func(*int) bool, func(int)) {
+}
+
+func (fn *GoodDoFnAllExtras) Setup(context.Context) error {
+ return nil
+}
+
+func (fn *GoodDoFnAllExtras) Teardown(context.Context) error {
+ return nil
+}
+
+type GoodDoFnUnexportedExtraMethod struct{}
+
+func (fn *GoodDoFnUnexportedExtraMethod) ProcessElement(int) int {
+ return 0
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) StartBundle() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) FinishBundle() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) Setup() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) Teardown() {
+}
+
+func (fn *GoodDoFnUnexportedExtraMethod) unexportedFunction() {
+}
+
+// Examples of incorrect DoFn signatures.
+// Embedding good DoFns avoids repetitive ProcessElement signatures when desired.
+// The immediately following examples are relating to emit parameter mismatches.
+
+type BadDoFnNoEmitsStartBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnNoEmitsStartBundle) StartBundle() {
+}
+
+type BadDoFnMissingEmitsStartBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnMissingEmitsStartBundle) StartBundle(func(int)) {
+}
+
+type BadDoFnMismatchedEmitsStartBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnMismatchedEmitsStartBundle) StartBundle(func(int), func(int)) {
+}
+
+type BadDoFnNoEmitsFinishBundle struct {
+ *GoodDoFnEmits
+}
+
+func (fn *BadDoFnNoEmitsFinishBundle) FinishBundle() {
+}
+
+// Examples of side input mismatches.
+
+type BadDoFnNoSideInputsStartBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnNoSideInputsStartBundle) StartBundle() {
+}
+
+type BadDoFnMissingSideInputsStartBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnMissingSideInputsStartBundle) StartBundle(func(*int) bool) {
+}
+
+type BadDoFnMismatchedSideInputsStartBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnMismatchedSideInputsStartBundle) StartBundle(func(*int) bool, int, func() func(*int)) {
+}
+
+type BadDoFnNoSideInputsFinishBundle struct {
+ *GoodDoFnSideInputs
+}
+
+func (fn *BadDoFnNoSideInputsFinishBundle) FinishBundle() {
+}
+
+// Examples of incorrect Setup/Teardown methods.
+
+type BadDoFnParamsInSetup struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnParamsInSetup) Setup(int) {
+}
+
+type BadDoFnParamsInTeardown struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnParamsInTeardown) Teardown(int) {
+}
+
+type BadDoFnReturnValuesInStartBundle struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInStartBundle) StartBundle() int {
+ return 0
+}
+
+type BadDoFnReturnValuesInFinishBundle struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInFinishBundle) FinishBundle() int {
+ return 0
+}
+
+type BadDoFnReturnValuesInSetup struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInSetup) Setup() int {
+ return 0
+}
+
+type BadDoFnReturnValuesInTeardown struct {
+ *GoodDoFn
+}
+
+func (*BadDoFnReturnValuesInTeardown) Teardown() int {
+ return 0
+}
// Examples of correct CombineFn signatures
+type MyAccum struct{}
+
type GoodCombineFn struct{}
func (fn *GoodCombineFn) MergeAccumulators(MyAccum, MyAccum) MyAccum {
@@ -152,7 +443,7 @@
// Examples of incorrect CombineFn signatures.
// Embedding *GoodCombineFn avoids repetitive MergeAccumulators signatures when desired.
-// The immeadiately following examples are relating to accumulator mismatches.
+// The immediately following examples are relating to accumulator mismatches.
type BadCombineFnNoMergeAccumulators struct{}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/combine_test.go b/sdks/go/pkg/beam/core/runtime/exec/combine_test.go
index c60ab52..49f2d45 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/combine_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/combine_test.go
@@ -168,7 +168,7 @@
inT := typex.NewCoGBK(typex.New(kt), typex.New(vtype))
in := g.NewNode(inT, window.DefaultWindowingStrategy(), true)
- edge, err := graph.NewCombine(g, g.Root(), fn, in, ac)
+ edge, err := graph.NewCombine(g, g.Root(), fn, in, ac, nil)
if err != nil {
t.Fatalf("invalid combinefn: %v", err)
}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/data.go b/sdks/go/pkg/beam/core/runtime/exec/data.go
index c4d0850..2b00a37 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/data.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/data.go
@@ -26,30 +26,22 @@
URL string
}
-// Target represents the static target of external operations.
-type Target struct {
- // ID is the transform ID.
- ID string
- // Name is a local name in the context of the transform.
- Name string
-}
-
// StreamID represents the static information needed to identify
// a data stream. Dynamic information, notably bundleID, is provided
// implicitly by the managers.
type StreamID struct {
- Port Port
- Target Target
+ Port Port
+ PtransformID string
}
func (id StreamID) String() string {
- return fmt.Sprintf("S[%v:%v@%v]", id.Target.ID, id.Target.Name, id.Port.URL)
+ return fmt.Sprintf("S[%v@%v]", id.PtransformID, id.Port.URL)
}
// DataContext holds connectors to various data connections, incl. state and side input.
type DataContext struct {
- Data DataManager
- SideInput SideInputReader
+ Data DataManager
+ State StateReader
}
// DataManager manages external data byte streams. Each data stream can be
@@ -61,10 +53,12 @@
OpenWrite(ctx context.Context, id StreamID) (io.WriteCloser, error)
}
-// SideInputReader is the interface for reading side input data.
-type SideInputReader interface {
- // Open opens a byte stream for reading iterable side input.
- Open(ctx context.Context, id StreamID, key, w []byte) (io.ReadCloser, error)
+// StateReader is the interface for reading side input data.
+type StateReader interface {
+ // OpenSideInput opens a byte stream for reading iterable side input.
+ OpenSideInput(ctx context.Context, id StreamID, sideInputID string, key, w []byte) (io.ReadCloser, error)
+ // OpenIterable opens a byte stream for reading unwindowed iterables from the runner.
+ OpenIterable(ctx context.Context, id StreamID, key []byte) (io.ReadCloser, error)
}
// TODO(herohde) 7/20/2018: user state management
diff --git a/sdks/go/pkg/beam/core/runtime/exec/datasource.go b/sdks/go/pkg/beam/core/runtime/exec/datasource.go
index 5d40b4a..79eae96 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/datasource.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/datasource.go
@@ -19,10 +19,12 @@
"context"
"fmt"
"io"
- "sync/atomic"
+ "math"
+ "sync"
"time"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/ioutilx"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
"github.com/apache/beam/sdks/go/pkg/beam/log"
)
@@ -31,29 +33,42 @@
type DataSource struct {
UID UnitID
SID StreamID
+ Name string
Coder *coder.Coder
Out Node
- source DataManager
- count int64
- start time.Time
+ source DataManager
+ state StateReader
+ index int64
+ splitIdx int64
+ start time.Time
+
+ mu sync.Mutex
}
+// ID returns the UnitID for this node.
func (n *DataSource) ID() UnitID {
return n.UID
}
+// Up initializes this datasource.
func (n *DataSource) Up(ctx context.Context) error {
return nil
}
+// StartBundle initializes this datasource for the bundle.
func (n *DataSource) StartBundle(ctx context.Context, id string, data DataContext) error {
+ n.mu.Lock()
n.source = data.Data
+ n.state = data.State
n.start = time.Now()
- atomic.StoreInt64(&n.count, 0)
+ n.index = -1
+ n.splitIdx = math.MaxInt64
+ n.mu.Unlock()
return n.Out.StartBundle(ctx, id, data)
}
+// Process opens the data source, reads and decodes data, kicking off element processing.
func (n *DataSource) Process(ctx context.Context) error {
r, err := n.source.OpenRead(ctx, n.SID)
if err != nil {
@@ -64,129 +79,161 @@
c := coder.SkipW(n.Coder)
wc := MakeWindowDecoder(n.Coder.Window)
+ var cp ElementDecoder // Decoder for the primary element or the key in CoGBKs.
+ var cvs []ElementDecoder // Decoders for each value stream in CoGBKs.
+
switch {
case coder.IsCoGBK(c):
- ck := MakeElementDecoder(c.Components[0])
- cv := MakeElementDecoder(c.Components[1])
+ cp = MakeElementDecoder(c.Components[0])
- for {
- ws, t, err := DecodeWindowedValueHeader(wc, r)
- if err != nil {
- if err == io.EOF {
- return nil
- }
- return errors.Wrap(err, "source failed")
+ // TODO(BEAM-490): Support multiple value streams (coder components) with
+ // with CoGBK.
+ cvs = []ElementDecoder{MakeElementDecoder(c.Components[1])}
+ default:
+ cp = MakeElementDecoder(c)
+ }
+
+ for {
+ if n.incrementIndexAndCheckSplit() {
+ return nil
+ }
+ ws, t, err := DecodeWindowedValueHeader(wc, r)
+ if err != nil {
+ if err == io.EOF {
+ return nil
}
-
- // Decode key
-
- key, err := ck.Decode(r)
- if err != nil {
- return errors.Wrap(err, "source decode failed")
- }
- key.Timestamp = t
- key.Windows = ws
-
- // TODO(herohde) 4/30/2017: the State API will be handle re-iterations
- // and only "small" value streams would be inline. Presumably, that
- // would entail buffering the whole stream. We do that for now.
-
- var buf []FullValue
-
- size, err := coder.DecodeInt32(r)
- if err != nil {
- return errors.Wrap(err, "stream size decoding failed")
- }
-
- if size > -1 {
- // Single chunk stream.
-
- // log.Printf("Fixed size=%v", size)
- atomic.AddInt64(&n.count, int64(size))
-
- for i := int32(0); i < size; i++ {
- value, err := cv.Decode(r)
- if err != nil {
- return errors.Wrap(err, "stream value decode failed")
- }
- buf = append(buf, *value)
- }
- } else {
- // Multi-chunked stream.
-
- for {
- chunk, err := coder.DecodeVarUint64(r)
- if err != nil {
- return errors.Wrap(err, "stream chunk size decoding failed")
- }
-
- // log.Printf("Chunk size=%v", chunk)
-
- if chunk == 0 {
- break
- }
-
- atomic.AddInt64(&n.count, int64(chunk))
- for i := uint64(0); i < chunk; i++ {
- value, err := cv.Decode(r)
- if err != nil {
- return errors.Wrap(err, "stream value decode failed")
- }
- buf = append(buf, *value)
- }
- }
- }
-
- values := &FixedReStream{Buf: buf}
- if err := n.Out.ProcessElement(ctx, key, values); err != nil {
- return err
- }
+ return errors.Wrap(err, "source failed")
}
- default:
- ec := MakeElementDecoder(c)
+ // Decode key or parallel element.
+ pe, err := cp.Decode(r)
+ if err != nil {
+ return errors.Wrap(err, "source decode failed")
+ }
+ pe.Timestamp = t
+ pe.Windows = ws
- for {
- atomic.AddInt64(&n.count, 1)
- ws, t, err := DecodeWindowedValueHeader(wc, r)
+ var valReStreams []ReStream
+ for _, cv := range cvs {
+ values, err := n.makeReStream(ctx, pe, cv, r)
if err != nil {
- if err == io.EOF {
- return nil
- }
- return errors.Wrap(err, "source failed")
- }
-
- elm, err := ec.Decode(r)
- if err != nil {
- return errors.Wrap(err, "source decode failed")
- }
- elm.Timestamp = t
- elm.Windows = ws
-
- // log.Printf("READ: %v %v", elm.Key.Type(), elm.Key.Interface())
-
- if err := n.Out.ProcessElement(ctx, elm); err != nil {
return err
}
+ valReStreams = append(valReStreams, values)
+ }
+
+ if err := n.Out.ProcessElement(ctx, pe, valReStreams...); err != nil {
+ return err
}
}
}
-func (n *DataSource) FinishBundle(ctx context.Context) error {
- log.Infof(ctx, "DataSource: %d elements in %d ns", atomic.LoadInt64(&n.count), time.Now().Sub(n.start))
- n.source = nil
- err := n.Out.FinishBundle(ctx)
- atomic.StoreInt64(&n.count, 0)
- return err
+func (n *DataSource) makeReStream(ctx context.Context, key *FullValue, cv ElementDecoder, r io.ReadCloser) (ReStream, error) {
+ size, err := coder.DecodeInt32(r)
+ if err != nil {
+ return nil, errors.Wrap(err, "stream size decoding failed")
+ }
+
+ switch {
+ case size >= 0:
+ // Single chunk streams are fully read in and buffered in memory.
+ var buf []FullValue
+ buf, err = readStreamToBuffer(cv, r, int64(size), buf)
+ if err != nil {
+ return nil, err
+ }
+ return &FixedReStream{Buf: buf}, nil
+ case size == -1: // Shouldn't this be 0?
+ // Multi-chunked stream.
+ var buf []FullValue
+ for {
+ chunk, err := coder.DecodeVarInt(r)
+ if err != nil {
+ return nil, errors.Wrap(err, "stream chunk size decoding failed")
+ }
+ // All done, escape out.
+ switch {
+ case chunk == 0: // End of stream, return buffer.
+ return &FixedReStream{Buf: buf}, nil
+ case chunk > 0: // Non-zero chunk, read that many elements from the stream, and buffer them.
+ buf, err = readStreamToBuffer(cv, r, chunk, buf)
+ if err != nil {
+ return nil, err
+ }
+ case chunk == -1: // State backed iterable!
+ chunk, err := coder.DecodeVarInt(r)
+ if err != nil {
+ return nil, err
+ }
+ token, err := ioutilx.ReadN(r, (int)(chunk))
+ if err != nil {
+ return nil, err
+ }
+ return &concatReStream{
+ first: &FixedReStream{Buf: buf},
+ next: &proxyReStream{
+ open: func() (Stream, error) {
+ r, err := n.state.OpenIterable(ctx, n.SID, token)
+ if err != nil {
+ return nil, err
+ }
+ return &elementStream{r: r, ec: cv}, nil
+ },
+ },
+ }, nil
+ default:
+ return nil, errors.Errorf("multi-chunk stream with invalid chunk size of %d", chunk)
+ }
+ }
+ default:
+ return nil, errors.Errorf("received stream with marker size of %d", size)
+ }
}
+func readStreamToBuffer(cv ElementDecoder, r io.ReadCloser, size int64, buf []FullValue) ([]FullValue, error) {
+ for i := int64(0); i < size; i++ {
+ value, err := cv.Decode(r)
+ if err != nil {
+ return nil, errors.Wrap(err, "stream value decode failed")
+ }
+ buf = append(buf, *value)
+ }
+ return buf, nil
+}
+
+// FinishBundle resets the source.
+func (n *DataSource) FinishBundle(ctx context.Context) error {
+ n.mu.Lock()
+ defer n.mu.Unlock()
+ log.Infof(ctx, "DataSource: %d elements in %d ns", n.index, time.Now().Sub(n.start))
+ n.source = nil
+ n.splitIdx = 0 // Ensure errors are returned for split requests if this plan is re-used.
+ return n.Out.FinishBundle(ctx)
+}
+
+// Down resets the source.
func (n *DataSource) Down(ctx context.Context) error {
n.source = nil
return nil
}
func (n *DataSource) String() string {
- return fmt.Sprintf("DataSource[%v] Coder:%v Out:%v", n.SID, n.Coder, n.Out.ID())
+ return fmt.Sprintf("DataSource[%v, %v] Coder:%v Out:%v", n.SID, n.Name, n.Coder, n.Out.ID())
+}
+
+// incrementIndexAndCheckSplit increments DataSource.index by one and checks if
+// the caller should abort further element processing, and finish the bundle.
+// Returns true if the new value of index is greater than or equal to the split
+// index, and false otherwise.
+func (n *DataSource) incrementIndexAndCheckSplit() bool {
+ b := false
+ n.mu.Lock()
+ n.index++
+ if n.index >= n.splitIdx {
+ b = true
+ }
+ n.mu.Unlock()
+ return b
}
// ProgressReportSnapshot captures the progress reading an input source.
@@ -200,5 +247,99 @@
if n == nil {
return ProgressReportSnapshot{}
}
- return ProgressReportSnapshot{n.SID.Target.ID, n.SID.Target.Name, atomic.LoadInt64(&n.count)}
+ n.mu.Lock()
+ // The count is the number of "completely processed elements"
+ // which matches the index of the currently processing element.
+ c := n.index
+ n.mu.Unlock()
+ // Do not sent negative progress reports, index is initialized to 0.
+ if c < 0 {
+ c = 0
+ }
+ return ProgressReportSnapshot{ID: n.SID.PtransformID, Name: n.Name, Count: c}
+}
+
+// Split takes a sorted set of potential split indices, selects and actuates
+// split on an appropriate split index, and returns the selected split index
+// if successful. Returns an error when unable to split.
+func (n *DataSource) Split(splits []int64, frac float32) (int64, error) {
+ if splits == nil {
+ return 0, fmt.Errorf("failed to split: requested splits were empty")
+ }
+ if n == nil {
+ return 0, fmt.Errorf("failed to split at requested splits: {%v}, DataSource not initialized", splits)
+ }
+ n.mu.Lock()
+ c := n.index
+ // Find the smallest split index that we haven't yet processed, and set
+ // the promised split index to this value.
+ for _, s := range splits {
+ if s > 0 && s >= c && s < n.splitIdx {
+ n.splitIdx = s
+ fs := n.splitIdx
+ n.mu.Unlock()
+ return fs, nil
+ }
+ }
+ n.mu.Unlock()
+ // If we can't find a suitable split index from the requested choices,
+ // return an error.
+ return 0, fmt.Errorf("failed to split at requested splits: {%v}, DataSource at index: %v", splits, c)
+}
+
+type concatReStream struct {
+ first, next ReStream
+}
+
+func (c *concatReStream) Open() (Stream, error) {
+ firstStream, err := c.first.Open()
+ if err != nil {
+ return nil, err
+ }
+ return &concatStream{first: firstStream, nextStream: c.next}, nil
+}
+
+type concatStream struct {
+ first Stream
+ nextStream ReStream
+}
+
+// Close nils the stream.
+func (s *concatStream) Close() error {
+ if s.first == nil {
+ return nil
+ }
+ defer func() {
+ s.first = nil
+ s.nextStream = nil
+ }()
+ return s.first.Close()
+}
+
+func (s *concatStream) Read() (*FullValue, error) {
+ if s.first == nil { // When the stream is closed.
+ return nil, io.EOF
+ }
+ fv, err := s.first.Read()
+ if err == nil {
+ return fv, nil
+ }
+ if err == io.EOF {
+ if err := s.first.Close(); err != nil {
+ s.nextStream = nil
+ return nil, err
+ }
+ if s.nextStream == nil {
+ s.first = nil
+ return nil, io.EOF
+ }
+ s.first, err = s.nextStream.Open()
+ s.nextStream = nil
+ if err != nil {
+ return nil, err
+ }
+ fv, err := s.first.Read()
+ return fv, err
+ }
+ return nil, err
}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go b/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go
new file mode 100644
index 0000000..0fa6d23
--- /dev/null
+++ b/sdks/go/pkg/beam/core/runtime/exec/datasource_test.go
@@ -0,0 +1,397 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package exec
+
+import (
+ "context"
+ "fmt"
+ "io"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph/mtime"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
+)
+
+func TestDataSource_PerElement(t *testing.T) {
+ tests := []struct {
+ name string
+ expected []interface{}
+ Coder *coder.Coder
+ driver func(*coder.Coder, io.WriteCloser, []interface{})
+ }{
+ {
+ name: "perElement",
+ expected: []interface{}{int64(1), int64(2), int64(3), int64(4), int64(5)},
+ Coder: coder.NewW(coder.NewVarInt(), coder.NewGlobalWindow()),
+ driver: func(c *coder.Coder, pw io.WriteCloser, expected []interface{}) {
+ wc := MakeWindowEncoder(c.Window)
+ ec := MakeElementEncoder(coder.SkipW(c))
+ for _, v := range expected {
+ EncodeWindowedValueHeader(wc, window.SingleGlobalWindow, mtime.ZeroTimestamp, pw)
+ ec.Encode(&FullValue{Elm: v}, pw)
+ }
+ pw.Close()
+ },
+ },
+ // TODO: Test progress.
+ }
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ out := &CaptureNode{UID: 1}
+ source := &DataSource{
+ UID: 2,
+ SID: StreamID{PtransformID: "myPTransform"},
+ Name: test.name,
+ Coder: test.Coder,
+ Out: out,
+ }
+ pr, pw := io.Pipe()
+ go test.driver(source.Coder, pw, test.expected)
+
+ constructAndExecutePlanWithContext(t, []Unit{out, source}, DataContext{
+ Data: &TestDataManager{R: pr},
+ })
+
+ validateSource(t, out, source, makeValues(test.expected...))
+ })
+ }
+}
+
+const tokenString = "token"
+
+// TestDataSource_Iterators per wire protocols for ITERABLEs beam_runner_api.proto
+func TestDataSource_Iterators(t *testing.T) {
+ extractCoders := func(c *coder.Coder) (WindowEncoder, ElementEncoder, ElementEncoder) {
+ wc := MakeWindowEncoder(c.Window)
+ cc := coder.SkipW(c)
+ kc := MakeElementEncoder(cc.Components[0])
+ vc := MakeElementEncoder(cc.Components[1])
+ return wc, kc, vc
+ }
+
+ tests := []struct {
+ name string
+ keys, vals []interface{}
+ Coder *coder.Coder
+ driver func(c *coder.Coder, dmw io.WriteCloser, siwFn func() io.WriteCloser, ks, vs []interface{})
+ }{
+ {
+ name: "beam:coder:iterable:v1-singleChunk",
+ keys: []interface{}{int64(42), int64(53)},
+ vals: []interface{}{int64(1), int64(2), int64(3), int64(4), int64(5)},
+ Coder: coder.NewW(coder.NewCoGBK([]*coder.Coder{coder.NewVarInt(), coder.NewVarInt()}), coder.NewGlobalWindow()),
+ driver: func(c *coder.Coder, dmw io.WriteCloser, _ func() io.WriteCloser, ks, vs []interface{}) {
+ wc, kc, vc := extractCoders(c)
+ for _, k := range ks {
+ EncodeWindowedValueHeader(wc, window.SingleGlobalWindow, mtime.ZeroTimestamp, dmw)
+ kc.Encode(&FullValue{Elm: k}, dmw)
+ coder.EncodeInt32(int32(len(vs)), dmw) // Number of elements.
+ for _, v := range vs {
+ vc.Encode(&FullValue{Elm: v}, dmw)
+ }
+ }
+ dmw.Close()
+ },
+ },
+ {
+ name: "beam:coder:iterable:v1-multiChunk",
+ keys: []interface{}{int64(42), int64(53)},
+ vals: []interface{}{int64(1), int64(2), int64(3), int64(4), int64(5)},
+ Coder: coder.NewW(coder.NewCoGBK([]*coder.Coder{coder.NewVarInt(), coder.NewVarInt()}), coder.NewGlobalWindow()),
+ driver: func(c *coder.Coder, dmw io.WriteCloser, _ func() io.WriteCloser, ks, vs []interface{}) {
+ wc, kc, vc := extractCoders(c)
+ for _, k := range ks {
+ EncodeWindowedValueHeader(wc, window.SingleGlobalWindow, mtime.ZeroTimestamp, dmw)
+ kc.Encode(&FullValue{Elm: k}, dmw)
+
+ coder.EncodeInt32(-1, dmw) // Mark this as a multi-Chunk (though beam runner proto says to use 0)
+ for _, v := range vs {
+ coder.EncodeVarInt(1, dmw) // Number of elements in this chunk.
+ vc.Encode(&FullValue{Elm: v}, dmw)
+ }
+ coder.EncodeVarInt(0, dmw) // Terminate the multi-chunk for this key.
+ }
+ dmw.Close()
+ },
+ },
+ {
+ name: "beam:coder:state_backed_iterable:v1",
+ keys: []interface{}{int64(42), int64(53)},
+ vals: []interface{}{int64(1), int64(2), int64(3), int64(4), int64(5)},
+ Coder: coder.NewW(coder.NewCoGBK([]*coder.Coder{coder.NewVarInt(), coder.NewVarInt()}), coder.NewGlobalWindow()),
+ driver: func(c *coder.Coder, dmw io.WriteCloser, swFn func() io.WriteCloser, ks, vs []interface{}) {
+ wc, kc, vc := extractCoders(c)
+ for _, k := range ks {
+ EncodeWindowedValueHeader(wc, window.SingleGlobalWindow, mtime.ZeroTimestamp, dmw)
+ kc.Encode(&FullValue{Elm: k}, dmw)
+ coder.EncodeInt32(-1, dmw) // Mark as multi-chunk (though beam, runner says to use 0)
+ coder.EncodeVarInt(-1, dmw) // Mark subsequent chunks as "state backed"
+
+ token := []byte(tokenString)
+ coder.EncodeVarInt(int64(len(token)), dmw) // token.
+ dmw.Write(token)
+ // Each state stream needs to be a different writer, so get a new writer.
+ sw := swFn()
+ for _, v := range vs {
+ vc.Encode(&FullValue{Elm: v}, sw)
+ }
+ sw.Close()
+ }
+ dmw.Close()
+ },
+ },
+ // TODO: Test progress.
+ }
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ out := &IteratorCaptureNode{CaptureNode: CaptureNode{UID: 1}}
+ source := &DataSource{
+ UID: 2,
+ SID: StreamID{PtransformID: "myPTransform"},
+ Name: test.name,
+ Coder: test.Coder,
+ Out: out,
+ }
+ dmr, dmw := io.Pipe()
+
+ // Simulate individual state channels with pipes and a channel.
+ sRc := make(chan io.ReadCloser)
+ swFn := func() io.WriteCloser {
+ sr, sw := io.Pipe()
+ sRc <- sr
+ return sw
+ }
+ go test.driver(source.Coder, dmw, swFn, test.keys, test.vals)
+
+ constructAndExecutePlanWithContext(t, []Unit{out, source}, DataContext{
+ Data: &TestDataManager{R: dmr},
+ State: &TestStateReader{Rc: sRc},
+ })
+ if len(out.CapturedInputs) == 0 {
+ t.Fatal("did not capture source output")
+ }
+
+ expectedKeys := makeValues(test.keys...)
+ expectedValues := makeValuesNoWindowOrTime(test.vals...)
+ if got, want := len(out.CapturedInputs), len(expectedKeys); got != want {
+ t.Fatalf("lengths don't match: got %v, want %v", got, want)
+ }
+ var iVals []FullValue
+ for _, i := range out.CapturedInputs {
+ iVals = append(iVals, i.Key)
+
+ if got, want := i.Values, expectedValues; !equalList(got, want) {
+ t.Errorf("DataSource => key(%v) = %#v, want %#v", i.Key, extractValues(got...), extractValues(want...))
+ }
+ }
+
+ if got, want := iVals, expectedKeys; !equalList(got, want) {
+ t.Errorf("DataSource => %#v, want %#v", extractValues(got...), extractValues(want...))
+ }
+ })
+ }
+}
+
+func TestDataSource_Split(t *testing.T) {
+ elements := []interface{}{int64(1), int64(2), int64(3), int64(4), int64(5)}
+ initSourceTest := func(name string) (*DataSource, *CaptureNode, io.ReadCloser) {
+ out := &CaptureNode{UID: 1}
+ c := coder.NewW(coder.NewVarInt(), coder.NewGlobalWindow())
+ source := &DataSource{
+ UID: 2,
+ SID: StreamID{PtransformID: "myPTransform"},
+ Name: name,
+ Coder: c,
+ Out: out,
+ }
+ pr, pw := io.Pipe()
+
+ go func(c *coder.Coder, pw io.WriteCloser, elements []interface{}) {
+ wc := MakeWindowEncoder(c.Window)
+ ec := MakeElementEncoder(coder.SkipW(c))
+ for _, v := range elements {
+ EncodeWindowedValueHeader(wc, window.SingleGlobalWindow, mtime.ZeroTimestamp, pw)
+ ec.Encode(&FullValue{Elm: v}, pw)
+ }
+ pw.Close()
+ }(c, pw, elements)
+ return source, out, pr
+ }
+
+ tests := []struct {
+ name string
+ expected []interface{}
+ splitIdx int64
+ }{
+ {splitIdx: 1},
+ {splitIdx: 2},
+ {splitIdx: 3},
+ {splitIdx: 4},
+ {splitIdx: 5},
+ {
+ name: "wellBeyondRange",
+ expected: elements,
+ splitIdx: 1000,
+ },
+ }
+ for _, test := range tests {
+ test := test
+ if len(test.name) == 0 {
+ test.name = fmt.Sprintf("atIndex%d", test.splitIdx)
+ }
+ if test.expected == nil {
+ test.expected = elements[:test.splitIdx]
+ }
+ t.Run(test.name, func(t *testing.T) {
+ source, out, pr := initSourceTest(test.name)
+ p, err := NewPlan("a", []Unit{out, source})
+ if err != nil {
+ t.Fatalf("failed to construct plan: %v", err)
+ }
+ dc := DataContext{Data: &TestDataManager{R: pr}}
+ ctx := context.Background()
+
+ // StartBundle resets the source, so no splits can be actuated before then,
+ // which means we need to actuate the plan manually, and insert the split request
+ // after StartBundle.
+ for i, root := range p.units {
+ if err := root.Up(ctx); err != nil {
+ t.Fatalf("error in root[%d].Up: %v", i, err)
+ }
+ }
+ p.status = Active
+
+ runOnRoots(ctx, t, p, "StartBundle", func(root Root, ctx context.Context) error { return root.StartBundle(ctx, "1", dc) })
+
+ // SDK never splits on 0, so check that every test.
+ if splitIdx, err := p.Split(SplitPoints{Splits: []int64{0, test.splitIdx}}); err != nil {
+ t.Fatalf("error in Split: %v", err)
+ } else if got, want := splitIdx, test.splitIdx; got != want {
+ t.Fatalf("error in Split: got splitIdx = %v, want %v ", got, want)
+ }
+ runOnRoots(ctx, t, p, "Process", Root.Process)
+ runOnRoots(ctx, t, p, "FinishBundle", Root.FinishBundle)
+
+ validateSource(t, out, source, makeValues(test.expected...))
+ })
+ }
+
+ // Test expects splitting errors, but for processing to be successful.
+ t.Run("errors", func(t *testing.T) {
+ source, out, pr := initSourceTest("noSplitsUntilStarted")
+ p, err := NewPlan("a", []Unit{out, source})
+ if err != nil {
+ t.Fatalf("failed to construct plan: %v", err)
+ }
+ dc := DataContext{Data: &TestDataManager{R: pr}}
+ ctx := context.Background()
+
+ if _, err := p.Split(SplitPoints{Splits: []int64{0, 3}, Frac: -1}); err == nil {
+ t.Fatal("plan uninitialized, expected error when splitting, got nil")
+ }
+ for i, root := range p.units {
+ if err := root.Up(ctx); err != nil {
+ t.Fatalf("error in root[%d].Up: %v", i, err)
+ }
+ }
+ p.status = Active
+ if _, err := p.Split(SplitPoints{Splits: []int64{0, 3}, Frac: -1}); err == nil {
+ t.Fatal("plan not started, expected error when splitting, got nil")
+ }
+ runOnRoots(ctx, t, p, "StartBundle", func(root Root, ctx context.Context) error { return root.StartBundle(ctx, "1", dc) })
+ if _, err := p.Split(SplitPoints{Splits: []int64{0}, Frac: -1}); err == nil {
+ t.Fatal("plan started, expected error when splitting, got nil")
+ }
+ runOnRoots(ctx, t, p, "Process", Root.Process)
+ if _, err := p.Split(SplitPoints{Splits: []int64{0}, Frac: -1}); err == nil {
+ t.Fatal("plan in progress, expected error when unable to get a desired split, got nil")
+ }
+ runOnRoots(ctx, t, p, "FinishBundle", Root.FinishBundle)
+ if _, err := p.Split(SplitPoints{Splits: []int64{0}, Frac: -1}); err == nil {
+ t.Fatal("plan finished, expected error when splitting, got nil")
+ }
+ validateSource(t, out, source, makeValues(elements...))
+ })
+
+ t.Run("sanity_errors", func(t *testing.T) {
+ var source *DataSource
+ if _, err := source.Split([]int64{0}, -1); err == nil {
+ t.Fatal("expected error splitting nil *DataSource")
+ }
+ if _, err := source.Split(nil, -1); err == nil {
+ t.Fatal("expected error splitting nil desired splits")
+ }
+ })
+}
+
+func runOnRoots(ctx context.Context, t *testing.T, p *Plan, name string, mthd func(Root, context.Context) error) {
+ t.Helper()
+ for i, root := range p.roots {
+ if err := mthd(root, ctx); err != nil {
+ t.Fatalf("error in root[%d].%s: %v", i, name, err)
+ }
+ }
+}
+
+type TestDataManager struct {
+ R io.ReadCloser
+}
+
+func (dm *TestDataManager) OpenRead(ctx context.Context, id StreamID) (io.ReadCloser, error) {
+ return dm.R, nil
+}
+
+func (dm *TestDataManager) OpenWrite(ctx context.Context, id StreamID) (io.WriteCloser, error) {
+ return nil, nil
+}
+
+// TestSideInputReader simulates state reads using channels.
+type TestStateReader struct {
+ StateReader
+ Rc <-chan io.ReadCloser
+}
+
+func (si *TestStateReader) OpenIterable(ctx context.Context, id StreamID, key []byte) (io.ReadCloser, error) {
+ return <-si.Rc, nil
+}
+
+func constructAndExecutePlanWithContext(t *testing.T, us []Unit, dc DataContext) {
+ p, err := NewPlan("a", us)
+ if err != nil {
+ t.Fatalf("failed to construct plan: %v", err)
+ }
+
+ if err := p.Execute(context.Background(), "1", dc); err != nil {
+ t.Fatalf("execute failed: %v", err)
+ }
+ if err := p.Down(context.Background()); err != nil {
+ t.Fatalf("down failed: %v", err)
+ }
+}
+
+func validateSource(t *testing.T, out *CaptureNode, source *DataSource, expected []FullValue) {
+ t.Helper()
+ if got, want := len(out.Elements), len(expected); got != want {
+ t.Fatalf("lengths don't match: got %v, want %v", got, want)
+ }
+ if got, want := source.Progress().Count, int64(len(expected)); got != want {
+ t.Fatalf("progress count didn't match: got %v, want %v", got, want)
+ }
+ if !equalList(out.Elements, expected) {
+ t.Errorf("DataSource => %#v, want %#v", extractValues(out.Elements...), extractValues(expected...))
+ }
+}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/fn.go b/sdks/go/pkg/beam/core/runtime/exec/fn.go
index b6c479c..1211989 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/fn.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/fn.go
@@ -265,7 +265,7 @@
for i := 0; i < len(side); i++ {
s, err := makeSideInput(in[i+1].Kind, fn.Param[param[i+offset]].T, side[i])
if err != nil {
- return nil, errors.WithContextf(err, "making side input %v", i)
+ return nil, errors.WithContextf(err, "making side input %v for %v", i, fn)
}
ret = append(ret, s)
}
@@ -303,7 +303,7 @@
return nil, err
}
if len(elms) != 1 {
- return nil, errors.Errorf("singleton side input %v for %v ill-defined", kind, t)
+ return nil, errors.Errorf("got %d values, want one value for %v side input of type %v", len(elms), graph.Singleton, t)
}
return &fixedValue{val: Convert(elms[0].Elm, t)}, nil
diff --git a/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go b/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
index 4be6fdc..9cdc67d 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
@@ -46,6 +46,16 @@
return ret
}
+func makeValuesNoWindowOrTime(vs ...interface{}) []FullValue {
+ var ret []FullValue
+ for _, v := range vs {
+ ret = append(ret, FullValue{
+ Elm: v,
+ })
+ }
+ return ret
+}
+
// makeKVValues returns a list of KV<K,V> inputs as a list of main inputs.
func makeKVInput(key interface{}, vs ...interface{}) []MainInput {
var ret []MainInput
diff --git a/sdks/go/pkg/beam/core/runtime/exec/pardo.go b/sdks/go/pkg/beam/core/runtime/exec/pardo.go
index 612e7ff..aba5d43 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/pardo.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/pardo.go
@@ -43,7 +43,7 @@
ctx context.Context
inv *invoker
- side SideInputReader
+ side StateReader
cache *cacheElm
status Status
@@ -93,7 +93,7 @@
return errors.Errorf("invalid status for pardo %v: %v, want Up", n.UID, n.status)
}
n.status = Active
- n.side = data.SideInput
+ n.side = data.State
// Allocating contexts all the time is expensive, but we seldom re-write them,
// and never accept modified contexts from users, so we will cache them per-bundle
// per-unit, to avoid the constant allocation overhead.
diff --git a/sdks/go/pkg/beam/core/runtime/exec/plan.go b/sdks/go/pkg/beam/core/runtime/exec/plan.go
index 5031c3e..d221c7e 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/plan.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/plan.go
@@ -87,6 +87,11 @@
return p.id
}
+// SourcePTransformID returns the ID of the data's origin PTransform.
+func (p *Plan) SourcePTransformID() string {
+ return p.source.SID.PtransformID
+}
+
// Execute executes the plan with the given data context and bundle id. Units
// are brought up on the first execution. If a bundle fails, the plan cannot
// be reused for further bundles. Does not panic. Blocking.
@@ -191,3 +196,21 @@
Ptransforms: transforms,
}
}
+
+// SplitPoints captures the split requested by the Runner.
+type SplitPoints struct {
+ // Splits is a list of desired split indices.
+ Splits []int64
+ Frac float32
+}
+
+// Split takes a set of potential split indexes, and if successful returns
+// the split index of the first element of the residual, on which processing
+// will be halted.
+// Returns an error when unable to split.
+func (p *Plan) Split(s SplitPoints) (int64, error) {
+ if p.source != nil {
+ return p.source.Split(s.Splits, s.Frac)
+ }
+ return 0, fmt.Errorf("failed to split at requested splits: {%v}, Source not initialized", s)
+}
diff --git a/sdks/go/pkg/beam/core/runtime/exec/sideinput.go b/sdks/go/pkg/beam/core/runtime/exec/sideinput.go
index e880f05..08cc279 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/sideinput.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/sideinput.go
@@ -32,19 +32,20 @@
// SideInputAdapter provides a concrete ReStream from a low-level side input reader. It
// encapsulates StreamID and coding as needed.
type SideInputAdapter interface {
- NewIterable(ctx context.Context, reader SideInputReader, w typex.Window) (ReStream, error)
+ NewIterable(ctx context.Context, reader StateReader, w typex.Window) (ReStream, error)
}
type sideInputAdapter struct {
- sid StreamID
- wc WindowEncoder
- kc ElementEncoder
- ec ElementDecoder
+ sid StreamID
+ sideInputID string
+ wc WindowEncoder
+ kc ElementEncoder
+ ec ElementDecoder
}
// NewSideInputAdapter returns a side input adapter for the given StreamID and coder.
// It expects a W<KV<K,V>> coder, because the protocol supports MultiSet access only.
-func NewSideInputAdapter(sid StreamID, c *coder.Coder) SideInputAdapter {
+func NewSideInputAdapter(sid StreamID, sideInputID string, c *coder.Coder) SideInputAdapter {
if !coder.IsW(c) || !coder.IsKV(coder.SkipW(c)) {
panic(fmt.Sprintf("expected WKV coder for side input %v: %v", sid, c))
}
@@ -52,10 +53,10 @@
wc := MakeWindowEncoder(c.Window)
kc := MakeElementEncoder(coder.SkipW(c).Components[0])
ec := MakeElementDecoder(coder.SkipW(c).Components[1])
- return &sideInputAdapter{sid: sid, wc: wc, kc: kc, ec: ec}
+ return &sideInputAdapter{sid: sid, sideInputID: sideInputID, wc: wc, kc: kc, ec: ec}
}
-func (s *sideInputAdapter) NewIterable(ctx context.Context, reader SideInputReader, w typex.Window) (ReStream, error) {
+func (s *sideInputAdapter) NewIterable(ctx context.Context, reader StateReader, w typex.Window) (ReStream, error) {
key, err := EncodeElement(s.kc, []byte(iterableSideInputKey))
if err != nil {
return nil, err
@@ -66,7 +67,7 @@
}
return &proxyReStream{
open: func() (Stream, error) {
- r, err := reader.Open(ctx, s.sid, key, win)
+ r, err := reader.OpenSideInput(ctx, s.sid, s.sideInputID, key, win)
if err != nil {
return nil, err
}
@@ -76,7 +77,7 @@
}
func (s *sideInputAdapter) String() string {
- return fmt.Sprintf("SideInputAdapter[%v]", s.sid)
+ return fmt.Sprintf("SideInputAdapter[%v, %v]", s.sid, s.sideInputID)
}
// proxyReStream is a simple wrapper of an open function.
diff --git a/sdks/go/pkg/beam/core/runtime/exec/translate.go b/sdks/go/pkg/beam/core/runtime/exec/translate.go
index e3981d3..e6caafc 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/translate.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/translate.go
@@ -38,8 +38,8 @@
// TODO(lostluck): 2018/05/28 Extract these from the canonical enums in beam_runner_api.proto
const (
- urnDataSource = "urn:org.apache.beam:source:runner:0.1"
- urnDataSink = "urn:org.apache.beam:sink:runner:0.1"
+ urnDataSource = "beam:source:runner:0.1"
+ urnDataSink = "beam:sink:runner:0.1"
urnPerKeyCombinePre = "beam:transform:combine_per_key_precombine:v1"
urnPerKeyCombineMerge = "beam:transform:combine_per_key_merge_accumulators:v1"
urnPerKeyCombineExtract = "beam:transform:combine_per_key_extract_outputs:v1"
@@ -67,7 +67,8 @@
u := &DataSource{UID: b.idgen.New()}
for key, pid := range transform.GetOutputs() {
- u.SID = StreamID{Target: Target{ID: id, Name: key}, Port: port}
+ u.SID = StreamID{PtransformID: id, Port: port}
+ u.Name = key
u.Out, err = b.makePCollection(pid)
if err != nil {
@@ -396,13 +397,11 @@
}
sid := StreamID{
- Port: Port{URL: b.desc.GetStateApiServiceDescriptor().GetUrl()},
- Target: Target{
- ID: id.to, // PTransformID
- Name: fmt.Sprintf("i%v", i), // SideInputID (= local id, "iN")
- },
+ Port: Port{URL: b.desc.GetStateApiServiceDescriptor().GetUrl()},
+ PtransformID: id.to,
}
- side := NewSideInputAdapter(sid, coder.NewW(ec, wc))
+ sideInputID := fmt.Sprintf("i%v", i) // SideInputID (= local id, "iN")
+ side := NewSideInputAdapter(sid, sideInputID, coder.NewW(ec, wc))
n.Side = append(n.Side, side)
}
u = n
@@ -494,6 +493,11 @@
case graphx.URNFlatten:
u = &Flatten{UID: b.idgen.New(), N: len(transform.Inputs), Out: out[0]}
+ // Use the same flatten instance for all the inputs links to this transform.
+ for i := 0; i < len(transform.Inputs); i++ {
+ b.links[linkID{id.to, i}] = u
+ }
+
case urnDataSink:
port, cid, err := unmarshalPort(payload)
if err != nil {
@@ -502,8 +506,8 @@
sink := &DataSink{UID: b.idgen.New()}
- for key, pid := range transform.GetInputs() {
- sink.SID = StreamID{Target: Target{ID: id.to, Name: key}, Port: port}
+ for _, pid := range transform.GetInputs() {
+ sink.SID = StreamID{PtransformID: id.to, Port: port}
if cid == "" {
c, wc, err := b.makeCoderForPCollection(pid)
diff --git a/sdks/go/pkg/beam/core/runtime/exec/unit_test.go b/sdks/go/pkg/beam/core/runtime/exec/unit_test.go
index 0d17746..19e54e8 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/unit_test.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/unit_test.go
@@ -17,6 +17,7 @@
import (
"context"
+ "io"
"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
@@ -76,6 +77,47 @@
return nil
}
+// iterInput keeps a key along with the list of associated values.
+type iterInput struct {
+ Key FullValue
+ Values []FullValue
+}
+
+// IteratorCaptureNode is a test Node that captures all KV pairs elements for
+// verification, including all streamed values. It also validates that it is
+// invoked correctly.
+type IteratorCaptureNode struct {
+ CaptureNode // embedded for the default unit methods
+ CapturedInputs []iterInput
+}
+
+func (n *IteratorCaptureNode) ProcessElement(ctx context.Context, elm *FullValue, values ...ReStream) error {
+ if n.CaptureNode.status != Active {
+ return errors.Errorf("invalid status for pardo %v: %v, want Active", n.CaptureNode.UID, n.CaptureNode.status)
+ }
+ var vs []FullValue
+ for _, iterV := range values {
+ s, err := iterV.Open()
+ if err != nil {
+ return err
+ }
+ defer s.Close()
+ v, err := s.Read()
+ for err == nil {
+ vs = append(vs, *v)
+ v, err = s.Read()
+ }
+ if err == io.EOF {
+ break
+ } else if err != nil {
+ return err
+ }
+ }
+
+ n.CapturedInputs = append(n.CapturedInputs, iterInput{Key: *elm, Values: vs})
+ return nil
+}
+
// FixedRoot is a test Root that emits a fixed number of elements.
type FixedRoot struct {
UID UnitID
@@ -117,7 +159,7 @@
Val ReStream
}
-func (a *FixedSideInputAdapter) NewIterable(ctx context.Context, reader SideInputReader, w typex.Window) (ReStream, error) {
+func (a *FixedSideInputAdapter) NewIterable(ctx context.Context, reader StateReader, w typex.Window) (ReStream, error) {
return a.Val, nil
}
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/coder.go b/sdks/go/pkg/beam/core/runtime/graphx/coder.go
index 50e4591..5ac1f59 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/coder.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/coder.go
@@ -20,7 +20,7 @@
"fmt"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
- "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
+ v1 "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
"github.com/apache/beam/sdks/go/pkg/beam/core/util/protox"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
@@ -31,12 +31,13 @@
const (
// Model constants
- urnBytesCoder = "beam:coder:bytes:v1"
- urnVarIntCoder = "beam:coder:varint:v1"
- urnLengthPrefixCoder = "beam:coder:length_prefix:v1"
- urnKVCoder = "beam:coder:kv:v1"
- urnIterableCoder = "beam:coder:iterable:v1"
- urnWindowedValueCoder = "beam:coder:windowed_value:v1"
+ urnBytesCoder = "beam:coder:bytes:v1"
+ urnVarIntCoder = "beam:coder:varint:v1"
+ urnLengthPrefixCoder = "beam:coder:length_prefix:v1"
+ urnKVCoder = "beam:coder:kv:v1"
+ urnIterableCoder = "beam:coder:iterable:v1"
+ urnStateBackedIterableCoder = "beam:coder:state_backed_iterable:v1"
+ urnWindowedValueCoder = "beam:coder:windowed_value:v1"
urnGlobalWindow = "beam:coder:global_window:v1"
urnIntervalWindow = "beam:coder:interval_window:v1"
@@ -130,7 +131,7 @@
return nil, err
}
- w := urnToWindowCoder(c.GetSpec().GetSpec().GetUrn())
+ w := urnToWindowCoder(c.GetSpec().GetUrn())
b.windowCoders[id] = w
return w, nil
}
@@ -147,7 +148,7 @@
}
func (b *CoderUnmarshaller) makeCoder(c *pb.Coder) (*coder.Coder, error) {
- urn := c.GetSpec().GetSpec().GetUrn()
+ urn := c.GetSpec().GetUrn()
components := c.GetComponentCoderIds()
switch urn {
@@ -175,8 +176,9 @@
if err != nil {
return nil, err
}
- isGBK := elm.GetSpec().GetSpec().GetUrn() == urnIterableCoder
- if isGBK {
+
+ switch elm.GetSpec().GetUrn() {
+ case urnIterableCoder, urnStateBackedIterableCoder:
id = elm.GetComponentCoderIds()[0]
kind = coder.CoGBK
root = typex.CoGBKType
@@ -217,13 +219,13 @@
}
// TODO(lostluck) 2018/10/17: Make this strict again, once dataflow can use
// the portable pipeline model directly (BEAM-2885)
- if elm.GetSpec().GetSpec().GetUrn() != "" && elm.GetSpec().GetSpec().GetUrn() != urnCustomCoder {
+ if elm.GetSpec().GetUrn() != "" && elm.GetSpec().GetUrn() != urnCustomCoder {
// TODO(herohde) 11/17/2017: revisit this restriction
return nil, errors.Errorf("could not unmarshal length prefix coder from %v, want a custom coder as a sub component but got %v", c, elm)
}
var ref v1.CustomCoder
- if err := protox.DecodeBase64(string(elm.GetSpec().GetSpec().GetPayload()), &ref); err != nil {
+ if err := protox.DecodeBase64(string(elm.GetSpec().GetPayload()), &ref); err != nil {
return nil, err
}
custom, err := decodeCustomCoder(&ref)
@@ -256,7 +258,7 @@
// TODO(herohde) 11/27/2017: we still see CoderRefs from Dataflow. Handle that
// case here, for now, so that the harness can use this logic.
- payload := c.GetSpec().GetSpec().GetPayload()
+ payload := c.GetSpec().GetPayload()
var ref CoderRef
if err := json.Unmarshal(payload, &ref); err != nil {
@@ -286,14 +288,14 @@
if err != nil {
return nil, false
}
- if elm.GetSpec().GetSpec().GetUrn() != urnLengthPrefixCoder {
+ if elm.GetSpec().GetUrn() != urnLengthPrefixCoder {
return nil, false
}
elm2, err := b.peek(elm.GetComponentCoderIds()[0])
if err != nil {
return nil, false
}
- if elm2.GetSpec().GetSpec().GetUrn() != urnCoGBKList {
+ if elm2.GetSpec().GetUrn() != urnCoGBKList {
return nil, false
}
return elm2.GetComponentCoderIds(), true
@@ -321,22 +323,18 @@
ref, err := encodeCustomCoder(c.Custom)
if err != nil {
typeName := c.Custom.Name
- panic(fmt.Sprintf("Failed to encode custom coder for type %s. "+
+ panic(errors.SetTopLevelMsgf(err, "Failed to encode custom coder for type %s. "+
"Make sure the type was registered before calling beam.Init. For example: "+
- "beam.RegisterType(reflect.TypeOf((*TypeName)(nil)).Elem())\n\n"+
- "Full error: %v", typeName, err))
+ "beam.RegisterType(reflect.TypeOf((*TypeName)(nil)).Elem())", typeName))
}
data, err := protox.EncodeBase64(ref)
if err != nil {
panic(errors.Wrapf(err, "Failed to marshal custom coder %v", c))
}
inner := b.internCoder(&pb.Coder{
- Spec: &pb.SdkFunctionSpec{
- Spec: &pb.FunctionSpec{
- Urn: urnCustomCoder,
- Payload: []byte(data),
- },
- // TODO(BEAM-3204): coders should not have environments.
+ Spec: &pb.FunctionSpec{
+ Urn: urnCustomCoder,
+ Payload: []byte(data),
},
})
return b.internBuiltInCoder(urnLengthPrefixCoder, inner)
@@ -356,6 +354,7 @@
value = b.internBuiltInCoder(urnLengthPrefixCoder, union)
}
+ // SDKs always provide iterableCoder to runners, but can receive StateBackedIterables in return.
stream := b.internBuiltInCoder(urnIterableCoder, value)
return b.internBuiltInCoder(urnKVCoder, comp[0], stream)
@@ -405,10 +404,8 @@
func (b *CoderMarshaller) internBuiltInCoder(urn string, components ...string) string {
return b.internCoder(&pb.Coder{
- Spec: &pb.SdkFunctionSpec{
- Spec: &pb.FunctionSpec{
- Urn: urn,
- },
+ Spec: &pb.FunctionSpec{
+ Urn: urn,
},
ComponentCoderIds: components,
})
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/serialize.go b/sdks/go/pkg/beam/core/runtime/graphx/serialize.go
index bce12b7..c79935d 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/serialize.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/serialize.go
@@ -27,7 +27,7 @@
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
"github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
- "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
+ v1 "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
"github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
@@ -426,6 +426,11 @@
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
+ if f.PkgPath != "" {
+ wrapped := errors.Errorf("type has unexported field: %v", f.Name)
+ return nil, errors.WithContextf(wrapped, "encoding struct %v", t)
+ }
+
fType, err := encodeType(f.Type)
if err != nil {
wrapped := errors.Wrap(err, "bad field type")
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/serialize_test.go b/sdks/go/pkg/beam/core/runtime/graphx/serialize_test.go
new file mode 100644
index 0000000..5a91019
--- /dev/null
+++ b/sdks/go/pkg/beam/core/runtime/graphx/serialize_test.go
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package graphx
+
+import (
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+ v1 "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/graphx/v1"
+)
+
+func TestEncodeType(t *testing.T) {
+ t.Run("NoUnexportedFields", func(t *testing.T) {
+ type MyAwesomeType struct {
+ ExportedField string
+ }
+ rt := reflect.TypeOf((*MyAwesomeType)(nil)).Elem()
+
+ pbT, err := encodeType(rt)
+ if err != nil {
+ t.Fatalf("got error = %v, want nil", err)
+ }
+ if got, want := pbT.Kind, v1.Type_STRUCT; got != want {
+ t.Fatalf("got pbT.Kind == %v, want %v", got, want)
+ }
+ })
+ t.Run("UnregisteredWithUnexportedField", func(t *testing.T) {
+ type MyProblematicType struct {
+ unexportedField string
+ }
+ rt := reflect.TypeOf((*MyProblematicType)(nil)).Elem()
+ pbT, err := encodeType(rt)
+ if err == nil {
+ t.Fatalf("got type = %v, nil, want unexported field error", pbT)
+ }
+ if !strings.Contains(err.Error(), "type has unexported field: unexportedField") {
+ t.Errorf("expected error about unexported field, got %q", err.Error())
+ }
+ })
+ t.Run("RegisteredWithUnexportedField", func(t *testing.T) {
+ type MyRegisteredType struct {
+ unexportedField string
+ }
+ rt := reflect.TypeOf((*MyRegisteredType)(nil)).Elem()
+ runtime.RegisterType(rt)
+ pbT, err := encodeType(rt)
+ if err != nil {
+ t.Fatalf("got error = %v, want nil", err)
+ }
+ if got, want := pbT.Kind, v1.Type_EXTERNAL; got != want {
+ t.Fatalf("got pbT.Kind == %v, want %v", got, want)
+ }
+ })
+}
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/translate.go b/sdks/go/pkg/beam/core/runtime/graphx/translate.go
index 3798be4..17dc253 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/translate.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/translate.go
@@ -34,7 +34,7 @@
// TODO(lostluck): 2018/05/28 Extract these from their enum descriptors in the pipeline_v1 proto
const (
URNImpulse = "beam:transform:impulse:v1"
- URNParDo = "urn:beam:transform:pardo:v1"
+ URNParDo = "beam:transform:pardo:v1"
URNFlatten = "beam:transform:flatten:v1"
URNGBK = "beam:transform:group_by_key:v1"
URNCombinePerKey = "beam:transform:combine_per_key:v1"
@@ -53,7 +53,7 @@
// URNJavaDoFn is the legacy constant for marking a DoFn.
// TODO: remove URNJavaDoFN when the Dataflow runner
// uses the model pipeline and no longer falls back to Java.
- URNJavaDoFn = "urn:beam:dofn:javasdk:0.1"
+ URNJavaDoFn = "beam:dofn:javasdk:0.1"
URNDoFn = "beam:go:transform:dofn:v1"
URNIterableSideInputKey = "beam:go:transform:iterablesideinputkey:v1"
@@ -128,7 +128,7 @@
var subtransforms []string
for _, edge := range s.Edges {
- subtransforms = append(subtransforms, m.addMultiEdge(edge))
+ subtransforms = append(subtransforms, m.addMultiEdge(edge)...)
}
for _, tree := range s.Children {
subtransforms = append(subtransforms, m.addScopeTree(tree))
@@ -173,14 +173,14 @@
transform.Spec = &pb.FunctionSpec{Urn: URNCombinePerKey, Payload: protox.MustEncode(payload)}
}
-func (m *marshaller) addMultiEdge(edge NamedEdge) string {
+func (m *marshaller) addMultiEdge(edge NamedEdge) []string {
id := edgeID(edge.Edge)
if _, exists := m.transforms[id]; exists {
- return id
+ return []string{id}
}
if edge.Edge.Op == graph.CoGBK && len(edge.Edge.Input) > 1 {
- return m.expandCoGBK(edge)
+ return []string{m.expandCoGBK(edge)}
}
inputs := make(map[string]string)
@@ -194,6 +194,8 @@
outputs[fmt.Sprintf("i%v", i)] = nodeID(out.To)
}
+ // allPIds tracks additional PTransformIDs generated for the pipeline
+ var allPIds []string
var spec *pb.FunctionSpec
switch edge.Edge.Op {
case graph.Impulse:
@@ -238,6 +240,7 @@
Outputs: map[string]string{"i0": out},
}
m.transforms[keyedID] = keyed
+ allPIds = append(allPIds, keyedID)
// Fixup input map
inputs[fmt.Sprintf("i%v", i)] = out
@@ -320,7 +323,8 @@
Outputs: outputs,
}
m.transforms[id] = transform
- return id
+ allPIds = append(allPIds, id)
+ return allPIds
}
func (m *marshaller) expandCoGBK(edge NamedEdge) string {
diff --git a/sdks/go/pkg/beam/core/runtime/graphx/translate_test.go b/sdks/go/pkg/beam/core/runtime/graphx/translate_test.go
index ec2a8b7..0026745 100644
--- a/sdks/go/pkg/beam/core/runtime/graphx/translate_test.go
+++ b/sdks/go/pkg/beam/core/runtime/graphx/translate_test.go
@@ -41,22 +41,36 @@
}
}
-func pick(t *testing.T, g *graph.Graph) *graph.MultiEdge {
- dofn, err := graph.NewDoFn(pickFn)
+func pickSideFn(a, side int, small, big func(int)) {
+ if a < side {
+ small(a)
+ } else {
+ big(a)
+ }
+}
+
+func addDoFn(t *testing.T, g *graph.Graph, fn interface{}, scope *graph.Scope, inputs []*graph.Node, outputCoders []*coder.Coder) {
+ t.Helper()
+ dofn, err := graph.NewDoFn(fn)
if err != nil {
t.Fatal(err)
}
+ e, err := graph.NewParDo(g, scope, dofn, inputs, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(outputCoders) != len(e.Output) {
+ t.Fatalf("%v has %d outputs, but only got %d coders", dofn.Name(), len(e.Output), len(outputCoders))
+ }
+ for i, c := range outputCoders {
+ e.Output[i].To.Coder = c
+ }
+}
+func newIntInput(g *graph.Graph) *graph.Node {
in := g.NewNode(intT(), window.DefaultWindowingStrategy(), true)
in.Coder = intCoder()
-
- e, err := graph.NewParDo(g, g.Root(), dofn, []*graph.Node{in}, nil)
- if err != nil {
- t.Fatal(err)
- }
- e.Output[0].To.Coder = intCoder()
- e.Output[1].To.Coder = intCoder()
- return e
+ return in
}
func intT() typex.FullType {
@@ -67,30 +81,82 @@
return custom("int", reflectx.Int)
}
-// TestParDo verifies that ParDo can be serialized.
-func TestParDo(t *testing.T) {
- g := graph.New()
- pick(t, g)
+// TestMarshal verifies that ParDo can be serialized.
+func TestMarshal(t *testing.T) {
+ tests := []struct {
+ name string
+ makeGraph func(t *testing.T, g *graph.Graph)
+ edges, transforms, roots int
+ }{
+ {
+ name: "ParDo",
+ makeGraph: func(t *testing.T, g *graph.Graph) {
+ addDoFn(t, g, pickFn, g.Root(), []*graph.Node{newIntInput(g)}, []*coder.Coder{intCoder(), intCoder()})
+ },
+ edges: 1,
+ transforms: 1,
+ roots: 1,
+ }, {
+ name: "ScopedParDo",
+ makeGraph: func(t *testing.T, g *graph.Graph) {
+ addDoFn(t, g, pickFn, g.NewScope(g.Root(), "sub"), []*graph.Node{newIntInput(g)}, []*coder.Coder{intCoder(), intCoder()})
+ },
+ edges: 1,
+ transforms: 2,
+ roots: 1,
+ }, {
+ name: "SideInput",
+ makeGraph: func(t *testing.T, g *graph.Graph) {
+ in := newIntInput(g)
+ side := newIntInput(g)
+ addDoFn(t, g, pickSideFn, g.Root(), []*graph.Node{in, side}, []*coder.Coder{intCoder(), intCoder()})
+ },
+ edges: 1,
+ transforms: 2,
+ roots: 2,
+ }, {
+ name: "ScopedSideInput",
+ makeGraph: func(t *testing.T, g *graph.Graph) {
+ in := newIntInput(g)
+ side := newIntInput(g)
+ addDoFn(t, g, pickSideFn, g.NewScope(g.Root(), "sub"), []*graph.Node{in, side}, []*coder.Coder{intCoder(), intCoder()})
+ },
+ edges: 1,
+ transforms: 3,
+ roots: 1,
+ },
+ }
+ for _, test := range tests {
+ test := test
+ t.Run(test.name, func(t *testing.T) {
- edges, _, err := g.Build()
- if err != nil {
- t.Fatal(err)
- }
- if len(edges) != 1 {
- t.Fatal("expected a single edge")
- }
+ g := graph.New()
+ test.makeGraph(t, g)
- payload, err := proto.Marshal(&pb.DockerPayload{ContainerImage: "foo"})
- if err != nil {
- t.Fatal(err)
- }
- p, err := graphx.Marshal(edges,
- &graphx.Options{Environment: pb.Environment{Urn: "beam:env:docker:v1", Payload: payload}})
- if err != nil {
- t.Fatal(err)
- }
+ edges, _, err := g.Build()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(edges) != test.edges {
+ t.Fatal("expected a single edge")
+ }
- if len(p.GetComponents().GetTransforms()) != 1 {
- t.Errorf("bad ParDo translation: %v", proto.MarshalTextString(p))
+ payload, err := proto.Marshal(&pb.DockerPayload{ContainerImage: "foo"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ p, err := graphx.Marshal(edges,
+ &graphx.Options{Environment: pb.Environment{Urn: "beam:env:docker:v1", Payload: payload}})
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ if got, want := len(p.GetComponents().GetTransforms()), test.transforms; got != want {
+ t.Errorf("got %d transforms, want %d : %v", got, want, proto.MarshalTextString(p))
+ }
+ if got, want := len(p.GetRootTransformIds()), test.roots; got != want {
+ t.Errorf("got %d roots, want %d : %v", got, want, proto.MarshalTextString(p))
+ }
+ })
}
}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/datamgr.go b/sdks/go/pkg/beam/core/runtime/harness/datamgr.go
index 2d985d4..453cf9f 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/datamgr.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/datamgr.go
@@ -54,7 +54,7 @@
if err != nil {
return nil, err
}
- return ch.OpenRead(ctx, id.Target, s.instID), nil
+ return ch.OpenRead(ctx, id.PtransformID, s.instID), nil
}
func (s *ScopedDataManager) OpenWrite(ctx context.Context, id exec.StreamID) (io.WriteCloser, error) {
@@ -62,7 +62,7 @@
if err != nil {
return nil, err
}
- return ch.OpenWrite(ctx, id.Target, s.instID), nil
+ return ch.OpenWrite(ctx, id.PtransformID, s.instID), nil
}
func (s *ScopedDataManager) open(ctx context.Context, port exec.Port) (*DataChannel, error) {
@@ -118,8 +118,8 @@
// clientID identifies a client of a connected channel.
type clientID struct {
- target exec.Target
- instID string
+ ptransformID string
+ instID string
}
// This is a reduced version of the full gRPC interface to help with testing.
@@ -147,12 +147,12 @@
func newDataChannel(ctx context.Context, port exec.Port) (*DataChannel, error) {
cc, err := dial(ctx, port.URL, 15*time.Second)
if err != nil {
- return nil, errors.Wrap(err, "failed to connect")
+ return nil, errors.Wrapf(err, "failed to connect to data service at %v", port.URL)
}
client, err := pb.NewBeamFnDataClient(cc).Data(ctx)
if err != nil {
cc.Close()
- return nil, errors.Wrap(err, "failed to connect to data service")
+ return nil, errors.Wrapf(err, "failed to create data client on %v", port.URL)
}
return makeDataChannel(ctx, port.URL, client), nil
}
@@ -169,12 +169,12 @@
return ret
}
-func (c *DataChannel) OpenRead(ctx context.Context, target exec.Target, instID string) io.ReadCloser {
- return c.makeReader(ctx, clientID{target: target, instID: instID})
+func (c *DataChannel) OpenRead(ctx context.Context, ptransformID string, instID string) io.ReadCloser {
+ return c.makeReader(ctx, clientID{ptransformID: ptransformID, instID: instID})
}
-func (c *DataChannel) OpenWrite(ctx context.Context, target exec.Target, instID string) io.WriteCloser {
- return c.makeWriter(ctx, clientID{target: target, instID: instID})
+func (c *DataChannel) OpenWrite(ctx context.Context, ptransformID string, instID string) io.WriteCloser {
+ return c.makeWriter(ctx, clientID{ptransformID: ptransformID, instID: instID})
}
func (c *DataChannel) read(ctx context.Context) {
@@ -184,10 +184,11 @@
if err != nil {
if err == io.EOF {
// TODO(herohde) 10/12/2017: can this happen before shutdown? Reconnect?
- log.Warnf(ctx, "DataChannel %v closed", c.id)
+ log.Warnf(ctx, "DataChannel.read %v closed", c.id)
return
}
- panic(errors.Wrapf(err, "channel %v bad", c.id))
+ log.Errorf(ctx, "DataChannel.read %v bad", c.id)
+ return
}
recordStreamReceive(msg)
@@ -197,7 +198,7 @@
// to reduce lock contention.
for _, elm := range msg.GetData() {
- id := clientID{target: exec.Target{ID: elm.GetTarget().PrimitiveTransformReference, Name: elm.GetTarget().GetName()}, instID: elm.GetInstructionReference()}
+ id := clientID{ptransformID: elm.TransformId, instID: elm.GetInstructionId()}
// log.Printf("Chan read (%v): %v\n", sid, elm.GetData())
@@ -329,12 +330,11 @@
w.ch.mu.Lock()
defer w.ch.mu.Unlock()
delete(w.ch.writers, w.id)
- target := &pb.Target{PrimitiveTransformReference: w.id.target.ID, Name: w.id.target.Name}
msg := &pb.Elements{
Data: []*pb.Elements_Data{
{
- InstructionReference: w.id.instID,
- Target: target,
+ InstructionId: w.id.instID,
+ TransformId: w.id.ptransformID,
// Empty data == sentinel
},
},
@@ -354,13 +354,12 @@
return nil
}
- target := &pb.Target{PrimitiveTransformReference: w.id.target.ID, Name: w.id.target.Name}
msg := &pb.Elements{
Data: []*pb.Elements_Data{
{
- InstructionReference: w.id.instID,
- Target: target,
- Data: w.buf,
+ InstructionId: w.id.instID,
+ TransformId: w.id.ptransformID,
+ Data: w.buf,
},
},
}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go b/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go
index 2a4b2d7..1bbf22e 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/datamgr_test.go
@@ -22,7 +22,6 @@
"log"
"testing"
- "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
)
@@ -36,12 +35,9 @@
f.calls++
data := []byte{1, 2, 3, 4}
elemData := pb.Elements_Data{
- InstructionReference: "inst_ref",
- Data: data,
- Target: &pb.Target{
- PrimitiveTransformReference: "ptr",
- Name: "instruction_name",
- },
+ InstructionId: "inst_ref",
+ Data: data,
+ TransformId: "ptr",
}
msg := pb.Elements{}
@@ -82,7 +78,7 @@
client := &fakeClient{t: t, done: done}
c := makeDataChannel(context.Background(), "id", client)
- r := c.OpenRead(context.Background(), exec.Target{ID: "ptr", Name: "instruction_name"}, "inst_ref")
+ r := c.OpenRead(context.Background(), "ptr", "inst_ref")
var read = make([]byte, 4)
// We don't read up all the buffered data, but immediately close the reader.
diff --git a/sdks/go/pkg/beam/core/runtime/harness/harness.go b/sdks/go/pkg/beam/core/runtime/harness/harness.go
index 0c40f27..dcc7922 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/harness.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/harness.go
@@ -76,7 +76,7 @@
log.Debugf(ctx, "RESP: %v", proto.MarshalTextString(resp))
if err := client.Send(resp); err != nil {
- log.Errorf(ctx, "Failed to respond: %v", err)
+ log.Errorf(ctx, "control.Send: Failed to respond: %v", err)
}
}
}()
@@ -102,7 +102,7 @@
recordFooter()
return nil
}
- return errors.Wrapf(err, "recv failed")
+ return errors.Wrapf(err, "control.Recv failed")
}
// Launch a goroutine to handle the control message.
@@ -180,7 +180,7 @@
log.Debugf(ctx, "PB: %v", msg)
- ref := msg.GetProcessBundleDescriptorReference()
+ ref := msg.GetProcessBundleDescriptorId()
c.mu.Lock()
plan, ok := c.plans[ref]
// Make the plan active, and remove it from candidates
@@ -194,10 +194,10 @@
}
data := NewScopedDataManager(c.data, id)
- side := NewScopedSideInputReader(c.state, id)
- err := plan.Execute(ctx, id, exec.DataContext{Data: data, SideInput: side})
+ state := NewScopedStateReader(c.state, id)
+ err := plan.Execute(ctx, id, exec.DataContext{Data: data, State: state})
data.Close()
- side.Close()
+ state.Close()
m := plan.Metrics()
// Move the plan back to the candidate state
@@ -224,7 +224,7 @@
// log.Debugf(ctx, "PB Progress: %v", msg)
- ref := msg.GetInstructionReference()
+ ref := msg.GetInstructionId()
c.mu.Lock()
plan, ok := c.active[ref]
c.mu.Unlock()
@@ -247,11 +247,36 @@
msg := req.GetProcessBundleSplit()
log.Debugf(ctx, "PB Split: %v", msg)
+ ref := msg.GetInstructionId()
+ c.mu.Lock()
+ plan, ok := c.active[ref]
+ c.mu.Unlock()
+ if !ok {
+ return fail(id, "execution plan for %v not found", ref)
+ }
+
+ // Get the desired splits for the root FnAPI read operation.
+ ds := msg.GetDesiredSplits()[plan.SourcePTransformID()]
+ if ds == nil {
+ return fail(id, "failed to split: desired splits for root was empty.")
+ }
+ split, err := plan.Split(exec.SplitPoints{ds.GetAllowedSplitPoints(), ds.GetFractionOfRemainder()})
+
+ if err != nil {
+ return fail(id, "unable to split: %v", err)
+ }
return &fnpb.InstructionResponse{
InstructionId: id,
Response: &fnpb.InstructionResponse_ProcessBundleSplit{
- ProcessBundleSplit: &fnpb.ProcessBundleSplitResponse{},
+ ProcessBundleSplit: &fnpb.ProcessBundleSplitResponse{
+ ChannelSplits: []*fnpb.ProcessBundleSplitResponse_ChannelSplit{
+ &fnpb.ProcessBundleSplitResponse_ChannelSplit{
+ LastPrimaryElement: int32(split - 1),
+ FirstResidualElement: int32(split),
+ },
+ },
+ },
},
}
diff --git a/sdks/go/pkg/beam/core/runtime/harness/init/init.go b/sdks/go/pkg/beam/core/runtime/harness/init/init.go
index ca637d0..889c33b 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/init/init.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/init/init.go
@@ -46,6 +46,22 @@
options = flag.String("options", "", "JSON-encoded pipeline options (required in worker mode).")
)
+type exitMode int
+
+const (
+ // Terminate means the hook should exit itself when the worker harness returns.
+ Terminate exitMode = iota
+ // Return means that the hook should return out, and allow the calling code to
+ // determine if and when the process exits.
+ // This may cause errors that caused worker failure to be ignored.
+ Return
+)
+
+var (
+ // ShutdownMode allows the runner to set how the worker harness should exit.
+ ShutdownMode = Terminate
+)
+
func init() {
runtime.RegisterInit(hook)
}
@@ -65,7 +81,7 @@
if *options != "" {
var opt runtime.RawOptionsWrapper
if err := json.Unmarshal([]byte(*options), &opt); err != nil {
- fmt.Fprintf(os.Stderr, "Failed to parse pipeline options '%v': %v", *options, err)
+ fmt.Fprintf(os.Stderr, "Failed to parse pipeline options '%v': %v\n", *options, err)
os.Exit(1)
}
runtime.GlobalOptions.Import(opt.Options)
@@ -73,9 +89,16 @@
defer func() {
if r := recover(); r != nil {
- fmt.Fprintf(os.Stderr, "Worker panic: %v", r)
+ fmt.Fprintf(os.Stderr, "Worker panic: %v\n", r)
debug.PrintStack()
- os.Exit(2)
+ switch ShutdownMode {
+ case Terminate:
+ os.Exit(2)
+ case Return:
+ return
+ default:
+ panic(fmt.Sprintf("unknown ShutdownMode: %v", ShutdownMode))
+ }
}
}()
@@ -84,11 +107,17 @@
ctx := grpcx.WriteWorkerID(context.Background(), *id)
if err := harness.Main(ctx, *loggingEndpoint, *controlEndpoint); err != nil {
- fmt.Fprintf(os.Stderr, "Worker failed: %v", err)
- os.Exit(1)
+ fmt.Fprintf(os.Stderr, "Worker failed: %v\n", err)
+ switch ShutdownMode {
+ case Terminate:
+ os.Exit(1)
+ case Return:
+ return
+ default:
+ panic(fmt.Sprintf("unknown ShutdownMode: %v", ShutdownMode))
+ }
}
-
- fmt.Fprint(os.Stderr, "Worker exited successfully!")
+ fmt.Fprintln(os.Stderr, "Worker exited successfully!")
for {
// Just hang around until we're terminated.
time.Sleep(time.Hour)
diff --git a/sdks/go/pkg/beam/core/runtime/harness/logging.go b/sdks/go/pkg/beam/core/runtime/harness/logging.go
index ad24148..63afe7c 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/logging.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/logging.go
@@ -29,7 +29,7 @@
)
// TODO(herohde) 10/12/2017: make this file a separate package. Then
-// populate InstructionReference and PrimitiveTransformReference properly.
+// populate InstructionId and TransformId properly.
// TODO(herohde) 10/13/2017: add top-level harness.Main panic handler that flushes logs.
// Also make logger flush on Fatal severity messages.
@@ -65,7 +65,7 @@
entry.LogLocation = fmt.Sprintf("%v:%v", file, line)
}
if id, ok := tryGetInstID(ctx); ok {
- entry.InstructionReference = id
+ entry.InstructionId = id
}
select {
diff --git a/sdks/go/pkg/beam/core/runtime/harness/statemgr.go b/sdks/go/pkg/beam/core/runtime/harness/statemgr.go
index f5d3102..ff20ff9 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/statemgr.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/statemgr.go
@@ -30,9 +30,9 @@
"github.com/golang/protobuf/proto"
)
-// ScopedSideInputReader scopes the global gRPC state manager to a single instruction
+// ScopedStateReader scopes the global gRPC state manager to a single instruction
// for side input use. The indirection makes it easier to control access.
-type ScopedSideInputReader struct {
+type ScopedStateReader struct {
mgr *StateChannelManager
instID string
@@ -41,12 +41,26 @@
mu sync.Mutex
}
-// NewScopedSideInputReader returns a ScopedSideInputReader for the given instruction.
-func NewScopedSideInputReader(mgr *StateChannelManager, instID string) *ScopedSideInputReader {
- return &ScopedSideInputReader{mgr: mgr, instID: instID}
+// NewScopedStateReader returns a ScopedStateReader for the given instruction.
+func NewScopedStateReader(mgr *StateChannelManager, instID string) *ScopedStateReader {
+ return &ScopedStateReader{mgr: mgr, instID: instID}
}
-func (s *ScopedSideInputReader) Open(ctx context.Context, id exec.StreamID, key, w []byte) (io.ReadCloser, error) {
+// OpenSideInput opens a byte stream for reading iterable side input.
+func (s *ScopedStateReader) OpenSideInput(ctx context.Context, id exec.StreamID, sideInputID string, key, w []byte) (io.ReadCloser, error) {
+ return s.openReader(ctx, id, func(ch *StateChannel) *stateKeyReader {
+ return newSideInputReader(ch, id, sideInputID, s.instID, key, w)
+ })
+}
+
+// OpenIterable opens a byte stream for reading unwindowed iterables from the runner.
+func (s *ScopedStateReader) OpenIterable(ctx context.Context, id exec.StreamID, key []byte) (io.ReadCloser, error) {
+ return s.openReader(ctx, id, func(ch *StateChannel) *stateKeyReader {
+ return newRunnerReader(ch, s.instID, key)
+ })
+}
+
+func (s *ScopedStateReader) openReader(ctx context.Context, id exec.StreamID, readerFn func(*StateChannel) *stateKeyReader) (*stateKeyReader, error) {
ch, err := s.open(ctx, id.Port)
if err != nil {
return nil, err
@@ -57,13 +71,13 @@
s.mu.Unlock()
return nil, errors.Errorf("instruction %v no longer processing", s.instID)
}
- ret := newSideInputReader(ch, id.Target, s.instID, key, w)
+ ret := readerFn(ch)
s.opened = append(s.opened, ret)
s.mu.Unlock()
return ret, nil
}
-func (s *ScopedSideInputReader) open(ctx context.Context, port exec.Port) (*StateChannel, error) {
+func (s *ScopedStateReader) open(ctx context.Context, port exec.Port) (*StateChannel, error) {
s.mu.Lock()
if s.closed {
s.mu.Unlock()
@@ -75,7 +89,8 @@
return local.Open(ctx, port) // don't hold lock over potentially slow operation
}
-func (s *ScopedSideInputReader) Close() error {
+// Close closes all open readers.
+func (s *ScopedStateReader) Close() error {
s.mu.Lock()
s.closed = true
s.mgr = nil
@@ -87,7 +102,7 @@
return nil
}
-type sideInputReader struct {
+type stateKeyReader struct {
instID string
key *pb.StateKey
@@ -100,25 +115,40 @@
mu sync.Mutex
}
-func newSideInputReader(ch *StateChannel, target exec.Target, instID string, k, w []byte) *sideInputReader {
+func newSideInputReader(ch *StateChannel, id exec.StreamID, sideInputID string, instID string, k, w []byte) *stateKeyReader {
key := &pb.StateKey{
Type: &pb.StateKey_MultimapSideInput_{
MultimapSideInput: &pb.StateKey_MultimapSideInput{
- PtransformId: target.ID,
- SideInputId: target.Name,
- Window: w,
- Key: k,
+ TransformId: id.PtransformID,
+ SideInputId: sideInputID,
+ Window: w,
+ Key: k,
},
},
}
- return &sideInputReader{
+ return &stateKeyReader{
instID: instID,
key: key,
ch: ch,
}
}
-func (r *sideInputReader) Read(buf []byte) (int, error) {
+func newRunnerReader(ch *StateChannel, instID string, k []byte) *stateKeyReader {
+ key := &pb.StateKey{
+ Type: &pb.StateKey_Runner_{
+ Runner: &pb.StateKey_Runner{
+ Key: k,
+ },
+ },
+ }
+ return &stateKeyReader{
+ instID: instID,
+ key: key,
+ ch: ch,
+ }
+}
+
+func (r *stateKeyReader) Read(buf []byte) (int, error) {
if r.buf == nil {
if r.eof {
return 0, io.EOF
@@ -136,8 +166,8 @@
req := &pb.StateRequest{
// Id: set by channel
- InstructionReference: r.instID,
- StateKey: r.key,
+ InstructionId: r.instID,
+ StateKey: r.key,
Request: &pb.StateRequest_Get{
Get: &pb.StateGetRequest{
ContinuationToken: r.token,
@@ -149,8 +179,12 @@
return 0, err
}
get := resp.GetGet()
- r.token = get.ContinuationToken
- r.buf = get.Data
+ if get == nil { // no data associated with this segment.
+ r.eof = true
+ return 0, io.EOF
+ }
+ r.token = get.GetContinuationToken()
+ r.buf = get.GetData()
if r.token == nil {
r.eof = true // no token == this is the last segment.
@@ -167,7 +201,7 @@
return n, nil
}
-func (r *sideInputReader) Close() error {
+func (r *stateKeyReader) Close() error {
r.mu.Lock()
r.closed = true
r.ch = nil
@@ -219,12 +253,12 @@
func newStateChannel(ctx context.Context, port exec.Port) (*StateChannel, error) {
cc, err := dial(ctx, port.URL, 15*time.Second)
if err != nil {
- return nil, errors.Wrap(err, "failed to connect")
+ return nil, errors.Wrapf(err, "failed to connect to state service %v", port.URL)
}
client, err := pb.NewBeamFnStateClient(cc).State(ctx)
if err != nil {
cc.Close()
- return nil, errors.Wrap(err, "failed to connect to data service")
+ return nil, errors.Wrapf(err, "failed to create state client %v", port.URL)
}
ret := &StateChannel{
@@ -245,10 +279,11 @@
if err != nil {
if err == io.EOF {
// TODO(herohde) 10/12/2017: can this happen before shutdown? Reconnect?
- log.Warnf(ctx, "StateChannel %v closed", c.id)
+ log.Warnf(ctx, "StateChannel[%v].read: closed", c.id)
return
}
- panic(errors.Wrapf(err, "state channel %v bad", c.id))
+ log.Errorf(ctx, "StateChannel[%v].read bad: %v", c.id, err)
+ return
}
c.mu.Lock()
@@ -258,7 +293,7 @@
if !ok {
// This can happen if Send returns an error that write handles, but
// the message was actually sent.
- log.Errorf(ctx, "no consumer for state response: %v", proto.MarshalTextString(msg))
+ log.Errorf(ctx, "StateChannel[%v].read: no consumer for state response: %v", c.id, proto.MarshalTextString(msg))
continue
}
@@ -266,7 +301,7 @@
case ch <- msg:
// ok
default:
- panic(fmt.Sprintf("failed to consume state response: %v", proto.MarshalTextString(msg)))
+ panic(fmt.Sprintf("StateChannel[%v].read: failed to consume state response: %v", c.id, proto.MarshalTextString(msg)))
}
}
}
diff --git a/sdks/go/pkg/beam/core/typex/fulltype.go b/sdks/go/pkg/beam/core/typex/fulltype.go
index a96e6f9..011c91d 100644
--- a/sdks/go/pkg/beam/core/typex/fulltype.go
+++ b/sdks/go/pkg/beam/core/typex/fulltype.go
@@ -180,6 +180,14 @@
return t
}
+// SkipK skips the key in a KV layer, if present. If no, returns the input.
+func SkipK(t FullType) FullType {
+ if t.Type() == KVType {
+ return t.Components()[1]
+ }
+ return t
+}
+
// IsKV returns true iff the type is a KV.
func IsKV(t FullType) bool {
return t.Type() == KVType
diff --git a/sdks/go/pkg/beam/core/util/dot/dot.go b/sdks/go/pkg/beam/core/util/dot/dot.go
index d4d09a2..2b17088 100644
--- a/sdks/go/pkg/beam/core/util/dot/dot.go
+++ b/sdks/go/pkg/beam/core/util/dot/dot.go
@@ -19,6 +19,7 @@
import (
"fmt"
"io"
+ "path"
"text/template"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
@@ -35,23 +36,26 @@
bgcolor="lightgray";
style="solid";
penwidth="0.5";
- concentrate="true";
+ concentrate="true";
- // Node definition used for multiedge
+ // Node definition used for multiedge
node [shape="rectangle" style="filled" fillcolor="honeydew" fontname="Ubuntu" penwidth="1.0" margin="0.05,0.0.05"];
- bgcolor="#e6ecfa";
+ bgcolor="#e6ecfa";
`
nodeText = ` "{{.Name}}" [ shape="ellipse" fillcolor = "lightblue" label="{{.Label}}"]
`
+ edgeDefnText = ` "{{.Name}}" [ label="{{.Label}}" ]
+`
edgeText = ` "{{.From}}" -> "{{.To}}"
`
footer = `
}
`
- nodeTmpl = template.Must(template.New("node").Parse(nodeText))
- edgeTmpl = template.Must(template.New("edge").Parse(edgeText))
+ nodeTmpl = template.Must(template.New("node").Parse(nodeText))
+ edgeDefnTmpl = template.Must(template.New("edge_defn").Parse(edgeDefnText))
+ edgeTmpl = template.Must(template.New("edge").Parse(edgeText))
)
type nodeLinks struct {
@@ -112,6 +116,13 @@
for _, edge := range edges {
e := fmt.Sprintf("%d: %s", edge.ID(), edge.Op)
+ label := fmt.Sprint(edge.Op)
+ if name := path.Base(edge.Name()); name != label {
+ label = fmt.Sprintf("%s\n%s", edge.Op, name)
+ }
+ if err := edgeDefnTmpl.Execute(w, struct{ Name, Label string }{e, label}); err != nil {
+ return errors.Wrap(err, "render DOT failed")
+ }
for _, ib := range edge.Input {
err := edgeTmpl.Execute(w, struct{ From, To string }{ib.From.String(), e})
if err != nil {
diff --git a/sdks/go/pkg/beam/core/util/reflectx/functions.go b/sdks/go/pkg/beam/core/util/reflectx/functions.go
index bd8cacef..4727a8e 100644
--- a/sdks/go/pkg/beam/core/util/reflectx/functions.go
+++ b/sdks/go/pkg/beam/core/util/reflectx/functions.go
@@ -40,6 +40,8 @@
// points to a valid function implementation.
func LoadFunction(ptr uintptr, t reflect.Type) interface{} {
v := reflect.New(t).Elem()
- *(*uintptr)(unsafe.Pointer(v.Addr().Pointer())) = (uintptr)(unsafe.Pointer(&ptr))
+ p := new(uintptr)
+ *p = ptr
+ *(*unsafe.Pointer)(unsafe.Pointer(v.Addr().Pointer())) = unsafe.Pointer(p)
return v.Interface()
}
diff --git a/sdks/go/pkg/beam/core/util/reflectx/functions_test.go b/sdks/go/pkg/beam/core/util/reflectx/functions_test.go
new file mode 100644
index 0000000..8d2e92c
--- /dev/null
+++ b/sdks/go/pkg/beam/core/util/reflectx/functions_test.go
@@ -0,0 +1,43 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package reflectx
+
+import (
+ "reflect"
+ "testing"
+)
+
+func testFunction() int64 {
+ return 42
+}
+
+func TestLoadFunction(t *testing.T) {
+ val := reflect.ValueOf(testFunction)
+ fi := uintptr(val.Pointer())
+ typ := val.Type()
+
+ callable := LoadFunction(fi, typ)
+
+ cv := reflect.ValueOf(callable)
+ out := cv.Call(nil)
+ if len(out) != 1 {
+ t.Errorf("got %d return values, wanted 1.", len(out))
+ }
+
+ if out[0].Int() != testFunction() {
+ t.Errorf("got %d, wanted %d", out[0].Int(), testFunction())
+ }
+}
diff --git a/sdks/go/pkg/beam/core/util/reflectx/structs.go b/sdks/go/pkg/beam/core/util/reflectx/structs.go
index 4c38cf5..84f9be9 100644
--- a/sdks/go/pkg/beam/core/util/reflectx/structs.go
+++ b/sdks/go/pkg/beam/core/util/reflectx/structs.go
@@ -44,7 +44,7 @@
}
key := t.String()
- if _, exists := funcs[key]; exists {
+ if _, exists := structFuncs[key]; exists {
log.Warnf(context.Background(), "StructWrapper for %v already registered. Overwriting.", key)
}
structFuncs[key] = wrapper
diff --git a/sdks/go/pkg/beam/core/util/symtab/symtab.go b/sdks/go/pkg/beam/core/util/symtab/symtab.go
index ded72a7..68abfd8 100644
--- a/sdks/go/pkg/beam/core/util/symtab/symtab.go
+++ b/sdks/go/pkg/beam/core/util/symtab/symtab.go
@@ -21,19 +21,46 @@
"debug/elf"
"debug/macho"
"debug/pe"
+ "fmt"
"os"
+ "reflect"
+ "runtime"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
)
// SymbolTable allows for mapping between symbols and their addresses.
type SymbolTable struct {
- data *dwarf.Data
+ data *dwarf.Data
+ offset uintptr // offset between file addresses and runtime addresses
}
// New creates a new symbol table based on the debug info
// read from the specified file.
func New(filename string) (*SymbolTable, error) {
+ d, err := dwarfData(filename)
+ if err != nil {
+ return nil, err
+ }
+
+ sym := &SymbolTable{data: d}
+
+ // Work out the offset between the file addresses and the
+ // runtime addreses, in case this is a position independent
+ // executable.
+ runtimeAddr := reflect.ValueOf(New).Pointer()
+ name := fnname()
+ fileAddr, err := sym.Sym2Addr(name)
+ if err != nil {
+ return nil, fmt.Errorf("failed to reverse lookup known function %s: %v", name, err)
+ }
+ sym.offset = runtimeAddr - fileAddr
+
+ return sym, nil
+}
+
+// dwarfData returns the debug info for the specified file.
+func dwarfData(filename string) (*dwarf.Data, error) {
f, err := os.Open(filename)
if err != nil {
return nil, err
@@ -51,7 +78,7 @@
f.Close()
return nil, errors.Wrap(err, "No working DWARF")
}
- return &SymbolTable{d}, nil
+ return d, nil
}
// then Mach-O
@@ -62,7 +89,7 @@
f.Close()
return nil, errors.Wrap(err, "No working DWARF")
}
- return &SymbolTable{d}, nil
+ return d, nil
}
// finally try Windows PE format
@@ -73,7 +100,7 @@
f.Close()
return nil, errors.Wrap(err, "No working DWARF")
}
- return &SymbolTable{d}, nil
+ return d, nil
}
// Give up, we don't recognize it
@@ -81,8 +108,18 @@
return nil, errors.New("Unknown file format")
}
+// fnname returns the name of the function that called it.
+func fnname() string {
+ var pcs [2]uintptr
+ n := runtime.Callers(2, pcs[:])
+ frames := runtime.CallersFrames(pcs[:n])
+ frame, _ := frames.Next()
+ return frame.Func.Name()
+}
+
// Addr2Sym returns the symbol name for the provided address.
func (s *SymbolTable) Addr2Sym(addr uintptr) (string, error) {
+ addr -= s.offset
reader := s.data.Reader()
for {
e, err := reader.Next()
@@ -121,7 +158,7 @@
nf := e.Field[0]
if nf.Attr.String() == "Name" && nf.Val.(string) == symbol {
addr := e.Field[1].Val.(uint64)
- return uintptr(addr), nil
+ return uintptr(addr) + s.offset, nil
}
}
}
diff --git a/sdks/go/pkg/beam/core/util/symtab/symtab_test.go b/sdks/go/pkg/beam/core/util/symtab/symtab_test.go
new file mode 100644
index 0000000..d60f7f0
--- /dev/null
+++ b/sdks/go/pkg/beam/core/util/symtab/symtab_test.go
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package symtab
+
+import (
+ "io/ioutil"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "testing"
+)
+
+// TestSym2Addr builds and runs this test program.
+const testprog = `
+package main
+
+import (
+ "fmt"
+ "os"
+ "runtime"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/symtab"
+)
+
+func die(format string, a ...interface{}) {
+ fmt.Fprintf(os.Stderr, format, a...)
+ os.Exit(1)
+}
+
+func main() {
+ syms, err := symtab.New(os.Args[0])
+ if err != nil {
+ die("%s: could not read symbols: %v", os.Args[0], err)
+ }
+
+ symPC, err := syms.Sym2Addr("main.main")
+ if err != nil {
+ die("Sym2Addr(%q) failed: %v", "main.main", err)
+ }
+
+ runtimePC := fnaddr()
+ if symPC != runtimePC {
+ die("PC from symbol table %x != runtime PC %x", symPC, runtimePC)
+ }
+}
+
+// fnaddr returns the entry address of its caller.
+func fnaddr() uintptr {
+ var pcs [2]uintptr
+ n := runtime.Callers(2, pcs[:])
+ frames := runtime.CallersFrames(pcs[:n])
+ frame, _ := frames.Next()
+ return frame.Func.Entry()
+}
+`
+
+func TestSym2Addr(t *testing.T) {
+ f, err := ioutil.TempFile("", "TestSym2Addr*.go")
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ fname := f.Name()
+ defer os.Remove(fname)
+
+ if _, err := f.WriteString(testprog); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.Close(); err != nil {
+ t.Fatal(err)
+ }
+
+ bin := strings.TrimSuffix(fname, ".go")
+ defer os.Remove(bin)
+
+ gotool := filepath.Join(runtime.GOROOT(), "bin", "go")
+
+ for _, arg := range []string{"-buildmode=exe", "-buildmode=pie"} {
+ args := []string{
+ gotool,
+ "build",
+ "-o",
+ bin,
+ arg,
+ fname,
+ }
+ if out, err := exec.Command(args[0], args[1:]...).CombinedOutput(); err != nil {
+ t.Logf("%s", out)
+ t.Errorf("%v failed: %v", args, err)
+ continue
+ }
+
+ if out, err := exec.Command(bin).CombinedOutput(); err != nil {
+ t.Logf("%s", out)
+ t.Errorf("test program built with %v failed: %v", args, err)
+ }
+ }
+}
diff --git a/sdks/go/pkg/beam/doc_test.go b/sdks/go/pkg/beam/doc_test.go
index 645926f..92a2b03 100644
--- a/sdks/go/pkg/beam/doc_test.go
+++ b/sdks/go/pkg/beam/doc_test.go
@@ -128,9 +128,9 @@
a := textio.Read(s, "...some file path...") // PCollection<string>
beam.Seq(s, a,
- strconv.Atoi, // string to int
+ strconv.Atoi, // string to int
func(i int) float64 { return float64(i) }, // int to float64
- math.Signbit, // float64 to bool
+ math.Signbit, // float64 to bool
) // PCollection<bool>
}
diff --git a/sdks/go/pkg/beam/internal/errors/errors.go b/sdks/go/pkg/beam/internal/errors/errors.go
index cd3269c..dde5d81 100644
--- a/sdks/go/pkg/beam/internal/errors/errors.go
+++ b/sdks/go/pkg/beam/internal/errors/errors.go
@@ -115,7 +115,7 @@
if be, ok := e.(*beamError); ok {
return be.top
}
- return e.Error()
+ return ""
}
// beamError represents one or more details about an error. They are usually
@@ -127,10 +127,10 @@
//
// * If no cause is present it indicates that this instance is the original
// error, and the message is assumed to be present.
-// * If both message and context are present, the context describes this error
-// not the next error.
-// * top is always assumed to be present since it is propogated up from the
-// original error if not explicitly set.
+// * If both message and context are present, the context describes this error,
+// not the cause of this error.
+// * top is always propogated up from the cause. If it's empty that means that
+// it was never set on any error in the sequence.
type beamError struct {
cause error // The error being wrapped. If nil then this is the first error.
context string // Adds additional context to this error and any following.
@@ -153,9 +153,8 @@
return builder.String()
}
-// printRecursive outputs the contexts and messages of beamErrors recursively
-// while ignoring the top-level error. This avoids calling Error recursively on
-// beamErrors since that would repeatedly print top-level messages.
+// printRecursive is a helper function for outputting the contexts and messages
+// of a sequence of beamErrors.
func (e *beamError) printRecursive(builder *strings.Builder) {
wraps := e.cause != nil
@@ -165,7 +164,7 @@
if e.msg != "" {
builder.WriteString(e.msg)
if wraps {
- builder.WriteString("\nCaused by:\n")
+ builder.WriteString("\n\tcaused by:\n")
}
}
diff --git a/sdks/go/pkg/beam/internal/errors/errors_test.go b/sdks/go/pkg/beam/internal/errors/errors_test.go
index 5df0da0..614609f 100644
--- a/sdks/go/pkg/beam/internal/errors/errors_test.go
+++ b/sdks/go/pkg/beam/internal/errors/errors_test.go
@@ -116,10 +116,10 @@
}{
{
err: New(base),
- want: base,
+ want: "",
}, {
err: Wrap(WithContext(New(base), ctx1), msg1),
- want: base,
+ want: "",
}, {
err: SetTopLevelMsg(New(base), top1),
want: top1,
diff --git a/sdks/go/pkg/beam/model/PROTOBUF.md b/sdks/go/pkg/beam/model/PROTOBUF.md
index 80d13e3..6a03268 100644
--- a/sdks/go/pkg/beam/model/PROTOBUF.md
+++ b/sdks/go/pkg/beam/model/PROTOBUF.md
@@ -26,5 +26,10 @@
* A proper Go development setup per `BUILD.md` (variables GOPATH and GOBIN set properly)
* `go get -u github.com/golang/protobuf/protoc-gen-go`
+> **Note:** Newer releases of the protobuf compiler may be incompatible with the
+> protobuf version in Beam. For guaranteed compatibility, use the latest release
+> available from the date of the golang/protobuf release used by Beam. (Currently
+> v3.5.2)
+
If all this setup is complete, simply run `go generate` in the current directory
-(`pkg/beam/model`).
\ No newline at end of file
+(`pkg/beam/model`).
diff --git a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
index fb1e7d3..55de402 100644
--- a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
+++ b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
@@ -74,97 +74,7 @@
return proto.EnumName(LogEntry_Severity_Enum_name, int32(x))
}
func (LogEntry_Severity_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{28, 1, 0}
-}
-
-// A representation of an input or output definition on a primitive transform.
-// Stable
-type Target struct {
- // (Required) The id of the PrimitiveTransform which is the target.
- PrimitiveTransformReference string `protobuf:"bytes,1,opt,name=primitive_transform_reference,json=primitiveTransformReference,proto3" json:"primitive_transform_reference,omitempty"`
- // (Required) The local name of an input or output defined on the primitive
- // transform.
- Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Target) Reset() { *m = Target{} }
-func (m *Target) String() string { return proto.CompactTextString(m) }
-func (*Target) ProtoMessage() {}
-func (*Target) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{0}
-}
-func (m *Target) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Target.Unmarshal(m, b)
-}
-func (m *Target) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Target.Marshal(b, m, deterministic)
-}
-func (dst *Target) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Target.Merge(dst, src)
-}
-func (m *Target) XXX_Size() int {
- return xxx_messageInfo_Target.Size(m)
-}
-func (m *Target) XXX_DiscardUnknown() {
- xxx_messageInfo_Target.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Target proto.InternalMessageInfo
-
-func (m *Target) GetPrimitiveTransformReference() string {
- if m != nil {
- return m.PrimitiveTransformReference
- }
- return ""
-}
-
-func (m *Target) GetName() string {
- if m != nil {
- return m.Name
- }
- return ""
-}
-
-// A repeated list of target definitions.
-type Target_List struct {
- Target []*Target `protobuf:"bytes,1,rep,name=target,proto3" json:"target,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Target_List) Reset() { *m = Target_List{} }
-func (m *Target_List) String() string { return proto.CompactTextString(m) }
-func (*Target_List) ProtoMessage() {}
-func (*Target_List) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{0, 0}
-}
-func (m *Target_List) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Target_List.Unmarshal(m, b)
-}
-func (m *Target_List) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Target_List.Marshal(b, m, deterministic)
-}
-func (dst *Target_List) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Target_List.Merge(dst, src)
-}
-func (m *Target_List) XXX_Size() int {
- return xxx_messageInfo_Target_List.Size(m)
-}
-func (m *Target_List) XXX_DiscardUnknown() {
- xxx_messageInfo_Target_List.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Target_List proto.InternalMessageInfo
-
-func (m *Target_List) GetTarget() []*Target {
- if m != nil {
- return m.Target
- }
- return nil
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{27, 1, 0}
}
// A descriptor for connecting to a remote port using the Beam Fn Data API.
@@ -187,7 +97,7 @@
func (m *RemoteGrpcPort) String() string { return proto.CompactTextString(m) }
func (*RemoteGrpcPort) ProtoMessage() {}
func (*RemoteGrpcPort) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{0}
}
func (m *RemoteGrpcPort) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RemoteGrpcPort.Unmarshal(m, b)
@@ -247,7 +157,7 @@
func (m *InstructionRequest) String() string { return proto.CompactTextString(m) }
func (*InstructionRequest) ProtoMessage() {}
func (*InstructionRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{2}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{1}
}
func (m *InstructionRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_InstructionRequest.Unmarshal(m, b)
@@ -503,7 +413,7 @@
func (m *InstructionResponse) String() string { return proto.CompactTextString(m) }
func (*InstructionResponse) ProtoMessage() {}
func (*InstructionResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{3}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{2}
}
func (m *InstructionResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_InstructionResponse.Unmarshal(m, b)
@@ -751,7 +661,7 @@
func (m *RegisterRequest) String() string { return proto.CompactTextString(m) }
func (*RegisterRequest) ProtoMessage() {}
func (*RegisterRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{4}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{3}
}
func (m *RegisterRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RegisterRequest.Unmarshal(m, b)
@@ -789,7 +699,7 @@
func (m *RegisterResponse) String() string { return proto.CompactTextString(m) }
func (*RegisterResponse) ProtoMessage() {}
func (*RegisterResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{5}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{4}
}
func (m *RegisterResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RegisterResponse.Unmarshal(m, b)
@@ -837,7 +747,7 @@
func (m *ProcessBundleDescriptor) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleDescriptor) ProtoMessage() {}
func (*ProcessBundleDescriptor) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{6}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{5}
}
func (m *ProcessBundleDescriptor) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleDescriptor.Unmarshal(m, b)
@@ -911,8 +821,8 @@
// https://docs.google.com/document/d/1tUDb45sStdR8u7-jBkGdw3OGFK7aa2-V7eo86zYSE_4/edit#heading=h.9g3g5weg2u9
// for further details.
type BundleApplication struct {
- // (Required) The primitive transform to which to pass the element
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ // (Required) The transform to which to pass the element
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Required) Name of the transform's input to which to pass the element.
InputId string `protobuf:"bytes,2,opt,name=input_id,json=inputId,proto3" json:"input_id,omitempty"`
// (Required) The encoded element to pass to the transform.
@@ -921,24 +831,21 @@
// value represents a lower bound on the timestamps of elements that
// are produced by this PTransform into each of its output PCollections
// when invoked with this application.
+ //
+ // If there is no watermark reported from RestrictionTracker, the runner will
+ // use MIN_TIMESTAMP by default.
OutputWatermarks map[string]*timestamp.Timestamp `protobuf:"bytes,4,rep,name=output_watermarks,json=outputWatermarks,proto3" json:"output_watermarks,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
- // (Required) An estimate for the amount outstanding work related to
- // this application.
- Backlog *BundleApplication_Backlog `protobuf:"bytes,5,opt,name=backlog,proto3" json:"backlog,omitempty"`
// (Required) Whether this application potentially produces an unbounded
// amount of data. Note that this should only be set to BOUNDED if and
// only if the application is known to produce a finite amount of output.
- //
- // Note that this is different from the backlog as the backlog represents
- // how much work there is currently outstanding.
- IsBounded pipeline_v1.IsBounded_Enum `protobuf:"varint,6,opt,name=is_bounded,json=isBounded,proto3,enum=org.apache.beam.model.pipeline.v1.IsBounded_Enum" json:"is_bounded,omitempty"`
+ IsBounded pipeline_v1.IsBounded_Enum `protobuf:"varint,5,opt,name=is_bounded,json=isBounded,proto3,enum=org.apache.beam.model.pipeline.v1.IsBounded_Enum" json:"is_bounded,omitempty"`
// Contains additional monitoring information related to this application.
//
// Each application is able to report information that some runners
- // will use consume when providing a UI or for making scaling and performance
+ // will use when providing a UI or for making scaling and performance
// decisions. See https://s.apache.org/beam-bundles-backlog-splitting for
// details about what types of signals may be useful to report.
- MonitoringInfos []*pipeline_v1.MonitoringInfo `protobuf:"bytes,7,rep,name=monitoring_infos,json=monitoringInfos,proto3" json:"monitoring_infos,omitempty"`
+ MonitoringInfos []*pipeline_v1.MonitoringInfo `protobuf:"bytes,6,rep,name=monitoring_infos,json=monitoringInfos,proto3" json:"monitoring_infos,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -948,7 +855,7 @@
func (m *BundleApplication) String() string { return proto.CompactTextString(m) }
func (*BundleApplication) ProtoMessage() {}
func (*BundleApplication) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{7}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{6}
}
func (m *BundleApplication) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_BundleApplication.Unmarshal(m, b)
@@ -968,9 +875,9 @@
var xxx_messageInfo_BundleApplication proto.InternalMessageInfo
-func (m *BundleApplication) GetPtransformId() string {
+func (m *BundleApplication) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -996,13 +903,6 @@
return nil
}
-func (m *BundleApplication) GetBacklog() *BundleApplication_Backlog {
- if m != nil {
- return m.Backlog
- }
- return nil
-}
-
func (m *BundleApplication) GetIsBounded() pipeline_v1.IsBounded_Enum {
if m != nil {
return m.IsBounded
@@ -1017,166 +917,6 @@
return nil
}
-// Represents an estimate for the amount of currently outstanding work.
-type BundleApplication_Backlog struct {
- // This informs Runners on how to aggregate the backlog
- // being reported across multiple active bundles. Backlogs
- // are aggregated using the set of partitions.
- //
- // For example SplittableDoFn's which consume elements from:
- // * a globally shared resource such as a Pubsub queue should set this
- // to “”.
- // * a shared partitioned resource should use the partition identifier.
- // * a uniquely partitioned resource such as a file range should set this
- // to
- // file name + start offset.
- Partition []byte `protobuf:"bytes,1,opt,name=partition,proto3" json:"partition,omitempty"`
- // The estimate for the backlog.
- //
- // Types that are valid to be assigned to Value:
- // *BundleApplication_Backlog_Bytes
- // *BundleApplication_Backlog_IsUnknown
- Value isBundleApplication_Backlog_Value `protobuf_oneof:"value"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *BundleApplication_Backlog) Reset() { *m = BundleApplication_Backlog{} }
-func (m *BundleApplication_Backlog) String() string { return proto.CompactTextString(m) }
-func (*BundleApplication_Backlog) ProtoMessage() {}
-func (*BundleApplication_Backlog) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{7, 1}
-}
-func (m *BundleApplication_Backlog) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_BundleApplication_Backlog.Unmarshal(m, b)
-}
-func (m *BundleApplication_Backlog) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_BundleApplication_Backlog.Marshal(b, m, deterministic)
-}
-func (dst *BundleApplication_Backlog) XXX_Merge(src proto.Message) {
- xxx_messageInfo_BundleApplication_Backlog.Merge(dst, src)
-}
-func (m *BundleApplication_Backlog) XXX_Size() int {
- return xxx_messageInfo_BundleApplication_Backlog.Size(m)
-}
-func (m *BundleApplication_Backlog) XXX_DiscardUnknown() {
- xxx_messageInfo_BundleApplication_Backlog.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_BundleApplication_Backlog proto.InternalMessageInfo
-
-type isBundleApplication_Backlog_Value interface {
- isBundleApplication_Backlog_Value()
-}
-
-type BundleApplication_Backlog_Bytes struct {
- Bytes []byte `protobuf:"bytes,1000,opt,name=bytes,proto3,oneof"`
-}
-type BundleApplication_Backlog_IsUnknown struct {
- IsUnknown bool `protobuf:"varint,1001,opt,name=is_unknown,json=isUnknown,proto3,oneof"`
-}
-
-func (*BundleApplication_Backlog_Bytes) isBundleApplication_Backlog_Value() {}
-func (*BundleApplication_Backlog_IsUnknown) isBundleApplication_Backlog_Value() {}
-
-func (m *BundleApplication_Backlog) GetValue() isBundleApplication_Backlog_Value {
- if m != nil {
- return m.Value
- }
- return nil
-}
-
-func (m *BundleApplication_Backlog) GetPartition() []byte {
- if m != nil {
- return m.Partition
- }
- return nil
-}
-
-func (m *BundleApplication_Backlog) GetBytes() []byte {
- if x, ok := m.GetValue().(*BundleApplication_Backlog_Bytes); ok {
- return x.Bytes
- }
- return nil
-}
-
-func (m *BundleApplication_Backlog) GetIsUnknown() bool {
- if x, ok := m.GetValue().(*BundleApplication_Backlog_IsUnknown); ok {
- return x.IsUnknown
- }
- return false
-}
-
-// XXX_OneofFuncs is for the internal use of the proto package.
-func (*BundleApplication_Backlog) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
- return _BundleApplication_Backlog_OneofMarshaler, _BundleApplication_Backlog_OneofUnmarshaler, _BundleApplication_Backlog_OneofSizer, []interface{}{
- (*BundleApplication_Backlog_Bytes)(nil),
- (*BundleApplication_Backlog_IsUnknown)(nil),
- }
-}
-
-func _BundleApplication_Backlog_OneofMarshaler(msg proto.Message, b *proto.Buffer) error {
- m := msg.(*BundleApplication_Backlog)
- // value
- switch x := m.Value.(type) {
- case *BundleApplication_Backlog_Bytes:
- b.EncodeVarint(1000<<3 | proto.WireBytes)
- b.EncodeRawBytes(x.Bytes)
- case *BundleApplication_Backlog_IsUnknown:
- t := uint64(0)
- if x.IsUnknown {
- t = 1
- }
- b.EncodeVarint(1001<<3 | proto.WireVarint)
- b.EncodeVarint(t)
- case nil:
- default:
- return fmt.Errorf("BundleApplication_Backlog.Value has unexpected type %T", x)
- }
- return nil
-}
-
-func _BundleApplication_Backlog_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
- m := msg.(*BundleApplication_Backlog)
- switch tag {
- case 1000: // value.bytes
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- x, err := b.DecodeRawBytes(true)
- m.Value = &BundleApplication_Backlog_Bytes{x}
- return true, err
- case 1001: // value.is_unknown
- if wire != proto.WireVarint {
- return true, proto.ErrInternalBadWireType
- }
- x, err := b.DecodeVarint()
- m.Value = &BundleApplication_Backlog_IsUnknown{x != 0}
- return true, err
- default:
- return false, nil
- }
-}
-
-func _BundleApplication_Backlog_OneofSizer(msg proto.Message) (n int) {
- m := msg.(*BundleApplication_Backlog)
- // value
- switch x := m.Value.(type) {
- case *BundleApplication_Backlog_Bytes:
- n += 2 // tag and wire
- n += proto.SizeVarint(uint64(len(x.Bytes)))
- n += len(x.Bytes)
- case *BundleApplication_Backlog_IsUnknown:
- n += 2 // tag and wire
- n += 1
- case nil:
- default:
- panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
- }
- return n
-}
-
// An Application should be scheduled for execution after a delay.
type DelayedBundleApplication struct {
// Recommended time at which the application should be scheduled to execute
@@ -1193,7 +933,7 @@
func (m *DelayedBundleApplication) String() string { return proto.CompactTextString(m) }
func (*DelayedBundleApplication) ProtoMessage() {}
func (*DelayedBundleApplication) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{8}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{7}
}
func (m *DelayedBundleApplication) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DelayedBundleApplication.Unmarshal(m, b)
@@ -1232,20 +972,20 @@
type ProcessBundleRequest struct {
// (Required) A reference to the process bundle descriptor that must be
// instantiated and executed by the SDK harness.
- ProcessBundleDescriptorReference string `protobuf:"bytes,1,opt,name=process_bundle_descriptor_reference,json=processBundleDescriptorReference,proto3" json:"process_bundle_descriptor_reference,omitempty"`
+ ProcessBundleDescriptorId string `protobuf:"bytes,1,opt,name=process_bundle_descriptor_id,json=processBundleDescriptorId,proto3" json:"process_bundle_descriptor_id,omitempty"`
// (Optional) A list of cache tokens that can be used by an SDK to reuse
// cached data returned by the State API across multiple bundles.
- CacheTokens [][]byte `protobuf:"bytes,2,rep,name=cache_tokens,json=cacheTokens,proto3" json:"cache_tokens,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
+ CacheTokens []*ProcessBundleRequest_CacheToken `protobuf:"bytes,2,rep,name=cache_tokens,json=cacheTokens,proto3" json:"cache_tokens,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
}
func (m *ProcessBundleRequest) Reset() { *m = ProcessBundleRequest{} }
func (m *ProcessBundleRequest) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleRequest) ProtoMessage() {}
func (*ProcessBundleRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{9}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{8}
}
func (m *ProcessBundleRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleRequest.Unmarshal(m, b)
@@ -1265,20 +1005,251 @@
var xxx_messageInfo_ProcessBundleRequest proto.InternalMessageInfo
-func (m *ProcessBundleRequest) GetProcessBundleDescriptorReference() string {
+func (m *ProcessBundleRequest) GetProcessBundleDescriptorId() string {
if m != nil {
- return m.ProcessBundleDescriptorReference
+ return m.ProcessBundleDescriptorId
}
return ""
}
-func (m *ProcessBundleRequest) GetCacheTokens() [][]byte {
+func (m *ProcessBundleRequest) GetCacheTokens() []*ProcessBundleRequest_CacheToken {
if m != nil {
return m.CacheTokens
}
return nil
}
+// A cache token which can be used by an SDK to check for the validity
+// of cached elements which have a cache token associated.
+type ProcessBundleRequest_CacheToken struct {
+ // The scope of a cache token.
+ //
+ // Types that are valid to be assigned to Type:
+ // *ProcessBundleRequest_CacheToken_UserState_
+ // *ProcessBundleRequest_CacheToken_SideInput_
+ Type isProcessBundleRequest_CacheToken_Type `protobuf_oneof:"type"`
+ // The cache token identifier which should be globally unique.
+ Token []byte `protobuf:"bytes,10,opt,name=token,proto3" json:"token,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *ProcessBundleRequest_CacheToken) Reset() { *m = ProcessBundleRequest_CacheToken{} }
+func (m *ProcessBundleRequest_CacheToken) String() string { return proto.CompactTextString(m) }
+func (*ProcessBundleRequest_CacheToken) ProtoMessage() {}
+func (*ProcessBundleRequest_CacheToken) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{8, 0}
+}
+func (m *ProcessBundleRequest_CacheToken) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken.Unmarshal(m, b)
+}
+func (m *ProcessBundleRequest_CacheToken) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken.Marshal(b, m, deterministic)
+}
+func (dst *ProcessBundleRequest_CacheToken) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_ProcessBundleRequest_CacheToken.Merge(dst, src)
+}
+func (m *ProcessBundleRequest_CacheToken) XXX_Size() int {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken.Size(m)
+}
+func (m *ProcessBundleRequest_CacheToken) XXX_DiscardUnknown() {
+ xxx_messageInfo_ProcessBundleRequest_CacheToken.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ProcessBundleRequest_CacheToken proto.InternalMessageInfo
+
+type isProcessBundleRequest_CacheToken_Type interface {
+ isProcessBundleRequest_CacheToken_Type()
+}
+
+type ProcessBundleRequest_CacheToken_UserState_ struct {
+ UserState *ProcessBundleRequest_CacheToken_UserState `protobuf:"bytes,1,opt,name=user_state,json=userState,proto3,oneof"`
+}
+type ProcessBundleRequest_CacheToken_SideInput_ struct {
+ SideInput *ProcessBundleRequest_CacheToken_SideInput `protobuf:"bytes,2,opt,name=side_input,json=sideInput,proto3,oneof"`
+}
+
+func (*ProcessBundleRequest_CacheToken_UserState_) isProcessBundleRequest_CacheToken_Type() {}
+func (*ProcessBundleRequest_CacheToken_SideInput_) isProcessBundleRequest_CacheToken_Type() {}
+
+func (m *ProcessBundleRequest_CacheToken) GetType() isProcessBundleRequest_CacheToken_Type {
+ if m != nil {
+ return m.Type
+ }
+ return nil
+}
+
+func (m *ProcessBundleRequest_CacheToken) GetUserState() *ProcessBundleRequest_CacheToken_UserState {
+ if x, ok := m.GetType().(*ProcessBundleRequest_CacheToken_UserState_); ok {
+ return x.UserState
+ }
+ return nil
+}
+
+func (m *ProcessBundleRequest_CacheToken) GetSideInput() *ProcessBundleRequest_CacheToken_SideInput {
+ if x, ok := m.GetType().(*ProcessBundleRequest_CacheToken_SideInput_); ok {
+ return x.SideInput
+ }
+ return nil
+}
+
+func (m *ProcessBundleRequest_CacheToken) GetToken() []byte {
+ if m != nil {
+ return m.Token
+ }
+ return nil
+}
+
+// XXX_OneofFuncs is for the internal use of the proto package.
+func (*ProcessBundleRequest_CacheToken) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
+ return _ProcessBundleRequest_CacheToken_OneofMarshaler, _ProcessBundleRequest_CacheToken_OneofUnmarshaler, _ProcessBundleRequest_CacheToken_OneofSizer, []interface{}{
+ (*ProcessBundleRequest_CacheToken_UserState_)(nil),
+ (*ProcessBundleRequest_CacheToken_SideInput_)(nil),
+ }
+}
+
+func _ProcessBundleRequest_CacheToken_OneofMarshaler(msg proto.Message, b *proto.Buffer) error {
+ m := msg.(*ProcessBundleRequest_CacheToken)
+ // type
+ switch x := m.Type.(type) {
+ case *ProcessBundleRequest_CacheToken_UserState_:
+ b.EncodeVarint(1<<3 | proto.WireBytes)
+ if err := b.EncodeMessage(x.UserState); err != nil {
+ return err
+ }
+ case *ProcessBundleRequest_CacheToken_SideInput_:
+ b.EncodeVarint(2<<3 | proto.WireBytes)
+ if err := b.EncodeMessage(x.SideInput); err != nil {
+ return err
+ }
+ case nil:
+ default:
+ return fmt.Errorf("ProcessBundleRequest_CacheToken.Type has unexpected type %T", x)
+ }
+ return nil
+}
+
+func _ProcessBundleRequest_CacheToken_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
+ m := msg.(*ProcessBundleRequest_CacheToken)
+ switch tag {
+ case 1: // type.user_state
+ if wire != proto.WireBytes {
+ return true, proto.ErrInternalBadWireType
+ }
+ msg := new(ProcessBundleRequest_CacheToken_UserState)
+ err := b.DecodeMessage(msg)
+ m.Type = &ProcessBundleRequest_CacheToken_UserState_{msg}
+ return true, err
+ case 2: // type.side_input
+ if wire != proto.WireBytes {
+ return true, proto.ErrInternalBadWireType
+ }
+ msg := new(ProcessBundleRequest_CacheToken_SideInput)
+ err := b.DecodeMessage(msg)
+ m.Type = &ProcessBundleRequest_CacheToken_SideInput_{msg}
+ return true, err
+ default:
+ return false, nil
+ }
+}
+
+func _ProcessBundleRequest_CacheToken_OneofSizer(msg proto.Message) (n int) {
+ m := msg.(*ProcessBundleRequest_CacheToken)
+ // type
+ switch x := m.Type.(type) {
+ case *ProcessBundleRequest_CacheToken_UserState_:
+ s := proto.Size(x.UserState)
+ n += 1 // tag and wire
+ n += proto.SizeVarint(uint64(s))
+ n += s
+ case *ProcessBundleRequest_CacheToken_SideInput_:
+ s := proto.Size(x.SideInput)
+ n += 1 // tag and wire
+ n += proto.SizeVarint(uint64(s))
+ n += s
+ case nil:
+ default:
+ panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
+ }
+ return n
+}
+
+// A flag to indicate a cache token is valid for user state.
+type ProcessBundleRequest_CacheToken_UserState struct {
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *ProcessBundleRequest_CacheToken_UserState) Reset() {
+ *m = ProcessBundleRequest_CacheToken_UserState{}
+}
+func (m *ProcessBundleRequest_CacheToken_UserState) String() string { return proto.CompactTextString(m) }
+func (*ProcessBundleRequest_CacheToken_UserState) ProtoMessage() {}
+func (*ProcessBundleRequest_CacheToken_UserState) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{8, 0, 0}
+}
+func (m *ProcessBundleRequest_CacheToken_UserState) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState.Unmarshal(m, b)
+}
+func (m *ProcessBundleRequest_CacheToken_UserState) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState.Marshal(b, m, deterministic)
+}
+func (dst *ProcessBundleRequest_CacheToken_UserState) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState.Merge(dst, src)
+}
+func (m *ProcessBundleRequest_CacheToken_UserState) XXX_Size() int {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState.Size(m)
+}
+func (m *ProcessBundleRequest_CacheToken_UserState) XXX_DiscardUnknown() {
+ xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ProcessBundleRequest_CacheToken_UserState proto.InternalMessageInfo
+
+// A flag to indicate a cache token is valid for a side input.
+type ProcessBundleRequest_CacheToken_SideInput struct {
+ // The id of a side input.
+ SideInput string `protobuf:"bytes,1,opt,name=side_input,json=sideInput,proto3" json:"side_input,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *ProcessBundleRequest_CacheToken_SideInput) Reset() {
+ *m = ProcessBundleRequest_CacheToken_SideInput{}
+}
+func (m *ProcessBundleRequest_CacheToken_SideInput) String() string { return proto.CompactTextString(m) }
+func (*ProcessBundleRequest_CacheToken_SideInput) ProtoMessage() {}
+func (*ProcessBundleRequest_CacheToken_SideInput) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{8, 0, 1}
+}
+func (m *ProcessBundleRequest_CacheToken_SideInput) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput.Unmarshal(m, b)
+}
+func (m *ProcessBundleRequest_CacheToken_SideInput) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput.Marshal(b, m, deterministic)
+}
+func (dst *ProcessBundleRequest_CacheToken_SideInput) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput.Merge(dst, src)
+}
+func (m *ProcessBundleRequest_CacheToken_SideInput) XXX_Size() int {
+ return xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput.Size(m)
+}
+func (m *ProcessBundleRequest_CacheToken_SideInput) XXX_DiscardUnknown() {
+ xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ProcessBundleRequest_CacheToken_SideInput proto.InternalMessageInfo
+
+func (m *ProcessBundleRequest_CacheToken_SideInput) GetSideInput() string {
+ if m != nil {
+ return m.SideInput
+ }
+ return ""
+}
+
type ProcessBundleResponse struct {
// (Optional) If metrics reporting is supported by the SDK, this represents
// the final metrics to record for this bundle.
@@ -1306,7 +1277,7 @@
func (m *ProcessBundleResponse) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleResponse) ProtoMessage() {}
func (*ProcessBundleResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{10}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{9}
}
func (m *ProcessBundleResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleResponse.Unmarshal(m, b)
@@ -1360,7 +1331,7 @@
type ProcessBundleProgressRequest struct {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -1370,7 +1341,7 @@
func (m *ProcessBundleProgressRequest) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleProgressRequest) ProtoMessage() {}
func (*ProcessBundleProgressRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{11}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{10}
}
func (m *ProcessBundleProgressRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleProgressRequest.Unmarshal(m, b)
@@ -1390,9 +1361,9 @@
var xxx_messageInfo_ProcessBundleProgressRequest proto.InternalMessageInfo
-func (m *ProcessBundleProgressRequest) GetInstructionReference() string {
+func (m *ProcessBundleProgressRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
@@ -1409,7 +1380,7 @@
func (m *Metrics) String() string { return proto.CompactTextString(m) }
func (*Metrics) ProtoMessage() {}
func (*Metrics) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11}
}
func (m *Metrics) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics.Unmarshal(m, b)
@@ -1440,7 +1411,7 @@
// These metrics are split into processed and active element groups for
// progress reporting purposes. This allows a Runner to see what is measured,
// what is estimated and what can be extrapolated to be able to accurately
-// estimate the backlog of remaining work.
+// estimate the amount of remaining work.
type Metrics_PTransform struct {
// (Required): Metrics for processed elements.
ProcessedElements *Metrics_PTransform_ProcessedElements `protobuf:"bytes,1,opt,name=processed_elements,json=processedElements,proto3" json:"processed_elements,omitempty"`
@@ -1462,7 +1433,7 @@
func (m *Metrics_PTransform) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform) ProtoMessage() {}
func (*Metrics_PTransform) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 0}
}
func (m *Metrics_PTransform) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform.Unmarshal(m, b)
@@ -1532,7 +1503,7 @@
func (m *Metrics_PTransform_Measured) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform_Measured) ProtoMessage() {}
func (*Metrics_PTransform_Measured) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 0, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 0, 0}
}
func (m *Metrics_PTransform_Measured) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform_Measured.Unmarshal(m, b)
@@ -1586,7 +1557,7 @@
func (m *Metrics_PTransform_ProcessedElements) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform_ProcessedElements) ProtoMessage() {}
func (*Metrics_PTransform_ProcessedElements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 0, 1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 0, 1}
}
func (m *Metrics_PTransform_ProcessedElements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform_ProcessedElements.Unmarshal(m, b)
@@ -1640,7 +1611,7 @@
func (m *Metrics_PTransform_ActiveElements) String() string { return proto.CompactTextString(m) }
func (*Metrics_PTransform_ActiveElements) ProtoMessage() {}
func (*Metrics_PTransform_ActiveElements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 0, 2}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 0, 2}
}
func (m *Metrics_PTransform_ActiveElements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_PTransform_ActiveElements.Unmarshal(m, b)
@@ -1701,7 +1672,7 @@
func (m *Metrics_User) String() string { return proto.CompactTextString(m) }
func (*Metrics_User) ProtoMessage() {}
func (*Metrics_User) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 1}
}
func (m *Metrics_User) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User.Unmarshal(m, b)
@@ -1882,7 +1853,7 @@
func (m *Metrics_User_MetricName) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_MetricName) ProtoMessage() {}
func (*Metrics_User_MetricName) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 1, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 1, 0}
}
func (m *Metrics_User_MetricName) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_MetricName.Unmarshal(m, b)
@@ -1928,7 +1899,7 @@
func (m *Metrics_User_CounterData) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_CounterData) ProtoMessage() {}
func (*Metrics_User_CounterData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 1, 1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 1, 1}
}
func (m *Metrics_User_CounterData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_CounterData.Unmarshal(m, b)
@@ -1970,7 +1941,7 @@
func (m *Metrics_User_DistributionData) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_DistributionData) ProtoMessage() {}
func (*Metrics_User_DistributionData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 1, 2}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 1, 2}
}
func (m *Metrics_User_DistributionData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_DistributionData.Unmarshal(m, b)
@@ -2031,7 +2002,7 @@
func (m *Metrics_User_GaugeData) String() string { return proto.CompactTextString(m) }
func (*Metrics_User_GaugeData) ProtoMessage() {}
func (*Metrics_User_GaugeData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{12, 1, 3}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{11, 1, 3}
}
func (m *Metrics_User_GaugeData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Metrics_User_GaugeData.Unmarshal(m, b)
@@ -2083,7 +2054,7 @@
func (m *ProcessBundleProgressResponse) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleProgressResponse) ProtoMessage() {}
func (*ProcessBundleProgressResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{13}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{12}
}
func (m *ProcessBundleProgressResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleProgressResponse.Unmarshal(m, b)
@@ -2128,19 +2099,7 @@
type ProcessBundleSplitRequest struct {
// (Required) A reference to an active process bundle request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
- // (Required) Specifies that the Runner would like the bundle to split itself
- // such that it performs no more work than the backlog specified for each
- // PTransform. The interpretation of how much work should be processed is up
- // to the PTransform.
- //
- // For example, A backlog of "" tells the SDK to perform as little work as
- // possible, effectively checkpointing when able. The remaining backlog
- // will be relative to the backlog reported during processing.
- //
- // If the backlog is unspecified for a PTransform, the runner would like
- // the SDK to process all data received for that PTransform.
- BacklogRemaining map[string][]byte `protobuf:"bytes,2,rep,name=backlog_remaining,json=backlogRemaining,proto3" json:"backlog_remaining,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
// (Required) Specifies the desired split for each transform.
//
// Currently only splits at GRPC read operations are supported.
@@ -2156,7 +2115,7 @@
func (m *ProcessBundleSplitRequest) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitRequest) ProtoMessage() {}
func (*ProcessBundleSplitRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{14}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{13}
}
func (m *ProcessBundleSplitRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitRequest.Unmarshal(m, b)
@@ -2176,20 +2135,13 @@
var xxx_messageInfo_ProcessBundleSplitRequest proto.InternalMessageInfo
-func (m *ProcessBundleSplitRequest) GetInstructionReference() string {
+func (m *ProcessBundleSplitRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
-func (m *ProcessBundleSplitRequest) GetBacklogRemaining() map[string][]byte {
- if m != nil {
- return m.BacklogRemaining
- }
- return nil
-}
-
func (m *ProcessBundleSplitRequest) GetDesiredSplits() map[string]*ProcessBundleSplitRequest_DesiredSplit {
if m != nil {
return m.DesiredSplits
@@ -2223,7 +2175,7 @@
func (m *ProcessBundleSplitRequest_DesiredSplit) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitRequest_DesiredSplit) ProtoMessage() {}
func (*ProcessBundleSplitRequest_DesiredSplit) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{14, 1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{13, 0}
}
func (m *ProcessBundleSplitRequest_DesiredSplit) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitRequest_DesiredSplit.Unmarshal(m, b)
@@ -2293,7 +2245,7 @@
func (m *ProcessBundleSplitResponse) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitResponse) ProtoMessage() {}
func (*ProcessBundleSplitResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{15}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{14}
}
func (m *ProcessBundleSplitResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitResponse.Unmarshal(m, b)
@@ -2344,17 +2296,15 @@
// as some range in an underlying dataset).
type ProcessBundleSplitResponse_ChannelSplit struct {
// (Required) The grpc read transform reading this channel.
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
- // (Required) Name of the transform's input to which to pass the element.
- InputId string `protobuf:"bytes,2,opt,name=input_id,json=inputId,proto3" json:"input_id,omitempty"`
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// The last element of the input channel that should be entirely considered
// part of the primary, identified by its absolute index in the (ordered)
// channel.
- LastPrimaryElement int32 `protobuf:"varint,3,opt,name=last_primary_element,json=lastPrimaryElement,proto3" json:"last_primary_element,omitempty"`
+ LastPrimaryElement int32 `protobuf:"varint,2,opt,name=last_primary_element,json=lastPrimaryElement,proto3" json:"last_primary_element,omitempty"`
// The first element of the input channel that should be entirely considered
// part of the residual, identified by its absolute index in the (ordered)
// channel.
- FirstResidualElement int32 `protobuf:"varint,4,opt,name=first_residual_element,json=firstResidualElement,proto3" json:"first_residual_element,omitempty"`
+ FirstResidualElement int32 `protobuf:"varint,3,opt,name=first_residual_element,json=firstResidualElement,proto3" json:"first_residual_element,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -2366,7 +2316,7 @@
func (m *ProcessBundleSplitResponse_ChannelSplit) String() string { return proto.CompactTextString(m) }
func (*ProcessBundleSplitResponse_ChannelSplit) ProtoMessage() {}
func (*ProcessBundleSplitResponse_ChannelSplit) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{15, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{14, 0}
}
func (m *ProcessBundleSplitResponse_ChannelSplit) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessBundleSplitResponse_ChannelSplit.Unmarshal(m, b)
@@ -2386,16 +2336,9 @@
var xxx_messageInfo_ProcessBundleSplitResponse_ChannelSplit proto.InternalMessageInfo
-func (m *ProcessBundleSplitResponse_ChannelSplit) GetPtransformId() string {
+func (m *ProcessBundleSplitResponse_ChannelSplit) GetTransformId() string {
if m != nil {
- return m.PtransformId
- }
- return ""
-}
-
-func (m *ProcessBundleSplitResponse_ChannelSplit) GetInputId() string {
- if m != nil {
- return m.InputId
+ return m.TransformId
}
return ""
}
@@ -2417,7 +2360,7 @@
type FinalizeBundleRequest struct {
// (Required) A reference to a completed process bundle request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
@@ -2427,7 +2370,7 @@
func (m *FinalizeBundleRequest) String() string { return proto.CompactTextString(m) }
func (*FinalizeBundleRequest) ProtoMessage() {}
func (*FinalizeBundleRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{16}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{15}
}
func (m *FinalizeBundleRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_FinalizeBundleRequest.Unmarshal(m, b)
@@ -2447,9 +2390,9 @@
var xxx_messageInfo_FinalizeBundleRequest proto.InternalMessageInfo
-func (m *FinalizeBundleRequest) GetInstructionReference() string {
+func (m *FinalizeBundleRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
@@ -2464,7 +2407,7 @@
func (m *FinalizeBundleResponse) String() string { return proto.CompactTextString(m) }
func (*FinalizeBundleResponse) ProtoMessage() {}
func (*FinalizeBundleResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{17}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{16}
}
func (m *FinalizeBundleResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_FinalizeBundleResponse.Unmarshal(m, b)
@@ -2498,7 +2441,7 @@
func (m *Elements) String() string { return proto.CompactTextString(m) }
func (*Elements) ProtoMessage() {}
func (*Elements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{18}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{17}
}
func (m *Elements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Elements.Unmarshal(m, b)
@@ -2526,11 +2469,11 @@
}
// Represents multiple encoded elements in nested context for a given named
-// instruction and target.
+// instruction and transform.
type Elements_Data struct {
// (Required) A reference to an active instruction request with the given
// instruction id.
- InstructionReference string `protobuf:"bytes,1,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,1,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
// (Required) A definition representing a consumer or producer of this data.
// If received by a harness, this represents the consumer within that
// harness that should consume these bytes. If sent by a harness, this
@@ -2538,15 +2481,14 @@
//
// Note that a single element may span multiple Data messages.
//
- // Note that a sending/receiving pair should share the same target
- // identifier.
- Target *Target `protobuf:"bytes,2,opt,name=target,proto3" json:"target,omitempty"`
+ // Note that a sending/receiving pair should share the same identifier.
+ TransformId string `protobuf:"bytes,2,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Optional) Represents a part of a logical byte stream. Elements within
// the logical byte stream are encoded in the nested context and
// concatenated together.
//
// An empty data block represents the end of stream for the given
- // instruction and target.
+ // instruction and transform.
Data []byte `protobuf:"bytes,3,opt,name=data,proto3" json:"data,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
@@ -2557,7 +2499,7 @@
func (m *Elements_Data) String() string { return proto.CompactTextString(m) }
func (*Elements_Data) ProtoMessage() {}
func (*Elements_Data) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{18, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{17, 0}
}
func (m *Elements_Data) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Elements_Data.Unmarshal(m, b)
@@ -2577,18 +2519,18 @@
var xxx_messageInfo_Elements_Data proto.InternalMessageInfo
-func (m *Elements_Data) GetInstructionReference() string {
+func (m *Elements_Data) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
-func (m *Elements_Data) GetTarget() *Target {
+func (m *Elements_Data) GetTransformId() string {
if m != nil {
- return m.Target
+ return m.TransformId
}
- return nil
+ return ""
}
func (m *Elements_Data) GetData() []byte {
@@ -2606,7 +2548,7 @@
// (Required) The associated instruction id of the work that is currently
// being processed. This allows for the runner to associate any modifications
// to state to be committed with the appropriate work execution.
- InstructionReference string `protobuf:"bytes,2,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
+ InstructionId string `protobuf:"bytes,2,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
// (Required) The state key this request is for.
StateKey *StateKey `protobuf:"bytes,3,opt,name=state_key,json=stateKey,proto3" json:"state_key,omitempty"`
// (Required) The action to take on this request.
@@ -2625,7 +2567,7 @@
func (m *StateRequest) String() string { return proto.CompactTextString(m) }
func (*StateRequest) ProtoMessage() {}
func (*StateRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{19}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{18}
}
func (m *StateRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateRequest.Unmarshal(m, b)
@@ -2677,9 +2619,9 @@
return ""
}
-func (m *StateRequest) GetInstructionReference() string {
+func (m *StateRequest) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
@@ -2814,9 +2756,6 @@
// A human readable string representing the reason as to why the request
// failed.
Error string `protobuf:"bytes,2,opt,name=error,proto3" json:"error,omitempty"`
- // (Optional) If this is specified, then the result of this state request
- // can be cached using the supplied token.
- CacheToken []byte `protobuf:"bytes,3,opt,name=cache_token,json=cacheToken,proto3" json:"cache_token,omitempty"`
// A corresponding response matching the request will be populated.
//
// Types that are valid to be assigned to Response:
@@ -2833,7 +2772,7 @@
func (m *StateResponse) String() string { return proto.CompactTextString(m) }
func (*StateResponse) ProtoMessage() {}
func (*StateResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{20}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{19}
}
func (m *StateResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateResponse.Unmarshal(m, b)
@@ -2892,13 +2831,6 @@
return ""
}
-func (m *StateResponse) GetCacheToken() []byte {
- if m != nil {
- return m.CacheToken
- }
- return nil
-}
-
func (m *StateResponse) GetGet() *StateGetResponse {
if x, ok := m.GetResponse().(*StateResponse_Get); ok {
return x.Get
@@ -3030,7 +2962,7 @@
func (m *StateKey) String() string { return proto.CompactTextString(m) }
func (*StateKey) ProtoMessage() {}
func (*StateKey) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{21}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{20}
}
func (m *StateKey) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey.Unmarshal(m, b)
@@ -3192,6 +3124,11 @@
type StateKey_Runner struct {
// (Required) Opaque information supplied by the runner. Used to support
// remote references.
+ // https://s.apache.org/beam-fn-api-send-and-receive-data
+ //
+ // Used by state backed iterable. And in this use case, request type can
+ // only be of type get. Details see:
+ // https://s.apache.org/beam-fn-api-state-backed-iterables
Key []byte `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
@@ -3202,7 +3139,7 @@
func (m *StateKey_Runner) String() string { return proto.CompactTextString(m) }
func (*StateKey_Runner) ProtoMessage() {}
func (*StateKey_Runner) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{21, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{20, 0}
}
func (m *StateKey_Runner) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey_Runner.Unmarshal(m, b)
@@ -3231,7 +3168,7 @@
type StateKey_MultimapSideInput struct {
// (Required) The id of the PTransform containing a side input.
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Required) The id of the side input.
SideInputId string `protobuf:"bytes,2,opt,name=side_input_id,json=sideInputId,proto3" json:"side_input_id,omitempty"`
// (Required) The window (after mapping the currently executing elements
@@ -3248,7 +3185,7 @@
func (m *StateKey_MultimapSideInput) String() string { return proto.CompactTextString(m) }
func (*StateKey_MultimapSideInput) ProtoMessage() {}
func (*StateKey_MultimapSideInput) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{21, 1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{20, 1}
}
func (m *StateKey_MultimapSideInput) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey_MultimapSideInput.Unmarshal(m, b)
@@ -3268,9 +3205,9 @@
var xxx_messageInfo_StateKey_MultimapSideInput proto.InternalMessageInfo
-func (m *StateKey_MultimapSideInput) GetPtransformId() string {
+func (m *StateKey_MultimapSideInput) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -3298,7 +3235,7 @@
type StateKey_BagUserState struct {
// (Required) The id of the PTransform containing user state.
- PtransformId string `protobuf:"bytes,1,opt,name=ptransform_id,json=ptransformId,proto3" json:"ptransform_id,omitempty"`
+ TransformId string `protobuf:"bytes,1,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Required) The id of the user state.
UserStateId string `protobuf:"bytes,2,opt,name=user_state_id,json=userStateId,proto3" json:"user_state_id,omitempty"`
// (Required) The window encoded in a nested context.
@@ -3315,7 +3252,7 @@
func (m *StateKey_BagUserState) String() string { return proto.CompactTextString(m) }
func (*StateKey_BagUserState) ProtoMessage() {}
func (*StateKey_BagUserState) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{21, 2}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{20, 2}
}
func (m *StateKey_BagUserState) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateKey_BagUserState.Unmarshal(m, b)
@@ -3335,9 +3272,9 @@
var xxx_messageInfo_StateKey_BagUserState proto.InternalMessageInfo
-func (m *StateKey_BagUserState) GetPtransformId() string {
+func (m *StateKey_BagUserState) GetTransformId() string {
if m != nil {
- return m.PtransformId
+ return m.TransformId
}
return ""
}
@@ -3380,7 +3317,7 @@
func (m *StateGetRequest) String() string { return proto.CompactTextString(m) }
func (*StateGetRequest) ProtoMessage() {}
func (*StateGetRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{22}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{21}
}
func (m *StateGetRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateGetRequest.Unmarshal(m, b)
@@ -3427,7 +3364,7 @@
func (m *StateGetResponse) String() string { return proto.CompactTextString(m) }
func (*StateGetResponse) ProtoMessage() {}
func (*StateGetResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{23}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{22}
}
func (m *StateGetResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateGetResponse.Unmarshal(m, b)
@@ -3476,7 +3413,7 @@
func (m *StateAppendRequest) String() string { return proto.CompactTextString(m) }
func (*StateAppendRequest) ProtoMessage() {}
func (*StateAppendRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{24}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{23}
}
func (m *StateAppendRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateAppendRequest.Unmarshal(m, b)
@@ -3514,7 +3451,7 @@
func (m *StateAppendResponse) String() string { return proto.CompactTextString(m) }
func (*StateAppendResponse) ProtoMessage() {}
func (*StateAppendResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{25}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{24}
}
func (m *StateAppendResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateAppendResponse.Unmarshal(m, b)
@@ -3545,7 +3482,7 @@
func (m *StateClearRequest) String() string { return proto.CompactTextString(m) }
func (*StateClearRequest) ProtoMessage() {}
func (*StateClearRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{26}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{25}
}
func (m *StateClearRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateClearRequest.Unmarshal(m, b)
@@ -3576,7 +3513,7 @@
func (m *StateClearResponse) String() string { return proto.CompactTextString(m) }
func (*StateClearResponse) ProtoMessage() {}
func (*StateClearResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{27}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{26}
}
func (m *StateClearResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateClearResponse.Unmarshal(m, b)
@@ -3609,10 +3546,10 @@
Trace string `protobuf:"bytes,4,opt,name=trace,proto3" json:"trace,omitempty"`
// (Optional) A reference to the instruction this log statement is associated
// with.
- InstructionReference string `protobuf:"bytes,5,opt,name=instruction_reference,json=instructionReference,proto3" json:"instruction_reference,omitempty"`
- // (Optional) A reference to the primitive transform this log statement is
+ InstructionId string `protobuf:"bytes,5,opt,name=instruction_id,json=instructionId,proto3" json:"instruction_id,omitempty"`
+ // (Optional) A reference to the transform this log statement is
// associated with.
- PrimitiveTransformReference string `protobuf:"bytes,6,opt,name=primitive_transform_reference,json=primitiveTransformReference,proto3" json:"primitive_transform_reference,omitempty"`
+ TransformId string `protobuf:"bytes,6,opt,name=transform_id,json=transformId,proto3" json:"transform_id,omitempty"`
// (Optional) Human-readable name of the function or method being invoked,
// with optional context such as the class or package name. The format can
// vary by language. For example:
@@ -3632,7 +3569,7 @@
func (m *LogEntry) String() string { return proto.CompactTextString(m) }
func (*LogEntry) ProtoMessage() {}
func (*LogEntry) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{28}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{27}
}
func (m *LogEntry) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogEntry.Unmarshal(m, b)
@@ -3680,16 +3617,16 @@
return ""
}
-func (m *LogEntry) GetInstructionReference() string {
+func (m *LogEntry) GetInstructionId() string {
if m != nil {
- return m.InstructionReference
+ return m.InstructionId
}
return ""
}
-func (m *LogEntry) GetPrimitiveTransformReference() string {
+func (m *LogEntry) GetTransformId() string {
if m != nil {
- return m.PrimitiveTransformReference
+ return m.TransformId
}
return ""
}
@@ -3722,7 +3659,7 @@
func (m *LogEntry_List) String() string { return proto.CompactTextString(m) }
func (*LogEntry_List) ProtoMessage() {}
func (*LogEntry_List) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{28, 0}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{27, 0}
}
func (m *LogEntry_List) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogEntry_List.Unmarshal(m, b)
@@ -3772,7 +3709,7 @@
func (m *LogEntry_Severity) String() string { return proto.CompactTextString(m) }
func (*LogEntry_Severity) ProtoMessage() {}
func (*LogEntry_Severity) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{28, 1}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{27, 1}
}
func (m *LogEntry_Severity) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogEntry_Severity.Unmarshal(m, b)
@@ -3802,7 +3739,7 @@
func (m *LogControl) String() string { return proto.CompactTextString(m) }
func (*LogControl) ProtoMessage() {}
func (*LogControl) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{29}
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{28}
}
func (m *LogControl) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_LogControl.Unmarshal(m, b)
@@ -3822,7 +3759,7 @@
var xxx_messageInfo_LogControl proto.InternalMessageInfo
-type NotifyRunnerAvailableRequest struct {
+type StartWorkerRequest struct {
WorkerId string `protobuf:"bytes,1,opt,name=worker_id,json=workerId,proto3" json:"worker_id,omitempty"`
ControlEndpoint *pipeline_v1.ApiServiceDescriptor `protobuf:"bytes,2,opt,name=control_endpoint,json=controlEndpoint,proto3" json:"control_endpoint,omitempty"`
LoggingEndpoint *pipeline_v1.ApiServiceDescriptor `protobuf:"bytes,3,opt,name=logging_endpoint,json=loggingEndpoint,proto3" json:"logging_endpoint,omitempty"`
@@ -3834,104 +3771,180 @@
XXX_sizecache int32 `json:"-"`
}
-func (m *NotifyRunnerAvailableRequest) Reset() { *m = NotifyRunnerAvailableRequest{} }
-func (m *NotifyRunnerAvailableRequest) String() string { return proto.CompactTextString(m) }
-func (*NotifyRunnerAvailableRequest) ProtoMessage() {}
-func (*NotifyRunnerAvailableRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{30}
+func (m *StartWorkerRequest) Reset() { *m = StartWorkerRequest{} }
+func (m *StartWorkerRequest) String() string { return proto.CompactTextString(m) }
+func (*StartWorkerRequest) ProtoMessage() {}
+func (*StartWorkerRequest) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{29}
}
-func (m *NotifyRunnerAvailableRequest) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_NotifyRunnerAvailableRequest.Unmarshal(m, b)
+func (m *StartWorkerRequest) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_StartWorkerRequest.Unmarshal(m, b)
}
-func (m *NotifyRunnerAvailableRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_NotifyRunnerAvailableRequest.Marshal(b, m, deterministic)
+func (m *StartWorkerRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_StartWorkerRequest.Marshal(b, m, deterministic)
}
-func (dst *NotifyRunnerAvailableRequest) XXX_Merge(src proto.Message) {
- xxx_messageInfo_NotifyRunnerAvailableRequest.Merge(dst, src)
+func (dst *StartWorkerRequest) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_StartWorkerRequest.Merge(dst, src)
}
-func (m *NotifyRunnerAvailableRequest) XXX_Size() int {
- return xxx_messageInfo_NotifyRunnerAvailableRequest.Size(m)
+func (m *StartWorkerRequest) XXX_Size() int {
+ return xxx_messageInfo_StartWorkerRequest.Size(m)
}
-func (m *NotifyRunnerAvailableRequest) XXX_DiscardUnknown() {
- xxx_messageInfo_NotifyRunnerAvailableRequest.DiscardUnknown(m)
+func (m *StartWorkerRequest) XXX_DiscardUnknown() {
+ xxx_messageInfo_StartWorkerRequest.DiscardUnknown(m)
}
-var xxx_messageInfo_NotifyRunnerAvailableRequest proto.InternalMessageInfo
+var xxx_messageInfo_StartWorkerRequest proto.InternalMessageInfo
-func (m *NotifyRunnerAvailableRequest) GetWorkerId() string {
+func (m *StartWorkerRequest) GetWorkerId() string {
if m != nil {
return m.WorkerId
}
return ""
}
-func (m *NotifyRunnerAvailableRequest) GetControlEndpoint() *pipeline_v1.ApiServiceDescriptor {
+func (m *StartWorkerRequest) GetControlEndpoint() *pipeline_v1.ApiServiceDescriptor {
if m != nil {
return m.ControlEndpoint
}
return nil
}
-func (m *NotifyRunnerAvailableRequest) GetLoggingEndpoint() *pipeline_v1.ApiServiceDescriptor {
+func (m *StartWorkerRequest) GetLoggingEndpoint() *pipeline_v1.ApiServiceDescriptor {
if m != nil {
return m.LoggingEndpoint
}
return nil
}
-func (m *NotifyRunnerAvailableRequest) GetArtifactEndpoint() *pipeline_v1.ApiServiceDescriptor {
+func (m *StartWorkerRequest) GetArtifactEndpoint() *pipeline_v1.ApiServiceDescriptor {
if m != nil {
return m.ArtifactEndpoint
}
return nil
}
-func (m *NotifyRunnerAvailableRequest) GetProvisionEndpoint() *pipeline_v1.ApiServiceDescriptor {
+func (m *StartWorkerRequest) GetProvisionEndpoint() *pipeline_v1.ApiServiceDescriptor {
if m != nil {
return m.ProvisionEndpoint
}
return nil
}
-func (m *NotifyRunnerAvailableRequest) GetParams() map[string]string {
+func (m *StartWorkerRequest) GetParams() map[string]string {
if m != nil {
return m.Params
}
return nil
}
-type NotifyRunnerAvailableResponse struct {
+type StartWorkerResponse struct {
Error string `protobuf:"bytes,1,opt,name=error,proto3" json:"error,omitempty"`
XXX_NoUnkeyedLiteral struct{} `json:"-"`
XXX_unrecognized []byte `json:"-"`
XXX_sizecache int32 `json:"-"`
}
-func (m *NotifyRunnerAvailableResponse) Reset() { *m = NotifyRunnerAvailableResponse{} }
-func (m *NotifyRunnerAvailableResponse) String() string { return proto.CompactTextString(m) }
-func (*NotifyRunnerAvailableResponse) ProtoMessage() {}
-func (*NotifyRunnerAvailableResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_fn_api_447e04abaef5392a, []int{31}
+func (m *StartWorkerResponse) Reset() { *m = StartWorkerResponse{} }
+func (m *StartWorkerResponse) String() string { return proto.CompactTextString(m) }
+func (*StartWorkerResponse) ProtoMessage() {}
+func (*StartWorkerResponse) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{30}
}
-func (m *NotifyRunnerAvailableResponse) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_NotifyRunnerAvailableResponse.Unmarshal(m, b)
+func (m *StartWorkerResponse) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_StartWorkerResponse.Unmarshal(m, b)
}
-func (m *NotifyRunnerAvailableResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_NotifyRunnerAvailableResponse.Marshal(b, m, deterministic)
+func (m *StartWorkerResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_StartWorkerResponse.Marshal(b, m, deterministic)
}
-func (dst *NotifyRunnerAvailableResponse) XXX_Merge(src proto.Message) {
- xxx_messageInfo_NotifyRunnerAvailableResponse.Merge(dst, src)
+func (dst *StartWorkerResponse) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_StartWorkerResponse.Merge(dst, src)
}
-func (m *NotifyRunnerAvailableResponse) XXX_Size() int {
- return xxx_messageInfo_NotifyRunnerAvailableResponse.Size(m)
+func (m *StartWorkerResponse) XXX_Size() int {
+ return xxx_messageInfo_StartWorkerResponse.Size(m)
}
-func (m *NotifyRunnerAvailableResponse) XXX_DiscardUnknown() {
- xxx_messageInfo_NotifyRunnerAvailableResponse.DiscardUnknown(m)
+func (m *StartWorkerResponse) XXX_DiscardUnknown() {
+ xxx_messageInfo_StartWorkerResponse.DiscardUnknown(m)
}
-var xxx_messageInfo_NotifyRunnerAvailableResponse proto.InternalMessageInfo
+var xxx_messageInfo_StartWorkerResponse proto.InternalMessageInfo
-func (m *NotifyRunnerAvailableResponse) GetError() string {
+func (m *StartWorkerResponse) GetError() string {
+ if m != nil {
+ return m.Error
+ }
+ return ""
+}
+
+type StopWorkerRequest struct {
+ WorkerId string `protobuf:"bytes,1,opt,name=worker_id,json=workerId,proto3" json:"worker_id,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *StopWorkerRequest) Reset() { *m = StopWorkerRequest{} }
+func (m *StopWorkerRequest) String() string { return proto.CompactTextString(m) }
+func (*StopWorkerRequest) ProtoMessage() {}
+func (*StopWorkerRequest) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{31}
+}
+func (m *StopWorkerRequest) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_StopWorkerRequest.Unmarshal(m, b)
+}
+func (m *StopWorkerRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_StopWorkerRequest.Marshal(b, m, deterministic)
+}
+func (dst *StopWorkerRequest) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_StopWorkerRequest.Merge(dst, src)
+}
+func (m *StopWorkerRequest) XXX_Size() int {
+ return xxx_messageInfo_StopWorkerRequest.Size(m)
+}
+func (m *StopWorkerRequest) XXX_DiscardUnknown() {
+ xxx_messageInfo_StopWorkerRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_StopWorkerRequest proto.InternalMessageInfo
+
+func (m *StopWorkerRequest) GetWorkerId() string {
+ if m != nil {
+ return m.WorkerId
+ }
+ return ""
+}
+
+type StopWorkerResponse struct {
+ Error string `protobuf:"bytes,1,opt,name=error,proto3" json:"error,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *StopWorkerResponse) Reset() { *m = StopWorkerResponse{} }
+func (m *StopWorkerResponse) String() string { return proto.CompactTextString(m) }
+func (*StopWorkerResponse) ProtoMessage() {}
+func (*StopWorkerResponse) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_fn_api_d24d1635dfa071c8, []int{32}
+}
+func (m *StopWorkerResponse) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_StopWorkerResponse.Unmarshal(m, b)
+}
+func (m *StopWorkerResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_StopWorkerResponse.Marshal(b, m, deterministic)
+}
+func (dst *StopWorkerResponse) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_StopWorkerResponse.Merge(dst, src)
+}
+func (m *StopWorkerResponse) XXX_Size() int {
+ return xxx_messageInfo_StopWorkerResponse.Size(m)
+}
+func (m *StopWorkerResponse) XXX_DiscardUnknown() {
+ xxx_messageInfo_StopWorkerResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_StopWorkerResponse proto.InternalMessageInfo
+
+func (m *StopWorkerResponse) GetError() string {
if m != nil {
return m.Error
}
@@ -3939,8 +3952,6 @@
}
func init() {
- proto.RegisterType((*Target)(nil), "org.apache.beam.model.fn_execution.v1.Target")
- proto.RegisterType((*Target_List)(nil), "org.apache.beam.model.fn_execution.v1.Target.List")
proto.RegisterType((*RemoteGrpcPort)(nil), "org.apache.beam.model.fn_execution.v1.RemoteGrpcPort")
proto.RegisterType((*InstructionRequest)(nil), "org.apache.beam.model.fn_execution.v1.InstructionRequest")
proto.RegisterType((*InstructionResponse)(nil), "org.apache.beam.model.fn_execution.v1.InstructionResponse")
@@ -3954,9 +3965,11 @@
proto.RegisterMapType((map[string]*pipeline_v1.WindowingStrategy)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleDescriptor.WindowingStrategiesEntry")
proto.RegisterType((*BundleApplication)(nil), "org.apache.beam.model.fn_execution.v1.BundleApplication")
proto.RegisterMapType((map[string]*timestamp.Timestamp)(nil), "org.apache.beam.model.fn_execution.v1.BundleApplication.OutputWatermarksEntry")
- proto.RegisterType((*BundleApplication_Backlog)(nil), "org.apache.beam.model.fn_execution.v1.BundleApplication.Backlog")
proto.RegisterType((*DelayedBundleApplication)(nil), "org.apache.beam.model.fn_execution.v1.DelayedBundleApplication")
proto.RegisterType((*ProcessBundleRequest)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleRequest")
+ proto.RegisterType((*ProcessBundleRequest_CacheToken)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleRequest.CacheToken")
+ proto.RegisterType((*ProcessBundleRequest_CacheToken_UserState)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleRequest.CacheToken.UserState")
+ proto.RegisterType((*ProcessBundleRequest_CacheToken_SideInput)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleRequest.CacheToken.SideInput")
proto.RegisterType((*ProcessBundleResponse)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleResponse")
proto.RegisterType((*ProcessBundleProgressRequest)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleProgressRequest")
proto.RegisterType((*Metrics)(nil), "org.apache.beam.model.fn_execution.v1.Metrics")
@@ -3976,7 +3989,6 @@
proto.RegisterType((*Metrics_User_GaugeData)(nil), "org.apache.beam.model.fn_execution.v1.Metrics.User.GaugeData")
proto.RegisterType((*ProcessBundleProgressResponse)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleProgressResponse")
proto.RegisterType((*ProcessBundleSplitRequest)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest")
- proto.RegisterMapType((map[string][]byte)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest.BacklogRemainingEntry")
proto.RegisterMapType((map[string]*ProcessBundleSplitRequest_DesiredSplit)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest.DesiredSplitsEntry")
proto.RegisterType((*ProcessBundleSplitRequest_DesiredSplit)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitRequest.DesiredSplit")
proto.RegisterType((*ProcessBundleSplitResponse)(nil), "org.apache.beam.model.fn_execution.v1.ProcessBundleSplitResponse")
@@ -4001,9 +4013,11 @@
proto.RegisterType((*LogEntry_List)(nil), "org.apache.beam.model.fn_execution.v1.LogEntry.List")
proto.RegisterType((*LogEntry_Severity)(nil), "org.apache.beam.model.fn_execution.v1.LogEntry.Severity")
proto.RegisterType((*LogControl)(nil), "org.apache.beam.model.fn_execution.v1.LogControl")
- proto.RegisterType((*NotifyRunnerAvailableRequest)(nil), "org.apache.beam.model.fn_execution.v1.NotifyRunnerAvailableRequest")
- proto.RegisterMapType((map[string]string)(nil), "org.apache.beam.model.fn_execution.v1.NotifyRunnerAvailableRequest.ParamsEntry")
- proto.RegisterType((*NotifyRunnerAvailableResponse)(nil), "org.apache.beam.model.fn_execution.v1.NotifyRunnerAvailableResponse")
+ proto.RegisterType((*StartWorkerRequest)(nil), "org.apache.beam.model.fn_execution.v1.StartWorkerRequest")
+ proto.RegisterMapType((map[string]string)(nil), "org.apache.beam.model.fn_execution.v1.StartWorkerRequest.ParamsEntry")
+ proto.RegisterType((*StartWorkerResponse)(nil), "org.apache.beam.model.fn_execution.v1.StartWorkerResponse")
+ proto.RegisterType((*StopWorkerRequest)(nil), "org.apache.beam.model.fn_execution.v1.StopWorkerRequest")
+ proto.RegisterType((*StopWorkerResponse)(nil), "org.apache.beam.model.fn_execution.v1.StopWorkerResponse")
proto.RegisterEnum("org.apache.beam.model.fn_execution.v1.LogEntry_Severity_Enum", LogEntry_Severity_Enum_name, LogEntry_Severity_Enum_value)
}
@@ -4415,7 +4429,10 @@
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream.
type BeamFnExternalWorkerPoolClient interface {
- NotifyRunnerAvailable(ctx context.Context, in *NotifyRunnerAvailableRequest, opts ...grpc.CallOption) (*NotifyRunnerAvailableResponse, error)
+ // Start the SDK worker with the given ID.
+ StartWorker(ctx context.Context, in *StartWorkerRequest, opts ...grpc.CallOption) (*StartWorkerResponse, error)
+ // Stop the SDK worker.
+ StopWorker(ctx context.Context, in *StopWorkerRequest, opts ...grpc.CallOption) (*StopWorkerResponse, error)
}
type beamFnExternalWorkerPoolClient struct {
@@ -4426,9 +4443,18 @@
return &beamFnExternalWorkerPoolClient{cc}
}
-func (c *beamFnExternalWorkerPoolClient) NotifyRunnerAvailable(ctx context.Context, in *NotifyRunnerAvailableRequest, opts ...grpc.CallOption) (*NotifyRunnerAvailableResponse, error) {
- out := new(NotifyRunnerAvailableResponse)
- err := c.cc.Invoke(ctx, "/org.apache.beam.model.fn_execution.v1.BeamFnExternalWorkerPool/NotifyRunnerAvailable", in, out, opts...)
+func (c *beamFnExternalWorkerPoolClient) StartWorker(ctx context.Context, in *StartWorkerRequest, opts ...grpc.CallOption) (*StartWorkerResponse, error) {
+ out := new(StartWorkerResponse)
+ err := c.cc.Invoke(ctx, "/org.apache.beam.model.fn_execution.v1.BeamFnExternalWorkerPool/StartWorker", in, out, opts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
+func (c *beamFnExternalWorkerPoolClient) StopWorker(ctx context.Context, in *StopWorkerRequest, opts ...grpc.CallOption) (*StopWorkerResponse, error) {
+ out := new(StopWorkerResponse)
+ err := c.cc.Invoke(ctx, "/org.apache.beam.model.fn_execution.v1.BeamFnExternalWorkerPool/StopWorker", in, out, opts...)
if err != nil {
return nil, err
}
@@ -4437,27 +4463,48 @@
// BeamFnExternalWorkerPoolServer is the server API for BeamFnExternalWorkerPool service.
type BeamFnExternalWorkerPoolServer interface {
- NotifyRunnerAvailable(context.Context, *NotifyRunnerAvailableRequest) (*NotifyRunnerAvailableResponse, error)
+ // Start the SDK worker with the given ID.
+ StartWorker(context.Context, *StartWorkerRequest) (*StartWorkerResponse, error)
+ // Stop the SDK worker.
+ StopWorker(context.Context, *StopWorkerRequest) (*StopWorkerResponse, error)
}
func RegisterBeamFnExternalWorkerPoolServer(s *grpc.Server, srv BeamFnExternalWorkerPoolServer) {
s.RegisterService(&_BeamFnExternalWorkerPool_serviceDesc, srv)
}
-func _BeamFnExternalWorkerPool_NotifyRunnerAvailable_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
- in := new(NotifyRunnerAvailableRequest)
+func _BeamFnExternalWorkerPool_StartWorker_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(StartWorkerRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
- return srv.(BeamFnExternalWorkerPoolServer).NotifyRunnerAvailable(ctx, in)
+ return srv.(BeamFnExternalWorkerPoolServer).StartWorker(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
- FullMethod: "/org.apache.beam.model.fn_execution.v1.BeamFnExternalWorkerPool/NotifyRunnerAvailable",
+ FullMethod: "/org.apache.beam.model.fn_execution.v1.BeamFnExternalWorkerPool/StartWorker",
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
- return srv.(BeamFnExternalWorkerPoolServer).NotifyRunnerAvailable(ctx, req.(*NotifyRunnerAvailableRequest))
+ return srv.(BeamFnExternalWorkerPoolServer).StartWorker(ctx, req.(*StartWorkerRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
+func _BeamFnExternalWorkerPool_StopWorker_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(StopWorkerRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(BeamFnExternalWorkerPoolServer).StopWorker(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: "/org.apache.beam.model.fn_execution.v1.BeamFnExternalWorkerPool/StopWorker",
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(BeamFnExternalWorkerPoolServer).StopWorker(ctx, req.(*StopWorkerRequest))
}
return interceptor(ctx, in, info, handler)
}
@@ -4467,220 +4514,217 @@
HandlerType: (*BeamFnExternalWorkerPoolServer)(nil),
Methods: []grpc.MethodDesc{
{
- MethodName: "NotifyRunnerAvailable",
- Handler: _BeamFnExternalWorkerPool_NotifyRunnerAvailable_Handler,
+ MethodName: "StartWorker",
+ Handler: _BeamFnExternalWorkerPool_StartWorker_Handler,
+ },
+ {
+ MethodName: "StopWorker",
+ Handler: _BeamFnExternalWorkerPool_StopWorker_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "beam_fn_api.proto",
}
-func init() { proto.RegisterFile("beam_fn_api.proto", fileDescriptor_beam_fn_api_447e04abaef5392a) }
+func init() { proto.RegisterFile("beam_fn_api.proto", fileDescriptor_beam_fn_api_d24d1635dfa071c8) }
-var fileDescriptor_beam_fn_api_447e04abaef5392a = []byte{
- // 3250 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5a, 0x4b, 0x6c, 0x24, 0x47,
- 0x19, 0x76, 0x7b, 0x66, 0xfc, 0xf8, 0x3d, 0xb6, 0x67, 0xca, 0xf6, 0xee, 0x6c, 0x67, 0x43, 0x36,
- 0x13, 0x22, 0xf9, 0x92, 0xd9, 0x27, 0xc9, 0x6e, 0x48, 0x36, 0xb1, 0xc7, 0xb3, 0xeb, 0x49, 0xbc,
- 0xde, 0xa1, 0xed, 0xcd, 0xc2, 0x06, 0xd2, 0x2a, 0x4f, 0xd7, 0xcc, 0x96, 0xb6, 0xa7, 0xbb, 0x53,
- 0xdd, 0x63, 0xaf, 0x97, 0x08, 0x04, 0x48, 0xe1, 0x21, 0x50, 0x0e, 0x48, 0x28, 0xe2, 0xc6, 0x43,
- 0x1c, 0xe1, 0xc6, 0x1d, 0x89, 0x23, 0x12, 0xe2, 0x0a, 0x39, 0x82, 0x14, 0x08, 0x1c, 0xb9, 0xa3,
- 0x7a, 0xf4, 0x63, 0x5e, 0xde, 0x79, 0x38, 0xdc, 0xba, 0xaa, 0xfa, 0xff, 0xbe, 0xbf, 0xff, 0xfa,
- 0xab, 0xfe, 0xff, 0xef, 0x2a, 0xc8, 0x1f, 0x10, 0xdc, 0x32, 0x1b, 0x8e, 0x89, 0x3d, 0x5a, 0xf2,
- 0x98, 0x1b, 0xb8, 0xe8, 0x45, 0x97, 0x35, 0x4b, 0xd8, 0xc3, 0xf5, 0x87, 0xa4, 0xc4, 0x47, 0x4b,
- 0x2d, 0xd7, 0x22, 0x76, 0xa9, 0xe1, 0x98, 0xe4, 0x31, 0xa9, 0xb7, 0x03, 0xea, 0x3a, 0xa5, 0xc3,
- 0xcb, 0xfa, 0x9a, 0x90, 0x64, 0x6d, 0xc7, 0x21, 0x2c, 0x96, 0xd6, 0x97, 0x89, 0x63, 0x79, 0x2e,
- 0x75, 0x02, 0x5f, 0x75, 0x5c, 0x68, 0xba, 0x6e, 0xd3, 0x26, 0x17, 0x45, 0xeb, 0xa0, 0xdd, 0xb8,
- 0x68, 0x11, 0xbf, 0xce, 0xa8, 0x17, 0xb8, 0x4c, 0xbd, 0xf1, 0x5c, 0xf7, 0x1b, 0x01, 0x6d, 0x11,
- 0x3f, 0xc0, 0x2d, 0x4f, 0xbd, 0xf0, 0x85, 0xee, 0x17, 0x8e, 0x18, 0xf6, 0x3c, 0xc2, 0x42, 0x8a,
- 0xc5, 0x16, 0x09, 0x18, 0xad, 0xab, 0x66, 0xf1, 0x77, 0x1a, 0xcc, 0xec, 0x63, 0xd6, 0x24, 0x01,
- 0xda, 0x84, 0x67, 0x3d, 0x46, 0x5b, 0x34, 0xa0, 0x87, 0xc4, 0x0c, 0x18, 0x76, 0xfc, 0x86, 0xcb,
- 0x5a, 0x26, 0x23, 0x0d, 0xc2, 0x88, 0x53, 0x27, 0x05, 0xed, 0x82, 0xb6, 0x3e, 0x6f, 0x3c, 0x13,
- 0xbd, 0xb4, 0x1f, 0xbe, 0x63, 0x84, 0xaf, 0x20, 0x04, 0x69, 0x07, 0xb7, 0x48, 0x61, 0x5a, 0xbc,
- 0x2a, 0x9e, 0xf5, 0x3b, 0x90, 0xde, 0xa1, 0x7e, 0x80, 0x2a, 0x30, 0x13, 0x08, 0xa6, 0x82, 0x76,
- 0x21, 0xb5, 0xbe, 0x70, 0xe5, 0xa5, 0xd2, 0x50, 0xc6, 0x2b, 0x49, 0xf5, 0x0c, 0x25, 0x5c, 0xfc,
- 0xb9, 0x06, 0x4b, 0x06, 0x69, 0xb9, 0x01, 0xb9, 0xcd, 0xbc, 0x7a, 0xcd, 0x65, 0x01, 0x6a, 0xc1,
- 0x19, 0xec, 0x51, 0xd3, 0x27, 0xec, 0x90, 0xd6, 0x89, 0x19, 0x1b, 0x4d, 0xa8, 0xbc, 0x70, 0xe5,
- 0x95, 0x01, 0x4c, 0x1e, 0xf5, 0x88, 0x4d, 0x1d, 0xc2, 0x59, 0x36, 0x3c, 0xba, 0x27, 0xe5, 0xb7,
- 0x22, 0x71, 0x63, 0x15, 0xf7, 0xe9, 0x45, 0xe7, 0x60, 0xae, 0xee, 0x5a, 0x84, 0x99, 0xd4, 0x52,
- 0x1f, 0x3a, 0x2b, 0xda, 0x55, 0xab, 0xf8, 0x8f, 0x34, 0xa0, 0xaa, 0xe3, 0x07, 0xac, 0x5d, 0xe7,
- 0xea, 0x1b, 0xe4, 0xfd, 0x36, 0xf1, 0x03, 0xf4, 0x22, 0x2c, 0xd1, 0xb8, 0x97, 0xcb, 0x49, 0x5b,
- 0x2e, 0x26, 0x7a, 0xab, 0x16, 0xba, 0x07, 0x73, 0x8c, 0x34, 0xa9, 0x1f, 0x10, 0x56, 0xf8, 0x74,
- 0x56, 0xa8, 0xfe, 0xf2, 0x90, 0x46, 0x32, 0x94, 0x9c, 0x62, 0xdc, 0x9e, 0x32, 0x22, 0x28, 0x44,
- 0x60, 0xc9, 0x63, 0x6e, 0x9d, 0xf8, 0xbe, 0x79, 0xd0, 0x76, 0x2c, 0x9b, 0x14, 0xfe, 0x29, 0xc1,
- 0xbf, 0x3c, 0x24, 0x78, 0x4d, 0x4a, 0x6f, 0x0a, 0xe1, 0x98, 0x61, 0xd1, 0x4b, 0xf6, 0xa3, 0x6f,
- 0xc1, 0xd9, 0x4e, 0x1a, 0xd3, 0x63, 0x6e, 0x93, 0x11, 0xdf, 0x2f, 0xfc, 0x4b, 0xf2, 0x95, 0xc7,
- 0xe1, 0xab, 0x29, 0x90, 0x98, 0x77, 0xcd, 0xeb, 0x37, 0x8e, 0xda, 0xb0, 0xda, 0xc5, 0xef, 0x7b,
- 0x36, 0x0d, 0x0a, 0x9f, 0x49, 0xf2, 0x37, 0xc7, 0x21, 0xdf, 0xe3, 0x08, 0x31, 0x33, 0xf2, 0x7a,
- 0x06, 0xd1, 0x43, 0x58, 0x6e, 0x50, 0x07, 0xdb, 0xf4, 0x09, 0x09, 0xcd, 0xfb, 0x6f, 0xc9, 0xf8,
- 0xda, 0x90, 0x8c, 0xb7, 0x94, 0x78, 0xb7, 0x7d, 0x97, 0x1a, 0x1d, 0x03, 0x9b, 0xf3, 0x30, 0xcb,
- 0xe4, 0x60, 0xf1, 0xbb, 0x19, 0x58, 0xe9, 0xf0, 0x33, 0xdf, 0x73, 0x1d, 0x9f, 0x0c, 0xeb, 0x68,
- 0xab, 0x90, 0x21, 0x8c, 0xb9, 0x4c, 0xb9, 0xaf, 0x6c, 0xa0, 0x77, 0x7a, 0xdd, 0xef, 0x95, 0x91,
- 0xdd, 0x4f, 0x2a, 0xd2, 0xe1, 0x7f, 0x8d, 0x41, 0xfe, 0xf7, 0xda, 0x78, 0xfe, 0x17, 0x51, 0x74,
- 0x39, 0xe0, 0xb7, 0x9f, 0xea, 0x80, 0x5b, 0x93, 0x39, 0x60, 0x44, 0x3c, 0xc0, 0x03, 0x0f, 0x4f,
- 0xf6, 0xc0, 0x8d, 0x09, 0x3c, 0x30, 0xa2, 0xee, 0xe7, 0x82, 0x74, 0xa0, 0x0b, 0xbe, 0x3e, 0xa6,
- 0x0b, 0x46, 0x74, 0xdd, 0x3e, 0x08, 0xdc, 0x47, 0xe4, 0x68, 0xf1, 0x27, 0x1a, 0x2c, 0x77, 0xed,
- 0x3b, 0xe8, 0x09, 0x9c, 0xeb, 0x32, 0x41, 0xc7, 0x6e, 0xcc, 0xf7, 0xfd, 0x9b, 0xe3, 0x98, 0x21,
- 0xb1, 0x29, 0x9f, 0xf5, 0xfa, 0x0f, 0x14, 0x11, 0xe4, 0xba, 0xfd, 0xb0, 0xf8, 0x2b, 0x80, 0xb3,
- 0x03, 0x80, 0xd0, 0x12, 0x4c, 0x47, 0x0b, 0x64, 0x9a, 0x5a, 0xc8, 0x01, 0x88, 0xc2, 0x9e, 0x5f,
- 0x98, 0x16, 0xca, 0xee, 0x4e, 0xa6, 0x6c, 0x29, 0x8a, 0x91, 0x7e, 0xc5, 0x09, 0xd8, 0xb1, 0x91,
- 0x60, 0x40, 0x01, 0x64, 0xbd, 0xba, 0x6b, 0xdb, 0x44, 0x2c, 0x4b, 0xbf, 0x90, 0x12, 0x8c, 0xb5,
- 0x09, 0x19, 0x6b, 0x09, 0x48, 0xc9, 0xd9, 0xc1, 0x82, 0x7e, 0xa4, 0xc1, 0xea, 0x11, 0x75, 0x2c,
- 0xf7, 0x88, 0x3a, 0x4d, 0xd3, 0x0f, 0x18, 0x0e, 0x48, 0x93, 0x12, 0xbf, 0x90, 0x16, 0xf4, 0xf7,
- 0x27, 0xa4, 0xbf, 0x1f, 0x42, 0xef, 0x45, 0xc8, 0x52, 0x8b, 0x95, 0xa3, 0xde, 0x11, 0x74, 0x00,
- 0x33, 0x22, 0x74, 0xfa, 0x85, 0x8c, 0x60, 0x7f, 0x6b, 0x42, 0xf6, 0xb2, 0x00, 0x93, 0x84, 0x0a,
- 0x99, 0x9b, 0x99, 0x38, 0x87, 0x94, 0xb9, 0x4e, 0x8b, 0x38, 0x81, 0x5f, 0x98, 0x39, 0x15, 0x33,
- 0x57, 0x12, 0x90, 0xca, 0xcc, 0x49, 0x16, 0xf4, 0x18, 0xce, 0xfb, 0x01, 0x0e, 0x88, 0x39, 0x20,
- 0x33, 0x99, 0x9d, 0x2c, 0x33, 0x39, 0x27, 0xc0, 0xfb, 0x0d, 0xe9, 0x36, 0x2c, 0x77, 0x79, 0x1d,
- 0xca, 0x41, 0xea, 0x11, 0x39, 0x56, 0xae, 0xce, 0x1f, 0x51, 0x19, 0x32, 0x87, 0xd8, 0x6e, 0xcb,
- 0x4c, 0x6d, 0x70, 0x2e, 0x96, 0xd4, 0xa3, 0x16, 0xe7, 0x7b, 0x52, 0xf6, 0xd5, 0xe9, 0xeb, 0x9a,
- 0xee, 0x42, 0xbe, 0xc7, 0xe3, 0xfa, 0xf0, 0x6d, 0x75, 0xf2, 0x95, 0x86, 0xe1, 0x2b, 0x47, 0xb0,
- 0x49, 0xc2, 0x0f, 0xa0, 0x30, 0xc8, 0xc7, 0xfa, 0xf0, 0xbe, 0xd5, 0xc9, 0x7b, 0x6d, 0x08, 0xde,
- 0x6e, 0xf4, 0xe3, 0x24, 0x7b, 0x1d, 0x16, 0x12, 0x3e, 0xd6, 0x87, 0xf0, 0x66, 0x27, 0xe1, 0xfa,
- 0x10, 0x84, 0x02, 0xb0, 0xcb, 0xa6, 0x3d, 0xee, 0x75, 0x3a, 0x36, 0x4d, 0xc0, 0x26, 0x08, 0x8b,
- 0x7f, 0xcc, 0x40, 0x5e, 0x7a, 0xf8, 0x86, 0xe7, 0xd9, 0xb4, 0x8e, 0xb9, 0xd1, 0xd1, 0x0b, 0xb0,
- 0xe8, 0xc5, 0x75, 0x40, 0xb4, 0x55, 0x66, 0xe3, 0xce, 0xaa, 0xc5, 0x93, 0x61, 0xea, 0x78, 0xed,
- 0x20, 0x91, 0x0c, 0x8b, 0x76, 0xd5, 0x42, 0x05, 0x98, 0x25, 0x36, 0xe1, 0x5c, 0x85, 0xd4, 0x05,
- 0x6d, 0x3d, 0x6b, 0x84, 0x4d, 0xf4, 0x4d, 0xc8, 0xbb, 0xed, 0x80, 0x4b, 0x1d, 0xe1, 0x80, 0xb0,
- 0x16, 0x66, 0x8f, 0xc2, 0xfd, 0x67, 0xd8, 0x0d, 0xb7, 0x47, 0xdd, 0xd2, 0x5d, 0x81, 0x78, 0x3f,
- 0x02, 0x94, 0xab, 0x32, 0xe7, 0x76, 0x75, 0xa3, 0x07, 0x30, 0x7b, 0x80, 0xeb, 0x8f, 0x6c, 0xb7,
- 0x59, 0xc8, 0x8c, 0x94, 0x19, 0xf6, 0x52, 0x6e, 0x4a, 0x1c, 0x23, 0x04, 0x44, 0x35, 0x00, 0xea,
- 0x9b, 0x07, 0x6e, 0xdb, 0xb1, 0x88, 0x55, 0x98, 0xb9, 0xa0, 0xad, 0x2f, 0x5d, 0xb9, 0x3c, 0xc4,
- 0xbc, 0x54, 0xfd, 0x4d, 0x29, 0x53, 0xaa, 0x38, 0xed, 0x96, 0x31, 0x4f, 0xc3, 0x36, 0xfa, 0x3a,
- 0xe4, 0x5a, 0xae, 0x43, 0x03, 0x97, 0xf1, 0xed, 0x9a, 0x3a, 0x0d, 0xd7, 0x2f, 0xcc, 0x0a, 0x4b,
- 0x0d, 0x83, 0x7b, 0x27, 0x12, 0xad, 0x3a, 0x0d, 0xd7, 0x58, 0x6e, 0x75, 0xb4, 0x7d, 0xdd, 0x84,
- 0xb5, 0xbe, 0x66, 0xeb, 0xe3, 0x6d, 0x97, 0x3a, 0xbd, 0x4d, 0x2f, 0xc9, 0x42, 0xb3, 0x14, 0x16,
- 0x9a, 0xa5, 0xfd, 0xb0, 0x12, 0x4d, 0xba, 0xf2, 0x23, 0x98, 0x55, 0x46, 0x42, 0xe7, 0x61, 0xde,
- 0xc3, 0x2c, 0xa0, 0xdc, 0x72, 0x02, 0x38, 0x6b, 0xc4, 0x1d, 0xe8, 0x2c, 0x64, 0x0e, 0x8e, 0x03,
- 0xe2, 0xcb, 0xcc, 0x33, 0xbb, 0x3d, 0x65, 0xc8, 0x36, 0xba, 0x20, 0x4c, 0xda, 0x76, 0x1e, 0x39,
- 0xee, 0x91, 0x23, 0x33, 0xc7, 0xb9, 0xed, 0x29, 0x6e, 0xa2, 0x7b, 0xb2, 0x6f, 0x73, 0x56, 0x69,
- 0x56, 0xfc, 0x93, 0x06, 0x85, 0x2d, 0x62, 0xe3, 0x63, 0x62, 0xf5, 0x7a, 0xf3, 0x3e, 0x14, 0x54,
- 0xf6, 0x4c, 0xac, 0x78, 0x5e, 0x4d, 0x5e, 0x3f, 0xab, 0x32, 0xf1, 0xa4, 0x4f, 0x3a, 0x13, 0xc9,
- 0x56, 0x42, 0x51, 0x3e, 0x88, 0x1e, 0xc0, 0x02, 0x8e, 0x49, 0x94, 0x6d, 0xae, 0x8f, 0xeb, 0x50,
- 0x46, 0x12, 0xac, 0xf8, 0x03, 0x0d, 0x56, 0xfb, 0x95, 0x5e, 0xe8, 0x0e, 0xbc, 0x30, 0x30, 0xc9,
- 0xea, 0xa9, 0xd7, 0x2f, 0x0c, 0x48, 0x97, 0xe2, 0xa2, 0xfd, 0x79, 0xc8, 0xd6, 0xb9, 0xaa, 0x66,
- 0xe0, 0x3e, 0x22, 0x8e, 0xcc, 0x7c, 0xb2, 0xc6, 0x82, 0xe8, 0xdb, 0x17, 0x5d, 0xc5, 0x4f, 0xa6,
- 0x61, 0xad, 0x6f, 0x16, 0x8e, 0xb6, 0x61, 0x56, 0xfd, 0x51, 0x50, 0x56, 0x2c, 0x0d, 0xf9, 0xf1,
- 0x77, 0xa4, 0x94, 0x11, 0x8a, 0xf3, 0x32, 0x81, 0x11, 0x9f, 0x5a, 0x6d, 0x6c, 0x9b, 0xcc, 0x75,
- 0x83, 0x30, 0x05, 0x7b, 0x63, 0x48, 0xc0, 0x41, 0x33, 0x6f, 0x2c, 0x86, 0xb0, 0x06, 0x47, 0xed,
- 0xbb, 0xa2, 0x52, 0xa7, 0xb5, 0xa2, 0xd0, 0x55, 0x58, 0xe3, 0xae, 0x42, 0x19, 0xf1, 0x4d, 0x95,
- 0x3b, 0x4b, 0xd7, 0x48, 0x73, 0xc7, 0x35, 0x56, 0xc3, 0xc1, 0x5b, 0x89, 0xb1, 0xe2, 0x1e, 0x9c,
- 0x3f, 0xa9, 0xe6, 0xe5, 0xa0, 0xc9, 0xb2, 0xae, 0x7b, 0x8a, 0x57, 0x69, 0xb2, 0x14, 0x54, 0x63,
- 0xc5, 0x8f, 0x57, 0x60, 0x56, 0x19, 0x19, 0x61, 0x58, 0xf0, 0x12, 0xb9, 0xad, 0x36, 0x92, 0x61,
- 0x15, 0x48, 0xa9, 0x16, 0x74, 0x25, 0xb3, 0x49, 0x4c, 0xfd, 0x93, 0x05, 0x80, 0x38, 0x45, 0x40,
- 0x4f, 0x20, 0xac, 0x54, 0xf8, 0x72, 0x93, 0xfb, 0x7e, 0xe8, 0x22, 0x6f, 0x8f, 0x4a, 0x1c, 0xc1,
- 0x86, 0xe9, 0x18, 0xb1, 0x2a, 0x0a, 0xd2, 0xc8, 0x7b, 0xdd, 0x5d, 0xe8, 0x7d, 0x58, 0xc6, 0x75,
- 0xf1, 0x1b, 0x2b, 0x22, 0x96, 0x0b, 0x73, 0x7b, 0x7c, 0xe2, 0x0d, 0x01, 0x18, 0xb1, 0x2e, 0xe1,
- 0x8e, 0x36, 0xa2, 0x00, 0x89, 0x50, 0x26, 0xdd, 0xa9, 0x3a, 0x3e, 0x5b, 0x77, 0x14, 0x4b, 0x80,
- 0xa3, 0xdb, 0x90, 0x6e, 0xfb, 0x84, 0xa9, 0x78, 0x79, 0x75, 0x44, 0x92, 0x7b, 0x3e, 0x61, 0x86,
- 0x00, 0xd0, 0xff, 0x9e, 0x82, 0xb9, 0x3b, 0x04, 0xfb, 0x6d, 0x46, 0x2c, 0xf4, 0x63, 0x0d, 0x56,
- 0x65, 0x20, 0x57, 0x36, 0x33, 0xeb, 0x6e, 0x5b, 0x4e, 0x19, 0xa7, 0x79, 0x30, 0xfe, 0xb7, 0x84,
- 0x14, 0xa5, 0x2a, 0x87, 0x57, 0x16, 0x2b, 0x0b, 0x70, 0xf9, 0x71, 0x88, 0xf6, 0x0c, 0xa0, 0x8f,
- 0x34, 0x58, 0x53, 0x29, 0x42, 0x97, 0x3e, 0x72, 0x53, 0x78, 0xf7, 0x14, 0xf4, 0x91, 0x91, 0xaf,
- 0x8f, 0x42, 0x2b, 0x6e, 0xef, 0x08, 0x5a, 0x87, 0x5c, 0xe0, 0x06, 0xd8, 0x16, 0x11, 0xc3, 0xf4,
- 0xbd, 0x30, 0xad, 0xd1, 0x8c, 0x25, 0xd1, 0xcf, 0xc3, 0xc1, 0x1e, 0xef, 0xd5, 0x2b, 0x70, 0x76,
- 0xc0, 0xa7, 0xf6, 0x09, 0xab, 0xab, 0xc9, 0xb0, 0x9a, 0x4a, 0x86, 0xce, 0x5b, 0x50, 0x18, 0xa4,
- 0xe1, 0x48, 0x38, 0x3e, 0xe4, 0x7b, 0x56, 0x0d, 0x7a, 0x0f, 0xe6, 0x5a, 0xca, 0x0e, 0x6a, 0x51,
- 0x6e, 0x4e, 0x6e, 0x51, 0x23, 0xc2, 0xd4, 0x3f, 0x4a, 0xc1, 0x52, 0xe7, 0x92, 0xf9, 0xbc, 0x29,
- 0xd1, 0x4b, 0x80, 0x1a, 0x0c, 0x87, 0x3b, 0x64, 0x0b, 0x53, 0x87, 0x3a, 0x4d, 0x61, 0x0e, 0xcd,
- 0xc8, 0x87, 0x23, 0x46, 0x38, 0x80, 0x7e, 0xa1, 0xc1, 0xb9, 0x4e, 0x0f, 0xf3, 0x13, 0x62, 0x72,
- 0x05, 0x93, 0xd3, 0xda, 0x2f, 0x3a, 0x7d, 0xcd, 0x8f, 0xb4, 0x90, 0xfe, 0x76, 0xd6, 0xed, 0x3f,
- 0xaa, 0xbf, 0x05, 0xe7, 0x4f, 0x12, 0x1c, 0xc9, 0x0d, 0x5e, 0x87, 0xe5, 0xa7, 0x27, 0x79, 0x83,
- 0xc5, 0xff, 0x9c, 0x81, 0x34, 0xdf, 0x3b, 0x90, 0x09, 0x0b, 0x32, 0x62, 0x9b, 0xe2, 0x4f, 0xbf,
- 0x9c, 0xc9, 0x9b, 0x63, 0xec, 0x42, 0xaa, 0xb1, 0x8b, 0x5b, 0xc4, 0x80, 0x56, 0xf4, 0x8c, 0x08,
- 0x64, 0xc5, 0x52, 0x27, 0xcc, 0xb4, 0x70, 0x80, 0xc3, 0x9f, 0x85, 0x6f, 0x8c, 0x43, 0x51, 0x96,
- 0x40, 0x5b, 0x38, 0xc0, 0xdb, 0x53, 0xc6, 0x42, 0x3d, 0x6e, 0xa2, 0x00, 0xf2, 0x16, 0xf5, 0x03,
- 0x46, 0x0f, 0x64, 0x22, 0x28, 0xb8, 0x46, 0xfc, 0x4f, 0xd8, 0xc1, 0xb5, 0x95, 0x40, 0x53, 0x84,
- 0x39, 0xab, 0xab, 0x0f, 0x99, 0x00, 0x4d, 0xdc, 0x6e, 0x12, 0x49, 0xf7, 0xd9, 0x68, 0x7f, 0xe9,
- 0x3a, 0xe8, 0x6e, 0x73, 0x18, 0xc5, 0x33, 0xdf, 0x0c, 0x1b, 0xfa, 0x4d, 0x80, 0xd8, 0xae, 0x3c,
- 0xe7, 0xe6, 0xb3, 0xe4, 0x7b, 0xb8, 0x1e, 0x1e, 0xca, 0xc4, 0x1d, 0xd1, 0x69, 0x4d, 0x2a, 0x71,
- 0x5a, 0xf3, 0x02, 0x2f, 0x70, 0x63, 0x2b, 0x45, 0x0e, 0xa1, 0x25, 0x1c, 0x42, 0x7f, 0x0f, 0x72,
- 0xdd, 0x5f, 0xcb, 0xdf, 0x14, 0xe6, 0x0d, 0xdf, 0x14, 0x0d, 0xee, 0x62, 0x7e, 0xbb, 0xa5, 0xdc,
- 0x89, 0x3f, 0xf2, 0x9e, 0x16, 0x75, 0x04, 0x67, 0xca, 0xe0, 0x8f, 0xa2, 0x07, 0x3f, 0x16, 0x09,
- 0x12, 0xef, 0xc1, 0x8f, 0xf5, 0x77, 0x61, 0x3e, 0xfa, 0xbc, 0xfe, 0x2a, 0xa0, 0xeb, 0x30, 0x1f,
- 0x1d, 0x7d, 0x0d, 0x51, 0x92, 0xc4, 0x2f, 0x6f, 0xce, 0x40, 0x9a, 0x1b, 0x5f, 0x3f, 0x86, 0x5c,
- 0x77, 0x46, 0xd3, 0x67, 0x45, 0xdc, 0xed, 0x2c, 0x7b, 0x6e, 0x8c, 0xbd, 0x23, 0x24, 0xeb, 0xed,
- 0xdf, 0x4c, 0xc3, 0xb3, 0x27, 0xfe, 0x63, 0x3e, 0xc5, 0xb4, 0xfa, 0xf3, 0x4d, 0x77, 0xbf, 0x01,
- 0x8b, 0x1e, 0xa3, 0x2d, 0xcc, 0x8e, 0x55, 0xce, 0x2e, 0xb3, 0x92, 0xf1, 0x2b, 0xa0, 0xac, 0x82,
- 0x13, 0xb9, 0x7a, 0xf1, 0xaf, 0x19, 0x38, 0x37, 0xf0, 0x40, 0x66, 0xac, 0xb4, 0x18, 0x7d, 0x4f,
- 0x83, 0xbc, 0x2a, 0xd7, 0x3b, 0xc2, 0x04, 0x57, 0xfb, 0x9d, 0x49, 0xcf, 0x88, 0xa2, 0x3f, 0x02,
- 0x9d, 0x1b, 0x7c, 0xee, 0xa0, 0xab, 0x1b, 0x3d, 0x81, 0x25, 0x8b, 0xf8, 0x94, 0x11, 0x4b, 0x9e,
- 0x11, 0x84, 0x73, 0xb2, 0x37, 0xb1, 0x06, 0x5b, 0x12, 0x56, 0xf4, 0xa9, 0x7c, 0x66, 0xd1, 0x4a,
- 0xf6, 0xe9, 0x65, 0x58, 0xeb, 0xab, 0xe6, 0xd3, 0xe2, 0x41, 0x36, 0x19, 0x0f, 0x7e, 0xab, 0x41,
- 0x36, 0x49, 0x85, 0xae, 0xc0, 0x5a, 0x14, 0x7e, 0xdd, 0x86, 0x32, 0xad, 0x45, 0xe4, 0x19, 0xec,
- 0xb4, 0xb1, 0x12, 0x0e, 0xde, 0x6d, 0x18, 0xe1, 0x10, 0xba, 0x04, 0xab, 0xd8, 0xb6, 0xdd, 0xa3,
- 0xd0, 0x0a, 0xa6, 0x3c, 0x0d, 0x17, 0xb6, 0x48, 0x19, 0x48, 0x8d, 0x09, 0xfc, 0x9a, 0x18, 0x41,
- 0xd7, 0xa1, 0x40, 0xfc, 0x80, 0xb6, 0x30, 0xaf, 0xe2, 0x3b, 0xf2, 0x55, 0x5f, 0x6d, 0x32, 0x67,
- 0xa2, 0xf1, 0x64, 0x12, 0xe6, 0xeb, 0x1f, 0x69, 0x80, 0x7a, 0x6d, 0xd3, 0xe7, 0x9b, 0xeb, 0x9d,
- 0x2b, 0xfe, 0xce, 0xa9, 0xce, 0x48, 0x72, 0x17, 0xf8, 0x69, 0x1a, 0xf4, 0xc1, 0x67, 0x3d, 0xbd,
- 0x4b, 0x4b, 0x3b, 0xcd, 0xa5, 0xf5, 0x7f, 0x2b, 0xb7, 0xdb, 0xb0, 0x54, 0x7f, 0x88, 0x1d, 0x87,
- 0xd8, 0x9d, 0x9e, 0xbe, 0x3b, 0xf1, 0x69, 0x58, 0xa9, 0x2c, 0x71, 0x65, 0xe7, 0x62, 0x3d, 0xd1,
- 0xf2, 0xf5, 0xdf, 0x6b, 0x90, 0x4d, 0x8e, 0x4f, 0xfc, 0x37, 0xf3, 0x12, 0xac, 0xda, 0xd8, 0x0f,
- 0xcc, 0x70, 0x4e, 0x92, 0xbf, 0x36, 0x33, 0x06, 0xe2, 0x63, 0x35, 0x39, 0xa4, 0x5c, 0x0e, 0x5d,
- 0x83, 0x33, 0x0d, 0xca, 0xfc, 0xc0, 0x8c, 0xec, 0x1c, 0xca, 0xa4, 0x85, 0xcc, 0xaa, 0x18, 0x35,
- 0xd4, 0xa0, 0x92, 0x2a, 0xee, 0xc0, 0x5a, 0xdf, 0x03, 0xe1, 0xf1, 0x7e, 0x02, 0x14, 0xe0, 0x4c,
- 0xff, 0xb3, 0xbd, 0xe2, 0x7f, 0x34, 0x98, 0x8b, 0x72, 0xf3, 0x6d, 0x19, 0x13, 0x95, 0x8b, 0x5d,
- 0x1b, 0x72, 0x6a, 0xa2, 0xec, 0x96, 0xc7, 0x69, 0x43, 0x46, 0xd5, 0x9f, 0x69, 0x90, 0x16, 0x61,
- 0x7b, 0xac, 0xcd, 0x39, 0xbe, 0x23, 0x72, 0xf2, 0xb9, 0xc4, 0xc9, 0x77, 0x44, 0x78, 0x62, 0x23,
- 0x3e, 0x47, 0xfe, 0x76, 0x16, 0xcf, 0xc5, 0x5f, 0xa6, 0x20, 0xbb, 0x17, 0xe0, 0x20, 0xb2, 0x67,
- 0xf7, 0xf1, 0xdf, 0x40, 0x85, 0xa7, 0x4f, 0x50, 0x78, 0x07, 0xe6, 0xe5, 0x31, 0x0f, 0xdf, 0x44,
- 0x52, 0x42, 0xe7, 0x8b, 0x43, 0xea, 0x2c, 0x94, 0x79, 0x9b, 0x1c, 0x1b, 0x73, 0xbe, 0x7a, 0x42,
- 0x6f, 0x43, 0x8a, 0x7f, 0xfb, 0x88, 0x77, 0x3f, 0x04, 0xd0, 0x6d, 0x92, 0xb8, 0xa7, 0xc0, 0x51,
- 0xd0, 0x3e, 0xcc, 0x60, 0xcf, 0x23, 0x8e, 0x15, 0x66, 0xd0, 0x37, 0x46, 0xc1, 0xdb, 0x10, 0xa2,
- 0x31, 0xa4, 0xc2, 0x42, 0x5f, 0x81, 0x4c, 0xdd, 0x26, 0x98, 0x85, 0xa9, 0xf2, 0xf5, 0x51, 0x40,
- 0xcb, 0x5c, 0x32, 0xc6, 0x94, 0x48, 0xc9, 0x7b, 0x0d, 0x7f, 0x9b, 0x86, 0x45, 0x35, 0x49, 0x6a,
- 0x17, 0xec, 0x9e, 0xa5, 0xfe, 0x57, 0x17, 0x9e, 0x83, 0x85, 0xc4, 0x2f, 0x4c, 0x35, 0xef, 0x10,
- 0xff, 0xc1, 0x44, 0x3b, 0x1d, 0x96, 0x7d, 0x65, 0x64, 0xcb, 0x46, 0x07, 0xe2, 0xc2, 0xb4, 0xf7,
- 0xba, 0x4d, 0xfb, 0xea, 0x38, 0xa6, 0x8d, 0x30, 0x43, 0xdb, 0x1a, 0x5d, 0xb6, 0xbd, 0x31, 0x86,
- 0x6d, 0x23, 0x50, 0x65, 0xdc, 0xe4, 0x81, 0xfd, 0xa7, 0x69, 0x98, 0x0b, 0xbd, 0x0e, 0xd5, 0x60,
- 0x46, 0x5e, 0x48, 0x53, 0x09, 0xe6, 0xcb, 0x23, 0xba, 0x6d, 0xc9, 0x10, 0xd2, 0x5c, 0x7d, 0x89,
- 0x83, 0x7c, 0x58, 0x69, 0xb5, 0x6d, 0x1e, 0x7c, 0x3d, 0xd3, 0xa7, 0x16, 0x91, 0xf1, 0x59, 0xad,
- 0xe4, 0x8d, 0x51, 0xe1, 0xef, 0x28, 0xa8, 0x3d, 0x6a, 0x11, 0x11, 0xc9, 0xb7, 0xa7, 0x8c, 0x7c,
- 0xab, 0xbb, 0x13, 0x59, 0xb0, 0x74, 0x80, 0x9b, 0x66, 0xdb, 0x27, 0xcc, 0x14, 0xeb, 0x48, 0xad,
- 0xc2, 0xd7, 0x46, 0xe5, 0xdb, 0xc4, 0x4d, 0x5e, 0x57, 0x89, 0xf6, 0xf6, 0x94, 0x91, 0x3d, 0x48,
- 0xb4, 0x75, 0x1d, 0x66, 0xe4, 0xe7, 0x26, 0xf3, 0x85, 0xac, 0xc8, 0x17, 0xf4, 0x0f, 0x35, 0xc8,
- 0xf7, 0x28, 0x3b, 0x5c, 0xb8, 0x29, 0xc2, 0x62, 0x6c, 0xa8, 0x38, 0xe6, 0x2c, 0xf8, 0x21, 0x4c,
- 0xd5, 0x42, 0x67, 0x60, 0x46, 0x9e, 0x9c, 0x2b, 0xaf, 0x56, 0xad, 0x50, 0x91, 0x74, 0xac, 0xc8,
- 0x77, 0x34, 0xc8, 0x26, 0xbf, 0x62, 0x68, 0x1d, 0x62, 0xe3, 0x25, 0x74, 0x68, 0x87, 0x30, 0xa3,
- 0xe8, 0xc0, 0x8b, 0xab, 0xe0, 0xd8, 0x23, 0xc5, 0x37, 0x61, 0xb9, 0x6b, 0x5b, 0x42, 0x2f, 0x01,
- 0xaa, 0xbb, 0x4e, 0x40, 0x9d, 0x36, 0x96, 0x27, 0x2f, 0x62, 0xa9, 0x4a, 0x43, 0xe6, 0x93, 0x23,
- 0x62, 0xc5, 0x16, 0xef, 0x41, 0xae, 0x7b, 0xf9, 0x8d, 0x08, 0x11, 0x85, 0x81, 0xe9, 0x44, 0x18,
- 0x58, 0x07, 0xd4, 0xbb, 0xbf, 0x45, 0x6f, 0x6a, 0x89, 0x37, 0xd7, 0x60, 0xa5, 0xcf, 0x72, 0x2d,
- 0xae, 0x40, 0xbe, 0x67, 0x2f, 0x2b, 0xae, 0x2a, 0xd4, 0x8e, 0x45, 0x58, 0xfc, 0x4b, 0x1a, 0xe6,
- 0x76, 0x5c, 0x95, 0x5c, 0x7f, 0x0d, 0xe6, 0x7c, 0x72, 0x48, 0x18, 0x0d, 0xa4, 0xf7, 0x2c, 0x0d,
- 0x5d, 0xf6, 0x87, 0x10, 0xa5, 0x3d, 0x25, 0x2f, 0x0f, 0x09, 0x23, 0xb8, 0xf1, 0x6b, 0x61, 0x54,
- 0xe0, 0x65, 0xa6, 0xef, 0xe3, 0x66, 0xf8, 0x13, 0x20, 0x6c, 0xf2, 0x7d, 0x36, 0x60, 0xb8, 0x4e,
- 0xc4, 0xe4, 0xce, 0x1b, 0xb2, 0x31, 0x38, 0x46, 0x66, 0x4e, 0x88, 0x91, 0x4f, 0xbd, 0x58, 0x3a,
- 0xf3, 0xf4, 0x8b, 0xa5, 0xcf, 0x43, 0x96, 0x17, 0x6c, 0xb6, 0xab, 0x0e, 0xda, 0x66, 0xa5, 0x93,
- 0xda, 0x6e, 0x73, 0x47, 0x75, 0x71, 0x27, 0x0d, 0x1e, 0x32, 0x82, 0xad, 0xc2, 0x9c, 0x18, 0x54,
- 0x2d, 0xfd, 0xab, 0xea, 0xfe, 0x69, 0x0d, 0xf8, 0xeb, 0x26, 0x71, 0x02, 0x46, 0x49, 0x98, 0x4d,
- 0x5f, 0x1c, 0x71, 0x0e, 0x0c, 0xb0, 0xe5, 0x13, 0x25, 0xbe, 0xce, 0x60, 0x2e, 0x9c, 0x92, 0x62,
- 0x03, 0xd2, 0x7c, 0x56, 0xd0, 0x32, 0x2c, 0xdc, 0xdb, 0xdd, 0xab, 0x55, 0xca, 0xd5, 0x5b, 0xd5,
- 0xca, 0x56, 0x6e, 0x0a, 0xcd, 0x43, 0x66, 0xdf, 0xd8, 0x28, 0x57, 0x72, 0x1a, 0x7f, 0xdc, 0xaa,
- 0x6c, 0xde, 0xbb, 0x9d, 0x9b, 0x46, 0x73, 0x90, 0xae, 0xee, 0xde, 0xba, 0x9b, 0x4b, 0x21, 0x80,
- 0x99, 0xdd, 0xbb, 0xfb, 0xd5, 0x72, 0x25, 0x97, 0xe6, 0xbd, 0xf7, 0x37, 0x8c, 0xdd, 0x5c, 0x86,
- 0xbf, 0x5a, 0x31, 0x8c, 0xbb, 0x46, 0x6e, 0x06, 0x65, 0x61, 0xae, 0x6c, 0x54, 0xf7, 0xab, 0xe5,
- 0x8d, 0x9d, 0xdc, 0x6c, 0x31, 0x0b, 0xb0, 0xe3, 0x36, 0xcb, 0xae, 0x13, 0x30, 0xd7, 0x2e, 0xfe,
- 0x37, 0x0d, 0xe7, 0x77, 0xdd, 0x80, 0x36, 0x8e, 0xe5, 0xf6, 0xb4, 0x71, 0x88, 0xa9, 0x8d, 0x0f,
- 0xe2, 0xa4, 0xf1, 0x19, 0x98, 0x3f, 0x72, 0xd9, 0x23, 0x79, 0x59, 0x55, 0x2e, 0xff, 0x39, 0xd9,
- 0x51, 0xb5, 0xd0, 0x01, 0xe4, 0xea, 0x12, 0xc8, 0x0c, 0x6f, 0x22, 0x2b, 0xf7, 0x19, 0xfb, 0x5e,
- 0xca, 0xb2, 0x02, 0xac, 0x28, 0x3c, 0xce, 0x61, 0xbb, 0xcd, 0x26, 0x75, 0x9a, 0x31, 0x47, 0x6a,
- 0x42, 0x0e, 0x05, 0x18, 0x71, 0x58, 0x90, 0xc7, 0x2c, 0xa0, 0x0d, 0x5c, 0x0f, 0x62, 0x92, 0xf4,
- 0x64, 0x24, 0xb9, 0x10, 0x31, 0x62, 0x69, 0x88, 0x13, 0xad, 0x43, 0xea, 0x73, 0xcf, 0x8f, 0x68,
- 0x32, 0x93, 0xd1, 0xe4, 0x23, 0xc8, 0x88, 0xa7, 0x09, 0x33, 0x1e, 0x66, 0xb8, 0xe5, 0x17, 0x40,
- 0xb8, 0xe8, 0xdd, 0x21, 0x5d, 0xf4, 0x24, 0x3f, 0x28, 0xd5, 0x04, 0xa2, 0xba, 0x18, 0x25, 0xe1,
- 0xf5, 0x1b, 0xb0, 0x90, 0xe8, 0x7e, 0x5a, 0xf5, 0x3f, 0x9f, 0x2c, 0x5d, 0xbf, 0x04, 0xcf, 0x0e,
- 0xa0, 0x53, 0x3b, 0x75, 0x94, 0xa6, 0x69, 0x89, 0x34, 0xed, 0xca, 0xc7, 0x1a, 0x2c, 0x6e, 0x12,
- 0xdc, 0xba, 0xe5, 0x28, 0x07, 0x46, 0x1f, 0x6a, 0x30, 0x1b, 0x3e, 0x0f, 0x9b, 0x44, 0xf5, 0xb9,
- 0xf8, 0xaa, 0xdf, 0x18, 0x47, 0x56, 0xee, 0xdd, 0x53, 0xeb, 0xda, 0x25, 0xed, 0xca, 0x07, 0x00,
- 0x52, 0x33, 0x51, 0xbc, 0x38, 0xaa, 0x88, 0xb9, 0x38, 0x62, 0x25, 0xa4, 0x8f, 0x2a, 0xa0, 0xd8,
- 0xbf, 0xaf, 0xc1, 0x82, 0xa4, 0x97, 0x91, 0xfb, 0x31, 0x64, 0xe4, 0xc3, 0xd5, 0x51, 0xd2, 0x18,
- 0xf5, 0x45, 0xfa, 0xb5, 0xd1, 0x84, 0x54, 0xb4, 0x92, 0x9a, 0xfc, 0x30, 0x9a, 0xa2, 0x1d, 0xb9,
- 0xca, 0xd0, 0x63, 0x98, 0x0d, 0x1f, 0xaf, 0x8d, 0x1a, 0xb1, 0xf8, 0xc6, 0xab, 0x5f, 0x1e, 0x5e,
- 0x2a, 0xdc, 0xd7, 0xa4, 0x2e, 0x7f, 0xd0, 0xa0, 0x20, 0x75, 0xa9, 0x3c, 0x0e, 0x08, 0x73, 0xb0,
- 0x7d, 0x5f, 0x6c, 0x5d, 0x35, 0xd7, 0xb5, 0xd1, 0xaf, 0x35, 0x58, 0xeb, 0xeb, 0x83, 0xa8, 0x7c,
- 0x0a, 0x0b, 0x46, 0xdf, 0x9a, 0x0c, 0x24, 0xb4, 0xe9, 0xe6, 0x06, 0x7c, 0x71, 0x10, 0x50, 0x12,
- 0x67, 0x73, 0x5e, 0x7e, 0xe8, 0x86, 0x47, 0x1f, 0x2c, 0x25, 0x86, 0xcc, 0xc3, 0xcb, 0x07, 0x33,
- 0x22, 0x86, 0x5f, 0xfd, 0x5f, 0x00, 0x00, 0x00, 0xff, 0xff, 0x51, 0x9e, 0x43, 0x4b, 0x80, 0x32,
- 0x00, 0x00,
+var fileDescriptor_beam_fn_api_d24d1635dfa071c8 = []byte{
+ // 3139 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5a, 0x5b, 0x6f, 0x1b, 0xc7,
+ 0xf5, 0xf7, 0x92, 0x94, 0x44, 0x1e, 0x52, 0x12, 0x35, 0x92, 0x6c, 0x7a, 0xff, 0xce, 0xbf, 0x0e,
+ 0xdb, 0x00, 0x42, 0x8a, 0xac, 0xaf, 0x48, 0xec, 0x34, 0x71, 0x22, 0x51, 0xb4, 0xcd, 0x44, 0xb6,
+ 0xd9, 0x95, 0x5c, 0xb7, 0x49, 0x93, 0xc5, 0x8a, 0x3b, 0xa4, 0x17, 0x5e, 0xee, 0x6e, 0x66, 0x96,
+ 0xb2, 0xe4, 0x06, 0x0d, 0x7a, 0x41, 0x8b, 0x16, 0x6d, 0xf3, 0xd2, 0x87, 0xb4, 0x6f, 0x6d, 0x81,
+ 0x02, 0x7d, 0xe9, 0x07, 0xc8, 0x37, 0x28, 0x50, 0xa0, 0x5f, 0x20, 0x2f, 0x45, 0x5b, 0xa0, 0x6d,
+ 0xfa, 0x5c, 0xa0, 0x6f, 0xc5, 0x5c, 0xf6, 0xc2, 0x25, 0xe9, 0x2c, 0x29, 0xa5, 0x6f, 0x3b, 0x73,
+ 0xf6, 0xfc, 0x7e, 0x33, 0x67, 0xcf, 0x9c, 0x39, 0x67, 0x66, 0x61, 0x65, 0x1f, 0x9b, 0x7d, 0xa3,
+ 0xeb, 0x1a, 0xa6, 0x6f, 0x6b, 0x3e, 0xf1, 0x02, 0x0f, 0x3d, 0xe7, 0x91, 0x9e, 0x66, 0xfa, 0x66,
+ 0xe7, 0x21, 0xd6, 0x98, 0x54, 0xeb, 0x7b, 0x16, 0x76, 0xb4, 0xae, 0x6b, 0xe0, 0x43, 0xdc, 0x19,
+ 0x04, 0xb6, 0xe7, 0x6a, 0x07, 0x97, 0xd4, 0x75, 0xae, 0x49, 0x06, 0xae, 0x8b, 0x49, 0xac, 0xad,
+ 0x2e, 0x63, 0xd7, 0xf2, 0x3d, 0xdb, 0x0d, 0xa8, 0xec, 0x38, 0xdf, 0xf3, 0xbc, 0x9e, 0x83, 0x2f,
+ 0xf0, 0xd6, 0xfe, 0xa0, 0x7b, 0xc1, 0xc2, 0xb4, 0x43, 0x6c, 0x3f, 0xf0, 0x88, 0x7c, 0xe3, 0x0b,
+ 0xe9, 0x37, 0x02, 0xbb, 0x8f, 0x69, 0x60, 0xf6, 0x7d, 0xf9, 0xc2, 0xff, 0xa7, 0x5f, 0x78, 0x4c,
+ 0x4c, 0xdf, 0xc7, 0x24, 0xa4, 0x58, 0xec, 0xe3, 0x80, 0xd8, 0x1d, 0xd9, 0xac, 0xff, 0x52, 0x81,
+ 0x25, 0x1d, 0xf7, 0xbd, 0x00, 0xdf, 0x22, 0x7e, 0xa7, 0xed, 0x91, 0x00, 0xf5, 0xe1, 0xb4, 0xe9,
+ 0xdb, 0x06, 0xc5, 0xe4, 0xc0, 0xee, 0x60, 0x23, 0x1e, 0x42, 0x4d, 0x39, 0xaf, 0x6c, 0x94, 0x2f,
+ 0xbf, 0xa4, 0x8d, 0x9f, 0xb4, 0x6f, 0xfb, 0xd8, 0xb1, 0x5d, 0xac, 0x1d, 0x5c, 0xd2, 0x36, 0x7d,
+ 0x7b, 0x57, 0xe8, 0x6f, 0x47, 0xea, 0xfa, 0x9a, 0x39, 0xa6, 0x17, 0x9d, 0x85, 0x62, 0xc7, 0xb3,
+ 0x30, 0x31, 0x6c, 0xab, 0x96, 0x3b, 0xaf, 0x6c, 0x94, 0xf4, 0x05, 0xde, 0x6e, 0x59, 0xf5, 0xbf,
+ 0x15, 0x00, 0xb5, 0x5c, 0x1a, 0x90, 0x41, 0x87, 0x59, 0x52, 0xc7, 0xef, 0x0d, 0x30, 0x0d, 0xd0,
+ 0x73, 0xb0, 0x64, 0xc7, 0xbd, 0x4c, 0x4f, 0xe1, 0x7a, 0x8b, 0x89, 0xde, 0x96, 0x85, 0xee, 0x43,
+ 0x91, 0xe0, 0x9e, 0x4d, 0x03, 0x4c, 0x6a, 0x7f, 0x5f, 0xe0, 0x43, 0x7f, 0x51, 0xcb, 0xf4, 0xbd,
+ 0x34, 0x5d, 0xea, 0x49, 0xc6, 0xdb, 0xa7, 0xf4, 0x08, 0x0a, 0x61, 0x58, 0xf2, 0x89, 0xd7, 0xc1,
+ 0x94, 0x1a, 0xfb, 0x03, 0xd7, 0x72, 0x70, 0xed, 0x1f, 0x02, 0xfc, 0x2b, 0x19, 0xc1, 0xdb, 0x42,
+ 0x7b, 0x8b, 0x2b, 0xc7, 0x0c, 0x8b, 0x7e, 0xb2, 0x1f, 0x7d, 0x1b, 0xce, 0x0c, 0xd3, 0x18, 0x3e,
+ 0xf1, 0x7a, 0x04, 0x53, 0x5a, 0xfb, 0xa7, 0xe0, 0x6b, 0xcc, 0xc2, 0xd7, 0x96, 0x20, 0x31, 0xef,
+ 0xba, 0x3f, 0x4e, 0x8e, 0x06, 0xb0, 0x96, 0xe2, 0xa7, 0xbe, 0x63, 0x07, 0xb5, 0x4f, 0x05, 0xf9,
+ 0xeb, 0xb3, 0x90, 0xef, 0x32, 0x84, 0x98, 0x19, 0xf9, 0x23, 0x42, 0xf4, 0x10, 0x96, 0xbb, 0xb6,
+ 0x6b, 0x3a, 0xf6, 0x13, 0x1c, 0x9a, 0xf7, 0x5f, 0x82, 0xf1, 0x95, 0x8c, 0x8c, 0x37, 0xa5, 0x7a,
+ 0xda, 0xbe, 0x4b, 0xdd, 0x21, 0xc1, 0x56, 0x09, 0x16, 0x88, 0x10, 0xd6, 0xbf, 0x3b, 0x07, 0xab,
+ 0x43, 0x7e, 0x46, 0x7d, 0xcf, 0xa5, 0x38, 0xab, 0xa3, 0xad, 0xc1, 0x1c, 0x26, 0xc4, 0x23, 0xd2,
+ 0x7d, 0x45, 0x03, 0x7d, 0x6d, 0xd4, 0xfd, 0x5e, 0x9a, 0xda, 0xfd, 0xc4, 0x40, 0x86, 0xfc, 0xaf,
+ 0x3b, 0xc9, 0xff, 0x5e, 0x99, 0xcd, 0xff, 0x22, 0x8a, 0x94, 0x03, 0x7e, 0xf0, 0x99, 0x0e, 0xb8,
+ 0x7d, 0x3c, 0x07, 0x8c, 0x88, 0x27, 0x78, 0xe0, 0xc1, 0xd3, 0x3d, 0x70, 0xf3, 0x18, 0x1e, 0x18,
+ 0x51, 0x8f, 0x73, 0x41, 0x7b, 0xa2, 0x0b, 0xbe, 0x3a, 0xa3, 0x0b, 0x46, 0x74, 0x69, 0x1f, 0x04,
+ 0xe6, 0x23, 0x42, 0x5a, 0xff, 0xa9, 0x02, 0xcb, 0xa9, 0xb8, 0x83, 0x9e, 0xc0, 0xd9, 0x94, 0x09,
+ 0x86, 0xa2, 0x71, 0x7e, 0xa3, 0x7c, 0xf9, 0xc6, 0x2c, 0x66, 0x48, 0x04, 0xe5, 0x33, 0xfe, 0x78,
+ 0x41, 0x1d, 0x41, 0x35, 0xed, 0x87, 0xf5, 0xdf, 0x00, 0x9c, 0x99, 0x00, 0x84, 0x96, 0x20, 0x17,
+ 0x2d, 0x90, 0x9c, 0x6d, 0x21, 0x17, 0x20, 0x20, 0xa6, 0x4b, 0xbb, 0x1e, 0xe9, 0xd3, 0x5a, 0x8e,
+ 0x0f, 0xf6, 0xee, 0xf1, 0x06, 0xab, 0xed, 0x45, 0x80, 0x4d, 0x37, 0x20, 0x47, 0x7a, 0x82, 0x01,
+ 0x05, 0x50, 0xf1, 0x3b, 0x9e, 0xe3, 0x60, 0xbe, 0x2c, 0x69, 0x2d, 0xcf, 0x19, 0xdb, 0xc7, 0x64,
+ 0x6c, 0x27, 0x20, 0x05, 0xe7, 0x10, 0x0b, 0xfa, 0xb1, 0x02, 0x6b, 0x8f, 0x6d, 0xd7, 0xf2, 0x1e,
+ 0xdb, 0x6e, 0xcf, 0xa0, 0x01, 0x31, 0x03, 0xdc, 0xb3, 0x31, 0xad, 0x15, 0x38, 0xfd, 0x83, 0x63,
+ 0xd2, 0x3f, 0x08, 0xa1, 0x77, 0x23, 0x64, 0x31, 0x8a, 0xd5, 0xc7, 0xa3, 0x12, 0xb4, 0x0f, 0xf3,
+ 0x7c, 0xeb, 0xa4, 0xb5, 0x39, 0xce, 0xfe, 0xc6, 0x31, 0xd9, 0x1b, 0x1c, 0x4c, 0x10, 0x4a, 0x64,
+ 0x66, 0x66, 0xec, 0x1e, 0xd8, 0xc4, 0x73, 0xfb, 0xd8, 0x0d, 0x68, 0x6d, 0xfe, 0x44, 0xcc, 0xdc,
+ 0x4c, 0x40, 0x4a, 0x33, 0x27, 0x59, 0xd0, 0x21, 0x9c, 0xa3, 0x81, 0x19, 0x60, 0x63, 0x42, 0x66,
+ 0xb2, 0x70, 0xbc, 0xcc, 0xe4, 0x2c, 0x07, 0x1f, 0x27, 0x52, 0x1d, 0x58, 0x4e, 0x79, 0x1d, 0xaa,
+ 0x42, 0xfe, 0x11, 0x3e, 0x92, 0xae, 0xce, 0x1e, 0x51, 0x03, 0xe6, 0x0e, 0x4c, 0x67, 0x80, 0xf9,
+ 0x0e, 0x50, 0xbe, 0xfc, 0x42, 0x86, 0x71, 0xb4, 0x23, 0x54, 0x5d, 0xe8, 0xbe, 0x9c, 0xbb, 0xa6,
+ 0xa8, 0x1e, 0xac, 0x8c, 0x78, 0xdc, 0x18, 0xbe, 0xed, 0x61, 0x3e, 0x2d, 0x0b, 0x5f, 0x23, 0x82,
+ 0x4d, 0x12, 0xbe, 0x0f, 0xb5, 0x49, 0x3e, 0x36, 0x86, 0xf7, 0x8d, 0x61, 0xde, 0xab, 0x19, 0x78,
+ 0xd3, 0xe8, 0x47, 0x49, 0xf6, 0x0e, 0x94, 0x13, 0x3e, 0x36, 0x86, 0xf0, 0xc6, 0x30, 0xe1, 0x46,
+ 0x06, 0x42, 0x0e, 0x98, 0xb2, 0xe9, 0x88, 0x7b, 0x9d, 0x8c, 0x4d, 0x13, 0xb0, 0x09, 0xc2, 0xfa,
+ 0x7f, 0xf2, 0xb0, 0x22, 0x3c, 0x7c, 0xd3, 0xf7, 0x1d, 0xbb, 0x63, 0x32, 0xa3, 0xa3, 0x67, 0xa1,
+ 0x12, 0x45, 0xab, 0x38, 0x95, 0x28, 0x47, 0x7d, 0x2d, 0x8b, 0xa5, 0xc2, 0xb6, 0xeb, 0x0f, 0x82,
+ 0x44, 0x2a, 0xcc, 0xdb, 0x2d, 0x0b, 0xd5, 0x60, 0x01, 0x3b, 0x98, 0x31, 0xd5, 0xf2, 0xe7, 0x95,
+ 0x8d, 0x8a, 0x1e, 0x36, 0xd1, 0xb7, 0x60, 0xc5, 0x1b, 0x04, 0x4c, 0xeb, 0xb1, 0x19, 0x60, 0xd2,
+ 0x37, 0xc9, 0xa3, 0x30, 0xfa, 0x64, 0x0d, 0xb7, 0x23, 0x83, 0xd5, 0xee, 0x71, 0xc4, 0x07, 0x11,
+ 0xa0, 0x58, 0x93, 0x55, 0x2f, 0xd5, 0x8d, 0xda, 0x00, 0x36, 0x35, 0xf6, 0xbd, 0x81, 0x6b, 0x61,
+ 0xab, 0x36, 0x77, 0x5e, 0xd9, 0x58, 0xba, 0x7c, 0x29, 0x83, 0xe5, 0x5a, 0x74, 0x4b, 0xe8, 0x68,
+ 0x4d, 0x77, 0xd0, 0xd7, 0x4b, 0x76, 0xd8, 0x46, 0xdf, 0x84, 0x6a, 0xdf, 0x73, 0xed, 0xc0, 0x23,
+ 0x2c, 0xa0, 0xda, 0x6e, 0xd7, 0x0b, 0x63, 0x4c, 0x16, 0xdc, 0x3b, 0x91, 0x6a, 0xcb, 0xed, 0x7a,
+ 0xfa, 0x72, 0x7f, 0xa8, 0x4d, 0x55, 0x03, 0xd6, 0xc7, 0x4e, 0x6d, 0x8c, 0x3f, 0x5c, 0x1c, 0xf6,
+ 0x07, 0x55, 0x13, 0x85, 0x95, 0x16, 0x16, 0x56, 0xda, 0x5e, 0x58, 0x79, 0x25, 0xbf, 0xfd, 0x1f,
+ 0x15, 0xa8, 0x6d, 0x63, 0xc7, 0x3c, 0xc2, 0xd6, 0xa8, 0x0b, 0xec, 0x41, 0x4d, 0xa6, 0x9c, 0xd8,
+ 0x8a, 0xbf, 0x80, 0xc1, 0x4a, 0x38, 0x59, 0x5b, 0x3d, 0x8d, 0xe5, 0x74, 0xa4, 0xdb, 0x0c, 0x55,
+ 0x99, 0x10, 0xbd, 0x05, 0x65, 0x33, 0x26, 0x91, 0xc3, 0xbd, 0x36, 0xeb, 0xa7, 0xd7, 0x93, 0x60,
+ 0xf5, 0x9f, 0x15, 0x60, 0x6d, 0x5c, 0xbd, 0x82, 0x5e, 0x83, 0x73, 0x13, 0x33, 0x93, 0xd8, 0xbb,
+ 0xcf, 0x4e, 0x48, 0x2e, 0x5a, 0x16, 0xb2, 0xa1, 0xd2, 0x61, 0x83, 0x33, 0x02, 0xef, 0x11, 0x76,
+ 0xc3, 0x04, 0xe1, 0xe6, 0x31, 0x6a, 0x28, 0xad, 0xc1, 0xb4, 0xf6, 0x18, 0x9c, 0x5e, 0xee, 0x44,
+ 0xcf, 0x54, 0xfd, 0x43, 0x0e, 0x20, 0x96, 0xa1, 0xf7, 0x00, 0x06, 0x14, 0x13, 0x83, 0xc7, 0x7c,
+ 0x69, 0xf7, 0xf6, 0xc9, 0xf0, 0x6a, 0xf7, 0x29, 0x26, 0xbb, 0x0c, 0xf7, 0xf6, 0x29, 0xbd, 0x34,
+ 0x08, 0x1b, 0x8c, 0x92, 0xda, 0x16, 0x36, 0xf8, 0x6a, 0x96, 0x5f, 0xe8, 0xa4, 0x28, 0x77, 0x6d,
+ 0x0b, 0xb7, 0x18, 0x2e, 0xa3, 0xa4, 0x61, 0x83, 0x15, 0x25, 0xdc, 0xb2, 0x35, 0xe0, 0xe1, 0x42,
+ 0x34, 0xd4, 0x32, 0x94, 0xa2, 0x21, 0xaa, 0xcf, 0x43, 0x29, 0x52, 0x46, 0xcf, 0x0c, 0x0d, 0x51,
+ 0x7c, 0xbe, 0x18, 0x6e, 0x6b, 0x1e, 0x0a, 0xc1, 0x91, 0x8f, 0xeb, 0x9f, 0xe4, 0x60, 0x7d, 0x6c,
+ 0x01, 0x81, 0x6e, 0xc3, 0x82, 0x3c, 0x5a, 0x90, 0x36, 0xd5, 0x32, 0x4e, 0xf0, 0x8e, 0xd0, 0xd2,
+ 0x43, 0x75, 0x56, 0xe1, 0x10, 0x4c, 0x6d, 0x6b, 0x60, 0x3a, 0x06, 0xf1, 0xbc, 0x20, 0x74, 0x8e,
+ 0xd7, 0x32, 0x02, 0x4e, 0x5a, 0x7f, 0xfa, 0x62, 0x08, 0xab, 0x33, 0xd4, 0xb1, 0xa1, 0x26, 0x7f,
+ 0x52, 0xa1, 0x06, 0x5d, 0x81, 0x75, 0xb6, 0x60, 0x6d, 0x82, 0xa9, 0x21, 0xd3, 0x7e, 0xb1, 0x40,
+ 0x0b, 0xe7, 0x95, 0x8d, 0xa2, 0xbe, 0x16, 0x0a, 0x6f, 0x26, 0x64, 0xf5, 0x26, 0x9c, 0x7b, 0x5a,
+ 0xb9, 0x9e, 0xb1, 0x22, 0xad, 0x7f, 0xb4, 0x0a, 0x0b, 0xd2, 0xac, 0xc8, 0x84, 0xb2, 0x9f, 0x48,
+ 0xc4, 0x95, 0xa9, 0x4c, 0x29, 0x41, 0xb4, 0x76, 0x90, 0xca, 0xbc, 0x93, 0x98, 0xea, 0x27, 0x65,
+ 0x80, 0x38, 0x9f, 0x41, 0x4f, 0x20, 0x2c, 0xab, 0x58, 0x98, 0x13, 0xdb, 0x54, 0xe8, 0x14, 0x6f,
+ 0x4e, 0x4b, 0x1c, 0xc1, 0x86, 0x0b, 0x01, 0x5b, 0x4d, 0x09, 0xa9, 0xaf, 0xf8, 0xe9, 0x2e, 0xf4,
+ 0x1e, 0x2c, 0x9b, 0x9d, 0xc0, 0x3e, 0xc0, 0x31, 0xb1, 0x58, 0x6e, 0xb7, 0x67, 0x27, 0xde, 0xe4,
+ 0x80, 0x11, 0xeb, 0x92, 0x39, 0xd4, 0x46, 0x36, 0x40, 0x62, 0xe7, 0x15, 0x0e, 0xd4, 0x9a, 0x9d,
+ 0x2d, 0xbd, 0xe9, 0x26, 0xc0, 0xd1, 0x2d, 0x28, 0xb0, 0xa0, 0x22, 0xb7, 0xf7, 0x2b, 0x53, 0x92,
+ 0xb0, 0x95, 0xaf, 0x73, 0x00, 0xf5, 0xaf, 0x79, 0x28, 0xde, 0xc1, 0x26, 0x1d, 0x10, 0x6c, 0xa1,
+ 0x9f, 0x28, 0xb0, 0x26, 0xf2, 0x0e, 0x69, 0x33, 0xa3, 0xe3, 0x0d, 0xc4, 0x27, 0x63, 0x34, 0x6f,
+ 0xcd, 0x3e, 0x97, 0x90, 0x42, 0xe3, 0x41, 0x44, 0x5a, 0xac, 0xc1, 0xc1, 0xc5, 0xe4, 0x90, 0x3d,
+ 0x22, 0x40, 0x1f, 0x2a, 0xb0, 0x2e, 0x33, 0x9a, 0xd4, 0x78, 0x44, 0x18, 0x78, 0xfb, 0x04, 0xc6,
+ 0x23, 0x92, 0x80, 0x31, 0x03, 0x5a, 0xf5, 0x46, 0x25, 0x68, 0x03, 0xaa, 0x81, 0x17, 0x98, 0x0e,
+ 0xdf, 0xa9, 0x0d, 0xea, 0x87, 0x59, 0x98, 0xa2, 0x2f, 0xf1, 0x7e, 0xb6, 0x0d, 0xef, 0xb2, 0x5e,
+ 0xb5, 0x09, 0x67, 0x26, 0x4c, 0x75, 0x4c, 0x86, 0xb1, 0x96, 0xcc, 0x30, 0xf2, 0xc9, 0x94, 0xf5,
+ 0x26, 0xd4, 0x26, 0x8d, 0x70, 0x2a, 0x1c, 0x0a, 0x2b, 0x23, 0xab, 0x06, 0xbd, 0x0b, 0xc5, 0xbe,
+ 0xb4, 0x83, 0x5c, 0x94, 0x5b, 0xc7, 0xb7, 0xa8, 0x1e, 0x61, 0xaa, 0x1f, 0xe6, 0x61, 0x69, 0x78,
+ 0xc9, 0x7c, 0xde, 0x94, 0xe8, 0x05, 0x40, 0x5d, 0x62, 0x8a, 0x98, 0x48, 0x70, 0xdf, 0xb4, 0x5d,
+ 0xdb, 0xed, 0x71, 0x73, 0x28, 0xfa, 0x4a, 0x28, 0xd1, 0x43, 0x01, 0xfa, 0x95, 0x02, 0x67, 0x87,
+ 0x3d, 0x8c, 0x26, 0xd4, 0xc4, 0x0a, 0xc6, 0x27, 0x15, 0x2f, 0x86, 0x7d, 0x8d, 0x46, 0xa3, 0x10,
+ 0xfe, 0x76, 0xc6, 0x1b, 0x2f, 0x55, 0xdf, 0x80, 0x73, 0x4f, 0x53, 0x9c, 0xca, 0x0d, 0x5e, 0x85,
+ 0xe5, 0xcf, 0xce, 0x77, 0x27, 0xab, 0xff, 0x69, 0x0e, 0x0a, 0x2c, 0x76, 0x20, 0x03, 0xca, 0x62,
+ 0x8f, 0x36, 0x5c, 0x33, 0x4a, 0x59, 0x6f, 0xcc, 0x10, 0x85, 0x64, 0xe3, 0xae, 0xd9, 0xc7, 0x3a,
+ 0xf4, 0xa3, 0x67, 0x84, 0xa1, 0xc2, 0x97, 0x3a, 0x26, 0x86, 0x65, 0x06, 0x66, 0x78, 0xb2, 0xf9,
+ 0xda, 0x2c, 0x14, 0x0d, 0x01, 0xb4, 0x6d, 0x06, 0xe6, 0xed, 0x53, 0x7a, 0xb9, 0x13, 0x37, 0x51,
+ 0x00, 0x2b, 0x96, 0x4d, 0x03, 0x62, 0xef, 0x8b, 0x04, 0x9c, 0x73, 0x4d, 0x79, 0xa8, 0x39, 0xc4,
+ 0xb5, 0x9d, 0x40, 0x93, 0x84, 0x55, 0x2b, 0xd5, 0x87, 0x0c, 0x80, 0x9e, 0x39, 0xe8, 0x61, 0x41,
+ 0xf7, 0xe9, 0x74, 0x47, 0x8a, 0x43, 0x74, 0xb7, 0x18, 0x8c, 0xe4, 0x29, 0xf5, 0xc2, 0x86, 0x7a,
+ 0x03, 0x20, 0xb6, 0x2b, 0x3a, 0x07, 0x25, 0xf6, 0x95, 0xa8, 0x6f, 0x76, 0xb0, 0xac, 0x26, 0xe3,
+ 0x0e, 0x84, 0xa0, 0xc0, 0xbf, 0x61, 0x9e, 0x0b, 0xf8, 0xb3, 0xfa, 0x45, 0x56, 0x8d, 0xc7, 0x56,
+ 0x8a, 0x1c, 0x42, 0x49, 0x38, 0x84, 0xfa, 0x2e, 0x54, 0xd3, 0xb3, 0x65, 0x6f, 0x72, 0xf3, 0x86,
+ 0x6f, 0xf2, 0x06, 0x73, 0x31, 0x3a, 0xe8, 0x4b, 0x77, 0x62, 0x8f, 0xac, 0xa7, 0x6f, 0xbb, 0x9c,
+ 0x33, 0xaf, 0xb3, 0x47, 0xde, 0x63, 0x1e, 0xf2, 0x94, 0x88, 0xf5, 0x98, 0x87, 0xea, 0xdb, 0x50,
+ 0x8a, 0xa6, 0x37, 0x7e, 0x08, 0xe8, 0x1a, 0x94, 0xa2, 0x5b, 0xaf, 0x0c, 0xd5, 0x59, 0xfc, 0x32,
+ 0xcb, 0x62, 0x99, 0xf1, 0xd5, 0x23, 0xa8, 0xa6, 0x33, 0x9a, 0x31, 0x2b, 0xe2, 0xde, 0x70, 0x05,
+ 0x78, 0x7d, 0xe6, 0x88, 0x90, 0x2c, 0x10, 0x7f, 0x9b, 0x83, 0x67, 0x9e, 0x7a, 0x20, 0x7e, 0x82,
+ 0x89, 0xf4, 0xe7, 0x9b, 0xe0, 0xbe, 0x03, 0x8b, 0x3e, 0xb1, 0xfb, 0x26, 0x39, 0x92, 0x59, 0xba,
+ 0xc8, 0x4a, 0x66, 0xaf, 0x3c, 0x2b, 0x12, 0x8e, 0x67, 0xe7, 0xf5, 0xef, 0x14, 0xe0, 0xec, 0xc4,
+ 0xdb, 0xa3, 0xac, 0x57, 0x33, 0x4f, 0x60, 0xc9, 0xc2, 0xd4, 0x26, 0xd8, 0x12, 0x97, 0x07, 0xe1,
+ 0xfc, 0x77, 0x8f, 0x7b, 0x7d, 0xa5, 0x6d, 0x0b, 0x58, 0xde, 0x27, 0x73, 0x87, 0x45, 0x2b, 0xd9,
+ 0xa7, 0xfe, 0x5e, 0x81, 0x4a, 0xf2, 0x2d, 0x74, 0x19, 0xd6, 0xa3, 0x5d, 0xca, 0xeb, 0xca, 0x1d,
+ 0xc7, 0xc2, 0xe2, 0x5e, 0x35, 0xa7, 0xaf, 0x86, 0xc2, 0x7b, 0x5d, 0x3d, 0x14, 0xa1, 0x8b, 0xb0,
+ 0x66, 0x3a, 0x8e, 0xf7, 0x38, 0x9c, 0x80, 0x21, 0xee, 0x8b, 0xf9, 0x34, 0xf2, 0x3a, 0x92, 0x32,
+ 0x8e, 0xdf, 0xe6, 0x12, 0x74, 0x0d, 0x6a, 0x98, 0x06, 0x76, 0xdf, 0x0c, 0xb0, 0x65, 0x0c, 0xa5,
+ 0x75, 0x54, 0xae, 0xc5, 0xd3, 0x91, 0x3c, 0x99, 0xab, 0x50, 0xf5, 0x43, 0x05, 0xd0, 0xe8, 0xb4,
+ 0xc6, 0x2c, 0x8c, 0xce, 0xf0, 0xc2, 0xb8, 0x73, 0xa2, 0xc6, 0x4c, 0x2e, 0x96, 0x7f, 0xe7, 0x41,
+ 0x9d, 0x7c, 0x7f, 0x33, 0xea, 0x81, 0xca, 0x49, 0x7a, 0xe0, 0xff, 0xac, 0x0e, 0x1d, 0xc0, 0x52,
+ 0xe7, 0xa1, 0xe9, 0xba, 0xd8, 0x19, 0x76, 0xd2, 0xbb, 0xc7, 0xbe, 0xe1, 0xd2, 0x1a, 0x02, 0x57,
+ 0x74, 0x2e, 0x76, 0x12, 0x2d, 0xaa, 0xfe, 0x42, 0x81, 0x4a, 0x52, 0x9e, 0xe5, 0x84, 0xf2, 0x22,
+ 0xac, 0x39, 0x26, 0x0d, 0x8c, 0xd0, 0xec, 0xe1, 0x99, 0x24, 0x73, 0x84, 0x39, 0x1d, 0x31, 0x59,
+ 0x5b, 0x88, 0xa4, 0x57, 0xa1, 0xab, 0x70, 0xba, 0x6b, 0x13, 0x1a, 0x18, 0x91, 0x29, 0x93, 0xe7,
+ 0x98, 0x73, 0xfa, 0x1a, 0x97, 0xea, 0x52, 0x28, 0xb5, 0xea, 0x37, 0x60, 0x7d, 0xec, 0x3d, 0x6e,
+ 0xd6, 0x02, 0xb8, 0x06, 0xa7, 0xc7, 0x5f, 0xc2, 0xd5, 0x3f, 0x56, 0xa0, 0x18, 0xe5, 0xa5, 0xb7,
+ 0xc5, 0x7e, 0x20, 0xfd, 0xe6, 0x6a, 0x46, 0x7b, 0x47, 0x99, 0x1d, 0xdb, 0xa3, 0x74, 0xb1, 0xa3,
+ 0x58, 0x50, 0xe0, 0x3b, 0x56, 0xc6, 0xb8, 0x94, 0x36, 0x75, 0x6e, 0xd4, 0xd4, 0x48, 0x8e, 0x4d,
+ 0x1c, 0xf7, 0xf2, 0xe7, 0xfa, 0xcf, 0xf3, 0x50, 0xe1, 0x67, 0x37, 0xa1, 0x39, 0xd2, 0x97, 0x6e,
+ 0xa3, 0xf4, 0xb9, 0x71, 0xf4, 0x3b, 0x50, 0x12, 0xd7, 0x29, 0x6c, 0x61, 0xe7, 0xf9, 0x22, 0xbe,
+ 0x90, 0x71, 0xf2, 0x9c, 0xfe, 0x4d, 0x7c, 0xa4, 0x17, 0xa9, 0x7c, 0x42, 0x6f, 0x42, 0xbe, 0x87,
+ 0x83, 0x69, 0xff, 0xb1, 0xe0, 0x40, 0xb7, 0x70, 0xe2, 0x7f, 0x00, 0x86, 0x82, 0xf6, 0x60, 0xde,
+ 0xf4, 0x7d, 0xec, 0x5a, 0x61, 0xf2, 0x77, 0x7d, 0x1a, 0xbc, 0x4d, 0xae, 0x1a, 0x43, 0x4a, 0x2c,
+ 0xf4, 0x55, 0x98, 0xeb, 0x38, 0xd8, 0x24, 0x61, 0x96, 0x77, 0x6d, 0x1a, 0xd0, 0x06, 0xd3, 0x8c,
+ 0x31, 0x05, 0x52, 0xf2, 0xff, 0x81, 0x8f, 0x73, 0xb0, 0x28, 0x3f, 0x8b, 0x8c, 0x4c, 0xe9, 0xef,
+ 0x32, 0xfe, 0x17, 0x81, 0x9d, 0x21, 0xc3, 0xbd, 0x34, 0xb5, 0xe1, 0xa2, 0x7b, 0x65, 0x6e, 0xb9,
+ 0xfb, 0x69, 0xcb, 0xbd, 0x3c, 0x8b, 0xe5, 0x22, 0xcc, 0xd0, 0x74, 0x7a, 0xca, 0x74, 0xd7, 0x67,
+ 0x30, 0x5d, 0x04, 0x2a, 0x6d, 0x97, 0xbc, 0xf7, 0xfe, 0x4b, 0x01, 0x8a, 0xa1, 0x53, 0xa1, 0x36,
+ 0xcc, 0x8b, 0xbf, 0xa4, 0x64, 0xea, 0xf3, 0xe2, 0x94, 0x5e, 0xa9, 0xe9, 0x5c, 0x9b, 0x0d, 0x5f,
+ 0xe0, 0x20, 0x0a, 0xab, 0xfd, 0x81, 0xc3, 0xf6, 0x3b, 0xdf, 0x18, 0x39, 0x83, 0xdd, 0x9c, 0x16,
+ 0xfe, 0x8e, 0x84, 0x4a, 0x1e, 0xba, 0xae, 0xf4, 0xd3, 0x9d, 0xc8, 0x82, 0xa5, 0x7d, 0xb3, 0x67,
+ 0x24, 0x8e, 0x99, 0xf3, 0x53, 0xfd, 0xa2, 0x11, 0xf1, 0x6d, 0x99, 0xbd, 0xe4, 0x91, 0x72, 0x65,
+ 0x3f, 0xd1, 0x56, 0x55, 0x98, 0x17, 0xd3, 0x4d, 0x6e, 0xd1, 0x15, 0xbe, 0x45, 0xab, 0xdf, 0x57,
+ 0x60, 0x65, 0x64, 0xb0, 0x59, 0x22, 0x7c, 0x1d, 0x16, 0x63, 0x33, 0x25, 0x42, 0x53, 0x74, 0x14,
+ 0xdc, 0xb2, 0xd0, 0x69, 0x98, 0x17, 0xd7, 0xcf, 0x32, 0x38, 0xc9, 0x56, 0x38, 0x8c, 0x42, 0x3c,
+ 0x8c, 0x0f, 0xa0, 0x92, 0x9c, 0x42, 0xc6, 0x01, 0xc4, 0x76, 0x4b, 0x0c, 0x20, 0x3a, 0x4d, 0x9f,
+ 0x66, 0x00, 0xd1, 0xb9, 0xf5, 0xeb, 0xb0, 0x9c, 0x0a, 0x38, 0xe8, 0x05, 0x40, 0x1d, 0xcf, 0x0d,
+ 0x6c, 0x77, 0x60, 0x8a, 0x6b, 0x18, 0x7e, 0x5c, 0x2e, 0x6c, 0xb8, 0x92, 0x94, 0xf0, 0x73, 0xf6,
+ 0xfa, 0x7d, 0xa8, 0xa6, 0x57, 0xde, 0x94, 0x10, 0x51, 0x48, 0xcf, 0x25, 0x42, 0xfa, 0x06, 0xa0,
+ 0xd1, 0xc8, 0x15, 0xbd, 0xa9, 0x24, 0xde, 0x5c, 0x87, 0xd5, 0x31, 0x2b, 0xb5, 0xbe, 0x0a, 0x2b,
+ 0x23, 0x51, 0xaa, 0xbe, 0x26, 0x51, 0x87, 0xd6, 0x5f, 0xfd, 0xd7, 0x05, 0x28, 0xee, 0x78, 0xf2,
+ 0x00, 0xe1, 0x1b, 0x50, 0xa4, 0xf8, 0x00, 0x13, 0x3b, 0x10, 0x8e, 0xb3, 0x94, 0xb9, 0x16, 0x0d,
+ 0x21, 0xb4, 0x5d, 0xa9, 0x2f, 0x2e, 0xf1, 0x22, 0xb8, 0xd9, 0x0b, 0x34, 0x54, 0x63, 0xb5, 0x0f,
+ 0xa5, 0x66, 0x2f, 0xac, 0x4c, 0xc3, 0x26, 0xbf, 0xcf, 0x20, 0xac, 0x94, 0x2d, 0x88, 0x08, 0xca,
+ 0x1b, 0x63, 0xf6, 0xbb, 0xb9, 0x2c, 0xdb, 0xed, 0xfc, 0xa8, 0xdb, 0x3d, 0x0b, 0x15, 0xc7, 0xeb,
+ 0x19, 0x8e, 0x27, 0xaf, 0xd1, 0x16, 0xc4, 0x2b, 0x8e, 0xd7, 0xdb, 0x91, 0x5d, 0xcc, 0xeb, 0x82,
+ 0x87, 0x04, 0x9b, 0x56, 0xad, 0xc8, 0x85, 0xb2, 0xa5, 0x7e, 0x1d, 0x0a, 0x3b, 0x36, 0x0d, 0x50,
+ 0x1b, 0xd8, 0xeb, 0x06, 0x76, 0x03, 0x62, 0xe3, 0x30, 0x19, 0xbd, 0x30, 0xa5, 0x51, 0x75, 0x70,
+ 0xc4, 0x93, 0x8d, 0xa9, 0x4a, 0xa0, 0x18, 0xda, 0xb8, 0xde, 0x85, 0x02, 0x33, 0x33, 0x5a, 0x86,
+ 0xf2, 0xfd, 0xbb, 0xbb, 0xed, 0x66, 0xa3, 0x75, 0xb3, 0xd5, 0xdc, 0xae, 0x9e, 0x42, 0x25, 0x98,
+ 0xdb, 0xd3, 0x37, 0x1b, 0xcd, 0xaa, 0xc2, 0x1e, 0xb7, 0x9b, 0x5b, 0xf7, 0x6f, 0x55, 0x73, 0xa8,
+ 0x08, 0x85, 0xd6, 0xdd, 0x9b, 0xf7, 0xaa, 0x79, 0x04, 0x30, 0x7f, 0xf7, 0xde, 0x5e, 0xab, 0xd1,
+ 0xac, 0x16, 0x58, 0xef, 0x83, 0x4d, 0xfd, 0x6e, 0x75, 0x8e, 0xbd, 0xda, 0xd4, 0xf5, 0x7b, 0x7a,
+ 0x75, 0x1e, 0x55, 0xa0, 0xd8, 0xd0, 0x5b, 0x7b, 0xad, 0xc6, 0xe6, 0x4e, 0x75, 0xa1, 0x5e, 0x01,
+ 0xd8, 0xf1, 0x7a, 0x0d, 0xcf, 0x0d, 0x88, 0xe7, 0xd4, 0xff, 0x5c, 0xe0, 0x9e, 0x44, 0x82, 0x07,
+ 0x1e, 0x79, 0x14, 0xff, 0x98, 0xf4, 0x7f, 0x50, 0x7a, 0xcc, 0x3b, 0xe2, 0x45, 0x5c, 0x14, 0x1d,
+ 0x2d, 0x0b, 0xed, 0x43, 0xb5, 0x23, 0xd4, 0x8d, 0xf0, 0x07, 0x57, 0xe9, 0x05, 0x33, 0xff, 0xa0,
+ 0xb1, 0x2c, 0x01, 0x9b, 0x12, 0x8f, 0x71, 0x38, 0x5e, 0xaf, 0xc7, 0xea, 0xda, 0x88, 0x23, 0x7f,
+ 0x4c, 0x0e, 0x09, 0x18, 0x71, 0x58, 0xb0, 0x62, 0x92, 0xc0, 0xee, 0x9a, 0x9d, 0x20, 0x26, 0x29,
+ 0x1c, 0x8f, 0xa4, 0x1a, 0x22, 0x46, 0x2c, 0x5d, 0x7e, 0x5b, 0x72, 0x60, 0x53, 0xe6, 0xc0, 0x11,
+ 0xcd, 0xdc, 0xf1, 0x68, 0x56, 0x22, 0xc8, 0x88, 0xe7, 0x1d, 0x98, 0xf7, 0x4d, 0x62, 0xf6, 0x69,
+ 0x0d, 0xb8, 0x63, 0x36, 0xb3, 0xef, 0x45, 0xa9, 0xaf, 0xaf, 0xb5, 0x39, 0x8e, 0xfc, 0x2f, 0x48,
+ 0x80, 0xaa, 0xd7, 0xa1, 0x9c, 0xe8, 0xfe, 0xac, 0xf3, 0xc5, 0x52, 0xb2, 0xca, 0xfb, 0x32, 0x0f,
+ 0x6c, 0x31, 0x89, 0x0c, 0xae, 0x51, 0xce, 0xa4, 0x24, 0x72, 0xa6, 0xfa, 0x45, 0x16, 0xee, 0x3c,
+ 0x3f, 0xbb, 0x3b, 0xd6, 0x9f, 0x67, 0x1e, 0x1c, 0x6b, 0x3c, 0x0d, 0xfd, 0xf2, 0x47, 0x0a, 0x2c,
+ 0x6e, 0x61, 0xb3, 0x7f, 0xd3, 0x95, 0x0b, 0x00, 0xfd, 0x40, 0x81, 0x85, 0xf0, 0x39, 0x6b, 0x42,
+ 0x35, 0xe6, 0x5f, 0x52, 0xf5, 0xfa, 0x2c, 0xba, 0x22, 0x98, 0x9f, 0xda, 0x50, 0x2e, 0x2a, 0x97,
+ 0xdf, 0x07, 0x10, 0x23, 0xe3, 0x75, 0x86, 0x2b, 0xeb, 0x8d, 0x0b, 0x53, 0xd6, 0x2c, 0xea, 0xb4,
+ 0x0a, 0x92, 0xfd, 0x87, 0x0a, 0x94, 0x05, 0xbd, 0xd8, 0xc8, 0x0f, 0x61, 0x4e, 0x3c, 0x5c, 0x99,
+ 0x26, 0xa5, 0x91, 0x33, 0x52, 0xaf, 0x4e, 0xa7, 0x24, 0xb7, 0x2f, 0x31, 0x92, 0x1f, 0x45, 0x9f,
+ 0x68, 0x47, 0xac, 0x57, 0x74, 0x08, 0x0b, 0xe1, 0xe3, 0xd5, 0x69, 0xb7, 0x30, 0x16, 0xb8, 0xd5,
+ 0x4b, 0xd9, 0xb5, 0xc2, 0xb8, 0x28, 0xc6, 0xf2, 0xbb, 0x1c, 0xd4, 0xc4, 0x58, 0x9a, 0x87, 0x01,
+ 0x26, 0xae, 0xe9, 0x08, 0x2f, 0x6b, 0x7b, 0xc2, 0x73, 0xca, 0x09, 0xbf, 0x46, 0xd7, 0x67, 0x5e,
+ 0x70, 0xea, 0xcb, 0xb3, 0xa8, 0x86, 0x56, 0x43, 0xdf, 0x53, 0x00, 0xe2, 0x15, 0x80, 0xb2, 0xd7,
+ 0x3e, 0xa9, 0x65, 0xa6, 0x5e, 0x9f, 0x41, 0x33, 0x1c, 0xc5, 0xd6, 0x26, 0x7c, 0x69, 0x92, 0x76,
+ 0x52, 0x79, 0xab, 0x24, 0x0c, 0xba, 0xe9, 0xdb, 0x6f, 0x2d, 0x25, 0x44, 0xc6, 0xc1, 0xa5, 0xfd,
+ 0x79, 0x9e, 0x3c, 0x5c, 0xf9, 0x6f, 0x00, 0x00, 0x00, 0xff, 0xff, 0x0f, 0x46, 0x65, 0x7e, 0x89,
+ 0x31, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go
index 9988c16..d2c8261 100644
--- a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go
+++ b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_artifact_api.pb.go
@@ -29,9 +29,6 @@
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
// (Optional) The Unix-like permissions of the artifact
Permissions uint32 `protobuf:"varint,2,opt,name=permissions,proto3" json:"permissions,omitempty"`
- // (Optional) The base64-encoded md5 checksum of the artifact. Used, among other things, by
- // harness boot code to validate the integrity of the artifact.
- Md5X string `protobuf:"bytes,3,opt,name=md5X,proto3" json:"md5X,omitempty"`
// (Optional) The hex-encoded sha256 checksum of the artifact. Used, among other things, by
// harness boot code to validate the integrity of the artifact.
Sha256 string `protobuf:"bytes,4,opt,name=sha256,proto3" json:"sha256,omitempty"`
@@ -44,7 +41,7 @@
func (m *ArtifactMetadata) String() string { return proto.CompactTextString(m) }
func (*ArtifactMetadata) ProtoMessage() {}
func (*ArtifactMetadata) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{0}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{0}
}
func (m *ArtifactMetadata) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ArtifactMetadata.Unmarshal(m, b)
@@ -78,13 +75,6 @@
return 0
}
-func (m *ArtifactMetadata) GetMd5X() string {
- if m != nil {
- return m.Md5X
- }
- return ""
-}
-
func (m *ArtifactMetadata) GetSha256() string {
if m != nil {
return m.Sha256
@@ -104,7 +94,7 @@
func (m *Manifest) String() string { return proto.CompactTextString(m) }
func (*Manifest) ProtoMessage() {}
func (*Manifest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{1}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{1}
}
func (m *Manifest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Manifest.Unmarshal(m, b)
@@ -144,7 +134,7 @@
func (m *ProxyManifest) String() string { return proto.CompactTextString(m) }
func (*ProxyManifest) ProtoMessage() {}
func (*ProxyManifest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{2}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{2}
}
func (m *ProxyManifest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProxyManifest.Unmarshal(m, b)
@@ -190,7 +180,7 @@
func (m *ProxyManifest_Location) String() string { return proto.CompactTextString(m) }
func (*ProxyManifest_Location) ProtoMessage() {}
func (*ProxyManifest_Location) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{2, 0}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{2, 0}
}
func (m *ProxyManifest_Location) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProxyManifest_Location.Unmarshal(m, b)
@@ -238,7 +228,7 @@
func (m *GetManifestRequest) String() string { return proto.CompactTextString(m) }
func (*GetManifestRequest) ProtoMessage() {}
func (*GetManifestRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{3}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{3}
}
func (m *GetManifestRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetManifestRequest.Unmarshal(m, b)
@@ -277,7 +267,7 @@
func (m *GetManifestResponse) String() string { return proto.CompactTextString(m) }
func (*GetManifestResponse) ProtoMessage() {}
func (*GetManifestResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{4}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{4}
}
func (m *GetManifestResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetManifestResponse.Unmarshal(m, b)
@@ -320,7 +310,7 @@
func (m *GetArtifactRequest) String() string { return proto.CompactTextString(m) }
func (*GetArtifactRequest) ProtoMessage() {}
func (*GetArtifactRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{5}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{5}
}
func (m *GetArtifactRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetArtifactRequest.Unmarshal(m, b)
@@ -366,7 +356,7 @@
func (m *ArtifactChunk) String() string { return proto.CompactTextString(m) }
func (*ArtifactChunk) ProtoMessage() {}
func (*ArtifactChunk) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{6}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{6}
}
func (m *ArtifactChunk) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ArtifactChunk.Unmarshal(m, b)
@@ -408,7 +398,7 @@
func (m *PutArtifactMetadata) String() string { return proto.CompactTextString(m) }
func (*PutArtifactMetadata) ProtoMessage() {}
func (*PutArtifactMetadata) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{7}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{7}
}
func (m *PutArtifactMetadata) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PutArtifactMetadata.Unmarshal(m, b)
@@ -459,7 +449,7 @@
func (m *PutArtifactRequest) String() string { return proto.CompactTextString(m) }
func (*PutArtifactRequest) ProtoMessage() {}
func (*PutArtifactRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{8}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{8}
}
func (m *PutArtifactRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PutArtifactRequest.Unmarshal(m, b)
@@ -598,7 +588,7 @@
func (m *PutArtifactResponse) String() string { return proto.CompactTextString(m) }
func (*PutArtifactResponse) ProtoMessage() {}
func (*PutArtifactResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{9}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{9}
}
func (m *PutArtifactResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PutArtifactResponse.Unmarshal(m, b)
@@ -635,7 +625,7 @@
func (m *CommitManifestRequest) String() string { return proto.CompactTextString(m) }
func (*CommitManifestRequest) ProtoMessage() {}
func (*CommitManifestRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{10}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{10}
}
func (m *CommitManifestRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CommitManifestRequest.Unmarshal(m, b)
@@ -684,7 +674,7 @@
func (m *CommitManifestResponse) String() string { return proto.CompactTextString(m) }
func (*CommitManifestResponse) ProtoMessage() {}
func (*CommitManifestResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_artifact_api_d51e52a8ee148278, []int{11}
+ return fileDescriptor_beam_artifact_api_09b5b695a8be46db, []int{11}
}
func (m *CommitManifestResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CommitManifestResponse.Unmarshal(m, b)
@@ -1011,49 +1001,48 @@
}
func init() {
- proto.RegisterFile("beam_artifact_api.proto", fileDescriptor_beam_artifact_api_d51e52a8ee148278)
+ proto.RegisterFile("beam_artifact_api.proto", fileDescriptor_beam_artifact_api_09b5b695a8be46db)
}
-var fileDescriptor_beam_artifact_api_d51e52a8ee148278 = []byte{
- // 626 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xac, 0x56, 0xc1, 0x6e, 0xd3, 0x40,
- 0x10, 0xed, 0xba, 0x55, 0x49, 0xc7, 0xb4, 0x54, 0x5b, 0xb5, 0x58, 0x39, 0x45, 0x46, 0xa2, 0x39,
- 0x59, 0xad, 0x51, 0x2b, 0x21, 0x0a, 0x55, 0xdb, 0x43, 0x7b, 0x68, 0xa4, 0xe2, 0x82, 0x84, 0xca,
- 0xc1, 0xda, 0x24, 0xdb, 0x64, 0x69, 0xbc, 0x6b, 0xec, 0x4d, 0x04, 0x77, 0x0e, 0x88, 0x1b, 0x57,
- 0x4e, 0x7c, 0x00, 0x3f, 0xc0, 0x27, 0xf0, 0x31, 0xfc, 0x03, 0xf2, 0xda, 0xeb, 0xc6, 0x8d, 0x23,
- 0x39, 0xa1, 0xb7, 0xc9, 0x6c, 0xde, 0x9b, 0x37, 0x6f, 0x66, 0x57, 0x86, 0xc7, 0x6d, 0x4a, 0x02,
- 0x9f, 0x44, 0x92, 0x5d, 0x93, 0x8e, 0xf4, 0x49, 0xc8, 0x9c, 0x30, 0x12, 0x52, 0xe0, 0x6d, 0x11,
- 0xf5, 0x1c, 0x12, 0x92, 0x4e, 0x9f, 0x3a, 0xc9, 0x7f, 0x9c, 0x40, 0x74, 0xe9, 0xc0, 0xf9, 0x20,
- 0xda, 0x7e, 0x40, 0x38, 0xe9, 0xd1, 0x80, 0x72, 0xe9, 0x8c, 0x76, 0x6d, 0x09, 0xeb, 0x47, 0x19,
- 0xbc, 0x45, 0x25, 0xe9, 0x12, 0x49, 0x30, 0x86, 0x25, 0x4e, 0x02, 0x6a, 0xa1, 0x06, 0x6a, 0xae,
- 0x78, 0x2a, 0xc6, 0x0d, 0x30, 0x43, 0x1a, 0x05, 0x2c, 0x8e, 0x99, 0xe0, 0xb1, 0x65, 0x34, 0x50,
- 0x73, 0xd5, 0x1b, 0x4f, 0x25, 0xa8, 0xa0, 0xbb, 0xf7, 0xce, 0x5a, 0x4c, 0x51, 0x49, 0x8c, 0xb7,
- 0x60, 0x39, 0xee, 0x13, 0x77, 0x6f, 0xdf, 0x5a, 0x52, 0xd9, 0xec, 0x97, 0x4d, 0xa0, 0xd6, 0x22,
- 0x9c, 0x5d, 0xd3, 0x58, 0xe2, 0xb7, 0x50, 0xd3, 0x0d, 0x58, 0xa8, 0xb1, 0xd8, 0x34, 0xdd, 0xe7,
- 0x4e, 0x45, 0xf5, 0xce, 0x5d, 0xe9, 0x5e, 0x4e, 0x65, 0xff, 0x45, 0xb0, 0x7a, 0x11, 0x89, 0x4f,
- 0x9f, 0xf3, 0x42, 0x2d, 0xa8, 0x05, 0x59, 0xac, 0x5a, 0x33, 0xdd, 0xdd, 0xca, 0x85, 0x34, 0x89,
- 0x97, 0x53, 0xe0, 0xf7, 0x50, 0x1b, 0x88, 0x0e, 0x91, 0x4c, 0x70, 0xcb, 0x50, 0xba, 0x0f, 0x2b,
- 0xd3, 0x15, 0x84, 0x39, 0xe7, 0x19, 0x8d, 0x97, 0x13, 0xd6, 0x77, 0xa0, 0xa6, 0xb3, 0xa5, 0xe3,
- 0x58, 0x87, 0xc5, 0x61, 0xc4, 0xd4, 0x18, 0x56, 0xbc, 0x24, 0xb4, 0x5f, 0x02, 0x3e, 0xa5, 0x32,
- 0xd7, 0x49, 0x3f, 0x0e, 0x13, 0x91, 0xdb, 0xf0, 0x28, 0xa2, 0x32, 0x62, 0x74, 0x44, 0x06, 0xbe,
- 0x14, 0x37, 0x94, 0x67, 0x34, 0x6b, 0x79, 0xfa, 0x4d, 0x92, 0xb5, 0xbb, 0xb0, 0x51, 0x80, 0xc7,
- 0xa1, 0xe0, 0x31, 0xbd, 0x67, 0xcf, 0xec, 0xd7, 0x4a, 0xa4, 0x9e, 0x9a, 0x16, 0x59, 0xd6, 0x60,
- 0x89, 0x70, 0xa3, 0x54, 0xf8, 0x13, 0x58, 0xd5, 0x7c, 0x27, 0xfd, 0x21, 0xbf, 0x49, 0xd8, 0x92,
- 0x55, 0x50, 0x6c, 0x0f, 0x3d, 0x15, 0xdb, 0x3f, 0x11, 0x6c, 0x5c, 0x0c, 0xe5, 0xc4, 0xa6, 0xbb,
- 0xb0, 0x19, 0x4b, 0xd2, 0x63, 0xbc, 0xe7, 0xc7, 0x54, 0xed, 0x71, 0xc1, 0xa4, 0x8d, 0xec, 0xf0,
- 0x32, 0x3d, 0x53, 0x05, 0x93, 0x7d, 0x0d, 0x32, 0xbc, 0x92, 0xf4, 0x7f, 0xfb, 0xaa, 0xa9, 0xec,
- 0x3f, 0x08, 0xf0, 0x98, 0x44, 0xed, 0xcd, 0xd5, 0x58, 0xb5, 0x74, 0x00, 0x07, 0xd5, 0xb7, 0x6c,
- 0xb2, 0xe3, 0xb3, 0x85, 0xdb, 0x92, 0xf8, 0x3c, 0x73, 0x2a, 0xed, 0x62, 0x7f, 0xe6, 0x2e, 0x94,
- 0xdf, 0x67, 0x0b, 0xa9, 0xc7, 0xc7, 0x2b, 0xf0, 0xa0, 0x23, 0xb8, 0xa4, 0x5c, 0xda, 0x9b, 0x05,
- 0xb7, 0xf5, 0x32, 0xd9, 0x3f, 0x10, 0x6c, 0x9e, 0x88, 0x20, 0x60, 0x13, 0x6b, 0x7a, 0xcf, 0x57,
- 0x73, 0xea, 0x58, 0x8d, 0xa9, 0x63, 0xb5, 0x8f, 0x60, 0xeb, 0xae, 0xb6, 0xec, 0x0e, 0x54, 0xbd,
- 0x43, 0xee, 0x6f, 0x03, 0xb6, 0x74, 0xd3, 0x97, 0xba, 0x44, 0x34, 0x62, 0x1d, 0x8a, 0xbf, 0x21,
- 0x30, 0xc7, 0x2c, 0xc1, 0x2f, 0xe6, 0x19, 0x62, 0xe6, 0x56, 0xfd, 0x60, 0x3e, 0x70, 0xda, 0x4e,
- 0x13, 0xe1, 0xef, 0x08, 0xd6, 0x8a, 0xbd, 0xe2, 0x57, 0x95, 0x29, 0x4b, 0x07, 0x58, 0x3f, 0x9c,
- 0x1b, 0x9f, 0xaa, 0x72, 0x7f, 0x19, 0x60, 0xdd, 0x4a, 0xcd, 0x6c, 0xd5, 0xee, 0x7d, 0x45, 0x60,
- 0x8e, 0xbd, 0x4e, 0x33, 0xb8, 0x37, 0xf9, 0x24, 0xce, 0xe0, 0x5e, 0xd9, 0x83, 0xf8, 0x25, 0x95,
- 0x32, 0xc7, 0x20, 0x27, 0x1f, 0xbe, 0xfa, 0x9c, 0x57, 0x6e, 0x07, 0x1d, 0x9f, 0xc2, 0xd3, 0xa9,
- 0xd0, 0x02, 0xf2, 0xd8, 0xd4, 0xd0, 0xa3, 0x90, 0x5d, 0xad, 0x17, 0x8e, 0xfd, 0xd1, 0x6e, 0x7b,
- 0x59, 0x7d, 0x2f, 0x3c, 0xfb, 0x17, 0x00, 0x00, 0xff, 0xff, 0x03, 0x0a, 0x3b, 0x02, 0x4a, 0x08,
- 0x00, 0x00,
+var fileDescriptor_beam_artifact_api_09b5b695a8be46db = []byte{
+ // 618 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xac, 0x96, 0xc1, 0x6e, 0xd3, 0x4c,
+ 0x10, 0xc7, 0xbb, 0x6e, 0xd5, 0x2f, 0x1d, 0x7f, 0x2d, 0xd5, 0x56, 0x2d, 0x56, 0x4e, 0x91, 0x91,
+ 0x68, 0x4e, 0x56, 0x6b, 0x44, 0x25, 0x44, 0xa1, 0x6a, 0x7a, 0x68, 0x0f, 0x8d, 0x54, 0x5c, 0xb8,
+ 0x94, 0x83, 0xd9, 0x24, 0xdb, 0x64, 0x69, 0xbc, 0x6b, 0xec, 0x4d, 0x04, 0x77, 0x0e, 0x88, 0x1b,
+ 0x57, 0x4e, 0x3c, 0x00, 0x2f, 0xc0, 0x23, 0xf0, 0x30, 0xbc, 0x03, 0xf2, 0xda, 0xeb, 0xc6, 0x8d,
+ 0x23, 0x39, 0xa1, 0xb7, 0xcd, 0x6e, 0xe6, 0x3f, 0xbf, 0xf9, 0xcf, 0xec, 0xca, 0xf0, 0xb0, 0x43,
+ 0x49, 0xe0, 0x93, 0x48, 0xb2, 0x6b, 0xd2, 0x95, 0x3e, 0x09, 0x99, 0x13, 0x46, 0x42, 0x0a, 0xbc,
+ 0x2b, 0xa2, 0xbe, 0x43, 0x42, 0xd2, 0x1d, 0x50, 0x27, 0xf9, 0x8f, 0x13, 0x88, 0x1e, 0x1d, 0x3a,
+ 0xef, 0x45, 0xc7, 0x0f, 0x08, 0x27, 0x7d, 0x1a, 0x50, 0x2e, 0x9d, 0xf1, 0xbe, 0xfd, 0x0e, 0x36,
+ 0x8f, 0xb3, 0xf0, 0x36, 0x95, 0xa4, 0x47, 0x24, 0xc1, 0x18, 0x56, 0x38, 0x09, 0xa8, 0x85, 0x1a,
+ 0xa8, 0xb9, 0xe6, 0xa9, 0x35, 0x6e, 0x80, 0x19, 0xd2, 0x28, 0x60, 0x71, 0xcc, 0x04, 0x8f, 0x2d,
+ 0xa3, 0x81, 0x9a, 0xeb, 0xde, 0xe4, 0x16, 0xde, 0x81, 0xd5, 0x78, 0x40, 0xdc, 0xa7, 0x07, 0xd6,
+ 0x8a, 0x8a, 0xcb, 0x7e, 0xd9, 0x04, 0x6a, 0x6d, 0xc2, 0xd9, 0x35, 0x8d, 0x25, 0x7e, 0x03, 0x35,
+ 0x0d, 0x6b, 0xa1, 0xc6, 0x72, 0xd3, 0x74, 0x9f, 0x39, 0x15, 0x49, 0x9d, 0xbb, 0x98, 0x5e, 0x2e,
+ 0x65, 0xff, 0x41, 0xb0, 0x7e, 0x11, 0x89, 0x8f, 0x9f, 0xf2, 0x44, 0x6d, 0xa8, 0x05, 0xd9, 0x5a,
+ 0x95, 0x61, 0xba, 0xfb, 0x95, 0x13, 0x69, 0x11, 0x2f, 0x97, 0xc0, 0x6f, 0xa1, 0x36, 0x14, 0x5d,
+ 0x22, 0x99, 0xe0, 0x96, 0xa1, 0xb8, 0x8f, 0x2a, 0xcb, 0x15, 0xc0, 0x9c, 0xf3, 0x4c, 0xc6, 0xcb,
+ 0x05, 0xeb, 0x7b, 0x50, 0xd3, 0xbb, 0xa5, 0xd6, 0x6f, 0xc2, 0xf2, 0x28, 0x62, 0xca, 0xf2, 0x35,
+ 0x2f, 0x59, 0xda, 0x2f, 0x00, 0x9f, 0x52, 0x99, 0x73, 0xd2, 0x0f, 0xa3, 0x04, 0x72, 0x17, 0x1e,
+ 0x44, 0x54, 0x46, 0x8c, 0x8e, 0xc9, 0xd0, 0x97, 0xe2, 0x86, 0xf2, 0x4c, 0x66, 0x23, 0xdf, 0x7e,
+ 0x9d, 0xec, 0xda, 0x3d, 0xd8, 0x2a, 0x84, 0xc7, 0xa1, 0xe0, 0x31, 0xbd, 0x67, 0xcf, 0xec, 0x57,
+ 0x0a, 0x52, 0x77, 0x4d, 0x43, 0x96, 0x15, 0x58, 0x02, 0x6e, 0x94, 0x82, 0x3f, 0x82, 0x75, 0xad,
+ 0x77, 0x32, 0x18, 0xf1, 0x9b, 0x44, 0x2d, 0x19, 0x05, 0xa5, 0xf6, 0xbf, 0xa7, 0xd6, 0xf6, 0x0f,
+ 0x04, 0x5b, 0x17, 0x23, 0x39, 0x35, 0xd5, 0x2e, 0x6c, 0xc7, 0x92, 0xf4, 0x19, 0xef, 0xfb, 0x31,
+ 0x55, 0x33, 0x5b, 0x30, 0x69, 0x2b, 0x3b, 0xbc, 0x4c, 0xcf, 0x54, 0xc2, 0x64, 0x5e, 0x83, 0x2c,
+ 0x5e, 0x21, 0xfd, 0xdb, 0xbc, 0x6a, 0x29, 0xfb, 0x37, 0x02, 0x3c, 0x81, 0xa8, 0xbd, 0xb9, 0x9a,
+ 0xc8, 0x96, 0x36, 0xe0, 0xb0, 0xfa, 0x94, 0x4d, 0x57, 0x7c, 0xb6, 0x74, 0x9b, 0x12, 0x9f, 0x67,
+ 0x4e, 0xa5, 0x55, 0x1c, 0xcc, 0x5d, 0x85, 0xf2, 0xfb, 0x6c, 0x29, 0xf5, 0xb8, 0xb5, 0x06, 0xff,
+ 0x75, 0x05, 0x97, 0x94, 0x4b, 0x7b, 0xbb, 0xe0, 0xb6, 0x1e, 0x26, 0xfb, 0x3b, 0x82, 0xed, 0x13,
+ 0x11, 0x04, 0x6c, 0x6a, 0x4c, 0xef, 0xf9, 0x6a, 0xce, 0x6c, 0xab, 0x31, 0xb3, 0xad, 0xf6, 0x31,
+ 0xec, 0xdc, 0x65, 0xcb, 0xee, 0x40, 0xd5, 0x3b, 0xe4, 0xfe, 0x32, 0x60, 0x47, 0x17, 0x7d, 0xa9,
+ 0x53, 0x44, 0x63, 0xd6, 0xa5, 0xf8, 0x2b, 0x02, 0x73, 0xc2, 0x12, 0xfc, 0x7c, 0x91, 0x26, 0x66,
+ 0x6e, 0xd5, 0x0f, 0x17, 0x0b, 0x4e, 0xcb, 0x69, 0x22, 0xfc, 0x0d, 0xc1, 0x46, 0xb1, 0x56, 0xfc,
+ 0xb2, 0xb2, 0x64, 0x69, 0x03, 0xeb, 0x47, 0x0b, 0xc7, 0xa7, 0x54, 0xee, 0x4f, 0x03, 0xac, 0x5b,
+ 0xd4, 0xcc, 0x56, 0xed, 0xde, 0x17, 0x04, 0xe6, 0xc4, 0xeb, 0x34, 0x87, 0x7b, 0xd3, 0x4f, 0xe2,
+ 0x1c, 0xee, 0x95, 0x3d, 0x88, 0x9f, 0x53, 0x94, 0x05, 0x1a, 0x39, 0xfd, 0xf0, 0xd5, 0x17, 0xbc,
+ 0x72, 0x7b, 0xa8, 0x75, 0x0a, 0x8f, 0x67, 0x86, 0x16, 0x22, 0x5b, 0xa6, 0x0e, 0x3d, 0x0e, 0xd9,
+ 0xd5, 0x66, 0xe1, 0xd8, 0x1f, 0xef, 0x77, 0x56, 0xd5, 0xb7, 0xc1, 0x93, 0xbf, 0x01, 0x00, 0x00,
+ 0xff, 0xff, 0xb2, 0x30, 0x58, 0x4f, 0x36, 0x08, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_expansion_api.pb.go b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_expansion_api.pb.go
index 761ea743..61718ce 100644
--- a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_expansion_api.pb.go
+++ b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_expansion_api.pb.go
@@ -46,7 +46,7 @@
func (m *ExpansionRequest) String() string { return proto.CompactTextString(m) }
func (*ExpansionRequest) ProtoMessage() {}
func (*ExpansionRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_expansion_api_493bf0692a755443, []int{0}
+ return fileDescriptor_beam_expansion_api_7d6074648ff0899a, []int{0}
}
func (m *ExpansionRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExpansionRequest.Unmarshal(m, b)
@@ -106,7 +106,7 @@
func (m *ExpansionResponse) String() string { return proto.CompactTextString(m) }
func (*ExpansionResponse) ProtoMessage() {}
func (*ExpansionResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_expansion_api_493bf0692a755443, []int{1}
+ return fileDescriptor_beam_expansion_api_7d6074648ff0899a, []int{1}
}
func (m *ExpansionResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExpansionResponse.Unmarshal(m, b)
@@ -225,28 +225,27 @@
}
func init() {
- proto.RegisterFile("beam_expansion_api.proto", fileDescriptor_beam_expansion_api_493bf0692a755443)
+ proto.RegisterFile("beam_expansion_api.proto", fileDescriptor_beam_expansion_api_7d6074648ff0899a)
}
-var fileDescriptor_beam_expansion_api_493bf0692a755443 = []byte{
- // 291 bytes of a gzipped FileDescriptorProto
+var fileDescriptor_beam_expansion_api_7d6074648ff0899a = []byte{
+ // 285 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xcc, 0x92, 0xc1, 0x4a, 0xc3, 0x40,
- 0x10, 0x86, 0x89, 0x62, 0xa1, 0xab, 0x87, 0xba, 0x28, 0x84, 0xe2, 0xa1, 0xe4, 0x20, 0xbd, 0x38,
- 0x90, 0xaa, 0x0f, 0x60, 0x45, 0x10, 0x44, 0x90, 0xe8, 0xc9, 0x4b, 0xd8, 0xa4, 0x63, 0x5d, 0xe9,
- 0xce, 0xac, 0xbb, 0x49, 0xf0, 0x15, 0x7c, 0x30, 0x6f, 0x3e, 0x94, 0x98, 0xd2, 0x24, 0x4a, 0x41,
- 0xbd, 0x79, 0xdc, 0xd9, 0xff, 0xfb, 0x97, 0x7f, 0xff, 0x11, 0x61, 0x86, 0xca, 0xa4, 0xf8, 0x62,
- 0x15, 0x79, 0xcd, 0x94, 0x2a, 0xab, 0xc1, 0x3a, 0x2e, 0x58, 0x46, 0xec, 0xe6, 0xa0, 0xac, 0xca,
- 0x1f, 0x11, 0x3e, 0x45, 0x60, 0x78, 0x86, 0x0b, 0x68, 0xa4, 0x50, 0xc5, 0xc3, 0xfd, 0x9a, 0x76,
- 0x25, 0x11, 0xba, 0x16, 0x8d, 0xde, 0x03, 0x31, 0xb8, 0x58, 0xe9, 0x12, 0x7c, 0x2e, 0xd1, 0x17,
- 0xf2, 0x5a, 0x88, 0x9c, 0x8d, 0x65, 0x42, 0x2a, 0x7c, 0x18, 0x8c, 0x82, 0xf1, 0xf6, 0xe4, 0x08,
- 0xd6, 0x3f, 0x62, 0xb5, 0xc5, 0x85, 0x26, 0x84, 0x2a, 0x86, 0xf3, 0x06, 0x4a, 0x3a, 0x06, 0xf2,
- 0x4a, 0xf4, 0x0b, 0xa7, 0xc8, 0x3f, 0xb0, 0x33, 0xe1, 0xc6, 0xaf, 0xdd, 0x6e, 0xee, 0x56, 0x50,
- 0xd2, 0xf2, 0xf2, 0x40, 0xf4, 0x49, 0x19, 0xf4, 0x56, 0xe5, 0x18, 0x6e, 0x8e, 0x82, 0x71, 0x3f,
- 0x69, 0x07, 0xd1, 0x5b, 0x20, 0x76, 0x3b, 0x71, 0xbc, 0x65, 0xf2, 0xf8, 0xaf, 0xf3, 0xec, 0x89,
- 0x2d, 0x74, 0x8e, 0x5d, 0x28, 0xea, 0x2c, 0xcb, 0xc3, 0xe4, 0xb5, 0x5b, 0xcb, 0x2d, 0xba, 0x4a,
- 0xe7, 0x28, 0x4b, 0xd1, 0xab, 0x67, 0x33, 0x79, 0x02, 0x3f, 0x37, 0x0e, 0xdf, 0x6b, 0x1d, 0x9e,
- 0xfe, 0x91, 0x5a, 0xfe, 0xde, 0xf4, 0x52, 0x1c, 0xae, 0xe7, 0x9e, 0x38, 0x33, 0x8a, 0xd4, 0x1c,
- 0x0d, 0x52, 0x01, 0x55, 0x3c, 0xdd, 0x69, 0xe0, 0x33, 0xab, 0xef, 0x07, 0x5f, 0xee, 0xd3, 0x2a,
- 0xce, 0x7a, 0xf5, 0xce, 0x1d, 0x7f, 0x04, 0x00, 0x00, 0xff, 0xff, 0xa4, 0x2a, 0x51, 0x65, 0xca,
- 0x02, 0x00, 0x00,
+ 0x10, 0x86, 0x89, 0x62, 0x21, 0xab, 0x87, 0xba, 0x28, 0x84, 0xe2, 0xa1, 0xe4, 0xd4, 0x8b, 0x03,
+ 0xa9, 0xfa, 0x00, 0x56, 0x3d, 0x89, 0x20, 0xd1, 0x93, 0x97, 0xb0, 0x49, 0xc7, 0xba, 0xd2, 0x9d,
+ 0x59, 0x77, 0x93, 0xe0, 0x2b, 0xf8, 0x60, 0xde, 0x7c, 0x28, 0x31, 0xa5, 0x49, 0x10, 0xc1, 0x7a,
+ 0xf3, 0xb8, 0xc3, 0xff, 0xcd, 0xf2, 0xcf, 0xff, 0x8b, 0x28, 0x47, 0x65, 0x32, 0x7c, 0xb5, 0x8a,
+ 0xbc, 0x66, 0xca, 0x94, 0xd5, 0x60, 0x1d, 0x97, 0x2c, 0x63, 0x76, 0x0b, 0x50, 0x56, 0x15, 0x4f,
+ 0x08, 0x5f, 0x22, 0x30, 0x3c, 0xc7, 0x25, 0xb4, 0x52, 0xa8, 0x93, 0xd1, 0x61, 0x43, 0xbb, 0x8a,
+ 0x08, 0x5d, 0x87, 0xc6, 0x1f, 0x81, 0x18, 0x5e, 0xad, 0x75, 0x29, 0xbe, 0x54, 0xe8, 0x4b, 0x79,
+ 0x23, 0x44, 0xc1, 0xc6, 0x32, 0x21, 0x95, 0x3e, 0x0a, 0xc6, 0xc1, 0x64, 0x77, 0x7a, 0x0c, 0x3f,
+ 0x7f, 0x62, 0xb5, 0xc5, 0xa5, 0x26, 0x84, 0x3a, 0x81, 0x8b, 0x16, 0x4a, 0x7b, 0x0b, 0xe4, 0xb5,
+ 0x08, 0x4b, 0xa7, 0xc8, 0x3f, 0xb2, 0x33, 0xd1, 0xd6, 0xc6, 0xdb, 0x6e, 0xef, 0xd7, 0x50, 0xda,
+ 0xf1, 0xf2, 0x48, 0x84, 0xa4, 0x0c, 0x7a, 0xab, 0x0a, 0x8c, 0xb6, 0xc7, 0xc1, 0x24, 0x4c, 0xbb,
+ 0x41, 0xfc, 0x1e, 0x88, 0xfd, 0x9e, 0x1d, 0x6f, 0x99, 0x3c, 0xfe, 0x6b, 0x3f, 0x07, 0x62, 0x07,
+ 0x9d, 0x63, 0x17, 0x89, 0xc6, 0xcb, 0xea, 0x31, 0x7d, 0xeb, 0xc7, 0x72, 0x87, 0xae, 0xd6, 0x05,
+ 0xca, 0x4a, 0x0c, 0x9a, 0xd9, 0x5c, 0x9e, 0xc2, 0xef, 0x89, 0xc3, 0xf7, 0x58, 0x47, 0x67, 0x7f,
+ 0xa4, 0x56, 0xd7, 0x9b, 0x5d, 0x8a, 0x0d, 0xfa, 0x35, 0xdb, 0x6b, 0xc1, 0x73, 0xab, 0x1f, 0x86,
+ 0xcf, 0x9c, 0x1b, 0x45, 0x6a, 0x81, 0x06, 0xa9, 0xcc, 0xea, 0x24, 0x1f, 0x34, 0x7d, 0x3b, 0xf9,
+ 0x0c, 0x00, 0x00, 0xff, 0xff, 0xb4, 0xc3, 0x3e, 0x66, 0xc6, 0x02, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go
index d5f95d9..55f8b16 100644
--- a/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go
+++ b/sdks/go/pkg/beam/model/jobmanagement_v1/beam_job_api.pb.go
@@ -57,7 +57,7 @@
return proto.EnumName(JobMessage_MessageImportance_name, int32(x))
}
func (JobMessage_MessageImportance) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{9, 0}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{14, 0}
}
type JobState_Enum int32
@@ -107,7 +107,7 @@
return proto.EnumName(JobState_Enum_name, int32(x))
}
func (JobState_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{11, 0}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{16, 0}
}
type PipelineOptionType_Enum int32
@@ -143,7 +143,7 @@
return proto.EnumName(PipelineOptionType_Enum_name, int32(x))
}
func (PipelineOptionType_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{16, 0}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{21, 0}
}
// Prepare is a synchronous request that returns a preparationId back
@@ -163,7 +163,7 @@
func (m *PrepareJobRequest) String() string { return proto.CompactTextString(m) }
func (*PrepareJobRequest) ProtoMessage() {}
func (*PrepareJobRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{0}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{0}
}
func (m *PrepareJobRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PrepareJobRequest.Unmarshal(m, b)
@@ -223,7 +223,7 @@
func (m *PrepareJobResponse) String() string { return proto.CompactTextString(m) }
func (*PrepareJobResponse) ProtoMessage() {}
func (*PrepareJobResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{1}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{1}
}
func (m *PrepareJobResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PrepareJobResponse.Unmarshal(m, b)
@@ -284,7 +284,7 @@
func (m *RunJobRequest) String() string { return proto.CompactTextString(m) }
func (*RunJobRequest) ProtoMessage() {}
func (*RunJobRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{2}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{2}
}
func (m *RunJobRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RunJobRequest.Unmarshal(m, b)
@@ -329,7 +329,7 @@
func (m *RunJobResponse) String() string { return proto.CompactTextString(m) }
func (*RunJobResponse) ProtoMessage() {}
func (*RunJobResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{3}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{3}
}
func (m *RunJobResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_RunJobResponse.Unmarshal(m, b)
@@ -370,7 +370,7 @@
func (m *CancelJobRequest) String() string { return proto.CompactTextString(m) }
func (*CancelJobRequest) ProtoMessage() {}
func (*CancelJobRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{4}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{4}
}
func (m *CancelJobRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CancelJobRequest.Unmarshal(m, b)
@@ -409,7 +409,7 @@
func (m *CancelJobResponse) String() string { return proto.CompactTextString(m) }
func (*CancelJobResponse) ProtoMessage() {}
func (*CancelJobResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{5}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{5}
}
func (m *CancelJobResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CancelJobResponse.Unmarshal(m, b)
@@ -436,6 +436,139 @@
return JobState_UNSPECIFIED
}
+// A subset of info provided by ProvisionApi.ProvisionInfo
+type JobInfo struct {
+ JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
+ JobName string `protobuf:"bytes,2,opt,name=job_name,json=jobName,proto3" json:"job_name,omitempty"`
+ PipelineOptions *_struct.Struct `protobuf:"bytes,3,opt,name=pipeline_options,json=pipelineOptions,proto3" json:"pipeline_options,omitempty"`
+ State JobState_Enum `protobuf:"varint,4,opt,name=state,proto3,enum=org.apache.beam.model.job_management.v1.JobState_Enum" json:"state,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *JobInfo) Reset() { *m = JobInfo{} }
+func (m *JobInfo) String() string { return proto.CompactTextString(m) }
+func (*JobInfo) ProtoMessage() {}
+func (*JobInfo) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{6}
+}
+func (m *JobInfo) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_JobInfo.Unmarshal(m, b)
+}
+func (m *JobInfo) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_JobInfo.Marshal(b, m, deterministic)
+}
+func (dst *JobInfo) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_JobInfo.Merge(dst, src)
+}
+func (m *JobInfo) XXX_Size() int {
+ return xxx_messageInfo_JobInfo.Size(m)
+}
+func (m *JobInfo) XXX_DiscardUnknown() {
+ xxx_messageInfo_JobInfo.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_JobInfo proto.InternalMessageInfo
+
+func (m *JobInfo) GetJobId() string {
+ if m != nil {
+ return m.JobId
+ }
+ return ""
+}
+
+func (m *JobInfo) GetJobName() string {
+ if m != nil {
+ return m.JobName
+ }
+ return ""
+}
+
+func (m *JobInfo) GetPipelineOptions() *_struct.Struct {
+ if m != nil {
+ return m.PipelineOptions
+ }
+ return nil
+}
+
+func (m *JobInfo) GetState() JobState_Enum {
+ if m != nil {
+ return m.State
+ }
+ return JobState_UNSPECIFIED
+}
+
+// GetJobs is a synchronus request that returns a list of invoked jobs back
+// Throws error GRPC_STATUS_UNAVAILABLE if server is down
+type GetJobsRequest struct {
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *GetJobsRequest) Reset() { *m = GetJobsRequest{} }
+func (m *GetJobsRequest) String() string { return proto.CompactTextString(m) }
+func (*GetJobsRequest) ProtoMessage() {}
+func (*GetJobsRequest) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{7}
+}
+func (m *GetJobsRequest) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_GetJobsRequest.Unmarshal(m, b)
+}
+func (m *GetJobsRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_GetJobsRequest.Marshal(b, m, deterministic)
+}
+func (dst *GetJobsRequest) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_GetJobsRequest.Merge(dst, src)
+}
+func (m *GetJobsRequest) XXX_Size() int {
+ return xxx_messageInfo_GetJobsRequest.Size(m)
+}
+func (m *GetJobsRequest) XXX_DiscardUnknown() {
+ xxx_messageInfo_GetJobsRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_GetJobsRequest proto.InternalMessageInfo
+
+type GetJobsResponse struct {
+ JobInfo []*JobInfo `protobuf:"bytes,1,rep,name=job_info,json=jobInfo,proto3" json:"job_info,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *GetJobsResponse) Reset() { *m = GetJobsResponse{} }
+func (m *GetJobsResponse) String() string { return proto.CompactTextString(m) }
+func (*GetJobsResponse) ProtoMessage() {}
+func (*GetJobsResponse) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{8}
+}
+func (m *GetJobsResponse) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_GetJobsResponse.Unmarshal(m, b)
+}
+func (m *GetJobsResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_GetJobsResponse.Marshal(b, m, deterministic)
+}
+func (dst *GetJobsResponse) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_GetJobsResponse.Merge(dst, src)
+}
+func (m *GetJobsResponse) XXX_Size() int {
+ return xxx_messageInfo_GetJobsResponse.Size(m)
+}
+func (m *GetJobsResponse) XXX_DiscardUnknown() {
+ xxx_messageInfo_GetJobsResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_GetJobsResponse proto.InternalMessageInfo
+
+func (m *GetJobsResponse) GetJobInfo() []*JobInfo {
+ if m != nil {
+ return m.JobInfo
+ }
+ return nil
+}
+
// GetState is a synchronus request that returns a job state back
// Throws error GRPC_STATUS_UNAVAILABLE if server is down
// Throws error NOT_FOUND if the jobId is not found
@@ -450,7 +583,7 @@
func (m *GetJobStateRequest) String() string { return proto.CompactTextString(m) }
func (*GetJobStateRequest) ProtoMessage() {}
func (*GetJobStateRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{6}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{9}
}
func (m *GetJobStateRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobStateRequest.Unmarshal(m, b)
@@ -488,7 +621,7 @@
func (m *GetJobStateResponse) String() string { return proto.CompactTextString(m) }
func (*GetJobStateResponse) ProtoMessage() {}
func (*GetJobStateResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{7}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{10}
}
func (m *GetJobStateResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobStateResponse.Unmarshal(m, b)
@@ -515,6 +648,85 @@
return JobState_UNSPECIFIED
}
+// GetPipeline is a synchronus request that returns a pipeline back
+// Throws error GRPC_STATUS_UNAVAILABLE if server is down
+// Throws error NOT_FOUND if the jobId is not found
+type GetJobPipelineRequest struct {
+ JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *GetJobPipelineRequest) Reset() { *m = GetJobPipelineRequest{} }
+func (m *GetJobPipelineRequest) String() string { return proto.CompactTextString(m) }
+func (*GetJobPipelineRequest) ProtoMessage() {}
+func (*GetJobPipelineRequest) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{11}
+}
+func (m *GetJobPipelineRequest) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_GetJobPipelineRequest.Unmarshal(m, b)
+}
+func (m *GetJobPipelineRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_GetJobPipelineRequest.Marshal(b, m, deterministic)
+}
+func (dst *GetJobPipelineRequest) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_GetJobPipelineRequest.Merge(dst, src)
+}
+func (m *GetJobPipelineRequest) XXX_Size() int {
+ return xxx_messageInfo_GetJobPipelineRequest.Size(m)
+}
+func (m *GetJobPipelineRequest) XXX_DiscardUnknown() {
+ xxx_messageInfo_GetJobPipelineRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_GetJobPipelineRequest proto.InternalMessageInfo
+
+func (m *GetJobPipelineRequest) GetJobId() string {
+ if m != nil {
+ return m.JobId
+ }
+ return ""
+}
+
+type GetJobPipelineResponse struct {
+ Pipeline *pipeline_v1.Pipeline `protobuf:"bytes,1,opt,name=pipeline,proto3" json:"pipeline,omitempty"`
+ XXX_NoUnkeyedLiteral struct{} `json:"-"`
+ XXX_unrecognized []byte `json:"-"`
+ XXX_sizecache int32 `json:"-"`
+}
+
+func (m *GetJobPipelineResponse) Reset() { *m = GetJobPipelineResponse{} }
+func (m *GetJobPipelineResponse) String() string { return proto.CompactTextString(m) }
+func (*GetJobPipelineResponse) ProtoMessage() {}
+func (*GetJobPipelineResponse) Descriptor() ([]byte, []int) {
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{12}
+}
+func (m *GetJobPipelineResponse) XXX_Unmarshal(b []byte) error {
+ return xxx_messageInfo_GetJobPipelineResponse.Unmarshal(m, b)
+}
+func (m *GetJobPipelineResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+ return xxx_messageInfo_GetJobPipelineResponse.Marshal(b, m, deterministic)
+}
+func (dst *GetJobPipelineResponse) XXX_Merge(src proto.Message) {
+ xxx_messageInfo_GetJobPipelineResponse.Merge(dst, src)
+}
+func (m *GetJobPipelineResponse) XXX_Size() int {
+ return xxx_messageInfo_GetJobPipelineResponse.Size(m)
+}
+func (m *GetJobPipelineResponse) XXX_DiscardUnknown() {
+ xxx_messageInfo_GetJobPipelineResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_GetJobPipelineResponse proto.InternalMessageInfo
+
+func (m *GetJobPipelineResponse) GetPipeline() *pipeline_v1.Pipeline {
+ if m != nil {
+ return m.Pipeline
+ }
+ return nil
+}
+
// GetJobMessages is a streaming api for streaming job messages from the service
// One request will connect you to the job and you'll get a stream of job state
// and job messages back; one is used for logging and the other for detecting
@@ -530,7 +742,7 @@
func (m *JobMessagesRequest) String() string { return proto.CompactTextString(m) }
func (*JobMessagesRequest) ProtoMessage() {}
func (*JobMessagesRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{8}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{13}
}
func (m *JobMessagesRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobMessagesRequest.Unmarshal(m, b)
@@ -571,7 +783,7 @@
func (m *JobMessage) String() string { return proto.CompactTextString(m) }
func (*JobMessage) ProtoMessage() {}
func (*JobMessage) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{9}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{14}
}
func (m *JobMessage) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobMessage.Unmarshal(m, b)
@@ -633,7 +845,7 @@
func (m *JobMessagesResponse) String() string { return proto.CompactTextString(m) }
func (*JobMessagesResponse) ProtoMessage() {}
func (*JobMessagesResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{10}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{15}
}
func (m *JobMessagesResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobMessagesResponse.Unmarshal(m, b)
@@ -773,7 +985,7 @@
func (m *JobState) String() string { return proto.CompactTextString(m) }
func (*JobState) ProtoMessage() {}
func (*JobState) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{11}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{16}
}
func (m *JobState) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_JobState.Unmarshal(m, b)
@@ -804,7 +1016,7 @@
func (m *GetJobMetricsRequest) String() string { return proto.CompactTextString(m) }
func (*GetJobMetricsRequest) ProtoMessage() {}
func (*GetJobMetricsRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{12}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{17}
}
func (m *GetJobMetricsRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobMetricsRequest.Unmarshal(m, b)
@@ -842,7 +1054,7 @@
func (m *GetJobMetricsResponse) String() string { return proto.CompactTextString(m) }
func (*GetJobMetricsResponse) ProtoMessage() {}
func (*GetJobMetricsResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{13}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{18}
}
func (m *GetJobMetricsResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_GetJobMetricsResponse.Unmarshal(m, b)
@@ -882,7 +1094,7 @@
func (m *MetricResults) String() string { return proto.CompactTextString(m) }
func (*MetricResults) ProtoMessage() {}
func (*MetricResults) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{14}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{19}
}
func (m *MetricResults) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MetricResults.Unmarshal(m, b)
@@ -930,7 +1142,7 @@
func (m *DescribePipelineOptionsRequest) String() string { return proto.CompactTextString(m) }
func (*DescribePipelineOptionsRequest) ProtoMessage() {}
func (*DescribePipelineOptionsRequest) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{15}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{20}
}
func (m *DescribePipelineOptionsRequest) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DescribePipelineOptionsRequest.Unmarshal(m, b)
@@ -962,7 +1174,7 @@
func (m *PipelineOptionType) String() string { return proto.CompactTextString(m) }
func (*PipelineOptionType) ProtoMessage() {}
func (*PipelineOptionType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{16}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{21}
}
func (m *PipelineOptionType) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PipelineOptionType.Unmarshal(m, b)
@@ -1003,7 +1215,7 @@
func (m *PipelineOptionDescriptor) String() string { return proto.CompactTextString(m) }
func (*PipelineOptionDescriptor) ProtoMessage() {}
func (*PipelineOptionDescriptor) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{17}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{22}
}
func (m *PipelineOptionDescriptor) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PipelineOptionDescriptor.Unmarshal(m, b)
@@ -1070,7 +1282,7 @@
func (m *DescribePipelineOptionsResponse) String() string { return proto.CompactTextString(m) }
func (*DescribePipelineOptionsResponse) ProtoMessage() {}
func (*DescribePipelineOptionsResponse) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_job_api_1bded2a84bccf720, []int{18}
+ return fileDescriptor_beam_job_api_0a1706a5eaabebe4, []int{23}
}
func (m *DescribePipelineOptionsResponse) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DescribePipelineOptionsResponse.Unmarshal(m, b)
@@ -1104,8 +1316,13 @@
proto.RegisterType((*RunJobResponse)(nil), "org.apache.beam.model.job_management.v1.RunJobResponse")
proto.RegisterType((*CancelJobRequest)(nil), "org.apache.beam.model.job_management.v1.CancelJobRequest")
proto.RegisterType((*CancelJobResponse)(nil), "org.apache.beam.model.job_management.v1.CancelJobResponse")
+ proto.RegisterType((*JobInfo)(nil), "org.apache.beam.model.job_management.v1.JobInfo")
+ proto.RegisterType((*GetJobsRequest)(nil), "org.apache.beam.model.job_management.v1.GetJobsRequest")
+ proto.RegisterType((*GetJobsResponse)(nil), "org.apache.beam.model.job_management.v1.GetJobsResponse")
proto.RegisterType((*GetJobStateRequest)(nil), "org.apache.beam.model.job_management.v1.GetJobStateRequest")
proto.RegisterType((*GetJobStateResponse)(nil), "org.apache.beam.model.job_management.v1.GetJobStateResponse")
+ proto.RegisterType((*GetJobPipelineRequest)(nil), "org.apache.beam.model.job_management.v1.GetJobPipelineRequest")
+ proto.RegisterType((*GetJobPipelineResponse)(nil), "org.apache.beam.model.job_management.v1.GetJobPipelineResponse")
proto.RegisterType((*JobMessagesRequest)(nil), "org.apache.beam.model.job_management.v1.JobMessagesRequest")
proto.RegisterType((*JobMessage)(nil), "org.apache.beam.model.job_management.v1.JobMessage")
proto.RegisterType((*JobMessagesResponse)(nil), "org.apache.beam.model.job_management.v1.JobMessagesResponse")
@@ -1139,8 +1356,12 @@
Prepare(ctx context.Context, in *PrepareJobRequest, opts ...grpc.CallOption) (*PrepareJobResponse, error)
// Submit the job for execution
Run(ctx context.Context, in *RunJobRequest, opts ...grpc.CallOption) (*RunJobResponse, error)
+ // Get a list of all invoked jobs
+ GetJobs(ctx context.Context, in *GetJobsRequest, opts ...grpc.CallOption) (*GetJobsResponse, error)
// Get the current state of the job
GetState(ctx context.Context, in *GetJobStateRequest, opts ...grpc.CallOption) (*GetJobStateResponse, error)
+ // Get the job's pipeline
+ GetPipeline(ctx context.Context, in *GetJobPipelineRequest, opts ...grpc.CallOption) (*GetJobPipelineResponse, error)
// Cancel the job
Cancel(ctx context.Context, in *CancelJobRequest, opts ...grpc.CallOption) (*CancelJobResponse, error)
// Subscribe to a stream of state changes of the job, will immediately return the current state of the job as the first response.
@@ -1179,6 +1400,15 @@
return out, nil
}
+func (c *jobServiceClient) GetJobs(ctx context.Context, in *GetJobsRequest, opts ...grpc.CallOption) (*GetJobsResponse, error) {
+ out := new(GetJobsResponse)
+ err := c.cc.Invoke(ctx, "/org.apache.beam.model.job_management.v1.JobService/GetJobs", in, out, opts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
func (c *jobServiceClient) GetState(ctx context.Context, in *GetJobStateRequest, opts ...grpc.CallOption) (*GetJobStateResponse, error) {
out := new(GetJobStateResponse)
err := c.cc.Invoke(ctx, "/org.apache.beam.model.job_management.v1.JobService/GetState", in, out, opts...)
@@ -1188,6 +1418,15 @@
return out, nil
}
+func (c *jobServiceClient) GetPipeline(ctx context.Context, in *GetJobPipelineRequest, opts ...grpc.CallOption) (*GetJobPipelineResponse, error) {
+ out := new(GetJobPipelineResponse)
+ err := c.cc.Invoke(ctx, "/org.apache.beam.model.job_management.v1.JobService/GetPipeline", in, out, opts...)
+ if err != nil {
+ return nil, err
+ }
+ return out, nil
+}
+
func (c *jobServiceClient) Cancel(ctx context.Context, in *CancelJobRequest, opts ...grpc.CallOption) (*CancelJobResponse, error) {
out := new(CancelJobResponse)
err := c.cc.Invoke(ctx, "/org.apache.beam.model.job_management.v1.JobService/Cancel", in, out, opts...)
@@ -1286,8 +1525,12 @@
Prepare(context.Context, *PrepareJobRequest) (*PrepareJobResponse, error)
// Submit the job for execution
Run(context.Context, *RunJobRequest) (*RunJobResponse, error)
+ // Get a list of all invoked jobs
+ GetJobs(context.Context, *GetJobsRequest) (*GetJobsResponse, error)
// Get the current state of the job
GetState(context.Context, *GetJobStateRequest) (*GetJobStateResponse, error)
+ // Get the job's pipeline
+ GetPipeline(context.Context, *GetJobPipelineRequest) (*GetJobPipelineResponse, error)
// Cancel the job
Cancel(context.Context, *CancelJobRequest) (*CancelJobResponse, error)
// Subscribe to a stream of state changes of the job, will immediately return the current state of the job as the first response.
@@ -1340,6 +1583,24 @@
return interceptor(ctx, in, info, handler)
}
+func _JobService_GetJobs_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(GetJobsRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(JobServiceServer).GetJobs(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: "/org.apache.beam.model.job_management.v1.JobService/GetJobs",
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(JobServiceServer).GetJobs(ctx, req.(*GetJobsRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
func _JobService_GetState_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(GetJobStateRequest)
if err := dec(in); err != nil {
@@ -1358,6 +1619,24 @@
return interceptor(ctx, in, info, handler)
}
+func _JobService_GetPipeline_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
+ in := new(GetJobPipelineRequest)
+ if err := dec(in); err != nil {
+ return nil, err
+ }
+ if interceptor == nil {
+ return srv.(JobServiceServer).GetPipeline(ctx, in)
+ }
+ info := &grpc.UnaryServerInfo{
+ Server: srv,
+ FullMethod: "/org.apache.beam.model.job_management.v1.JobService/GetPipeline",
+ }
+ handler := func(ctx context.Context, req interface{}) (interface{}, error) {
+ return srv.(JobServiceServer).GetPipeline(ctx, req.(*GetJobPipelineRequest))
+ }
+ return interceptor(ctx, in, info, handler)
+}
+
func _JobService_Cancel_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(CancelJobRequest)
if err := dec(in); err != nil {
@@ -1467,10 +1746,18 @@
Handler: _JobService_Run_Handler,
},
{
+ MethodName: "GetJobs",
+ Handler: _JobService_GetJobs_Handler,
+ },
+ {
MethodName: "GetState",
Handler: _JobService_GetState_Handler,
},
{
+ MethodName: "GetPipeline",
+ Handler: _JobService_GetPipeline_Handler,
+ },
+ {
MethodName: "Cancel",
Handler: _JobService_Cancel_Handler,
},
@@ -1498,88 +1785,96 @@
Metadata: "beam_job_api.proto",
}
-func init() { proto.RegisterFile("beam_job_api.proto", fileDescriptor_beam_job_api_1bded2a84bccf720) }
+func init() { proto.RegisterFile("beam_job_api.proto", fileDescriptor_beam_job_api_0a1706a5eaabebe4) }
-var fileDescriptor_beam_job_api_1bded2a84bccf720 = []byte{
- // 1274 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x57, 0x5f, 0x6f, 0xdb, 0x54,
- 0x14, 0x9f, 0xd3, 0x24, 0x4d, 0x4e, 0x97, 0xd4, 0xbd, 0x5d, 0xd5, 0x2c, 0x82, 0x51, 0x8c, 0x60,
- 0x43, 0x13, 0xde, 0x9a, 0x49, 0x4c, 0x6c, 0x80, 0x70, 0x1a, 0x2f, 0x73, 0xd5, 0x26, 0xd1, 0xb5,
- 0x0b, 0x02, 0x1e, 0x8c, 0x93, 0xdc, 0x06, 0x8f, 0xd8, 0xd7, 0xd8, 0x37, 0xd1, 0xf6, 0x02, 0x12,
- 0x12, 0x8f, 0xc0, 0x17, 0xe0, 0x0b, 0x20, 0x21, 0x21, 0x9e, 0xf9, 0x34, 0x48, 0x3c, 0xf1, 0xcc,
- 0x1b, 0x2f, 0xe8, 0xda, 0xd7, 0x49, 0xdc, 0xad, 0x2c, 0xcd, 0x84, 0x78, 0xca, 0xbd, 0xe7, 0xcf,
- 0xef, 0xfc, 0xf5, 0x3d, 0x27, 0x80, 0xfa, 0xc4, 0xf1, 0xec, 0x47, 0xb4, 0x6f, 0x3b, 0x81, 0xab,
- 0x06, 0x21, 0x65, 0x14, 0x5d, 0xa7, 0xe1, 0x48, 0x75, 0x02, 0x67, 0xf0, 0x39, 0x51, 0x39, 0x5b,
- 0xf5, 0xe8, 0x90, 0x8c, 0x55, 0x2e, 0xe4, 0x39, 0xbe, 0x33, 0x22, 0x1e, 0xf1, 0x99, 0x3a, 0xdd,
- 0xaf, 0xef, 0xc4, 0xca, 0xe1, 0xc4, 0xf7, 0x49, 0x38, 0xd7, 0xaf, 0x6f, 0x12, 0x7f, 0x18, 0x50,
- 0xd7, 0x67, 0x91, 0x20, 0xbc, 0x34, 0xa2, 0x74, 0x34, 0x26, 0xb7, 0xe2, 0x5b, 0x7f, 0x72, 0x7a,
- 0x2b, 0x62, 0xe1, 0x64, 0xc0, 0x04, 0xb7, 0xe2, 0x11, 0x16, 0xba, 0x03, 0x21, 0xac, 0xfc, 0x26,
- 0xc1, 0x56, 0x2f, 0x24, 0x81, 0x13, 0x92, 0x43, 0xda, 0xc7, 0xe4, 0xcb, 0x09, 0x89, 0x18, 0x6a,
- 0x43, 0x29, 0x70, 0x03, 0x32, 0x76, 0x7d, 0x52, 0x93, 0xf6, 0xa4, 0x1b, 0x1b, 0x8d, 0x9b, 0xea,
- 0xb3, 0xdd, 0x4c, 0xc5, 0xd4, 0xe9, 0xbe, 0xda, 0x13, 0x67, 0x3c, 0x53, 0x46, 0x4d, 0x90, 0xd3,
- 0xb3, 0x4d, 0x03, 0xe6, 0x52, 0x3f, 0xaa, 0xe5, 0x62, 0xc0, 0x5d, 0x35, 0x71, 0x53, 0x4d, 0xdd,
- 0x54, 0xcd, 0xd8, 0x4d, 0xbc, 0x99, 0x2a, 0x74, 0x13, 0x79, 0x74, 0x15, 0x4a, 0x3c, 0x19, 0xbe,
- 0xe3, 0x91, 0xda, 0xda, 0x9e, 0x74, 0xa3, 0x8c, 0xd7, 0x1f, 0xd1, 0x7e, 0xc7, 0xf1, 0x88, 0xf2,
- 0x87, 0x04, 0x68, 0xd1, 0xfb, 0x28, 0xa0, 0x7e, 0x44, 0xd0, 0xeb, 0x50, 0x0d, 0x62, 0xaa, 0xc3,
- 0x11, 0x6c, 0x77, 0x18, 0x07, 0x51, 0xc6, 0x95, 0x05, 0xaa, 0x31, 0x44, 0x11, 0x5c, 0x75, 0x42,
- 0xe6, 0x9e, 0x3a, 0x03, 0x66, 0x47, 0xcc, 0x19, 0xb9, 0xfe, 0xc8, 0x4e, 0x93, 0x29, 0xbc, 0xbc,
- 0xbb, 0x44, 0xd8, 0x5a, 0xe0, 0x9a, 0x24, 0x9c, 0xba, 0x03, 0xd2, 0x22, 0xd1, 0x20, 0x74, 0x03,
- 0x46, 0x43, 0xbc, 0x9b, 0x22, 0x9b, 0x09, 0xb0, 0x2e, 0x70, 0x51, 0x03, 0x76, 0x52, 0x5b, 0x11,
- 0x89, 0x22, 0xee, 0x1f, 0xa3, 0x5f, 0x10, 0x5f, 0x84, 0xb6, 0x2d, 0x98, 0x66, 0xc2, 0xb3, 0x38,
- 0x4b, 0xb1, 0xa1, 0x82, 0x27, 0xfe, 0x42, 0x7d, 0x96, 0x0c, 0xf0, 0x3a, 0x6c, 0x86, 0xbc, 0xda,
- 0x64, 0xea, 0x8c, 0x85, 0x95, 0x5c, 0x2c, 0x57, 0x9d, 0x91, 0x13, 0x03, 0xd7, 0xa1, 0x9a, 0x1a,
- 0x10, 0x29, 0xdc, 0x81, 0x22, 0x4f, 0xfa, 0x0c, 0xb9, 0xf0, 0x88, 0xf6, 0x8d, 0xa1, 0xf2, 0x26,
- 0xc8, 0x07, 0x8e, 0x3f, 0x20, 0xe3, 0x05, 0x67, 0xce, 0x11, 0x75, 0x60, 0x6b, 0x41, 0x54, 0xc0,
- 0x1e, 0x41, 0x21, 0x62, 0x0e, 0x4b, 0xba, 0xaa, 0xda, 0x78, 0x5b, 0x5d, 0xb2, 0xf9, 0xd5, 0x43,
- 0xda, 0x37, 0xb9, 0xa2, 0xaa, 0xfb, 0x13, 0x0f, 0x27, 0x20, 0xca, 0x4d, 0x40, 0x6d, 0xc2, 0x52,
- 0xd6, 0x73, 0xfc, 0x19, 0xc0, 0x76, 0x46, 0xf8, 0xbf, 0xf2, 0xe8, 0x90, 0xf6, 0x8f, 0x49, 0x14,
- 0x39, 0x23, 0x12, 0x3d, 0xc7, 0xa3, 0xbf, 0x73, 0x00, 0x73, 0x69, 0xf4, 0x32, 0x80, 0x97, 0x1c,
- 0xe7, 0x92, 0x65, 0x41, 0x31, 0x86, 0x08, 0x41, 0x9e, 0xb9, 0x1e, 0x11, 0x15, 0x8c, 0xcf, 0x88,
- 0x00, 0xb8, 0x5e, 0x40, 0x43, 0xc6, 0x13, 0x1d, 0x77, 0x50, 0xb5, 0xa1, 0x5f, 0x24, 0x02, 0x61,
- 0x5b, 0x15, 0xbf, 0xc6, 0x0c, 0x0c, 0x2f, 0x00, 0xa3, 0x57, 0xe1, 0x72, 0xea, 0x19, 0x23, 0x8f,
- 0x59, 0x2d, 0x1f, 0xbb, 0xb0, 0x21, 0x68, 0x16, 0x79, 0xcc, 0x94, 0x5f, 0x24, 0xd8, 0x7a, 0x0a,
- 0x04, 0x29, 0x70, 0xed, 0x58, 0x37, 0x4d, 0xad, 0xad, 0xdb, 0xc6, 0x71, 0xaf, 0x8b, 0x2d, 0xad,
- 0x73, 0xa0, 0xdb, 0x27, 0x1d, 0xb3, 0xa7, 0x1f, 0x18, 0x0f, 0x0c, 0xbd, 0x25, 0x5f, 0x42, 0x3b,
- 0xb0, 0x75, 0xd8, 0x6d, 0xda, 0xa9, 0x5c, 0x4b, 0x6f, 0x9e, 0xb4, 0x65, 0x09, 0xd5, 0xe0, 0x4a,
- 0x96, 0x6c, 0x69, 0xc6, 0x91, 0xde, 0x92, 0x73, 0x67, 0x15, 0x9a, 0x9a, 0x69, 0x1c, 0xc8, 0x6b,
- 0x68, 0x17, 0xb6, 0x17, 0xc9, 0x1f, 0x69, 0xb8, 0x63, 0x74, 0xda, 0x72, 0xfe, 0xac, 0xbc, 0x8e,
- 0x71, 0x17, 0xcb, 0x05, 0xe5, 0x4f, 0x09, 0xb6, 0x33, 0xb5, 0x12, 0x0d, 0xf1, 0x19, 0xc8, 0x69,
- 0xb0, 0xa1, 0xa0, 0x89, 0x37, 0xf0, 0xce, 0x0a, 0x99, 0x7d, 0x78, 0x09, 0x6f, 0x0a, 0xb8, 0x99,
- 0x05, 0x02, 0xd5, 0xb8, 0x5b, 0xe6, 0xf8, 0xc9, 0x63, 0xf3, 0xee, 0xd2, 0xf8, 0xcf, 0x68, 0xe4,
- 0x87, 0x97, 0x70, 0x25, 0x5a, 0x24, 0x34, 0x01, 0x4a, 0xa9, 0x01, 0xe5, 0x27, 0x09, 0x4a, 0xa9,
- 0x86, 0xf2, 0xa3, 0x04, 0x79, 0xde, 0xb4, 0x68, 0x13, 0x36, 0xb2, 0xb5, 0xd8, 0x80, 0x75, 0xd3,
- 0xea, 0xf6, 0x7a, 0x7a, 0x4b, 0x96, 0xf8, 0x05, 0x9f, 0x74, 0xe2, 0x24, 0xe6, 0x50, 0x09, 0xf2,
- 0xad, 0x6e, 0x47, 0x97, 0xd7, 0x10, 0x40, 0xf1, 0x41, 0x52, 0x8a, 0x3c, 0xaa, 0x40, 0xf9, 0x80,
- 0x97, 0xf4, 0x88, 0x5f, 0x0b, 0x5c, 0xe3, 0xa4, 0xd7, 0xd2, 0x2c, 0xbd, 0x25, 0x17, 0xd1, 0x65,
- 0x28, 0xb5, 0xb0, 0x66, 0xc4, 0xfa, 0xeb, 0x9c, 0x15, 0xdf, 0xf4, 0x96, 0x5c, 0xe2, 0x2c, 0xd3,
- 0xd2, 0xb0, 0xc5, 0x59, 0x65, 0x54, 0x05, 0x10, 0x20, 0xfc, 0x0e, 0xca, 0x5b, 0x70, 0x25, 0x89,
- 0xef, 0x38, 0x99, 0x54, 0xcf, 0xf9, 0x8a, 0x5c, 0xd8, 0x39, 0x23, 0x2e, 0xd2, 0xdc, 0x83, 0x75,
- 0x31, 0xeb, 0x44, 0xfd, 0x96, 0xff, 0xb6, 0x13, 0x28, 0x4c, 0xa2, 0xc9, 0x98, 0x45, 0x38, 0x85,
- 0x51, 0x7e, 0x95, 0xa0, 0x92, 0x61, 0xa1, 0x2e, 0x94, 0x1d, 0xc6, 0x88, 0x17, 0x30, 0xc2, 0xdd,
- 0x5a, 0xbb, 0xb1, 0xd1, 0xd8, 0x5f, 0x62, 0x64, 0x1c, 0x53, 0xdf, 0x65, 0x34, 0x74, 0xfd, 0x91,
- 0xe1, 0x9f, 0x52, 0x3c, 0xc7, 0xe0, 0x80, 0x03, 0xea, 0x79, 0x2e, 0xe3, 0x80, 0xb9, 0x95, 0x01,
- 0x67, 0x18, 0xca, 0x1e, 0x5c, 0x4b, 0xc6, 0x52, 0x9f, 0xf4, 0xb2, 0x83, 0x55, 0xe4, 0x55, 0x21,
- 0x80, 0xb2, 0x1c, 0xeb, 0x49, 0x40, 0x94, 0xae, 0xe8, 0x11, 0x80, 0xa2, 0x69, 0x61, 0x5e, 0x99,
- 0xb8, 0x3d, 0x9a, 0xdd, 0xee, 0x91, 0xae, 0x75, 0x92, 0xf6, 0x30, 0x3a, 0x96, 0xde, 0xd6, 0xb1,
- 0x9c, 0xe3, 0x52, 0x9d, 0x93, 0xe3, 0xa6, 0x8e, 0xe5, 0x35, 0x54, 0x86, 0x82, 0x86, 0xb1, 0xf6,
- 0xb1, 0x9c, 0xe7, 0xe4, 0x6e, 0xf3, 0x50, 0x3f, 0xb0, 0xe4, 0x82, 0xf2, 0xbb, 0x04, 0xb5, 0xac,
- 0x9d, 0xf9, 0xb8, 0xe4, 0x8f, 0x5b, 0x3c, 0xdf, 0x93, 0xca, 0xc6, 0x67, 0x64, 0x41, 0x9e, 0x3d,
- 0x09, 0x92, 0x8f, 0xa3, 0xda, 0xf8, 0x60, 0xe9, 0xe2, 0x3d, 0x1d, 0x4c, 0xf2, 0x44, 0xc7, 0x68,
- 0x68, 0x0f, 0x36, 0x86, 0xc2, 0xae, 0x4b, 0xd3, 0xa9, 0xbb, 0x48, 0x42, 0xaf, 0x41, 0x65, 0x48,
- 0x4e, 0x9d, 0xc9, 0x98, 0xd9, 0x53, 0x67, 0x3c, 0x21, 0xe2, 0xb9, 0xbb, 0x2c, 0x88, 0x1f, 0x72,
- 0x1a, 0xba, 0x02, 0x85, 0x51, 0x48, 0x27, 0x41, 0xad, 0x90, 0xf4, 0x62, 0x7c, 0x51, 0xbe, 0x82,
- 0x57, 0xce, 0x4d, 0xb6, 0xe8, 0xca, 0x4f, 0x61, 0x3d, 0x5d, 0x84, 0x92, 0x7e, 0xd1, 0x56, 0x0c,
- 0x6c, 0x61, 0xd9, 0x48, 0x11, 0x1b, 0x7f, 0x95, 0xe2, 0x89, 0x22, 0xd6, 0x11, 0xf4, 0x8d, 0x04,
- 0xeb, 0x62, 0x3d, 0x42, 0xf7, 0x96, 0x37, 0x73, 0x76, 0x1d, 0xac, 0xdf, 0x5f, 0x49, 0x57, 0x04,
- 0x3c, 0x85, 0x35, 0x3c, 0xf1, 0xd1, 0xf2, 0x1f, 0x5f, 0x66, 0xd5, 0xa9, 0xdf, 0xbd, 0xb0, 0x9e,
- 0xb0, 0xfb, 0xad, 0x04, 0xa5, 0x36, 0x61, 0xf1, 0x93, 0x87, 0xee, 0xaf, 0xf6, 0xb4, 0x26, 0x2e,
- 0xbc, 0xd0, 0xbb, 0x8c, 0xbe, 0x86, 0x62, 0xb2, 0x07, 0xa1, 0x77, 0x96, 0xc6, 0x39, 0xbb, 0x63,
- 0xd5, 0xef, 0xad, 0xa2, 0x2a, 0x1c, 0xf8, 0x4e, 0x82, 0x6a, 0x9a, 0x08, 0x93, 0x85, 0xc4, 0xf1,
- 0xfe, 0xc7, 0x74, 0xdc, 0x96, 0xd0, 0x0f, 0x12, 0xc8, 0x6d, 0xc2, 0xc4, 0x80, 0xbc, 0xb0, 0x47,
- 0x4f, 0xef, 0x57, 0x17, 0xf0, 0xe8, 0x19, 0x03, 0xff, 0xb6, 0x84, 0xbe, 0x97, 0xa0, 0x92, 0x99,
- 0x21, 0xe8, 0xbd, 0x0b, 0xc6, 0x98, 0x1d, 0x55, 0xf5, 0xf7, 0x57, 0x55, 0x17, 0x25, 0xfb, 0x59,
- 0x82, 0xdd, 0x73, 0x1e, 0x12, 0xd4, 0x5e, 0x1a, 0xfb, 0xdf, 0xdf, 0xfd, 0xfa, 0xc3, 0x17, 0x07,
- 0x12, 0x9b, 0x46, 0x13, 0xde, 0x38, 0x17, 0x2a, 0x83, 0xd4, 0x2c, 0x1e, 0xd2, 0xbe, 0x16, 0xb8,
- 0x9f, 0xc8, 0x19, 0x8e, 0x3d, 0xdd, 0xef, 0x17, 0xe3, 0xff, 0x81, 0x77, 0xfe, 0x09, 0x00, 0x00,
- 0xff, 0xff, 0xb1, 0xc2, 0xcc, 0x7d, 0x23, 0x0f, 0x00, 0x00,
+var fileDescriptor_beam_job_api_0a1706a5eaabebe4 = []byte{
+ // 1401 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xc4, 0x57, 0xdd, 0x8e, 0xdb, 0x44,
+ 0x14, 0xae, 0xf3, 0x9f, 0x93, 0x26, 0xeb, 0x9d, 0xed, 0xb2, 0x69, 0x04, 0x65, 0x31, 0x82, 0x16,
+ 0x55, 0xb8, 0xdd, 0x54, 0xa2, 0xd0, 0x42, 0xc1, 0xd9, 0xb8, 0x69, 0xc2, 0x6e, 0x12, 0x4d, 0xb2,
+ 0x20, 0x40, 0x22, 0x38, 0xc9, 0x6c, 0x70, 0x89, 0x3d, 0xc6, 0x9e, 0x44, 0xad, 0x84, 0x40, 0x02,
+ 0x71, 0x09, 0xbc, 0x00, 0x2f, 0x80, 0x84, 0x84, 0xb8, 0xe6, 0x19, 0x78, 0x08, 0x24, 0xae, 0x78,
+ 0x05, 0x6e, 0xd0, 0xd8, 0xe3, 0x6c, 0xbc, 0x3f, 0x6c, 0x92, 0x16, 0x71, 0x15, 0xcf, 0xf9, 0xf9,
+ 0xce, 0xef, 0x9c, 0x39, 0x01, 0xd4, 0x27, 0x86, 0xd5, 0x7b, 0x48, 0xfb, 0x3d, 0xc3, 0x31, 0x55,
+ 0xc7, 0xa5, 0x8c, 0xa2, 0xab, 0xd4, 0x1d, 0xa9, 0x86, 0x63, 0x0c, 0x3e, 0x25, 0x2a, 0x67, 0xab,
+ 0x16, 0x1d, 0x92, 0xb1, 0xca, 0x85, 0x2c, 0xc3, 0x36, 0x46, 0xc4, 0x22, 0x36, 0x53, 0xa7, 0x3b,
+ 0xa5, 0x4d, 0x5f, 0xd9, 0x9d, 0xd8, 0x36, 0x71, 0x8f, 0xf4, 0x4b, 0x6b, 0xc4, 0x1e, 0x3a, 0xd4,
+ 0xb4, 0x99, 0x27, 0x08, 0xcf, 0x8e, 0x28, 0x1d, 0x8d, 0xc9, 0x0d, 0xff, 0xd4, 0x9f, 0x1c, 0xde,
+ 0xf0, 0x98, 0x3b, 0x19, 0x30, 0xc1, 0xcd, 0x5b, 0x84, 0xb9, 0xe6, 0x40, 0x08, 0x2b, 0xbf, 0x49,
+ 0xb0, 0xde, 0x76, 0x89, 0x63, 0xb8, 0xa4, 0x41, 0xfb, 0x98, 0x7c, 0x3e, 0x21, 0x1e, 0x43, 0x35,
+ 0xc8, 0x38, 0xa6, 0x43, 0xc6, 0xa6, 0x4d, 0x8a, 0xd2, 0xb6, 0x74, 0x2d, 0x57, 0xbe, 0xae, 0x9e,
+ 0xee, 0x66, 0x28, 0xa6, 0x4e, 0x77, 0xd4, 0xb6, 0xf8, 0xc6, 0x33, 0x65, 0x54, 0x01, 0x39, 0xfc,
+ 0xee, 0x51, 0x87, 0x99, 0xd4, 0xf6, 0x8a, 0x31, 0x1f, 0x70, 0x4b, 0x0d, 0xdc, 0x54, 0x43, 0x37,
+ 0xd5, 0x8e, 0xef, 0x26, 0x5e, 0x0b, 0x15, 0x5a, 0x81, 0x3c, 0xba, 0x0c, 0x19, 0x9e, 0x0c, 0xdb,
+ 0xb0, 0x48, 0x31, 0xbe, 0x2d, 0x5d, 0xcb, 0xe2, 0xf4, 0x43, 0xda, 0x6f, 0x1a, 0x16, 0x51, 0xfe,
+ 0x94, 0x00, 0xcd, 0x7b, 0xef, 0x39, 0xd4, 0xf6, 0x08, 0x7a, 0x09, 0x0a, 0x8e, 0x4f, 0x35, 0x38,
+ 0x42, 0xcf, 0x1c, 0xfa, 0x41, 0x64, 0x71, 0x7e, 0x8e, 0x5a, 0x1f, 0x22, 0x0f, 0x2e, 0x1b, 0x2e,
+ 0x33, 0x0f, 0x8d, 0x01, 0xeb, 0x79, 0xcc, 0x18, 0x99, 0xf6, 0xa8, 0x17, 0x26, 0x53, 0x78, 0x79,
+ 0x7b, 0x81, 0xb0, 0x35, 0xc7, 0xec, 0x10, 0x77, 0x6a, 0x0e, 0x48, 0x95, 0x78, 0x03, 0xd7, 0x74,
+ 0x18, 0x75, 0xf1, 0x56, 0x88, 0xdc, 0x09, 0x80, 0x75, 0x81, 0x8b, 0xca, 0xb0, 0x19, 0xda, 0xf2,
+ 0x88, 0xe7, 0x71, 0xff, 0x18, 0xfd, 0x8c, 0xd8, 0x22, 0xb4, 0x0d, 0xc1, 0xec, 0x04, 0xbc, 0x2e,
+ 0x67, 0x29, 0x3d, 0xc8, 0xe3, 0x89, 0x3d, 0x57, 0x9f, 0x05, 0x03, 0xbc, 0x0a, 0x6b, 0x2e, 0xaf,
+ 0x36, 0x99, 0x1a, 0x63, 0x61, 0x25, 0xe6, 0xcb, 0x15, 0x66, 0xe4, 0xc0, 0xc0, 0x55, 0x28, 0x84,
+ 0x06, 0x44, 0x0a, 0x37, 0x21, 0xc5, 0x93, 0x3e, 0x43, 0x4e, 0x3e, 0xa4, 0xfd, 0xfa, 0x50, 0x79,
+ 0x05, 0xe4, 0x5d, 0xc3, 0x1e, 0x90, 0xf1, 0x9c, 0x33, 0x67, 0x88, 0x1a, 0xb0, 0x3e, 0x27, 0x2a,
+ 0x60, 0xf7, 0x20, 0xe9, 0x31, 0x83, 0x05, 0x5d, 0x55, 0x28, 0xbf, 0xa6, 0x2e, 0xd8, 0xfc, 0x6a,
+ 0x83, 0xf6, 0x3b, 0x5c, 0x51, 0xd5, 0xed, 0x89, 0x85, 0x03, 0x10, 0xe5, 0x77, 0x09, 0xd2, 0x0d,
+ 0xda, 0xaf, 0xdb, 0x87, 0xf4, 0x0c, 0x2f, 0x22, 0xcd, 0x13, 0x8b, 0x34, 0xcf, 0xa9, 0xbd, 0x19,
+ 0x5f, 0xb2, 0x37, 0x67, 0xf1, 0x24, 0x9e, 0x46, 0x3c, 0x32, 0x14, 0x6a, 0x84, 0x35, 0x68, 0xdf,
+ 0x13, 0xb9, 0x55, 0x3e, 0x86, 0xb5, 0x19, 0x45, 0xa4, 0xf0, 0xdd, 0x20, 0x22, 0xd3, 0x3e, 0xa4,
+ 0x45, 0x69, 0x3b, 0x7e, 0x2d, 0x57, 0xbe, 0xb9, 0x8c, 0x55, 0x9e, 0x2c, 0x3f, 0x07, 0xfc, 0x43,
+ 0xb9, 0x0e, 0x28, 0xc0, 0xf7, 0x9d, 0x39, 0xa7, 0xa2, 0x03, 0xd8, 0x88, 0x08, 0xff, 0x27, 0x35,
+ 0x55, 0x61, 0x33, 0x30, 0x32, 0x9b, 0x26, 0xe7, 0xb5, 0xd9, 0x33, 0xc7, 0xe5, 0x85, 0x5f, 0x4f,
+ 0x6b, 0x88, 0xf1, 0x24, 0x35, 0x68, 0x7f, 0x9f, 0x78, 0x9e, 0x31, 0x22, 0xde, 0x39, 0xfe, 0xfc,
+ 0x1d, 0x03, 0x38, 0x92, 0x46, 0xcf, 0x01, 0x58, 0xc1, 0xe7, 0x91, 0x64, 0x56, 0x50, 0xea, 0x43,
+ 0x84, 0x20, 0xc1, 0xcc, 0x59, 0x6b, 0xfa, 0xdf, 0x88, 0x00, 0x98, 0x96, 0x43, 0x5d, 0xc6, 0x6f,
+ 0x8f, 0xdf, 0x91, 0x85, 0xb2, 0xbe, 0x4c, 0x52, 0x85, 0x6d, 0x55, 0xfc, 0xd6, 0x67, 0x60, 0x78,
+ 0x0e, 0x18, 0xbd, 0x00, 0x17, 0x43, 0xcf, 0x18, 0x79, 0xc4, 0xfc, 0x0e, 0xce, 0xe2, 0x9c, 0xa0,
+ 0x75, 0xc9, 0x23, 0xa6, 0xfc, 0x22, 0xc1, 0xfa, 0x09, 0x10, 0xa4, 0xc0, 0x95, 0x7d, 0xbd, 0xd3,
+ 0xd1, 0x6a, 0x7a, 0xaf, 0xbe, 0xdf, 0x6e, 0xe1, 0xae, 0xd6, 0xdc, 0xd5, 0x7b, 0x07, 0xcd, 0x4e,
+ 0x5b, 0xdf, 0xad, 0xdf, 0xaf, 0xeb, 0x55, 0xf9, 0x02, 0xda, 0x84, 0xf5, 0x46, 0xab, 0xd2, 0x0b,
+ 0xe5, 0xaa, 0x7a, 0xe5, 0xa0, 0x26, 0x4b, 0xa8, 0x08, 0x97, 0xa2, 0xe4, 0xae, 0x56, 0xdf, 0xd3,
+ 0xab, 0x72, 0xec, 0xb8, 0x42, 0x45, 0xeb, 0xd4, 0x77, 0xe5, 0x38, 0xda, 0x82, 0x8d, 0x79, 0xf2,
+ 0xfb, 0x1a, 0x6e, 0xd6, 0x9b, 0x35, 0x39, 0x71, 0x5c, 0x5e, 0xc7, 0xb8, 0x85, 0xe5, 0xa4, 0xf2,
+ 0x97, 0x04, 0x1b, 0x91, 0x5a, 0x89, 0x5e, 0xf8, 0x04, 0xe4, 0x30, 0x58, 0x57, 0xd0, 0x44, 0x4f,
+ 0xdc, 0x5a, 0x21, 0xb3, 0x0f, 0x2e, 0xe0, 0x35, 0x01, 0x37, 0xb3, 0x40, 0xa0, 0xe0, 0x37, 0xf0,
+ 0x11, 0x7e, 0xf0, 0x82, 0xbc, 0xb9, 0x30, 0xfe, 0x29, 0x77, 0xeb, 0xc1, 0x05, 0x9c, 0xf7, 0xe6,
+ 0x09, 0x15, 0x80, 0x4c, 0x68, 0x40, 0xf9, 0x49, 0x82, 0x4c, 0xa8, 0xa1, 0xfc, 0x28, 0x41, 0x82,
+ 0xdf, 0x23, 0xb4, 0x06, 0xb9, 0x68, 0x2d, 0x72, 0x90, 0xee, 0x74, 0x5b, 0xed, 0xb6, 0x5e, 0x95,
+ 0x25, 0x7e, 0xc0, 0x07, 0x4d, 0x3f, 0x89, 0x31, 0x94, 0x81, 0x44, 0xb5, 0xd5, 0xd4, 0xe5, 0x38,
+ 0x02, 0x48, 0xdd, 0x0f, 0x4a, 0x91, 0x40, 0x79, 0xc8, 0xee, 0xf2, 0x92, 0xee, 0xf1, 0x63, 0x92,
+ 0x6b, 0x1c, 0xb4, 0xab, 0x5a, 0x57, 0xaf, 0xca, 0x29, 0x74, 0x11, 0x32, 0x55, 0xac, 0xd5, 0x7d,
+ 0xfd, 0x34, 0x67, 0xf9, 0x27, 0xbd, 0x2a, 0x67, 0x38, 0xab, 0xd3, 0xd5, 0x70, 0x97, 0xb3, 0xb2,
+ 0xa8, 0x00, 0x20, 0x40, 0xf8, 0x19, 0x94, 0x57, 0xe1, 0x52, 0x10, 0xdf, 0x7e, 0xb0, 0x7e, 0x9c,
+ 0x73, 0x8b, 0xcc, 0x70, 0x0a, 0xcc, 0xc4, 0x45, 0x9a, 0xdb, 0x90, 0x16, 0x0b, 0x8c, 0xa8, 0xdf,
+ 0xe2, 0xe3, 0x26, 0x80, 0xc2, 0xc4, 0x9b, 0x8c, 0x99, 0x87, 0x43, 0x18, 0xe5, 0x57, 0x09, 0xf2,
+ 0x11, 0x16, 0x6a, 0x41, 0xd6, 0x60, 0x8c, 0x58, 0x0e, 0x23, 0x43, 0x31, 0x62, 0x77, 0x16, 0x98,
+ 0x1c, 0xfb, 0xd4, 0x36, 0x19, 0x75, 0x4d, 0x7b, 0xe4, 0xcf, 0xd8, 0x23, 0x0c, 0x0e, 0x38, 0xa0,
+ 0x96, 0x65, 0x32, 0x0e, 0x18, 0x5b, 0x19, 0x70, 0x86, 0xa1, 0x6c, 0xc3, 0x95, 0x60, 0xd7, 0xe8,
+ 0x93, 0x76, 0xf4, 0x45, 0x0a, 0x1f, 0x0e, 0x02, 0x28, 0xca, 0xe9, 0x3e, 0x76, 0x88, 0xd2, 0x12,
+ 0x3d, 0x02, 0x90, 0xea, 0x74, 0x31, 0xaf, 0x8c, 0xdf, 0x1e, 0x95, 0x56, 0x6b, 0x4f, 0xd7, 0x9a,
+ 0x41, 0x7b, 0xd4, 0x9b, 0x5d, 0xbd, 0xa6, 0x63, 0x39, 0xc6, 0xa5, 0x9a, 0x07, 0xfb, 0x15, 0x1d,
+ 0xcb, 0x71, 0x94, 0x85, 0xa4, 0x86, 0xb1, 0xf6, 0x81, 0x9c, 0xe0, 0xe4, 0x56, 0xa5, 0xa1, 0xef,
+ 0x76, 0xe5, 0xa4, 0xf2, 0x87, 0x04, 0xc5, 0xa8, 0x9d, 0xa3, 0x1d, 0x88, 0x0f, 0x37, 0xff, 0xdd,
+ 0x0d, 0x2a, 0xeb, 0x7f, 0xa3, 0x2e, 0x24, 0xd8, 0x63, 0x27, 0xb8, 0x1c, 0x85, 0xf2, 0x3b, 0x0b,
+ 0x17, 0xef, 0x64, 0x30, 0xc1, 0xab, 0xe1, 0xa3, 0xa1, 0x6d, 0xc8, 0x0d, 0x85, 0x5d, 0x93, 0x86,
+ 0xab, 0xd4, 0x3c, 0x09, 0xbd, 0x08, 0xf9, 0x21, 0x39, 0x34, 0x26, 0x63, 0xd6, 0x9b, 0x1a, 0xe3,
+ 0x09, 0x11, 0xe3, 0xee, 0xa2, 0x20, 0xbe, 0xc7, 0x69, 0xe8, 0x12, 0x24, 0x47, 0x2e, 0x9d, 0x38,
+ 0xc5, 0x64, 0xd0, 0x8b, 0xfe, 0x41, 0xf9, 0x12, 0x9e, 0x3f, 0x33, 0xd9, 0xa2, 0x2b, 0x3f, 0x82,
+ 0x74, 0xb8, 0x41, 0x04, 0xfd, 0xa2, 0xad, 0x18, 0xd8, 0xdc, 0x06, 0x19, 0x22, 0x96, 0xbf, 0xc9,
+ 0xf9, 0x2f, 0x8a, 0xd8, 0x31, 0xd1, 0xd7, 0x12, 0xa4, 0xc5, 0xce, 0x8b, 0xee, 0x2c, 0x6e, 0xe6,
+ 0xf8, 0x8e, 0x5f, 0xba, 0xbb, 0x92, 0xae, 0x08, 0x78, 0x0a, 0x71, 0x3c, 0xb1, 0xd1, 0xe2, 0x97,
+ 0x2f, 0xb2, 0xbf, 0x96, 0x6e, 0x2f, 0xad, 0x27, 0xec, 0x7e, 0x01, 0x69, 0xb1, 0x0f, 0xa1, 0xdb,
+ 0x4b, 0x0e, 0xd6, 0xf0, 0x6a, 0x94, 0x5e, 0x5f, 0x5e, 0x51, 0x58, 0xff, 0x56, 0x82, 0x4c, 0x8d,
+ 0x30, 0x7f, 0xe0, 0xa2, 0xbb, 0xab, 0x0d, 0xf6, 0xc0, 0x87, 0x27, 0x7a, 0x15, 0xd0, 0xf7, 0x12,
+ 0xe4, 0x6a, 0x84, 0x85, 0xad, 0x83, 0xee, 0x2d, 0x89, 0x76, 0x6c, 0xb5, 0x2a, 0xbd, 0xbd, 0xb2,
+ 0xbe, 0x70, 0xe8, 0x2b, 0x48, 0x05, 0xbb, 0x3e, 0x7a, 0x63, 0x61, 0xa8, 0xe3, 0xff, 0x23, 0x4a,
+ 0x77, 0x56, 0x51, 0x15, 0x0e, 0x7c, 0x27, 0xf9, 0xab, 0xb3, 0x9f, 0xa6, 0x0e, 0x73, 0x89, 0x61,
+ 0xfd, 0x8f, 0xf5, 0xb9, 0x29, 0xa1, 0x1f, 0x24, 0x90, 0x6b, 0x84, 0x89, 0x7d, 0x61, 0x69, 0x8f,
+ 0x4e, 0xae, 0x9b, 0x4b, 0x78, 0x74, 0xca, 0xfe, 0x73, 0x53, 0xe2, 0x3d, 0x93, 0x8f, 0x3c, 0xa9,
+ 0xe8, 0xad, 0x25, 0x63, 0x8c, 0xbe, 0xdc, 0xa5, 0x7b, 0xab, 0xaa, 0x8b, 0x92, 0xfd, 0x2c, 0xc1,
+ 0xd6, 0x19, 0x73, 0x15, 0xd5, 0x16, 0xc6, 0xfe, 0xf7, 0x67, 0xb0, 0xf4, 0xe0, 0xc9, 0x81, 0xc4,
+ 0xe2, 0x55, 0x81, 0x97, 0xcf, 0x84, 0x8a, 0x20, 0x55, 0x52, 0x0d, 0xda, 0xd7, 0x1c, 0xf3, 0x43,
+ 0x39, 0xc2, 0xe9, 0x4d, 0x77, 0xfa, 0x29, 0xff, 0xff, 0xe4, 0xad, 0x7f, 0x02, 0x00, 0x00, 0xff,
+ 0xff, 0xa3, 0xce, 0x16, 0x90, 0x07, 0x12, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go b/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
index 103e3ea..8d8face 100644
--- a/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
+++ b/sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
@@ -51,7 +51,7 @@
return proto.EnumName(BeamConstants_Constants_name, int32(x))
}
func (BeamConstants_Constants) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{0, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{0, 0}
}
type StandardPTransforms_Primitives int32
@@ -130,7 +130,7 @@
return proto.EnumName(StandardPTransforms_Primitives_name, int32(x))
}
func (StandardPTransforms_Primitives) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{4, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{4, 0}
}
type StandardPTransforms_DeprecatedPrimitives int32
@@ -157,7 +157,7 @@
return proto.EnumName(StandardPTransforms_DeprecatedPrimitives_name, int32(x))
}
func (StandardPTransforms_DeprecatedPrimitives) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{4, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{4, 1}
}
type StandardPTransforms_Composites int32
@@ -196,63 +196,53 @@
return proto.EnumName(StandardPTransforms_Composites_name, int32(x))
}
func (StandardPTransforms_Composites) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{4, 2}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{4, 2}
}
// Payload for all of these: CombinePayload
type StandardPTransforms_CombineComponents int32
const (
- // TODO(BEAM-6199): Remove these old URNs.
- StandardPTransforms_COMBINE_PGBKCV StandardPTransforms_CombineComponents = 0
- StandardPTransforms_COMBINE_MERGE_ACCUMULATORS StandardPTransforms_CombineComponents = 1
- StandardPTransforms_COMBINE_EXTRACT_OUTPUTS StandardPTransforms_CombineComponents = 2
// Represents the Pre-Combine part of a lifted Combine Per Key, as described
// in the following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.ta0g6ase8z07
// Payload: CombinePayload
- StandardPTransforms_COMBINE_PER_KEY_PRECOMBINE StandardPTransforms_CombineComponents = 3
+ StandardPTransforms_COMBINE_PER_KEY_PRECOMBINE StandardPTransforms_CombineComponents = 0
// Represents the Merge Accumulators part of a lifted Combine Per Key, as
// described in the following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.jco9rvatld5m
// Payload: CombinePayload
- StandardPTransforms_COMBINE_PER_KEY_MERGE_ACCUMULATORS StandardPTransforms_CombineComponents = 4
+ StandardPTransforms_COMBINE_PER_KEY_MERGE_ACCUMULATORS StandardPTransforms_CombineComponents = 1
// Represents the Extract Outputs part of a lifted Combine Per Key, as
// described in the following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.i9i6p8gtl6ku
// Payload: CombinePayload
- StandardPTransforms_COMBINE_PER_KEY_EXTRACT_OUTPUTS StandardPTransforms_CombineComponents = 5
+ StandardPTransforms_COMBINE_PER_KEY_EXTRACT_OUTPUTS StandardPTransforms_CombineComponents = 2
// Represents the Combine Grouped Values transform, as described in the
// following document:
// https://s.apache.org/beam-runner-api-combine-model#heading=h.aj86ew4v1wk
// Payload: CombinePayload
- StandardPTransforms_COMBINE_GROUPED_VALUES StandardPTransforms_CombineComponents = 6
+ StandardPTransforms_COMBINE_GROUPED_VALUES StandardPTransforms_CombineComponents = 3
)
var StandardPTransforms_CombineComponents_name = map[int32]string{
- 0: "COMBINE_PGBKCV",
- 1: "COMBINE_MERGE_ACCUMULATORS",
- 2: "COMBINE_EXTRACT_OUTPUTS",
- 3: "COMBINE_PER_KEY_PRECOMBINE",
- 4: "COMBINE_PER_KEY_MERGE_ACCUMULATORS",
- 5: "COMBINE_PER_KEY_EXTRACT_OUTPUTS",
- 6: "COMBINE_GROUPED_VALUES",
+ 0: "COMBINE_PER_KEY_PRECOMBINE",
+ 1: "COMBINE_PER_KEY_MERGE_ACCUMULATORS",
+ 2: "COMBINE_PER_KEY_EXTRACT_OUTPUTS",
+ 3: "COMBINE_GROUPED_VALUES",
}
var StandardPTransforms_CombineComponents_value = map[string]int32{
- "COMBINE_PGBKCV": 0,
- "COMBINE_MERGE_ACCUMULATORS": 1,
- "COMBINE_EXTRACT_OUTPUTS": 2,
- "COMBINE_PER_KEY_PRECOMBINE": 3,
- "COMBINE_PER_KEY_MERGE_ACCUMULATORS": 4,
- "COMBINE_PER_KEY_EXTRACT_OUTPUTS": 5,
- "COMBINE_GROUPED_VALUES": 6,
+ "COMBINE_PER_KEY_PRECOMBINE": 0,
+ "COMBINE_PER_KEY_MERGE_ACCUMULATORS": 1,
+ "COMBINE_PER_KEY_EXTRACT_OUTPUTS": 2,
+ "COMBINE_GROUPED_VALUES": 3,
}
func (x StandardPTransforms_CombineComponents) String() string {
return proto.EnumName(StandardPTransforms_CombineComponents_name, int32(x))
}
func (StandardPTransforms_CombineComponents) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{4, 3}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{4, 3}
}
// Payload for all of these: ParDoPayload containing the user's SDF
@@ -312,7 +302,7 @@
return proto.EnumName(StandardPTransforms_SplittableParDoComponents_name, int32(x))
}
func (StandardPTransforms_SplittableParDoComponents) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{4, 4}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{4, 4}
}
type StandardSideInputTypes_Enum int32
@@ -335,7 +325,7 @@
return proto.EnumName(StandardSideInputTypes_Enum_name, int32(x))
}
func (StandardSideInputTypes_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{5, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{5, 0}
}
type Parameter_Type_Enum int32
@@ -364,7 +354,7 @@
return proto.EnumName(Parameter_Type_Enum_name, int32(x))
}
func (Parameter_Type_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{8, 0, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{8, 0, 0}
}
type IsBounded_Enum int32
@@ -390,7 +380,7 @@
return proto.EnumName(IsBounded_Enum_name, int32(x))
}
func (IsBounded_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{16, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{16, 0}
}
type StandardCoders_Enum int32
@@ -419,7 +409,7 @@
// If the length is unknown, it is batched up into groups of size b1..bM
// and encoded as
//
- // fixed32(0)
+ // fixed32(-1)
// varInt64(b1) encode(e1) encode(e2) ... encode(e_b1)
// varInt64(b2) encode(e_(b1+1)) encode(e_(b1+2)) ... encode(e_(b1+b2))
// ...
@@ -507,69 +497,7 @@
return proto.EnumName(StandardCoders_Enum_name, int32(x))
}
func (StandardCoders_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{23, 0}
-}
-
-type Schema_TypeName int32
-
-const (
- Schema_BYTE Schema_TypeName = 0
- Schema_INT16 Schema_TypeName = 1
- Schema_INT32 Schema_TypeName = 2
- Schema_INT64 Schema_TypeName = 3
- Schema_DECIMAL Schema_TypeName = 4
- Schema_FLOAT Schema_TypeName = 5
- Schema_DOUBLE Schema_TypeName = 6
- Schema_STRING Schema_TypeName = 7
- Schema_DATETIME Schema_TypeName = 8
- Schema_BOOLEAN Schema_TypeName = 9
- Schema_BYTES Schema_TypeName = 10
- Schema_ARRAY Schema_TypeName = 11
- Schema_MAP Schema_TypeName = 13
- Schema_ROW Schema_TypeName = 14
- Schema_LOGICAL_TYPE Schema_TypeName = 15
-)
-
-var Schema_TypeName_name = map[int32]string{
- 0: "BYTE",
- 1: "INT16",
- 2: "INT32",
- 3: "INT64",
- 4: "DECIMAL",
- 5: "FLOAT",
- 6: "DOUBLE",
- 7: "STRING",
- 8: "DATETIME",
- 9: "BOOLEAN",
- 10: "BYTES",
- 11: "ARRAY",
- 13: "MAP",
- 14: "ROW",
- 15: "LOGICAL_TYPE",
-}
-var Schema_TypeName_value = map[string]int32{
- "BYTE": 0,
- "INT16": 1,
- "INT32": 2,
- "INT64": 3,
- "DECIMAL": 4,
- "FLOAT": 5,
- "DOUBLE": 6,
- "STRING": 7,
- "DATETIME": 8,
- "BOOLEAN": 9,
- "BYTES": 10,
- "ARRAY": 11,
- "MAP": 13,
- "ROW": 14,
- "LOGICAL_TYPE": 15,
-}
-
-func (x Schema_TypeName) String() string {
- return proto.EnumName(Schema_TypeName_name, int32(x))
-}
-func (Schema_TypeName) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{24, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{23, 0}
}
type MergeStatus_Enum int32
@@ -606,7 +534,7 @@
return proto.EnumName(MergeStatus_Enum_name, int32(x))
}
func (MergeStatus_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{26, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{25, 0}
}
type AccumulationMode_Enum int32
@@ -617,24 +545,28 @@
AccumulationMode_DISCARDING AccumulationMode_Enum = 1
// The aggregation is accumulated across outputs
AccumulationMode_ACCUMULATING AccumulationMode_Enum = 2
+ // The aggregation emits retractions when it is output
+ AccumulationMode_RETRACTING AccumulationMode_Enum = 3
)
var AccumulationMode_Enum_name = map[int32]string{
0: "UNSPECIFIED",
1: "DISCARDING",
2: "ACCUMULATING",
+ 3: "RETRACTING",
}
var AccumulationMode_Enum_value = map[string]int32{
"UNSPECIFIED": 0,
"DISCARDING": 1,
"ACCUMULATING": 2,
+ "RETRACTING": 3,
}
func (x AccumulationMode_Enum) String() string {
return proto.EnumName(AccumulationMode_Enum_name, int32(x))
}
func (AccumulationMode_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{27, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{26, 0}
}
type ClosingBehavior_Enum int32
@@ -663,7 +595,7 @@
return proto.EnumName(ClosingBehavior_Enum_name, int32(x))
}
func (ClosingBehavior_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{28, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{27, 0}
}
type OnTimeBehavior_Enum int32
@@ -692,7 +624,7 @@
return proto.EnumName(OnTimeBehavior_Enum_name, int32(x))
}
func (OnTimeBehavior_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{29, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{28, 0}
}
type OutputTime_Enum int32
@@ -726,7 +658,7 @@
return proto.EnumName(OutputTime_Enum_name, int32(x))
}
func (OutputTime_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{30, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{29, 0}
}
type TimeDomain_Enum int32
@@ -763,7 +695,7 @@
return proto.EnumName(TimeDomain_Enum_name, int32(x))
}
func (TimeDomain_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{31, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{30, 0}
}
type StandardEnvironments_Environments int32
@@ -789,7 +721,7 @@
return proto.EnumName(StandardEnvironments_Environments_name, int32(x))
}
func (StandardEnvironments_Environments) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{36, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{35, 0}
}
type DisplayData_Type_Enum int32
@@ -830,7 +762,7 @@
return proto.EnumName(DisplayData_Type_Enum_name, int32(x))
}
func (DisplayData_Type_Enum) EnumDescriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{42, 2, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{41, 2, 0}
}
type BeamConstants struct {
@@ -843,7 +775,7 @@
func (m *BeamConstants) String() string { return proto.CompactTextString(m) }
func (*BeamConstants) ProtoMessage() {}
func (*BeamConstants) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{0}
}
func (m *BeamConstants) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_BeamConstants.Unmarshal(m, b)
@@ -885,7 +817,7 @@
func (m *Components) String() string { return proto.CompactTextString(m) }
func (*Components) ProtoMessage() {}
func (*Components) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{1}
}
func (m *Components) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Components.Unmarshal(m, b)
@@ -969,7 +901,7 @@
func (m *Pipeline) String() string { return proto.CompactTextString(m) }
func (*Pipeline) ProtoMessage() {}
func (*Pipeline) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{2}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{2}
}
func (m *Pipeline) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Pipeline.Unmarshal(m, b)
@@ -1039,7 +971,7 @@
// For some special composite transforms, the payload is also officially
// defined:
//
- // - when the URN is "urn:beam:transforms:combine" it is a CombinePayload
+ // - when the URN is "beam:transforms:combine" it is a CombinePayload
//
Spec *FunctionSpec `protobuf:"bytes,1,opt,name=spec,proto3" json:"spec,omitempty"`
// (Optional) if this node is a composite, a list of the ids of
@@ -1083,7 +1015,7 @@
func (m *PTransform) String() string { return proto.CompactTextString(m) }
func (*PTransform) ProtoMessage() {}
func (*PTransform) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{3}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{3}
}
func (m *PTransform) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PTransform.Unmarshal(m, b)
@@ -1155,7 +1087,7 @@
func (m *StandardPTransforms) String() string { return proto.CompactTextString(m) }
func (*StandardPTransforms) ProtoMessage() {}
func (*StandardPTransforms) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{4}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{4}
}
func (m *StandardPTransforms) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardPTransforms.Unmarshal(m, b)
@@ -1185,7 +1117,7 @@
func (m *StandardSideInputTypes) String() string { return proto.CompactTextString(m) }
func (*StandardSideInputTypes) ProtoMessage() {}
func (*StandardSideInputTypes) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{5}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{5}
}
func (m *StandardSideInputTypes) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardSideInputTypes.Unmarshal(m, b)
@@ -1235,7 +1167,7 @@
func (m *PCollection) String() string { return proto.CompactTextString(m) }
func (*PCollection) ProtoMessage() {}
func (*PCollection) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{6}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{6}
}
func (m *PCollection) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_PCollection.Unmarshal(m, b)
@@ -1320,7 +1252,7 @@
func (m *ParDoPayload) String() string { return proto.CompactTextString(m) }
func (*ParDoPayload) ProtoMessage() {}
func (*ParDoPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{7}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{7}
}
func (m *ParDoPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ParDoPayload.Unmarshal(m, b)
@@ -1421,7 +1353,7 @@
func (m *Parameter) String() string { return proto.CompactTextString(m) }
func (*Parameter) ProtoMessage() {}
func (*Parameter) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{8}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{8}
}
func (m *Parameter) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Parameter.Unmarshal(m, b)
@@ -1458,7 +1390,7 @@
func (m *Parameter_Type) String() string { return proto.CompactTextString(m) }
func (*Parameter_Type) ProtoMessage() {}
func (*Parameter_Type) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{8, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{8, 0}
}
func (m *Parameter_Type) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Parameter_Type.Unmarshal(m, b)
@@ -1495,7 +1427,7 @@
func (m *StateSpec) String() string { return proto.CompactTextString(m) }
func (*StateSpec) ProtoMessage() {}
func (*StateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{9}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{9}
}
func (m *StateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StateSpec.Unmarshal(m, b)
@@ -1725,7 +1657,7 @@
func (m *ValueStateSpec) String() string { return proto.CompactTextString(m) }
func (*ValueStateSpec) ProtoMessage() {}
func (*ValueStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{10}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{10}
}
func (m *ValueStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ValueStateSpec.Unmarshal(m, b)
@@ -1763,7 +1695,7 @@
func (m *BagStateSpec) String() string { return proto.CompactTextString(m) }
func (*BagStateSpec) ProtoMessage() {}
func (*BagStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{11}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{11}
}
func (m *BagStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_BagStateSpec.Unmarshal(m, b)
@@ -1802,7 +1734,7 @@
func (m *CombiningStateSpec) String() string { return proto.CompactTextString(m) }
func (*CombiningStateSpec) ProtoMessage() {}
func (*CombiningStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{12}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{12}
}
func (m *CombiningStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CombiningStateSpec.Unmarshal(m, b)
@@ -1848,7 +1780,7 @@
func (m *MapStateSpec) String() string { return proto.CompactTextString(m) }
func (*MapStateSpec) ProtoMessage() {}
func (*MapStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{13}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{13}
}
func (m *MapStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MapStateSpec.Unmarshal(m, b)
@@ -1893,7 +1825,7 @@
func (m *SetStateSpec) String() string { return proto.CompactTextString(m) }
func (*SetStateSpec) ProtoMessage() {}
func (*SetStateSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{14}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{14}
}
func (m *SetStateSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_SetStateSpec.Unmarshal(m, b)
@@ -1932,7 +1864,7 @@
func (m *TimerSpec) String() string { return proto.CompactTextString(m) }
func (*TimerSpec) ProtoMessage() {}
func (*TimerSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{15}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{15}
}
func (m *TimerSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimerSpec.Unmarshal(m, b)
@@ -1976,7 +1908,7 @@
func (m *IsBounded) String() string { return proto.CompactTextString(m) }
func (*IsBounded) ProtoMessage() {}
func (*IsBounded) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{16}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{16}
}
func (m *IsBounded) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_IsBounded.Unmarshal(m, b)
@@ -2011,7 +1943,7 @@
func (m *ReadPayload) String() string { return proto.CompactTextString(m) }
func (*ReadPayload) ProtoMessage() {}
func (*ReadPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{17}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{17}
}
func (m *ReadPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ReadPayload.Unmarshal(m, b)
@@ -2058,7 +1990,7 @@
func (m *WindowIntoPayload) String() string { return proto.CompactTextString(m) }
func (*WindowIntoPayload) ProtoMessage() {}
func (*WindowIntoPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{18}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{18}
}
func (m *WindowIntoPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_WindowIntoPayload.Unmarshal(m, b)
@@ -2100,7 +2032,7 @@
func (m *CombinePayload) String() string { return proto.CompactTextString(m) }
func (*CombinePayload) ProtoMessage() {}
func (*CombinePayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{19}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{19}
}
func (m *CombinePayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_CombinePayload.Unmarshal(m, b)
@@ -2148,7 +2080,7 @@
func (m *TestStreamPayload) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload) ProtoMessage() {}
func (*TestStreamPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{20}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{20}
}
func (m *TestStreamPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload.Unmarshal(m, b)
@@ -2197,7 +2129,7 @@
func (m *TestStreamPayload_Event) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_Event) ProtoMessage() {}
func (*TestStreamPayload_Event) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{20, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{20, 0}
}
func (m *TestStreamPayload_Event) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event.Unmarshal(m, b)
@@ -2369,7 +2301,7 @@
func (m *TestStreamPayload_Event_AdvanceWatermark) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_Event_AdvanceWatermark) ProtoMessage() {}
func (*TestStreamPayload_Event_AdvanceWatermark) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{20, 0, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{20, 0, 0}
}
func (m *TestStreamPayload_Event_AdvanceWatermark) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event_AdvanceWatermark.Unmarshal(m, b)
@@ -2411,7 +2343,7 @@
}
func (*TestStreamPayload_Event_AdvanceProcessingTime) ProtoMessage() {}
func (*TestStreamPayload_Event_AdvanceProcessingTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{20, 0, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{20, 0, 1}
}
func (m *TestStreamPayload_Event_AdvanceProcessingTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event_AdvanceProcessingTime.Unmarshal(m, b)
@@ -2449,7 +2381,7 @@
func (m *TestStreamPayload_Event_AddElements) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_Event_AddElements) ProtoMessage() {}
func (*TestStreamPayload_Event_AddElements) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{20, 0, 2}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{20, 0, 2}
}
func (m *TestStreamPayload_Event_AddElements) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_Event_AddElements.Unmarshal(m, b)
@@ -2488,7 +2420,7 @@
func (m *TestStreamPayload_TimestampedElement) String() string { return proto.CompactTextString(m) }
func (*TestStreamPayload_TimestampedElement) ProtoMessage() {}
func (*TestStreamPayload_TimestampedElement) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{20, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{20, 1}
}
func (m *TestStreamPayload_TimestampedElement) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TestStreamPayload_TimestampedElement.Unmarshal(m, b)
@@ -2540,7 +2472,7 @@
func (m *WriteFilesPayload) String() string { return proto.CompactTextString(m) }
func (*WriteFilesPayload) ProtoMessage() {}
func (*WriteFilesPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{21}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{21}
}
func (m *WriteFilesPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_WriteFilesPayload.Unmarshal(m, b)
@@ -2602,8 +2534,8 @@
// may be a cross-language agreed-upon format, or it may be a "custom coder"
// that can only be used by a particular SDK. It does not include component
// coders, as it is beneficial for these to be comprehensible to a runner
- // regardless of whether the binary format is agree-upon.
- Spec *SdkFunctionSpec `protobuf:"bytes,1,opt,name=spec,proto3" json:"spec,omitempty"`
+ // regardless of whether the binary format is agreed-upon.
+ Spec *FunctionSpec `protobuf:"bytes,1,opt,name=spec,proto3" json:"spec,omitempty"`
// (Optional) If this coder is parametric, such as ListCoder(VarIntCoder),
// this is a list of the components. In order for encodings to be identical,
// the SdkFunctionSpec and all components must be identical, recursively.
@@ -2617,7 +2549,7 @@
func (m *Coder) String() string { return proto.CompactTextString(m) }
func (*Coder) ProtoMessage() {}
func (*Coder) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{22}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{22}
}
func (m *Coder) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Coder.Unmarshal(m, b)
@@ -2637,7 +2569,7 @@
var xxx_messageInfo_Coder proto.InternalMessageInfo
-func (m *Coder) GetSpec() *SdkFunctionSpec {
+func (m *Coder) GetSpec() *FunctionSpec {
if m != nil {
return m.Spec
}
@@ -2661,7 +2593,7 @@
func (m *StandardCoders) String() string { return proto.CompactTextString(m) }
func (*StandardCoders) ProtoMessage() {}
func (*StandardCoders) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{23}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{23}
}
func (m *StandardCoders) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardCoders.Unmarshal(m, b)
@@ -2681,452 +2613,6 @@
var xxx_messageInfo_StandardCoders proto.InternalMessageInfo
-// Experimental: A representation of a Beam Schema.
-type Schema struct {
- Fields []*Schema_Field `protobuf:"bytes,1,rep,name=fields,proto3" json:"fields,omitempty"`
- Id string `protobuf:"bytes,2,opt,name=id,proto3" json:"id,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema) Reset() { *m = Schema{} }
-func (m *Schema) String() string { return proto.CompactTextString(m) }
-func (*Schema) ProtoMessage() {}
-func (*Schema) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{24}
-}
-func (m *Schema) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema.Unmarshal(m, b)
-}
-func (m *Schema) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema.Marshal(b, m, deterministic)
-}
-func (dst *Schema) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema.Merge(dst, src)
-}
-func (m *Schema) XXX_Size() int {
- return xxx_messageInfo_Schema.Size(m)
-}
-func (m *Schema) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema proto.InternalMessageInfo
-
-func (m *Schema) GetFields() []*Schema_Field {
- if m != nil {
- return m.Fields
- }
- return nil
-}
-
-func (m *Schema) GetId() string {
- if m != nil {
- return m.Id
- }
- return ""
-}
-
-type Schema_LogicalType struct {
- Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
- Args string `protobuf:"bytes,2,opt,name=args,proto3" json:"args,omitempty"`
- BaseType *Schema_FieldType `protobuf:"bytes,3,opt,name=base_type,json=baseType,proto3" json:"base_type,omitempty"`
- SerializedClass []byte `protobuf:"bytes,4,opt,name=serialized_class,json=serializedClass,proto3" json:"serialized_class,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_LogicalType) Reset() { *m = Schema_LogicalType{} }
-func (m *Schema_LogicalType) String() string { return proto.CompactTextString(m) }
-func (*Schema_LogicalType) ProtoMessage() {}
-func (*Schema_LogicalType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{24, 0}
-}
-func (m *Schema_LogicalType) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_LogicalType.Unmarshal(m, b)
-}
-func (m *Schema_LogicalType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_LogicalType.Marshal(b, m, deterministic)
-}
-func (dst *Schema_LogicalType) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_LogicalType.Merge(dst, src)
-}
-func (m *Schema_LogicalType) XXX_Size() int {
- return xxx_messageInfo_Schema_LogicalType.Size(m)
-}
-func (m *Schema_LogicalType) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_LogicalType.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_LogicalType proto.InternalMessageInfo
-
-func (m *Schema_LogicalType) GetId() string {
- if m != nil {
- return m.Id
- }
- return ""
-}
-
-func (m *Schema_LogicalType) GetArgs() string {
- if m != nil {
- return m.Args
- }
- return ""
-}
-
-func (m *Schema_LogicalType) GetBaseType() *Schema_FieldType {
- if m != nil {
- return m.BaseType
- }
- return nil
-}
-
-func (m *Schema_LogicalType) GetSerializedClass() []byte {
- if m != nil {
- return m.SerializedClass
- }
- return nil
-}
-
-type Schema_MapType struct {
- KeyType *Schema_FieldType `protobuf:"bytes,1,opt,name=key_type,json=keyType,proto3" json:"key_type,omitempty"`
- ValueType *Schema_FieldType `protobuf:"bytes,2,opt,name=value_type,json=valueType,proto3" json:"value_type,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_MapType) Reset() { *m = Schema_MapType{} }
-func (m *Schema_MapType) String() string { return proto.CompactTextString(m) }
-func (*Schema_MapType) ProtoMessage() {}
-func (*Schema_MapType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{24, 1}
-}
-func (m *Schema_MapType) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_MapType.Unmarshal(m, b)
-}
-func (m *Schema_MapType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_MapType.Marshal(b, m, deterministic)
-}
-func (dst *Schema_MapType) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_MapType.Merge(dst, src)
-}
-func (m *Schema_MapType) XXX_Size() int {
- return xxx_messageInfo_Schema_MapType.Size(m)
-}
-func (m *Schema_MapType) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_MapType.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_MapType proto.InternalMessageInfo
-
-func (m *Schema_MapType) GetKeyType() *Schema_FieldType {
- if m != nil {
- return m.KeyType
- }
- return nil
-}
-
-func (m *Schema_MapType) GetValueType() *Schema_FieldType {
- if m != nil {
- return m.ValueType
- }
- return nil
-}
-
-type Schema_FieldType struct {
- TypeName Schema_TypeName `protobuf:"varint,1,opt,name=type_name,json=typeName,proto3,enum=org.apache.beam.model.pipeline.v1.Schema_TypeName" json:"type_name,omitempty"`
- Nullable bool `protobuf:"varint,2,opt,name=nullable,proto3" json:"nullable,omitempty"`
- // Types that are valid to be assigned to TypeInfo:
- // *Schema_FieldType_CollectionElementType
- // *Schema_FieldType_MapType
- // *Schema_FieldType_RowSchema
- // *Schema_FieldType_LogicalType
- TypeInfo isSchema_FieldType_TypeInfo `protobuf_oneof:"type_info"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_FieldType) Reset() { *m = Schema_FieldType{} }
-func (m *Schema_FieldType) String() string { return proto.CompactTextString(m) }
-func (*Schema_FieldType) ProtoMessage() {}
-func (*Schema_FieldType) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{24, 2}
-}
-func (m *Schema_FieldType) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_FieldType.Unmarshal(m, b)
-}
-func (m *Schema_FieldType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_FieldType.Marshal(b, m, deterministic)
-}
-func (dst *Schema_FieldType) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_FieldType.Merge(dst, src)
-}
-func (m *Schema_FieldType) XXX_Size() int {
- return xxx_messageInfo_Schema_FieldType.Size(m)
-}
-func (m *Schema_FieldType) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_FieldType.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_FieldType proto.InternalMessageInfo
-
-type isSchema_FieldType_TypeInfo interface {
- isSchema_FieldType_TypeInfo()
-}
-
-type Schema_FieldType_CollectionElementType struct {
- CollectionElementType *Schema_FieldType `protobuf:"bytes,3,opt,name=collection_element_type,json=collectionElementType,proto3,oneof"`
-}
-type Schema_FieldType_MapType struct {
- MapType *Schema_MapType `protobuf:"bytes,4,opt,name=map_type,json=mapType,proto3,oneof"`
-}
-type Schema_FieldType_RowSchema struct {
- RowSchema *Schema `protobuf:"bytes,5,opt,name=row_schema,json=rowSchema,proto3,oneof"`
-}
-type Schema_FieldType_LogicalType struct {
- LogicalType *Schema_LogicalType `protobuf:"bytes,6,opt,name=logical_type,json=logicalType,proto3,oneof"`
-}
-
-func (*Schema_FieldType_CollectionElementType) isSchema_FieldType_TypeInfo() {}
-func (*Schema_FieldType_MapType) isSchema_FieldType_TypeInfo() {}
-func (*Schema_FieldType_RowSchema) isSchema_FieldType_TypeInfo() {}
-func (*Schema_FieldType_LogicalType) isSchema_FieldType_TypeInfo() {}
-
-func (m *Schema_FieldType) GetTypeInfo() isSchema_FieldType_TypeInfo {
- if m != nil {
- return m.TypeInfo
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetTypeName() Schema_TypeName {
- if m != nil {
- return m.TypeName
- }
- return Schema_BYTE
-}
-
-func (m *Schema_FieldType) GetNullable() bool {
- if m != nil {
- return m.Nullable
- }
- return false
-}
-
-func (m *Schema_FieldType) GetCollectionElementType() *Schema_FieldType {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_CollectionElementType); ok {
- return x.CollectionElementType
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetMapType() *Schema_MapType {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_MapType); ok {
- return x.MapType
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetRowSchema() *Schema {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_RowSchema); ok {
- return x.RowSchema
- }
- return nil
-}
-
-func (m *Schema_FieldType) GetLogicalType() *Schema_LogicalType {
- if x, ok := m.GetTypeInfo().(*Schema_FieldType_LogicalType); ok {
- return x.LogicalType
- }
- return nil
-}
-
-// XXX_OneofFuncs is for the internal use of the proto package.
-func (*Schema_FieldType) XXX_OneofFuncs() (func(msg proto.Message, b *proto.Buffer) error, func(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error), func(msg proto.Message) (n int), []interface{}) {
- return _Schema_FieldType_OneofMarshaler, _Schema_FieldType_OneofUnmarshaler, _Schema_FieldType_OneofSizer, []interface{}{
- (*Schema_FieldType_CollectionElementType)(nil),
- (*Schema_FieldType_MapType)(nil),
- (*Schema_FieldType_RowSchema)(nil),
- (*Schema_FieldType_LogicalType)(nil),
- }
-}
-
-func _Schema_FieldType_OneofMarshaler(msg proto.Message, b *proto.Buffer) error {
- m := msg.(*Schema_FieldType)
- // type_info
- switch x := m.TypeInfo.(type) {
- case *Schema_FieldType_CollectionElementType:
- b.EncodeVarint(3<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.CollectionElementType); err != nil {
- return err
- }
- case *Schema_FieldType_MapType:
- b.EncodeVarint(4<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.MapType); err != nil {
- return err
- }
- case *Schema_FieldType_RowSchema:
- b.EncodeVarint(5<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.RowSchema); err != nil {
- return err
- }
- case *Schema_FieldType_LogicalType:
- b.EncodeVarint(6<<3 | proto.WireBytes)
- if err := b.EncodeMessage(x.LogicalType); err != nil {
- return err
- }
- case nil:
- default:
- return fmt.Errorf("Schema_FieldType.TypeInfo has unexpected type %T", x)
- }
- return nil
-}
-
-func _Schema_FieldType_OneofUnmarshaler(msg proto.Message, tag, wire int, b *proto.Buffer) (bool, error) {
- m := msg.(*Schema_FieldType)
- switch tag {
- case 3: // type_info.collection_element_type
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema_FieldType)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_CollectionElementType{msg}
- return true, err
- case 4: // type_info.map_type
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema_MapType)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_MapType{msg}
- return true, err
- case 5: // type_info.row_schema
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_RowSchema{msg}
- return true, err
- case 6: // type_info.logical_type
- if wire != proto.WireBytes {
- return true, proto.ErrInternalBadWireType
- }
- msg := new(Schema_LogicalType)
- err := b.DecodeMessage(msg)
- m.TypeInfo = &Schema_FieldType_LogicalType{msg}
- return true, err
- default:
- return false, nil
- }
-}
-
-func _Schema_FieldType_OneofSizer(msg proto.Message) (n int) {
- m := msg.(*Schema_FieldType)
- // type_info
- switch x := m.TypeInfo.(type) {
- case *Schema_FieldType_CollectionElementType:
- s := proto.Size(x.CollectionElementType)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case *Schema_FieldType_MapType:
- s := proto.Size(x.MapType)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case *Schema_FieldType_RowSchema:
- s := proto.Size(x.RowSchema)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case *Schema_FieldType_LogicalType:
- s := proto.Size(x.LogicalType)
- n += 1 // tag and wire
- n += proto.SizeVarint(uint64(s))
- n += s
- case nil:
- default:
- panic(fmt.Sprintf("proto: unexpected type %T in oneof", x))
- }
- return n
-}
-
-type Schema_Field struct {
- Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
- Description string `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"`
- Type *Schema_FieldType `protobuf:"bytes,3,opt,name=type,proto3" json:"type,omitempty"`
- Id int32 `protobuf:"varint,4,opt,name=id,proto3" json:"id,omitempty"`
- EncodingPosition int32 `protobuf:"varint,5,opt,name=encoding_position,json=encodingPosition,proto3" json:"encoding_position,omitempty"`
- XXX_NoUnkeyedLiteral struct{} `json:"-"`
- XXX_unrecognized []byte `json:"-"`
- XXX_sizecache int32 `json:"-"`
-}
-
-func (m *Schema_Field) Reset() { *m = Schema_Field{} }
-func (m *Schema_Field) String() string { return proto.CompactTextString(m) }
-func (*Schema_Field) ProtoMessage() {}
-func (*Schema_Field) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{24, 3}
-}
-func (m *Schema_Field) XXX_Unmarshal(b []byte) error {
- return xxx_messageInfo_Schema_Field.Unmarshal(m, b)
-}
-func (m *Schema_Field) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
- return xxx_messageInfo_Schema_Field.Marshal(b, m, deterministic)
-}
-func (dst *Schema_Field) XXX_Merge(src proto.Message) {
- xxx_messageInfo_Schema_Field.Merge(dst, src)
-}
-func (m *Schema_Field) XXX_Size() int {
- return xxx_messageInfo_Schema_Field.Size(m)
-}
-func (m *Schema_Field) XXX_DiscardUnknown() {
- xxx_messageInfo_Schema_Field.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_Schema_Field proto.InternalMessageInfo
-
-func (m *Schema_Field) GetName() string {
- if m != nil {
- return m.Name
- }
- return ""
-}
-
-func (m *Schema_Field) GetDescription() string {
- if m != nil {
- return m.Description
- }
- return ""
-}
-
-func (m *Schema_Field) GetType() *Schema_FieldType {
- if m != nil {
- return m.Type
- }
- return nil
-}
-
-func (m *Schema_Field) GetId() int32 {
- if m != nil {
- return m.Id
- }
- return 0
-}
-
-func (m *Schema_Field) GetEncodingPosition() int32 {
- if m != nil {
- return m.EncodingPosition
- }
- return 0
-}
-
// A windowing strategy describes the window function, triggering, allowed
// lateness, and accumulation mode for a PCollection.
//
@@ -3178,7 +2664,7 @@
func (m *WindowingStrategy) String() string { return proto.CompactTextString(m) }
func (*WindowingStrategy) ProtoMessage() {}
func (*WindowingStrategy) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{25}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{24}
}
func (m *WindowingStrategy) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_WindowingStrategy.Unmarshal(m, b)
@@ -3281,7 +2767,7 @@
func (m *MergeStatus) String() string { return proto.CompactTextString(m) }
func (*MergeStatus) ProtoMessage() {}
func (*MergeStatus) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{26}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{25}
}
func (m *MergeStatus) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MergeStatus.Unmarshal(m, b)
@@ -3314,7 +2800,7 @@
func (m *AccumulationMode) String() string { return proto.CompactTextString(m) }
func (*AccumulationMode) ProtoMessage() {}
func (*AccumulationMode) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{27}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{26}
}
func (m *AccumulationMode) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_AccumulationMode.Unmarshal(m, b)
@@ -3346,7 +2832,7 @@
func (m *ClosingBehavior) String() string { return proto.CompactTextString(m) }
func (*ClosingBehavior) ProtoMessage() {}
func (*ClosingBehavior) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{28}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{27}
}
func (m *ClosingBehavior) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ClosingBehavior.Unmarshal(m, b)
@@ -3378,7 +2864,7 @@
func (m *OnTimeBehavior) String() string { return proto.CompactTextString(m) }
func (*OnTimeBehavior) ProtoMessage() {}
func (*OnTimeBehavior) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{29}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{28}
}
func (m *OnTimeBehavior) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_OnTimeBehavior.Unmarshal(m, b)
@@ -3410,7 +2896,7 @@
func (m *OutputTime) String() string { return proto.CompactTextString(m) }
func (*OutputTime) ProtoMessage() {}
func (*OutputTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{30}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{29}
}
func (m *OutputTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_OutputTime.Unmarshal(m, b)
@@ -3441,7 +2927,7 @@
func (m *TimeDomain) String() string { return proto.CompactTextString(m) }
func (*TimeDomain) ProtoMessage() {}
func (*TimeDomain) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{31}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{30}
}
func (m *TimeDomain) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimeDomain.Unmarshal(m, b)
@@ -3491,7 +2977,7 @@
func (m *Trigger) String() string { return proto.CompactTextString(m) }
func (*Trigger) ProtoMessage() {}
func (*Trigger) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31}
}
func (m *Trigger) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger.Unmarshal(m, b)
@@ -3932,7 +3418,7 @@
func (m *Trigger_AfterAll) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterAll) ProtoMessage() {}
func (*Trigger_AfterAll) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 0}
}
func (m *Trigger_AfterAll) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterAll.Unmarshal(m, b)
@@ -3971,7 +3457,7 @@
func (m *Trigger_AfterAny) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterAny) ProtoMessage() {}
func (*Trigger_AfterAny) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 1}
}
func (m *Trigger_AfterAny) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterAny.Unmarshal(m, b)
@@ -4011,7 +3497,7 @@
func (m *Trigger_AfterEach) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterEach) ProtoMessage() {}
func (*Trigger_AfterEach) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 2}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 2}
}
func (m *Trigger_AfterEach) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterEach.Unmarshal(m, b)
@@ -4057,7 +3543,7 @@
func (m *Trigger_AfterEndOfWindow) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterEndOfWindow) ProtoMessage() {}
func (*Trigger_AfterEndOfWindow) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 3}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 3}
}
func (m *Trigger_AfterEndOfWindow) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterEndOfWindow.Unmarshal(m, b)
@@ -4105,7 +3591,7 @@
func (m *Trigger_AfterProcessingTime) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterProcessingTime) ProtoMessage() {}
func (*Trigger_AfterProcessingTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 4}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 4}
}
func (m *Trigger_AfterProcessingTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterProcessingTime.Unmarshal(m, b)
@@ -4146,7 +3632,7 @@
func (m *Trigger_AfterSynchronizedProcessingTime) String() string { return proto.CompactTextString(m) }
func (*Trigger_AfterSynchronizedProcessingTime) ProtoMessage() {}
func (*Trigger_AfterSynchronizedProcessingTime) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 5}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 5}
}
func (m *Trigger_AfterSynchronizedProcessingTime) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_AfterSynchronizedProcessingTime.Unmarshal(m, b)
@@ -4178,7 +3664,7 @@
func (m *Trigger_Default) String() string { return proto.CompactTextString(m) }
func (*Trigger_Default) ProtoMessage() {}
func (*Trigger_Default) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 6}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 6}
}
func (m *Trigger_Default) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Default.Unmarshal(m, b)
@@ -4210,7 +3696,7 @@
func (m *Trigger_ElementCount) String() string { return proto.CompactTextString(m) }
func (*Trigger_ElementCount) ProtoMessage() {}
func (*Trigger_ElementCount) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 7}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 7}
}
func (m *Trigger_ElementCount) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_ElementCount.Unmarshal(m, b)
@@ -4249,7 +3735,7 @@
func (m *Trigger_Never) String() string { return proto.CompactTextString(m) }
func (*Trigger_Never) ProtoMessage() {}
func (*Trigger_Never) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 8}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 8}
}
func (m *Trigger_Never) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Never.Unmarshal(m, b)
@@ -4281,7 +3767,7 @@
func (m *Trigger_Always) String() string { return proto.CompactTextString(m) }
func (*Trigger_Always) ProtoMessage() {}
func (*Trigger_Always) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 9}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 9}
}
func (m *Trigger_Always) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Always.Unmarshal(m, b)
@@ -4317,7 +3803,7 @@
func (m *Trigger_OrFinally) String() string { return proto.CompactTextString(m) }
func (*Trigger_OrFinally) ProtoMessage() {}
func (*Trigger_OrFinally) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 10}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 10}
}
func (m *Trigger_OrFinally) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_OrFinally.Unmarshal(m, b)
@@ -4365,7 +3851,7 @@
func (m *Trigger_Repeat) String() string { return proto.CompactTextString(m) }
func (*Trigger_Repeat) ProtoMessage() {}
func (*Trigger_Repeat) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{32, 11}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{31, 11}
}
func (m *Trigger_Repeat) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Trigger_Repeat.Unmarshal(m, b)
@@ -4410,7 +3896,7 @@
func (m *TimestampTransform) String() string { return proto.CompactTextString(m) }
func (*TimestampTransform) ProtoMessage() {}
func (*TimestampTransform) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{33}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{32}
}
func (m *TimestampTransform) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimestampTransform.Unmarshal(m, b)
@@ -4551,7 +4037,7 @@
func (m *TimestampTransform_Delay) String() string { return proto.CompactTextString(m) }
func (*TimestampTransform_Delay) ProtoMessage() {}
func (*TimestampTransform_Delay) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{33, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{32, 0}
}
func (m *TimestampTransform_Delay) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimestampTransform_Delay.Unmarshal(m, b)
@@ -4594,7 +4080,7 @@
func (m *TimestampTransform_AlignTo) String() string { return proto.CompactTextString(m) }
func (*TimestampTransform_AlignTo) ProtoMessage() {}
func (*TimestampTransform_AlignTo) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{33, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{32, 1}
}
func (m *TimestampTransform_AlignTo) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_TimestampTransform_AlignTo.Unmarshal(m, b)
@@ -4637,7 +4123,7 @@
// interface for accessing a side input.
//
// The only access pattern intended for Beam, because of its superior
- // performance possibilities, is "urn:beam:sideinput:multimap" (or some such
+ // performance possibilities, is "beam:sideinput:multimap" (or some such
// URN)
AccessPattern *FunctionSpec `protobuf:"bytes,1,opt,name=access_pattern,json=accessPattern,proto3" json:"access_pattern,omitempty"`
// (Required) The SdkFunctionSpec of the UDF that adapts a particular
@@ -4662,7 +4148,7 @@
func (m *SideInput) String() string { return proto.CompactTextString(m) }
func (*SideInput) ProtoMessage() {}
func (*SideInput) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{34}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{33}
}
func (m *SideInput) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_SideInput.Unmarshal(m, b)
@@ -4720,7 +4206,7 @@
func (m *Environment) String() string { return proto.CompactTextString(m) }
func (*Environment) ProtoMessage() {}
func (*Environment) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{35}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{34}
}
func (m *Environment) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_Environment.Unmarshal(m, b)
@@ -4764,7 +4250,7 @@
func (m *StandardEnvironments) String() string { return proto.CompactTextString(m) }
func (*StandardEnvironments) ProtoMessage() {}
func (*StandardEnvironments) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{36}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{35}
}
func (m *StandardEnvironments) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_StandardEnvironments.Unmarshal(m, b)
@@ -4796,7 +4282,7 @@
func (m *DockerPayload) String() string { return proto.CompactTextString(m) }
func (*DockerPayload) ProtoMessage() {}
func (*DockerPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{37}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{36}
}
func (m *DockerPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DockerPayload.Unmarshal(m, b)
@@ -4837,7 +4323,7 @@
func (m *ProcessPayload) String() string { return proto.CompactTextString(m) }
func (*ProcessPayload) ProtoMessage() {}
func (*ProcessPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{38}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{37}
}
func (m *ProcessPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ProcessPayload.Unmarshal(m, b)
@@ -4897,7 +4383,7 @@
func (m *ExternalPayload) String() string { return proto.CompactTextString(m) }
func (*ExternalPayload) ProtoMessage() {}
func (*ExternalPayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{39}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{38}
}
func (m *ExternalPayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExternalPayload.Unmarshal(m, b)
@@ -4948,7 +4434,7 @@
func (m *SdkFunctionSpec) String() string { return proto.CompactTextString(m) }
func (*SdkFunctionSpec) ProtoMessage() {}
func (*SdkFunctionSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{40}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{39}
}
func (m *SdkFunctionSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_SdkFunctionSpec.Unmarshal(m, b)
@@ -4998,12 +4484,12 @@
// one should bear in mind:
//
// 1. The runner understands the URN. For example, it might be
-// a well-known URN like "urn:beam:transform:Top" or
-// "urn:beam:windowfn:FixedWindows" with
+// a well-known URN like "beam:transform:Top" or
+// "beam:windowfn:FixedWindows" with
// an agreed-upon payload (e.g. a number or duration,
// respectively).
// 2. The runner does not understand the URN. It might be an
-// SDK specific URN such as "urn:beam:dofn:javasdk:1.0"
+// SDK specific URN such as "beam:dofn:javasdk:1.0"
// that indicates to the SDK what the payload is,
// such as a serialized Java DoFn from a particular
// version of the Beam Java SDK. The payload will often
@@ -5027,7 +4513,7 @@
func (m *FunctionSpec) String() string { return proto.CompactTextString(m) }
func (*FunctionSpec) ProtoMessage() {}
func (*FunctionSpec) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{41}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{40}
}
func (m *FunctionSpec) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_FunctionSpec.Unmarshal(m, b)
@@ -5074,7 +4560,7 @@
func (m *DisplayData) String() string { return proto.CompactTextString(m) }
func (*DisplayData) ProtoMessage() {}
func (*DisplayData) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{42}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{41}
}
func (m *DisplayData) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData.Unmarshal(m, b)
@@ -5118,7 +4604,7 @@
func (m *DisplayData_Identifier) String() string { return proto.CompactTextString(m) }
func (*DisplayData_Identifier) ProtoMessage() {}
func (*DisplayData_Identifier) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{42, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{41, 0}
}
func (m *DisplayData_Identifier) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData_Identifier.Unmarshal(m, b)
@@ -5182,7 +4668,7 @@
func (m *DisplayData_Item) String() string { return proto.CompactTextString(m) }
func (*DisplayData_Item) ProtoMessage() {}
func (*DisplayData_Item) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{42, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{41, 1}
}
func (m *DisplayData_Item) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData_Item.Unmarshal(m, b)
@@ -5254,7 +4740,7 @@
func (m *DisplayData_Type) String() string { return proto.CompactTextString(m) }
func (*DisplayData_Type) ProtoMessage() {}
func (*DisplayData_Type) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{42, 2}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{41, 2}
}
func (m *DisplayData_Type) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_DisplayData_Type.Unmarshal(m, b)
@@ -5308,7 +4794,7 @@
func (m *MessageWithComponents) String() string { return proto.CompactTextString(m) }
func (*MessageWithComponents) ProtoMessage() {}
func (*MessageWithComponents) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{43}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{42}
}
func (m *MessageWithComponents) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_MessageWithComponents.Unmarshal(m, b)
@@ -5752,7 +5238,7 @@
func (m *ExecutableStagePayload) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload) ProtoMessage() {}
func (*ExecutableStagePayload) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{44}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{43}
}
func (m *ExecutableStagePayload) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload.Unmarshal(m, b)
@@ -5844,7 +5330,7 @@
func (m *ExecutableStagePayload_SideInputId) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload_SideInputId) ProtoMessage() {}
func (*ExecutableStagePayload_SideInputId) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{44, 0}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{43, 0}
}
func (m *ExecutableStagePayload_SideInputId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload_SideInputId.Unmarshal(m, b)
@@ -5894,7 +5380,7 @@
func (m *ExecutableStagePayload_UserStateId) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload_UserStateId) ProtoMessage() {}
func (*ExecutableStagePayload_UserStateId) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{44, 1}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{43, 1}
}
func (m *ExecutableStagePayload_UserStateId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload_UserStateId.Unmarshal(m, b)
@@ -5944,7 +5430,7 @@
func (m *ExecutableStagePayload_TimerId) String() string { return proto.CompactTextString(m) }
func (*ExecutableStagePayload_TimerId) ProtoMessage() {}
func (*ExecutableStagePayload_TimerId) Descriptor() ([]byte, []int) {
- return fileDescriptor_beam_runner_api_198a59238d98c078, []int{44, 2}
+ return fileDescriptor_beam_runner_api_d5fa30116074ddde, []int{43, 2}
}
func (m *ExecutableStagePayload_TimerId) XXX_Unmarshal(b []byte) error {
return xxx_messageInfo_ExecutableStagePayload_TimerId.Unmarshal(m, b)
@@ -6038,11 +5524,6 @@
proto.RegisterMapType((map[string]*SideInput)(nil), "org.apache.beam.model.pipeline.v1.WriteFilesPayload.SideInputsEntry")
proto.RegisterType((*Coder)(nil), "org.apache.beam.model.pipeline.v1.Coder")
proto.RegisterType((*StandardCoders)(nil), "org.apache.beam.model.pipeline.v1.StandardCoders")
- proto.RegisterType((*Schema)(nil), "org.apache.beam.model.pipeline.v1.Schema")
- proto.RegisterType((*Schema_LogicalType)(nil), "org.apache.beam.model.pipeline.v1.Schema.LogicalType")
- proto.RegisterType((*Schema_MapType)(nil), "org.apache.beam.model.pipeline.v1.Schema.MapType")
- proto.RegisterType((*Schema_FieldType)(nil), "org.apache.beam.model.pipeline.v1.Schema.FieldType")
- proto.RegisterType((*Schema_Field)(nil), "org.apache.beam.model.pipeline.v1.Schema.Field")
proto.RegisterType((*WindowingStrategy)(nil), "org.apache.beam.model.pipeline.v1.WindowingStrategy")
proto.RegisterType((*MergeStatus)(nil), "org.apache.beam.model.pipeline.v1.MergeStatus")
proto.RegisterType((*AccumulationMode)(nil), "org.apache.beam.model.pipeline.v1.AccumulationMode")
@@ -6095,7 +5576,6 @@
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.Parameter_Type_Enum", Parameter_Type_Enum_name, Parameter_Type_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.IsBounded_Enum", IsBounded_Enum_name, IsBounded_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.StandardCoders_Enum", StandardCoders_Enum_name, StandardCoders_Enum_value)
- proto.RegisterEnum("org.apache.beam.model.pipeline.v1.Schema_TypeName", Schema_TypeName_name, Schema_TypeName_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.MergeStatus_Enum", MergeStatus_Enum_name, MergeStatus_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.AccumulationMode_Enum", AccumulationMode_Enum_name, AccumulationMode_Enum_value)
proto.RegisterEnum("org.apache.beam.model.pipeline.v1.ClosingBehavior_Enum", ClosingBehavior_Enum_name, ClosingBehavior_Enum_value)
@@ -6109,358 +5589,327 @@
}
func init() {
- proto.RegisterFile("beam_runner_api.proto", fileDescriptor_beam_runner_api_198a59238d98c078)
+ proto.RegisterFile("beam_runner_api.proto", fileDescriptor_beam_runner_api_d5fa30116074ddde)
}
-var fileDescriptor_beam_runner_api_198a59238d98c078 = []byte{
- // 5582 bytes of a gzipped FileDescriptorProto
- 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5c, 0x5b, 0x6f, 0x24, 0xc7,
- 0x75, 0x9e, 0xfb, 0xe5, 0xcc, 0x70, 0xd8, 0xac, 0xbd, 0x88, 0x6a, 0xcb, 0xd2, 0xaa, 0x25, 0x4b,
- 0x2b, 0x59, 0x1a, 0xed, 0x72, 0x57, 0x7b, 0xa1, 0x6d, 0xc9, 0x43, 0x4e, 0x93, 0xec, 0xdd, 0xb9,
- 0xb9, 0x67, 0xc8, 0xdd, 0x95, 0x1d, 0xb5, 0x8b, 0xd3, 0x35, 0x64, 0x83, 0x3d, 0xdd, 0xe3, 0xee,
- 0x1e, 0xae, 0x68, 0xc4, 0x30, 0x90, 0x07, 0x23, 0x40, 0x80, 0x20, 0x79, 0xc8, 0x83, 0x91, 0x87,
- 0x00, 0x36, 0x90, 0x87, 0x04, 0xb9, 0xda, 0x08, 0x90, 0x3c, 0xda, 0xce, 0x2f, 0x70, 0x80, 0x00,
- 0xf9, 0x0d, 0x79, 0x49, 0x82, 0x3c, 0x24, 0x4f, 0x41, 0x5d, 0xba, 0xa7, 0x67, 0x48, 0xae, 0x66,
- 0xc8, 0x45, 0xde, 0xa6, 0x4f, 0xd5, 0xf9, 0x4e, 0x5d, 0x4e, 0x9d, 0x3a, 0xe7, 0x54, 0xd5, 0xc0,
- 0xb5, 0x7d, 0x82, 0x87, 0x86, 0x37, 0x76, 0x1c, 0xe2, 0x19, 0x78, 0x64, 0x55, 0x47, 0x9e, 0x1b,
- 0xb8, 0xe8, 0x4d, 0xd7, 0x3b, 0xa8, 0xe2, 0x11, 0xee, 0x1f, 0x92, 0x2a, 0xad, 0x51, 0x1d, 0xba,
- 0x26, 0xb1, 0xab, 0x23, 0x6b, 0x44, 0x6c, 0xcb, 0x21, 0xd5, 0xe3, 0xdb, 0xf2, 0x32, 0x71, 0xcc,
- 0x91, 0x6b, 0x39, 0x81, 0xcf, 0x79, 0xe4, 0x57, 0x0f, 0x5c, 0xf7, 0xc0, 0x26, 0x1f, 0xb1, 0xaf,
- 0xfd, 0xf1, 0xe0, 0x23, 0xec, 0x9c, 0x88, 0xa2, 0x1b, 0xb3, 0x45, 0x26, 0xf1, 0xfb, 0x9e, 0x35,
- 0x0a, 0x5c, 0x8f, 0xd7, 0x50, 0x7e, 0x95, 0x84, 0xa5, 0x0d, 0x82, 0x87, 0x9b, 0xae, 0xe3, 0x07,
- 0xd8, 0x09, 0x7c, 0xe5, 0x6f, 0x93, 0x50, 0x8c, 0xbe, 0xd0, 0x6d, 0xb8, 0xda, 0xd4, 0x5a, 0x46,
- 0x4f, 0x6b, 0xaa, 0xdd, 0x5e, 0xad, 0xd9, 0x31, 0x9a, 0x5a, 0xa3, 0xa1, 0x75, 0xa5, 0x84, 0xfc,
- 0xca, 0x5f, 0xfc, 0xf2, 0x7f, 0x7f, 0x95, 0x5d, 0xf9, 0xf0, 0xe1, 0xda, 0xda, 0x9d, 0x3b, 0xf7,
- 0xd7, 0x6e, 0xdd, 0xb9, 0xf7, 0xe0, 0xe3, 0xbb, 0xf7, 0xef, 0x7f, 0x8c, 0x6e, 0xc1, 0xd5, 0x66,
- 0xed, 0xe9, 0x69, 0x96, 0xa4, 0x7c, 0x9d, 0xb1, 0x48, 0xa7, 0x38, 0x3e, 0x01, 0x65, 0xbb, 0xd1,
- 0xde, 0xa8, 0x35, 0x8c, 0x27, 0x5a, 0xab, 0xde, 0x7e, 0x62, 0x9c, 0xc9, 0x9f, 0x9a, 0xe6, 0xbf,
- 0xfd, 0xf0, 0xe3, 0x5b, 0x77, 0x19, 0xbf, 0xf2, 0x0f, 0x05, 0x80, 0x4d, 0x77, 0x38, 0x72, 0x1d,
- 0x42, 0xdb, 0xfc, 0x3b, 0x00, 0x81, 0x87, 0x1d, 0x7f, 0xe0, 0x7a, 0x43, 0x7f, 0x35, 0x79, 0x23,
- 0x7d, 0xb3, 0xb4, 0xf6, 0xad, 0xea, 0x97, 0x8e, 0x6c, 0x75, 0x02, 0x51, 0xed, 0x45, 0xfc, 0xaa,
- 0x13, 0x78, 0x27, 0x7a, 0x0c, 0x10, 0xf5, 0xa1, 0x3c, 0xea, 0xbb, 0xb6, 0x4d, 0xfa, 0x81, 0xe5,
- 0x3a, 0xfe, 0x6a, 0x8a, 0x09, 0xf8, 0x74, 0x31, 0x01, 0x9d, 0x18, 0x02, 0x17, 0x31, 0x05, 0x8a,
- 0x4e, 0xe0, 0xea, 0x73, 0xcb, 0x31, 0xdd, 0xe7, 0x96, 0x73, 0x60, 0xf8, 0x81, 0x87, 0x03, 0x72,
- 0x60, 0x11, 0x7f, 0x35, 0xcd, 0x84, 0x6d, 0x2d, 0x26, 0xec, 0x49, 0x88, 0xd4, 0x8d, 0x80, 0xb8,
- 0xcc, 0x2b, 0xcf, 0x4f, 0x97, 0xa0, 0xef, 0x40, 0xae, 0xef, 0x9a, 0xc4, 0xf3, 0x57, 0x33, 0x4c,
- 0xd8, 0xc3, 0xc5, 0x84, 0x6d, 0x32, 0x5e, 0x8e, 0x2f, 0x80, 0xe8, 0x90, 0x11, 0xe7, 0xd8, 0xf2,
- 0x5c, 0x67, 0x48, 0xeb, 0xac, 0x66, 0x2f, 0x32, 0x64, 0x6a, 0x0c, 0x41, 0x0c, 0x59, 0x1c, 0x54,
- 0xb6, 0x61, 0x79, 0x66, 0xda, 0x90, 0x04, 0xe9, 0x23, 0x72, 0xb2, 0x9a, 0xbc, 0x91, 0xbc, 0x59,
- 0xd4, 0xe9, 0x4f, 0xb4, 0x09, 0xd9, 0x63, 0x6c, 0x8f, 0xc9, 0x6a, 0xea, 0x46, 0xf2, 0x66, 0x69,
- 0xed, 0xc3, 0x39, 0x9a, 0xd0, 0x89, 0x50, 0x75, 0xce, 0xbb, 0x9e, 0x7a, 0x90, 0x94, 0x5d, 0x58,
- 0x39, 0x35, 0x87, 0x67, 0xc8, 0xab, 0x4f, 0xcb, 0xab, 0xce, 0x23, 0x6f, 0x33, 0x82, 0x8d, 0x0b,
- 0xfc, 0x5d, 0x58, 0x3d, 0x6f, 0x1e, 0xcf, 0x90, 0xfb, 0x68, 0x5a, 0xee, 0xdd, 0x39, 0xe4, 0xce,
- 0xa2, 0x9f, 0xc4, 0xa5, 0xf7, 0xa1, 0x14, 0x9b, 0xd8, 0x33, 0x04, 0x7e, 0x32, 0x2d, 0xf0, 0xe6,
- 0x5c, 0x73, 0x6b, 0x12, 0x6f, 0x66, 0x4c, 0x4f, 0x4d, 0xf2, 0xcb, 0x19, 0xd3, 0x18, 0x6c, 0x4c,
- 0xa0, 0xf2, 0x6f, 0x49, 0x28, 0x74, 0x44, 0x35, 0xd4, 0x04, 0xe8, 0x47, 0xda, 0xc6, 0xe4, 0xcd,
- 0xa7, 0x1f, 0x13, 0x15, 0xd5, 0x63, 0x00, 0xe8, 0x03, 0x40, 0x9e, 0xeb, 0x06, 0x46, 0x64, 0x39,
- 0x0c, 0xcb, 0xe4, 0xc6, 0xa2, 0xa8, 0x4b, 0xb4, 0x24, 0x52, 0x2b, 0xcd, 0xa4, 0x8b, 0xae, 0x6c,
- 0x5a, 0xfe, 0xc8, 0xc6, 0x27, 0x86, 0x89, 0x03, 0xbc, 0x9a, 0x9e, 0xbb, 0x6b, 0x75, 0xce, 0x56,
- 0xc7, 0x01, 0xd6, 0x4b, 0xe6, 0xe4, 0x43, 0xf9, 0x83, 0x0c, 0xc0, 0x44, 0x77, 0xd1, 0x1b, 0x50,
- 0x1a, 0x3b, 0xd6, 0x0f, 0xc6, 0xc4, 0x70, 0xf0, 0x90, 0xac, 0x66, 0xd9, 0x78, 0x02, 0x27, 0xb5,
- 0xf0, 0x90, 0xa0, 0x4d, 0xc8, 0xf8, 0x23, 0xd2, 0x17, 0x3d, 0xff, 0x68, 0x0e, 0xd1, 0x5b, 0x63,
- 0x87, 0xa9, 0x69, 0x77, 0x44, 0xfa, 0x3a, 0x63, 0x46, 0x6f, 0xc3, 0x92, 0x3f, 0xde, 0x8f, 0x99,
- 0x5f, 0xde, 0xe1, 0x69, 0x22, 0x35, 0x31, 0x96, 0x33, 0x1a, 0x07, 0xa1, 0x3d, 0x7b, 0xb8, 0xd0,
- 0x32, 0xac, 0x6a, 0x8c, 0x57, 0x98, 0x18, 0x0e, 0x84, 0x7a, 0x90, 0x77, 0xc7, 0x01, 0xc3, 0xe4,
- 0x66, 0x6b, 0x7d, 0x31, 0xcc, 0x36, 0x67, 0xe6, 0xa0, 0x21, 0xd4, 0xa9, 0x69, 0xc9, 0x5d, 0x7a,
- 0x5a, 0xe4, 0x87, 0x50, 0x8a, 0xb5, 0xff, 0x0c, 0xf5, 0xbe, 0x1a, 0x57, 0xef, 0x62, 0x7c, 0x7d,
- 0xac, 0x43, 0x39, 0xde, 0xcc, 0x45, 0x78, 0x95, 0x3f, 0x5d, 0x86, 0x2b, 0xdd, 0x00, 0x3b, 0x26,
- 0xf6, 0xcc, 0x49, 0xb7, 0x7d, 0xe5, 0xef, 0xd2, 0x00, 0x1d, 0xcf, 0x1a, 0x5a, 0x81, 0x75, 0x4c,
- 0x7c, 0xf4, 0x21, 0xe4, 0x3a, 0x35, 0xdd, 0xa8, 0xb7, 0xa5, 0x84, 0xfc, 0xe6, 0xcf, 0xe8, 0x76,
- 0xfb, 0x95, 0xb1, 0xe7, 0xac, 0xd3, 0x4e, 0xae, 0x47, 0x13, 0xb8, 0x3e, 0xc2, 0x9e, 0xe9, 0xae,
- 0x1f, 0xdf, 0x46, 0x1f, 0x40, 0x7e, 0xab, 0x51, 0xeb, 0xf5, 0xd4, 0x96, 0x94, 0x94, 0xdf, 0x60,
- 0xf5, 0x5f, 0x9d, 0xa9, 0x3b, 0xb0, 0x71, 0x10, 0x10, 0x87, 0xd6, 0xbe, 0x07, 0xe5, 0x6d, 0xbd,
- 0xbd, 0xdb, 0x31, 0x36, 0x9e, 0x19, 0x8f, 0xd5, 0x67, 0x52, 0x4a, 0x7e, 0x9b, 0xb1, 0xbc, 0x3e,
- 0xc3, 0x72, 0xe0, 0xb9, 0xe3, 0x91, 0xb1, 0x7f, 0x62, 0x1c, 0x91, 0x13, 0x21, 0x45, 0x6b, 0x76,
- 0x76, 0x1b, 0x5d, 0x55, 0x4a, 0x9f, 0x23, 0xc5, 0x1a, 0x8e, 0xc6, 0xb6, 0x4f, 0x68, 0xed, 0xfb,
- 0x50, 0xa9, 0x75, 0xbb, 0xda, 0x76, 0x4b, 0x78, 0x13, 0x5d, 0x29, 0x23, 0xbf, 0xc5, 0x98, 0xbe,
- 0x3a, 0xc3, 0xc4, 0x77, 0x3f, 0xc3, 0x72, 0x02, 0x97, 0x33, 0x96, 0x7a, 0x6a, 0xb7, 0x67, 0x74,
- 0x7b, 0xba, 0x5a, 0x6b, 0x4a, 0x59, 0xf9, 0x1d, 0xc6, 0x75, 0xe3, 0x8c, 0x01, 0x08, 0x88, 0x1f,
- 0xf8, 0x81, 0x47, 0x89, 0xc7, 0xb7, 0xd1, 0x5d, 0x28, 0x35, 0x6b, 0x9d, 0x48, 0x5c, 0xee, 0x1c,
- 0x71, 0x43, 0x3c, 0x32, 0xb8, 0x48, 0x9f, 0x72, 0x3d, 0x80, 0xa5, 0xa6, 0xaa, 0x6f, 0xab, 0x11,
- 0x5f, 0x5e, 0xfe, 0x1a, 0xe3, 0x7b, 0x63, 0x96, 0x8f, 0x78, 0x07, 0x24, 0xc6, 0xa9, 0x04, 0x70,
- 0xb5, 0x4e, 0x46, 0x1e, 0xe9, 0xe3, 0x80, 0x98, 0xb1, 0xc9, 0x7b, 0x07, 0x32, 0xba, 0x5a, 0xab,
- 0x4b, 0x09, 0xf9, 0x35, 0x06, 0x74, 0x7d, 0x06, 0xc8, 0x23, 0xd8, 0x14, 0xed, 0xdd, 0xd4, 0xd5,
- 0x5a, 0x4f, 0x35, 0xf6, 0x34, 0xf5, 0x89, 0x94, 0x3c, 0xa7, 0xbd, 0x7d, 0x8f, 0xe0, 0x80, 0x18,
- 0xc7, 0x16, 0x79, 0x4e, 0xa5, 0xfe, 0x67, 0x52, 0x78, 0x59, 0xbe, 0x15, 0x10, 0x1f, 0x7d, 0x13,
- 0x96, 0x37, 0xdb, 0xcd, 0x0d, 0xad, 0xa5, 0x1a, 0x1d, 0x55, 0x67, 0xf3, 0x99, 0x90, 0xdf, 0x65,
- 0x40, 0x6f, 0xce, 0x02, 0xb9, 0xc3, 0x7d, 0xcb, 0x21, 0xc6, 0x88, 0x78, 0xe1, 0x94, 0x7e, 0x02,
- 0x52, 0xc8, 0xcd, 0x5d, 0xbf, 0xc6, 0x33, 0x29, 0x29, 0xdf, 0x64, 0xec, 0xca, 0x39, 0xec, 0x07,
- 0xb6, 0xbb, 0x8f, 0x6d, 0x9b, 0xf1, 0xdf, 0x82, 0xa2, 0xae, 0x76, 0x77, 0x76, 0xb7, 0xb6, 0x1a,
- 0xaa, 0x94, 0x0a, 0x55, 0xf5, 0x54, 0x7f, 0xfd, 0xc3, 0xf1, 0x60, 0x60, 0x13, 0xd1, 0xe9, 0x27,
- 0xba, 0xd6, 0x53, 0x8d, 0x2d, 0xad, 0xa1, 0x76, 0xa5, 0xf4, 0x79, 0x3a, 0xe1, 0x59, 0x01, 0x31,
- 0x06, 0x96, 0x4d, 0xd8, 0x50, 0xff, 0x36, 0x03, 0x2b, 0x9b, 0x5c, 0x7e, 0xcc, 0xc3, 0x5c, 0x87,
- 0x4a, 0xd4, 0xf7, 0xed, 0x8d, 0xc7, 0x9b, 0x7b, 0x52, 0x22, 0x54, 0x96, 0xf3, 0xba, 0x7e, 0xb0,
- 0x7f, 0xd4, 0x3f, 0xa6, 0xed, 0xd0, 0x41, 0x0e, 0x79, 0xf9, 0xf4, 0xd7, 0x36, 0x37, 0x77, 0x9b,
- 0xbb, 0x8d, 0x5a, 0xaf, 0xad, 0x53, 0x27, 0x79, 0x8d, 0xe1, 0x7c, 0x70, 0x0e, 0x0e, 0xd7, 0x05,
- 0xdc, 0xef, 0x8f, 0x87, 0x63, 0x1b, 0x07, 0xae, 0xc7, 0x54, 0xa9, 0x01, 0xaf, 0x84, 0x98, 0xea,
- 0xd3, 0x9e, 0x5e, 0xdb, 0xec, 0x19, 0xed, 0xdd, 0x5e, 0x67, 0xb7, 0x47, 0xbd, 0xe6, 0x8f, 0x18,
- 0xe0, 0x7b, 0xe7, 0x00, 0x92, 0x2f, 0x02, 0x0f, 0xf7, 0x03, 0x43, 0x58, 0xbc, 0x99, 0x16, 0x8a,
- 0x99, 0x35, 0x3a, 0xba, 0x2a, 0x48, 0x52, 0xfa, 0x4b, 0x5a, 0x28, 0x26, 0xd9, 0xa0, 0xfa, 0xc9,
- 0x49, 0x14, 0x73, 0x1f, 0x94, 0x59, 0xcc, 0x33, 0x7a, 0x9f, 0x91, 0xd7, 0x19, 0xf6, 0xdd, 0x2f,
- 0xc1, 0x3e, 0x7b, 0x14, 0xbe, 0x07, 0x6f, 0xcc, 0xca, 0x98, 0x1d, 0x8d, 0xac, 0x7c, 0x9f, 0x09,
- 0xb8, 0xfd, 0x25, 0x02, 0xce, 0x18, 0x95, 0x47, 0x70, 0x3d, 0xd2, 0x58, 0x6a, 0xc4, 0xd4, 0xba,
- 0xb1, 0x57, 0x6b, 0xec, 0xaa, 0x74, 0xbd, 0x57, 0x19, 0xe8, 0xcd, 0xf3, 0xf4, 0x96, 0x9a, 0x33,
- 0x62, 0x1a, 0xcc, 0x18, 0x33, 0xad, 0xfa, 0xc3, 0x0c, 0xbc, 0xda, 0x1d, 0xd9, 0x56, 0x10, 0xe0,
- 0x7d, 0x9b, 0x74, 0xb0, 0x57, 0x77, 0x63, 0xda, 0xd5, 0x80, 0x6b, 0x9d, 0x9a, 0xa6, 0x1b, 0x4f,
- 0xb4, 0xde, 0x8e, 0xa1, 0xab, 0xdd, 0x9e, 0xae, 0x6d, 0xf6, 0xb4, 0x76, 0x4b, 0x4a, 0xc8, 0xb7,
- 0x99, 0xa0, 0xaf, 0xcf, 0x08, 0xf2, 0xcd, 0x81, 0x31, 0xc2, 0x96, 0x67, 0x3c, 0xb7, 0x82, 0x43,
- 0xc3, 0x23, 0x7e, 0xe0, 0x59, 0x6c, 0x63, 0xa6, 0xed, 0xae, 0xc3, 0x4a, 0xb7, 0xd3, 0xd0, 0x7a,
- 0x53, 0x48, 0x49, 0xf9, 0x43, 0x86, 0xf4, 0xee, 0x19, 0x48, 0x3e, 0x6d, 0xd8, 0x2c, 0x4a, 0x0b,
- 0xae, 0x77, 0xf4, 0xf6, 0xa6, 0xda, 0xed, 0xd2, 0x71, 0x55, 0xeb, 0x86, 0xda, 0x50, 0x9b, 0x6a,
- 0x8b, 0x29, 0xd8, 0xd9, 0xfa, 0xc0, 0x1a, 0xe5, 0xb9, 0x7d, 0xe2, 0xfb, 0x74, 0x48, 0x89, 0x69,
- 0x10, 0x9b, 0x30, 0xbf, 0x8e, 0xe2, 0x6d, 0x80, 0x14, 0xe2, 0x45, 0x48, 0x69, 0xf9, 0x03, 0x86,
- 0xf4, 0xce, 0x0b, 0x90, 0xe2, 0x18, 0x4f, 0xe1, 0x2b, 0xbc, 0x67, 0xb5, 0x56, 0xdd, 0xe8, 0x6a,
- 0x9f, 0xa9, 0xf1, 0x2e, 0x52, 0x65, 0x3a, 0x7b, 0xae, 0x27, 0x7d, 0xc4, 0x8e, 0x69, 0xf8, 0xd6,
- 0x0f, 0x49, 0xbc, 0xb3, 0x0c, 0xd9, 0x85, 0x77, 0xc3, 0xd6, 0x51, 0xdc, 0x49, 0x6f, 0x99, 0xa8,
- 0x29, 0x29, 0x59, 0x79, 0x83, 0x49, 0xf9, 0xe6, 0x0b, 0x1a, 0x4d, 0x65, 0x44, 0xdd, 0x67, 0x52,
- 0x67, 0x04, 0x2a, 0xbf, 0x97, 0x84, 0xeb, 0xe1, 0xee, 0xdc, 0xb5, 0x4c, 0xc2, 0x3c, 0x84, 0xde,
- 0xc9, 0x88, 0xf8, 0xca, 0x21, 0x64, 0x54, 0x67, 0x3c, 0x44, 0x1f, 0x41, 0x41, 0xeb, 0xa9, 0x7a,
- 0x6d, 0xa3, 0xa1, 0x4e, 0xf6, 0x66, 0x26, 0xd4, 0xb7, 0x4c, 0x62, 0x30, 0x37, 0x68, 0xdd, 0x0a,
- 0x88, 0x47, 0x55, 0x8a, 0x76, 0xe2, 0x23, 0x28, 0x34, 0x77, 0x1b, 0x3d, 0xad, 0x59, 0xeb, 0x48,
- 0xc9, 0xf3, 0x18, 0x86, 0x63, 0x3b, 0xb0, 0x86, 0x78, 0x44, 0x1b, 0xf1, 0xb3, 0x14, 0x94, 0x62,
- 0xc1, 0xc7, 0xac, 0xc7, 0x98, 0x3c, 0xe5, 0x31, 0xbe, 0x0a, 0x05, 0x16, 0xe0, 0x19, 0x96, 0x29,
- 0x1c, 0x8e, 0x3c, 0xfb, 0xd6, 0x4c, 0xd4, 0x01, 0xb0, 0x7c, 0x63, 0xdf, 0x1d, 0x3b, 0x26, 0x31,
- 0x99, 0x37, 0x5b, 0x59, 0xbb, 0x3d, 0x87, 0xdb, 0xa4, 0xf9, 0x1b, 0x9c, 0xa7, 0x4a, 0x3b, 0xad,
- 0x17, 0xad, 0xf0, 0x1b, 0xad, 0xc1, 0xb5, 0x53, 0x11, 0xf1, 0x09, 0x95, 0x9c, 0x61, 0x92, 0x4f,
- 0x85, 0xb2, 0x27, 0x9a, 0x79, 0xca, 0x7d, 0xcb, 0x5e, 0xde, 0xab, 0xfe, 0x69, 0x1e, 0xca, 0x6c,
- 0xc1, 0x76, 0xf0, 0x89, 0xed, 0x62, 0x13, 0x6d, 0x43, 0xd6, 0x74, 0x8d, 0x81, 0x23, 0xfc, 0xe6,
- 0xb5, 0x39, 0xc0, 0xbb, 0xe6, 0xd1, 0xb4, 0xeb, 0x6c, 0xba, 0x5b, 0x0e, 0x6a, 0x00, 0x8c, 0xb0,
- 0x87, 0x87, 0x24, 0xa0, 0xb1, 0x37, 0xcf, 0x2a, 0x7c, 0x30, 0x8f, 0x13, 0x1b, 0x32, 0xe9, 0x31,
- 0x7e, 0xf4, 0x7d, 0x28, 0x4d, 0xa6, 0x39, 0xf4, 0xb3, 0x3f, 0x9d, 0x0f, 0x2e, 0xea, 0x5c, 0x35,
- 0xd2, 0xc5, 0x30, 0x0f, 0xe2, 0x47, 0x04, 0x26, 0x21, 0xa0, 0x0e, 0x02, 0x75, 0xfc, 0x43, 0xaf,
- 0x7b, 0x71, 0x09, 0x14, 0x82, 0x8e, 0x42, 0x24, 0x21, 0x22, 0x50, 0x09, 0x81, 0x35, 0x24, 0x9e,
- 0x90, 0x90, 0xbd, 0x98, 0x84, 0x1e, 0x85, 0x88, 0x4b, 0x08, 0x22, 0x02, 0x7a, 0x1d, 0xc0, 0x8f,
- 0xec, 0x30, 0xf3, 0xee, 0x0b, 0x7a, 0x8c, 0x82, 0x6e, 0xc1, 0xd5, 0xd8, 0x52, 0x35, 0x22, 0x6d,
- 0xcf, 0x33, 0x9d, 0x43, 0xb1, 0xb2, 0x4d, 0xa1, 0xf8, 0x77, 0xe0, 0x9a, 0x47, 0x7e, 0x30, 0xa6,
- 0xfe, 0xa1, 0x31, 0xb0, 0x1c, 0x6c, 0x5b, 0x3f, 0xc4, 0xb4, 0x7c, 0xb5, 0xc0, 0xc0, 0xaf, 0x86,
- 0x85, 0x5b, 0xb1, 0x32, 0xf9, 0x08, 0x96, 0x67, 0x46, 0xfa, 0x0c, 0xdf, 0x7e, 0x63, 0x3a, 0xec,
- 0x9d, 0x47, 0x35, 0x22, 0xd0, 0x78, 0x14, 0x41, 0x85, 0x4d, 0x0f, 0xfa, 0x4b, 0x12, 0x16, 0x82,
- 0xce, 0x08, 0x9b, 0x19, 0xff, 0x97, 0x23, 0x2c, 0x02, 0x8d, 0xc7, 0x38, 0xbf, 0x48, 0x42, 0x31,
- 0x5a, 0x0d, 0xe8, 0x11, 0x64, 0x82, 0x93, 0x11, 0xb7, 0x5b, 0x95, 0xb5, 0x7b, 0x8b, 0xac, 0xa4,
- 0x2a, 0x35, 0xbd, 0xdc, 0x02, 0x31, 0x0c, 0xf9, 0x33, 0xc8, 0x50, 0x92, 0xa2, 0x0b, 0x63, 0xbc,
- 0x0c, 0xa5, 0xdd, 0x56, 0xb7, 0xa3, 0x6e, 0x6a, 0x5b, 0x9a, 0x5a, 0x97, 0x12, 0x08, 0x20, 0xc7,
- 0xdd, 0x78, 0x29, 0x89, 0xae, 0x82, 0xd4, 0xd1, 0x3a, 0x6a, 0x83, 0xba, 0x0a, 0xed, 0x0e, 0xdf,
- 0x26, 0x52, 0xe8, 0x15, 0xb8, 0x12, 0xdb, 0x38, 0x0c, 0xea, 0x97, 0x3c, 0x56, 0x75, 0x29, 0x4d,
- 0x23, 0xb0, 0x62, 0x34, 0x76, 0x48, 0x07, 0x60, 0x1d, 0x32, 0x62, 0xb1, 0xf8, 0x3c, 0x86, 0x73,
- 0x8f, 0x32, 0x45, 0x30, 0x3b, 0x09, 0xbd, 0xc8, 0x60, 0x18, 0x66, 0x03, 0x0a, 0xfb, 0xf8, 0x80,
- 0x23, 0xa6, 0xe6, 0x8e, 0xee, 0x37, 0xf0, 0x41, 0x1c, 0x2f, 0xbf, 0x8f, 0x0f, 0x18, 0xda, 0xe7,
- 0x50, 0xe1, 0x9e, 0x0d, 0x33, 0xc4, 0x14, 0x93, 0x27, 0x2b, 0x3e, 0x9e, 0x2f, 0x57, 0xc2, 0x19,
- 0xe3, 0xc8, 0x4b, 0x11, 0x5c, 0xd8, 0x5a, 0x1a, 0x29, 0x31, 0xe4, 0xcc, 0xdc, 0xad, 0x6d, 0xe2,
- 0xd1, 0x54, 0x6b, 0x87, 0x78, 0x14, 0xa2, 0xf9, 0x24, 0xe0, 0x68, 0xd9, 0xb9, 0xd1, 0xba, 0x24,
- 0x98, 0x42, 0xf3, 0x49, 0x40, 0x7f, 0x6e, 0xe4, 0x78, 0x8e, 0x44, 0xf9, 0x3a, 0x54, 0xa6, 0x07,
- 0x7c, 0x6a, 0x2f, 0x4c, 0x4e, 0xed, 0x85, 0xca, 0x03, 0x28, 0xc7, 0xc7, 0x12, 0xdd, 0x04, 0x29,
- 0xf4, 0x05, 0x66, 0x58, 0x2a, 0x82, 0x2e, 0x8c, 0x89, 0xf2, 0xd3, 0x24, 0xa0, 0xd3, 0x43, 0x46,
- 0xad, 0x52, 0xcc, 0xf7, 0x9d, 0x05, 0x41, 0xb1, 0xb2, 0xd0, 0x2a, 0x7d, 0x87, 0xe5, 0xb6, 0x98,
- 0x37, 0x3a, 0x70, 0x84, 0x0e, 0x5c, 0x64, 0xa7, 0x2a, 0x0a, 0x94, 0x2d, 0x47, 0xd9, 0x83, 0x72,
- 0x7c, 0xcc, 0xd1, 0x0d, 0x28, 0x53, 0xcf, 0x79, 0xa6, 0x31, 0x70, 0x44, 0x4e, 0xc2, 0x46, 0xbc,
- 0x0d, 0x15, 0xae, 0xda, 0x33, 0x4e, 0x43, 0x99, 0x51, 0x37, 0x27, 0xa3, 0x15, 0x1f, 0xfd, 0x05,
- 0x46, 0xeb, 0x27, 0x49, 0x28, 0x46, 0x76, 0x01, 0x75, 0xf9, 0xe6, 0x61, 0x98, 0xee, 0x10, 0x5b,
- 0x8e, 0xb0, 0x02, 0x6b, 0x73, 0x9a, 0x96, 0x3a, 0x63, 0xe2, 0x16, 0x80, 0xed, 0x17, 0x9c, 0x40,
- 0xbb, 0xc0, 0x77, 0xa4, 0xd9, 0x2e, 0x30, 0x6a, 0xd8, 0x90, 0x6f, 0x43, 0x31, 0xf2, 0x63, 0x94,
- 0x3b, 0xe7, 0x99, 0x8c, 0x25, 0x28, 0xee, 0xb6, 0x36, 0xda, 0xbb, 0xad, 0xba, 0x5a, 0x97, 0x92,
- 0xa8, 0x04, 0xf9, 0xf0, 0x23, 0xa5, 0xfc, 0x65, 0x12, 0x4a, 0x3a, 0xc1, 0x66, 0xe8, 0x64, 0x3c,
- 0x82, 0x9c, 0xef, 0x8e, 0xbd, 0x3e, 0xb9, 0x84, 0x97, 0x21, 0x10, 0x66, 0x5c, 0xb3, 0xd4, 0xe5,
- 0x5d, 0x33, 0xc5, 0x84, 0x15, 0x9e, 0x3c, 0xd6, 0x9c, 0x20, 0xf2, 0x8b, 0xda, 0x50, 0x14, 0xf9,
- 0x95, 0x4b, 0xf9, 0x46, 0x05, 0x0e, 0xb2, 0xe5, 0x28, 0x7f, 0x92, 0x84, 0x8a, 0x08, 0xc5, 0x43,
- 0x19, 0xd3, 0x6a, 0x9d, 0x7c, 0x09, 0x6a, 0x7d, 0xee, 0xda, 0x4a, 0x9d, 0xb7, 0xb6, 0x94, 0xdf,
- 0xe6, 0x60, 0xa5, 0x47, 0xfc, 0xa0, 0xcb, 0xf2, 0x41, 0x61, 0xd3, 0xce, 0xb7, 0x07, 0x48, 0x87,
- 0x1c, 0x39, 0x66, 0x49, 0xe6, 0xd4, 0xdc, 0x99, 0xca, 0x53, 0x02, 0xaa, 0x2a, 0x85, 0xd0, 0x05,
- 0x92, 0xfc, 0x1f, 0x19, 0xc8, 0x32, 0x0a, 0x3a, 0x86, 0xe5, 0xe7, 0x38, 0x20, 0xde, 0x10, 0x7b,
- 0x47, 0x06, 0x2b, 0x15, 0x03, 0xf3, 0xf8, 0xe2, 0x62, 0xaa, 0x35, 0xf3, 0x18, 0x3b, 0x7d, 0xf2,
- 0x24, 0x04, 0xde, 0x49, 0xe8, 0x95, 0x48, 0x0a, 0x97, 0xfb, 0x93, 0x24, 0x5c, 0x13, 0x01, 0x0f,
- 0xdd, 0x18, 0xd8, 0xda, 0xe3, 0xe2, 0xb9, 0xb9, 0xe9, 0x5c, 0x5e, 0x7c, 0x27, 0x82, 0xa7, 0x6b,
- 0x74, 0x27, 0xa1, 0x5f, 0x19, 0x4d, 0x51, 0x78, 0x43, 0x86, 0xb0, 0x14, 0x1a, 0x0c, 0x2e, 0x9f,
- 0x6f, 0x4f, 0x5b, 0x97, 0x92, 0x6f, 0xaa, 0x22, 0xf0, 0xdc, 0x49, 0xe8, 0x65, 0x01, 0xcf, 0xca,
- 0xe4, 0xfb, 0x20, 0xcd, 0x8e, 0x0e, 0x7a, 0x0b, 0x96, 0x1c, 0xf2, 0xdc, 0x88, 0x46, 0x88, 0xcd,
- 0x40, 0x5a, 0x2f, 0x3b, 0xe4, 0x79, 0x54, 0x49, 0xde, 0x80, 0x6b, 0x67, 0xf6, 0x0b, 0xbd, 0x07,
- 0x12, 0xe6, 0x05, 0x86, 0x39, 0xf6, 0xb8, 0xf7, 0xc8, 0x01, 0x96, 0x05, 0xbd, 0x2e, 0xc8, 0xb2,
- 0x07, 0xa5, 0x58, 0xdb, 0x50, 0x1f, 0x0a, 0x61, 0x80, 0x2c, 0xce, 0x3d, 0xb7, 0x2f, 0xd4, 0x6b,
- 0xda, 0x0c, 0x3f, 0xc0, 0xc3, 0x11, 0x09, 0xb1, 0xf5, 0x08, 0x78, 0x23, 0x0f, 0x59, 0x36, 0xae,
- 0xf2, 0x77, 0x01, 0x9d, 0xae, 0x88, 0xde, 0x85, 0x65, 0xe2, 0x50, 0x55, 0x8f, 0x22, 0x5e, 0xd6,
- 0xf8, 0xb2, 0x5e, 0x11, 0xe4, 0xb0, 0xe2, 0x6b, 0x50, 0x0c, 0x42, 0x76, 0xa6, 0x23, 0x69, 0x7d,
- 0x42, 0x50, 0xfe, 0x2b, 0x0d, 0x2b, 0x4f, 0x3c, 0x2b, 0x20, 0x5b, 0x96, 0x4d, 0xfc, 0x70, 0x55,
- 0x6d, 0x41, 0xc6, 0xb7, 0x9c, 0xa3, 0xcb, 0xc4, 0x5a, 0x94, 0x1f, 0x7d, 0x17, 0x96, 0x69, 0x94,
- 0x8e, 0x03, 0x63, 0x20, 0x0a, 0x2f, 0xb1, 0x29, 0x56, 0x38, 0x54, 0x48, 0xa3, 0x23, 0xc0, 0x8d,
- 0x16, 0x31, 0x0d, 0x96, 0x4e, 0xf4, 0x99, 0x0a, 0x16, 0xf4, 0x4a, 0x48, 0x66, 0x1d, 0xf3, 0xd1,
- 0x37, 0x41, 0x16, 0x37, 0x00, 0x4c, 0xea, 0x75, 0x0e, 0x2d, 0x87, 0x98, 0x86, 0x7f, 0x88, 0x3d,
- 0xd3, 0x72, 0x0e, 0x98, 0xef, 0x53, 0xd0, 0x57, 0x79, 0x8d, 0x7a, 0x54, 0xa1, 0x2b, 0xca, 0x11,
- 0x99, 0x8e, 0xf0, 0x78, 0x74, 0x54, 0x9f, 0xe7, 0xa0, 0x6f, 0x76, 0x58, 0x5f, 0x14, 0xe6, 0xfd,
- 0xbf, 0xc6, 0x26, 0xca, 0x8f, 0x21, 0xcb, 0xcc, 0x2a, 0x9b, 0xe8, 0x89, 0x03, 0x7c, 0xb1, 0x89,
- 0xa6, 0x5e, 0x40, 0x15, 0xae, 0x44, 0x67, 0x72, 0x91, 0x31, 0x0f, 0x4f, 0xa5, 0x56, 0xa2, 0x22,
- 0x61, 0xcb, 0x7d, 0xe5, 0x5f, 0x33, 0x50, 0x09, 0x13, 0x31, 0xfc, 0xc0, 0x53, 0xf9, 0x4d, 0x46,
- 0xec, 0xe0, 0x6f, 0x43, 0x76, 0xe3, 0x59, 0x4f, 0xed, 0x4a, 0x09, 0xf9, 0x55, 0x96, 0x4d, 0xb9,
- 0xc2, 0xb2, 0x29, 0x0c, 0x75, 0x7d, 0xff, 0x24, 0x60, 0xb9, 0x3d, 0x74, 0x0b, 0x4a, 0xd4, 0xcb,
- 0x6f, 0x6d, 0x1b, 0xbb, 0xbd, 0xad, 0x07, 0x12, 0x4c, 0x1d, 0x58, 0xf0, 0xba, 0x34, 0x68, 0x74,
- 0x0e, 0x8c, 0x71, 0x30, 0x78, 0x40, 0x39, 0x5e, 0x87, 0xd4, 0xe3, 0x3d, 0x29, 0x29, 0x5f, 0x67,
- 0x15, 0xa5, 0x58, 0xc5, 0x23, 0x96, 0x31, 0x7e, 0x07, 0x72, 0x7b, 0x35, 0x5d, 0x6b, 0xf5, 0xa4,
- 0x94, 0x2c, 0xb3, 0x3a, 0x57, 0x63, 0x75, 0x8e, 0xb1, 0x67, 0x39, 0x81, 0xa8, 0x57, 0x6f, 0xef,
- 0x6e, 0x34, 0x54, 0xa9, 0x74, 0x46, 0x3d, 0xd3, 0x1d, 0x8b, 0xc4, 0xd0, 0xfb, 0xb1, 0x4c, 0x52,
- 0x7a, 0xea, 0xa8, 0x80, 0xd7, 0x8c, 0x27, 0x91, 0xde, 0x86, 0x6c, 0x4f, 0x6b, 0xaa, 0xba, 0x94,
- 0x39, 0xa3, 0xcf, 0xcc, 0xe9, 0xe1, 0x47, 0x19, 0xcb, 0x5a, 0xab, 0xa7, 0xea, 0x7b, 0xd1, 0x15,
- 0x0e, 0x29, 0x3b, 0x95, 0x5f, 0x17, 0xc0, 0x4e, 0x40, 0xbc, 0x63, 0x6c, 0x8b, 0xb3, 0x0c, 0x9e,
- 0x95, 0x5f, 0x6a, 0xa8, 0xad, 0xed, 0xde, 0x8e, 0xd1, 0xd1, 0xd5, 0x2d, 0xed, 0xa9, 0x94, 0x9b,
- 0xca, 0x54, 0x71, 0x3e, 0x9b, 0x38, 0x07, 0xc1, 0xa1, 0x31, 0xf2, 0xc8, 0xc0, 0xfa, 0x42, 0x70,
- 0x4d, 0x5d, 0x18, 0x91, 0xf2, 0x67, 0x70, 0xf1, 0xe3, 0x82, 0x98, 0xac, 0x7b, 0x50, 0xe1, 0xd5,
- 0xc3, 0xd4, 0xad, 0x54, 0x90, 0x15, 0xc6, 0xf6, 0x5a, 0x8c, 0x2d, 0x5a, 0xba, 0x5c, 0x2b, 0x59,
- 0x06, 0xf5, 0x5a, 0xb7, 0x57, 0xeb, 0xa9, 0xc6, 0x06, 0x0d, 0xd9, 0xea, 0x46, 0x34, 0x78, 0x45,
- 0xf9, 0x3d, 0xc6, 0xfe, 0xd6, 0xd4, 0xdc, 0xe2, 0x80, 0x18, 0xfb, 0xb8, 0x7f, 0x44, 0x4c, 0x23,
- 0x36, 0x92, 0xca, 0x2f, 0x01, 0x72, 0xdd, 0xfe, 0x21, 0x19, 0x62, 0xb4, 0x0d, 0xb9, 0x81, 0x45,
- 0x6c, 0x33, 0x34, 0xd2, 0x73, 0x45, 0x24, 0x8c, 0xb5, 0xba, 0x45, 0xf9, 0x74, 0xc1, 0x8e, 0x2a,
- 0x90, 0x8a, 0x5c, 0x93, 0x94, 0x65, 0xca, 0x7f, 0x9d, 0x84, 0x52, 0xc3, 0x3d, 0xb0, 0xfa, 0xd8,
- 0xa6, 0xe1, 0xaa, 0x28, 0x4f, 0x86, 0xe5, 0x08, 0x41, 0x06, 0x7b, 0x07, 0xbe, 0xe0, 0x60, 0xbf,
- 0x51, 0x07, 0x8a, 0xfb, 0xd8, 0x27, 0x06, 0x8b, 0x95, 0xf9, 0x56, 0x79, 0x67, 0xc1, 0xf6, 0x50,
- 0x59, 0x7a, 0x81, 0xa2, 0x30, 0xa9, 0xef, 0x81, 0xe4, 0x13, 0xcf, 0xc2, 0x36, 0x4b, 0x7b, 0xf6,
- 0x6d, 0xec, 0xfb, 0xcc, 0x98, 0x95, 0xf5, 0xe5, 0x09, 0x7d, 0x93, 0x92, 0xe5, 0xbf, 0x4a, 0x42,
- 0xbe, 0x89, 0x47, 0x8c, 0xad, 0x05, 0x05, 0x1a, 0x40, 0x44, 0x31, 0xfb, 0x05, 0xdb, 0x91, 0x3f,
- 0x22, 0x27, 0x0c, 0x2f, 0x8a, 0xa4, 0x19, 0x62, 0xea, 0xe2, 0x88, 0x3c, 0x92, 0xa6, 0x3f, 0xe5,
- 0x7f, 0x4f, 0x43, 0x31, 0x2a, 0xa0, 0x2e, 0x2e, 0xc5, 0x9e, 0xa4, 0x47, 0xe7, 0x0b, 0x30, 0x84,
- 0x00, 0x0a, 0xd1, 0xc2, 0x43, 0xa2, 0x17, 0x02, 0xf1, 0x0b, 0xc9, 0x50, 0x70, 0xc6, 0xb6, 0xcd,
- 0x92, 0x51, 0x29, 0x66, 0xfe, 0xa3, 0x6f, 0x34, 0x84, 0x57, 0x26, 0xf7, 0x4d, 0xa2, 0x64, 0xf2,
- 0x25, 0x67, 0x6d, 0x27, 0xa1, 0x5f, 0x9b, 0xa0, 0x8a, 0x9d, 0x39, 0x9c, 0x0d, 0x1a, 0x85, 0x33,
- 0xfc, 0xcc, 0xdc, 0x59, 0x08, 0x81, 0x2f, 0xa6, 0x54, 0xc4, 0xe1, 0x0c, 0xef, 0x11, 0x80, 0xe7,
- 0x3e, 0x37, 0x7c, 0x56, 0x41, 0x44, 0xe2, 0xef, 0xcd, 0x8d, 0xb8, 0x93, 0xd0, 0x8b, 0x9e, 0xfb,
- 0x5c, 0xac, 0x9f, 0xcf, 0xa0, 0x6c, 0x73, 0x2d, 0xe7, 0xed, 0xcb, 0xcd, 0x9d, 0x7f, 0x10, 0xed,
- 0x8b, 0xad, 0x91, 0x9d, 0x84, 0x5e, 0xb2, 0x27, 0x9f, 0x1b, 0x25, 0x31, 0xa7, 0x96, 0x33, 0x70,
- 0xe5, 0x5f, 0x27, 0x21, 0xcb, 0xc6, 0x8a, 0xae, 0x9c, 0x58, 0x12, 0x9c, 0xfd, 0x46, 0x37, 0xa0,
- 0x14, 0xde, 0xa7, 0x0b, 0x1d, 0x88, 0xa2, 0x1e, 0x27, 0xa1, 0x6d, 0x91, 0x82, 0xba, 0xc4, 0xb2,
- 0x62, 0x00, 0x62, 0x21, 0xd3, 0x79, 0xc8, 0xb2, 0x85, 0xfc, 0x75, 0x58, 0x61, 0xde, 0x14, 0xdd,
- 0x46, 0xd8, 0x81, 0x2c, 0x6d, 0x40, 0x96, 0x15, 0x4b, 0x61, 0x41, 0x47, 0xd0, 0x95, 0x7f, 0x4a,
- 0x42, 0x21, 0x54, 0x36, 0x54, 0x80, 0x0c, 0xdd, 0xc4, 0xa4, 0x04, 0x2a, 0x42, 0x56, 0x6b, 0xf5,
- 0x6e, 0xdf, 0x93, 0x92, 0xe2, 0xe7, 0x9d, 0x35, 0x29, 0x25, 0x7e, 0xde, 0xbb, 0x2b, 0xa5, 0x69,
- 0x44, 0x5a, 0x57, 0x37, 0xb5, 0x66, 0xad, 0x21, 0x65, 0x28, 0x7d, 0xab, 0xd1, 0xae, 0xf5, 0xa4,
- 0x2c, 0x82, 0x68, 0x9f, 0xc9, 0xd1, 0xdf, 0x7c, 0xb7, 0x93, 0xf2, 0xa8, 0x0c, 0x85, 0x7a, 0xad,
- 0xa7, 0xd2, 0xfd, 0x42, 0x2a, 0xf0, 0x78, 0xb6, 0xdd, 0x50, 0x6b, 0x2d, 0xa9, 0x48, 0xb9, 0xf9,
- 0xd6, 0x09, 0xf4, 0x67, 0x4d, 0xd7, 0x6b, 0xcf, 0xa4, 0x12, 0xca, 0x43, 0xba, 0x59, 0xeb, 0x48,
- 0x4b, 0xf4, 0x87, 0xde, 0x7e, 0x22, 0x55, 0x90, 0x04, 0xe5, 0x46, 0x7b, 0x5b, 0xdb, 0xac, 0x35,
- 0x8c, 0xde, 0xb3, 0x8e, 0x2a, 0x2d, 0x2b, 0xbf, 0x9f, 0x0b, 0x83, 0xcb, 0x58, 0x6a, 0xff, 0xa5,
- 0x07, 0x97, 0x68, 0x0f, 0xca, 0xfc, 0x50, 0x91, 0xda, 0xef, 0xb1, 0x2f, 0xc2, 0xe2, 0x79, 0x66,
- 0xac, 0x49, 0xd9, 0xba, 0x8c, 0x8b, 0x07, 0xc6, 0xa5, 0xe1, 0x84, 0x82, 0xde, 0x09, 0x7d, 0xc1,
- 0x49, 0x24, 0x99, 0x66, 0x7a, 0xb2, 0xc4, 0xc9, 0x61, 0x6e, 0xa4, 0x0e, 0xf9, 0xc0, 0xb3, 0x0e,
- 0x0e, 0x88, 0x27, 0x56, 0xdb, 0xfb, 0xf3, 0x38, 0xee, 0x9c, 0x43, 0x0f, 0x59, 0x11, 0x81, 0x95,
- 0x28, 0x40, 0xa5, 0x56, 0x82, 0xb2, 0x30, 0xb5, 0xa8, 0xac, 0x3d, 0x98, 0x03, 0xaf, 0x16, 0xe3,
- 0x6d, 0xba, 0xa6, 0xc8, 0x80, 0x4a, 0x78, 0x86, 0x8c, 0xba, 0x50, 0xe2, 0x07, 0xa3, 0x2c, 0xca,
- 0x63, 0xcb, 0x6f, 0x3e, 0xcb, 0xc7, 0x6f, 0xaf, 0xd0, 0xa0, 0x41, 0xa4, 0x56, 0xdc, 0x88, 0x80,
- 0xf6, 0x41, 0xea, 0xdb, 0x2e, 0x8b, 0x1d, 0xf7, 0xc9, 0x21, 0x3e, 0xb6, 0x5c, 0x8f, 0xa5, 0xd9,
- 0x2b, 0x6b, 0xf7, 0xe7, 0x49, 0x2c, 0x72, 0xd6, 0x0d, 0xc1, 0xc9, 0xe1, 0x97, 0xfb, 0xd3, 0x54,
- 0x16, 0x59, 0xd9, 0x36, 0xdb, 0xdd, 0x6d, 0x1c, 0x10, 0x87, 0xf8, 0x3e, 0xcb, 0xcb, 0xd3, 0xc8,
- 0x8a, 0xd3, 0x1b, 0x82, 0x8c, 0x3e, 0x87, 0x4a, 0xdb, 0xa1, 0x0d, 0x0b, 0x99, 0x57, 0x8b, 0x73,
- 0xe7, 0x91, 0xa7, 0x19, 0x79, 0x5b, 0x66, 0xd0, 0xd0, 0x6d, 0xb8, 0x86, 0x7d, 0xdf, 0x3a, 0x70,
- 0x7c, 0x23, 0x70, 0x0d, 0xd7, 0x09, 0x2f, 0x78, 0xac, 0x02, 0xb3, 0xfb, 0x48, 0x14, 0xf6, 0xdc,
- 0xb6, 0x43, 0xb8, 0xfe, 0x2b, 0xdf, 0x83, 0x52, 0x4c, 0xd9, 0x94, 0xe6, 0x79, 0x89, 0xa5, 0x65,
- 0x28, 0xb5, 0xda, 0x2d, 0x76, 0xbe, 0x4e, 0x17, 0x66, 0x92, 0x11, 0x54, 0xb5, 0xde, 0xe5, 0x47,
- 0xee, 0x52, 0x0a, 0x21, 0xa8, 0xd4, 0x1a, 0xba, 0x5a, 0xab, 0x8b, 0x53, 0xf8, 0xba, 0x94, 0x56,
- 0x9a, 0x20, 0xcd, 0xce, 0xbf, 0xf2, 0xf0, 0x3c, 0x11, 0x15, 0x80, 0xba, 0xd6, 0xdd, 0xac, 0xe9,
- 0x75, 0x2e, 0x41, 0x82, 0x72, 0x74, 0x90, 0x4f, 0x29, 0x29, 0xe5, 0x3b, 0xb0, 0x3c, 0x33, 0x27,
- 0xca, 0x27, 0x2f, 0x68, 0xb0, 0xda, 0xd4, 0x7a, 0x46, 0xad, 0xf1, 0xa4, 0xf6, 0xac, 0xcb, 0x33,
- 0xe8, 0x8c, 0xa0, 0x6d, 0x19, 0xad, 0x76, 0x4b, 0x6d, 0x76, 0x7a, 0xcf, 0xa4, 0x94, 0xd2, 0x99,
- 0x9d, 0x92, 0x17, 0x22, 0x6e, 0x69, 0xba, 0x3a, 0x85, 0xc8, 0x08, 0xd3, 0x88, 0xfb, 0x00, 0x13,
- 0x95, 0x54, 0x7a, 0xe7, 0xa1, 0xad, 0xc0, 0x92, 0xda, 0xaa, 0x1b, 0xed, 0x2d, 0x23, 0xca, 0xf1,
- 0x23, 0xa8, 0x34, 0x6a, 0xec, 0xb6, 0x90, 0xd6, 0x32, 0x3a, 0xb5, 0x16, 0x1d, 0x55, 0xda, 0xea,
- 0x9a, 0xde, 0xd0, 0xe2, 0xd4, 0xb4, 0x62, 0x03, 0x4c, 0x32, 0x8a, 0xca, 0xe7, 0x2f, 0x18, 0x51,
- 0x75, 0x4f, 0x6d, 0xf5, 0xd8, 0xbd, 0x67, 0x29, 0x89, 0xae, 0xc0, 0xb2, 0x38, 0x82, 0xa6, 0xa1,
- 0x04, 0x23, 0xa6, 0xd0, 0x0d, 0x78, 0xad, 0xfb, 0xac, 0xb5, 0xb9, 0xa3, 0xb7, 0x5b, 0xec, 0x58,
- 0x7a, 0xb6, 0x46, 0x5a, 0xf9, 0xb9, 0x04, 0x79, 0x61, 0x16, 0x90, 0x0e, 0x45, 0x3c, 0x08, 0x88,
- 0x67, 0x60, 0xdb, 0x5e, 0xc0, 0xa3, 0x12, 0xec, 0xd5, 0x1a, 0xe5, 0xad, 0xd9, 0xf6, 0x4e, 0x42,
- 0x2f, 0x60, 0xf1, 0x3b, 0x86, 0xe9, 0x9c, 0x2c, 0xe0, 0x53, 0x4d, 0x63, 0x3a, 0x27, 0x13, 0x4c,
- 0xe7, 0x04, 0xed, 0x02, 0x70, 0x4c, 0x82, 0xfb, 0x87, 0x62, 0xaf, 0xbc, 0xbb, 0x28, 0xa8, 0x8a,
- 0xfb, 0x87, 0xd4, 0x4b, 0xc0, 0xe1, 0x07, 0xb2, 0xe1, 0x8a, 0x80, 0x75, 0x4c, 0xc3, 0x1d, 0x84,
- 0xeb, 0x8b, 0x9b, 0xd7, 0x6f, 0x2c, 0x8c, 0xef, 0x98, 0xed, 0x01, 0x5f, 0x88, 0x3b, 0x09, 0x5d,
- 0xc2, 0x33, 0x34, 0x14, 0xc0, 0x35, 0x2e, 0x6d, 0x26, 0x07, 0x26, 0x5c, 0x9d, 0x4f, 0x16, 0x95,
- 0x77, 0x3a, 0xd7, 0x85, 0x4f, 0x93, 0xd1, 0x4f, 0x93, 0xa0, 0x70, 0xb1, 0xfe, 0x89, 0xd3, 0x3f,
- 0xf4, 0x5c, 0x87, 0xf9, 0xdc, 0xb3, 0x6d, 0xe0, 0x0e, 0xd2, 0xa3, 0x45, 0xdb, 0xd0, 0x8d, 0x61,
- 0x9e, 0x6a, 0xcf, 0x1b, 0xf8, 0xc5, 0x55, 0xd0, 0x63, 0xc8, 0x61, 0xfb, 0x39, 0x3e, 0xf1, 0x57,
- 0xcb, 0x73, 0xfb, 0x8f, 0x91, 0x78, 0xc6, 0xb8, 0x93, 0xd0, 0x05, 0x04, 0x6a, 0x41, 0xde, 0x24,
- 0x03, 0x3c, 0xb6, 0x03, 0xb6, 0x29, 0xcc, 0xb7, 0xdd, 0x87, 0x68, 0x75, 0xce, 0x49, 0xdd, 0x51,
- 0x01, 0x82, 0x3e, 0x9f, 0x24, 0x09, 0xfb, 0xee, 0xd8, 0x09, 0xd8, 0x36, 0x50, 0x9a, 0x6b, 0xab,
- 0x09, 0x51, 0xd5, 0xf0, 0xf4, 0x61, 0xec, 0x04, 0xb1, 0xac, 0x20, 0xfb, 0x46, 0x3b, 0x90, 0x75,
- 0xc8, 0x31, 0xe1, 0xbb, 0x46, 0x69, 0xed, 0xd6, 0x02, 0xb8, 0x2d, 0xca, 0xb7, 0x93, 0xd0, 0x39,
- 0x00, 0x5d, 0x1d, 0xae, 0xc7, 0x8f, 0x92, 0xed, 0x13, 0xb6, 0x3b, 0x2c, 0xb6, 0x3a, 0xda, 0xde,
- 0x16, 0xe7, 0xa5, 0xab, 0xc3, 0x0d, 0x3f, 0xe8, 0xec, 0x78, 0x64, 0x44, 0x70, 0xb0, 0x5a, 0x5a,
- 0x78, 0x76, 0x74, 0xc6, 0x48, 0x67, 0x87, 0x43, 0xc8, 0x4f, 0xa1, 0x10, 0x5a, 0x0b, 0xd4, 0x80,
- 0x12, 0xbb, 0xec, 0xcb, 0xaa, 0x86, 0x11, 0xee, 0x22, 0xde, 0x4c, 0x9c, 0x7d, 0x82, 0xec, 0x9c,
- 0xbc, 0x64, 0xe4, 0x67, 0x50, 0x8c, 0x0c, 0xc7, 0x4b, 0x86, 0xfe, 0xfb, 0x24, 0x48, 0xb3, 0x46,
- 0x03, 0xb5, 0x61, 0x89, 0x60, 0xcf, 0x3e, 0x31, 0x06, 0x96, 0x67, 0x39, 0x07, 0xe1, 0x0d, 0xf3,
- 0x45, 0x84, 0x94, 0x19, 0xc0, 0x16, 0xe7, 0x47, 0x4d, 0x28, 0x53, 0x27, 0x26, 0xc2, 0x4b, 0x2d,
- 0x8c, 0x57, 0xa2, 0xfc, 0x02, 0x4e, 0xfe, 0x31, 0x5c, 0x39, 0xc3, 0xf0, 0xa0, 0x43, 0xb8, 0x1a,
- 0x25, 0x65, 0x8d, 0x53, 0xcf, 0x6a, 0x3e, 0x9e, 0xf3, 0x3c, 0x8d, 0xb1, 0x4f, 0xde, 0x51, 0x5c,
- 0x09, 0x4e, 0xd1, 0x7c, 0xf9, 0x4d, 0x78, 0xe3, 0x4b, 0xac, 0x8e, 0x5c, 0x84, 0xbc, 0x58, 0xcb,
- 0xf2, 0x1d, 0x28, 0xc7, 0x17, 0x20, 0x7a, 0x6b, 0x76, 0x41, 0x27, 0x59, 0x34, 0x34, 0xb5, 0x2a,
- 0xe5, 0x3c, 0x64, 0xd9, 0xea, 0x92, 0x0b, 0x90, 0xe3, 0x26, 0x46, 0xfe, 0xe3, 0x24, 0x14, 0xa3,
- 0x25, 0x82, 0x3e, 0x81, 0x4c, 0x74, 0x5a, 0xb8, 0xd8, 0x58, 0x32, 0x3e, 0xea, 0xc6, 0x87, 0x2b,
- 0x75, 0xf1, 0xe9, 0x08, 0x59, 0xe5, 0x1e, 0xe4, 0xf8, 0x12, 0xa3, 0x51, 0xf3, 0x44, 0xb1, 0x2e,
- 0xd0, 0xaa, 0x18, 0xf7, 0x46, 0x31, 0x0a, 0x31, 0x94, 0x5f, 0xa7, 0x62, 0xa9, 0xfb, 0xc9, 0x13,
- 0x81, 0x2e, 0x64, 0x4d, 0x62, 0xe3, 0x13, 0x21, 0xe8, 0x1b, 0x17, 0x9a, 0xdc, 0x6a, 0x9d, 0x42,
- 0x50, 0xfb, 0xc5, 0xb0, 0xd0, 0x67, 0x50, 0xc0, 0xb6, 0x75, 0xe0, 0x18, 0x81, 0x2b, 0xc6, 0xe4,
- 0x5b, 0x17, 0xc3, 0xad, 0x51, 0x94, 0x9e, 0x4b, 0xad, 0x38, 0xe6, 0x3f, 0xe5, 0xf7, 0x21, 0xcb,
- 0xa4, 0xa1, 0x37, 0xa1, 0xcc, 0xa4, 0x19, 0x43, 0xcb, 0xb6, 0x2d, 0x5f, 0x1c, 0x97, 0x94, 0x18,
- 0xad, 0xc9, 0x48, 0xf2, 0x43, 0xc8, 0x0b, 0x04, 0x74, 0x1d, 0x72, 0x23, 0xe2, 0x59, 0x2e, 0x8f,
- 0xc5, 0xd2, 0xba, 0xf8, 0xa2, 0x74, 0x77, 0x30, 0xf0, 0x49, 0xc0, 0x9c, 0x84, 0xb4, 0x2e, 0xbe,
- 0x36, 0xae, 0xc1, 0x95, 0x33, 0xd6, 0x80, 0xf2, 0x47, 0x29, 0x28, 0x46, 0x59, 0x6c, 0xb4, 0x07,
- 0x15, 0xdc, 0x67, 0xd7, 0xfd, 0x46, 0x38, 0x08, 0x88, 0xe7, 0x5c, 0xf4, 0x21, 0xc5, 0x12, 0x87,
- 0xe9, 0x70, 0x14, 0xf4, 0x18, 0xf2, 0xc7, 0x16, 0x79, 0x7e, 0xb9, 0x73, 0xfb, 0x1c, 0x85, 0xd8,
- 0x72, 0xd0, 0xe7, 0xb0, 0x22, 0xc2, 0xd1, 0x21, 0x1e, 0x8d, 0xa8, 0x7f, 0x30, 0x70, 0x84, 0xc7,
- 0x75, 0x11, 0x58, 0x11, 0xdb, 0x36, 0x39, 0xd6, 0x96, 0xa3, 0x7c, 0x0a, 0xa5, 0xd8, 0x53, 0x1b,
- 0x24, 0x41, 0x7a, 0xec, 0x85, 0x99, 0x11, 0xfa, 0x13, 0xad, 0x42, 0x7e, 0xc4, 0x0f, 0x1d, 0x98,
- 0xd8, 0xb2, 0x1e, 0x7e, 0x3e, 0xca, 0x14, 0x92, 0x52, 0x4a, 0xf9, 0xb3, 0x24, 0x5c, 0x0d, 0xf3,
- 0xef, 0xf1, 0xb7, 0x40, 0xca, 0x4f, 0x92, 0x50, 0x8e, 0x13, 0xd0, 0xdb, 0x90, 0xab, 0xb7, 0xd9,
- 0x15, 0x9a, 0x84, 0xbc, 0xca, 0xd2, 0xb0, 0x88, 0xa5, 0x61, 0x89, 0x73, 0xbc, 0x6e, 0xba, 0xfd,
- 0x23, 0x9e, 0x99, 0x7e, 0x07, 0xf2, 0xc2, 0x49, 0x96, 0x92, 0x53, 0x19, 0x6c, 0x5a, 0x4d, 0xb8,
- 0x49, 0xb4, 0xde, 0x4d, 0x28, 0xa8, 0x4f, 0x7b, 0xaa, 0xde, 0xaa, 0x35, 0x66, 0xb2, 0xec, 0xb4,
- 0x22, 0xf9, 0x82, 0x4e, 0x05, 0xb6, 0xd7, 0x8f, 0x6f, 0x2b, 0x0f, 0x60, 0xa9, 0xce, 0xe0, 0xc3,
- 0x33, 0xa9, 0x77, 0x61, 0xb9, 0xef, 0x3a, 0x01, 0xb6, 0x1c, 0x1a, 0xdf, 0x0f, 0xf1, 0x41, 0x98,
- 0x25, 0xaa, 0x44, 0x64, 0x8d, 0x52, 0x95, 0x7f, 0x49, 0x42, 0x45, 0x18, 0xb4, 0x90, 0xb7, 0x02,
- 0x29, 0xd7, 0x0f, 0x13, 0xb4, 0xae, 0xcf, 0x13, 0xb4, 0xfd, 0xc3, 0x49, 0x82, 0xb6, 0x7f, 0x48,
- 0x87, 0xac, 0xef, 0x0e, 0x87, 0xd8, 0x09, 0x53, 0x07, 0xe1, 0x27, 0x6a, 0x40, 0x9a, 0x38, 0xc7,
- 0x8b, 0xbc, 0x77, 0x99, 0x92, 0x5e, 0x55, 0x9d, 0x63, 0x7e, 0xde, 0x43, 0x61, 0xe4, 0x7b, 0x50,
- 0x08, 0x09, 0x0b, 0xbd, 0x2c, 0xf9, 0x9f, 0x24, 0x2c, 0xab, 0x62, 0x80, 0xc2, 0x7e, 0x75, 0xa1,
- 0x10, 0x3e, 0x53, 0x15, 0xcb, 0x60, 0x1e, 0xcf, 0xaa, 0x36, 0xb2, 0xba, 0xc4, 0x3b, 0xb6, 0xfa,
- 0xa4, 0x1e, 0xbd, 0x53, 0xd5, 0x23, 0x20, 0xb4, 0x07, 0x39, 0x76, 0xc1, 0x31, 0x3c, 0x37, 0x9f,
- 0xc7, 0xa7, 0x9e, 0x69, 0x18, 0xbf, 0xe2, 0x15, 0x3e, 0x1d, 0xe2, 0x68, 0xf2, 0x43, 0x28, 0xc5,
- 0xc8, 0x0b, 0xf5, 0xfd, 0x47, 0xb0, 0x3c, 0xb3, 0x26, 0x5e, 0xce, 0x33, 0xaa, 0xaf, 0x41, 0x25,
- 0xf6, 0xb6, 0x71, 0x72, 0xff, 0x60, 0x29, 0x46, 0xd5, 0x4c, 0x65, 0x1d, 0xca, 0x53, 0xb2, 0xc5,
- 0x7a, 0x4b, 0xce, 0xb1, 0xde, 0x94, 0xff, 0xce, 0x40, 0x29, 0x76, 0xcb, 0x15, 0x69, 0x90, 0xb5,
- 0x02, 0x12, 0xed, 0xec, 0x77, 0x16, 0xbb, 0x24, 0x5b, 0xd5, 0x02, 0x32, 0xd4, 0x39, 0x82, 0x3c,
- 0x00, 0xd0, 0x4c, 0xe2, 0x04, 0xd6, 0xc0, 0x22, 0x1e, 0xb5, 0xcd, 0xf1, 0x37, 0x70, 0xa2, 0x75,
- 0xa5, 0x60, 0xf2, 0xfc, 0x8d, 0x6e, 0xde, 0x93, 0x2a, 0x13, 0x8b, 0x31, 0xe1, 0xdb, 0xf5, 0x9c,
- 0x70, 0x5e, 0xd2, 0xd1, 0xbc, 0xc8, 0xbf, 0x48, 0x41, 0x86, 0xca, 0x45, 0x5a, 0x74, 0xce, 0x31,
- 0xdf, 0x5b, 0xb2, 0xa9, 0x86, 0x47, 0x2d, 0x65, 0x99, 0xd5, 0x86, 0x48, 0xd9, 0xa6, 0xe6, 0xce,
- 0x9a, 0xc5, 0xc1, 0x66, 0xee, 0x0d, 0xa2, 0xf7, 0x43, 0xcd, 0xe1, 0x36, 0xf6, 0x6a, 0x95, 0x3f,
- 0xc8, 0xae, 0x86, 0x0f, 0xb2, 0xab, 0x35, 0x27, 0x7c, 0x66, 0x89, 0x3e, 0x86, 0x92, 0x7f, 0xe8,
- 0x7a, 0x01, 0x3f, 0x78, 0x12, 0x71, 0xea, 0xd9, 0x1c, 0xc0, 0x2a, 0xb2, 0x1b, 0x68, 0x54, 0x39,
- 0x6d, 0xbc, 0x4f, 0x6c, 0xf1, 0xa2, 0x8f, 0x7f, 0xa0, 0x57, 0xa1, 0x60, 0x5b, 0xce, 0x91, 0x31,
- 0xf6, 0x6c, 0x16, 0xfd, 0x15, 0xf5, 0x3c, 0xfd, 0xde, 0xf5, 0x6c, 0xf9, 0x47, 0xe2, 0x2e, 0xe3,
- 0xf8, 0x05, 0x77, 0x19, 0x45, 0x4e, 0x97, 0xdd, 0x4a, 0xd2, 0x5a, 0x3d, 0x75, 0x5b, 0xd5, 0x79,
- 0x6e, 0x98, 0xe7, 0x80, 0xd3, 0xf1, 0xec, 0x6e, 0x06, 0x2d, 0x41, 0x31, 0x7a, 0xad, 0x2d, 0x65,
- 0x59, 0x1e, 0x78, 0x57, 0xaf, 0xb1, 0x87, 0x06, 0x39, 0x54, 0x01, 0x78, 0x54, 0xdb, 0xab, 0x19,
- 0x9b, 0x8d, 0x5a, 0xb7, 0x2b, 0xe5, 0x95, 0x7f, 0x2c, 0xc0, 0xb5, 0x26, 0xf1, 0x7d, 0x7c, 0x40,
- 0x9e, 0x58, 0xc1, 0x61, 0xec, 0xdd, 0xc3, 0x4b, 0x7e, 0x80, 0xf9, 0x6d, 0xc8, 0xb2, 0x9c, 0xeb,
- 0xa2, 0x2f, 0x52, 0xa9, 0xeb, 0xc2, 0x18, 0xd1, 0xf7, 0xa8, 0x65, 0x17, 0x0f, 0x43, 0x62, 0x8b,
- 0x68, 0xbe, 0x60, 0x69, 0xfa, 0xaa, 0xd2, 0x4e, 0x42, 0x17, 0xb7, 0x26, 0xa3, 0xcb, 0x4b, 0xdf,
- 0x87, 0x15, 0xdf, 0x3c, 0x8a, 0x2e, 0x20, 0xc4, 0x2f, 0x3c, 0x5e, 0x60, 0x2f, 0xde, 0x49, 0xe8,
- 0xcb, 0xfe, 0x8c, 0x29, 0x7a, 0x02, 0x95, 0x11, 0xf6, 0x0c, 0xd3, 0x8d, 0x9a, 0x9f, 0x9b, 0xdb,
- 0x28, 0xc5, 0xaf, 0x50, 0xd3, 0xe8, 0x76, 0x14, 0xbf, 0xf3, 0xde, 0x06, 0x18, 0x45, 0x6b, 0x53,
- 0x04, 0xe4, 0x8b, 0x3d, 0xa5, 0xde, 0x49, 0xe8, 0x31, 0x08, 0xa4, 0x43, 0x29, 0xf6, 0xfc, 0x5d,
- 0x04, 0xe3, 0x0b, 0x3e, 0x96, 0xde, 0x49, 0xe8, 0x71, 0x10, 0xd4, 0x85, 0xb2, 0x47, 0xb0, 0x19,
- 0xf5, 0xbd, 0x38, 0x37, 0x68, 0xec, 0xe6, 0x1d, 0x05, 0xf5, 0x62, 0x17, 0xf1, 0x9a, 0x00, 0x93,
- 0x4b, 0x17, 0x22, 0x74, 0x5e, 0xe8, 0xb6, 0x03, 0x8d, 0xc2, 0xa3, 0xdb, 0x15, 0x68, 0x00, 0x57,
- 0x62, 0x8f, 0x10, 0xa3, 0xa6, 0x96, 0x17, 0x7c, 0xb4, 0x1d, 0xbb, 0x77, 0xb7, 0x93, 0xd0, 0x85,
- 0x8b, 0x17, 0xbf, 0x8c, 0x47, 0x00, 0x9d, 0x7e, 0x3c, 0xb1, 0xba, 0x74, 0xf1, 0xb7, 0xe1, 0x13,
- 0x31, 0xf1, 0x63, 0x99, 0x3d, 0x58, 0x9a, 0x56, 0xe7, 0xca, 0x85, 0x36, 0x41, 0xaa, 0x6f, 0x83,
- 0xd8, 0xf7, 0x46, 0x0e, 0x32, 0x9e, 0xeb, 0x06, 0xca, 0xcf, 0x73, 0x70, 0x5d, 0xfd, 0x82, 0xf4,
- 0xc7, 0xec, 0x76, 0x7e, 0x37, 0xc0, 0x07, 0xd1, 0x6a, 0xea, 0x40, 0x29, 0xb6, 0x37, 0x0a, 0xeb,
- 0xb1, 0xe8, 0xd3, 0xf0, 0x38, 0x04, 0x35, 0xac, 0x7c, 0x96, 0xc5, 0xae, 0x6f, 0x89, 0x19, 0x3b,
- 0xe3, 0x5d, 0x85, 0x3a, 0x97, 0x27, 0x72, 0x56, 0xbb, 0x27, 0x8a, 0xa1, 0x99, 0x53, 0xaf, 0x2b,
- 0x5e, 0x9f, 0xfa, 0x13, 0x8b, 0x0c, 0xbb, 0xaf, 0x12, 0xff, 0x17, 0x8a, 0xd5, 0xc9, 0x7b, 0xe7,
- 0x2c, 0x2b, 0x8c, 0xde, 0x2c, 0x4f, 0x9b, 0xd1, 0xdc, 0x65, 0xcd, 0xe8, 0x00, 0x4a, 0x63, 0x9f,
- 0x78, 0xec, 0x60, 0x8c, 0xf8, 0xab, 0xf9, 0xcb, 0x76, 0x78, 0xd7, 0x27, 0x1e, 0xbb, 0xdd, 0x4b,
- 0x3b, 0x3c, 0x0e, 0x3f, 0x7c, 0xf4, 0x0c, 0x72, 0xec, 0x3e, 0x89, 0xbf, 0x5a, 0x60, 0x22, 0x6a,
- 0x17, 0x17, 0xc1, 0x2e, 0x01, 0x6b, 0xa6, 0x2e, 0x00, 0xe5, 0x36, 0x94, 0x62, 0xc3, 0x3c, 0x8f,
- 0x43, 0xf2, 0x55, 0x00, 0xdb, 0xed, 0x63, 0x9b, 0x1f, 0xed, 0x73, 0x05, 0x28, 0x32, 0x4a, 0x0b,
- 0x0f, 0x09, 0x05, 0x8c, 0x75, 0xe3, 0x25, 0x00, 0x3e, 0x86, 0xbc, 0x68, 0xf4, 0xe5, 0xc1, 0xd6,
- 0x3f, 0x85, 0x02, 0xfb, 0x77, 0x19, 0xea, 0xff, 0xbd, 0x79, 0xca, 0x7f, 0xa0, 0x7b, 0x3e, 0xf3,
- 0x1c, 0xda, 0x23, 0xfe, 0xff, 0x25, 0xbf, 0xf9, 0xf3, 0xbf, 0x79, 0xca, 0x3d, 0x04, 0xca, 0xb5,
- 0xeb, 0x39, 0xeb, 0x1a, 0x2c, 0x31, 0x80, 0xbe, 0xf8, 0x1b, 0x98, 0x79, 0x50, 0xfe, 0x39, 0x44,
- 0x29, 0xef, 0xc7, 0xfe, 0x4e, 0x66, 0xe3, 0x1b, 0xf0, 0xe5, 0x7f, 0x69, 0xb3, 0x51, 0xd4, 0xd9,
- 0x1d, 0xb7, 0xda, 0xc8, 0xfa, 0xac, 0x14, 0xd2, 0x8d, 0xe3, 0xdb, 0xfb, 0x39, 0x26, 0xee, 0xce,
- 0xff, 0x05, 0x00, 0x00, 0xff, 0xff, 0x02, 0x6d, 0xf4, 0xf1, 0x2d, 0x47, 0x00, 0x00,
+var fileDescriptor_beam_runner_api_d5fa30116074ddde = []byte{
+ // 5086 bytes of a gzipped FileDescriptorProto
+ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x5c, 0xdd, 0x6f, 0xdb, 0x58,
+ 0x76, 0xd7, 0xb7, 0xa5, 0x23, 0x59, 0xa6, 0xaf, 0x9d, 0xac, 0xc3, 0x9d, 0x9d, 0x24, 0x9c, 0xec,
+ 0x4c, 0x76, 0x76, 0x46, 0x93, 0x38, 0xc9, 0x24, 0xf1, 0xcc, 0x66, 0x56, 0xb2, 0xa8, 0x98, 0x89,
+ 0xbe, 0x86, 0x92, 0x9d, 0x64, 0x76, 0x76, 0xb8, 0xb4, 0x78, 0x65, 0x13, 0xa6, 0x48, 0x2d, 0x49,
+ 0x39, 0xa3, 0xc5, 0x2e, 0x0a, 0xf4, 0x61, 0x50, 0xa0, 0x40, 0xd1, 0x3e, 0xf4, 0x61, 0x9e, 0x0a,
+ 0xec, 0x02, 0x05, 0xda, 0x3e, 0xf4, 0x63, 0xdb, 0x02, 0x7d, 0xdd, 0x6e, 0xff, 0x82, 0x16, 0x28,
+ 0xd0, 0xff, 0xa2, 0x2d, 0xf6, 0xa1, 0x7d, 0x2a, 0xee, 0x07, 0x29, 0x4a, 0xb6, 0x33, 0x92, 0x1d,
+ 0xf4, 0x4d, 0x3c, 0xbc, 0xe7, 0x77, 0xee, 0x3d, 0xf7, 0xde, 0x73, 0xcf, 0x39, 0xf7, 0x50, 0x70,
+ 0x69, 0x1f, 0xeb, 0x03, 0xcd, 0x1d, 0xd9, 0x36, 0x76, 0x35, 0x7d, 0x68, 0x96, 0x86, 0xae, 0xe3,
+ 0x3b, 0xe8, 0xba, 0xe3, 0x1e, 0x94, 0xf4, 0xa1, 0xde, 0x3b, 0xc4, 0x25, 0xd2, 0xa2, 0x34, 0x70,
+ 0x0c, 0x6c, 0x95, 0x86, 0xe6, 0x10, 0x5b, 0xa6, 0x8d, 0x4b, 0xc7, 0xb7, 0xc5, 0x15, 0x6c, 0x1b,
+ 0x43, 0xc7, 0xb4, 0x7d, 0x8f, 0xf1, 0x88, 0x57, 0x0e, 0x1c, 0xe7, 0xc0, 0xc2, 0x1f, 0xd0, 0xa7,
+ 0xfd, 0x51, 0xff, 0x03, 0xdd, 0x1e, 0xf3, 0x57, 0xd7, 0x66, 0x5f, 0x19, 0xd8, 0xeb, 0xb9, 0xe6,
+ 0xd0, 0x77, 0x5c, 0xd6, 0x42, 0xfa, 0x4d, 0x1c, 0x96, 0x2b, 0x58, 0x1f, 0x6c, 0x3b, 0xb6, 0xe7,
+ 0xeb, 0xb6, 0xef, 0x49, 0x7f, 0x13, 0x87, 0x5c, 0xf8, 0x84, 0x6e, 0xc3, 0x7a, 0x43, 0x69, 0x6a,
+ 0x5d, 0xa5, 0x21, 0x77, 0xba, 0xe5, 0x46, 0x5b, 0x6b, 0x28, 0xf5, 0xba, 0xd2, 0x11, 0x62, 0xe2,
+ 0xb7, 0xfe, 0xf2, 0xef, 0xff, 0xf7, 0x37, 0xe9, 0xd5, 0xf7, 0x1f, 0x6e, 0x6e, 0xde, 0xb9, 0x73,
+ 0x7f, 0xf3, 0xd6, 0x9d, 0x0f, 0x1f, 0xdc, 0xbb, 0x7b, 0xff, 0xfe, 0x3d, 0x74, 0x0b, 0xd6, 0x1b,
+ 0xe5, 0xe7, 0x27, 0x59, 0xe2, 0xe2, 0x65, 0xca, 0x22, 0x9c, 0xe0, 0x78, 0x04, 0xd2, 0xe3, 0x7a,
+ 0xab, 0x52, 0xae, 0x6b, 0xcf, 0x94, 0x66, 0xb5, 0xf5, 0x4c, 0x3b, 0x95, 0x3f, 0x31, 0xcd, 0x7f,
+ 0xfb, 0xe1, 0xbd, 0x5b, 0x77, 0x29, 0xbf, 0xf4, 0x8f, 0x59, 0x80, 0x6d, 0x67, 0x30, 0x74, 0x6c,
+ 0x4c, 0xfa, 0xfc, 0x63, 0x00, 0xdf, 0xd5, 0x6d, 0xaf, 0xef, 0xb8, 0x03, 0x6f, 0x23, 0x7e, 0x2d,
+ 0x79, 0x33, 0xbf, 0xf9, 0x83, 0xd2, 0x37, 0x6a, 0xb6, 0x34, 0x81, 0x28, 0x75, 0x43, 0x7e, 0xd9,
+ 0xf6, 0xdd, 0xb1, 0x1a, 0x01, 0x44, 0x3d, 0x28, 0x0c, 0x7b, 0x8e, 0x65, 0xe1, 0x9e, 0x6f, 0x3a,
+ 0xb6, 0xb7, 0x91, 0xa0, 0x02, 0x3e, 0x59, 0x4c, 0x40, 0x3b, 0x82, 0xc0, 0x44, 0x4c, 0x81, 0xa2,
+ 0x31, 0xac, 0xbf, 0x34, 0x6d, 0xc3, 0x79, 0x69, 0xda, 0x07, 0x9a, 0xe7, 0xbb, 0xba, 0x8f, 0x0f,
+ 0x4c, 0xec, 0x6d, 0x24, 0xa9, 0xb0, 0xda, 0x62, 0xc2, 0x9e, 0x05, 0x48, 0x9d, 0x10, 0x88, 0xc9,
+ 0x5c, 0x7b, 0x79, 0xf2, 0x0d, 0xfa, 0x14, 0x32, 0x3d, 0xc7, 0xc0, 0xae, 0xb7, 0x91, 0xa2, 0xc2,
+ 0x1e, 0x2e, 0x26, 0x6c, 0x9b, 0xf2, 0x32, 0x7c, 0x0e, 0x44, 0x54, 0x86, 0xed, 0x63, 0xd3, 0x75,
+ 0xec, 0x01, 0x69, 0xb3, 0x91, 0x3e, 0x8f, 0xca, 0xe4, 0x08, 0x02, 0x57, 0x59, 0x14, 0x54, 0xb4,
+ 0x60, 0x65, 0x66, 0xda, 0x90, 0x00, 0xc9, 0x23, 0x3c, 0xde, 0x88, 0x5f, 0x8b, 0xdf, 0xcc, 0xa9,
+ 0xe4, 0x27, 0xda, 0x86, 0xf4, 0xb1, 0x6e, 0x8d, 0xf0, 0x46, 0xe2, 0x5a, 0xfc, 0x66, 0x7e, 0xf3,
+ 0xfd, 0x39, 0xba, 0xd0, 0x0e, 0x51, 0x55, 0xc6, 0xbb, 0x95, 0x78, 0x10, 0x17, 0x1d, 0x58, 0x3d,
+ 0x31, 0x87, 0xa7, 0xc8, 0xab, 0x4e, 0xcb, 0x2b, 0xcd, 0x23, 0x6f, 0x3b, 0x84, 0x8d, 0x0a, 0xfc,
+ 0x39, 0x6c, 0x9c, 0x35, 0x8f, 0xa7, 0xc8, 0x7d, 0x32, 0x2d, 0xf7, 0xee, 0x1c, 0x72, 0x67, 0xd1,
+ 0xc7, 0x51, 0xe9, 0x3d, 0xc8, 0x47, 0x26, 0xf6, 0x14, 0x81, 0x8f, 0xa6, 0x05, 0xde, 0x9c, 0x6b,
+ 0x6e, 0x0d, 0xec, 0xce, 0xe8, 0xf4, 0xc4, 0x24, 0xbf, 0x1e, 0x9d, 0x46, 0x60, 0x23, 0x02, 0xa5,
+ 0xff, 0x88, 0x43, 0xb6, 0xcd, 0x9b, 0xa1, 0x06, 0x40, 0x2f, 0x5c, 0x6d, 0x54, 0xde, 0x7c, 0xeb,
+ 0x63, 0xb2, 0x44, 0xd5, 0x08, 0x00, 0x7a, 0x0f, 0x90, 0xeb, 0x38, 0xbe, 0x16, 0x5a, 0x0e, 0xcd,
+ 0x34, 0x98, 0xb1, 0xc8, 0xa9, 0x02, 0x79, 0x13, 0x2e, 0x2b, 0xc5, 0x20, 0x9b, 0xae, 0x60, 0x98,
+ 0xde, 0xd0, 0xd2, 0xc7, 0x9a, 0xa1, 0xfb, 0xfa, 0x46, 0x72, 0xee, 0xa1, 0x55, 0x19, 0x5b, 0x55,
+ 0xf7, 0x75, 0x35, 0x6f, 0x4c, 0x1e, 0xa4, 0x3f, 0x4c, 0x01, 0x4c, 0xd6, 0x2e, 0xba, 0x0a, 0xf9,
+ 0x91, 0x6d, 0xfe, 0x74, 0x84, 0x35, 0x5b, 0x1f, 0xe0, 0x8d, 0x34, 0xd5, 0x27, 0x30, 0x52, 0x53,
+ 0x1f, 0x60, 0xb4, 0x0d, 0x29, 0x6f, 0x88, 0x7b, 0x7c, 0xe4, 0x1f, 0xcc, 0x21, 0xba, 0x36, 0xb2,
+ 0xe9, 0x32, 0xed, 0x0c, 0x71, 0x4f, 0xa5, 0xcc, 0xe8, 0x06, 0x2c, 0x7b, 0xa3, 0xfd, 0x88, 0xf9,
+ 0x65, 0x03, 0x9e, 0x26, 0x12, 0x13, 0x63, 0xda, 0xc3, 0x91, 0x1f, 0xd8, 0xb3, 0x87, 0x0b, 0x6d,
+ 0xc3, 0x92, 0x42, 0x79, 0xb9, 0x89, 0x61, 0x40, 0xa8, 0x0b, 0x4b, 0xce, 0xc8, 0xa7, 0x98, 0xcc,
+ 0x6c, 0x6d, 0x2d, 0x86, 0xd9, 0x62, 0xcc, 0x0c, 0x34, 0x80, 0x3a, 0x31, 0x2d, 0x99, 0x0b, 0x4f,
+ 0x8b, 0xf8, 0x10, 0xf2, 0x91, 0xfe, 0x9f, 0xb2, 0xbc, 0xd7, 0xa3, 0xcb, 0x3b, 0x17, 0xdd, 0x1f,
+ 0x5b, 0x50, 0x88, 0x76, 0x73, 0x11, 0x5e, 0xe9, 0x1f, 0x96, 0x61, 0xad, 0xe3, 0xeb, 0xb6, 0xa1,
+ 0xbb, 0xc6, 0x64, 0xd8, 0x9e, 0xf4, 0x17, 0x49, 0x80, 0xb6, 0x6b, 0x0e, 0x4c, 0xdf, 0x3c, 0xc6,
+ 0x1e, 0xfa, 0x1e, 0x64, 0xda, 0x65, 0x55, 0xab, 0xb6, 0x84, 0x98, 0xf8, 0x9d, 0x5f, 0x92, 0xe3,
+ 0xf6, 0x5b, 0x64, 0x80, 0x5b, 0xe1, 0xe4, 0x6d, 0x0d, 0x75, 0xd7, 0x70, 0xb6, 0x8e, 0x6f, 0xa3,
+ 0xf7, 0x60, 0xa9, 0x56, 0x2f, 0x77, 0xbb, 0x72, 0x53, 0x88, 0x8b, 0x57, 0x69, 0xdb, 0x2b, 0x33,
+ 0x6d, 0xfb, 0x96, 0xee, 0xfb, 0xd8, 0x26, 0xad, 0x3f, 0x84, 0xc2, 0x63, 0xb5, 0xb5, 0xdb, 0xd6,
+ 0x2a, 0x2f, 0xb4, 0xa7, 0xf2, 0x0b, 0x21, 0x21, 0xde, 0xa0, 0x2c, 0x6f, 0xce, 0xb0, 0x1c, 0xb8,
+ 0xce, 0x68, 0xa8, 0xed, 0x8f, 0xb5, 0x23, 0x3c, 0xe6, 0x52, 0x94, 0x46, 0x7b, 0xb7, 0xde, 0x91,
+ 0x85, 0xe4, 0x19, 0x52, 0xcc, 0xc1, 0x70, 0x64, 0x79, 0x98, 0xb4, 0xbe, 0x0f, 0xc5, 0x72, 0xa7,
+ 0xa3, 0x3c, 0x6e, 0x72, 0x4f, 0xa2, 0x23, 0xa4, 0xc4, 0xb7, 0x28, 0xd3, 0x77, 0x66, 0x98, 0xd8,
+ 0xc9, 0xa7, 0x99, 0xb6, 0x4f, 0x07, 0x73, 0x07, 0xf2, 0x5d, 0xb9, 0xd3, 0xd5, 0x3a, 0x5d, 0x55,
+ 0x2e, 0x37, 0x84, 0xb4, 0x28, 0x51, 0xae, 0x37, 0x66, 0xb8, 0x7c, 0xec, 0xf9, 0x9e, 0xef, 0x12,
+ 0xe2, 0xf1, 0x6d, 0x74, 0x17, 0xf2, 0x8d, 0x72, 0x3b, 0x14, 0x95, 0x39, 0x43, 0xd4, 0x40, 0x1f,
+ 0x6a, 0x4c, 0x9c, 0x47, 0xb8, 0x1e, 0xc0, 0x72, 0x43, 0x56, 0x1f, 0xcb, 0x21, 0xdf, 0x92, 0xf8,
+ 0x5d, 0xca, 0x77, 0x75, 0x96, 0x0f, 0xbb, 0x07, 0x38, 0xc2, 0x29, 0xf9, 0xb0, 0x5e, 0xc5, 0x43,
+ 0x17, 0xf7, 0x74, 0x1f, 0x1b, 0x91, 0x49, 0x7b, 0x1b, 0x52, 0xaa, 0x5c, 0xae, 0x0a, 0x31, 0xf1,
+ 0x0d, 0x0a, 0x74, 0x79, 0x06, 0xc8, 0xc5, 0xba, 0xc1, 0xfb, 0xbb, 0xad, 0xca, 0xe5, 0xae, 0xac,
+ 0xed, 0x29, 0xf2, 0x33, 0x21, 0x7e, 0x46, 0x7f, 0x7b, 0x2e, 0xd6, 0x7d, 0xac, 0x1d, 0x9b, 0xf8,
+ 0x25, 0x91, 0xfa, 0x5f, 0x71, 0xee, 0x5d, 0x79, 0xa6, 0x8f, 0x3d, 0xf4, 0x31, 0xac, 0x6c, 0xb7,
+ 0x1a, 0x15, 0xa5, 0x29, 0x6b, 0x6d, 0x59, 0xa5, 0x73, 0x19, 0x13, 0xdf, 0xa1, 0x40, 0xd7, 0x67,
+ 0x81, 0x9c, 0xc1, 0xbe, 0x69, 0x63, 0x6d, 0x88, 0xdd, 0x60, 0x3a, 0x1f, 0x81, 0x10, 0x70, 0x33,
+ 0x97, 0xaf, 0xfe, 0x42, 0x88, 0x8b, 0x37, 0x29, 0xbb, 0x74, 0x06, 0xfb, 0x81, 0xe5, 0xec, 0xeb,
+ 0x96, 0x45, 0xf9, 0x6f, 0x41, 0x4e, 0x95, 0x3b, 0x3b, 0xbb, 0xb5, 0x5a, 0x5d, 0x16, 0x12, 0xe2,
+ 0x75, 0xca, 0xf8, 0xed, 0x13, 0xe3, 0xf5, 0x0e, 0x47, 0xfd, 0xbe, 0x85, 0xf9, 0xa0, 0x9f, 0xa9,
+ 0x4a, 0x57, 0xd6, 0x6a, 0x4a, 0x5d, 0xee, 0x08, 0xc9, 0xb3, 0xd6, 0x83, 0x6b, 0xfa, 0x58, 0xeb,
+ 0x9b, 0x16, 0xa6, 0xaa, 0xfe, 0x5d, 0x02, 0x56, 0xb7, 0x99, 0xfc, 0x88, 0x67, 0xa9, 0x82, 0x38,
+ 0x33, 0x76, 0xad, 0xad, 0xca, 0x9c, 0x24, 0xc4, 0xc4, 0x4d, 0x0a, 0xfd, 0xde, 0xab, 0xd5, 0xa0,
+ 0x91, 0x19, 0x64, 0x24, 0xd2, 0xbf, 0x7d, 0x90, 0x66, 0x31, 0xd9, 0xf2, 0x28, 0x6f, 0x6f, 0xef,
+ 0x36, 0x76, 0xeb, 0xe5, 0x6e, 0x4b, 0x25, 0xce, 0xf3, 0x16, 0xc5, 0xbe, 0xfb, 0x0d, 0xd8, 0x6c,
+ 0xcd, 0xe8, 0xbd, 0xde, 0x68, 0x30, 0xb2, 0x74, 0xdf, 0x71, 0xe9, 0x92, 0xfb, 0x1c, 0xae, 0xce,
+ 0xca, 0x90, 0x9f, 0x77, 0xd5, 0xf2, 0x76, 0x57, 0x6b, 0xed, 0x76, 0xdb, 0xbb, 0x5d, 0xe2, 0x5d,
+ 0xdf, 0xa7, 0x02, 0x6e, 0x7f, 0x83, 0x00, 0xfc, 0xa5, 0xef, 0xea, 0x3d, 0x5f, 0xe3, 0x16, 0x92,
+ 0xa0, 0x3f, 0x81, 0xcb, 0xe1, 0x9c, 0x92, 0x2d, 0x2e, 0x57, 0xb5, 0xbd, 0x72, 0x7d, 0x97, 0x2a,
+ 0xbb, 0x44, 0x41, 0x6f, 0x9e, 0x35, 0xb3, 0x64, 0xb3, 0x63, 0x43, 0xa3, 0x66, 0x8a, 0xea, 0xfd,
+ 0x8f, 0x52, 0x70, 0xa5, 0x33, 0xb4, 0x4c, 0xdf, 0xd7, 0xf7, 0x2d, 0xdc, 0xd6, 0xdd, 0xaa, 0x13,
+ 0xd1, 0x7f, 0x1d, 0x2e, 0xb5, 0xcb, 0x8a, 0xaa, 0x3d, 0x53, 0xba, 0x3b, 0x9a, 0x2a, 0x77, 0xba,
+ 0xaa, 0xb2, 0xdd, 0x55, 0x5a, 0x4d, 0x21, 0x26, 0xde, 0xa6, 0x82, 0xbe, 0x3f, 0x23, 0xc8, 0x33,
+ 0xfa, 0xda, 0x50, 0x37, 0x5d, 0xed, 0xa5, 0xe9, 0x1f, 0x6a, 0x2e, 0xf6, 0x7c, 0xd7, 0xa4, 0x47,
+ 0x16, 0xe9, 0x77, 0x15, 0x56, 0x3b, 0xed, 0xba, 0xd2, 0x9d, 0x42, 0x8a, 0x8b, 0xef, 0x53, 0xa4,
+ 0x77, 0x4e, 0x41, 0xf2, 0x48, 0xc7, 0x66, 0x51, 0x9a, 0x70, 0xb9, 0xad, 0xb6, 0xb6, 0xe5, 0x4e,
+ 0x87, 0xe8, 0x55, 0xae, 0x6a, 0x72, 0x5d, 0x6e, 0xc8, 0x4d, 0xaa, 0xd2, 0xd3, 0xd7, 0x03, 0xed,
+ 0x94, 0xeb, 0xf4, 0xb0, 0xe7, 0x11, 0x95, 0x62, 0x43, 0xc3, 0x16, 0xa6, 0x1e, 0x0f, 0xc1, 0xab,
+ 0x80, 0x10, 0xe0, 0x85, 0x48, 0x49, 0xf1, 0x3d, 0x8a, 0xf4, 0xf6, 0x2b, 0x90, 0xa2, 0x18, 0xcf,
+ 0xe1, 0xdb, 0x6c, 0x64, 0xe5, 0x66, 0x55, 0xeb, 0x28, 0x9f, 0xc9, 0xd1, 0x21, 0x12, 0x9b, 0x78,
+ 0xfa, 0x5c, 0x4f, 0xc6, 0xa8, 0xdb, 0x86, 0xe6, 0x99, 0x3f, 0xc3, 0xd1, 0xc1, 0x52, 0x64, 0x07,
+ 0xde, 0x09, 0x7a, 0x47, 0x70, 0x27, 0xa3, 0xa5, 0xa2, 0xa6, 0xa4, 0xa4, 0xc5, 0x0a, 0x95, 0xf2,
+ 0xf1, 0x2b, 0x3a, 0x4d, 0x64, 0x84, 0xc3, 0xa7, 0x52, 0x67, 0x04, 0x4a, 0xbf, 0x1f, 0x87, 0xcb,
+ 0xc1, 0xb9, 0xd5, 0x31, 0x0d, 0x4c, 0xcf, 0xce, 0xee, 0x78, 0x88, 0x3d, 0xe9, 0x10, 0x52, 0xb2,
+ 0x3d, 0x1a, 0xa0, 0x0f, 0x20, 0xab, 0x74, 0x65, 0xb5, 0x5c, 0xa9, 0x93, 0x3d, 0x18, 0x35, 0x09,
+ 0x9e, 0x69, 0x60, 0x8d, 0x3a, 0x08, 0x5b, 0xa6, 0x8f, 0x5d, 0xb2, 0xa4, 0xc8, 0x20, 0x3e, 0x80,
+ 0x6c, 0x63, 0xb7, 0xde, 0x55, 0x1a, 0xe5, 0xb6, 0x10, 0x3f, 0x8b, 0x61, 0x30, 0xb2, 0x7c, 0x73,
+ 0xa0, 0x0f, 0x49, 0x27, 0x7e, 0x99, 0x80, 0x7c, 0xc4, 0x2d, 0x9f, 0xf5, 0xa5, 0xe2, 0x27, 0x7c,
+ 0xa9, 0x2b, 0x90, 0xa5, 0xa1, 0x8f, 0x66, 0x1a, 0xfc, 0x28, 0x5e, 0xa2, 0xcf, 0x8a, 0x81, 0xda,
+ 0x00, 0xa6, 0xa7, 0xed, 0x3b, 0x23, 0xdb, 0xc0, 0x06, 0xf5, 0xf3, 0x8a, 0x9b, 0xb7, 0xe7, 0x70,
+ 0x28, 0x14, 0xaf, 0xc2, 0x78, 0x4a, 0x64, 0xd0, 0x6a, 0xce, 0x0c, 0x9e, 0xd1, 0x26, 0x5c, 0x3a,
+ 0x11, 0x2b, 0x8e, 0x89, 0xe4, 0x14, 0x95, 0x7c, 0x22, 0xc8, 0x1b, 0x2b, 0xc6, 0x09, 0xc7, 0x26,
+ 0x7d, 0x71, 0x7f, 0xf3, 0xeb, 0x25, 0x28, 0xd0, 0x0d, 0xdb, 0xd6, 0xc7, 0x96, 0xa3, 0x1b, 0xe8,
+ 0x31, 0xa4, 0x0d, 0x47, 0xeb, 0xdb, 0xdc, 0xa3, 0xdc, 0x9c, 0x03, 0xbc, 0x63, 0x1c, 0x4d, 0x3b,
+ 0x95, 0x86, 0x53, 0xb3, 0x51, 0x1d, 0x60, 0xa8, 0xbb, 0xfa, 0x00, 0xfb, 0x24, 0x2a, 0x65, 0xf1,
+ 0xf6, 0x7b, 0xf3, 0xb8, 0x77, 0x01, 0x93, 0x1a, 0xe1, 0x47, 0x3f, 0x81, 0xfc, 0x64, 0x9a, 0x03,
+ 0x0f, 0xf4, 0x93, 0xf9, 0xe0, 0xc2, 0xc1, 0x95, 0xc2, 0xb5, 0x18, 0x64, 0x08, 0xbc, 0x90, 0x40,
+ 0x25, 0xf8, 0xe4, 0x08, 0x25, 0x2e, 0x71, 0xe0, 0x8f, 0x2e, 0x2e, 0x81, 0x40, 0x10, 0x2d, 0x84,
+ 0x12, 0x42, 0x02, 0x91, 0xe0, 0x9b, 0x03, 0xec, 0x72, 0x09, 0xe9, 0xf3, 0x49, 0xe8, 0x12, 0x88,
+ 0xa8, 0x04, 0x3f, 0x24, 0xa0, 0x37, 0x01, 0xbc, 0xd0, 0x0e, 0x53, 0xbf, 0x37, 0xab, 0x46, 0x28,
+ 0xe8, 0x16, 0xac, 0x47, 0xb6, 0xaa, 0x16, 0xae, 0xf6, 0x25, 0xba, 0xe6, 0x50, 0xe4, 0xdd, 0x36,
+ 0x5f, 0xf8, 0x77, 0xe0, 0x92, 0x8b, 0x7f, 0x3a, 0x22, 0x1e, 0x94, 0xd6, 0x37, 0x6d, 0xdd, 0x32,
+ 0x7f, 0xa6, 0x93, 0xf7, 0x1b, 0x59, 0x0a, 0xbe, 0x1e, 0xbc, 0xac, 0x45, 0xde, 0x89, 0x47, 0xb0,
+ 0x32, 0xa3, 0xe9, 0x53, 0xbc, 0xde, 0xca, 0x74, 0x40, 0x38, 0xcf, 0xd2, 0x08, 0x41, 0xa3, 0xfe,
+ 0x35, 0x11, 0x36, 0xad, 0xf4, 0xd7, 0x24, 0x2c, 0x00, 0x9d, 0x11, 0x36, 0xa3, 0xff, 0xd7, 0x23,
+ 0x2c, 0x04, 0x8d, 0x7a, 0xff, 0xbf, 0x8e, 0x43, 0x2e, 0xdc, 0x0d, 0xe8, 0x09, 0xa4, 0xfc, 0xf1,
+ 0x90, 0xd9, 0xad, 0xe2, 0xe6, 0x87, 0x8b, 0xec, 0xa4, 0x12, 0x31, 0xbd, 0xcc, 0x02, 0x51, 0x0c,
+ 0xf1, 0x33, 0x48, 0x11, 0x92, 0xa4, 0x72, 0x63, 0xbc, 0x02, 0xf9, 0xdd, 0x66, 0xa7, 0x2d, 0x6f,
+ 0x2b, 0x35, 0x45, 0xae, 0x0a, 0x31, 0x04, 0x90, 0x61, 0x8e, 0xae, 0x10, 0x47, 0xeb, 0x20, 0xb4,
+ 0x95, 0xb6, 0x5c, 0x27, 0xae, 0x42, 0xab, 0xcd, 0x8e, 0x89, 0x04, 0xfa, 0x16, 0xac, 0x45, 0x0e,
+ 0x0e, 0x8d, 0xf8, 0x25, 0x4f, 0x65, 0x55, 0x48, 0x4a, 0x7f, 0x9b, 0x84, 0x5c, 0xa8, 0x3b, 0xa4,
+ 0x02, 0xd0, 0x01, 0x69, 0x91, 0x28, 0x75, 0x1e, 0xc3, 0xb9, 0x47, 0x98, 0x42, 0x98, 0x9d, 0x98,
+ 0x9a, 0xa3, 0x30, 0x14, 0xb3, 0x0e, 0xd9, 0x7d, 0xfd, 0x80, 0x21, 0x26, 0xe6, 0x8e, 0x7b, 0x2b,
+ 0xfa, 0x41, 0x14, 0x6f, 0x69, 0x5f, 0x3f, 0xa0, 0x68, 0x5f, 0x40, 0x91, 0x79, 0x36, 0xd4, 0x10,
+ 0x13, 0x4c, 0x16, 0xc6, 0xdf, 0x9b, 0x2f, 0x8b, 0xc0, 0x18, 0xa3, 0xc8, 0xcb, 0x21, 0x5c, 0xd0,
+ 0x5b, 0x12, 0x4b, 0x50, 0xe4, 0xd4, 0xdc, 0xbd, 0x6d, 0xe8, 0xc3, 0xa9, 0xde, 0x0e, 0xf4, 0x61,
+ 0x80, 0xe6, 0x61, 0x9f, 0xa1, 0xa5, 0xe7, 0x46, 0xeb, 0x60, 0x7f, 0x0a, 0xcd, 0xc3, 0x3e, 0xf9,
+ 0x59, 0xc9, 0xb0, 0xec, 0x81, 0xf4, 0x7d, 0x28, 0x4e, 0x2b, 0x7c, 0xea, 0x2c, 0x8c, 0x4f, 0x9d,
+ 0x85, 0xd2, 0x03, 0x28, 0x44, 0x75, 0x89, 0x6e, 0x82, 0x10, 0xf8, 0x02, 0x33, 0x2c, 0x45, 0x4e,
+ 0xe7, 0xc6, 0x44, 0xfa, 0x3a, 0x0e, 0xe8, 0xa4, 0xca, 0x88, 0x55, 0x8a, 0xf8, 0xbe, 0xb3, 0x20,
+ 0x28, 0xf2, 0x2e, 0xb0, 0x4a, 0x9f, 0xd2, 0xac, 0x0f, 0xf5, 0x46, 0xfb, 0x36, 0x5f, 0x03, 0xe7,
+ 0x39, 0xa9, 0x72, 0x1c, 0xa5, 0x66, 0x4b, 0x7b, 0x50, 0x88, 0xea, 0x1c, 0x5d, 0x83, 0x02, 0xf1,
+ 0x9c, 0x67, 0x3a, 0x03, 0x47, 0x78, 0x1c, 0x74, 0xe2, 0x06, 0x14, 0xd9, 0xd2, 0x9e, 0x71, 0x1a,
+ 0x0a, 0x94, 0xba, 0x3d, 0xd1, 0x56, 0x54, 0xfb, 0x0b, 0x68, 0xeb, 0xab, 0x38, 0xe4, 0x42, 0xbb,
+ 0x80, 0x3a, 0xec, 0xf0, 0xd0, 0x0c, 0x67, 0xa0, 0x9b, 0x36, 0xb7, 0x02, 0x9b, 0x73, 0x9a, 0x96,
+ 0x2a, 0x65, 0x62, 0x16, 0x80, 0x9e, 0x17, 0x8c, 0x40, 0x86, 0xc0, 0x4e, 0xa4, 0xd9, 0x21, 0x50,
+ 0x6a, 0xd0, 0x91, 0x1f, 0x42, 0x2e, 0xf4, 0x63, 0xa4, 0x3b, 0x67, 0x99, 0x8c, 0x65, 0xc8, 0xed,
+ 0x36, 0x2b, 0xad, 0xdd, 0x66, 0x55, 0xae, 0x0a, 0x71, 0x94, 0x87, 0xa5, 0xe0, 0x21, 0x21, 0xfd,
+ 0x55, 0x1c, 0xf2, 0x2a, 0xd6, 0x8d, 0xc0, 0xc9, 0x78, 0x02, 0x19, 0xcf, 0x19, 0xb9, 0x3d, 0x7c,
+ 0x01, 0x2f, 0x83, 0x23, 0xcc, 0xb8, 0x66, 0x89, 0x8b, 0xbb, 0x66, 0x92, 0x01, 0xab, 0x2c, 0xad,
+ 0xaa, 0xd8, 0x7e, 0xe8, 0x17, 0xb5, 0x20, 0xc7, 0xb3, 0x0f, 0x17, 0xf2, 0x8d, 0xb2, 0x0c, 0xa4,
+ 0x66, 0x4b, 0x7f, 0x1a, 0x87, 0x22, 0x0f, 0x56, 0x03, 0x19, 0xd3, 0xcb, 0x3a, 0xfe, 0x1a, 0x96,
+ 0xf5, 0x99, 0x7b, 0x2b, 0x71, 0xd6, 0xde, 0x92, 0xfe, 0x35, 0x03, 0xab, 0x5d, 0xec, 0xf9, 0x1d,
+ 0x9a, 0x31, 0x09, 0xba, 0x76, 0xb6, 0x3d, 0x40, 0x2a, 0x64, 0xf0, 0x31, 0x4d, 0xbf, 0x26, 0xe6,
+ 0xce, 0xe1, 0x9d, 0x10, 0x50, 0x92, 0x09, 0x84, 0xca, 0x91, 0xc4, 0xff, 0x4c, 0x41, 0x9a, 0x52,
+ 0xd0, 0x31, 0xac, 0xbc, 0xd4, 0x7d, 0xec, 0x0e, 0x74, 0xf7, 0x48, 0xa3, 0x6f, 0xb9, 0x62, 0x9e,
+ 0x9e, 0x5f, 0x4c, 0xa9, 0x6c, 0x1c, 0xeb, 0x76, 0x0f, 0x3f, 0x0b, 0x80, 0x77, 0x62, 0x6a, 0x31,
+ 0x94, 0xc2, 0xe4, 0x7e, 0x15, 0x87, 0x4b, 0x3c, 0xe0, 0x21, 0x07, 0x03, 0xdd, 0x7b, 0x4c, 0x3c,
+ 0x33, 0x37, 0xed, 0x8b, 0x8b, 0x6f, 0x87, 0xf0, 0x64, 0x8f, 0xee, 0xc4, 0xd4, 0xb5, 0xe1, 0x14,
+ 0x85, 0x75, 0x64, 0x00, 0xcb, 0x81, 0xc1, 0x60, 0xf2, 0xd9, 0xf1, 0x54, 0xbb, 0x90, 0x7c, 0x43,
+ 0xe6, 0x81, 0xe7, 0x4e, 0x4c, 0x2d, 0x70, 0x78, 0xfa, 0x4e, 0xbc, 0x0f, 0xc2, 0xac, 0x76, 0xd0,
+ 0x5b, 0xb0, 0x6c, 0xe3, 0x97, 0x5a, 0xa8, 0x21, 0x3a, 0x03, 0x49, 0xb5, 0x60, 0xe3, 0x97, 0x61,
+ 0x23, 0xb1, 0x02, 0x97, 0x4e, 0x1d, 0x17, 0xfa, 0x1e, 0x08, 0x3a, 0x7b, 0xa1, 0x19, 0x23, 0x97,
+ 0x79, 0x8f, 0x0c, 0x60, 0x85, 0xd3, 0xab, 0x9c, 0x2c, 0xba, 0x90, 0x8f, 0xf4, 0x0d, 0xf5, 0x20,
+ 0x1b, 0x04, 0xc8, 0xfc, 0x46, 0xf0, 0xf1, 0xb9, 0x46, 0x4d, 0xba, 0xe1, 0xf9, 0xfa, 0x60, 0x88,
+ 0x03, 0x6c, 0x35, 0x04, 0xae, 0x2c, 0x41, 0x9a, 0xea, 0x55, 0xfc, 0x11, 0xa0, 0x93, 0x0d, 0xd1,
+ 0x3b, 0xb0, 0x82, 0x6d, 0xb2, 0xd4, 0xc3, 0x88, 0x97, 0x76, 0xbe, 0xa0, 0x16, 0x39, 0x39, 0x68,
+ 0xf8, 0x06, 0xe4, 0xfc, 0x80, 0x9d, 0xae, 0x91, 0xa4, 0x3a, 0x21, 0x48, 0xff, 0x9d, 0x84, 0xd5,
+ 0x67, 0xae, 0xe9, 0xe3, 0x9a, 0x69, 0x61, 0x2f, 0xd8, 0x55, 0x35, 0x48, 0x79, 0xa6, 0x7d, 0x74,
+ 0x91, 0x58, 0x8b, 0xf0, 0xa3, 0x1f, 0xc1, 0x0a, 0x89, 0xd2, 0x75, 0x5f, 0xeb, 0xf3, 0x97, 0x17,
+ 0x38, 0x14, 0x8b, 0x0c, 0x2a, 0xa0, 0x11, 0x0d, 0x30, 0xa3, 0x85, 0x0d, 0x8d, 0x26, 0xdc, 0x3c,
+ 0xba, 0x04, 0xb3, 0x6a, 0x31, 0x20, 0xd3, 0x81, 0x79, 0xe8, 0x63, 0x10, 0xf9, 0xdd, 0xb8, 0x41,
+ 0xbc, 0xce, 0x81, 0x69, 0x63, 0x43, 0xf3, 0x0e, 0x75, 0xd7, 0x30, 0xed, 0x03, 0xea, 0xfb, 0x64,
+ 0xd5, 0x0d, 0xd6, 0xa2, 0x1a, 0x36, 0xe8, 0xf0, 0xf7, 0x08, 0x4f, 0x47, 0x78, 0x2c, 0x3a, 0xaa,
+ 0xce, 0x73, 0x05, 0x36, 0xab, 0xd6, 0x57, 0x85, 0x79, 0xff, 0xaf, 0xb1, 0x89, 0xf4, 0x73, 0x48,
+ 0x53, 0xb3, 0xfa, 0x7a, 0xae, 0x69, 0x4a, 0xb0, 0x16, 0x5e, 0x55, 0x85, 0x96, 0x3c, 0xb8, 0xac,
+ 0x59, 0x0d, 0x5f, 0x71, 0x43, 0xee, 0x49, 0xff, 0x9e, 0x82, 0x62, 0x90, 0x85, 0x61, 0xf7, 0x80,
+ 0xd2, 0x6f, 0x53, 0xfc, 0xf8, 0xbe, 0x01, 0xe9, 0xca, 0x8b, 0xae, 0xdc, 0x11, 0x62, 0xe2, 0x15,
+ 0x9a, 0x4a, 0x59, 0xa3, 0xa9, 0x14, 0x8a, 0xba, 0xb5, 0x3f, 0xf6, 0x69, 0x62, 0x0f, 0xdd, 0x82,
+ 0x3c, 0x71, 0xf1, 0x9b, 0x8f, 0xb5, 0xdd, 0x6e, 0xed, 0x81, 0x00, 0x53, 0xb9, 0x7c, 0xd6, 0x96,
+ 0x44, 0x8c, 0xf6, 0x81, 0x36, 0xf2, 0xfb, 0x0f, 0x08, 0xc7, 0x9b, 0x90, 0x78, 0xba, 0x27, 0xc4,
+ 0xc5, 0xcb, 0xb4, 0xa1, 0x10, 0x69, 0x78, 0x74, 0x4c, 0xde, 0xbf, 0x0d, 0x99, 0xbd, 0xb2, 0xaa,
+ 0x34, 0xbb, 0x42, 0x42, 0x14, 0x69, 0x9b, 0xf5, 0x48, 0x9b, 0x63, 0xdd, 0x35, 0x6d, 0x9f, 0xb7,
+ 0xab, 0xb6, 0x76, 0x2b, 0x75, 0x59, 0xc8, 0x9f, 0xd2, 0xce, 0x70, 0x46, 0x3c, 0x2b, 0xf4, 0x6e,
+ 0x24, 0x8d, 0x94, 0x9c, 0xca, 0xa4, 0xb3, 0x96, 0xd1, 0x0c, 0xd2, 0x0d, 0x48, 0x77, 0x95, 0x86,
+ 0xac, 0x0a, 0xa9, 0x53, 0xc6, 0x4c, 0x3d, 0x1e, 0x96, 0xe9, 0x5f, 0x51, 0x9a, 0x5d, 0x59, 0xdd,
+ 0x0b, 0x2b, 0x1b, 0x84, 0xf4, 0x54, 0xfa, 0x99, 0x03, 0xdb, 0x3e, 0x76, 0x8f, 0x75, 0x8b, 0xa7,
+ 0xfa, 0x59, 0xd2, 0x7a, 0xb9, 0x2e, 0x37, 0x1f, 0x77, 0x77, 0xb4, 0xb6, 0x2a, 0xd7, 0x94, 0xe7,
+ 0x42, 0x66, 0x2a, 0x4d, 0xc5, 0xf8, 0x2c, 0x6c, 0x1f, 0xf8, 0x87, 0xda, 0xd0, 0xc5, 0x7d, 0xf3,
+ 0x4b, 0xce, 0x35, 0x55, 0x47, 0x21, 0x2c, 0x9d, 0xc2, 0xc5, 0xb2, 0xe9, 0x11, 0x59, 0x1f, 0x42,
+ 0x91, 0x35, 0x0f, 0xf2, 0xb6, 0x42, 0x76, 0xea, 0xf6, 0x83, 0xb1, 0x85, 0xfb, 0x96, 0x2d, 0x49,
+ 0x9a, 0x3e, 0xbd, 0xd4, 0xe9, 0x96, 0xbb, 0xb2, 0x56, 0x21, 0xf1, 0x5a, 0x55, 0x0b, 0x95, 0x97,
+ 0x13, 0xbf, 0x47, 0xd9, 0xdf, 0x9a, 0x9a, 0x5b, 0xdd, 0xc7, 0xda, 0xbe, 0xde, 0x3b, 0xc2, 0x86,
+ 0x16, 0xd1, 0xa4, 0xf4, 0x07, 0x99, 0xc0, 0x45, 0x8a, 0x24, 0xa8, 0x5e, 0xbb, 0x8b, 0x84, 0xf6,
+ 0xa0, 0xc0, 0x52, 0xe3, 0xa4, 0x23, 0x23, 0x8f, 0x3b, 0x77, 0x77, 0xe6, 0x09, 0x9f, 0x08, 0x5b,
+ 0x87, 0x72, 0x31, 0xf7, 0x2e, 0x3f, 0x98, 0x50, 0xd0, 0xdb, 0x81, 0x45, 0x9b, 0xf8, 0x43, 0x49,
+ 0xba, 0xf9, 0x97, 0x19, 0x39, 0xf0, 0xf0, 0xab, 0xb0, 0xe4, 0xbb, 0xe6, 0xc1, 0x01, 0x76, 0x79,
+ 0xe4, 0xf6, 0xee, 0x3c, 0xc7, 0x0f, 0xe3, 0x50, 0x03, 0x56, 0x84, 0x61, 0x35, 0x74, 0xb3, 0x4c,
+ 0xc7, 0xd6, 0x08, 0x0b, 0x8d, 0xdd, 0x8a, 0x9b, 0x0f, 0xe6, 0xc0, 0x2b, 0x47, 0x78, 0x1b, 0x8e,
+ 0xc1, 0xe3, 0x78, 0x41, 0x9f, 0x21, 0x93, 0x00, 0x81, 0xa5, 0xf7, 0xa9, 0xaf, 0x42, 0x93, 0x3f,
+ 0xf3, 0x05, 0x08, 0xec, 0x76, 0x92, 0x1c, 0x7d, 0x3c, 0x40, 0x70, 0x42, 0x02, 0xda, 0x07, 0xa1,
+ 0x67, 0x39, 0xd4, 0x03, 0xda, 0xc7, 0x87, 0xfa, 0xb1, 0xe9, 0xb8, 0x34, 0x59, 0x54, 0xdc, 0xbc,
+ 0x3f, 0x4f, 0x78, 0xcc, 0x58, 0x2b, 0x9c, 0x93, 0xc1, 0xaf, 0xf4, 0xa6, 0xa9, 0xd4, 0x3f, 0xb0,
+ 0x2c, 0xba, 0x4c, 0x2d, 0xdd, 0xc7, 0x36, 0xf6, 0x3c, 0x9a, 0x5d, 0x22, 0xfe, 0x01, 0xa3, 0xd7,
+ 0x39, 0x99, 0xc4, 0xea, 0x2d, 0x9b, 0x74, 0x2c, 0x60, 0xde, 0xc8, 0xcd, 0x9d, 0x0d, 0x99, 0x66,
+ 0x64, 0x7d, 0x99, 0x41, 0x43, 0xb7, 0xe1, 0x92, 0xee, 0x79, 0xe6, 0x81, 0xed, 0x69, 0xbe, 0xa3,
+ 0x39, 0x76, 0x70, 0x91, 0xb7, 0x01, 0xf4, 0xf0, 0x42, 0xfc, 0x65, 0xd7, 0x69, 0xd9, 0x98, 0xad,
+ 0x7f, 0xe9, 0x73, 0xc8, 0x47, 0x16, 0x9b, 0xd4, 0x38, 0x2b, 0x3c, 0x5a, 0x81, 0x7c, 0xb3, 0xd5,
+ 0xa4, 0xb7, 0x44, 0x4a, 0xf3, 0xb1, 0x10, 0xa7, 0x04, 0x59, 0xae, 0x76, 0xd8, 0xc5, 0x91, 0x90,
+ 0x40, 0x08, 0x8a, 0xe5, 0xba, 0x2a, 0x97, 0xab, 0xfc, 0x2e, 0xa9, 0x2a, 0x24, 0xa5, 0x1f, 0x83,
+ 0x30, 0x3b, 0xff, 0x92, 0x72, 0x96, 0x88, 0x22, 0x40, 0x55, 0xe9, 0x6c, 0x97, 0xd5, 0x2a, 0x93,
+ 0x20, 0x40, 0x21, 0xbc, 0x8e, 0x22, 0x94, 0x04, 0x69, 0xa1, 0xca, 0xf4, 0x0a, 0x89, 0x3c, 0x27,
+ 0xa5, 0x4f, 0x61, 0x65, 0x66, 0x8e, 0xa4, 0x47, 0xaf, 0x18, 0x80, 0xdc, 0x50, 0xba, 0x5a, 0xb9,
+ 0xfe, 0xac, 0xfc, 0xa2, 0xc3, 0xf2, 0x42, 0x94, 0xa0, 0xd4, 0xb4, 0x66, 0xab, 0x29, 0x37, 0xda,
+ 0xdd, 0x17, 0x42, 0x42, 0x6a, 0xcf, 0x4e, 0xd1, 0x2b, 0x11, 0x6b, 0x8a, 0x2a, 0x4f, 0x21, 0x52,
+ 0xc2, 0x34, 0xe2, 0x3e, 0xc0, 0x64, 0x89, 0x4a, 0xdd, 0xb3, 0xd0, 0x56, 0x61, 0x59, 0x6e, 0x56,
+ 0xb5, 0x56, 0x4d, 0x0b, 0x33, 0x57, 0x08, 0x8a, 0xf5, 0x32, 0xbd, 0x21, 0x56, 0x9a, 0x5a, 0xbb,
+ 0xdc, 0x24, 0x5a, 0x26, 0xbd, 0x2e, 0xab, 0x75, 0x25, 0x4a, 0x4d, 0x4a, 0x16, 0xc0, 0x24, 0x4e,
+ 0x96, 0xbe, 0x78, 0x85, 0x86, 0xe5, 0x3d, 0xb9, 0xd9, 0xa5, 0x75, 0x6e, 0x42, 0x1c, 0xad, 0xc1,
+ 0x0a, 0xbf, 0x58, 0x21, 0x67, 0x24, 0x25, 0x26, 0xd0, 0x35, 0x78, 0xa3, 0xf3, 0xa2, 0xb9, 0xbd,
+ 0xa3, 0xb6, 0x9a, 0xf4, 0xb2, 0x65, 0xb6, 0x45, 0x52, 0xfa, 0x95, 0x00, 0x4b, 0xdc, 0x4c, 0x20,
+ 0x15, 0x72, 0x7a, 0xdf, 0xc7, 0xae, 0xa6, 0x5b, 0x16, 0x37, 0x9a, 0x77, 0xe6, 0xb7, 0x32, 0xa5,
+ 0x32, 0xe1, 0x2d, 0x5b, 0xd6, 0x4e, 0x4c, 0xcd, 0xea, 0xfc, 0x77, 0x04, 0xd3, 0x1e, 0x73, 0x17,
+ 0x66, 0x71, 0x4c, 0x7b, 0x3c, 0xc1, 0xb4, 0xc7, 0x68, 0x17, 0x80, 0x61, 0x62, 0xbd, 0x77, 0xc8,
+ 0x63, 0x90, 0xbb, 0x8b, 0x82, 0xca, 0x7a, 0xef, 0x70, 0x27, 0xa6, 0xb2, 0xde, 0x91, 0x07, 0x64,
+ 0xc1, 0x1a, 0x87, 0xb5, 0x0d, 0xcd, 0xe9, 0x07, 0xfb, 0x8d, 0x99, 0xdb, 0x8f, 0x16, 0xc6, 0xb7,
+ 0x8d, 0x56, 0x9f, 0x6d, 0xcc, 0x9d, 0x98, 0x2a, 0xe8, 0x33, 0x34, 0xe4, 0xc3, 0x25, 0x26, 0x6d,
+ 0x26, 0xb2, 0xe3, 0xa9, 0xb4, 0x47, 0x8b, 0xca, 0x3b, 0x19, 0xc1, 0xe9, 0x27, 0xc9, 0xe8, 0xeb,
+ 0x38, 0x48, 0x4c, 0xac, 0x37, 0xb6, 0x7b, 0x87, 0xae, 0x63, 0xd3, 0x0b, 0xb4, 0xd9, 0x3e, 0xb0,
+ 0x32, 0x95, 0x27, 0x8b, 0xf6, 0xa1, 0x13, 0xc1, 0x3c, 0xd1, 0x9f, 0xab, 0xfa, 0xab, 0x9b, 0xa0,
+ 0xa7, 0x90, 0xd1, 0xad, 0x97, 0xfa, 0xd8, 0xdb, 0x28, 0xcc, 0x9d, 0x9b, 0x0d, 0xc5, 0x53, 0xc6,
+ 0x9d, 0x98, 0xca, 0x21, 0x50, 0x13, 0x96, 0x0c, 0xdc, 0xd7, 0x47, 0x96, 0x4f, 0x0f, 0x89, 0xf9,
+ 0x8e, 0xff, 0x00, 0xad, 0xca, 0x38, 0x77, 0x62, 0x6a, 0x00, 0x82, 0xbe, 0x98, 0x84, 0xbe, 0x3d,
+ 0x67, 0x64, 0xfb, 0xf4, 0x58, 0xc8, 0xcf, 0x75, 0xf4, 0x04, 0xa8, 0x72, 0x90, 0x53, 0x1b, 0xd9,
+ 0x7e, 0x24, 0xd6, 0xa5, 0xcf, 0x68, 0x07, 0xd2, 0x36, 0x3e, 0xc6, 0xec, 0x14, 0xc9, 0x6f, 0xde,
+ 0x5a, 0x00, 0xb7, 0x49, 0xf8, 0x76, 0x62, 0x2a, 0x03, 0x20, 0xbb, 0xc3, 0x71, 0xd9, 0x05, 0x89,
+ 0x35, 0xa6, 0xa7, 0xc5, 0x62, 0xbb, 0xa3, 0xe5, 0xd6, 0x18, 0x2f, 0xd9, 0x1d, 0x4e, 0xf0, 0x40,
+ 0x66, 0xc7, 0xc5, 0x43, 0xac, 0xfb, 0x1b, 0xf9, 0x85, 0x67, 0x47, 0xa5, 0x8c, 0x64, 0x76, 0x18,
+ 0x84, 0xf8, 0x1c, 0xb2, 0x81, 0xb5, 0x40, 0x75, 0xc8, 0xd3, 0xe2, 0x2e, 0xda, 0x34, 0x08, 0xae,
+ 0x17, 0xf1, 0x6e, 0xa2, 0xec, 0x13, 0x64, 0x7b, 0xfc, 0x9a, 0x91, 0x5f, 0x40, 0x2e, 0x34, 0x1c,
+ 0xaf, 0x19, 0xfa, 0xef, 0xe2, 0x20, 0xcc, 0x1a, 0x0d, 0xd4, 0x82, 0x65, 0xac, 0xbb, 0xd6, 0x58,
+ 0xeb, 0x9b, 0x24, 0xac, 0x09, 0x2a, 0x0a, 0x17, 0x11, 0x52, 0xa0, 0x00, 0x35, 0xc6, 0x8f, 0x1a,
+ 0x50, 0x20, 0x4e, 0x4d, 0x88, 0x97, 0x58, 0x18, 0x2f, 0x4f, 0xf8, 0x39, 0x9c, 0xf8, 0x7b, 0xb0,
+ 0x76, 0x8a, 0xe1, 0x41, 0x87, 0xb0, 0x1e, 0xa6, 0x1a, 0xb4, 0x13, 0x65, 0xd4, 0xf7, 0xe6, 0xcc,
+ 0x12, 0x53, 0xf6, 0x49, 0xdd, 0xec, 0x9a, 0x7f, 0x82, 0xe6, 0x89, 0xd7, 0xe1, 0xea, 0x37, 0x58,
+ 0x1d, 0x31, 0x07, 0x4b, 0x7c, 0x2f, 0x8b, 0x77, 0xa0, 0x10, 0xdd, 0x80, 0xe8, 0xad, 0xd9, 0x0d,
+ 0x4d, 0xd4, 0x9b, 0x9e, 0xde, 0x95, 0xe2, 0x12, 0xa4, 0xe9, 0xee, 0x12, 0xb3, 0x90, 0x61, 0x26,
+ 0x46, 0xfc, 0x93, 0x38, 0xe4, 0xc2, 0x2d, 0x82, 0x1e, 0x41, 0x2a, 0xcc, 0x81, 0x2f, 0xa6, 0x4b,
+ 0xca, 0x47, 0xdc, 0xfa, 0x60, 0xa7, 0x2e, 0x3e, 0x1d, 0x01, 0xab, 0xd8, 0x85, 0x0c, 0xdb, 0x62,
+ 0xe8, 0x09, 0xc0, 0x64, 0x61, 0x9d, 0xa3, 0x57, 0x11, 0xee, 0x4a, 0x2e, 0x0c, 0x39, 0xa4, 0x7f,
+ 0x4e, 0x44, 0x12, 0x52, 0x93, 0x92, 0xd0, 0x0e, 0xa4, 0x0d, 0x6c, 0xe9, 0x63, 0x2e, 0xe8, 0xa3,
+ 0x73, 0x4d, 0x6e, 0xa9, 0x4a, 0x20, 0x88, 0xfd, 0xa2, 0x58, 0xe8, 0x33, 0xc8, 0xea, 0x96, 0x79,
+ 0x60, 0x6b, 0xbe, 0xc3, 0x75, 0xf2, 0x83, 0xf3, 0xe1, 0x96, 0x09, 0x4a, 0xd7, 0x21, 0x56, 0x5c,
+ 0x67, 0x3f, 0xc5, 0x77, 0x21, 0x4d, 0xa5, 0xa1, 0xeb, 0x50, 0xa0, 0xd2, 0xb4, 0x81, 0x69, 0x59,
+ 0xa6, 0xc7, 0x93, 0x80, 0x79, 0x4a, 0x6b, 0x50, 0x92, 0xf8, 0x10, 0x96, 0x38, 0x02, 0xba, 0x0c,
+ 0x99, 0x21, 0x76, 0x4d, 0x87, 0xc5, 0x66, 0x49, 0x95, 0x3f, 0x11, 0xba, 0xd3, 0xef, 0x7b, 0xd8,
+ 0xa7, 0x4e, 0x42, 0x52, 0xe5, 0x4f, 0x95, 0x4b, 0xb0, 0x76, 0xca, 0x1e, 0x90, 0xfe, 0x38, 0x01,
+ 0xb9, 0x30, 0x37, 0x83, 0xf6, 0xa0, 0xa8, 0xf7, 0x68, 0x11, 0xcb, 0x50, 0xf7, 0x7d, 0xec, 0xda,
+ 0xe7, 0xcd, 0xc8, 0x2c, 0x33, 0x98, 0x36, 0x43, 0x41, 0x4f, 0x61, 0xe9, 0xd8, 0xc4, 0x2f, 0x2f,
+ 0x76, 0x1b, 0x95, 0x21, 0x10, 0x35, 0x1b, 0x7d, 0x01, 0xab, 0x3c, 0x3c, 0x1d, 0xe8, 0xc3, 0x21,
+ 0xf1, 0x0f, 0xfa, 0x36, 0xf7, 0xb8, 0xce, 0x03, 0xcb, 0x63, 0xdd, 0x06, 0xc3, 0xaa, 0xd9, 0xd2,
+ 0x27, 0x90, 0x8f, 0x94, 0x56, 0x23, 0x01, 0x92, 0x23, 0xd7, 0xe6, 0x37, 0x02, 0xe4, 0x27, 0xda,
+ 0x80, 0xa5, 0x21, 0x4b, 0xa5, 0x51, 0xb1, 0x05, 0x35, 0x78, 0x7c, 0x92, 0xca, 0xc6, 0x85, 0x84,
+ 0xf4, 0x67, 0x71, 0x58, 0x0f, 0x12, 0x4b, 0xd1, 0xda, 0x6f, 0xe9, 0xab, 0x38, 0x14, 0xa2, 0x04,
+ 0x74, 0x03, 0x32, 0xd5, 0x16, 0xbd, 0x18, 0x8e, 0x89, 0x1b, 0x34, 0xbf, 0x80, 0x68, 0x7e, 0x01,
+ 0xdb, 0xc7, 0x5b, 0x86, 0xd3, 0x3b, 0x62, 0x29, 0x97, 0xb7, 0x61, 0x89, 0x3b, 0xc9, 0x42, 0x7c,
+ 0x2a, 0x35, 0x43, 0x9a, 0x71, 0x37, 0x89, 0xb4, 0xbb, 0x09, 0x59, 0xf9, 0x79, 0x57, 0x56, 0x9b,
+ 0xe5, 0xfa, 0x4c, 0xfa, 0x88, 0x34, 0xc4, 0x5f, 0x92, 0xa9, 0xd0, 0xad, 0xad, 0xe3, 0xdb, 0xd2,
+ 0x03, 0x58, 0xae, 0x52, 0xf8, 0x20, 0xd3, 0xfa, 0x0e, 0xac, 0xf4, 0x1c, 0xdb, 0xd7, 0x4d, 0x9b,
+ 0xc4, 0xfb, 0x03, 0xfd, 0x20, 0x28, 0x00, 0x2a, 0x86, 0x64, 0x85, 0x50, 0xa5, 0x7f, 0x8b, 0x43,
+ 0x91, 0x1b, 0xb4, 0x80, 0xb7, 0x08, 0x09, 0xc7, 0xe3, 0xcd, 0x13, 0x8e, 0x87, 0x10, 0xa4, 0x74,
+ 0xb7, 0x77, 0xc8, 0x35, 0x46, 0x7f, 0x13, 0x95, 0xf5, 0x9c, 0xc1, 0x40, 0xb7, 0x83, 0x54, 0x42,
+ 0xf0, 0x88, 0xea, 0x90, 0xc4, 0xf6, 0xf1, 0x22, 0xf5, 0xcd, 0x53, 0xd2, 0x4b, 0xb2, 0x7d, 0xcc,
+ 0xb2, 0x98, 0x04, 0x46, 0xfc, 0x10, 0xb2, 0x01, 0x61, 0xa1, 0x4a, 0xe2, 0xff, 0x89, 0xc3, 0x8a,
+ 0xcc, 0x15, 0x14, 0x8c, 0xab, 0x03, 0xd9, 0xe0, 0xb3, 0x24, 0xbe, 0x0d, 0xe6, 0xf1, 0xac, 0xca,
+ 0x43, 0xb3, 0x83, 0xdd, 0x63, 0xb3, 0x87, 0xab, 0xe1, 0x77, 0x49, 0x6a, 0x08, 0x84, 0xf6, 0x20,
+ 0x43, 0xcb, 0x76, 0x82, 0xdb, 0xa0, 0x79, 0x7c, 0xea, 0x99, 0x8e, 0xb1, 0xc2, 0x85, 0xa0, 0x54,
+ 0x9c, 0xa1, 0x89, 0x0f, 0x21, 0x1f, 0x21, 0x2f, 0x34, 0xf6, 0x5f, 0xc0, 0xca, 0xcc, 0x9e, 0x78,
+ 0x3d, 0xf9, 0xd8, 0xef, 0x42, 0x31, 0xf2, 0x2d, 0xcb, 0xe4, 0x56, 0x6d, 0x39, 0x42, 0x55, 0x0c,
+ 0x69, 0x0b, 0x0a, 0x53, 0xb2, 0xf9, 0x7e, 0x8b, 0xcf, 0xb1, 0xdf, 0xa4, 0xdf, 0xa5, 0x20, 0x1f,
+ 0xa9, 0xdd, 0x42, 0x0a, 0xa4, 0x4d, 0x1f, 0x87, 0x27, 0xfb, 0x9d, 0xc5, 0x4a, 0xbf, 0x4a, 0x8a,
+ 0x8f, 0x07, 0x2a, 0x43, 0x10, 0xfb, 0x00, 0x8a, 0x81, 0x6d, 0xdf, 0xec, 0x9b, 0xd8, 0x25, 0xb6,
+ 0x39, 0xfa, 0xcd, 0x03, 0xef, 0x5d, 0xde, 0x9f, 0x7c, 0xee, 0x40, 0x0e, 0xef, 0x49, 0x93, 0x89,
+ 0xc5, 0x98, 0xf0, 0xed, 0xba, 0x76, 0x30, 0x2f, 0xc9, 0x70, 0x5e, 0xc4, 0x5f, 0x27, 0x20, 0x45,
+ 0xe4, 0x22, 0x05, 0x12, 0x1c, 0x78, 0xbe, 0x6f, 0x07, 0xa6, 0x3a, 0x1e, 0xf6, 0x54, 0x4d, 0x98,
+ 0x64, 0x4f, 0xb1, 0x5a, 0x98, 0xc4, 0xdc, 0x59, 0xb4, 0x28, 0xd8, 0x4c, 0x35, 0x0c, 0x7a, 0x37,
+ 0x58, 0x39, 0xcc, 0xc6, 0xae, 0x97, 0xd8, 0x07, 0x78, 0xa5, 0xe0, 0x03, 0xbc, 0x52, 0xd9, 0x0e,
+ 0x3e, 0xab, 0x41, 0xf7, 0x20, 0xef, 0x1d, 0x3a, 0xae, 0xcf, 0x32, 0xaa, 0x3c, 0x4e, 0x3d, 0x9d,
+ 0x03, 0x68, 0x43, 0x5a, 0x57, 0x41, 0x16, 0xa7, 0xa5, 0xef, 0x63, 0x8b, 0x7f, 0xc1, 0xc1, 0x1e,
+ 0xd0, 0x15, 0xc8, 0x5a, 0xa6, 0x7d, 0xa4, 0x8d, 0x5c, 0x8b, 0x46, 0x7f, 0x39, 0x75, 0x89, 0x3c,
+ 0xef, 0xba, 0x96, 0xf8, 0x0b, 0x5e, 0xa1, 0x33, 0x7a, 0x45, 0x85, 0x0e, 0x4b, 0xcd, 0xb3, 0xbb,
+ 0x76, 0xa5, 0xd9, 0x95, 0x1f, 0xcb, 0xaa, 0x90, 0x40, 0x39, 0x48, 0xd7, 0xea, 0xad, 0x72, 0x57,
+ 0x48, 0xb2, 0x3b, 0xf8, 0x56, 0x5d, 0x2e, 0x37, 0x85, 0x14, 0x5a, 0x86, 0x5c, 0xf8, 0x75, 0x9e,
+ 0x90, 0x46, 0x05, 0xc8, 0x56, 0x77, 0xd5, 0x32, 0x2d, 0x9f, 0xcd, 0xa0, 0x22, 0xc0, 0x93, 0xf2,
+ 0x5e, 0x59, 0xdb, 0xae, 0x97, 0x3b, 0x1d, 0x61, 0x49, 0xfa, 0xa7, 0x2c, 0x5c, 0x6a, 0x60, 0xcf,
+ 0xd3, 0x0f, 0xf0, 0x33, 0xd3, 0x3f, 0x8c, 0x54, 0xf3, 0xbe, 0xe6, 0x0f, 0x6e, 0x7e, 0x08, 0x69,
+ 0x9a, 0x83, 0x5d, 0xf4, 0x0b, 0x24, 0xe2, 0xba, 0x50, 0x46, 0xf4, 0x39, 0xb1, 0xec, 0xbc, 0xdc,
+ 0x39, 0xb2, 0x89, 0xe6, 0x0b, 0x96, 0xa6, 0x2f, 0xe0, 0x77, 0x62, 0x2a, 0xaf, 0x05, 0x0a, 0xaf,
+ 0xe4, 0x7f, 0x02, 0xab, 0x9e, 0x71, 0x14, 0x5e, 0xab, 0x45, 0xcb, 0x78, 0xce, 0x71, 0x16, 0xef,
+ 0xc4, 0xd4, 0x15, 0x6f, 0xc6, 0x14, 0x3d, 0x83, 0xe2, 0x50, 0x77, 0x35, 0xc3, 0x09, 0xbb, 0x9f,
+ 0x99, 0xdb, 0x28, 0x45, 0x0b, 0x03, 0x49, 0x74, 0x3b, 0x8c, 0x56, 0x72, 0xb6, 0x00, 0x86, 0xe1,
+ 0xde, 0xe4, 0x01, 0xf9, 0x62, 0x9f, 0xce, 0xed, 0xc4, 0xd4, 0x08, 0x04, 0x52, 0x21, 0x1f, 0xf9,
+ 0xdc, 0x91, 0x07, 0xe3, 0x0b, 0x7e, 0x1c, 0xb7, 0x13, 0x53, 0xa3, 0x20, 0xa8, 0x03, 0x05, 0x17,
+ 0xeb, 0x46, 0x38, 0xf6, 0xdc, 0xdc, 0xa0, 0x91, 0x7a, 0x12, 0x02, 0xea, 0x46, 0xca, 0x4b, 0x1a,
+ 0x00, 0x93, 0xab, 0x44, 0x1e, 0x3a, 0x2f, 0x74, 0x87, 0x47, 0xa2, 0xf0, 0xf0, 0xce, 0x10, 0xf5,
+ 0x61, 0x2d, 0xf2, 0xe1, 0x49, 0xd8, 0xd5, 0xc2, 0x82, 0x1f, 0xe9, 0x45, 0xaa, 0x49, 0x76, 0x62,
+ 0x2a, 0x77, 0xf1, 0xa2, 0x25, 0x26, 0x18, 0xd0, 0xc9, 0x92, 0xe0, 0x8d, 0xe5, 0xf3, 0x7f, 0x0b,
+ 0x38, 0x11, 0x13, 0xbd, 0xa6, 0xd9, 0x83, 0xe5, 0xe9, 0xe5, 0x5c, 0x3c, 0xd7, 0x21, 0x48, 0xd6,
+ 0x5b, 0x3f, 0xf2, 0x5c, 0xc9, 0x40, 0xca, 0x75, 0x1c, 0x5f, 0xfa, 0x55, 0x06, 0x2e, 0xcb, 0x5f,
+ 0xe2, 0xde, 0x88, 0xd6, 0x9c, 0x76, 0x7c, 0xfd, 0x20, 0xdc, 0x4d, 0x6d, 0xc8, 0x47, 0xce, 0x46,
+ 0x6e, 0x3d, 0x16, 0xfd, 0x14, 0x30, 0x0a, 0x41, 0x0c, 0x2b, 0x9b, 0x65, 0x7e, 0xea, 0x9b, 0x7c,
+ 0xc6, 0x4e, 0xa9, 0x16, 0x96, 0xe7, 0xf2, 0x44, 0x4e, 0xeb, 0xf7, 0x64, 0x61, 0x28, 0xc6, 0x54,
+ 0xcd, 0xf0, 0x9b, 0x53, 0x1f, 0x2d, 0xa7, 0xe8, 0x45, 0x6c, 0xf4, 0xab, 0xe3, 0x8d, 0xc9, 0xf7,
+ 0x6d, 0x69, 0xfa, 0x32, 0xfc, 0x46, 0x6d, 0xda, 0x8c, 0x66, 0x2e, 0x6a, 0x46, 0xfb, 0x90, 0x1f,
+ 0x79, 0xd8, 0xa5, 0x17, 0x65, 0xd8, 0xdb, 0x58, 0xba, 0xe8, 0x80, 0x77, 0x3d, 0xec, 0xd2, 0x9a,
+ 0x35, 0x32, 0xe0, 0x51, 0xf0, 0xe0, 0xa1, 0x17, 0x90, 0xa1, 0x17, 0xa5, 0xde, 0x46, 0x96, 0x8a,
+ 0x28, 0x9f, 0x5f, 0x04, 0x2d, 0x6d, 0x53, 0x0c, 0x95, 0x03, 0x8a, 0x2d, 0xc8, 0x47, 0xd4, 0x3c,
+ 0x8f, 0x43, 0xf2, 0x1d, 0x00, 0xcb, 0xe9, 0xe9, 0x16, 0xab, 0xe7, 0x67, 0x0b, 0x20, 0x47, 0x29,
+ 0x4d, 0x7d, 0x80, 0x09, 0x60, 0x64, 0x18, 0xaf, 0x01, 0xf0, 0x29, 0x2c, 0xf1, 0x4e, 0x5f, 0x1c,
+ 0x6c, 0xeb, 0x13, 0xc8, 0xd2, 0x7f, 0x13, 0x20, 0xfe, 0xdf, 0xf5, 0x13, 0xfe, 0x03, 0x39, 0xf3,
+ 0xa9, 0xe7, 0xd0, 0x1a, 0xb2, 0xef, 0xd5, 0x7f, 0xfb, 0xe7, 0x7f, 0xfd, 0x9c, 0x79, 0x08, 0x84,
+ 0x6b, 0xd7, 0xb5, 0xb7, 0x14, 0x58, 0xa6, 0x00, 0x3d, 0xfe, 0xd9, 0xff, 0x3c, 0x28, 0xff, 0x12,
+ 0xa0, 0x14, 0xf6, 0x23, 0x7f, 0x1f, 0x50, 0xf9, 0x08, 0xbe, 0xf9, 0x2f, 0x0c, 0x2a, 0x39, 0x95,
+ 0x56, 0x6e, 0x94, 0x87, 0xe6, 0x67, 0xf9, 0x80, 0xae, 0x1d, 0xdf, 0xde, 0xcf, 0x50, 0x71, 0x77,
+ 0xfe, 0x2f, 0x00, 0x00, 0xff, 0xff, 0xa0, 0xb1, 0x4c, 0x75, 0x1d, 0x41, 0x00, 0x00,
}
diff --git a/sdks/go/pkg/beam/options/jobopts/options.go b/sdks/go/pkg/beam/options/jobopts/options.go
index 86beb94..f7c090d 100644
--- a/sdks/go/pkg/beam/options/jobopts/options.go
+++ b/sdks/go/pkg/beam/options/jobopts/options.go
@@ -61,6 +61,10 @@
// Async determines whether to wait for job completion.
Async = flag.Bool("async", false, "Do not wait for job completion.")
+
+ // Strict mode applies additional validation to user pipelines before
+ // executing them and fails early if the pipelines don't pass.
+ Strict = flag.Bool("beam_strict", false, "Apply additional validation to pipelines.")
)
// GetEndpoint returns the endpoint, if non empty and exits otherwise. Runners
@@ -104,7 +108,7 @@
// Convenience function.
func GetEnvironmentConfig(ctx context.Context) string {
if *EnvironmentConfig == "" {
- *EnvironmentConfig = os.ExpandEnv("$USER-docker-apache.bintray.io/beam/go:latest")
+ *EnvironmentConfig = os.ExpandEnv("apachebeam/go_sdk:latest")
log.Infof(ctx, "No environment config specified. Using default config: '%v'", *EnvironmentConfig)
}
return *EnvironmentConfig
diff --git a/sdks/go/pkg/beam/pardo.go b/sdks/go/pkg/beam/pardo.go
index 9c23b91..41283f7 100644
--- a/sdks/go/pkg/beam/pardo.go
+++ b/sdks/go/pkg/beam/pardo.go
@@ -252,7 +252,7 @@
// Beam makes heavy use of this modular, composable style, trusting to the
// runner to "flatten out" all the compositions into highly optimized stages.
//
-// See https://beam.apache.org/documentation/programming-guide/#transforms-pardo"
+// See https://beam.apache.org/documentation/programming-guide/#pardo
// for the web documentation for ParDo
func ParDo(s Scope, dofn interface{}, col PCollection, opts ...Option) PCollection {
ret := MustN(TryParDo(s, dofn, col, opts...))
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflow.go b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
index 2600188..7cdaa09 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflow.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflow.go
@@ -49,17 +49,20 @@
stagingLocation = flag.String("staging_location", "", "GCS staging location (required).")
image = flag.String("worker_harness_container_image", "", "Worker harness container image (required).")
labels = flag.String("labels", "", "JSON-formatted map[string]string of job labels (optional).")
+ serviceAccountEmail = flag.String("service_account_email", "", "Service account email (optional).")
numWorkers = flag.Int64("num_workers", 0, "Number of workers (optional).")
maxNumWorkers = flag.Int64("max_num_workers", 0, "Maximum number of workers during scaling (optional).")
autoscalingAlgorithm = flag.String("autoscaling_algorithm", "", "Autoscaling mode to use (optional).")
zone = flag.String("zone", "", "GCP zone (optional)")
- region = flag.String("region", "us-central1", "GCP Region (optional)")
+ region = flag.String("region", "", "GCP Region (optional but encouraged)")
network = flag.String("network", "", "GCP network (optional)")
+ subnetwork = flag.String("subnetwork", "", "GCP subnetwork (optional)")
tempLocation = flag.String("temp_location", "", "Temp location (optional)")
machineType = flag.String("worker_machine_type", "", "GCE machine type (optional)")
minCPUPlatform = flag.String("min_cpu_platform", "", "GCE minimum cpu platform (optional)")
workerJar = flag.String("dataflow_worker_jar", "", "Dataflow worker jar (optional)")
+ executeAsync = flag.Bool("execute_async", false, "Asynchronous execution. Submit the job and return immediately.")
dryRun = flag.Bool("dry_run", false, "Dry run. Just print the job, but don't submit it.")
teardownPolicy = flag.String("teardown_policy", "", "Job teardown policy (internal only).")
@@ -89,6 +92,12 @@
if *stagingLocation == "" {
return errors.New("no GCS staging location specified. Use --staging_location=gs://<bucket>/<path>")
}
+ if *region == "" {
+ *region = "us-central1"
+ log.Warn(ctx, "--region not set; will default to us-central1. Future releases of Beam will "+
+ "require the user to set the region explicitly. "+
+ "https://cloud.google.com/compute/docs/regions-zones/regions-zones")
+ }
if *image == "" {
*image = getContainerImage(ctx)
}
@@ -126,22 +135,24 @@
}
opts := &dataflowlib.JobOptions{
- Name: jobopts.GetJobName(),
- Experiments: experiments,
- Options: beam.PipelineOptions.Export(),
- Project: project,
- Region: *region,
- Zone: *zone,
- Network: *network,
- NumWorkers: *numWorkers,
- MaxNumWorkers: *maxNumWorkers,
- Algorithm: *autoscalingAlgorithm,
- MachineType: *machineType,
- Labels: jobLabels,
- TempLocation: *tempLocation,
- Worker: *jobopts.WorkerBinary,
- WorkerJar: *workerJar,
- TeardownPolicy: *teardownPolicy,
+ Name: jobopts.GetJobName(),
+ Experiments: experiments,
+ Options: beam.PipelineOptions.Export(),
+ Project: project,
+ Region: *region,
+ Zone: *zone,
+ Network: *network,
+ Subnetwork: *subnetwork,
+ NumWorkers: *numWorkers,
+ MaxNumWorkers: *maxNumWorkers,
+ Algorithm: *autoscalingAlgorithm,
+ MachineType: *machineType,
+ Labels: jobLabels,
+ ServiceAccountEmail: *serviceAccountEmail,
+ TempLocation: *tempLocation,
+ Worker: *jobopts.WorkerBinary,
+ WorkerJar: *workerJar,
+ TeardownPolicy: *teardownPolicy,
}
if opts.TempLocation == "" {
opts.TempLocation = gcsx.Join(*stagingLocation, "tmp")
@@ -177,7 +188,7 @@
return nil
}
- _, err = dataflowlib.Execute(ctx, model, opts, workerURL, jarURL, modelURL, *endpoint, false)
+ _, err = dataflowlib.Execute(ctx, model, opts, workerURL, jarURL, modelURL, *endpoint, *executeAsync)
return err
}
func gcsRecorderHook(opts []string) perf.CaptureHook {
diff --git a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
index 3979d6e..ef24348 100644
--- a/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
+++ b/sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
@@ -41,13 +41,15 @@
// Pipeline options
Options runtime.RawOptions
- Project string
- Region string
- Zone string
- Network string
- NumWorkers int64
- MachineType string
- Labels map[string]string
+ Project string
+ Region string
+ Zone string
+ Network string
+ Subnetwork string
+ NumWorkers int64
+ MachineType string
+ Labels map[string]string
+ ServiceAccountEmail string
// Autoscaling settings
Algorithm string
@@ -108,6 +110,7 @@
Name: opts.Name,
Type: jobType,
Environment: &df.Environment{
+ ServiceAccountEmail: opts.ServiceAccountEmail,
UserAgent: newMsg(userAgent{
Name: "Apache Beam SDK for Go",
Version: "0.5.0",
@@ -135,6 +138,7 @@
NumWorkers: 1,
MachineType: opts.MachineType,
Network: opts.Network,
+ Subnetwork: opts.Subnetwork,
Zone: opts.Zone,
}},
TempStoragePrefix: opts.TempLocation,
@@ -241,6 +245,7 @@
addIfNonEmpty("region", opts.Region)
addIfNonEmpty("zone", opts.Zone)
addIfNonEmpty("network", opts.Network)
+ addIfNonEmpty("subnetwork", opts.Subnetwork)
addIfNonEmpty("machine_type", opts.MachineType)
addIfNonEmpty("container_images", strings.Join(images, ","))
addIfNonEmpty("temp_location", opts.TempLocation)
diff --git a/sdks/go/pkg/beam/runners/direct/buffer.go b/sdks/go/pkg/beam/runners/direct/buffer.go
index d652b81..05c1526 100644
--- a/sdks/go/pkg/beam/runners/direct/buffer.go
+++ b/sdks/go/pkg/beam/runners/direct/buffer.go
@@ -64,7 +64,7 @@
return nil
}
-func (n *buffer) NewIterable(ctx context.Context, reader exec.SideInputReader, w typex.Window) (exec.ReStream, error) {
+func (n *buffer) NewIterable(ctx context.Context, reader exec.StateReader, w typex.Window) (exec.ReStream, error) {
if !n.done {
panic(fmt.Sprintf("buffer[%v] incomplete: %v", n.uid, len(n.buf)))
}
@@ -86,11 +86,16 @@
instID string
mgr exec.DataContext
- buf []exec.FullValue
+ buf []bufElement
ready int // guards ready
done bool // FinishBundle called for main input?
}
+type bufElement struct {
+ elm exec.FullValue
+ values []exec.ReStream
+}
+
func (w *wait) ID() exec.UnitID {
return w.UID
}
@@ -112,8 +117,8 @@
if err := w.next.StartBundle(ctx, w.instID, w.mgr); err != nil {
return err
}
- for _, elm := range w.buf {
- if err := w.next.ProcessElement(ctx, &elm); err != nil {
+ for _, element := range w.buf {
+ if err := w.next.ProcessElement(ctx, &element.elm, element.values...); err != nil {
return err
}
}
@@ -139,12 +144,12 @@
func (w *wait) ProcessElement(ctx context.Context, elm *exec.FullValue, values ...exec.ReStream) error {
if w.ready < w.need {
// log.Printf("buffer[%v]: %v", w.UID, elm)
- w.buf = append(w.buf, *elm)
+ w.buf = append(w.buf, bufElement{elm: *elm, values: values})
return nil
}
// log.Printf("NOT buffer[%v]: %v", w.UID, elm)
- return w.next.ProcessElement(ctx, elm)
+ return w.next.ProcessElement(ctx, elm, values...)
}
func (w *wait) FinishBundle(ctx context.Context) error {
diff --git a/sdks/go/pkg/beam/runners/direct/direct.go b/sdks/go/pkg/beam/runners/direct/direct.go
index bd1d324..55e4836 100644
--- a/sdks/go/pkg/beam/runners/direct/direct.go
+++ b/sdks/go/pkg/beam/runners/direct/direct.go
@@ -28,6 +28,8 @@
"github.com/apache/beam/sdks/go/pkg/beam/core/typex"
"github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
"github.com/apache/beam/sdks/go/pkg/beam/log"
+ "github.com/apache/beam/sdks/go/pkg/beam/options/jobopts"
+ "github.com/apache/beam/sdks/go/pkg/beam/runners/vet"
)
func init() {
@@ -45,6 +47,14 @@
log.Info(ctx, "Pipeline:")
log.Info(ctx, p)
+ if *jobopts.Strict {
+ log.Info(ctx, "Strict mode enabled, applying additional validation.")
+ if err := vet.Execute(ctx, p); err != nil {
+ return errors.Wrap(err, "strictness check failed")
+ }
+ log.Info(ctx, "Strict mode validation passed.")
+ }
+
edges, _, err := p.Build()
if err != nil {
return errors.Wrap(err, "invalid pipeline")
diff --git a/sdks/go/pkg/beam/runners/session/session.go b/sdks/go/pkg/beam/runners/session/session.go
index cf3b8a2..8725bc1 100644
--- a/sdks/go/pkg/beam/runners/session/session.go
+++ b/sdks/go/pkg/beam/runners/session/session.go
@@ -180,11 +180,11 @@
for _, desc := range rr.GetProcessBundleDescriptor() {
for beamPort, t := range desc.GetTransforms() {
s := t.GetSpec()
- if s.GetUrn() == "urn:org.apache.beam:source:runner:0.1" {
+ if s.GetUrn() == "beam:source:runner:0.1" {
tcpPort := extractPortSpec(s)
c.establishDataChannel(beamPort, tcpPort)
}
- if s.GetUrn() == "urn:org.apache.beam:sink:runner:0.1" {
+ if s.GetUrn() == "beam:sink:runner:0.1" {
tcpPort := extractPortSpec(s)
c.establishDataChannel(beamPort, tcpPort)
}
diff --git a/sdks/go/pkg/beam/runners/spark/spark.go b/sdks/go/pkg/beam/runners/spark/spark.go
new file mode 100644
index 0000000..c216c59
--- /dev/null
+++ b/sdks/go/pkg/beam/runners/spark/spark.go
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package spark contains the Spark runner.
+package spark
+
+import (
+ "context"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/runners/universal"
+)
+
+func init() {
+ beam.RegisterRunner("spark", Execute)
+}
+
+// Execute runs the given pipeline on Spark. Convenience wrapper over the
+// universal runner.
+func Execute(ctx context.Context, p *beam.Pipeline) error {
+ return universal.Execute(ctx, p)
+}
diff --git a/sdks/go/pkg/beam/runners/universal/universal.go b/sdks/go/pkg/beam/runners/universal/universal.go
index 0e22db7..51f74da 100644
--- a/sdks/go/pkg/beam/runners/universal/universal.go
+++ b/sdks/go/pkg/beam/runners/universal/universal.go
@@ -30,6 +30,7 @@
pb "github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
"github.com/apache/beam/sdks/go/pkg/beam/options/jobopts"
"github.com/apache/beam/sdks/go/pkg/beam/runners/universal/runnerlib"
+ "github.com/apache/beam/sdks/go/pkg/beam/runners/vet"
"github.com/golang/protobuf/proto"
)
@@ -40,6 +41,18 @@
// Execute executes the pipeline on a universal beam runner.
func Execute(ctx context.Context, p *beam.Pipeline) error {
+ if !beam.Initialized() {
+ panic(fmt.Sprint("Beam has not been initialized. Call beam.Init() before pipeline construction."))
+ }
+
+ if *jobopts.Strict {
+ log.Info(ctx, "Strict mode enabled, applying additional validation.")
+ if err := vet.Execute(ctx, p); err != nil {
+ return errors.Wrap(err, "strictness check failed")
+ }
+ log.Info(ctx, "Strict mode validation passed.")
+ }
+
endpoint, err := jobopts.GetEndpoint()
if err != nil {
return err
diff --git a/sdks/go/pkg/beam/runners/vet/testpipeline/functions.go b/sdks/go/pkg/beam/runners/vet/testpipeline/functions.go
new file mode 100644
index 0000000..ce3d3cf
--- /dev/null
+++ b/sdks/go/pkg/beam/runners/vet/testpipeline/functions.go
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package testpipeline
+
+import "github.com/apache/beam/sdks/go/pkg/beam"
+
+//go:generate go install github.com/apache/beam/sdks/go/cmd/starcgen
+//go:generate starcgen --package=testpipeline --identifiers=VFn,KvFn,KvEmitFn,SCombine
+//go:generate go fmt
+
+// VFn is a do nothing example function with a k and v.
+func VFn(v int) (string, int) {
+ return "key", v
+}
+
+// KvFn is a do nothing example function with a k and v.
+func KvFn(k string, v int) (string, int) {
+ return k, v
+}
+
+// KvEmitFn is a do nothing example function with a k and v that uses an emit
+// instead of a return.
+func KvEmitFn(k string, v int, emit func(string, int)) {
+ emit(k, v)
+}
+
+// SCombine is a Do Nothing structural doFn to ensure that generating things for
+// combinefn structs works.
+type SCombine struct{}
+
+// MergeAccumulators lifecycle method.
+func (s *SCombine) MergeAccumulators(a, b int) int { return a + b }
+
+// otherMethod should never have a shim generated for it.
+// Unfortunately, outside of a manual inspection, or parsing of the
+// generated file, this is difficult to test.
+func (s *SCombine) otherMethod(v beam.V) beam.V {
+ return v
+}
diff --git a/sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.go b/sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.go
new file mode 100644
index 0000000..3ce3c17
--- /dev/null
+++ b/sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.go
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package testpipeline exports small test pipelines for testing the vet
+// runner. Shims must be generated for this package in order for tests to run
+// correctly. These shims should be regenerated if changes are made to this
+// package or to the shim generator.
+package testpipeline
+
+import (
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+// Performant constructs a performant pipeline.
+func Performant(s beam.Scope) {
+ vs := beam.Create(s, 1, 2, 3)
+ kvs := beam.ParDo(s, VFn, vs)
+ kv1 := beam.ParDo(s, KvFn, kvs)
+ kv2 := beam.ParDo(s, KvEmitFn, kvs)
+ flatKvs := beam.Flatten(s, kv1, kv2)
+
+ beam.CombinePerKey(s, &SCombine{}, flatKvs)
+}
+
+// FunctionReg constructs a sub optimal pipeline that needs function registration.
+func FunctionReg(s beam.Scope) {
+ vs := beam.Create(s, float64(1), float64(2), float64(3))
+ kvs := beam.ParDo(s, VFloat64Fn, vs)
+ beam.CombinePerKey(s, &SCombine{}, kvs)
+}
+
+// ShimNeeded constructs a sub optimal pipeline that needs a function shim registration.
+func ShimNeeded(s beam.Scope) {
+ vs := beam.Create(s, float64(1), float64(2), float64(3))
+ kvs := beam.ParDo(s, vFloat64Fn, vs)
+ beam.CombinePerKey(s, &SCombine{}, kvs)
+}
+
+// TypeReg constructs a sub optimal pipeline that needs type registration.
+func TypeReg(s beam.Scope) {
+ vs := beam.Create(s, 1, 2, 3)
+ kvs := beam.ParDo(s, VFn, vs)
+
+ c := beam.CombinePerKey(s, &SCombine{}, kvs)
+ beam.ParDo(s, toFooFn, c)
+}
+
+// VFloat64Fn is an unregistered function without type shims.
+func VFloat64Fn(v float64) (string, int) {
+ return "key", 0
+}
+
+func init() {
+ beam.RegisterFunction(vFloat64Fn)
+}
+
+// vFloat64Fn is a registered function without type shims.
+func vFloat64Fn(v float64) (string, int) {
+ return "key", 0
+}
+
+// foo is an unregistered, unexported user type.
+type foo struct {
+ K string
+ V int
+}
+
+// toFooFn is an unregistered function, that uses an unregistered user type,
+// without a shim.
+func toFooFn(k string, v int) foo {
+ return foo{K: k, V: v}
+}
diff --git a/sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.shims.go b/sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.shims.go
new file mode 100644
index 0000000..9d32d1f
--- /dev/null
+++ b/sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.shims.go
@@ -0,0 +1,190 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Code generated by starcgen. DO NOT EDIT.
+// File: testpipeline.shims.go
+
+package testpipeline
+
+import (
+ "context"
+ "reflect"
+
+ // Library imports
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
+)
+
+func init() {
+ runtime.RegisterFunction(KvEmitFn)
+ runtime.RegisterFunction(KvFn)
+ runtime.RegisterFunction(VFn)
+ runtime.RegisterType(reflect.TypeOf((*SCombine)(nil)).Elem())
+ reflectx.RegisterStructWrapper(reflect.TypeOf((*SCombine)(nil)).Elem(), wrapMakerSCombine)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int, int) int)(nil)).Elem(), funcMakerIntIntГInt)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int) (string, int))(nil)).Elem(), funcMakerIntГStringInt)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(string, int, func(string, int)))(nil)).Elem(), funcMakerStringIntEmitStringIntГ)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(string, int) (string, int))(nil)).Elem(), funcMakerStringIntГStringInt)
+ exec.RegisterEmitter(reflect.TypeOf((*func(string, int))(nil)).Elem(), emitMakerStringInt)
+}
+
+func wrapMakerSCombine(fn interface{}) map[string]reflectx.Func {
+ dfn := fn.(*SCombine)
+ return map[string]reflectx.Func{
+ "MergeAccumulators": reflectx.MakeFunc(func(a0 int, a1 int) int { return dfn.MergeAccumulators(a0, a1) }),
+ }
+}
+
+type callerIntIntГInt struct {
+ fn func(int, int) int
+}
+
+func funcMakerIntIntГInt(fn interface{}) reflectx.Func {
+ f := fn.(func(int, int) int)
+ return &callerIntIntГInt{fn: f}
+}
+
+func (c *callerIntIntГInt) Name() string {
+ return reflectx.FunctionName(c.fn)
+}
+
+func (c *callerIntIntГInt) Type() reflect.Type {
+ return reflect.TypeOf(c.fn)
+}
+
+func (c *callerIntIntГInt) Call(args []interface{}) []interface{} {
+ out0 := c.fn(args[0].(int), args[1].(int))
+ return []interface{}{out0}
+}
+
+func (c *callerIntIntГInt) Call2x1(arg0, arg1 interface{}) interface{} {
+ return c.fn(arg0.(int), arg1.(int))
+}
+
+type callerIntГStringInt struct {
+ fn func(int) (string, int)
+}
+
+func funcMakerIntГStringInt(fn interface{}) reflectx.Func {
+ f := fn.(func(int) (string, int))
+ return &callerIntГStringInt{fn: f}
+}
+
+func (c *callerIntГStringInt) Name() string {
+ return reflectx.FunctionName(c.fn)
+}
+
+func (c *callerIntГStringInt) Type() reflect.Type {
+ return reflect.TypeOf(c.fn)
+}
+
+func (c *callerIntГStringInt) Call(args []interface{}) []interface{} {
+ out0, out1 := c.fn(args[0].(int))
+ return []interface{}{out0, out1}
+}
+
+func (c *callerIntГStringInt) Call1x2(arg0 interface{}) (interface{}, interface{}) {
+ return c.fn(arg0.(int))
+}
+
+type callerStringIntEmitStringIntГ struct {
+ fn func(string, int, func(string, int))
+}
+
+func funcMakerStringIntEmitStringIntГ(fn interface{}) reflectx.Func {
+ f := fn.(func(string, int, func(string, int)))
+ return &callerStringIntEmitStringIntГ{fn: f}
+}
+
+func (c *callerStringIntEmitStringIntГ) Name() string {
+ return reflectx.FunctionName(c.fn)
+}
+
+func (c *callerStringIntEmitStringIntГ) Type() reflect.Type {
+ return reflect.TypeOf(c.fn)
+}
+
+func (c *callerStringIntEmitStringIntГ) Call(args []interface{}) []interface{} {
+ c.fn(args[0].(string), args[1].(int), args[2].(func(string, int)))
+ return []interface{}{}
+}
+
+func (c *callerStringIntEmitStringIntГ) Call3x0(arg0, arg1, arg2 interface{}) {
+ c.fn(arg0.(string), arg1.(int), arg2.(func(string, int)))
+}
+
+type callerStringIntГStringInt struct {
+ fn func(string, int) (string, int)
+}
+
+func funcMakerStringIntГStringInt(fn interface{}) reflectx.Func {
+ f := fn.(func(string, int) (string, int))
+ return &callerStringIntГStringInt{fn: f}
+}
+
+func (c *callerStringIntГStringInt) Name() string {
+ return reflectx.FunctionName(c.fn)
+}
+
+func (c *callerStringIntГStringInt) Type() reflect.Type {
+ return reflect.TypeOf(c.fn)
+}
+
+func (c *callerStringIntГStringInt) Call(args []interface{}) []interface{} {
+ out0, out1 := c.fn(args[0].(string), args[1].(int))
+ return []interface{}{out0, out1}
+}
+
+func (c *callerStringIntГStringInt) Call2x2(arg0, arg1 interface{}) (interface{}, interface{}) {
+ return c.fn(arg0.(string), arg1.(int))
+}
+
+type emitNative struct {
+ n exec.ElementProcessor
+ fn interface{}
+
+ ctx context.Context
+ ws []typex.Window
+ et typex.EventTime
+ value exec.FullValue
+}
+
+func (e *emitNative) Init(ctx context.Context, ws []typex.Window, et typex.EventTime) error {
+ e.ctx = ctx
+ e.ws = ws
+ e.et = et
+ return nil
+}
+
+func (e *emitNative) Value() interface{} {
+ return e.fn
+}
+
+func emitMakerStringInt(n exec.ElementProcessor) exec.ReusableEmitter {
+ ret := &emitNative{n: n}
+ ret.fn = ret.invokeStringInt
+ return ret
+}
+
+func (e *emitNative) invokeStringInt(key string, val int) {
+ e.value = exec.FullValue{Windows: e.ws, Timestamp: e.et, Elm: key, Elm2: val}
+ if err := e.n.ProcessElement(e.ctx, &e.value); err != nil {
+ panic(err)
+ }
+}
+
+// DO NOT MODIFY: GENERATED CODE
diff --git a/sdks/go/pkg/beam/runners/vet/vet.go b/sdks/go/pkg/beam/runners/vet/vet.go
new file mode 100644
index 0000000..4ce0e2f
--- /dev/null
+++ b/sdks/go/pkg/beam/runners/vet/vet.go
@@ -0,0 +1,598 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package vet is a Beam runner that "runs" a pipeline by producing
+// generated code to avoid symbol table lookups and reflection in pipeline
+// execution.
+//
+// This runner isn't necessarily intended to be run by itself. Other runners
+// can use this as a sanity check on whether a given pipeline avoids known
+// performance bottlenecks.
+//
+// TODO(BEAM-7374): Add usage documentation.
+package vet
+
+import (
+ "bytes"
+ "context"
+ "fmt"
+ "reflect"
+ "strings"
+ "unicode"
+ "unicode/utf8"
+
+ "github.com/apache/beam/sdks/go/pkg/beam/util/shimx"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/funcx"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
+ "github.com/apache/beam/sdks/go/pkg/beam/internal/errors"
+)
+
+func init() {
+ beam.RegisterRunner("vet", Execute)
+}
+
+// We want clear failures when looking up symbols so we can tell if something has been
+// registered properly or not.
+type disabledResolver bool
+
+func (p disabledResolver) Sym2Addr(name string) (uintptr, error) {
+ return 0, errors.Errorf("%v not found. Use runtime.RegisterFunction in unit tests", name)
+}
+
+// Execute evaluates the pipeline on whether it can run without reflection.
+func Execute(ctx context.Context, p *beam.Pipeline) error {
+ e, err := Evaluate(ctx, p)
+ if err != nil {
+ return errors.WithContext(err, "validating pipeline with vet runner")
+ }
+ if !e.Performant() {
+ e.summary()
+ e.Generate("main")
+ e.diag("*/\n")
+ err := errors.Errorf("pipeline is not performant, see diagnostic summary:\n%s\n%s", string(e.d.Bytes()), string(e.Bytes()))
+ err = errors.WithContext(err, "validating pipeline with vet runner")
+ return errors.SetTopLevelMsg(err, "pipeline is not performant")
+ }
+ // Pipeline nas no further tasks.
+ return nil
+}
+
+// Evaluate returns an object that can generate necessary shims and inits.
+func Evaluate(_ context.Context, p *beam.Pipeline) (*Eval, error) {
+ // Disable the resolver so we can see functions that are that are already registered.
+ r := runtime.Resolver
+ runtime.Resolver = disabledResolver(false)
+ // Reinstate the resolver when we're through.
+ defer func() { runtime.Resolver = r }()
+
+ edges, _, err := p.Build()
+ if err != nil {
+ return nil, errors.New("can't get data to generate")
+ }
+
+ e := newEval()
+
+ e.diag("/**\n")
+ e.extractFromMultiEdges(edges)
+ return e, nil
+}
+
+func newEval() *Eval {
+ return &Eval{
+ functions: make(map[string]*funcx.Fn),
+ types: make(map[string]reflect.Type),
+ funcs: make(map[string]reflect.Type),
+ emits: make(map[string]reflect.Type),
+ iters: make(map[string]reflect.Type),
+ imports: make(map[string]struct{}),
+ allExported: true,
+ }
+}
+
+// Eval contains and uniquifies the cache of types and things that need to be generated.
+type Eval struct {
+ // d is a buffer for the diagnostic information produced when evaluating the pipeline.
+ // w is the primary output buffer.
+ d, w bytes.Buffer
+
+ // Register and uniquify the needed shims for each kind.
+ // Functions to Register
+ functions map[string]*funcx.Fn
+ // Types to Register (structs, essentially)
+ types map[string]reflect.Type
+ // FuncShims needed
+ funcs map[string]reflect.Type
+ // Emitter Shims needed
+ emits map[string]reflect.Type
+ // Iterator Shims needed
+ iters map[string]reflect.Type
+
+ // list of packages we need to import.
+ imports map[string]struct{}
+
+ allExported bool // Marks if all ptransforms are exported and available in main.
+}
+
+// extractFromMultiEdges audits the given pipeline edges so we can determine if
+// this pipeline will run without reflection.
+func (e *Eval) extractFromMultiEdges(edges []*graph.MultiEdge) {
+ e.diag("PTransform Audit:\n")
+ for _, edge := range edges {
+ switch edge.Op {
+ case graph.ParDo:
+ // Gets the ParDo's identifier
+ e.diagf("pardo %s", edge.Name())
+ e.extractGraphFn((*graph.Fn)(edge.DoFn))
+ case graph.Combine:
+ e.diagf("combine %s", edge.Name())
+ e.extractGraphFn((*graph.Fn)(edge.CombineFn))
+ default:
+ continue
+ }
+ e.diag("\n")
+ }
+}
+
+// Performant returns whether this pipeline needs additional registrations
+// to avoid reflection, or symbol lookups at runtime.
+func (e *Eval) Performant() bool {
+ return !e.RequiresRegistrations() && !e.UsesDefaultReflectionShims()
+}
+
+// RequiresRegistrations returns if there are any types or functions that require
+// registrations.
+func (e *Eval) RequiresRegistrations() bool {
+ return (len(e.functions) + len(e.types)) > 0
+}
+
+// UsesDefaultReflectionShims returns whether the default reflection shims are going
+// to be used by the pipeline.
+func (e *Eval) UsesDefaultReflectionShims() bool {
+ return (len(e.funcs) + len(e.emits) + len(e.iters)) > 0
+}
+
+// AllExported returns whether all values in the pipeline are exported,
+// and thus it may be possible to patch the pipeline's package with
+// generated shims.
+// Using exported vs unexported identifiers does not affect pipeline performance
+// but does matter on if the pipeline package can do anything about it.
+func (e *Eval) AllExported() bool {
+ return e.allExported
+}
+
+func (e *Eval) summary() {
+ e.diag("\n")
+ e.diag("Summary\n")
+ e.diagf("All exported?: %v\n", e.AllExported())
+ e.diagf("%d\t Imports\n", len(e.imports))
+ e.diagf("%d\t Functions\n", len(e.functions))
+ e.diagf("%d\t Types\n", len(e.types))
+ e.diagf("%d\t Shims\n", len(e.funcs))
+ e.diagf("%d\t Emits\n", len(e.emits))
+ e.diagf("%d\t Inputs\n", len(e.iters))
+
+ if e.Performant() {
+ e.diag("Pipeline is performant!\n")
+ } else {
+ e.diag("Pipeline is not performant:\n")
+ if e.RequiresRegistrations() {
+ e.diag("\trequires additional type or function registration\n")
+ }
+ if e.UsesDefaultReflectionShims() {
+ e.diag("\trequires additional shim generation\n")
+ }
+ if e.AllExported() {
+ e.diag("\tGood News! All identifiers are exported; the pipeline's package can be patched with generated output.\n")
+ }
+ }
+}
+
+// NameType turns a reflect.Type into a string based on its name.
+// It prefixes Emit or Iter if the function satisfies the constraints of those types.
+func NameType(t reflect.Type) string {
+ if emt, ok := makeEmitter(t); ok {
+ return "Emit" + emt.Name
+ }
+ if ipt, ok := makeInput(t); ok {
+ return "Iter" + ipt.Name
+ }
+ return shimx.Name(t.String())
+}
+
+// Generate produces a go file under the given package.
+func (e *Eval) Generate(packageName string) {
+ // Here's where we shove everything into the Top template type.
+ // Need to swap in typex.* for beam.* where appropriate.
+ e.diag("Diagnostic output pre-amble for the code generator\n")
+
+ e.diag("Functions\n")
+ var functions []string
+ for fn, t := range e.functions {
+ e.diagf("%s, %v\n", fn, t)
+ n := strings.Split(fn, ".")
+ // If this is the main package, we don't need the package qualifier
+ if n[0] == "main" {
+ functions = append(functions, n[1])
+ } else {
+ functions = append(functions, fn)
+ }
+ }
+ e.diag("Types\n")
+ var types []string
+ for fn, t := range e.types {
+ e.diagf("%s, %v\n", fn, t)
+ n := strings.Split(fn, ".")
+ // If this is the main package, we don't need the package qualifier
+ if n[0] == "main" {
+ types = append(types, n[1])
+ } else {
+ types = append(types, fn)
+ }
+ }
+ e.diag("Shims\n")
+ var shims []shimx.Func
+ for fn, t := range e.funcs {
+ e.diagf("%s, %v\n", fn, t)
+ shim := shimx.Func{Type: t.String()}
+ var inNames []string
+ for i := 0; i < t.NumIn(); i++ {
+ s := t.In(i)
+ shim.In = append(shim.In, s.String())
+ inNames = append(inNames, NameType(s))
+ }
+ var outNames []string
+ for i := 0; i < t.NumOut(); i++ {
+ s := t.Out(i)
+ shim.Out = append(shim.Out, s.String())
+ outNames = append(outNames, NameType(s))
+ }
+ shim.Name = shimx.FuncName(inNames, outNames)
+ shims = append(shims, shim)
+ }
+ e.diag("Emitters\n")
+ var emitters []shimx.Emitter
+ for k, t := range e.emits {
+ e.diagf("%s, %v\n", k, t)
+ emt, ok := makeEmitter(t)
+ if !ok {
+ panic(fmt.Sprintf("%v is not an emit, but we expected it to be one.", t))
+ }
+ emitters = append(emitters, emt)
+ }
+ e.diag("Iterators \n")
+ var inputs []shimx.Input
+ for ipt, t := range e.iters {
+ e.diagf("%s, %v\n", ipt, t)
+ itr, ok := makeInput(t)
+ if !ok {
+ panic(fmt.Sprintf("%v is not an emit, but we expected it to be one.", t))
+ }
+ inputs = append(inputs, itr)
+ }
+ var imports []string
+ for k := range e.imports {
+ if k == "" {
+ continue
+ }
+ imports = append(imports, k)
+ }
+
+ top := shimx.Top{
+ Package: packageName,
+ Imports: imports,
+ Functions: functions,
+ Types: types,
+ Shims: shims,
+ Emitters: emitters,
+ Inputs: inputs,
+ }
+ shimx.File(&e.w, &top)
+}
+
+func makeEmitter(t reflect.Type) (shimx.Emitter, bool) {
+ types, isEmit := funcx.UnfoldEmit(t)
+ if !isEmit {
+ return shimx.Emitter{}, false
+ }
+ emt := shimx.Emitter{Type: t.String()}
+ switch len(types) {
+ case 1:
+ emt.Time = false
+ emt.Val = types[0].String()
+ case 2:
+ if types[0] == typex.EventTimeType {
+ emt.Time = true
+ } else {
+ emt.Key = types[0].String()
+ }
+ emt.Val = types[1].String()
+ case 3:
+ // If there's 3, the first one must be typex.EvalentTime.
+ emt.Time = true
+ emt.Key = types[1].String()
+ emt.Val = types[2].String()
+ }
+ if emt.Time {
+ emt.Name = fmt.Sprintf("ET%s%s", shimx.Name(emt.Key), shimx.Name(emt.Val))
+ } else {
+ emt.Name = fmt.Sprintf("%s%s", shimx.Name(emt.Key), shimx.Name(emt.Val))
+ }
+ return emt, true
+}
+
+func makeInput(t reflect.Type) (shimx.Input, bool) {
+ itr := shimx.Input{Type: t.String()}
+ types, isIter := funcx.UnfoldIter(t)
+ if !isIter {
+ return shimx.Input{}, false
+ }
+ switch len(types) {
+ case 1:
+ itr.Time = false
+ itr.Val = types[0].String()
+ case 2:
+ if types[0] == typex.EventTimeType {
+ itr.Time = true
+ } else {
+ itr.Key = types[0].String()
+ }
+ itr.Val = types[1].String()
+ case 3:
+ // If there's 3, the first one must be typex.EventTime.
+ itr.Time = true
+ itr.Key = types[1].String()
+ itr.Val = types[2].String()
+ }
+ if itr.Time {
+ itr.Name = fmt.Sprintf("ET%s%s", shimx.Name(itr.Key), shimx.Name(itr.Val))
+ } else {
+ itr.Name = fmt.Sprintf("%s%s", shimx.Name(itr.Key), shimx.Name(itr.Val))
+ }
+ return itr, true
+}
+
+// needFunction marks the function itself needs to be registered
+func (e *Eval) needFunction(fn *funcx.Fn) {
+ k := fn.Fn.Name()
+ if _, ok := e.functions[k]; ok {
+ e.diag(" FUNCTION_COVERED")
+ } else {
+ e.diag(" NEED_FUNCTION") // Needs a RegisterFunction
+ e.functions[k] = fn
+ e.needImport(fn.Fn.Name())
+ }
+}
+
+// needImport registers the given identifier's import for including in generation.
+func (e *Eval) needImport(p string) {
+ // If this is a reflect.methodValueCall, this is covered by the type
+ // check already, so we don't need to do anything.
+ if p == "reflect.methodValueCall" {
+ return
+ }
+ // Split at last '.' to get full package name and identifier name.
+ splitInd := strings.LastIndexByte(p, '.')
+ pp := p[:splitInd]
+
+ // If it's ad-hoc, or in main we can't/won't import it.
+ if pp == "main" || pp == "" {
+ return
+ }
+
+ // Check if the identifier is exported
+ r, _ := utf8.DecodeRuneInString(p[splitInd+1:])
+ if !unicode.IsUpper(r) {
+ e.allExported = false
+ return
+ }
+ e.imports[pp] = struct{}{}
+ e.diagf("\n%s\n", pp)
+}
+
+// needShim marks the function's type signature as needing to be specialized.
+func (e *Eval) needShim(fn *funcx.Fn) {
+ k := fn.Fn.Type().String()
+ if _, ok := e.funcs[k]; ok {
+ e.diag(" SHIM_COVERED")
+ } else {
+ e.diag(" NEED_SHIM") // Needs a RegisterFunc
+ e.funcs[k] = fn.Fn.Type()
+ e.needImport(fn.Fn.Name())
+ }
+}
+
+// needType marks the struct's type signature as needing to be specialized.
+func (e *Eval) needType(k string, rt reflect.Type) {
+ if _, ok := e.types[k]; ok {
+ e.diag(" OK")
+ } else {
+ e.diag(" NEED_TYPE") // Needs a RegisterType
+ e.types[k] = rt
+ e.needImport(k)
+ }
+}
+
+// needEmit marks the emit parameter as needed specialization
+func (e *Eval) needEmit(rt reflect.Type) {
+ k := fmt.Sprintf("%v", rt)
+ if exec.IsEmitterRegistered(rt) {
+ e.diag(" OK")
+ return
+ }
+ if _, ok := e.emits[k]; ok {
+ e.diag(" EMIT_COVERED")
+ } else {
+ e.diagf(" NEED_EMIT[%v]", rt) // Needs a RegisterEmit
+ e.emits[k] = rt
+ }
+}
+
+// needInput marks the iterator parameter as needed specialization
+func (e *Eval) needInput(rt reflect.Type) {
+ k := fmt.Sprintf("%v", rt)
+ if exec.IsInputRegistered(rt) {
+ e.diag(" OK")
+ return
+ }
+ if _, ok := e.iters[k]; ok {
+ e.diag(" INPUT_COVERED")
+ } else {
+ e.diagf(" NEED_INPUT[%v]", rt) // Needs a RegisterInput
+ e.iters[k] = rt
+ }
+}
+
+// diag invokes fmt.Fprint on the diagnostic buffer.
+func (e *Eval) diag(s string) {
+ fmt.Fprint(&e.d, s)
+}
+
+// diag invokes fmt.Fprintf on the diagnostic buffer.
+func (e *Eval) diagf(f string, args ...interface{}) {
+ fmt.Fprintf(&e.d, f, args...)
+}
+
+// Print invokes fmt.Fprint on the Eval buffer.
+func (e *Eval) Print(s string) {
+ fmt.Fprint(&e.w, s)
+}
+
+// Printf invokes fmt.Fprintf on the Eval buffer.
+func (e *Eval) Printf(f string, args ...interface{}) {
+ fmt.Fprintf(&e.w, f, args...)
+}
+
+// Bytes returns the Eval buffer's bytes for file writing.
+func (e *Eval) Bytes() []byte {
+ return e.w.Bytes()
+}
+
+// We need to take graph.Fns (which can be created from interface{} from graph.NewFn)
+// and convert them to all needed function caller signatures,
+// and emitters.
+//
+// The type assertion shim Funcs need to be registered with reflectx.RegisterFunc
+// Emitters need to be registered with exec.RegisterEmitter
+// Iterators with exec.RegisterInput
+// The types need to be registered with beam.RegisterType
+// The user functions need to be registered with beam.RegisterFunction
+//
+// Registrations are all on the concrete element type, rather than the
+// pointer type.
+
+// extractGraphFn does the analysis of the function and determines what things need generating.
+// A single line is used, unless it's a struct, at which point one line per implemented method
+// is used.
+func (e *Eval) extractGraphFn(fn *graph.Fn) {
+ if fn.DynFn != nil {
+ // TODO(BEAM-7375) handle dynamics if necessary (probably not since it's got general function handling)
+ e.diag(" dynamic function")
+ return
+ }
+ if fn.Recv != nil {
+ e.diagf(" struct[[%T]]", fn.Recv)
+
+ rt := reflectx.SkipPtr(reflect.TypeOf(fn.Recv)) // We need the value not the pointer that's used.
+ if tk, ok := runtime.TypeKey(rt); ok {
+ if t, found := runtime.LookupType(tk); !found {
+ e.needType(tk, rt)
+ } else {
+ e.diagf(" FOUND %v", t) // Doesn't need a RegisterType
+ }
+ } else {
+ e.diagf(" CANT REGISTER %v %v %v", rt, rt.PkgPath(), rt.Name())
+ }
+ e.extractFromDoFn((*graph.DoFn)(fn))
+ e.extractFromCombineFn((*graph.CombineFn)(fn))
+ }
+
+ if fn.Fn != nil {
+ // This goes here since methods don't need registering. That's handled by the type.
+ f := fn.Fn.Fn
+ if _, err := runtime.ResolveFunction(f.Name(), f.Type()); err != nil {
+ e.needFunction(fn.Fn) // Need a RegisterFunction
+ }
+ e.extractFuncxFn(fn.Fn)
+ }
+}
+
+type mthd struct {
+ m func() *funcx.Fn
+ name string
+}
+
+func (e *Eval) extractFromCombineFn(cmbfn *graph.CombineFn) {
+ methods := []mthd{
+ {cmbfn.SetupFn, "SetupFn"},
+ {cmbfn.CreateAccumulatorFn, "CreateAccumulatorFn"},
+ {cmbfn.AddInputFn, "AddInputFn"},
+ {cmbfn.MergeAccumulatorsFn, "MergeAccumulatorsFn"},
+ {cmbfn.ExtractOutputFn, "ExtractOutputFn"},
+ {cmbfn.CompactFn, "CompactFn"},
+ {cmbfn.TeardownFn, "TeardownFn"},
+ }
+ e.extractMethods(methods)
+}
+
+func (e *Eval) extractFromDoFn(dofn *graph.DoFn) {
+ methods := []mthd{
+ {dofn.SetupFn, "SetupFn"},
+ {dofn.StartBundleFn, "StartBundleFn"},
+ {dofn.ProcessElementFn, "ProcessElementFn"},
+ {dofn.FinishBundleFn, "FinishBundleFn"},
+ {dofn.TeardownFn, "TeardownFn"},
+ }
+ e.extractMethods(methods)
+}
+
+func (e *Eval) extractMethods(methods []mthd) {
+ for _, m := range methods {
+ if mfn := m.m(); mfn != nil {
+ e.diag("\n\t- ")
+ e.diag(m.name)
+ e.extractFuncxFn(mfn)
+ }
+ }
+}
+
+// extractFuncxFn writes everything to the same line marking things as registered or not as needed.
+func (e *Eval) extractFuncxFn(fn *funcx.Fn) {
+ t := fn.Fn.Type()
+ e.diagf(" function[[%v]]", t)
+ // We don't have access to the maps directly, so we can sanity check if we need
+ // a shim by checking against this type.
+ if shim := fmt.Sprintf("%T", fn.Fn); shim == "*reflectx.reflectFunc" {
+ e.needShim(fn) // Need a generated Shim and RegisterFunc
+ }
+ // Need to extract emitter types and iterator types for specialization.
+ // We're "stuck" always generating these all the time, since we
+ // can't tell what's already registered at this level.
+ for _, p := range fn.Param {
+ switch p.Kind {
+ case funcx.FnEmit:
+ e.needEmit(p.T) // Need a generated emitter and RegisterEmitter
+ case funcx.FnIter:
+ e.needInput(p.T) // Need a generated iter and RegisterInput
+ case funcx.FnReIter:
+ e.needInput(p.T) // ???? Might be unnecessary?
+ }
+ }
+}
diff --git a/sdks/go/pkg/beam/runners/vet/vet_test.go b/sdks/go/pkg/beam/runners/vet/vet_test.go
new file mode 100644
index 0000000..dc7076a
--- /dev/null
+++ b/sdks/go/pkg/beam/runners/vet/vet_test.go
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package vet
+
+import (
+ "context"
+ "github.com/apache/beam/sdks/go/pkg/beam/runners/vet/testpipeline"
+ "testing"
+
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func TestEvaluate(t *testing.T) {
+ tests := []struct {
+ name string
+ c func(beam.Scope)
+ perf, exp, ref, reg bool
+ }{
+ {name: "Performant", c: testpipeline.Performant, perf: true},
+ {name: "FunctionReg", c: testpipeline.FunctionReg, exp: true, ref: true, reg: true},
+ {name: "ShimNeeded", c: testpipeline.ShimNeeded, ref: true},
+ {name: "TypeReg", c: testpipeline.TypeReg, ref: true, reg: true},
+ }
+ for _, test := range tests {
+ test := test
+ t.Run(test.name, func(t *testing.T) {
+ p, s := beam.NewPipelineWithRoot()
+ test.c(s)
+ e, err := Evaluate(context.Background(), p)
+ if err != nil {
+ t.Fatalf("failed to evaluate testpipeline.Pipeline: %v", err)
+ }
+ if e.Performant() != test.perf {
+ t.Fatalf("e.Performant() = %v, want %v", e.Performant(), test.perf)
+ }
+ // Abort early for performant pipelines.
+ if test.perf {
+ return
+ }
+ e.summary()
+ if e.AllExported() != test.exp {
+ t.Errorf("e.AllExported() = %v, want %v", e.AllExported(), test.exp)
+ }
+ if e.RequiresRegistrations() != test.reg {
+ t.Errorf("e.RequiresRegistrations() = %v, want %v\n%v", e.RequiresRegistrations(), test.reg, string(e.d.Bytes()))
+ }
+ if e.UsesDefaultReflectionShims() != test.ref {
+ t.Errorf("e.UsesDefaultReflectionShims() = %v, want %v", e.UsesDefaultReflectionShims(), test.ref)
+ }
+ })
+ }
+}
diff --git a/sdks/go/pkg/beam/transforms/stats/stats.shims.go b/sdks/go/pkg/beam/transforms/stats/stats.shims.go
index cd2a95a..c13babe 100644
--- a/sdks/go/pkg/beam/transforms/stats/stats.shims.go
+++ b/sdks/go/pkg/beam/transforms/stats/stats.shims.go
@@ -68,41 +68,41 @@
runtime.RegisterType(reflect.TypeOf((*meanAccum)(nil)).Elem())
runtime.RegisterType(reflect.TypeOf((*meanFn)(nil)).Elem())
reflectx.RegisterStructWrapper(reflect.TypeOf((*meanFn)(nil)).Elem(), wrapMakerMeanFn)
- reflectx.RegisterFunc(reflect.TypeOf((*func(float32,float32) (float32))(nil)).Elem(), funcMakerFloat32Float32ГFloat32)
- reflectx.RegisterFunc(reflect.TypeOf((*func(float64,float64) (float64))(nil)).Elem(), funcMakerFloat64Float64ГFloat64)
- reflectx.RegisterFunc(reflect.TypeOf((*func(int16,int16) (int16))(nil)).Elem(), funcMakerInt16Int16ГInt16)
- reflectx.RegisterFunc(reflect.TypeOf((*func(int32,int32) (int32))(nil)).Elem(), funcMakerInt32Int32ГInt32)
- reflectx.RegisterFunc(reflect.TypeOf((*func(int64,int64) (int64))(nil)).Elem(), funcMakerInt64Int64ГInt64)
- reflectx.RegisterFunc(reflect.TypeOf((*func(int8,int8) (int8))(nil)).Elem(), funcMakerInt8Int8ГInt8)
- reflectx.RegisterFunc(reflect.TypeOf((*func(int,int) (int))(nil)).Elem(), funcMakerIntIntГInt)
- reflectx.RegisterFunc(reflect.TypeOf((*func(meanAccum,meanAccum) (meanAccum))(nil)).Elem(), funcMakerMeanAccumMeanAccumГMeanAccum)
- reflectx.RegisterFunc(reflect.TypeOf((*func(meanAccum,typex.T) (meanAccum))(nil)).Elem(), funcMakerMeanAccumTypex۰TГMeanAccum)
- reflectx.RegisterFunc(reflect.TypeOf((*func(meanAccum) (float64))(nil)).Elem(), funcMakerMeanAccumГFloat64)
- reflectx.RegisterFunc(reflect.TypeOf((*func(typex.T) (typex.T,int))(nil)).Elem(), funcMakerTypex۰TГTypex۰TInt)
- reflectx.RegisterFunc(reflect.TypeOf((*func(uint16,uint16) (uint16))(nil)).Elem(), funcMakerUint16Uint16ГUint16)
- reflectx.RegisterFunc(reflect.TypeOf((*func(uint32,uint32) (uint32))(nil)).Elem(), funcMakerUint32Uint32ГUint32)
- reflectx.RegisterFunc(reflect.TypeOf((*func(uint64,uint64) (uint64))(nil)).Elem(), funcMakerUint64Uint64ГUint64)
- reflectx.RegisterFunc(reflect.TypeOf((*func(uint8,uint8) (uint8))(nil)).Elem(), funcMakerUint8Uint8ГUint8)
- reflectx.RegisterFunc(reflect.TypeOf((*func(uint,uint) (uint))(nil)).Elem(), funcMakerUintUintГUint)
- reflectx.RegisterFunc(reflect.TypeOf((*func() (meanAccum))(nil)).Elem(), funcMakerГMeanAccum)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(float32, float32) float32)(nil)).Elem(), funcMakerFloat32Float32ГFloat32)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(float64, float64) float64)(nil)).Elem(), funcMakerFloat64Float64ГFloat64)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int16, int16) int16)(nil)).Elem(), funcMakerInt16Int16ГInt16)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int32, int32) int32)(nil)).Elem(), funcMakerInt32Int32ГInt32)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int64, int64) int64)(nil)).Elem(), funcMakerInt64Int64ГInt64)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int8, int8) int8)(nil)).Elem(), funcMakerInt8Int8ГInt8)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(int, int) int)(nil)).Elem(), funcMakerIntIntГInt)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(meanAccum, meanAccum) meanAccum)(nil)).Elem(), funcMakerMeanAccumMeanAccumГMeanAccum)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(meanAccum, typex.T) meanAccum)(nil)).Elem(), funcMakerMeanAccumTypex۰TГMeanAccum)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(meanAccum) float64)(nil)).Elem(), funcMakerMeanAccumГFloat64)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(typex.T) (typex.T, int))(nil)).Elem(), funcMakerTypex۰TГTypex۰TInt)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(uint16, uint16) uint16)(nil)).Elem(), funcMakerUint16Uint16ГUint16)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(uint32, uint32) uint32)(nil)).Elem(), funcMakerUint32Uint32ГUint32)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(uint64, uint64) uint64)(nil)).Elem(), funcMakerUint64Uint64ГUint64)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(uint8, uint8) uint8)(nil)).Elem(), funcMakerUint8Uint8ГUint8)
+ reflectx.RegisterFunc(reflect.TypeOf((*func(uint, uint) uint)(nil)).Elem(), funcMakerUintUintГUint)
+ reflectx.RegisterFunc(reflect.TypeOf((*func() meanAccum)(nil)).Elem(), funcMakerГMeanAccum)
}
func wrapMakerMeanFn(fn interface{}) map[string]reflectx.Func {
dfn := fn.(*meanFn)
return map[string]reflectx.Func{
- "AddInput": reflectx.MakeFunc(func(a0 meanAccum, a1 typex.T) (meanAccum) { return dfn.AddInput(a0, a1) }),
- "CreateAccumulator": reflectx.MakeFunc(func() (meanAccum) { return dfn.CreateAccumulator() }),
- "ExtractOutput": reflectx.MakeFunc(func(a0 meanAccum) (float64) { return dfn.ExtractOutput(a0) }),
- "MergeAccumulators": reflectx.MakeFunc(func(a0 meanAccum, a1 meanAccum) (meanAccum) { return dfn.MergeAccumulators(a0, a1) }),
+ "AddInput": reflectx.MakeFunc(func(a0 meanAccum, a1 typex.T) meanAccum { return dfn.AddInput(a0, a1) }),
+ "CreateAccumulator": reflectx.MakeFunc(func() meanAccum { return dfn.CreateAccumulator() }),
+ "ExtractOutput": reflectx.MakeFunc(func(a0 meanAccum) float64 { return dfn.ExtractOutput(a0) }),
+ "MergeAccumulators": reflectx.MakeFunc(func(a0 meanAccum, a1 meanAccum) meanAccum { return dfn.MergeAccumulators(a0, a1) }),
}
}
type callerFloat32Float32ГFloat32 struct {
- fn func(float32,float32) (float32)
+ fn func(float32, float32) float32
}
func funcMakerFloat32Float32ГFloat32(fn interface{}) reflectx.Func {
- f := fn.(func(float32,float32) (float32))
+ f := fn.(func(float32, float32) float32)
return &callerFloat32Float32ГFloat32{fn: f}
}
@@ -119,16 +119,16 @@
return []interface{}{out0}
}
-func (c *callerFloat32Float32ГFloat32) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerFloat32Float32ГFloat32) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(float32), arg1.(float32))
}
type callerFloat64Float64ГFloat64 struct {
- fn func(float64,float64) (float64)
+ fn func(float64, float64) float64
}
func funcMakerFloat64Float64ГFloat64(fn interface{}) reflectx.Func {
- f := fn.(func(float64,float64) (float64))
+ f := fn.(func(float64, float64) float64)
return &callerFloat64Float64ГFloat64{fn: f}
}
@@ -145,16 +145,16 @@
return []interface{}{out0}
}
-func (c *callerFloat64Float64ГFloat64) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerFloat64Float64ГFloat64) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(float64), arg1.(float64))
}
type callerInt16Int16ГInt16 struct {
- fn func(int16,int16) (int16)
+ fn func(int16, int16) int16
}
func funcMakerInt16Int16ГInt16(fn interface{}) reflectx.Func {
- f := fn.(func(int16,int16) (int16))
+ f := fn.(func(int16, int16) int16)
return &callerInt16Int16ГInt16{fn: f}
}
@@ -171,16 +171,16 @@
return []interface{}{out0}
}
-func (c *callerInt16Int16ГInt16) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerInt16Int16ГInt16) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(int16), arg1.(int16))
}
type callerInt32Int32ГInt32 struct {
- fn func(int32,int32) (int32)
+ fn func(int32, int32) int32
}
func funcMakerInt32Int32ГInt32(fn interface{}) reflectx.Func {
- f := fn.(func(int32,int32) (int32))
+ f := fn.(func(int32, int32) int32)
return &callerInt32Int32ГInt32{fn: f}
}
@@ -197,16 +197,16 @@
return []interface{}{out0}
}
-func (c *callerInt32Int32ГInt32) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerInt32Int32ГInt32) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(int32), arg1.(int32))
}
type callerInt64Int64ГInt64 struct {
- fn func(int64,int64) (int64)
+ fn func(int64, int64) int64
}
func funcMakerInt64Int64ГInt64(fn interface{}) reflectx.Func {
- f := fn.(func(int64,int64) (int64))
+ f := fn.(func(int64, int64) int64)
return &callerInt64Int64ГInt64{fn: f}
}
@@ -223,16 +223,16 @@
return []interface{}{out0}
}
-func (c *callerInt64Int64ГInt64) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerInt64Int64ГInt64) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(int64), arg1.(int64))
}
type callerInt8Int8ГInt8 struct {
- fn func(int8,int8) (int8)
+ fn func(int8, int8) int8
}
func funcMakerInt8Int8ГInt8(fn interface{}) reflectx.Func {
- f := fn.(func(int8,int8) (int8))
+ f := fn.(func(int8, int8) int8)
return &callerInt8Int8ГInt8{fn: f}
}
@@ -249,16 +249,16 @@
return []interface{}{out0}
}
-func (c *callerInt8Int8ГInt8) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerInt8Int8ГInt8) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(int8), arg1.(int8))
}
type callerIntIntГInt struct {
- fn func(int,int) (int)
+ fn func(int, int) int
}
func funcMakerIntIntГInt(fn interface{}) reflectx.Func {
- f := fn.(func(int,int) (int))
+ f := fn.(func(int, int) int)
return &callerIntIntГInt{fn: f}
}
@@ -275,16 +275,16 @@
return []interface{}{out0}
}
-func (c *callerIntIntГInt) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerIntIntГInt) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(int), arg1.(int))
}
type callerMeanAccumMeanAccumГMeanAccum struct {
- fn func(meanAccum,meanAccum) (meanAccum)
+ fn func(meanAccum, meanAccum) meanAccum
}
func funcMakerMeanAccumMeanAccumГMeanAccum(fn interface{}) reflectx.Func {
- f := fn.(func(meanAccum,meanAccum) (meanAccum))
+ f := fn.(func(meanAccum, meanAccum) meanAccum)
return &callerMeanAccumMeanAccumГMeanAccum{fn: f}
}
@@ -301,16 +301,16 @@
return []interface{}{out0}
}
-func (c *callerMeanAccumMeanAccumГMeanAccum) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerMeanAccumMeanAccumГMeanAccum) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(meanAccum), arg1.(meanAccum))
}
type callerMeanAccumTypex۰TГMeanAccum struct {
- fn func(meanAccum,typex.T) (meanAccum)
+ fn func(meanAccum, typex.T) meanAccum
}
func funcMakerMeanAccumTypex۰TГMeanAccum(fn interface{}) reflectx.Func {
- f := fn.(func(meanAccum,typex.T) (meanAccum))
+ f := fn.(func(meanAccum, typex.T) meanAccum)
return &callerMeanAccumTypex۰TГMeanAccum{fn: f}
}
@@ -327,16 +327,16 @@
return []interface{}{out0}
}
-func (c *callerMeanAccumTypex۰TГMeanAccum) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerMeanAccumTypex۰TГMeanAccum) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(meanAccum), arg1.(typex.T))
}
type callerMeanAccumГFloat64 struct {
- fn func(meanAccum) (float64)
+ fn func(meanAccum) float64
}
func funcMakerMeanAccumГFloat64(fn interface{}) reflectx.Func {
- f := fn.(func(meanAccum) (float64))
+ f := fn.(func(meanAccum) float64)
return &callerMeanAccumГFloat64{fn: f}
}
@@ -353,16 +353,16 @@
return []interface{}{out0}
}
-func (c *callerMeanAccumГFloat64) Call1x1(arg0 interface{}) (interface{}) {
+func (c *callerMeanAccumГFloat64) Call1x1(arg0 interface{}) interface{} {
return c.fn(arg0.(meanAccum))
}
type callerTypex۰TГTypex۰TInt struct {
- fn func(typex.T) (typex.T,int)
+ fn func(typex.T) (typex.T, int)
}
func funcMakerTypex۰TГTypex۰TInt(fn interface{}) reflectx.Func {
- f := fn.(func(typex.T) (typex.T,int))
+ f := fn.(func(typex.T) (typex.T, int))
return &callerTypex۰TГTypex۰TInt{fn: f}
}
@@ -384,11 +384,11 @@
}
type callerUint16Uint16ГUint16 struct {
- fn func(uint16,uint16) (uint16)
+ fn func(uint16, uint16) uint16
}
func funcMakerUint16Uint16ГUint16(fn interface{}) reflectx.Func {
- f := fn.(func(uint16,uint16) (uint16))
+ f := fn.(func(uint16, uint16) uint16)
return &callerUint16Uint16ГUint16{fn: f}
}
@@ -405,16 +405,16 @@
return []interface{}{out0}
}
-func (c *callerUint16Uint16ГUint16) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerUint16Uint16ГUint16) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(uint16), arg1.(uint16))
}
type callerUint32Uint32ГUint32 struct {
- fn func(uint32,uint32) (uint32)
+ fn func(uint32, uint32) uint32
}
func funcMakerUint32Uint32ГUint32(fn interface{}) reflectx.Func {
- f := fn.(func(uint32,uint32) (uint32))
+ f := fn.(func(uint32, uint32) uint32)
return &callerUint32Uint32ГUint32{fn: f}
}
@@ -431,16 +431,16 @@
return []interface{}{out0}
}
-func (c *callerUint32Uint32ГUint32) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerUint32Uint32ГUint32) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(uint32), arg1.(uint32))
}
type callerUint64Uint64ГUint64 struct {
- fn func(uint64,uint64) (uint64)
+ fn func(uint64, uint64) uint64
}
func funcMakerUint64Uint64ГUint64(fn interface{}) reflectx.Func {
- f := fn.(func(uint64,uint64) (uint64))
+ f := fn.(func(uint64, uint64) uint64)
return &callerUint64Uint64ГUint64{fn: f}
}
@@ -457,16 +457,16 @@
return []interface{}{out0}
}
-func (c *callerUint64Uint64ГUint64) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerUint64Uint64ГUint64) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(uint64), arg1.(uint64))
}
type callerUint8Uint8ГUint8 struct {
- fn func(uint8,uint8) (uint8)
+ fn func(uint8, uint8) uint8
}
func funcMakerUint8Uint8ГUint8(fn interface{}) reflectx.Func {
- f := fn.(func(uint8,uint8) (uint8))
+ f := fn.(func(uint8, uint8) uint8)
return &callerUint8Uint8ГUint8{fn: f}
}
@@ -483,16 +483,16 @@
return []interface{}{out0}
}
-func (c *callerUint8Uint8ГUint8) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerUint8Uint8ГUint8) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(uint8), arg1.(uint8))
}
type callerUintUintГUint struct {
- fn func(uint,uint) (uint)
+ fn func(uint, uint) uint
}
func funcMakerUintUintГUint(fn interface{}) reflectx.Func {
- f := fn.(func(uint,uint) (uint))
+ f := fn.(func(uint, uint) uint)
return &callerUintUintГUint{fn: f}
}
@@ -509,16 +509,16 @@
return []interface{}{out0}
}
-func (c *callerUintUintГUint) Call2x1(arg0, arg1 interface{}) (interface{}) {
+func (c *callerUintUintГUint) Call2x1(arg0, arg1 interface{}) interface{} {
return c.fn(arg0.(uint), arg1.(uint))
}
type callerГMeanAccum struct {
- fn func() (meanAccum)
+ fn func() meanAccum
}
func funcMakerГMeanAccum(fn interface{}) reflectx.Func {
- f := fn.(func() (meanAccum))
+ f := fn.(func() meanAccum)
return &callerГMeanAccum{fn: f}
}
@@ -535,9 +535,8 @@
return []interface{}{out0}
}
-func (c *callerГMeanAccum) Call0x1() (interface{}) {
+func (c *callerГMeanAccum) Call0x1() interface{} {
return c.fn()
}
-
// DO NOT MODIFY: GENERATED CODE
diff --git a/sdks/go/pkg/beam/x/beamx/run.go b/sdks/go/pkg/beam/x/beamx/run.go
index 3380b06..9a1d27a 100644
--- a/sdks/go/pkg/beam/x/beamx/run.go
+++ b/sdks/go/pkg/beam/x/beamx/run.go
@@ -30,6 +30,7 @@
_ "github.com/apache/beam/sdks/go/pkg/beam/runners/direct"
_ "github.com/apache/beam/sdks/go/pkg/beam/runners/dot"
_ "github.com/apache/beam/sdks/go/pkg/beam/runners/flink"
+ _ "github.com/apache/beam/sdks/go/pkg/beam/runners/spark"
_ "github.com/apache/beam/sdks/go/pkg/beam/runners/universal"
)
diff --git a/sdks/go/test/build.gradle b/sdks/go/test/build.gradle
index a55698a..77fd3be 100644
--- a/sdks/go/test/build.gradle
+++ b/sdks/go/test/build.gradle
@@ -48,13 +48,29 @@
}
task flinkValidatesRunner {
- dependsOn ":beam-sdks-go-test:build"
- dependsOn ":beam-runners-flink_2.11-job-server:shadowJar"
+ dependsOn ":sdks:go:test:goBuild"
+ dependsOn ":runners:flink:1.8:job-server:shadowJar"
doLast {
def options = [
"--runner flink",
"--parallel 1", // prevent memory overuse
- "--flink_job_server_jar ${project(":beam-runners-flink_2.11-job-server:").shadowJar.archivePath}",
+ "--flink_job_server_jar ${project(":runners:flink:1.8:job-server").shadowJar.archivePath}",
+ ]
+ exec {
+ executable "sh"
+ args "-c", "./run_integration_tests.sh ${options.join(' ')}"
+ }
+ }
+}
+
+task sparkValidatesRunner {
+ dependsOn ":sdks:go:test:goBuild"
+ dependsOn ":runners:spark:job-server:shadowJar"
+ doLast {
+ def options = [
+ "--runner spark",
+ "--parallel 1", // prevent memory overuse
+ "--spark_job_server_jar ${project(":runners:spark:job-server").shadowJar.archivePath}",
]
exec {
executable "sh"
diff --git a/sdks/go/test/integration/primitives/pardo.go b/sdks/go/test/integration/primitives/pardo.go
index ae7d9b2..62654e8 100644
--- a/sdks/go/test/integration/primitives/pardo.go
+++ b/sdks/go/test/integration/primitives/pardo.go
@@ -53,7 +53,8 @@
p, s := beam.NewPipelineWithRoot()
in := beam.Create(s, 1, 2, 3, 4, 5, 6, 7, 8, 9)
- out := beam.ParDo(s, sumValuesFn, beam.Impulse(s), beam.SideInput{Input: in})
+ sub := s.Scope("subscope") // Ensure scoping works with side inputs. See: BEAM-5354
+ out := beam.ParDo(sub, sumValuesFn, beam.Impulse(s), beam.SideInput{Input: in})
passert.Sum(s, out, "out", 1, 45)
return p
diff --git a/sdks/go/test/run_integration_tests.sh b/sdks/go/test/run_integration_tests.sh
index 51906f5..bfa29b4 100755
--- a/sdks/go/test/run_integration_tests.sh
+++ b/sdks/go/test/run_integration_tests.sh
@@ -74,6 +74,11 @@
shift # past argument
shift # past value
;;
+ --spark_job_server_jar)
+ SPARK_JOB_SERVER_JAR="$2"
+ shift # past argument
+ shift # past value
+ ;;
--endpoint)
ENDPOINT="$2"
shift # past argument
@@ -120,7 +125,7 @@
# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
-CONTAINER=us.gcr.io/$PROJECT/$USER/go
+CONTAINER=us.gcr.io/$PROJECT/$USER/go_sdk
echo "Using container $CONTAINER"
./gradlew :sdks:go:container:docker -Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
@@ -135,7 +140,7 @@
DATAFLOW_WORKER_JAR=$(find ./runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-*.jar)
fi
echo "Using Dataflow worker jar: $DATAFLOW_WORKER_JAR"
-elif [[ "$RUNNER" == "flink" ]]; then
+elif [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" ]]; then
if [[ -z "$ENDPOINT" ]]; then
# Hacky python script to find a free port. Note there is a small chance the chosen port could
# get taken before being claimed by the job server.
@@ -148,12 +153,20 @@
"
JOB_PORT=$(python -c "$SOCKET_SCRIPT")
ENDPOINT="localhost:$JOB_PORT"
- echo "No endpoint specified; starting a new Flink job server on $ENDPOINT"
- java \
- -jar $FLINK_JOB_SERVER_JAR \
- --flink-master-url [local] \
- --job-port $JOB_PORT \
- --artifact-port 0 &
+ echo "No endpoint specified; starting a new $RUNNER job server on $ENDPOINT"
+ if [[ "$RUNNER" == "flink" ]]; then
+ java \
+ -jar $FLINK_JOB_SERVER_JAR \
+ --flink-master-url [local] \
+ --job-port $JOB_PORT \
+ --artifact-port 0 &
+ else
+ java \
+ -jar $SPARK_JOB_SERVER_JAR \
+ --spark-master-url local \
+ --job-port $JOB_PORT \
+ --artifact-port 0 &
+ fi
fi
fi
diff --git a/sdks/java/build-tools/build.gradle b/sdks/java/build-tools/build.gradle
index 53f88b7..5916470 100644
--- a/sdks/java/build-tools/build.gradle
+++ b/sdks/java/build-tools/build.gradle
@@ -17,6 +17,6 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, publish: false)
description = "Apache Beam :: SDKs :: Java :: Build Tools"
diff --git a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
index 3d7fe48..26265e3 100644
--- a/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
@@ -106,6 +106,38 @@
<property name="message" value="You are using raw guava, please use vendored guava classes."/>
</module>
+ <!-- Forbid guava imports from gRPC vendored guava version. -->
+ <module name="RegexpSinglelineJava">
+ <property name="id" value="ForbidGrpcVendoredGuava"/>
+ <property name="format" value="(\sorg\.apache\.beam\.vendor\.grpc\.(.*)\.com\.google\.common\.(?!testing))|(\sorg\.apache\.beam\.vendor\.grpc\.(.*)\.com\.google\.thirdparty)"/>
+ <property name="severity" value="error"/>
+ <property name="message" value="You are using raw guava, please use vendored guava classes."/>
+ </module>
+
+ <!-- Forbid TestNG imports that may leak because of dependencies. -->
+ <module name="RegexpSinglelineJava">
+ <property name="id" value="ForbidTestNG"/>
+ <property name="format" value="(\sorg\.testng)"/>
+ <property name="severity" value="error"/>
+ <property name="message" value="You should not use TestNG classes in Beam."/>
+ </module>
+
+ <!-- Forbid Non-vendored byte-buddy imports. -->
+ <module name="RegexpSinglelineJava">
+ <property name="id" value="ForbidByteBuddy"/>
+ <property name="format" value="(\snet\.bytebuddy)"/>
+ <property name="severity" value="error"/>
+ <property name="message" value="You are using raw byte-buddy, please use vendored byte-buddy classes."/>
+ </module>
+
+ <!-- Forbid Non-vendored calcite imports. -->
+ <module name="RegexpSinglelineJava">
+ <property name="id" value="ForbidCalcite"/>
+ <property name="format" value="(\sorg\.apache\.calcite)"/>
+ <property name="severity" value="error"/>
+ <property name="message" value="You are using raw calcite, please use vendored calcite classes."/>
+ </module>
+
<module name="UnusedImports">
<property name="severity" value="error"/>
<property name="processJavadoc" value="true"/>
diff --git a/sdks/java/build-tools/src/main/resources/beam/spotbugs-filter.xml b/sdks/java/build-tools/src/main/resources/beam/spotbugs-filter.xml
index c0de9c8..7811398 100644
--- a/sdks/java/build-tools/src/main/resources/beam/spotbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/spotbugs-filter.xml
@@ -179,25 +179,6 @@
</Match>
<Match>
- <Class name="org.apache.beam.runners.direct.portable.ExecutorServiceParallelExecutor$QueueMessageReceiver" />
- <Or>
- <Method name="failed" />
- <Method name="cancelled" />
- <Method name="completed" />
- </Or>
- <Bug pattern="RV_RETURN_VALUE_IGNORED_BAD_PRACTICE" />
- <!-- updates is a non-capacity-limited LinkedBlockingQueue, which
- can never refuse an offered update -->
- </Match>
-
- <Match>
- <Class name="org.apache.beam.runners.direct.portable.job.ReferenceRunnerJobService" />
- <Method name="run" />
- <Bug pattern="RV_RETURN_VALUE_IGNORED_BAD_PRACTICE" />
- <!-- the success of future methods aren't based on the returned value of the future -->
- </Match>
-
- <Match>
<Class name="org.apache.beam.runners.spark.util.BroadcastHelper$CodedBroadcastHelper"/>
<Or>
<Field name="bcast" />
@@ -430,4 +411,23 @@
<Bug pattern="URF_UNREAD_FIELD"/>
<!-- Fix build. -->
</Match>
+
+ <Match>
+ <!--
+ This is a false positive. Spotbugs does not recognize the use of try-with-resources, so it thinks that
+ the connection is not correctly closed.
+ -->
+ <Or>
+ <And>
+ <Class name="org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn"/>
+ <Method name="processElement"/>
+ </And>
+ <And>
+ <Class name="org.apache.beam.sdk.io.jdbc.JdbcIO$ReadRows"/>
+ <Method name="inferBeamSchema"/>
+ </And>
+ </Or>
+
+ <Bug pattern="OBL_UNSATISFIED_OBLIGATION"/>
+ </Match>
</FindBugsFilter>
diff --git a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
index a6b25a0..203d92b 100644
--- a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
@@ -80,9 +80,13 @@
<suppress id="ForbidNonVendoredGuava" files=".*kinesis.*KinesisIO\.java" />
<suppress id="ForbidNonVendoredGuava" files=".*kinesis.*KinesisProducerMock\.java" />
<suppress id="ForbidNonVendoredGuava" files=".*bigtable.*VendoredListenableFutureAdapter\.java" />
+ <suppress id="ForbidNonVendoredGuava" files=".*bigtable.*VendoredListenableFutureAdapter\.java" />
<suppress id="ForbidNonVendoredGuava" files=".*bigtable.*BigtableServiceImplTest\.java" />
<suppress id="ForbidNonVendoredGuava" files=".*sql.*BeamValuesRel\.java" />
<suppress id="ForbidNonVendoredGuava" files=".*sql.*BeamEnumerableConverterTest\.java" />
+ <suppress id="ForbidNonVendoredGuava" files=".*zetasql.*TableScanConverter\.java" />
+ <suppress id="ForbidNonVendoredGuava" files=".*zetasql.*ExpressionConverter\.java" />
+ <suppress id="ForbidNonVendoredGuava" files=".*zetasql.*ZetaSQLPlannerImpl\.java" />
<!-- Flink -->
<!-- Checkstyle does not correctly detect package files across multiple source directories. -->
diff --git a/sdks/java/container/build.gradle b/sdks/java/container/build.gradle
index 33e094b..63e40e4 100644
--- a/sdks/java/container/build.gradle
+++ b/sdks/java/container/build.gradle
@@ -42,7 +42,9 @@
dockerDependency library.java.slf4j_jdk14
dockerDependency project(path: ":sdks:java:harness", configuration: "shadow")
// For executing KafkaIO, e.g. as an external transform
- dockerDependency project(path: ":sdks:java:io:kafka", configuration: "shadow")
+ dockerDependency project(":sdks:java:io:kafka")
+ // This dependency is set to 'provided' scope in :sdks:java:io:kafka
+ dockerDependency library.java.kafka_clients
}
def dockerfileName = project.findProperty('dockerfile') ?: 'Dockerfile'
@@ -67,7 +69,10 @@
}
docker {
- name containerImageName(name: "java")
+ name containerImageName(
+ name: "java_sdk",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] : "apachebeam")
dockerfile project.file("./${dockerfileName}")
files "./build/"
}
diff --git a/sdks/java/core/build.gradle b/sdks/java/core/build.gradle
index 0290683..a7ed6c2 100644
--- a/sdks/java/core/build.gradle
+++ b/sdks/java/core/build.gradle
@@ -17,19 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(shadowClosure: DEFAULT_SHADOW_CLOSURE << {
- dependencies {
- include(dependency(library.java.protobuf_java))
- include(dependency(library.java.byte_buddy))
- include(dependency("org.apache.commons:.*"))
- include(dependency(library.java.antlr_runtime))
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk',
+ shadowClosure: {
+ dependencies {
+ include(dependency(library.java.protobuf_java))
+ include(dependency("org.apache.commons:.*"))
+ include(dependency(library.java.antlr_runtime))
+ }
+ relocate "com.google.thirdparty", getJavaRelocatedPath("com.google.thirdparty")
+ relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
+ relocate "org.apache.commons", getJavaRelocatedPath("org.apache.commons")
+ relocate "org.antlr.v4", getJavaRelocatedPath("org.antlr.v4")
}
- relocate "com.google.thirdparty", getJavaRelocatedPath("com.google.thirdparty")
- relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
- relocate "net.bytebuddy", getJavaRelocatedPath("net.bytebuddy")
- relocate "org.apache.commons", getJavaRelocatedPath("org.apache.commons")
- relocate "org.antlr.v4", getJavaRelocatedPath("org.antlr.v4")
-})
+)
applyAvroNature()
applyAntlrNature()
@@ -62,13 +63,12 @@
// Required to load constants from the model, e.g. max timestamp for global window
shadow project(path: ":model:pipeline", configuration: "shadow")
shadow project(path: ":model:job-management", configuration: "shadow")
- shadow library.java.vendored_guava_20_0
+ shadow library.java.vendored_bytebuddy_1_9_3
+ shadow library.java.vendored_guava_26_0_jre
compile library.java.antlr_runtime
compile library.java.protobuf_java
- compile library.java.byte_buddy
compile library.java.commons_compress
compile library.java.commons_lang3
- compile library.java.guava_testlib
shadow library.java.jackson_core
shadow library.java.jackson_annotations
shadow library.java.jackson_databind
diff --git a/sdks/java/core/src/main/antlr/org/apache/beam/sdk/schemas/parser/generated/FieldSpecifierNotation.g4 b/sdks/java/core/src/main/antlr/org/apache/beam/sdk/schemas/parser/generated/FieldSpecifierNotation.g4
index a869304..07bd8ac 100644
--- a/sdks/java/core/src/main/antlr/org/apache/beam/sdk/schemas/parser/generated/FieldSpecifierNotation.g4
+++ b/sdks/java/core/src/main/antlr/org/apache/beam/sdk/schemas/parser/generated/FieldSpecifierNotation.g4
@@ -45,7 +45,7 @@
| '{*}'
;
-IDENTIFIER: [a-zA-Z0-9]+ ;
+IDENTIFIER: [a-zA-Z0-9_]+ ;
WILDCARD: '*';
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index b49e8a1..fa784f6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -17,15 +17,17 @@
*/
package org.apache.beam.sdk;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.transform;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.transform;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.coders.CoderRegistry;
@@ -48,16 +50,16 @@
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Collections2;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Collections2;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SetMultimap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -664,8 +666,11 @@
this.instances = instancePerName;
}
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public String apply(final Map.Entry<String, Collection<PTransform<?, ?>>> input) {
+ public String apply(@Nonnull final Map.Entry<String, Collection<PTransform<?, ?>>> input) {
final Collection<PTransform<?, ?>> values = instances.get(input.getKey());
return "- name="
+ input.getKey()
@@ -676,15 +681,21 @@
private static class KeysExtractor
implements Function<Map.Entry<String, Collection<PTransform<?, ?>>>, String> {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public String apply(final Map.Entry<String, Collection<PTransform<?, ?>>> input) {
+ public String apply(@Nonnull final Map.Entry<String, Collection<PTransform<?, ?>>> input) {
return input.getKey();
}
}
private static class IsUnique<K, V> implements Predicate<Map.Entry<K, Collection<V>>> {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public boolean apply(final Map.Entry<K, Collection<V>> input) {
+ public boolean apply(@Nonnull final Map.Entry<K, Collection<V>> input) {
return input != null && input.getValue().size() == 1;
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineResult.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineResult.java
index f84b7fd..2c9d06f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineResult.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineResult.java
@@ -75,7 +75,7 @@
*/
enum State {
- /** The job state could not be obtained or was not specified. */
+ /** The job state was not specified or unknown to a runner. */
UNKNOWN(false, false),
/** The job has been paused, or has not yet started. */
@@ -94,7 +94,10 @@
CANCELLED(true, false),
/** The job has been updated. */
- UPDATED(true, true);
+ UPDATED(true, true),
+
+ /** The job state reported by a runner cannot be interpreted by the SDK. */
+ UNRECOGNIZED(false, false);
private final boolean terminal;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineRunner.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineRunner.java
index 2d819a0..6a5eb05 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineRunner.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
index 0921490..b044165 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java
@@ -34,6 +34,7 @@
import javax.annotation.Nullable;
import org.apache.avro.AvroRuntimeException;
import org.apache.avro.Schema;
+import org.apache.avro.data.TimeConversions.TimestampConversion;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
@@ -56,8 +57,8 @@
import org.apache.avro.util.Utf8;
import org.apache.beam.sdk.util.EmptyOnDeserializationThreadLocal;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
/**
* A {@link Coder} using Avro binary format.
@@ -131,8 +132,7 @@
* Returns an {@code AvroCoder} instance for the provided element type using the provided Avro
* schema.
*
- * <p>If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema
- * must correspond to the type provided.
+ * <p>The schema must correspond to the type provided.
*
* @param <T> the element type
*/
@@ -237,7 +237,9 @@
@Override
public ReflectData get() {
- return new ReflectData(clazz.getClassLoader());
+ ReflectData reflectData = new ReflectData(clazz.getClassLoader());
+ reflectData.addLogicalTypeConversion(new TimestampConversion());
+ return reflectData;
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
index e705120..bc9ce55 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.InputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianLongCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianLongCoder.java
index c13bacb..b5d0b60 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianLongCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianLongCoder.java
@@ -26,7 +26,7 @@
import java.io.UTFDataFormatException;
import org.apache.beam.sdk.values.TypeDescriptor;
-/** A {@link BigEndianLongCoder} encodes {@link Long}s in 8 bytes, big-endian. */
+/** A {@link BigEndianLongCoder} encodes {@link Long Longs} in 8 bytes, big-endian. */
public class BigEndianLongCoder extends AtomicCoder<Long> {
public static BigEndianLongCoder of() {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianShortCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianShortCoder.java
index 2ea605f..5ef21ad 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianShortCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigEndianShortCoder.java
@@ -26,7 +26,7 @@
import java.io.UTFDataFormatException;
import org.apache.beam.sdk.values.TypeDescriptor;
-/** A {@link BigEndianShortCoder} encodes {@link Short Shorts} in 4 bytes, big-endian. */
+/** A {@link BigEndianShortCoder} encodes {@link Short Shorts} in 2 bytes, big-endian. */
public class BigEndianShortCoder extends AtomicCoder<Short> {
public static BigEndianShortCoder of() {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java
index a878e6c..0c497e1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.InputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
index e7f7543..9ccbc12 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BooleanCoder.java
@@ -39,7 +39,13 @@
@Override
public Boolean decode(InputStream is) throws IOException {
- return BYTE_CODER.decode(is) == 1;
+ Byte value = BYTE_CODER.decode(is);
+ if (value == 0) {
+ return false;
+ } else if (value == 1) {
+ return true;
+ }
+ throw new IOException(String.format("Expected 0 or 1, got %d", value));
}
@Override
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
index 45f4cca..fd555bc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ByteArrayCoder.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.util.StreamUtils;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A {@link Coder} for {@code byte[]}.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
index 1ba5537..fb405d2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
@@ -32,11 +32,11 @@
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
/**
* A {@link Coder Coder<T>} defines how to encode and decode values of type {@code T} into
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderProviders.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderProviders.java
index 71fd5df..2def8c4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderProviders.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderProviders.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
@@ -25,7 +25,7 @@
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Static utility methods for creating and working with {@link CoderProvider}s. */
public final class CoderProviders {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
index a7bb2ef..1e218ff 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
@@ -51,15 +51,15 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSetMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SetMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -103,8 +103,7 @@
Boolean.class, CoderProviders.fromStaticMethods(Boolean.class, BooleanCoder.class));
builder.put(Byte.class, CoderProviders.fromStaticMethods(Byte.class, ByteCoder.class));
builder.put(BitSet.class, CoderProviders.fromStaticMethods(BitSet.class, BitSetCoder.class));
- builder.put(
- FloatCoder.class, CoderProviders.fromStaticMethods(Float.class, FloatCoder.class));
+ builder.put(Float.class, CoderProviders.fromStaticMethods(Float.class, FloatCoder.class));
builder.put(Double.class, CoderProviders.fromStaticMethods(Double.class, DoubleCoder.class));
builder.put(
Instant.class, CoderProviders.fromStaticMethods(Instant.class, InstantCoder.class));
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
index c80f15c..5974150 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DefaultCoder.java
@@ -28,7 +28,7 @@
import javax.annotation.CheckForNull;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DelegateCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DelegateCoder.java
index fc83bdc..2abfdf1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DelegateCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/DelegateCoder.java
@@ -23,8 +23,8 @@
import java.io.Serializable;
import javax.annotation.Nullable;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/**
* A {@code DelegateCoder<T, IntermediateT>} wraps a {@link Coder} for {@code IntermediateT} and
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/FloatCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/FloatCoder.java
index b131c4e..44dcc4a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/FloatCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/FloatCoder.java
@@ -26,7 +26,7 @@
import java.io.UTFDataFormatException;
import org.apache.beam.sdk.values.TypeDescriptor;
-/** A {@link FloatCoder} encodes {@link Float} values in 8 bytes using Java serialization. */
+/** A {@link FloatCoder} encodes {@link Float} values in 4 bytes using Java serialization. */
public class FloatCoder extends AtomicCoder<Float> {
public static FloatCoder of() {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/IterableLikeCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/IterableLikeCoder.java
index a3c5761..fcd744c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/IterableLikeCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/IterableLikeCoder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.DataInputStream;
import java.io.DataOutputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/LengthPrefixCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/LengthPrefixCoder.java
index e724d08..a86524d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/LengthPrefixCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/LengthPrefixCoder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
@@ -25,8 +25,8 @@
import java.io.OutputStream;
import java.util.List;
import org.apache.beam.sdk.util.VarInt;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A {@link Coder} which is able to take any existing coder and wrap it such that it is only invoked
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java
index 346ca08..1a95c2e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeParameter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* A {@link Coder} for {@link Map Maps} that encodes them according to provided coders for keys and
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
index 637b9c4..2bacbab 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java
@@ -24,8 +24,8 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link NullableCoder} encodes nullable values of type {@code T} using a nested {@code Coder<T>}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
index dd507d5..b94f30f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java
@@ -17,234 +17,38 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-import java.util.stream.Collectors;
-import javax.annotation.Nullable;
+import java.util.Objects;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.Field;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-/** A {@link Coder} for {@link Row}. It wraps the {@link Coder} for each element directly. */
+/** A sub-class of SchemaCoder that can only encode {@link Row} instances. */
@Experimental
-public class RowCoder extends CustomCoder<Row> {
- // This contains a map of primitive types to their coders.
- static final ImmutableMap<TypeName, Coder> CODER_MAP =
- ImmutableMap.<TypeName, Coder>builder()
- .put(TypeName.BYTE, ByteCoder.of())
- .put(TypeName.BYTES, ByteArrayCoder.of())
- .put(TypeName.INT16, BigEndianShortCoder.of())
- .put(TypeName.INT32, VarIntCoder.of())
- .put(TypeName.INT64, VarLongCoder.of())
- .put(TypeName.DECIMAL, BigDecimalCoder.of())
- .put(TypeName.FLOAT, FloatCoder.of())
- .put(TypeName.DOUBLE, DoubleCoder.of())
- .put(TypeName.STRING, StringUtf8Coder.of())
- .put(TypeName.DATETIME, InstantCoder.of())
- .put(TypeName.BOOLEAN, BooleanCoder.of())
- .build();
-
- private static final ImmutableMap<TypeName, Integer> ESTIMATED_FIELD_SIZES =
- ImmutableMap.<TypeName, Integer>builder()
- .put(TypeName.BYTE, Byte.BYTES)
- .put(TypeName.INT16, Short.BYTES)
- .put(TypeName.INT32, Integer.BYTES)
- .put(TypeName.INT64, Long.BYTES)
- .put(TypeName.FLOAT, Float.BYTES)
- .put(TypeName.DOUBLE, Double.BYTES)
- .put(TypeName.DECIMAL, 32)
- .put(TypeName.BOOLEAN, 1)
- .put(TypeName.DATETIME, Long.BYTES)
- .build();
-
- private final Schema schema;
-
- public UUID getId() {
- return id;
- }
-
- private final UUID id;
- @Nullable private transient Coder<Row> delegateCoder = null;
-
+public class RowCoder extends SchemaCoder<Row> {
public static RowCoder of(Schema schema) {
- UUID id = (schema.getUUID() == null) ? UUID.randomUUID() : schema.getUUID();
- return new RowCoder(schema, id);
+ return new RowCoder(schema);
}
- private RowCoder(Schema schema, UUID id) {
- if (schema.getUUID() != null) {
- checkArgument(
- schema.getUUID().equals(id),
- "Schema has a UUID that doesn't match argument to constructor. %s v.s. %s",
- schema.getUUID(),
- id);
- } else {
- // Clone the schema before modifying the Java object.
- schema = SerializableUtils.clone(schema);
- setSchemaIds(schema, id);
- }
- this.schema = schema;
- this.id = id;
- }
-
- // Sets the schema id, and then recursively ensures that all schemas have ids set.
- private void setSchemaIds(Schema schema, UUID id) {
- if (schema.getUUID() == null) {
- schema.setUUID(id);
- }
- for (Field field : schema.getFields()) {
- setSchemaIds(field.getType());
- }
- }
-
- private void setSchemaIds(FieldType fieldType) {
- switch (fieldType.getTypeName()) {
- case ROW:
- setSchemaIds(fieldType.getRowSchema(), UUID.randomUUID());
- return;
- case MAP:
- setSchemaIds(fieldType.getMapKeyType());
- setSchemaIds(fieldType.getMapValueType());
- return;
- case LOGICAL_TYPE:
- setSchemaIds(fieldType.getLogicalType().getBaseType());
- return;
-
- case ARRAY:
- setSchemaIds(fieldType.getCollectionElementType());
- return;
-
- default:
- return;
- }
- }
-
- // Return the generated coder class for this schema.
- private Coder<Row> getDelegateCoder() {
- if (delegateCoder == null) {
- // RowCoderGenerator caches based on id, so if a new instance of this RowCoder is
- // deserialized, we don't need to run ByteBuddy again to construct the class.
- delegateCoder = RowCoderGenerator.generate(schema);
- }
- return delegateCoder;
+ private RowCoder(Schema schema) {
+ super(schema, SerializableFunctions.identity(), SerializableFunctions.identity());
}
@Override
- public void encode(Row value, OutputStream outStream) throws IOException {
- getDelegateCoder().encode(value, outStream);
- }
-
- @Override
- public Row decode(InputStream inStream) throws IOException {
- return getDelegateCoder().decode(inStream);
- }
-
- public Schema getSchema() {
- return schema;
- }
-
- @Override
- public void verifyDeterministic()
- throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
- verifyDeterministic(schema);
- }
-
- private void verifyDeterministic(Schema schema)
- throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
-
- List<Coder<?>> coders =
- schema.getFields().stream()
- .map(Field::getType)
- .map(RowCoder::coderForFieldType)
- .collect(Collectors.toList());
-
- Coder.verifyDeterministic(this, "All fields must have deterministic encoding", coders);
- }
-
- @Override
- public boolean consistentWithEquals() {
- return true;
- }
-
- /** Returns the coder used for a given primitive type. */
- public static <T> Coder<T> coderForFieldType(FieldType fieldType) {
- switch (fieldType.getTypeName()) {
- case ROW:
- return (Coder<T>) RowCoder.of(fieldType.getRowSchema());
- case ARRAY:
- return (Coder<T>) ListCoder.of(coderForFieldType(fieldType.getCollectionElementType()));
- case MAP:
- return (Coder<T>)
- MapCoder.of(
- coderForFieldType(fieldType.getMapKeyType()),
- coderForFieldType(fieldType.getMapValueType()));
- default:
- return (Coder<T>) CODER_MAP.get(fieldType.getTypeName());
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
}
- }
-
- /** Return the estimated serialized size of a give row object. */
- public static long estimatedSizeBytes(Row row) {
- Schema schema = row.getSchema();
- int fieldCount = schema.getFieldCount();
- int bitmapSize = (((fieldCount - 1) >> 6) + 1) * 8;
-
- int fieldsSize = 0;
- for (int i = 0; i < schema.getFieldCount(); ++i) {
- fieldsSize += (int) estimatedSizeBytes(schema.getField(i).getType(), row.getValue(i));
+ if (o == null || getClass() != o.getClass()) {
+ return false;
}
- return (long) bitmapSize + fieldsSize;
- }
-
- private static long estimatedSizeBytes(FieldType typeDescriptor, Object value) {
- switch (typeDescriptor.getTypeName()) {
- case LOGICAL_TYPE:
- return estimatedSizeBytes(typeDescriptor.getLogicalType().getBaseType(), value);
- case ROW:
- return estimatedSizeBytes((Row) value);
- case ARRAY:
- List list = (List) value;
- long listSizeBytes = 0;
- for (Object elem : list) {
- listSizeBytes += estimatedSizeBytes(typeDescriptor.getCollectionElementType(), elem);
- }
- return 4 + listSizeBytes;
- case BYTES:
- byte[] bytes = (byte[]) value;
- return 4L + bytes.length;
- case MAP:
- Map<Object, Object> map = (Map<Object, Object>) value;
- long mapSizeBytes = 0;
- for (Map.Entry<Object, Object> elem : map.entrySet()) {
- mapSizeBytes +=
- typeDescriptor.getMapKeyType().getTypeName().equals(TypeName.STRING)
- ? ((String) elem.getKey()).length()
- : ESTIMATED_FIELD_SIZES.get(typeDescriptor.getMapKeyType().getTypeName());
- mapSizeBytes += estimatedSizeBytes(typeDescriptor.getMapValueType(), elem.getValue());
- }
- return 4 + mapSizeBytes;
- case STRING:
- // Not always accurate - String.getBytes().length() would be more accurate here, but slower.
- return ((String) value).length();
- default:
- return ESTIMATED_FIELD_SIZES.get(typeDescriptor.getTypeName());
- }
+ RowCoder rowCoder = (RowCoder) o;
+ return schema.equals(rowCoder.schema);
}
@Override
- public String toString() {
- String string = "Schema: " + schema + " UUID: " + id + " delegateCoder: " + getDelegateCoder();
- return string;
+ public int hashCode() {
+ return Objects.hash(schema);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
index 351b04e..b084a09 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
@@ -27,35 +27,36 @@
import java.util.List;
import java.util.Map;
import java.util.UUID;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.modifier.FieldManifestation;
-import net.bytebuddy.description.modifier.Ownership;
-import net.bytebuddy.description.modifier.Visibility;
-import net.bytebuddy.description.type.TypeDescription;
-import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.implementation.FixedValue;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.Duplication;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
-import net.bytebuddy.implementation.bytecode.TypeCreation;
-import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
-import net.bytebuddy.implementation.bytecode.member.FieldAccess;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.FieldManifestation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.Ownership;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.Visibility;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.FixedValue;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Duplication;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.TypeCreation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* A utility for automatically generating a {@link Coder} for {@link Row} objects corresponding to a
@@ -114,7 +115,7 @@
// Initialize the CODER_MAP with the StackManipulations to create the primitive coders.
// Assumes that each class contains a static of() constructor method.
CODER_MAP = Maps.newHashMap();
- for (Map.Entry<TypeName, Coder> entry : RowCoder.CODER_MAP.entrySet()) {
+ for (Map.Entry<TypeName, Coder> entry : SchemaCoder.CODER_MAP.entrySet()) {
StackManipulation stackManipulation =
MethodInvocation.invoke(
new ForLoadedType(entry.getValue().getClass())
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
index ea84dbd..641952d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SerializableCoder.java
@@ -31,7 +31,7 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ShardedKeyCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ShardedKeyCoder.java
index eb18d62..2487ed2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ShardedKeyCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/ShardedKeyCoder.java
@@ -23,7 +23,7 @@
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.values.ShardedKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** A {@link Coder} for {@link ShardedKey}, using a wrapped key {@link Coder}. */
@VisibleForTesting
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SnappyCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SnappyCoder.java
index 3f436a4..cf410ac 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SnappyCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/SnappyCoder.java
@@ -22,7 +22,7 @@
import java.io.OutputStream;
import java.util.List;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.xerial.snappy.Snappy;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StringUtf8Coder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StringUtf8Coder.java
index 4414cb3..33e6153 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StringUtf8Coder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StringUtf8Coder.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.util.StreamUtils;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Utf8;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Utf8;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A {@link Coder} that encodes {@link String Strings} in UTF-8 encoding. If in a nested context,
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StructuralByteArray.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StructuralByteArray.java
index e20b6b5..79f944a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StructuralByteArray.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/StructuralByteArray.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.coders;
import java.util.Arrays;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
/**
* A wrapper around a byte[] that uses structural, value-based equality rather than byte[]'s normal
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
index 26ee379..da19881 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java
@@ -18,12 +18,14 @@
package org.apache.beam.sdk.expansion;
import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.transforms.ExternalTransformBuilder;
/**
* A registrar which contains a mapping from URNs to available {@link ExternalTransformBuilder}s.
* Should be used with {@link com.google.auto.service.AutoService}.
*/
+@Experimental
public interface ExternalTransformRegistrar {
/** A mapping from URN to an {@link ExternalTransformBuilder} class. */
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java
new file mode 100644
index 0000000..c3d4938
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.harness;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.options.PipelineOptions;
+
+/**
+ * A service interface for defining one-time initialization of the JVM during pipeline execution.
+ *
+ * <p>During pipeline execution, {@code onStartup} and {@code beforeProcessing} will be invoked at
+ * the appropriate stage of execution after the JVM is launched. Currently this is only supported in
+ * portable pipelines or when using Google Cloud Dataflow.
+ *
+ * <p>{@link java.util.ServiceLoader} is used to discover implementations of {@link JvmInitializer},
+ * note that you will need to register your implementation with the appropriate resources to ensure
+ * your code is executed. You can use a tool like {@link com.google.auto.service.AutoService} to
+ * automate this.
+ */
+@Experimental
+public interface JvmInitializer {
+
+ /**
+ * Implement onStartup to run some custom initialization immediately after the JVM is launched for
+ * pipeline execution.
+ *
+ * <p>In general users should prefer to implement {@code beforeProcessing} to perform custom
+ * initialization so that basic services such as logging can be initialized first, but {@code
+ * onStartup} is also provided if initialization absolutely needs to be run immediately after
+ * starting.
+ */
+ default void onStartup() {}
+
+ /**
+ * Implement beforeProcessing to run some custom initialization after basic services such as
+ * logging, but before data processing begins.
+ *
+ * @param options The pipeline options passed to the worker.
+ */
+ default void beforeProcessing(PipelineOptions options) {}
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/package-info.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/package-info.java
new file mode 100644
index 0000000..bd6df7b
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/harness/package-info.java
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Utilities for configuring worker environment. */
+@DefaultAnnotation(NonNull.class)
+package org.apache.beam.sdk.harness;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index 4706a7d..bb0e062 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.io;
import static org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -33,6 +33,7 @@
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumWriter;
import org.apache.beam.sdk.annotations.Experimental;
@@ -59,11 +60,13 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Duration;
/**
@@ -74,7 +77,7 @@
* <p>To read a {@link PCollection} from one or more Avro files with the same schema known at
* pipeline construction time, use {@link #read}, using {@link AvroIO.Read#from} to specify the
* filename or filepattern to read from. If the filepatterns to be read are themselves in a {@link
- * PCollection} you can use {@link FileIO} to match them and {@link TextIO#readFiles} to read them.
+ * PCollection} you can use {@link FileIO} to match them and {@link AvroIO#readFiles} to read them.
* If the schema is unknown at pipeline construction time, use {@link #parseGenericRecords} or
* {@link #parseFilesGenericRecords}.
*
@@ -148,7 +151,7 @@
*
* PCollection<String> filepatterns = p.apply(...);
* PCollection<AvroAutoGenClass> records =
- * filepatterns.apply(AvroIO.read(AvroAutoGenClass.class));
+ * filepatterns.apply(AvroIO.readAll(AvroAutoGenClass.class));
* PCollection<AvroAutoGenClass> records =
* filepatterns
* .apply(FileIO.matchAll())
@@ -185,6 +188,49 @@
* scalability. Note that it may decrease performance if the filepattern matches only a small number
* of files.
*
+ * <h3>Inferring Beam schemas from Avro files</h3>
+ *
+ * <p>If you want to use SQL or schema based operations on an Avro-based PCollection, you must
+ * configure the read transform to infer the Beam schema and automatically setup the Beam related
+ * coders by doing:
+ *
+ * <pre>{@code
+ * PCollection<AvroAutoGenClass> records =
+ * p.apply(AvroIO.read(...).from(...).withBeamSchemas(true);
+ * }</pre>
+ *
+ * <h3>Inferring Beam schemas from Avro PCollections</h3>
+ *
+ * <p>If you created an Avro-based PCollection by other means e.g. reading records from Kafka or as
+ * the output of another PTransform, you may be interested on making your PCollection schema-aware
+ * so you can use the Schema-based APIs or Beam's SqlTransform.
+ *
+ * <p>If you are using Avro specific records (generated classes from an Avro schema), you can
+ * register a schema provider for the specific Avro class to make any PCollection of these objects
+ * schema-aware.
+ *
+ * <pre>{@code
+ * pipeline.getSchemaRegistry().registerSchemaProvider(AvroAutoGenClass.class, AvroAutoGenClass.getClassSchema());
+ * }</pre>
+ *
+ * You can also manually set an Avro-backed Schema coder for a PCollection using {@link
+ * org.apache.beam.sdk.schemas.utils.AvroUtils#schemaCoder(Class, Schema)} to make it schema-aware.
+ *
+ * <pre>{@code
+ * PCollection<AvroAutoGenClass> records = ...
+ * AvroCoder<AvroAutoGenClass> coder = (AvroCoder<AvroAutoGenClass>) users.getCoder();
+ * records.setCoder(AvroUtils.schemaCoder(coder.getType(), coder.getSchema()));
+ * }</pre>
+ *
+ * <p>If you are using GenericRecords you may need to set a specific Beam schema coder for each
+ * PCollection to match their internal Avro schema.
+ *
+ * <pre>{@code
+ * org.apache.avro.Schema avroSchema = ...
+ * PCollection<GenericRecord> records = ...
+ * records.setCoder(AvroUtils.schemaCoder(avroSchema));
+ * }</pre>
+ *
* <h2>Writing Avro files</h2>
*
* <p>To write a {@link PCollection} to one or more Avro files, use {@link AvroIO.Write}, using
@@ -624,6 +670,10 @@
return toBuilder().setHintMatchesManyFiles(true).build();
}
+ /**
+ * If set to true, a Beam schema will be inferred from the AVRO schema. This allows the output
+ * to be used by SQL and by the schema-transform library.
+ */
@Experimental(Kind.SCHEMAS)
public Read<T> withBeamSchemas(boolean withBeamSchemas) {
return toBuilder().setInferBeamSchema(withBeamSchemas).build();
@@ -863,7 +913,9 @@
CreateSourceFn(Class<T> recordClass, String jsonSchema) {
this.recordClass = recordClass;
- this.schemaSupplier = AvroUtils.serializableSchemaSupplier(jsonSchema);
+ this.schemaSupplier =
+ Suppliers.memoize(
+ Suppliers.compose(new JsonToSchema(), Suppliers.ofInstance(jsonSchema)));
}
@Override
@@ -874,6 +926,13 @@
recordClass,
schemaSupplier.get());
}
+
+ private static class JsonToSchema implements Function<String, Schema>, Serializable {
+ @Override
+ public Schema apply(String input) {
+ return new Schema.Parser().parse(input);
+ }
+ }
}
/////////////////////////////////////////////////////////////////////////////
@@ -1705,6 +1764,28 @@
/**
* A {@link Sink} for use with {@link FileIO#write} and {@link FileIO#writeDynamic}, writing
+ * elements with a given (common) schema, like {@link #writeGenericRecords(Schema)}.
+ */
+ @Experimental
+ public static <ElementT extends IndexedRecord> Sink<ElementT> sink(Schema schema) {
+ return sink(schema.toString());
+ }
+
+ /**
+ * A {@link Sink} for use with {@link FileIO#write} and {@link FileIO#writeDynamic}, writing
+ * elements with a given (common) schema, like {@link #writeGenericRecords(String)}.
+ */
+ @Experimental
+ public static <ElementT extends IndexedRecord> Sink<ElementT> sink(String jsonSchema) {
+ return new AutoValue_AvroIO_Sink.Builder<ElementT>()
+ .setJsonSchema(jsonSchema)
+ .setMetadata(ImmutableMap.of())
+ .setCodec(TypedWrite.DEFAULT_SERIALIZABLE_CODEC)
+ .build();
+ }
+
+ /**
+ * A {@link Sink} for use with {@link FileIO#write} and {@link FileIO#writeDynamic}, writing
* elements by converting each one to a {@link GenericRecord} with a given (common) schema, like
* {@link #writeCustomTypeToGenericRecords()}.
*
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java
index 412da15..4062a4b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java
@@ -39,7 +39,7 @@
ValueProvider<ResourceId> outputPrefix,
DynamicAvroDestinations<UserT, DestinationT, OutputT> dynamicDestinations,
boolean genericRecords) {
- // Avro handle compression internally using the codec.
+ // Avro handles compression internally using the codec.
super(outputPrefix, dynamicDestinations, Compression.UNCOMPRESSED);
this.genericRecords = genericRecords;
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
index 1f31b02..217ba32 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.io;
import static org.apache.beam.sdk.io.FileBasedSource.Mode.SINGLE_FILE_OR_SUBRANGE;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.ByteArrayInputStream;
import java.io.EOFException;
@@ -64,8 +64,8 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
import org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream;
import org.apache.commons.compress.compressors.xz.XZCompressorInputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroUtils.java
deleted file mode 100644
index f4ff114..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroUtils.java
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io;
-
-import java.io.Serializable;
-import org.apache.avro.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-
-/** Helpers for working with Avro. */
-class AvroUtils {
- /** Helper to get around the fact that {@link Schema} itself is not serializable. */
- public static Supplier<Schema> serializableSchemaSupplier(String jsonSchema) {
- return Suppliers.memoize(
- Suppliers.compose(new JsonToSchema(), Suppliers.ofInstance(jsonSchema)));
- }
-
- private static class JsonToSchema implements Function<String, Schema>, Serializable {
- @Override
- public Schema apply(String input) {
- return new Schema.Parser().parse(input);
- }
- }
-}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
index 2cd9d6d..064cfc2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.ValueWithRecordId;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
index f46f466..ae878e7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CompressedSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.Serializable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java
index 04ced7d..e4f3624 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java
@@ -27,9 +27,9 @@
import java.util.zip.GZIPOutputStream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream;
import org.apache.commons.compress.compressors.deflate.DeflateCompressorInputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/ConstantAvroDestination.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/ConstantAvroDestination.java
index ceb2631..02b33ad 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/ConstantAvroDestination.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/ConstantAvroDestination.java
@@ -26,18 +26,17 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.transforms.display.HasDisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
/** Always returns a constant {@link FilenamePolicy}, {@link Schema}, metadata, and codec. */
class ConstantAvroDestination<UserT, OutputT>
extends DynamicAvroDestinations<UserT, Void, OutputT> {
private static class SchemaFunction implements Serializable, Function<String, Schema> {
- @Nullable
@Override
- public Schema apply(@Nullable String input) {
+ public Schema apply(String input) {
return new Schema.Parser().parse(input);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
index 8179e24..f24246b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.List;
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DefaultFilenamePolicy.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DefaultFilenamePolicy.java
index 34e240d..bf79b6e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DefaultFilenamePolicy.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DefaultFilenamePolicy.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import java.io.IOException;
import java.io.InputStream;
@@ -44,9 +44,9 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/**
* A default {@link FilenamePolicy} for windowed and unwindowed files. This policy is constructed
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicAvroDestinations.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicAvroDestinations.java
index 398fba6..023d397 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicAvroDestinations.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicAvroDestinations.java
@@ -21,7 +21,7 @@
import org.apache.avro.Schema;
import org.apache.avro.file.CodecFactory;
import org.apache.beam.sdk.io.FileBasedSink.DynamicDestinations;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A specialization of {@link DynamicDestinations} for {@link AvroIO}. In addition to dynamic file
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicFileDestinations.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicFileDestinations.java
index cb4b5fb..d992a83 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicFileDestinations.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicFileDestinations.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
import org.apache.beam.sdk.coders.Coder;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index eff8a7c..21a2cf4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -19,11 +19,11 @@
import static org.apache.beam.sdk.io.WriteFiles.UNKNOWN_SHARDNUM;
import static org.apache.beam.sdk.values.TypeDescriptors.extractFromTypeParameters;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verifyNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verifyNotNull;
import java.io.IOException;
import java.io.InputStream;
@@ -40,7 +40,6 @@
import java.util.Objects;
import java.util.Set;
import java.util.UUID;
-import java.util.concurrent.atomic.AtomicLong;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -74,17 +73,14 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors.TypeVariableExtractor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.joda.time.Instant;
-import org.joda.time.format.DateTimeFormat;
-import org.joda.time.format.DateTimeFormatter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -510,7 +506,7 @@
*
* <p>Default is a uniquely named subdirectory of the provided tempDirectory, e.g. if
* tempDirectory is /path/to/foo/, the temporary directory will be
- * /path/to/foo/temp-beam-foo-$date.
+ * /path/to/foo/.temp-beam-$uuid.
*
* @param sink the FileBasedSink that will be used to configure this write operation.
*/
@@ -522,20 +518,12 @@
private static class TemporaryDirectoryBuilder
implements SerializableFunction<ResourceId, ResourceId> {
- private static final AtomicLong TEMP_COUNT = new AtomicLong(0);
- private static final DateTimeFormatter TEMPDIR_TIMESTAMP =
- DateTimeFormat.forPattern("yyyy-MM-dd_HH-mm-ss");
- // The intent of the code is to have a consistent value of tempDirectory across
- // all workers, which wouldn't happen if now() was called inline.
- private final String timestamp = Instant.now().toString(TEMPDIR_TIMESTAMP);
- // Multiple different sinks may be used in the same output directory; use tempId to create a
- // separate temp directory for each.
- private final Long tempId = TEMP_COUNT.getAndIncrement();
+ private final UUID tempUUID = UUID.randomUUID();
@Override
public ResourceId apply(ResourceId tempDirectory) {
- // Temp directory has a timestamp and a unique ID
- String tempDirName = String.format(TEMP_DIRECTORY_PREFIX + "-%s-%s", timestamp, tempId);
+ // Temp directory has a random UUID postfix (BEAM-7689)
+ String tempDirName = String.format(TEMP_DIRECTORY_PREFIX + "-%s", tempUUID);
return tempDirectory
.getCurrentDirectory()
.resolve(tempDirName, StandardResolveOptions.RESOLVE_DIRECTORY);
@@ -803,7 +791,10 @@
FileSystems.match(Collections.singletonList(tempDir.toString() + "*")));
for (Metadata matchResult : singleMatch.metadata()) {
if (allMatches.add(matchResult.resourceId())) {
- LOG.info("Will also remove unknown temporary file {}", matchResult.resourceId());
+ LOG.warn(
+ "Will also remove unknown temporary file {}. This might indicate that other process/job is using "
+ + "the same temporary folder and result in data consistency issues.",
+ matchResult.resourceId());
}
}
} catch (Exception e) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
index 98a85ee..7f8271cd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verify;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verify;
import java.io.IOException;
import java.nio.channels.ReadableByteChannel;
@@ -37,7 +37,7 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
index 5447e86..3339508 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
@@ -19,8 +19,8 @@
import static org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions.RESOLVE_FILE;
import static org.apache.beam.sdk.transforms.Contextful.fn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -72,10 +72,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -237,7 +237,7 @@
* type to the sink's <i>output type</i>.
*
* <p>However, when using dynamic destinations, in many such cases the destination needs to be
- * extract from the original type, so such a conversion is not possible. For example, one might
+ * extracted from the original type, so such a conversion is not possible. For example, one might
* write events of a custom class {@code Event} to a text sink, using the event's "type" as a
* destination. In that case, specify an <i>output function</i> in {@link Write#via(Contextful,
* Contextful)} or {@link Write#via(Contextful, Sink)}.
@@ -729,6 +729,55 @@
builder.add(DisplayData.item("directoryTreatment", getDirectoryTreatment().toString()));
}
+ /**
+ * @return True if metadata is a directory and directory Treatment is SKIP.
+ * @throws java.lang.IllegalArgumentException if metadata is a directory and directoryTreatment
+ * is Prohibited.
+ * @throws java.lang.UnsupportedOperationException if metadata is a directory and
+ * directoryTreatment is not SKIP or PROHIBIT.
+ */
+ static boolean shouldSkipDirectory(
+ MatchResult.Metadata metadata, DirectoryTreatment directoryTreatment) {
+ if (metadata.resourceId().isDirectory()) {
+ switch (directoryTreatment) {
+ case SKIP:
+ return true;
+ case PROHIBIT:
+ throw new IllegalArgumentException(
+ "Trying to read " + metadata.resourceId() + " which is a directory");
+
+ default:
+ throw new UnsupportedOperationException(
+ "Unknown DirectoryTreatment: " + directoryTreatment);
+ }
+ }
+
+ return false;
+ }
+
+ /**
+ * Converts metadata to readableFile. Make sure {@link
+ * #shouldSkipDirectory(org.apache.beam.sdk.io.fs.MatchResult.Metadata,
+ * org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment)} returns false before using.
+ */
+ static ReadableFile matchToReadableFile(
+ MatchResult.Metadata metadata, Compression compression) {
+
+ compression =
+ (compression == Compression.AUTO)
+ ? Compression.detect(metadata.resourceId().getFilename())
+ : compression;
+ return new ReadableFile(
+ MatchResult.Metadata.builder()
+ .setResourceId(metadata.resourceId())
+ .setSizeBytes(metadata.sizeBytes())
+ .setLastModifiedMillis(metadata.lastModifiedMillis())
+ .setIsReadSeekEfficient(
+ metadata.isReadSeekEfficient() && compression == Compression.UNCOMPRESSED)
+ .build(),
+ compression);
+ }
+
private static class ToReadableFileFn extends DoFn<MatchResult.Metadata, ReadableFile> {
private final ReadMatches spec;
@@ -738,36 +787,11 @@
@ProcessElement
public void process(ProcessContext c) {
- MatchResult.Metadata metadata = c.element();
- if (metadata.resourceId().isDirectory()) {
- switch (spec.getDirectoryTreatment()) {
- case SKIP:
- return;
-
- case PROHIBIT:
- throw new IllegalArgumentException(
- "Trying to read " + metadata.resourceId() + " which is a directory");
-
- default:
- throw new UnsupportedOperationException(
- "Unknown DirectoryTreatment: " + spec.getDirectoryTreatment());
- }
+ if (shouldSkipDirectory(c.element(), spec.getDirectoryTreatment())) {
+ return;
}
-
- Compression compression =
- (spec.getCompression() == Compression.AUTO)
- ? Compression.detect(metadata.resourceId().getFilename())
- : spec.getCompression();
- c.output(
- new ReadableFile(
- MatchResult.Metadata.builder()
- .setResourceId(metadata.resourceId())
- .setSizeBytes(metadata.sizeBytes())
- .setLastModifiedMillis(metadata.lastModifiedMillis())
- .setIsReadSeekEfficient(
- metadata.isReadSeekEfficient() && compression == Compression.UNCOMPRESSED)
- .build(),
- compression));
+ ReadableFile r = matchToReadableFile(c.element(), spec.getCompression());
+ c.output(r);
}
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
index 71948c2..34ba69d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
@@ -35,7 +35,7 @@
* <p>It defines APIs for writing file systems agnostic code.
*
* <p>All methods are protected, and they are for file system providers to implement. Clients should
- * use {@link FileSystems} utility.
+ * use the {@link FileSystems} utility.
*/
@Experimental(Kind.FILESYSTEM)
public abstract class FileSystem<ResourceIdT extends ResourceId> {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
index db43788..4f26711 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java
@@ -17,10 +17,11 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verify;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verify;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.channels.ReadableByteChannel;
@@ -51,17 +52,17 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TreeMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeMultimap;
/** Clients facing {@link FileSystem} utility. */
@Experimental(Kind.FILESYSTEM)
@@ -348,6 +349,9 @@
.filter(matchResult -> !matchResult.status().equals(Status.NOT_FOUND))
.transformAndConcat(
new Function<MatchResult, Iterable<Metadata>>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Nonnull
@Override
public Iterable<Metadata> apply(@Nonnull MatchResult input) {
@@ -362,6 +366,9 @@
})
.transform(
new Function<Metadata, ResourceId>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Nonnull
@Override
public ResourceId apply(@Nonnull Metadata input) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenerateSequence.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenerateSequence.java
index 6a33f38..01a0203 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenerateSequence.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenerateSequence.java
@@ -17,12 +17,13 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import com.google.auto.value.AutoValue;
import java.util.Map;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.expansion.ExternalTransformRegistrar;
import org.apache.beam.sdk.transforms.ExternalTransformBuilder;
import org.apache.beam.sdk.transforms.PTransform;
@@ -30,8 +31,8 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -88,6 +89,7 @@
abstract Builder toBuilder();
@AutoValue.Builder
+ @Experimental
abstract static class Builder
implements ExternalTransformBuilder<
External.ExternalConfiguration, PBegin, PCollection<Long>> {
@@ -128,6 +130,7 @@
}
/** Exposes GenerateSequence as an external transform for cross-language usage. */
+ @Experimental
@AutoService(ExternalTransformRegistrar.class)
public static class External implements ExternalTransformRegistrar {
@@ -139,6 +142,7 @@
}
/** Parameters class to expose the transform to an external SDK. */
+ @Experimental
public static class ExternalConfiguration {
private Long start;
@Nullable private Long stop;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
index dc4a918..4a3f11d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files.fileTreeTraverser;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files.fileTraverser;
import java.io.BufferedOutputStream;
import java.io.File;
@@ -40,15 +40,16 @@
import java.util.Collections;
import java.util.List;
import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;
import org.apache.beam.sdk.io.fs.CreateOptions;
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.io.fs.MatchResult.Status;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.commons.lang3.SystemUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -82,6 +83,9 @@
private static final Logger LOG = LoggerFactory.getLogger(LocalFileSystem.class);
+ /** Matches a glob containing a wildcard, capturing the portion before the first wildcard. */
+ private static final Pattern GLOB_PREFIX = Pattern.compile("(?<PREFIX>[^\\[*?]*)[\\[*?].*");
+
LocalFileSystem() {}
@Override
@@ -224,7 +228,7 @@
return MatchResult.create(Status.OK, ImmutableList.of(toMetadata(file)));
}
- File parent = file.getAbsoluteFile().getParentFile();
+ File parent = getSpecNonGlobPrefixParentFile(spec);
if (!parent.exists()) {
return MatchResult.create(Status.NOT_FOUND, Collections.emptyList());
}
@@ -244,12 +248,12 @@
java.nio.file.FileSystems.getDefault().getPathMatcher("glob:" + pathToMatch);
// TODO: Avoid iterating all files: https://issues.apache.org/jira/browse/BEAM-1309
- Iterable<File> files = fileTreeTraverser().preOrderTraversal(parent);
+ Iterable<File> files = fileTraverser().depthFirstPreOrder(parent);
Iterable<File> matchedFiles =
StreamSupport.stream(files.spliterator(), false)
.filter(
Predicates.and(
- org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files.isFile(),
+ org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files.isFile(),
input -> matcher.matches(input.toPath()))
::apply)
.collect(Collectors.toList());
@@ -268,6 +272,14 @@
}
}
+ private File getSpecNonGlobPrefixParentFile(String spec) {
+ String specNonWildcardPrefix = getNonWildcardPrefix(spec);
+ File file = new File(specNonWildcardPrefix);
+ return specNonWildcardPrefix.endsWith(File.separator)
+ ? file
+ : file.getAbsoluteFile().getParentFile();
+ }
+
private Metadata toMetadata(File file) {
return Metadata.builder()
.setResourceId(LocalResourceId.fromPath(file.toPath(), file.isDirectory()))
@@ -276,4 +288,9 @@
.setLastModifiedMillis(file.lastModified())
.build();
}
+
+ private static String getNonWildcardPrefix(String globExp) {
+ Matcher m = GLOB_PREFIX.matcher(globExp);
+ return !m.matches() ? globExp : m.group("PREFIX");
+ }
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystemRegistrar.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystemRegistrar.java
index 171ff5e..38b8c41 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystemRegistrar.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystemRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link AutoService} registrar for the {@link LocalFileSystem}. */
@AutoService(FileSystemRegistrar.class)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalResourceId.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalResourceId.java
index a370cdf..02b6394 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalResourceId.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalResourceId.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.File;
import java.nio.file.Path;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/OffsetBasedSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/OffsetBasedSource.java
index e1439e8..7b10174 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/OffsetBasedSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/OffsetBasedSource.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.ArrayList;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
index 75a10d5..4fb9a60 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
@@ -22,8 +22,8 @@
import static org.apache.avro.file.DataFileConstants.NULL_CODEC;
import static org.apache.avro.file.DataFileConstants.SNAPPY_CODEC;
import static org.apache.avro.file.DataFileConstants.XZ_CODEC;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Externalizable;
import java.io.IOException;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Source.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Source.java
index b32d7a8..9740bfc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Source.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Source.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.transforms.display.HasDisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java
index 96a753a..bb65e77 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -39,14 +39,15 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.util.MimeTypes;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
/**
* {@link PTransform}s for reading and writing TensorFlow TFRecord files.
@@ -74,6 +75,14 @@
}
/**
+ * Like {@link #read}, but reads each file in a {@link PCollection} of {@link
+ * FileIO.ReadableFile}, returned by {@link FileIO#readMatches}.
+ */
+ public static ReadFiles readFiles() {
+ return new AutoValue_TFRecordIO_ReadFiles.Builder().build();
+ }
+
+ /**
* A {@link PTransform} that writes a {@link PCollection} to TFRecord file (or multiple TFRecord
* files matching a sharding pattern), with each element of the input collection encoded into its
* own record.
@@ -211,6 +220,38 @@
/////////////////////////////////////////////////////////////////////////////
+ /** Implementation of {@link #readFiles}. */
+ @AutoValue
+ public abstract static class ReadFiles
+ extends PTransform<PCollection<FileIO.ReadableFile>, PCollection<byte[]>> {
+
+ abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract TFRecordIO.ReadFiles build();
+ }
+
+ @Override
+ public PCollection<byte[]> expand(PCollection<FileIO.ReadableFile> input) {
+ return input.apply(
+ "Read all via FileBasedSource",
+ new ReadAllViaFileBasedSource<>(
+ Long.MAX_VALUE, new CreateSourceFn(), DEFAULT_BYTE_ARRAY_CODER));
+ }
+
+ private static class CreateSourceFn
+ implements SerializableFunction<String, FileBasedSource<byte[]>> {
+
+ @Override
+ public FileBasedSource<byte[]> apply(String input) {
+ return new TFRecordSource(StaticValueProvider.of(input));
+ }
+ }
+ }
+
+ /////////////////////////////////////////////////////////////////////////////
+
/** Implementation of {@link #write}. */
@AutoValue
public abstract static class Write extends PTransform<PCollection<byte[]>, PDone> {
@@ -600,7 +641,7 @@
/**
* Codec for TFRecords file format. See
- * https://www.tensorflow.org/api_guides/python/python_io#TFRecords_Format_Details
+ * https://www.tensorflow.org/versions/r1.11/api_guides/python/python_io#TFRecords_Format_Details
*/
private static class TFRecordCodec {
private static final int HEADER_LEN = (Long.SIZE + Integer.SIZE) / Byte.SIZE;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
index 330b5fc..e7f4785 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.io;
import static org.apache.beam.sdk.io.FileIO.ReadMatches.DirectoryTreatment;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.apache.commons.compress.utils.CharsetNames.UTF_8;
import com.google.auto.value.AutoValue;
@@ -57,10 +57,10 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
/**
@@ -180,6 +180,8 @@
* DynamicDestinations} interface for advanced features via {@link Write#to(DynamicDestinations)}.
*/
public class TextIO {
+ private static final long DEFAULT_BUNDLE_SIZE_BYTES = 64 * 1024 * 1024L;
+
/**
* A {@link PTransform} that reads from one or more text files and returns a bounded {@link
* PCollection} containing one element for each line of the input files.
@@ -224,7 +226,7 @@
// 64MB is a reasonable value that allows to amortize the cost of opening files,
// but is not so large as to exhaust a typical runner's maximum amount of output per
// ProcessElement call.
- .setDesiredBundleSizeBytes(64 * 1024 * 1024L)
+ .setDesiredBundleSizeBytes(DEFAULT_BUNDLE_SIZE_BYTES)
.build();
}
@@ -1316,7 +1318,8 @@
if (getFooter() != null) {
writer.println(getFooter());
}
- writer.close();
+ // BEAM-7813: don't close writer here
+ writer.flush();
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextRowCountEstimator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextRowCountEstimator.java
new file mode 100644
index 0000000..ad26fb1
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextRowCountEstimator.java
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.io.fs.EmptyMatchTreatment;
+import org.apache.beam.sdk.io.fs.MatchResult;
+import org.apache.beam.sdk.io.range.OffsetRange;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
+
+/** This returns a row count estimation for files associated with a file pattern. */
+@AutoValue
+public abstract class TextRowCountEstimator {
+ private static final long DEFAULT_NUM_BYTES_PER_FILE = 64 * 1024L;
+ private static final Compression DEFAULT_COMPRESSION = Compression.AUTO;
+ private static final FileIO.ReadMatches.DirectoryTreatment DEFAULT_DIRECTORY_TREATMENT =
+ FileIO.ReadMatches.DirectoryTreatment.SKIP;
+ private static final EmptyMatchTreatment DEFAULT_EMPTY_MATCH_TREATMENT =
+ EmptyMatchTreatment.DISALLOW;
+ private static final SamplingStrategy DEFAULT_SAMPLING_STRATEGY = new SampleAllFiles();
+
+ public abstract long getNumSampledBytesPerFile();
+
+ @Nullable
+ @SuppressWarnings("mutable")
+ public abstract byte[] getDelimiters();
+
+ public abstract String getFilePattern();
+
+ public abstract Compression getCompression();
+
+ public abstract SamplingStrategy getSamplingStrategy();
+
+ public abstract EmptyMatchTreatment getEmptyMatchTreatment();
+
+ public abstract FileIO.ReadMatches.DirectoryTreatment getDirectoryTreatment();
+
+ public static TextRowCountEstimator.Builder builder() {
+ return (new AutoValue_TextRowCountEstimator.Builder())
+ .setSamplingStrategy(DEFAULT_SAMPLING_STRATEGY)
+ .setNumSampledBytesPerFile(DEFAULT_NUM_BYTES_PER_FILE)
+ .setCompression(DEFAULT_COMPRESSION)
+ .setDirectoryTreatment(DEFAULT_DIRECTORY_TREATMENT)
+ .setEmptyMatchTreatment(DEFAULT_EMPTY_MATCH_TREATMENT);
+ }
+
+ /**
+ * Estimates the number of non empty rows. It samples NumSampledBytesPerFile bytes from every file
+ * until the condition in sampling strategy is met. Then it takes the average line size of the
+ * rows and divides the total file sizes by that number. If all the sampled rows are empty, and it
+ * has not sampled all the lines (due to sampling strategy) it throws Exception.
+ *
+ * @return Number of estimated rows.
+ * @throws org.apache.beam.sdk.io.TextRowCountEstimator.NoEstimationException if all the sampled
+ * lines are empty and we have not read all the lines in the matched files.
+ */
+ public Double estimateRowCount(PipelineOptions pipelineOptions)
+ throws IOException, NoEstimationException {
+ long linesSize = 0;
+ int numberOfReadLines = 0;
+ long totalFileSizes = 0;
+ long totalSampledBytes = 0;
+ int numberOfReadFiles = 0;
+ boolean sampledEverything = true;
+
+ MatchResult match = FileSystems.match(getFilePattern(), getEmptyMatchTreatment());
+
+ for (MatchResult.Metadata metadata : match.metadata()) {
+
+ if (getSamplingStrategy().stopSampling(numberOfReadFiles, totalSampledBytes)) {
+ sampledEverything = false;
+ break;
+ }
+
+ if (FileIO.ReadMatches.shouldSkipDirectory(metadata, getDirectoryTreatment())) {
+ continue;
+ }
+
+ FileIO.ReadableFile file = FileIO.ReadMatches.matchToReadableFile(metadata, getCompression());
+
+ // We use this as an estimate of the size of the sampled lines. Since the last sampled line
+ // may exceed this range, we are over estimating the number of lines in our estimation. (If
+ // each line is larger than readingWindowSize we will read one line any way and that line is
+ // the last line)
+ long readingWindowSize = Math.min(getNumSampledBytesPerFile(), metadata.sizeBytes());
+ sampledEverything = metadata.sizeBytes() == readingWindowSize && sampledEverything;
+ OffsetRange range = new OffsetRange(0, readingWindowSize);
+
+ TextSource textSource =
+ new TextSource(
+ ValueProvider.StaticValueProvider.of(file.getMetadata().resourceId().toString()),
+ getEmptyMatchTreatment(),
+ getDelimiters());
+ FileBasedSource<String> source =
+ CompressedSource.from(textSource).withCompression(file.getCompression());
+ try (BoundedSource.BoundedReader<String> reader =
+ source
+ .createForSubrangeOfFile(file.getMetadata(), range.getFrom(), range.getTo())
+ .createReader(pipelineOptions)) {
+
+ int numberOfNonEmptyLines = 0;
+ for (boolean more = reader.start(); more; more = reader.advance()) {
+ numberOfNonEmptyLines += reader.getCurrent().trim().equals("") ? 0 : 1;
+ }
+ numberOfReadLines += numberOfNonEmptyLines;
+ linesSize += (numberOfNonEmptyLines == 0) ? 0 : readingWindowSize;
+ }
+ long fileSize = metadata.sizeBytes();
+ numberOfReadFiles += fileSize == 0 ? 0 : 1;
+ totalFileSizes += fileSize;
+ }
+
+ if (numberOfReadLines == 0 && sampledEverything) {
+ return 0d;
+ }
+
+ if (numberOfReadLines == 0) {
+ throw new NoEstimationException(
+ "Cannot estimate the row count. All the sampled lines are empty");
+ }
+
+ // This is total file sizes divided by average line size.
+ return (double) totalFileSizes * numberOfReadLines / linesSize;
+ }
+
+ /** Builder for {@link org.apache.beam.sdk.io.TextRowCountEstimator}. */
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setNumSampledBytesPerFile(long numSampledBytes);
+
+ public abstract Builder setDirectoryTreatment(
+ FileIO.ReadMatches.DirectoryTreatment directoryTreatment);
+
+ public abstract Builder setCompression(Compression compression);
+
+ public abstract Builder setDelimiters(byte[] delimiters);
+
+ public abstract Builder setFilePattern(String filePattern);
+
+ public abstract Builder setEmptyMatchTreatment(EmptyMatchTreatment emptyMatchTreatment);
+
+ public abstract Builder setSamplingStrategy(SamplingStrategy samplingStrategy);
+
+ public abstract TextRowCountEstimator build();
+ }
+
+ /**
+ * An exception that will be thrown if the estimator cannot get an estimation of the number of
+ * lines.
+ */
+ public static class NoEstimationException extends Exception {
+ NoEstimationException(String message) {
+ super(message);
+ }
+ }
+
+ /** Sampling Strategy shows us when should we stop reading further files. * */
+ public interface SamplingStrategy {
+ boolean stopSampling(int numberOfFiles, long totalReadBytes);
+ }
+
+ /** This strategy samples all the files. */
+ public static class SampleAllFiles implements SamplingStrategy {
+
+ @Override
+ public boolean stopSampling(int numberOfSampledFiles, long totalReadBytes) {
+ return false;
+ }
+ }
+
+ /** This strategy stops sampling if we sample enough number of bytes. */
+ public static class LimitNumberOfFiles implements SamplingStrategy {
+ int limit;
+
+ public LimitNumberOfFiles(int limit) {
+ this.limit = limit;
+ }
+
+ @Override
+ public boolean stopSampling(int numberOfFiles, long totalReadBytes) {
+ return numberOfFiles > limit;
+ }
+ }
+
+ /**
+ * This strategy stops sampling when total number of sampled bytes are more than some threshold.
+ */
+ public static class LimitNumberOfTotalBytes implements SamplingStrategy {
+ long limit;
+
+ public LimitNumberOfTotalBytes(long limit) {
+ this.limit = limit;
+ }
+
+ @Override
+ public boolean stopSampling(int numberOfFiles, long totalReadBytes) {
+ return totalReadBytes > limit;
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSink.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSink.java
index e5119c0..179ff43 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSink.java
@@ -80,7 +80,6 @@
/** A {@link Writer Writer} for text files. */
private static class TextWriter<DestinationT> extends Writer<DestinationT, String> {
- private static final String NEWLINE = "\n";
@Nullable private final String header;
@Nullable private final String footer;
private final char[] delimiter;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java
index 8ce1fdb..5e2d0bf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.protobuf.ByteString;
import java.io.IOException;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.ValueProvider;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Implementation detail of {@link TextIO.Read}.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
index 2d56516..09d0580 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -77,12 +77,12 @@
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFilesResult.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFilesResult.java
index 7d1bd74..7e5e6be 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFilesResult.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFilesResult.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** The result of a {@link WriteFiles} transform. */
public class WriteFilesResult<DestinationT> implements POutput {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/ResourceIdTester.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/ResourceIdTester.java
index 92e5ce6..79eba30 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/ResourceIdTester.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/ResourceIdTester.java
@@ -19,14 +19,17 @@
import static org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions.RESOLVE_DIRECTORY;
import static org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions.RESOLVE_FILE;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
-import com.google.common.testing.EqualsTester;
import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -77,31 +80,58 @@
allResourceIds.add(dir2);
// ResourceIds in equality groups.
- new EqualsTester()
- .addEqualityGroup(file1)
- .addEqualityGroup(file2, file2a)
- .addEqualityGroup(dir1, dir1.getCurrentDirectory())
- .addEqualityGroup(dir2, dir2a, dir2.getCurrentDirectory())
- .addEqualityGroup(baseDirectory, file1.getCurrentDirectory(), file2.getCurrentDirectory())
- .testEquals();
+ assertEqualityGroups(
+ Arrays.asList(
+ Arrays.asList(file1),
+ Arrays.asList(file2, file2a),
+ Arrays.asList(dir1, dir1.getCurrentDirectory()),
+ Arrays.asList(dir2, dir2a, dir2.getCurrentDirectory()),
+ Arrays.asList(
+ baseDirectory, file1.getCurrentDirectory(), file2.getCurrentDirectory())));
// ResourceId toString() in equality groups.
- new EqualsTester()
- .addEqualityGroup(file1.toString())
- .addEqualityGroup(file2.toString(), file2a.toString())
- .addEqualityGroup(dir1.toString(), dir1.getCurrentDirectory().toString())
- .addEqualityGroup(dir2.toString(), dir2a.toString(), dir2.getCurrentDirectory().toString())
- .addEqualityGroup(
- baseDirectory.toString(),
- file1.getCurrentDirectory().toString(),
- file2.getCurrentDirectory().toString())
- .testEquals();
+ assertEqualityGroups(
+ Arrays.asList(
+ Arrays.asList(file1.toString()),
+ Arrays.asList(file2.toString(), file2a.toString()),
+ Arrays.asList(dir1.toString(), dir1.getCurrentDirectory().toString()),
+ Arrays.asList(dir2.toString(), dir2a.toString(), dir2.getCurrentDirectory().toString()),
+ Arrays.asList(
+ baseDirectory.toString(),
+ file1.getCurrentDirectory().toString(),
+ file2.getCurrentDirectory().toString())));
// TODO: test resolving strings that need to be escaped.
// Possible spec: https://tools.ietf.org/html/rfc3986#section-2
// May need options to be filesystem-independent, e.g., if filesystems ban certain chars.
}
+ /**
+ * Asserts that all elements in each group are equal to each other but not equal to any other
+ * element in another group.
+ */
+ private static <T> void assertEqualityGroups(List<List<T>> equalityGroups) {
+ for (int i = 0; i < equalityGroups.size(); ++i) {
+ List<T> current = equalityGroups.get(i);
+ for (int j = 0; j < current.size(); ++j) {
+ for (int k = 0; k < current.size(); ++k) {
+ assertEquals(
+ "Value at " + j + " should equal value at " + k + " in equality group " + i,
+ current.get(j),
+ current.get(k));
+ }
+ }
+ for (int j = 0; j < equalityGroups.size(); ++j) {
+ if (i == j) {
+ continue;
+ }
+ assertTrue(
+ current + " should not match any in " + equalityGroups.get(j),
+ Collections.disjoint(current, equalityGroups.get(j)));
+ }
+ }
+ }
+
private static void validateFailureResolvingIds(ResourceId baseDirectory) {
try {
ResourceId badFile = baseDirectory.resolve("file/", RESOLVE_FILE);
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKey.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKey.java
index 586acbf..f89ca7b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKey.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKey.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.range;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.protobuf.ByteString;
import com.google.protobuf.ByteString.ByteIterator;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
index 2a74ccf..42b8040 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRange.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.io.range;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verify;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verify;
import java.io.Serializable;
import java.math.BigDecimal;
@@ -29,8 +29,8 @@
import java.util.List;
import java.util.Objects;
import org.apache.beam.sdk.transforms.splittabledofn.HasDefaultTracker;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
index d3da59b..a54843c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/ByteKeyRangeTracker.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.range;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.toStringHelper;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.toStringHelper;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRange.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRange.java
index e398a9d..afe3102 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRange.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRange.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.range;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.InputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRangeTracker.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRangeTracker.java
index 39f39f8..b76ad3c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRangeTracker.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/range/OffsetRangeTracker.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.io.range;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricFiltering.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricFiltering.java
index bab5e81..1e64468 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricFiltering.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricFiltering.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.metrics;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/**
* Implements matching for metrics filters. Specifically, matching for metric name, namespace, and
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
index ae23ff0..4a83172 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricName.java
@@ -17,13 +17,13 @@
*/
package org.apache.beam.sdk.metrics;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
/**
* The name of a metric consists of a {@link #getNamespace} and a {@link #getName}. The {@link
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricNameFilter.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricNameFilter.java
index ea6a2ea..c32423f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricNameFilter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricNameFilter.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.metrics;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import javax.annotation.Nullable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java
index b75ef20..a3341c6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricQueryResults.java
@@ -21,7 +21,7 @@
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* The results of a query for metrics. Allows accessing all of the metrics that matched the filter.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsFilter.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsFilter.java
index bc4137c..dbdec82 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsFilter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsFilter.java
@@ -21,7 +21,7 @@
import java.util.Set;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/** Simple POJO representing a filter for querying metrics. */
@Experimental(Kind.METRICS)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/SourceMetrics.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/SourceMetrics.java
index ef3d1e4..e73aa02 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/SourceMetrics.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/SourceMetrics.java
@@ -19,7 +19,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
/** Standard {@link org.apache.beam.sdk.io.Source} Metrics. */
@Experimental(Kind.METRICS)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/DefaultPipelineOptionsRegistrar.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/DefaultPipelineOptionsRegistrar.java
index fb08f9a..62b54ba 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/DefaultPipelineOptionsRegistrar.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/DefaultPipelineOptionsRegistrar.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.options;
import com.google.auto.service.AutoService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link PipelineOptionsRegistrar} containing the {@link PipelineOptions} subclasses available by
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ExperimentalOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ExperimentalOptions.java
index 9abf7d6..b9825ca 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ExperimentalOptions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ExperimentalOptions.java
@@ -20,7 +20,7 @@
import java.util.List;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* Apache Beam provides a number of experimental features that can be enabled with this flag. If
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ManualDockerEnvironmentOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ManualDockerEnvironmentOptions.java
index 38c29a8..b329ddd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ManualDockerEnvironmentOptions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ManualDockerEnvironmentOptions.java
@@ -19,7 +19,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Pipeline options to tune DockerEnvironment. */
@Experimental
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
index e9d92f5..ae3cc77 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.transforms.display.HasDisplayData;
import org.apache.beam.sdk.util.ReleaseInfo;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.DateTimeUtils;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 2d7ff17..b887394 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -18,12 +18,13 @@
package org.apache.beam.sdk.options;
import static java.util.Locale.ROOT;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.beans.BeanInfo;
import java.beans.IntrospectionException;
import java.beans.Introspector;
@@ -46,7 +47,6 @@
import java.util.List;
import java.util.Map;
import java.util.NoSuchElementException;
-import java.util.ServiceLoader;
import java.util.Set;
import java.util.SortedMap;
import java.util.SortedSet;
@@ -65,30 +65,30 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.util.StringUtils;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.CaseFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSortedSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.RowSortedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SortedSetMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TreeBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TreeMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.CaseFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.RowSortedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SortedSetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeMultimap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -592,7 +592,7 @@
checkNotNull(iface);
CACHE.get().validateWellFormed(iface);
- Set<PipelineOptionSpec> properties = PipelineOptionsReflector.getOptionSpecs(iface);
+ Set<PipelineOptionSpec> properties = PipelineOptionsReflector.getOptionSpecs(iface, true);
RowSortedTable<Class<?>, String, Method> ifacePropGetterTable =
TreeBasedTable.create(ClassNameComparator.INSTANCE, Ordering.natural());
@@ -661,7 +661,7 @@
for (Class<? extends PipelineOptions> iface : ifaces) {
CACHE.get().validateWellFormed(iface);
- Set<PipelineOptionSpec> properties = PipelineOptionsReflector.getOptionSpecs(iface);
+ Set<PipelineOptionSpec> properties = PipelineOptionsReflector.getOptionSpecs(iface, false);
RowSortedTable<Class<?>, String, Method> ifacePropGetterTable =
TreeBasedTable.create(ClassNameComparator.INSTANCE, Ordering.natural());
@@ -1073,6 +1073,9 @@
FluentIterable.from(gettersWithTheAnnotation)
.transformAndConcat(
new Function<Method, Iterable<? extends Annotation>>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Nonnull
@Override
public Iterable<? extends Annotation> apply(@Nonnull Method method) {
@@ -1089,6 +1092,9 @@
FluentIterable.from(gettersWithTheAnnotation)
.transformAndConcat(
new Function<Method, Iterable<String>>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Nonnull
@Override
public Iterable<String> apply(final @Nonnull Method method) {
@@ -1096,6 +1102,10 @@
.filter(annotationPredicates.forAnnotation)
.transform(
new Function<Annotation, String>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification =
+ "https://github.com/google/guava/issues/920")
@Nonnull
@Override
public String apply(@Nonnull Annotation annotation) {
@@ -1449,8 +1459,11 @@
private static class ReturnTypeFetchingFunction implements Function<Method, Class<?>> {
static final ReturnTypeFetchingFunction INSTANCE = new ReturnTypeFetchingFunction();
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public Class<?> apply(Method input) {
+ public Class<?> apply(@Nonnull Method input) {
return input.getReturnType();
}
}
@@ -1459,8 +1472,11 @@
private static class MethodToDeclaringClassFunction implements Function<Method, Class<?>> {
static final MethodToDeclaringClassFunction INSTANCE = new MethodToDeclaringClassFunction();
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
- public Class<?> apply(Method input) {
+ public Class<?> apply(@Nonnull Method input) {
return input.getDeclaringClass();
}
}
@@ -1784,15 +1800,11 @@
private Cache() {
final ClassLoader loader = ReflectHelpers.findClassLoader();
-
- Set<PipelineRunnerRegistrar> pipelineRunnerRegistrars =
- Sets.newTreeSet(ReflectHelpers.ObjectsClassComparator.INSTANCE);
- pipelineRunnerRegistrars.addAll(
- Lists.newArrayList(ServiceLoader.load(PipelineRunnerRegistrar.class, loader)));
// Store the list of all available pipeline runners.
ImmutableMap.Builder<String, Class<? extends PipelineRunner<?>>> builder =
ImmutableMap.builder();
- for (PipelineRunnerRegistrar registrar : pipelineRunnerRegistrars) {
+ for (PipelineRunnerRegistrar registrar :
+ ReflectHelpers.loadServicesOrdered(PipelineRunnerRegistrar.class, loader)) {
for (Class<? extends PipelineRunner<?>> klass : registrar.getPipelineRunners()) {
String runnerName = klass.getSimpleName().toLowerCase();
builder.put(runnerName, klass);
@@ -1807,12 +1819,8 @@
/** Load and register the list of all classes that extend PipelineOptions. */
private void initializeRegistry(final ClassLoader loader) {
- register(PipelineOptions.class);
- Set<PipelineOptionsRegistrar> pipelineOptionsRegistrars =
- Sets.newTreeSet(ReflectHelpers.ObjectsClassComparator.INSTANCE);
- pipelineOptionsRegistrars.addAll(
- Lists.newArrayList(ServiceLoader.load(PipelineOptionsRegistrar.class, loader)));
- for (PipelineOptionsRegistrar registrar : pipelineOptionsRegistrars) {
+ for (PipelineOptionsRegistrar registrar :
+ ReflectHelpers.loadServicesOrdered(PipelineOptionsRegistrar.class, loader)) {
for (Class<? extends PipelineOptions> klass : registrar.getPipelineOptions()) {
register(klass);
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsReflector.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsReflector.java
index 5d808b7..5a9e9db 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsReflector.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsReflector.java
@@ -22,21 +22,22 @@
import java.util.Map;
import java.util.Set;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
/** Utilities to reflect over {@link PipelineOptions}. */
class PipelineOptionsReflector {
private PipelineOptionsReflector() {}
/**
- * Retrieve metadata for the full set of pipeline options visible within the type hierarchy of a
- * single {@link PipelineOptions} interface.
+ * Retrieve metadata for the full set of pipeline options within the type hierarchy of a single
+ * {@link PipelineOptions} interface with optional filtering {@link Hidden} options.
*
* @see PipelineOptionsReflector#getOptionSpecs(Iterable)
*/
- static Set<PipelineOptionSpec> getOptionSpecs(Class<? extends PipelineOptions> optionsInterface) {
+ static Set<PipelineOptionSpec> getOptionSpecs(
+ Class<? extends PipelineOptions> optionsInterface, boolean skipHidden) {
Iterable<Method> methods = ReflectHelpers.getClosureOfMethodsOnInterface(optionsInterface);
Multimap<String, Method> propsToGetters = getPropertyNamesToGetters(methods);
@@ -53,7 +54,7 @@
continue;
}
- if (declaringClass.isAnnotationPresent(Hidden.class)) {
+ if (skipHidden && declaringClass.isAnnotationPresent(Hidden.class)) {
continue;
}
@@ -77,7 +78,7 @@
Iterable<Class<? extends PipelineOptions>> optionsInterfaces) {
ImmutableSet.Builder<PipelineOptionSpec> setBuilder = ImmutableSet.builder();
for (Class<? extends PipelineOptions> optionsInterface : optionsInterfaces) {
- setBuilder.addAll(getOptionSpecs(optionsInterface));
+ setBuilder.addAll(getOptionSpecs(optionsInterface, true));
}
return setBuilder.build();
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsValidator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsValidator.java
index 4345c11..babdffc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsValidator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsValidator.java
@@ -17,18 +17,18 @@
*/
package org.apache.beam.sdk.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.Collection;
import org.apache.beam.sdk.options.Validation.Required;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Collections2;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SortedSetMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.TreeMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Collections2;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SortedSetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TreeMultimap;
/** Validates that the {@link PipelineOptions} conforms to all the {@link Validation} criteria. */
public class PipelineOptionsValidator {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
index 4c28b90..2eb4d56 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java
@@ -80,4 +80,29 @@
int getEnvironmentCacheMillis();
void setEnvironmentCacheMillis(int environmentCacheMillis);
+
+ @Description("Duration in milliseconds for environment expiration. 0 means no expiration.")
+ @Default.Integer(0)
+ int getEnvironmentExpirationMillis();
+
+ void setEnvironmentExpirationMillis(int environmentExpirationMillis);
+
+ @Description("The output path for the executable file to be created.")
+ String getOutputExecutablePath();
+
+ void setOutputExecutablePath(String outputExecutablePath);
+
+ /** Enumeration of the different implementations of the artifact retrieval service. */
+ enum RetrievalServiceType {
+ /** Artifacts are to be retrieved from a {@link org.apache.beam.sdk.io.FileSystem}. */
+ FILE_SYSTEM,
+ /** Artifacts are to be retrieved from the runtime {@link ClassLoader}. */
+ CLASSLOADER,
+ }
+
+ @Description("The artifact retrieval service to be used.")
+ @Default.Enum("FILE_SYSTEM")
+ RetrievalServiceType getRetrievalServiceType();
+
+ void setRetrievalServiceType(RetrievalServiceType retrievalServiceType);
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
index df2df86..ea4a5be 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.core.JsonGenerator;
@@ -64,18 +64,18 @@
import org.apache.beam.sdk.transforms.display.HasDisplayData;
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Defaults;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ClassToInstanceMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MutableClassToInstanceMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Defaults;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ClassToInstanceMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.MutableClassToInstanceMap;
/**
- * Represents and {@link InvocationHandler} for a {@link Proxy}. The invocation handler uses bean
+ * Represents an {@link InvocationHandler} for a {@link Proxy}. The invocation handler uses bean
* introspection of the proxy class to store and retrieve values based off of the property name.
*
* <p>Unset properties use the {@code @Default} metadata on the getter to return values. If there is
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java
new file mode 100644
index 0000000..caa447b
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Options that are used to control configuration of the remote environment. */
+@Experimental
+@Hidden
+public interface RemoteEnvironmentOptions extends PipelineOptions {
+
+ // The default should be null (no default), so that the environment can pick its suitable tmp
+ // directory when nothing is specified by the user
+ @Description("Local semi-persistent directory")
+ String getSemiPersistDir();
+
+ void setSemiPersistDir(String value);
+
+ /** Register the {@link RemoteEnvironmentOptions}. */
+ @AutoService(PipelineOptionsRegistrar.class)
+ class Options implements PipelineOptionsRegistrar {
+ @Override
+ public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
+ return ImmutableList.of(RemoteEnvironmentOptions.class);
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java
index 701087a..3bd428e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.annotation.JsonCreator;
import java.util.Arrays;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
index a8cbaec..92f0644 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonParser;
@@ -43,7 +43,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A {@link ValueProvider} abstracts the notion of fetching a value that may or may not be currently
@@ -230,6 +230,7 @@
Method method = klass.getMethod(methodName);
PipelineOptions methodOptions = options.as(klass);
InvocationHandler handler = Proxy.getInvocationHandler(methodOptions);
+ @SuppressWarnings("unchecked")
ValueProvider<T> result = (ValueProvider<T>) handler.invoke(methodOptions, method, null);
// Two cases: If we have deserialized a new value from JSON, it will
// be wrapped in a StaticValueProvider, which we can provide here. If
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProviders.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProviders.java
index cb7d32a..1617860 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProviders.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProviders.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.io.IOException;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/PTransformMatcher.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/PTransformMatcher.java
index 463aefa..c7d17b5 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/PTransformMatcher.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/PTransformMatcher.java
@@ -34,7 +34,7 @@
/**
* An {@link AppliedPTransform} matched by a {@link PTransformMatcher} will be replaced during
- * pipeline surgery, and is often expected to be gone the new pipeline. For the {@link
+ * pipeline surgery, and is often expected to be gone in the new pipeline. For the {@link
* AppliedPTransform} that is expected to remain in the pipeline after surgery, the corresponding
* {@link PTransformMatcher} should override this method, such that it will not be matched during
* the validation.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
index 5468537..7ba5d83 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/runners/TransformHierarchy.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.runners;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collections;
@@ -42,10 +42,10 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
index 91659c8..1a11a62 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
import org.apache.beam.sdk.schemas.utils.ReflectUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** A {@link SchemaProvider} for AutoValue classes. */
public class AutoValueSchema extends GetterBasedSchemaProvider {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldAccessDescriptor.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldAccessDescriptor.java
index 55b9939..15cea8d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldAccessDescriptor.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldAccessDescriptor.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.schemas;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoOneOf;
import com.google.auto.value.AutoValue;
@@ -45,14 +45,14 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.schemas.parser.FieldAccessDescriptorParser;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* Used inside of a {@link org.apache.beam.sdk.transforms.DoFn} to describe which fields in a schema
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
index 5ec41f3..13835a0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldTypeDescriptors.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.ParameterizedType;
import java.util.Collection;
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableBiMap;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
index 983195f..85f688a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldValueTypeInformation.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
index 1673b24..2f731dc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FromRowUsingCreator.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.schemas;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.lang.reflect.Type;
import java.util.List;
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.RowWithGetters;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Function to convert a {@link Row} to a user type using a creator factory. */
class FromRowUsingCreator<T> implements SerializableFunction<Row, T> {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
index 176f97d..8540b3d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaBeanSchema.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.schemas.utils.JavaBeanUtils;
import org.apache.beam.sdk.schemas.utils.ReflectUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link SchemaProvider} for Java Bean objects.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java
index cba5448..1d63cd7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/JavaFieldSchema.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.schemas.utils.POJOUtils;
import org.apache.beam.sdk.schemas.utils.ReflectUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link SchemaProvider} for Java POJO objects.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
index 0b903fc..8aead16 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/LogicalTypes.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import org.apache.beam.sdk.schemas.Schema.FieldType;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index d659b25..e096887 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -36,12 +36,12 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBiMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** {@link Schema} describes the fields in {@link Row}. */
@Experimental(Kind.SCHEMAS)
@@ -302,7 +302,7 @@
return equivalent(other, EquivalenceNullablePolicy.WEAKEN);
}
- /** Returns true if this Schema can be assigned to another Schema, igmoring nullable. * */
+ /** Returns true if this Schema can be assigned to another Schema, ignoring nullable. * */
public boolean assignableToIgnoreNullable(Schema other) {
return equivalent(other, EquivalenceNullablePolicy.IGNORE);
}
@@ -334,10 +334,11 @@
@Override
public String toString() {
StringBuilder builder = new StringBuilder();
- builder.append("Fields:\n");
+ builder.append("Fields:");
+ builder.append(System.lineSeparator());
for (Field field : fields) {
builder.append(field);
- builder.append("\n");
+ builder.append(System.lineSeparator());
}
return builder.toString();
};
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
index 0199534..8b2e126 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
@@ -20,28 +20,74 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
+import java.util.Objects;
+import java.util.UUID;
+import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.coders.BigDecimalCoder;
+import org.apache.beam.sdk.coders.BigEndianShortCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.ByteCoder;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.DoubleCoder;
+import org.apache.beam.sdk.coders.FloatCoder;
+import org.apache.beam.sdk.coders.InstantCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.coders.RowCoderGenerator;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
+import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** {@link SchemaCoder} is used as the coder for types that have schemas registered. */
@Experimental(Kind.SCHEMAS)
public class SchemaCoder<T> extends CustomCoder<T> {
- private final RowCoder rowCoder;
+
+ // This contains a map of primitive types to their coders.
+ public static final ImmutableMap<TypeName, Coder> CODER_MAP =
+ ImmutableMap.<TypeName, Coder>builder()
+ .put(TypeName.BYTE, ByteCoder.of())
+ .put(TypeName.BYTES, ByteArrayCoder.of())
+ .put(TypeName.INT16, BigEndianShortCoder.of())
+ .put(TypeName.INT32, VarIntCoder.of())
+ .put(TypeName.INT64, VarLongCoder.of())
+ .put(TypeName.DECIMAL, BigDecimalCoder.of())
+ .put(TypeName.FLOAT, FloatCoder.of())
+ .put(TypeName.DOUBLE, DoubleCoder.of())
+ .put(TypeName.STRING, StringUtf8Coder.of())
+ .put(TypeName.DATETIME, InstantCoder.of())
+ .put(TypeName.BOOLEAN, BooleanCoder.of())
+ .build();
+
+ protected final Schema schema;
private final SerializableFunction<T, Row> toRowFunction;
private final SerializableFunction<Row, T> fromRowFunction;
+ @Nullable private transient Coder<Row> delegateCoder;
- private SchemaCoder(
+ protected SchemaCoder(
Schema schema,
SerializableFunction<T, Row> toRowFunction,
SerializableFunction<Row, T> fromRowFunction) {
+ if (schema.getUUID() == null) {
+ // Clone the schema before modifying the Java object.
+ schema = SerializableUtils.clone(schema);
+ setSchemaIds(schema);
+ }
this.toRowFunction = toRowFunction;
this.fromRowFunction = fromRowFunction;
- this.rowCoder = RowCoder.of(schema);
+ this.schema = schema;
}
/**
@@ -55,15 +101,31 @@
return new SchemaCoder<>(schema, toRowFunction, fromRowFunction);
}
- /** Returns a {@link SchemaCoder} for {@link Row} classes. */
+ /** Returns a {@link SchemaCoder} for {@link Row} instances with the given {@code schema}. */
public static SchemaCoder<Row> of(Schema schema) {
- return new SchemaCoder<>(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ return RowCoder.of(schema);
+ }
+
+ /** Returns the coder used for a given primitive type. */
+ public static <T> Coder<T> coderForFieldType(FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ROW:
+ return (Coder<T>) SchemaCoder.of(fieldType.getRowSchema());
+ case ARRAY:
+ return (Coder<T>) ListCoder.of(coderForFieldType(fieldType.getCollectionElementType()));
+ case MAP:
+ return (Coder<T>)
+ MapCoder.of(
+ coderForFieldType(fieldType.getMapKeyType()),
+ coderForFieldType(fieldType.getMapValueType()));
+ default:
+ return (Coder<T>) CODER_MAP.get(fieldType.getTypeName());
+ }
}
/** Returns the schema associated with this type. */
public Schema getSchema() {
- return rowCoder.getSchema();
+ return schema;
}
/** Returns the toRow conversion function. */
@@ -76,28 +138,106 @@
return toRowFunction;
}
+ private Coder<Row> getDelegateCoder() {
+ if (delegateCoder == null) {
+ // RowCoderGenerator caches based on id, so if a new instance of this RowCoder is
+ // deserialized, we don't need to run ByteBuddy again to construct the class.
+ delegateCoder = RowCoderGenerator.generate(schema);
+ }
+ return delegateCoder;
+ }
+
@Override
public void encode(T value, OutputStream outStream) throws IOException {
- rowCoder.encode(toRowFunction.apply(value), outStream);
+ getDelegateCoder().encode(toRowFunction.apply(value), outStream);
}
@Override
public T decode(InputStream inStream) throws IOException {
- return fromRowFunction.apply(rowCoder.decode(inStream));
+ return fromRowFunction.apply(getDelegateCoder().decode(inStream));
}
@Override
- public void verifyDeterministic() throws NonDeterministicException {
- rowCoder.verifyDeterministic();
+ public void verifyDeterministic()
+ throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
+ verifyDeterministic(schema);
+ }
+
+ private void verifyDeterministic(Schema schema)
+ throws org.apache.beam.sdk.coders.Coder.NonDeterministicException {
+
+ ImmutableList<Coder<?>> coders =
+ schema.getFields().stream()
+ .map(Field::getType)
+ .map(SchemaCoder::coderForFieldType)
+ .collect(ImmutableList.toImmutableList());
+
+ Coder.verifyDeterministic(this, "All fields must have deterministic encoding", coders);
}
@Override
public boolean consistentWithEquals() {
- return rowCoder.consistentWithEquals();
+ return true;
}
@Override
public String toString() {
- return "SchemaCoder: " + rowCoder.toString();
+ return "SchemaCoder<Schema: "
+ + schema
+ + " UUID: "
+ + schema.getUUID()
+ + " delegateCoder: "
+ + getDelegateCoder();
+ }
+
+ // Sets the schema id, and then recursively ensures that all schemas have ids set.
+ private static void setSchemaIds(Schema schema) {
+ if (schema.getUUID() == null) {
+ schema.setUUID(UUID.randomUUID());
+ }
+ for (Field field : schema.getFields()) {
+ setSchemaIds(field.getType());
+ }
+ }
+
+ private static void setSchemaIds(FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ROW:
+ setSchemaIds(fieldType.getRowSchema());
+ return;
+ case MAP:
+ setSchemaIds(fieldType.getMapKeyType());
+ setSchemaIds(fieldType.getMapValueType());
+ return;
+ case LOGICAL_TYPE:
+ setSchemaIds(fieldType.getLogicalType().getBaseType());
+ return;
+
+ case ARRAY:
+ setSchemaIds(fieldType.getCollectionElementType());
+ return;
+
+ default:
+ return;
+ }
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SchemaCoder<?> that = (SchemaCoder<?>) o;
+ return schema.equals(that.schema)
+ && toRowFunction.equals(that.toRowFunction)
+ && fromRowFunction.equals(that.fromRowFunction);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(schema, toRowFunction, fromRowFunction);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
index 8074500..f190623 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
@@ -32,10 +32,10 @@
import org.apache.beam.sdk.util.common.ReflectHelpers.ObjectsClassComparator;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A {@link SchemaRegistry} allows registering {@link Schema}s for a given Java {@link Class} or a
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeConstructorCreator.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeConstructorCreator.java
index 3890f06..ed62ac7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeConstructorCreator.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUserTypeConstructorCreator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/DefaultSchema.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/DefaultSchema.java
index d3b7d10..61c38fa 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/DefaultSchema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/DefaultSchema.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.annotations;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.lang.annotation.Documented;
@@ -38,8 +38,8 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* The {@link DefaultSchema} annotation specifies a {@link SchemaProvider} class to handle obtaining
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/parser/FieldAccessDescriptorParser.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/parser/FieldAccessDescriptorParser.java
index a418a6e..68a5278 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/parser/FieldAccessDescriptorParser.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/parser/FieldAccessDescriptorParser.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.parser;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.List;
import java.util.stream.Collectors;
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.schemas.parser.generated.FieldSpecifierNotationParser.QualifyComponentContext;
import org.apache.beam.sdk.schemas.parser.generated.FieldSpecifierNotationParser.SimpleIdentifierContext;
import org.apache.beam.sdk.schemas.parser.generated.FieldSpecifierNotationParser.WildcardContext;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** Parser for textual field-access selector. */
public class FieldAccessDescriptorParser {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/AddFields.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/AddFields.java
index 1019d0e..435c635 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/AddFields.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/AddFields.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.schemas.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
@@ -40,12 +40,12 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimaps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimaps;
/**
* A transform to add new nullable fields to a PCollection's schema. Elements are extended to have
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Cast.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Cast.java
index 8eb936e..b59db34 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Cast.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Cast.java
@@ -38,9 +38,9 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Set of utilities for casting rows between schemas. */
@Experimental(Experimental.Kind.SCHEMAS)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/CoGroup.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/CoGroup.java
index 5f95855..0123347 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/CoGroup.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/CoGroup.java
@@ -47,9 +47,9 @@
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* A transform that performs equijoins across multiple schema {@link PCollection}s.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
index b137c6a..1625cf2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Convert.java
@@ -27,6 +27,7 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
@@ -126,27 +127,41 @@
ConvertHelpers.getConvertedSchemaInformation(
input.getSchema(), outputTypeDescriptor, registry);
boolean unbox = converted.unboxedType != null;
- PCollection<OutputT> output =
- input.apply(
- ParDo.of(
- new DoFn<InputT, OutputT>() {
- @ProcessElement
- public void processElement(@Element Row row, OutputReceiver<OutputT> o) {
- // Read the row, potentially unboxing if necessary.
- Object input = unbox ? row.getValue(0) : row;
- // The output has a schema, so we need to convert to the appropriate type.
- o.output(converted.outputSchemaCoder.getFromRowFunction().apply((Row) input));
- }
- }));
+ PCollection<OutputT> output;
if (converted.outputSchemaCoder != null) {
output =
+ input.apply(
+ ParDo.of(
+ new DoFn<InputT, OutputT>() {
+ @ProcessElement
+ public void processElement(@Element Row row, OutputReceiver<OutputT> o) {
+ // Read the row, potentially unboxing if necessary.
+ Object input = unbox ? row.getValue(0) : row;
+ // The output has a schema, so we need to convert to the appropriate type.
+ o.output(
+ converted.outputSchemaCoder.getFromRowFunction().apply((Row) input));
+ }
+ }));
+ output =
output.setSchema(
converted.outputSchemaCoder.getSchema(),
converted.outputSchemaCoder.getToRowFunction(),
converted.outputSchemaCoder.getFromRowFunction());
} else {
- // TODO: Support full unboxing and boxing in Create.
- throw new RuntimeException("Unboxing is not yet supported in the Create transform");
+ SerializableFunction<?, OutputT> convertPrimitive =
+ ConvertHelpers.getConvertPrimitive(converted.unboxedType, outputTypeDescriptor);
+ output =
+ input.apply(
+ ParDo.of(
+ new DoFn<InputT, OutputT>() {
+ @ProcessElement
+ public void processElement(@Element Row row, OutputReceiver<OutputT> o) {
+ o.output(convertPrimitive.apply(row.getValue(0)));
+ }
+ }));
+
+ output.setTypeDescriptor(outputTypeDescriptor);
+ // TODO: Support boxing in Convert (e.g. Long -> Row with Schema { Long }).
}
return output;
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/DropFields.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/DropFields.java
index d1c4d12..c3b039b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/DropFields.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/DropFields.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
import java.util.Set;
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A transform to drop fields from a schema.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Filter.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Filter.java
index 5f56261..60f86b1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Filter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Filter.java
@@ -17,20 +17,23 @@
*/
package org.apache.beam.sdk.schemas.transforms;
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
import java.util.List;
-import java.util.Map;
import java.util.stream.Collectors;
+import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.SelectHelpers;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* A {@link PTransform} for filtering a collection of schema types.
@@ -84,95 +87,128 @@
/** Implementation of the filter. */
public static class Inner<T> extends PTransform<PCollection<T>, PCollection<T>> {
- private final Map<String, SerializableFunction<?, Boolean>> fieldNameFilters =
- Maps.newHashMap();
- private final Map<Integer, SerializableFunction<?, Boolean>> fieldIdFilters = Maps.newHashMap();
- private final Map<List<String>, SerializableFunction<Row, Boolean>> fieldNamesFilters =
- Maps.newHashMap();
- private final Map<List<Integer>, SerializableFunction<Row, Boolean>> fieldIdsFilters =
- Maps.newHashMap();
+ @AutoValue
+ abstract static class FilterDescription<FieldT> implements Serializable {
+ abstract FieldAccessDescriptor getFieldAccessDescriptor();
+
+ abstract SerializableFunction<FieldT, Boolean> getPredicate();
+
+ @Nullable
+ abstract Schema getSelectedSchema();
+
+ abstract boolean getSelectsSingleField();
+
+ abstract Builder<FieldT> toBuilder();
+
+ @AutoValue.Builder
+ abstract static class Builder<FieldT> {
+ abstract Builder<FieldT> setFieldAccessDescriptor(
+ FieldAccessDescriptor fieldAccessDescriptor);
+
+ abstract Builder<FieldT> setPredicate(SerializableFunction<FieldT, Boolean> predicate);
+
+ abstract Builder<FieldT> setSelectedSchema(@Nullable Schema selectedSchema);
+
+ abstract Builder<FieldT> setSelectsSingleField(boolean unbox);
+
+ abstract FilterDescription<FieldT> build();
+ }
+ }
+
+ private final List<FilterDescription<?>> filters = Lists.newArrayList();
/** Set a predicate based on the value of a field, where the field is specified by name. */
- public Inner<T> whereFieldName(String fieldName, SerializableFunction<?, Boolean> predicate) {
- fieldNameFilters.put(fieldName, predicate);
+ public <FieldT> Inner<T> whereFieldName(
+ String fieldName, SerializableFunction<FieldT, Boolean> predicate) {
+ filters.add(
+ new AutoValue_Filter_Inner_FilterDescription.Builder<FieldT>()
+ .setFieldAccessDescriptor(FieldAccessDescriptor.withFieldNames(fieldName))
+ .setPredicate(predicate)
+ .setSelectsSingleField(true)
+ .build());
return this;
}
/** Set a predicate based on the value of a field, where the field is specified by id. */
- public Inner<T> whereFieldId(int fieldId, SerializableFunction<?, Boolean> predicate) {
- fieldIdFilters.put(fieldId, predicate);
+ public <FieldT> Inner<T> whereFieldId(
+ int fieldId, SerializableFunction<FieldT, Boolean> predicate) {
+ filters.add(
+ new AutoValue_Filter_Inner_FilterDescription.Builder<FieldT>()
+ .setFieldAccessDescriptor(FieldAccessDescriptor.withFieldIds(fieldId))
+ .setPredicate(predicate)
+ .setSelectsSingleField(true)
+ .build());
return this;
}
/** Set a predicate based on the value of multipled fields, specified by name. */
public Inner<T> whereFieldNames(
List<String> fieldNames, SerializableFunction<Row, Boolean> predicate) {
- fieldNamesFilters.put(fieldNames, predicate);
+ filters.add(
+ new AutoValue_Filter_Inner_FilterDescription.Builder<Row>()
+ .setFieldAccessDescriptor(FieldAccessDescriptor.withFieldNames(fieldNames))
+ .setPredicate(predicate)
+ .setSelectsSingleField(false)
+ .build());
return this;
}
/** Set a predicate based on the value of multipled fields, specified by id. */
public Inner<T> whereFieldIds(
List<Integer> fieldIds, SerializableFunction<Row, Boolean> predicate) {
- fieldIdsFilters.put(fieldIds, predicate);
+ filters.add(
+ new AutoValue_Filter_Inner_FilterDescription.Builder<Row>()
+ .setFieldAccessDescriptor(FieldAccessDescriptor.withFieldIds(fieldIds))
+ .setPredicate(predicate)
+ .setSelectsSingleField(false)
+ .build());
return this;
}
@Override
public PCollection<T> expand(PCollection<T> input) {
- // Validate that all referenced fields are in the schema.
- Schema schema = input.getSchema();
- for (String fieldName :
- Sets.union(
- fieldNameFilters.keySet(),
- fieldNamesFilters.keySet().stream()
- .flatMap(List::stream)
- .collect(Collectors.toSet()))) {
- schema.getField(fieldName);
- }
- for (int fieldIndex :
- Sets.union(
- fieldIdFilters.keySet(),
- fieldIdsFilters.keySet().stream()
- .flatMap(List::stream)
- .collect(Collectors.toSet()))) {
- if (fieldIndex >= schema.getFieldCount() || fieldIndex < 0) {
- throw new IllegalArgumentException(
- "Field index " + fieldIndex + " does not exist in the schema.");
- }
- }
-
- // TODO: Once BEAM-4457 is fixed, tag this ParDo with a FieldAccessDescriptor so that Beam
- // knows which fields are being accessed.
+ Schema inputSchema = input.getSchema();
+ List<FilterDescription> resolvedFilters =
+ filters.stream()
+ .map(
+ f ->
+ f.toBuilder()
+ .setFieldAccessDescriptor(
+ f.getFieldAccessDescriptor().resolve(inputSchema))
+ .build())
+ .map(
+ f ->
+ f.toBuilder()
+ .setSelectedSchema(
+ SelectHelpers.getOutputSchema(
+ inputSchema, f.getFieldAccessDescriptor()))
+ .build())
+ .collect(Collectors.toList());
return input.apply(
ParDo.of(
new DoFn<T, T>() {
@ProcessElement
public void process(@Element Row row, OutputReceiver<Row> o) {
- for (Map.Entry<String, SerializableFunction<?, Boolean>> entry :
- fieldNameFilters.entrySet()) {
- if (!entry.getValue().apply(row.getValue(entry.getKey()))) {
- return;
- }
- }
-
- for (Map.Entry<Integer, SerializableFunction<?, Boolean>> entry :
- fieldIdFilters.entrySet()) {
- if (!entry.getValue().apply(row.getValue(entry.getKey()))) {
- return;
- }
- }
-
- for (SerializableFunction<Row, Boolean> predicate : fieldNamesFilters.values()) {
- if (!predicate.apply(row)) {
- return;
- }
- }
-
- for (SerializableFunction<Row, Boolean> predicate : fieldIdsFilters.values()) {
- if (!predicate.apply(row)) {
- return;
+ for (FilterDescription filter : resolvedFilters) {
+ Row selected =
+ SelectHelpers.selectRow(
+ row,
+ filter.getFieldAccessDescriptor(),
+ inputSchema,
+ filter.getSelectedSchema());
+ if (filter.getSelectsSingleField()) {
+ SerializableFunction<Object, Boolean> predicate =
+ (SerializableFunction<Object, Boolean>) filter.getPredicate();
+ if (!predicate.apply(selected.getValue(0))) {
+ return;
+ }
+ } else {
+ SerializableFunction<Row, Boolean> predicate =
+ (SerializableFunction<Row, Boolean>) filter.getPredicate();
+ if (!predicate.apply(selected)) {
+ return;
+ }
}
}
// All filters passed. Output the row.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/RenameFields.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/RenameFields.java
index 07d8499..a851034 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/RenameFields.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/RenameFields.java
@@ -33,11 +33,11 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.apache.commons.compress.utils.Lists;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
index 77db218..b82ecaf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaAggregateFn.java
@@ -27,7 +27,6 @@
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.schemas.FieldAccessDescriptor;
import org.apache.beam.sdk.schemas.FieldTypeDescriptors;
import org.apache.beam.sdk.schemas.Schema;
@@ -43,7 +42,7 @@
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** This is the builder used by {@link Group} to build up a composed {@link CombineFn}. */
@Experimental(Kind.SCHEMAS)
@@ -158,11 +157,11 @@
if (fieldAggregation.unnestedInputSubSchema.getFieldCount() == 1) {
extractFunction = new ExtractSingleFieldFunction<>(fieldAggregation, toRowFunction);
extractOutputCoder =
- RowCoder.coderForFieldType(
+ SchemaCoder.coderForFieldType(
fieldAggregation.unnestedInputSubSchema.getField(0).getType());
} else {
extractFunction = new ExtractFieldsFunction<>(fieldAggregation, toRowFunction);
- extractOutputCoder = RowCoder.of(fieldAggregation.inputSubSchema);
+ extractOutputCoder = SchemaCoder.of(fieldAggregation.inputSubSchema);
}
if (i == 0) {
composedCombineFn =
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
index a94e77e..cfe40b1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* A {@link PTransform} to unnest nested rows.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
index 0dec260..565b12b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
@@ -31,26 +31,6 @@
import java.util.function.Function;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
-import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
-import net.bytebuddy.implementation.bytecode.Duplication;
-import net.bytebuddy.implementation.bytecode.Removal;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.TypeCreation;
-import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
-import net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
-import net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
@@ -59,7 +39,27 @@
import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.InjectPackageStrategy;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Duplication;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Removal;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.TypeCreation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/** Utilities for managing AutoValue schemas. */
public class AutoValueUtils {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
index fc9ae0b..cd320ae 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
@@ -20,17 +20,6 @@
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.util.Map;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.implementation.MethodCall;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
-import net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
-import net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.avro.specific.SpecificRecord;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
@@ -39,7 +28,18 @@
import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.MethodCall;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
class AvroByteBuddyUtils {
private static final ByteBuddy BYTE_BUDDY = new ByteBuddy();
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
index 5d897f3..7b43961 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.schemas.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Method;
import java.math.BigDecimal;
@@ -36,7 +36,6 @@
import org.apache.avro.Schema.Type;
import org.apache.avro.data.TimeConversions;
import org.apache.avro.generic.GenericData;
-import org.apache.avro.generic.GenericEnumSymbol;
import org.apache.avro.generic.GenericFixed;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.GenericRecordBuilder;
@@ -47,6 +46,7 @@
import org.apache.avro.specific.SpecificRecord;
import org.apache.avro.util.Utf8;
import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.schemas.AvroRecordSchema;
import org.apache.beam.sdk.schemas.FieldValueGetter;
import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
@@ -55,13 +55,17 @@
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.schemas.SchemaUserTypeCreator;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.CaseFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.CaseFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.joda.time.Days;
+import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
@@ -158,11 +162,26 @@
}
/** Convert to an AVRO type. */
- public org.apache.avro.Schema toAvroType() {
- return org.apache.avro.Schema.createFixed(null, "", "", size);
+ public org.apache.avro.Schema toAvroType(String name, String namespace) {
+ return org.apache.avro.Schema.createFixed(name, null, namespace, size);
}
}
+ /** Get Beam Field from avro Field. */
+ public static Schema.Field toBeamField(org.apache.avro.Schema.Field field) {
+ TypeWithNullability nullableType = new TypeWithNullability(field.schema());
+ FieldType beamFieldType = toFieldType(nullableType);
+ return Field.of(field.name(), beamFieldType);
+ }
+
+ /** Get Avro Field from Beam Field. */
+ public static org.apache.avro.Schema.Field toAvroField(Schema.Field field, String namespace) {
+ org.apache.avro.Schema fieldSchema =
+ getFieldSchema(field.getType(), field.getName(), namespace);
+ return new org.apache.avro.Schema.Field(
+ field.getName(), fieldSchema, field.getDescription(), (Object) null);
+ }
+
private AvroUtils() {}
/**
@@ -174,8 +193,7 @@
Schema.Builder builder = Schema.builder();
for (org.apache.avro.Schema.Field field : schema.getFields()) {
- TypeWithNullability nullableType = new TypeWithNullability(field.schema());
- Field beamField = Field.of(field.name(), toFieldType(nullableType));
+ Field beamField = toBeamField(field);
if (field.doc() != null) {
beamField = beamField.withDescription(field.doc());
}
@@ -186,17 +204,22 @@
}
/** Converts a Beam Schema into an AVRO schema. */
- public static org.apache.avro.Schema toAvroSchema(Schema beamSchema) {
+ public static org.apache.avro.Schema toAvroSchema(
+ Schema beamSchema, @Nullable String name, @Nullable String namespace) {
+ final String schemaName = Strings.isNullOrEmpty(name) ? "topLevelRecord" : name;
+ final String schemaNamespace = namespace == null ? "" : namespace;
+ String childNamespace =
+ !"".equals(schemaNamespace) ? schemaNamespace + "." + schemaName : schemaName;
List<org.apache.avro.Schema.Field> fields = Lists.newArrayList();
for (Schema.Field field : beamSchema.getFields()) {
- org.apache.avro.Schema fieldSchema = getFieldSchema(field.getType());
- org.apache.avro.Schema.Field recordField =
- new org.apache.avro.Schema.Field(
- field.getName(), fieldSchema, field.getDescription(), (Object) null);
+ org.apache.avro.Schema.Field recordField = toAvroField(field, childNamespace);
fields.add(recordField);
}
- org.apache.avro.Schema avroSchema = org.apache.avro.Schema.createRecord(fields);
- return avroSchema;
+ return org.apache.avro.Schema.createRecord(schemaName, null, schemaNamespace, false, fields);
+ }
+
+ public static org.apache.avro.Schema toAvroSchema(Schema beamSchema) {
+ return toAvroSchema(beamSchema, null, null);
}
/**
@@ -298,6 +321,57 @@
return g -> toGenericRecord(g, avroSchema);
}
+ /**
+ * Returns an {@code SchemaCoder} instance for the provided element type.
+ *
+ * @param <T> the element type
+ */
+ public static <T> SchemaCoder<T> schemaCoder(TypeDescriptor<T> type) {
+ @SuppressWarnings("unchecked")
+ Class<T> clazz = (Class<T>) type.getRawType();
+ return schemaCoder(clazz);
+ }
+
+ /**
+ * Returns an {@code SchemaCoder} instance for the provided element class.
+ *
+ * @param <T> the element type
+ */
+ public static <T> SchemaCoder<T> schemaCoder(Class<T> clazz) {
+ return schemaCoder(clazz, new ReflectData(clazz.getClassLoader()).getSchema(clazz));
+ }
+
+ /**
+ * Returns an {@code SchemaCoder} instance for the Avro schema. The implicit type is
+ * GenericRecord.
+ */
+ public static SchemaCoder<GenericRecord> schemaCoder(org.apache.avro.Schema schema) {
+ return schemaCoder(GenericRecord.class, schema);
+ }
+
+ /**
+ * Returns an {@code SchemaCoder} instance for the provided element type using the provided Avro
+ * schema.
+ *
+ * <p>If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema
+ * must correspond to the type provided.
+ *
+ * @param <T> the element type
+ */
+ public static <T> SchemaCoder<T> schemaCoder(Class<T> clazz, org.apache.avro.Schema schema) {
+ return SchemaCoder.of(
+ getSchema(clazz, schema), getToRowFunction(clazz, schema), getFromRowFunction(clazz));
+ }
+
+ /**
+ * Returns an {@code SchemaCoder} instance based on the provided AvroCoder for the element type.
+ *
+ * @param <T> the element type
+ */
+ public static <T> SchemaCoder<T> schemaCoder(AvroCoder<T> avroCoder) {
+ return schemaCoder(avroCoder.getType(), avroCoder.getSchema());
+ }
+
private static final class AvroSpecificRecordFieldValueTypeSupplier
implements FieldValueTypeSupplier {
@Override
@@ -398,6 +472,8 @@
// TODO: There is a desire to move Beam schema DATETIME to a micros representation. When
// this is done, this logical type needs to be changed.
fieldType = FieldType.DATETIME;
+ } else if (logicalType instanceof LogicalTypes.Date) {
+ fieldType = FieldType.DATETIME;
}
}
@@ -470,7 +546,8 @@
return fieldType;
}
- private static org.apache.avro.Schema getFieldSchema(Schema.FieldType fieldType) {
+ private static org.apache.avro.Schema getFieldSchema(
+ Schema.FieldType fieldType, String fieldName, String namespace) {
org.apache.avro.Schema baseType;
switch (fieldType.getTypeName()) {
case BYTE:
@@ -519,7 +596,7 @@
case LOGICAL_TYPE:
FixedBytesField fixedBytesField = FixedBytesField.fromBeamFieldType(fieldType);
if (fixedBytesField != null) {
- baseType = fixedBytesField.toAvroType();
+ baseType = fixedBytesField.toAvroType("fixed", namespace + "." + fieldName);
} else {
throw new RuntimeException(
"Unhandled logical type " + fieldType.getLogicalType().getIdentifier());
@@ -529,20 +606,22 @@
case ARRAY:
baseType =
org.apache.avro.Schema.createArray(
- getFieldSchema(fieldType.getCollectionElementType()));
+ getFieldSchema(fieldType.getCollectionElementType(), fieldName, namespace));
break;
case MAP:
if (fieldType.getMapKeyType().getTypeName().isStringType()) {
// Avro only supports string keys in maps.
- baseType = org.apache.avro.Schema.createMap(getFieldSchema(fieldType.getMapValueType()));
+ baseType =
+ org.apache.avro.Schema.createMap(
+ getFieldSchema(fieldType.getMapValueType(), fieldName, namespace));
} else {
throw new IllegalArgumentException("Avro only supports maps with string keys");
}
break;
case ROW:
- baseType = toAvroSchema(fieldType.getRowSchema());
+ baseType = toAvroSchema(fieldType.getRowSchema(), fieldName, namespace);
break;
default:
@@ -587,8 +666,16 @@
return new Conversions.DecimalConversion().toBytes(decimal, null, logicalType);
case DATETIME:
- ReadableInstant instant = (ReadableInstant) value;
- return instant.getMillis();
+ if (typeWithNullability.type.getType() == Type.INT) {
+ ReadableInstant instant = (ReadableInstant) value;
+ return (int) Days.daysBetween(Instant.EPOCH, instant).getDays();
+ } else if (typeWithNullability.type.getType() == Type.LONG) {
+ ReadableInstant instant = (ReadableInstant) value;
+ return (long) instant.getMillis();
+ } else {
+ throw new IllegalArgumentException(
+ "Can't represent " + fieldType + " as " + typeWithNullability.type.getType());
+ }
case BYTES:
return ByteBuffer.wrap((byte[]) value);
@@ -669,7 +756,18 @@
.fromBytes(byteBuffer.duplicate(), type.type, logicalType);
return convertDecimal(bigDecimal, fieldType);
} else if (logicalType instanceof LogicalTypes.TimestampMillis) {
- return convertDateTimeStrict((Long) value, fieldType);
+ if (value instanceof ReadableInstant) {
+ return convertDateTimeStrict(((ReadableInstant) value).getMillis(), fieldType);
+ } else {
+ return convertDateTimeStrict((Long) value, fieldType);
+ }
+ } else if (logicalType instanceof LogicalTypes.Date) {
+ if (value instanceof ReadableInstant) {
+ int epochDays = Days.daysBetween(Instant.EPOCH, (ReadableInstant) value).getDays();
+ return convertDateStrict(epochDays, fieldType);
+ } else {
+ return convertDateStrict((Integer) value, fieldType);
+ }
}
}
@@ -702,7 +800,9 @@
return convertRecordStrict((GenericRecord) value, fieldType);
case ENUM:
- return convertEnumStrict((GenericEnumSymbol) value, fieldType);
+ // enums are either Java enums, or GenericEnumSymbol,
+ // they don't share common interface, but override toString()
+ return convertEnumStrict(value, fieldType);
case ARRAY:
return convertArrayStrict((List<Object>) value, type.type.getElementType(), fieldType);
@@ -762,6 +862,11 @@
return value;
}
+ private static Object convertDateStrict(Integer epochDays, Schema.FieldType fieldType) {
+ checkTypeName(fieldType.getTypeName(), TypeName.DATETIME, "date");
+ return Instant.EPOCH.plus(Duration.standardDays(epochDays));
+ }
+
private static Object convertDateTimeStrict(Long value, Schema.FieldType fieldType) {
checkTypeName(fieldType.getTypeName(), TypeName.DATETIME, "dateTime");
return new Instant(value);
@@ -782,7 +887,7 @@
return value;
}
- private static Object convertEnumStrict(GenericEnumSymbol value, Schema.FieldType fieldType) {
+ private static Object convertEnumStrict(Object value, Schema.FieldType fieldType) {
checkTypeName(fieldType.getTypeName(), Schema.TypeName.STRING, "enum");
return value.toString();
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
index 2b2a6f4..e98e532 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
@@ -25,50 +25,55 @@
import java.lang.reflect.Parameter;
import java.lang.reflect.Type;
import java.nio.ByteBuffer;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.NamingStrategy;
-import net.bytebuddy.NamingStrategy.SuffixingRandom.BaseNameResolver;
-import net.bytebuddy.description.method.MethodDescription.ForLoadedConstructor;
-import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
-import net.bytebuddy.description.type.TypeDescription;
-import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
-import net.bytebuddy.implementation.bytecode.Duplication;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
-import net.bytebuddy.implementation.bytecode.TypeCreation;
-import net.bytebuddy.implementation.bytecode.assign.Assigner;
-import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
-import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
-import net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
-import net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
-import net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
-import net.bytebuddy.utility.RandomString;
import org.apache.avro.generic.GenericFixed;
import org.apache.beam.sdk.schemas.FieldValueGetter;
import org.apache.beam.sdk.schemas.FieldValueSetter;
import org.apache.beam.sdk.schemas.FieldValueTypeInformation;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeParameter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.NamingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.NamingStrategy.SuffixingRandom.BaseNameResolver;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.method.MethodDescription.ForLoadedConstructor;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Duplication;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.TypeCreation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayFactory;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.utility.RandomString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.ClassUtils;
+import org.joda.time.DateTimeZone;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
+import org.joda.time.ReadablePartial;
+import org.joda.time.base.BaseLocal;
class ByteBuddyUtils {
private static final ForLoadedType ARRAYS_TYPE = new ForLoadedType(Arrays.class);
@@ -77,9 +82,13 @@
private static final ForLoadedType BYTE_BUFFER_TYPE = new ForLoadedType(ByteBuffer.class);
private static final ForLoadedType CHAR_SEQUENCE_TYPE = new ForLoadedType(CharSequence.class);
private static final ForLoadedType INSTANT_TYPE = new ForLoadedType(Instant.class);
+ private static final ForLoadedType DATE_TIME_ZONE_TYPE = new ForLoadedType(DateTimeZone.class);
private static final ForLoadedType LIST_TYPE = new ForLoadedType(List.class);
private static final ForLoadedType READABLE_INSTANT_TYPE =
new ForLoadedType(ReadableInstant.class);
+ private static final ForLoadedType READABLE_PARTIAL_TYPE =
+ new ForLoadedType(ReadablePartial.class);
+ private static final ForLoadedType OBJECT_TYPE = new ForLoadedType(Object.class);
/**
* A naming strategy for ByteBuddy classes.
@@ -151,6 +160,8 @@
return convertMap(typeDescriptor);
} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(ReadableInstant.class))) {
return convertDateTime(typeDescriptor);
+ } else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(ReadablePartial.class))) {
+ return convertDateTime(typeDescriptor);
} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(ByteBuffer.class))) {
return convertByteBuffer(typeDescriptor);
} else if (typeDescriptor.isSubtypeOf(TypeDescriptor.of(GenericFixed.class))) {
@@ -160,6 +171,8 @@
return convertCharSequence(typeDescriptor);
} else if (typeDescriptor.getRawType().isPrimitive()) {
return convertPrimitive(typeDescriptor);
+ } else if (typeDescriptor.getRawType().isEnum()) {
+ return convertEnum(typeDescriptor);
} else {
return convertDefault(typeDescriptor);
}
@@ -181,6 +194,8 @@
protected abstract T convertPrimitive(TypeDescriptor<?> type);
+ protected abstract T convertEnum(TypeDescriptor<?> type);
+
protected abstract T convertDefault(TypeDescriptor<?> type);
}
@@ -251,6 +266,11 @@
}
@Override
+ protected Type convertEnum(TypeDescriptor<?> type) {
+ return String.class;
+ }
+
+ @Override
protected Type convertDefault(TypeDescriptor<?> type) {
return returnRawTypes ? type.getRawType() : type.getType();
}
@@ -328,22 +348,64 @@
if (Instant.class.isAssignableFrom(type.getRawType())) {
return readValue;
}
- // Otherwise, generate the following code:
- // return new Instant(value.getMillis());
- return new StackManipulation.Compound(
- // Create a new instance of the target type.
- TypeCreation.of(INSTANT_TYPE),
- Duplication.SINGLE,
- readValue,
- TypeCasting.to(READABLE_INSTANT_TYPE),
- // Call ReadableInstant.getMillis to extract the millis since the epoch.
+ // Otherwise, generate the following code:
+ //
+ // for ReadableInstant:
+ // return new Instant(value.getMillis());
+ //
+ // for ReadablePartial:
+ // return new Instant((value.toDateTime(Instant.EPOCH)).getMillis());
+
+ List<StackManipulation> stackManipulations = new ArrayList<>();
+
+ // Create a new instance of the target type.
+ stackManipulations.add(TypeCreation.of(INSTANT_TYPE));
+ stackManipulations.add(Duplication.SINGLE);
+
+ // if value is ReadablePartial, convert it to ReadableInstant first
+ if (ReadablePartial.class.isAssignableFrom(type.getRawType())) {
+ // Generate the following code: .toDateTime(Instant.EPOCH)
+
+ // Load the parameter and cast it to ReadablePartial.
+ stackManipulations.add(readValue);
+ stackManipulations.add(TypeCasting.to(READABLE_PARTIAL_TYPE));
+
+ // Get Instant.EPOCH
+ stackManipulations.add(
+ FieldAccess.forField(
+ INSTANT_TYPE
+ .getDeclaredFields()
+ .filter(ElementMatchers.named("EPOCH"))
+ .getOnly())
+ .read());
+
+ // Call ReadablePartial.toDateTime
+ stackManipulations.add(
+ MethodInvocation.invoke(
+ READABLE_PARTIAL_TYPE
+ .getDeclaredMethods()
+ .filter(
+ ElementMatchers.named("toDateTime")
+ .and(ElementMatchers.takesArguments(READABLE_INSTANT_TYPE)))
+ .getOnly()));
+ } else {
+ // Otherwise, parameter is already ReadableInstant.
+ // Load the parameter and cast it to ReadableInstant.
+ stackManipulations.add(readValue);
+ stackManipulations.add(TypeCasting.to(READABLE_INSTANT_TYPE));
+ }
+
+ // Call ReadableInstant.getMillis to extract the millis since the epoch.
+ stackManipulations.add(
MethodInvocation.invoke(
READABLE_INSTANT_TYPE
.getDeclaredMethods()
.filter(ElementMatchers.named("getMillis"))
- .getOnly()),
- // Construct a DateTime object containing the millis.
+ .getOnly()));
+
+ // Construct a Instant object containing the millis.
+ stackManipulations.add(
MethodInvocation.invoke(
INSTANT_TYPE
.getDeclaredMethods()
@@ -351,6 +413,8 @@
ElementMatchers.isConstructor()
.and(ElementMatchers.takesArguments(ForLoadedType.of(long.class))))
.getOnly()));
+
+ return new StackManipulation.Compound(stackManipulations);
}
@Override
@@ -402,7 +466,7 @@
MethodInvocation.invoke(
CHAR_SEQUENCE_TYPE
.getDeclaredMethods()
- .filter(ElementMatchers.named("toString"))
+ .filter(ElementMatchers.named("toString").and(ElementMatchers.takesArguments(0)))
.getOnly()));
}
@@ -417,6 +481,17 @@
}
@Override
+ protected StackManipulation convertEnum(TypeDescriptor<?> type) {
+ return new Compound(
+ readValue,
+ MethodInvocation.invoke(
+ OBJECT_TYPE
+ .getDeclaredMethods()
+ .filter(ElementMatchers.named("toString").and(ElementMatchers.takesArguments(0)))
+ .getOnly()));
+ }
+
+ @Override
protected StackManipulation convertDefault(TypeDescriptor<?> type) {
return readValue;
}
@@ -502,31 +577,62 @@
// that the POJO can accept.
// Generate the following code:
- // return new T(value.getMillis());
+ // return new T(value.getMillis());
+ // Unless T is a sub-class of BaseLocal. Then generate:
+ // return new T(value.getMillis(), DateTimeZone.UTC);
ForLoadedType loadedType = new ForLoadedType(type.getRawType());
- return new Compound(
- // Create a new instance of the target type.
- TypeCreation.of(loadedType),
- Duplication.SINGLE,
- // Load the parameter and cast it to a ReadableInstant.
- readValue,
- TypeCasting.to(READABLE_INSTANT_TYPE),
- // Call ReadableInstant.getMillis to extract the millis since the epoch.
+ List<StackManipulation> stackManipulations = new ArrayList<>();
+
+ // Create a new instance of the target ype.
+ stackManipulations.add(TypeCreation.of(loadedType));
+ stackManipulations.add(Duplication.SINGLE);
+ // Load the parameter and cast it to a ReadableInstant.
+ stackManipulations.add(readValue);
+ stackManipulations.add(TypeCasting.to(READABLE_INSTANT_TYPE));
+ // Call ReadableInstant.getMillis to extract the millis since the epoch.
+ stackManipulations.add(
MethodInvocation.invoke(
READABLE_INSTANT_TYPE
.getDeclaredMethods()
.filter(ElementMatchers.named("getMillis"))
- .getOnly()),
- // All subclasses of ReadableInstant contain a ()(long) constructor that takes in a millis
- // argument. Call that constructor of the field to initialize it.
- MethodInvocation.invoke(
- loadedType
- .getDeclaredMethods()
- .filter(
- ElementMatchers.isConstructor()
- .and(ElementMatchers.takesArguments(ForLoadedType.of(long.class))))
.getOnly()));
+ if (type.isSubtypeOf(TypeDescriptor.of(BaseLocal.class))) {
+ // Access DateTimeZone.UTC
+ stackManipulations.add(
+ FieldAccess.forField(
+ DATE_TIME_ZONE_TYPE
+ .getDeclaredFields()
+ .filter(ElementMatchers.named("UTC"))
+ .getOnly())
+ .read());
+ // All subclasses of BaseLocal contain a ()(long, DateTimeZone) constructor
+ // that takes in a millis and time zone argument. Call that constructor of the field to
+ // initialize it.
+ stackManipulations.add(
+ MethodInvocation.invoke(
+ loadedType
+ .getDeclaredMethods()
+ .filter(
+ ElementMatchers.isConstructor()
+ .and(
+ ElementMatchers.takesArguments(
+ ForLoadedType.of(long.class), DATE_TIME_ZONE_TYPE)))
+ .getOnly()));
+ } else {
+ // All subclasses of ReadableInstant and ReadablePartial contain a ()(long) constructor
+ // that takes in a millis argument. Call that constructor of the field to initialize it.
+ stackManipulations.add(
+ MethodInvocation.invoke(
+ loadedType
+ .getDeclaredMethods()
+ .filter(
+ ElementMatchers.isConstructor()
+ .and(ElementMatchers.takesArguments(ForLoadedType.of(long.class))))
+ .getOnly()));
+ }
+
+ return new Compound(stackManipulations);
}
@Override
@@ -551,13 +657,14 @@
@Override
protected StackManipulation convertGenericFixed(TypeDescriptor<?> type) {
// Generate the following code:
- // return T((byte[]) value);
+ // return new T((byte[]) value);
// TODO: Refactor AVRO-specific code out of this class.
ForLoadedType loadedType = new ForLoadedType(type.getRawType());
return new Compound(
TypeCreation.of(loadedType),
Duplication.SINGLE,
+ // Load the parameter and cast it to a byte[].
readValue,
TypeCasting.to(BYTE_ARRAY_TYPE),
// Create a new instance that wraps this byte[].
@@ -610,6 +717,23 @@
}
@Override
+ protected StackManipulation convertEnum(TypeDescriptor<?> type) {
+ ForLoadedType loadedType = new ForLoadedType(type.getRawType());
+
+ return new Compound(
+ readValue,
+ MethodInvocation.invoke(
+ loadedType
+ .getDeclaredMethods()
+ .filter(
+ ElementMatchers.named("valueOf")
+ .and(
+ ElementMatchers.isStatic()
+ .and(ElementMatchers.takesArguments(String.class))))
+ .getOnly()));
+ }
+
+ @Override
protected StackManipulation convertDefault(TypeDescriptor<?> type) {
return readValue;
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java
index e74f85f..259d6f3 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ConvertHelpers.java
@@ -21,18 +21,6 @@
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Type;
import javax.annotation.Nullable;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.type.TypeDescription;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.schemas.JavaFieldSchema.JavaFieldTypeSupplier;
import org.apache.beam.sdk.schemas.NoSuchSchemaException;
import org.apache.beam.sdk.schemas.Schema;
@@ -46,7 +34,19 @@
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Primitives;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Primitives;
/** Helper functions for converting between equivalent schema types. */
public class ConvertHelpers {
@@ -54,10 +54,14 @@
public static class ConvertedSchemaInformation<T> implements Serializable {
// If the output type is a composite type, this is the schema coder.
@Nullable public final SchemaCoder<T> outputSchemaCoder;
+ // If the input schema has a single field and the output type's schema matches that field, this
+ // is the output type.
@Nullable public final FieldType unboxedType;
public ConvertedSchemaInformation(
@Nullable SchemaCoder<T> outputSchemaCoder, @Nullable FieldType unboxedType) {
+ assert outputSchemaCoder != null || unboxedType != null;
+
this.outputSchemaCoder = outputSchemaCoder;
this.unboxedType = unboxedType;
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
index 9ee266c..a47224f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtils.java
@@ -24,21 +24,6 @@
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.implementation.FixedValue;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
-import net.bytebuddy.implementation.bytecode.Removal;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.schemas.FieldValueGetter;
@@ -53,7 +38,22 @@
import org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.StaticFactoryMethodInstruction;
import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.FixedValue;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Removal;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** A set of utilities to generate getter and setter classes for JavaBean objects. */
@Experimental(Kind.SCHEMAS)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/POJOUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/POJOUtils.java
index f4de779..27b0db2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/POJOUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/POJOUtils.java
@@ -25,27 +25,6 @@
import java.util.Map;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.field.FieldDescription.ForLoadedField;
-import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.implementation.FixedValue;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
-import net.bytebuddy.implementation.bytecode.Duplication;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.TypeCreation;
-import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
-import net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
-import net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
-import net.bytebuddy.implementation.bytecode.member.FieldAccess;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.schemas.FieldValueGetter;
@@ -61,7 +40,28 @@
import org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.field.FieldDescription.ForLoadedField;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.FixedValue;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender.Size;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Duplication;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.TypeCreation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** A set of utilities to generate getter and setter classes for POJOs. */
@Experimental(Kind.SCHEMAS)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
index 3204bb1..b9f1ae5 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ReflectUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
@@ -33,8 +33,8 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** A set of reflection helper methods. */
public class ReflectUtils {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SchemaZipFold.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SchemaZipFold.java
index 4b225f5..eb9dcfe 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SchemaZipFold.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SchemaZipFold.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Visitor that zips schemas, and accepts pairs of fields and their types.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
index a8f1274..bb4972f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.schemas.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.List;
import java.util.Map;
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Helper methods to select subrows out of rows. */
public class SelectHelpers {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
index 7de5fae..8fb13bb 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/StaticSchemaInference.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.schemas.utils;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.ParameterizedType;
import java.math.BigDecimal;
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.ReadableInstant;
/** A set of utilities for inferring a Beam {@link Schema} from static Java types. */
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/StateSpecs.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/StateSpecs.java
index 894f678..7227727 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/state/StateSpecs.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/state/StateSpecs.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.state;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Objects;
import javax.annotation.Nullable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/Annotations.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/Annotations.java
index 1d1fa5f..22d9952 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/Annotations.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/Annotations.java
@@ -19,8 +19,8 @@
import java.lang.annotation.Annotation;
import java.util.Arrays;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
import org.junit.experimental.categories.Category;
/** A utility class for querying annotations. */
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java
index d5d2bbc..f10e95b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CoderProperties.java
@@ -39,12 +39,12 @@
import org.apache.beam.sdk.util.UnownedInputStream;
import org.apache.beam.sdk.util.UnownedOutputStream;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingInputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingInputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
/**
* Properties for use in {@link Coder} tests. These are implemented with junit assertions rather
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
index 62e2172..28349ab 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/FileChecksumMatcher.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.util.NumberedShardedFile;
import org.apache.beam.sdk.util.ShardedFile;
import org.apache.beam.sdk.util.Sleeper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashCode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
import org.joda.time.Duration;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherDeserializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherDeserializer.java
index 12fb782..5a8228d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherDeserializer.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherDeserializer.java
@@ -24,7 +24,7 @@
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.io.IOException;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
/** MatcherDeserializer is used with Jackson to enable deserialization of SerializableMatchers. */
class MatcherDeserializer extends JsonDeserializer<SerializableMatcher<?>> {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherSerializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherSerializer.java
index d6c205d..1295b9a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherSerializer.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/MatcherSerializer.java
@@ -23,7 +23,7 @@
import com.fasterxml.jackson.databind.SerializerProvider;
import java.io.IOException;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
/** MatcherSerializer is used with Jackson to enable serialization of SerializableMatchers. */
class MatcherSerializer extends JsonSerializer<SerializableMatcher<?>> {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index ac7a71c..e80205a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.not;
@@ -72,9 +72,9 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.ValueInSingleWindow;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SerializableMatchers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SerializableMatchers.java
index ff671c8..209c867 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SerializableMatchers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SerializableMatchers.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java
index a866c0d..d31cadf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SourceTestUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
import static org.junit.Assert.assertEquals;
@@ -43,8 +43,8 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Assert;
import org.slf4j.Logger;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StaticWindows.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StaticWindows.java
index 1712210..a9615a2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StaticWindows.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/StaticWindows.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Collections;
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
index 7a915fa..317c5f1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
@@ -22,8 +22,8 @@
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.util.SerializableThrowable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Output of {@link PAssert}. Passed to a conclude function to act upon. */
@DefaultCoder(SerializableCoder.class)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index 6967a64..8f5b7d0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
@@ -45,12 +45,12 @@
import org.apache.beam.sdk.runners.TransformHierarchy;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.junit.experimental.categories.Category;
import org.junit.rules.TestRule;
import org.junit.runner.Description;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
index 6ab4c0f..41a46ab 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -47,8 +47,8 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -74,6 +74,10 @@
return new Builder<>(coder);
}
+ public static Builder<Row> create(Schema schema) {
+ return create(SchemaCoder.of(schema));
+ }
+
public static <T> Builder<T> create(
Schema schema,
SerializableFunction<T, Row> toRowFunction,
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesKms.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesKms.java
new file mode 100644
index 0000000..f67d647
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesKms.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/**
+ * Category tag for validation tests which utilize --tempRoot from {@link TestPipelineOptions} and
+ * and expect a default KMS key enable for the bucket specified.
+ *
+ * <p>Currently only applicable to GCP-based tests.
+ */
+public interface UsesKms {}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputsWithDifferentCoders.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputsWithDifferentCoders.java
new file mode 100644
index 0000000..ab50313
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputsWithDifferentCoders.java
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+/** Category tag for validation tests which use multiple side inputs with different coders. */
+public interface UsesSideInputsWithDifferentCoders {}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowFnTestUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowFnTestUtils.java
index a76176d..9da9593 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowFnTestUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowFnTestUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.greaterThan;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertThat;
@@ -38,8 +38,8 @@
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowSupplier.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowSupplier.java
index 8e1ba38..ebae6d6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowSupplier.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/WindowSupplier.java
@@ -24,9 +24,9 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/**
* A {@link Supplier} that returns a static set of {@link BoundedWindow BoundedWindows}. The
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
index 1e5fd48..d58b2cc 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateQuantiles.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.DataInputStream;
import java.io.DataOutputStream;
@@ -47,9 +47,9 @@
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.UnmodifiableIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.UnmodifiableIterator;
/**
* {@code PTransform}s for getting an idea of a {@code PCollection}'s data distribution using
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateUnique.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateUnique.java
index fd12e45..2b46641 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateUnique.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ApproximateUnique.java
@@ -32,16 +32,30 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashingOutputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* {@code PTransform}s for estimating the number of distinct elements in a {@code PCollection}, or
* the number of distinct values associated with each key in a {@code PCollection} of {@code KV}s.
+ *
+ * <p>Consider using {@code HllCount} in the {@code zetasketch} extension module if you need better
+ * performance or need to save intermediate aggregation result into a sketch for later processing.
+ *
+ * <p>For example, to estimate the number of distinct elements in a {@code PCollection<String>}:
+ *
+ * <pre>{@code
+ * PCollection<String> input = ...;
+ * PCollection<Long> countDistinct =
+ * input.apply(HllCount.Init.forStrings().globally()).apply(HllCount.Extract.globally());
+ * }</pre>
+ *
+ * For more details about using {@code HllCount} and the {@code zetasketch} extension module, see
+ * https://s.apache.org/hll-in-beam#bookmark=id.v6chsij1ixo7.
*/
public class ApproximateUnique {
@@ -144,7 +158,7 @@
*
* @param <T> the type of the elements in the input {@code PCollection}
*/
- static class Globally<T> extends PTransform<PCollection<T>, PCollection<Long>> {
+ public static final class Globally<T> extends PTransform<PCollection<T>, PCollection<Long>> {
/**
* The number of entries in the statistical sample; the higher this number, the more accurate
@@ -200,7 +214,8 @@
* @param <K> the type of the keys in the input and output {@code PCollection}s
* @param <V> the type of the values in the input {@code PCollection}
*/
- static class PerKey<K, V> extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Long>>> {
+ public static final class PerKey<K, V>
+ extends PTransform<PCollection<KV<K, V>>, PCollection<KV<K, Long>>> {
/**
* The number of entries in the statistical sample; the higher this number, the more accurate
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
index 02538df..2bd8365 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
@@ -64,13 +64,14 @@
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PCollectionViews;
+import org.apache.beam.sdk.values.PCollectionViews.TypeDescriptorSupplier;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* {@code PTransform}s for combining {@code PCollection} elements globally and per-key.
@@ -585,6 +586,11 @@
this.present = true;
this.value = value;
}
+
+ @Override
+ public String toString() {
+ return "Combine.Holder(value=" + value + ", present=" + present + ")";
+ }
}
/** A {@link Coder} for a {@link Holder}. */
@@ -1300,9 +1306,12 @@
input.apply(Combine.<InputT, OutputT>globally(fn).withoutDefaults().withFanout(fanout));
PCollection<KV<Void, OutputT>> materializationInput =
combined.apply(new VoidKeyToMultimapMaterialization<>());
+ Coder<OutputT> outputCoder = combined.getCoder();
PCollectionView<OutputT> view =
PCollectionViews.singletonView(
materializationInput,
+ (TypeDescriptorSupplier<OutputT>)
+ () -> outputCoder != null ? outputCoder.getEncodedTypeDescriptor() : null,
input.getWindowingStrategy(),
insertDefault,
insertDefault ? fn.defaultValue() : null,
@@ -2054,10 +2063,10 @@
* <pre>{@code
* PCollection<KV<String, Integer>> pc = ...;
* PCollection<KV<String, Iterable<Integer>>> groupedByKey = pc.apply(
- * new GroupByKey<String, Integer>());
- * PCollection<KV<String, Integer>> sumByKey = groupedByKey.apply(
- * Combine.<String, Integer>groupedValues(
- * new Sum.SumIntegerFn()));
+ * GroupByKey.create());
+ * PCollection<KV<String, Integer>> sumByKey =
+ * groupedByKey.apply(Combine.groupedValues(
+ * Sum.ofIntegers()));
* }</pre>
*
* <p>See also {@link #perKey}/{@link PerKey Combine.PerKey}, which captures the common pattern of
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFnBase.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFnBase.java
index 78368ee..08f518a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFnBase.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFnBase.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.transforms.display.HasDisplayData;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* <b><i>For internal use only; no backwards-compatibility guarantees.</i></b>
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
index 6a848c7..d32654e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/CombineFns.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.InputStream;
@@ -42,12 +42,12 @@
import org.apache.beam.sdk.transforms.display.HasDisplayData;
import org.apache.beam.sdk.util.CombineFnUtil;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Static utility methods that create combine function instances. */
public class CombineFns {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
index 0739994..7d19850 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
@@ -21,7 +21,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Pair of a bit of user code (a "closure") and the {@link Requirements} needed to run it. */
@Experimental(Kind.CONTEXTFUL)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java
index 3496558..09fa260 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Create.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.ArrayList;
@@ -63,10 +63,10 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TimestampedValue.TimestampedValueCoder;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index 069215a..4aa8dbf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -586,6 +586,8 @@
*
* <ul>
* <li>It <i>must</i> define a {@link GetInitialRestriction} method.
+ * <li>It <i>may</i> define a {@link GetSize} method.
+ * <li>It <i>may</i> define a {@link GetPartitition} method.
* <li>It <i>may</i> define a {@link SplitRestriction} method.
* <li>It <i>may</i> define a {@link NewTracker} method returning a subtype of {@code
* RestrictionTracker<R>} where {@code R} is the restriction type returned by {@link
@@ -623,6 +625,14 @@
@Target(ElementType.PARAMETER)
public @interface Timestamp {}
+ /** Parameter annotation for the SideInput for a {@link ProcessElement} method. */
+ @Documented
+ @Retention(RetentionPolicy.RUNTIME)
+ @Target(ElementType.PARAMETER)
+ public @interface SideInput {
+ /** The SideInput tag ID. */
+ String value();
+ }
/**
* <b><i>Experimental - no backwards compatibility guarantees. The exact name or usage of this
* feature may change.</i></b>
@@ -724,6 +734,75 @@
public @interface GetInitialRestriction {}
/**
+ * Annotation for the method that returns the corresponding size for an element and restriction
+ * pair.
+ *
+ * <p>Signature: {@code double getSize(InputT element, RestrictionT restriction);}
+ *
+ * <p>Returns a double representing the size of the element and restriction.
+ *
+ * <p>A representation for the amount of known work represented as a size. Size representations
+ * should preferably represent a linear space and be comparable within the same partition (see
+ * {@link GetPartition} for details on partition identifiers}).
+ *
+ * <p>Splittable {@link DoFn}s should only provide this method if the default implementation
+ * within the {@link RestrictionTracker} is an inaccurate representation of known work.
+ *
+ * <p>It is up to each splittable {@DoFn} to convert between their natural representation of
+ * outstanding work and this representation. For example:
+ *
+ * <ul>
+ * <li>Block based file source (e.g. Avro): From the end of the current block, the remaining
+ * number of bytes to the end of the restriction.
+ * <li>Pull based queue based source (e.g. Pubsub): The local/global size available in number of
+ * messages or number of {@code message bytes} that have not been processed.
+ * <li>Key range based source (e.g. Shuffle, Bigtable, ...): Scale the start key to be one and
+ * end key to be zero and interpolate the position of the next splittable key as the size.
+ * If information about the probability density function or cumulative distribution function
+ * is available, size interpolation can be improved. Alternatively, if the number of encoded
+ * bytes for the keys and values is known for the key range, the number of remaining bytes
+ * can be used.
+ * </ul>
+ */
+ @Documented
+ @Retention(RetentionPolicy.RUNTIME)
+ @Target(ElementType.METHOD)
+ @Experimental(Kind.SPLITTABLE_DO_FN)
+ public @interface GetSize {}
+
+ /**
+ * Annotation for the method that returns the corresponding partition identifier for an element
+ * and restriction pair.
+ *
+ * <p>Signature: {@code byte[] getPartitition(InputT element, RestrictionT restriction);}
+ *
+ * <p>Returns an immutable representation of the partition identifier as a byte[].
+ *
+ * <p>By default, the partition identifier is represented as the encoded element and restriction
+ * pair and should only be provided if the splittable {@link DoFn} can only provide a size over a
+ * shared resource such as a message queue that potentially multiple element and restriction pairs
+ * are doing work on. The partition identifier is used by runners for various size calculations.
+ * Sizes reported with the same partition identifier represent a point in time reporting of the
+ * size for that partition. For example, a runner can compute a global size by summing all
+ * reported sizes over all unique partition identifiers while it can compute the size of a
+ * specific partition based upon the last reported value.
+ *
+ * <p>For example splittable {@link DoFn}s which consume elements from:
+ *
+ * <ul>
+ * <li>a globally shared resource such as a Pubsub queue should set this to "".
+ * <li>a shared partitioned resource should use the partition identifier.
+ * <li>a uniquely partitioned resource such as a file and offset range should not override this
+ * since the default element and restriction pair should suffice.
+ * </ul>
+ */
+ @Documented
+ @Retention(RetentionPolicy.RUNTIME)
+ @Target(ElementType.METHOD)
+ @Experimental(Kind.SPLITTABLE_DO_FN)
+ public @interface GetPartition {}
+
+ /**
* Annotation for the method that returns the coder to use for the restriction of a <a
* href="https://s.apache.org/splittable-do-fn">splittable</a> {@link DoFn}.
*
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnOutputReceivers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnOutputReceivers.java
index ab924f1..c0c220b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnOutputReceivers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnOutputReceivers.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Map;
import javax.annotation.Nullable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnSchemaInformation.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnSchemaInformation.java
index ab54fb3..3f7c035 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnSchemaInformation.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnSchemaInformation.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.schemas.utils.SelectHelpers;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Represents information about how a DoFn extracts schemas. */
@AutoValue
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index 1c5f4b6..9ceb37e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Arrays;
@@ -52,8 +52,8 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -251,6 +251,11 @@
}
@Override
+ public InputT sideInput(String sideInputTag) {
+ throw new UnsupportedOperationException("SideInputs are not supported by DoFnTester");
+ }
+
+ @Override
public InputT schemaElement(int index) {
throw new UnsupportedOperationException("Schemas are not supported by DoFnTester");
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
index 59914f8..c2a935e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
+import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.expansion.ExternalTransformRegistrar;
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.POutput;
@@ -37,6 +38,7 @@
* @param <InputT> The input type of the externally configured PTransform.
* @param <OutputT> The output type of the externally configured PTransform.
*/
+@Experimental
public interface ExternalTransformBuilder<ConfigT, InputT extends PInput, OutputT extends POutput> {
/** Builds the transform after it has been configured. */
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/FlatMapElements.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/FlatMapElements.java
index 286e8ab..2dcb78e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/FlatMapElements.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/FlatMapElements.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java
index dbc54e4..3f724a9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
@@ -33,8 +33,8 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -80,6 +80,11 @@
return new GroupIntoBatches<>(batchSize);
}
+ /** Returns the size of the batch. */
+ public long getBatchSize() {
+ return batchSize;
+ }
+
@Override
public PCollection<KV<K, Iterable<InputT>>> expand(PCollection<KV<K, InputT>> input) {
Duration allowedLateness = input.getWindowingStrategy().getAllowedLateness();
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
index b8546b4..33fb2a6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/JsonToRow.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.sdk.util.JsonToRowUtils.jsonToRow;
-import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.jsonToRow;
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
import com.fasterxml.jackson.databind.ObjectMapper;
import javax.annotation.Nullable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Latest.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Latest.java
index bdd03e9..906e7e0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Latest.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Latest.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Iterator;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* {@link PTransform} and {@link Combine.CombineFn} for computing the latest element in a {@link
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/MapElements.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/MapElements.java
index 09fef37..5f66731 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/MapElements.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/MapElements.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Mean.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Mean.java
index 2188735..0f7c0d9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Mean.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Mean.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.CoderRegistry;
import org.apache.beam.sdk.coders.DoubleCoder;
import org.apache.beam.sdk.transforms.Combine.AccumulatingCombineFn.Accumulator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* {@code PTransform}s for computing the arithmetic mean (a.k.a. average) of the elements in a
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 08bae25..fb8524a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -17,15 +17,16 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
import java.util.Arrays;
import java.util.Collections;
-import java.util.List;
import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
import javax.annotation.Nullable;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineRunner;
@@ -52,6 +53,7 @@
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.MethodWithExtraParameters;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.OnTimerMethod;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SideInputParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -64,7 +66,7 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* {@link ParDo} is the core element-wise transform in Apache Beam, invoking a user-specified
@@ -223,18 +225,18 @@
* public void processElement(@Element String word, MultiOutputReceiver r) {
* if (word.length() <= wordLengthCutOff) {
* // Emit this short word to the main output.
- * r.output(wordsBelowCutOffTag, word);
+ * r.get(wordsBelowCutOffTag).output(word);
* } else {
* // Emit this long word's length to a specified output.
- * r.output(wordLengthsAboveCutOffTag, word.length());
+ * r.get(wordLengthsAboveCutOffTag).output(word.length());
* }
* if (word.startsWith("MARKER")) {
* // Emit this word to a different specified output.
- * r.output(markedWordsTag, word);
+ * r.get(markedWordsTag).output(word);
* }
* if (word.startsWith("SPECIAL")) {
* // Emit this word to the unconsumed output.
- * r.output(specialWordsTag, word);
+ * r.get(specialWordsTag).output(word);
* }
* }}})
* // Specify the main and consumed output tags of the
@@ -382,8 +384,8 @@
* Beam makes heavy use of this modular, composable style, trusting to the runner to "flatten out"
* all the compositions into highly optimized stages.
*
- * @see <a href= "https://beam.apache.org/documentation/programming-guide/#transforms-pardo"> the
- * web documentation for ParDo</a>
+ * @see <a href= "https://beam.apache.org/documentation/programming-guide/#pardo"> the web
+ * documentation for ParDo</a>
*/
public class ParDo {
@@ -395,7 +397,7 @@
*/
public static <InputT, OutputT> SingleOutput<InputT, OutputT> of(DoFn<InputT, OutputT> fn) {
validate(fn);
- return new SingleOutput<>(fn, Collections.emptyList(), displayDataForFn(fn));
+ return new SingleOutput<>(fn, Collections.emptyMap(), displayDataForFn(fn));
}
private static <T> DisplayData.ItemSpec<? extends Class<?>> displayDataForFn(T fn) {
@@ -436,6 +438,29 @@
}
}
+ private static void validateSideInputTypes(
+ Map<String, PCollectionView<?>> sideInputs, DoFn<?, ?> fn) {
+ DoFnSignature signature = DoFnSignatures.getSignature(fn.getClass());
+ DoFnSignature.ProcessElementMethod processElementMethod = signature.processElement();
+ for (SideInputParameter sideInput : processElementMethod.getSideInputParameters()) {
+ PCollectionView<?> view = sideInputs.get(sideInput.sideInputId());
+ checkArgument(
+ view != null,
+ "the ProcessElement method expects a side input identified with the tag %s, but no such side input was"
+ + " supplied. Use withSideInput(String, PCollectionView) to supply this side input.",
+ sideInput.sideInputId());
+ TypeDescriptor<?> viewType = view.getViewFn().getTypeDescriptor();
+
+ // Currently check that the types exactly match, even if the types are convertible.
+ checkArgument(
+ viewType.equals(sideInput.elementT()),
+ "Side Input with tag %s and type %s cannot be bound to ProcessElement parameter with type %s",
+ sideInput.sideInputId(),
+ viewType,
+ sideInput.elementT());
+ }
+ }
+
private static FieldAccessDescriptor getFieldAccessDescriptorFromParameter(
@Nullable String fieldAccessString,
Schema inputSchema,
@@ -562,7 +587,6 @@
fn.getClass().getName()));
}
}
-
/**
* Extract information on how the DoFn uses schemas. In particular, if the schema of an element
* parameter does not match the input PCollection's schema, convert.
@@ -629,19 +653,18 @@
private static final String MAIN_OUTPUT_TAG = "output";
- private final List<PCollectionView<?>> sideInputs;
+ private final Map<String, PCollectionView<?>> sideInputs;
private final DoFn<InputT, OutputT> fn;
private final DisplayData.ItemSpec<? extends Class<?>> fnDisplayData;
SingleOutput(
DoFn<InputT, OutputT> fn,
- List<PCollectionView<?>> sideInputs,
+ Map<String, PCollectionView<?>> sideInputs,
DisplayData.ItemSpec<? extends Class<?>> fnDisplayData) {
this.fn = fn;
this.fnDisplayData = fnDisplayData;
this.sideInputs = sideInputs;
}
-
/**
* Returns a new {@link ParDo} {@link PTransform} that's like this {@link PTransform} but with
* the specified additional side inputs. Does not modify this {@link PTransform}.
@@ -660,16 +683,39 @@
*/
public SingleOutput<InputT, OutputT> withSideInputs(
Iterable<? extends PCollectionView<?>> sideInputs) {
+ Map<String, PCollectionView<?>> mappedInputs =
+ StreamSupport.stream(sideInputs.spliterator(), false)
+ .collect(Collectors.toMap(v -> v.getTagInternal().getId(), v -> v));
+ return withSideInputs(mappedInputs);
+ }
+
+ /**
+ * Returns a new {@link ParDo} {@link PTransform} that's like this {@link PTransform} but with
+ * the specified additional side inputs. Does not modify this {@link PTransform}.
+ *
+ * <p>See the discussion of Side Inputs above for more explanation.
+ */
+ public SingleOutput<InputT, OutputT> withSideInputs(
+ Map<String, PCollectionView<?>> sideInputs) {
return new SingleOutput<>(
fn,
- ImmutableList.<PCollectionView<?>>builder()
- .addAll(this.sideInputs)
- .addAll(sideInputs)
+ ImmutableMap.<String, PCollectionView<?>>builder()
+ .putAll(this.sideInputs)
+ .putAll(sideInputs)
.build(),
fnDisplayData);
}
/**
+ * Returns a new {@link ParDo} {@link PTransform} that's like this {@link PTransform} but with
+ * the specified additional side inputs. Does not modify this {@link PTransform}.
+ */
+ public SingleOutput<InputT, OutputT> withSideInput(
+ String tagId, PCollectionView<?> pCollectionView) {
+ return withSideInputs(Collections.singletonMap(tagId, pCollectionView));
+ }
+
+ /**
* Returns a new multi-output {@link ParDo} {@link PTransform} that's like this {@link
* PTransform} but with the specified output tags. Does not modify this {@link PTransform}.
*
@@ -685,7 +731,6 @@
SchemaRegistry schemaRegistry = input.getPipeline().getSchemaRegistry();
CoderRegistry registry = input.getPipeline().getCoderRegistry();
finishSpecifyingStateSpecs(fn, registry, input.getCoder());
-
TupleTag<OutputT> mainOutput = new TupleTag<>(MAIN_OUTPUT_TAG);
PCollection<OutputT> res =
input.apply(withOutputTags(mainOutput, TupleTagList.empty())).get(mainOutput);
@@ -732,7 +777,7 @@
return fn;
}
- public List<PCollectionView<?>> getSideInputs() {
+ public Map<String, PCollectionView<?>> getSideInputs() {
return sideInputs;
}
@@ -743,7 +788,7 @@
*/
@Override
public Map<TupleTag<?>, PValue> getAdditionalInputs() {
- return PCollectionViews.toAdditionalInputs(sideInputs);
+ return PCollectionViews.toAdditionalInputs(sideInputs.values());
}
@Override
@@ -763,7 +808,7 @@
*/
public static class MultiOutput<InputT, OutputT>
extends PTransform<PCollection<? extends InputT>, PCollectionTuple> {
- private final List<PCollectionView<?>> sideInputs;
+ private final Map<String, PCollectionView<?>> sideInputs;
private final TupleTag<OutputT> mainOutputTag;
private final TupleTagList additionalOutputTags;
private final DisplayData.ItemSpec<? extends Class<?>> fnDisplayData;
@@ -771,7 +816,7 @@
MultiOutput(
DoFn<InputT, OutputT> fn,
- List<PCollectionView<?>> sideInputs,
+ Map<String, PCollectionView<?>> sideInputs,
TupleTag<OutputT> mainOutputTag,
TupleTagList additionalOutputTags,
ItemSpec<? extends Class<?>> fnDisplayData) {
@@ -793,6 +838,14 @@
return withSideInputs(Arrays.asList(sideInputs));
}
+ public MultiOutput<InputT, OutputT> withSideInputs(
+ Iterable<? extends PCollectionView<?>> sideInputs) {
+ Map<String, PCollectionView<?>> mappedInputs =
+ StreamSupport.stream(sideInputs.spliterator(), false)
+ .collect(Collectors.toMap(v -> v.getTagInternal().getId(), v -> v));
+ return withSideInputs(mappedInputs);
+ }
+
/**
* Returns a new multi-output {@link ParDo} {@link PTransform} that's like this {@link
* PTransform} but with the specified additional side inputs. Does not modify this {@link
@@ -800,19 +853,28 @@
*
* <p>See the discussion of Side Inputs above for more explanation.
*/
- public MultiOutput<InputT, OutputT> withSideInputs(
- Iterable<? extends PCollectionView<?>> sideInputs) {
+ public MultiOutput<InputT, OutputT> withSideInputs(Map<String, PCollectionView<?>> sideInputs) {
return new MultiOutput<>(
fn,
- ImmutableList.<PCollectionView<?>>builder()
- .addAll(this.sideInputs)
- .addAll(sideInputs)
+ ImmutableMap.<String, PCollectionView<?>>builder()
+ .putAll(this.sideInputs)
+ .putAll(sideInputs)
.build(),
mainOutputTag,
additionalOutputTags,
fnDisplayData);
}
+ /**
+ * Returns a new multi-output {@link ParDo} {@link PTransform} that's like this {@link
+ * PTransform} but with the specified additional side inputs. Does not modify this {@link
+ * PTransform}.
+ */
+ public MultiOutput<InputT, OutputT> withSideInput(
+ String tagId, PCollectionView<?> pCollectionView) {
+ return withSideInputs(Collections.singletonMap(tagId, pCollectionView));
+ }
+
@Override
public PCollectionTuple expand(PCollection<? extends InputT> input) {
// SplittableDoFn should be forbidden on the runner-side.
@@ -827,6 +889,8 @@
validateStateApplicableForInput(fn, input);
}
+ validateSideInputTypes(sideInputs, fn);
+
// TODO: We should validate OutputReceiver<Row> only happens if the output PCollection
// as schema. However coder/schema inference may not have happened yet at this point.
// Need to figure out where to validate this.
@@ -883,7 +947,7 @@
return additionalOutputTags;
}
- public List<PCollectionView<?>> getSideInputs() {
+ public Map<String, PCollectionView<?>> getSideInputs() {
return sideInputs;
}
@@ -894,7 +958,7 @@
*/
@Override
public Map<TupleTag<?>, PValue> getAdditionalInputs() {
- return PCollectionViews.toAdditionalInputs(sideInputs);
+ return PCollectionViews.toAdditionalInputs(sideInputs.values());
}
@Override
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Requirements.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Requirements.java
index a6db99f..064c383 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Requirements.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Requirements.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** Describes the run-time requirements of a {@link Contextful}, such as access to side inputs. */
@Experimental(Kind.CONTEXTFUL)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java
index ca7615d..0e24b7e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java
@@ -135,7 +135,7 @@
// http://hydronitrogen.com/poor-hash-partitioning-of-timestamps-integers-and-longs-in-
// spark.html
// This hashing strategy is copied from
- // org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Hashing.smear().
+ // org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Hashing.smear().
int hashOfShard = 0x1b873593 * Integer.rotateLeft(shard * 0xcc9e2d51, 15);
r.output(KV.of(hashOfShard, element));
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
index e119328..2594dee 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Iterator;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
new file mode 100644
index 0000000..28d6c46
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.rowToJson;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.util.RowJsonSerializer;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/**
+ * <i>Experimental</i>
+ *
+ * <p>Creates a {@link PTransform} that serializes UTF-8 JSON objects from a {@link Schema}-aware
+ * PCollection (i.e. {@link PCollection#hasSchema()} returns true). JSON format is compatible with
+ * {@link JsonToRow}.
+ *
+ * <p>For specifics of JSON serialization see {@link RowJsonSerializer}.
+ */
+@Experimental
+public class ToJson<T> extends PTransform<PCollection<T>, PCollection<String>> {
+ private transient volatile @Nullable ObjectMapper objectMapper;
+
+ static <T> ToJson<T> of() {
+ return new ToJson<T>();
+ }
+
+ @Override
+ public PCollection<String> expand(PCollection<T> rows) {
+ Schema inputSchema = rows.getSchema();
+ SerializableFunction<T, Row> toRow = rows.getToRowFunction();
+ return rows.apply(
+ ParDo.of(
+ new DoFn<T, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext context) {
+ context.output(
+ rowToJson(objectMapper(inputSchema), toRow.apply(context.element())));
+ }
+ }));
+ }
+
+ private ObjectMapper objectMapper(Schema schema) {
+ if (this.objectMapper == null) {
+ synchronized (this) {
+ if (this.objectMapper == null) {
+ this.objectMapper = newObjectMapperWith(RowJsonSerializer.forSchema(schema));
+ }
+ }
+ }
+
+ return this.objectMapper;
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
index 7164e27..015e517 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
@@ -19,7 +19,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
/**
* {@link PTransform PTransforms} for converting a {@link PCollection PCollection<?>}, {@link
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Top.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Top.java
index 425cd7d..e3f4533 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Top.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Top.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.InputStream;
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* {@code PTransform}s for finding the largest (or smallest) set of elements in a {@code
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
index df1debb..75583ce 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
@@ -32,6 +32,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PCollectionViews;
+import org.apache.beam.sdk.values.PCollectionViews.TypeDescriptorSupplier;
/**
* Transforms for creating {@link PCollectionView PCollectionViews} from {@link PCollection
@@ -233,9 +234,12 @@
PCollection<KV<Void, T>> materializationInput =
input.apply(new VoidKeyToMultimapMaterialization<>());
+ Coder<T> inputCoder = input.getCoder();
PCollectionView<List<T>> view =
PCollectionViews.listView(
- materializationInput, materializationInput.getWindowingStrategy());
+ materializationInput,
+ (TypeDescriptorSupplier<T>) inputCoder::getEncodedTypeDescriptor,
+ materializationInput.getWindowingStrategy());
materializationInput.apply(CreatePCollectionView.of(view));
return view;
}
@@ -263,9 +267,12 @@
PCollection<KV<Void, T>> materializationInput =
input.apply(new VoidKeyToMultimapMaterialization<>());
+ Coder<T> inputCoder = input.getCoder();
PCollectionView<Iterable<T>> view =
PCollectionViews.iterableView(
- materializationInput, materializationInput.getWindowingStrategy());
+ materializationInput,
+ (TypeDescriptorSupplier<T>) inputCoder::getEncodedTypeDescriptor,
+ materializationInput.getWindowingStrategy());
materializationInput.apply(CreatePCollectionView.of(view));
return view;
}
@@ -402,11 +409,17 @@
throw new IllegalStateException("Unable to create a side-input view from input", e);
}
+ KvCoder<K, V> kvCoder = (KvCoder<K, V>) input.getCoder();
+ Coder<K> keyCoder = kvCoder.getKeyCoder();
+ Coder<V> valueCoder = kvCoder.getValueCoder();
PCollection<KV<Void, KV<K, V>>> materializationInput =
input.apply(new VoidKeyToMultimapMaterialization<>());
PCollectionView<Map<K, Iterable<V>>> view =
PCollectionViews.multimapView(
- materializationInput, materializationInput.getWindowingStrategy());
+ materializationInput,
+ (TypeDescriptorSupplier<K>) keyCoder::getEncodedTypeDescriptor,
+ (TypeDescriptorSupplier<V>) valueCoder::getEncodedTypeDescriptor,
+ materializationInput.getWindowingStrategy());
materializationInput.apply(CreatePCollectionView.of(view));
return view;
}
@@ -438,11 +451,18 @@
throw new IllegalStateException("Unable to create a side-input view from input", e);
}
+ KvCoder<K, V> kvCoder = (KvCoder<K, V>) input.getCoder();
+ Coder<K> keyCoder = kvCoder.getKeyCoder();
+ Coder<V> valueCoder = kvCoder.getValueCoder();
+
PCollection<KV<Void, KV<K, V>>> materializationInput =
input.apply(new VoidKeyToMultimapMaterialization<>());
PCollectionView<Map<K, V>> view =
PCollectionViews.mapView(
- materializationInput, materializationInput.getWindowingStrategy());
+ materializationInput,
+ (TypeDescriptorSupplier<K>) keyCoder::getEncodedTypeDescriptor,
+ (TypeDescriptorSupplier<V>) valueCoder::getEncodedTypeDescriptor,
+ materializationInput.getWindowingStrategy());
materializationInput.apply(CreatePCollectionView.of(view));
return view;
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
index 17a9e9c..41a3a4f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ViewFn.java
@@ -21,6 +21,7 @@
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.TypeDescriptor;
/**
* <b><i>For internal use only; no backwards-compatibility guarantees.</i></b>
@@ -45,4 +46,7 @@
/** A function to adapt a primitive view type to a desired view type. */
public abstract ViewT apply(PrimitiveViewT primitiveViewT);
+
+ /** Return the {@link TypeDescriptor} describing the output of this fn. */
+ public abstract TypeDescriptor<ViewT> getTypeDescriptor();
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java
index 7d83ace..f3583b6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Wait.java
@@ -30,8 +30,8 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* Delays processing of each window in a {@link PCollection} until signaled.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java
index 093cf43..8d5ed4b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java
@@ -20,9 +20,9 @@
import static org.apache.beam.sdk.transforms.Contextful.Fn.Context.wrapProcessContext;
import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.resume;
import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.stop;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -65,16 +65,16 @@
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.TypeDescriptors.TypeVariableExtractor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnel;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnels;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashCode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnel;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnels;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.ReadableDuration;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithFailures.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithFailures.java
index 357b1d4..795d232 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithFailures.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithFailures.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A collection of utilities for writing transforms that can handle exceptions raised during
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithKeys.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithKeys.java
index 6d50889..60bd4a9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithKeys.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithKeys.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import javax.annotation.CheckForNull;
import javax.annotation.Nullable;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
index d0abbbb..c410b84 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.sdk.io.Source;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
index 24d6047..94b2943 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/DisplayData.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms.display;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.annotation.JsonGetter;
import com.fasterxml.jackson.annotation.JsonIgnore;
@@ -34,11 +34,11 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.DateTimeFormatter;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGbkResult.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGbkResult.java
index df97041..028884f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGbkResult.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/CoGbkResult.java
@@ -33,10 +33,10 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.PeekingIterator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.PeekingIterator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/KeyedPCollectionTuple.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/KeyedPCollectionTuple.java
index 7bb2781..9818474 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/KeyedPCollectionTuple.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/join/KeyedPCollectionTuple.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* An immutable tuple of keyed {@link PCollection PCollections} with key type K. ({@link PCollection
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
index 457ee55..2f54c27 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.transforms.reflect;
import static org.apache.beam.sdk.util.common.ReflectHelpers.findClassLoader;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
@@ -29,40 +29,6 @@
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.field.FieldDescription;
-import net.bytebuddy.description.method.MethodDescription;
-import net.bytebuddy.description.modifier.Visibility;
-import net.bytebuddy.description.type.TypeDescription;
-import net.bytebuddy.description.type.TypeDescription.ForLoadedType;
-import net.bytebuddy.description.type.TypeList;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.dynamic.scaffold.subclass.ConstructorStrategy;
-import net.bytebuddy.implementation.ExceptionMethod;
-import net.bytebuddy.implementation.FixedValue;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.Implementation.Context;
-import net.bytebuddy.implementation.MethodDelegation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
-import net.bytebuddy.implementation.bytecode.Throw;
-import net.bytebuddy.implementation.bytecode.assign.Assigner;
-import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
-import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
-import net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
-import net.bytebuddy.implementation.bytecode.constant.TextConstant;
-import net.bytebuddy.implementation.bytecode.member.FieldAccess;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.jar.asm.Label;
-import net.bytebuddy.jar.asm.MethodVisitor;
-import net.bytebuddy.jar.asm.Opcodes;
-import net.bytebuddy.jar.asm.Type;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderRegistry;
@@ -79,6 +45,7 @@
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.ProcessContextParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SideInputParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StartBundleContextParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TaggedOutputReceiverParameter;
@@ -90,7 +57,41 @@
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Primitives;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.field.FieldDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.method.MethodDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.Visibility;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeList;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.subclass.ConstructorStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.ExceptionMethod;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.FixedValue;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation.Context;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.MethodDelegation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation.Compound;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.Throw;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.assign.TypeCasting;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.TextConstant;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.Label;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.MethodVisitor;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.Opcodes;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.Type;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Primitives;
/** Dynamically generates a {@link DoFnInvoker} instances for invoking a {@link DoFn}. */
public class ByteBuddyDoFnInvokerFactory implements DoFnInvokerFactory {
@@ -112,6 +113,7 @@
public static final String RESTRICTION_TRACKER_PARAMETER_METHOD = "restrictionTracker";
public static final String STATE_PARAMETER_METHOD = "state";
public static final String TIMER_PARAMETER_METHOD = "timer";
+ public static final String SIDE_INPUT_PARAMETER_METHOD = "sideInput";
/**
* Returns a {@link ByteBuddyDoFnInvokerFactory} shared with all other invocations, so that its
@@ -787,6 +789,16 @@
public StackManipulation dispatch(DoFnSignature.Parameter.PipelineOptionsParameter p) {
return simpleExtraContextParameter(PIPELINE_OPTIONS_PARAMETER_METHOD);
}
+
+ @Override
+ public StackManipulation dispatch(SideInputParameter p) {
+ return new StackManipulation.Compound(
+ new TextConstant(p.sideInputId()),
+ MethodInvocation.invoke(
+ getExtraContextFactoryMethodDescription(
+ SIDE_INPUT_PARAMETER_METHOD, String.class)),
+ TypeCasting.to(new TypeDescription.ForLoadedType(p.elementT().getRawType())));
+ }
});
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
index c0902a0..b997653 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
@@ -22,32 +22,32 @@
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.util.concurrent.ExecutionException;
-import net.bytebuddy.ByteBuddy;
-import net.bytebuddy.description.modifier.FieldManifestation;
-import net.bytebuddy.description.modifier.Visibility;
-import net.bytebuddy.description.type.TypeDescription;
-import net.bytebuddy.dynamic.DynamicType;
-import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
-import net.bytebuddy.dynamic.scaffold.InstrumentedType;
-import net.bytebuddy.dynamic.scaffold.subclass.ConstructorStrategy;
-import net.bytebuddy.implementation.Implementation;
-import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
-import net.bytebuddy.implementation.bytecode.StackManipulation;
-import net.bytebuddy.implementation.bytecode.member.FieldAccess;
-import net.bytebuddy.implementation.bytecode.member.MethodInvocation;
-import net.bytebuddy.implementation.bytecode.member.MethodReturn;
-import net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
-import net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFn.OnTimer;
import org.apache.beam.sdk.transforms.DoFn.TimerId;
import org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory.DoFnMethodWithExtraParametersDelegation;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.CharMatcher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.FieldManifestation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.modifier.Visibility;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.scaffold.subclass.ConstructorStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.Implementation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.StackManipulation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.CharMatcher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
/**
* Dynamically generates {@link OnTimerInvoker} instances for invoking a particular {@link TimerId}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
index d8504ee..fe31c64 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
@@ -135,6 +135,9 @@
/** Provide a reference to the input element. */
InputT element(DoFn<InputT, OutputT> doFn);
+ /** Provide a reference to the input sideInput with the specified tag. */
+ Object sideInput(String tagId);
+
/**
* Provide a reference to the selected schema field corresponding to the input argument
* specified by index.
@@ -191,6 +194,14 @@
}
@Override
+ public InputT sideInput(String tagId) {
+ throw new UnsupportedOperationException(
+ String.format(
+ "Should never call non-overridden methods of %s",
+ FakeArgumentProvider.class.getSimpleName()));
+ }
+
+ @Override
public InputT schemaElement(int index) {
throw new UnsupportedOperationException(
String.format(
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 0b09c9e..5737ac9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -40,6 +40,7 @@
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.OutputReceiverParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.RestrictionTrackerParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SideInputParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimerParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.WindowParameter;
@@ -47,7 +48,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
/**
* Describes the signature of a {@link DoFn}, in particular, which features it uses, which extra
@@ -242,6 +243,8 @@
return cases.dispatch((TaggedOutputReceiverParameter) this);
} else if (this instanceof TimeDomainParameter) {
return cases.dispatch((TimeDomainParameter) this);
+ } else if (this instanceof SideInputParameter) {
+ return cases.dispatch((SideInputParameter) this);
} else {
throw new IllegalStateException(
String.format(
@@ -284,6 +287,8 @@
ResultT dispatch(PipelineOptionsParameter p);
+ ResultT dispatch(SideInputParameter p);
+
/** A base class for a visitor with a default method for cases it is not interested in. */
abstract class WithDefault<ResultT> implements Cases<ResultT> {
@@ -368,6 +373,11 @@
public ResultT dispatch(PipelineOptionsParameter p) {
return dispatchDefault(p);
}
+
+ @Override
+ public ResultT dispatch(SideInputParameter p) {
+ return dispatchDefault(p);
+ }
}
}
@@ -413,6 +423,14 @@
return TIMESTAMP_PARAMETER;
}
+ public static SideInputParameter sideInputParameter(
+ TypeDescriptor<?> elementT, String sideInputId) {
+ return new AutoValue_DoFnSignature_Parameter_SideInputParameter.Builder()
+ .setElementT(elementT)
+ .setSideInputId(sideInputId)
+ .build();
+ }
+
public static TimeDomainParameter timeDomainParameter() {
return TIME_DOMAIN_PARAMETER;
}
@@ -556,6 +574,28 @@
TimeDomainParameter() {}
}
+ /** Descriptor for a {@link Parameter} of type {@link DoFn.SideInput}. */
+ @AutoValue
+ public abstract static class SideInputParameter extends Parameter {
+ SideInputParameter() {}
+
+ public abstract TypeDescriptor<?> elementT();
+
+ public abstract String sideInputId();
+
+ /** Builder class. */
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract SideInputParameter.Builder setElementT(TypeDescriptor<?> elementT);
+
+ public abstract SideInputParameter.Builder setSideInputId(String sideInput);
+
+ public abstract SideInputParameter build();
+ }
+
+ public abstract SideInputParameter.Builder toBuilder();
+ }
+
/**
* Descriptor for a {@link Parameter} of type {@link DoFn.OutputReceiver}.
*
@@ -721,6 +761,14 @@
.collect(Collectors.toList());
}
+ @Nullable
+ public List<SideInputParameter> getSideInputParameters() {
+ return extraParameters().stream()
+ .filter(Predicates.instanceOf(SideInputParameter.class)::apply)
+ .map(SideInputParameter.class::cast)
+ .collect(Collectors.toList());
+ }
+
/** The {@link OutputReceiverParameter} for a main output, or null if there is none. */
@Nullable
public OutputReceiverParameter getMainOutputReceiver() {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
index 69b03a4..b3fde4f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms.reflect;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.lang.annotation.Annotation;
@@ -49,6 +49,7 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver;
import org.apache.beam.sdk.transforms.DoFn.OutputReceiver;
+import org.apache.beam.sdk.transforms.DoFn.SideInput;
import org.apache.beam.sdk.transforms.DoFn.StateId;
import org.apache.beam.sdk.transforms.DoFn.TimerId;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.FieldAccessDeclaration;
@@ -69,11 +70,11 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeParameter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.Instant;
/** Utilities for working with {@link DoFnSignature}. See {@link #getSignature}. */
@@ -96,7 +97,8 @@
Parameter.PaneInfoParameter.class,
Parameter.PipelineOptionsParameter.class,
Parameter.TimerParameter.class,
- Parameter.StateParameter.class);
+ Parameter.StateParameter.class,
+ Parameter.SideInputParameter.class);
private static final ImmutableList<Class<? extends Parameter>>
ALLOWED_SPLITTABLE_PROCESS_ELEMENT_PARAMETERS =
@@ -107,7 +109,8 @@
Parameter.OutputReceiverParameter.class,
Parameter.TaggedOutputReceiverParameter.class,
Parameter.ProcessContextParameter.class,
- Parameter.RestrictionTrackerParameter.class);
+ Parameter.RestrictionTrackerParameter.class,
+ Parameter.SideInputParameter.class);
private static final ImmutableList<Class<? extends Parameter>> ALLOWED_ON_TIMER_PARAMETERS =
ImmutableList.of(
@@ -255,7 +258,6 @@
public Map<String, TimerParameter> getTimerParameters() {
return Collections.unmodifiableMap(timerParameters);
}
-
/** Extra parameters in their entirety. Unmodifiable. */
public List<Parameter> getExtraParameters() {
return Collections.unmodifiableList(extraParameters);
@@ -371,7 +373,7 @@
TimerDeclaration timerDecl = fnContext.getTimerDeclarations().get(id);
errors.checkArgument(
- timerDecl.field().getDeclaringClass().equals(onTimerMethod.getDeclaringClass()),
+ timerDecl.field().getDeclaringClass().equals(getDeclaringClass(onTimerMethod)),
"Callback %s is for timer %s declared in a different class %s."
+ " Timer callbacks must be declared in the same lexical scope as their timer",
onTimerMethod,
@@ -479,6 +481,14 @@
return signature;
}
+ private static Class<?> getDeclaringClass(Method onTimerMethod) {
+ Class<?> declaringClass = onTimerMethod.getDeclaringClass();
+ if (declaringClass.getName().contains("$MockitoMock$")) {
+ declaringClass = declaringClass.getSuperclass();
+ }
+ return declaringClass;
+ }
+
/**
* Infers the boundedness of the {@link DoFn.ProcessElement} method (whether or not it performs a
* bounded amount of work per element) using the following criteria:
@@ -896,6 +906,11 @@
return Parameter.timestampParameter();
} else if (rawType.equals(TimeDomain.class)) {
return Parameter.timeDomainParameter();
+ } else if (hasSideInputAnnotation(param.getAnnotations())) {
+ String sideInputId = getSideInputId(param.getAnnotations());
+ paramErrors.checkArgument(
+ sideInputId != null, "%s missing %s annotation", SideInput.class.getSimpleName());
+ return Parameter.sideInputParameter(paramT, sideInputId);
} else if (rawType.equals(PaneInfo.class)) {
return Parameter.paneInfoParameter();
} else if (rawType.equals(DoFn.ProcessContext.class)) {
@@ -969,7 +984,7 @@
id);
paramErrors.checkArgument(
- timerDecl.field().getDeclaringClass().equals(param.getMethod().getDeclaringClass()),
+ timerDecl.field().getDeclaringClass().equals(getDeclaringClass(param.getMethod())),
"%s %s declared in a different class %s."
+ " Timers may be referenced only in the lexical scope where they are declared.",
TimerId.class.getSimpleName(),
@@ -1008,7 +1023,7 @@
formatType(stateDecl.stateType()));
paramErrors.checkArgument(
- stateDecl.field().getDeclaringClass().equals(param.getMethod().getDeclaringClass()),
+ stateDecl.field().getDeclaringClass().equals(getDeclaringClass(param.getMethod())),
"%s %s declared in a different class %s."
+ " State may be referenced only in the class where it is declared.",
StateId.class.getSimpleName(),
@@ -1048,6 +1063,12 @@
}
@Nullable
+ private static String getSideInputId(List<Annotation> annotations) {
+ DoFn.SideInput sideInputId = findFirstOfType(annotations, DoFn.SideInput.class);
+ return sideInputId != null ? sideInputId.value() : null;
+ }
+
+ @Nullable
static <T> T findFirstOfType(List<Annotation> annotations, Class<T> clazz) {
Optional<Annotation> annotation =
annotations.stream().filter(a -> a.annotationType().equals(clazz)).findFirst();
@@ -1062,6 +1083,10 @@
return annotations.stream().anyMatch(a -> a.annotationType().equals(DoFn.Timestamp.class));
}
+ private static boolean hasSideInputAnnotation(List<Annotation> annotations) {
+ return annotations.stream().anyMatch(a -> a.annotationType().equals(DoFn.SideInput.class));
+ }
+
@Nullable
private static TypeDescriptor<?> getTrackerType(TypeDescriptor<?> fnClass, Method method) {
Type[] params = method.getGenericParameterTypes();
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
index 0cf2002..06541ab 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/StableInvokerNamingStrategy.java
@@ -17,13 +17,13 @@
*/
package org.apache.beam.sdk.transforms.reflect;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import com.google.auto.value.AutoValue;
import javax.annotation.Nullable;
-import net.bytebuddy.NamingStrategy;
-import net.bytebuddy.description.type.TypeDescription;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.NamingStrategy;
+import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription;
/**
* A naming strategy for ByteBuddy invokers ({@link DoFnInvoker} and {@link OnTimerInvoker}) that is
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlog.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlog.java
deleted file mode 100644
index 2712f9b..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlog.java
+++ /dev/null
@@ -1,90 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms.splittabledofn;
-
-import com.google.auto.value.AutoValue;
-import java.math.BigDecimal;
-import javax.annotation.Nullable;
-
-/**
- * A representation for the amount of known work represented as a backlog. Note that backlog {@code
- * byte[]} representations must be lexicographically comparable. Backlog {@code byte[]}
- * representations should preferably represent a linear space and be comparable within the same
- * partition.
- *
- * <p>It is up to each restriction tracker to convert between their natural representation of
- * outstanding work and this representation. For example:
- *
- * <ul>
- * <li>Block based file source (e.g. Avro): From the end of the current block, the remaining
- * number of bytes to the end of the restriction represented as a big endian 64 bit unsigned
- * integer.
- * <li>Pull based queue based source (e.g. Pubsub): The local/global backlog available in number
- * of messages or number of {@code messages / bytes} that have not been processed represented
- * as a big endian 64 bit unsigned integer.
- * <li>Key range based source (e.g. Shuffle, Bigtable, ...): Scale the start key to be one and end
- * key to be zero and interpolate the position of the next splittable key as the backlog. If
- * information about the probability density function or cumulative distribution function is
- * available, backlog interpolation can be improved. Alternatively, if the number of encoded
- * bytes for the keys and values is known for the key range, the backlog of remaining bytes
- * can be used.
- * </ul>
- *
- * <p>{@link RestrictionTracker}s should implement {@link Backlogs.HasBacklog} to report a backlog
- * where the element and restriction pair uniquely identify the resource. Otherwise {@link
- * RestrictionTracker}s should implement {@link Backlogs.HasPartitionedBacklog} to report a backlog
- * for a shared resource such as a message queue.
- *
- * <p>See <a href="https://s.apache.org/beam-bundles-backlog-splitting">Bundles w/ SplittableDoFns:
- * Backlog & Splitting</a> for further details.
- */
-@AutoValue
-public abstract class Backlog {
- private static final Backlog UNKNOWN_BACKLOG = new AutoValue_Backlog(null);
-
- /** Returns a backlog represented by the specified bytes. */
- public static Backlog of(BigDecimal backlog) {
- return new AutoValue_Backlog(backlog);
- }
-
- /** Returns an unknown backlog. */
- public static Backlog unknown() {
- return UNKNOWN_BACKLOG;
- }
-
- /** Returns whether this backlog is known or unknown. */
- public boolean isUnknown() {
- return backlogInternal() == null;
- }
-
- /**
- * Returns the {@code byte[]} representation of the backlog if it is known.
- *
- * @throws IllegalStateException if the backlog is unknown.
- */
- public BigDecimal backlog() {
- if (isUnknown()) {
- throw new IllegalStateException("Backlog is unknown, there is no byte[] representation.");
- }
- return backlogInternal();
- }
-
- @SuppressWarnings("mutable")
- @Nullable
- protected abstract BigDecimal backlogInternal();
-}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlogs.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlogs.java
deleted file mode 100644
index 2bd21e6..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlogs.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms.splittabledofn;
-
-/** Definitions and convenience methods for reporting/consuming/updating backlogs. */
-public final class Backlogs {
- /**
- * {@link RestrictionTracker}s which can provide a backlog should implement this interface.
- * Implementations that do not implement this interface will be assumed to have an unknown
- * backlog.
- *
- * <p>By default, the backlog partition identifier is represented as the encoded element and
- * restriction pair. See {@link HasPartitionedBacklog} for {@link RestrictionTracker}s that report
- * backlogs over a shared resource.
- */
- public interface HasBacklog {
- Backlog getBacklog();
- }
-
- /**
- * {@link RestrictionTracker}s which can provide a backlog that is from a shared resource such as
- * a message queue should implement this interface to provide the partition identifier. The
- * partition identifier is used by runners for various backlog calculations. Backlogs reported
- * with the same partition identifier represent a point in time reporting of the backlog for that
- * partition. For example, a runner can compute a global backlog by summing all reported backlogs
- * over all unique partition identifiers.
- *
- * <p>For example SplittableDoFn's which consume elements from:
- *
- * <ul>
- * <li>a globally shared resource such as a Pubsub queue should set this to "".
- * <li>a shared partitioned resource should use the partition identifier.
- * <li>a uniquely partitioned resource such as a file range should set this to file name + start
- * offset. Note that the default for {@link RestrictionTracker}s is to use the encoded
- * element and restriction pair.
- * </ul>
- *
- * <p>Returns an immutable representation of the partition identifier.
- */
- public interface HasPartitionedBacklog extends HasBacklog {
- byte[] getBacklogPartition();
- }
-}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java
index 44f2f0b..f8a577d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java
@@ -17,17 +17,16 @@
*/
package org.apache.beam.sdk.transforms.splittabledofn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
-import java.math.BigDecimal;
import javax.annotation.Nullable;
import org.apache.beam.sdk.io.range.ByteKey;
import org.apache.beam.sdk.io.range.ByteKeyRange;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Bytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
/**
* A {@link RestrictionTracker} for claiming {@link ByteKey}s in a {@link ByteKeyRange} in a
@@ -38,7 +37,7 @@
* to process.
*/
public class ByteKeyRangeTracker extends RestrictionTracker<ByteKeyRange, ByteKey>
- implements Backlogs.HasBacklog {
+ implements Sizes.HasSize {
/* An empty range which contains no keys. */
@VisibleForTesting
static final ByteKeyRange NO_KEYS = ByteKeyRange.of(ByteKey.EMPTY, ByteKey.of(0x00));
@@ -184,28 +183,26 @@
private static final byte[] ZERO_BYTE_ARRAY = new byte[] {0};
@Override
- public Backlog getBacklog() {
+ public double getSize() {
// Return 0 for the empty range which is implicitly done.
// This case can occur if the range tracker is checkpointed before any keys have been claimed
// or if the range tracker is checkpointed once the range is done.
if (NO_KEYS.equals(range)) {
- return Backlog.of(BigDecimal.ZERO);
+ return 0;
}
// If we are attempting to get the backlog without processing a single key, we return 1.0
if (lastAttemptedKey == null) {
- return Backlog.of(BigDecimal.ONE);
+ return 1;
}
// Return 0 if the last attempted key was the empty key representing the end of range for
// all ranges or the last attempted key is beyond the end of the range.
if (lastAttemptedKey.isEmpty()
|| !(range.getEndKey().isEmpty() || range.getEndKey().compareTo(lastAttemptedKey) > 0)) {
- return Backlog.of(BigDecimal.ZERO);
+ return 0;
}
- // TODO: Use the ability of BigDecimal's additional precision to more accurately report backlog
- // for keys which are long.
- return Backlog.of(BigDecimal.valueOf(range.estimateFractionForKey(lastAttemptedKey)));
+ return range.estimateFractionForKey(lastAttemptedKey);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTracker.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTracker.java
index 9d90c69..69641a7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTracker.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTracker.java
@@ -17,21 +17,20 @@
*/
package org.apache.beam.sdk.transforms.splittabledofn;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
-import java.math.BigDecimal;
import javax.annotation.Nullable;
import org.apache.beam.sdk.io.range.OffsetRange;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A {@link RestrictionTracker} for claiming offsets in an {@link OffsetRange} in a monotonically
* increasing fashion.
*/
public class OffsetRangeTracker extends RestrictionTracker<OffsetRange, Long>
- implements Backlogs.HasBacklog {
+ implements Sizes.HasSize {
private OffsetRange range;
@Nullable private Long lastClaimedOffset = null;
@Nullable private Long lastAttemptedOffset = null;
@@ -101,14 +100,14 @@
}
@Override
- public Backlog getBacklog() {
+ public double getSize() {
// If we have never attempted an offset, we return the length of the entire range.
if (lastAttemptedOffset == null) {
- return Backlog.of(BigDecimal.valueOf(range.getTo() - range.getFrom()));
+ return range.getTo() - range.getFrom();
}
// Otherwise we return the length from where we are to where we are attempting to get to
// with a minimum of zero in case we have claimed beyond the end of the range.
- return Backlog.of(BigDecimal.valueOf(Math.max(range.getTo() - lastAttemptedOffset, 0)));
+ return Math.max(range.getTo() - lastAttemptedOffset, 0);
}
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Sizes.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Sizes.java
new file mode 100644
index 0000000..3bbcb7e
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Sizes.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms.splittabledofn;
+
+import org.apache.beam.sdk.transforms.DoFn;
+
+/** Definitions and convenience methods for reporting sizes for SplittableDoFns. */
+public final class Sizes {
+ /**
+ * {@link RestrictionTracker}s which can provide a size should implement this interface.
+ * Implementations that do not implement this interface will be assumed to have an equivalent
+ * size.
+ */
+ public interface HasSize {
+ /**
+ * A representation for the amount of known work represented as a size. Size {@code double}
+ * representations should preferably represent a linear space and be comparable within the same
+ * partition.
+ *
+ * <p>It is up to each restriction tracker to convert between their natural representation of
+ * outstanding work and this representation. For example:
+ *
+ * <ul>
+ * <li>Block based file source (e.g. Avro): From the end of the current block, the remaining
+ * number of bytes to the end of the restriction.
+ * <li>Pull based queue based source (e.g. Pubsub): The local/global size available in number
+ * of messages or number of {@code message bytes} that have not been processed.
+ * <li>Key range based source (e.g. Shuffle, Bigtable, ...): Scale the start key to be one and
+ * end key to be zero and interpolate the position of the next splittable key as the size.
+ * If information about the probability density function or cumulative distribution
+ * function is available, size interpolation can be improved. Alternatively, if the number
+ * of encoded bytes for the keys and values is known for the key range, the number of
+ * remaining bytes can be used.
+ * </ul>
+ *
+ * <p>{@link DoFn}s should provide a method annotated with {@link DoFn.GetPartition} to report a
+ * partition identifier if the element and restriction represent the size for a shared resource
+ * such as a message queue topic. See {@link DoFn.GetPartition} for additional details.
+ */
+ double getSize();
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
index 71a4473..35c0d94 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.joda.time.Instant;
/** A composite {@link Trigger} that fires when all of its sub-triggers are ready. */
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
index 891ece1..2ce2fdf 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
index c61937d..b131935 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
@@ -17,13 +17,13 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterProcessingTime.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterProcessingTime.java
index 48a9510..d17aaa9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterProcessingTime.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterProcessingTime.java
@@ -23,7 +23,7 @@
import java.util.Objects;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.format.PeriodFormat;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
index c5274ff..6c1446b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
@@ -19,7 +19,7 @@
import java.util.List;
import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
index 4f64ba1..2be41de 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.List;
import java.util.Objects;
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.state.TimeDomain;
import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
index f92de2b..a8f6659 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
@@ -19,7 +19,7 @@
import java.util.Arrays;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
index d3c51ee..6e1969e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.InputStream;
@@ -30,8 +30,8 @@
import org.apache.beam.sdk.transforms.DoFn.WindowedContext;
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.util.VarInt;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Provides information about the pane an element belongs to. Every pane is implicitly associated
@@ -165,7 +165,7 @@
private final long nonSpeculativeIndex;
/**
- * {@code PaneInfo} to use for elements on (and before) initial window assignemnt (including
+ * {@code PaneInfo} to use for elements on (and before) initial window assignment (including
* elements read from sources) before they have passed through a {@link GroupByKey} and are
* associated with a particular trigger firing.
*/
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Sessions.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Sessions.java
index 1681369..9bd4798 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Sessions.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Sessions.java
@@ -22,6 +22,7 @@
import java.util.Objects;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.joda.time.Duration;
/**
@@ -86,6 +87,11 @@
}
@Override
+ public TypeDescriptor<IntervalWindow> getWindowTypeDescriptor() {
+ return TypeDescriptor.of(IntervalWindow.class);
+ }
+
+ @Override
public WindowMappingFn<IntervalWindow> getDefaultWindowMappingFn() {
throw new UnsupportedOperationException("Sessions is not allowed in side inputs");
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/TimestampCombiner.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/TimestampCombiner.java
index be7b35a..93d2908 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/TimestampCombiner.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/TimestampCombiner.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.transforms.windowing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.Collections;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
index 50f9d6e..ffddebd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index 90bc45c..c4d1a0c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
index f59b95f..7add6e3 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ApiSurface.java
@@ -35,22 +35,22 @@
import java.util.List;
import java.util.Set;
import java.util.regex.Pattern;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimaps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.Invokable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.Parameter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.TypeToken;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimaps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.Invokable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.Parameter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.TypeToken;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
import org.hamcrest.StringDescription;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AppliedCombineFn.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AppliedCombineFn.java
index ebb5a9f..e3f8fb4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AppliedCombineFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AppliedCombineFn.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link GlobalCombineFn} with a fixed accumulator coder. This is created from a specific
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BucketingFunction.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BucketingFunction.java
index cbe3979..638b345 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BucketingFunction.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BucketingFunction.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.HashMap;
import java.util.Map;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
index 514e8a5..79b3d15 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStream.java
@@ -23,7 +23,7 @@
import java.util.concurrent.ArrayBlockingQueue;
import javax.annotation.concurrent.NotThreadSafe;
import org.apache.beam.sdk.coders.Coder.Context;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Provides an efficient encoding for {@link Iterable}s containing small values by buffering up to
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ClassPath.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ClassPath.java
index 378490c..f3f43df 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ClassPath.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ClassPath.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.File;
import java.io.IOException;
@@ -37,22 +37,22 @@
import java.util.jar.JarFile;
import java.util.jar.Manifest;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.Beta;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.CharMatcher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.MultimapBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SetMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Resources;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.Reflection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.Beta;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.CharMatcher;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.MultimapBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Resources;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.Reflection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CoderUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CoderUtils.java
index f798c6b..6aa5d09 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CoderUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/CoderUtils.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
/** Utilities for working with Coders. */
public final class CoderUtils {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnInfo.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnInfo.java
index 32998b4..9097277 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnInfo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnInfo.java
@@ -41,6 +41,7 @@
Map<TupleTag<?>, Coder<?>> outputCoders;
private final TupleTag<OutputT> mainOutput;
private final DoFnSchemaInformation doFnSchemaInformation;
+ private final Map<String, PCollectionView<?>> sideInputMapping;
/**
* Creates a {@link DoFnInfo} for the given {@link DoFn}.
@@ -54,7 +55,8 @@
Iterable<PCollectionView<?>> sideInputViews,
Coder<InputT> inputCoder,
TupleTag<OutputT> mainOutput,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
return new DoFnInfo<>(
doFn,
windowingStrategy,
@@ -62,7 +64,8 @@
inputCoder,
Collections.emptyMap(),
mainOutput,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
}
/** Creates a {@link DoFnInfo} for the given {@link DoFn}. */
@@ -73,7 +76,8 @@
Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
TupleTag<OutputT> mainOutput,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
return new DoFnInfo<>(
doFn,
windowingStrategy,
@@ -81,7 +85,8 @@
inputCoder,
outputCoders,
mainOutput,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
}
public DoFnInfo<InputT, OutputT> withFn(DoFn<InputT, OutputT> newFn) {
@@ -92,7 +97,8 @@
inputCoder,
outputCoders,
mainOutput,
- doFnSchemaInformation);
+ doFnSchemaInformation,
+ sideInputMapping);
}
private DoFnInfo(
@@ -102,7 +108,8 @@
Coder<InputT> inputCoder,
Map<TupleTag<?>, Coder<?>> outputCoders,
TupleTag<OutputT> mainOutput,
- DoFnSchemaInformation doFnSchemaInformation) {
+ DoFnSchemaInformation doFnSchemaInformation,
+ Map<String, PCollectionView<?>> sideInputMapping) {
this.doFn = doFn;
this.windowingStrategy = windowingStrategy;
this.sideInputViews = sideInputViews;
@@ -110,6 +117,7 @@
this.outputCoders = outputCoders;
this.mainOutput = mainOutput;
this.doFnSchemaInformation = doFnSchemaInformation;
+ this.sideInputMapping = sideInputMapping;
}
/** Returns the embedded function. */
@@ -140,4 +148,8 @@
public DoFnSchemaInformation getDoFnSchemaInformation() {
return doFnSchemaInformation;
}
+
+ public Map<String, PCollectionView<?>> getSideInputMapping() {
+ return sideInputMapping;
+ }
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnWithExecutionInformation.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnWithExecutionInformation.java
index a812eca..0306a15 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnWithExecutionInformation.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/DoFnWithExecutionInformation.java
@@ -19,21 +19,29 @@
import com.google.auto.value.AutoValue;
import java.io.Serializable;
+import java.util.Map;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
+import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
/** The data that the Java SDK harness needs to execute a DoFn. */
@AutoValue
public abstract class DoFnWithExecutionInformation implements Serializable {
public static DoFnWithExecutionInformation of(
- DoFn<?, ?> fn, TupleTag<?> tag, DoFnSchemaInformation doFnSchemaInformation) {
- return new AutoValue_DoFnWithExecutionInformation(fn, tag, doFnSchemaInformation);
+ DoFn<?, ?> fn,
+ TupleTag<?> tag,
+ Map<String, PCollectionView<?>> sideInputMapping,
+ DoFnSchemaInformation doFnSchemaInformation) {
+ return new AutoValue_DoFnWithExecutionInformation(
+ fn, tag, sideInputMapping, doFnSchemaInformation);
}
public abstract DoFn<?, ?> getDoFn();
public abstract TupleTag<?> getMainOutputTag();
+ public abstract Map<String, PCollectionView<?>> getSideInputMapping();
+
public abstract DoFnSchemaInformation getSchemaInformation();
}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExplicitShardedFile.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExplicitShardedFile.java
index fa9148d..2a5ec6d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExplicitShardedFile.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ExplicitShardedFile.java
@@ -27,10 +27,10 @@
import java.util.List;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharStreams;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFile.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFile.java
index ea231c2..4e2f610 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFile.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFile.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.Reader;
@@ -29,10 +29,10 @@
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharStreams;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java
index c8a2418..81e8617 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FluentBackoff.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/InstanceBuilder.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/InstanceBuilder.java
index 57998a9..ddbd158 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/InstanceBuilder.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/InstanceBuilder.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
@@ -28,7 +28,7 @@
import java.util.List;
import javax.annotation.Nullable;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
/**
* Utility for creating objects dynamically.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
deleted file mode 100644
index 8ac834c..0000000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import com.fasterxml.jackson.core.JsonParseException;
-import com.fasterxml.jackson.databind.JsonMappingException;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.module.SimpleModule;
-import java.io.IOException;
-import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
-import org.apache.beam.sdk.values.Row;
-
-/** JsonToRowUtils. */
-@Internal
-public class JsonToRowUtils {
-
- public static ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
- SimpleModule module = new SimpleModule("rowDeserializationModule");
- module.addDeserializer(Row.class, deserializer);
-
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.registerModule(module);
-
- return objectMapper;
- }
-
- public static Row jsonToRow(ObjectMapper objectMapper, String jsonString) {
- try {
- return objectMapper.readValue(jsonString, Row.class);
- } catch (JsonParseException | JsonMappingException jsonException) {
- throw new UnsupportedRowJsonException("Unable to parse Row", jsonException);
- } catch (IOException e) {
- throw new IllegalArgumentException("Unable to parse json object: " + jsonString, e);
- }
- }
-}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
index 06df322..a7e4c45 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import org.apache.beam.sdk.transforms.Combine;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NameUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NameUtils.java
index 3a72cc6..bd7afb1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NameUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NameUtils.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
/** Helpers for extracting the name of objects and classes. */
public class NameUtils {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NumberedShardedFile.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NumberedShardedFile.java
index 6029274..880ce68 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NumberedShardedFile.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/NumberedShardedFile.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.Reader;
@@ -31,11 +31,11 @@
import java.util.regex.Pattern;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharStreams;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReleaseInfo.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReleaseInfo.java
index f964614..741f373 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReleaseInfo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReleaseInfo.java
@@ -23,7 +23,7 @@
import java.io.Serializable;
import java.util.Map;
import java.util.Properties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
index 3b1805f..1929726 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
@@ -53,7 +53,7 @@
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.util.RowJsonValueExtractors.ValueExtractor;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Jackson deserializer for {@link Row Rows}.
@@ -88,7 +88,7 @@
.put(DECIMAL, decimalValueExtractor())
.build();
- private Schema schema;
+ private final Schema schema;
/** Creates a deserializer for a {@link Row} {@link Schema}. */
public static RowJsonDeserializer forSchema(Schema schema) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonSerializer.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonSerializer.java
new file mode 100644
index 0000000..0cb1672
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonSerializer.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.ser.std.StdSerializer;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.List;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.Row;
+
+public class RowJsonSerializer extends StdSerializer<Row> {
+
+ private final Schema schema;
+
+ /** Creates a serializer for a {@link Row} {@link Schema}. */
+ public static RowJsonSerializer forSchema(Schema schema) {
+ schema.getFields().forEach(RowJsonValidation::verifyFieldTypeSupported);
+ return new RowJsonSerializer(schema);
+ }
+
+ private RowJsonSerializer(Schema schema) {
+ super(Row.class);
+ this.schema = schema;
+ }
+
+ @Override
+ public void serialize(Row value, JsonGenerator gen, SerializerProvider provider)
+ throws IOException {
+ writeRow(value, this.schema, gen);
+ }
+
+ // TODO: ByteBuddy generate based on schema?
+ private void writeRow(Row row, Schema schema, JsonGenerator gen) throws IOException {
+ gen.writeStartObject();
+ for (int i = 0; i < schema.getFieldCount(); ++i) {
+ Field field = schema.getField(i);
+ Object value = row.getValue(i);
+ gen.writeFieldName(field.getName());
+ if (field.getType().getNullable() && value == null) {
+ gen.writeNull();
+ continue;
+ }
+ writeValue(gen, field.getType(), value);
+ }
+ gen.writeEndObject();
+ }
+
+ private void writeValue(JsonGenerator gen, FieldType type, Object value) throws IOException {
+ switch (type.getTypeName()) {
+ case BOOLEAN:
+ gen.writeBoolean((boolean) value);
+ break;
+ case STRING:
+ gen.writeString((String) value);
+ break;
+ case BYTE:
+ gen.writeNumber((byte) value);
+ break;
+ case DOUBLE:
+ gen.writeNumber((double) value);
+ break;
+ case FLOAT:
+ gen.writeNumber((float) value);
+ break;
+ case INT16:
+ gen.writeNumber((short) value);
+ break;
+ case INT32:
+ gen.writeNumber((int) value);
+ break;
+ case INT64:
+ gen.writeNumber((long) value);
+ break;
+ case DECIMAL:
+ gen.writeNumber((BigDecimal) value);
+ break;
+ case ARRAY:
+ gen.writeStartArray();
+ for (Object element : (List<Object>) value) {
+ writeValue(gen, type.getCollectionElementType(), element);
+ }
+ gen.writeEndArray();
+ break;
+ case ROW:
+ writeRow((Row) value, type.getRowSchema(), gen);
+ break;
+ default:
+ throw new IllegalArgumentException("Unsupported field type: " + type);
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
new file mode 100644
index 0000000..598dc74
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.fasterxml.jackson.core.JsonParseException;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonMappingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import java.io.IOException;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.values.Row;
+
+/** Utilities for working with {@link RowJsonSerializer} and {@link RowJsonDeserializer}. */
+@Internal
+public class RowJsonUtils {
+
+ public static ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
+ SimpleModule module = new SimpleModule("rowDeserializationModule");
+ module.addDeserializer(Row.class, deserializer);
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(module);
+
+ return objectMapper;
+ }
+
+ public static ObjectMapper newObjectMapperWith(RowJsonSerializer serializer) {
+ SimpleModule module = new SimpleModule("rowSerializationModule");
+ module.addSerializer(Row.class, serializer);
+
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(module);
+
+ return objectMapper;
+ }
+
+ public static Row jsonToRow(ObjectMapper objectMapper, String jsonString) {
+ try {
+ return objectMapper.readValue(jsonString, Row.class);
+ } catch (JsonParseException | JsonMappingException jsonException) {
+ throw new UnsupportedRowJsonException("Unable to parse Row", jsonException);
+ } catch (IOException e) {
+ throw new IllegalArgumentException("Unable to parse json object: " + jsonString, e);
+ }
+ }
+
+ public static String rowToJson(ObjectMapper objectMapper, Row row) {
+ try {
+ return objectMapper.writeValueAsString(row);
+ } catch (JsonProcessingException e) {
+ throw new IllegalArgumentException("Unable to serilize row: " + row);
+ }
+ }
+}
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java
index 0d2f716..2ab7aec 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonValidation.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/**
* Validates if the types specified in {@link Row} {@link Schema} are supported for conversion from
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java
index a6b5047..2e06f3c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableUtils.java
@@ -19,7 +19,7 @@
import static org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray;
import static org.apache.beam.sdk.util.CoderUtils.encodeToByteArray;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
index ac95d40..cf980b9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/StringUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.ArrayList;
import java.util.List;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedInputStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedInputStream.java
index a0b07fc..a0cda03 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedInputStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedInputStream.java
@@ -21,7 +21,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A {@link OutputStream} wrapper which protects against the user attempting to modify the
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedOutputStream.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedOutputStream.java
index f818132..1181758 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedOutputStream.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/UnownedOutputStream.java
@@ -20,7 +20,7 @@
import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* A {@link OutputStream} wrapper which protects against the user attempting to modify the
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
index 98f6620..84b1253 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/WindowedValue.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.InputStream;
@@ -40,8 +40,8 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.PaneInfoCoder;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
/**
@@ -136,6 +136,11 @@
/** Returns the pane of this {@code WindowedValue} in its window. */
public abstract PaneInfo getPane();
+ /** Returns {@code true} if this WindowedValue has exactly one window. */
+ public boolean isSingleWindowedValue() {
+ return false;
+ }
+
/**
* Returns a collection of {@link WindowedValue WindowedValues} identical to this one, except each
* is in exactly one of the windows that this {@link WindowedValue} is in.
@@ -177,11 +182,19 @@
private static final Collection<? extends BoundedWindow> GLOBAL_WINDOWS =
Collections.singletonList(GlobalWindow.INSTANCE);
+ /** A {@link WindowedValue} which holds exactly single window per value. */
+ public interface SingleWindowedValue {
+
+ /** @return the single window associated with this value. */
+ BoundedWindow getWindow();
+ }
+
/**
* An abstract superclass for implementations of {@link WindowedValue} that stores the value and
* pane info.
*/
private abstract static class SimpleWindowedValue<T> extends WindowedValue<T> {
+
private final T value;
private final PaneInfo pane;
@@ -214,7 +227,9 @@
}
/** The representation of a WindowedValue where timestamp == MIN and windows == {GlobalWindow}. */
- private static class ValueInGlobalWindow<T> extends MinTimestampWindowedValue<T> {
+ private static class ValueInGlobalWindow<T> extends MinTimestampWindowedValue<T>
+ implements SingleWindowedValue {
+
public ValueInGlobalWindow(T value, PaneInfo pane) {
super(value, pane);
}
@@ -230,6 +245,16 @@
}
@Override
+ public boolean isSingleWindowedValue() {
+ return true;
+ }
+
+ @Override
+ public BoundedWindow getWindow() {
+ return GlobalWindow.INSTANCE;
+ }
+
+ @Override
public boolean equals(Object o) {
if (o instanceof ValueInGlobalWindow) {
ValueInGlobalWindow<?> that = (ValueInGlobalWindow<?>) o;
@@ -273,7 +298,9 @@
* The representation of a WindowedValue where timestamp {@code >} MIN and windows ==
* {GlobalWindow}.
*/
- private static class TimestampedValueInGlobalWindow<T> extends TimestampedWindowedValue<T> {
+ private static class TimestampedValueInGlobalWindow<T> extends TimestampedWindowedValue<T>
+ implements SingleWindowedValue {
+
public TimestampedValueInGlobalWindow(T value, Instant timestamp, PaneInfo pane) {
super(value, timestamp, pane);
}
@@ -289,6 +316,16 @@
}
@Override
+ public boolean isSingleWindowedValue() {
+ return true;
+ }
+
+ @Override
+ public BoundedWindow getWindow() {
+ return GlobalWindow.INSTANCE;
+ }
+
+ @Override
public boolean equals(Object o) {
if (o instanceof TimestampedValueInGlobalWindow) {
TimestampedValueInGlobalWindow<?> that = (TimestampedValueInGlobalWindow<?>) o;
@@ -323,7 +360,9 @@
* The representation of a WindowedValue where timestamp is arbitrary and windows == a single
* non-Global window.
*/
- private static class TimestampedValueInSingleWindow<T> extends TimestampedWindowedValue<T> {
+ private static class TimestampedValueInSingleWindow<T> extends TimestampedWindowedValue<T>
+ implements SingleWindowedValue {
+
private final BoundedWindow window;
public TimestampedValueInSingleWindow(
@@ -343,6 +382,16 @@
}
@Override
+ public boolean isSingleWindowedValue() {
+ return true;
+ }
+
+ @Override
+ public BoundedWindow getWindow() {
+ return window;
+ }
+
+ @Override
public boolean equals(Object o) {
if (o instanceof TimestampedValueInSingleWindow) {
TimestampedValueInSingleWindow<?> that = (TimestampedValueInSingleWindow<?>) o;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java
index 09915f0..c2293ab 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.BufferedOutputStream;
import java.io.File;
@@ -32,12 +32,12 @@
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Closer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Closer;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
/**
* Functions for zipping a directory (including a subdirectory) into a ZIP-file or unzipping it
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
index f84bd21..dd20129 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/common/ReflectHelpers.java
@@ -18,9 +18,10 @@
package org.apache.beam.sdk.util.common;
import static java.util.Arrays.asList;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.lang.annotation.Annotation;
import java.lang.reflect.GenericArrayType;
import java.lang.reflect.Method;
@@ -36,11 +37,12 @@
import java.util.ServiceLoader;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Queues;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Queues;
/** Utilities for working with with {@link Class Classes} and {@link Method Methods}. */
public class ReflectHelpers {
@@ -50,6 +52,9 @@
/** A {@link Function} that turns a method into a simple method signature. */
public static final Function<Method, String> METHOD_FORMATTER =
new Function<Method, String>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
public String apply(@Nonnull Method input) {
String parameterTypes =
@@ -63,6 +68,9 @@
/** A {@link Function} that turns a method into the declaring class + method signature. */
public static final Function<Method, String> CLASS_AND_METHOD_FORMATTER =
new Function<Method, String>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
public String apply(@Nonnull Method input) {
return String.format(
@@ -90,6 +98,9 @@
/** A {@link Function} that formats types. */
public static final Function<Type, String> TYPE_SIMPLE_DESCRIPTION =
new Function<Type, String>() {
+ @SuppressFBWarnings(
+ value = "NP_METHOD_PARAMETER_TIGHTENS_ANNOTATION",
+ justification = "https://github.com/google/guava/issues/920")
@Override
@Nullable
public String apply(@Nonnull Type input) {
@@ -158,7 +169,7 @@
}
};
- /** A {@link Comparator} that uses the object's classes canonical name to compare them. */
+ /** A {@link Comparator} that uses the object's class' canonical name to compare them. */
public static class ObjectsClassComparator implements Comparator<Object> {
public static final ObjectsClassComparator INSTANCE = new ObjectsClassComparator();
@@ -201,6 +212,34 @@
}
/**
+ * Returns instances of all implementations of the the specified {@code iface}. Instances are
+ * sorted by their class' name to ensure deterministic execution.
+ *
+ * @param iface The interface to load implementations of
+ * @param classLoader The class loader to use
+ * @param <T> The type of {@code iface}
+ * @return An iterable of instances of T, ordered by their class' canonical name
+ */
+ public static <T> Iterable<T> loadServicesOrdered(Class<T> iface, ClassLoader classLoader) {
+ ServiceLoader<T> loader = ServiceLoader.load(iface, classLoader);
+ ImmutableSortedSet.Builder<T> builder =
+ new ImmutableSortedSet.Builder<>(ObjectsClassComparator.INSTANCE);
+ builder.addAll(loader);
+ return builder.build();
+ }
+
+ /**
+ * A version of {@code loadServicesOrdered} that uses a default class loader.
+ *
+ * @param iface The interface to load implementations of
+ * @param <T> The type of {@code iface}
+ * @return An iterable of instances of T, ordered by their class' canonical name
+ */
+ public static <T> Iterable<T> loadServicesOrdered(Class<T> iface) {
+ return loadServicesOrdered(iface, ReflectHelpers.findClassLoader());
+ }
+
+ /**
* Finds the appropriate {@code ClassLoader} to be used by the {@link ServiceLoader#load} call,
* which by default would use the proposed {@code ClassLoader}, which can be null. The fallback is
* as follows: context ClassLoader, class ClassLoader and finaly the system ClassLoader.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/KV.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/KV.java
index 10a979a..a45ddcb 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/KV.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/KV.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.transforms.GroupByKey;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.SerializableComparator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* An immutable key/value pair.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
index 88dffb7..fe37364 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollection.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.values;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Collections;
import java.util.Map;
@@ -296,7 +296,7 @@
/**
* Sets a schema on this PCollection.
*
- * <p>Can only be called on a {@link PCollection}.
+ * <p>Can only be called on a {@link PCollection<Row>}.
*/
@Experimental(Kind.SCHEMAS)
public PCollection<T> setRowSchema(Schema schema) {
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionList.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionList.java
index f9375af..d4ea50e 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionList.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionList.java
@@ -25,8 +25,8 @@
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.Partition;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A {@link PCollectionList PCollectionList<T>} is an immutable list of homogeneously typed
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionTuple.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionTuple.java
index 92fe0ee..6a24218 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionTuple.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionTuple.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection.IsBounded;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A {@link PCollectionTuple} is an immutable tuple of heterogeneously-typed {@link PCollection
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
index 74363c0..038cf42 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
@@ -58,7 +58,6 @@
@Nullable
@Internal
PCollection<?> getPCollection();
-
/**
* <b>For internal use only.</b>
*
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java
index e9b7959..0f73699 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.values;
import java.io.IOException;
+import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
@@ -25,6 +26,7 @@
import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Objects;
+import java.util.function.Supplier;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -39,11 +41,11 @@
import org.apache.beam.sdk.transforms.windowing.InvalidWindows;
import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
/**
* <b>For internal use only; no backwards compatibility guarantees.</b>
@@ -52,6 +54,7 @@
*/
@Internal
public class PCollectionViews {
+ public interface TypeDescriptorSupplier<T> extends Supplier<TypeDescriptor<T>>, Serializable {}
/**
* Returns a {@code PCollectionView<T>} capable of processing elements windowed using the provided
@@ -62,13 +65,14 @@
*/
public static <T, W extends BoundedWindow> PCollectionView<T> singletonView(
PCollection<KV<Void, T>> pCollection,
+ TypeDescriptorSupplier<T> typeDescriptorSupplier,
WindowingStrategy<?, W> windowingStrategy,
boolean hasDefault,
@Nullable T defaultValue,
Coder<T> defaultValueCoder) {
return new SimplePCollectionView<>(
pCollection,
- new SingletonViewFn<>(hasDefault, defaultValue, defaultValueCoder),
+ new SingletonViewFn<T>(hasDefault, defaultValue, defaultValueCoder, typeDescriptorSupplier),
windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
windowingStrategy);
}
@@ -78,10 +82,12 @@
* the provided {@link WindowingStrategy}.
*/
public static <T, W extends BoundedWindow> PCollectionView<Iterable<T>> iterableView(
- PCollection<KV<Void, T>> pCollection, WindowingStrategy<?, W> windowingStrategy) {
+ PCollection<KV<Void, T>> pCollection,
+ TypeDescriptorSupplier<T> typeDescriptorSupplier,
+ WindowingStrategy<?, W> windowingStrategy) {
return new SimplePCollectionView<>(
pCollection,
- new IterableViewFn<T>(),
+ new IterableViewFn<T>(typeDescriptorSupplier),
windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
windowingStrategy);
}
@@ -91,23 +97,27 @@
* provided {@link WindowingStrategy}.
*/
public static <T, W extends BoundedWindow> PCollectionView<List<T>> listView(
- PCollection<KV<Void, T>> pCollection, WindowingStrategy<?, W> windowingStrategy) {
+ PCollection<KV<Void, T>> pCollection,
+ TypeDescriptorSupplier<T> typeDescriptorSupplier,
+ WindowingStrategy<?, W> windowingStrategy) {
return new SimplePCollectionView<>(
pCollection,
- new ListViewFn<T>(),
+ new ListViewFn<>(typeDescriptorSupplier),
windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
windowingStrategy);
}
-
/**
* Returns a {@code PCollectionView<Map<K, V>>} capable of processing elements windowed using the
* provided {@link WindowingStrategy}.
*/
public static <K, V, W extends BoundedWindow> PCollectionView<Map<K, V>> mapView(
- PCollection<KV<Void, KV<K, V>>> pCollection, WindowingStrategy<?, W> windowingStrategy) {
+ PCollection<KV<Void, KV<K, V>>> pCollection,
+ TypeDescriptorSupplier<K> keyTypeDescriptorSupplier,
+ TypeDescriptorSupplier<V> valueTypeDescriptorSupplier,
+ WindowingStrategy<?, W> windowingStrategy) {
return new SimplePCollectionView<>(
pCollection,
- new MapViewFn<K, V>(),
+ new MapViewFn<>(keyTypeDescriptorSupplier, valueTypeDescriptorSupplier),
windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
windowingStrategy);
}
@@ -117,10 +127,13 @@
* using the provided {@link WindowingStrategy}.
*/
public static <K, V, W extends BoundedWindow> PCollectionView<Map<K, Iterable<V>>> multimapView(
- PCollection<KV<Void, KV<K, V>>> pCollection, WindowingStrategy<?, W> windowingStrategy) {
+ PCollection<KV<Void, KV<K, V>>> pCollection,
+ TypeDescriptorSupplier<K> keyTypeDescriptorSupplier,
+ TypeDescriptorSupplier<V> valueTypeDescriptorSupplier,
+ WindowingStrategy<?, W> windowingStrategy) {
return new SimplePCollectionView<>(
pCollection,
- new MultimapViewFn<K, V>(),
+ new MultimapViewFn<>(keyTypeDescriptorSupplier, valueTypeDescriptorSupplier),
windowingStrategy.getWindowFn().getDefaultWindowMappingFn(),
windowingStrategy);
}
@@ -150,11 +163,17 @@
@Nullable private transient T defaultValue;
@Nullable private Coder<T> valueCoder;
private boolean hasDefault;
+ private TypeDescriptorSupplier<T> typeDescriptorSupplier;
- private SingletonViewFn(boolean hasDefault, T defaultValue, Coder<T> valueCoder) {
+ private SingletonViewFn(
+ boolean hasDefault,
+ T defaultValue,
+ Coder<T> valueCoder,
+ TypeDescriptorSupplier<T> typeDescriptorSupplier) {
this.hasDefault = hasDefault;
this.defaultValue = defaultValue;
this.valueCoder = valueCoder;
+ this.typeDescriptorSupplier = typeDescriptorSupplier;
if (hasDefault) {
try {
this.encodedDefaultValue = CoderUtils.encodeToByteArray(valueCoder, defaultValue);
@@ -213,6 +232,11 @@
"PCollection with more than one element accessed as a singleton view.");
}
}
+
+ @Override
+ public TypeDescriptor<T> getTypeDescriptor() {
+ return typeDescriptorSupplier.get();
+ }
}
/**
@@ -224,6 +248,11 @@
*/
@Experimental(Kind.CORE_RUNNERS_ONLY)
public static class IterableViewFn<T> extends ViewFn<MultimapView<Void, T>, Iterable<T>> {
+ private TypeDescriptorSupplier<T> typeDescriptorSupplier;
+
+ public IterableViewFn(TypeDescriptorSupplier<T> typeDescriptorSupplier) {
+ this.typeDescriptorSupplier = typeDescriptorSupplier;
+ }
@Override
public Materialization<MultimapView<Void, T>> getMaterialization() {
@@ -234,6 +263,11 @@
public Iterable<T> apply(MultimapView<Void, T> primitiveViewT) {
return Iterables.unmodifiableIterable(primitiveViewT.get(null));
}
+
+ @Override
+ public TypeDescriptor<Iterable<T>> getTypeDescriptor() {
+ return TypeDescriptors.iterables(typeDescriptorSupplier.get());
+ }
}
/**
@@ -245,6 +279,12 @@
*/
@Experimental(Kind.CORE_RUNNERS_ONLY)
public static class ListViewFn<T> extends ViewFn<MultimapView<Void, T>, List<T>> {
+ private TypeDescriptorSupplier<T> typeDescriptorSupplier;
+
+ public ListViewFn(TypeDescriptorSupplier<T> typeDescriptorSupplier) {
+ this.typeDescriptorSupplier = typeDescriptorSupplier;
+ }
+
@Override
public Materialization<MultimapView<Void, T>> getMaterialization() {
return Materializations.multimap();
@@ -260,6 +300,11 @@
}
@Override
+ public TypeDescriptor<List<T>> getTypeDescriptor() {
+ return TypeDescriptors.lists(typeDescriptorSupplier.get());
+ }
+
+ @Override
public boolean equals(Object other) {
return other instanceof ListViewFn;
}
@@ -281,6 +326,16 @@
@Experimental(Kind.CORE_RUNNERS_ONLY)
public static class MultimapViewFn<K, V>
extends ViewFn<MultimapView<Void, KV<K, V>>, Map<K, Iterable<V>>> {
+ private TypeDescriptorSupplier<K> keyTypeDescriptorSupplier;
+ private TypeDescriptorSupplier<V> valueTypeDescriptorSupplier;
+
+ public MultimapViewFn(
+ TypeDescriptorSupplier<K> keyTypeDescriptorSupplier,
+ TypeDescriptorSupplier<V> valueTypeDescriptorSupplier) {
+ this.keyTypeDescriptorSupplier = keyTypeDescriptorSupplier;
+ this.valueTypeDescriptorSupplier = valueTypeDescriptorSupplier;
+ }
+
@Override
public Materialization<MultimapView<Void, KV<K, V>>> getMaterialization() {
return Materializations.multimap();
@@ -299,6 +354,13 @@
Map<K, Iterable<V>> resultMap = (Map) multimap.asMap();
return Collections.unmodifiableMap(resultMap);
}
+
+ @Override
+ public TypeDescriptor<Map<K, Iterable<V>>> getTypeDescriptor() {
+ return TypeDescriptors.maps(
+ keyTypeDescriptorSupplier.get(),
+ TypeDescriptors.iterables(valueTypeDescriptorSupplier.get()));
+ }
}
/**
@@ -310,6 +372,15 @@
*/
@Experimental(Kind.CORE_RUNNERS_ONLY)
public static class MapViewFn<K, V> extends ViewFn<MultimapView<Void, KV<K, V>>, Map<K, V>> {
+ private TypeDescriptorSupplier<K> keyTypeDescriptorSupplier;
+ private TypeDescriptorSupplier<V> valueTypeDescriptorSupplier;
+
+ public MapViewFn(
+ TypeDescriptorSupplier<K> keyTypeDescriptorSupplier,
+ TypeDescriptorSupplier<V> valueTypeDescriptorSupplier) {
+ this.keyTypeDescriptorSupplier = keyTypeDescriptorSupplier;
+ this.valueTypeDescriptorSupplier = valueTypeDescriptorSupplier;
+ }
@Override
public Materialization<MultimapView<Void, KV<K, V>>> getMaterialization() {
@@ -329,6 +400,12 @@
}
return Collections.unmodifiableMap(map);
}
+
+ @Override
+ public TypeDescriptor<Map<K, V>> getTypeDescriptor() {
+ return TypeDescriptors.maps(
+ keyTypeDescriptorSupplier.get(), valueTypeDescriptorSupplier.get());
+ }
}
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
index 63d5598..c71f695 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PValueBase.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.values;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
import org.apache.beam.sdk.Pipeline;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
index 66c4c65..48c708c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.values;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Serializable;
import java.math.BigDecimal;
@@ -39,9 +39,9 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.LogicalType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.joda.time.DateTime;
import org.joda.time.Instant;
import org.joda.time.ReadableDateTime;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
index 1014cbc..e50f818 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowWithGetters.java
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* A Concrete subclass of {@link Row} that delegates to a set of provided {@link FieldValueGetter}s.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TaggedPValue.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TaggedPValue.java
index 46611aa..6b1193c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TaggedPValue.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TaggedPValue.java
@@ -19,7 +19,7 @@
import com.google.auto.value.AutoValue;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* <b><i>For internal use only; no backwards-compatibility guarantees.</i></b>
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TimestampedValue.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TimestampedValue.java
index 54d9999..059f747 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TimestampedValue.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TimestampedValue.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.values;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.io.InputStream;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTag.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTag.java
index 6c0184a..707a2f8 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTag.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTag.java
@@ -20,8 +20,8 @@
import java.io.Serializable;
import java.util.Random;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multiset;
/**
* A {@link TupleTag} is a typed tag to use as the key of a heterogeneously typed tuple, like {@link
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTagList.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTagList.java
index 85d2ce5..9578cc5 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTagList.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TupleTagList.java
@@ -22,8 +22,8 @@
import java.util.Collections;
import java.util.List;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A {@link TupleTagList} is an immutable list of heterogeneously typed {@link TupleTag TupleTags}.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
index ceb673a..db8fb06 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeDescriptor.java
@@ -25,11 +25,11 @@
import java.lang.reflect.TypeVariable;
import java.util.List;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.Invokable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.Parameter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.TypeResolver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.TypeToken;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.Invokable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.Parameter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.TypeResolver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.TypeToken;
/**
* A description of a Java type, including actual generic parameters where possible.
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
index 21e8abe..f685952 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/TypeParameter.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.values;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueInSingleWindow.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueInSingleWindow.java
index 330f044..ea27271 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueInSingleWindow.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueInSingleWindow.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueWithRecordId.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueWithRecordId.java
index 3035997..461b5dd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueWithRecordId.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueWithRecordId.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.StructuredCoder;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/**
* <b>For internal use only; no backwards compatibility guarantees.</b>
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
index ab6b427..6b2c4d6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/WindowingStrategy.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.transforms.windowing.Window.ClosingBehavior;
import org.apache.beam.sdk.transforms.windowing.Window.OnTimeBehavior;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Duration;
/**
@@ -44,10 +44,18 @@
*/
public class WindowingStrategy<T, W extends BoundedWindow> implements Serializable {
- /** The accumulation modes that can be used with windowing. */
+ /**
+ * The accumulation modes that can be used with windowing.
+ *
+ * <p>Experimental {@link AccumulationMode.RETRACTING_FIRED_PANES} for enabling retractions in
+ * pipelines. There is no backwards-compatibility guarantees.
+ */
public enum AccumulationMode {
DISCARDING_FIRED_PANES,
- ACCUMULATING_FIRED_PANES
+ ACCUMULATING_FIRED_PANES,
+ // RETRACTING_FIRED_PANES is experimental. There is no backwards-compatibility guarantees.
+ @Experimental
+ RETRACTING_FIRED_PANES,
}
private static final Duration DEFAULT_ALLOWED_LATENESS = Duration.ZERO;
diff --git a/sdks/java/core/src/test/avro/org/apache/beam/sdk/schemas/test.avsc b/sdks/java/core/src/test/avro/org/apache/beam/sdk/schemas/test.avsc
index 9ed18bf..061bb27 100644
--- a/sdks/java/core/src/test/avro/org/apache/beam/sdk/schemas/test.avsc
+++ b/sdks/java/core/src/test/avro/org/apache/beam/sdk/schemas/test.avsc
@@ -11,8 +11,9 @@
{ "name": "string", "type": ["string", "null"]},
{ "name": "bytes", "type": ["bytes", "null"]},
{ "name": "fixed", "type": {"type": "fixed", "size": 4, "name": "fixed4"} },
- { "name": "timestampMillis", "type":
- [ {"type": "long", "logicalType": "timestamp-millis"}, "null"]},
+ { "name": "date", "type": {"type": "int", "logicalType": "date"} },
+ { "name": "timestampMillis", "type": {"type": "long", "logicalType": "timestamp-millis"} },
+ { "name": "testEnum", "type": {"name": "TestEnum", "type": "enum", "symbols": ["abc","cde"] } },
{ "name": "row", "type": ["null", {
"type": "record",
"name": "TestAvroNested",
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
index 432f36a..de9b885 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
@@ -67,8 +67,8 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TaggedPValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.hamcrest.Matchers;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
index 05d3886..2741d3a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java
@@ -73,6 +73,8 @@
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.hamcrest.TypeSafeMatcher;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
@@ -85,18 +87,27 @@
@RunWith(JUnit4.class)
public class AvroCoderTest {
+ public static final DateTime DATETIME_A =
+ new DateTime().withDate(1994, 10, 31).withZone(DateTimeZone.UTC);
+ public static final DateTime DATETIME_B =
+ new DateTime().withDate(1997, 4, 25).withZone(DateTimeZone.UTC);
+
@DefaultCoder(AvroCoder.class)
private static class Pojo {
public String text;
public int count;
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestamp;
+
// Empty constructor required for Avro decoding.
@SuppressWarnings("unused")
public Pojo() {}
- public Pojo(String text, int count) {
+ public Pojo(String text, int count, DateTime timestamp) {
this.text = text;
this.count = count;
+ this.timestamp = timestamp;
}
// auto-generated
@@ -117,6 +128,9 @@
if (text != null ? !text.equals(pojo.text) : pojo.text != null) {
return false;
}
+ if (timestamp != null ? !timestamp.equals(pojo.timestamp) : pojo.timestamp != null) {
+ return false;
+ }
return true;
}
@@ -128,7 +142,15 @@
@Override
public String toString() {
- return "Pojo{" + "text='" + text + '\'' + ", count=" + count + '}';
+ return "Pojo{"
+ + "text='"
+ + text
+ + '\''
+ + ", count="
+ + count
+ + ", timestamp="
+ + timestamp
+ + '}';
}
}
@@ -147,9 +169,9 @@
CoderProperties.coderSerializable(coder);
AvroCoder<Pojo> copy = SerializableUtils.clone(coder);
- Pojo pojo = new Pojo("foo", 3);
- Pojo equalPojo = new Pojo("foo", 3);
- Pojo otherPojo = new Pojo("bar", -19);
+ Pojo pojo = new Pojo("foo", 3, DATETIME_A);
+ Pojo equalPojo = new Pojo("foo", 3, DATETIME_A);
+ Pojo otherPojo = new Pojo("bar", -19, DATETIME_B);
CoderProperties.coderConsistentWithEquals(coder, pojo, equalPojo);
CoderProperties.coderConsistentWithEquals(copy, pojo, equalPojo);
CoderProperties.coderConsistentWithEquals(coder, pojo, otherPojo);
@@ -205,7 +227,7 @@
*/
@Test
public void testTransientFieldInitialization() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
// Serialization of object
@@ -228,7 +250,7 @@
*/
@Test
public void testKryoSerialization() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
// Kryo instantiation
@@ -266,7 +288,7 @@
@Test
public void testPojoEncoding() throws Exception {
- Pojo value = new Pojo("Hello", 42);
+ Pojo value = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
CoderProperties.coderDecodeEncodeEqual(coder, value);
@@ -302,7 +324,7 @@
// This test ensures that the coder doesn't read ahead and buffer data.
// Reading ahead causes a problem if the stream consists of records of different
// types.
- Pojo before = new Pojo("Hello", 42);
+ Pojo before = new Pojo("Hello", 42, DATETIME_A);
AvroCoder<Pojo> coder = AvroCoder.of(Pojo.class);
SerializableCoder<Integer> intCoder = SerializableCoder.of(Integer.class);
@@ -329,7 +351,7 @@
// a coder (this uses the default coders, which may not be AvroCoder).
PCollection<String> output =
pipeline
- .apply(Create.of(new Pojo("hello", 1), new Pojo("world", 2)))
+ .apply(Create.of(new Pojo("hello", 1, DATETIME_A), new Pojo("world", 2, DATETIME_B)))
.apply(ParDo.of(new GetTextFn()));
PAssert.that(output).containsInAnyOrder("hello", "world");
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTestPojo.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTestPojo.java
index 24a596e..bb04ec6 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTestPojo.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTestPojo.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.coders;
import java.util.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** A Pojo at the top level for use in tests. */
class AvroCoderTestPojo {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigDecimalCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigDecimalCoderTest.java
index 4acc0bc..9e53399 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigDecimalCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigDecimalCoderTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.testing.CoderProperties.TestElementByteSizeObserver;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java
index 37ce59d..d94451d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.testing.CoderProperties.TestElementByteSizeObserver;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
index 3f9d917..f06d791 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -77,6 +77,10 @@
public void testSimpleDefaultCoder() throws Exception {
CoderRegistry registry = CoderRegistry.createDefault();
assertEquals(StringUtf8Coder.of(), registry.getCoder(String.class));
+ assertEquals(VarIntCoder.of(), registry.getCoder(Integer.class));
+ assertEquals(VarLongCoder.of(), registry.getCoder(Long.class));
+ assertEquals(FloatCoder.of(), registry.getCoder(Float.class));
+ assertEquals(DoubleCoder.of(), registry.getCoder(Double.class));
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DefaultCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DefaultCoderTest.java
index b45e956..f34dc35 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DefaultCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DefaultCoderTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.instanceOf;
import static org.junit.Assert.assertThat;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DelegateCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DelegateCoderTest.java
index fe143a1..e152e8e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DelegateCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DelegateCoderTest.java
@@ -30,8 +30,8 @@
import java.util.Set;
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DurationCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DurationCoderTest.java
index 9959452..e49658d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DurationCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/DurationCoderTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.ReadableDuration;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/FloatCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/FloatCoderTest.java
new file mode 100644
index 0000000..17f2629
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/FloatCoderTest.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.coders;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test case for {@link FloatCoder}. */
+@RunWith(JUnit4.class)
+public class FloatCoderTest {
+
+ private static final Coder<Float> TEST_CODER = FloatCoder.of();
+
+ private static final List<Float> TEST_VALUES =
+ Arrays.asList(
+ 0.0f,
+ -0.5f,
+ 0.5f,
+ 0.3f,
+ -0.3f,
+ 1.0f,
+ -43.895687f,
+ (float) Math.PI,
+ Float.MAX_VALUE,
+ Float.MIN_VALUE,
+ Float.POSITIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ Float.NaN);
+
+ @Test
+ public void testDecodeEncodeEqual() throws Exception {
+ for (Float value : TEST_VALUES) {
+ CoderProperties.coderDecodeEncodeEqual(TEST_CODER, value);
+ }
+ }
+
+ /**
+ * Generated data to check that the wire format has not changed. To regenerate, see {@link
+ * org.apache.beam.sdk.coders.PrintBase64Encodings}.
+ */
+ private static final List<String> TEST_ENCODINGS =
+ Arrays.asList(
+ "AAAAAA", "vwAAAA", "PwAAAA", "PpmZmg", "vpmZmg", "P4AAAA", "wi-VLw", "QEkP2w", "f3___w",
+ "AAAAAQ", "f4AAAA", "_4AAAA", "f8AAAA");
+
+ @Test
+ public void testWireFormatEncode() throws Exception {
+ System.out.println(TEST_ENCODINGS);
+ CoderProperties.coderEncodesBase64(TEST_CODER, TEST_VALUES, TEST_ENCODINGS);
+ }
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void encodeNullThrowsCoderException() throws Exception {
+ thrown.expect(CoderException.class);
+ thrown.expectMessage("cannot encode a null Float");
+
+ CoderUtils.encodeToBase64(TEST_CODER, null);
+ }
+
+ @Test
+ public void testEncodedTypeDescriptor() throws Exception {
+ assertThat(TEST_CODER.getEncodedTypeDescriptor(), equalTo(TypeDescriptor.of(Float.class)));
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/InstantCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/InstantCoderTest.java
index ec00fb5..082ea80 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/InstantCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/InstantCoderTest.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
import org.joda.time.Instant;
import org.junit.Assert;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/LengthPrefixCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/LengthPrefixCoderTest.java
index 3b5172d..8f21d94 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/LengthPrefixCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/LengthPrefixCoderTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/MapCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/MapCoderTest.java
index 74dfa7b..788bfd9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/MapCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/MapCoderTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
index 66c5bb50..461bac1 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/NullableCoderTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.coders;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.theInstance;
import static org.junit.Assert.assertEquals;
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/PrintBase64Encodings.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/PrintBase64Encodings.java
index 89f1bc0..5b97ecf 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/PrintBase64Encodings.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/PrintBase64Encodings.java
@@ -21,8 +21,8 @@
import java.lang.reflect.Modifier;
import java.util.List;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* A command-line utility for printing the base-64 encodings of test values, for generating exact
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/RowCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/RowCoderTest.java
index f0d814c..79aa1c7 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/RowCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/RowCoderTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.junit.Assume;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuralByteArrayTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuralByteArrayTest.java
index 90cb6e1..f95c02d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuralByteArrayTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuralByteArrayTest.java
@@ -20,7 +20,7 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuredCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuredCoderTest.java
index 396e272..71e2e07 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuredCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/StructuredCoderTest.java
@@ -26,7 +26,7 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.CoreMatchers;
import org.junit.Assert;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
index 012d0ae..a64f868 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
@@ -23,7 +23,7 @@
import static org.apache.beam.sdk.transforms.Contextful.fn;
import static org.apache.beam.sdk.transforms.Requirements.requiresSideInputs;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.hasItem;
import static org.junit.Assert.assertArrayEquals;
@@ -34,12 +34,15 @@
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
import java.io.Serializable;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Objects;
@@ -56,6 +59,8 @@
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.reflect.ReflectDatumReader;
import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy;
@@ -89,16 +94,16 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
@@ -261,7 +266,11 @@
private enum WriteMethod {
AVROIO_WRITE,
- AVROIO_SINK
+ AVROIO_SINK_WITH_CLASS,
+ AVROIO_SINK_WITH_SCHEMA,
+ /** @deprecated Test code for the deprecated {AvroIO.RecordFormatter}. */
+ @Deprecated
+ AVROIO_SINK_WITH_FORMATTER
}
private static final String SCHEMA_STRING =
@@ -870,7 +879,7 @@
@Test
@Category({NeedsRunner.class, UsesTestStream.class})
public void testWindowedAvroIOWriteViaSink() throws Throwable {
- testWindowedAvroIOWriteUsingMethod(WriteMethod.AVROIO_SINK);
+ testWindowedAvroIOWriteUsingMethod(WriteMethod.AVROIO_SINK_WITH_CLASS);
}
void testWindowedAvroIOWriteUsingMethod(WriteMethod method) throws IOException {
@@ -943,7 +952,7 @@
break;
}
- case AVROIO_SINK:
+ case AVROIO_SINK_WITH_CLASS:
{
write =
FileIO.<GenericClass>write()
@@ -976,9 +985,11 @@
case AVROIO_WRITE:
expectedFiles.add(new File(baseAndWindow + "-" + shard + "-of-2-pane-0-last.avro"));
break;
- case AVROIO_SINK:
+ case AVROIO_SINK_WITH_CLASS:
expectedFiles.add(new File(baseAndWindow + "-0000" + shard + "-of-00002.avro"));
break;
+ default:
+ throw new UnsupportedOperationException("Unknown write method " + method);
}
}
}
@@ -1000,7 +1011,7 @@
private static final String SCHEMA_TEMPLATE_STRING =
"{\"namespace\": \"example.avro\",\n"
+ " \"type\": \"record\",\n"
- + " \"name\": \"TestTemplateSchema$$\",\n"
+ + " \"name\": \"$$TestTemplateSchema\",\n"
+ " \"fields\": [\n"
+ " {\"name\": \"$$full\", \"type\": \"string\"},\n"
+ " {\"name\": \"$$suffix\", \"type\": [\"string\", \"null\"]}\n"
@@ -1067,6 +1078,50 @@
}
}
+ /**
+ * Example of a {@link Coder} for a collection of Avro records with different schemas.
+ *
+ * <p>All the schemas are known at pipeline construction, and are keyed internally on the prefix
+ * character (lower byte only for UTF-8 data).
+ */
+ private static class AvroMultiplexCoder extends Coder<GenericRecord> {
+
+ /** Lookup table for the possible schemas, keyed on the prefix character. */
+ private final Map<Character, AvroCoder<GenericRecord>> coderMap = Maps.newHashMap();
+
+ protected AvroMultiplexCoder(Map<String, String> schemaMap) {
+ for (Map.Entry<String, String> entry : schemaMap.entrySet()) {
+ coderMap.put(
+ entry.getKey().charAt(0), AvroCoder.of(new Schema.Parser().parse(entry.getValue())));
+ }
+ }
+
+ @Override
+ public void encode(GenericRecord value, OutputStream outStream) throws IOException {
+ char prefix = value.getSchema().getName().charAt(0);
+ outStream.write(prefix); // Only reads and writes the low byte.
+ coderMap.get(prefix).encode(value, outStream);
+ }
+
+ @Override
+ public GenericRecord decode(InputStream inStream) throws CoderException, IOException {
+ char prefix = (char) inStream.read();
+ return coderMap.get(prefix).decode(inStream);
+ }
+
+ @Override
+ public List<? extends Coder<?>> getCoderArguments() {
+ return Collections.emptyList();
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {
+ for (AvroCoder<GenericRecord> internalCoder : coderMap.values()) {
+ internalCoder.verifyDeterministic();
+ }
+ }
+ }
+
private void testDynamicDestinationsUnwindowedWithSharding(
WriteMethod writeMethod, Sharding sharding) throws Exception {
final ResourceId baseDir =
@@ -1116,7 +1171,69 @@
break;
}
- case AVROIO_SINK:
+ case AVROIO_SINK_WITH_SCHEMA:
+ {
+ FileIO.Write<String, GenericRecord> write =
+ FileIO.<String, GenericRecord>writeDynamic()
+ .by(
+ fn(
+ (element, c) -> {
+ c.sideInput(schemaView); // Ignore result
+ return element.getSchema().getName().substring(0, 1);
+ },
+ requiresSideInputs(schemaView)))
+ .via(
+ fn(
+ (dest, c) -> {
+ Schema schema =
+ new Schema.Parser().parse(c.sideInput(schemaView).get(dest));
+ return AvroIO.sink(schema);
+ },
+ requiresSideInputs(schemaView)))
+ .to(baseDir.toString())
+ .withNaming(
+ fn(
+ (dest, c) -> {
+ c.sideInput(schemaView); // Ignore result
+ return FileIO.Write.defaultNaming("file_" + dest, ".avro");
+ },
+ requiresSideInputs(schemaView)))
+ .withTempDirectory(baseDir.toString())
+ .withDestinationCoder(StringUtf8Coder.of())
+ .withIgnoreWindowing();
+ switch (sharding) {
+ case RUNNER_DETERMINED:
+ break;
+ case WITHOUT_SHARDING:
+ write = write.withNumShards(1);
+ break;
+ case FIXED_3_SHARDS:
+ write = write.withNumShards(3);
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown sharding " + sharding);
+ }
+
+ MapElements<String, GenericRecord> toRecord =
+ MapElements.via(
+ new SimpleFunction<String, GenericRecord>() {
+ @Override
+ public GenericRecord apply(String element) {
+ String prefix = element.substring(0, 1);
+ GenericRecord record =
+ new GenericData.Record(
+ new Schema.Parser().parse(schemaFromPrefix(prefix)));
+ record.put(prefix + "full", element);
+ record.put(prefix + "suffix", element.substring(1));
+ return record;
+ }
+ });
+
+ input.apply(toRecord).setCoder(new AvroMultiplexCoder(schemaMap)).apply(write);
+ break;
+ }
+
+ case AVROIO_SINK_WITH_FORMATTER:
{
final AvroIO.RecordFormatter<String> formatter =
(element, schema) -> {
@@ -1170,6 +1287,8 @@
input.apply(write);
break;
}
+ default:
+ throw new UnsupportedOperationException("Unknown write method " + writeMethod);
}
writePipeline.run();
@@ -1230,21 +1349,43 @@
@Category(NeedsRunner.class)
public void testDynamicDestinationsViaSinkRunnerDeterminedSharding() throws Exception {
testDynamicDestinationsUnwindowedWithSharding(
- WriteMethod.AVROIO_SINK, Sharding.RUNNER_DETERMINED);
+ WriteMethod.AVROIO_SINK_WITH_SCHEMA, Sharding.RUNNER_DETERMINED);
}
@Test
@Category(NeedsRunner.class)
public void testDynamicDestinationsViaSinkWithoutSharding() throws Exception {
testDynamicDestinationsUnwindowedWithSharding(
- WriteMethod.AVROIO_SINK, Sharding.WITHOUT_SHARDING);
+ WriteMethod.AVROIO_SINK_WITH_SCHEMA, Sharding.WITHOUT_SHARDING);
}
@Test
@Category(NeedsRunner.class)
public void testDynamicDestinationsViaSinkWithNumShards() throws Exception {
testDynamicDestinationsUnwindowedWithSharding(
- WriteMethod.AVROIO_SINK, Sharding.FIXED_3_SHARDS);
+ WriteMethod.AVROIO_SINK_WITH_SCHEMA, Sharding.FIXED_3_SHARDS);
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testDynamicDestinationsViaSinkWithFormatterRunnerDeterminedSharding()
+ throws Exception {
+ testDynamicDestinationsUnwindowedWithSharding(
+ WriteMethod.AVROIO_SINK_WITH_FORMATTER, Sharding.RUNNER_DETERMINED);
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testDynamicDestinationsViaSinkWithFormatterWithoutSharding() throws Exception {
+ testDynamicDestinationsUnwindowedWithSharding(
+ WriteMethod.AVROIO_SINK_WITH_FORMATTER, Sharding.WITHOUT_SHARDING);
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testDynamicDestinationsViaSinkWithFormatterWithNumShards() throws Exception {
+ testDynamicDestinationsUnwindowedWithSharding(
+ WriteMethod.AVROIO_SINK_WITH_FORMATTER, Sharding.FIXED_3_SHARDS);
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java
index 4e0bce3..398dc65 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroSourceTest.java
@@ -60,7 +60,7 @@
import org.apache.beam.sdk.testing.SourceTestUtils;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CompressedSourceTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CompressedSourceTest.java
index f5756e6..7afde41 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CompressedSourceTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/CompressedSourceTest.java
@@ -59,11 +59,11 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultiset;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Bytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultiset;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream;
import org.apache.commons.compress.compressors.deflate.DeflateCompressorOutputStream;
import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
index 912723a..5da3290 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.io;
import static org.apache.beam.sdk.io.WriteFiles.UNKNOWN_SHARDNUM;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets.UTF_8;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets.UTF_8;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertEquals;
@@ -47,6 +47,7 @@
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
+import java.util.Set;
import java.util.zip.GZIPInputStream;
import org.apache.beam.sdk.io.FileBasedSink.CompressionType;
import org.apache.beam.sdk.io.FileBasedSink.FileResult;
@@ -58,7 +59,8 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream;
import org.apache.commons.compress.compressors.deflate.DeflateCompressorInputStream;
import org.junit.Rule;
@@ -141,6 +143,20 @@
}
}
+ /** Test whether WriteOperation can create a unique temporary directory. */
+ @Test
+ public void testTemporaryDirectoryUniqueness() {
+ List<SimpleSink.SimpleWriteOperation<Void>> writeOps = Lists.newArrayListWithCapacity(1000);
+ for (int i = 0; i < 1000; i++) {
+ writeOps.add(buildWriteOperation());
+ }
+ Set<String> tempDirectorySet = Sets.newHashSetWithExpectedSize(1000);
+ for (SimpleSink.SimpleWriteOperation<Void> op : writeOps) {
+ tempDirectorySet.add(op.getTempDirectory().toString());
+ }
+ assertEquals(1000, tempDirectorySet.size());
+ }
+
/** Removes temporary files when temporary and output directories differ. */
@Test
public void testRemoveWithTempFilename() throws Exception {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java
index a34c37b..f219ee8 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java
@@ -56,7 +56,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java
index 6681554..d22e64d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileSystemsTest.java
@@ -35,10 +35,10 @@
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.apache.commons.lang3.SystemUtils;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemRegistrarTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemRegistrarTest.java
index d491b3d..af7e1f4 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemRegistrarTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemRegistrarTest.java
@@ -24,7 +24,7 @@
import java.util.ServiceLoader;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java
index 81d8aac..4100bff 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/LocalFileSystemTest.java
@@ -34,17 +34,19 @@
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.nio.file.Paths;
+import java.util.Collections;
import java.util.List;
import org.apache.beam.sdk.io.fs.CreateOptions.StandardCreateOptions;
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.LineReader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.LineReader;
import org.apache.commons.lang3.SystemUtils;
import org.hamcrest.Matchers;
import org.junit.Rule;
@@ -183,6 +185,31 @@
}
@Test
+ public void testMatchWithGlob() throws Exception {
+ String globPattern = "/A/a=[0-9][0-9][0-9]/*/*";
+ File baseFolder = temporaryFolder.newFolder("A");
+ File folder1 = new File(baseFolder, "a=100");
+ File folder2 = new File(baseFolder, "a=233");
+ File dataFolder1 = new File(folder1, "data1");
+ File dataFolder2 = new File(folder2, "data_dir");
+ File expectedFile1 = new File(dataFolder1, "file1");
+ File expectedFile2 = new File(dataFolder2, "data_file2");
+
+ createEmptyFile(expectedFile1);
+ createEmptyFile(expectedFile2);
+
+ List<String> expected =
+ ImmutableList.of(expectedFile1.getAbsolutePath(), expectedFile2.getAbsolutePath());
+
+ List<MatchResult> matchResults =
+ matchGlobWithPathPrefix(temporaryFolder.getRoot().toPath(), globPattern);
+
+ assertThat(
+ toFilenames(matchResults),
+ containsInAnyOrder(expected.toArray(new String[expected.size()])));
+ }
+
+ @Test
public void testMatchExact() throws Exception {
List<String> expected = ImmutableList.of(temporaryFolder.newFile("a").toString());
temporaryFolder.newFile("aa");
@@ -197,6 +224,25 @@
}
@Test
+ public void testMatchDirectory() throws Exception {
+ final Path dir = temporaryFolder.newFolder("dir").toPath();
+ final MatchResult matchResult =
+ Iterables.getOnlyElement(localFileSystem.match(Collections.singletonList(dir.toString())));
+ assertThat(
+ matchResult,
+ equalTo(
+ MatchResult.create(
+ MatchResult.Status.OK,
+ ImmutableList.of(
+ MatchResult.Metadata.builder()
+ .setResourceId(LocalResourceId.fromPath(dir, true))
+ .setIsReadSeekEfficient(true)
+ .setSizeBytes(dir.toFile().length())
+ .setLastModifiedMillis(dir.toFile().lastModified())
+ .build()))));
+ }
+
+ @Test
public void testMatchPatternNone() throws Exception {
temporaryFolder.newFile("a");
temporaryFolder.newFile("aa");
@@ -401,4 +447,10 @@
.transform(metadata -> ((LocalResourceId) metadata.resourceId()).getPath().toString())
.toList();
}
+
+ private static void createEmptyFile(File file) throws IOException {
+ if (!file.getParentFile().mkdirs() || !file.createNewFile()) {
+ throw new IOException("Failed creating empty file " + file.getAbsolutePath());
+ }
+ }
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/SimpleSink.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/SimpleSink.java
index 8ad5ace..1501a81 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/SimpleSink.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/SimpleSink.java
@@ -100,6 +100,10 @@
public SimpleWriter<DestinationT> createWriter() {
return new SimpleWriter<>(this);
}
+
+ public ResourceId getTempDirectory() {
+ return tempDirectory.get();
+ }
}
static final class SimpleWriter<DestinationT> extends Writer<DestinationT, String> {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
index efd9463..37ddc59 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TFRecordIOTest.java
@@ -24,10 +24,10 @@
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.hamcrest.CoreMatchers.containsString;
import static org.hamcrest.CoreMatchers.instanceOf;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.in;
import static org.hamcrest.core.Is.is;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
import static org.junit.internal.matchers.ThrowableMessageMatcher.hasMessage;
import java.io.File;
@@ -40,6 +40,8 @@
import java.util.Collections;
import java.util.List;
import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.FileIO.ReadableFile;
+import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.testing.NeedsRunner;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -48,10 +50,10 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -101,19 +103,39 @@
@Test
public void testReadNamed() {
- writePipeline.enableAbandonedNodeEnforcement(false);
+ readPipeline.enableAbandonedNodeEnforcement(false);
assertEquals(
"TFRecordIO.Read/Read.out",
- writePipeline.apply(TFRecordIO.read().from("foo.*").withoutValidation()).getName());
+ readPipeline.apply(TFRecordIO.read().from("foo.*").withoutValidation()).getName());
assertEquals(
"MyRead/Read.out",
- writePipeline
+ readPipeline
.apply("MyRead", TFRecordIO.read().from("foo.*").withoutValidation())
.getName());
}
@Test
+ public void testReadFilesNamed() {
+ readPipeline.enableAbandonedNodeEnforcement(false);
+
+ Metadata metadata =
+ Metadata.builder()
+ .setResourceId(FileSystems.matchNewResource("file", false /* isDirectory */))
+ .setIsReadSeekEfficient(true)
+ .setSizeBytes(1024)
+ .build();
+ Create.Values<ReadableFile> create = Create.of(new ReadableFile(metadata, Compression.AUTO));
+
+ assertEquals(
+ "TFRecordIO.ReadFiles/Read all via FileBasedSource/Read ranges/ParMultiDo(ReadFileRanges).output",
+ readPipeline.apply(create).apply(TFRecordIO.readFiles()).getName());
+ assertEquals(
+ "MyRead/Read all via FileBasedSource/Read ranges/ParMultiDo(ReadFileRanges).output",
+ readPipeline.apply(create).apply("MyRead", TFRecordIO.readFiles()).getName());
+ }
+
+ @Test
public void testReadDisplayData() {
TFRecordIO.Read read =
TFRecordIO.read().from("foo.*").withCompression(GZIP).withoutValidation();
@@ -200,6 +222,7 @@
runTestRead(BaseEncoding.base64().decode(base64), expected);
}
+ /** Tests both {@link TFRecordIO.Read} and {@link TFRecordIO.ReadFiles}. */
private void runTestRead(byte[] data, String[] expected) throws IOException {
File tmpFile =
Files.createTempFile(tempFolder.getRoot().toPath(), "file", ".tfrecords").toFile();
@@ -210,10 +233,20 @@
}
TFRecordIO.Read read = TFRecordIO.read().from(filename);
- PCollection<String> output = writePipeline.apply(read).apply(ParDo.of(new ByteArrayToString()));
-
+ PCollection<String> output = readPipeline.apply(read).apply(ParDo.of(new ByteArrayToString()));
PAssert.that(output).containsInAnyOrder(expected);
- writePipeline.run();
+
+ Compression compression = AUTO;
+ PAssert.that(
+ readPipeline
+ .apply("Create_Paths_ReadFiles_" + tmpFile, Create.of(tmpFile.getPath()))
+ .apply("Match_" + tmpFile, FileIO.matchAll())
+ .apply("ReadMatches_" + tmpFile, FileIO.readMatches().withCompression(compression))
+ .apply("ReadFiles_" + compression.toString(), TFRecordIO.readFiles())
+ .apply("ToString", ParDo.of(new ByteArrayToString())))
+ .containsInAnyOrder(expected);
+
+ readPipeline.run();
}
private void runTestWrite(String[] elems, String... base64) throws IOException {
@@ -346,7 +379,7 @@
TFRecordIO.read()
.from(baseFilenameViaWrite + "*")
.withCompression(readCompression))
- .apply("To string first", ParDo.of(new ByteArrayToString())))
+ .apply("To string read from write", ParDo.of(new ByteArrayToString())))
.containsInAnyOrder(elems);
PAssert.that(
readPipeline
@@ -355,7 +388,28 @@
TFRecordIO.read()
.from(baseFilenameViaSink + "*")
.withCompression(readCompression))
- .apply("To string second", ParDo.of(new ByteArrayToString())))
+ .apply("To string read from sink", ParDo.of(new ByteArrayToString())))
+ .containsInAnyOrder(elems);
+ PAssert.that(
+ readPipeline
+ .apply(
+ "Create_Paths_ReadFiles_" + baseFilenameViaWrite,
+ Create.of(baseFilenameViaWrite + "*"))
+ .apply("Match_" + baseFilenameViaWrite, FileIO.matchAll())
+ .apply(
+ "ReadMatches_" + baseFilenameViaWrite,
+ FileIO.readMatches().withCompression(readCompression))
+ .apply("ReadFiles written by TFRecordIO.write", TFRecordIO.readFiles())
+ .apply("To string readFiles from write", ParDo.of(new ByteArrayToString())))
+ .containsInAnyOrder(elems);
+ PAssert.that(
+ readPipeline
+ .apply(
+ "ReadFiles written by TFRecordIO.sink",
+ TFRecordIO.read()
+ .from(baseFilenameViaSink + "*")
+ .withCompression(readCompression))
+ .apply("To string readFiles from sink", ParDo.of(new ByteArrayToString())))
.containsInAnyOrder(elems);
readPipeline.run();
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
index 8d93c2a..fdc5eba 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java
@@ -79,11 +79,11 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream;
import org.apache.commons.compress.compressors.deflate.DeflateCompressorOutputStream;
import org.joda.time.Duration;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
index 23627e2..e358ff9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOWriteTest.java
@@ -21,7 +21,7 @@
import static org.apache.beam.sdk.TestUtils.LINES_ARRAY;
import static org.apache.beam.sdk.TestUtils.NO_LINES_ARRAY;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
@@ -66,15 +66,15 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Functions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Functions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextRowCountEstimatorTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextRowCountEstimatorTest.java
new file mode 100644
index 0000000..6f53d1e
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextRowCountEstimatorTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.Writer;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Tests for {@link org.apache.beam.sdk.io.TextRowCountEstimator}. */
+@RunWith(JUnit4.class)
+public class TextRowCountEstimatorTest {
+ @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
+ private static final Logger LOG = LoggerFactory.getLogger(TextRowCountEstimatorTest.class);
+
+ @Test
+ public void testNonEmptyFiles() throws Exception {
+ File file1 = temporaryFolder.newFile("file1.txt");
+ Writer writer = Files.newWriter(file1, Charsets.UTF_8);
+ for (int i = 0; i < 100; i++) {
+ writer.write("123123123\n");
+ }
+ writer.flush();
+ writer.close();
+ temporaryFolder.newFolder("testfolder");
+ temporaryFolder.newFolder("testfolder2");
+ file1 = temporaryFolder.newFile("testfolder/test2.txt");
+ writer = Files.newWriter(file1, Charsets.UTF_8);
+ for (int i = 0; i < 50; i++) {
+ writer.write("123123123\n");
+ }
+
+ writer.flush();
+ writer.close();
+ TextRowCountEstimator textRowCountEstimator =
+ TextRowCountEstimator.builder().setFilePattern(temporaryFolder.getRoot() + "/**").build();
+ Double rows = textRowCountEstimator.estimateRowCount(PipelineOptionsFactory.create());
+ Assert.assertNotNull(rows);
+ Assert.assertEquals(150d, rows, 0.01);
+ }
+
+ @Test(expected = FileNotFoundException.class)
+ public void testEmptyFolder() throws Exception {
+ TextRowCountEstimator textRowCountEstimator =
+ TextRowCountEstimator.builder().setFilePattern(temporaryFolder.getRoot() + "/**").build();
+ Double rows = textRowCountEstimator.estimateRowCount(PipelineOptionsFactory.create());
+ }
+
+ @Test
+ public void testEmptyFile() throws Exception {
+ File file1 = temporaryFolder.newFile("file1.txt");
+ Writer writer = Files.newWriter(file1, Charsets.UTF_8);
+ for (int i = 0; i < 100; i++) {
+ writer.write("\n");
+ }
+ writer.flush();
+ writer.close();
+ TextRowCountEstimator textRowCountEstimator =
+ TextRowCountEstimator.builder().setFilePattern(temporaryFolder.getRoot() + "/**").build();
+ Double rows = textRowCountEstimator.estimateRowCount(PipelineOptionsFactory.create());
+ Assert.assertEquals(0d, rows, 0.01);
+ }
+
+ @Test(expected = TextRowCountEstimator.NoEstimationException.class)
+ public void lotsOfNewLines() throws Exception {
+ File file1 = temporaryFolder.newFile("file1.txt");
+ Writer writer = Files.newWriter(file1, Charsets.UTF_8);
+ for (int i = 0; i < 1000; i++) {
+ writer.write("\n");
+ }
+ writer.write("123123123");
+ writer.flush();
+ writer.close();
+ TextRowCountEstimator textRowCountEstimator =
+ TextRowCountEstimator.builder()
+ .setNumSampledBytesPerFile(10L)
+ .setFilePattern(temporaryFolder.getRoot() + "/**")
+ .build();
+ textRowCountEstimator.estimateRowCount(PipelineOptionsFactory.create());
+ }
+
+ @Test(expected = FileNotFoundException.class)
+ public void testNonExistence() throws Exception {
+ TextRowCountEstimator textRowCountEstimator =
+ TextRowCountEstimator.builder()
+ .setFilePattern(temporaryFolder.getRoot() + "/something/**")
+ .build();
+ Double rows = textRowCountEstimator.estimateRowCount(PipelineOptionsFactory.create());
+ Assert.assertNull(rows);
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java
index afe5852..01bb331 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java
@@ -19,7 +19,7 @@
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.includesDisplayDataFor;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
@@ -87,9 +87,9 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.ShardedKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.commons.compress.utils.Sets;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeEstimateFractionTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeEstimateFractionTest.java
index 8b63c8f..7093d11 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeEstimateFractionTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeEstimateFractionTest.java
@@ -20,7 +20,7 @@
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.junit.Assert.assertThat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeInterpolateKeyTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeInterpolateKeyTest.java
index 607574e..0835402 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeInterpolateKeyTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeInterpolateKeyTest.java
@@ -22,7 +22,7 @@
import static org.hamcrest.Matchers.greaterThanOrEqualTo;
import static org.junit.Assert.assertThat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeTest.java
index 90994e3..0025e3e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/range/ByteKeyRangeTest.java
@@ -28,7 +28,7 @@
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricResultsMatchers.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricResultsMatchers.java
index db437c3..0519255 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricResultsMatchers.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricResultsMatchers.java
@@ -44,8 +44,49 @@
}
/**
+ * Matches a {@link MetricResult} with the given namespace, name and step, and a matcher for the
+ * value for either committed or attempted (based on {@code isCommitted}) metrics.
+ */
+ public static <T> Matcher<MetricResult<T>> metricsResult(
+ final String namespace,
+ final String name,
+ final String step,
+ final Matcher<T> valueMatcher,
+ final boolean isCommitted) {
+
+ final String metricState = isCommitted ? "committed" : "attempted";
+ return new MatchNameAndKey<T>(namespace, name, step) {
+ @Override
+ protected boolean matchesSafely(MetricResult<T> item) {
+ final T metricValue = isCommitted ? item.getCommitted() : item.getAttempted();
+ return super.matchesSafely(item) && valueMatcher.matches(metricValue);
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ super.describeTo(description);
+ description.appendText(String.format(", %s=", metricState));
+ valueMatcher.describeTo(description);
+ description.appendText("}");
+ }
+
+ @Override
+ protected void describeMismatchSafely(MetricResult<T> item, Description mismatchDescription) {
+ final T metricValue = isCommitted ? item.getCommitted() : item.getAttempted();
+ super.describeMismatchSafely(item, mismatchDescription);
+ mismatchDescription.appendText(String.format("%s: ", metricState));
+ valueMatcher.describeMismatch(metricValue, mismatchDescription);
+ mismatchDescription.appendText("}");
+ }
+ };
+ }
+
+ /**
* Matches a {@link MetricResult} with the given namespace, name and step, and whose value equals
* the given value for either committed or attempted (based on {@code isCommitted}) metrics.
+ *
+ * <p>For metrics with a {@link {@link GaugeResult}}, only the value is matched and the timestamp
+ * is ignored.
*/
public static <T> Matcher<MetricResult<T>> metricsResult(
final String namespace,
@@ -54,24 +95,17 @@
final T value,
final boolean isCommitted) {
final String metricState = isCommitted ? "committed" : "attempted";
- return new TypeSafeMatcher<MetricResult<T>>() {
+ return new MatchNameAndKey<T>(namespace, name, step) {
@Override
protected boolean matchesSafely(MetricResult<T> item) {
final T metricValue = isCommitted ? item.getCommitted() : item.getAttempted();
- return MetricFiltering.matches(MetricsFilter.builder().addStep(step).build(), item.getKey())
- && Objects.equals(MetricName.named(namespace, name), item.getName())
- && metricResultsEqual(value, metricValue);
+ return super.matchesSafely(item) && metricResultsEqual(value, metricValue);
}
@Override
public void describeTo(Description description) {
+ super.describeTo(description);
description
- .appendText("MetricResult{inNamespace=")
- .appendValue(namespace)
- .appendText(", name=")
- .appendValue(name)
- .appendText(", step=")
- .appendValue(step)
.appendText(String.format(", %s=", metricState))
.appendValue(value)
.appendText("}");
@@ -79,11 +113,8 @@
@Override
protected void describeMismatchSafely(MetricResult<T> item, Description mismatchDescription) {
- mismatchDescription.appendText("MetricResult{");
final T metricValue = isCommitted ? item.getCommitted() : item.getAttempted();
-
- describeMetricsResultMembersMismatch(item, mismatchDescription, namespace, name, step);
-
+ super.describeMismatchSafely(item, mismatchDescription);
if (!Objects.equals(value, metricValue)) {
mismatchDescription
.appendText(String.format("%s: ", metricState))
@@ -94,15 +125,15 @@
mismatchDescription.appendText("}");
}
- };
- }
- private static <T> boolean metricResultsEqual(T result1, T result2) {
- if (result1 instanceof GaugeResult) {
- return (((GaugeResult) result1).getValue()) == (((GaugeResult) result2).getValue());
- } else {
- return Objects.equals(result1, result2);
- }
+ private boolean metricResultsEqual(T result1, T result2) {
+ if (result1 instanceof GaugeResult) {
+ return (((GaugeResult) result1).getValue()) == (((GaugeResult) result2).getValue());
+ } else {
+ return Objects.equals(result1, result2);
+ }
+ }
+ };
}
static Matcher<MetricResult<DistributionResult>> distributionAttemptedMinMax(
@@ -131,25 +162,20 @@
final Long max,
final boolean isCommitted) {
final String metricState = isCommitted ? "committed" : "attempted";
- return new TypeSafeMatcher<MetricResult<DistributionResult>>() {
+ return new MatchNameAndKey<DistributionResult>(namespace, name, step) {
@Override
protected boolean matchesSafely(MetricResult<DistributionResult> item) {
- DistributionResult metricValue = isCommitted ? item.getCommitted() : item.getAttempted();
- return MetricFiltering.matches(MetricsFilter.builder().addStep(step).build(), item.getKey())
- && Objects.equals(MetricName.named(namespace, name), item.getName())
+ final DistributionResult metricValue =
+ isCommitted ? item.getCommitted() : item.getAttempted();
+ return super.matchesSafely(item)
&& Objects.equals(min, metricValue.getMin())
&& Objects.equals(max, metricValue.getMax());
}
@Override
public void describeTo(Description description) {
+ super.describeTo(description);
description
- .appendText("MetricResult{inNamespace=")
- .appendValue(namespace)
- .appendText(", name=")
- .appendValue(name)
- .appendText(", step=")
- .appendValue(step)
.appendText(String.format(", %sMin=", metricState))
.appendValue(min)
.appendText(String.format(", %sMax=", metricState))
@@ -160,11 +186,9 @@
@Override
protected void describeMismatchSafely(
MetricResult<DistributionResult> item, Description mismatchDescription) {
- mismatchDescription.appendText("MetricResult{");
-
- describeMetricsResultMembersMismatch(item, mismatchDescription, namespace, name, step);
- DistributionResult metricValue = isCommitted ? item.getCommitted() : item.getAttempted();
-
+ final DistributionResult metricValue =
+ isCommitted ? item.getCommitted() : item.getAttempted();
+ super.describeMismatchSafely(item, mismatchDescription);
if (!Objects.equals(min, metricValue.getMin())) {
mismatchDescription
.appendText(String.format("%sMin: ", metricState))
@@ -186,19 +210,53 @@
};
}
- private static <T> void describeMetricsResultMembersMismatch(
- MetricResult<T> item,
- Description mismatchDescription,
- String namespace,
- String name,
- String step) {
- MetricKey key = MetricKey.create(step, MetricName.named(namespace, name));
- if (!Objects.equals(key, item.getKey())) {
- mismatchDescription
- .appendText("inKey: ")
- .appendValue(key)
- .appendText(" != ")
- .appendValue(item.getKey());
+ private static class MatchNameAndKey<T> extends TypeSafeMatcher<MetricResult<T>> {
+
+ private final String namespace;
+ private final String name;
+ private final String step;
+
+ MatchNameAndKey(String namespace, String name, String step) {
+ this.namespace = namespace;
+ this.name = name;
+ this.step = step;
+ }
+
+ @Override
+ protected boolean matchesSafely(MetricResult<T> item) {
+ return MetricFiltering.matches(MetricsFilter.builder().addStep(step).build(), item.getKey())
+ && Objects.equals(MetricName.named(namespace, name), item.getName());
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description
+ .appendText("MetricResult{inNamespace=")
+ .appendValue(namespace)
+ .appendText(", name=")
+ .appendValue(name)
+ .appendText(", step=")
+ .appendValue(step);
+ if (this.getClass() == MatchNameAndKey.class) {
+ description.appendText("}");
+ }
+ }
+
+ @Override
+ protected void describeMismatchSafely(MetricResult<T> item, Description mismatchDescription) {
+ mismatchDescription.appendText("MetricResult{");
+ MetricKey key = MetricKey.create(step, MetricName.named(namespace, name));
+ if (!Objects.equals(key, item.getKey())) {
+ mismatchDescription
+ .appendText("inKey: ")
+ .appendValue(key)
+ .appendText(" != ")
+ .appendValue(item.getKey());
+ }
+
+ if (this.getClass() == MatchNameAndKey.class) {
+ mismatchDescription.appendText("}");
+ }
}
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
index 5f91baa..ab2da73 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsFactoryTest.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.options;
import static java.util.Locale.ROOT;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps.uniqueIndex;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps.uniqueIndex;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
@@ -67,13 +67,13 @@
import org.apache.beam.sdk.testing.InterceptingUrlClassLoader;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Collections2;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Collections2;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java
index 27a809f..ba165cc 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsReflectorTest.java
@@ -27,7 +27,7 @@
import com.fasterxml.jackson.annotation.JsonIgnore;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.hamcrest.FeatureMatcher;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
@@ -41,7 +41,7 @@
@Test
public void testGetOptionSpecs() throws NoSuchMethodException {
Set<PipelineOptionSpec> properties =
- PipelineOptionsReflector.getOptionSpecs(SimpleOptions.class);
+ PipelineOptionsReflector.getOptionSpecs(SimpleOptions.class, true);
assertThat(
properties,
@@ -60,7 +60,7 @@
@Test
public void testFiltersNonGetterMethods() {
Set<PipelineOptionSpec> properties =
- PipelineOptionsReflector.getOptionSpecs(OnlyTwoValidGetters.class);
+ PipelineOptionsReflector.getOptionSpecs(OnlyTwoValidGetters.class, true);
assertThat(properties, not(hasItem(hasName(isOneOf("misspelled", "hasParameter", "prefix")))));
}
@@ -91,7 +91,7 @@
@Test
public void testBaseClassOptions() {
Set<PipelineOptionSpec> props =
- PipelineOptionsReflector.getOptionSpecs(ExtendsSimpleOptions.class);
+ PipelineOptionsReflector.getOptionSpecs(ExtendsSimpleOptions.class, true);
assertThat(props, hasItem(allOf(hasName("foo"), hasClass(SimpleOptions.class))));
assertThat(props, hasItem(allOf(hasName("foo"), hasClass(ExtendsSimpleOptions.class))));
@@ -114,7 +114,7 @@
@Test
public void testExcludesNonPipelineOptionsMethods() {
Set<PipelineOptionSpec> properties =
- PipelineOptionsReflector.getOptionSpecs(ExtendsNonPipelineOptions.class);
+ PipelineOptionsReflector.getOptionSpecs(ExtendsNonPipelineOptions.class, true);
assertThat(properties, not(hasItem(hasName("foo"))));
}
@@ -132,11 +132,19 @@
@Test
public void testExcludesHiddenInterfaces() {
Set<PipelineOptionSpec> properties =
- PipelineOptionsReflector.getOptionSpecs(HiddenOptions.class);
+ PipelineOptionsReflector.getOptionSpecs(HiddenOptions.class, true);
assertThat(properties, not(hasItem(hasName("foo"))));
}
+ @Test
+ public void testIncludesHiddenInterfaces() {
+ Set<PipelineOptionSpec> properties =
+ PipelineOptionsReflector.getOptionSpecs(HiddenOptions.class, false);
+
+ assertThat(properties, hasItem(hasName("foo")));
+ }
+
/** Test interface. */
@Hidden
public interface HiddenOptions extends PipelineOptions {
@@ -148,7 +156,7 @@
@Test
public void testShouldSerialize() {
Set<PipelineOptionSpec> properties =
- PipelineOptionsReflector.getOptionSpecs(JsonIgnoreOptions.class);
+ PipelineOptionsReflector.getOptionSpecs(JsonIgnoreOptions.class, true);
assertThat(properties, hasItem(allOf(hasName("notIgnored"), shouldSerialize())));
assertThat(properties, hasItem(allOf(hasName("ignored"), not(shouldSerialize()))));
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsTest.java
index 6c50b18..09ac451 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/PipelineOptionsTest.java
@@ -27,7 +27,7 @@
import java.util.List;
import java.util.Map;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
index 5937bf7..0dc76ec 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ProxyInvocationHandlerTest.java
@@ -52,10 +52,10 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java
new file mode 100644
index 0000000..ef4b164
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.options;
+
+import static org.junit.Assert.assertEquals;
+
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link RemoteEnvironmentOptions}. */
+@RunWith(JUnit4.class)
+public class RemoteEnvironmentOptionsTest {
+
+ @Test
+ public void testSemiDirectory() {
+ RemoteEnvironmentOptions options = PipelineOptionsFactory.as(RemoteEnvironmentOptions.class);
+ assertEquals(null, options.getSemiPersistDir());
+
+ String semiDir = "/ab/cd";
+ options.setSemiPersistDir(semiDir);
+ assertEquals(semiDir, options.getSemiPersistDir());
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/SdkHarnessOptionsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/SdkHarnessOptionsTest.java
index 2a05fe0..c574495 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/SdkHarnessOptionsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/SdkHarnessOptionsTest.java
@@ -23,7 +23,7 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.beam.sdk.options.SdkHarnessOptions.SdkHarnessLogLevelOverrides;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
index bce90e7..6d410f7 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProvidersTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProvidersTest.java
index f717f05..5d033a8 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProvidersTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProvidersTest.java
@@ -22,7 +22,7 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
index 37651a6..4c3bc35 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
@@ -60,7 +60,7 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.junit.Before;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
index 6f350f1..06b2f9a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
@@ -19,6 +19,7 @@
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
import com.google.auto.value.AutoValue;
import java.math.BigDecimal;
@@ -219,7 +220,7 @@
assertEquals((short) 2, (Object) row.getInt16("aShort"));
assertEquals((int) 3, (Object) row.getInt32("anInt"));
assertEquals((long) 4, (Object) row.getInt64("aLong"));
- assertEquals(true, (Object) row.getBoolean("aBoolean"));
+ assertTrue(row.getBoolean("aBoolean"));
assertEquals(DATE.toInstant(), row.getDateTime("dateTime"));
assertEquals(DATE.toInstant(), row.getDateTime("instant"));
assertArrayEquals(BYTE_ARRAY, row.getBytes("bytes"));
@@ -234,7 +235,7 @@
assertEquals((short) 2, value.getaShort());
assertEquals((int) 3, value.getAnInt());
assertEquals((long) 4, value.getaLong());
- assertEquals(true, value.isaBoolean());
+ assertTrue(value.isaBoolean());
assertEquals(DATE, value.getDateTime());
assertEquals(DATE.toInstant(), value.getInstant());
assertArrayEquals("not equal", BYTE_ARRAY, value.getBytes());
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
index cd14213..f107332 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AvroSchemaTest.java
@@ -32,15 +32,28 @@
import org.apache.avro.reflect.AvroSchema;
import org.apache.avro.util.Utf8;
import org.apache.beam.sdk.schemas.LogicalTypes.FixedBytes;
+import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.schemas.transforms.Group;
import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Days;
+import org.joda.time.LocalDate;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
/** Tests for AVRO schema classes. */
public class AvroSchemaTest {
@@ -105,9 +118,17 @@
@org.apache.avro.reflect.Nullable public ByteBuffer bytes;
@AvroSchema("{\"type\": \"fixed\", \"size\": 4, \"name\": \"fixed4\"}")
- @org.apache.avro.reflect.Nullable
public byte[] fixed;
+ @AvroSchema("{\"type\": \"int\", \"logicalType\": \"date\"}")
+ public LocalDate date;
+
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestampMillis;
+
+ @AvroSchema("{\"name\": \"TestEnum\", \"type\": \"enum\", \"symbols\": [\"abc\",\"cde\"] }")
+ public TestEnum testEnum;
+
@org.apache.avro.reflect.Nullable public AvroSubPojo row;
@org.apache.avro.reflect.Nullable public List<AvroSubPojo> array;
@org.apache.avro.reflect.Nullable public Map<String, AvroSubPojo> map;
@@ -130,6 +151,9 @@
&& Objects.equals(string, avroPojo.string)
&& Objects.equals(bytes, avroPojo.bytes)
&& Arrays.equals(fixed, avroPojo.fixed)
+ && Objects.equals(date, avroPojo.date)
+ && Objects.equals(timestampMillis, avroPojo.timestampMillis)
+ && Objects.equals(testEnum, avroPojo.testEnum)
&& Objects.equals(row, avroPojo.row)
&& Objects.equals(array, avroPojo.array)
&& Objects.equals(map, avroPojo.map);
@@ -146,6 +170,9 @@
string,
bytes,
Arrays.hashCode(fixed),
+ date,
+ timestampMillis,
+ testEnum,
row,
array,
map);
@@ -160,6 +187,9 @@
String string,
ByteBuffer bytes,
byte[] fixed,
+ LocalDate date,
+ DateTime timestampMillis,
+ TestEnum testEnum,
AvroSubPojo row,
List<AvroSubPojo> array,
Map<String, AvroSubPojo> map) {
@@ -171,6 +201,9 @@
this.string = string;
this.bytes = bytes;
this.fixed = fixed;
+ this.date = date;
+ this.timestampMillis = timestampMillis;
+ this.testEnum = testEnum;
this.row = row;
this.array = array;
this.map = map;
@@ -197,7 +230,9 @@
.addNullableField("string", FieldType.STRING)
.addNullableField("bytes", FieldType.BYTES)
.addField("fixed", FieldType.logicalType(FixedBytes.of(4)))
- .addNullableField("timestampMillis", FieldType.DATETIME)
+ .addField("date", FieldType.DATETIME)
+ .addField("timestampMillis", FieldType.DATETIME)
+ .addField("testEnum", FieldType.STRING)
.addNullableField("row", SUB_TYPE)
.addNullableField("array", FieldType.array(SUB_TYPE))
.addNullableField("map", FieldType.map(FieldType.STRING, SUB_TYPE))
@@ -213,6 +248,9 @@
.addNullableField("string", FieldType.STRING)
.addNullableField("bytes", FieldType.BYTES)
.addField("fixed", FieldType.logicalType(FixedBytes.of(4)))
+ .addField("date", FieldType.DATETIME)
+ .addField("timestampMillis", FieldType.DATETIME)
+ .addField("testEnum", FieldType.STRING)
.addNullableField("row", SUB_TYPE)
.addNullableField("array", FieldType.array(SUB_TYPE.withNullable(false)))
.addNullableField("map", FieldType.map(FieldType.STRING, SUB_TYPE.withNullable(false)))
@@ -220,7 +258,8 @@
private static final byte[] BYTE_ARRAY = new byte[] {1, 2, 3, 4};
private static final DateTime DATE_TIME =
- new DateTime().withDate(1979, 03, 14).withTime(1, 2, 3, 4);
+ new DateTime().withDate(1979, 3, 14).withTime(1, 2, 3, 4);
+ private static final LocalDate DATE = new LocalDate(1979, 3, 14);
private static final TestAvroNested AVRO_NESTED_SPECIFIC_RECORD = new TestAvroNested(true, 42);
private static final TestAvro AVRO_SPECIFIC_RECORD =
new TestAvro(
@@ -232,7 +271,9 @@
"mystring",
ByteBuffer.wrap(BYTE_ARRAY),
new fixed4(BYTE_ARRAY),
+ DATE,
DATE_TIME,
+ TestEnum.abc,
AVRO_NESTED_SPECIFIC_RECORD,
ImmutableList.of(AVRO_NESTED_SPECIFIC_RECORD, AVRO_NESTED_SPECIFIC_RECORD),
ImmutableMap.of("k1", AVRO_NESTED_SPECIFIC_RECORD, "k2", AVRO_NESTED_SPECIFIC_RECORD));
@@ -255,7 +296,9 @@
GenericData.get()
.createFixed(
null, BYTE_ARRAY, org.apache.avro.Schema.createFixed("fixed4", "", "", 4)))
+ .set("date", (int) Days.daysBetween(new LocalDate(1970, 1, 1), DATE).getDays())
.set("timestampMillis", DATE_TIME.getMillis())
+ .set("testEnum", TestEnum.abc)
.set("row", AVRO_NESTED_GENERIC_RECORD)
.set("array", ImmutableList.of(AVRO_NESTED_GENERIC_RECORD, AVRO_NESTED_GENERIC_RECORD))
.set(
@@ -277,7 +320,9 @@
"mystring",
ByteBuffer.wrap(BYTE_ARRAY),
BYTE_ARRAY,
+ DATE.toDateTimeAtStartOfDay(DateTimeZone.UTC),
DATE_TIME,
+ "abc",
NESTED_ROW,
ImmutableList.of(NESTED_ROW, NESTED_ROW),
ImmutableMap.of("k1", NESTED_ROW, "k2", NESTED_ROW))
@@ -332,6 +377,9 @@
"mystring",
ByteBuffer.wrap(BYTE_ARRAY),
BYTE_ARRAY,
+ DATE,
+ DATE_TIME,
+ TestEnum.abc,
SUB_POJO,
ImmutableList.of(SUB_POJO, SUB_POJO),
ImmutableMap.of("k1", SUB_POJO, "k2", SUB_POJO));
@@ -347,6 +395,9 @@
"mystring",
ByteBuffer.wrap(BYTE_ARRAY),
BYTE_ARRAY,
+ DATE.toDateTimeAtStartOfDay(DateTimeZone.UTC),
+ DATE_TIME,
+ "abc",
NESTED_ROW,
ImmutableList.of(NESTED_ROW, NESTED_ROW),
ImmutableMap.of("k1", NESTED_ROW, "k2", NESTED_ROW))
@@ -365,4 +416,23 @@
new AvroRecordSchema().fromRowFunction(TypeDescriptor.of(AvroPojo.class));
assertEquals(AVRO_POJO, fromRow.apply(ROW_FOR_POJO));
}
+
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ @Category(ValidatesRunner.class)
+ public void testAvroPipelineGroupBy() {
+ PCollection<Row> input = pipeline.apply(Create.of(ROW_FOR_POJO)).setRowSchema(POJO_SCHEMA);
+
+ PCollection<KV<Row, Iterable<Row>>> output = input.apply(Group.byFieldNames("string"));
+ PAssert.that(output)
+ .containsInAnyOrder(
+ KV.of(
+ Row.withSchema(Schema.of(Field.of("string", FieldType.STRING)))
+ .addValue("mystring")
+ .build(),
+ ImmutableList.of(ROW_FOR_POJO)));
+
+ pipeline.run();
+ }
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/FieldAccessDescriptorTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/FieldAccessDescriptorTest.java
index 598ffc9..a82bd4b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/FieldAccessDescriptorTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/FieldAccessDescriptorTest.java
@@ -23,8 +23,8 @@
import java.util.stream.Collectors;
import org.apache.beam.sdk.schemas.FieldAccessDescriptor.FieldDescriptor;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
index d5df895..0842201 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
@@ -26,6 +26,7 @@
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
import java.math.BigDecimal;
import java.nio.charset.Charset;
@@ -42,10 +43,10 @@
import org.apache.beam.sdk.schemas.utils.TestJavaBeans.SimpleBean;
import org.apache.beam.sdk.schemas.utils.TestJavaBeans.SimpleBeanWithAnnotations;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
import org.joda.time.DateTime;
import org.junit.Rule;
import org.junit.Test;
@@ -123,7 +124,7 @@
assertEquals((short) 2, (Object) row.getInt16("aShort"));
assertEquals((int) 3, (Object) row.getInt32("anInt"));
assertEquals((long) 4, (Object) row.getInt64("aLong"));
- assertEquals(true, (Object) row.getBoolean("aBoolean"));
+ assertTrue(row.getBoolean("aBoolean"));
assertEquals(DATE.toInstant(), row.getDateTime("dateTime"));
assertEquals(DATE.toInstant(), row.getDateTime("instant"));
assertArrayEquals(BYTE_ARRAY, row.getBytes("bytes"));
@@ -143,7 +144,7 @@
assertEquals((short) 2, bean.getaShort());
assertEquals((int) 3, bean.getAnInt());
assertEquals((long) 4, bean.getaLong());
- assertEquals(true, bean.isaBoolean());
+ assertTrue(bean.isaBoolean());
assertEquals(DATE, bean.getDateTime());
assertEquals(DATE.toInstant(), bean.getInstant());
assertArrayEquals("not equal", BYTE_ARRAY, bean.getBytes());
@@ -177,7 +178,7 @@
assertEquals((short) 2, (Object) nestedRow.getInt16("aShort"));
assertEquals((int) 3, (Object) nestedRow.getInt32("anInt"));
assertEquals((long) 4, (Object) nestedRow.getInt64("aLong"));
- assertEquals(true, nestedRow.getBoolean("aBoolean"));
+ assertTrue(nestedRow.getBoolean("aBoolean"));
assertEquals(DATE.toInstant(), nestedRow.getDateTime("dateTime"));
assertEquals(DATE.toInstant(), nestedRow.getDateTime("instant"));
assertArrayEquals("not equal", BYTE_ARRAY, nestedRow.getBytes("bytes"));
@@ -199,7 +200,7 @@
assertEquals((short) 2, bean.getNested().getaShort());
assertEquals((int) 3, bean.getNested().getAnInt());
assertEquals((long) 4, bean.getNested().getaLong());
- assertEquals(true, bean.getNested().isaBoolean());
+ assertTrue(bean.getNested().isaBoolean());
assertEquals(DATE, bean.getNested().getDateTime());
assertEquals(DATE.toInstant(), bean.getNested().getInstant());
assertArrayEquals("not equal", BYTE_ARRAY, bean.getNested().getBytes());
@@ -367,7 +368,7 @@
assertEquals((short) 2, (Object) row.getInt16("aShort"));
assertEquals((int) 3, (Object) row.getInt32("anInt"));
assertEquals((long) 4, (Object) row.getInt64("aLong"));
- assertEquals(true, (Object) row.getBoolean("aBoolean"));
+ assertTrue(row.getBoolean("aBoolean"));
assertEquals(DATE.toInstant(), row.getDateTime("dateTime"));
assertEquals(DATE.toInstant(), row.getDateTime("instant"));
assertArrayEquals(BYTE_ARRAY, row.getBytes("bytes"));
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java
index 4d3db7a..a907def 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaFieldSchemaTest.java
@@ -30,6 +30,7 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
import java.math.BigDecimal;
import java.nio.ByteBuffer;
@@ -50,10 +51,10 @@
import org.apache.beam.sdk.schemas.utils.TestPOJOs.SimplePOJO;
import org.apache.beam.sdk.schemas.utils.TestPOJOs.StaticCreationSimplePojo;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Ints;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Ints;
import org.joda.time.DateTime;
import org.joda.time.Instant;
import org.junit.Test;
@@ -151,7 +152,7 @@
assertEquals((short) 2, (Object) row.getInt16("aShort"));
assertEquals((int) 3, (Object) row.getInt32("anInt"));
assertEquals((long) 4, (Object) row.getInt64("aLong"));
- assertEquals(true, row.getBoolean("aBoolean"));
+ assertTrue(row.getBoolean("aBoolean"));
assertEquals(DATE.toInstant(), row.getDateTime("dateTime"));
assertEquals(INSTANT, row.getDateTime("instant").toInstant());
assertArrayEquals(BYTE_ARRAY, row.getBytes("bytes"));
@@ -171,7 +172,7 @@
assertEquals((short) 2, pojo.aShort);
assertEquals((int) 3, pojo.anInt);
assertEquals((long) 4, pojo.aLong);
- assertEquals(true, pojo.aBoolean);
+ assertTrue(pojo.aBoolean);
assertEquals(DATE, pojo.dateTime);
assertEquals(INSTANT, pojo.instant);
assertArrayEquals("not equal", BYTE_ARRAY, pojo.bytes);
@@ -205,7 +206,7 @@
assertEquals((short) 2, (Object) nestedRow.getInt16("aShort"));
assertEquals((int) 3, (Object) nestedRow.getInt32("anInt"));
assertEquals((long) 4, (Object) nestedRow.getInt64("aLong"));
- assertEquals(true, nestedRow.getBoolean("aBoolean"));
+ assertTrue(nestedRow.getBoolean("aBoolean"));
assertEquals(DATE.toInstant(), nestedRow.getDateTime("dateTime"));
assertEquals(INSTANT, nestedRow.getDateTime("instant").toInstant());
assertArrayEquals("not equal", BYTE_ARRAY, nestedRow.getBytes("bytes"));
@@ -227,7 +228,7 @@
assertEquals((short) 2, pojo.nested.aShort);
assertEquals((int) 3, pojo.nested.anInt);
assertEquals((long) 4, pojo.nested.aLong);
- assertEquals(true, pojo.nested.aBoolean);
+ assertTrue(pojo.nested.aBoolean);
assertEquals(DATE, pojo.nested.dateTime);
assertEquals(INSTANT, pojo.nested.instant);
assertArrayEquals("not equal", BYTE_ARRAY, pojo.nested.bytes);
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaRegistryTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaRegistryTest.java
index 5c333b0..60f8538 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaRegistryTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaRegistryTest.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/AddFieldsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/AddFieldsTest.java
index af166e2..d837995 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/AddFieldsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/AddFieldsTest.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastTest.java
index 189554a..0dbed0e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastValidatorTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastValidatorTest.java
index b9b1d75..46533c3 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastValidatorTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CastValidatorTest.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.testing.UsesSchema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java
index 9e8d489..73aa085 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/CoGroupTest.java
@@ -38,8 +38,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java
index 4c6a6fa..5320f6b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/ConvertTest.java
@@ -32,8 +32,10 @@
import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -221,4 +223,28 @@
PAssert.that(pojos).containsInAnyOrder(new POJO2());
pipeline.run();
}
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFromRowsUnboxingRow() {
+ PCollection<POJO1Nested> pojos =
+ pipeline
+ .apply(Create.of(new POJO1()))
+ .apply(Select.fieldNames("field3"))
+ .apply(Convert.to(TypeDescriptor.of(POJO1Nested.class)));
+ PAssert.that(pojos).containsInAnyOrder(new POJO1Nested());
+ pipeline.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFromRowsUnboxingPrimitive() {
+ PCollection<Long> longs =
+ pipeline
+ .apply(Create.of(new POJO1()))
+ .apply(Select.fieldNames("field2"))
+ .apply(Convert.to(TypeDescriptors.longs()));
+ PAssert.that(longs).containsInAnyOrder((Long) EXPECTED_ROW1.getValue("field2"));
+ pipeline.run();
+ }
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java
index 52c67a4..abb3dd8 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/FilterTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/FilterTest.java
index 9d9fe31..9a5e2e4 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/FilterTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/FilterTest.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.schemas.transforms;
-import java.util.Objects;
-import org.apache.beam.sdk.schemas.JavaFieldSchema;
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
import org.apache.beam.sdk.testing.NeedsRunner;
import org.apache.beam.sdk.testing.PAssert;
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.testing.UsesSchema;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -41,48 +41,29 @@
@Rule public final transient TestPipeline pipeline = TestPipeline.create();
@Rule public transient ExpectedException thrown = ExpectedException.none();
- /** POJO used to test schemas. * */
- @DefaultSchema(JavaFieldSchema.class)
- public static class POJO {
- public String field1;
- public int field2;
- public int field3;
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ abstract static class Simple {
+ abstract String getField1();
- public POJO(String field1, int field2, int field3) {
- this.field1 = field1;
- this.field2 = field2;
- this.field3 = field3;
- }
+ abstract int getField2();
- public POJO() {}
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (o == null || getClass() != o.getClass()) {
- return false;
- }
- POJO pojo = (POJO) o;
- return Objects.equals(field1, pojo.field1)
- && Objects.equals(field2, pojo.field2)
- && Objects.equals(field3, pojo.field3);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(field1, field2, field3);
- }
+ abstract int getField3();
};
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ abstract static class Nested {
+ abstract Simple getNested();
+ }
+
@Test
@Category(NeedsRunner.class)
public void testMissingFieldName() {
thrown.expect(IllegalArgumentException.class);
pipeline
- .apply(Create.of(new POJO("pass", 52, 2)))
- .apply(Filter.<POJO>create().whereFieldName("missing", f -> true));
+ .apply(Create.of(new AutoValue_FilterTest_Simple("pass", 52, 2)))
+ .apply(Filter.<AutoValue_FilterTest_Simple>create().whereFieldName("missing", f -> true));
pipeline.run();
}
@@ -91,8 +72,8 @@
public void testMissingFieldIndex() {
thrown.expect(IllegalArgumentException.class);
pipeline
- .apply(Create.of(new POJO("pass", 52, 2)))
- .apply(Filter.<POJO>create().whereFieldId(23, f -> true));
+ .apply(Create.of(new AutoValue_FilterTest_Simple("pass", 52, 2)))
+ .apply(Filter.<AutoValue_FilterTest_Simple>create().whereFieldId(23, f -> true));
pipeline.run();
}
@@ -100,16 +81,40 @@
@Category(NeedsRunner.class)
public void testFilterFieldsByName() {
// Pass only elements where field1 == "pass && field2 > 50.
- PCollection<POJO> filtered =
+ PCollection<AutoValue_FilterTest_Simple> filtered =
pipeline
.apply(
Create.of(
- new POJO("pass", 52, 2), new POJO("pass", 2, 2), new POJO("fail", 100, 100)))
+ new AutoValue_FilterTest_Simple("pass", 52, 2),
+ new AutoValue_FilterTest_Simple("pass", 2, 2),
+ new AutoValue_FilterTest_Simple("fail", 100, 100)))
.apply(
- Filter.<POJO>create()
+ Filter.<AutoValue_FilterTest_Simple>create()
.whereFieldName("field1", s -> "pass".equals(s))
- .whereFieldName("field2", i -> (Integer) i > 50));
- PAssert.that(filtered).containsInAnyOrder(new POJO("pass", 52, 2));
+ .whereFieldName("field2", (Integer i) -> i > 50));
+ PAssert.that(filtered).containsInAnyOrder(new AutoValue_FilterTest_Simple("pass", 52, 2));
+ pipeline.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testFilterOnNestedField() {
+ // Pass only elements where field1 == "pass && field2 > 50.
+ PCollection<AutoValue_FilterTest_Nested> filtered =
+ pipeline
+ .apply(
+ Create.of(
+ new AutoValue_FilterTest_Nested(new AutoValue_FilterTest_Simple("pass", 52, 2)),
+ new AutoValue_FilterTest_Nested(new AutoValue_FilterTest_Simple("pass", 2, 2)),
+ new AutoValue_FilterTest_Nested(
+ new AutoValue_FilterTest_Simple("fail", 100, 100))))
+ .apply(
+ Filter.<AutoValue_FilterTest_Nested>create()
+ .whereFieldName("nested.field1", s -> "pass".equals(s))
+ .whereFieldName("nested.field2", (Integer i) -> i > 50));
+ PAssert.that(filtered)
+ .containsInAnyOrder(
+ new AutoValue_FilterTest_Nested(new AutoValue_FilterTest_Simple("pass", 52, 2)));
pipeline.run();
}
@@ -117,15 +122,22 @@
@Category(NeedsRunner.class)
public void testFilterMultipleFields() {
// Pass only elements where field1 + field2 >= 100.
- PCollection<POJO> filtered =
+ PCollection<AutoValue_FilterTest_Simple> filtered =
pipeline
- .apply(Create.of(new POJO("", 52, 48), new POJO("", 52, 2), new POJO("", 70, 33)))
.apply(
- Filter.<POJO>create()
+ Create.of(
+ new AutoValue_FilterTest_Simple("", 52, 48),
+ new AutoValue_FilterTest_Simple("", 52, 2),
+ new AutoValue_FilterTest_Simple("", 70, 33)))
+ .apply(
+ Filter.<AutoValue_FilterTest_Simple>create()
.whereFieldNames(
Lists.newArrayList("field2", "field3"),
r -> r.getInt32("field2") + r.getInt32("field3") >= 100));
- PAssert.that(filtered).containsInAnyOrder(new POJO("", 52, 48), new POJO("", 70, 33));
+ PAssert.that(filtered)
+ .containsInAnyOrder(
+ new AutoValue_FilterTest_Simple("", 52, 48),
+ new AutoValue_FilterTest_Simple("", 70, 33));
pipeline.run();
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/GroupTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/GroupTest.java
index 419b40e..7e6b404 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/GroupTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/GroupTest.java
@@ -48,8 +48,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matcher;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
index 26839a1..b8d776c 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTestUtils.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTestUtils.java
index 2ef6370..3a7b421 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTestUtils.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTestUtils.java
@@ -22,7 +22,7 @@
import java.util.stream.Collectors;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
class JoinTestUtils {
static List<Row> innerJoin(
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/RenameFieldsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/RenameFieldsTest.java
index b940e78..963fff8 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/RenameFieldsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/RenameFieldsTest.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/SelectTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/SelectTest.java
index e803c11..15d1379 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/SelectTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/SelectTest.java
@@ -28,8 +28,8 @@
import org.apache.beam.sdk.testing.UsesSchema;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroGenerators.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroGenerators.java
index 985aaff..cbeceab 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroGenerators.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroGenerators.java
@@ -29,9 +29,9 @@
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.apache.avro.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ObjectArrays;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ObjectArrays;
/** QuickCheck generators for AVRO. */
class AvroGenerators {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java
index 89c9d3c..cedeb77 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java
@@ -19,13 +19,13 @@
import static org.hamcrest.Matchers.not;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.junit.Assume.assumeThat;
import com.pholser.junit.quickcheck.From;
import com.pholser.junit.quickcheck.Property;
import com.pholser.junit.quickcheck.runner.JUnitQuickcheck;
-import java.io.IOException;
import java.math.BigDecimal;
import java.nio.ByteBuffer;
import java.util.List;
@@ -40,16 +40,21 @@
import org.apache.avro.generic.GenericRecordBuilder;
import org.apache.avro.reflect.ReflectData;
import org.apache.avro.util.Utf8;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.io.AvroGeneratedUser;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.utils.AvroGenerators.RecordSchemaGenerator;
import org.apache.beam.sdk.schemas.utils.AvroUtils.TypeWithNullability;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.joda.time.DateTime;
@@ -82,8 +87,8 @@
@Property(trials = 1000)
@SuppressWarnings("unchecked")
- public void avroToBeamRoudTrip(
- @From(RecordSchemaGenerator.class) org.apache.avro.Schema avroSchema) throws IOException {
+ public void avroToBeamRoundTrip(
+ @From(RecordSchemaGenerator.class) org.apache.avro.Schema avroSchema) {
// not everything is possible to translate
assumeThat(avroSchema, not(containsField(AvroUtilsTest::hasNonNullUnion)));
// roundtrip for enums returns strings because Beam doesn't have enum type
@@ -140,17 +145,54 @@
typeWithNullability.type);
}
- private org.apache.avro.Schema getAvroSubSchema() {
+ @Test
+ public void testNullableArrayFieldToBeamArrayField() {
+ org.apache.avro.Schema.Field avroField =
+ new org.apache.avro.Schema.Field(
+ "arrayField",
+ ReflectData.makeNullable(
+ org.apache.avro.Schema.createArray((org.apache.avro.Schema.create(Type.INT)))),
+ "",
+ null);
+
+ Field expectedBeamField = Field.nullable("arrayField", FieldType.array(FieldType.INT32));
+
+ Field beamField = AvroUtils.toBeamField(avroField);
+ assertEquals(expectedBeamField, beamField);
+ }
+
+ @Test
+ public void testNullableBeamArrayFieldToAvroField() {
+ Field beamField = Field.nullable("arrayField", FieldType.array(FieldType.INT32));
+
+ org.apache.avro.Schema.Field expectedAvroField =
+ new org.apache.avro.Schema.Field(
+ "arrayField",
+ ReflectData.makeNullable(
+ org.apache.avro.Schema.createArray((org.apache.avro.Schema.create(Type.INT)))),
+ "",
+ null);
+
+ org.apache.avro.Schema.Field avroField = AvroUtils.toAvroField(beamField, "ignored");
+ assertEquals(expectedAvroField, avroField);
+ }
+
+ private static List<org.apache.avro.Schema.Field> getAvroSubSchemaFields() {
List<org.apache.avro.Schema.Field> fields = Lists.newArrayList();
fields.add(
new org.apache.avro.Schema.Field(
"bool", org.apache.avro.Schema.create(Type.BOOLEAN), "", null));
fields.add(
new org.apache.avro.Schema.Field("int", org.apache.avro.Schema.create(Type.INT), "", null));
- return org.apache.avro.Schema.createRecord(fields);
+ return fields;
}
- private org.apache.avro.Schema getAvroSchema() {
+ private static org.apache.avro.Schema getAvroSubSchema(String name) {
+ return org.apache.avro.Schema.createRecord(
+ name, null, "topLevelRecord", false, getAvroSubSchemaFields());
+ }
+
+ private static org.apache.avro.Schema getAvroSchema() {
List<org.apache.avro.Schema.Field> fields = Lists.newArrayList();
fields.add(
new org.apache.avro.Schema.Field(
@@ -186,17 +228,20 @@
LogicalTypes.timestampMillis().addToSchema(org.apache.avro.Schema.create(Type.LONG)),
"",
(Object) null));
- fields.add(new org.apache.avro.Schema.Field("row", getAvroSubSchema(), "", (Object) null));
+ fields.add(new org.apache.avro.Schema.Field("row", getAvroSubSchema("row"), "", (Object) null));
fields.add(
new org.apache.avro.Schema.Field(
- "array", org.apache.avro.Schema.createArray(getAvroSubSchema()), "", (Object) null));
+ "array",
+ org.apache.avro.Schema.createArray(getAvroSubSchema("array")),
+ "",
+ (Object) null));
fields.add(
new org.apache.avro.Schema.Field(
- "map", org.apache.avro.Schema.createMap(getAvroSubSchema()), "", (Object) null));
- return org.apache.avro.Schema.createRecord(fields);
+ "map", org.apache.avro.Schema.createMap(getAvroSubSchema("map")), "", (Object) null));
+ return org.apache.avro.Schema.createRecord("topLevelRecord", null, null, false, fields);
}
- private Schema getBeamSubSchema() {
+ private static Schema getBeamSubSchema() {
return new Schema.Builder()
.addField(Field.of("bool", FieldType.BOOLEAN))
.addField(Field.of("int", FieldType.INT32))
@@ -221,10 +266,10 @@
.build();
}
- static final byte[] BYTE_ARRAY = new byte[] {1, 2, 3, 4};
- static final DateTime DATE_TIME =
- new DateTime().withDate(1979, 03, 14).withTime(1, 2, 3, 4).withZone(DateTimeZone.UTC);
- static final BigDecimal BIG_DECIMAL = new BigDecimal(3600);
+ private static final byte[] BYTE_ARRAY = new byte[] {1, 2, 3, 4};
+ private static final DateTime DATE_TIME =
+ new DateTime().withDate(1979, 3, 14).withTime(1, 2, 3, 4).withZone(DateTimeZone.UTC);
+ private static final BigDecimal BIG_DECIMAL = new BigDecimal(3600);
private Row getBeamRow() {
Row subRow = Row.withSchema(getBeamSubSchema()).addValues(true, 42).build();
@@ -244,10 +289,14 @@
.build();
}
- private GenericRecord getGenericRecord() {
+ private static GenericRecord getSubGenericRecord(String name) {
+ return new GenericRecordBuilder(getAvroSubSchema(name))
+ .set("bool", true)
+ .set("int", 42)
+ .build();
+ }
- GenericRecord subRecord =
- new GenericRecordBuilder(getAvroSubSchema()).set("bool", true).set("int", 42).build();
+ private static GenericRecord getGenericRecord() {
LogicalType decimalType =
LogicalTypes.decimal(Integer.MAX_VALUE)
@@ -266,9 +315,15 @@
.set("bytes", ByteBuffer.wrap(BYTE_ARRAY))
.set("decimal", encodedDecimal)
.set("timestampMillis", DATE_TIME.getMillis())
- .set("row", subRecord)
- .set("array", ImmutableList.of(subRecord, subRecord))
- .set("map", ImmutableMap.of(new Utf8("k1"), subRecord, new Utf8("k2"), subRecord))
+ .set("row", getSubGenericRecord("row"))
+ .set("array", ImmutableList.of(getSubGenericRecord("array"), getSubGenericRecord("array")))
+ .set(
+ "map",
+ ImmutableMap.of(
+ new Utf8("k1"),
+ getSubGenericRecord("map"),
+ new Utf8("k2"),
+ getSubGenericRecord("map")))
.build();
}
@@ -285,6 +340,61 @@
}
@Test
+ public void testAvroSchemaFromBeamSchemaCanBeParsed() {
+ org.apache.avro.Schema convertedSchema = AvroUtils.toAvroSchema(getBeamSchema());
+ org.apache.avro.Schema validatedSchema =
+ new org.apache.avro.Schema.Parser().parse(convertedSchema.toString());
+ assertEquals(convertedSchema, validatedSchema);
+ }
+
+ @Test
+ public void testAvroSchemaFromBeamSchemaWithFieldCollisionCanBeParsed() {
+
+ // Two similar schemas, the only difference is the "street" field type in the nested record.
+ Schema contact =
+ new Schema.Builder()
+ .addField(Field.of("name", FieldType.STRING))
+ .addField(
+ Field.of(
+ "address",
+ FieldType.row(
+ new Schema.Builder()
+ .addField(Field.of("street", FieldType.STRING))
+ .addField(Field.of("city", FieldType.STRING))
+ .build())))
+ .build();
+
+ Schema contactMultiline =
+ new Schema.Builder()
+ .addField(Field.of("name", FieldType.STRING))
+ .addField(
+ Field.of(
+ "address",
+ FieldType.row(
+ new Schema.Builder()
+ .addField(Field.of("street", FieldType.array(FieldType.STRING)))
+ .addField(Field.of("city", FieldType.STRING))
+ .build())))
+ .build();
+
+ // Ensure that no collisions happen between two sibling fields with same-named child fields
+ // (with different schemas, between a parent field and a sub-record field with the same name,
+ // and artificially with the generated field name.
+ Schema beamSchema =
+ new Schema.Builder()
+ .addField(Field.of("home", FieldType.row(contact)))
+ .addField(Field.of("work", FieldType.row(contactMultiline)))
+ .addField(Field.of("address", FieldType.row(contact)))
+ .addField(Field.of("topLevelRecord", FieldType.row(contactMultiline)))
+ .build();
+
+ org.apache.avro.Schema convertedSchema = AvroUtils.toAvroSchema(beamSchema);
+ org.apache.avro.Schema validatedSchema =
+ new org.apache.avro.Schema.Parser().parse(convertedSchema.toString());
+ assertEquals(convertedSchema, validatedSchema);
+ }
+
+ @Test
public void testNullableFieldInAvroSchema() {
List<org.apache.avro.Schema.Field> fields = Lists.newArrayList();
fields.add(
@@ -304,7 +414,8 @@
ReflectData.makeNullable(org.apache.avro.Schema.create(Type.INT))),
"",
null));
- org.apache.avro.Schema avroSchema = org.apache.avro.Schema.createRecord(fields);
+ org.apache.avro.Schema avroSchema =
+ org.apache.avro.Schema.createRecord("topLevelRecord", null, null, false, fields);
Schema expectedSchema =
Schema.builder()
@@ -358,7 +469,8 @@
ReflectData.makeNullable(org.apache.avro.Schema.create(Type.INT))),
"",
null));
- org.apache.avro.Schema avroSchema = org.apache.avro.Schema.createRecord(fields);
+ org.apache.avro.Schema avroSchema =
+ org.apache.avro.Schema.createRecord("topLevelRecord", null, null, false, fields);
assertEquals(avroSchema, AvroUtils.toAvroSchema(beamSchema));
Map<Utf8, Object> nullMapUtf8 = Maps.newHashMap();
@@ -394,6 +506,36 @@
assertEquals(getBeamRow(), row);
}
+ @Test
+ public void testAvroSchemaCoders() {
+ Pipeline pipeline = Pipeline.create();
+ org.apache.avro.Schema schema =
+ org.apache.avro.Schema.createRecord(
+ "TestSubRecord",
+ "TestSubRecord doc",
+ "org.apache.beam.sdk.schemas.utils",
+ false,
+ getAvroSubSchemaFields());
+ GenericRecord record =
+ new GenericRecordBuilder(getAvroSubSchema("simple"))
+ .set("bool", true)
+ .set("int", 42)
+ .build();
+
+ PCollection<GenericRecord> records =
+ pipeline.apply(Create.of(record).withCoder(AvroCoder.of(schema)));
+ assertFalse(records.hasSchema());
+ records.setCoder(AvroUtils.schemaCoder(schema));
+ assertTrue(records.hasSchema());
+
+ AvroGeneratedUser user = new AvroGeneratedUser("foo", 42, "green");
+ PCollection<AvroGeneratedUser> users =
+ pipeline.apply(Create.of(user).withCoder(AvroCoder.of(AvroGeneratedUser.class)));
+ assertFalse(users.hasSchema());
+ users.setCoder(AvroUtils.schemaCoder((AvroCoder<AvroGeneratedUser>) users.getCoder()));
+ assertTrue(users.hasSchema());
+ }
+
public static ContainsField containsField(Function<org.apache.avro.Schema, Boolean> predicate) {
return new ContainsField(predicate);
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtilsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtilsTest.java
index 5467483..b551a5e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtilsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/JavaBeanUtilsTest.java
@@ -142,7 +142,7 @@
assertEquals((short) 42, getters.get(2).get(simpleBean));
assertEquals((int) 43, getters.get(3).get(simpleBean));
assertEquals((long) 44, getters.get(4).get(simpleBean));
- assertEquals(true, getters.get(5).get(simpleBean));
+ assertTrue((Boolean) getters.get(5).get(simpleBean));
assertEquals(DateTime.parse("1979-03-14").toInstant(), getters.get(6).get(simpleBean));
assertEquals(DateTime.parse("1979-03-15").toInstant(), getters.get(7).get(simpleBean));
assertArrayEquals(
@@ -182,7 +182,7 @@
assertEquals((short) 42, simpleBean.getaShort());
assertEquals((int) 43, simpleBean.getAnInt());
assertEquals((long) 44, simpleBean.getaLong());
- assertEquals(true, simpleBean.isaBoolean());
+ assertTrue(simpleBean.isaBoolean());
assertEquals(DateTime.parse("1979-03-14"), simpleBean.getDateTime());
assertEquals(DateTime.parse("1979-03-15").toInstant(), simpleBean.getInstant());
assertArrayEquals(
@@ -211,7 +211,7 @@
assertEquals((short) 42, getters.get(1).get(bean));
assertEquals((int) 43, getters.get(2).get(bean));
assertEquals((long) 44, getters.get(3).get(bean));
- assertEquals(true, getters.get(4).get(bean));
+ assertTrue((Boolean) getters.get(4).get(bean));
}
@Test
@@ -231,7 +231,7 @@
assertEquals((short) 42, bean.getaShort().shortValue());
assertEquals((int) 43, bean.getAnInt().intValue());
assertEquals((long) 44, bean.getaLong().longValue());
- assertEquals(true, bean.getaBoolean().booleanValue());
+ assertTrue(bean.getaBoolean().booleanValue());
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/POJOUtilsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/POJOUtilsTest.java
index 697000c..6a75e25 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/POJOUtilsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/POJOUtilsTest.java
@@ -142,7 +142,7 @@
assertEquals((short) 42, getters.get(2).get(simplePojo));
assertEquals((int) 43, getters.get(3).get(simplePojo));
assertEquals((long) 44, getters.get(4).get(simplePojo));
- assertEquals(true, getters.get(5).get(simplePojo));
+ assertTrue((Boolean) getters.get(5).get(simplePojo));
assertEquals(DATE.toInstant(), getters.get(6).get(simplePojo));
assertEquals(INSTANT, getters.get(7).get(simplePojo));
assertArrayEquals("Unexpected bytes", BYTE_ARRAY, (byte[]) getters.get(8).get(simplePojo));
@@ -177,7 +177,7 @@
assertEquals((short) 42, simplePojo.aShort);
assertEquals((int) 43, simplePojo.anInt);
assertEquals((long) 44, simplePojo.aLong);
- assertEquals(true, simplePojo.aBoolean);
+ assertTrue(simplePojo.aBoolean);
assertEquals(DATE, simplePojo.dateTime);
assertEquals(INSTANT, simplePojo.instant);
assertArrayEquals("Unexpected bytes", BYTE_ARRAY, simplePojo.bytes);
@@ -199,7 +199,7 @@
assertEquals((short) 42, getters.get(1).get(pojo));
assertEquals((int) 43, getters.get(2).get(pojo));
assertEquals((long) 44, getters.get(3).get(pojo));
- assertEquals(true, getters.get(4).get(pojo));
+ assertTrue((Boolean) getters.get(4).get(pojo));
}
@Test
@@ -221,7 +221,7 @@
assertEquals((short) 42, pojo.aShort.shortValue());
assertEquals((int) 43, pojo.anInt.intValue());
assertEquals((long) 44, pojo.aLong.longValue());
- assertEquals(true, pojo.aBoolean.booleanValue());
+ assertTrue(pojo.aBoolean.booleanValue());
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SchemaZipFoldTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SchemaZipFoldTest.java
index 72ac451..35bb57a 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SchemaZipFoldTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SchemaZipFoldTest.java
@@ -27,7 +27,7 @@
import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.junit.Test;
/** Tests for {@link SchemaZipFold} with examples. */
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SelectHelpersTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SelectHelpersTest.java
index 77183bf..2682fc0 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SelectHelpersTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SelectHelpersTest.java
@@ -23,9 +23,9 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
/** Tests for {@link SelectHelpers}. */
@@ -35,8 +35,10 @@
.addStringField("field1")
.addInt32Field("field2")
.addDoubleField("field3")
+ .addStringField("field_extra")
.build();
- static final Row FLAT_ROW = Row.withSchema(FLAT_SCHEMA).addValues("first", 42, 3.14).build();
+ static final Row FLAT_ROW =
+ Row.withSchema(FLAT_SCHEMA).addValues("first", 42, 3.14, "extra").build();
static final Schema NESTED_SCHEMA =
Schema.builder().addRowField("nested", FLAT_SCHEMA).addStringField("foo").build();
@@ -101,6 +103,19 @@
}
@Test
+ public void testsSimpleSelectSingleWithUnderscore() {
+ FieldAccessDescriptor fieldAccessDescriptor =
+ FieldAccessDescriptor.withFieldNames("field_extra").resolve(FLAT_SCHEMA);
+ Schema outputSchema = SelectHelpers.getOutputSchema(FLAT_SCHEMA, fieldAccessDescriptor);
+ Schema expectedSchema = Schema.builder().addStringField("field_extra").build();
+ assertEquals(expectedSchema, outputSchema);
+
+ Row row = SelectHelpers.selectRow(FLAT_ROW, fieldAccessDescriptor, FLAT_SCHEMA, outputSchema);
+ Row expectedRow = Row.withSchema(expectedSchema).addValue("extra").build();
+ assertEquals(expectedRow, row);
+ }
+
+ @Test
public void testsSimpleSelectMultiple() {
FieldAccessDescriptor fieldAccessDescriptor =
FieldAccessDescriptor.withFieldNames("field1", "field3").resolve(FLAT_SCHEMA);
@@ -273,6 +288,7 @@
.addMapField("field1", FieldType.INT32, FieldType.STRING)
.addMapField("field2", FieldType.INT32, FieldType.INT32)
.addMapField("field3", FieldType.INT32, FieldType.DOUBLE)
+ .addMapField("field_extra", FieldType.INT32, FieldType.STRING)
.build();
assertEquals(expectedSchema, outputSchema);
@@ -282,6 +298,7 @@
.addValue(ImmutableMap.of(1, FLAT_ROW.getValue(0)))
.addValue(ImmutableMap.of(1, FLAT_ROW.getValue(1)))
.addValue(ImmutableMap.of(1, FLAT_ROW.getValue(2)))
+ .addValue(ImmutableMap.of(1, FLAT_ROW.getValue(3)))
.build();
assertEquals(expectedRow, row);
}
diff --git a/sdks/java/core/src/test/avro/org/apache/beam/sdk/state/StateContextsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/state/StateContextsTest.java
similarity index 100%
rename from sdks/java/core/src/test/avro/org/apache/beam/sdk/state/StateContextsTest.java
rename to sdks/java/core/src/test/java/org/apache/beam/sdk/state/StateContextsTest.java
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CoderPropertiesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CoderPropertiesTest.java
index 948048a..78c2325 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CoderPropertiesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CoderPropertiesTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.CustomCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.hamcrest.CoreMatchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CombineFnTesterTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CombineFnTesterTest.java
index b49abb8..3c054f5 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CombineFnTesterTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/CombineFnTesterTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
index a85a7c3..fd3927f 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/FileChecksumMatcherTest.java
@@ -25,7 +25,7 @@
import java.nio.charset.StandardCharsets;
import java.util.regex.Pattern;
import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/GatherAllPanesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/GatherAllPanesTest.java
index e961456..6c43126 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/GatherAllPanesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/GatherAllPanesTest.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/InterceptingUrlClassLoader.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/InterceptingUrlClassLoader.java
index 75ab715..39e11ef 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/InterceptingUrlClassLoader.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/InterceptingUrlClassLoader.java
@@ -19,9 +19,9 @@
import java.io.IOException;
import java.util.function.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A classloader that intercepts loading of specifically named classes. This classloader copies the
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
index 8037d28..268d280 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PAssertTest.java
@@ -21,6 +21,7 @@
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -56,8 +57,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
@@ -511,7 +512,7 @@
new MatcherCheckerFn(SerializableMatchers.contains(11)));
String stacktrace = Throwables.getStackTraceAsString(res.assertionError());
- assertEquals(false, res.isSuccess());
+ assertFalse(res.isSuccess());
assertThat(stacktrace, containsString("PAssertionSite.capture"));
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
index 7cdd63f..db46736 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/** Methods for testing {@link PCollectionView}s. */
public final class PCollectionViewTesting {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PaneExtractorsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PaneExtractorsTest.java
index 106c06c..aa86f29 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PaneExtractorsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PaneExtractorsTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SerializableMatchersTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SerializableMatchersTest.java
index fbfc6a0..7e17f75 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SerializableMatchersTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SerializableMatchersTest.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SourceTestUtilsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SourceTestUtilsTest.java
index dedeef4..81c002b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SourceTestUtilsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/SourceTestUtilsTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.io.CountingSource;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/StaticWindowsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/StaticWindowsTest.java
index da9e4e7..c5d47f6 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/StaticWindowsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/StaticWindowsTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.transforms.windowing.IncompatibleWindowException;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/WindowSupplierTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/WindowSupplierTest.java
index 1b7fb73..676d820 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/WindowSupplierTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/WindowSupplierTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.util.SerializableUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
index acf87a7..6d6e07f 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateQuantilesTest.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.CoreMatchers;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateUniqueTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateUniqueTest.java
index 8d28c4e..bfa4f8d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateUniqueTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ApproximateUniqueTest.java
@@ -46,8 +46,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matcher;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
index dd37c05..1786b9e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineFnsTest.java
@@ -48,8 +48,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
index 6a87c82..654768f 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CombineTest.java
@@ -21,8 +21,8 @@
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasNamespace;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.includesDisplayDataFor;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.empty;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
@@ -80,10 +80,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.MatcherAssert;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java
index ccb9a9b..0cfb3fc 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/CreateTest.java
@@ -63,8 +63,8 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matchers;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DistinctTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DistinctTest.java
index be3df72..f2e7517 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DistinctTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DistinctTest.java
@@ -49,8 +49,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnTesterTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnTesterTest.java
index abecc4b..0f12ee0 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnTesterTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/DoFnTesterTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.contains;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.hasItems;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java
index bd7a7ee..9e606b3 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlatMapElementsTest.java
@@ -21,8 +21,8 @@
import static org.apache.beam.sdk.transforms.Requirements.requiresSideInputs;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
import java.io.Serializable;
import java.util.Collections;
@@ -43,8 +43,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
index aa6891b..8b988a7 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java
@@ -60,7 +60,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.junit.Assert;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
index a0e8311..6d8408e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
@@ -71,7 +71,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matcher;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupIntoBatchesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupIntoBatchesTest.java
index a54ab0d..dab99e6 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupIntoBatchesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupIntoBatchesTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LatestFnTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LatestFnTest.java
index 740ff6e..7e90578 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LatestFnTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/LatestFnTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.coders.NullableCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java
index 727c1fe..390a613 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MapElementsTest.java
@@ -21,12 +21,12 @@
import static org.apache.beam.sdk.transforms.Requirements.requiresSideInputs;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
+import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.hasKey;
import static org.hamcrest.Matchers.hasSize;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
import java.io.Serializable;
import java.util.Map;
@@ -46,7 +46,6 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
@@ -538,8 +537,6 @@
PAssert.that(result.output()).containsInAnyOrder(1);
- Map<String, String> expectedFailureInfo =
- ImmutableMap.of("className", "java.lang.ArithmeticException");
PAssert.thatSingleton(result.failures())
.satisfies(
kv -> {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MaxTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MaxTest.java
index 463e2d1..e540e63 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MaxTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MaxTest.java
@@ -23,7 +23,7 @@
import static org.junit.Assert.assertEquals;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MeanTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MeanTest.java
index a4ad8c5..6693447 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MeanTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MeanTest.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.transforms.Mean.CountSum;
import org.apache.beam.sdk.transforms.Mean.CountSumCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MinTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MinTest.java
index 770d6ae..d930ad2 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MinTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MinTest.java
@@ -23,7 +23,7 @@
import static org.junit.Assert.assertEquals;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
index f6815db..0685644 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java
@@ -34,6 +34,7 @@
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.sdk.state.StateSpec;
import org.apache.beam.sdk.state.StateSpecs;
@@ -335,6 +336,7 @@
// exception is not necessarily thrown on every instance. But we expect at least
// one during tests
static AtomicBoolean exceptionWasThrown = new AtomicBoolean(false);
+ static AtomicInteger noOfInstancesToTearDown = new AtomicInteger(0);
private final MethodForException toThrow;
private boolean thrown;
@@ -348,6 +350,7 @@
assertThat(
"lifecycle methods should not have been called", callStateMap.get(id()), is(nullValue()));
initCallState();
+ noOfInstancesToTearDown.incrementAndGet();
throwIfNecessary(MethodForException.SETUP);
}
@@ -391,8 +394,8 @@
@Teardown
public void after() {
- if (!exceptionWasThrown.get()) {
- fail("Excepted to have a processing method throw an exception");
+ if (noOfInstancesToTearDown.decrementAndGet() == 0 && !exceptionWasThrown.get()) {
+ fail("Expected to have a processing method throw an exception");
}
assertThat(
"some lifecycle method should have been called",
@@ -402,7 +405,11 @@
}
private void initCallState() {
- callStateMap.put(id(), new DelayedCallStateTracker(CallState.SETUP));
+ DelayedCallStateTracker previousTracker =
+ callStateMap.put(id(), new DelayedCallStateTracker(CallState.SETUP));
+ if (previousTracker != null) {
+ fail(CallState.SETUP + " method called multiple times");
+ }
}
private int id() {
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java
index b229d33..2c03a04 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoSchemaTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index 4586a4f..ee7c784 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -24,7 +24,7 @@
import static org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray;
import static org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString;
import static org.apache.beam.sdk.util.StringUtils.jsonStringToByteArray;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.anyOf;
import static org.hamcrest.Matchers.containsString;
@@ -53,7 +53,6 @@
import java.util.Set;
import org.apache.beam.sdk.coders.AtomicCoder;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.ListCoder;
import org.apache.beam.sdk.coders.SetCoder;
@@ -83,6 +82,7 @@
import org.apache.beam.sdk.testing.UsesMapState;
import org.apache.beam.sdk.testing.UsesSetState;
import org.apache.beam.sdk.testing.UsesSideInputs;
+import org.apache.beam.sdk.testing.UsesSideInputsWithDifferentCoders;
import org.apache.beam.sdk.testing.UsesStatefulParDo;
import org.apache.beam.sdk.testing.UsesTestStream;
import org.apache.beam.sdk.testing.UsesTestStreamWithProcessingTime;
@@ -110,12 +110,12 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -739,6 +739,290 @@
}
@Test
+ @Category({NeedsRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotationFailedValidationMissing() {
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(@SideInput(sideInputTag1) String tag1) {}
+ };
+
+ thrown.expect(IllegalArgumentException.class);
+ PCollection<List<Integer>> output =
+ pipeline.apply("Create main input", Create.of(2)).apply(ParDo.of(fn));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({NeedsRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotationFailedValidationSingletonType() {
+
+ final PCollectionView<Integer> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2))
+ .apply("ViewSideInput1", View.asSingleton());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(@SideInput(sideInputTag1) String tag1) {}
+ };
+
+ thrown.expect(IllegalArgumentException.class);
+ PCollection<List<Integer>> output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({NeedsRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotationFailedValidationListType() {
+
+ final PCollectionView<List<Integer>> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2, 1, 0))
+ .apply("ViewSideInput1", View.asList());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(@SideInput(sideInputTag1) List<String> tag1) {}
+ };
+
+ thrown.expect(IllegalArgumentException.class);
+ PCollection<List<Integer>> output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({NeedsRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotationFailedValidationIterableType() {
+
+ final PCollectionView<Iterable<Integer>> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2, 1, 0))
+ .apply("ViewSideInput1", View.asIterable());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(@SideInput(sideInputTag1) List<String> tag1) {}
+ };
+
+ thrown.expect(IllegalArgumentException.class);
+ PCollection<List<Integer>> output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({NeedsRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotationFailedValidationMapType() {
+
+ final PCollectionView<Map<Integer, Integer>> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(KV.of(1, 2), KV.of(2, 3), KV.of(3, 4)))
+ .apply("ViewSideInput1", View.asMap());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(@SideInput(sideInputTag1) Map<String, String> tag1) {}
+ };
+
+ thrown.expect(IllegalArgumentException.class);
+ PCollection<List<Integer>> output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({NeedsRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotationFailedValidationMultiMapType() {
+
+ final PCollectionView<Map<Integer, Iterable<Integer>>> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(KV.of(1, 2), KV.of(1, 3), KV.of(3, 4)))
+ .apply("ViewSideInput1", View.asMultimap());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(@SideInput(sideInputTag1) Map<Integer, Integer> tag1) {}
+ };
+
+ thrown.expect(IllegalArgumentException.class);
+ PCollection<List<Integer>> output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({ValidatesRunner.class, UsesSideInputs.class})
+ public void testSideInputAnnotation() {
+
+ final PCollectionView<List<Integer>> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2, 1, 0))
+ .apply("ViewSideInput1", View.asList());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ DoFn<Integer, List<Integer>> fn =
+ new DoFn<Integer, List<Integer>>() {
+ @ProcessElement
+ public void processElement(
+ OutputReceiver<List<Integer>> r, @SideInput(sideInputTag1) List<Integer> tag1) {
+
+ List<Integer> sideSorted = Lists.newArrayList(tag1);
+ Collections.sort(sideSorted);
+ r.output(sideSorted);
+ }
+ };
+
+ PCollection<List<Integer>> output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput1));
+
+ PAssert.that(output).containsInAnyOrder(Lists.newArrayList(0, 1, 2));
+ pipeline.run();
+ }
+
+ @Test
+ @Category({
+ ValidatesRunner.class,
+ UsesSideInputs.class,
+ UsesSideInputsWithDifferentCoders.class
+ })
+ public void testSideInputAnnotationWithMultipleSideInputs() {
+
+ final List<Integer> side1Data = ImmutableList.of(2, 0);
+ final PCollectionView<List<Integer>> sideInput1 =
+ pipeline
+ .apply("CreateSideInput1", Create.of(side1Data))
+ .apply("ViewSideInput1", View.asList());
+
+ final Integer side2Data = 5;
+ final PCollectionView<Integer> sideInput2 =
+ pipeline
+ .apply("CreateSideInput2", Create.of(side2Data))
+ .apply("ViewSideInput2", View.asSingleton());
+
+ final List<Integer> side3Data = ImmutableList.of(1, 3);
+ final PCollectionView<Iterable<Integer>> sideInput3 =
+ pipeline
+ .apply("CreateSideInput3", Create.of(side3Data))
+ .apply("ViewSideInput3", View.asIterable());
+
+ final List<KV<Integer, Integer>> side4Data =
+ ImmutableList.of(KV.of(1, 2), KV.of(2, 3), KV.of(3, 4));
+ final PCollectionView<Map<Integer, Integer>> sideInput4 =
+ pipeline
+ .apply("CreateSideInput4", Create.of(side4Data))
+ .apply("ViewSideInput4", View.asMap());
+
+ final List<KV<Integer, Integer>> side5Data =
+ ImmutableList.of(KV.of(1, 2), KV.of(1, 3), KV.of(3, 4));
+ final PCollectionView<Map<Integer, Iterable<Integer>>> sideInput5 =
+ pipeline
+ .apply("CreateSideInput5", Create.of(side5Data))
+ .apply("ViewSideInput5", View.asMultimap());
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+ final String sideInputTag2 = "tag2";
+ final String sideInputTag3 = "tag3";
+ final String sideInputTag4 = "tag4";
+ final String sideInputTag5 = "tag5";
+
+ final TupleTag<Integer> outputTag1 = new TupleTag<>();
+ final TupleTag<Integer> outputTag2 = new TupleTag<>();
+ final TupleTag<Integer> outputTag3 = new TupleTag<>();
+ final TupleTag<KV<Integer, Integer>> outputTag4 = new TupleTag<>();
+ final TupleTag<KV<Integer, Integer>> outputTag5 = new TupleTag<>();
+
+ DoFn<Integer, Integer> fn =
+ new DoFn<Integer, Integer>() {
+ @ProcessElement
+ public void processElement(
+ MultiOutputReceiver r,
+ @SideInput(sideInputTag1) List<Integer> side1,
+ @SideInput(sideInputTag2) Integer side2,
+ @SideInput(sideInputTag3) Iterable<Integer> side3,
+ @SideInput(sideInputTag4) Map<Integer, Integer> side4,
+ @SideInput(sideInputTag5) Map<Integer, Iterable<Integer>> side5) {
+ side1.forEach(i -> r.get(outputTag1).output(i));
+ r.get(outputTag2).output(side2);
+ side3.forEach(i -> r.get(outputTag3).output(i));
+ side4.forEach((k, v) -> r.get(outputTag4).output(KV.of(k, v)));
+ side5.forEach((k, v) -> v.forEach(v2 -> r.get(outputTag5).output(KV.of(k, v2))));
+ }
+ };
+
+ PCollectionTuple output =
+ pipeline
+ .apply("Create main input", Create.of(2))
+ .apply(
+ ParDo.of(fn)
+ .withSideInput(sideInputTag1, sideInput1)
+ .withSideInput(sideInputTag2, sideInput2)
+ .withSideInput(sideInputTag3, sideInput3)
+ .withSideInput(sideInputTag4, sideInput4)
+ .withSideInput(sideInputTag5, sideInput5)
+ .withOutputTags(
+ outputTag1,
+ TupleTagList.of(outputTag2)
+ .and(outputTag3)
+ .and(outputTag4)
+ .and(outputTag5)));
+
+ output.get(outputTag1).setCoder(VarIntCoder.of());
+ output.get(outputTag2).setCoder(VarIntCoder.of());
+ output.get(outputTag3).setCoder(VarIntCoder.of());
+ output.get(outputTag4).setCoder(KvCoder.of(VarIntCoder.of(), VarIntCoder.of()));
+ output.get(outputTag5).setCoder(KvCoder.of(VarIntCoder.of(), VarIntCoder.of()));
+
+ PAssert.that(output.get(outputTag1)).containsInAnyOrder(side1Data);
+ PAssert.that(output.get(outputTag2)).containsInAnyOrder(side2Data);
+ PAssert.that(output.get(outputTag3)).containsInAnyOrder(side3Data);
+ PAssert.that(output.get(outputTag4)).containsInAnyOrder(side4Data);
+ PAssert.that(output.get(outputTag5)).containsInAnyOrder(side5Data);
+
+ pipeline.run();
+ }
+
+ @Test
@Category({ValidatesRunner.class, UsesSideInputs.class})
public void testParDoWithSideInputsIsCumulative() {
@@ -1967,7 +2251,6 @@
final PCollectionView<List<Integer>> listView =
pipeline.apply("Create list for side input", Create.of(2, 1, 0)).apply(View.asList());
-
final String stateId = "foo";
DoFn<KV<String, Integer>, List<Integer>> fn =
new DoFn<KV<String, Integer>, List<Integer>>() {
@@ -2011,6 +2294,64 @@
.containsInAnyOrder(Lists.newArrayList(12, 42, 84, 97), Lists.newArrayList(0, 1, 2));
pipeline.run();
}
+
+ @Test
+ @Category({ValidatesRunner.class, UsesStatefulParDo.class, UsesSideInputs.class})
+ public void testStateSideInput() {
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+
+ Coder<MyInteger> myIntegerCoder = MyIntegerCoder.of();
+ pipeline.getCoderRegistry().registerCoderForClass(MyInteger.class, myIntegerCoder);
+ final PCollectionView<Integer> sideInput =
+ pipeline
+ .apply("CreateSideInput1", Create.of(2))
+ .apply("ViewSideInput1", View.asSingleton());
+
+ TestSimpleStatefulDoFn fn = new TestSimpleStatefulDoFn(sideInput);
+ pipeline
+ .apply(Create.of(KV.of(1, 2)))
+ .apply(ParDo.of(fn).withSideInput(sideInputTag1, sideInput));
+
+ pipeline.run();
+ }
+
+ private static class TestSimpleStatefulDoFn extends DoFn<KV<Integer, Integer>, Integer> {
+
+ // SideInput tag id
+ final String sideInputTag1 = "tag1";
+ private final PCollectionView<Integer> view;
+
+ final String stateId = "foo";
+ Coder<MyInteger> myIntegerCoder = MyIntegerCoder.of();
+
+ @StateId(stateId)
+ private final StateSpec<BagState<MyInteger>> bufferState = StateSpecs.bag();
+
+ private TestSimpleStatefulDoFn(PCollectionView<Integer> view) {
+ this.view = view;
+ }
+
+ @ProcessElement
+ public void processElem(
+ ProcessContext c,
+ @SideInput(sideInputTag1) Integer sideInputTag,
+ @StateId(stateId) BagState<MyInteger> state) {
+ state.add(new MyInteger(sideInputTag));
+ c.output(sideInputTag);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ return other instanceof TestSimpleStatefulDoFn;
+ }
+
+ @Override
+ public int hashCode() {
+ return getClass().hashCode();
+ }
+ }
}
/** Tests for state coder inference behaviors. */
@@ -3337,11 +3678,10 @@
}
@Override
- public void encode(TestDummy value, OutputStream outStream)
- throws CoderException, IOException {}
+ public void encode(TestDummy value, OutputStream outStream) throws IOException {}
@Override
- public TestDummy decode(InputStream inStream) throws CoderException, IOException {
+ public TestDummy decode(InputStream inStream) throws IOException {
return new TestDummy();
}
@@ -3445,12 +3785,12 @@
}
@Override
- public void encode(MyInteger value, OutputStream outStream) throws CoderException, IOException {
+ public void encode(MyInteger value, OutputStream outStream) throws IOException {
delegate.encode(value.getValue(), outStream);
}
@Override
- public MyInteger decode(InputStream inStream) throws CoderException, IOException {
+ public MyInteger decode(InputStream inStream) throws IOException {
return new MyInteger(delegate.decode(inStream));
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java
index 04f6d8d..55c62cb 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SampleTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SampleTest.java
index bd09ee0..9f7e89f 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SampleTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SampleTest.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.transforms;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.everyItem;
@@ -49,8 +49,8 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
index c86f8f6..eeb5d6d 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java
@@ -19,7 +19,7 @@
import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.resume;
import static org.apache.beam.sdk.transforms.DoFn.ProcessContinuation.stop;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
@@ -61,7 +61,7 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.MutableDateTime;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SumTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SumTest.java
index 44ae0cb..3486f6b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SumTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SumTest.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.coders.DoubleCoder;
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
new file mode 100644
index 0000000..76a29a9
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.UsesSchema;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link JsonToRow}. */
+@RunWith(JUnit4.class)
+@Category(UsesSchema.class)
+public class ToJsonTest implements Serializable {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ abstract static class Person {
+ public static Person of(String name, Integer height, Boolean knowsJavascript) {
+ return new AutoValue_ToJsonTest_Person(name, height, knowsJavascript);
+ }
+
+ public abstract String getName();
+
+ public abstract Integer getHeight();
+
+ public abstract Boolean getKnowsJavascript();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testSerializesParseableJson() throws Exception {
+ PCollection<Person> persons =
+ pipeline.apply(
+ "jsonPersons",
+ Create.of(
+ Person.of("person1", 80, true),
+ Person.of("person2", 70, false),
+ Person.of("person3", 60, true),
+ Person.of("person4", 50, false),
+ Person.of("person5", 40, true)));
+
+ Schema personSchema =
+ Schema.builder()
+ .addStringField("name")
+ .addInt32Field("height")
+ .addBooleanField("knowsJavascript")
+ .build();
+
+ PCollection<Row> personRows =
+ persons.apply(ToJson.of()).apply(JsonToRow.withSchema(personSchema));
+
+ PAssert.that(personRows)
+ .containsInAnyOrder(
+ row(personSchema, "person1", 80, true),
+ row(personSchema, "person2", 70, false),
+ row(personSchema, "person3", 60, true),
+ row(personSchema, "person4", 50, false),
+ row(personSchema, "person5", 40, true));
+
+ pipeline.run();
+ }
+
+ private Row row(Schema schema, Object... values) {
+ return Row.withSchema(schema).addValues(values).build();
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java
index 74a8f3a..3fdec45 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.transforms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import static org.hamcrest.Matchers.isA;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
@@ -61,7 +61,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WaitTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WaitTest.java
index 61e9d73..c1410d3 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WaitTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WaitTest.java
@@ -40,8 +40,8 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WatchTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WatchTest.java
index 7ac1796..615eaf6 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WatchTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WatchTest.java
@@ -60,15 +60,15 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnel;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Funnels;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashCode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnel;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Funnels;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.joda.time.ReadableDuration;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WithFailuresTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WithFailuresTest.java
index 5841674..827331b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WithFailuresTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/WithFailuresTest.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataEvaluator.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataEvaluator.java
index ae4f093..5cdfc41 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataEvaluator.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataEvaluator.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* Test utilities to evaluate the {@link DisplayData} in the context of a {@link PipelineRunner}.
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataMatchers.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataMatchers.java
index c39447d..de5d0ab 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataMatchers.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataMatchers.java
@@ -23,7 +23,7 @@
import java.util.Arrays;
import java.util.Collection;
import org.apache.beam.sdk.transforms.display.DisplayData.Item;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.CustomTypeSafeMatcher;
import org.hamcrest.Description;
import org.hamcrest.FeatureMatcher;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
index ee75c66..fea68ff 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/display/DisplayDataTest.java
@@ -58,10 +58,10 @@
import org.apache.beam.sdk.transforms.display.DisplayData.Item;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.hamcrest.CustomTypeSafeMatcher;
import org.hamcrest.FeatureMatcher;
import org.hamcrest.Matcher;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGbkResultCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGbkResultCoderTest.java
index 6e94b83..b52709b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGbkResultCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGbkResultCoderTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
index 28bef71..09553d1 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/CoGroupByKeyTest.java
@@ -45,8 +45,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/UnionCoderTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/UnionCoderTest.java
index d5511ec1..7a764e8 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/UnionCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/join/UnionCoderTest.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.coders.DoubleCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.testing.CoderProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
index da673da..bac8459 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesSplittableDoFnTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
index 8c80cfc..e0427f2 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java
@@ -56,6 +56,7 @@
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.PipelineOptionsParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.ProcessContextParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SchemaElementParameter;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.SideInputParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.StateParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TaggedOutputReceiverParameter;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature.Parameter.TimeDomainParameter;
@@ -110,10 +111,12 @@
BoundedWindow window,
PaneInfo paneInfo,
OutputReceiver<String> receiver,
- PipelineOptions options) {}
+ PipelineOptions options,
+ @SideInput("tag1") String input1,
+ @SideInput("tag2") Integer input2) {}
}.getClass());
- assertThat(sig.processElement().extraParameters().size(), equalTo(6));
+ assertThat(sig.processElement().extraParameters().size(), equalTo(8));
assertThat(sig.processElement().extraParameters().get(0), instanceOf(ElementParameter.class));
assertThat(sig.processElement().extraParameters().get(1), instanceOf(TimestampParameter.class));
assertThat(sig.processElement().extraParameters().get(2), instanceOf(WindowParameter.class));
@@ -122,6 +125,8 @@
sig.processElement().extraParameters().get(4), instanceOf(OutputReceiverParameter.class));
assertThat(
sig.processElement().extraParameters().get(5), instanceOf(PipelineOptionsParameter.class));
+ assertThat(sig.processElement().extraParameters().get(6), instanceOf(SideInputParameter.class));
+ assertThat(sig.processElement().extraParameters().get(7), instanceOf(SideInputParameter.class));
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java
index bc6da06..da7cd53 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java
@@ -26,7 +26,6 @@
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertTrue;
-import java.math.BigDecimal;
import org.apache.beam.sdk.io.range.ByteKey;
import org.apache.beam.sdk.io.range.ByteKeyRange;
import org.junit.Rule;
@@ -261,35 +260,31 @@
@Test
public void testBacklogUnstarted() {
ByteKeyRangeTracker tracker = ByteKeyRangeTracker.of(ByteKeyRange.ALL_KEYS);
- assertEquals(BigDecimal.ONE, tracker.getBacklog().backlog());
+ assertEquals(1., tracker.getSize(), 0.001);
tracker = ByteKeyRangeTracker.of(ByteKeyRange.of(ByteKey.of(0x10), ByteKey.of(0xc0)));
- assertEquals(BigDecimal.ONE, tracker.getBacklog().backlog());
+ assertEquals(1., tracker.getSize(), 0.001);
}
@Test
public void testBacklogFinished() {
ByteKeyRangeTracker tracker = ByteKeyRangeTracker.of(ByteKeyRange.ALL_KEYS);
tracker.tryClaim(ByteKey.EMPTY);
- assertEquals(BigDecimal.ZERO, tracker.getBacklog().backlog());
+ assertEquals(0., tracker.getSize(), 0.001);
tracker = ByteKeyRangeTracker.of(ByteKeyRange.of(ByteKey.of(0x10), ByteKey.of(0xc0)));
tracker.tryClaim(ByteKey.of(0xd0));
- assertEquals(BigDecimal.ZERO, tracker.getBacklog().backlog());
+ assertEquals(0., tracker.getSize(), 0.001);
}
@Test
public void testBacklogPartiallyCompleted() {
ByteKeyRangeTracker tracker = ByteKeyRangeTracker.of(ByteKeyRange.ALL_KEYS);
tracker.tryClaim(ByteKey.of(0xa0));
- assertThat(
- tracker.getBacklog().backlog(),
- allOf(greaterThan(BigDecimal.ZERO), lessThan(BigDecimal.ONE)));
+ assertThat(tracker.getSize(), allOf(greaterThan(0.), lessThan(1.)));
tracker = ByteKeyRangeTracker.of(ByteKeyRange.of(ByteKey.of(0x10), ByteKey.of(0xc0)));
tracker.tryClaim(ByteKey.of(0xa0));
- assertThat(
- tracker.getBacklog().backlog(),
- allOf(greaterThan(BigDecimal.ZERO), lessThan(BigDecimal.ONE)));
+ assertThat(tracker.getSize(), allOf(greaterThan(0.), lessThan(1.)));
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java
index 2f162c6..f209247 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTrackerTest.java
@@ -21,7 +21,6 @@
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
-import java.math.BigDecimal;
import org.apache.beam.sdk.io.range.OffsetRange;
import org.junit.Rule;
import org.junit.Test;
@@ -159,31 +158,31 @@
@Test
public void testBacklogUnstarted() {
OffsetRangeTracker tracker = new OffsetRangeTracker(new OffsetRange(0, 200));
- assertEquals(BigDecimal.valueOf(200), tracker.getBacklog().backlog());
+ assertEquals(200, tracker.getSize(), 0.001);
tracker = new OffsetRangeTracker(new OffsetRange(100, 200));
- assertEquals(BigDecimal.valueOf(100), tracker.getBacklog().backlog());
+ assertEquals(100, tracker.getSize(), 0.001);
}
@Test
public void testBacklogFinished() {
OffsetRangeTracker tracker = new OffsetRangeTracker(new OffsetRange(0, 200));
tracker.tryClaim(300L);
- assertEquals(BigDecimal.ZERO, tracker.getBacklog().backlog());
+ assertEquals(0, tracker.getSize(), 0.001);
tracker = new OffsetRangeTracker(new OffsetRange(100, 200));
tracker.tryClaim(300L);
- assertEquals(BigDecimal.ZERO, tracker.getBacklog().backlog());
+ assertEquals(0., tracker.getSize(), 0.001);
}
@Test
public void testBacklogPartiallyCompleted() {
OffsetRangeTracker tracker = new OffsetRangeTracker(new OffsetRange(0, 200));
tracker.tryClaim(150L);
- assertEquals(BigDecimal.valueOf(50), tracker.getBacklog().backlog());
+ assertEquals(50., tracker.getSize(), 0.001);
tracker = new OffsetRangeTracker(new OffsetRange(100, 200));
tracker.tryClaim(150L);
- assertEquals(BigDecimal.valueOf(50), tracker.getBacklog().backlog());
+ assertEquals(50., tracker.getSize(), 0.001);
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/GlobalWindowTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/GlobalWindowTest.java
index 54ef7b5..6bd2d86 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/GlobalWindowTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/GlobalWindowTest.java
@@ -21,8 +21,8 @@
import org.apache.beam.sdk.coders.Coder.Context;
import org.apache.beam.sdk.testing.CoderProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/IntervalWindowTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/IntervalWindowTest.java
index ebc5b5f..5267423 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/IntervalWindowTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/IntervalWindowTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.coders.InstantCoder;
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/SessionsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/SessionsTest.java
index 6cf8da8..a7e922e 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/SessionsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/SessionsTest.java
@@ -33,7 +33,7 @@
import java.util.Set;
import org.apache.beam.sdk.testing.WindowFnTestUtils;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/StubTrigger.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/StubTrigger.java
index 081b418..d4ee23b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/StubTrigger.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/StubTrigger.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.transforms.windowing;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
/** No-op {@link OnceTrigger} implementation for testing. */
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
index 025550f..02ae679 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
@@ -70,7 +70,7 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowingTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowingTest.java
index 7e61313..c975615 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowingTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowingTest.java
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ApiSurfaceTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ApiSurfaceTest.java
index 979fb39..94b8995 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ApiSurfaceTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ApiSurfaceTest.java
@@ -23,10 +23,10 @@
import static org.junit.Assert.assertThat;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStreamTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStreamTest.java
index ba0e53b..205f1af 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStreamTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/BufferedElementCountingOutputStreamTest.java
@@ -34,8 +34,8 @@
import java.util.List;
import java.util.Random;
import org.apache.beam.sdk.coders.ByteArrayCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.collection.IsIterableContainingInOrder;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/CombineFnUtilTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/CombineFnUtilTest.java
index 19cb314..084b067 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/CombineFnUtilTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/CombineFnUtilTest.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.transforms.CombineWithContext.CombineFnWithContext;
import org.apache.beam.sdk.transforms.CombineWithContext.Context;
import org.apache.beam.sdk.transforms.Sum;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayInputStreamTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayInputStreamTest.java
index 0c8d70f..03fe8ed 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayInputStreamTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayInputStreamTest.java
@@ -24,7 +24,7 @@
import java.io.ByteArrayInputStream;
import java.io.IOException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStreamTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStreamTest.java
index a345190..cbcb86c 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStreamTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExposedByteArrayOutputStreamTest.java
@@ -24,7 +24,7 @@
import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFileTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFileTest.java
index 4353660..51214ed 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFileTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/FilePatternMatchingShardedFileTest.java
@@ -30,7 +30,7 @@
import java.nio.charset.StandardCharsets;
import org.apache.beam.sdk.io.LocalResources;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MutationDetectorsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MutationDetectorsTest.java
index 068085f..113cae9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MutationDetectorsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/MutationDetectorsTest.java
@@ -33,9 +33,9 @@
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.ListCoder;
import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/NumberedShardedFileTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/NumberedShardedFileTest.java
index 889b4f6..9d7fb82 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/NumberedShardedFileTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/NumberedShardedFileTest.java
@@ -31,7 +31,7 @@
import java.util.regex.Pattern;
import org.apache.beam.sdk.io.LocalResources;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
deleted file mode 100644
index 2bbabc3..0000000
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
+++ /dev/null
@@ -1,496 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util;
-
-import static org.hamcrest.Matchers.allOf;
-import static org.hamcrest.Matchers.hasProperty;
-import static org.hamcrest.Matchers.stringContainsInOrder;
-import static org.junit.Assert.assertEquals;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.module.SimpleModule;
-import java.math.BigDecimal;
-import java.util.Arrays;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
-import org.apache.beam.sdk.values.Row;
-import org.hamcrest.Matcher;
-import org.hamcrest.Matchers;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Unit tests for {@link RowJsonDeserializer}. */
-public class RowJsonDeserializerTest {
- private static final Boolean BOOLEAN_TRUE_VALUE = true;
- private static final String BOOLEAN_TRUE_STRING = "true";
- private static final Byte BYTE_VALUE = 126;
- private static final String BYTE_STRING = "126";
- private static final Short SHORT_VALUE = 32766;
- private static final String SHORT_STRING = "32766";
- private static final Integer INT_VALUE = 2147483646;
- private static final String INT_STRING = "2147483646";
- private static final Long LONG_VALUE = 9223372036854775806L;
- private static final String LONG_STRING = "9223372036854775806";
- private static final Float FLOAT_VALUE = 1.02e5f;
- private static final String FLOAT_STRING = "1.02e5";
- private static final Double DOUBLE_VALUE = 1.02d;
- private static final String DOUBLE_STRING = "1.02";
-
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testParsesFlatRow() throws Exception {
- Schema schema =
- Schema.builder()
- .addByteField("f_byte")
- .addInt16Field("f_int16")
- .addInt32Field("f_int32")
- .addInt64Field("f_int64")
- .addFloatField("f_float")
- .addDoubleField("f_double")
- .addBooleanField("f_boolean")
- .addStringField("f_string")
- .addDecimalField("f_decimal")
- .build();
-
- String rowString =
- "{\n"
- + "\"f_byte\" : 12,\n"
- + "\"f_int16\" : 22,\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_int64\" : 42,\n"
- + "\"f_float\" : 1.02E5,\n"
- + "\"f_double\" : 62.2,\n"
- + "\"f_boolean\" : true,\n"
- + "\"f_string\" : \"hello\",\n"
- + "\"f_decimal\" : 123.12\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(
- (byte) 12,
- (short) 22,
- 32,
- (long) 42,
- 1.02E5f,
- 62.2d,
- true,
- "hello",
- new BigDecimal("123.12"))
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testParsesArrayField() throws Exception {
- Schema schema =
- Schema.builder()
- .addInt32Field("f_int32")
- .addArrayField("f_intArray", FieldType.INT32)
- .build();
-
- String rowString = "{\n" + "\"f_int32\" : 32,\n" + "\"f_intArray\" : [ 1, 2, 3, 4, 5]\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow = Row.withSchema(schema).addValues(32, Arrays.asList(1, 2, 3, 4, 5)).build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testParsesArrayOfArrays() throws Exception {
-
- Schema schema =
- Schema.builder()
- .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
- .build();
-
- String rowString = "{\n" + "\"f_arrayOfIntArrays\" : [ [1, 2], [3, 4], [5]]\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addArray(Arrays.asList(1, 2), Arrays.asList(3, 4), Arrays.asList(5))
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMismatchedArrayField() throws Exception {
-
- Schema schema =
- Schema.builder()
- .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
- .build();
-
- String rowString =
- "{\n"
- + "\"f_arrayOfIntArrays\" : { }\n" // expect array, get object
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("Expected JSON array");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testParsesRowField() throws Exception {
- Schema nestedRowSchema =
- Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
-
- Schema schema =
- Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_row\" : {\n"
- + " \"f_nestedInt32\" : 54,\n"
- + " \"f_nestedString\" : \"foo\"\n"
- + " }\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(32, Row.withSchema(nestedRowSchema).addValues(54, "foo").build())
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMismatchedRowField() throws Exception {
- Schema nestedRowSchema =
- Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
-
- Schema schema =
- Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_int32\" : 32,\n"
- + "\"f_row\" : []\n" // expect object, get array
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("Expected JSON object");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testParsesNestedRowField() throws Exception {
-
- Schema doubleNestedRowSchema = Schema.builder().addStringField("f_doubleNestedString").build();
-
- Schema nestedRowSchema =
- Schema.builder().addRowField("f_nestedRow", doubleNestedRowSchema).build();
-
- Schema schema = Schema.builder().addRowField("f_row", nestedRowSchema).build();
-
- String rowString =
- "{\n"
- + "\"f_row\" : {\n"
- + " \"f_nestedRow\" : {\n"
- + " \"f_doubleNestedString\":\"foo\"\n"
- + " }\n"
- + " }\n"
- + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow =
- Row.withSchema(schema)
- .addValues(
- Row.withSchema(nestedRowSchema)
- .addValues(Row.withSchema(doubleNestedRowSchema).addValues("foo").build())
- .build())
- .build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForUnsupportedType() throws Exception {
- Schema schema = Schema.builder().addDateTimeField("f_dateTime").build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testThrowsForUnsupportedArrayElementType() throws Exception {
- Schema schema = Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testThrowsForUnsupportedNestedFieldType() throws Exception {
- Schema nestedSchema =
- Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
-
- Schema schema = Schema.builder().addRowField("f_nestedRow", nestedSchema).build();
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("DATETIME is not supported");
-
- RowJsonDeserializer.forSchema(schema);
- }
-
- @Test
- public void testParsesNulls() throws Exception {
- Schema schema =
- Schema.builder()
- .addByteField("f_byte")
- .addNullableField("f_string", FieldType.STRING)
- .build();
-
- String rowString = "{\n" + "\"f_byte\" : 12,\n" + "\"f_string\" : null\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- Row parsedRow = newObjectMapperWith(deserializer).readValue(rowString, Row.class);
-
- Row expectedRow = Row.withSchema(schema).addValues((byte) 12, null).build();
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testThrowsForMissingNotNullableField() throws Exception {
- Schema schema = Schema.builder().addByteField("f_byte").addStringField("f_string").build();
-
- String rowString = "{\n" + "\"f_byte\" : 12\n" + "}";
-
- RowJsonDeserializer deserializer = RowJsonDeserializer.forSchema(schema);
-
- thrown.expect(UnsupportedRowJsonException.class);
- thrown.expectMessage("'f_string' is not present");
-
- newObjectMapperWith(deserializer).readValue(rowString, Row.class);
- }
-
- @Test
- public void testSupportedBooleanConversions() throws Exception {
- testSupportedConversion(FieldType.BOOLEAN, BOOLEAN_TRUE_STRING, BOOLEAN_TRUE_VALUE);
- }
-
- @Test
- public void testSupportedStringConversions() throws Exception {
- testSupportedConversion(FieldType.STRING, quoted(FLOAT_STRING), FLOAT_STRING);
- }
-
- @Test
- public void testSupportedByteConversions() throws Exception {
- testSupportedConversion(FieldType.BYTE, BYTE_STRING, BYTE_VALUE);
- }
-
- @Test
- public void testSupportedShortConversions() throws Exception {
- testSupportedConversion(FieldType.INT16, BYTE_STRING, (short) BYTE_VALUE);
- testSupportedConversion(FieldType.INT16, SHORT_STRING, SHORT_VALUE);
- }
-
- @Test
- public void testSupportedIntConversions() throws Exception {
- testSupportedConversion(FieldType.INT32, BYTE_STRING, (int) BYTE_VALUE);
- testSupportedConversion(FieldType.INT32, SHORT_STRING, (int) SHORT_VALUE);
- testSupportedConversion(FieldType.INT32, INT_STRING, INT_VALUE);
- }
-
- @Test
- public void testSupportedLongConversions() throws Exception {
- testSupportedConversion(FieldType.INT64, BYTE_STRING, (long) BYTE_VALUE);
- testSupportedConversion(FieldType.INT64, SHORT_STRING, (long) SHORT_VALUE);
- testSupportedConversion(FieldType.INT64, INT_STRING, (long) INT_VALUE);
- testSupportedConversion(FieldType.INT64, LONG_STRING, LONG_VALUE);
- }
-
- @Test
- public void testSupportedFloatConversions() throws Exception {
- testSupportedConversion(FieldType.FLOAT, FLOAT_STRING, FLOAT_VALUE);
- testSupportedConversion(FieldType.FLOAT, SHORT_STRING, (float) SHORT_VALUE);
- }
-
- @Test
- public void testSupportedDoubleConversions() throws Exception {
- testSupportedConversion(FieldType.DOUBLE, DOUBLE_STRING, DOUBLE_VALUE);
- testSupportedConversion(FieldType.DOUBLE, FLOAT_STRING, (double) FLOAT_VALUE);
- testSupportedConversion(FieldType.DOUBLE, INT_STRING, (double) INT_VALUE);
- }
-
- private void testSupportedConversion(
- FieldType fieldType, String jsonFieldValue, Object expectedRowFieldValue) throws Exception {
-
- String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
- Schema schema = schemaWithField(fieldName, fieldType);
- Row expectedRow = Row.withSchema(schema).addValues(expectedRowFieldValue).build();
- ObjectMapper jsonParser = newObjectMapperWith(RowJsonDeserializer.forSchema(schema));
-
- Row parsedRow = jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
-
- assertEquals(expectedRow, parsedRow);
- }
-
- @Test
- public void testUnsupportedBooleanConversions() throws Exception {
- testUnsupportedConversion(FieldType.BOOLEAN, quoted(BOOLEAN_TRUE_STRING));
- testUnsupportedConversion(FieldType.BOOLEAN, BYTE_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, SHORT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, INT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, LONG_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, FLOAT_STRING);
- testUnsupportedConversion(FieldType.BOOLEAN, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedStringConversions() throws Exception {
- testUnsupportedConversion(FieldType.STRING, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.STRING, BYTE_STRING);
- testUnsupportedConversion(FieldType.STRING, SHORT_STRING);
- testUnsupportedConversion(FieldType.STRING, INT_STRING);
- testUnsupportedConversion(FieldType.STRING, LONG_STRING);
- testUnsupportedConversion(FieldType.STRING, FLOAT_STRING);
- testUnsupportedConversion(FieldType.STRING, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedByteConversions() throws Exception {
- testUnsupportedConversion(FieldType.BYTE, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.BYTE, quoted(BYTE_STRING));
- testUnsupportedConversion(FieldType.BYTE, SHORT_STRING);
- testUnsupportedConversion(FieldType.BYTE, INT_STRING);
- testUnsupportedConversion(FieldType.BYTE, LONG_STRING);
- testUnsupportedConversion(FieldType.BYTE, FLOAT_STRING);
- testUnsupportedConversion(FieldType.BYTE, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedShortConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT16, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT16, quoted(SHORT_STRING));
- testUnsupportedConversion(FieldType.INT16, INT_STRING);
- testUnsupportedConversion(FieldType.INT16, LONG_STRING);
- testUnsupportedConversion(FieldType.INT16, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT16, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedIntConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT32, quoted(INT_STRING));
- testUnsupportedConversion(FieldType.INT32, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT32, LONG_STRING);
- testUnsupportedConversion(FieldType.INT32, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT32, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedLongConversions() throws Exception {
- testUnsupportedConversion(FieldType.INT64, quoted(LONG_STRING));
- testUnsupportedConversion(FieldType.INT64, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.INT64, FLOAT_STRING);
- testUnsupportedConversion(FieldType.INT64, DOUBLE_STRING);
- }
-
- @Test
- public void testUnsupportedFloatConversions() throws Exception {
- testUnsupportedConversion(FieldType.FLOAT, quoted(FLOAT_STRING));
- testUnsupportedConversion(FieldType.FLOAT, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.FLOAT, DOUBLE_STRING);
- testUnsupportedConversion(FieldType.FLOAT, INT_STRING); // too large to fit
- }
-
- @Test
- public void testUnsupportedDoubleConversions() throws Exception {
- testUnsupportedConversion(FieldType.DOUBLE, quoted(DOUBLE_STRING));
- testUnsupportedConversion(FieldType.DOUBLE, BOOLEAN_TRUE_STRING);
- testUnsupportedConversion(FieldType.DOUBLE, LONG_STRING); // too large to fit
- }
-
- private void testUnsupportedConversion(FieldType fieldType, String jsonFieldValue)
- throws Exception {
-
- String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
-
- ObjectMapper jsonParser =
- newObjectMapperWith(RowJsonDeserializer.forSchema(schemaWithField(fieldName, fieldType)));
-
- thrown.expectMessage(fieldName);
- thrown.expectCause(unsupportedWithMessage(jsonFieldValue, "out of range"));
-
- jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
- }
-
- private String quoted(String string) {
- return "\"" + string + "\"";
- }
-
- private Schema schemaWithField(String fieldName, FieldType fieldType) {
- return Schema.builder().addField(fieldName, fieldType).build();
- }
-
- private String jsonObjectWith(String fieldName, String fieldValue) {
- return "{\n" + "\"" + fieldName + "\" : " + fieldValue + "\n" + "}";
- }
-
- private Matcher<UnsupportedRowJsonException> unsupportedWithMessage(String... message) {
- return allOf(
- Matchers.isA(UnsupportedRowJsonException.class),
- hasProperty("message", stringContainsInOrder(Arrays.asList(message))));
- }
-
- private ObjectMapper newObjectMapperWith(RowJsonDeserializer deserializer) {
- SimpleModule simpleModule = new SimpleModule("rowSerializationTesModule");
- simpleModule.addDeserializer(Row.class, deserializer);
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.registerModule(simpleModule);
- return objectMapper;
- }
-}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
new file mode 100644
index 0000000..f1f3fe9
--- /dev/null
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
@@ -0,0 +1,520 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.hamcrest.Matchers.allOf;
+import static org.hamcrest.Matchers.hasProperty;
+import static org.hamcrest.Matchers.stringContainsInOrder;
+import static org.junit.Assert.assertEquals;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.Arrays;
+import java.util.Collection;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.hamcrest.Matcher;
+import org.hamcrest.Matchers;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+/** Unit tests for {@link RowJsonDeserializer} and {@link RowJsonSerializer}. */
+@RunWith(Enclosed.class)
+public class RowJsonTest {
+ @RunWith(Parameterized.class)
+ public static class ValueTests {
+ @Parameter(0)
+ public String name;
+
+ @Parameter(1)
+ public Schema schema;
+
+ @Parameter(2)
+ public String serializedString;
+
+ @Parameter(3)
+ public Row row;
+
+ @Parameters(name = "{0}")
+ public static Collection<Object[]> data() {
+ return ImmutableList.of(
+ makeFlatRowTestCase(),
+ makeArrayFieldTestCase(),
+ makeArrayOfArraysTestCase(),
+ makeNestedRowTestCase(),
+ makeDoublyNestedRowTestCase(),
+ makeNullsTestCase());
+ }
+
+ private static Object[] makeFlatRowTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addByteField("f_byte")
+ .addInt16Field("f_int16")
+ .addInt32Field("f_int32")
+ .addInt64Field("f_int64")
+ .addFloatField("f_float")
+ .addDoubleField("f_double")
+ .addBooleanField("f_boolean")
+ .addStringField("f_string")
+ .addDecimalField("f_decimal")
+ .build();
+
+ String rowString =
+ "{\n"
+ + "\"f_byte\" : 12,\n"
+ + "\"f_int16\" : 22,\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_int64\" : 42,\n"
+ + "\"f_float\" : 1.02E5,\n"
+ + "\"f_double\" : 62.2,\n"
+ + "\"f_boolean\" : true,\n"
+ + "\"f_string\" : \"hello\",\n"
+ + "\"f_decimal\" : 123.12\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(
+ (byte) 12,
+ (short) 22,
+ 32,
+ (long) 42,
+ 1.02E5f,
+ 62.2d,
+ true,
+ "hello",
+ new BigDecimal("123.12"))
+ .build();
+
+ return new Object[] {"Flat row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeArrayFieldTestCase() {
+
+ Schema schema =
+ Schema.builder()
+ .addInt32Field("f_int32")
+ .addArrayField("f_intArray", FieldType.INT32)
+ .build();
+
+ String rowString =
+ "{\n" + "\"f_int32\" : 32,\n" + "\"f_intArray\" : [ 1, 2, 3, 4, 5]\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues(32, Arrays.asList(1, 2, 3, 4, 5)).build();
+
+ return new Object[] {"Array field", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeArrayOfArraysTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
+ .build();
+
+ String rowString = "{\n" + "\"f_arrayOfIntArrays\" : [ [1, 2], [3, 4], [5]]\n" + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addArray(Arrays.asList(1, 2), Arrays.asList(3, 4), Arrays.asList(5))
+ .build();
+
+ return new Object[] {"Array of arrays", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeNestedRowTestCase() {
+ Schema nestedRowSchema =
+ Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_row\" : {\n"
+ + " \"f_nestedInt32\" : 54,\n"
+ + " \"f_nestedString\" : \"foo\"\n"
+ + " }\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(32, Row.withSchema(nestedRowSchema).addValues(54, "foo").build())
+ .build();
+
+ return new Object[] {"Nested row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeDoublyNestedRowTestCase() {
+
+ Schema doubleNestedRowSchema =
+ Schema.builder().addStringField("f_doubleNestedString").build();
+
+ Schema nestedRowSchema =
+ Schema.builder().addRowField("f_nestedRow", doubleNestedRowSchema).build();
+
+ Schema schema = Schema.builder().addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_row\" : {\n"
+ + " \"f_nestedRow\" : {\n"
+ + " \"f_doubleNestedString\":\"foo\"\n"
+ + " }\n"
+ + " }\n"
+ + "}";
+
+ Row expectedRow =
+ Row.withSchema(schema)
+ .addValues(
+ Row.withSchema(nestedRowSchema)
+ .addValues(Row.withSchema(doubleNestedRowSchema).addValues("foo").build())
+ .build())
+ .build();
+
+ return new Object[] {"Doubly nested row", schema, rowString, expectedRow};
+ }
+
+ private static Object[] makeNullsTestCase() {
+ Schema schema =
+ Schema.builder()
+ .addByteField("f_byte")
+ .addNullableField("f_string", FieldType.STRING)
+ .build();
+
+ String rowString = "{\n" + "\"f_byte\" : 12,\n" + "\"f_string\" : null\n" + "}";
+
+ Row expectedRow = Row.withSchema(schema).addValues((byte) 12, null).build();
+
+ return new Object[] {"Nulls", schema, rowString, expectedRow};
+ }
+
+ @Test
+ public void testDeserialize() throws IOException {
+ Row parsedRow = newObjectMapperFor(schema).readValue(serializedString, Row.class);
+
+ assertEquals(row, parsedRow);
+ }
+
+ // This serves to validate RowJsonSerializer. We don't have tests to check that the output
+ // string matches exactly what we expect, just that the string we produced can be deserialized
+ // again into an equal row.
+ @Test
+ public void testRoundTrip() throws IOException {
+ ObjectMapper objectMapper = newObjectMapperFor(schema);
+ Row parsedRow = objectMapper.readValue(objectMapper.writeValueAsString(row), Row.class);
+
+ assertEquals(row, parsedRow);
+ }
+ }
+
+ @RunWith(JUnit4.class)
+ public static class DeserializerTests {
+ private static final Boolean BOOLEAN_TRUE_VALUE = true;
+ private static final String BOOLEAN_TRUE_STRING = "true";
+ private static final Byte BYTE_VALUE = 126;
+ private static final String BYTE_STRING = "126";
+ private static final Short SHORT_VALUE = 32766;
+ private static final String SHORT_STRING = "32766";
+ private static final Integer INT_VALUE = 2147483646;
+ private static final String INT_STRING = "2147483646";
+ private static final Long LONG_VALUE = 9223372036854775806L;
+ private static final String LONG_STRING = "9223372036854775806";
+ private static final Float FLOAT_VALUE = 1.02e5f;
+ private static final String FLOAT_STRING = "1.02e5";
+ private static final Double DOUBLE_VALUE = 1.02d;
+ private static final String DOUBLE_STRING = "1.02";
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testThrowsForMismatchedArrayField() throws Exception {
+
+ Schema schema =
+ Schema.builder()
+ .addArrayField("f_arrayOfIntArrays", FieldType.array(FieldType.INT32))
+ .build();
+
+ String rowString =
+ "{\n"
+ + "\"f_arrayOfIntArrays\" : { }\n" // expect array, get object
+ + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("Expected JSON array");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testThrowsForMismatchedRowField() throws Exception {
+ Schema nestedRowSchema =
+ Schema.builder().addInt32Field("f_nestedInt32").addStringField("f_nestedString").build();
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_int32").addRowField("f_row", nestedRowSchema).build();
+
+ String rowString =
+ "{\n"
+ + "\"f_int32\" : 32,\n"
+ + "\"f_row\" : []\n" // expect object, get array
+ + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("Expected JSON object");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testThrowsForMissingNotNullableField() throws Exception {
+ Schema schema = Schema.builder().addByteField("f_byte").addStringField("f_string").build();
+
+ String rowString = "{\n" + "\"f_byte\" : 12\n" + "}";
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("'f_string' is not present");
+
+ newObjectMapperFor(schema).readValue(rowString, Row.class);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedType() throws Exception {
+ Schema schema = Schema.builder().addDateTimeField("f_dateTime").build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedArrayElementType() throws Exception {
+ Schema schema = Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testDeserializerThrowsForUnsupportedNestedFieldType() throws Exception {
+ Schema nestedSchema =
+ Schema.builder().addArrayField("f_dateTimeArray", FieldType.DATETIME).build();
+
+ Schema schema = Schema.builder().addRowField("f_nestedRow", nestedSchema).build();
+
+ thrown.expect(UnsupportedRowJsonException.class);
+ thrown.expectMessage("DATETIME is not supported");
+
+ RowJsonDeserializer.forSchema(schema);
+ }
+
+ @Test
+ public void testSupportedBooleanConversions() throws Exception {
+ testSupportedConversion(FieldType.BOOLEAN, BOOLEAN_TRUE_STRING, BOOLEAN_TRUE_VALUE);
+ }
+
+ @Test
+ public void testSupportedStringConversions() throws Exception {
+ testSupportedConversion(FieldType.STRING, quoted(FLOAT_STRING), FLOAT_STRING);
+ }
+
+ @Test
+ public void testSupportedByteConversions() throws Exception {
+ testSupportedConversion(FieldType.BYTE, BYTE_STRING, BYTE_VALUE);
+ }
+
+ @Test
+ public void testSupportedShortConversions() throws Exception {
+ testSupportedConversion(FieldType.INT16, BYTE_STRING, (short) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT16, SHORT_STRING, SHORT_VALUE);
+ }
+
+ @Test
+ public void testSupportedIntConversions() throws Exception {
+ testSupportedConversion(FieldType.INT32, BYTE_STRING, (int) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT32, SHORT_STRING, (int) SHORT_VALUE);
+ testSupportedConversion(FieldType.INT32, INT_STRING, INT_VALUE);
+ }
+
+ @Test
+ public void testSupportedLongConversions() throws Exception {
+ testSupportedConversion(FieldType.INT64, BYTE_STRING, (long) BYTE_VALUE);
+ testSupportedConversion(FieldType.INT64, SHORT_STRING, (long) SHORT_VALUE);
+ testSupportedConversion(FieldType.INT64, INT_STRING, (long) INT_VALUE);
+ testSupportedConversion(FieldType.INT64, LONG_STRING, LONG_VALUE);
+ }
+
+ @Test
+ public void testSupportedFloatConversions() throws Exception {
+ testSupportedConversion(FieldType.FLOAT, FLOAT_STRING, FLOAT_VALUE);
+ testSupportedConversion(FieldType.FLOAT, SHORT_STRING, (float) SHORT_VALUE);
+ }
+
+ @Test
+ public void testSupportedDoubleConversions() throws Exception {
+ testSupportedConversion(FieldType.DOUBLE, DOUBLE_STRING, DOUBLE_VALUE);
+ testSupportedConversion(FieldType.DOUBLE, FLOAT_STRING, (double) FLOAT_VALUE);
+ testSupportedConversion(FieldType.DOUBLE, INT_STRING, (double) INT_VALUE);
+ }
+
+ private void testSupportedConversion(
+ FieldType fieldType, String jsonFieldValue, Object expectedRowFieldValue) throws Exception {
+
+ String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
+ Schema schema = schemaWithField(fieldName, fieldType);
+ Row expectedRow = Row.withSchema(schema).addValues(expectedRowFieldValue).build();
+ ObjectMapper jsonParser = newObjectMapperFor(schema);
+
+ Row parsedRow = jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
+
+ assertEquals(expectedRow, parsedRow);
+ }
+
+ @Test
+ public void testUnsupportedBooleanConversions() throws Exception {
+ testUnsupportedConversion(FieldType.BOOLEAN, quoted(BOOLEAN_TRUE_STRING));
+ testUnsupportedConversion(FieldType.BOOLEAN, BYTE_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, SHORT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, INT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, LONG_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.BOOLEAN, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedStringConversions() throws Exception {
+ testUnsupportedConversion(FieldType.STRING, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.STRING, BYTE_STRING);
+ testUnsupportedConversion(FieldType.STRING, SHORT_STRING);
+ testUnsupportedConversion(FieldType.STRING, INT_STRING);
+ testUnsupportedConversion(FieldType.STRING, LONG_STRING);
+ testUnsupportedConversion(FieldType.STRING, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.STRING, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedByteConversions() throws Exception {
+ testUnsupportedConversion(FieldType.BYTE, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.BYTE, quoted(BYTE_STRING));
+ testUnsupportedConversion(FieldType.BYTE, SHORT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, INT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, LONG_STRING);
+ testUnsupportedConversion(FieldType.BYTE, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.BYTE, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedShortConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT16, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT16, quoted(SHORT_STRING));
+ testUnsupportedConversion(FieldType.INT16, INT_STRING);
+ testUnsupportedConversion(FieldType.INT16, LONG_STRING);
+ testUnsupportedConversion(FieldType.INT16, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT16, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedIntConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT32, quoted(INT_STRING));
+ testUnsupportedConversion(FieldType.INT32, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT32, LONG_STRING);
+ testUnsupportedConversion(FieldType.INT32, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT32, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedLongConversions() throws Exception {
+ testUnsupportedConversion(FieldType.INT64, quoted(LONG_STRING));
+ testUnsupportedConversion(FieldType.INT64, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.INT64, FLOAT_STRING);
+ testUnsupportedConversion(FieldType.INT64, DOUBLE_STRING);
+ }
+
+ @Test
+ public void testUnsupportedFloatConversions() throws Exception {
+ testUnsupportedConversion(FieldType.FLOAT, quoted(FLOAT_STRING));
+ testUnsupportedConversion(FieldType.FLOAT, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.FLOAT, DOUBLE_STRING);
+ testUnsupportedConversion(FieldType.FLOAT, INT_STRING); // too large to fit
+ }
+
+ @Test
+ public void testUnsupportedDoubleConversions() throws Exception {
+ testUnsupportedConversion(FieldType.DOUBLE, quoted(DOUBLE_STRING));
+ testUnsupportedConversion(FieldType.DOUBLE, BOOLEAN_TRUE_STRING);
+ testUnsupportedConversion(FieldType.DOUBLE, LONG_STRING); // too large to fit
+ }
+
+ private void testUnsupportedConversion(FieldType fieldType, String jsonFieldValue)
+ throws Exception {
+
+ String fieldName = "f_" + fieldType.getTypeName().name().toLowerCase();
+
+ Schema schema = schemaWithField(fieldName, fieldType);
+ ObjectMapper jsonParser = newObjectMapperFor(schema);
+
+ thrown.expectMessage(fieldName);
+ thrown.expectCause(unsupportedWithMessage(jsonFieldValue, "out of range"));
+
+ jsonParser.readValue(jsonObjectWith(fieldName, jsonFieldValue), Row.class);
+ }
+ }
+
+ private static String quoted(String string) {
+ return "\"" + string + "\"";
+ }
+
+ private static Schema schemaWithField(String fieldName, FieldType fieldType) {
+ return Schema.builder().addField(fieldName, fieldType).build();
+ }
+
+ private static String jsonObjectWith(String fieldName, String fieldValue) {
+ return "{\n" + "\"" + fieldName + "\" : " + fieldValue + "\n" + "}";
+ }
+
+ private static Matcher<UnsupportedRowJsonException> unsupportedWithMessage(String... message) {
+ return allOf(
+ Matchers.isA(UnsupportedRowJsonException.class),
+ hasProperty("message", stringContainsInOrder(Arrays.asList(message))));
+ }
+
+ private static ObjectMapper newObjectMapperFor(Schema schema) {
+ SimpleModule simpleModule = new SimpleModule("rowSerializationTesModule");
+ simpleModule.addSerializer(Row.class, RowJsonSerializer.forSchema(schema));
+ simpleModule.addDeserializer(Row.class, RowJsonDeserializer.forSchema(schema));
+ ObjectMapper objectMapper = new ObjectMapper();
+ objectMapper.registerModule(simpleModule);
+ return objectMapper;
+ }
+}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/SerializableUtilsTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/SerializableUtilsTest.java
index f69ef35..9f96bce 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/SerializableUtilsTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/SerializableUtilsTest.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.testing.InterceptingUrlClassLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/WindowedValueTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/WindowedValueTest.java
index 83f526a..206b548 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/WindowedValueTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/WindowedValueTest.java
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.junit.Assert;
import org.junit.Rule;
@@ -120,5 +120,25 @@
WindowedValue.of("foo", now, futureWindow, pane),
WindowedValue.of("foo", now, centerWindow, pane),
WindowedValue.of("foo", now, pastWindow, pane)));
+
+ assertThat(value.isSingleWindowedValue(), equalTo(false));
+ }
+
+ @Test
+ public void testSingleWindowedValueInGlobalWindow() {
+ WindowedValue<Integer> value =
+ WindowedValue.of(1, Instant.now(), GlobalWindow.INSTANCE, PaneInfo.NO_FIRING);
+ assertThat(value.isSingleWindowedValue(), equalTo(true));
+ assertThat(
+ ((WindowedValue.SingleWindowedValue) value).getWindow(), equalTo(GlobalWindow.INSTANCE));
+ }
+
+ @Test
+ public void testSingleWindowedValueInFixedWindow() {
+ Instant now = Instant.now();
+ BoundedWindow w = new IntervalWindow(now, now.plus(1));
+ WindowedValue<Integer> value = WindowedValue.of(1, now, w, PaneInfo.NO_FIRING);
+ assertThat(value.isSingleWindowedValue(), equalTo(true));
+ assertThat(((WindowedValue.SingleWindowedValue) value).getWindow(), equalTo(w));
}
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ZipFilesTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ZipFilesTest.java
index 2317a1c..f9e2e12 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ZipFilesTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ZipFilesTest.java
@@ -35,9 +35,9 @@
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CharSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CharSource;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -194,7 +194,7 @@
assertTrue(zipDir.mkdir());
ZipFiles.zipDirectory(tmpDir, zipFile);
File invalidDirectory = new File("/foo/bar");
- assertTrue(!invalidDirectory.exists());
+ assertFalse(invalidDirectory.exists());
try {
ZipFiles.unzipFile(zipFile, invalidDirectory);
fail("We expect the IllegalArgumentException, but it never occured");
@@ -280,7 +280,7 @@
// This is not generally safe as it does not handle symlinks, etc. However it is safe
// enough for these tests.
private static void removeRecursive(Path path) throws IOException {
- Iterable<File> files = Files.fileTreeTraverser().postOrderTraversal(path.toFile());
+ Iterable<File> files = Files.fileTraverser().depthFirstPostOrder(path.toFile());
for (File f : files) {
java.nio.file.Files.delete(f.toPath());
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
index 32568d7..822c392 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java
@@ -17,9 +17,12 @@
*/
package org.apache.beam.sdk.util.common;
+import static org.hamcrest.Matchers.contains;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
import com.fasterxml.jackson.annotation.JsonIgnore;
+import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.beam.sdk.options.Default;
@@ -170,4 +173,42 @@
thread.join();
assertEquals(cl, classLoader[0]);
}
+
+ /**
+ * Test service interface and implementations for loadServicesOrdered.
+ *
+ * <p>Note that rather than using AutoService to create resources, AlphaImpl and ZetaImpl are
+ * listed in reverse lexicographical order in
+ * sdks/java/core/src/test/resources/META-INF/services/org.apache.beam.sdk.util.common.ReflectHelpersTest$FakeService
+ * so that we can verify loadServicesOrdered properly re-orders them.
+ */
+ public interface FakeService {
+ String getName();
+ }
+
+ /** Alpha implemnetation of FakeService. Should be loaded first */
+ public static class AlphaImpl implements FakeService {
+ @Override
+ public String getName() {
+ return "Alpha";
+ }
+ }
+
+ /** Zeta implemnetation of FakeService. Should be loaded second */
+ public static class ZetaImpl implements FakeService {
+ @Override
+ public String getName() {
+ return "Zeta";
+ }
+ }
+
+ @Test
+ public void testLoadServicesOrderedReordersClassesByName() {
+ List<String> names = new ArrayList<>();
+ for (FakeService service : ReflectHelpers.loadServicesOrdered(FakeService.class)) {
+ names.add(service.getName());
+ }
+
+ assertThat(names, contains("Alpha", "Zeta"));
+ }
}
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/KVTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/KVTest.java
index 7311ea9..44c0824 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/KVTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/KVTest.java
@@ -23,7 +23,7 @@
import static org.junit.Assert.assertThat;
import java.util.Comparator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionListTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionListTest.java
index 694cea7..21b384b 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionListTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionListTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.io.GenerateSequence;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
index 26b2446..e302c51 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/PCollectionTupleTest.java
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PCollection.IsBounded;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
index 3f773c8..40e5ee8 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/RowTest.java
@@ -20,6 +20,7 @@
import static org.apache.beam.sdk.schemas.Schema.toSchema;
import static org.apache.beam.sdk.values.Row.toRow;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNull;
@@ -32,9 +33,9 @@
import java.util.stream.Stream;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.junit.Assert;
@@ -143,8 +144,8 @@
assertEquals("str", row.getString(7));
assertEquals(dateTime, row.getDateTime("f_datetime"));
assertEquals(dateTime, row.getDateTime(8));
- assertEquals(false, row.getBoolean("f_boolean"));
- assertEquals(false, row.getBoolean(9));
+ assertFalse(row.getBoolean("f_boolean"));
+ assertFalse(row.getBoolean(9));
}
@Test
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TupleTagTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TupleTagTest.java
index f53d022..4daac49 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TupleTagTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TupleTagTest.java
@@ -23,8 +23,8 @@
import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertThat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TypeDescriptorTest.java b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TypeDescriptorTest.java
index 020ded8..dbad49c 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TypeDescriptorTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/values/TypeDescriptorTest.java
@@ -23,7 +23,7 @@
import java.lang.reflect.TypeVariable;
import java.util.List;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.reflect.TypeToken;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.reflect.TypeToken;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/core/src/test/resources/META-INF/services/org.apache.beam.sdk.util.common.ReflectHelpersTest$FakeService b/sdks/java/core/src/test/resources/META-INF/services/org.apache.beam.sdk.util.common.ReflectHelpersTest$FakeService
new file mode 100644
index 0000000..ea9e9fc
--- /dev/null
+++ b/sdks/java/core/src/test/resources/META-INF/services/org.apache.beam.sdk.util.common.ReflectHelpersTest$FakeService
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Load in reverse lexicographical order, so we can verify they are re-ordered
+org.apache.beam.sdk.util.common.ReflectHelpersTest$ZetaImpl
+org.apache.beam.sdk.util.common.ReflectHelpersTest$AlphaImpl
diff --git a/sdks/java/extensions/euphoria/README.md b/sdks/java/extensions/euphoria/README.md
index e7b07cb..0d9254e 100644
--- a/sdks/java/extensions/euphoria/README.md
+++ b/sdks/java/extensions/euphoria/README.md
@@ -29,7 +29,7 @@
Euphoria is located in `dsl-euphoria` branch. To build `euphoria` subprojects use command:
```
-./gradlew :beam-sdks-java-extensions-euphoria:build
+./gradlew :sdks:java:extensions:euphoria:build
```
## Documentation
diff --git a/sdks/java/extensions/euphoria/build.gradle b/sdks/java/extensions/euphoria/build.gradle
index bd73791..93f6ebd 100644
--- a/sdks/java/extensions/euphoria/build.gradle
+++ b/sdks/java/extensions/euphoria/build.gradle
@@ -17,21 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.extensions.euphoria')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Euphoria Java 8 DSL"
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.guava
+ compile project(path: ":sdks:java:core", configuration: "shadow")
testCompile library.java.mockito_core
- testCompile project(path: ":sdks:java:extensions:kryo")
+ testCompile project(":sdks:java:extensions:kryo")
testCompile library.java.slf4j_api
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.mockito_core
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testRuntimeOnly project(':runners:direct-java')
+ testRuntimeOnly project(":runners:direct-java")
}
test {
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
index 01c17d5..a617aa4 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/CountByKey.java
@@ -265,7 +265,7 @@
.of(PCollectionLists.getOnlyElement(inputs))
.keyBy(getKeyExtractor(), getKeyType().orElse(null))
.valueBy(v -> 1L, TypeDescriptors.longs())
- .combineBy(Sums.ofLongs(), TypeDescriptors.longs())
+ .combineBy(Sums.ofLongs())
.applyIf(
getWindow().isPresent(),
builder -> {
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Distinct.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Distinct.java
index 62c8d88..dab5e91 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Distinct.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Distinct.java
@@ -45,7 +45,7 @@
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/FlatMap.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/FlatMap.java
index 3fa25d4..b71be1b 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/FlatMap.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/FlatMap.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKey.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKey.java
index 67769c9..5adcf69 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKey.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKey.java
@@ -27,10 +27,12 @@
import org.apache.beam.sdk.extensions.euphoria.core.annotation.operator.Recommended;
import org.apache.beam.sdk.extensions.euphoria.core.annotation.operator.StateComplexity;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.BinaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.CombinableBinaryFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.CombinableReduceFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.ReduceFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.ReduceFunctor;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.UnaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.VoidFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.io.Collector;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.Builders;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.OptionalMethodBuilder;
@@ -39,6 +41,7 @@
import org.apache.beam.sdk.extensions.euphoria.core.client.type.TypeAware;
import org.apache.beam.sdk.extensions.euphoria.core.client.type.TypeAwareness;
import org.apache.beam.sdk.extensions.euphoria.core.translate.OperatorTransform;
+import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.Trigger;
@@ -83,6 +86,7 @@
* @param <InputT> Type of input records
* @param <KeyT> Output type of #keyBy method
* @param <ValueT> Output type of #valueBy method
+ * @param <AccT> type of accumulator (if CombineFn used)
* @param <OutputT> Type of output value
*/
@Audience(Audience.Type.CLIENT)
@@ -93,10 +97,27 @@
+ "can be efficiently used in the executor-specific implementation",
state = StateComplexity.CONSTANT_IF_COMBINABLE,
repartitions = 1)
-public class ReduceByKey<InputT, KeyT, ValueT, OutputT>
+public class ReduceByKey<InputT, KeyT, ValueT, AccT, OutputT>
extends ShuffleOperator<InputT, KeyT, KV<KeyT, OutputT>> implements TypeAware.Value<ValueT> {
/**
+ * A syntactic sugar interface to enable #combineBy(Sums.ofLongs()) to use Combine.CombineFn style
+ * combine logic.
+ *
+ * @param <T> type paramter
+ */
+ public interface CombineFunctionWithIdentity<T> extends CombinableBinaryFunction<T> {
+
+ /** @return identity value with respect to the combining function. */
+ T identity();
+
+ /** @return type descriptor of the value */
+ default TypeDescriptor<T> valueDesc() {
+ return null;
+ }
+ }
+
+ /**
* Starts building a nameless {@link ReduceByKey} operator to process the given input dataset.
*
* @param <InputT> the type of elements of the input dataset
@@ -187,6 +208,9 @@
* function is combinable (associative and commutative) so it can be used to compute partial
* results before shuffle.
*
+ * <p>Note: this might be less efficient, so you should use #combineBy(ValueT identity,
+ * BinaryFunction reducer) whenever it is possible.
+ *
* @param reducer function that reduces all values into one output object
* @return next builder to complete the setup of the {@link ReduceByKey} operator
*/
@@ -198,6 +222,74 @@
CombinableReduceFunction<ValueT> reducer, TypeDescriptor<ValueT> outputType) {
return reduceBy(ReduceFunctor.of(reducer), outputType);
}
+
+ /**
+ * Syntactic sugar to enable #combineBy to take only single argument and be used in helpers like
+ * #combineBy(Sums.ofLongs()).
+ */
+ default WindowByBuilder<KeyT, ValueT> combineBy(CombineFunctionWithIdentity<ValueT> reduce) {
+ return combineBy(reduce.identity(), reduce, reduce.valueDesc());
+ }
+
+ /**
+ * Syntactic sugar to enable #combineBy to take only single argument and be used in helpers like
+ * #combineBy(Sums.ofLongs()).
+ *
+ * @deprecated Replaced by @{link #combineBy(CombineFunctionWithIdentity)}.
+ */
+ @Deprecated
+ default WindowByBuilder<KeyT, ValueT> combineBy(
+ CombineFunctionWithIdentity<ValueT> reduce, TypeDescriptor<ValueT> ignored) {
+ return combineBy(reduce);
+ }
+
+ /**
+ * Define a function that reduces all values related to one key into one result object.
+ *
+ * @param identity zero (identity) element, must be {@link java.io.Serializable}.
+ * @param reducer function combining two values into result
+ * @return next builder to complete the setup of the {@link ReduceByKey} operator
+ */
+ default WindowByBuilder<KeyT, ValueT> combineBy(
+ ValueT identity, CombinableBinaryFunction<ValueT> reducer) {
+ return combineBy(identity, reducer, null);
+ }
+
+ @SuppressWarnings("unchecked")
+ default WindowByBuilder<KeyT, ValueT> combineBy(
+ ValueT identity,
+ CombinableBinaryFunction<ValueT> reducer,
+ @Nullable TypeDescriptor<ValueT> valueType) {
+ return (WindowByBuilder<KeyT, ValueT>)
+ this.<ValueT, ValueT>combineBy(
+ () -> identity, (BinaryFunction) reducer, reducer, e -> e, valueType, valueType);
+ }
+
+ /**
+ * Combine with full semantics defined by {@link Combine.CombineFn}.
+ *
+ * @param <AccT> type parameter of accumulator
+ * @param accumulatorFactory factory of accumulator
+ * @param accumulate accumulation function
+ * @param mergeAccumulators merging function
+ * @param outputFn output function
+ * @return next builder to complete the setup of the {@link ReduceByKey} operator
+ */
+ default <AccT> WindowByBuilder<KeyT, ValueT> combineBy(
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, ValueT> outputFn) {
+ return combineBy(accumulatorFactory, accumulate, mergeAccumulators, outputFn, null, null);
+ }
+
+ <AccT, OutputT> WindowByBuilder<KeyT, OutputT> combineBy(
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, OutputT> outputFn,
+ @Nullable TypeDescriptor<AccT> accumulatorDescriptor,
+ @Nullable TypeDescriptor<OutputT> outputDescriptor);
}
/** Builder for 'valueBy' / 'reduceBy' step. */
@@ -301,9 +393,10 @@
* @param <InputT> type of input
* @param <KeyT> type of key
* @param <ValueT> type of value
- * @param <OutputT> type ouf output
+ * @param <AccT> type of accumulator (if using CombineFn)
+ * @param <OutputT> type of output
*/
- static class Builder<InputT, KeyT, ValueT, OutputT>
+ static class Builder<InputT, KeyT, ValueT, AccT, OutputT>
implements OfBuilder,
KeyByBuilder<InputT>,
ValueByReduceByBuilder<InputT, KeyT, ValueT>,
@@ -323,19 +416,78 @@
@Nullable private TypeDescriptor<KeyT> keyType;
@Nullable private UnaryFunction<InputT, ValueT> valueExtractor;
@Nullable private TypeDescriptor<ValueT> valueType;
- private ReduceFunctor<ValueT, OutputT> reducer;
@Nullable private TypeDescriptor<OutputT> outputType;
@Nullable private BinaryFunction<ValueT, ValueT, Integer> valueComparator;
+ // following are defined for RBK using ReduceFunctor
+ @Nullable private final ReduceFunctor<ValueT, OutputT> reducer;
+
+ // following are defined when combineFnStyle == true
+ @Nullable private final VoidFunction<AccT> accumulatorFactory;
+ @Nullable private final BinaryFunction<AccT, ValueT, AccT> accumulate;
+ @Nullable private final CombinableBinaryFunction<AccT> mergeAccumulators;
+ @Nullable private final UnaryFunction<AccT, OutputT> outputFn;
+ @Nullable private final TypeDescriptor<AccT> accumulatorTypeDescriptor;
+
Builder(@Nullable String name) {
this.name = name;
+ this.reducer = null;
+ this.accumulatorFactory = null;
+ this.accumulate = null;
+ this.mergeAccumulators = null;
+ this.outputFn = null;
+ this.accumulatorTypeDescriptor = null;
+ }
+
+ // constructor for combine style
+ private Builder(
+ Builder parent,
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, OutputT> outputFn,
+ @Nullable TypeDescriptor<AccT> accumulatorTypeDescriptor) {
+ this.name = parent.name;
+ this.input = parent.input;
+ this.keyExtractor = parent.keyExtractor;
+ this.keyType = parent.keyType;
+ this.valueExtractor = parent.valueExtractor;
+ this.valueType = parent.valueType;
+ this.outputType = parent.outputType;
+ this.valueComparator = parent.valueComparator;
+
+ this.accumulatorFactory = requireNonNull(accumulatorFactory);
+ this.accumulate = requireNonNull(accumulate);
+ this.mergeAccumulators = requireNonNull(mergeAccumulators);
+ this.outputFn = requireNonNull(outputFn);
+ this.accumulatorTypeDescriptor = accumulatorTypeDescriptor;
+ this.reducer = null;
+ }
+
+ // constructor for ReduceFunctor style
+ private Builder(Builder parent, ReduceFunctor<ValueT, OutputT> reducer) {
+ this.name = parent.name;
+ this.input = parent.input;
+ this.keyExtractor = parent.keyExtractor;
+ this.keyType = parent.keyType;
+ this.valueExtractor = parent.valueExtractor;
+ this.valueType = parent.valueType;
+ this.outputType = parent.outputType;
+ this.valueComparator = parent.valueComparator;
+
+ this.accumulatorFactory = null;
+ this.accumulate = null;
+ this.mergeAccumulators = null;
+ this.outputFn = null;
+ this.accumulatorTypeDescriptor = null;
+ this.reducer = requireNonNull(reducer);
}
@Override
@SuppressWarnings("unchecked")
public <T> KeyByBuilder<T> of(PCollection<T> input) {
@SuppressWarnings("unchecked")
- final Builder<T, ?, ?, ?> cast = (Builder) this;
+ final Builder<T, ?, ?, ?, ?> cast = (Builder) this;
cast.input = input;
return cast;
}
@@ -344,7 +496,7 @@
public <T> ValueByReduceByBuilder<InputT, T, InputT> keyBy(
UnaryFunction<InputT, T> keyExtractor, @Nullable TypeDescriptor<T> keyType) {
@SuppressWarnings("unchecked")
- final Builder<InputT, T, InputT, ?> cast = (Builder) this;
+ final Builder<InputT, T, InputT, ?, ?> cast = (Builder) this;
cast.keyExtractor = requireNonNull(keyExtractor);
cast.keyType = keyType;
return cast;
@@ -354,28 +506,43 @@
public <T> ReduceByBuilder<KeyT, T> valueBy(
UnaryFunction<InputT, T> valueExtractor, @Nullable TypeDescriptor<T> valueType) {
@SuppressWarnings("unchecked")
- final Builder<InputT, KeyT, T, ?> cast = (Builder) this;
+ final Builder<InputT, KeyT, T, ?, ?> cast = (Builder) this;
cast.valueExtractor = requireNonNull(valueExtractor);
cast.valueType = valueType;
return cast;
}
@Override
- @SuppressWarnings("unchecked")
public <T> WithSortedValuesBuilder<KeyT, ValueT, T> reduceBy(
ReduceFunctor<ValueT, T> reducer, @Nullable TypeDescriptor<T> outputType) {
- if (valueExtractor == null) {
- // if the valueExtractor was not set in 'valueBy' step, we use untouched input element
- valueExtractor = (UnaryFunction) UnaryFunction.identity();
- }
@SuppressWarnings("unchecked")
- final Builder<InputT, KeyT, ValueT, T> cast = (Builder) this;
- cast.reducer = requireNonNull(reducer);
+ final Builder<InputT, KeyT, ValueT, ?, T> cast = new Builder(this, reducer);
cast.outputType = outputType;
return cast;
}
@Override
+ public <NewAccT, T> WindowByBuilder<KeyT, T> combineBy(
+ VoidFunction<NewAccT> accumulatorFactory,
+ BinaryFunction<NewAccT, ValueT, NewAccT> accumulate,
+ CombinableBinaryFunction<NewAccT> mergeAccumulators,
+ UnaryFunction<NewAccT, T> outputFn,
+ TypeDescriptor<NewAccT> accumulatorDescriptor,
+ TypeDescriptor<T> outputDescriptor) {
+ Builder<InputT, KeyT, ValueT, NewAccT, T> ret =
+ new Builder<>(
+ this,
+ accumulatorFactory,
+ accumulate,
+ mergeAccumulators,
+ outputFn,
+ accumulatorDescriptor);
+ ret.valueType = valueType;
+ ret.outputType = outputDescriptor;
+ return ret;
+ }
+
+ @Override
public WindowByBuilder<KeyT, OutputT> withSortedValues(
BinaryFunction<ValueT, ValueT, Integer> valueComparator) {
this.valueComparator = requireNonNull(valueComparator);
@@ -445,23 +612,58 @@
new OutputValues<>(name, outputType, createOperator()), PCollectionList.of(input));
}
- private ReduceByKey<InputT, KeyT, ValueT, OutputT> createOperator() {
+ private ReduceByKey<InputT, KeyT, ValueT, AccT, OutputT> createOperator() {
+ if (valueExtractor == null) {
+ valueExtractor = identity();
+ }
+ if (reducer != null) {
+ return new ReduceByKey<>(
+ name,
+ keyExtractor,
+ keyType,
+ valueExtractor,
+ valueType,
+ reducer,
+ valueComparator,
+ windowBuilder.getWindow().orElse(null),
+ TypeDescriptors.kvs(
+ TypeAwareness.orObjects(Optional.ofNullable(keyType)),
+ TypeAwareness.orObjects(Optional.ofNullable(outputType))));
+ }
return new ReduceByKey<>(
name,
keyExtractor,
keyType,
valueExtractor,
valueType,
- reducer,
+ accumulatorFactory,
+ accumulate,
+ mergeAccumulators,
+ outputFn,
+ accumulatorTypeDescriptor,
valueComparator,
windowBuilder.getWindow().orElse(null),
TypeDescriptors.kvs(
TypeAwareness.orObjects(Optional.ofNullable(keyType)),
TypeAwareness.orObjects(Optional.ofNullable(outputType))));
}
+
+ @SuppressWarnings("unchecked")
+ private UnaryFunction<InputT, ValueT> identity() {
+ return (UnaryFunction) UnaryFunction.identity();
+ }
}
- private final ReduceFunctor<ValueT, OutputT> reducer;
+ // ReduceFunctor variant
+ private final @Nullable ReduceFunctor<ValueT, OutputT> reducer;
+
+ // CombineFn variant
+ private final @Nullable VoidFunction<AccT> accumulatorFactory;
+ private final @Nullable BinaryFunction<AccT, ValueT, AccT> accumulate;
+ private final @Nullable CombinableBinaryFunction<AccT> mergeAccumulators;
+ private final @Nullable UnaryFunction<AccT, OutputT> outputFn;
+ private final @Nullable TypeDescriptor<AccT> accumulatorType;
+
private final UnaryFunction<InputT, ValueT> valueExtractor;
@Nullable private final BinaryFunction<ValueT, ValueT, Integer> valueComparator;
@Nullable private final TypeDescriptor<ValueT> valueType;
@@ -475,20 +677,81 @@
ReduceFunctor<ValueT, OutputT> reducer,
@Nullable BinaryFunction<ValueT, ValueT, Integer> valueComparator,
@Nullable Window<InputT> window,
- TypeDescriptor<KV<KeyT, OutputT>> outputType) {
+ @Nullable TypeDescriptor<KV<KeyT, OutputT>> outputType) {
super(name, outputType, keyExtractor, keyType, window);
- this.reducer = reducer;
- this.valueExtractor = valueExtractor;
+ this.reducer = requireNonNull(reducer);
+ this.valueExtractor = requireNonNull(valueExtractor);
this.valueType = valueType;
this.valueComparator = valueComparator;
+
+ this.accumulatorFactory = null;
+ this.accumulate = null;
+ this.mergeAccumulators = null;
+ this.outputFn = null;
+ this.accumulatorType = null;
+ }
+
+ private ReduceByKey(
+ @Nullable String name,
+ UnaryFunction<InputT, KeyT> keyExtractor,
+ @Nullable TypeDescriptor<KeyT> keyType,
+ UnaryFunction<InputT, ValueT> valueExtractor,
+ @Nullable TypeDescriptor<ValueT> valueType,
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, OutputT> outputFn,
+ TypeDescriptor<AccT> accumulatorType,
+ @Nullable BinaryFunction<ValueT, ValueT, Integer> valueComparator,
+ @Nullable Window<InputT> window,
+ @Nullable TypeDescriptor<KV<KeyT, OutputT>> outputType) {
+ super(name, outputType, keyExtractor, keyType, window);
+ this.reducer = null;
+ this.valueExtractor = requireNonNull(valueExtractor);
+ this.valueType = valueType;
+ this.valueComparator = valueComparator;
+
+ this.accumulatorFactory = requireNonNull(accumulatorFactory);
+ this.accumulate = requireNonNull(accumulate);
+ this.mergeAccumulators = requireNonNull(mergeAccumulators);
+ this.outputFn = requireNonNull(outputFn);
+ this.accumulatorType = accumulatorType;
+ }
+
+ public boolean isCombineFnStyle() {
+ return reducer == null;
}
public ReduceFunctor<ValueT, OutputT> getReducer() {
- return reducer;
+ return requireNonNull(reducer, "Don't call #getReducer when #isCombinableFnStyle() == true");
+ }
+
+ public VoidFunction<AccT> getAccumulatorFactory() {
+ return requireNonNull(
+ accumulatorFactory,
+ "Don't vall #getAccumulatorFactory when #isCombinableFnStyle() == false");
+ }
+
+ public BinaryFunction<AccT, ValueT, AccT> getAccumulate() {
+ return requireNonNull(
+ accumulate, "Don't vall #getAccumulate when #isCombinableFnStyle() == false");
+ }
+
+ public CombinableBinaryFunction<AccT> getMergeAccumulators() {
+ return requireNonNull(
+ mergeAccumulators, "Don't vall #getMergeAccumulators when #isCombinableFnStyle() == false");
+ }
+
+ public UnaryFunction<AccT, OutputT> getOutputFn() {
+ return requireNonNull(outputFn, "Don't vall #getOutputFn when #isCombinableFnStyle() == false");
+ }
+
+ public TypeDescriptor<AccT> getAccumulatorType() {
+ return accumulatorType;
}
public boolean isCombinable() {
- return reducer.isCombinable();
+ return isCombineFnStyle() || reducer.isCombinable();
}
public UnaryFunction<InputT, ValueT> getValueExtractor() {
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceWindow.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceWindow.java
index 4c26518..8a8cad3 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceWindow.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceWindow.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.euphoria.core.client.operator;
import static java.util.Objects.requireNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Optional;
import java.util.stream.Stream;
@@ -27,10 +27,12 @@
import org.apache.beam.sdk.extensions.euphoria.core.annotation.operator.Derived;
import org.apache.beam.sdk.extensions.euphoria.core.annotation.operator.StateComplexity;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.BinaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.CombinableBinaryFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.CombinableReduceFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.ReduceFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.ReduceFunctor;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.UnaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.VoidFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.io.Collector;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.Builders;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.OptionalMethodBuilder;
@@ -76,7 +78,8 @@
*/
@Audience(Audience.Type.CLIENT)
@Derived(state = StateComplexity.CONSTANT_IF_COMBINABLE, repartitions = 1)
-public class ReduceWindow<InputT, ValueT, OutputT> extends ShuffleOperator<InputT, Byte, OutputT>
+public class ReduceWindow<InputT, ValueT, AccT, OutputT>
+ extends ShuffleOperator<InputT, Byte, OutputT>
implements TypeAware.Value<ValueT>, CompositeOperator<InputT, OutputT> {
private static final Byte B_ZERO = (byte) 0;
@@ -169,6 +172,75 @@
CombinableReduceFunction<ValueT> reducer, TypeDescriptor<ValueT> outputType) {
return reduceBy(ReduceFunctor.of(reducer), outputType);
}
+
+ /**
+ * Syntactic sugar to enable #combineBy to take only single argument and be used in helpers like
+ * #combineBy(Sums.ofLongs()).
+ */
+ default WindowByBuilder<ValueT> combineBy(
+ ReduceByKey.CombineFunctionWithIdentity<ValueT> reduce) {
+ return combineBy(reduce.identity(), reduce, reduce.valueDesc());
+ }
+
+ /**
+ * Syntactic sugar to enable #combineBy to take only single argument and be used in helpers like
+ * #combineBy(Sums.ofLongs()).
+ *
+ * @deprecated Replaced by @{link #combineBy(ReduceByKey.CombineFunctionWithIdentity)}.
+ */
+ @Deprecated
+ default WindowByBuilder<ValueT> combineBy(
+ ReduceByKey.CombineFunctionWithIdentity<ValueT> reduce, TypeDescriptor<ValueT> ignored) {
+ return combineBy(reduce.identity(), reduce, reduce.valueDesc());
+ }
+
+ /**
+ * Define a function that reduces all values related to one key into one result object.
+ *
+ * @param identity zero (identity) element, must be {@link java.io.Serializable}.
+ * @param reducer function combining two values into result
+ * @return next builder to complete the setup of the {@link ReduceByKey} operator
+ */
+ default WindowByBuilder<ValueT> combineBy(
+ ValueT identity, CombinableBinaryFunction<ValueT> reducer) {
+ return combineBy(identity, reducer, null);
+ }
+
+ @SuppressWarnings("unchecked")
+ default WindowByBuilder<ValueT> combineBy(
+ ValueT identity,
+ CombinableBinaryFunction<ValueT> reducer,
+ @Nullable TypeDescriptor<ValueT> valueType) {
+ return (WindowByBuilder<ValueT>)
+ this.<ValueT, ValueT>combineBy(
+ () -> identity, (BinaryFunction) reducer, reducer, e -> e, valueType, valueType);
+ }
+
+ /**
+ * Combine with full semantics defined by {@link Combine.CombineFn}.
+ *
+ * @param <AccT> type parameter of accumulator
+ * @param accumulatorFactory factory of accumulator
+ * @param accumulate accumulation function
+ * @param mergeAccumulators merging function
+ * @param outputFn output function
+ * @return next builder to complete the setup of the {@link ReduceByKey} operator
+ */
+ default <AccT> WindowByBuilder<ValueT> combineBy(
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, ValueT> outputFn) {
+ return combineBy(accumulatorFactory, accumulate, mergeAccumulators, outputFn, null, null);
+ }
+
+ <AccT, OutputT> WindowByBuilder<OutputT> combineBy(
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, OutputT> outputFn,
+ @Nullable TypeDescriptor<AccT> accumulatorDescriptor,
+ @Nullable TypeDescriptor<OutputT> outptuDescriptor);
}
/** Builder for 'valueBy' / 'reduceBy' step. */
@@ -249,7 +321,7 @@
* @param <ValueT> type of value
* @param <OutputT> type ouf output
*/
- private static class Builder<InputT, ValueT, OutputT>
+ private static class Builder<InputT, ValueT, AccT, OutputT>
implements OfBuilder,
ValueByReduceByBuilder<InputT, ValueT>,
WithSortedValuesBuilder<ValueT, OutputT>,
@@ -262,21 +334,70 @@
private final WindowBuilder<InputT> windowBuilder = new WindowBuilder<>();
@Nullable private final String name;
+ @Nullable private final ReduceFunctor<ValueT, OutputT> reducer;
+ @Nullable private final VoidFunction<AccT> accumulatorFactory;
+ @Nullable private final BinaryFunction<AccT, ValueT, AccT> accumulate;
+ @Nullable private final CombinableBinaryFunction<AccT> mergeAccumulators;
+ @Nullable private final UnaryFunction<AccT, OutputT> outputFn;
+ @Nullable private final TypeDescriptor<AccT> accumulatorType;
+
private PCollection<InputT> input;
@Nullable private UnaryFunction<InputT, ValueT> valueExtractor;
@Nullable private TypeDescriptor<ValueT> valueType;
- private ReduceFunctor<ValueT, OutputT> reducer;
+
@Nullable private TypeDescriptor<OutputT> outputType;
@Nullable private BinaryFunction<ValueT, ValueT, Integer> valueComparator;
Builder(@Nullable String name) {
this.name = name;
+ this.reducer = null;
+ this.accumulatorFactory = null;
+ this.accumulate = null;
+ this.mergeAccumulators = null;
+ this.outputFn = null;
+ this.accumulatorType = null;
+ }
+
+ private Builder(Builder parent, ReduceFunctor<ValueT, OutputT> reducer) {
+ this.name = parent.name;
+ this.reducer = requireNonNull(reducer);
+ this.accumulatorFactory = null;
+ this.accumulate = null;
+ this.mergeAccumulators = null;
+ this.outputFn = null;
+ this.accumulatorType = null;
+ this.input = parent.input;
+ this.valueExtractor = parent.valueExtractor;
+ this.valueType = parent.valueType;
+ this.outputType = parent.outputType;
+ this.valueComparator = parent.valueComparator;
+ }
+
+ private Builder(
+ Builder parent,
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, OutputT> outputFn,
+ TypeDescriptor<AccT> accumulatorType) {
+ this.name = parent.name;
+ this.reducer = null;
+ this.accumulatorFactory = requireNonNull(accumulatorFactory);
+ this.accumulate = requireNonNull(accumulate);
+ this.mergeAccumulators = requireNonNull(mergeAccumulators);
+ this.outputFn = requireNonNull(outputFn);
+ this.accumulatorType = accumulatorType;
+ this.input = parent.input;
+ this.valueExtractor = parent.valueExtractor;
+ this.valueType = parent.valueType;
+ this.outputType = parent.outputType;
+ this.valueComparator = parent.valueComparator;
}
@Override
public <T> ValueByReduceByBuilder<T, T> of(PCollection<T> input) {
@SuppressWarnings("unchecked")
- final Builder<T, T, ?> cast = (Builder) this;
+ final Builder<T, T, ?, ?> cast = (Builder) this;
cast.input = requireNonNull(input);
return cast;
}
@@ -285,28 +406,43 @@
public <T> ReduceByBuilder<T> valueBy(
UnaryFunction<InputT, T> valueExtractor, @Nullable TypeDescriptor<T> valueType) {
@SuppressWarnings("unchecked")
- final Builder<InputT, T, ?> cast = (Builder) this;
+ final Builder<InputT, T, ?, ?> cast = (Builder) this;
cast.valueExtractor = requireNonNull(valueExtractor);
cast.valueType = valueType;
return cast;
}
@Override
- @SuppressWarnings("unchecked")
public <T> WithSortedValuesBuilder<ValueT, T> reduceBy(
ReduceFunctor<ValueT, T> reducer, @Nullable TypeDescriptor<T> outputType) {
- if (valueExtractor == null) {
- // if the valueExtractor was not set in 'valueBy' step, we use untouched input element
- valueExtractor = (UnaryFunction) UnaryFunction.identity();
- }
@SuppressWarnings("unchecked")
- final Builder<InputT, ValueT, T> cast = (Builder) this;
- cast.reducer = requireNonNull(reducer);
+ final Builder<InputT, ValueT, ?, T> cast = new Builder<>(this, reducer);
cast.outputType = outputType;
return cast;
}
@Override
+ public <NewAccT, T> WindowByBuilder<T> combineBy(
+ VoidFunction<NewAccT> accumulatorFactory,
+ BinaryFunction<NewAccT, ValueT, NewAccT> accumulate,
+ CombinableBinaryFunction<NewAccT> mergeAccumulators,
+ UnaryFunction<NewAccT, T> outputFn,
+ @Nullable TypeDescriptor<NewAccT> accumulatorDescriptor,
+ @Nullable TypeDescriptor<T> outputDescriptor) {
+ Builder<InputT, ValueT, NewAccT, T> ret =
+ new Builder<>(
+ this,
+ accumulatorFactory,
+ accumulate,
+ mergeAccumulators,
+ outputFn,
+ accumulatorDescriptor);
+ ret.valueType = valueType;
+ ret.outputType = outputDescriptor;
+ return ret;
+ }
+
+ @Override
public WindowByBuilder<OutputT> withSortedValues(
BinaryFunction<ValueT, ValueT, Integer> valueComparator) {
this.valueComparator = requireNonNull(valueComparator);
@@ -361,20 +497,53 @@
@Override
public PCollection<OutputT> output(OutputHint... outputHints) {
- final ReduceWindow<InputT, ValueT, OutputT> rw =
- new ReduceWindow<>(
- name,
- valueExtractor,
- valueType,
- reducer,
- valueComparator,
- windowBuilder.getWindow().orElse(null),
- outputType);
+ final ReduceWindow<InputT, ValueT, ?, OutputT> rw;
+ if (valueExtractor == null) {
+ valueExtractor = identity();
+ }
+ if (reducer != null) {
+ rw =
+ new ReduceWindow<>(
+ name,
+ valueExtractor,
+ valueType,
+ reducer,
+ valueComparator,
+ windowBuilder.getWindow().orElse(null),
+ outputType);
+ } else {
+ rw =
+ new ReduceWindow<>(
+ name,
+ valueExtractor,
+ valueType,
+ accumulatorFactory,
+ accumulate,
+ mergeAccumulators,
+ outputFn,
+ accumulatorType,
+ valueComparator,
+ windowBuilder.getWindow().orElse(null),
+ outputType);
+ }
return OperatorTransform.apply(rw, PCollectionList.of(input));
}
+
+ @SuppressWarnings("unchecked")
+ private UnaryFunction<InputT, ValueT> identity() {
+ return (UnaryFunction) UnaryFunction.identity();
+ }
}
- private final ReduceFunctor<ValueT, OutputT> reducer;
+ // ReduceFunctor style
+ private final @Nullable ReduceFunctor<ValueT, OutputT> reducer;
+ // CombineFn style
+ private final @Nullable VoidFunction<AccT> accumulatorFactory;
+ private final @Nullable BinaryFunction<AccT, ValueT, AccT> accumulate;
+ private final @Nullable CombinableBinaryFunction<AccT> mergeAccumulators;
+ private final @Nullable UnaryFunction<AccT, OutputT> outputFn;
+ private final @Nullable TypeDescriptor<AccT> accumulatorType;
+
private final UnaryFunction<InputT, ValueT> valueExtractor;
@Nullable private final BinaryFunction<ValueT, ValueT, Integer> valueComparator;
@Nullable private final TypeDescriptor<ValueT> valueType;
@@ -389,18 +558,52 @@
TypeDescriptor<OutputT> outputType) {
super(name, outputType, e -> B_ZERO, TypeDescriptors.bytes(), window);
- this.reducer = reducer;
+ this.reducer = requireNonNull(reducer);
this.valueExtractor = valueExtractor;
this.valueType = valueType;
this.valueComparator = valueComparator;
+
+ this.accumulatorFactory = null;
+ this.accumulate = null;
+ this.mergeAccumulators = null;
+ this.outputFn = null;
+ this.accumulatorType = null;
+ }
+
+ private ReduceWindow(
+ @Nullable String name,
+ UnaryFunction<InputT, ValueT> valueExtractor,
+ @Nullable TypeDescriptor<ValueT> valueType,
+ VoidFunction<AccT> accumulatorFactory,
+ BinaryFunction<AccT, ValueT, AccT> accumulate,
+ CombinableBinaryFunction<AccT> mergeAccumulators,
+ UnaryFunction<AccT, OutputT> outputFn,
+ @Nullable TypeDescriptor<AccT> accumulatorType,
+ @Nullable BinaryFunction<ValueT, ValueT, Integer> valueComparator,
+ @Nullable Window<InputT> window,
+ TypeDescriptor<OutputT> outputType) {
+ super(name, outputType, e -> B_ZERO, TypeDescriptors.bytes(), window);
+ this.accumulatorFactory = requireNonNull(accumulatorFactory);
+ this.accumulate = requireNonNull(accumulate);
+ this.mergeAccumulators = requireNonNull(mergeAccumulators);
+ this.outputFn = requireNonNull(outputFn);
+ this.accumulatorType = accumulatorType;
+ this.valueExtractor = requireNonNull(valueExtractor);
+ this.valueType = valueType;
+ this.valueComparator = valueComparator;
+ this.reducer = null;
+ }
+
+ public boolean isCombineFnStyle() {
+ return reducer == null;
}
public ReduceFunctor<ValueT, OutputT> getReducer() {
- return reducer;
+ return requireNonNull(reducer, "Don't call #getReducer is #isCombineFnStyle() == true");
}
public boolean isCombinable() {
- return reducer.isCombinable();
+ return isCombineFnStyle() || reducer.isCombinable();
}
public UnaryFunction<InputT, ValueT> getValueExtractor() {
@@ -419,21 +622,38 @@
@Override
@SuppressWarnings("unchecked")
public PCollection<OutputT> expand(PCollectionList<InputT> inputs) {
- final ReduceByKey.ReduceByBuilder<Byte, ValueT> reduceBy =
- ReduceByKey.named(getName().orElse("") + "::reduce-by")
- .of(PCollectionLists.getOnlyElement(inputs))
- .keyBy(e -> B_ZERO)
- .valueBy(valueExtractor, valueType);
- final ReduceByKey.WithSortedValuesBuilder<Byte, ValueT, OutputT> sortBy =
- reduceBy.reduceBy(reducer);
if (isCombinable()) {
// sanity check
checkState(valueComparator == null, "Sorting is not supported for combinable reducers.");
}
- final ReduceByKey.WindowByBuilder<Byte, OutputT> windowBy =
- getValueComparator().isPresent()
- ? sortBy.withSortedValues(getValueComparator().get())
- : sortBy;
+ final ReduceByKey.WindowByBuilder<Byte, OutputT> windowBy;
+ if (isCombineFnStyle()) {
+ windowBy =
+ ReduceByKey.named(getName().orElse("") + "::reduce-by")
+ .of(PCollectionLists.getOnlyElement(inputs))
+ .keyBy(getKeyExtractor())
+ .valueBy(valueExtractor, valueType)
+ .combineBy(
+ accumulatorFactory,
+ accumulate,
+ mergeAccumulators,
+ outputFn,
+ accumulatorType,
+ getOutputType().orElse(null));
+ } else {
+ ReduceByKey.WithSortedValuesBuilder<Byte, ValueT, OutputT> reduceBy;
+ reduceBy =
+ ReduceByKey.named(getName().orElse("") + "::reduce-by")
+ .of(PCollectionLists.getOnlyElement(inputs))
+ .keyBy(getKeyExtractor())
+ .valueBy(valueExtractor, valueType)
+ .reduceBy(reducer, getOutputType().orElse(null));
+ if (getValueComparator().isPresent()) {
+ windowBy = reduceBy.withSortedValues(valueComparator);
+ } else {
+ windowBy = reduceBy;
+ }
+ }
return windowBy
.applyIf(
getWindow().isPresent(),
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Union.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Union.java
index 9a48bca..d6b0ac8 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Union.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Union.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.euphoria.core.client.operator;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Arrays;
import java.util.List;
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/WindowBuilder.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/WindowBuilder.java
index 523938e..5e7c00f 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/WindowBuilder.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/WindowBuilder.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.euphoria.core.client.operator;
import static java.util.Objects.requireNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Optional;
import javax.annotation.Nullable;
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java
index 5e544c1..8666f01 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java
@@ -19,7 +19,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Utilities related to {@link PCollection}s. */
public class PCollectionLists {
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/Sums.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/Sums.java
index 46cf1d3..e662c42 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/Sums.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/Sums.java
@@ -17,26 +17,68 @@
*/
package org.apache.beam.sdk.extensions.euphoria.core.client.util;
-import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.euphoria.core.annotation.audience.Audience;
-import org.apache.beam.sdk.extensions.euphoria.core.client.functional.CombinableReduceFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.BinaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.ReduceByKey;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
/** Provides commonly used function objects around computing sums. */
@Audience(Audience.Type.CLIENT)
public class Sums {
- private static final CombinableReduceFunction<Long> SUMS_OF_LONG =
- (CombinableReduceFunction<Long>) s -> s.collect(Collectors.summingLong(e -> e));
- private static final CombinableReduceFunction<Integer> SUMS_OF_INT =
- (CombinableReduceFunction<Integer>) s -> s.collect(Collectors.summingInt(e -> e));
+ private static class SumFunction<T> implements ReduceByKey.CombineFunctionWithIdentity<T> {
+
+ private final T identity;
+ private final TypeDescriptor<T> valueDesc;
+ private final BinaryFunction<T, T, T> reduce;
+
+ SumFunction(T identity, TypeDescriptor<T> valueDesc, BinaryFunction<T, T, T> reduce) {
+ this.identity = identity;
+ this.valueDesc = valueDesc;
+ this.reduce = reduce;
+ }
+
+ @Override
+ public T identity() {
+ return identity;
+ }
+
+ @Override
+ public TypeDescriptor<T> valueDesc() {
+ return valueDesc;
+ }
+
+ @Override
+ public T apply(T left, T right) {
+ return reduce.apply(left, right);
+ }
+ }
+
+ private static final SumFunction<Long> SUMS_OF_LONG =
+ new SumFunction<>(0L, TypeDescriptors.longs(), (a, b) -> a + b);
+ private static final SumFunction<Integer> SUMS_OF_INT =
+ new SumFunction<>(0, TypeDescriptors.integers(), (a, b) -> a + b);
+ private static final SumFunction<Float> SUMS_OF_FLOAT =
+ new SumFunction<>(0.0f, TypeDescriptors.floats(), (a, b) -> a + b);
+ private static final SumFunction<Double> SUMS_OF_DOUBLE =
+ new SumFunction<>(0.0, TypeDescriptors.doubles(), (a, b) -> a + b);
private Sums() {}
- public static CombinableReduceFunction<Long> ofLongs() {
+ public static ReduceByKey.CombineFunctionWithIdentity<Long> ofLongs() {
return SUMS_OF_LONG;
}
- public static CombinableReduceFunction<Integer> ofInts() {
+ public static ReduceByKey.CombineFunctionWithIdentity<Integer> ofInts() {
return SUMS_OF_INT;
}
+
+ public static ReduceByKey.CombineFunctionWithIdentity<Float> ofFloats() {
+ return SUMS_OF_FLOAT;
+ }
+
+ public static ReduceByKey.CombineFunctionWithIdentity<Double> ofDoubles() {
+ return SUMS_OF_DOUBLE;
+ }
}
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/AbstractJoinTranslator.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/AbstractJoinTranslator.java
index 4d70dec..b1c04f1 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/AbstractJoinTranslator.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/AbstractJoinTranslator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.euphoria.core.translate;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.Join;
import org.apache.beam.sdk.extensions.euphoria.core.client.type.TypeAwareness;
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/BroadcastHashJoinTranslator.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/BroadcastHashJoinTranslator.java
index 501d22b..5216a05 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/BroadcastHashJoinTranslator.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/BroadcastHashJoinTranslator.java
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table;
/**
* Translator for {@link org.apache.beam.sdk.extensions.euphoria.core.client.operator.RightJoin} and
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/CompositeOperatorTranslator.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/CompositeOperatorTranslator.java
index 0edb215..894f7be 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/CompositeOperatorTranslator.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/CompositeOperatorTranslator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.euphoria.core.translate;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.CompositeOperator;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.base.Operator;
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/OperatorTransform.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/OperatorTransform.java
index 54d07fa..460ea54 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/OperatorTransform.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/OperatorTransform.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/**
* Expand operator to a beam {@link PTransform}.
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/ReduceByKeyTranslator.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/ReduceByKeyTranslator.java
index 64d5d78..8bf3cf6 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/ReduceByKeyTranslator.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/ReduceByKeyTranslator.java
@@ -18,13 +18,19 @@
package org.apache.beam.sdk.extensions.euphoria.core.translate;
import static java.util.Objects.requireNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.stream.StreamSupport;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
import org.apache.beam.sdk.extensions.euphoria.core.client.accumulators.AccumulatorProvider;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.BinaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.CombinableBinaryFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.ReduceFunctor;
import org.apache.beam.sdk.extensions.euphoria.core.client.functional.UnaryFunction;
+import org.apache.beam.sdk.extensions.euphoria.core.client.functional.VoidFunction;
import org.apache.beam.sdk.extensions.euphoria.core.client.operator.ReduceByKey;
import org.apache.beam.sdk.extensions.euphoria.core.client.type.TypeAwareness;
import org.apache.beam.sdk.extensions.euphoria.core.client.util.PCollectionLists;
@@ -40,23 +46,23 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
/** Translator for {@code ReduceByKey} operator. */
public class ReduceByKeyTranslator<InputT, KeyT, ValueT, OutputT>
implements OperatorTranslator<
- InputT, KV<KeyT, OutputT>, ReduceByKey<InputT, KeyT, ValueT, OutputT>> {
+ InputT, KV<KeyT, OutputT>, ReduceByKey<InputT, KeyT, ValueT, ?, OutputT>> {
@Override
public PCollection<KV<KeyT, OutputT>> translate(
- ReduceByKey<InputT, KeyT, ValueT, OutputT> operator, PCollectionList<InputT> inputs) {
+ ReduceByKey<InputT, KeyT, ValueT, ?, OutputT> operator, PCollectionList<InputT> inputs) {
// todo Could we even do values sorting in Beam ? And do we want it?
checkState(!operator.getValueComparator().isPresent(), "Values sorting is not supported.");
final UnaryFunction<InputT, KeyT> keyExtractor = operator.getKeyExtractor();
final UnaryFunction<InputT, ValueT> valueExtractor = operator.getValueExtractor();
- final ReduceFunctor<ValueT, OutputT> reducer = operator.getReducer();
final PCollection<InputT> input =
operator
@@ -81,10 +87,18 @@
if (operator.isCombinable()) {
// if operator is combinable we can process it in more efficient way
- final PCollection<KV<KeyT, ValueT>> combined =
- extracted.apply(
- "combine",
- Combine.perKey(asCombiner(reducer, accumulators, operator.getName().orElse(null))));
+ @SuppressWarnings("unchecked")
+ final PCollection combined;
+ if (operator.isCombineFnStyle()) {
+ combined = extracted.apply("combine", Combine.perKey(asCombineFn(operator)));
+ } else {
+ combined =
+ extracted.apply(
+ "combine",
+ Combine.perKey(
+ asCombiner(
+ operator.getReducer(), accumulators, operator.getName().orElse(null))));
+ }
@SuppressWarnings("unchecked")
final PCollection<KV<KeyT, OutputT>> cast = (PCollection) combined;
return cast.setTypeDescriptor(
@@ -102,7 +116,9 @@
TypeDescriptors.iterables(TypeAwareness.orObjects(operator.getValueType()))))
.apply(
"reduce",
- ParDo.of(new ReduceDoFn<>(reducer, accumulators, operator.getName().orElse(null))))
+ ParDo.of(
+ new ReduceDoFn<>(
+ operator.getReducer(), accumulators, operator.getName().orElse(null))))
.setTypeDescriptor(
operator
.getOutputType()
@@ -116,6 +132,57 @@
return !operator.getValueComparator().isPresent();
}
+ private static <InputT, KeyT, ValueT, AccT, OutputT>
+ Combine.CombineFn<ValueT, AccT, OutputT> asCombineFn(
+ ReduceByKey<InputT, KeyT, ValueT, AccT, OutputT> operator) {
+
+ @SuppressWarnings("unchecked")
+ ReduceByKey<InputT, KeyT, ValueT, AccT, OutputT> cast = (ReduceByKey) operator;
+
+ VoidFunction<AccT> accumulatorFactory = cast.getAccumulatorFactory();
+ BinaryFunction<AccT, ValueT, AccT> accumulate = cast.getAccumulate();
+ CombinableBinaryFunction<AccT> mergeAccumulators = cast.getMergeAccumulators();
+ UnaryFunction<AccT, OutputT> outputFn = cast.getOutputFn();
+ TypeDescriptor<AccT> accumulatorType = cast.getAccumulatorType();
+
+ return new Combine.CombineFn<ValueT, AccT, OutputT>() {
+
+ @Override
+ public AccT createAccumulator() {
+ return accumulatorFactory.apply();
+ }
+
+ @Override
+ public Coder<AccT> getAccumulatorCoder(CoderRegistry registry, Coder<ValueT> inputCoder)
+ throws CannotProvideCoderException {
+ return registry.getCoder(accumulatorType);
+ }
+
+ @Override
+ public AccT addInput(AccT mutableAccumulator, ValueT input) {
+ return accumulate.apply(mutableAccumulator, input);
+ }
+
+ @Override
+ public AccT mergeAccumulators(Iterable<AccT> accumulators) {
+ AccT accumulated = null;
+ for (AccT o : accumulators) {
+ if (accumulated == null) {
+ accumulated = o;
+ } else {
+ accumulated = mergeAccumulators.apply(accumulated, o);
+ }
+ }
+ return accumulated;
+ }
+
+ @Override
+ public OutputT extractOutput(AccT accumulator) {
+ return outputFn.apply(accumulator);
+ }
+ };
+ }
+
private static <InputT, OutputT> SerializableFunction<Iterable<InputT>, InputT> asCombiner(
ReduceFunctor<InputT, OutputT> reducer,
AccumulatorProvider accumulatorProvider,
diff --git a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/GenericTranslatorProvider.java b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/GenericTranslatorProvider.java
index f9ea163..6bf8ae9 100644
--- a/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/GenericTranslatorProvider.java
+++ b/sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/translate/provider/GenericTranslatorProvider.java
@@ -37,7 +37,7 @@
import org.apache.beam.sdk.extensions.euphoria.core.translate.ReduceByKeyTranslator;
import org.apache.beam.sdk.extensions.euphoria.core.translate.TranslatorProvider;
import org.apache.beam.sdk.extensions.euphoria.core.translate.UnionTranslator;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/**
* Adjustable {@link TranslatorProvider} that selects first suitable translation for the registered
diff --git a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKeyTest.java b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKeyTest.java
index 441cca5..cbf5474 100644
--- a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKeyTest.java
+++ b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/ReduceByKeyTest.java
@@ -23,8 +23,11 @@
import static org.junit.Assert.assertSame;
import static org.junit.Assert.assertTrue;
-import java.util.stream.StreamSupport;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.euphoria.core.client.type.TypePropagationAssert;
+import org.apache.beam.sdk.extensions.euphoria.core.client.util.Sums;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
@@ -35,6 +38,8 @@
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.junit.Test;
@@ -51,7 +56,7 @@
.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .combineBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .combineBy(Sums.ofLongs())
.windowBy(windowing)
.triggeredBy(trigger)
.discardingFiredPanes()
@@ -63,7 +68,12 @@
assertEquals("ReduceByKey1", reduce.getName().get());
assertNotNull(reduce.getKeyExtractor());
assertNotNull(reduce.getValueExtractor());
- assertNotNull(reduce.getReducer());
+ assertTrue(reduce.isCombineFnStyle());
+ assertNotNull(reduce.getAccumulatorFactory());
+ assertNotNull(reduce.getAccumulate());
+ assertNotNull(reduce.getAccumulatorType());
+ assertNotNull(reduce.getMergeAccumulators());
+ assertNotNull(reduce.getOutputFn());
assertTrue(reduce.getWindow().isPresent());
@SuppressWarnings("unchecked")
@@ -82,7 +92,7 @@
.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .reduceBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .combineBy(Sums.ofLongs())
.outputValues();
final OutputValues outputValues = (OutputValues) TestUtils.getProducer(reduced);
@@ -94,11 +104,7 @@
public void testBuild_ImplicitName() {
final PCollection<String> dataset = TestUtils.createMockDataset(TypeDescriptors.strings());
final PCollection<KV<String, Long>> reduced =
- ReduceByKey.of(dataset)
- .keyBy(s -> s)
- .valueBy(s -> 1L)
- .combineBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
- .output();
+ ReduceByKey.of(dataset).keyBy(s -> s).valueBy(s -> 1L).combineBy(Sums.ofLongs()).output();
final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
assertFalse(reduce.getName().isPresent());
}
@@ -110,10 +116,71 @@
ReduceByKey.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .reduceBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .reduceBy(s -> s.mapToLong(e -> e).sum())
.output();
final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
assertNotNull(reduce.getReducer());
+ assertFalse(reduce.isCombineFnStyle());
+ }
+
+ @Test
+ public void testBuild_CombineByStream() {
+ final PCollection<String> dataset = TestUtils.createMockDataset(TypeDescriptors.strings());
+ final PCollection<KV<String, Long>> reduced =
+ ReduceByKey.of(dataset)
+ .keyBy(s -> s)
+ .valueBy(s -> 1L)
+ .combineBy(s -> s.mapToLong(e -> e).sum())
+ .output();
+ final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
+ assertNotNull(reduce.getReducer());
+ assertFalse(reduce.isCombineFnStyle());
+ }
+
+ @Test
+ public void testBuild_CombineByFull() {
+ final PCollection<String> dataset = TestUtils.createMockDataset(TypeDescriptors.strings());
+ final PCollection<KV<String, Integer>> reduced =
+ ReduceByKey.of(dataset)
+ .keyBy(s -> s)
+ .valueBy(s -> 1L)
+ .combineBy(
+ () -> new ArrayList<>(),
+ (acc, e) -> {
+ acc.add(e);
+ return acc;
+ },
+ (l, r) -> Lists.newArrayList(Iterables.concat(l, r)),
+ List::size,
+ TypeDescriptors.lists(TypeDescriptors.longs()),
+ TypeDescriptors.integers())
+ .output();
+ final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
+ assertTrue(reduce.isCombineFnStyle());
+ assertNotNull(reduce.getAccumulatorFactory());
+ assertNotNull(reduce.getAccumulatorType());
+ assertNotNull(reduce.getAccumulate());
+ assertNotNull(reduce.getMergeAccumulators());
+ assertNotNull(reduce.getOutputFn());
+ assertTrue(reduce.getOutputType().isPresent());
+ }
+
+ @Test
+ public void testBuild_CombineBy() {
+ final PCollection<String> dataset = TestUtils.createMockDataset(TypeDescriptors.strings());
+ final PCollection<KV<String, Long>> reduced =
+ ReduceByKey.of(dataset)
+ .keyBy(s -> s)
+ .valueBy(s -> 1L)
+ .combineBy(0L, (a, b) -> a + b)
+ .output();
+ final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
+ assertTrue(reduce.isCombineFnStyle());
+ assertNotNull(reduce.getAccumulatorFactory());
+ assertNotNull(reduce.getAccumulate());
+ assertNotNull(reduce.getMergeAccumulators());
+ assertNotNull(reduce.getOutputFn());
+ assertTrue(reduce.getOutputType().isPresent());
}
@Test
@@ -123,7 +190,7 @@
ReduceByKey.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .combineBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .combineBy(Sums.ofLongs())
.windowBy(FixedWindows.of(Duration.standardHours(1)))
.triggeredBy(DefaultTrigger.of())
.accumulationMode(AccumulationMode.DISCARDING_FIRED_PANES)
@@ -144,11 +211,11 @@
@Test
public void testBuild_sortedValues() {
final PCollection<String> dataset = TestUtils.createMockDataset(TypeDescriptors.strings());
- final PCollection<KV<String, Long>> reduced =
+ final PCollection<KV<String, List<Long>>> reduced =
ReduceByKey.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .reduceBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .reduceBy(s -> s.collect(Collectors.toList()))
.withSortedValues(Long::compare)
.windowBy(FixedWindows.of(Duration.standardHours(1)))
.triggeredBy(DefaultTrigger.of())
@@ -161,11 +228,11 @@
@Test
public void testBuild_sortedValuesWithNoWindowing() {
final PCollection<String> dataset = TestUtils.createMockDataset(TypeDescriptors.strings());
- final PCollection<KV<String, Long>> reduced =
+ final PCollection<KV<String, List<Long>>> reduced =
ReduceByKey.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .reduceBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .reduceBy(s -> s.collect(Collectors.toList()))
.withSortedValues(Long::compare)
.output();
final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
@@ -179,7 +246,7 @@
ReduceByKey.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .reduceBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .combineBy(Sums.ofLongs())
.applyIf(
true,
b ->
@@ -204,7 +271,7 @@
ReduceByKey.of(dataset)
.keyBy(s -> s)
.valueBy(s -> 1L)
- .reduceBy(n -> StreamSupport.stream(n.spliterator(), false).mapToLong(Long::new).sum())
+ .combineBy(Sums.ofLongs())
.applyIf(
false,
b ->
@@ -227,7 +294,7 @@
ReduceByKey.of(dataset)
.keyBy(s -> s, keyType)
.valueBy(s -> 1L, valueType)
- .combineBy(n -> n.mapToLong(l -> l).sum(), outputType)
+ .combineBy(Sums.ofLongs())
.output();
final ReduceByKey reduce = (ReduceByKey) TestUtils.getProducer(reduced);
TypePropagationAssert.assertOperatorTypeAwareness(reduce, keyType, valueType, outputType);
diff --git a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/SumsTest.java b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/SumsTest.java
new file mode 100644
index 0000000..38efa7b
--- /dev/null
+++ b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/SumsTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.euphoria.core.client.util;
+
+import static org.junit.Assert.assertEquals;
+
+import java.util.stream.Stream;
+import org.apache.beam.sdk.extensions.euphoria.core.client.operator.ReduceByKey;
+import org.junit.Test;
+
+/** Test suite for @{link Sums}. */
+public class SumsTest {
+
+ @Test
+ public void testSumOfInts() {
+ assertEquals(6, (int) apply(Stream.of(1, 2, 3), Sums.ofInts()));
+ }
+
+ @Test
+ public void testSumOfLongs() {
+ assertEquals(6L, (long) apply(Stream.of(1L, 2L, 3L), Sums.ofLongs()));
+ }
+
+ @Test
+ public void testSumOfFloats() {
+ assertEquals(6f, (float) apply(Stream.of(1f, 2f, 3f), Sums.ofFloats()), 0.001);
+ }
+
+ @Test
+ public void testSumOfDoubles() {
+ assertEquals(6.0, (double) apply(Stream.of(1.0, 2.0, 3.0), Sums.ofDoubles()), 0.001);
+ }
+
+ private <T> T apply(Stream<T> stream, ReduceByKey.CombineFunctionWithIdentity<T> fn) {
+ return stream.reduce(fn.identity(), fn::apply);
+ }
+}
diff --git a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/docs/DocumentationExamplesTest.java b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/docs/DocumentationExamplesTest.java
index b078572..404d6d1 100644
--- a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/docs/DocumentationExamplesTest.java
+++ b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/docs/DocumentationExamplesTest.java
@@ -69,7 +69,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.joda.time.Duration;
import org.junit.Assert;
import org.junit.Before;
diff --git a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/ReduceByKeyTest.java b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/ReduceByKeyTest.java
index 5d4ca76..863a7c2 100644
--- a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/ReduceByKeyTest.java
+++ b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/ReduceByKeyTest.java
@@ -51,7 +51,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Ignore;
@@ -295,7 +297,7 @@
return ReduceByKey.of(input)
.keyBy(e -> e, TypeDescriptor.of(String.class))
.valueBy(e -> 1L, TypeDescriptor.of(Long.class))
- .combineBy(Sums.ofLongs(), TypeDescriptor.of(Long.class))
+ .combineBy(Sums.ofLongs())
.output();
}
});
@@ -502,6 +504,49 @@
});
}
+ @Test
+ public void testCombineFull() {
+ execute(
+ new AbstractTestCase<Integer, KV<Integer, Integer>>() {
+
+ @Override
+ protected List<Integer> getInput() {
+ return Arrays.asList(1, 2, 3, 4, 5, 6, 7, 9);
+ }
+
+ @Override
+ protected TypeDescriptor<Integer> getInputType() {
+ return TypeDescriptors.integers();
+ }
+
+ @Override
+ protected PCollection<KV<Integer, Integer>> getOutput(PCollection<Integer> input) {
+ return ReduceByKey.of(input)
+ .keyBy(e -> e % 2)
+ .valueBy(e -> e)
+ .combineBy(
+ () -> new ArrayList<>(),
+ (acc, e) -> {
+ acc.add(e);
+ return acc;
+ },
+ (l, r) -> Lists.newArrayList(Iterables.concat(l, r)),
+ List::size,
+ TypeDescriptors.lists(TypeDescriptors.integers()),
+ TypeDescriptors.integers())
+ .windowBy(new GlobalWindows())
+ .triggeredBy(AfterWatermark.pastEndOfWindow())
+ .discardingFiredPanes()
+ .output();
+ }
+
+ @Override
+ public List<KV<Integer, Integer>> getUnorderedOutput() {
+ return Arrays.asList(KV.of(0, 3), KV.of(1, 5));
+ }
+ });
+ }
+
private static class TestWindowFn extends WindowFn<Number, CountWindow> {
@Override
diff --git a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/accumulators/NanosecondTimer.java b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/accumulators/NanosecondTimer.java
index d4339ca..7d7c666 100644
--- a/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/accumulators/NanosecondTimer.java
+++ b/sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/accumulators/NanosecondTimer.java
@@ -21,7 +21,7 @@
import java.util.Map;
import java.util.concurrent.TimeUnit;
import org.apache.beam.sdk.extensions.euphoria.core.client.accumulators.Timer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
final class NanosecondTimer implements Timer, Snapshotable<Map<Duration, Long>> {
diff --git a/sdks/java/extensions/google-cloud-platform-core/build.gradle b/sdks/java/extensions/google-cloud-platform-core/build.gradle
index 7324fa9..9ffa229 100644
--- a/sdks/java/extensions/google-cloud-platform-core/build.gradle
+++ b/sdks/java/extensions/google-cloud-platform-core/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.gcp')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core"
ext.summary = """Common components used to support multiple
@@ -34,36 +34,35 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.google_http_client_jackson2
- shadow library.java.google_auth_library_oauth2_http
- shadow library.java.google_api_client
- shadow library.java.bigdataoss_gcsio
- shadow library.java.bigdataoss_util
- shadow library.java.google_api_services_cloudresourcemanager
- shadow library.java.google_api_services_storage
- shadow library.java.google_auth_library_credentials
- shadow library.java.google_http_client
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.google_http_client_jackson2
+ compile library.java.google_auth_library_oauth2_http
+ compile library.java.google_api_client
+ compile library.java.bigdataoss_gcsio
+ compile library.java.bigdataoss_util
+ compile library.java.google_api_services_cloudresourcemanager
+ compile library.java.google_api_services_storage
+ compile library.java.google_auth_library_credentials
+ compile library.java.google_http_client
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
provided library.java.hamcrest_core
provided library.java.junit
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.hamcrest_library
- testCompile library.java.guava_testlib
testCompile library.java.mockito_core
testRuntimeOnly library.java.slf4j_jdk14
}
// Note that no runner is specified here, so tests running under this task should not be running
// pipelines.
-task integrationTest(type: Test) {
+task integrationTestKms(type: Test) {
group = "Verification"
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
- def gcpTempRoot = project.findProperty('gcpTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests'
+ def gcpTempRoot = project.findProperty('gcpTempRootKms') ?: 'gs://temp-storage-for-end-to-end-tests-cmek'
def dataflowKmsKey = project.findProperty('dataflowKmsKey') ?: "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test"
systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
"--project=${gcpProject}",
@@ -78,11 +77,13 @@
maxParallelForks 4
classpath = sourceSets.test.runtimeClasspath
testClassesDirs = sourceSets.test.output.classesDirs
- useJUnit { }
+ useJUnit {
+ includeCategories "org.apache.beam.sdk.testing.UsesKms"
+ }
}
task postCommit {
group = "Verification"
description = "Integration tests of GCP connectors using the DirectRunner."
- dependsOn integrationTest
+ dependsOn integrationTestKms
}
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
index 61c28c48..37cf6bc 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.gcp.options;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings.isNullOrEmpty;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings.isNullOrEmpty;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.google.api.client.http.HttpRequestInitializer;
@@ -27,7 +27,6 @@
import com.google.api.services.cloudresourcemanager.CloudResourceManager;
import com.google.api.services.cloudresourcemanager.model.Project;
import com.google.api.services.storage.model.Bucket;
-import com.google.api.services.storage.model.Bucket.Encryption;
import com.google.auth.Credentials;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.cloud.hadoop.util.ChainingHttpRequestInitializer;
@@ -59,9 +58,9 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.InstanceBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -95,6 +94,7 @@
*/
@Description(
"GCP availability zone for running GCP operations. "
+ + "and GCE availability zone for launching workers "
+ "Default is up to the individual service.")
String getZone();
@@ -295,6 +295,10 @@
static String tryCreateDefaultBucket(PipelineOptions options, CloudResourceManager crmClient) {
GcsOptions gcsOptions = options.as(GcsOptions.class);
+ checkArgument(
+ isNullOrEmpty(gcsOptions.getDataflowKmsKey()),
+ "Cannot create a default bucket when --dataflowKmsKey is set.");
+
final String projectId = gcsOptions.getProject();
checkArgument(!isNullOrEmpty(projectId), "--project is a required option.");
@@ -312,11 +316,7 @@
}
final String bucketName = "dataflow-staging-" + region + "-" + projectNumber;
LOG.info("No tempLocation specified, attempting to use default bucket: {}", bucketName);
- Bucket bucket =
- new Bucket()
- .setName(bucketName)
- .setLocation(region)
- .setEncryption(new Encryption().setDefaultKmsKeyName(gcsOptions.getDataflowKmsKey()));
+ Bucket bucket = new Bucket().setName(bucketName).setLocation(region);
// Always try to create the bucket before checking access, so that we do not
// race with other pipelines that may be attempting to do the same thing.
try {
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpPipelineOptionsRegistrar.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpPipelineOptionsRegistrar.java
index 46761ee..17d2d78 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpPipelineOptionsRegistrar.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpPipelineOptionsRegistrar.java
@@ -20,7 +20,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A registrar containing the default GCP options. */
@AutoService(PipelineOptionsRegistrar.class)
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java
index dc3d1f4..7248d98 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.options.Hidden;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.InstanceBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
/** Options used to configure Google Cloud Storage. */
public interface GcsOptions extends ApplicationNameOptions, GcpOptions, PipelineOptions {
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
index 3fd507c..e2ca634 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.extensions.gcp.storage;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.DateTime;
import com.google.api.services.storage.model.Objects;
@@ -48,11 +48,11 @@
import org.apache.beam.sdk.io.fs.MatchResult.Status;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.java
index 599906a..6caff00 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.gcp.storage;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.service.AutoService;
import javax.annotation.Nonnull;
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.io.FileSystem;
import org.apache.beam.sdk.io.FileSystemRegistrar;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link AutoService} registrar for the {@link GcsFileSystem}. */
@AutoService(FileSystemRegistrar.class)
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
index e109634..8965f23 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsPathValidator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.gcp.storage;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceId.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceId.java
index 397736e..fe893da 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceId.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsResourceId.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.extensions.gcp.storage;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import javax.annotation.Nullable;
import org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
index 0f066b8..e669c29 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.gcp.util;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.googleapis.batch.BatchRequest;
import com.google.api.client.googleapis.batch.json.JsonBatchCallback;
@@ -70,10 +70,10 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
index 1f6b8d1..8759f8e 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java
@@ -17,8 +17,6 @@
*/
package org.apache.beam.sdk.extensions.gcp.util;
-import static com.google.api.client.util.BackOffUtils.next;
-
import com.google.api.client.http.HttpIOExceptionHandler;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
@@ -36,6 +34,8 @@
import java.util.HashSet;
import java.util.Set;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -68,6 +68,9 @@
private final BackOff ioExceptionBackOff;
private final BackOff unsuccessfulResponseBackOff;
private final Set<Integer> ignoredResponseCodes;
+ // aggregate the total time spent in exponential backoff
+ private final Counter throttlingSeconds =
+ Metrics.counter(LoggingHttpBackOffHandler.class, "cumulativeThrottlingSeconds");
private int ioExceptionRetries;
private int unsuccessfulResponseRetries;
@Nullable private CustomHttpErrors customHttpErrors;
@@ -172,7 +175,13 @@
/** Returns true iff performing the backoff was successful. */
private boolean backOffWasSuccessful(BackOff backOff) {
try {
- return next(sleeper, backOff);
+ long backOffTime = backOff.nextBackOffMillis();
+ if (backOffTime == BackOff.STOP) {
+ return false;
+ }
+ throttlingSeconds.inc(backOffTime / 1000);
+ sleeper.sleep(backOffTime);
+ return true;
} catch (InterruptedException | IOException e) {
return false;
}
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
index 5662c45..52a01d6 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/Transport.java
@@ -33,7 +33,7 @@
import java.security.GeneralSecurityException;
import org.apache.beam.sdk.extensions.gcp.auth.NullCredentialInitializer;
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Helpers for cloud communication. */
public class Transport {
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/gcsfs/GcsPath.java b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/gcsfs/GcsPath.java
index a0a4a52..99f2fb4 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/gcsfs/GcsPath.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/gcsfs/GcsPath.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.gcp.util.gcsfs;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings.isNullOrEmpty;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings.isNullOrEmpty;
import com.google.api.services.storage.model.StorageObject;
import java.io.File;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
index 5bee366..9a041e9 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/GcpCoreApiSurfaceTest.java
@@ -23,7 +23,7 @@
import java.util.Set;
import org.apache.beam.sdk.util.ApiSurface;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.hamcrest.Matcher;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java
index e5a0f83..ab03de0 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptionsTest.java
@@ -45,8 +45,8 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.RestoreSystemProperties;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -63,7 +63,7 @@
/** Tests for the majority of methods. */
@RunWith(JUnit4.class)
- public static class Common {
+ public static class CommonTests {
@Rule public TestRule restoreSystemProperties = new RestoreSystemProperties();
@Rule public TemporaryFolder tmpFolder = new TemporaryFolder();
@Rule public ExpectedException thrown = ExpectedException.none();
@@ -199,7 +199,7 @@
/** Tests related to determining the GCP temp location. */
@RunWith(JUnit4.class)
- public static class GcpTempLocation {
+ public static class GcpTempLocationTest {
@Rule public ExpectedException thrown = ExpectedException.none();
@Mock private GcsUtil mockGcsUtil;
@Mock private CloudResourceManager mockCrmClient;
@@ -271,6 +271,16 @@
}
@Test
+ public void testCreateBucketCreateWithKmsFails() throws Exception {
+ doReturn(fakeProject).when(mockGet).execute();
+ options.as(GcpOptions.class).setDataflowKmsKey("kms_key");
+
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("dataflowKmsKey");
+ GcpTempLocationFactory.tryCreateDefaultBucket(options, mockCrmClient);
+ }
+
+ @Test
public void regionFromZone() throws Exception {
assertEquals("us-central1", GcpTempLocationFactory.getRegionFromZone("us-central1-a"));
assertEquals("asia-east", GcpTempLocationFactory.getRegionFromZone("asia-east-a"));
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrarTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrarTest.java
index 3c1446a..83a213b 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrarTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrarTest.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.io.FileSystem;
import org.apache.beam.sdk.io.FileSystemRegistrar;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemTest.java
index 7bd3f9d..3547411 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemTest.java
@@ -40,8 +40,8 @@
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.MatchResult.Status;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.FluentIterable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.FluentIterable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/CustomHttpErrorsTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/CustomHttpErrorsTest.java
index c33933b..0bb274f 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/CustomHttpErrorsTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/CustomHttpErrorsTest.java
@@ -19,8 +19,8 @@
import static junit.framework.TestCase.assertEquals;
import static junit.framework.TestCase.assertNull;
-import static org.mockito.BDDMockito.given;
-import static org.mockito.BDDMockito.mock;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.json.Json;
@@ -59,13 +59,13 @@
private HttpRequestWrapper createHttpRequest(String url) throws MalformedURLException {
HttpRequestWrapper request = mock(HttpRequestWrapper.class);
GenericUrl genericUrl = new GenericUrl(new URL(url));
- given(request.getUrl()).willReturn(genericUrl);
+ when(request.getUrl()).thenReturn(genericUrl);
return request;
}
private HttpResponseWrapper createHttpResponse(int statusCode) {
HttpResponseWrapper response = mock(HttpResponseWrapper.class);
- given(response.getStatusCode()).willReturn(statusCode);
+ when(response.getStatusCode()).thenReturn(statusCode);
return response;
}
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilIT.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilIT.java
index 9a3ebf7..ff96566 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilIT.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilIT.java
@@ -28,8 +28,10 @@
import org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.sdk.testing.UsesKms;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -41,17 +43,19 @@
* this test should only be run against single runner (such as DirectRunner).
*/
@RunWith(JUnit4.class)
+@Category(UsesKms.class)
public class GcsUtilIT {
/** Tests a rewrite operation that requires multiple API calls (using a continuation token). */
@Test
public void testRewriteMultiPart() throws IOException {
TestPipelineOptions options =
TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);
- GcsOptions gcsOptions = options.as(GcsOptions.class);
- // Setting the KMS key is necessary to trigger multi-part rewrites (gcpTempLocation is created
+ // Using a KMS key is necessary to trigger multi-part rewrites (bucket is created
// with a bucket default key).
- assertNotNull(gcsOptions.getDataflowKmsKey());
+ assertNotNull(options.getTempRoot());
+ options.setTempLocation(options.getTempRoot() + "/testRewriteMultiPart");
+ GcsOptions gcsOptions = options.as(GcsOptions.class);
GcsUtil gcsUtil = gcsOptions.getGcsUtil();
String srcFilename = "gs://dataflow-samples/wikipedia_edits/wiki_data-000000000000.json";
String dstFilename =
diff --git a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java
index 759296e..6bc0a75 100644
--- a/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java
+++ b/sdks/java/extensions/google-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java
@@ -80,8 +80,8 @@
import org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/extensions/jackson/build.gradle b/sdks/java/extensions/jackson/build.gradle
index e38de09..3d1e692 100644
--- a/sdks/java/extensions/jackson/build.gradle
+++ b/sdks/java/extensions/jackson/build.gradle
@@ -17,18 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-extensions-json-jackson'
-applyJavaNature()
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.jackson',
+ archivesBaseName: 'beam-sdks-java-extensions-json-jackson'
+)
description = "Apache Beam :: SDKs :: Java :: Extensions :: Jackson"
ext.summary = "Jackson extension provides PTransforms for deserializing and generating JSON strings."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.jackson_databind
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.jackson_databind
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
index 935e75d..9f1a199 100644
--- a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
+++ b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
@@ -17,13 +17,27 @@
*/
package org.apache.beam.sdk.extensions.jackson;
+import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
+import edu.umd.cs.findbugs.annotations.Nullable;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.Contextful;
+import org.apache.beam.sdk.transforms.InferableFunction;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ProcessFunction;
+import org.apache.beam.sdk.transforms.Requirements;
import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.WithFailures;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* {@link PTransform} for serializing objects to JSON {@link String Strings}. Transforms a {@code
@@ -41,12 +55,12 @@
* into a {@link PCollection} of JSON {@link String Strings} representing those objects using a
* Jackson {@link ObjectMapper}.
*/
- public static <OutputT> AsJsons<OutputT> of(Class<? extends OutputT> outputClass) {
- return new AsJsons<>(outputClass);
+ public static <InputT> AsJsons<InputT> of(Class<? extends InputT> inputClass) {
+ return new AsJsons<>(inputClass);
}
- private AsJsons(Class<? extends InputT> outputClass) {
- this.inputClass = outputClass;
+ private AsJsons(Class<? extends InputT> inputClass) {
+ this.inputClass = inputClass;
}
/** Use custom Jackson {@link ObjectMapper} instead of the default one. */
@@ -56,6 +70,83 @@
return newTransform;
}
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, with the given type descriptor used for the failure collection but the
+ * exception handler yet to be specified using {@link
+ * AsJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <NewFailureT> AsJsonsWithFailures<NewFailureT> exceptionsInto(
+ TypeDescriptor<NewFailureT> failureTypeDescriptor) {
+ return new AsJsonsWithFailures<>(null, failureTypeDescriptor);
+ }
+
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, passing the raised exception instance and the input element being
+ * processed through the given {@code exceptionHandler} and emitting the result to a failure
+ * collection.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ * pojos.apply(
+ * AsJsons.of(MyPojo.class)
+ * .exceptionsVia(new WithFailures.ExceptionAsMapHandler<MyPojo>() {}));
+ *
+ * PCollection<String> output = result.output(); // valid json elements
+ * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <FailureT> AsJsonsWithFailures<FailureT> exceptionsVia(
+ InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler) {
+ return new AsJsonsWithFailures<>(exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, passing the raised exception instance and the input element being
+ * processed through the default exception handler {@link DefaultExceptionAsMapHandler} and
+ * emitting the result to a failure collection.
+ *
+ * <p>See {@link DefaultExceptionAsMapHandler} for more details about default handler behavior.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ * pojos.apply(
+ * AsJsons.of(MyPojo.class)
+ * .exceptionsVia());
+ *
+ * PCollection<String> output = result.output(); // valid json elements
+ * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public AsJsonsWithFailures<KV<InputT, Map<String, String>>> exceptionsVia() {
+ DefaultExceptionAsMapHandler<InputT> exceptionHandler =
+ new DefaultExceptionAsMapHandler<InputT>() {};
+ return new AsJsonsWithFailures<>(exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ private String writeValue(InputT input) throws JsonProcessingException {
+ ObjectMapper mapper = Optional.ofNullable(customMapper).orElse(DEFAULT_MAPPER);
+ return mapper.writeValueAsString(input);
+ }
+
@Override
public PCollection<String> expand(PCollection<InputT> input) {
return input.apply(
@@ -64,8 +155,7 @@
@Override
public String apply(InputT input) {
try {
- ObjectMapper mapper = Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
- return mapper.writeValueAsString(input);
+ return writeValue(input);
} catch (IOException e) {
throw new RuntimeException(
"Failed to serialize " + inputClass.getName() + " value: " + input, e);
@@ -73,4 +163,93 @@
}
}));
}
+
+ /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+ public class AsJsonsWithFailures<FailureT>
+ extends PTransform<PCollection<InputT>, WithFailures.Result<PCollection<String>, FailureT>> {
+
+ @Nullable
+ private InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler;
+
+ @Nullable private final transient TypeDescriptor<FailureT> failureType;
+
+ AsJsonsWithFailures(
+ InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler,
+ TypeDescriptor<FailureT> failureType) {
+ this.exceptionHandler = exceptionHandler;
+ this.failureType = failureType;
+ }
+
+ /**
+ * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
+ * writing JSON elements, passing the raised exception instance and the input element being
+ * processed through the given {@code exceptionHandler} and emitting the result to a failure
+ * collection. It is supposed to be used along with {@link
+ * AsJsons#exceptionsInto(TypeDescriptor)} and get lambda function as exception handler.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ * pojos.apply(
+ * AsJsons.of(MyPojo.class)
+ * .exceptionsInto(
+ * TypeDescriptors.kvs(
+ * TypeDescriptor.of(MyPojo.class), TypeDescriptors.strings()))
+ * .exceptionsVia(
+ * f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ *
+ * PCollection<String> output = result.output(); // valid json elements
+ * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ public AsJsonsWithFailures<FailureT> exceptionsVia(
+ ProcessFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler) {
+ return new AsJsonsWithFailures<>(
+ new InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT>(
+ exceptionHandler) {},
+ failureType);
+ }
+
+ @Override
+ public WithFailures.Result<PCollection<String>, FailureT> expand(PCollection<InputT> input) {
+ return input.apply(
+ MapElements.into(TypeDescriptors.strings())
+ .via(
+ Contextful.fn(
+ (Contextful.Fn<InputT, String>) (input1, c) -> writeValue(input1),
+ Requirements.empty()))
+ .exceptionsInto(failureType)
+ .exceptionsVia(exceptionHandler));
+ }
+ }
+
+ /**
+ * A default handler that extracts information from an exception to a {@code Map<String, String>}
+ * and returns a {@link KV} where the key is the input element that failed processing, and the
+ * value is the map of exception attributes. It handles only {@code JsonProcessingException},
+ * other type of exceptions will be rethrown as {@code RuntimeException}.
+ *
+ * <p>The keys populated in the map are "className", "message", and "stackTrace" of the exception.
+ */
+ private static class DefaultExceptionAsMapHandler<InputT>
+ extends SimpleFunction<
+ WithFailures.ExceptionElement<InputT>, KV<InputT, Map<String, String>>> {
+ @Override
+ public KV<InputT, Map<String, String>> apply(WithFailures.ExceptionElement<InputT> f)
+ throws RuntimeException {
+ if (!(f.exception() instanceof JsonProcessingException)) {
+ throw new RuntimeException(f.exception());
+ }
+ return KV.of(
+ f.element(),
+ ImmutableMap.of(
+ "className", f.exception().getClass().getName(),
+ "message", f.exception().getMessage(),
+ "stackTrace", Arrays.toString(f.exception().getStackTrace())));
+ }
+ }
}
diff --git a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
index a5c55db..a92041b 100644
--- a/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
+++ b/sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
@@ -19,11 +19,23 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.Optional;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.Contextful;
+import org.apache.beam.sdk.transforms.InferableFunction;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ProcessFunction;
+import org.apache.beam.sdk.transforms.Requirements;
import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.WithFailures;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* {@link PTransform} for parsing JSON {@link String Strings}. Parse {@link PCollection} of {@link
@@ -55,6 +67,84 @@
return newTransform;
}
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, with the given type descriptor used for the failure collection but the
+ * exception handler yet to be specified using {@link
+ * ParseJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <NewFailureT> ParseJsonsWithFailures<NewFailureT> exceptionsInto(
+ TypeDescriptor<NewFailureT> failureTypeDescriptor) {
+ return new ParseJsonsWithFailures<>(null, failureTypeDescriptor);
+ }
+
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, passing the raised exception instance and the input element being processed
+ * through the given {@code exceptionHandler} and emitting the result to a failure collection.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ * json.apply(
+ * ParseJsons.of(MyPojo.class)
+ * .exceptionsVia(new WithFailures.ExceptionAsMapHandler<String>() {}));
+ *
+ * PCollection<MyPojo> output = result.output(); // valid POJOs
+ * PCollection<KV<String, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public <FailureT> ParseJsonsWithFailures<FailureT> exceptionsVia(
+ InferableFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler) {
+ return new ParseJsonsWithFailures<>(
+ exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, passing the raised exception instance and the input element being processed
+ * through the default exception handler {@link DefaultExceptionAsMapHandler} and emitting the
+ * result to a failure collection.
+ *
+ * <p>See {@link DefaultExceptionAsMapHandler} for more details about default handler behavior.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ * json.apply(
+ * ParseJsons.of(MyPojo.class)
+ * .exceptionsVia());
+ *
+ * PCollection<MyPojo> output = result.output(); // valid POJOs
+ * PCollection<KV<String, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+ public ParseJsonsWithFailures<KV<String, Map<String, String>>> exceptionsVia() {
+ DefaultExceptionAsMapHandler<String> exceptionHandler =
+ new DefaultExceptionAsMapHandler<String>() {};
+ return new ParseJsonsWithFailures<>(
+ exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+ }
+
+ private OutputT readValue(String input) throws IOException {
+ ObjectMapper mapper = Optional.ofNullable(customMapper).orElse(DEFAULT_MAPPER);
+ return mapper.readValue(input, outputClass);
+ }
+
@Override
public PCollection<OutputT> expand(PCollection<String> input) {
return input.apply(
@@ -63,8 +153,7 @@
@Override
public OutputT apply(String input) {
try {
- ObjectMapper mapper = Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
- return mapper.readValue(input, outputClass);
+ return readValue(input);
} catch (IOException e) {
throw new RuntimeException(
"Failed to parse a " + outputClass.getName() + " from JSON value: " + input,
@@ -73,4 +162,91 @@
}
}));
}
+
+ /** A {@code PTransform} that adds exception handling to {@link ParseJsons}. */
+ public class ParseJsonsWithFailures<FailureT>
+ extends PTransform<PCollection<String>, WithFailures.Result<PCollection<OutputT>, FailureT>> {
+ @Nullable
+ private InferableFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler;
+
+ @Nullable private final transient TypeDescriptor<FailureT> failureType;
+
+ ParseJsonsWithFailures(
+ InferableFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler,
+ TypeDescriptor<FailureT> failureType) {
+ this.exceptionHandler = exceptionHandler;
+ this.failureType = failureType;
+ }
+
+ /**
+ * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
+ * parsing elements, passing the raised exception instance and the input element being processed
+ * through the given {@code exceptionHandler} and emitting the result to a failure collection.
+ * It is supposed to be used along with {@link ParseJsons#exceptionsInto(TypeDescriptor)} and
+ * get lambda function as exception handler.
+ *
+ * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
+ * WithFailures.Result}.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ * json.apply(
+ * ParseJsons.of(MyPojo.class)
+ * .exceptionsInto(
+ * TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
+ * .exceptionsVia(
+ * f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ *
+ * PCollection<MyPojo> output = result.output(); // valid POJOs
+ * PCollection<KV<String, Map<String, String>>> failures = result.failures();
+ * }</pre>
+ */
+ public ParseJsonsWithFailures<FailureT> exceptionsVia(
+ ProcessFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler) {
+ return new ParseJsonsWithFailures<>(
+ new InferableFunction<WithFailures.ExceptionElement<String>, FailureT>(
+ exceptionHandler) {},
+ failureType);
+ }
+
+ @Override
+ public WithFailures.Result<PCollection<OutputT>, FailureT> expand(PCollection<String> input) {
+ return input.apply(
+ MapElements.into(new TypeDescriptor<OutputT>() {})
+ .via(
+ Contextful.fn(
+ (Contextful.Fn<String, OutputT>) (input1, c) -> readValue(input1),
+ Requirements.empty()))
+ .exceptionsInto(failureType)
+ .exceptionsVia(exceptionHandler));
+ }
+ }
+
+ /**
+ * A default handler that extracts information from an exception to a {@code Map<String, String>}
+ * and returns a {@link KV} where the key is the input element that failed processing, and the
+ * value is the map of exception attributes. It handles only {@code IOException}, other type of
+ * exceptions will be rethrown as {@code RuntimeException}.
+ *
+ * <p>The keys populated in the map are "className", "message", and "stackTrace" of the exception.
+ */
+ private static class DefaultExceptionAsMapHandler<OutputT>
+ extends SimpleFunction<
+ WithFailures.ExceptionElement<OutputT>, KV<OutputT, Map<String, String>>> {
+ @Override
+ public KV<OutputT, Map<String, String>> apply(WithFailures.ExceptionElement<OutputT> f)
+ throws RuntimeException {
+ if (!(f.exception() instanceof IOException)) {
+ throw new RuntimeException(f.exception());
+ }
+ return KV.of(
+ f.element(),
+ ImmutableMap.of(
+ "className", f.exception().getClass().getName(),
+ "message", f.exception().getMessage(),
+ "stackTrace", Arrays.toString(f.exception().getStackTrace())));
+ }
+ }
}
diff --git a/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java b/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java
index bd502e0..4414465 100644
--- a/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java
+++ b/sdks/java/extensions/jackson/src/test/java/org/apache/beam/sdk/extensions/jackson/JacksonTransformsTest.java
@@ -17,25 +17,38 @@
*/
package org.apache.beam.sdk.extensions.jackson;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.hasKey;
+import static org.hamcrest.Matchers.hasSize;
+import static org.junit.Assert.assertEquals;
+
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import java.io.Serializable;
import java.util.Arrays;
import java.util.List;
+import java.util.Map;
import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.MapCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.WithFailures;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Rule;
import org.junit.Test;
/** Test Jackson transforms {@link ParseJsons} and {@link AsJsons}. */
-public class JacksonTransformsTest {
+public class JacksonTransformsTest implements Serializable {
private static final List<String> VALID_JSONS =
Arrays.asList("{\"myString\":\"abc\",\"myInt\":3}", "{\"myString\":\"def\",\"myInt\":4}");
@@ -51,6 +64,9 @@
private static final List<MyPojo> POJOS =
Arrays.asList(new MyPojo("abc", 3), new MyPojo("def", 4));
+ private static final List<MyInvalidPojo> INVALID_POJOS =
+ Arrays.asList(new MyInvalidPojo("aaa", 5), new MyInvalidPojo("bbb", 6));
+
private static final List<MyEmptyBean> EMPTY_BEANS =
Arrays.asList(new MyEmptyBean("abc", 3), new MyEmptyBean("def", 4));
@@ -82,6 +98,83 @@
pipeline.run();
}
+ @Test
+ public void testParsingInvalidJsonsWithFailuresDefaultHandler() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(ParseJsons.of(MyPojo.class).exceptionsVia());
+
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testParsingInvalidJsonsWithFailuresAsMap() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(
+ ParseJsons.of(MyPojo.class)
+ .exceptionsVia(new WithFailures.ExceptionAsMapHandler<String>() {}));
+
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testParsingInvalidJsonsWithFailuresSimpleFunction() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, String>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(
+ ParseJsons.of(MyPojo.class)
+ .exceptionsVia(
+ new SimpleFunction<
+ WithFailures.ExceptionElement<String>, KV<String, String>>() {
+ @Override
+ public KV<String, String> apply(
+ WithFailures.ExceptionElement<String> failure) {
+ return KV.of(
+ failure.element(),
+ failure.exception().getClass().getCanonicalName());
+ }
+ }));
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testParsingInvalidJsonsWithFailuresLambda() {
+ WithFailures.Result<PCollection<MyPojo>, KV<String, String>> result =
+ pipeline
+ .apply(Create.of(Iterables.concat(VALID_JSONS, INVALID_JSONS)))
+ .apply(
+ ParseJsons.of(MyPojo.class)
+ .exceptionsInto(
+ TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
+ .exceptionsVia(
+ f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ result.output().setCoder(SerializableCoder.of(MyPojo.class));
+
+ PAssert.that(result.output()).containsInAnyOrder(POJOS);
+ assertParsingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
@Test(expected = Pipeline.PipelineExecutionException.class)
public void failParsingWithoutCustomMapper() {
PCollection<MyPojo> output =
@@ -150,6 +243,99 @@
pipeline.run();
}
+ @Test
+ public void testWritingInvalidJsonsWithFailuresDefaultHandler() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(AsJsons.of(MyPojo.class).exceptionsVia());
+
+ result.output().setCoder(StringUtf8Coder.of());
+
+ result
+ .failures()
+ .setCoder(
+ KvCoder.of(
+ SerializableCoder.of(MyPojo.class),
+ MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())));
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWritingInvalidJsonsWithFailuresAsMap() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(
+ AsJsons.of(MyPojo.class)
+ .exceptionsVia(new WithFailures.ExceptionAsMapHandler<MyPojo>() {}));
+
+ result.output().setCoder(StringUtf8Coder.of());
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorMapHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWritingInvalidJsonsWithFailuresSimpleFunction() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, String>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(
+ AsJsons.of(MyPojo.class)
+ .exceptionsVia(
+ new SimpleFunction<
+ WithFailures.ExceptionElement<MyPojo>, KV<MyPojo, String>>() {
+ @Override
+ public KV<MyPojo, String> apply(
+ WithFailures.ExceptionElement<MyPojo> failure) {
+ return KV.of(
+ failure.element(),
+ failure.exception().getClass().getCanonicalName());
+ }
+ }));
+ result.output().setCoder(StringUtf8Coder.of());
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testWritingInvalidJsonsWithFailuresLambda() {
+ WithFailures.Result<PCollection<String>, KV<MyPojo, String>> result =
+ pipeline
+ .apply(
+ Create.of(Iterables.concat(POJOS, INVALID_POJOS))
+ .withCoder(SerializableCoder.of(MyPojo.class)))
+ .apply(
+ AsJsons.of(MyPojo.class)
+ .exceptionsInto(
+ TypeDescriptors.kvs(
+ TypeDescriptor.of(MyPojo.class), TypeDescriptors.strings()))
+ .exceptionsVia(
+ f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
+ result.output().setCoder(StringUtf8Coder.of());
+
+ PAssert.that(result.output()).containsInAnyOrder(VALID_JSONS);
+ assertWritingWithErrorFunctionHandler(result);
+
+ pipeline.run();
+ }
+
/** Pojo for tests. */
@SuppressWarnings({"WeakerAccess", "unused"})
public static class MyPojo implements Serializable {
@@ -238,4 +424,84 @@
return result;
}
}
+
+ /** Pojo for tests. */
+ @SuppressWarnings({"WeakerAccess", "unused"})
+ public static class MyInvalidPojo extends MyPojo {
+ public MyInvalidPojo(String myString, int myInt) {
+ super(myString, myInt);
+ }
+
+ @Override
+ public String getMyString() {
+ throw new RuntimeException("Unknown error!");
+ }
+ }
+
+ private void assertParsingWithErrorMapHandler(
+ WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result) {
+ PAssert.that(result.failures())
+ .satisfies(
+ kv -> {
+ for (KV<String, Map<String, String>> entry : kv) {
+ if (entry.getKey().equals(INVALID_JSONS.get(0))) {
+ assertEquals(
+ "com.fasterxml.jackson.core.JsonParseException",
+ entry.getValue().get("className"));
+ } else if (entry.getKey().equals(INVALID_JSONS.get(1))) {
+ assertEquals(
+ "com.fasterxml.jackson.core.io.JsonEOFException",
+ entry.getValue().get("className"));
+ } else if (entry.getKey().equals(INVALID_JSONS.get(2))) {
+ assertEquals(
+ "com.fasterxml.jackson.databind.exc.MismatchedInputException",
+ entry.getValue().get("className"));
+ } else {
+ throw new AssertionError(
+ "Unexpected key is found in failures result: \"" + entry.getKey() + "\"");
+ }
+ assertThat(entry.getValue().entrySet(), hasSize(3));
+ assertThat(entry.getValue(), hasKey("stackTrace"));
+ assertThat(entry.getValue(), hasKey("message"));
+ }
+
+ return null;
+ });
+ }
+
+ private void assertParsingWithErrorFunctionHandler(
+ WithFailures.Result<PCollection<MyPojo>, KV<String, String>> result) {
+ PAssert.that(result.failures())
+ .containsInAnyOrder(
+ KV.of(INVALID_JSONS.get(0), "com.fasterxml.jackson.core.JsonParseException"),
+ KV.of(INVALID_JSONS.get(1), "com.fasterxml.jackson.core.io.JsonEOFException"),
+ KV.of(
+ INVALID_JSONS.get(2),
+ "com.fasterxml.jackson.databind.exc.MismatchedInputException"));
+ }
+
+ private void assertWritingWithErrorMapHandler(
+ WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result) {
+ PAssert.that(result.failures())
+ .satisfies(
+ kv -> {
+ for (KV<MyPojo, Map<String, String>> entry : kv) {
+ assertThat(entry.getValue().entrySet(), hasSize(3));
+ assertThat(entry.getValue(), hasKey("stackTrace"));
+ assertThat(entry.getValue(), hasKey("message"));
+ assertEquals(
+ "com.fasterxml.jackson.databind.JsonMappingException",
+ entry.getValue().get("className"));
+ }
+ return null;
+ });
+ }
+
+ private void assertWritingWithErrorFunctionHandler(
+ WithFailures.Result<PCollection<String>, KV<MyPojo, String>> result) {
+ PAssert.that(result.failures())
+ .containsInAnyOrder(
+ KV.of(INVALID_POJOS.get(0), "com.fasterxml.jackson.databind.JsonMappingException"),
+ KV.of(INVALID_POJOS.get(1), "com.fasterxml.jackson.databind.JsonMappingException"));
+ }
}
diff --git a/sdks/java/extensions/join-library/build.gradle b/sdks/java/extensions/join-library/build.gradle
index 93b6b45..c3c79e9 100644
--- a/sdks/java/extensions/join-library/build.gradle
+++ b/sdks/java/extensions/join-library/build.gradle
@@ -17,15 +17,15 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.joinlibrary')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Join library"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java b/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
index 73386b2..3861b9c 100644
--- a/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
+++ b/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.joinlibrary;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.transforms.DoFn;
diff --git a/sdks/java/extensions/kryo/build.gradle b/sdks/java/extensions/kryo/build.gradle
index e949b8b..24c8e0c 100644
--- a/sdks/java/extensions/kryo/build.gradle
+++ b/sdks/java/extensions/kryo/build.gradle
@@ -23,8 +23,9 @@
}
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.kryo',
exportJavadoc: false,
- shadowClosure: DEFAULT_SHADOW_CLOSURE << {
+ shadowClosure: {
dependencies {
include(dependency('com.esotericsoftware:.*'))
include(dependency('org.ow2.asm:asm'))
diff --git a/sdks/java/extensions/kryo/src/main/java/org/apache/beam/sdk/extensions/kryo/KryoCoderProvider.java b/sdks/java/extensions/kryo/src/main/java/org/apache/beam/sdk/extensions/kryo/KryoCoderProvider.java
index a25b5cb..11e9c5a 100644
--- a/sdks/java/extensions/kryo/src/main/java/org/apache/beam/sdk/extensions/kryo/KryoCoderProvider.java
+++ b/sdks/java/extensions/kryo/src/main/java/org/apache/beam/sdk/extensions/kryo/KryoCoderProvider.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Implementation of {@link CoderProvider}, which provides {@link KryoCoder} for any type registered
diff --git a/sdks/java/extensions/protobuf/build.gradle b/sdks/java/extensions/protobuf/build.gradle
index ece59da..2f068b2 100644
--- a/sdks/java/extensions/protobuf/build.gradle
+++ b/sdks/java/extensions/protobuf/build.gradle
@@ -17,16 +17,16 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.protobuf')
applyGrpcNature()
description = "Apache Beam :: SDKs :: Java :: Extensions :: Protobuf"
ext.summary = "Add support to Apache Beam for Google Protobuf."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.protobuf_java
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.protobuf_java
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
index d8dba2d..effae47 100644
--- a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A {@link Coder} for {@link ByteString} objects based on their encoded Protocol Buffer form.
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java
index 9ee30c6..e2a919a 100644
--- a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoder.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.protobuf;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.protobuf.ExtensionRegistry;
import com.google.protobuf.Message;
@@ -43,8 +43,8 @@
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A {@link Coder} using Google Protocol Buffers binary format. {@link ProtoCoder} supports both
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufCoderProviderRegistrar.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufCoderProviderRegistrar.java
index a1da20b..9c056ee 100644
--- a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufCoderProviderRegistrar.java
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufCoderProviderRegistrar.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.CoderProviderRegistrar;
import org.apache.beam.sdk.coders.CoderProviders;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A {@link CoderProviderRegistrar} for standard types used with Google Protobuf. */
@AutoService(CoderProviderRegistrar.class)
diff --git a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java
index 8428e57..791ed63 100644
--- a/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java
+++ b/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtil.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.protobuf;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.protobuf.Descriptors.Descriptor;
import com.google.protobuf.Descriptors.FieldDescriptor;
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoderTest.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoderTest.java
index 0d21ec7..744de901 100644
--- a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoderTest.java
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoderTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
index 76069eb..04ed9a6 100644
--- a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoCoderTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtilTest.java b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtilTest.java
index 40cd2d6..ac2971b 100644
--- a/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtilTest.java
+++ b/sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtobufUtilTest.java
@@ -36,8 +36,8 @@
import org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageC;
import org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.MessageWithMap;
import org.apache.beam.sdk.extensions.protobuf.Proto2CoderTestMessages.ReferencesMessageWithMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/extensions/sketching/build.gradle b/sdks/java/extensions/sketching/build.gradle
index 7932d07..1f403ee 100644
--- a/sdks/java/extensions/sketching/build.gradle
+++ b/sdks/java/extensions/sketching/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sketching')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Sketching"
@@ -25,16 +25,16 @@
def tdigest_version = "3.2"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow "com.clearspring.analytics:stream:$streamlib_version"
- shadow "com.tdunning:t-digest:$tdigest_version"
- shadow library.java.slf4j_api
- shadowTest library.java.avro
- shadowTest library.java.commons_lang3
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.junit
- testRuntimeOnly project(path: ":runners:direct-java")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile "com.clearspring.analytics:stream:$streamlib_version"
+ compile "com.tdunning:t-digest:$tdigest_version"
+ compile library.java.slf4j_api
+ testCompile library.java.avro
+ testCompile library.java.commons_lang3
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.junit
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
index 6822350..f279670 100644
--- a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
+++ b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/ApproximateDistinct.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sketching;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.clearspring.analytics.stream.cardinality.CardinalityMergeException;
import com.clearspring.analytics.stream.cardinality.HyperLogLogPlus;
@@ -200,6 +200,11 @@
*
* }</pre>
*
+ * Consider using the {@code HllCount.Init} transform in the {@code zetasketch} extension module if
+ * you need to create sketches compatible with Google Cloud BigQuery. For more details about using
+ * {@code HllCount} and the {@code zetasketch} extension module, see
+ * https://s.apache.org/hll-in-beam#bookmark=id.v6chsij1ixo7
+ *
* <p><b>Warning: this class is experimental.</b> Its API is subject to change in future versions of
* Beam. For example, it may be merged with the {@link
* org.apache.beam.sdk.transforms.ApproximateUnique} transform.
diff --git a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
index e6a2817..3586c83 100644
--- a/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
+++ b/sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
/**
* {@code PTransform}s to compute the estimate frequency of each element in a stream.
diff --git a/sdks/java/extensions/sorter/build.gradle b/sdks/java/extensions/sorter/build.gradle
index 7f0c72c..94cbba5 100644
--- a/sdks/java/extensions/sorter/build.gradle
+++ b/sdks/java/extensions/sorter/build.gradle
@@ -17,18 +17,18 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sorter')
description = "Apache Beam :: SDKs :: Java :: Extensions :: Sorter"
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.vendored_guava_20_0
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_guava_26_0_jre
provided library.java.hadoop_mapreduce_client_core
provided library.java.hadoop_common
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.mockito_core
testCompile library.java.junit
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/BufferedExternalSorter.java b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/BufferedExternalSorter.java
index 6032b00..e9640de 100644
--- a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/BufferedExternalSorter.java
+++ b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/BufferedExternalSorter.java
@@ -17,10 +17,11 @@
*/
package org.apache.beam.sdk.extensions.sorter;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.io.Serializable;
+import org.apache.beam.sdk.extensions.sorter.ExternalSorter.Options.SorterType;
import org.apache.beam.sdk.values.KV;
/**
@@ -29,17 +30,19 @@
*/
public class BufferedExternalSorter implements Sorter {
public static Options options() {
- return new Options("/tmp", 100);
+ return new Options("/tmp", 100, SorterType.HADOOP);
}
/** Contains configuration for the sorter. */
public static class Options implements Serializable {
private final String tempLocation;
private final int memoryMB;
+ private final SorterType sorterType;
- private Options(String tempLocation, int memoryMB) {
+ private Options(String tempLocation, int memoryMB, SorterType sorterType) {
this.tempLocation = tempLocation;
this.memoryMB = memoryMB;
+ this.sorterType = sorterType;
}
/** Sets the path to a temporary location where the sorter writes intermediate files. */
@@ -48,7 +51,7 @@
!tempLocation.startsWith("gs://"),
"BufferedExternalSorter does not support GCS temporary location");
- return new Options(tempLocation, memoryMB);
+ return new Options(tempLocation, memoryMB, sorterType);
}
/** Returns the configured temporary location. */
@@ -66,13 +69,23 @@
// Hadoop's external sort stores the number of available memory bytes in an int, this prevents
// overflow
checkArgument(memoryMB < 2048, "memoryMB must be less than 2048");
- return new Options(tempLocation, memoryMB);
+ return new Options(tempLocation, memoryMB, sorterType);
}
/** Returns the configured size of the memory buffer. */
public int getMemoryMB() {
return memoryMB;
}
+
+ /** Sets the external sorter type. */
+ public Options withExternalSorterType(SorterType sorterType) {
+ return new Options(tempLocation, memoryMB, sorterType);
+ }
+
+ /** Returns the external sorter type. */
+ public SorterType getExternalSorterType() {
+ return sorterType;
+ }
}
private final ExternalSorter externalSorter;
@@ -89,6 +102,7 @@
ExternalSorter.Options externalSorterOptions = new ExternalSorter.Options();
externalSorterOptions.setMemoryMB(options.getMemoryMB());
externalSorterOptions.setTempLocation(options.getTempLocation());
+ externalSorterOptions.setSorterType(options.getExternalSorterType());
InMemorySorter.Options inMemorySorterOptions = new InMemorySorter.Options();
inMemorySorterOptions.setMemoryMB(options.getMemoryMB());
diff --git a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/ExternalSorter.java b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/ExternalSorter.java
index 192b8ca..1595d5a 100644
--- a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/ExternalSorter.java
+++ b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/ExternalSorter.java
@@ -17,54 +17,25 @@
*/
package org.apache.beam.sdk.extensions.sorter;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.DataInputStream;
-import java.io.DataOutputStream;
-import java.io.IOException;
import java.io.Serializable;
-import java.util.Iterator;
-import java.util.NoSuchElementException;
-import java.util.UUID;
-import javax.annotation.Nonnull;
-import org.apache.beam.sdk.values.KV;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.io.SequenceFile;
-import org.apache.hadoop.io.SequenceFile.CompressionType;
-import org.apache.hadoop.io.SequenceFile.Sorter.RawKeyValueIterator;
-import org.apache.hadoop.io.SequenceFile.Writer;
-import org.apache.hadoop.mapred.JobConf;
-/** Does an external sort of the provided values using Hadoop's {@link SequenceFile}. */
-class ExternalSorter implements Sorter {
- private final Options options;
-
- /** Whether {@link #sort()} was already called. */
- private boolean sortCalled = false;
-
- /** SequenceFile Writer for writing all input data to a file. */
- private Writer writer;
-
- /** Sorter used to sort the input file. */
- private SequenceFile.Sorter sorter;
-
- /** Temporary directory for input and intermediate files. */
- private Path tempDir;
-
- /** The list of input files to be sorted. */
- private Path[] paths;
-
- private boolean initialized = false;
+/** Does an external sort of the provided values. */
+public abstract class ExternalSorter implements Sorter {
+ protected final Options options;
/** {@link Options} contains configuration of the sorter. */
public static class Options implements Serializable {
private String tempLocation = "/tmp";
private int memoryMB = 100;
+ private SorterType sorterType = SorterType.HADOOP;
+
+ /** Sorter type. */
+ public enum SorterType {
+ HADOOP,
+ NATIVE
+ }
/** Sets the path to a temporary location where the sorter writes intermediate files. */
public Options setTempLocation(String tempLocation) {
@@ -98,144 +69,27 @@
public int getMemoryMB() {
return memoryMB;
}
+
+ /** Sets the sorter type. */
+ public Options setSorterType(SorterType sorterType) {
+ this.sorterType = sorterType;
+ return this;
+ }
+
+ /** Returns the sorter type. */
+ public SorterType getSorterType() {
+ return sorterType;
+ }
}
/** Returns a {@link Sorter} configured with the given {@link Options}. */
public static ExternalSorter create(Options options) {
- return new ExternalSorter(options);
+ return options.getSorterType() == Options.SorterType.HADOOP
+ ? HadoopExternalSorter.create(options)
+ : NativeExternalSorter.create(options);
}
- @Override
- public void add(KV<byte[], byte[]> record) throws IOException {
- checkState(!sortCalled, "Records can only be added before sort()");
-
- initHadoopSorter();
-
- BytesWritable key = new BytesWritable(record.getKey());
- BytesWritable value = new BytesWritable(record.getValue());
-
- writer.append(key, value);
- }
-
- @Override
- public Iterable<KV<byte[], byte[]>> sort() throws IOException {
- checkState(!sortCalled, "sort() can only be called once.");
- sortCalled = true;
-
- initHadoopSorter();
-
- writer.close();
-
- return new SortedRecordsIterable();
- }
-
- private ExternalSorter(Options options) {
+ ExternalSorter(Options options) {
this.options = options;
}
-
- /**
- * Initializes the hadoop sorter. Does some local file system setup, and is somewhat expensive
- * (~20 ms on local machine). Only executed when necessary.
- */
- private void initHadoopSorter() throws IOException {
- if (!initialized) {
- tempDir = new Path(options.getTempLocation(), "tmp" + UUID.randomUUID().toString());
- paths = new Path[] {new Path(tempDir, "test.seq")};
-
- JobConf conf = new JobConf();
- // Sets directory for intermediate files created during merge of merge sort
- conf.set("io.seqfile.local.dir", tempDir.toUri().getPath());
-
- writer =
- SequenceFile.createWriter(
- conf,
- Writer.valueClass(BytesWritable.class),
- Writer.keyClass(BytesWritable.class),
- Writer.file(paths[0]),
- Writer.compression(CompressionType.NONE));
-
- FileSystem fs = FileSystem.getLocal(conf);
- // Directory has to exist for Hadoop to recognize it as deletable on exit
- fs.mkdirs(tempDir);
- fs.deleteOnExit(tempDir);
-
- sorter =
- new SequenceFile.Sorter(
- fs, new BytesWritable.Comparator(), BytesWritable.class, BytesWritable.class, conf);
- sorter.setMemory(options.getMemoryMB() * 1024 * 1024);
-
- initialized = true;
- }
- }
-
- /** An {@link Iterable} producing the iterators over sorted data. */
- private class SortedRecordsIterable implements Iterable<KV<byte[], byte[]>> {
- @Nonnull
- @Override
- public Iterator<KV<byte[], byte[]>> iterator() {
- return new SortedRecordsIterator();
- }
- }
-
- /** An {@link Iterator} producing the sorted data. */
- private class SortedRecordsIterator implements Iterator<KV<byte[], byte[]>> {
- private RawKeyValueIterator iterator;
-
- /** Next {@link KV} to return from {@link #next()}. */
- private KV<byte[], byte[]> nextKV;
-
- SortedRecordsIterator() {
- try {
- this.iterator = sorter.sortAndIterate(paths, tempDir, false);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
-
- nextKV = KV.of(null, null); // A dummy value that will be overwritten by next().
- next();
- }
-
- @Override
- public boolean hasNext() {
- return nextKV != null;
- }
-
- @Override
- public KV<byte[], byte[]> next() {
- if (nextKV == null) {
- throw new NoSuchElementException();
- }
-
- KV<byte[], byte[]> current = nextKV;
-
- try {
- if (iterator.next()) {
- // Parse key from DataOutputBuffer.
- ByteArrayInputStream keyStream = new ByteArrayInputStream(iterator.getKey().getData());
- BytesWritable key = new BytesWritable();
- key.readFields(new DataInputStream(keyStream));
-
- // Parse value from ValueBytes.
- ByteArrayOutputStream valOutStream = new ByteArrayOutputStream();
- iterator.getValue().writeUncompressedBytes(new DataOutputStream(valOutStream));
- ByteArrayInputStream valInStream = new ByteArrayInputStream(valOutStream.toByteArray());
- BytesWritable value = new BytesWritable();
- value.readFields(new DataInputStream(valInStream));
-
- nextKV = KV.of(key.copyBytes(), value.copyBytes());
- } else {
- nextKV = null;
- }
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
-
- return current;
- }
-
- @Override
- public void remove() {
- throw new UnsupportedOperationException("Iterator does not support remove");
- }
- }
}
diff --git a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/HadoopExternalSorter.java b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/HadoopExternalSorter.java
new file mode 100644
index 0000000..c9c8f16
--- /dev/null
+++ b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/HadoopExternalSorter.java
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sorter;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.UUID;
+import javax.annotation.Nonnull;
+import org.apache.beam.sdk.values.KV;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.SequenceFile;
+import org.apache.hadoop.io.SequenceFile.CompressionType;
+import org.apache.hadoop.io.SequenceFile.Sorter.RawKeyValueIterator;
+import org.apache.hadoop.io.SequenceFile.Writer;
+import org.apache.hadoop.mapred.JobConf;
+
+/** Does an external sort of the provided values using Hadoop's {@link SequenceFile}. */
+class HadoopExternalSorter extends ExternalSorter {
+
+ /** Whether {@link #sort()} was already called. */
+ private boolean sortCalled = false;
+
+ /** SequenceFile Writer for writing all input data to a file. */
+ private Writer writer;
+
+ /** Sorter used to sort the input file. */
+ private SequenceFile.Sorter sorter;
+
+ /** Temporary directory for input and intermediate files. */
+ private Path tempDir;
+
+ /** The list of input files to be sorted. */
+ private Path[] paths;
+
+ private boolean initialized = false;
+
+ /** Returns a {@link Sorter} configured with the given {@link Options}. */
+ public static HadoopExternalSorter create(Options options) {
+ return new HadoopExternalSorter(options);
+ }
+
+ @Override
+ public void add(KV<byte[], byte[]> record) throws IOException {
+ checkState(!sortCalled, "Records can only be added before sort()");
+
+ initHadoopSorter();
+
+ BytesWritable key = new BytesWritable(record.getKey());
+ BytesWritable value = new BytesWritable(record.getValue());
+
+ writer.append(key, value);
+ }
+
+ @Override
+ public Iterable<KV<byte[], byte[]>> sort() throws IOException {
+ checkState(!sortCalled, "sort() can only be called once.");
+ sortCalled = true;
+
+ initHadoopSorter();
+
+ writer.close();
+
+ return new SortedRecordsIterable();
+ }
+
+ private HadoopExternalSorter(Options options) {
+ super(options);
+ }
+
+ /**
+ * Initializes the hadoop sorter. Does some local file system setup, and is somewhat expensive
+ * (~20 ms on local machine). Only executed when necessary.
+ */
+ private void initHadoopSorter() throws IOException {
+ if (!initialized) {
+ tempDir = new Path(options.getTempLocation(), "tmp" + UUID.randomUUID().toString());
+ paths = new Path[] {new Path(tempDir, "test.seq")};
+
+ JobConf conf = new JobConf();
+ // Sets directory for intermediate files created during merge of merge sort
+ conf.set("io.seqfile.local.dir", tempDir.toUri().getPath());
+
+ writer =
+ SequenceFile.createWriter(
+ conf,
+ Writer.valueClass(BytesWritable.class),
+ Writer.keyClass(BytesWritable.class),
+ Writer.file(paths[0]),
+ Writer.compression(CompressionType.NONE));
+
+ FileSystem fs = FileSystem.getLocal(conf);
+ // Directory has to exist for Hadoop to recognize it as deletable on exit
+ fs.mkdirs(tempDir);
+ fs.deleteOnExit(tempDir);
+
+ sorter =
+ new SequenceFile.Sorter(
+ fs, new BytesWritable.Comparator(), BytesWritable.class, BytesWritable.class, conf);
+ sorter.setMemory(options.getMemoryMB() * 1024 * 1024);
+
+ initialized = true;
+ }
+ }
+
+ /** An {@link Iterable} producing the iterators over sorted data. */
+ private class SortedRecordsIterable implements Iterable<KV<byte[], byte[]>> {
+ @Nonnull
+ @Override
+ public Iterator<KV<byte[], byte[]>> iterator() {
+ return new SortedRecordsIterator();
+ }
+ }
+
+ /** An {@link Iterator} producing the sorted data. */
+ private class SortedRecordsIterator implements Iterator<KV<byte[], byte[]>> {
+ private RawKeyValueIterator iterator;
+
+ /** Next {@link KV} to return from {@link #next()}. */
+ private KV<byte[], byte[]> nextKV;
+
+ SortedRecordsIterator() {
+ try {
+ this.iterator = sorter.sortAndIterate(paths, tempDir, false);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ nextKV = KV.of(null, null); // A dummy value that will be overwritten by next().
+ next();
+ }
+
+ @Override
+ public boolean hasNext() {
+ return nextKV != null;
+ }
+
+ @Override
+ public KV<byte[], byte[]> next() {
+ if (nextKV == null) {
+ throw new NoSuchElementException();
+ }
+
+ KV<byte[], byte[]> current = nextKV;
+
+ try {
+ if (iterator.next()) {
+ // Parse key from DataOutputBuffer.
+ ByteArrayInputStream keyStream = new ByteArrayInputStream(iterator.getKey().getData());
+ BytesWritable key = new BytesWritable();
+ key.readFields(new DataInputStream(keyStream));
+
+ // Parse value from ValueBytes.
+ ByteArrayOutputStream valOutStream = new ByteArrayOutputStream();
+ iterator.getValue().writeUncompressedBytes(new DataOutputStream(valOutStream));
+ ByteArrayInputStream valInStream = new ByteArrayInputStream(valOutStream.toByteArray());
+ BytesWritable value = new BytesWritable();
+ value.readFields(new DataInputStream(valInStream));
+
+ nextKV = KV.of(key.copyBytes(), value.copyBytes());
+ } else {
+ nextKV = null;
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+
+ return current;
+ }
+
+ @Override
+ public void remove() {
+ throw new UnsupportedOperationException("Iterator does not support remove");
+ }
+ }
+}
diff --git a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/InMemorySorter.java b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/InMemorySorter.java
index 8aa280d..8e67963 100644
--- a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/InMemorySorter.java
+++ b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/InMemorySorter.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.extensions.sorter;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
/**
* Sorts {@code <key, value>} pairs in memory. Based on the configured size of the memory buffer,
diff --git a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeExternalSorter.java b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeExternalSorter.java
new file mode 100644
index 0000000..d54476d
--- /dev/null
+++ b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeExternalSorter.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sorter;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import org.apache.beam.sdk.values.KV;
+
+/** Does an external sort of the provided values. */
+class NativeExternalSorter extends ExternalSorter {
+
+ /** Whether {@link #sort()} was already called. */
+ private boolean sortCalled = false;
+
+ /** Sorter used to sort the input. */
+ private NativeFileSorter sorter;
+
+ private boolean initialized = false;
+
+ /** Returns a {@link Sorter} configured with the given {@link Options}. */
+ public static NativeExternalSorter create(Options options) {
+ return new NativeExternalSorter(options);
+ }
+
+ @Override
+ public void add(KV<byte[], byte[]> record) throws IOException {
+ checkState(!sortCalled, "Records can only be added before sort()");
+
+ initSorter();
+
+ sorter.add(record.getKey(), record.getValue());
+ }
+
+ @Override
+ public Iterable<KV<byte[], byte[]>> sort() throws IOException {
+ checkState(!sortCalled, "sort() can only be called once.");
+ sortCalled = true;
+
+ initSorter();
+
+ return sorter.sort();
+ }
+
+ private NativeExternalSorter(Options options) {
+ super(options);
+ }
+
+ /**
+ * Initializes the sorter. Does some local file system setup, and is somewhat expensive (~20 ms on
+ * local machine). Only executed when necessary.
+ */
+ private void initSorter() throws IOException {
+ if (!initialized) {
+ sorter =
+ new NativeFileSorter(
+ Paths.get(options.getTempLocation()), (long) options.getMemoryMB() * 1024 * 1024);
+ initialized = true;
+ }
+ }
+}
diff --git a/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
new file mode 100644
index 0000000..7fd8015
--- /dev/null
+++ b/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sorter;
+
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.EOFException;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * External Sorter based on <a
+ * href="https://github.com/lemire/externalsortinginjava">lemire/externalsortinginjava</a>.
+ */
+class NativeFileSorter {
+
+ private static final Logger LOG = LoggerFactory.getLogger(NativeFileSorter.class);
+
+ private static final int MAX_TEMP_FILES = 1024;
+ private static final long OBJECT_OVERHEAD = getObjectOverhead();
+
+ private static final Comparator<byte[]> COMPARATOR = UnsignedBytes.lexicographicalComparator();
+ private static final Comparator<KV<byte[], byte[]>> KV_COMPARATOR =
+ (x, y) -> COMPARATOR.compare(x.getKey(), y.getKey());
+ private static final ByteArrayCoder CODER = ByteArrayCoder.of();
+
+ private final Path tempDir;
+ private final long maxMemory;
+ private final File dataFile;
+ private final OutputStream dataStream;
+
+ private boolean sortCalled = false;
+
+ /** Create a new file sorter. */
+ public NativeFileSorter(Path tempDir, long maxMemory) throws IOException {
+ this.tempDir = tempDir;
+ this.maxMemory = maxMemory;
+
+ this.dataFile = Files.createTempFile(tempDir, "input", "seq").toFile();
+ this.dataStream = new BufferedOutputStream(new FileOutputStream(dataFile));
+ dataFile.deleteOnExit();
+
+ LOG.debug("Created input file {}", dataFile);
+ }
+
+ /**
+ * Adds a given record to the sorter.
+ *
+ * <p>Records can only be added before calling {@link #sort()}.
+ */
+ public void add(byte[] key, byte[] value) throws IOException {
+ Preconditions.checkState(!sortCalled, "Records can only be added before sort()");
+ CODER.encode(key, dataStream);
+ CODER.encode(value, dataStream);
+ }
+
+ /**
+ * Sorts the added elements and returns an {@link Iterable} over the sorted elements.
+ *
+ * <p>Can be called at most once.
+ */
+ public Iterable<KV<byte[], byte[]>> sort() throws IOException {
+ Preconditions.checkState(!sortCalled, "sort() can only be called once.");
+ sortCalled = true;
+
+ dataStream.close();
+
+ return mergeSortedFiles(sortInBatch());
+ }
+
+ ////////////////////////////////////////////////////////////////////////////////
+
+ /**
+ * Loads the file by blocks of records, sorts in memory, and writes the result to temporary files
+ * that have to be merged later.
+ */
+ private List<File> sortInBatch() throws IOException {
+ final long fileSize = Files.size(dataFile.toPath());
+ final long memory = maxMemory > 0 ? maxMemory : estimateAvailableMemory();
+ final long blockSize = estimateBestBlockSize(fileSize, memory); // in bytes
+ LOG.debug(
+ "Sort in batch with fileSize: {}, memory: {}, blockSize: {}", fileSize, memory, blockSize);
+
+ final List<File> files = new ArrayList<>();
+ InputStream inputStream = new BufferedInputStream(new FileInputStream(dataFile));
+ try {
+ final List<KV<byte[], byte[]>> tempList = new ArrayList<>();
+ KV<byte[], byte[]> kv = KV.of(null, null);
+ while (kv != null) {
+ long currentBlockSize = 0;
+ while ((currentBlockSize < blockSize) && (kv = readKeyValue(inputStream)) != null) {
+ // as long as you have enough memory
+ tempList.add(kv);
+ currentBlockSize += estimateSizeOf(kv);
+ }
+ files.add(sortAndSave(tempList));
+ tempList.clear();
+ }
+ } finally {
+ inputStream.close();
+ }
+ return files;
+ }
+
+ /** Sort a list and save it to a temporary file. */
+ private File sortAndSave(List<KV<byte[], byte[]>> tempList) throws IOException {
+ final File tempFile = Files.createTempFile(tempDir, "sort", "seq").toFile();
+ tempFile.deleteOnExit();
+ LOG.debug("Sort and save {}", tempFile);
+
+ tempList.sort(KV_COMPARATOR);
+
+ OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(tempFile));
+ try {
+ for (KV<byte[], byte[]> kv : tempList) {
+ CODER.encode(kv.getKey(), outputStream);
+ CODER.encode(kv.getValue(), outputStream);
+ }
+ } finally {
+ outputStream.close();
+ }
+ return tempFile;
+ }
+
+ /** Merges a list of temporary flat files. */
+ private Iterable<KV<byte[], byte[]>> mergeSortedFiles(List<File> files) {
+ return () -> {
+ final List<Iterator<KV<byte[], byte[]>>> iterators = new ArrayList<>();
+ for (File file : files) {
+ try {
+ iterators.add(iterateFile(file));
+ } catch (FileNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+
+ return Iterators.mergeSorted(iterators, KV_COMPARATOR);
+ };
+ }
+
+ /** Creates an {@link Iterator} over the key-value pairs in a file. */
+ private Iterator<KV<byte[], byte[]>> iterateFile(File file) throws FileNotFoundException {
+ final InputStream inputStream = new BufferedInputStream(new FileInputStream(file));
+ return new Iterator<KV<byte[], byte[]>>() {
+ KV<byte[], byte[]> nextKv = readNext();
+
+ @Override
+ public boolean hasNext() {
+ return nextKv != null;
+ }
+
+ @Override
+ public KV<byte[], byte[]> next() {
+ KV<byte[], byte[]> r = nextKv;
+ nextKv = readNext();
+ return r;
+ }
+
+ private KV<byte[], byte[]> readNext() {
+ try {
+ return readKeyValue(inputStream);
+ } catch (EOFException e) {
+ return null;
+ } catch (IOException e) {
+ throw new IllegalStateException(e);
+ }
+ }
+ };
+ }
+
+ /** Reads the next key-value pair from a file. */
+ private KV<byte[], byte[]> readKeyValue(InputStream inputStream) throws IOException {
+ try {
+ final byte[] keyBytes = CODER.decode(inputStream);
+ final byte[] valueBytes = CODER.decode(inputStream);
+ return KV.of(keyBytes, valueBytes);
+ } catch (EOFException e) {
+ return null;
+ }
+ }
+
+ ////////////////////////////////////////////////////////////////////////////////
+
+ private int bufferSize(int numFiles) {
+ final long memory = maxMemory > 0 ? maxMemory : estimateAvailableMemory();
+ return (int) (memory / numFiles / 2);
+ }
+
+ /**
+ * This method calls the garbage collector and then returns the free memory. This avoids problems
+ * with applications where the GC hasn't reclaimed memory and reports no available memory.
+ */
+ @SuppressFBWarnings("DM_GC")
+ private static long estimateAvailableMemory() {
+ System.gc();
+ // http://stackoverflow.com/questions/12807797/java-get-available-memory
+ final Runtime r = Runtime.getRuntime();
+ final long allocatedMemory = r.totalMemory() - r.freeMemory();
+ return r.maxMemory() - allocatedMemory;
+ }
+
+ /**
+ * We divide the file into small blocks. If the blocks are too small, we shall create too many
+ * temporary files. If they are too big, we shall be using too much memory.
+ *
+ * @param sizeOfFile how much data (in bytes) can we expect
+ * @param maxMemory Maximum memory to use (in bytes)
+ */
+ private static long estimateBestBlockSize(final long sizeOfFile, final long maxMemory) {
+ // we don't want to open up much more than MAX_TEMP_FILES temporary files, better run out of
+ // memory first.
+ long blockSize = sizeOfFile / MAX_TEMP_FILES + (sizeOfFile % MAX_TEMP_FILES == 0 ? 0 : 1);
+
+ // on the other hand, we don't want to create many temporary files for naught. If blockSize is
+ // smaller than half the free memory, grow it.
+ if (blockSize < maxMemory / 2) {
+ blockSize = maxMemory / 2;
+ }
+ return blockSize;
+ }
+
+ private static long getObjectOverhead() {
+ // By default we assume 64 bit JVM
+ // (defensive approach since we will get larger estimations in case we are not sure)
+ boolean is64BitJvm = true;
+ // check the system property "sun.arch.data.model"
+ // not very safe, as it might not work for all JVM implementations
+ // nevertheless the worst thing that might happen is that the JVM is 32bit
+ // but we assume its 64bit, so we will be counting a few extra bytes per string object
+ // no harm done here since this is just an approximation.
+ String arch = System.getProperty("sun.arch.data.model");
+ if (arch != null && arch.contains("32")) {
+ // If exists and is 32 bit then we assume a 32bit JVM
+ is64BitJvm = false;
+ }
+ // The sizes below are a bit rough as we don't take into account
+ // advanced JVM options such as compressed oops
+ // however if our calculation is not accurate it'll be a bit over
+ // so there is no danger of an out of memory error because of this.
+ long objectHeader = is64BitJvm ? 16 : 8;
+ long arrayHeader = is64BitJvm ? 24 : 12;
+ long objectRef = is64BitJvm ? 8 : 4;
+ return objectHeader + (objectRef + arrayHeader) * 2;
+ }
+
+ private static long estimateSizeOf(KV<byte[], byte[]> kv) {
+ return kv.getKey().length + kv.getValue().length + OBJECT_OVERHEAD;
+ }
+}
diff --git a/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterBenchmark.java b/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterBenchmark.java
new file mode 100644
index 0000000..620711f
--- /dev/null
+++ b/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterBenchmark.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sorter;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.file.Files;
+import java.util.UUID;
+import org.apache.beam.sdk.extensions.sorter.ExternalSorter.Options.SorterType;
+import org.apache.beam.sdk.values.KV;
+
+/** {@link ExternalSorter} benchmarks. */
+public class ExternalSorterBenchmark {
+ private static final int N = 1000 * 1000; // 1m * (36 * 2) ~= 72MB per 1 million KVs
+
+ public static void main(String[] args) throws IOException {
+ File tempDirectory = Files.createTempDirectory("sorter").toFile();
+ tempDirectory.deleteOnExit();
+
+ ExternalSorter.Options options =
+ new ExternalSorter.Options().setMemoryMB(32).setTempLocation(tempDirectory.toString());
+
+ options.setSorterType(SorterType.HADOOP);
+ benchmark(ExternalSorter.create(options));
+
+ options.setSorterType(SorterType.NATIVE);
+ benchmark(ExternalSorter.create(options));
+ }
+
+ private static void benchmark(Sorter sorter) throws IOException {
+ long start = System.currentTimeMillis();
+ for (int i = 0; i < N; i++) {
+ sorter.add(
+ KV.of(
+ UUID.randomUUID().toString().getBytes(Charset.defaultCharset()),
+ UUID.randomUUID().toString().getBytes(Charset.defaultCharset())));
+ }
+ int i = 0;
+ for (KV<byte[], byte[]> ignored : sorter.sort()) {
+ i++;
+ }
+ long end = System.currentTimeMillis();
+ System.out.println(
+ String.format("%s: %fs", sorter.getClass().getSimpleName(), (end - start) / 1000.0));
+ }
+}
diff --git a/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterTest.java b/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterTest.java
index fb0a7b8..f3b391e 100644
--- a/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterTest.java
+++ b/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterTest.java
@@ -25,20 +25,30 @@
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
+import java.util.Arrays;
+import java.util.Collection;
+import org.apache.beam.sdk.extensions.sorter.ExternalSorter.Options.SorterType;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
/** Tests for Sorter. */
-@RunWith(JUnit4.class)
+@RunWith(Parameterized.class)
public class ExternalSorterTest {
@Rule public ExpectedException thrown = ExpectedException.none();
private static Path tmpLocation;
+ public ExternalSorterTest(SorterType sorterType) {
+ this.sorterType = sorterType;
+ }
+
+ private final SorterType sorterType;
+
@BeforeClass
public static void setupTempDir() throws IOException {
tmpLocation = Files.createTempDirectory("tmp");
@@ -64,32 +74,46 @@
});
}
+ @Parameters
+ public static Collection<SorterType[]> data() {
+ return Arrays.asList(
+ new SorterType[] {SorterType.HADOOP}, new SorterType[] {SorterType.NATIVE});
+ }
+
@Test
public void testEmpty() throws Exception {
SorterTestUtils.testEmpty(
ExternalSorter.create(
- new ExternalSorter.Options().setTempLocation(tmpLocation.toString())));
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)));
}
@Test
public void testSingleElement() throws Exception {
SorterTestUtils.testSingleElement(
ExternalSorter.create(
- new ExternalSorter.Options().setTempLocation(tmpLocation.toString())));
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)));
}
@Test
public void testEmptyKeyValueElement() throws Exception {
SorterTestUtils.testEmptyKeyValueElement(
ExternalSorter.create(
- new ExternalSorter.Options().setTempLocation(tmpLocation.toString())));
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)));
}
@Test
public void testMultipleIterations() throws Exception {
SorterTestUtils.testMultipleIterations(
ExternalSorter.create(
- new ExternalSorter.Options().setTempLocation(tmpLocation.toString())));
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)));
}
@Test
@@ -97,7 +121,9 @@
SorterTestUtils.testRandom(
() ->
ExternalSorter.create(
- new ExternalSorter.Options().setTempLocation(tmpLocation.toString())),
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)),
1,
1000000);
}
@@ -105,7 +131,10 @@
@Test
public void testAddAfterSort() throws Exception {
SorterTestUtils.testAddAfterSort(
- ExternalSorter.create(new ExternalSorter.Options().setTempLocation(tmpLocation.toString())),
+ ExternalSorter.create(
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)),
thrown);
fail();
}
@@ -113,7 +142,10 @@
@Test
public void testSortTwice() throws Exception {
SorterTestUtils.testSortTwice(
- ExternalSorter.create(new ExternalSorter.Options().setTempLocation(tmpLocation.toString())),
+ ExternalSorter.create(
+ new ExternalSorter.Options()
+ .setTempLocation(tmpLocation.toString())
+ .setSorterType(sorterType)),
thrown);
fail();
}
@@ -122,7 +154,7 @@
public void testNegativeMemory() {
thrown.expect(IllegalArgumentException.class);
thrown.expectMessage("memoryMB must be greater than zero");
- ExternalSorter.Options options = new ExternalSorter.Options();
+ ExternalSorter.Options options = new ExternalSorter.Options().setSorterType(sorterType);
options.setMemoryMB(-1);
}
@@ -130,7 +162,7 @@
public void testZeroMemory() {
thrown.expect(IllegalArgumentException.class);
thrown.expectMessage("memoryMB must be greater than zero");
- ExternalSorter.Options options = new ExternalSorter.Options();
+ ExternalSorter.Options options = new ExternalSorter.Options().setSorterType(sorterType);
options.setMemoryMB(0);
}
@@ -138,7 +170,7 @@
public void testMemoryTooLarge() {
thrown.expect(IllegalArgumentException.class);
thrown.expectMessage("memoryMB must be less than 2048");
- ExternalSorter.Options options = new ExternalSorter.Options();
+ ExternalSorter.Options options = new ExternalSorter.Options().setSorterType(sorterType);
options.setMemoryMB(2048);
}
}
diff --git a/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/SorterTestUtils.java b/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/SorterTestUtils.java
index 63dd2c0..0a978fa 100644
--- a/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/SorterTestUtils.java
+++ b/sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/SorterTestUtils.java
@@ -25,7 +25,7 @@
import java.util.Random;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
import org.junit.rules.ExpectedException;
/** A set of basic tests for {@link Sorter}s. */
diff --git a/sdks/java/extensions/sql/build.gradle b/sdks/java/extensions/sql/build.gradle
index 59afe49..78d7383 100644
--- a/sdks/java/extensions/sql/build.gradle
+++ b/sdks/java/extensions/sql/build.gradle
@@ -23,29 +23,9 @@
id 'ca.coglinc.javacc'
}
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.sql',
// javacc generated code produces lint warnings
- disableLintWarnings: ['dep-ann'],
- testShadowJar: true,
- enableStrictDependencies: true,
- shadowClosure: DEFAULT_SHADOW_CLOSURE << {
- dependencies {
- include(dependency(library.java.protobuf_java))
- include(dependency(library.java.protobuf_java_util))
- include(dependency("org.apache.calcite:.*"))
- include(dependency("org.apache.calcite.avatica:.*"))
- include(dependency("org.codehaus.janino:.*"))
- }
- relocate "com.google.protobuf", getJavaRelocatedPath("com.google.protobuf")
- relocate "org.apache.calcite", getJavaRelocatedPath("org.apache.calcite")
-
- // Looking up the compiler factory in Calcite depends on having a properties
- // file in the right location. We package one that is shading compatible
- // in src/main/resources. Note that if this shaded path changes, that
- // files name and contents need to be updated as well. TODO, swap to use
- // getJavaRelocatedPath once the Maven build is no longer also shading this
- // module.
- relocate "org.codehaus", "org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus"
-})
+ disableLintWarnings: ['dep-ann'])
description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL"
ext.summary = "Beam SQL provides a new interface to generate a Beam pipeline from SQL statement"
@@ -58,56 +38,34 @@
fmppTemplates
}
-def calcite_version = "1.19.0"
-def avatica_version = "1.15.0"
-
dependencies {
javacc "net.java.dev.javacc:javacc:4.0"
fmppTask "com.googlecode.fmpp-maven-plugin:fmpp-maven-plugin:1.0"
fmppTask "org.freemarker:freemarker:2.3.28"
- fmppTemplates "org.apache.calcite:calcite-core:$calcite_version"
- shadow library.java.vendored_guava_20_0 // Internal use
- compile library.java.guava // Interfaces with Calcite use this
- compile "org.apache.calcite:calcite-core:$calcite_version"
- compile "org.apache.calcite:calcite-linq4j:$calcite_version"
- compile "org.apache.calcite.avatica:avatica-core:$avatica_version"
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:join-library", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow library.java.commons_codec
- shadow library.java.commons_csv
- shadow library.java.commons_lang3
- shadow library.java.jackson_databind
- shadow library.java.jackson_dataformat_yaml
- shadow library.java.joda_time
- shadow "com.alibaba:fastjson:1.2.49"
- shadow "com.jayway.jsonpath:json-path:2.4.0"
- shadow project(path: ":runners:direct-java", configuration: "shadow")
- provided project(path: ":sdks:java:io:kafka", configuration: "shadow")
- provided project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
+ fmppTemplates library.java.vendored_calcite_1_20_0
+ compile project(":sdks:java:core")
+ compile project(":sdks:java:extensions:join-library")
+ compile project(":runners:direct-java")
+ compile library.java.commons_csv
+ compile library.java.vendored_calcite_1_20_0
+ compile "com.alibaba:fastjson:1.2.49"
+ compile "org.codehaus.janino:janino:3.0.11"
+ compile "org.codehaus.janino:commons-compiler:3.0.11"
+ provided project(":sdks:java:io:kafka")
+ provided project(":sdks:java:io:google-cloud-platform")
+ provided project(":sdks:java:io:parquet")
provided library.java.kafka_clients
- shadowTest library.java.junit
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.mockito_core
- shadowTest library.java.quickcheck_core
- shadowTestRuntimeClasspath library.java.slf4j_jdk14
-
- // Dependencies that we don't directly reference
- permitUnusedDeclared "com.jayway.jsonpath:json-path:2.4.0"
- permitUnusedDeclared library.java.jackson_dataformat_yaml
-
- // Dependencies that are bundled in when we bundle Calcite
- permitUsedUndeclared "org.codehaus.janino:janino:3.0.11"
- permitUsedUndeclared "org.codehaus.janino:commons-compiler:3.0.11"
-
- // Dependencies where one or the other appears "used" depending on classpath,
- // but it doesn't matter which is used
- permitUsedUndeclared "com.google.code.findbugs:jsr305:3.0.2"
- permitUnusedDeclared "net.jcip:jcip-annotations:1.0"
+ testCompile library.java.vendored_calcite_1_20_0
+ testCompile library.java.vendored_guava_26_0_jre
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.quickcheck_core
+ testRuntimeClasspath library.java.slf4j_jdk14
}
-// Copy Caclcite templates and our own template into the build directory
+// Copy Calcite templates and our own template into the build directory
// so we have one location for the FMPP task to parse.
task copyFmppTemplatesFromSrc(type: Copy) {
from "src/main/codegen"
@@ -116,11 +74,19 @@
task copyFmppTemplatesFromCalciteCore(type: Copy) {
dependsOn configurations.fmppTemplates
File calciteCoreJar = files(configurations.fmppTemplates.files).filter {
- it.name.startsWith("calcite-core")
+ it.name.startsWith("beam-vendor-calcite")
}.singleFile
from zipTree(calciteCoreJar)
include "**/Parser.jj"
into "${project.buildDir}/templates-fmpp"
+ filter{
+ line ->
+ line.replace('import org.apache.calcite.', 'import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.')
+ }
+ filter{
+ line ->
+ line.replace('import static org.apache.calcite.', 'import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.')
+ }
}
// Generate the FMPP sources from the FMPP templates.
@@ -189,6 +155,7 @@
systemProperty "beamTestPipelineOptions", JsonOutput.toJson(pipelineOptions)
include '**/*IT.class'
+ exclude '**/KafkaCSVTableIT.java'
maxParallelForks 4
classpath = project(":sdks:java:extensions:sql")
.sourceSets
diff --git a/sdks/java/extensions/sql/datacatalog/build.gradle b/sdks/java/extensions/sql/datacatalog/build.gradle
index 3496a9e..20a91ce 100644
--- a/sdks/java/extensions/sql/datacatalog/build.gradle
+++ b/sdks/java/extensions/sql/datacatalog/build.gradle
@@ -1,5 +1,3 @@
-import groovy.json.JsonOutput
-
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
@@ -18,16 +16,16 @@
* limitations under the License.
*/
+import groovy.json.JsonOutput
+
plugins { id 'org.apache.beam.module' }
-applyJavaNature(
- testShadowJar: true
-)
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.datacatalog')
dependencies {
- shadow library.java.grpc_google_cloud_datacatalog_v1beta1
- shadow library.java.proto_google_cloud_datacatalog_v1beta1
- provided project(path: ":sdks:java:extensions:sql", configuration: "shadow")
+ compile library.java.grpc_google_cloud_datacatalog_v1beta1
+ compile library.java.proto_google_cloud_datacatalog_v1beta1
+ provided project(":sdks:java:extensions:sql")
// For Data Catalog GRPC client
provided library.java.grpc_all
@@ -36,10 +34,9 @@
provided library.java.netty_tcnative_boringssl_static
// Dependencies for the example
- provided project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- provided library.java.vendored_guava_20_0
+ provided project(":sdks:java:io:google-cloud-platform")
provided library.java.slf4j_api
- shadowTestRuntimeClasspath library.java.slf4j_simple
+ testRuntimeOnly library.java.slf4j_simple
}
task runDataCatalogExample(type: JavaExec) {
@@ -61,3 +58,34 @@
"--tempLocation=${gcsTempRoot}",
]
}
+
+task integrationTest(type: Test) {
+ group = "Verification"
+ def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
+ def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests/'
+
+ // Disable Gradle cache (it should not be used because the IT's won't run).
+ outputs.upToDateWhen { false }
+
+ def pipelineOptions = [
+ "--project=${gcpProject}",
+ "--tempLocation=${gcsTempRoot}",
+ "--blockOnRun=false"]
+
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(pipelineOptions)
+
+ include '**/*IT.class'
+ maxParallelForks 4
+ classpath = project(":sdks:java:extensions:sql:datacatalog")
+ .sourceSets
+ .test
+ .runtimeClasspath
+ testClassesDirs = files(project(":sdks:java:extensions:sql:datacatalog").sourceSets.test.output.classesDirs)
+ useJUnit {}
+}
+
+task postCommit {
+ group = "Verification"
+ description = "Various integration tests"
+ dependsOn integrationTest
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java
index 8793842..81595cb 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlDataCatalogExample.java
@@ -21,6 +21,7 @@
import java.util.stream.Collectors;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog.DataCatalogPipelineOptions;
import org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog.DataCatalogTableProvider;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.options.Description;
@@ -30,7 +31,7 @@
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -71,7 +72,9 @@
.apply(
"SQL Query",
SqlTransform.query(options.getQueryString())
- .withDefaultTableProvider("datacatalog", DataCatalogTableProvider.create(options)))
+ .withDefaultTableProvider(
+ "datacatalog",
+ DataCatalogTableProvider.create(options.as(DataCatalogPipelineOptions.class))))
.apply("Convert to Strings", rowsToStrings())
.apply("Write output", TextIO.write().to(options.getOutputFilePrefix()));
@@ -91,7 +94,7 @@
"ERROR: SQL query or output file is not specified."
+ "To run this example:\n"
+ "./gradlew "
- + ":beam-sdks-java-extensions-sql-datacatalog:runDataCatalogExample "
+ + ":sdks:java:extensions:sql:datacatalog:runDataCatalogExample "
+ "-PgcpProject=<project> "
+ "-PgcsTempRoot=<GCS temp location> "
+ "-PqueryString=<query> "
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogClientAdapter.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogClientAdapter.java
deleted file mode 100644
index 973c3a8..0000000
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogClientAdapter.java
+++ /dev/null
@@ -1,94 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
-
-import com.google.auth.oauth2.GoogleCredentials;
-import com.google.cloud.datacatalog.DataCatalogGrpc;
-import com.google.cloud.datacatalog.DataCatalogGrpc.DataCatalogBlockingStub;
-import com.google.cloud.datacatalog.Entry;
-import com.google.cloud.datacatalog.LookupEntryRequest;
-import io.grpc.CallCredentials;
-import io.grpc.CallOptions;
-import io.grpc.Channel;
-import io.grpc.ClientCall;
-import io.grpc.ClientInterceptor;
-import io.grpc.ClientInterceptors;
-import io.grpc.ManagedChannelBuilder;
-import io.grpc.MethodDescriptor;
-import io.grpc.auth.MoreCallCredentials;
-import java.io.IOException;
-import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.meta.Table;
-
-/** Wraps DataCatalog GRPC client and exposes simplified APIS for Data Catalog Table Provider. */
-class DataCatalogClientAdapter {
-
- private DataCatalogBlockingStub dcClient;
-
- private DataCatalogClientAdapter(DataCatalogBlockingStub dcClient) {
- this.dcClient = dcClient;
- }
-
- /** Prod endpoint (default set in pipeline options): datacatalog.googleapis.com. */
- public static DataCatalogClientAdapter withDefaultCredentials(String endpoint)
- throws IOException {
- return new DataCatalogClientAdapter(newClient(endpoint));
- }
-
- private static DataCatalogBlockingStub newClient(String endpoint) throws IOException {
- Channel authedChannel =
- ClientInterceptors.intercept(
- ManagedChannelBuilder.forTarget(endpoint).build(),
- CredentialsInterceptor.defaultCredentials());
- return DataCatalogGrpc.newBlockingStub(authedChannel);
- }
-
- public @Nullable Table getTable(String tableName) {
- Entry entry = dcClient.lookupEntry(sqlResource(tableName));
- return TableUtils.toBeamTable(tableName, entry);
- }
-
- private LookupEntryRequest sqlResource(String tableName) {
- return LookupEntryRequest.newBuilder().setSqlResource(tableName).build();
- }
-
- /** Provides default credentials. */
- private static final class CredentialsInterceptor implements ClientInterceptor {
-
- private CallCredentials callCredentials;
-
- private CredentialsInterceptor(CallCredentials callCredentials) {
- this.callCredentials = callCredentials;
- }
-
- public static CredentialsInterceptor defaultCredentials() throws IOException {
- GoogleCredentials defaultCredentials = GoogleCredentials.getApplicationDefault();
- return of(MoreCallCredentials.from(defaultCredentials));
- }
-
- public static CredentialsInterceptor of(CallCredentials credentials) {
- return new CredentialsInterceptor(credentials);
- }
-
- @Override
- public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
- MethodDescriptor<ReqT, RespT> method, CallOptions callOptions, Channel next) {
- return next.newCall(method, callOptions.withCallCredentials(callCredentials));
- }
- }
-}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
index 44c6a78..0e8d6d9 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogTableProvider.java
@@ -19,44 +19,54 @@
import static java.util.stream.Collectors.toMap;
+import com.google.cloud.datacatalog.DataCatalogGrpc;
+import com.google.cloud.datacatalog.DataCatalogGrpc.DataCatalogBlockingStub;
+import com.google.cloud.datacatalog.LookupEntryRequest;
+import io.grpc.ManagedChannelBuilder;
import io.grpc.Status;
import io.grpc.StatusRuntimeException;
-import java.io.IOException;
+import io.grpc.auth.MoreCallCredentials;
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Stream;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.FullNameTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubJsonTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/** Uses DataCatalog to get the source type and schema for a table. */
-public class DataCatalogTableProvider implements TableProvider {
+public class DataCatalogTableProvider extends FullNameTableProvider {
private Map<String, TableProvider> delegateProviders;
private Map<String, Table> tableCache;
- private DataCatalogClientAdapter dataCatalog;
+ private DataCatalogBlockingStub dataCatalog;
private DataCatalogTableProvider(
- Map<String, TableProvider> delegateProviders, DataCatalogClientAdapter dataCatalogClient) {
+ Map<String, TableProvider> delegateProviders, DataCatalogBlockingStub dataCatalog) {
this.tableCache = new HashMap<>();
this.delegateProviders = ImmutableMap.copyOf(delegateProviders);
- this.dataCatalog = dataCatalogClient;
+ this.dataCatalog = dataCatalog;
}
- public static DataCatalogTableProvider create(PipelineOptions pipelineOptions)
- throws IOException {
+ public static DataCatalogTableProvider create(DataCatalogPipelineOptions options) {
+ return new DataCatalogTableProvider(getSupportedProviders(), createDataCatalogClient(options));
+ }
- DataCatalogPipelineOptions options = pipelineOptions.as(DataCatalogPipelineOptions.class);
-
- return new DataCatalogTableProvider(
- getSupportedProviders(), getDataCatalogClient(options.getDataCatalogEndpoint()));
+ private static DataCatalogBlockingStub createDataCatalogClient(
+ DataCatalogPipelineOptions options) {
+ return DataCatalogGrpc.newBlockingStub(
+ ManagedChannelBuilder.forTarget(options.getDataCatalogEndpoint()).build())
+ .withCallCredentials(
+ MoreCallCredentials.from(options.as(GcpOptions.class).getGcpCredential()));
}
private static Map<String, TableProvider> getSupportedProviders() {
@@ -65,10 +75,6 @@
.collect(toMap(TableProvider::getTableType, p -> p));
}
- private static DataCatalogClientAdapter getDataCatalogClient(String endpoint) throws IOException {
- return DataCatalogClientAdapter.withDefaultCredentials(endpoint);
- }
-
@Override
public String getTableType() {
return "google.cloud.datacatalog";
@@ -92,8 +98,23 @@
}
@Override
- public @Nullable Table getTable(String tableName) {
- return loadTable(tableName);
+ public @Nullable Table getTable(String tableNamePart) {
+ throw new UnsupportedOperationException(
+ "Loading a table by partial name '" + tableNamePart + "' is unsupported");
+ }
+
+ @Override
+ public @Nullable Table getTableByFullName(TableName fullTableName) {
+
+ ImmutableList<String> allNameParts =
+ ImmutableList.<String>builder()
+ .addAll(fullTableName.getPath())
+ .add(fullTableName.getTableName())
+ .build();
+
+ String fullEscapedTableName = ZetaSqlIdUtils.escapeAndJoin(allNameParts);
+
+ return loadTable(fullEscapedTableName);
}
private @Nullable Table loadTable(String tableName) {
@@ -106,7 +127,10 @@
private Table loadTableFromDC(String tableName) {
try {
- return dataCatalog.getTable(tableName);
+ return TableUtils.toBeamTable(
+ tableName,
+ dataCatalog.lookupEntry(
+ LookupEntryRequest.newBuilder().setSqlResource(tableName).build()));
} catch (StatusRuntimeException e) {
if (e.getStatus().equals(Status.INVALID_ARGUMENT)) {
return null;
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsUtils.java
new file mode 100644
index 0000000..d354e9d
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsUtils.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.cloud.datacatalog.Entry;
+import com.google.cloud.datacatalog.GcsFilesetSpec;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/** Utils to handle GCS entries from Cloud Data Catalog. */
+class GcsUtils {
+
+ /** Check if the entry represents a GCS fileset in Data Catalog. */
+ static boolean isGcs(Entry entry) {
+ return entry.hasGcsFilesetSpec();
+ }
+
+ /** Creates a Beam SQL table description from a GCS fileset entry. */
+ static Table.Builder tableBuilder(Entry entry) {
+ GcsFilesetSpec gcsFilesetSpec = entry.getGcsFilesetSpec();
+ List<String> filePatterns = gcsFilesetSpec.getFilePatternsList();
+
+ // We support exactly one 'file_patterns' field and nothing else at the moment
+ if (filePatterns.size() != 1) {
+ throw new UnsupportedOperationException(
+ "Unable to parse GCS entry '" + entry.getName() + "'");
+ }
+
+ String filePattern = filePatterns.get(0);
+
+ if (!filePattern.startsWith("gs://")) {
+ throw new UnsupportedOperationException(
+ "Unsupported file pattern. "
+ + "Only file patterns with 'gs://' schema are supported at the moment.");
+ }
+
+ return Table.builder()
+ .type("text")
+ .location(filePattern)
+ .properties(new JSONObject())
+ .comment("");
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
index 69288a8..31174c7 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
@@ -26,8 +26,8 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
class SchemaUtils {
@@ -39,6 +39,7 @@
.put("DATETIME", FieldType.DATETIME)
.put("DOUBLE", FieldType.DOUBLE)
.put("FLOAT", FieldType.DOUBLE)
+ .put("FLOAT64", FieldType.DOUBLE)
.put("INT32", FieldType.INT32)
.put("INT64", FieldType.INT64)
.put("STRING", FieldType.STRING)
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
index ebf3b99..6c0b62e 100644
--- a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
@@ -22,7 +22,7 @@
import java.util.Map;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/** Common utilities to create Beam SQL tables from Data Catalog schemas. */
class TableUtils {
@@ -45,15 +45,24 @@
+ "' in Data Catalog: "
+ entry.toString());
}
+ Schema schema = SchemaUtils.fromDataCatalog(entry.getSchema());
String service = URI.create(entry.getLinkedResource()).getAuthority().toLowerCase();
- if (!TABLE_FACTORIES.containsKey(service)) {
- throw new UnsupportedOperationException(
- "Unsupported SQL source kind: " + entry.getLinkedResource());
+ Table.Builder table = null;
+ if (TABLE_FACTORIES.containsKey(service)) {
+ table = TABLE_FACTORIES.get(service).tableBuilder(entry);
}
- Schema schema = SchemaUtils.fromDataCatalog(entry.getSchema());
- return TABLE_FACTORIES.get(service).tableBuilder(entry).schema(schema).name(tableName).build();
+ if (GcsUtils.isGcs(entry)) {
+ table = GcsUtils.tableBuilder(entry);
+ }
+
+ if (table != null) {
+ return table.schema(schema).name(tableName).build();
+ }
+
+ throw new UnsupportedOperationException(
+ "Unsupported SQL source kind: " + entry.getLinkedResource());
}
}
diff --git a/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtils.java b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtils.java
new file mode 100644
index 0000000..eac82c4
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtils.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import static java.util.stream.Collectors.joining;
+
+import java.util.List;
+import java.util.regex.Pattern;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** Utils to work with ZetaSQL-compatible IDs. */
+class ZetaSqlIdUtils {
+
+ /**
+ * Some special characters we explicitly handle.
+ *
+ * <p>Everything else is ignored, e.g. tabs, newlines, etc.
+ */
+ private static final Pattern SPECIAL_CHARS_ESCAPE =
+ Pattern.compile(
+ "(?<SpecialChar>["
+ + "\\\\" // slash
+ + "`" // backtick
+ + "'" // single quote
+ + "\"" // double quote
+ + "?" // question mark
+ + "])");
+
+ private static final ImmutableMap<String, String> WHITESPACES =
+ ImmutableMap.of(
+ "\n", "\\\\n",
+ "\t", "\\\\t",
+ "\r", "\\\\r",
+ "\f", "\\\\f");
+
+ private static final Pattern SIMPLE_ID = Pattern.compile("[A-Za-z_][A-Za-z_0-9]*");
+
+ /**
+ * Joins parts into a single compound ZetaSQL identifier.
+ *
+ * <p>Escapes backticks, slashes, double and single quotes, doesn't handle other special
+ * characters for now.
+ */
+ static String escapeAndJoin(List<String> parts) {
+ return parts.stream()
+ .map(ZetaSqlIdUtils::escapeSpecialChars)
+ .map(ZetaSqlIdUtils::replaceWhitespaces)
+ .map(ZetaSqlIdUtils::backtickIfNeeded)
+ .collect(joining("."));
+ }
+
+ private static String escapeSpecialChars(String str) {
+ return SPECIAL_CHARS_ESCAPE.matcher(str).replaceAll("\\\\${SpecialChar}");
+ }
+
+ private static String replaceWhitespaces(String s) {
+ return WHITESPACES.keySet().stream()
+ .reduce(s, (str, whitespace) -> str.replaceAll(whitespace, WHITESPACES.get(whitespace)));
+ }
+
+ private static String backtickIfNeeded(String s) {
+ return SIMPLE_ID.matcher(s).matches() ? s : ("`" + s + "`");
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
new file mode 100644
index 0000000..53599e9
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT64;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableReference;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
+import org.apache.beam.sdk.io.gcp.bigquery.TestBigQuery;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration tests for DataCatalog+BigQuery. */
+@RunWith(JUnit4.class)
+public class DataCatalogBigQueryIT {
+
+ private static final Schema ID_NAME_SCHEMA =
+ Schema.builder().addNullableField("id", INT64).addNullableField("name", STRING).build();
+
+ @Rule public transient TestPipeline writeToBQPipeline = TestPipeline.create();
+ @Rule public transient TestPipeline readPipeline = TestPipeline.create();
+ @Rule public transient TestBigQuery bigQuery = TestBigQuery.create(ID_NAME_SCHEMA);
+
+ @Test
+ public void testReadWrite() throws Exception {
+ createBQTableWith(
+ new TableRow().set("id", 1).set("name", "name1"),
+ new TableRow().set("id", 2).set("name", "name2"),
+ new TableRow().set("id", 3).set("name", "name3"));
+
+ TableReference bqTable = bigQuery.tableReference();
+ String tableId =
+ String.format(
+ "bigquery.`table`.`%s`.`%s`.`%s`",
+ bqTable.getProjectId(), bqTable.getDatasetId(), bqTable.getTableId());
+
+ PCollection<Row> result =
+ readPipeline.apply(
+ "query",
+ SqlTransform.query("SELECT id, name FROM " + tableId)
+ .withDefaultTableProvider(
+ "datacatalog",
+ DataCatalogTableProvider.create(
+ readPipeline.getOptions().as(DataCatalogPipelineOptions.class))));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "name1"), row(2, "name2"), row(3, "name3"));
+ readPipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private Row row(long id, String name) {
+ return Row.withSchema(ID_NAME_SCHEMA).addValues(id, name).build();
+ }
+
+ private void createBQTableWith(TableRow r1, TableRow r2, TableRow r3) {
+ writeToBQPipeline
+ .apply(Create.of(r1, r2, r3).withCoder(TableRowJsonCoder.of()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(bigQuery.tableSpec())
+ .withSchema(
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema().setName("id").setType("INTEGER"),
+ new TableFieldSchema().setName("name").setType("STRING"))))
+ .withoutValidation());
+ writeToBQPipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
new file mode 100644
index 0000000..ffd0d93
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+
+import java.io.Serializable;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration tests for DataCatalog+GCS. */
+@RunWith(JUnit4.class)
+public class DataCatalogGCSIT implements Serializable {
+
+ private static final Schema ID_NAME_TYPE_SCHEMA =
+ Schema.builder()
+ .addNullableField("id", INT32)
+ .addNullableField("name", STRING)
+ .addNullableField("type", STRING)
+ .build();
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ public void testReadFromGCS() throws Exception {
+ String gcsEntryId =
+ "`datacatalog`" // this is part of the resource name in DataCatalog, so it has to be
+ + ".`entry`" // different from the table provider name ("dc" in this test)
+ + ".`apache-beam-testing`"
+ + ".`us-central1`"
+ + ".`samples`"
+ + ".`integ_test_small_csv_test_1`";
+
+ PCollection<Row> result =
+ pipeline.apply(
+ "query",
+ SqlTransform.query("SELECT id, name, type FROM " + gcsEntryId)
+ .withDefaultTableProvider(
+ "dc",
+ DataCatalogTableProvider.create(
+ pipeline.getOptions().as(DataCatalogPipelineOptions.class))));
+
+ pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(true);
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(1, "customer1", "test"),
+ row(2, "customer2", "test"),
+ row(3, "customer1", "test"),
+ row(4, "customer2", "test"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private Row row(int id, String name, String type) {
+ return Row.withSchema(ID_NAME_TYPE_SCHEMA).addValues(id, name, type).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtilsTest.java b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtilsTest.java
new file mode 100644
index 0000000..ff34ef8
--- /dev/null
+++ b/sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtilsTest.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog;
+
+import static org.junit.Assert.assertEquals;
+
+import java.util.Arrays;
+import java.util.List;
+import org.junit.Test;
+
+/** Unit tests for {@link ZetaSqlIdUtils}. */
+public class ZetaSqlIdUtilsTest {
+
+ @Test
+ public void testHandlesSimpleIds() {
+ List<String> id = Arrays.asList("aaa", "BbB", "zAzzz00");
+ assertEquals("aaa.BbB.zAzzz00", ZetaSqlIdUtils.escapeAndJoin(id));
+ }
+
+ @Test
+ public void testHandlesMixedIds() {
+ List<String> id = Arrays.asList("aaa", "Bb---B", "zAzzz00");
+ assertEquals("aaa.`Bb---B`.zAzzz00", ZetaSqlIdUtils.escapeAndJoin(id));
+ }
+
+ @Test
+ public void testHandlesSpecialChars() {
+ List<String> id = Arrays.asList("a\\a", "b`b", "c'c", "d\"d", "e?e");
+ assertEquals("`a\\\\a`.`b\\`b`.`c\\'c`.`d\\\"d`.`e\\?e`", ZetaSqlIdUtils.escapeAndJoin(id));
+ }
+
+ @Test
+ public void testHandlesSpecialCharsInOnePart() {
+ List<String> id = Arrays.asList("a\\ab`bc'cd\"de?e");
+ assertEquals("`a\\\\ab\\`bc\\'cd\\\"de\\?e`", ZetaSqlIdUtils.escapeAndJoin(id));
+ }
+
+ @Test
+ public void testHandlesWhiteSpaces() {
+ List<String> id = Arrays.asList("a\na", "b\tb", "c\rc", "d\fd");
+ assertEquals("`a\\na`.`b\\tb`.`c\\rc`.`d\\fd`", ZetaSqlIdUtils.escapeAndJoin(id));
+ }
+
+ @Test
+ public void testHandlesWhiteSpacesInOnePart() {
+ List<String> id = Arrays.asList("a\nab\tbc\rcd\fd");
+ assertEquals("`a\\nab\\tbc\\rcd\\fd`", ZetaSqlIdUtils.escapeAndJoin(id));
+ }
+}
diff --git a/sdks/java/extensions/sql/hcatalog/build.gradle b/sdks/java/extensions/sql/hcatalog/build.gradle
index 9a15eec..0994ea4 100644
--- a/sdks/java/extensions/sql/hcatalog/build.gradle
+++ b/sdks/java/extensions/sql/hcatalog/build.gradle
@@ -20,19 +20,19 @@
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.meta.provider.hcatalog')
def hive_version = "2.1.0"
def netty_version = "4.1.30.Final"
dependencies {
- provided project(path: ":sdks:java:extensions:sql", configuration: "shadow")
- provided project(path: ":sdks:java:io:hcatalog", configuration: "shadow")
+ provided project(":sdks:java:extensions:sql")
+ provided project(":sdks:java:io:hcatalog")
// Needed for HCatalogTableProvider tests,
// they use HCat* types
- shadowTest "io.netty:netty-all:$netty_version"
- shadowTest("org.apache.hive.hcatalog:hive-hcatalog-core:$hive_version") {
+ testCompile "io.netty:netty-all:$netty_version"
+ testCompile("org.apache.hive.hcatalog:hive-hcatalog-core:$hive_version") {
// Hive brings full Calcite 1.6 + Avatica with JDBC driver which
// gets registered and gets started instead of ours
exclude group: "org.apache.calcite", module:"calcite-avatica"
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
index 86cb47d..4b7f900 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/DatabaseProvider.java
@@ -20,12 +20,12 @@
import com.alibaba.fastjson.JSONObject;
import java.util.Map;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
/**
* Metastore has a structure of 'db.table'.
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
index c96c657..8599732 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTable.java
@@ -20,9 +20,11 @@
import com.google.auto.value.AutoValue;
import java.util.Map;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
import org.apache.beam.sdk.io.hcatalog.HCatToRow;
import org.apache.beam.sdk.io.hcatalog.HCatalogIO;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -36,7 +38,7 @@
*/
@AutoValue
@Experimental
-public abstract class HCatalogTable implements BeamSqlTable {
+public abstract class HCatalogTable extends BaseBeamTable {
public abstract Schema schema();
@@ -68,6 +70,11 @@
}
@Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+
+ @Override
public Schema getSchema() {
return schema();
}
diff --git a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
index 8a35f9b..f38c173 100644
--- a/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
+++ b/sdks/java/extensions/sql/hcatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/HCatalogTableProvider.java
@@ -22,7 +22,7 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema;
diff --git a/sdks/java/extensions/sql/hcatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/BeamSqlHiveSchemaTest.java b/sdks/java/extensions/sql/hcatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/BeamSqlHiveSchemaTest.java
index 7c84202..706cf69 100644
--- a/sdks/java/extensions/sql/hcatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/BeamSqlHiveSchemaTest.java
+++ b/sdks/java/extensions/sql/hcatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/hcatalog/BeamSqlHiveSchemaTest.java
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
diff --git a/sdks/java/extensions/sql/jdbc/build.gradle b/sdks/java/extensions/sql/jdbc/build.gradle
index 70cbad1..acddedf 100644
--- a/sdks/java/extensions/sql/jdbc/build.gradle
+++ b/sdks/java/extensions/sql/jdbc/build.gradle
@@ -20,6 +20,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.extensions.sql.jdbc',
exportJavadoc: false,
testShadowJar: true,
validateShadowJar: false,
@@ -31,12 +32,12 @@
}
dependencies {
- compile project(path: ":sdks:java:extensions:sql", configuration: "shadow")
+ compile project(":sdks:java:extensions:sql")
compile "jline:jline:2.14.6"
compile "sqlline:sqlline:1.4.0"
compile library.java.slf4j_jdk14
compile library.java.guava
- testCompile project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:google-cloud-platform", configuration: "testRuntime")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
diff --git a/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java
index 9aef2ac..47b8ef6 100644
--- a/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java
+++ b/sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java
@@ -39,8 +39,8 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
import org.apache.beam.sdk.io.gcp.pubsub.TestPubsub;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.ObjectNode;
import org.hamcrest.collection.IsIn;
@@ -48,6 +48,7 @@
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
@@ -100,6 +101,7 @@
}
@Test
+ @Ignore("https://jira.apache.org/jira/browse/BEAM-7582")
public void testSelectFromPubsub() throws Exception {
String[] args =
buildArgs(
@@ -134,6 +136,7 @@
}
@Test
+ @Ignore("https://jira.apache.org/jira/browse/BEAM-7582")
public void testFilterForSouthManhattan() throws Exception {
String[] args =
buildArgs(
diff --git a/sdks/java/extensions/sql/shell/build.gradle b/sdks/java/extensions/sql/shell/build.gradle
index 50d281f..7422a94 100644
--- a/sdks/java/extensions/sql/shell/build.gradle
+++ b/sdks/java/extensions/sql/shell/build.gradle
@@ -22,14 +22,14 @@
}
dependencies {
- compile project(path: ":sdks:java:extensions:sql:jdbc", configuration: "shadow")
- permitUnusedDeclared project(path: ":sdks:java:extensions:sql:jdbc", configuration: "shadow")
+ compile project(":sdks:java:extensions:sql:jdbc")
+ permitUnusedDeclared project(":sdks:java:extensions:sql:jdbc")
if (project.hasProperty("beam.sql.shell.bundled")) {
project.getProperty("beam.sql.shell.bundled").tokenize(",").each {
subproject ->
- compile project(path: subproject, configuration: "shadow")
- permitUnusedDeclared project(path: subproject, configuration: "shadow")
+ compile project(path: subproject)
+ permitUnusedDeclared project(path: subproject)
}
}
}
diff --git a/sdks/java/extensions/sql/src/main/codegen/config.fmpp b/sdks/java/extensions/sql/src/main/codegen/config.fmpp
index 5b627eb..8dcb04b 100644
--- a/sdks/java/extensions/sql/src/main/codegen/config.fmpp
+++ b/sdks/java/extensions/sql/src/main/codegen/config.fmpp
@@ -21,15 +21,15 @@
# List of import statements.
imports: [
- "org.apache.calcite.schema.ColumnStrategy"
- "org.apache.calcite.sql.SqlCreate"
- "org.apache.calcite.sql.SqlDrop"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ColumnStrategy"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCreate"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDrop"
+ "org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlCreateExternalTable"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlDdlNodes"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlSetOptionBeam"
- "org.apache.beam.sdk.schemas.Schema"
"org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils"
- "org.apache.calcite.sql.type.SqlTypeName"
+ "org.apache.beam.sdk.schemas.Schema"
]
# List of keywords.
@@ -385,6 +385,10 @@
"parserImpls.ftl"
]
+ # List of methods for parsing builtin function calls.
+ builtinFunctionCallMethods: [
+ ]
+
includeCompoundIdentifier: true
includeBraces: true
includeAdditionalDeclarations: false
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
index 5e44c6c..8bdb1bf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
@@ -25,6 +25,7 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.meta.store.MetaStore;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
/** {@link BeamSqlCli} provides methods to execute Beam SQL with an interactive client. */
@Experimental
@@ -34,15 +35,17 @@
private MetaStore metaStore;
public BeamSqlCli metaStore(MetaStore metaStore) {
- return metaStore(metaStore, false);
+ return metaStore(metaStore, false, PipelineOptionsFactory.create());
}
- public BeamSqlCli metaStore(MetaStore metaStore, boolean autoLoadUdfUdaf) {
+ public BeamSqlCli metaStore(
+ MetaStore metaStore, boolean autoLoadUdfUdaf, PipelineOptions pipelineOptions) {
this.metaStore = metaStore;
BeamSqlEnv.BeamSqlEnvBuilder builder = BeamSqlEnv.builder(metaStore);
if (autoLoadUdfUdaf) {
builder.autoLoadUserDefinedFunctions();
}
+ builder.setPipelineOptions(pipelineOptions);
this.env = builder.build();
return this;
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
deleted file mode 100644
index 14f1b80..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql;
-
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.sdk.values.Row;
-
-/** This interface defines a Beam Sql Table. */
-public interface BeamSqlTable {
- /** create a {@code PCollection<Row>} from source. */
- PCollection<Row> buildIOReader(PBegin begin);
-
- /** create a {@code IO.write()} instance to write to target. */
- POutput buildIOWriter(PCollection<Row> input);
-
- /** Whether this table is bounded (known to be finite) or unbounded (may or may not be finite). */
- PCollection.IsBounded isBounded();
-
- /** Get the schema info of the table. */
- Schema getSchema();
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
index e45daca..e061632 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
@@ -29,6 +29,7 @@
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlPipelineOptions;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.impl.schema.BeamPCollectionTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.transforms.Combine;
@@ -40,8 +41,8 @@
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* {@link SqlTransform} is the DSL interface of Beam SQL. It translates a SQL query as a {@link
@@ -118,6 +119,8 @@
sqlEnvBuilder.setQueryPlannerClassName(
input.getPipeline().getOptions().as(BeamSqlPipelineOptions.class).getPlannerName());
+ sqlEnvBuilder.setPipelineOptions(input.getPipeline().getOptions());
+
BeamSqlEnv sqlEnv = sqlEnvBuilder.build();
return BeamSqlRelUtils.toPCollection(input.getPipeline(), sqlEnv.parseQuery(queryString()));
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
new file mode 100644
index 0000000..c6b1774
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql;
+
+import static java.util.stream.Collectors.toList;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlAsOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSelect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSetOperator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Helper class to extract table identifiers from the query.
+ *
+ * <p>Supports queries:
+ *
+ * <pre>
+ * ... FROM table...
+ * ... FROM table1, table2 AS x...
+ * ... FROM table1 JOIN (LEFT, INNER, OUTER etc) table2 JOIN table3 ...
+ * ... FROM table1 UNION (INTERSECT etc) SELECT ...
+ * </pre>
+ */
+public class TableNameExtractionUtils {
+
+ public static List<TableName> extractTableNamesFromNode(SqlNode node) {
+ if (node instanceof SqlSelect) {
+ return extractTableFromSelect((SqlSelect) node);
+ }
+
+ if (node instanceof SqlIdentifier) {
+ return extractFromIdentifier((SqlIdentifier) node);
+ }
+
+ if (node instanceof SqlJoin) {
+ return extractFromJoin((SqlJoin) node);
+ }
+
+ if (node instanceof SqlCall) {
+ return extractFromCall((SqlCall) node);
+ }
+
+ return Collections.emptyList();
+ }
+
+ private static List<TableName> extractTableFromSelect(SqlSelect node) {
+ return extractTableNamesFromNode(node.getFrom());
+ }
+
+ private static List<TableName> extractFromCall(SqlCall node) {
+ if (node.getOperator() instanceof SqlAsOperator) {
+ return extractTableNamesFromNode(node.getOperandList().get(0));
+ }
+
+ if (node.getOperator() instanceof SqlSetOperator) {
+ return node.getOperandList().stream()
+ .map(TableNameExtractionUtils::extractTableNamesFromNode)
+ .flatMap(Collection::stream)
+ .collect(toList());
+ }
+
+ return Collections.emptyList();
+ }
+
+ private static List<TableName> extractFromJoin(SqlJoin join) {
+ return ImmutableList.<TableName>builder()
+ .addAll(extractTableNamesFromNode(join.getLeft()))
+ .addAll(extractTableNamesFromNode(join.getRight()))
+ .build();
+ }
+
+ private static List<TableName> extractFromIdentifier(SqlIdentifier identifier) {
+ return ImmutableList.of(TableName.create(identifier.names));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
index a4f697d..8496a71 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
@@ -17,7 +17,6 @@
*/
package org.apache.beam.sdk.extensions.sql.example;
-import javax.annotation.Nullable;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -25,7 +24,6 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -48,7 +46,7 @@
*/
class BeamSqlExample {
public static void main(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).as(PipelineOptions.class);
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
Pipeline p = Pipeline.create(options);
// define the input row format
@@ -61,11 +59,7 @@
// create a source PCollection with Create.of();
PCollection<Row> inputTable =
- PBegin.in(p)
- .apply(
- Create.of(row1, row2, row3)
- .withSchema(
- type, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ PBegin.in(p).apply(Create.of(row1, row2, row3).withRowSchema(type));
// Case 1. run a simple SQL query over input PCollection with BeamSql.simpleQuery;
PCollection<Row> outputStream =
@@ -75,14 +69,14 @@
outputStream.apply(
"log_result",
MapElements.via(
- new SimpleFunction<Row, Void>() {
+ new SimpleFunction<Row, Row>() {
@Override
- public @Nullable Void apply(Row input) {
+ public Row apply(Row input) {
// expect output:
// PCOLLECTION: [3, row, 3.0]
// PCOLLECTION: [2, row, 2.0]
System.out.println("PCOLLECTION: " + input.getValues());
- return null;
+ return input;
}
}));
@@ -95,13 +89,13 @@
outputStream2.apply(
"log_result",
MapElements.via(
- new SimpleFunction<Row, Void>() {
+ new SimpleFunction<Row, Row>() {
@Override
- public @Nullable Void apply(Row input) {
+ public Row apply(Row input) {
// expect output:
// CASE1_RESULT: [row, 5.0]
System.out.println("CASE1_RESULT: " + input.getValues());
- return null;
+ return input;
}
}));
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlPojoExample.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlPojoExample.java
index 7e5995b..865d059 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlPojoExample.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlPojoExample.java
@@ -17,7 +17,6 @@
*/
package org.apache.beam.sdk.extensions.sql.example;
-import javax.annotation.Nullable;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.extensions.sql.example.model.Customer;
@@ -54,7 +53,8 @@
*/
class BeamSqlPojoExample {
public static void main(String[] args) {
- Pipeline pipeline = createPipeline(args);
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).create();
+ Pipeline pipeline = Pipeline.create(options);
// First step is to get PCollections of source objects.
// In this example we create them directly in memory using Create.of().
@@ -101,22 +101,17 @@
pipeline.run().waitUntilFinish();
}
- private static MapElements<Row, Void> logRecords(String suffix) {
+ private static MapElements<Row, Row> logRecords(String suffix) {
return MapElements.via(
- new SimpleFunction<Row, Void>() {
+ new SimpleFunction<Row, Row>() {
@Override
- public @Nullable Void apply(Row input) {
+ public Row apply(Row input) {
System.out.println(input.getValues() + suffix);
- return null;
+ return input;
}
});
}
- private static Pipeline createPipeline(String[] args) {
- PipelineOptions options = PipelineOptionsFactory.fromArgs(args).as(PipelineOptions.class);
- return Pipeline.create(options);
- }
-
private static PCollection<Customer> loadCustomers(Pipeline pipeline) {
return pipeline.apply(
Create.of(
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
index 1da4aae..e8a1f5f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
@@ -24,13 +24,13 @@
import java.util.Set;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.rel.type.RelProtoDataType;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.Schema;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.SchemaVersion;
-import org.apache.calcite.schema.Schemas;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaVersion;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schemas;
/** Adapter from {@link TableProvider} to {@link Schema}. */
public class BeamCalciteSchema implements Schema {
@@ -99,12 +99,16 @@
}
@Override
- public org.apache.calcite.schema.Table getTable(String name) {
+ public org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table getTable(
+ String name) {
Table table = tableProvider.getTable(name);
if (table == null) {
return null;
}
- return new BeamCalciteTable(tableProvider.buildBeamSqlTable(table), getPipelineOptions());
+ return new BeamCalciteTable(
+ tableProvider.buildBeamSqlTable(table),
+ getPipelineOptions(),
+ connection.getPipelineOptions());
}
@Override
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java
index f6a016b..2c6151a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchemaFactory.java
@@ -27,16 +27,16 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.store.InMemoryMetaStore;
import org.apache.beam.sdk.extensions.sql.meta.store.MetaStore;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.rel.type.RelProtoDataType;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.Schema;
-import org.apache.calcite.schema.SchemaFactory;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.SchemaVersion;
-import org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelProtoDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaVersion;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
/**
* Factory classes that Calcite uses to create initial schema for JDBC connection.
@@ -51,8 +51,8 @@
* a normal JDBC path, e.g. when CLI connects to {@link JdbcDriver} (without any extra connection
* properties).
*
- * <p>{@link Empty} is an override used in {@link JdbcDriver#connect(TableProvider)} to avoid
- * loading all available table providers.
+ * <p>{@link Empty} is an override used in {@link JdbcDriver#connect(TableProvider,
+ * org.apache.beam.sdk.options.PipelineOptions)} to avoid loading all available table providers.
*/
class BeamCalciteSchemaFactory {
@@ -97,10 +97,10 @@
}
/**
- * This is the override to create an empty schema, used in {@link
- * JdbcDriver#connect(TableProvider)} to avoid loading all table providers. This schema is
- * expected to be replaced by an actual functional schema by the same code that specified this
- * override in the first place.
+ * This is the override to create an empty schema, used in {@link JdbcDriver#connect(TableProvider
+ * , org.apache.beam.sdk.options.PipelineOptions)} to avoid loading all table providers. This
+ * schema is expected to be replaced by an actual functional schema by the same code that
+ * specified this override in the first place.
*/
public static class Empty extends InitialEmptySchema implements SchemaFactory {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
index e800c82..bb2f212 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteTable.java
@@ -20,40 +20,49 @@
import java.util.Collection;
import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSinkRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSourceRel;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.adapter.java.AbstractQueryableTable;
-import org.apache.calcite.linq4j.QueryProvider;
-import org.apache.calcite.linq4j.Queryable;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.prepare.Prepare;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.TableModify;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.schema.ModifiableTable;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.schema.TranslatableTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.AbstractQueryableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.QueryProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Queryable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.Prepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableModify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ModifiableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
/** Adapter from {@link BeamSqlTable} to a calcite Table. */
public class BeamCalciteTable extends AbstractQueryableTable
implements ModifiableTable, TranslatableTable {
private final BeamSqlTable beamTable;
- private final Map<String, String> pipelineOptions;
+ // These two options should be unified.
+ // https://issues.apache.org/jira/projects/BEAM/issues/BEAM-7590
+ private final Map<String, String> pipelineOptionsMap;
+ private PipelineOptions pipelineOptions;
- BeamCalciteTable(BeamSqlTable beamTable, Map<String, String> pipelineOptions) {
+ BeamCalciteTable(
+ BeamSqlTable beamTable,
+ Map<String, String> pipelineOptionsMap,
+ PipelineOptions pipelineOptions) {
super(Object[].class);
this.beamTable = beamTable;
+ this.pipelineOptionsMap = pipelineOptionsMap;
this.pipelineOptions = pipelineOptions;
}
public static BeamCalciteTable of(BeamSqlTable table) {
- return new BeamCalciteTable(table, ImmutableMap.of());
+ return new BeamCalciteTable(table, ImmutableMap.of(), null);
}
@Override
@@ -61,9 +70,34 @@
return CalciteUtils.toCalciteRowType(this.beamTable.getSchema(), typeFactory);
}
+ private PipelineOptions getPipelineOptions() {
+ if (pipelineOptions != null) {
+ return pipelineOptions;
+ }
+
+ pipelineOptions = BeamEnumerableConverter.createPipelineOptions(pipelineOptionsMap);
+ return pipelineOptions;
+ }
+
+ @Override
+ public BeamTableStatistics getStatistic() {
+ /*
+ Changing class loader is required for the JDBC path. It is similar to what done in
+ {@link BeamEnumerableConverter#toRowList} and {@link BeamEnumerableConverter#toEnumerable }.
+ */
+ final ClassLoader originalClassLoader = Thread.currentThread().getContextClassLoader();
+ try {
+ Thread.currentThread().setContextClassLoader(BeamEnumerableConverter.class.getClassLoader());
+ return beamTable.getTableStatistics(getPipelineOptions());
+ } finally {
+ Thread.currentThread().setContextClassLoader(originalClassLoader);
+ }
+ }
+
@Override
public RelNode toRel(RelOptTable.ToRelContext context, RelOptTable relOptTable) {
- return new BeamIOSourceRel(context.getCluster(), relOptTable, beamTable, pipelineOptions);
+ return new BeamIOSourceRel(
+ context.getCluster(), relOptTable, beamTable, pipelineOptionsMap, this);
}
@Override
@@ -97,6 +131,6 @@
sourceExpressionList,
flattened,
beamTable,
- pipelineOptions);
+ pipelineOptionsMap);
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
index 02b3e69..f27cb2e 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkNotNull;
import java.lang.reflect.Method;
import java.sql.SQLException;
@@ -30,21 +30,25 @@
import java.util.Set;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.BeamSqlUdf;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
import org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.UdfUdafProvider;
import org.apache.beam.sdk.extensions.sql.meta.store.InMemoryMetaStore;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.plan.RelOptUtil;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
/**
* Contains the metadata of tables/UDF functions, and exposes APIs to
@@ -66,14 +70,26 @@
return new BeamSqlEnvBuilder(tableProvider);
}
+ /**
+ * This method creates {@link org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv} using empty
+ * Pipeline Options. It should only be used in tests.
+ */
public static BeamSqlEnv readOnly(String tableType, Map<String, BeamSqlTable> tables) {
return withTableProvider(new ReadOnlyTableProvider(tableType, tables));
}
+ /**
+ * This method creates {@link org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv} using empty
+ * Pipeline Options. It should only be used in tests.
+ */
public static BeamSqlEnv withTableProvider(TableProvider tableProvider) {
- return builder(tableProvider).build();
+ return builder(tableProvider).setPipelineOptions(PipelineOptionsFactory.create()).build();
}
+ /**
+ * This method creates {@link org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv} using empty *
+ * Pipeline Options. It should only be used in tests.
+ */
public static BeamSqlEnv inMemory(TableProvider... tableProviders) {
InMemoryMetaStore inMemoryMetaStore = new InMemoryMetaStore();
for (TableProvider tableProvider : tableProviders) {
@@ -123,6 +139,8 @@
private Set<Map.Entry<String, Function>> functionSet;
private boolean autoLoadBuiltinFunctions;
private boolean autoLoadUdfs;
+ private PipelineOptions pipelineOptions;
+ private RuleSet[] ruleSets;
private BeamSqlEnvBuilder(TableProvider tableProvider) {
checkNotNull(tableProvider, "Table provider for the default schema must be sets.");
@@ -133,6 +151,8 @@
functionSet = new HashSet<>();
autoLoadUdfs = false;
autoLoadBuiltinFunctions = false;
+ pipelineOptions = null;
+ ruleSets = BeamRuleSets.getRuleSets();
}
/** Add a top-level schema backed by the table provider. */
@@ -151,6 +171,11 @@
return this;
}
+ /** Set the ruleSet used for query optimizer. */
+ public BeamSqlEnvBuilder setRuleSets(RuleSet[] ruleSets) {
+ this.ruleSets = ruleSets;
+ return this;
+ }
/** Register a UDF function which can be used in SQL expression. */
public BeamSqlEnvBuilder addUdf(String functionName, Class<?> clazz, String method) {
functionSet.add(new SimpleEntry<>(functionName, UdfImpl.create(clazz, method)));
@@ -194,14 +219,20 @@
return this;
}
+ public BeamSqlEnvBuilder setPipelineOptions(PipelineOptions pipelineOptions) {
+ this.pipelineOptions = pipelineOptions;
+ return this;
+ }
+
/**
* Build function to create an instance of BeamSqlEnv based on preset fields.
*
* @return BeamSqlEnv.
*/
public BeamSqlEnv build() {
+ checkNotNull(pipelineOptions);
- JdbcConnection jdbcConnection = JdbcDriver.connect(defaultTableProvider);
+ JdbcConnection jdbcConnection = JdbcDriver.connect(defaultTableProvider, pipelineOptions);
configureSchemas(jdbcConnection);
@@ -211,7 +242,7 @@
addUdfsUdafs(jdbcConnection);
- QueryPlanner planner = instantiatePlanner(jdbcConnection);
+ QueryPlanner planner = instantiatePlanner(jdbcConnection, ruleSets);
return new BeamSqlEnv(jdbcConnection, planner);
}
@@ -272,17 +303,12 @@
}
}
- private QueryPlanner instantiatePlanner(JdbcConnection jdbcConnection) {
-
- if (queryPlannerClassName.equals(CALCITE_PLANNER)) {
- return new CalciteQueryPlanner(jdbcConnection);
- }
-
+ private QueryPlanner instantiatePlanner(JdbcConnection jdbcConnection, RuleSet[] ruleSets) {
try {
return (QueryPlanner)
Class.forName(queryPlannerClassName)
- .getConstructor(JdbcConnection.class)
- .newInstance(jdbcConnection);
+ .getConstructor(JdbcConnection.class, RuleSet[].class)
+ .newInstance(jdbcConnection, ruleSets);
} catch (Exception e) {
throw new RuntimeException(
String.format("Cannot construct query planner %s", queryPlannerClassName), e);
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java
index 4f5cc37..5a1d313 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptionsRegistrar.java
@@ -20,7 +20,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
/** {@link AutoService} registrar for {@link BeamSqlPipelineOptions}. */
@AutoService(PipelineOptionsRegistrar.class)
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java
new file mode 100644
index 0000000..b5d6a2e
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl;
+
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelDistribution;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelDistributionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelReferentialConstraint;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Statistic;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
+
+/** This class stores row count statistics. */
+@Experimental
+@Internal
+public class BeamTableStatistics implements Serializable, Statistic {
+ public static final BeamTableStatistics BOUNDED_UNKNOWN = new BeamTableStatistics(100d, 0d, true);
+ public static final BeamTableStatistics UNBOUNDED_UNKNOWN =
+ new BeamTableStatistics(0d, 0.1, true);
+ private final boolean unknown;
+ private final Double rowCount;
+ private final Double rate;
+
+ private BeamTableStatistics(Double rowCount, Double rate, boolean isUnknown) {
+ this.rowCount = rowCount;
+ this.rate = rate;
+ this.unknown = isUnknown;
+ }
+
+ private BeamTableStatistics(Double rowCount, Double rate) {
+ this(rowCount, rate, false);
+ }
+
+ public static BeamTableStatistics createBoundedTableStatistics(Double rowCount) {
+ return new BeamTableStatistics(rowCount, 0d);
+ }
+
+ public static BeamTableStatistics createUnboundedTableStatistics(Double rate) {
+ return new BeamTableStatistics(0d, rate);
+ }
+
+ public Double getRate() {
+ return rate;
+ }
+
+ public boolean isUnknown() {
+ return unknown;
+ }
+
+ @Override
+ public Double getRowCount() {
+ return rowCount;
+ }
+
+ @Override
+ public boolean isKey(ImmutableBitSet columns) {
+ return false;
+ }
+
+ @Override
+ public List<RelReferentialConstraint> getReferentialConstraints() {
+ return ImmutableList.of();
+ }
+
+ @Override
+ public List<RelCollation> getCollations() {
+ return ImmutableList.of();
+ }
+
+ @Override
+ public RelDistribution getDistribution() {
+ return RelDistributionTraitDef.INSTANCE.getDefault();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
index e376ae5..0bdab10 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
@@ -35,14 +35,14 @@
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.Executor;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.config.CalciteConnectionConfig;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.linq4j.Enumerator;
-import org.apache.calcite.linq4j.Queryable;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Queryable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
/**
* Abstract wrapper for {@link CalciteConnection} to simplify extension.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java
index a039154..6bd714f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteFactoryWrapper.java
@@ -21,18 +21,18 @@
import java.sql.SQLException;
import java.util.Properties;
import java.util.TimeZone;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.avatica.AvaticaConnection;
-import org.apache.calcite.avatica.AvaticaFactory;
-import org.apache.calcite.avatica.AvaticaPreparedStatement;
-import org.apache.calcite.avatica.AvaticaResultSet;
-import org.apache.calcite.avatica.AvaticaSpecificDatabaseMetaData;
-import org.apache.calcite.avatica.AvaticaStatement;
-import org.apache.calcite.avatica.Meta;
-import org.apache.calcite.avatica.QueryState;
-import org.apache.calcite.avatica.UnregisteredDriver;
-import org.apache.calcite.jdbc.CalciteFactory;
-import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaPreparedStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaResultSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaSpecificDatabaseMetaData;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.Meta;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.QueryState;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.UnregisteredDriver;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
/**
* Wrapper for {@link CalciteFactory}.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java
index beab059..93c26fa 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteQueryPlanner.java
@@ -17,33 +17,49 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.RelMdNodeStats;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.config.CalciteConnectionConfig;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.plan.Contexts;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptPlanner.CannotPlanException;
-import org.apache.calcite.plan.RelOptUtil;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.prepare.CalciteCatalogReader;
-import org.apache.calcite.rel.RelRoot;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperatorTable;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
-import org.apache.calcite.sql.parser.SqlParseException;
-import org.apache.calcite.sql.parser.SqlParser;
-import org.apache.calcite.sql.parser.SqlParserImplFactory;
-import org.apache.calcite.sql.util.ChainedSqlOperatorTable;
-import org.apache.calcite.tools.FrameworkConfig;
-import org.apache.calcite.tools.Frameworks;
-import org.apache.calcite.tools.Planner;
-import org.apache.calcite.tools.RelConversionException;
-import org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner.CannotPlanException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.BuiltInMetadata;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ChainedRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.JaninoRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataHandler;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParser;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Planner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.BuiltInMethod;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -55,12 +71,14 @@
private static final Logger LOG = LoggerFactory.getLogger(CalciteQueryPlanner.class);
private final Planner planner;
+ private final JdbcConnection connection;
- CalciteQueryPlanner(JdbcConnection connection) {
- planner = Frameworks.getPlanner(defaultConfig(connection));
+ public CalciteQueryPlanner(JdbcConnection connection, RuleSet[] ruleSets) {
+ this.connection = connection;
+ this.planner = Frameworks.getPlanner(defaultConfig(connection, ruleSets));
}
- public FrameworkConfig defaultConfig(JdbcConnection connection) {
+ public FrameworkConfig defaultConfig(JdbcConnection connection, RuleSet[] ruleSets) {
final CalciteConnectionConfig config = connection.config();
final SqlParser.ConfigBuilder parserConfig =
SqlParser.configBuilder()
@@ -94,8 +112,8 @@
.defaultSchema(defaultSchema)
.traitDefs(traitDefs)
.context(Contexts.of(connection.config()))
- .ruleSets(BeamRuleSets.getRuleSets())
- .costFactory(null)
+ .ruleSets(ruleSets)
+ .costFactory(BeamCostModel.FACTORY)
.typeSystem(connection.getTypeFactory().getTypeSystem())
.operatorTable(ChainedSqlOperatorTable.of(opTab0, catalogReader))
.build();
@@ -122,6 +140,7 @@
BeamRelNode beamRelNode;
try {
SqlNode parsed = planner.parse(sqlStatement);
+ TableResolutionUtils.setupCustomTableResolution(connection, parsed);
SqlNode validated = planner.validate(parsed);
LOG.info("SQL:\n" + validated);
@@ -135,8 +154,18 @@
.replace(BeamLogicalConvention.INSTANCE)
.replace(root.collation)
.simplify();
-
// beam physical plan
+ root.rel
+ .getCluster()
+ .setMetadataProvider(
+ ChainedRelMetadataProvider.of(
+ ImmutableList.of(
+ NonCumulativeCostImpl.SOURCE,
+ RelMdNodeStats.SOURCE,
+ root.rel.getCluster().getMetadataProvider())));
+ RelMetadataQuery.THREAD_PROVIDERS.set(
+ JaninoRelMetadataProvider.of(root.rel.getCluster().getMetadataProvider()));
+ root.rel.getCluster().invalidateMetadataQuery();
beamRelNode = (BeamRelNode) planner.transform(0, desiredTraits, root.rel);
LOG.info("BEAMPlan>\n" + RelOptUtil.toString(beamRelNode));
} catch (RelConversionException | CannotPlanException e) {
@@ -149,4 +178,43 @@
}
return beamRelNode;
}
+
+ // It needs to be public so that the generated code in Calcite can access it.
+ public static class NonCumulativeCostImpl
+ implements MetadataHandler<BuiltInMetadata.NonCumulativeCost> {
+
+ public static final RelMetadataProvider SOURCE =
+ ReflectiveRelMetadataProvider.reflectiveSource(
+ BuiltInMethod.NON_CUMULATIVE_COST.method, new NonCumulativeCostImpl());
+
+ @Override
+ public MetadataDef<BuiltInMetadata.NonCumulativeCost> getDef() {
+ return BuiltInMetadata.NonCumulativeCost.DEF;
+ }
+
+ @SuppressWarnings("UnusedDeclaration")
+ public RelOptCost getNonCumulativeCost(RelNode rel, RelMetadataQuery mq) {
+ // This is called by a generated code in calcite MetadataQuery.
+ // If the rel is Calcite rel or we are in JDBC path and cost factory is not set yet we should
+ // use calcite cost estimation
+ if (!(rel instanceof BeamRelNode)) {
+ return rel.computeSelfCost(rel.getCluster().getPlanner(), mq);
+ }
+
+ // Currently we do nothing in this case, however, we can plug our own cost estimation method
+ // here and based on the design we also need to remove the cached values
+
+ // We need to first remove the cached values.
+ List<List> costKeys =
+ mq.map.entrySet().stream()
+ .filter(entry -> entry.getValue() instanceof BeamCostModel)
+ .filter(entry -> ((BeamCostModel) entry.getValue()).isInfinite())
+ .map(Map.Entry::getKey)
+ .collect(Collectors.toList());
+
+ costKeys.forEach(mq.map::remove);
+
+ return ((BeamRelNode) rel).beamComputeSelfCost(rel.getCluster().getPlanner(), mq);
+ }
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
index b8ad7f0..1783f53 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
@@ -25,10 +25,10 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
/**
* Beam JDBC Connection.
@@ -45,6 +45,7 @@
private static final String PIPELINE_OPTION_PREFIX = "beam.";
private Map<String, String> pipelineOptionsMap;
+ private PipelineOptions pipelineOptions;
private JdbcConnection(CalciteConnection connection) throws SQLException {
super(connection);
@@ -97,6 +98,14 @@
this.pipelineOptionsMap = ImmutableMap.copyOf(pipelineOptionsMap);
}
+ public void setPipelineOptions(PipelineOptions pipelineOptions) {
+ this.pipelineOptions = pipelineOptions;
+ }
+
+ public PipelineOptions getPipelineOptions() {
+ return this.pipelineOptions;
+ }
+
/** Get the current default schema from the root schema. */
@SuppressWarnings("TypeParameterUnusedInFormals")
<T> T getCurrentBeamSchema() {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
index bb7cc42..f012fc2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriver.java
@@ -17,9 +17,10 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import static org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
-import static org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.CompilerFactoryFactory.getDefaultCompilerFactory;
+import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.auto.service.AutoService;
import java.sql.Connection;
import java.sql.SQLException;
@@ -31,19 +32,19 @@
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.calcite.avatica.AvaticaFactory;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalciteFactory;
-import org.apache.calcite.jdbc.Driver;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.prepare.CalcitePrepareImpl;
-import org.apache.calcite.rel.RelCollationTraitDef;
-import org.apache.calcite.rel.rules.CalcRemoveRule;
-import org.apache.calcite.rel.rules.SortRemoveRule;
-import org.apache.calcite.runtime.Hook;
-import org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.Driver;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalcitePrepareImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcRemoveRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.SortRemoveRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.Hook;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
/**
* Calcite JDBC driver with Beam defaults.
@@ -100,6 +101,8 @@
INSTANCE.register();
}
+ public static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
@Override
protected AvaticaFactory createFactory() {
return JdbcFactory.wrap((CalciteFactory) super.createFactory());
@@ -141,7 +144,7 @@
* not this path. The CLI ends up using the schema factory that populates the default schema with
* all table providers it can find. See {@link BeamCalciteSchemaFactory}.
*/
- public static JdbcConnection connect(TableProvider tableProvider) {
+ public static JdbcConnection connect(TableProvider tableProvider, PipelineOptions options) {
try {
Properties properties = new Properties();
properties.setProperty(
@@ -149,6 +152,7 @@
JdbcConnection connection =
(JdbcConnection) INSTANCE.connect(CONNECT_STRING_PREFIX, properties);
connection.setSchema(TOP_LEVEL_BEAM_SCHEMA, tableProvider);
+ connection.setPipelineOptions(options);
return connection;
} catch (SQLException e) {
throw new RuntimeException(e);
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java
index 70c1229..22a6a52 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java
@@ -18,35 +18,37 @@
package org.apache.beam.sdk.extensions.sql.impl;
import static org.apache.beam.sdk.extensions.sql.impl.JdbcDriver.TOP_LEVEL_BEAM_SCHEMA;
-import static org.apache.calcite.avatica.BuiltInConnectionProperty.TIME_ZONE;
-import static org.apache.calcite.config.CalciteConnectionProperty.LEX;
-import static org.apache.calcite.config.CalciteConnectionProperty.PARSER_FACTORY;
-import static org.apache.calcite.config.CalciteConnectionProperty.SCHEMA;
-import static org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
-import static org.apache.calcite.config.CalciteConnectionProperty.TYPE_SYSTEM;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.BuiltInConnectionProperty.TIME_ZONE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.LEX;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.PARSER_FACTORY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.SCHEMA;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.SCHEMA_FACTORY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionProperty.TYPE_SYSTEM;
import java.util.Properties;
import org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.avatica.AvaticaConnection;
-import org.apache.calcite.avatica.AvaticaFactory;
-import org.apache.calcite.avatica.ConnectionProperty;
-import org.apache.calcite.avatica.UnregisteredDriver;
-import org.apache.calcite.config.Lex;
-import org.apache.calcite.jdbc.CalciteFactory;
-import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.AvaticaFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.ConnectionProperty;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.UnregisteredDriver;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.Lex;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
/**
* Implements {@link CalciteFactory} that is used by Clacite JDBC driver to instantiate different
* JDBC objects, like connections, result sets, etc.
*
* <p>The purpose of this class is to intercept the connection creation and force a cache-less root
- * schema ({@link org.apache.calcite.jdbc.SimpleCalciteSchema}). Otherwise Calcite uses {@link
- * org.apache.calcite.jdbc.CachingCalciteSchema} that eagerly caches table information. This
- * behavior does not work well for dynamic table providers.
+ * schema ({@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.SimpleCalciteSchema}). Otherwise
+ * Calcite uses {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CachingCalciteSchema} that eagerly
+ * caches table information. This behavior does not work well for dynamic table providers.
*/
class JdbcFactory extends CalciteFactoryWrapper {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java
index 0593921..cec0045 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/QueryPlanner.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.impl;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
/**
* An interface that planners should implement to convert sql statement to {@link BeamRelNode} or
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
index 149ae13..3ef4d9f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
@@ -17,28 +17,39 @@
*/
package org.apache.beam.sdk.extensions.sql.impl;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
+import java.lang.reflect.Type;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMultimap;
-import org.apache.calcite.adapter.enumerable.CallImplementor;
-import org.apache.calcite.adapter.enumerable.NullPolicy;
-import org.apache.calcite.adapter.enumerable.ReflectiveCallNotNullImplementor;
-import org.apache.calcite.adapter.enumerable.RexImpTable;
-import org.apache.calcite.linq4j.function.SemiStrict;
-import org.apache.calcite.linq4j.function.Strict;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.ImplementableFunction;
-import org.apache.calcite.schema.ScalarFunction;
-import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMultimap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.CallImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.NullPolicy;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.ReflectiveCallNotNullImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexImpTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexToLixTranslator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.SemiStrict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ImplementableFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ScalarFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperatorBinding;
/**
- * Beam-customized version from {@link org.apache.calcite.schema.impl.ScalarFunctionImpl}, to
+ * Beam-customized version from {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.ScalarFunctionImpl} , to
* address BEAM-5921.
*/
public class ScalarFunctionImpl extends UdfImplReflectiveFunctionBase
@@ -52,7 +63,10 @@
this.implementor = implementor;
}
- /** Creates {@link org.apache.calcite.schema.Function} for each method in a given class. */
+ /**
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} for
+ * each method in a given class.
+ */
public static ImmutableMultimap<String, Function> createAll(Class<?> clazz) {
final ImmutableMultimap.Builder<String, Function> builder = ImmutableMultimap.builder();
for (Method method : clazz.getMethods()) {
@@ -69,7 +83,8 @@
}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given class.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given class.
*
* <p>If a method of the given name is not found or it does not suit, returns {@code null}.
*
@@ -86,8 +101,8 @@
}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given method. When {@code eval} method
- * does not suit, {@code null} is returned.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given method. When {@code eval} method does not suit, {@code null} is returned.
*
* @param method method that is used to implement the function
* @return created {@link Function} or null
@@ -117,10 +132,65 @@
return implementor;
}
- private static CallImplementor createImplementor(final Method method) {
+ /**
+ * Version of {@link ReflectiveCallNotNullImplementor} that does parameter conversion for Beam
+ * UDFs.
+ */
+ private static class ScalarReflectiveCallNotNullImplementor
+ extends ReflectiveCallNotNullImplementor {
+ ScalarReflectiveCallNotNullImplementor(Method method) {
+ super(method);
+ }
+
+ private static List<Expression> translate(List<Type> types, List<Expression> expressions) {
+ // https://issues.apache.org/jira/browse/BEAM-8241
+ // In user defined functions Calcite allows variants with fewer arguments than Beam defined.
+ Preconditions.checkArgument(
+ types.size() >= expressions.size(), "types.size() < expressions.size()");
+
+ final List<Expression> translated = new ArrayList<>();
+ for (int i = 0; i < expressions.size(); i++) {
+ // TODO: [BEAM-8255] Add support for user defined function with var-arg
+ // Ex: types: [String[].class], expression: [param1, param2, ...]
+ translated.add(translate(types.get(i), expressions.get(i)));
+ }
+
+ return translated;
+ }
+
+ private static Expression translate(Type type, Expression expression) {
+ // NB: base class is called ReflectiveCallNotNullImplementor, but nulls are possible
+ //
+ // Calcite infers our UDF parameters as nullable, and WILL pass nullable expressions to this
+ // method. We could revisit this by explicitly asking users to add @Nullable annotation
+ // to UDF parameters, and not treating them as nullable by default, and then we can better
+ // determine if expression is possibly nullable by using reflection.
+
+ if (type == byte[].class && expression.type == ByteString.class) {
+ return Expressions.condition(
+ Expressions.equal(expression, Expressions.constant(null)),
+ Expressions.constant(null),
+ Expressions.call(expression, "getBytes"));
+ }
+
+ return expression;
+ }
+
+ @Override
+ public Expression implement(
+ RexToLixTranslator translator, RexCall call, List<Expression> translatedOperands) {
+ final List<Expression> translated =
+ translate(Arrays.asList(method.getParameterTypes()), translatedOperands);
+
+ // delegate to the underlying implementation to do the rest of translations
+ return super.implement(translator, call, translated);
+ }
+ }
+
+ private static CallImplementor createImplementor(Method method) {
final NullPolicy nullPolicy = getNullPolicy(method);
return RexImpTable.createImplementor(
- new ReflectiveCallNotNullImplementor(method), nullPolicy, false);
+ new ScalarReflectiveCallNotNullImplementor(method), nullPolicy, false);
}
private static NullPolicy getNullPolicy(Method m) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableName.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableName.java
new file mode 100644
index 0000000..282f0c2
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableName.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.auto.value.AutoValue;
+import java.util.Collections;
+import java.util.List;
+
+/** Represents a parsed table name that is specified in a FROM clause (and other places). */
+@AutoValue
+public abstract class TableName {
+
+ /**
+ * Table path up to the leaf table name.
+ *
+ * <p>Does not necessarily start from a schema name.
+ *
+ * <p>Does not include the actual table name, see {@link #getTableName()}.
+ */
+ public abstract List<String> getPath();
+
+ /** Table name, the last element of the fully-specified table name with path. */
+ public abstract String getTableName();
+
+ /** Full table name with path. */
+ public static TableName create(List<String> fullPath) {
+ checkNotNull(fullPath, "Full table path cannot be null");
+ checkArgument(fullPath.size() > 0, "Full table path has to have at least one element");
+ return create(fullPath.subList(0, fullPath.size() - 1), fullPath.get(fullPath.size() - 1));
+ }
+
+ /** Table name plus the path up to but not including table name. */
+ public static TableName create(List<String> path, String tableName) {
+ checkNotNull(tableName, "Table name cannot be null");
+ return new AutoValue_TableName(path == null ? Collections.emptyList() : path, tableName);
+ }
+
+ /** Whether it's a compound table name (with multiple path components). */
+ public boolean isCompound() {
+ return getPath().size() > 0;
+ }
+
+ /** Whether it's a simple name, with a single name component. */
+ public boolean isSimple() {
+ return getPath().size() == 0;
+ }
+
+ /** First element in the path. */
+ public String getPrefix() {
+ checkState(isCompound());
+ return getPath().get(0);
+ }
+
+ /**
+ * Remove prefix, e.g. this is helpful when stripping the top-level schema to register a table
+ * name with a provider.
+ */
+ public TableName removePrefix() {
+ List<String> pathPostfix = getPath().stream().skip(1).collect(toList());
+ return TableName.create(pathPostfix, getTableName());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java
new file mode 100644
index 0000000..1659e87
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl;
+
+import static java.util.stream.Collectors.toList;
+import static java.util.stream.Collectors.toMap;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.beam.sdk.extensions.sql.TableNameExtractionUtils;
+import org.apache.beam.sdk.extensions.sql.meta.CustomTableResolver;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Utils to wire up the custom table resolution into Calcite's planner. */
+class TableResolutionUtils {
+
+ private static final Logger LOG = LoggerFactory.getLogger(TableResolutionUtils.class);
+
+ /**
+ * Extract table names from the FROM clauses, register them with root TableProviders that support
+ * custom table schema resolution, e.g. DataCatalog.
+ *
+ * <p>Go over top-level schemas in the JdbcConnection, and for all top-level table providers that
+ * support custom table resolution, register all the parsed table names with them.
+ *
+ * <p>This way when a table provider has custom name-resolution strategy it can analyze whether it
+ * supports the name without using Calcite's logic. E.g. for DataCatalog we need to assemble the
+ * table name back into a single string and then query the back-end, whereas Calcite would require
+ * us to call the back-end for each part of the table name.
+ *
+ * <p>The logic is:
+ *
+ * <pre>
+ * - if it's a compound identifier (table name contains multiple parts):
+ * - get the first part of the identifier, assume it represents a top-level schema;
+ * - find a top-level table provider with the same name;
+ * - register the table identifier with it, if supported;
+ * - if not supported, then ignore the table identifier, everything will be resolved using
+ * existing Calcite's logic;
+ *
+ * - if it's a simple identifier (contains only a table name without a schema part),
+ * or if there was no matching top-level schema:
+ * - register with the default schema, if it supports custom table resolution;
+ * - if it does not, existing Calcite logic will still work as is;
+ * </pre>
+ */
+ static void setupCustomTableResolution(JdbcConnection connection, SqlNode parsed) {
+ List<TableName> tableNames = TableNameExtractionUtils.extractTableNamesFromNode(parsed);
+ String currentSchemaName = getCurrentSchemaName(connection);
+
+ SchemaWithName defaultSchema = SchemaWithName.create(connection, currentSchemaName);
+
+ if (defaultSchema.supportsCustomResolution()) {
+ registerWithDefaultSchema(connection, tableNames, defaultSchema);
+ }
+
+ registerWithTopLevelSchemas(connection, tableNames);
+ }
+
+ /** Current (default) schema name in the JdbcConnection. */
+ private static String getCurrentSchemaName(JdbcConnection connection) {
+ try {
+ return connection.getSchema();
+ } catch (SQLException e) {
+ throw new IllegalStateException(
+ "Unable to get current schema name from JdbcConnection. "
+ + "Assuming table names in the query are fully-qualified from the root.",
+ e);
+ }
+ }
+
+ /**
+ * Simple identifiers have to be resolved by the default schema, as well as compoung identifiers
+ * that don't have a matching top-level schema (meaning that a user didn't specify a top-level
+ * schema and expected it to be inferred).
+ */
+ private static void registerWithDefaultSchema(
+ JdbcConnection connection, List<TableName> tableNames, SchemaWithName defaultSchema) {
+ Set<String> topLevelSchemas = connection.getRootSchema().getSubSchemaNames();
+
+ List<TableName> simpleIdentifiers =
+ tableNames.stream().filter(TableName::isSimple).collect(toList());
+
+ List<TableName> withoutMatchingSchemas =
+ tableNames.stream()
+ .filter(name -> name.isCompound() && !topLevelSchemas.contains(name.getPrefix()))
+ .collect(toList());
+
+ List<TableName> explicitlyInDefaulSchema =
+ tableNames.stream()
+ .filter(name -> name.isCompound() && name.getPrefix().equals(defaultSchema.name))
+ .map(TableName::removePrefix)
+ .collect(toList());
+
+ List<TableName> shouldGoIntoDefaultSchema =
+ ImmutableList.<TableName>builder()
+ .addAll(simpleIdentifiers)
+ .addAll(withoutMatchingSchemas)
+ .addAll(explicitlyInDefaulSchema)
+ .build();
+
+ defaultSchema.getCustomTableResolver().registerKnownTableNames(shouldGoIntoDefaultSchema);
+ }
+
+ /**
+ * Register compound table identifiers with the matching custom resolvers that correspond to the
+ * top-level schemas.
+ */
+ private static void registerWithTopLevelSchemas(
+ JdbcConnection connection, List<TableName> tableNames) {
+
+ Map<String, CustomTableResolver> topLevelResolvers = getCustomTopLevelResolvers(connection);
+
+ topLevelResolvers.forEach(
+ (topLevelSchemaName, resolver) ->
+ resolver.registerKnownTableNames(tablesForSchema(tableNames, topLevelSchemaName)));
+ }
+
+ /** Get the custom schema resolvers for all top-level schemas that support custom resolution. */
+ private static Map<String, CustomTableResolver> getCustomTopLevelResolvers(
+ JdbcConnection connection) {
+ return connection.getRootSchema().getSubSchemaNames().stream()
+ .map(topLevelSchemaName -> SchemaWithName.create(connection, topLevelSchemaName))
+ .filter(schema -> !schema.getName().equals(getCurrentSchemaName(connection)))
+ .filter(SchemaWithName::supportsCustomResolution)
+ .collect(toMap(SchemaWithName::getName, SchemaWithName::getCustomTableResolver));
+ }
+
+ /**
+ * Get the compound identifiers that have the first component matching the given top-level schema
+ * name and remove the first component.
+ */
+ private static List<TableName> tablesForSchema(
+ List<TableName> tableNames, String topLevelSchema) {
+ return tableNames.stream()
+ .filter(TableName::isCompound)
+ .filter(t -> t.getPrefix().equals(topLevelSchema))
+ .map(TableName::removePrefix)
+ .collect(toList());
+ }
+
+ /**
+ * A utility class that keeps track of schema name and other properties.
+ *
+ * <p>Sole purpose is to reduce inline boilerplate and encapsulate stuff.
+ */
+ private static class SchemaWithName {
+ String name;
+ org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema schema;
+
+ static SchemaWithName create(JdbcConnection connection, String name) {
+ SchemaWithName schemaWithName = new SchemaWithName();
+ schemaWithName.name = name;
+ schemaWithName.schema =
+ CalciteSchema.from(connection.getRootSchema().getSubSchema(name)).schema;
+ return schemaWithName;
+ }
+
+ /** Whether this schema/table provider supports custom table resolution. */
+ boolean supportsCustomResolution() {
+ return isBeamSchema() && tableProviderSupportsCustomResolution();
+ }
+
+ /** Whether this Calcite schema is actually an instance of BeamCalciteSchema. */
+ boolean isBeamSchema() {
+ return schema instanceof BeamCalciteSchema;
+ }
+
+ /** Whether the table provider is an instance of CustomTableResolver. */
+ boolean tableProviderSupportsCustomResolution() {
+ return getTableProvider() instanceof CustomTableResolver;
+ }
+
+ /** Gets the table provider that backs the BeamCalciteSchema. */
+ TableProvider getTableProvider() {
+ checkState(isBeamSchema());
+ return ((BeamCalciteSchema) schema).getTableProvider();
+ }
+
+ /** Schema name. */
+ String getName() {
+ return name;
+ }
+
+ /** Custom table resolver in the provider. */
+ CustomTableResolver getCustomTableResolver() {
+ checkState(supportsCustomResolution());
+ return (CustomTableResolver) getTableProvider();
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java
index 70aa89d..532232b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdafImpl.java
@@ -25,12 +25,12 @@
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.calcite.adapter.enumerable.AggImplementor;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.schema.AggregateFunction;
-import org.apache.calcite.schema.FunctionParameter;
-import org.apache.calcite.schema.ImplementableAggFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.AggImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.AggregateFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ImplementableAggFunction;
/** Implement {@link AggregateFunction} to take a {@link CombineFn} as UDAF. */
@Experimental
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
index ba5848e..34f5683 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.extensions.sql.impl;
import java.lang.reflect.Method;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.TranslatableTable;
-import org.apache.calcite.schema.impl.TableMacroImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.TableMacroImpl;
/** Beam-customized facade behind {@link Function} to address BEAM-5921. */
class UdfImpl {
@@ -28,7 +28,8 @@
private UdfImpl() {}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given class.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given class.
*
* <p>If a method of the given name is not found or it does not suit, returns {@code null}.
*
@@ -45,7 +46,8 @@
}
/**
- * Creates {@link org.apache.calcite.schema.Function} from given method.
+ * Creates {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function} from
+ * given method.
*
* @param method method that is used to implement the function
* @return created {@link Function} or null
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java
index acc7551..be13523 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImplReflectiveFunctionBase.java
@@ -23,13 +23,13 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.schema.Function;
-import org.apache.calcite.schema.FunctionParameter;
-import org.apache.calcite.schema.impl.ReflectiveFunctionBase;
-import org.apache.calcite.util.ReflectUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.ReflectiveFunctionBase;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ReflectUtil;
/** Beam-customized version from {@link ReflectiveFunctionBase}, to address BEAM-5921. */
public abstract class UdfImplReflectiveFunctionBase implements Function {
@@ -95,7 +95,10 @@
return new ParameterListBuilder();
}
- /** Helps build lists of {@link org.apache.calcite.schema.FunctionParameter}. */
+ /**
+ * Helps build lists of {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter}.
+ */
public static class ParameterListBuilder {
final List<FunctionParameter> builder = new ArrayList<>();
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java
index de6a6f3..a6d145d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCheckConstraint.java
@@ -18,15 +18,15 @@
package org.apache.beam.sdk.extensions.sql.impl.parser;
import java.util.List;
-import org.apache.calcite.sql.SqlCall;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.ImmutableNullableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableNullableList;
/**
* Parse tree for {@code UNIQUE}, {@code PRIMARY KEY} constraints.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java
index fbec56c..1ffe80f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlColumnDeclaration.java
@@ -18,16 +18,16 @@
package org.apache.beam.sdk.extensions.sql.impl.parser;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.sql.SqlCall;
-import org.apache.calcite.sql.SqlDataTypeSpec;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
/** Parse tree for column. */
public class SqlColumnDeclaration extends SqlCall {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java
index 7041e33..bd331a9 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateExternalTable.java
@@ -19,8 +19,8 @@
import static com.alibaba.fastjson.JSON.parseObject;
import static org.apache.beam.sdk.schemas.Schema.toSchema;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import com.alibaba.fastjson.JSONObject;
import java.util.List;
@@ -28,19 +28,19 @@
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlCreate;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.SqlUtil;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlCreate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/** Parse tree for {@code CREATE EXTERNAL TABLE} statement. */
public class SqlCreateExternalTable extends SqlCreate implements SqlExecutableStatement {
@@ -135,8 +135,8 @@
private void unparseColumn(SqlWriter writer, Schema.Field column) {
writer.sep(",");
- writer.identifier(column.getName());
- writer.identifier(CalciteUtils.toSqlTypeName(column.getType()).name());
+ writer.identifier(column.getName(), false);
+ writer.identifier(CalciteUtils.toSqlTypeName(column.getType()).name(), false);
if (column.getType().getNullable() != null && !column.getType().getNullable()) {
writer.keyword("NOT NULL");
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
index d9ceeb5..dbeb98d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDdlNodes.java
@@ -19,16 +19,16 @@
import java.util.List;
import javax.annotation.Nullable;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlDataTypeSpec;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlLiteral;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.NlsString;
-import org.apache.calcite.util.Pair;
-import org.apache.calcite.util.Util;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDataTypeSpec;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.NlsString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
/** Utilities concerning {@link SqlNode} for DDL. */
public class SqlDdlNodes {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
index 1463892..2801dcd 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropObject.java
@@ -17,21 +17,21 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.parser;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlDrop;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlUtil;
-import org.apache.calcite.sql.SqlWriter;
-import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDrop;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
/**
* Base class for parse trees of {@code DROP TABLE}, {@code DROP VIEW} and {@code DROP MATERIALIZED
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java
index 3714cf6..9541242 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlDropTable.java
@@ -17,11 +17,11 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.parser;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.SqlSpecialOperator;
-import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
/** Parse tree for {@code DROP TABLE} statement. */
public class SqlDropTable extends SqlDropObject {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java
index 7314305..c74a1fb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlSetOptionBeam.java
@@ -17,18 +17,18 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.parser;
-import static org.apache.calcite.util.Static.RESOURCE;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Static.RESOURCE;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
-import org.apache.calcite.jdbc.CalcitePrepare;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.sql.SqlExecutableStatement;
-import org.apache.calcite.sql.SqlIdentifier;
-import org.apache.calcite.sql.SqlNode;
-import org.apache.calcite.sql.SqlSetOption;
-import org.apache.calcite.sql.SqlUtil;
-import org.apache.calcite.sql.parser.SqlParserPos;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalcitePrepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlExecutableStatement;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlSetOption;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/** SQL parse tree node to represent {@code SET} and {@code RESET} statements. */
public class SqlSetOptionBeam extends SqlSetOption implements SqlExecutableStatement {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java
new file mode 100644
index 0000000..2e57cb1
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModel.java
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import java.util.Objects;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptUtil;
+
+/**
+ * <code>VolcanoCost</code> represents the cost of a plan node.
+ *
+ * <p>This class is immutable: none of the methods modify any member variables.
+ */
+public class BeamCostModel implements RelOptCost {
+ private static final double RATE_IMPORTANCE = 3600;
+
+ static final BeamCostModel INFINITY =
+ new BeamCostModel(Double.POSITIVE_INFINITY, Double.POSITIVE_INFINITY) {
+ @Override
+ public String toString() {
+ return "{inf}";
+ }
+ };
+
+ static final BeamCostModel HUGE =
+ new BeamCostModel(Double.MAX_VALUE, Double.MAX_VALUE) {
+ @Override
+ public String toString() {
+ return "{huge}";
+ }
+ };
+
+ static final BeamCostModel ZERO =
+ new BeamCostModel(0.0, 0.0) {
+ @Override
+ public String toString() {
+ return "{0}";
+ }
+ };
+
+ static final BeamCostModel TINY =
+ new BeamCostModel(1.0, 0.001) {
+ @Override
+ public String toString() {
+ return "{tiny}";
+ }
+ };
+
+ public static final BeamCostModel.Factory FACTORY = new BeamCostModel.Factory();
+
+ final double cpu;
+ final double cpuRate;
+
+ BeamCostModel(double cpu, double cpuRate) {
+ this.cpu = Math.max(cpu, 0);
+ this.cpuRate = Math.max(cpuRate, 0);
+ }
+
+ @Override
+ public double getCpu() {
+ return cpu;
+ }
+
+ @Override
+ public boolean isInfinite() {
+ return (this.equals(INFINITY))
+ || (this.cpu == Double.POSITIVE_INFINITY)
+ || (this.cpuRate == Double.POSITIVE_INFINITY);
+ }
+
+ @Override
+ public double getIo() {
+ return 0;
+ }
+
+ public double getCpuRate() {
+ return cpuRate;
+ }
+
+ @Override
+ public boolean isLe(RelOptCost other) {
+ BeamCostModel that = (BeamCostModel) other;
+ // This if is to make sure Infinity.isLe(Huge) wont be true.
+ // Without this both of the costCombinations are infinity and therefore, this will return true.
+
+ // if one of them is infinite then the only thing that matters is "that" being infinite.
+ if (this.isInfinite() || that.isInfinite()) {
+ return that.isInfinite();
+ }
+
+ return getCostCombination(this) <= getCostCombination(that);
+ }
+
+ @Override
+ public boolean isLt(RelOptCost other) {
+ BeamCostModel that = (BeamCostModel) other;
+ // This is to make sure Huge.isLt(Infinity) returns true
+ if (that.isInfinite() || this.isInfinite()) {
+ return !this.isInfinite();
+ }
+
+ return getCostCombination(this) < getCostCombination(that);
+ }
+
+ private static double getCostCombination(BeamCostModel cost) {
+ return cost.cpu + cost.cpuRate * RATE_IMPORTANCE;
+ }
+
+ @Override
+ public double getRows() {
+ return 0;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(cpu, cpuRate);
+ }
+
+ @SuppressWarnings("NonOverridingEquals")
+ @Override
+ public boolean equals(RelOptCost other) {
+ return other instanceof BeamCostModel
+ && (this.cpu == ((BeamCostModel) other).cpu)
+ && (this.cpuRate == ((BeamCostModel) other).cpuRate);
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ if (obj instanceof BeamCostModel) {
+ return equals((BeamCostModel) obj);
+ }
+ return false;
+ }
+
+ @Override
+ public boolean isEqWithEpsilon(RelOptCost other) {
+ if (!(other instanceof BeamCostModel)) {
+ return false;
+ }
+ BeamCostModel that = (BeamCostModel) other;
+ return ((Math.abs(this.cpu - that.cpu) < RelOptUtil.EPSILON)
+ && (Math.abs(this.cpuRate - that.cpuRate) < RelOptUtil.EPSILON));
+ }
+
+ @Override
+ public BeamCostModel minus(RelOptCost other) {
+ if (this.equals(INFINITY)) {
+ return this;
+ }
+ BeamCostModel that = (BeamCostModel) other;
+ return new BeamCostModel(this.cpu - that.cpu, this.cpuRate - that.cpuRate);
+ }
+
+ @Override
+ public BeamCostModel multiplyBy(double factor) {
+ if (this.equals(INFINITY)) {
+ return this;
+ }
+ return new BeamCostModel(cpu * factor, cpuRate * factor);
+ }
+
+ @Override
+ public double divideBy(RelOptCost cost) {
+ // Compute the geometric average of the ratios of all of the factors
+ // which are non-zero and finite. (Except the window size)
+ BeamCostModel that = (BeamCostModel) cost;
+
+ if ((getCostCombination(this) != 0)
+ && !Double.isInfinite(getCostCombination(this))
+ && (getCostCombination(that) != 0)
+ && !Double.isInfinite(getCostCombination(that))) {
+ return getCostCombination(this) / getCostCombination(that);
+ }
+
+ return 1.0;
+ }
+
+ @Override
+ public BeamCostModel plus(RelOptCost other) {
+ BeamCostModel that = (BeamCostModel) other;
+ if (this.equals(INFINITY) || that.equals(INFINITY)) {
+ return INFINITY;
+ }
+ return new BeamCostModel(this.cpu + that.cpu, this.cpuRate + that.cpuRate);
+ }
+
+ @Override
+ public String toString() {
+ return "{" + cpu + " cpu, " + cpuRate + " cpuRate " + "}";
+ }
+
+ public static BeamCostModel convertRelOptCost(RelOptCost ic) {
+ BeamCostModel inputCost;
+ if (ic instanceof BeamCostModel) {
+ inputCost = ((BeamCostModel) ic);
+ } else {
+ inputCost = BeamCostModel.FACTORY.makeCost(ic.getRows(), ic.getCpu(), ic.getIo());
+ }
+ return inputCost;
+ }
+
+ /**
+ * Implementation of {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCostFactory} that creates
+ * {@link BeamCostModel}s.
+ */
+ public static class Factory implements RelOptCostFactory {
+
+ @Override
+ public BeamCostModel makeCost(double dRows, double dCpu, double dIo) {
+ return BeamCostModel.INFINITY;
+ }
+
+ public BeamCostModel makeCost(double dCpu, double dCpuRate) {
+ return new BeamCostModel(dCpu, dCpuRate);
+ }
+
+ @Override
+ public BeamCostModel makeHugeCost() {
+ return BeamCostModel.HUGE;
+ }
+
+ @Override
+ public BeamCostModel makeInfiniteCost() {
+ return BeamCostModel.INFINITY;
+ }
+
+ @Override
+ public BeamCostModel makeTinyCost() {
+ return BeamCostModel.TINY;
+ }
+
+ @Override
+ public BeamCostModel makeZeroCost() {
+ return BeamCostModel.ZERO;
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java
index 8d6114e..bc67b93 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamJavaTypeFactory.java
@@ -18,12 +18,12 @@
package org.apache.beam.sdk.extensions.sql.impl.planner;
import java.lang.reflect.Type;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.sql.type.BasicSqlType;
-import org.apache.calcite.sql.type.IntervalSqlType;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.BasicSqlType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.IntervalSqlType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
/** customized data type in Beam. */
public class BeamJavaTypeFactory extends JavaTypeFactoryImpl {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
index 4356e82..b83a1bf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRelDataTypeSystem.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.planner;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.rel.type.RelDataTypeSystemImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystemImpl;
/** customized data type in Beam. */
public class BeamRelDataTypeSystem extends RelDataTypeSystemImpl {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
index 38cfcc9..33d69dd 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
@@ -23,42 +23,47 @@
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamBasicAggregationRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCalcRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCoGBKJoinRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamEnumerableConverterRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIntersectRule;
-import org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinAssociateRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinPushThroughJoinRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamMinusRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamSideInputJoinRule;
+import org.apache.beam.sdk.extensions.sql.impl.rule.BeamSideInputLookupJoinRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamSortRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamUncollectRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamUnionRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamUnnestRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamValuesRule;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.rules.AggregateJoinTransposeRule;
-import org.apache.calcite.rel.rules.AggregateProjectMergeRule;
-import org.apache.calcite.rel.rules.AggregateRemoveRule;
-import org.apache.calcite.rel.rules.AggregateUnionAggregateRule;
-import org.apache.calcite.rel.rules.CalcMergeRule;
-import org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
-import org.apache.calcite.rel.rules.FilterCalcMergeRule;
-import org.apache.calcite.rel.rules.FilterJoinRule;
-import org.apache.calcite.rel.rules.FilterProjectTransposeRule;
-import org.apache.calcite.rel.rules.FilterSetOpTransposeRule;
-import org.apache.calcite.rel.rules.FilterToCalcRule;
-import org.apache.calcite.rel.rules.JoinPushExpressionsRule;
-import org.apache.calcite.rel.rules.ProjectCalcMergeRule;
-import org.apache.calcite.rel.rules.ProjectFilterTransposeRule;
-import org.apache.calcite.rel.rules.ProjectMergeRule;
-import org.apache.calcite.rel.rules.ProjectSetOpTransposeRule;
-import org.apache.calcite.rel.rules.ProjectSortTransposeRule;
-import org.apache.calcite.rel.rules.ProjectToCalcRule;
-import org.apache.calcite.rel.rules.PruneEmptyRules;
-import org.apache.calcite.rel.rules.SortProjectTransposeRule;
-import org.apache.calcite.rel.rules.UnionEliminatorRule;
-import org.apache.calcite.rel.rules.UnionToDistinctRule;
-import org.apache.calcite.tools.RuleSet;
-import org.apache.calcite.tools.RuleSets;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateJoinTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateProjectMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateRemoveRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AggregateUnionAggregateRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.CalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterJoinRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterProjectTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterSetOpTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.FilterToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinCommuteRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinPushExpressionsRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectCalcMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectFilterTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectMergeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectSetOpTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectSortTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.ProjectToCalcRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.PruneEmptyRules;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.SortProjectTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.UnionEliminatorRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.UnionToDistinctRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
/**
* {@link RuleSet} used in {@code BeamQueryPlanner}. It translates a standard Calcite {@link
@@ -103,6 +108,10 @@
// join rules
JoinPushExpressionsRule.INSTANCE,
+ JoinCommuteRule.INSTANCE,
+ BeamJoinAssociateRule.INSTANCE,
+ BeamJoinPushThroughJoinRule.RIGHT,
+ BeamJoinPushThroughJoinRule.LEFT,
// remove union with only a single child
UnionEliminatorRule.INSTANCE,
@@ -144,7 +153,9 @@
BeamUnionRule.INSTANCE,
BeamUncollectRule.INSTANCE,
BeamUnnestRule.INSTANCE,
- BeamJoinRule.INSTANCE);
+ BeamSideInputJoinRule.INSTANCE,
+ BeamCoGBKJoinRule.INSTANCE,
+ BeamSideInputLookupJoinRule.INSTANCE);
private static final List<RelOptRule> BEAM_TO_ENUMERABLE =
ImmutableList.of(BeamEnumerableConverterRule.INSTANCE);
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStats.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStats.java
new file mode 100644
index 0000000..88d7ad2
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStats.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import com.google.auto.value.AutoValue;
+
+/** This is a utility class to represent rowCount, rate and window. */
+@AutoValue
+public abstract class NodeStats {
+
+ /**
+ * Returns an instance with all values set to INFINITY. This will be only used when the node is
+ * not a BeamRelNode and we don't have an estimation implementation for it in the metadata
+ * handler. In this case we return INFINITE and it will be propagated up in the estimates.
+ */
+ public static final NodeStats UNKNOWN =
+ create(Double.POSITIVE_INFINITY, Double.POSITIVE_INFINITY, Double.POSITIVE_INFINITY);
+
+ public abstract double getRowCount();
+
+ public abstract double getRate();
+
+ /**
+ * This method returns the number of tuples in each window. It is different than the windowing
+ * notion of Beam.
+ */
+ public abstract double getWindow();
+
+ public static NodeStats create(double rowCount, double rate, double window) {
+ if (window < 0 || rate < 0 || rowCount < 0) {
+ throw new IllegalArgumentException("All the estimates in NodeStats should be positive");
+ }
+ return new AutoValue_NodeStats(rowCount, rate, window);
+ }
+
+ /** It creates an instance with rate=0 and window=rowCount for bounded sources. */
+ public static NodeStats create(double rowCount) {
+ return create(rowCount, 0d, rowCount);
+ }
+
+ /** If any of the values for rowCount, rate or window is infinite, it returns true. */
+ public boolean isUnknown() {
+ return Double.isInfinite(getRowCount())
+ || Double.isInfinite(getRate())
+ || Double.isInfinite(getWindow());
+ }
+
+ public NodeStats multiply(double factor) {
+ return create(getRowCount() * factor, getRate() * factor, getWindow() * factor);
+ }
+
+ public NodeStats plus(NodeStats that) {
+ if (this.isUnknown() || that.isUnknown()) {
+ return UNKNOWN;
+ }
+ return create(
+ this.getRowCount() + that.getRowCount(),
+ this.getRate() + that.getRate(),
+ this.getWindow() + that.getWindow());
+ }
+
+ public NodeStats minus(NodeStats that) {
+ if (this.isUnknown() || that.isUnknown()) {
+ return UNKNOWN;
+ }
+ return create(
+ Math.max(this.getRowCount() - that.getRowCount(), 0),
+ Math.max(this.getRate() - that.getRate(), 0),
+ Math.max(this.getWindow() - that.getWindow(), 0));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java
new file mode 100644
index 0000000..f0991af
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsMetadata.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import java.lang.reflect.Method;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Types;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.Metadata;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataHandler;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+
+/**
+ * This is a metadata used for row count and rate estimation. It extends Calcite's Metadata
+ * interface so that we can use MetadataQuery to get our own estimates.
+ */
+public interface NodeStatsMetadata extends Metadata {
+ Method METHOD = Types.lookupMethod(NodeStatsMetadata.class, "getNodeStats");
+
+ MetadataDef<NodeStatsMetadata> DEF =
+ MetadataDef.of(NodeStatsMetadata.class, NodeStatsMetadata.Handler.class, METHOD);
+
+ // In order to use this we need to call it by relNode.metadata(NodeStatsMetadata.class,
+ // mq).getNodeStats() where mq is the MetadataQuery (can be obtained by
+ // relNode.getCluster().getMetadataQuery()). After this, Calcite looks for the implementation of
+ // this metadata that we have registered in MetadataProvider (it is RelMdNodeStats.class in
+ // this case and we have registered it in CalciteQueryPlanner). Then Calcite's generated Code
+ // decides the type of the rel node and calls appropriate method in RelMdNodeStats.
+ // For instance: Join is a subclass of RelNode and if we have both getNodeStats(RelNode rel,
+ // RelMetadataQuery mq) and getNodeStats(Join rel, RelMetadataQuery mq) then if the rel is an
+ // instance of Join it will call getNodeStats((Join) rel, mq).
+ // Currently we only register it in SQLTransform path. JDBC does not register this and it does not
+ // use it. (because it does not register the our NonCumulativeMetadata implementation either).
+ NodeStats getNodeStats();
+
+ /** Handler API. */
+ interface Handler extends MetadataHandler<NodeStatsMetadata> {
+ NodeStats getNodeStats(RelNode r, RelMetadataQuery mq);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java
new file mode 100644
index 0000000..0619a4b
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/RelMdNodeStats.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.MetadataHandler;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+
+/**
+ * This is the implementation of NodeStatsMetadata. Methods to estimate rate and row count for
+ * Calcite's logical nodes be implemented here.
+ */
+public class RelMdNodeStats implements MetadataHandler<NodeStatsMetadata> {
+
+ public static final RelMetadataProvider SOURCE =
+ ReflectiveRelMetadataProvider.reflectiveSource(
+ NodeStatsMetadata.METHOD, new RelMdNodeStats());
+
+ @Override
+ public MetadataDef<NodeStatsMetadata> getDef() {
+ return NodeStatsMetadata.DEF;
+ }
+
+ @SuppressWarnings("UnusedDeclaration")
+ public NodeStats getNodeStats(RelNode rel, RelMetadataQuery mq) {
+
+ if (rel instanceof BeamRelNode) {
+ return this.getBeamNodeStats((BeamRelNode) rel, mq);
+ }
+
+ // We can later define custom methods for all different RelNodes to prevent hitting this point.
+ // Similar to RelMdRowCount in calcite.
+
+ return NodeStats.UNKNOWN;
+ }
+
+ private NodeStats getBeamNodeStats(BeamRelNode rel, RelMetadataQuery mq) {
+
+ // Removing the unknown results.
+ // Calcite caches previous results in mq.map. This is done to prevent cyclic calls of this
+ // method and also improving the performance. However, we might have returned an unknown result
+ // because one of the inputs of the node was unknown (it is a logical node that we have not
+ // implemented getNodeStats for it). Later we should not get the Unknown, therefore we need to
+ // remove unknown results everyTime that this method is called.
+ // Results are also cached in CachingRelMetadataProvider because calcite PlannerImpl#Transform
+ // wraps the metadata provider with CachingRelMetadataProvider. However,
+ // CachingRelMetadataProvider checks timestamp before returning previous results. Therefore,
+ // there wouldn't be a problem in that case.
+ List<List> keys =
+ mq.map.entrySet().stream()
+ .filter(entry -> entry.getValue() instanceof NodeStats)
+ .filter(entry -> ((NodeStats) entry.getValue()).isUnknown())
+ .map(Map.Entry::getKey)
+ .collect(Collectors.toList());
+
+ for (List key : keys) {
+ mq.map.remove(key);
+ }
+
+ return rel.estimateNodeStats(mq);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
index 419cef1..453c648 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRel.java
@@ -19,11 +19,13 @@
import static java.util.stream.Collectors.toList;
import static org.apache.beam.sdk.values.PCollection.IsBounded.BOUNDED;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.util.List;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.transform.agg.AggregationCombineFnAdapter;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.schemas.Schema;
@@ -48,14 +50,16 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelWriter;
-import org.apache.calcite.rel.core.Aggregate;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.AggregateCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
/** {@link BeamRelNode} to replace a {@link Aggregate} node. */
@@ -81,6 +85,68 @@
}
@Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+
+ NodeStats inputStat = BeamSqlRelUtils.getNodeStats(this.input, mq);
+ inputStat = computeWindowingCostEffect(inputStat);
+
+ // Aggregates with more aggregate functions cost a bit more
+ float multiplier = 1f + (float) aggCalls.size() * 0.125f;
+ for (AggregateCall aggCall : aggCalls) {
+ if (aggCall.getAggregation().getName().equals("SUM")) {
+ // Pretend that SUM costs a little bit more than $SUM0,
+ // to make things deterministic.
+ multiplier += 0.0125f;
+ }
+ }
+
+ return BeamCostModel.FACTORY.makeCost(
+ inputStat.getRowCount() * multiplier, inputStat.getRate() * multiplier);
+ }
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+
+ NodeStats inputEstimate = BeamSqlRelUtils.getNodeStats(this.input, mq);
+
+ inputEstimate = computeWindowingCostEffect(inputEstimate);
+
+ NodeStats estimate;
+ // groupCount shows how many columns do we have in group by. One of them might be the windowing.
+ int groupCount = groupSet.cardinality() - (windowFn == null ? 0 : 1);
+ // This is similar to what Calcite does.If groupCount is zero then then we have only one value
+ // per window for unbounded and we have only one value for bounded. e.g select count(*) from A
+ // If group count is none zero then more column we include in the group by, more rows will be
+ // preserved.
+ return (groupCount == 0)
+ ? NodeStats.create(
+ Math.min(inputEstimate.getRowCount(), 1d),
+ inputEstimate.getRate() / inputEstimate.getWindow(),
+ 1d)
+ : inputEstimate.multiply(1.0 - Math.pow(.5, groupCount));
+ }
+
+ private NodeStats computeWindowingCostEffect(NodeStats inputStat) {
+ if (windowFn == null) {
+ return inputStat;
+ }
+ WindowFn w = windowFn;
+ double multiplicationFactor = 1;
+ // If the window is SlidingWindow, the number of tuples will increase. (Because, some of the
+ // tuples repeat in multiple windows).
+ if (w instanceof SlidingWindows) {
+ multiplicationFactor =
+ ((double) ((SlidingWindows) w).getSize().getStandardSeconds())
+ / ((SlidingWindows) w).getPeriod().getStandardSeconds();
+ }
+
+ return NodeStats.create(
+ inputStat.getRowCount() * multiplicationFactor,
+ inputStat.getRate() * multiplicationFactor,
+ BeamIOSourceRel.CONSTANT_WINDOW_SIZE);
+ }
+
+ @Override
public RelWriter explainTerms(RelWriter pw) {
super.explainTerms(pw);
if (this.windowFn != null) {
@@ -274,7 +340,6 @@
public Aggregate copy(
RelTraitSet traitSet,
RelNode input,
- boolean indicator,
ImmutableBitSet groupSet,
List<ImmutableBitSet> groupSets,
List<AggregateCall> aggCalls) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
index 790bcf2..3d666aa 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java
@@ -19,18 +19,23 @@
import static org.apache.beam.sdk.schemas.Schema.FieldType;
import static org.apache.beam.sdk.schemas.Schema.TypeName;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
import java.lang.reflect.Type;
import java.math.BigDecimal;
import java.util.AbstractList;
+import java.util.AbstractMap;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamJavaTypeFactory;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.CharType;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.DateType;
@@ -44,34 +49,41 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.DataContext;
-import org.apache.calcite.adapter.enumerable.JavaRowFormat;
-import org.apache.calcite.adapter.enumerable.PhysType;
-import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
-import org.apache.calcite.adapter.enumerable.RexToLixTranslator;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.linq4j.QueryProvider;
-import org.apache.calcite.linq4j.tree.BlockBuilder;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.linq4j.tree.Expressions;
-import org.apache.calcite.linq4j.tree.GotoExpressionKind;
-import org.apache.calcite.linq4j.tree.ParameterExpression;
-import org.apache.calcite.linq4j.tree.Types;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPredicateList;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Calc;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexProgram;
-import org.apache.calcite.rex.RexSimplify;
-import org.apache.calcite.rex.RexUtil;
-import org.apache.calcite.schema.SchemaPlus;
-import org.apache.calcite.sql.validate.SqlConformance;
-import org.apache.calcite.sql.validate.SqlConformanceEnum;
-import org.apache.calcite.util.BuiltInMethod;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.DataContext;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.JavaRowFormat;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexToLixTranslator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.QueryProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.GotoExpressionKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.MemberDeclaration;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.ParameterExpression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Types;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPredicateList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLocalRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexProgram;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexSimplify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlConformance;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.BuiltInMethod;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.codehaus.commons.compiler.CompileException;
import org.codehaus.janino.ScriptEvaluator;
import org.joda.time.DateTime;
@@ -206,6 +218,35 @@
throw new RuntimeException("Could not get the limit count from a non BeamSortRel input.");
}
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ NodeStats inputStat = BeamSqlRelUtils.getNodeStats(this.input, mq);
+ double selectivity = estimateFilterSelectivity(getInput(), program, mq);
+
+ return inputStat.multiply(selectivity);
+ }
+
+ private static double estimateFilterSelectivity(
+ RelNode child, RexProgram program, RelMetadataQuery mq) {
+ // Similar to calcite, if the calc node is representing filter operation we estimate the filter
+ // selectivity based on the number of equality conditions, number of inequality conditions, ....
+ RexLocalRef programCondition = program.getCondition();
+ RexNode condition;
+ if (programCondition == null) {
+ condition = null;
+ } else {
+ condition = program.expandLocalRef(programCondition);
+ }
+ // Currently this gets the selectivity based on Calcite's Selectivity Handler (RelMdSelectivity)
+ return mq.getSelectivity(child, condition);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats inputStat = BeamSqlRelUtils.getNodeStats(this.input, mq);
+ return BeamCostModel.FACTORY.makeCost(inputStat.getRowCount(), inputStat.getRate());
+ }
+
public boolean isInputSortRelAndLimitOnly() {
return (input instanceof BeamSortRel) && ((BeamSortRel) input).isLimitOnly();
}
@@ -276,7 +317,13 @@
} else if (toType.getTypeName() == TypeName.DECIMAL
&& !Types.isAssignableFrom(BigDecimal.class, (Class) value.getType())) {
return Expressions.new_(BigDecimal.class, value);
+ } else if (toType.getTypeName() == TypeName.BYTES
+ && Types.isAssignableFrom(ByteString.class, (Class) value.getType())) {
+ return Expressions.condition(
+ Expressions.equal(value, Expressions.constant(null)),
+ Expressions.constant(null),
+ Expressions.call(value, "getBytes"));
} else if (((Class) value.getType()).isPrimitive()
|| Types.isAssignableFrom(Number.class, (Class) value.getType())) {
Type rawType = rawTypeMap.get(toType.getTypeName());
@@ -328,7 +375,7 @@
}
private static class InputGetterImpl implements RexToLixTranslator.InputGetter {
- private static final Map<TypeName, String> typeGetterMap =
+ private static final Map<TypeName, String> TYPE_GETTER_MAP =
ImmutableMap.<TypeName, String>builder()
.put(TypeName.BYTE, "getByte")
.put(TypeName.BYTES, "getBytes")
@@ -346,7 +393,7 @@
.put(TypeName.ROW, "getRow")
.build();
- private static final Map<String, String> logicalTypeGetterMap =
+ private static final Map<String, String> LOGICAL_TYPE_GETTER_MAP =
ImmutableMap.<String, String>builder()
.put(DateType.IDENTIFIER, "getDateTime")
.put(TimeType.IDENTIFIER, "getDateTime")
@@ -365,48 +412,143 @@
@Override
public Expression field(BlockBuilder list, int index, Type storageType) {
- if (index >= inputSchema.getFieldCount() || index < 0) {
- throw new IllegalArgumentException("Unable to find field #" + index);
+ return value(list, index, storageType, input, inputSchema);
+ }
+
+ private static Expression value(
+ BlockBuilder list, int index, Type storageType, Expression input, Schema schema) {
+ if (index >= schema.getFieldCount() || index < 0) {
+ throw new IllegalArgumentException("Unable to find value #" + index);
}
- final Expression expression = list.append("current", input);
+ final Expression expression = list.append(list.newName("current"), input);
if (storageType == Object.class) {
return Expressions.convert_(
Expressions.call(expression, "getValue", Expressions.constant(index)), Object.class);
}
- FieldType fromType = inputSchema.getField(index).getType();
+ FieldType fromType = schema.getField(index).getType();
String getter;
if (fromType.getTypeName().isLogicalType()) {
- getter = logicalTypeGetterMap.get(fromType.getLogicalType().getIdentifier());
+ getter = LOGICAL_TYPE_GETTER_MAP.get(fromType.getLogicalType().getIdentifier());
} else {
- getter = typeGetterMap.get(fromType.getTypeName());
+ getter = TYPE_GETTER_MAP.get(fromType.getTypeName());
}
if (getter == null) {
throw new IllegalArgumentException("Unable to get " + fromType.getTypeName());
}
- Expression field = Expressions.call(expression, getter, Expressions.constant(index));
- if (fromType.getTypeName().isLogicalType()) {
- field = Expressions.call(field, "getMillis");
- String logicalId = fromType.getLogicalType().getIdentifier();
+
+ Expression value = Expressions.call(expression, getter, Expressions.constant(index));
+
+ return value(value, fromType);
+ }
+
+ private static Expression value(Expression value, Schema.FieldType type) {
+ if (type.getTypeName().isLogicalType()) {
+ Expression millisField = Expressions.call(value, "getMillis");
+ String logicalId = type.getLogicalType().getIdentifier();
if (logicalId.equals(TimeType.IDENTIFIER)) {
- field = Expressions.convert_(field, int.class);
+ return nullOr(value, Expressions.convert_(millisField, int.class));
} else if (logicalId.equals(DateType.IDENTIFIER)) {
- field =
- Expressions.convert_(
- Expressions.modulo(field, Expressions.constant(MILLIS_PER_DAY)), int.class);
+ value =
+ nullOr(
+ value,
+ Expressions.convert_(
+ Expressions.divide(millisField, Expressions.constant(MILLIS_PER_DAY)),
+ int.class));
} else if (!logicalId.equals(CharType.IDENTIFIER)) {
throw new IllegalArgumentException(
- "Unknown LogicalType " + fromType.getLogicalType().getIdentifier());
+ "Unknown LogicalType " + type.getLogicalType().getIdentifier());
}
- } else if (CalciteUtils.isDateTimeType(fromType)) {
- field = Expressions.call(field, "getMillis");
- } else if (fromType.getTypeName().isCompositeType()
- || (fromType.getTypeName().isCollectionType()
- && fromType.getCollectionElementType().getTypeName().isCompositeType())) {
- field = Expressions.call(WrappedList.class, "of", field);
+ } else if (type.getTypeName().isMapType()) {
+ return nullOr(value, map(value, type.getMapValueType()));
+ } else if (CalciteUtils.isDateTimeType(type)) {
+ return nullOr(value, Expressions.call(value, "getMillis"));
+ } else if (type.getTypeName().isCompositeType()) {
+ return nullOr(value, row(value, type.getRowSchema()));
+ } else if (type.getTypeName().isCollectionType()) {
+ return nullOr(value, list(value, type.getCollectionElementType()));
+ } else if (type.getTypeName() == TypeName.BYTES) {
+ return nullOr(
+ value, Expressions.new_(ByteString.class, Types.castIfNecessary(byte[].class, value)));
}
- return field;
+
+ return value;
}
+
+ private static Expression list(Expression input, FieldType elementType) {
+ ParameterExpression value = Expressions.parameter(Object.class);
+
+ BlockBuilder block = new BlockBuilder();
+ block.add(value(value, elementType));
+
+ return Expressions.new_(
+ WrappedList.class,
+ ImmutableList.of(Types.castIfNecessary(List.class, input)),
+ ImmutableList.<MemberDeclaration>of(
+ Expressions.methodDecl(
+ Modifier.PUBLIC,
+ Object.class,
+ "value",
+ ImmutableList.of(value),
+ block.toBlock())));
+ }
+
+ private static Expression map(Expression input, FieldType mapValueType) {
+ ParameterExpression value = Expressions.parameter(Object.class);
+
+ BlockBuilder block = new BlockBuilder();
+ block.add(value(value, mapValueType));
+
+ return Expressions.new_(
+ WrappedMap.class,
+ ImmutableList.of(Types.castIfNecessary(Map.class, input)),
+ ImmutableList.<MemberDeclaration>of(
+ Expressions.methodDecl(
+ Modifier.PUBLIC,
+ Object.class,
+ "value",
+ ImmutableList.of(value),
+ block.toBlock())));
+ }
+
+ private static Expression row(Expression input, Schema schema) {
+ ParameterExpression row = Expressions.parameter(Row.class);
+ ParameterExpression index = Expressions.parameter(int.class);
+ BlockBuilder body = new BlockBuilder(/* optimizing= */ false);
+
+ for (int i = 0; i < schema.getFieldCount(); i++) {
+ BlockBuilder list = new BlockBuilder(/* optimizing= */ false, body);
+ Expression returnValue = value(list, i, /* storageType= */ null, row, schema);
+
+ list.append(returnValue);
+
+ body.append(
+ "if i=" + i,
+ Expressions.block(
+ Expressions.ifThen(
+ Expressions.equal(index, Expressions.constant(i, int.class)), list.toBlock())));
+ }
+
+ body.add(Expressions.throw_(Expressions.new_(IndexOutOfBoundsException.class)));
+
+ return Expressions.new_(
+ WrappedRow.class,
+ ImmutableList.of(Types.castIfNecessary(Row.class, input)),
+ ImmutableList.<MemberDeclaration>of(
+ Expressions.methodDecl(
+ Modifier.PUBLIC,
+ Object.class,
+ "field",
+ ImmutableList.of(row, index),
+ body.toBlock())));
+ }
+ }
+
+ private static Expression nullOr(Expression field, Expression ifNotNull) {
+ return Expressions.condition(
+ Expressions.equal(field, Expressions.constant(null)),
+ Expressions.constant(null),
+ Expressions.box(ifNotNull));
}
private static final DataContext CONTEXT_INSTANCE = new SlimDataContext();
@@ -439,40 +581,73 @@
}
}
- /** WrappedList translates {@code Row} and {@code List} on access. */
- public static class WrappedList extends AbstractList<Object> {
+ /** WrappedRow translates {@code Row} on access. */
+ public abstract static class WrappedRow extends AbstractList<Object> {
+ private final Row row;
- private final List<Object> list;
-
- private WrappedList(List<Object> list) {
- this.list = list;
- }
-
- public static List<Object> of(List list) {
- if (list instanceof WrappedList) {
- return list;
- }
- return new WrappedList(list);
- }
-
- public static List<Object> of(Row row) {
- return new WrappedList(row.getValues());
+ protected WrappedRow(Row row) {
+ this.row = row;
}
@Override
public Object get(int index) {
- Object obj = list.get(index);
- if (obj instanceof Row) {
- obj = of((Row) obj);
- } else if (obj instanceof List) {
- obj = of((List) obj);
- }
- return obj;
+ return field(row, index);
}
+ // we could override get(int index) if we knew how to access `this.row` in linq4j
+ // for now we keep it consistent with WrappedList
+ protected abstract Object field(Row row, int index);
+
@Override
public int size() {
- return list.size();
+ return row.getFieldCount();
+ }
+ }
+
+ /** WrappedMap translates {@code Map} on access. */
+ public abstract static class WrappedMap<V> extends AbstractMap<Object, V> {
+ private final Map<Object, Object> map;
+
+ protected WrappedMap(Map<Object, Object> map) {
+ this.map = map;
+ }
+
+ // TODO transform keys, in this case, we need to do lookup, so it should be both ways:
+ //
+ // public abstract Object fromKey(K key)
+ // public abstract K toKey(Object key)
+
+ @Override
+ public Set<Entry<Object, V>> entrySet() {
+ return Maps.transformValues(map, val -> (val == null) ? null : value(val)).entrySet();
+ }
+
+ @Override
+ public V get(Object key) {
+ return value(map.get(key));
+ }
+
+ protected abstract V value(Object value);
+ }
+
+ /** WrappedList translates {@code List} on access. */
+ public abstract static class WrappedList<T> extends AbstractList<T> {
+ private final List<Object> values;
+
+ protected WrappedList(List<Object> values) {
+ this.values = values;
+ }
+
+ @Override
+ public T get(int index) {
+ return value(values.get(index));
+ }
+
+ protected abstract T value(Object value);
+
+ @Override
+ public int size() {
+ return values.size();
}
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java
new file mode 100644
index 0000000..bef3cb7
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import static org.apache.beam.sdk.values.PCollection.IsBounded.UNBOUNDED;
+import static org.joda.time.Duration.ZERO;
+
+import java.util.Set;
+import org.apache.beam.sdk.extensions.sql.impl.transform.BeamJoinTransforms;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.IncompatibleWindowException;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * A {@code BeamJoinRel} which does CoGBK Join
+ *
+ * <p>This Join Covers the cases:
+ *
+ * <ul>
+ * <li>BoundedTable JOIN BoundedTable
+ * <li>UnboundedTable JOIN UnboundedTable
+ * </ul>
+ *
+ * <p>A CoGBK join is utilized as long as the windowFn of the both sides match. For more info refer
+ * <a href="https://issues.apache.org/jira/browse/BEAM-3345">BEAM-3345</a>
+ *
+ * <p>General constraints:
+ *
+ * <ul>
+ * <li>Only equi-join is supported.
+ * <li>CROSS JOIN is not supported.
+ * </ul>
+ */
+public class BeamCoGBKJoinRel extends BeamJoinRel {
+
+ public BeamCoGBKJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ Set<CorrelationId> variablesSet,
+ JoinRelType joinType) {
+ super(cluster, traitSet, left, right, condition, variablesSet, joinType);
+ }
+
+ @Override
+ public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
+ return new StandardJoin();
+ }
+
+ private class StandardJoin extends PTransform<PCollectionList<Row>, PCollection<Row>> {
+
+ @Override
+ public PCollection<Row> expand(PCollectionList<Row> pinput) {
+ Schema leftSchema = CalciteUtils.toSchema(left.getRowType());
+ Schema rightSchema = CalciteUtils.toSchema(right.getRowType());
+
+ PCollectionList<KV<Row, Row>> keyedInputs = pinput.apply(new ExtractJoinKeys());
+
+ PCollection<KV<Row, Row>> extractedLeftRows = keyedInputs.get(0);
+ PCollection<KV<Row, Row>> extractedRightRows = keyedInputs.get(1);
+
+ WindowFn leftWinFn = extractedLeftRows.getWindowingStrategy().getWindowFn();
+ WindowFn rightWinFn = extractedRightRows.getWindowingStrategy().getWindowFn();
+
+ try {
+ leftWinFn.verifyCompatibility(rightWinFn);
+ } catch (IncompatibleWindowException e) {
+ throw new IllegalArgumentException(
+ "WindowFns must match for a bounded-vs-bounded/unbounded-vs-unbounded join.", e);
+ }
+
+ verifySupportedTrigger(extractedLeftRows);
+ verifySupportedTrigger(extractedRightRows);
+
+ return standardJoin(extractedLeftRows, extractedRightRows, leftSchema, rightSchema);
+ }
+ }
+
+ private <T> void verifySupportedTrigger(PCollection<T> pCollection) {
+ WindowingStrategy windowingStrategy = pCollection.getWindowingStrategy();
+
+ if (UNBOUNDED.equals(pCollection.isBounded()) && !triggersOncePerWindow(windowingStrategy)) {
+ throw new UnsupportedOperationException(
+ "Joining unbounded PCollections is currently only supported for "
+ + "non-global windows with triggers that are known to produce output once per window,"
+ + "such as the default trigger with zero allowed lateness. "
+ + "In these cases Beam can guarantee it joins all input elements once per window. "
+ + windowingStrategy
+ + " is not supported");
+ }
+ }
+
+ private boolean triggersOncePerWindow(WindowingStrategy windowingStrategy) {
+ Trigger trigger = windowingStrategy.getTrigger();
+
+ return !(windowingStrategy.getWindowFn() instanceof GlobalWindows)
+ && trigger instanceof DefaultTrigger
+ && ZERO.equals(windowingStrategy.getAllowedLateness());
+ }
+
+ private PCollection<Row> standardJoin(
+ PCollection<KV<Row, Row>> extractedLeftRows,
+ PCollection<KV<Row, Row>> extractedRightRows,
+ Schema leftSchema,
+ Schema rightSchema) {
+ PCollection<KV<Row, KV<Row, Row>>> joinedRows = null;
+
+ switch (joinType) {
+ case LEFT:
+ {
+ Schema rigthNullSchema = buildNullSchema(rightSchema);
+ Row rightNullRow = Row.nullRow(rigthNullSchema);
+
+ extractedRightRows = setValueCoder(extractedRightRows, SchemaCoder.of(rigthNullSchema));
+
+ joinedRows =
+ org.apache.beam.sdk.extensions.joinlibrary.Join.leftOuterJoin(
+ extractedLeftRows, extractedRightRows, rightNullRow);
+
+ break;
+ }
+ case RIGHT:
+ {
+ Schema leftNullSchema = buildNullSchema(leftSchema);
+ Row leftNullRow = Row.nullRow(leftNullSchema);
+
+ extractedLeftRows = setValueCoder(extractedLeftRows, SchemaCoder.of(leftNullSchema));
+
+ joinedRows =
+ org.apache.beam.sdk.extensions.joinlibrary.Join.rightOuterJoin(
+ extractedLeftRows, extractedRightRows, leftNullRow);
+ break;
+ }
+ case FULL:
+ {
+ Schema leftNullSchema = buildNullSchema(leftSchema);
+ Schema rightNullSchema = buildNullSchema(rightSchema);
+
+ Row leftNullRow = Row.nullRow(leftNullSchema);
+ Row rightNullRow = Row.nullRow(rightNullSchema);
+
+ extractedLeftRows = setValueCoder(extractedLeftRows, SchemaCoder.of(leftNullSchema));
+ extractedRightRows = setValueCoder(extractedRightRows, SchemaCoder.of(rightNullSchema));
+
+ joinedRows =
+ org.apache.beam.sdk.extensions.joinlibrary.Join.fullOuterJoin(
+ extractedLeftRows, extractedRightRows, leftNullRow, rightNullRow);
+ break;
+ }
+ case INNER:
+ default:
+ joinedRows =
+ org.apache.beam.sdk.extensions.joinlibrary.Join.innerJoin(
+ extractedLeftRows, extractedRightRows);
+ break;
+ }
+
+ Schema schema = CalciteUtils.toSchema(getRowType());
+ return joinedRows
+ .apply(
+ "JoinParts2WholeRow",
+ MapElements.via(new BeamJoinTransforms.JoinParts2WholeRow(schema)))
+ .setRowSchema(schema);
+ }
+
+ @Override
+ public Join copy(
+ RelTraitSet traitSet,
+ RexNode conditionExpr,
+ RelNode left,
+ RelNode right,
+ JoinRelType joinType,
+ boolean semiJoinDone) {
+ return new BeamCoGBKJoinRel(
+ getCluster(), traitSet, left, right, conditionExpr, variablesSet, joinType);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
index a7dc532..cdd1444 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverter.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
import java.io.IOException;
import java.util.Iterator;
@@ -54,24 +54,24 @@
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.adapter.enumerable.EnumerableRel;
-import org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
-import org.apache.calcite.adapter.enumerable.PhysType;
-import org.apache.calcite.adapter.enumerable.PhysTypeImpl;
-import org.apache.calcite.linq4j.Enumerable;
-import org.apache.calcite.linq4j.Linq4j;
-import org.apache.calcite.linq4j.tree.BlockBuilder;
-import org.apache.calcite.linq4j.tree.Expression;
-import org.apache.calcite.linq4j.tree.Expressions;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptCost;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterImpl;
-import org.apache.calcite.rel.metadata.RelMetadataQuery;
-import org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableRelImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.PhysTypeImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Linq4j;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.BlockBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
import org.joda.time.Duration;
import org.joda.time.ReadableInstant;
import org.slf4j.Logger;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
index d302a22..3af4509 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSinkRel.java
@@ -17,25 +17,28 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.List;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamIOSinkRule;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.prepare.Prepare;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.TableModify;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql2rel.RelStructuredTypeFlattener;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.Prepare;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableModify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql2rel.RelStructuredTypeFlattener;
/** BeamRelNode to replace a {@code TableModify} node. */
public class BeamIOSinkRel extends TableModify
@@ -71,6 +74,17 @@
}
@Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ return BeamSqlRelUtils.getNodeStats(this.input, mq);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats inputEstimates = BeamSqlRelUtils.getNodeStats(this.input, mq);
+ return BeamCostModel.FACTORY.makeCost(inputEstimates.getRowCount(), inputEstimates.getRate());
+ }
+
+ @Override
public RelNode copy(RelTraitSet traitSet, List<RelNode> inputs) {
boolean flattened = isFlattened() || isFlattening;
BeamIOSinkRel newRel =
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
index 706ebe9..480ccab 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRel.java
@@ -17,37 +17,67 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelOptTable;
-import org.apache.calcite.rel.core.TableScan;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCost;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableScan;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** BeamRelNode to replace a {@code TableScan} node. */
public class BeamIOSourceRel extends TableScan implements BeamRelNode {
-
- private final BeamSqlTable sqlTable;
+ public static final double CONSTANT_WINDOW_SIZE = 10d;
+ private final BeamSqlTable beamTable;
+ private final BeamCalciteTable calciteTable;
private final Map<String, String> pipelineOptions;
public BeamIOSourceRel(
RelOptCluster cluster,
RelOptTable table,
- BeamSqlTable sqlTable,
- Map<String, String> pipelineOptions) {
+ BeamSqlTable beamTable,
+ Map<String, String> pipelineOptions,
+ BeamCalciteTable calciteTable) {
super(cluster, cluster.traitSetOf(BeamLogicalConvention.INSTANCE), table);
- this.sqlTable = sqlTable;
+ this.beamTable = beamTable;
+ this.calciteTable = calciteTable;
this.pipelineOptions = pipelineOptions;
}
@Override
+ public double estimateRowCount(RelMetadataQuery mq) {
+ BeamTableStatistics rowCountStatistics = calciteTable.getStatistic();
+ if (beamTable.isBounded() == PCollection.IsBounded.BOUNDED) {
+ return rowCountStatistics.getRowCount();
+ } else {
+ return rowCountStatistics.getRate();
+ }
+ }
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ BeamTableStatistics rowCountStatistics = calciteTable.getStatistic();
+ double window =
+ (beamTable.isBounded() == PCollection.IsBounded.BOUNDED)
+ ? rowCountStatistics.getRowCount()
+ : CONSTANT_WINDOW_SIZE;
+ return NodeStats.create(rowCountStatistics.getRowCount(), rowCountStatistics.getRate(), window);
+ }
+
+ @Override
public PCollection.IsBounded isBounded() {
- return sqlTable.isBounded();
+ return beamTable.isBounded();
}
@Override
@@ -64,12 +94,26 @@
"Should not have received input for %s: %s",
BeamIOSourceRel.class.getSimpleName(),
input);
- return sqlTable.buildIOReader(input.getPipeline().begin());
+ return beamTable.buildIOReader(input.getPipeline().begin());
}
}
+ @Override
+ public RelOptCost computeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ // We should technically avoid this function. This happens if we are in JDBC path or the
+ // costFactory is not set correctly.
+ double rowCount = this.estimateRowCount(mq);
+ return planner.getCostFactory().makeCost(rowCount, rowCount, rowCount);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats estimates = BeamSqlRelUtils.getNodeStats(this, mq);
+ return BeamCostModel.FACTORY.makeCost(estimates.getRowCount(), estimates.getRate());
+ }
+
protected BeamSqlTable getBeamSqlTable() {
- return sqlTable;
+ return beamTable;
}
@Override
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java
index cc2590e..80db503 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRel.java
@@ -18,15 +18,19 @@
package org.apache.beam.sdk.extensions.sql.impl.rel;
import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Intersect;
-import org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Intersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* {@code BeamRelNode} to replace a {@code Intersect} node.
@@ -49,4 +53,33 @@
public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
return new BeamSetOperatorRelBase(this, BeamSetOperatorRelBase.OpType.INTERSECT, all);
}
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ // This takes the minimum of the inputs for all the estimate factors.
+ double minimumRows = Double.POSITIVE_INFINITY;
+ double minimumWindowSize = Double.POSITIVE_INFINITY;
+ double minimumRate = Double.POSITIVE_INFINITY;
+
+ for (RelNode input : inputs) {
+ NodeStats inputEstimates = BeamSqlRelUtils.getNodeStats(input, mq);
+ minimumRows = Math.min(minimumRows, inputEstimates.getRowCount());
+ minimumRate = Math.min(minimumRate, inputEstimates.getRate());
+ minimumWindowSize = Math.min(minimumWindowSize, inputEstimates.getWindow());
+ }
+
+ return NodeStats.create(minimumRows, minimumRate, minimumWindowSize).multiply(0.5);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+
+ NodeStats inputsStatSummation =
+ inputs.stream()
+ .map(input -> BeamSqlRelUtils.getNodeStats(input, mq))
+ .reduce(NodeStats.create(0, 0, 0), NodeStats::plus);
+
+ return BeamCostModel.FACTORY.makeCost(
+ inputsStatSummation.getRowCount(), inputsStatSummation.getRate());
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
index 08a1a33..b595bdb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java
@@ -18,89 +18,64 @@
package org.apache.beam.sdk.extensions.sql.impl.rel;
import static org.apache.beam.sdk.schemas.Schema.toSchema;
-import static org.apache.beam.sdk.values.PCollection.IsBounded.UNBOUNDED;
-import static org.joda.time.Duration.ZERO;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.List;
-import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.transform.BeamJoinTransforms;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
-import org.apache.beam.sdk.transforms.windowing.IncompatibleWindowException;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
-import org.apache.beam.sdk.transforms.windowing.Trigger;
import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
-import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.CorrelationId;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Optional;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/**
- * {@code BeamRelNode} to replace a {@code Join} node.
+ * An abstract {@code BeamRelNode} to implement Join Rels.
*
- * <p>Support for join can be categorized into 3 cases:
+ * <p>Support for join can be categorized into 4 cases:
*
* <ul>
* <li>BoundedTable JOIN BoundedTable
* <li>UnboundedTable JOIN UnboundedTable
* <li>BoundedTable JOIN UnboundedTable
- * </ul>
- *
- * <p>For the first two cases, a standard join is utilized as long as the windowFn of the both sides
- * match.
- *
- * <p>For the third case, {@code sideInput} is utilized to implement the join, so there are some
- * constraints:
- *
- * <ul>
- * <li>{@code FULL OUTER JOIN} is not supported.
- * <li>If it's a {@code LEFT OUTER JOIN}, the unbounded table should on the left side.
- * <li>If it's a {@code RIGHT OUTER JOIN}, the unbounded table should on the right side.
- * </ul>
- *
- * <p>There are also some general constraints:
- *
- * <ul>
- * <li>Only equi-join is supported.
- * <li>CROSS JOIN is not supported.
+ * <li>SeekableTable JOIN non SeekableTable
* </ul>
*/
-public class BeamJoinRel extends Join implements BeamRelNode {
+public abstract class BeamJoinRel extends Join implements BeamRelNode {
- public BeamJoinRel(
+ protected BeamJoinRel(
RelOptCluster cluster,
RelTraitSet traits,
RelNode left,
@@ -112,18 +87,6 @@
}
@Override
- public Join copy(
- RelTraitSet traitSet,
- RexNode conditionExpr,
- RelNode left,
- RelNode right,
- JoinRelType joinType,
- boolean semiJoinDone) {
- return new BeamJoinRel(
- getCluster(), traitSet, left, right, conditionExpr, variablesSet, joinType);
- }
-
- @Override
public List<RelNode> getPCollectionInputs() {
if (isSideInputLookupJoin()) {
return ImmutableList.of(
@@ -133,52 +96,11 @@
}
}
- @Override
- public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
- if (isSideInputLookupJoin()) {
- return new SideInputLookupJoin();
- } else if (isSideInputJoin()) {
- // if one of the sides is Bounded & the other is Unbounded
- // then do a sideInput join
- // when doing a sideInput join, the windowFn does not need to match
- // Only support INNER JOIN & LEFT OUTER JOIN where left side of the join must be
- // the unbounded
- if (joinType == JoinRelType.FULL) {
- throw new UnsupportedOperationException(
- "FULL OUTER JOIN is not supported when join "
- + "a bounded table with an unbounded table.");
- }
-
- BeamRelNode leftRelNode = BeamSqlRelUtils.getBeamRelInput(left);
- BeamRelNode rightRelNode = BeamSqlRelUtils.getBeamRelInput(right);
-
- if ((joinType == JoinRelType.LEFT && leftRelNode.isBounded() == PCollection.IsBounded.BOUNDED)
- || (joinType == JoinRelType.RIGHT
- && rightRelNode.isBounded() == PCollection.IsBounded.BOUNDED)) {
- throw new UnsupportedOperationException(
- "LEFT side of an OUTER JOIN must be Unbounded table.");
- }
-
- return new SideInputJoin();
- } else {
- return new StandardJoin();
- }
- }
-
- private boolean isSideInputJoin() {
- BeamRelNode leftRelNode = BeamSqlRelUtils.getBeamRelInput(left);
- BeamRelNode rightRelNode = BeamSqlRelUtils.getBeamRelInput(right);
- return (leftRelNode.isBounded() == PCollection.IsBounded.BOUNDED
- && rightRelNode.isBounded() == UNBOUNDED)
- || (leftRelNode.isBounded() == UNBOUNDED
- && rightRelNode.isBounded() == PCollection.IsBounded.BOUNDED);
- }
-
- private boolean isSideInputLookupJoin() {
+ protected boolean isSideInputLookupJoin() {
return seekableInputIndex().isPresent() && nonSeekableInputIndex().isPresent();
}
- private Optional<Integer> seekableInputIndex() {
+ protected Optional<Integer> seekableInputIndex() {
BeamRelNode leftRelNode = BeamSqlRelUtils.getBeamRelInput(left);
BeamRelNode rightRelNode = BeamSqlRelUtils.getBeamRelInput(right);
return seekable(leftRelNode)
@@ -186,7 +108,7 @@
: seekable(rightRelNode) ? Optional.of(1) : Optional.absent();
}
- private Optional<Integer> nonSeekableInputIndex() {
+ protected Optional<Integer> nonSeekableInputIndex() {
BeamRelNode leftRelNode = BeamSqlRelUtils.getBeamRelInput(left);
BeamRelNode rightRelNode = BeamSqlRelUtils.getBeamRelInput(right);
return !seekable(leftRelNode)
@@ -194,50 +116,65 @@
: !seekable(rightRelNode) ? Optional.of(1) : Optional.absent();
}
- private class SideInputLookupJoin extends PTransform<PCollectionList<Row>, PCollection<Row>> {
-
- @Override
- public PCollection<Row> expand(PCollectionList<Row> pinput) {
- Schema schema = CalciteUtils.toSchema(getRowType());
-
- BeamRelNode seekableRel =
- BeamSqlRelUtils.getBeamRelInput(getInput(seekableInputIndex().get()));
- BeamRelNode nonSeekableRel =
- BeamSqlRelUtils.getBeamRelInput(getInput(nonSeekableInputIndex().get()));
-
- // Offset field references according to which table is on the left
- int factColOffset =
- nonSeekableInputIndex().get() == 0
- ? 0
- : CalciteUtils.toSchema(seekableRel.getRowType()).getFieldCount();
- int lkpColOffset =
- seekableInputIndex().get() == 0
- ? 0
- : CalciteUtils.toSchema(nonSeekableRel.getRowType()).getFieldCount();
-
- // HACK: if the input is an immediate instance of a seekable IO, we can do lookups
- // so we ignore the PCollection
- BeamIOSourceRel seekableInput = (BeamIOSourceRel) seekableRel;
- BeamSqlSeekableTable seekableTable = (BeamSqlSeekableTable) seekableInput.getBeamSqlTable();
-
- // getPCollectionInputs() ensures that there is only one and it is the non-seekable input
- PCollection<Row> nonSeekableInput = pinput.get(0);
-
- return nonSeekableInput
- .apply(
- "join_as_lookup",
- new BeamJoinTransforms.JoinAsLookup(
- condition,
- seekableTable,
- CalciteUtils.toSchema(seekableInput.getRowType()),
- schema,
- factColOffset,
- lkpColOffset))
- .setRowSchema(schema);
+ /** check if {@code BeamRelNode} implements {@code BeamSeekableTable}. */
+ public static boolean seekable(BeamRelNode relNode) {
+ if (relNode instanceof BeamIOSourceRel) {
+ BeamIOSourceRel srcRel = (BeamIOSourceRel) relNode;
+ BeamSqlTable sourceTable = srcRel.getBeamSqlTable();
+ if (sourceTable instanceof BeamSqlSeekableTable) {
+ return true;
+ }
}
+ return false;
}
- private class ExtractJoinKeys
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats leftEstimates = BeamSqlRelUtils.getNodeStats(this.left, mq);
+ NodeStats rightEstimates = BeamSqlRelUtils.getNodeStats(this.right, mq);
+ NodeStats selfEstimates = BeamSqlRelUtils.getNodeStats(this, mq);
+ NodeStats summation = selfEstimates.plus(leftEstimates).plus(rightEstimates);
+ return BeamCostModel.FACTORY.makeCost(summation.getRowCount(), summation.getRate());
+ }
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ double selectivity = mq.getSelectivity(this, getCondition());
+ NodeStats leftEstimates = BeamSqlRelUtils.getNodeStats(this.left, mq);
+ NodeStats rightEstimates = BeamSqlRelUtils.getNodeStats(this.right, mq);
+
+ if (leftEstimates.isUnknown() || rightEstimates.isUnknown()) {
+ return NodeStats.UNKNOWN;
+ }
+ // If any of the inputs are unbounded row count becomes zero (one of them would be zero)
+ // If one is bounded and one unbounded the rate will be window of the bounded (= its row count)
+ // multiplied by the rate of the unbounded one
+ // If both are unbounded, the rate will be multiplication of each rate into the window of the
+ // other.
+ return NodeStats.create(
+ leftEstimates.getRowCount() * rightEstimates.getRowCount() * selectivity,
+ (leftEstimates.getRate() * rightEstimates.getWindow()
+ + rightEstimates.getRate() * leftEstimates.getWindow())
+ * selectivity,
+ leftEstimates.getWindow() * rightEstimates.getWindow() * selectivity);
+ }
+
+ /**
+ * This method checks if a join is legal and can be converted into Beam SQL. It is used during
+ * planning and applying {@link
+ * org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinAssociateRule} and {@link
+ * org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinPushThroughJoinRule}
+ */
+ public static boolean isJoinLegal(Join join) {
+ try {
+ extractJoinRexNodes(join.getCondition());
+ } catch (UnsupportedOperationException e) {
+ return false;
+ }
+ return true;
+ }
+
+ protected class ExtractJoinKeys
extends PTransform<PCollectionList<Row>, PCollectionList<KV<Row, Row>>> {
@Override
@@ -254,7 +191,7 @@
int leftRowColumnCount = leftRelNode.getRowType().getFieldCount();
// extract the join fields
- List<Pair<RexNode, RexNode>> pairs = extractJoinRexNodes();
+ List<Pair<RexNode, RexNode>> pairs = extractJoinRexNodes(condition);
// build the extract key type
// the name of the join field is not important
@@ -298,189 +235,7 @@
}
}
- private class SideInputJoin extends PTransform<PCollectionList<Row>, PCollection<Row>> {
-
- @Override
- public PCollection<Row> expand(PCollectionList<Row> pinput) {
- Schema leftSchema = CalciteUtils.toSchema(left.getRowType());
- Schema rightSchema = CalciteUtils.toSchema(right.getRowType());
-
- PCollectionList<KV<Row, Row>> keyedInputs = pinput.apply(new ExtractJoinKeys());
-
- PCollection<KV<Row, Row>> extractedLeftRows = keyedInputs.get(0);
- PCollection<KV<Row, Row>> extractedRightRows = keyedInputs.get(1);
-
- return sideInputJoin(extractedLeftRows, extractedRightRows, leftSchema, rightSchema);
- }
- }
-
- private class StandardJoin extends PTransform<PCollectionList<Row>, PCollection<Row>> {
-
- @Override
- public PCollection<Row> expand(PCollectionList<Row> pinput) {
- Schema leftSchema = CalciteUtils.toSchema(left.getRowType());
- Schema rightSchema = CalciteUtils.toSchema(right.getRowType());
-
- PCollectionList<KV<Row, Row>> keyedInputs = pinput.apply(new ExtractJoinKeys());
-
- PCollection<KV<Row, Row>> extractedLeftRows = keyedInputs.get(0);
- PCollection<KV<Row, Row>> extractedRightRows = keyedInputs.get(1);
-
- WindowFn leftWinFn = extractedLeftRows.getWindowingStrategy().getWindowFn();
- WindowFn rightWinFn = extractedRightRows.getWindowingStrategy().getWindowFn();
-
- try {
- leftWinFn.verifyCompatibility(rightWinFn);
- } catch (IncompatibleWindowException e) {
- throw new IllegalArgumentException(
- "WindowFns must match for a bounded-vs-bounded/unbounded-vs-unbounded join.", e);
- }
-
- verifySupportedTrigger(extractedLeftRows);
- verifySupportedTrigger(extractedRightRows);
-
- return standardJoin(extractedLeftRows, extractedRightRows, leftSchema, rightSchema);
- }
- }
-
- private <T> void verifySupportedTrigger(PCollection<T> pCollection) {
- WindowingStrategy windowingStrategy = pCollection.getWindowingStrategy();
-
- if (UNBOUNDED.equals(pCollection.isBounded()) && !triggersOncePerWindow(windowingStrategy)) {
- throw new UnsupportedOperationException(
- "Joining unbounded PCollections is currently only supported for "
- + "non-global windows with triggers that are known to produce output once per window,"
- + "such as the default trigger with zero allowed lateness. "
- + "In these cases Beam can guarantee it joins all input elements once per window. "
- + windowingStrategy
- + " is not supported");
- }
- }
-
- private boolean triggersOncePerWindow(WindowingStrategy windowingStrategy) {
- Trigger trigger = windowingStrategy.getTrigger();
-
- return !(windowingStrategy.getWindowFn() instanceof GlobalWindows)
- && trigger instanceof DefaultTrigger
- && ZERO.equals(windowingStrategy.getAllowedLateness());
- }
-
- private PCollection<Row> standardJoin(
- PCollection<KV<Row, Row>> extractedLeftRows,
- PCollection<KV<Row, Row>> extractedRightRows,
- Schema leftSchema,
- Schema rightSchema) {
- PCollection<KV<Row, KV<Row, Row>>> joinedRows = null;
-
- switch (joinType) {
- case LEFT:
- {
- Schema rigthNullSchema = buildNullSchema(rightSchema);
- Row rightNullRow = Row.nullRow(rigthNullSchema);
-
- extractedRightRows = setValueCoder(extractedRightRows, SchemaCoder.of(rigthNullSchema));
-
- joinedRows =
- org.apache.beam.sdk.extensions.joinlibrary.Join.leftOuterJoin(
- extractedLeftRows, extractedRightRows, rightNullRow);
-
- break;
- }
- case RIGHT:
- {
- Schema leftNullSchema = buildNullSchema(leftSchema);
- Row leftNullRow = Row.nullRow(leftNullSchema);
-
- extractedLeftRows = setValueCoder(extractedLeftRows, SchemaCoder.of(leftNullSchema));
-
- joinedRows =
- org.apache.beam.sdk.extensions.joinlibrary.Join.rightOuterJoin(
- extractedLeftRows, extractedRightRows, leftNullRow);
- break;
- }
- case FULL:
- {
- Schema leftNullSchema = buildNullSchema(leftSchema);
- Schema rightNullSchema = buildNullSchema(rightSchema);
-
- Row leftNullRow = Row.nullRow(leftNullSchema);
- Row rightNullRow = Row.nullRow(rightNullSchema);
-
- extractedLeftRows = setValueCoder(extractedLeftRows, SchemaCoder.of(leftNullSchema));
- extractedRightRows = setValueCoder(extractedRightRows, SchemaCoder.of(rightNullSchema));
-
- joinedRows =
- org.apache.beam.sdk.extensions.joinlibrary.Join.fullOuterJoin(
- extractedLeftRows, extractedRightRows, leftNullRow, rightNullRow);
- break;
- }
- case INNER:
- default:
- joinedRows =
- org.apache.beam.sdk.extensions.joinlibrary.Join.innerJoin(
- extractedLeftRows, extractedRightRows);
- break;
- }
-
- Schema schema = CalciteUtils.toSchema(getRowType());
- return joinedRows
- .apply(
- "JoinParts2WholeRow",
- MapElements.via(new BeamJoinTransforms.JoinParts2WholeRow(schema)))
- .setRowSchema(schema);
- }
-
- public PCollection<Row> sideInputJoin(
- PCollection<KV<Row, Row>> extractedLeftRows,
- PCollection<KV<Row, Row>> extractedRightRows,
- Schema leftSchema,
- Schema rightSchema) {
- // we always make the Unbounded table on the left to do the sideInput join
- // (will convert the result accordingly before return)
- boolean swapped = (extractedLeftRows.isBounded() == PCollection.IsBounded.BOUNDED);
- JoinRelType realJoinType =
- (swapped && joinType != JoinRelType.INNER) ? JoinRelType.LEFT : joinType;
-
- PCollection<KV<Row, Row>> realLeftRows = swapped ? extractedRightRows : extractedLeftRows;
- PCollection<KV<Row, Row>> realRightRows = swapped ? extractedLeftRows : extractedRightRows;
-
- Row realRightNullRow;
- if (swapped) {
- Schema leftNullSchema = buildNullSchema(leftSchema);
-
- realRightRows = setValueCoder(realRightRows, SchemaCoder.of(leftNullSchema));
- realRightNullRow = Row.nullRow(leftNullSchema);
- } else {
- Schema rightNullSchema = buildNullSchema(rightSchema);
-
- realRightRows = setValueCoder(realRightRows, SchemaCoder.of(rightNullSchema));
- realRightNullRow = Row.nullRow(rightNullSchema);
- }
-
- // swapped still need to pass down because, we need to swap the result back.
- return sideInputJoinHelper(
- realJoinType, realLeftRows, realRightRows, realRightNullRow, swapped);
- }
-
- private PCollection<Row> sideInputJoinHelper(
- JoinRelType joinType,
- PCollection<KV<Row, Row>> leftRows,
- PCollection<KV<Row, Row>> rightRows,
- Row rightNullRow,
- boolean swapped) {
- final PCollectionView<Map<Row, Iterable<Row>>> rowsView = rightRows.apply(View.asMultimap());
-
- Schema schema = CalciteUtils.toSchema(getRowType());
- return leftRows
- .apply(
- ParDo.of(
- new BeamJoinTransforms.SideInputJoinDoFn(
- joinType, rightNullRow, rowsView, swapped, schema))
- .withSideInputs(rowsView))
- .setRowSchema(schema);
- }
-
- private Schema buildNullSchema(Schema schema) {
+ protected Schema buildNullSchema(Schema schema) {
Schema.Builder builder = Schema.builder();
builder.addFields(
@@ -489,7 +244,7 @@
return builder.build();
}
- private static <K, V> PCollection<KV<K, V>> setValueCoder(
+ protected static <K, V> PCollection<KV<K, V>> setValueCoder(
PCollection<KV<K, V>> kvs, Coder<V> valueCoder) {
// safe case because PCollection of KV always has KvCoder
KvCoder<K, V> coder = (KvCoder<K, V>) kvs.getCoder();
@@ -540,7 +295,7 @@
return curField;
}
- private List<Pair<RexNode, RexNode>> extractJoinRexNodes() {
+ static List<Pair<RexNode, RexNode>> extractJoinRexNodes(RexNode condition) {
// it's a CROSS JOIN because: condition == true
if (condition instanceof RexLiteral && (Boolean) ((RexLiteral) condition).getValue()) {
throw new UnsupportedOperationException("CROSS JOIN is not supported!");
@@ -564,7 +319,7 @@
return pairs;
}
- private Pair<RexNode, RexNode> extractJoinPairOfRexNodes(RexCall rexCall) {
+ private static Pair<RexNode, RexNode> extractJoinPairOfRexNodes(RexCall rexCall) {
if (!rexCall.getOperator().getName().equals("=")) {
throw new UnsupportedOperationException("Non equi-join is not supported");
}
@@ -584,14 +339,14 @@
}
// Only support {RexInputRef | RexFieldAccess} = {RexInputRef | RexFieldAccess}
- private boolean isIllegalJoinConjunctionClause(RexCall rexCall) {
+ private static boolean isIllegalJoinConjunctionClause(RexCall rexCall) {
return (!(rexCall.getOperands().get(0) instanceof RexInputRef)
&& !(rexCall.getOperands().get(0) instanceof RexFieldAccess))
|| (!(rexCall.getOperands().get(1) instanceof RexInputRef)
&& !(rexCall.getOperands().get(1) instanceof RexFieldAccess));
}
- private int getColumnIndex(RexNode rexNode) {
+ private static int getColumnIndex(RexNode rexNode) {
if (rexNode instanceof RexInputRef) {
return ((RexInputRef) rexNode).getIndex();
} else if (rexNode instanceof RexFieldAccess) {
@@ -601,15 +356,68 @@
throw new UnsupportedOperationException("Cannot get column index from " + rexNode.getType());
}
- /** check if {@code BeamRelNode} implements {@code BeamSeekableTable}. */
- private boolean seekable(BeamRelNode relNode) {
- if (relNode instanceof BeamIOSourceRel) {
- BeamIOSourceRel srcRel = (BeamIOSourceRel) relNode;
- BeamSqlTable sourceTable = srcRel.getBeamSqlTable();
- if (sourceTable instanceof BeamSqlSeekableTable) {
+ /**
+ * This method returns the Boundedness of a RelNode. It is used during planning and applying
+ * {@link org.apache.beam.sdk.extensions.sql.impl.rule.BeamCoGBKJoinRule} and {@link
+ * org.apache.beam.sdk.extensions.sql.impl.rule.BeamSideInputJoinRule}
+ *
+ * <p>The Volcano planner works in a top-down fashion. It starts by transforming the root and move
+ * towards the leafs of the plan. Due to this when transforming a logical join its inputs are
+ * still in the logical convention. So, Recursively visit the inputs of the RelNode till
+ * BeamIOSourceRel is encountered and propagate the boundedness upwards.
+ *
+ * <p>The Boundedness of each child of a RelNode is stored in a list. If any of the children are
+ * Unbounded, the RelNode is Unbounded. Else, the RelNode is Bounded.
+ *
+ * @param relNode the RelNode whose Boundedness has to be determined
+ * @return {@code PCollection.isBounded}
+ */
+ public static PCollection.IsBounded getBoundednessOfRelNode(RelNode relNode) {
+ if (relNode instanceof BeamRelNode) {
+ return (((BeamRelNode) relNode).isBounded());
+ }
+ List<PCollection.IsBounded> boundednessOfInputs = new ArrayList<>();
+ for (RelNode inputRel : relNode.getInputs()) {
+ if (inputRel instanceof RelSubset) {
+ // Consider the RelNode with best cost in the RelSubset. If best cost RelNode cannot be
+ // determined, consider the first RelNode in the RelSubset
+ RelNode rel = ((RelSubset) inputRel).getBest();
+ if (rel == null) {
+ rel = ((RelSubset) inputRel).getRelList().get(0);
+ }
+ boundednessOfInputs.add(getBoundednessOfRelNode(rel));
+ } else {
+ boundednessOfInputs.add(getBoundednessOfRelNode(inputRel));
+ }
+ }
+ // If one of the input is Unbounded, the result is Unbounded.
+ return (boundednessOfInputs.contains(PCollection.IsBounded.UNBOUNDED)
+ ? PCollection.IsBounded.UNBOUNDED
+ : PCollection.IsBounded.BOUNDED);
+ }
+
+ /**
+ * This method returns whether any of the children of the relNode are Seekable. It is used during
+ * planning and applying {@link org.apache.beam.sdk.extensions.sql.impl.rule.BeamCoGBKJoinRule}
+ * and {@link org.apache.beam.sdk.extensions.sql.impl.rule.BeamSideInputJoinRule} and {@link
+ * org.apache.beam.sdk.extensions.sql.impl.rule.BeamSideInputLookupJoinRule}
+ *
+ * @param relNode the relNode whose children can be Seekable
+ * @return A boolean
+ */
+ public static boolean containsSeekableInput(RelNode relNode) {
+ for (RelNode relInput : relNode.getInputs()) {
+ if (relInput instanceof RelSubset) {
+ relInput = ((RelSubset) relInput).getBest();
+ }
+ // input is Seekable
+ if (relInput != null
+ && relInput instanceof BeamRelNode
+ && (BeamJoinRel.seekable((BeamRelNode) relInput))) {
return true;
}
}
+ // None of the inputs are Seekable
return false;
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java
index f134686..4133f0a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamLogicalConvention.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.plan.ConventionTraitDef;
-import org.apache.calcite.plan.RelOptPlanner;
-import org.apache.calcite.plan.RelTrait;
-import org.apache.calcite.plan.RelTraitDef;
-import org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTrait;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
/** Convertion for Beam SQL. */
public enum BeamLogicalConvention implements Convention {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java
index 482b8be..5e9e075 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRel.java
@@ -18,15 +18,19 @@
package org.apache.beam.sdk.extensions.sql.impl.rel;
import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Minus;
-import org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Minus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* {@code BeamRelNode} to replace a {@code Minus} node.
@@ -41,6 +45,17 @@
}
@Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats inputsEstimatesSummation =
+ inputs.stream()
+ .map(input -> BeamSqlRelUtils.getNodeStats(input, mq))
+ .reduce(NodeStats.create(0, 0, 0), NodeStats::plus);
+
+ return BeamCostModel.FACTORY.makeCost(
+ inputsEstimatesSummation.getRowCount(), inputsEstimatesSummation.getRate());
+ }
+
+ @Override
public SetOp copy(RelTraitSet traitSet, List<RelNode> inputs, boolean all) {
return new BeamMinusRel(getCluster(), traitSet, inputs, all);
}
@@ -49,4 +64,15 @@
public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
return new BeamSetOperatorRelBase(this, BeamSetOperatorRelBase.OpType.MINUS, all);
}
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ NodeStats firstInputEstimates = BeamSqlRelUtils.getNodeStats(inputs.get(0), mq);
+ // The first input minus half of the others. (We are assuming half of them have intersection)
+ for (int i = 1; i < inputs.size(); i++) {
+ NodeStats inputEstimate = BeamSqlRelUtils.getNodeStats(inputs.get(i), mq);
+ firstInputEstimates = firstInputEstimates.minus(inputEstimate.multiply(0.5));
+ }
+ return firstInputEstimates;
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java
index 29e53d1..1b549b4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamRelNode.java
@@ -19,11 +19,15 @@
import java.util.List;
import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** A {@link RelNode} that can also give a {@link PTransform} that implements the expression. */
public interface BeamRelNode extends RelNode {
@@ -61,4 +65,28 @@
}
return options;
}
+
+ /**
+ * This method is called by {@code
+ * org.apache.beam.sdk.extensions.sql.impl.planner.RelMdNodeStats}. This is currently only used in
+ * SQLTransform Path (and not JDBC path). When a RelNode wants to calculate its BeamCost or
+ * estimate its NodeStats, it may need NodeStat of its inputs. However, it should not call this
+ * directly (because maybe its inputs are not physical yet). It should call {@link
+ * org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils#getNodeStats(
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode,
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery)}
+ * instead.
+ */
+ NodeStats estimateNodeStats(RelMetadataQuery mq);
+
+ /**
+ * This method is called by {@code
+ * org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.NonCumulativeCostImpl}. This is
+ * currently only used in SQLTransform Path (and not JDBC path). This is needed when Calcite Query
+ * Planner wants to get the cost of a plan. Instead of calling this directly for a node, if we
+ * needed that it should be obtained by calling mq.getNonCumulativeCost. This way RelMetadataQuery
+ * will call this method instead of ComputeSelfCost if the handler is set correctly (see {@code
+ * org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner#convertToBeamRel(String)})
+ */
+ BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq);
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
index 18b803d..1dfeb0e 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import org.apache.beam.sdk.extensions.sql.impl.transform.BeamSetOperatorsTransforms;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java
new file mode 100644
index 0000000..06011a9
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.beam.sdk.extensions.sql.impl.transform.BeamJoinTransforms;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaCoder;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * A {@code BeamJoinRel} which does sideinput Join
+ *
+ * <p>This Join Covers the case:
+ *
+ * <ul>
+ * <li>BoundedTable JOIN UnboundedTable
+ * </ul>
+ *
+ * <p>{@code sideInput} is utilized to implement the join, so there are some constraints:
+ *
+ * <ul>
+ * <li>{@code FULL OUTER JOIN} is not supported.
+ * <li>If it's a {@code LEFT OUTER JOIN}, the unbounded table should on the left side.
+ * <li>If it's a {@code RIGHT OUTER JOIN}, the unbounded table should on the right side.
+ * </ul>
+ *
+ * <p>General constraints:
+ *
+ * <ul>
+ * <li>Only equi-join is supported.
+ * <li>CROSS JOIN is not supported.
+ * </ul>
+ */
+public class BeamSideInputJoinRel extends BeamJoinRel {
+
+ public BeamSideInputJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ Set<CorrelationId> variablesSet,
+ JoinRelType joinType) {
+ super(cluster, traitSet, left, right, condition, variablesSet, joinType);
+ }
+
+ @Override
+ public Join copy(
+ RelTraitSet traitSet,
+ RexNode conditionExpr,
+ RelNode left,
+ RelNode right,
+ JoinRelType joinType,
+ boolean semiJoinDone) {
+ return new BeamSideInputJoinRel(
+ getCluster(), traitSet, left, right, conditionExpr, variablesSet, joinType);
+ }
+
+ @Override
+ public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
+ // if one of the sides is Bounded & the other is Unbounded
+ // then do a sideInput join.
+ // When doing a sideInput join, the windowFn does not need to match.
+ // Only support INNER JOIN & LEFT OUTER JOIN where left side of the join must be
+ // the unbounded & RIGHT OUTER JOIN where right side of the join must be the unbounded
+ if (joinType == JoinRelType.FULL) {
+ throw new UnsupportedOperationException(
+ "FULL OUTER JOIN is not supported when join "
+ + "a bounded table with an unbounded table.");
+ }
+
+ BeamRelNode leftRelNode = BeamSqlRelUtils.getBeamRelInput(left);
+ BeamRelNode rightRelNode = BeamSqlRelUtils.getBeamRelInput(right);
+
+ if ((joinType == JoinRelType.LEFT && leftRelNode.isBounded() == PCollection.IsBounded.BOUNDED)
+ || (joinType == JoinRelType.RIGHT
+ && rightRelNode.isBounded() == PCollection.IsBounded.BOUNDED)) {
+ throw new UnsupportedOperationException(
+ String.format("%s side of an OUTER JOIN must be Unbounded table.", joinType.name()));
+ }
+ return new SideInputJoin();
+ }
+
+ private class SideInputJoin extends PTransform<PCollectionList<Row>, PCollection<Row>> {
+
+ @Override
+ public PCollection<Row> expand(PCollectionList<Row> pinput) {
+ Schema leftSchema = CalciteUtils.toSchema(left.getRowType());
+ Schema rightSchema = CalciteUtils.toSchema(right.getRowType());
+
+ PCollectionList<KV<Row, Row>> keyedInputs = pinput.apply(new ExtractJoinKeys());
+
+ PCollection<KV<Row, Row>> extractedLeftRows = keyedInputs.get(0);
+ PCollection<KV<Row, Row>> extractedRightRows = keyedInputs.get(1);
+
+ return sideInputJoin(extractedLeftRows, extractedRightRows, leftSchema, rightSchema);
+ }
+ }
+
+ public PCollection<Row> sideInputJoin(
+ PCollection<KV<Row, Row>> extractedLeftRows,
+ PCollection<KV<Row, Row>> extractedRightRows,
+ Schema leftSchema,
+ Schema rightSchema) {
+ // we always make the Unbounded table on the left to do the sideInput join
+ // (will convert the result accordingly before return)
+ boolean swapped = (extractedLeftRows.isBounded() == PCollection.IsBounded.BOUNDED);
+ JoinRelType realJoinType =
+ (swapped && joinType != JoinRelType.INNER) ? JoinRelType.LEFT : joinType;
+
+ PCollection<KV<Row, Row>> realLeftRows = swapped ? extractedRightRows : extractedLeftRows;
+ PCollection<KV<Row, Row>> realRightRows = swapped ? extractedLeftRows : extractedRightRows;
+
+ Row realRightNullRow;
+ if (swapped) {
+ Schema leftNullSchema = buildNullSchema(leftSchema);
+
+ realRightRows = BeamJoinRel.setValueCoder(realRightRows, SchemaCoder.of(leftNullSchema));
+ realRightNullRow = Row.nullRow(leftNullSchema);
+ } else {
+ Schema rightNullSchema = buildNullSchema(rightSchema);
+
+ realRightRows = BeamJoinRel.setValueCoder(realRightRows, SchemaCoder.of(rightNullSchema));
+ realRightNullRow = Row.nullRow(rightNullSchema);
+ }
+
+ // swapped still need to pass down because, we need to swap the result back.
+ return sideInputJoinHelper(
+ realJoinType, realLeftRows, realRightRows, realRightNullRow, swapped);
+ }
+
+ private PCollection<Row> sideInputJoinHelper(
+ JoinRelType joinType,
+ PCollection<KV<Row, Row>> leftRows,
+ PCollection<KV<Row, Row>> rightRows,
+ Row rightNullRow,
+ boolean swapped) {
+ final PCollectionView<Map<Row, Iterable<Row>>> rowsView = rightRows.apply(View.asMultimap());
+
+ Schema schema = CalciteUtils.toSchema(getRowType());
+ return leftRows
+ .apply(
+ ParDo.of(
+ new BeamJoinTransforms.SideInputJoinDoFn(
+ joinType, rightNullRow, rowsView, swapped, schema))
+ .withSideInputs(rowsView))
+ .setRowSchema(schema);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java
new file mode 100644
index 0000000..b4dbd56
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.util.Set;
+import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
+import org.apache.beam.sdk.extensions.sql.impl.transform.BeamJoinTransforms;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * A {@code BeamJoinRel} which does Lookup Join
+ *
+ * <p>This Join Covers the case:
+ *
+ * <ul>
+ * <li>SeekableTable JOIN non SeekableTable
+ * </ul>
+ *
+ * <p>As Join is implemented as lookup, there are some constraints:
+ *
+ * <ul>
+ * <li>{@code FULL OUTER JOIN} is not supported.
+ * <li>If it's a {@code LEFT OUTER JOIN}, the non Seekable table should on the left side.
+ * <li>If it's a {@code RIGHT OUTER JOIN}, the non Seekable table should on the right side.
+ * </ul>
+ *
+ * <p>General constraints:
+ *
+ * <ul>
+ * <li>Only equi-join is supported.
+ * <li>CROSS JOIN is not supported.
+ * </ul>
+ */
+public class BeamSideInputLookupJoinRel extends BeamJoinRel {
+
+ public BeamSideInputLookupJoinRel(
+ RelOptCluster cluster,
+ RelTraitSet traitSet,
+ RelNode left,
+ RelNode right,
+ RexNode condition,
+ Set<CorrelationId> variablesSet,
+ JoinRelType joinType) {
+ super(cluster, traitSet, left, right, condition, variablesSet, joinType);
+ }
+
+ @Override
+ public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
+ // if one of the sides is Seekable & the other is non Seekable
+ // then do a sideInputLookup join.
+ // When doing a sideInputLookup join, the windowFn does not need to match.
+ // Only support INNER JOIN & LEFT OUTER JOIN where left side of the join must be
+ // non Seekable & RIGHT OUTER JOIN where right side of the join must be non Seekable
+ if (joinType == JoinRelType.FULL) {
+ throw new UnsupportedOperationException(
+ "FULL OUTER JOIN is not supported when join "
+ + "a Seekable table with a non Seekable table.");
+ }
+
+ if ((joinType == JoinRelType.LEFT && seekableInputIndex().get() == 0)
+ || (joinType == JoinRelType.RIGHT && seekableInputIndex().get() == 1)) {
+ throw new UnsupportedOperationException(
+ String.format("%s side of an OUTER JOIN must be a non Seekable table.", joinType.name()));
+ }
+ return new SideInputLookupJoin();
+ }
+
+ private class SideInputLookupJoin extends PTransform<PCollectionList<Row>, PCollection<Row>> {
+
+ @Override
+ public PCollection<Row> expand(PCollectionList<Row> pinput) {
+ Schema schema = CalciteUtils.toSchema(getRowType());
+
+ BeamRelNode seekableRel =
+ BeamSqlRelUtils.getBeamRelInput(getInput(seekableInputIndex().get()));
+ BeamRelNode nonSeekableRel =
+ BeamSqlRelUtils.getBeamRelInput(getInput(nonSeekableInputIndex().get()));
+
+ // Offset field references according to which table is on the left
+ int factColOffset =
+ nonSeekableInputIndex().get() == 0
+ ? 0
+ : CalciteUtils.toSchema(seekableRel.getRowType()).getFieldCount();
+ int lkpColOffset =
+ seekableInputIndex().get() == 0
+ ? 0
+ : CalciteUtils.toSchema(nonSeekableRel.getRowType()).getFieldCount();
+
+ // HACK: if the input is an immediate instance of a seekable IO, we can do lookups
+ // so we ignore the PCollection
+ BeamIOSourceRel seekableInput = (BeamIOSourceRel) seekableRel;
+ BeamSqlSeekableTable seekableTable = (BeamSqlSeekableTable) seekableInput.getBeamSqlTable();
+
+ // getPCollectionInputs() ensures that there is only one and it is the non-seekable input
+ PCollection<Row> nonSeekableInput = pinput.get(0);
+
+ return nonSeekableInput
+ .apply(
+ "join_as_lookup",
+ new BeamJoinTransforms.JoinAsLookup(
+ condition,
+ seekableTable,
+ CalciteUtils.toSchema(seekableInput.getRowType()),
+ schema,
+ factColOffset,
+ lkpColOffset))
+ .setRowSchema(schema);
+ }
+ }
+
+ @Override
+ public Join copy(
+ RelTraitSet traitSet,
+ RexNode conditionExpr,
+ RelNode left,
+ RelNode right,
+ JoinRelType joinType,
+ boolean semiJoinDone) {
+ return new BeamSideInputLookupJoinRel(
+ getCluster(), traitSet, left, right, conditionExpr, variablesSet, joinType);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java
index ce2b2d3..afe18b0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.math.BigDecimal;
@@ -30,6 +30,8 @@
import org.apache.beam.sdk.coders.ListCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.state.StateSpec;
@@ -39,7 +41,6 @@
import org.apache.beam.sdk.transforms.Flatten;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.Top;
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
@@ -49,17 +50,19 @@
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelCollation;
-import org.apache.calcite.rel.RelCollationImpl;
-import org.apache.calcite.rel.RelFieldCollation;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Sort;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Sort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
/**
* {@code BeamRelNode} to replace a {@code Sort} node.
@@ -134,6 +137,22 @@
}
}
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ // Sorting does not change rate or row count of the input.
+ return BeamSqlRelUtils.getNodeStats(this.input, mq);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats inputEstimates = BeamSqlRelUtils.getNodeStats(this.input, mq);
+
+ final double rowSize = getRowType().getFieldCount();
+ final double cpu = inputEstimates.getRowCount() * inputEstimates.getRowCount() * rowSize;
+ final double cpuRate = inputEstimates.getRate() * inputEstimates.getWindow() * rowSize;
+ return BeamCostModel.FACTORY.makeCost(cpu, cpuRate);
+ }
+
public boolean isLimitOnly() {
return fieldIndices.isEmpty();
}
@@ -204,10 +223,7 @@
return rawStream
.apply("flatten", Flatten.iterables())
- .setSchema(
- CalciteUtils.toSchema(getRowType()),
- SerializableFunctions.identity(),
- SerializableFunctions.identity());
+ .setRowSchema(CalciteUtils.toSchema(getRowType()));
}
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java
index 9f0fd55..9bf45c7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSqlRelUtils.java
@@ -22,12 +22,15 @@
import java.util.Map;
import java.util.stream.Collectors;
import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStatsMetadata;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.volcano.RelSubset;
-import org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** Utilities for {@code BeamRelNode}. */
public class BeamSqlRelUtils {
@@ -76,4 +79,20 @@
}
return (BeamRelNode) input;
}
+
+ public static RelNode getInput(RelNode input) {
+ RelNode result = input;
+ if (input instanceof RelSubset) {
+ // go with known best input
+ result = ((RelSubset) input).getBest();
+ result = result == null ? ((RelSubset) input).getOriginal() : result;
+ }
+
+ return result;
+ }
+
+ public static NodeStats getNodeStats(RelNode input, RelMetadataQuery mq) {
+ input = getInput(input);
+ return input.metadata(NodeStatsMetadata.class, mq).getNodeStats();
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java
index a0c66ac..2b2511d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRel.java
@@ -17,8 +17,10 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.DoFn;
@@ -27,10 +29,12 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/** {@link BeamRelNode} to implement an uncorrelated {@link Uncollect}, aka UNNEST. */
public class BeamUncollectRel extends Uncollect implements BeamRelNode {
@@ -72,6 +76,20 @@
}
}
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ // We estimate the average length of each array by a constant.
+ // We might be able to get an estimate of the length by making a MetadataHandler for this
+ // purpose, and get the estimate by reading the first couple of the rows in the source.
+ return BeamSqlRelUtils.getNodeStats(this.input, mq).multiply(2);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats estimates = BeamSqlRelUtils.getNodeStats(this, mq);
+ return BeamCostModel.FACTORY.makeCost(estimates.getRowCount(), estimates.getRate());
+ }
+
private static class UncollectDoFn extends DoFn<Row, Row> {
private final Schema schema;
@@ -83,7 +101,12 @@
@ProcessElement
public void process(@Element Row inputRow, OutputReceiver<Row> output) {
for (Object element : inputRow.getArray(0)) {
- output.output(Row.withSchema(schema).addValue(element).build());
+ if (element instanceof Row) {
+ Row nestedRow = (Row) element;
+ output.output(Row.withSchema(schema).addValues(nestedRow.getValues()).build());
+ } else {
+ output.output(Row.withSchema(schema).addValue(element).build());
+ }
}
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java
index 175b139..5fc3d07 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRel.java
@@ -18,16 +18,20 @@
package org.apache.beam.sdk.extensions.sql.impl.rel;
import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.SetOp;
-import org.apache.calcite.rel.core.Union;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.SetOp;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Union;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
/**
* {@link BeamRelNode} to replace a {@link Union}.
@@ -76,4 +80,28 @@
public PTransform<PCollectionList<Row>, PCollection<Row>> buildPTransform() {
return new BeamSetOperatorRelBase(this, BeamSetOperatorRelBase.OpType.UNION, all);
}
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ // The summation of the input stats
+ NodeStats summationOfEstimates =
+ inputs.stream()
+ .map(input -> BeamSqlRelUtils.getNodeStats(input, mq))
+ .reduce(NodeStats.create(0, 0, 0), NodeStats::plus);
+ // If all is set then we propagate duplicated values. Otherwise we assume a constant factor of
+ // them are duplicate.
+ summationOfEstimates = all ? summationOfEstimates : summationOfEstimates.multiply(0.5);
+ return summationOfEstimates;
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats summationOfEstimates =
+ inputs.stream()
+ .map(input -> BeamSqlRelUtils.getNodeStats(input, mq))
+ .reduce(NodeStats.create(0, 0, 0), NodeStats::plus);
+
+ return BeamCostModel.FACTORY.makeCost(
+ summationOfEstimates.getRowCount(), summationOfEstimates.getRate());
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
index d6c0de5..1263b3d 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel.java
@@ -19,6 +19,8 @@
import java.util.List;
import javax.annotation.Nullable;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.DoFn;
@@ -27,16 +29,18 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.RelWriter;
-import org.apache.calcite.rel.core.Correlate;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.sql.validate.SqlValidatorUtil;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelWriter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Correlate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlValidatorUtil;
/**
* {@link BeamRelNode} to implement UNNEST, supporting specifically only {@link Correlate} with
@@ -75,6 +79,20 @@
}
@Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ // We estimate the average length of each array by a constant.
+ // We might be able to get an estimate of the length by making a MetadataHandler for this
+ // purpose, and get the estimate by reading the first couple of the rows in the source.
+ return BeamSqlRelUtils.getNodeStats(this.input, mq).multiply(2);
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats estimates = BeamSqlRelUtils.getNodeStats(this, mq);
+ return BeamCostModel.FACTORY.makeCost(estimates.getRowCount(), estimates.getRate());
+ }
+
+ @Override
public RelWriter explainTerms(RelWriter pw) {
return super.explainTerms(pw).item("unnestIndex", Integer.toString(unnestIndex));
}
@@ -116,13 +134,24 @@
if (rawValues == null) {
return;
}
+ Schema.TypeName typeName =
+ outputSchema.getField(unnestIndex).getType().getCollectionElementType().getTypeName();
for (Object uncollectedValue : rawValues) {
- out.output(
- Row.withSchema(outputSchema)
- .addValues(row.getValues())
- .addValue(uncollectedValue)
- .build());
+ if (typeName.equals(Schema.TypeName.ROW)) {
+ Row nestedRow = (Row) uncollectedValue;
+ out.output(
+ Row.withSchema(outputSchema)
+ .addValues(row.getValues())
+ .addValues(nestedRow.getValues())
+ .build());
+ } else {
+ out.output(
+ Row.withSchema(outputSchema)
+ .addValues(row.getValues())
+ .addValue(uncollectedValue)
+ .build());
+ }
}
}
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
index 9799ed8..9fa5037 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRel.java
@@ -20,12 +20,13 @@
import static java.util.stream.Collectors.toList;
import static org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils.autoCastField;
import static org.apache.beam.sdk.values.Row.toRow;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
-import com.google.common.collect.ImmutableList;
import java.util.List;
import java.util.Map;
import java.util.stream.IntStream;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Create;
@@ -33,12 +34,15 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.core.Values;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Values;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
/**
* {@code BeamRelNode} to replace a {@code Values} node.
@@ -91,4 +95,15 @@
.mapToObj(i -> autoCastField(schema.getField(i), tuple.get(i).getValue()))
.collect(toRow(schema));
}
+
+ @Override
+ public NodeStats estimateNodeStats(RelMetadataQuery mq) {
+ return NodeStats.create(tuples.size(), 0, tuples.size());
+ }
+
+ @Override
+ public BeamCostModel beamComputeSelfCost(RelOptPlanner planner, RelMetadataQuery mq) {
+ NodeStats estimates = BeamSqlRelUtils.getNodeStats(this, mq);
+ return BeamCostModel.FACTORY.makeCost(estimates.getRowCount(), estimates.getRate());
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
index d09d802..6c74569 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/package-info.java
@@ -16,7 +16,10 @@
* limitations under the License.
*/
-/** BeamSQL specified nodes, to replace {@link org.apache.calcite.rel.RelNode}. */
+/**
+ * BeamSQL specified nodes, to replace {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode}.
+ */
@DefaultAnnotation(NonNull.class)
package org.apache.beam.sdk.extensions.sql.impl.rel;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
index bf52652..cbdcf6a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
@@ -26,18 +26,18 @@
import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Aggregate;
-import org.apache.calcite.rel.core.Project;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.tools.RelBuilderFactory;
-import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Project;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
import org.joda.time.Duration;
/** Rule to detect the window/trigger settings. */
@@ -57,6 +57,10 @@
final Aggregate aggregate = call.rel(0);
final Project project = call.rel(1);
RelNode x = updateWindow(call, aggregate, project);
+ if (x == null) {
+ // Non-windowed case should be handled by the BeamBasicAggregationRule
+ return;
+ }
call.transformTo(x);
}
@@ -82,6 +86,10 @@
}
}
+ if (windowFn == null) {
+ return null;
+ }
+
final Project newProject =
project.copy(project.getTraitSet(), project.getInput(), projects, project.getRowType());
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
index eb93911..9e1def7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamBasicAggregationRule.java
@@ -17,15 +17,24 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.rule;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamAggregationRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.core.Aggregate;
-import org.apache.calcite.rel.core.RelFactories;
-import org.apache.calcite.rel.core.TableScan;
-import org.apache.calcite.tools.RelBuilderFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Aggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Filter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Project;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
/**
* Aggregation rule that doesn't include projection.
@@ -40,15 +49,22 @@
public BeamBasicAggregationRule(
Class<? extends Aggregate> aggregateClass, RelBuilderFactory relBuilderFactory) {
- super(operand(aggregateClass, operand(TableScan.class, any())), relBuilderFactory, null);
+ super(operand(aggregateClass, operand(RelNode.class, any())), relBuilderFactory, null);
}
@Override
public void onMatch(RelOptRuleCall call) {
Aggregate aggregate = call.rel(0);
- TableScan tableScan = call.rel(1);
+ RelNode relNode = call.rel(1);
- RelNode newTableScan = tableScan.copy(tableScan.getTraitSet(), tableScan.getInputs());
+ if (relNode instanceof Project || relNode instanceof Calc || relNode instanceof Filter) {
+ if (isWindowed(relNode) || hasWindowedParents(relNode)) {
+ // This case is expected to get handled by the 'BeamAggregationRule'
+ return;
+ }
+ }
+
+ RelNode newTableScan = relNode.copy(relNode.getTraitSet(), relNode.getInputs());
call.transformTo(
new BeamAggregationRel(
@@ -63,4 +79,54 @@
null,
-1));
}
+
+ private static boolean isWindowed(RelNode node) {
+ List<RexNode> projects = null;
+
+ if (node instanceof Project) {
+ projects = new ArrayList<>(((Project) node).getProjects());
+ } else if (node instanceof Calc) {
+ projects =
+ ((Calc) node)
+ .getProgram().getProjectList().stream()
+ .map(
+ rexLocalRef ->
+ ((Calc) node).getProgram().getExprList().get(rexLocalRef.getIndex()))
+ .collect(Collectors.toList());
+ }
+
+ if (projects != null) {
+ for (RexNode projNode : projects) {
+ if (!(projNode instanceof RexCall)) {
+ continue;
+ }
+
+ SqlKind sqlKind = ((RexCall) projNode).op.kind;
+ if (sqlKind == SqlKind.SESSION || sqlKind == SqlKind.HOP || sqlKind == SqlKind.TUMBLE) {
+ return true;
+ }
+ }
+ }
+
+ return false;
+ }
+
+ private static boolean hasWindowedParents(RelNode node) {
+ List<RelNode> parents = new ArrayList<>();
+
+ for (RelNode inputNode : node.getInputs()) {
+ if (inputNode instanceof RelSubset) {
+ parents.addAll(((RelSubset) inputNode).getParentRels());
+ parents.addAll(((RelSubset) inputNode).getRelList());
+ }
+ }
+
+ for (RelNode parent : parents) {
+ if (isWindowed(parent)) {
+ return true;
+ }
+ }
+
+ return false;
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java
index b5aee81..824e5fc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCalcRule.java
@@ -19,13 +19,13 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Calc;
-import org.apache.calcite.rel.logical.LogicalCalc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalCalc;
/** A {@code ConverterRule} to replace {@link Calc} with {@link BeamCalcRel}. */
public class BeamCalcRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java
new file mode 100644
index 0000000..516bc09
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+
+/**
+ * Rule to convert {@code LogicalJoin} node to {@code BeamCoGBKJoinRel} node.
+ *
+ * <p>This rule is matched when both the inputs to {@code LogicalJoin} node have the same
+ * Boundedness i.e. either when both the inputs are {@code PCollection.IsBounded.BOUNDED} or when
+ * both the inputs are {@code PCollection.IsBounded.UNBOUNDED}
+ *
+ * <p>As {@code BeamSideInputLookupJoinRel} also matches this condition when both the inputs are
+ * {@code PCollection.IsBounded.BOUNDED}, to avoid conflicts, this rule is not matched when any of
+ * the inputs to {@code LogicalJoin} node are Seekable.
+ */
+public class BeamCoGBKJoinRule extends RelOptRule {
+ public static final BeamCoGBKJoinRule INSTANCE = new BeamCoGBKJoinRule();
+
+ private BeamCoGBKJoinRule() {
+ super(
+ operand(LogicalJoin.class, operand(RelNode.class, any()), operand(RelNode.class, any())),
+ RelFactories.LOGICAL_BUILDER,
+ "BeamCoGBKJoinRule");
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+ // The Rule does not match when any of the inputs are Seekable
+ if (BeamJoinRel.containsSeekableInput(call.rel(0))) {
+ return false;
+ }
+ PCollection.IsBounded boundednessOfLeftRel = BeamJoinRel.getBoundednessOfRelNode(call.rel(1));
+ PCollection.IsBounded boundednessOfRightRel = BeamJoinRel.getBoundednessOfRelNode(call.rel(2));
+ return (boundednessOfLeftRel == boundednessOfRightRel);
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+ Join join = (Join) call.rel(0);
+
+ BeamCoGBKJoinRel rel =
+ new BeamCoGBKJoinRel(
+ join.getCluster(),
+ join.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
+ convert(
+ join.getLeft(),
+ join.getLeft().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+ convert(
+ join.getRight(),
+ join.getRight().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+ join.getCondition(),
+ join.getVariablesSet(),
+ join.getJoinType());
+ call.transformTo(rel);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java
index ec64b44..773fef1 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamEnumerableConverterRule.java
@@ -20,10 +20,10 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import org.apache.calcite.adapter.enumerable.EnumerableConvention;
-import org.apache.calcite.plan.RelTraitSet;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
/** A {@code ConverterRule} to Convert {@link BeamRelNode} to {@link EnumerableConvention}. */
public class BeamEnumerableConverterRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java
index 9c6bfa6..d67e106 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOSinkRule.java
@@ -20,9 +20,9 @@
import java.util.Arrays;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIOSinkRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.TableModify;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableModify;
/** A {@code ConverterRule} to replace {@link TableModify} with {@link BeamIOSinkRel}. */
public class BeamIOSinkRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java
index 2ffe983a..1a91e4c 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIntersectRule.java
@@ -20,11 +20,11 @@
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamIntersectRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Intersect;
-import org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Intersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalIntersect;
/** {@code ConverterRule} to replace {@code Intersect} with {@code BeamIntersectRel}. */
public class BeamIntersectRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java
new file mode 100644
index 0000000..3eb7ab5
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinAssociateRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
+
+/**
+ * This is very similar to {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinAssociateRule}. It only
+ * checks if the resulting condition is supported before transforming.
+ */
+public class BeamJoinAssociateRule extends JoinAssociateRule {
+
+ public static final org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinAssociateRule INSTANCE =
+ new org.apache.beam.sdk.extensions.sql.impl.rule.BeamJoinAssociateRule(
+ RelFactories.LOGICAL_BUILDER);
+
+ private BeamJoinAssociateRule(RelBuilderFactory relBuilderFactory) {
+ super(relBuilderFactory);
+ }
+
+ @Override
+ public void onMatch(final RelOptRuleCall call) {
+ super.onMatch(
+ new JoinRelOptRuleCall(
+ call,
+ rel -> {
+ Join topJoin = (Join) rel;
+ Join bottomJoin = (Join) ((Join) rel).getRight();
+ return BeamJoinRel.isJoinLegal(topJoin) && BeamJoinRel.isJoinLegal(bottomJoin);
+ }));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java
new file mode 100644
index 0000000..f2a10b9
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinPushThroughJoinRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilderFactory;
+
+/**
+ * This is exactly similar to {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinPushThroughJoinRule}. It
+ * only checks if the condition of the new bottom join is supported.
+ */
+public class BeamJoinPushThroughJoinRule extends JoinPushThroughJoinRule {
+ /** Instance of the rule that works on logical joins only, and pushes to the right. */
+ public static final RelOptRule RIGHT =
+ new BeamJoinPushThroughJoinRule(
+ "BeamJoinPushThroughJoinRule:right",
+ true,
+ LogicalJoin.class,
+ RelFactories.LOGICAL_BUILDER);
+
+ /** Instance of the rule that works on logical joins only, and pushes to the left. */
+ public static final RelOptRule LEFT =
+ new BeamJoinPushThroughJoinRule(
+ "BeamJoinPushThroughJoinRule:left",
+ false,
+ LogicalJoin.class,
+ RelFactories.LOGICAL_BUILDER);
+
+ /** Creates a JoinPushThroughJoinRule. */
+ private BeamJoinPushThroughJoinRule(
+ String description,
+ boolean right,
+ Class<? extends Join> clazz,
+ RelBuilderFactory relBuilderFactory) {
+ super(description, right, clazz, relBuilderFactory);
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+ super.onMatch(
+ new JoinRelOptRuleCall(
+ call,
+ rel -> {
+ Join topJoin = (Join) rel.getInput(0);
+ Join bottomJoin = (Join) ((Join) rel.getInput(0)).getLeft();
+ return BeamJoinRel.isJoinLegal(topJoin) && BeamJoinRel.isJoinLegal(bottomJoin);
+ }));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinRule.java
deleted file mode 100644
index c94dd67..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinRule.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.rule;
-
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
-import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Join;
-import org.apache.calcite.rel.logical.LogicalJoin;
-
-/** {@code ConverterRule} to replace {@code Join} with {@code BeamJoinRel}. */
-public class BeamJoinRule extends ConverterRule {
- public static final BeamJoinRule INSTANCE = new BeamJoinRule();
-
- private BeamJoinRule() {
- super(LogicalJoin.class, Convention.NONE, BeamLogicalConvention.INSTANCE, "BeamJoinRule");
- }
-
- @Override
- public RelNode convert(RelNode rel) {
- Join join = (Join) rel;
-
- return new BeamJoinRel(
- join.getCluster(),
- join.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
- convert(
- join.getLeft(), join.getLeft().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
- convert(
- join.getRight(), join.getRight().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
- join.getCondition(),
- join.getVariablesSet(),
- join.getJoinType());
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java
index 73ac601..29d4a97 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule.java
@@ -20,11 +20,11 @@
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamMinusRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Minus;
-import org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Minus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalMinus;
/** {@code ConverterRule} to replace {@code Minus} with {@code BeamMinusRel}. */
public class BeamMinusRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java
new file mode 100644
index 0000000..98227bb
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSideInputJoinRel;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.RelFactories;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+
+/**
+ * Rule to convert {@code LogicalJoin} node to {@code BeamSideInputJoinRel} node.
+ *
+ * <p>This rule is matched when one of the input to {@code LogicalJoin} node is {@code
+ * PCollection.IsBounded.BOUNDED} and the other node is {@code PCollection.IsBounded.UNBOUNDED}
+ *
+ * <p>As {@code BeamSideInputLookupJoinRel} also matches this condition, to avoid conflicts, this
+ * rule is not matched when any of the inputs to {@code LogicalJoin} node are Seekable.
+ */
+public class BeamSideInputJoinRule extends RelOptRule {
+ public static final BeamSideInputJoinRule INSTANCE = new BeamSideInputJoinRule();
+
+ private BeamSideInputJoinRule() {
+ super(
+ operand(LogicalJoin.class, operand(RelNode.class, any()), operand(RelNode.class, any())),
+ RelFactories.LOGICAL_BUILDER,
+ "BeamSideInputJoinRule");
+ }
+
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+ // The Rule does not match when any of the inputs are Seekable
+ if (BeamJoinRel.containsSeekableInput(call.rel(0))) {
+ return false;
+ }
+ PCollection.IsBounded boundednessOfLeftRel = BeamJoinRel.getBoundednessOfRelNode(call.rel(1));
+ PCollection.IsBounded boundednessOfRightRel = BeamJoinRel.getBoundednessOfRelNode(call.rel(2));
+ return (boundednessOfLeftRel == PCollection.IsBounded.BOUNDED
+ ? boundednessOfRightRel == PCollection.IsBounded.UNBOUNDED
+ : boundednessOfRightRel == PCollection.IsBounded.BOUNDED);
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+ Join join = (Join) call.rel(0);
+
+ BeamSideInputJoinRel rel =
+ new BeamSideInputJoinRel(
+ join.getCluster(),
+ join.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
+ convert(
+ join.getLeft(),
+ join.getLeft().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+ convert(
+ join.getRight(),
+ join.getRight().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+ join.getCondition(),
+ join.getVariablesSet(),
+ join.getJoinType());
+ call.transformTo(rel);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java
new file mode 100644
index 0000000..2c96bd95
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRel;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSideInputLookupJoinRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+
+/**
+ * Rule to convert {@code LogicalJoin} node to {@code BeamSideInputLookupJoinRel} node.
+ *
+ * <p>This rule is matched when any of the inputs to {@code LogicalJoin} node are Seekable
+ */
+public class BeamSideInputLookupJoinRule extends ConverterRule {
+ public static final BeamSideInputLookupJoinRule INSTANCE = new BeamSideInputLookupJoinRule();
+
+ public BeamSideInputLookupJoinRule() {
+ super(
+ LogicalJoin.class,
+ Convention.NONE,
+ BeamLogicalConvention.INSTANCE,
+ "BeamSideInputLookupJoinRule");
+ }
+
+ // The Rule is Matched when any of the inputs are Seekable
+ @Override
+ public boolean matches(RelOptRuleCall call) {
+ RelNode joinRel = call.rel(0);
+ boolean matches = BeamJoinRel.containsSeekableInput(joinRel);
+ return (matches);
+ }
+
+ @Override
+ public RelNode convert(RelNode rel) {
+ Join join = (Join) rel;
+
+ return (new BeamSideInputLookupJoinRel(
+ join.getCluster(),
+ join.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
+ convert(
+ join.getLeft(), join.getLeft().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+ convert(
+ join.getRight(), join.getRight().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+ join.getCondition(),
+ join.getVariablesSet(),
+ join.getJoinType()));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java
index 18c24f4..1647bf7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSortRule.java
@@ -19,11 +19,11 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSortRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Sort;
-import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Sort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalSort;
/** {@code ConverterRule} to replace {@code Sort} with {@code BeamSortRel}. */
public class BeamSortRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java
index 6ce75fc..393882b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule.java
@@ -19,10 +19,10 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamUncollectRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
/** A {@code ConverterRule} to replace {@link Uncollect} with {@link BeamUncollectRule}. */
public class BeamUncollectRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java
index 1d4a637..7b84e25 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnionRule.java
@@ -19,14 +19,15 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamUnionRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Union;
-import org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Union;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalUnion;
/**
- * A {@code ConverterRule} to replace {@link org.apache.calcite.rel.core.Union} with {@link
+ * A {@code ConverterRule} to replace {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Union} with {@link
* BeamUnionRule}.
*/
public class BeamUnionRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java
index 03805a5..cc10225 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamUnnestRule.java
@@ -19,18 +19,18 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamUnnestRel;
-import org.apache.calcite.plan.RelOptRule;
-import org.apache.calcite.plan.RelOptRuleCall;
-import org.apache.calcite.plan.volcano.RelSubset;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.SingleRel;
-import org.apache.calcite.rel.core.Correlate;
-import org.apache.calcite.rel.core.Uncollect;
-import org.apache.calcite.rel.logical.LogicalCorrelate;
-import org.apache.calcite.rel.logical.LogicalProject;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.sql.SemiJoinType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.SingleRel;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Correlate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalCorrelate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/**
* A {@code ConverterRule} to replace {@link Correlate} {@link Uncollect} with {@link
@@ -61,7 +61,7 @@
// can only unnest a single column
return;
}
- if (correlate.getJoinType() != SemiJoinType.INNER) {
+ if (correlate.getJoinType() != JoinRelType.INNER) {
return;
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java
index 68c626e..6fbe1e0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamValuesRule.java
@@ -19,11 +19,11 @@
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamValuesRel;
-import org.apache.calcite.plan.Convention;
-import org.apache.calcite.rel.RelNode;
-import org.apache.calcite.rel.convert.ConverterRule;
-import org.apache.calcite.rel.core.Values;
-import org.apache.calcite.rel.logical.LogicalValues;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.ConverterRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Values;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
/** {@code ConverterRule} to replace {@code Values} with {@code BeamValuesRel}. */
public class BeamValuesRule extends ConverterRule {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java
new file mode 100644
index 0000000..27d8168
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelBuilder;
+
+/**
+ * This is a class to catch the built join and check if it is a legal join before passing it to the
+ * actual RelOptRuleCall.
+ */
+public class JoinRelOptRuleCall extends RelOptRuleCall {
+ private final RelOptRuleCall originalCall;
+ private final JoinChecker checker;
+
+ JoinRelOptRuleCall(RelOptRuleCall originalCall, JoinChecker checker) {
+ super(originalCall.getPlanner(), originalCall.getOperand0(), originalCall.rels, null, null);
+ this.originalCall = originalCall;
+ this.checker = checker;
+ }
+
+ // This is the only method that is different than originalCall. Everything else is delegated to
+ // originalCall.
+ @Override
+ public void transformTo(RelNode rel, Map<RelNode, RelNode> equiv) {
+ if (checker.check(rel)) {
+ originalCall.transformTo(rel, equiv);
+ }
+ }
+
+ /** This is a function gets the output relation and checks if it is a legal relational node. */
+ public interface JoinChecker {
+ boolean check(RelNode rel);
+ }
+
+ // Methods that are delegated to originalCall.
+
+ @Override
+ public RelOptRuleOperand getOperand0() {
+ return originalCall.getOperand0();
+ }
+
+ @Override
+ public RelOptRule getRule() {
+ return originalCall.getRule();
+ }
+
+ @Override
+ public List<RelNode> getRelList() {
+ return originalCall.getRelList();
+ }
+
+ @Override
+ @SuppressWarnings("TypeParameterUnusedInFormals")
+ public <T extends RelNode> T rel(int ordinal) {
+ return originalCall.rel(ordinal);
+ }
+
+ @Override
+ public List<RelNode> getChildRels(RelNode rel) {
+ return originalCall.getChildRels(rel);
+ }
+
+ @Override
+ public RelOptPlanner getPlanner() {
+ return originalCall.getPlanner();
+ }
+
+ @Override
+ public RelMetadataQuery getMetadataQuery() {
+ return originalCall.getMetadataQuery();
+ }
+
+ @Override
+ public List<RelNode> getParents() {
+ return originalCall.getParents();
+ }
+
+ @Override
+ public RelBuilder builder() {
+ return originalCall.builder();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java
index 6f82253..7c3d0b2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/package-info.java
@@ -17,8 +17,8 @@
*/
/**
- * {@link org.apache.calcite.plan.RelOptRule} to generate {@link
- * org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode}.
+ * {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule} to generate
+ * {@link org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode}.
*/
@DefaultAnnotation(NonNull.class)
package org.apache.beam.sdk.extensions.sql.impl.rule;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
deleted file mode 100644
index c8f911c..0000000
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.schema;
-
-import java.io.Serializable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
-import org.apache.beam.sdk.schemas.Schema;
-
-/** Each IO in Beam has one table schema, by extending {@link BaseBeamTable}. */
-public abstract class BaseBeamTable implements BeamSqlTable, Serializable {
- protected Schema schema;
-
- public BaseBeamTable(Schema schema) {
- this.schema = schema;
- }
-
- @Override
- public Schema getSchema() {
- return schema;
- }
-}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
index 661aec9..de3ff67 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
@@ -17,6 +17,9 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.schema;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.transforms.Convert;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -27,7 +30,7 @@
* {@code BeamPCollectionTable} converts a {@code PCollection<Row>} as a virtual table, then a
* downstream query can query directly.
*/
-public class BeamPCollectionTable<InputT> extends BaseBeamTable {
+public class BeamPCollectionTable<InputT> extends SchemaBaseBeamTable {
private transient PCollection<InputT> upstream;
public BeamPCollectionTable(PCollection<InputT> upstream) {
@@ -53,4 +56,9 @@
public POutput buildIOWriter(PCollection<Row> input) {
throw new IllegalArgumentException("cannot use [BeamPCollectionTable] as target");
}
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
index a76302c..c3761bb 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java
@@ -31,7 +31,7 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.util.NlsString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.NlsString;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVPrinter;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
index 7799245..ad99c28 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
@@ -40,7 +40,7 @@
import org.apache.beam.sdk.transforms.Min;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/** Built-in aggregations functions for COUNT/MAX/MIN/SUM/AVG/VAR_POP/VAR_SAMP. */
public class BeamBuiltinAggregations {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
index 3ea6d4c..47b9912 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamJoinTransforms.java
@@ -36,11 +36,11 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.core.JoinRelType;
-import org.apache.calcite.rex.RexCall;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexNode;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
/** Collections of {@code PTransform} and {@code DoFn} used to perform JOIN operation. */
public class BeamJoinTransforms {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
index 6556afe..4ea7dcf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamSetOperatorsTransforms.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterators;
/** Collections of {@code PTransform} and {@code DoFn} used to perform Set operations. */
public abstract class BeamSetOperatorsTransforms {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java
index 972438d..3905eb6 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/AggregationCombineFnAdapter.java
@@ -27,8 +27,8 @@
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.rel.core.AggregateCall;
-import org.apache.calcite.sql.validate.SqlUserDefinedAggFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.AggregateCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlUserDefinedAggFunction;
/** Wrapper {@link CombineFn}s for aggregation function calls. */
public class AggregationCombineFnAdapter<T> {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
index 36ea838..825aad8 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.transform.agg;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.math.BigDecimal;
import java.math.MathContext;
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
/**
* {@link Combine.CombineFn} for <em>Covariance</em> on {@link Number} types.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
index a0353e5..8114ee4 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/VarianceFn.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.calcite.runtime.SqlFunctions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
/**
* {@link Combine.CombineFn} for <em>Variance</em> on {@link Number} types.
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java
index 97d3ffc..1a90bf5 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinStringFunctions.java
@@ -22,6 +22,7 @@
import com.google.auto.service.AutoService;
import java.util.Arrays;
import org.apache.beam.sdk.schemas.Schema.TypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.lang3.ArrayUtils;
@@ -31,24 +32,20 @@
@AutoService(BeamBuiltinFunctionProvider.class)
public class BuiltinStringFunctions extends BeamBuiltinFunctionProvider {
- // return a explicitly null for Boolean has NP_BOOLEAN_RETURN_NULL warning.
- // return null for boolean is not allowed.
- // TODO: handle null input.
@UDF(
funcName = "ENDS_WITH",
parameterArray = {TypeName.STRING},
returnType = TypeName.STRING)
+ @Strict
public Boolean endsWith(String str1, String str2) {
return str1.endsWith(str2);
}
- // return a explicitly null for Boolean has NP_BOOLEAN_RETURN_NULL warning.
- // return null for boolean is not allowed.
- // TODO: handle null input.
@UDF(
funcName = "STARTS_WITH",
parameterArray = {TypeName.STRING},
returnType = TypeName.STRING)
+ @Strict
public Boolean startsWith(String str1, String str2) {
return str1.startsWith(str2);
}
@@ -57,10 +54,8 @@
funcName = "LENGTH",
parameterArray = {TypeName.STRING},
returnType = TypeName.INT64)
+ @Strict
public Long lengthString(String str) {
- if (str == null) {
- return null;
- }
return (long) str.length();
}
@@ -68,10 +63,8 @@
funcName = "LENGTH",
parameterArray = {TypeName.BYTES},
returnType = TypeName.INT64)
+ @Strict
public Long lengthBytes(byte[] bytes) {
- if (bytes == null) {
- return null;
- }
return (long) bytes.length;
}
@@ -79,10 +72,8 @@
funcName = "REVERSE",
parameterArray = {TypeName.STRING},
returnType = TypeName.STRING)
+ @Strict
public String reverseString(String str) {
- if (str == null) {
- return null;
- }
return new StringBuilder(str).reverse().toString();
}
@@ -90,10 +81,8 @@
funcName = "REVERSE",
parameterArray = {TypeName.BYTES},
returnType = TypeName.BYTES)
+ @Strict
public byte[] reverseBytes(byte[] bytes) {
- if (bytes == null) {
- return null;
- }
byte[] ret = Arrays.copyOf(bytes, bytes.length);
ArrayUtils.reverse(ret);
return ret;
@@ -103,11 +92,8 @@
funcName = "FROM_HEX",
parameterArray = {TypeName.STRING},
returnType = TypeName.BYTES)
+ @Strict
public byte[] fromHex(String str) {
- if (str == null) {
- return null;
- }
-
try {
return Hex.decodeHex(str.toCharArray());
} catch (DecoderException e) {
@@ -119,11 +105,8 @@
funcName = "TO_HEX",
parameterArray = {TypeName.BYTES},
returnType = TypeName.STRING)
+ @Strict
public String toHex(byte[] bytes) {
- if (bytes == null) {
- return null;
- }
-
return Hex.encodeHexString(bytes);
}
@@ -131,6 +114,7 @@
funcName = "LPAD",
parameterArray = {TypeName.STRING, TypeName.INT64},
returnType = TypeName.STRING)
+ @Strict
public String lpad(String originalValue, Long returnLength) {
return lpad(originalValue, returnLength, " ");
}
@@ -139,11 +123,8 @@
funcName = "LPAD",
parameterArray = {TypeName.STRING, TypeName.INT64, TypeName.STRING},
returnType = TypeName.STRING)
+ @Strict
public String lpad(String originalValue, Long returnLength, String pattern) {
- if (originalValue == null || returnLength == null || pattern == null) {
- return null;
- }
-
if (returnLength < -1 || pattern.isEmpty()) {
throw new IllegalArgumentException("returnLength cannot be 0 or pattern cannot be empty.");
}
@@ -162,6 +143,7 @@
funcName = "LPAD",
parameterArray = {TypeName.BYTES, TypeName.INT64},
returnType = TypeName.BYTES)
+ @Strict
public byte[] lpad(byte[] originalValue, Long returnLength) {
return lpad(originalValue, returnLength, " ".getBytes(UTF_8));
}
@@ -170,10 +152,8 @@
funcName = "LPAD",
parameterArray = {TypeName.BYTES, TypeName.INT64, TypeName.BYTES},
returnType = TypeName.BYTES)
+ @Strict
public byte[] lpad(byte[] originalValue, Long returnLength, byte[] pattern) {
- if (originalValue == null || returnLength == null || pattern == null) {
- return null;
- }
if (returnLength < -1 || pattern.length == 0) {
throw new IllegalArgumentException("returnLength cannot be 0 or pattern cannot be empty.");
}
@@ -205,6 +185,7 @@
funcName = "RPAD",
parameterArray = {TypeName.STRING, TypeName.INT64},
returnType = TypeName.STRING)
+ @Strict
public String rpad(String originalValue, Long returnLength) {
return lpad(originalValue, returnLength, " ");
}
@@ -213,11 +194,8 @@
funcName = "RPAD",
parameterArray = {TypeName.STRING, TypeName.INT64, TypeName.STRING},
returnType = TypeName.STRING)
+ @Strict
public String rpad(String originalValue, Long returnLength, String pattern) {
- if (originalValue == null || returnLength == null || pattern == null) {
- return null;
- }
-
if (returnLength < -1 || pattern.isEmpty()) {
throw new IllegalArgumentException("returnLength cannot be 0 or pattern cannot be empty.");
}
@@ -236,6 +214,7 @@
funcName = "RPAD",
parameterArray = {TypeName.BYTES, TypeName.INT64},
returnType = TypeName.BYTES)
+ @Strict
public byte[] rpad(byte[] originalValue, Long returnLength) {
return lpad(originalValue, returnLength, " ".getBytes(UTF_8));
}
@@ -244,10 +223,8 @@
funcName = "RPAD",
parameterArray = {TypeName.BYTES, TypeName.INT64, TypeName.BYTES},
returnType = TypeName.BYTES)
+ @Strict
public byte[] rpad(byte[] originalValue, Long returnLength, byte[] pattern) {
- if (originalValue == null || returnLength == null || pattern == null) {
- return null;
- }
if (returnLength < -1 || pattern.length == 0) {
throw new IllegalArgumentException("returnLength cannot be 0 or pattern cannot be empty.");
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java
index b79f48b..d00e6d6 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/BigDecimalConverter.java
@@ -21,7 +21,7 @@
import java.util.Map;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* Provides converters from {@link BigDecimal} to other numeric types based on the input {@link
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
index 207900f..dad5647 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
@@ -25,14 +25,14 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.BiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableBiMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.avatica.util.ByteString;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelDataTypeField;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.BiMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableBiMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.joda.time.Instant;
import org.joda.time.base.AbstractInstant;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java
index 6bf3cc2..ce75b92 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexFieldAccess.java
@@ -20,8 +20,8 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
/** SerializableRexFieldAccess. */
public class SerializableRexFieldAccess extends SerializableRexNode {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java
index 0b40c98..4d4d364 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexInputRef.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.utils;
-import org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
/** SerializableRexInputRef. */
public class SerializableRexInputRef extends SerializableRexNode {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java
index 31d5ab9..9796bf3 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/SerializableRexNode.java
@@ -18,9 +18,9 @@
package org.apache.beam.sdk.extensions.sql.impl.utils;
import java.io.Serializable;
-import org.apache.calcite.rex.RexFieldAccess;
-import org.apache.calcite.rex.RexInputRef;
-import org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexFieldAccess;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
/** SerializableRexNode. */
public abstract class SerializableRexNode implements Serializable {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
new file mode 100644
index 0000000..fd16ca6
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+
+import java.util.List;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Basic implementation of {@link BeamSqlTable} methods used by predicate and filter push-down. */
+public abstract class BaseBeamTable implements BeamSqlTable {
+
+ @Override
+ public PCollection<Row> buildIOReader(
+ PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames) {
+ String error = "%s does not support predicate/project push-down, yet non-empty %s is passed.";
+ checkArgument(
+ filters instanceof DefaultTableFilter, error, this.getClass().getName(), "'filters'");
+ checkArgument(fieldNames.isEmpty(), error, this.getClass().getName(), "'fieldNames'");
+
+ return buildIOReader(begin);
+ }
+
+ @Override
+ public BeamSqlTableFilter constructFilter(List<RexNode> filter) {
+ return new DefaultTableFilter(filter);
+ }
+
+ @Override
+ public boolean supportsProjects() {
+ return false;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
new file mode 100644
index 0000000..125bdd0
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** This interface defines a Beam Sql Table. */
+public interface BeamSqlTable {
+ /** create a {@code PCollection<Row>} from source. */
+ PCollection<Row> buildIOReader(PBegin begin);
+
+ /** create a {@code PCollection<Row>} from source with predicate and/or project pushed-down. */
+ PCollection<Row> buildIOReader(PBegin begin, BeamSqlTableFilter filters, List<String> fieldNames);
+
+ /** create a {@code IO.write()} instance to write to target. */
+ POutput buildIOWriter(PCollection<Row> input);
+
+ /** Generate an IO implementation of {@code BeamSqlTableFilter} for predicate push-down. */
+ BeamSqlTableFilter constructFilter(List<RexNode> filter);
+
+ /** Whether project push-down is supported by the IO API. */
+ boolean supportsProjects();
+
+ /** Whether this table is bounded (known to be finite) or unbounded (may or may not be finite). */
+ PCollection.IsBounded isBounded();
+
+ /** Get the schema info of the table. */
+ Schema getSchema();
+
+ /**
+ * Estimates the number of rows or the rate for unbounded Tables. If it is not possible to
+ * estimate the row count or rate it will return BeamTableStatistics.BOUNDED_UNKNOWN.
+ */
+ BeamTableStatistics getTableStatistics(PipelineOptions options);
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
new file mode 100644
index 0000000..80d4440
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** This interface defines Beam SQL Table Filter. */
+public interface BeamSqlTableFilter {
+ /**
+ * Identify parts of a predicate that are not supported by the IO push-down capabilities to be
+ * preserved in a {@code Calc} following {@code BeamIOSourceRel}.
+ *
+ * @return {@code List<RexNode>} unsupported by the IO API. Should be empty when an entire
+ * condition is supported, or an unchanged {@code List<RexNode>} when predicate push-down is
+ * not supported at all.
+ */
+ List<RexNode> getNotSupported();
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolver.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolver.java
new file mode 100644
index 0000000..fc066bd
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolver.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+
+/**
+ * Interface that table providers can implement if they require custom table name resolution.
+ *
+ * <p>{@link #registerKnownTableNames(List)} is called by the parser/planner and takes the list of
+ * all tables mentioned in the query. Then when normal Calcite lifecycle is executed the table
+ * provider can now check against this list and perform custom resolution. This is a workaround for
+ * lack of context in Calcite's logic, e.g. it's impossible to receive the whole table name at once,
+ * or understand that it has done querying sub-schemas and expects a table.
+ */
+public interface CustomTableResolver extends TableProvider {
+
+ /**
+ * Register the table names as extracted from the FROM clause.
+ *
+ * <p>Calcite doesn't provide these full names to table providers and queries them with individual
+ * parts of the identifiers without giving any extra context. So if a table provider needs to
+ * implement some custom table name resolution strategy it doesn't have information to do so. E.g.
+ * if you want to take the compound SQL identifiers that were originally split by dots, join them
+ * into a single string, and then query a back-end service, this interface makes this possible.
+ */
+ void registerKnownTableNames(List<TableName> tableNames);
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
new file mode 100644
index 0000000..685ce9e
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * This default implementation of {@link BeamSqlTableFilter} interface. Assumes that predicate
+ * push-down is not supported.
+ */
+public final class DefaultTableFilter implements BeamSqlTableFilter {
+ private final List<RexNode> filters;
+
+ DefaultTableFilter(List<RexNode> filters) {
+ this.filters = filters;
+ }
+
+ /**
+ * Since predicate push-down is assumed not to be supported by default - return an unchanged list
+ * of filters to be preserved.
+ *
+ * @return Predicate {@code List<RexNode>} which are not supported. To make a single RexNode
+ * expression all of the nodes must be joined by a logical AND.
+ */
+ @Override
+ public List<RexNode> getNotSupported() {
+ return filters;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
new file mode 100644
index 0000000..9842393
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.schemas.Schema;
+
+/** Each IO in Beam has one table schema, by extending {@link SchemaBaseBeamTable}. */
+public abstract class SchemaBaseBeamTable extends BaseBeamTable implements Serializable {
+ protected Schema schema;
+
+ public SchemaBaseBeamTable(Schema schema) {
+ this.schema = schema;
+ }
+
+ @Override
+ public Schema getSchema() {
+ return schema;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
new file mode 100644
index 0000000..a06632a
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider;
+
+import static java.util.stream.Collectors.toList;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.CustomTableResolver;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+
+/**
+ * Base class for table providers that look up table metadata using full table names, instead of
+ * querying it by parts of the name separately.
+ */
+@Experimental
+public abstract class FullNameTableProvider implements TableProvider, CustomTableResolver {
+
+ private List<TableName> knownTables;
+
+ protected FullNameTableProvider() {
+ knownTables = new ArrayList<>();
+ }
+
+ public abstract Table getTableByFullName(TableName fullTableName);
+
+ @Override
+ public void registerKnownTableNames(List<TableName> tableNames) {
+ knownTables.addAll(tableNames);
+ }
+
+ @Override
+ public TableProvider getSubProvider(String name) {
+ // TODO: implement with trie
+
+ // If 'name' matches a sub-schema/sub-provider we start tracking
+ // the subsequent calls to getSubProvider().
+ //
+ // Simple table ids and final table lookup
+ //
+ // If there is no matching sub-schema then returning null from here indicates
+ // that 'name' is either not part of this schema or it's a table, not a sub-schema,
+ // this will be checked right after this in a getTable() call.
+ //
+ // Because this is a getSubProvider() call it means Calcite expects
+ // the sub-schema/sub-provider to be returned, not a table,
+ // so we only need to check against known compound table identifiers.
+ // If 'name' acutally represents a simple identifier then it will be checked
+ // in a 'getTable()' call later. Unless there's the same sub-provider name,
+ // in which case it's a conflict and we will use the sub-schema and not assume it's a table.
+ // Calcite does the same.
+ //
+ // Here we find if there are any parsed tables that start from 'name' that belong to this
+ // table provider.
+ // We then create a fake tracking provider that in a trie-manner collects
+ // getSubProvider()/getTable() calls by checking whether there are known parsed table names
+ // matching what Calcite asks us for.
+ List<TableName> tablesToLookFor =
+ knownTables.stream()
+ .filter(TableName::isCompound)
+ .filter(tableName -> tableName.getPrefix().equals(name))
+ .collect(toList());
+
+ return tablesToLookFor.size() > 0 ? new TableNameTrackingProvider(1, tablesToLookFor) : null;
+ }
+
+ /**
+ * Calcite calls getSubProvider()/getTable() on this class when resolving a table name. This class
+ * keeps track of these calls and checks against known table names (extracted from a query), so
+ * that when a full table name is parsed out it calls the actual table provider to get a table
+ * based on the full name, instead of calling it component by component.
+ *
+ * <p>This class nables table providers to query their metadata source using full table names.
+ */
+ class TableNameTrackingProvider extends InMemoryMetaTableProvider {
+ int schemaLevel;
+ List<TableName> tableNames;
+
+ TableNameTrackingProvider(int schemaLevel, List<TableName> tableNames) {
+ this.schemaLevel = schemaLevel;
+ this.tableNames = tableNames;
+ }
+
+ @Override
+ public TableProvider getSubProvider(String name) {
+ // Find if any of the parsed table names have 'name' as part
+ // of their path at current index.
+ //
+ // If there are, return a new tracking provider for such tables and incremented index.
+ //
+ // If there are none, it means something weird has happened and returning null
+ // will make Calcite try other schemas. Maybe things will work out.
+ //
+ // However since we originally register all parsed table names for the given schema
+ // in this provider we should only receive a getSubProvider() call for something unknown
+ // when it's a leaf path element, i.e. actual table name, which will be handled in
+ // getTable() call.
+ List<TableName> matchingTables =
+ tableNames.stream()
+ .filter(TableName::isCompound)
+ .filter(tableName -> tableName.getPath().size() > schemaLevel)
+ .filter(tableName -> tableName.getPath().get(schemaLevel).equals(name))
+ .collect(toList());
+
+ return matchingTables.size() > 0
+ ? new TableNameTrackingProvider(schemaLevel + 1, matchingTables)
+ : null;
+ }
+
+ @Override
+ public String getTableType() {
+ return "google.cloud.datacatalog.subprovider";
+ }
+
+ @Nullable
+ @Override
+ public Table getTable(String name) {
+
+ // This is called only after getSubProvider() returned null,
+ // and since we are tracking the actual parsed table names, this should
+ // be it, there should exist a parsed table that matches the 'name'.
+
+ Optional<TableName> matchingTable =
+ tableNames.stream()
+ .filter(tableName -> tableName.getTableName().equals(name))
+ .findFirst();
+
+ TableName fullTableName =
+ matchingTable.orElseThrow(
+ () ->
+ new IllegalStateException(
+ "Unexpected table '"
+ + name
+ + "' requested. Current schema level is "
+ + schemaLevel
+ + ". Current known table names: "
+ + tableNames.toString()));
+ return FullNameTableProvider.this.getTableByFullName(fullTableName);
+ }
+
+ @Override
+ public synchronized BeamSqlTable buildBeamSqlTable(Table table) {
+ return FullNameTableProvider.this.buildBeamSqlTable(table);
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
index 2645787..ab826fc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/ReadOnlyTableProvider.java
@@ -18,10 +18,10 @@
package org.apache.beam.sdk.extensions.sql.meta.provider;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* A {@code ReadOnlyTableProvider} provides in-memory read only set of {@code BeamSqlTable
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
index a93dcf8..d38cd7a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/TableProvider.java
@@ -21,9 +21,9 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
/**
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
new file mode 100644
index 0000000..b335003
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import java.io.Serializable;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.transforms.Convert;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.Row;
+
+/** {@link AvroTable} is a {@link BeamSqlTable}. */
+public class AvroTable extends SchemaBaseBeamTable implements Serializable {
+ private final String filePattern;
+ private final String tableName;
+
+ public AvroTable(String tableName, Schema beamSchema, String filePattern) {
+ super(beamSchema);
+ this.filePattern = filePattern;
+ this.tableName = tableName;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+
+ return begin
+ .apply(
+ "AvroIORead",
+ AvroIO.readGenericRecords(AvroUtils.toAvroSchema(schema, tableName, null))
+ .withBeamSchemas(true)
+ .from(filePattern))
+ .apply("GenericRecordToRow", Convert.toRows());
+ }
+
+ @Override
+ public PDone buildIOWriter(PCollection<Row> input) {
+ PTransform<PCollection<Row>, PCollection<GenericRecord>> writeConverter =
+ GenericRecordWriteConverter.builder().beamSchema(schema).build();
+
+ return input
+ .apply("GenericRecordToRow", writeConverter)
+ .apply(
+ "AvroIOWrite",
+ AvroIO.writeGenericRecords(AvroUtils.toAvroSchema(schema, tableName, null))
+ .to(filePattern)
+ .withoutSharding());
+ }
+
+ @Override
+ public PCollection.IsBounded isBounded() {
+ return PCollection.IsBounded.BOUNDED;
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
new file mode 100644
index 0000000..b2ffc59
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+
+/**
+ * {@link TableProvider} for {@link AvroTable}.
+ *
+ * <p>A sample of avro table is:
+ *
+ * <pre>{@code
+ * CREATE EXTERNAL TABLE ORDERS(
+ * name VARCHAR,
+ * favorite_color VARCHAR,
+ * favorite_numbers ARRAY<INTEGER>
+ * )
+ * TYPE 'avro'
+ * LOCATION '/tmp/persons.avro'
+ * }</pre>
+ */
+@AutoService(TableProvider.class)
+public class AvroTableProvider extends InMemoryMetaTableProvider {
+ @Override
+ public String getTableType() {
+ return "avro";
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return new AvroTable(table.getName(), table.getSchema(), table.getLocation());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java
new file mode 100644
index 0000000..6ca1621
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/** A {@link PTransform} to convert {@link Row} to {@link GenericRecord}. */
+@AutoValue
+public abstract class GenericRecordWriteConverter
+ extends PTransform<PCollection<Row>, PCollection<GenericRecord>> implements Serializable {
+
+ public abstract Schema beamSchema();
+
+ public static Builder builder() {
+ return new AutoValue_GenericRecordWriteConverter.Builder();
+ }
+
+ @Override
+ public PCollection<GenericRecord> expand(PCollection<Row> input) {
+ return input
+ .apply(
+ "RowsToGenericRecord",
+ ParDo.of(
+ new DoFn<Row, GenericRecord>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ GenericRecord genericRecord =
+ AvroUtils.toGenericRecord(
+ c.element(), AvroUtils.toAvroSchema(beamSchema()));
+ c.output(genericRecord);
+ }
+ }))
+ .setCoder(AvroCoder.of(GenericRecord.class, AvroUtils.toAvroSchema(beamSchema())));
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ public abstract Builder beamSchema(Schema beamSchema);
+
+ public abstract GenericRecordWriteConverter build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java
new file mode 100644
index 0000000..f50679c
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Table schema for AvroIO. */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
index 621f149..be42b86 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
@@ -17,33 +17,86 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+import java.io.IOException;
import java.io.Serializable;
+import java.math.BigInteger;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.ConversionOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* {@code BigQueryTable} represent a BigQuery table as a target. This provider does not currently
* support being a source.
*/
@Experimental
-class BigQueryTable extends BaseBeamTable implements Serializable {
+class BigQueryTable extends SchemaBaseBeamTable implements Serializable {
+ @VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
+ private BeamTableStatistics rowCountStatistics = null;
+ private static final Logger LOGGER = LoggerFactory.getLogger(BigQueryTable.class);
+ @VisibleForTesting final Method method;
BigQueryTable(Table table, BigQueryUtils.ConversionOptions options) {
super(table.getSchema());
this.conversionOptions = options;
this.bqLocation = table.getLocation();
+
+ if (table.getProperties().containsKey(METHOD_PROPERTY)) {
+ List<String> validMethods =
+ Arrays.stream(Method.values()).map(Enum::toString).collect(Collectors.toList());
+ // toUpperCase should make it case-insensitive
+ String selectedMethod = table.getProperties().getString(METHOD_PROPERTY).toUpperCase();
+
+ if (validMethods.contains(selectedMethod)) {
+ method = Method.valueOf(selectedMethod);
+ } else {
+ InvalidPropertyException e =
+ new InvalidPropertyException(
+ "Invalid method "
+ + "'"
+ + selectedMethod
+ + "'. "
+ + "Supported methods are: "
+ + validMethods.toString()
+ + ".");
+
+ throw e;
+ }
+ } else {
+ method = Method.DEFAULT;
+ }
+
+ LOGGER.info("BigQuery method is set to: " + method.toString());
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+
+ if (rowCountStatistics == null) {
+ rowCountStatistics = getRowCountFromBQ(options, bqLocation);
+ }
+
+ return rowCountStatistics;
}
@Override
@@ -59,6 +112,7 @@
BigQueryIO.read(
record ->
BigQueryUtils.toBeamRow(record.getRecord(), getSchema(), conversionOptions))
+ .withMethod(method)
.from(bqLocation)
.withCoder(SchemaCoder.of(getSchema())))
.setRowSchema(getSchema());
@@ -72,4 +126,29 @@
.withFormatFunction(BigQueryUtils.toTableRow())
.to(bqLocation));
}
+
+ private static BeamTableStatistics getRowCountFromBQ(PipelineOptions o, String bqLocation) {
+ try {
+ BigInteger rowCount =
+ BigQueryHelpers.getNumRows(
+ o.as(BigQueryOptions.class), BigQueryHelpers.parseTableSpec(bqLocation));
+
+ if (rowCount == null) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+
+ return BeamTableStatistics.createBoundedTableStatistics(rowCount.doubleValue());
+
+ } catch (IOException | InterruptedException e) {
+ LOGGER.warn("Could not get the row count for the table " + bqLocation, e);
+ }
+
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+
+ public static class InvalidPropertyException extends UnsupportedOperationException {
+ private InvalidPropertyException(String s) {
+ super(s);
+ }
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
index b4ddf29..9c4266b 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProvider.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
import com.google.auto.service.AutoService;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
index 156c0c9..893d1ca 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java
@@ -17,13 +17,20 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
+import java.util.Collection;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Properties;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
import org.apache.beam.sdk.coders.ByteArrayCoder;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.kafka.KafkaIO;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.KV;
@@ -31,19 +38,29 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* {@code BeamKafkaTable} represent a Kafka topic, as source or target. Need to extend to convert
* between {@code BeamSqlRow} and {@code KV<byte[], byte[]>}.
*/
-public abstract class BeamKafkaTable extends BaseBeamTable {
+public abstract class BeamKafkaTable extends SchemaBaseBeamTable {
private String bootstrapServers;
private List<String> topics;
private List<TopicPartition> topicPartitions;
private Map<String, Object> configUpdates;
+ private BeamTableStatistics rowCountStatistics = null;
+ private static final Logger LOGGER = LoggerFactory.getLogger(BeamKafkaTable.class);
+ // This is the number of records looked from each partition when the rate is estimated
+ protected int numberOfRecordsForRate = 50;
protected BeamKafkaTable(Schema beamSchema) {
super(beamSchema);
@@ -53,6 +70,7 @@
super(beamSchema);
this.bootstrapServers = bootstrapServers;
this.topics = topics;
+ this.configUpdates = new HashMap<>();
}
public BeamKafkaTable(
@@ -80,7 +98,14 @@
@Override
public PCollection<Row> buildIOReader(PBegin begin) {
- KafkaIO.Read<byte[], byte[]> kafkaRead = null;
+ return begin
+ .apply("read", createKafkaRead().withoutMetadata())
+ .apply("in_format", getPTransformForInput())
+ .setRowSchema(getSchema());
+ }
+
+ KafkaIO.Read<byte[], byte[]> createKafkaRead() {
+ KafkaIO.Read<byte[], byte[]> kafkaRead;
if (topics != null) {
kafkaRead =
KafkaIO.<byte[], byte[]>read()
@@ -100,28 +125,25 @@
} else {
throw new IllegalArgumentException("One of topics and topicPartitions must be configurated.");
}
-
- return begin
- .apply("read", kafkaRead.withoutMetadata())
- .apply("in_format", getPTransformForInput())
- .setRowSchema(getSchema());
+ return kafkaRead;
}
@Override
public POutput buildIOWriter(PCollection<Row> input) {
checkArgument(
topics != null && topics.size() == 1, "Only one topic can be acceptable as output.");
- assert topics != null;
return input
.apply("out_reformat", getPTransformForOutput())
- .apply(
- "persistent",
- KafkaIO.<byte[], byte[]>write()
- .withBootstrapServers(bootstrapServers)
- .withTopic(topics.get(0))
- .withKeySerializer(ByteArraySerializer.class)
- .withValueSerializer(ByteArraySerializer.class));
+ .apply("persistent", createKafkaWrite());
+ }
+
+ private KafkaIO.Write<byte[], byte[]> createKafkaWrite() {
+ return KafkaIO.<byte[], byte[]>write()
+ .withBootstrapServers(bootstrapServers)
+ .withTopic(topics.get(0))
+ .withKeySerializer(ByteArraySerializer.class)
+ .withValueSerializer(ByteArraySerializer.class);
}
public String getBootstrapServers() {
@@ -131,4 +153,108 @@
public List<String> getTopics() {
return topics;
}
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ if (rowCountStatistics == null) {
+ try {
+ rowCountStatistics =
+ BeamTableStatistics.createUnboundedTableStatistics(
+ this.computeRate(numberOfRecordsForRate));
+ } catch (Exception e) {
+ LOGGER.warn("Could not get the row count for the topics " + getTopics(), e);
+ rowCountStatistics = BeamTableStatistics.UNBOUNDED_UNKNOWN;
+ }
+ }
+
+ return rowCountStatistics;
+ }
+
+ /**
+ * This method returns the estimate of the computeRate for this table using last numberOfRecords
+ * tuples in each partition.
+ */
+ double computeRate(int numberOfRecords) throws NoEstimationException {
+ Properties props = new Properties();
+
+ props.put("bootstrap.servers", bootstrapServers);
+ props.put("session.timeout.ms", "30000");
+ props.put("key.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer");
+ props.put("value.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer");
+
+ KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
+
+ return computeRate(consumer, numberOfRecords);
+ }
+
+ <T> double computeRate(Consumer<T, T> consumer, int numberOfRecordsToCheck)
+ throws NoEstimationException {
+
+ Stream<TopicPartition> c =
+ getTopics().stream()
+ .map(consumer::partitionsFor)
+ .flatMap(Collection::stream)
+ .map(parInf -> new TopicPartition(parInf.topic(), parInf.partition()));
+ List<TopicPartition> topicPartitions = c.collect(Collectors.toList());
+
+ consumer.assign(topicPartitions);
+ // This will return current offset of all the partitions that are assigned to the consumer. (It
+ // will be the last record in those partitions). Note that each topic can have multiple
+ // partitions. Since the consumer is not assigned to any consumer group, changing the offset or
+ // consuming messages does not have any effect on the other consumers (and the data that our
+ // table is receiving)
+ Map<TopicPartition, Long> offsets = consumer.endOffsets(topicPartitions);
+ long nParsSeen = 0;
+ for (TopicPartition par : topicPartitions) {
+ long offset = offsets.get(par);
+ nParsSeen = (offset == 0) ? nParsSeen : nParsSeen + 1;
+ consumer.seek(par, Math.max(0L, offset - numberOfRecordsToCheck));
+ }
+
+ if (nParsSeen == 0) {
+ throw new NoEstimationException("There is no partition with messages in it.");
+ }
+
+ ConsumerRecords<T, T> records = consumer.poll(1000);
+
+ // Kafka guarantees the delivery of messages in order they arrive to each partition.
+ // Therefore the first message seen from each partition is the first message arrived to that.
+ // We pick all the first messages of the partitions, and then consider the latest one as the
+ // starting point
+ // and discard all the messages that have arrived sooner than that in the rate estimation.
+ Map<Integer, Long> minTimeStamps = new HashMap<>();
+ long maxMinTimeStamp = 0;
+ for (ConsumerRecord<T, T> record : records) {
+ if (!minTimeStamps.containsKey(record.partition())) {
+ minTimeStamps.put(record.partition(), record.timestamp());
+
+ nParsSeen--;
+ maxMinTimeStamp = Math.max(record.timestamp(), maxMinTimeStamp);
+ if (nParsSeen == 0) {
+ break;
+ }
+ }
+ }
+
+ int numberOfRecords = 0;
+ long maxTimeStamp = 0;
+ for (ConsumerRecord<T, T> record : records) {
+ maxTimeStamp = Math.max(maxTimeStamp, record.timestamp());
+ numberOfRecords =
+ record.timestamp() > maxMinTimeStamp ? numberOfRecords + 1 : numberOfRecords;
+ }
+
+ if (maxTimeStamp == maxMinTimeStamp) {
+ throw new NoEstimationException("Arrival time of all records are the same.");
+ }
+
+ return (numberOfRecords * 1000.) / ((double) maxTimeStamp - maxMinTimeStamp);
+ }
+
+ /** Will be thrown if we cannot estimate the rate for kafka table. */
+ static class NoEstimationException extends Exception {
+ NoEstimationException(String message) {
+ super(message);
+ }
+ }
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
index fe59b83..c288c92 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider.java
@@ -22,7 +22,7 @@
import com.google.auto.service.AutoService;
import java.util.ArrayList;
import java.util.List;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordReadConverter.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordReadConverter.java
new file mode 100644
index 0000000..d5ba45b
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordReadConverter.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+
+/** A {@link PTransform} to convert {@link GenericRecord} to {@link Row}. */
+@AutoValue
+public abstract class GenericRecordReadConverter
+ extends PTransform<PCollection<GenericRecord>, PCollection<Row>> implements Serializable {
+
+ public abstract Schema beamSchema();
+
+ public static Builder builder() {
+ return new AutoValue_GenericRecordReadConverter.Builder();
+ }
+
+ @Override
+ public PCollection<Row> expand(PCollection<GenericRecord> input) {
+ return input
+ .apply(
+ "GenericRecordsToRows",
+ ParDo.of(
+ new DoFn<GenericRecord, Row>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ Row row = AvroUtils.toBeamRowStrict(c.element(), beamSchema());
+ c.output(row);
+ }
+ }))
+ .setRowSchema(beamSchema());
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ public abstract Builder beamSchema(Schema beamSchema);
+
+ public abstract GenericRecordReadConverter build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
new file mode 100644
index 0000000..00f46b9
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
+
+import java.io.Serializable;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.io.parquet.ParquetIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.Row;
+
+/** {@link ParquetTable} is a {@link BeamSqlTable}. */
+public class ParquetTable extends SchemaBaseBeamTable implements Serializable {
+ private final String filePattern;
+
+ public ParquetTable(Schema beamSchema, String filePattern) {
+ super(beamSchema);
+ this.filePattern = filePattern;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ PTransform<PCollection<GenericRecord>, PCollection<Row>> readConverter =
+ GenericRecordReadConverter.builder().beamSchema(schema).build();
+
+ return begin
+ .apply("ParquetIORead", ParquetIO.read(AvroUtils.toAvroSchema(schema)).from(filePattern))
+ .apply("GenericRecordToRow", readConverter);
+ }
+
+ @Override
+ public PDone buildIOWriter(PCollection<Row> input) {
+ throw new UnsupportedOperationException("Writing to a Parquet file is not supported");
+ }
+
+ @Override
+ public PCollection.IsBounded isBounded() {
+ return PCollection.IsBounded.BOUNDED;
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
new file mode 100644
index 0000000..8a7e5fa
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+
+/**
+ * {@link TableProvider} for {@link ParquetTable}.
+ *
+ * <p>A sample of parquet table is:
+ *
+ * <pre>{@code
+ * CREATE TABLE ORDERS(
+ * name VARCHAR,
+ * favorite_color VARCHAR,
+ * favorite_numbers ARRAY<INTEGER>
+ * )
+ * TYPE 'parquet'
+ * LOCATION '/home/admin/users.parquet'
+ * }</pre>
+ */
+@AutoService(TableProvider.class)
+public class ParquetTableProvider extends InMemoryMetaTableProvider {
+ @Override
+ public String getTableType() {
+ return "parquet";
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return new ParquetTable(table.getSchema(), table.getLocation());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/package-info.java
new file mode 100644
index 0000000..2f8c2f9
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Table schema for ParquetIO. */
+package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
index a3d338a..551e5a0 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java
@@ -25,9 +25,12 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubIO;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PBegin;
@@ -86,7 +89,7 @@
@AutoValue
@Internal
@Experimental
-abstract class PubsubIOJsonTable implements BeamSqlTable, Serializable {
+abstract class PubsubIOJsonTable extends BaseBeamTable implements Serializable {
/**
* Optional attribute key of the Pubsub message from which to extract the event timestamp.
@@ -186,6 +189,11 @@
throw new UnsupportedOperationException("Writing to a Pubsub topic is not supported");
}
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.UNBOUNDED_UNKNOWN;
+ }
+
@AutoValue.Builder
abstract static class Builder {
abstract Builder setSchema(Schema schema);
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
index 9c13c8a..dc49771 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProvider.java
@@ -28,7 +28,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
index d770380..654e722 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRow.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.extensions.sql.meta.provider.pubsub;
import static java.util.stream.Collectors.toList;
-import static org.apache.beam.sdk.util.JsonToRowUtils.newObjectMapperWith;
+import static org.apache.beam.sdk.util.RowJsonUtils.newObjectMapperWith;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.auto.value.AutoValue;
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.TypeName;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.util.JsonToRowUtils;
import org.apache.beam.sdk.util.RowJsonDeserializer;
import org.apache.beam.sdk.util.RowJsonDeserializer.UnsupportedRowJsonException;
+import org.apache.beam.sdk.util.RowJsonUtils;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.joda.time.Instant;
@@ -127,7 +127,7 @@
objectMapper = newObjectMapperWith(RowJsonDeserializer.forSchema(payloadSchema()));
}
- return JsonToRowUtils.jsonToRow(objectMapper, payloadJson);
+ return RowJsonUtils.jsonToRow(objectMapper, payloadJson);
}
@AutoValue.Builder
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
new file mode 100644
index 0000000..eea7bc4
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.seqgen;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollection.IsBounded;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+class GenerateSequenceTable extends SchemaBaseBeamTable implements Serializable {
+ public static final Schema TABLE_SCHEMA =
+ Schema.of(Field.of("sequence", FieldType.INT64), Field.of("event_time", FieldType.DATETIME));
+
+ Integer elementsPerSecond = 5;
+
+ GenerateSequenceTable(Table table) {
+ super(TABLE_SCHEMA);
+ if (table.getProperties().containsKey("elementsPerSecond")) {
+ elementsPerSecond = table.getProperties().getInteger("elementsPerSecond");
+ }
+ }
+
+ @Override
+ public PCollection.IsBounded isBounded() {
+ return IsBounded.UNBOUNDED;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ return begin
+ .apply(GenerateSequence.from(0).withRate(elementsPerSecond, Duration.standardSeconds(1)))
+ .apply(
+ MapElements.into(TypeDescriptor.of(Row.class))
+ .via(elm -> Row.withSchema(TABLE_SCHEMA).addValues(elm, Instant.now()).build()))
+ .setRowSchema(getSchema());
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.createUnboundedTableStatistics((double) elementsPerSecond);
+ }
+
+ @Override
+ public POutput buildIOWriter(PCollection<Row> input) {
+ throw new UnsupportedOperationException("buildIOWriter unsupported!");
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
new file mode 100644
index 0000000..1344223
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.seqgen;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+
+/**
+ * Sequence generator table provider.
+ *
+ * <p>A sample of text table is:
+ *
+ * <pre>{@code
+ * CREATE EXTERNAL TABLE MY_SEQUENCE(
+ * sequence BIGINT COMMENT 'this is the primary key',
+ * event_time TIMESTAMP COMMENT 'this is the element timestamp'
+ * )
+ * TYPE 'sequence';
+ * }</pre>
+ */
+@AutoService(TableProvider.class)
+public class GenerateSequenceTableProvider extends InMemoryMetaTableProvider {
+
+ @Override
+ public String getTableType() {
+ return "sequence";
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return new GenerateSequenceTable(table);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/package-info.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/package-info.java
new file mode 100644
index 0000000..d7841db
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Table schema for streaming sequence generator. */
+package org.apache.beam.sdk.extensions.sql.meta.provider.seqgen;
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestBoundedTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestBoundedTable.java
index 5c92c47..6f88b15 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestBoundedTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestBoundedTable.java
@@ -22,6 +22,8 @@
import java.util.List;
import java.util.concurrent.ConcurrentLinkedQueue;
import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
@@ -45,6 +47,11 @@
}
@Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.createBoundedTableStatistics((double) rows.size());
+ }
+
+ @Override
public PCollection.IsBounded isBounded() {
return PCollection.IsBounded.BOUNDED;
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
index c0bfd0d..807fc2a 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable.java
@@ -19,7 +19,7 @@
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.POutput;
@@ -27,7 +27,7 @@
/** Base class for mocked table. */
@Experimental
-public abstract class TestTable extends BaseBeamTable {
+public abstract class TestTable extends SchemaBaseBeamTable {
public static final AtomicInteger COUNTER = new AtomicInteger();
public TestTable(Schema beamSchema) {
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
index 3689ba8..b9d7bd7 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProvider.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.test;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.Serializable;
@@ -29,16 +29,18 @@
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
@@ -117,7 +119,7 @@
}
}
- private static class InMemoryTable implements BeamSqlTable {
+ private static class InMemoryTable extends BaseBeamTable {
private TableWithRows tableWithRows;
@Override
@@ -130,10 +132,13 @@
}
public Coder<Row> rowCoder() {
- return SchemaCoder.of(
- tableWithRows.table.getSchema(),
- SerializableFunctions.identity(),
- SerializableFunctions.identity());
+ return SchemaCoder.of(tableWithRows.table.getSchema());
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.createBoundedTableStatistics(
+ (double) tableWithRows.getRows().size());
}
@Override
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java
index d133600..7ebbf91 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableUtils.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Lists;
/** Utility functions for mock classes. */
@Experimental
@@ -55,11 +55,25 @@
.collect(toSchema());
}
+ public static Schema buildBeamSqlNullableSchema(Object... args) {
+ return Stream.iterate(0, i -> i + 3)
+ .limit(args.length / 3)
+ .map(i -> toNullableRecordField(args, i))
+ .collect(toSchema());
+ }
+
// TODO: support nested.
public static Schema.Field toRecordField(Object[] args, int i) {
return Schema.Field.of((String) args[i + 1], (FieldType) args[i]);
}
+ public static Schema.Field toNullableRecordField(Object[] args, int i) {
+ if ((boolean) args[i + 2]) {
+ return Schema.Field.nullable((String) args[i + 1], (FieldType) args[i]);
+ }
+ return Schema.Field.of((String) args[i + 1], (FieldType) args[i]);
+ }
+
/**
* Convenient way to build a {@code BeamSqlRow}s.
*
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java
index 5d41fef..22c1bd2 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestUnboundedTable.java
@@ -21,14 +21,15 @@
import java.util.Arrays;
import java.util.List;
import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -40,6 +41,8 @@
/** specify the index of column in the row which stands for the event time field. */
private int timestampField;
+ private BeamTableStatistics statistics = BeamTableStatistics.UNBOUNDED_UNKNOWN;
+
private TestUnboundedTable(Schema beamSchema) {
super(beamSchema);
}
@@ -61,6 +64,16 @@
return new TestUnboundedTable(TestTableUtils.buildBeamSqlSchema(args));
}
+ public TestUnboundedTable setStatistics(BeamTableStatistics statistics) {
+ this.statistics = statistics;
+ return this;
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return this.statistics;
+ }
+
public TestUnboundedTable timestampColumnIndex(int idx) {
this.timestampField = idx;
return this;
@@ -94,9 +107,7 @@
@Override
public PCollection<Row> buildIOReader(PBegin begin) {
- TestStream.Builder<Row> values =
- TestStream.create(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(schema);
for (Pair<Duration, List<Row>> pair : timestampedRows) {
values = values.advanceWatermarkTo(new Instant(0).plus(pair.getKey()));
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
index 94674de..acf4ccf 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable.java
@@ -17,9 +17,14 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.text;
+import java.io.IOException;
import org.apache.beam.sdk.annotations.Internal;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.TextRowCountEstimator;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PBegin;
@@ -27,21 +32,27 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
import org.apache.commons.csv.CSVFormat;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
- * {@link TextTable} is a {@link org.apache.beam.sdk.extensions.sql.BeamSqlTable} that reads text
- * files and converts them according to the specified format.
+ * {@link TextTable} is a {@link BeamSqlTable} that reads text files and converts them according to
+ * the specified format.
*
* <p>Support formats are {@code "csv"} and {@code "lines"}.
*
* <p>{@link CSVFormat} itself has many dialects, check its javadoc for more info.
*/
@Internal
-public class TextTable extends BaseBeamTable {
+public class TextTable extends SchemaBaseBeamTable {
private final PTransform<PCollection<String>, PCollection<Row>> readConverter;
private final PTransform<PCollection<Row>, PCollection<String>> writeConverter;
+ private static final TextRowCountEstimator.SamplingStrategy DEFAULT_SAMPLING_STRATEGY =
+ new TextRowCountEstimator.LimitNumberOfTotalBytes(1024 * 1024L);
private final String filePattern;
+ private BeamTableStatistics rowCountStatistics = null;
+ private static final Logger LOGGER = LoggerFactory.getLogger(TextTable.class);
/** Text table with the specified read and write transforms. */
public TextTable(
@@ -60,6 +71,31 @@
}
@Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ if (rowCountStatistics == null) {
+ rowCountStatistics = getTextRowEstimate(options, getFilePattern());
+ }
+
+ return rowCountStatistics;
+ }
+
+ private static BeamTableStatistics getTextRowEstimate(
+ PipelineOptions options, String filePattern) {
+ TextRowCountEstimator textRowCountEstimator =
+ TextRowCountEstimator.builder()
+ .setFilePattern(filePattern)
+ .setSamplingStrategy(DEFAULT_SAMPLING_STRATEGY)
+ .build();
+ try {
+ Double rows = textRowCountEstimator.estimateRowCount(options);
+ return BeamTableStatistics.createBoundedTableStatistics(rows);
+ } catch (IOException | TextRowCountEstimator.NoEstimationException e) {
+ LOGGER.warn("Could not get the row count for the text table " + filePattern, e);
+ }
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+
+ @Override
public PCollection.IsBounded isBounded() {
return PCollection.IsBounded.BOUNDED;
}
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
index 38d4e1f..229e7cc 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider.java
@@ -19,13 +19,13 @@
import static org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils.beamRow2CsvLine;
import static org.apache.beam.sdk.extensions.sql.impl.schema.BeamTableUtils.csvLines2BeamRows;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import com.alibaba.fastjson.JSONObject;
import com.google.auto.service.AutoService;
import java.io.Serializable;
import javax.annotation.Nullable;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.InMemoryMetaTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
@@ -37,9 +37,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
import org.apache.commons.csv.CSVFormat;
/**
diff --git a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
index 266449a..7a3fb1f 100644
--- a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
+++ b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStore.java
@@ -19,10 +19,10 @@
import java.util.HashMap;
import java.util.Map;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/**
* A {@link MetaStore} which stores the meta info in memory.
diff --git a/sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties b/sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties
deleted file mode 100644
index 72a4eec..0000000
--- a/sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties
+++ /dev/null
@@ -1,18 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-compilerFactory=org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.janino.CompilerFactory
diff --git a/sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties b/sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties
new file mode 100644
index 0000000..ab9a234
--- /dev/null
+++ b/sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties
@@ -0,0 +1,18 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+compilerFactory=org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.janino.CompilerFactory
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
index fbdf625..24d23c9 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamComplexTypeTest.java
@@ -18,6 +18,8 @@
package org.apache.beam.sdk.extensions.sql;
import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
@@ -26,10 +28,14 @@
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.joda.time.DateTime;
import org.joda.time.Duration;
+import org.joda.time.Instant;
import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
@@ -45,6 +51,12 @@
.addArrayField("array_field", FieldType.INT64)
.build();
+ private static final Schema nullableInnerRowSchema =
+ Schema.builder()
+ .addNullableField("inner_row_field", FieldType.row(innerRowSchema))
+ .addNullableField("array_field", FieldType.array(FieldType.row(innerRowSchema)))
+ .build();
+
private static final Schema nestedRowWithArraySchema =
Schema.builder()
.addStringField("field1")
@@ -53,6 +65,13 @@
.addArrayField("field4", FieldType.array(FieldType.STRING))
.build();
+ private static final Schema nullableNestedRowWithArraySchema =
+ Schema.builder()
+ .addNullableField("field1", FieldType.row(innerRowWithArraySchema))
+ .addNullableField("field2", FieldType.array(FieldType.row(innerRowWithArraySchema)))
+ .addNullableField("field3", FieldType.row(nullableInnerRowSchema))
+ .build();
+
private static final Schema nestedRowSchema =
Schema.builder()
.addStringField("nonRowfield1")
@@ -165,6 +184,19 @@
}
@Test
+ public void testArrayConstructor() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(readOnlyTableProvider);
+ PCollection<Row> stream =
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery("SELECT ARRAY[1, 2, 3] f_arr"));
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addArrayField("f_arr", FieldType.INT32).build())
+ .addValue(Arrays.asList(1, 2, 3))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
public void testRowWithArray() {
BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(readOnlyTableProvider);
PCollection<Row> stream =
@@ -213,6 +245,55 @@
}
@Test
+ public void testNestedBytes() {
+ byte[] bytes = new byte[] {-70, -83, -54, -2};
+
+ Schema nestedInputSchema = Schema.of(Schema.Field.of("c_bytes", Schema.FieldType.BYTES));
+ Schema inputSchema =
+ Schema.of(Schema.Field.of("nested", Schema.FieldType.row(nestedInputSchema)));
+
+ Schema outputSchema = Schema.of(Schema.Field.of("f0", Schema.FieldType.BYTES));
+
+ Row nestedRow = Row.withSchema(nestedInputSchema).addValue(bytes).build();
+ Row row = Row.withSchema(inputSchema).addValue(nestedRow).build();
+ Row expected = Row.withSchema(outputSchema).addValue(bytes).build();
+
+ PCollection<Row> result =
+ pipeline
+ .apply(Create.of(row).withRowSchema(inputSchema))
+ .apply(SqlTransform.query("SELECT t.nested.c_bytes AS f0 FROM PCOLLECTION t"));
+
+ PAssert.that(result).containsInAnyOrder(expected);
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testNestedArrayOfBytes() {
+ byte[] bytes = new byte[] {-70, -83, -54, -2};
+
+ Schema nestedInputSchema =
+ Schema.of(Schema.Field.of("c_bytes", Schema.FieldType.array(Schema.FieldType.BYTES)));
+ Schema inputSchema =
+ Schema.of(Schema.Field.of("nested", Schema.FieldType.row(nestedInputSchema)));
+
+ Schema outputSchema = Schema.of(Schema.Field.of("f0", Schema.FieldType.BYTES));
+
+ Row nestedRow = Row.withSchema(nestedInputSchema).addValue(ImmutableList.of(bytes)).build();
+ Row row = Row.withSchema(inputSchema).addValue(nestedRow).build();
+ Row expected = Row.withSchema(outputSchema).addValue(bytes).build();
+
+ PCollection<Row> result =
+ pipeline
+ .apply(Create.of(row).withRowSchema(inputSchema))
+ .apply(SqlTransform.query("SELECT t.nested.c_bytes[1] AS f0 FROM PCOLLECTION t"));
+
+ PAssert.that(result).containsInAnyOrder(expected);
+
+ pipeline.run();
+ }
+
+ @Test
public void testRowConstructor() {
BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(readOnlyTableProvider);
PCollection<Row> stream =
@@ -233,4 +314,249 @@
.build());
pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
}
+
+ @Test
+ public void testNullRows() {
+
+ Row nullRow = Row.nullRow(nullableNestedRowWithArraySchema);
+
+ PCollection<Row> outputRow =
+ pipeline
+ .apply(Create.of(nullRow))
+ .setRowSchema(nullableNestedRowWithArraySchema)
+ .apply(
+ SqlTransform.query(
+ "select PCOLLECTION.field1.string_field as row_string_field, PCOLLECTION.field2[2].string_field as array_string_field from PCOLLECTION"));
+
+ PAssert.that(outputRow)
+ .containsInAnyOrder(
+ Row.nullRow(
+ Schema.builder()
+ .addNullableField("row_string_field", FieldType.STRING)
+ .addNullableField("array_string_field", FieldType.STRING)
+ .build()));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testNullInnerRow() {
+
+ Row nestedInnerRow = Row.withSchema(innerRowSchema).addValues("str", 1000L).build();
+
+ Row innerRow =
+ Row.withSchema(nullableInnerRowSchema)
+ .addValues(null, Arrays.asList(nestedInnerRow))
+ .build();
+
+ Row row =
+ Row.withSchema(nullableNestedRowWithArraySchema).addValues(null, null, innerRow).build();
+
+ PCollection<Row> outputRow =
+ pipeline
+ .apply(Create.of(row))
+ .setRowSchema(nullableNestedRowWithArraySchema)
+ .apply(
+ SqlTransform.query(
+ "select PCOLLECTION.field3.inner_row_field.string_field as string_field, PCOLLECTION.field3.array_field[1].long_field as long_field from PCOLLECTION"));
+
+ PAssert.that(outputRow)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addNullableField("string_field", FieldType.STRING)
+ .addInt64Field("long_field")
+ .build())
+ .addValues(null, 1000L)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private static class DummySqlTimeType implements Schema.LogicalType<Long, Instant> {
+ @Override
+ public String getIdentifier() {
+ return "SqlTimeType";
+ }
+
+ @Override
+ public Schema.FieldType getBaseType() {
+ return Schema.FieldType.DATETIME;
+ }
+
+ @Override
+ public Instant toBaseType(Long input) {
+ return new Instant((long) input);
+ }
+
+ @Override
+ public Long toInputType(Instant base) {
+ return base.getMillis();
+ }
+ }
+
+ private static class DummySqlDateType implements Schema.LogicalType<Long, Instant> {
+ @Override
+ public String getIdentifier() {
+ return "SqlDateType";
+ }
+
+ @Override
+ public Schema.FieldType getBaseType() {
+ return Schema.FieldType.DATETIME;
+ }
+
+ @Override
+ public Instant toBaseType(Long input) {
+ return new Instant((long) input);
+ }
+
+ @Override
+ public Long toInputType(Instant base) {
+ return base.getMillis();
+ }
+ }
+
+ @Test
+ public void testNullDatetimeFields() {
+ Instant current = new Instant(1561671380000L); // Long value corresponds to 27/06/2019
+ DateTime date = new DateTime(3600000);
+
+ Schema dateTimeFieldSchema =
+ Schema.builder()
+ .addField("dateTimeField", FieldType.DATETIME)
+ .addNullableField("nullableDateTimeField", FieldType.DATETIME)
+ .addField("timeTypeField", FieldType.logicalType(new DummySqlTimeType()))
+ .addNullableField(
+ "nullableTimeTypeField", FieldType.logicalType(new DummySqlTimeType()))
+ .addField("dateTypeField", FieldType.logicalType(new DummySqlDateType()))
+ .addNullableField(
+ "nullableDateTypeField", FieldType.logicalType(new DummySqlDateType()))
+ .build();
+
+ Row dateTimeRow =
+ Row.withSchema(dateTimeFieldSchema)
+ .addValues(current, null, date.getMillis(), null, current.getMillis(), null)
+ .build();
+
+ PCollection<Row> outputRow =
+ pipeline
+ .apply(Create.of(dateTimeRow))
+ .setRowSchema(dateTimeFieldSchema)
+ .apply(
+ SqlTransform.query(
+ "select EXTRACT(YEAR from dateTimeField) as yyyy, "
+ + " EXTRACT(YEAR from nullableDateTimeField) as year_with_null, "
+ + " EXTRACT(MONTH from dateTimeField) as mm, "
+ + " EXTRACT(MONTH from nullableDateTimeField) as month_with_null, "
+ + " timeTypeField + interval '1' hour as time_with_hour_added, "
+ + " nullableTimeTypeField + interval '1' hour as hour_added_with_null, "
+ + " timeTypeField - INTERVAL '60' SECOND as time_with_seconds_added, "
+ + " nullableTimeTypeField - INTERVAL '60' SECOND as seconds_added_with_null, "
+ + " EXTRACT(DAY from dateTypeField) as dd, "
+ + " EXTRACT(DAY from nullableDateTypeField) as day_with_null "
+ + " from PCOLLECTION"));
+
+ Schema outputRowSchema =
+ Schema.builder()
+ .addField("yyyy", FieldType.INT64)
+ .addNullableField("year_with_null", FieldType.INT64)
+ .addField("mm", FieldType.INT64)
+ .addNullableField("month_with_null", FieldType.INT64)
+ .addField("time_with_hour_added", FieldType.DATETIME)
+ .addNullableField("hour_added_with_null", FieldType.DATETIME)
+ .addField("time_with_seconds_added", FieldType.DATETIME)
+ .addNullableField("seconds_added_with_null", FieldType.DATETIME)
+ .addField("dd", FieldType.INT64)
+ .addNullableField("day_with_null", FieldType.INT64)
+ .build();
+
+ Instant futureTime = date.toInstant().plus(3600 * 1000);
+ Instant pastTime = date.toInstant().minus(60 * 1000);
+
+ PAssert.that(outputRow)
+ .containsInAnyOrder(
+ Row.withSchema(outputRowSchema)
+ .addValues(2019L, null, 06L, null, futureTime, null, pastTime, null, 27L, null)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testMapWithRowAsValue() {
+
+ Schema inputSchema =
+ Schema.builder()
+ .addMapField("mapWithValueAsRow", FieldType.STRING, FieldType.row(rowWithArraySchema))
+ .build();
+
+ Map<String, Row> mapWithValueAsRow = new HashMap<>();
+ Row complexRow =
+ Row.withSchema(rowWithArraySchema)
+ .addValues("RED", 5L, Arrays.asList(10L, 20L, 30L))
+ .build();
+ mapWithValueAsRow.put("key", complexRow);
+
+ Row rowOfMap = Row.withSchema(inputSchema).addValue(mapWithValueAsRow).build();
+
+ PCollection<Row> outputRow =
+ pipeline
+ .apply(Create.of(rowOfMap))
+ .setRowSchema(inputSchema)
+ .apply(
+ SqlTransform.query(
+ "select PCOLLECTION.mapWithValueAsRow['key'].field1 as color, PCOLLECTION.mapWithValueAsRow['key'].field3[2] as num from PCOLLECTION"));
+
+ Row expectedRow =
+ Row.withSchema(Schema.builder().addStringField("color").addInt64Field("num").build())
+ .addValues("RED", 20L)
+ .build();
+
+ PAssert.that(outputRow).containsInAnyOrder(expectedRow);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(1));
+ }
+
+ @Test
+ public void testMapWithNullRowFields() {
+
+ Schema nullableInnerSchema =
+ Schema.builder()
+ .addNullableField("strField", FieldType.STRING)
+ .addNullableField("arrField", FieldType.array(FieldType.INT64))
+ .build();
+ Schema inputSchema =
+ Schema.builder()
+ .addMapField("mapField", FieldType.STRING, FieldType.row(nullableInnerSchema))
+ .addNullableField(
+ "nullableMapField",
+ FieldType.map(FieldType.STRING, FieldType.row(nullableInnerSchema)))
+ .build();
+
+ Row mapValue = Row.withSchema(nullableInnerSchema).addValues("str", null).build();
+ Map<String, Row> mapWithValueAsRow = new HashMap<>();
+ mapWithValueAsRow.put("key", mapValue);
+
+ Row inputRow = Row.withSchema(inputSchema).addValues(mapWithValueAsRow, null).build();
+
+ PCollection<Row> outputRow =
+ pipeline
+ .apply(Create.of(inputRow))
+ .setRowSchema(inputSchema)
+ .apply(
+ SqlTransform.query(
+ "select PCOLLECTION.mapField['key'].strField as str, PCOLLECTION.mapField['key'].arrField[1] as arr, PCOLLECTION.nullableMapField['key'].arrField[1] as nullableField from PCOLLECTION"));
+
+ Row expectedRow =
+ Row.withSchema(
+ Schema.builder()
+ .addStringField("str")
+ .addNullableField("arr", FieldType.INT64)
+ .addNullableField("nullableField", FieldType.INT64)
+ .build())
+ .addValues("str", null, null)
+ .build();
+ PAssert.that(outputRow).containsInAnyOrder(expectedRow);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(1));
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java
index 12b3a3c..01872a5 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCastTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.joda.time.DateTime;
@@ -46,10 +45,7 @@
PCollection<Row> input =
pipeline.apply(
Create.of(Row.withSchema(INPUT_ROW_SCHEMA).addValues(1).addValue("20181018").build())
- .withSchema(
- INPUT_ROW_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ .withRowSchema(INPUT_ROW_SCHEMA));
Schema resultType =
Schema.builder().addInt32Field("f_int").addNullableField("f_date", DATETIME).build();
@@ -78,10 +74,7 @@
PCollection<Row> input =
pipeline.apply(
Create.of(Row.withSchema(INPUT_ROW_SCHEMA).addValues(1).addValue("20181018").build())
- .withSchema(
- INPUT_ROW_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ .withRowSchema(INPUT_ROW_SCHEMA));
Schema resultType = Schema.builder().addInt32Field("f_int").addDateTimeField("f_date").build();
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
index 3cd8ee6..edb765a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationCovarianceTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.junit.Before;
@@ -59,11 +58,7 @@
2.0, 6, 4, 0, 8.0, 4.0, 1.0, 8, 4, 0)
.getRows();
- boundedInput =
- pipeline.apply(
- Create.of(rowsInTableB)
- .withSchema(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ boundedInput = pipeline.apply(Create.of(rowsInTableB).withRowSchema(schema));
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java
index 91e19db..91925f0 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationNullableTest.java
@@ -18,7 +18,6 @@
package org.apache.beam.sdk.extensions.sql;
import static org.apache.beam.sdk.extensions.sql.utils.RowAsserts.matchesScalar;
-import static org.apache.beam.sdk.transforms.SerializableFunctions.identity;
import static org.junit.Assert.assertEquals;
import java.util.List;
@@ -60,8 +59,7 @@
.addRows(3, 2, 1)
.getRows();
- boundedInput =
- PBegin.in(pipeline).apply(Create.of(rows).withSchema(schema, identity(), identity()));
+ boundedInput = PBegin.in(pipeline).apply(Create.of(rows).withRowSchema(schema));
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
index f395a46..9b9723c 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
@@ -36,7 +36,6 @@
import org.apache.beam.sdk.testing.UsesTestStream;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.windowing.AfterPane;
import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
@@ -104,13 +103,7 @@
.getRows();
boundedInput3 =
- pipeline.apply(
- "boundedInput3",
- Create.of(rowsInTableB)
- .withSchema(
- schemaInTableB,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput3", Create.of(rowsInTableB).withRowSchema(schemaInTableB));
}
/** GROUP-BY with single aggregation function with bounded PCollection. */
@@ -424,8 +417,7 @@
PCollection<Row> input =
pipeline.apply(
- TestStream.create(
- inputSchema, SerializableFunctions.identity(), SerializableFunctions.identity())
+ TestStream.create(inputSchema)
.addElements(
Row.withSchema(inputSchema)
.addValues(1, parseTimestampWithoutTimeZone("2017-01-01 01:01:01"))
@@ -696,7 +688,7 @@
2, 4,
2, 5)
.getRows()))
- .setSchema(schema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ .setRowSchema(schema);
String sql = "SELECT SUM(f_intValue) FROM PCOLLECTION GROUP BY f_intGroupingKey";
@@ -708,6 +700,37 @@
}
@Test
+ public void testSupportsAggregationWithFilterWithoutProjection() throws Exception {
+ pipeline.enableAbandonedNodeEnforcement(false);
+
+ Schema schema =
+ Schema.builder().addInt32Field("f_intGroupingKey").addInt32Field("f_intValue").build();
+
+ PCollection<Row> inputRows =
+ pipeline
+ .apply(
+ Create.of(
+ TestUtils.rowsBuilderOf(schema)
+ .addRows(
+ 0, 1,
+ 0, 2,
+ 1, 3,
+ 2, 4,
+ 2, 5)
+ .getRows()))
+ .setRowSchema(schema);
+
+ String sql =
+ "SELECT SUM(f_intValue) FROM PCOLLECTION WHERE f_intValue < 5 GROUP BY f_intGroupingKey";
+
+ PCollection<Row> result = inputRows.apply("sql", SqlTransform.query(sql));
+
+ PAssert.that(result).containsInAnyOrder(rowsWithSingleIntField("sum", Arrays.asList(3, 3, 4)));
+
+ pipeline.run();
+ }
+
+ @Test
public void testSupportsNonGlobalWindowWithCustomTrigger() {
DateTime startTime = parseTimestampWithoutTimeZone("2017-1-1 0:0:0");
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
index 5f0dc12..808b27a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationVarianceTest.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.junit.Before;
@@ -55,11 +54,7 @@
1, 1.0, 0, 4, 4.0, 0, 7, 7.0, 0, 13, 13.0, 0, 5, 5.0, 0, 10, 10.0, 0, 17, 17.0, 0)
.getRows();
- boundedInput =
- pipeline.apply(
- Create.of(rowsInTableB)
- .withSchema(
- schema, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ boundedInput = pipeline.apply(Create.of(rowsInTableB).withRowSchema(schema));
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
index eb90274..a9da87a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslArrayTest.java
@@ -22,7 +22,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
@@ -113,13 +112,7 @@
Row inputRow = Row.withSchema(INPUT_SCHEMA).addValues(1).addArray(Arrays.asList("111")).build();
PCollection<Row> input =
- pipeline.apply(
- "boundedInput1",
- Create.of(inputRow)
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput1", Create.of(inputRow).withRowSchema(INPUT_SCHEMA));
Schema resultType = Schema.builder().addStringField("f_arrElem").build();
@@ -154,11 +147,7 @@
PCollection<Row> input =
pipeline.apply(
"boundedInput1",
- Create.empty(TypeDescriptor.of(Row.class))
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.empty(TypeDescriptor.of(Row.class)).withRowSchema(INPUT_SCHEMA));
// Because we have a multi-part FROM the DSL considers it multi-input
TupleTag<Row> mainTag = new TupleTag<Row>("main") {};
@@ -184,11 +173,7 @@
PCollection<Row> input =
pipeline.apply(
"boundedInput1",
- Create.empty(TypeDescriptor.of(Row.class))
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.empty(TypeDescriptor.of(Row.class)).withRowSchema(INPUT_SCHEMA));
// Because we have a multi-part FROM the DSL considers it multi-input
TupleTag<Row> mainTag = new TupleTag<Row>("main") {};
@@ -222,13 +207,7 @@
Row.withSchema(INPUT_SCHEMA).addValues(13).addArray(Arrays.asList("444", "555")).build();
PCollection<Row> input =
- pipeline.apply(
- "boundedInput1",
- Create.of(row1, row2)
- .withSchema(
- INPUT_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput1", Create.of(row1, row2).withRowSchema(INPUT_SCHEMA));
// Because we have a multi-part FROM the DSL considers it multi-input
TupleTag<Row> mainTag = new TupleTag<Row>("main") {};
@@ -287,8 +266,7 @@
Row.withSchema(elementSchema).addValues("CC", 33).build(),
Row.withSchema(elementSchema).addValues("DD", 44).build()))
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -343,8 +321,7 @@
Row.withSchema(elementSchema).addValues("CC", 33).build(),
Row.withSchema(elementSchema).addValues("DD", 44).build()))
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -389,8 +366,7 @@
Row.withSchema(elementSchema).addValues("CC", 33).build(),
Row.withSchema(elementSchema).addValues("DD", 44).build()))
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -417,7 +393,6 @@
.addValues(2)
.addArray(Arrays.asList("33", "44", "55"))
.build())
- .withSchema(
- INPUT_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(INPUT_SCHEMA));
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java
index 8ad2b92..ad26d4a 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java
@@ -29,7 +29,6 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.PBegin;
@@ -256,66 +255,34 @@
@Before
public void preparePCollections() {
boundedInput1 =
- pipeline.apply(
- "boundedInput1",
- Create.of(rowsInTableA)
- .withSchema(
- schemaInTableA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInput1", Create.of(rowsInTableA).withRowSchema(schemaInTableA));
boundedInput2 =
pipeline.apply(
- "boundedInput2",
- Create.of(rowsInTableA.get(0))
- .withSchema(
- schemaInTableA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ "boundedInput2", Create.of(rowsInTableA.get(0)).withRowSchema(schemaInTableA));
boundedInputFloatDouble =
pipeline.apply(
"boundedInputFloatDouble",
- Create.of(rowsOfFloatDouble)
- .withSchema(
- schemaFloatDouble,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.of(rowsOfFloatDouble).withRowSchema(schemaFloatDouble));
boundedInputBytes =
- pipeline.apply(
- "boundedInputBytes",
- Create.of(rowsOfBytes)
- .withSchema(
- schemaBytes,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ pipeline.apply("boundedInputBytes", Create.of(rowsOfBytes).withRowSchema(schemaBytes));
boundedInputBytesPaddingTest =
pipeline.apply(
"boundedInputBytesPaddingTest",
- Create.of(rowsOfBytesPaddingTest)
- .withSchema(
- schemaBytesPaddingTest,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ Create.of(rowsOfBytesPaddingTest).withRowSchema(schemaBytesPaddingTest));
boundedInputMonthly =
pipeline.apply(
- "boundedInputMonthly",
- Create.of(monthlyRowsInTableA)
- .withSchema(
- schemaInTableA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ "boundedInputMonthly", Create.of(monthlyRowsInTableA).withRowSchema(schemaInTableA));
unboundedInput1 = prepareUnboundedPCollection1();
unboundedInput2 = prepareUnboundedPCollection2();
}
private PCollection<Row> prepareUnboundedPCollection1() {
- TestStream.Builder<Row> values =
- TestStream.create(
- schemaInTableA, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(schemaInTableA);
for (Row row : rowsInTableA) {
values = values.advanceWatermarkTo(new Instant(row.getDateTime("f_timestamp")));
@@ -330,9 +297,7 @@
}
private PCollection<Row> prepareUnboundedPCollection2() {
- TestStream.Builder<Row> values =
- TestStream.create(
- schemaInTableA, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(schemaInTableA);
Row row = rowsInTableA.get(0);
values = values.advanceWatermarkTo(new Instant(row.getDateTime("f_timestamp")));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslFilterTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslFilterTest.java
index 057140b..c00f3c9 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslFilterTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslFilterTest.java
@@ -21,7 +21,9 @@
import static org.junit.internal.matchers.ThrowableMessageMatcher.hasMessage;
import org.apache.beam.sdk.extensions.sql.impl.ParseException;
+import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
@@ -128,4 +130,24 @@
pipeline.run().waitUntilFinish();
}
+
+ @Test
+ public void testFilterBytes() {
+ String sql = "SELECT c_bytes FROM PCOLLECTION WHERE c_bytes = x'ff'";
+
+ Schema schema = Schema.builder().addByteArrayField("c_bytes").build();
+ PCollection<Row> input =
+ pipeline.apply(
+ Create.of(
+ Row.withSchema(schema).addValue(new byte[] {-1}).build(),
+ Row.withSchema(schema).addValue(new byte[] {127}).build())
+ .withRowSchema(schema));
+
+ PCollection<Row> result = input.apply(SqlTransform.query(sql));
+
+ PAssert.that(result)
+ .containsInAnyOrder(Row.withSchema(schema).addValue(new byte[] {-1}).build());
+
+ pipeline.run().waitUntilFinish();
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslJoinTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslJoinTest.java
index d643391..b210bb0 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslJoinTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslJoinTest.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.extensions.sql;
import static org.apache.beam.sdk.extensions.sql.TestUtils.tuple;
-import static org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRelBoundedVsBoundedTest.ORDER_DETAILS1;
-import static org.apache.beam.sdk.extensions.sql.impl.rel.BeamJoinRelBoundedVsBoundedTest.ORDER_DETAILS2;
+import static org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRelBoundedVsBoundedTest.ORDER_DETAILS1;
+import static org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRelBoundedVsBoundedTest.ORDER_DETAILS2;
import static org.apache.beam.sdk.extensions.sql.utils.DateTimeUtils.parseTimestampWithoutTimeZone;
import static org.hamcrest.Matchers.stringContainsInOrder;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java
index 2f1f3c0..db6cc14 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslNestedRowsTest.java
@@ -22,7 +22,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.junit.Rule;
@@ -62,8 +61,7 @@
.addValues(
1, Row.withSchema(nestedSchema).addValues(312, "CC", 313).build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -106,8 +104,7 @@
.addValues(
1, Row.withSchema(nestedSchema).addValues(312, "CC", 313).build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -148,8 +145,7 @@
.addValues(
2, Row.withSchema(nestedSchema).addValues(412, "DD", 413).build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -200,8 +196,7 @@
.addValues(412, "DD", 413, Arrays.asList("three", "four"))
.build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
@@ -251,8 +246,7 @@
.addValues(412, "DD", 413, Arrays.asList("three", "four"))
.build())
.build())
- .withSchema(
- inputType, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ .withRowSchema(inputType));
PCollection<Row> result =
input
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslProjectTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslProjectTest.java
index 068aeee..998013f 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslProjectTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslProjectTest.java
@@ -217,4 +217,18 @@
pipeline.run();
}
+
+ @Test
+ public void testBytesLiteral() {
+ Schema outputSchema = Schema.of(Schema.Field.of("c_bytes", Schema.FieldType.BYTES));
+
+ PCollection<Row> result =
+ PCollectionTuple.empty(pipeline).apply(SqlTransform.query("SELECT x'baadcafe' as c_bytes"));
+
+ PAssert.that(result)
+ .containsInAnyOrder(
+ Row.withSchema(outputSchema).addValue(new byte[] {-70, -83, -54, -2}).build());
+
+ pipeline.run();
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
index dc7c111..28ac9f4 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslSqlStdOperatorsTest.java
@@ -43,14 +43,14 @@
import org.apache.beam.sdk.extensions.sql.integrationtest.BeamSqlBuiltinFunctionsIntegrationTestBase;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
-import org.apache.calcite.runtime.SqlFunctions;
-import org.apache.calcite.sql.SqlKind;
-import org.apache.calcite.sql.SqlOperator;
-import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
@@ -58,7 +58,7 @@
/**
* DSL compliance tests for the row-level operators of {@link
- * org.apache.calcite.sql.fun.SqlStdOperatorTable}.
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable}.
*/
public class BeamSqlDslSqlStdOperatorsTest extends BeamSqlBuiltinFunctionsIntegrationTestBase {
private static final BigDecimal ZERO = BigDecimal.valueOf(0.0);
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
index a4c65da..75e8a08 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
@@ -37,9 +37,9 @@
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.linq4j.function.Parameter;
-import org.apache.calcite.schema.TranslatableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Parameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
import org.joda.time.Instant;
import org.junit.Test;
@@ -174,7 +174,9 @@
pipeline.run().waitUntilFinish();
}
- /** test {@link org.apache.calcite.schema.TableMacro} UDF. */
+ /**
+ * test {@link org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TableMacro} UDF.
+ */
@Test
public void testTableMacroUdf() throws Exception {
String sql1 = "SELECT * FROM table(range_udf(0, 3))";
@@ -345,7 +347,10 @@
}
}
- /** UDF to test support for {@link org.apache.calcite.schema.TableMacro}. */
+ /**
+ * UDF to test support for {@link
+ * org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TableMacro}.
+ */
public static final class RangeUdf implements BeamSqlUdf {
public static TranslatableTable eval(int startInclusive, int endExclusive) {
Schema schema = Schema.of(Schema.Field.of("f0", Schema.FieldType.INT32));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
index 2a476ce..3f0d2f2 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
@@ -21,9 +21,9 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.store.InMemoryMetaStore;
-import org.apache.calcite.sql.parser.SqlParseException;
-import org.apache.calcite.tools.RelConversionException;
-import org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException;
import org.junit.Before;
import org.junit.Ignore;
@@ -68,7 +68,7 @@
assertEquals(
"BeamCalcRel(expr#0..3=[{inputs}], c1=[$t0], c2=[$t3])\n"
- + " BeamJoinRel(condition=[=($0, $3)], joinType=[inner])\n"
+ + " BeamCoGBKJoinRel(condition=[=($0, $3)], joinType=[inner])\n"
+ " BeamCalcRel(expr#0..1=[{inputs}], expr#2=[0], expr#3=[>($t0, $t2)],"
+ " proj#0..1=[{exprs}], $condition=[$t3])\n"
+ " BeamIOSourceRel(table=[[beam, A]])\n"
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java
index da24f0c..e175530 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMapTest.java
@@ -21,10 +21,9 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -145,9 +144,6 @@
.addValues(2)
.addValue(ImmutableMap.of("key33", 33, "key44", 44, "key55", 55))
.build())
- .withSchema(
- INPUT_ROW_TYPE,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()));
+ .withRowSchema(INPUT_ROW_TYPE));
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java
index 7107c0b..41f916b 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlMultipleSchemasTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteCannotParseSimpleIdentifiersTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteCannotParseSimpleIdentifiersTest.java
new file mode 100644
index 0000000..c696c28
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteCannotParseSimpleIdentifiersTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql;
+
+import static org.junit.Assert.assertThrows;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.ParseException;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.function.ThrowingRunnable;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+/**
+ * Examples of simple identifiers that Calcite is unable to parse.
+ *
+ * <p>Not an exhaustive list.
+ */
+@RunWith(Parameterized.class)
+public class CalciteCannotParseSimpleIdentifiersTest implements Serializable {
+
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+ private final String input;
+
+ @Parameters(name = "{0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(
+ new Object[] {
+ "field id",
+ "field\nid",
+ "`field\nid`",
+ "field`id",
+ "field\\id",
+ "field``id",
+ "field\bid",
+ "field=id",
+ "field+id",
+ "field{id}",
+ "field.id",
+ "field\r_id",
+ "`field\r_id`"
+ });
+ }
+
+ public CalciteCannotParseSimpleIdentifiersTest(String input) {
+ this.input = input;
+ }
+
+ @Test
+ public void testFailsToParseAlias() {
+ assertThrows(ParseException.class, attemptParse(input));
+ }
+
+ private ThrowingRunnable attemptParse(String alias) {
+ return () -> BeamSqlEnv.inMemory().isDdl(String.format("SELECT 321 AS %s", alias));
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteParsesSimpleIdentifiersTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteParsesSimpleIdentifiersTest.java
new file mode 100644
index 0000000..2532c02
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteParsesSimpleIdentifiersTest.java
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+
+import java.io.Serializable;
+import java.util.Arrays;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.hamcrest.BaseMatcher;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+/**
+ * Examples of simple identifiers that Calcite is able to parse.
+ *
+ * <p>Not an exhaustive list.
+ */
+@RunWith(Parameterized.class)
+public class CalciteParsesSimpleIdentifiersTest implements Serializable {
+
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+ private final String input;
+ private final String expected;
+
+ @Parameters(name = "{0}")
+ public static Iterable<Object[]> data() {
+ return Arrays.asList(
+ new Object[][] {
+ // --------------------------------
+ // user input | parsed as |
+ // --------------------------------
+ {"field_id", "field_id"},
+ {"`field_id`", "field_id"},
+ {"`field``id`", "field`id"},
+ {"`field id`", "field id"},
+ {"`field-id`", "field-id"},
+ {"`field=id`", "field=id"},
+ {"`field.id`", "field.id"},
+ {"`field{id}`", "field{id}"},
+ {"`field|id`", "field|id"},
+ {"`field\\id`", "field\\id"},
+ {"`field\\a_id`", "field\\a_id"},
+ {"`field\b_id`", "field\b_id"},
+ {"`field\\b_id`", "field\\b_id"},
+ {"`field\\f_id`", "field\\f_id"},
+ {"`field\\n_id`", "field\\n_id"},
+ {"`field\\r_id`", "field\\r_id"},
+ {"`field\tid`", "field\tid"},
+ {"`field\\t_id`", "field\\t_id"},
+ {"`field\\v_id`", "field\\v_id"},
+ {"`field\\\\_id`", "field\\\\_id"},
+ {"`field\\?_id`", "field\\?_id"}
+ });
+ }
+
+ public CalciteParsesSimpleIdentifiersTest(String input, String expected) {
+ this.input = input;
+ this.expected = expected;
+ }
+
+ @Test
+ public void testParsesAlias() {
+ assertThat(alias(input), parsedAs(expected));
+ }
+
+ /** PCollection with a single row with a single field with the specified alias. */
+ private PCollection<Row> alias(String alias) {
+ return pipeline.apply(SqlTransform.query(String.format("SELECT 321 AS %s", alias)));
+ }
+
+ /**
+ * Asserts that the specified field alias is parsed as expected.
+ *
+ * <p>SQL parser un-escapes the qouted identifiers, for example.
+ */
+ private Matcher<PCollection<Row>> parsedAs(String expected) {
+ return new BaseMatcher<PCollection<Row>>() {
+ @Override
+ public boolean matches(Object actual) {
+ PCollection<Row> result = (PCollection<Row>) actual;
+ PAssert.thatSingleton(result).satisfies(assertFieldNameIs(expected));
+ pipeline.run();
+ return true;
+ }
+
+ @Override
+ public void describeTo(Description description) {
+ description.appendText("field alias matches");
+ }
+ };
+ }
+
+ /** Assert that field name of the only field matches the expected value. */
+ private SerializableFunction<Row, Void> assertFieldNameIs(String expected) {
+ return row -> {
+ assertEquals(expected, onlyField(row).getName());
+ return null;
+ };
+ }
+
+ /** Returns the only field in the row. */
+ private Schema.Field onlyField(Row row) {
+ assertEquals(1, row.getFieldCount());
+ return row.getSchema().getField(0);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java
index 1903348..dc73b20 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/PubsubToBigqueryIT.java
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java
index 36c7051..3ab0ddd 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/TestUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import java.util.ArrayList;
import java.util.Arrays;
@@ -27,7 +27,6 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
@@ -104,6 +103,14 @@
return builder;
}
+ public static RowsBuilder ofNullable(final Object... args) {
+ Schema beamSQLSchema = TestTableUtils.buildBeamSqlNullableSchema(args);
+ RowsBuilder builder = new RowsBuilder();
+ builder.type = beamSQLSchema;
+
+ return builder;
+ }
+
/**
* Create a RowsBuilder with the specified row type info.
*
@@ -201,9 +208,7 @@
type = rows.get(0).getSchema();
}
- TestStream.Builder<Row> values =
- TestStream.create(
- type, SerializableFunctions.identity(), SerializableFunctions.identity());
+ TestStream.Builder<Row> values = TestStream.create(type);
for (Row row : rows) {
if (timestampField != null) {
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnvTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnvTest.java
index 517309d..3b6dda0 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnvTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnvTest.java
@@ -24,6 +24,7 @@
import java.sql.Connection;
import java.sql.ResultSet;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -42,6 +43,7 @@
BeamSqlEnv.builder(root)
.addSchema("nested", nested)
.addSchema("anotherOne", anotherOne)
+ .setPipelineOptions(PipelineOptionsFactory.create())
.build();
Connection connection = env.connection;
@@ -60,6 +62,9 @@
exceptions.expectCause(hasMessage(containsString("org.test.ClassNotFound")));
TestTableProvider root = new TestTableProvider();
- BeamSqlEnv.builder(root).setQueryPlannerClassName("org.test.ClassNotFound").build();
+ BeamSqlEnv.builder(root)
+ .setQueryPlannerClassName("org.test.ClassNotFound")
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .build();
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java
index 6f36173..5d01661 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JdbcDriverTest.java
@@ -46,13 +46,14 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.util.ReleaseInfo;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.jdbc.CalciteSchema;
-import org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.joda.time.ReadableInstant;
@@ -197,7 +198,7 @@
@Test
public void testSelectsFromExistingTable() throws Exception {
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
connection
.createStatement()
@@ -219,7 +220,7 @@
@Test
public void testTimestampWithDefaultTimezone() throws Exception {
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
// A table with one TIMESTAMP column
Schema schema = Schema.builder().addDateTimeField("ts").build();
@@ -250,7 +251,7 @@
public void testTimestampWithNonzeroTimezone() throws Exception {
Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("Asia/Tokyo"), Locale.ROOT);
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
// A table with one TIMESTAMP column
Schema schema = Schema.builder().addDateTimeField("ts").build();
@@ -280,7 +281,7 @@
public void testTimestampWithZeroTimezone() throws Exception {
Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC"), Locale.ROOT);
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
// A table with one TIMESTAMP column
Schema schema = Schema.builder().addDateTimeField("ts").build();
@@ -309,7 +310,7 @@
@Test
public void testSelectsFromExistingComplexTable() throws Exception {
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
connection
.createStatement()
@@ -343,7 +344,7 @@
@Test
public void testInsertIntoCreatedTable() throws Exception {
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
connection
.createStatement()
@@ -369,7 +370,8 @@
@Test
public void testInternalConnect_boundedTable() throws Exception {
- CalciteConnection connection = JdbcDriver.connect(BOUNDED_TABLE);
+ CalciteConnection connection =
+ JdbcDriver.connect(BOUNDED_TABLE, PipelineOptionsFactory.create());
Statement statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery("SELECT * FROM test");
assertTrue(resultSet.next());
@@ -392,7 +394,8 @@
.addRows(1, "second first")
.addRows(2, "second")));
- CalciteConnection connection = JdbcDriver.connect(tableProvider);
+ CalciteConnection connection =
+ JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
Statement statement = connection.createStatement();
ResultSet resultSet1 = statement.executeQuery("SELECT * FROM test LIMIT 5");
assertTrue(resultSet1.next());
@@ -432,7 +435,8 @@
.timestampColumnIndex(3)
.addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)));
- CalciteConnection connection = JdbcDriver.connect(tableProvider);
+ CalciteConnection connection =
+ JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
Statement statement = connection.createStatement();
ResultSet resultSet1 = statement.executeQuery("SELECT * FROM test LIMIT 1");
@@ -470,7 +474,8 @@
@Test
public void testInternalConnect_setDirectRunner() throws Exception {
- CalciteConnection connection = JdbcDriver.connect(BOUNDED_TABLE);
+ CalciteConnection connection =
+ JdbcDriver.connect(BOUNDED_TABLE, PipelineOptionsFactory.create());
Statement statement = connection.createStatement();
assertEquals(0, statement.executeUpdate("SET runner = direct"));
assertTrue(statement.execute("SELECT * FROM test"));
@@ -480,7 +485,8 @@
public void testInternalConnect_setBogusRunner() throws Exception {
thrown.expectMessage("Unknown 'runner' specified 'bogus'");
- CalciteConnection connection = JdbcDriver.connect(BOUNDED_TABLE);
+ CalciteConnection connection =
+ JdbcDriver.connect(BOUNDED_TABLE, PipelineOptionsFactory.create());
Statement statement = connection.createStatement();
assertEquals(0, statement.executeUpdate("SET runner = bogus"));
assertTrue(statement.execute("SELECT * FROM test"));
@@ -488,7 +494,8 @@
@Test
public void testInternalConnect_resetAll() throws Exception {
- CalciteConnection connection = JdbcDriver.connect(BOUNDED_TABLE);
+ CalciteConnection connection =
+ JdbcDriver.connect(BOUNDED_TABLE, PipelineOptionsFactory.create());
Statement statement = connection.createStatement();
assertEquals(0, statement.executeUpdate("SET runner = bogus"));
assertEquals(0, statement.executeUpdate("RESET ALL"));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java
index 39b5f0d..b472bba 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLNestedTypesTest.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.extensions.sql.utils.QuickCheckGenerators.PrimitiveTypes;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
import org.junit.runner.RunWith;
/**
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
index b7f4215..f6db1ce 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/parser/BeamDDLTest.java
@@ -31,6 +31,7 @@
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.schemas.Schema;
import org.junit.Test;
@@ -62,6 +63,21 @@
tableProvider.getTables().get("person"));
}
+ @Test
+ public void testParseCreateExternalTable_WithComplexFields() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ env.executeDdl(
+ "CREATE EXTERNAL TABLE PersonDetails"
+ + " ( personInfo MAP<VARCHAR, ROW<field_1 INTEGER,field_2 VARCHAR>> , "
+ + " additionalInfo ROW<field_0 TIMESTAMP,field_1 INTEGER,field_2 TINYINT> )"
+ + " TYPE 'text'"
+ + " LOCATION '/home/admin/person'");
+
+ assertNotNull(tableProvider.getTables().get("PersonDetails"));
+ }
+
@Test(expected = ParseException.class)
public void testParseCreateExternalTable_withoutType() throws Exception {
BeamSqlEnv env = BeamSqlEnv.withTableProvider(new TestTableProvider());
@@ -167,7 +183,11 @@
TestTableProvider rootProvider = new TestTableProvider();
TestTableProvider testProvider = new TestTableProvider();
- BeamSqlEnv env = BeamSqlEnv.builder(rootProvider).addSchema("test", testProvider).build();
+ BeamSqlEnv env =
+ BeamSqlEnv.builder(rootProvider)
+ .addSchema("test", testProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .build();
assertNull(testProvider.getTables().get("person"));
env.executeDdl("CREATE EXTERNAL TABLE test.person (id INT) TYPE text");
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModelTest.java
new file mode 100644
index 0000000..26f83ff
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamCostModelTest.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+/** Tests the behavior of BeamCostModel. */
+public class BeamCostModelTest {
+
+ @Test
+ public void testDefaultConstructorIsInfinite() {
+ BeamCostModel cost = BeamCostModel.FACTORY.makeCost(1, 1, 1);
+ Assert.assertTrue(cost.isInfinite());
+ }
+
+ @Test
+ public void testOneInfiniteValue() {
+ BeamCostModel cost = BeamCostModel.FACTORY.makeCost(Double.POSITIVE_INFINITY, 1);
+ Assert.assertTrue(cost.isInfinite());
+ }
+
+ @Test
+ public void testComparisonOfBoundedCost() {
+ BeamCostModel cost1 = BeamCostModel.FACTORY.makeCost(10, 0);
+ BeamCostModel cost2 = BeamCostModel.FACTORY.makeCost(1, 0);
+ BeamCostModel cost3 = BeamCostModel.FACTORY.makeCost(10, 0);
+ Assert.assertTrue(cost2.isLt(cost1));
+ Assert.assertFalse(cost1.isLt(cost2));
+
+ Assert.assertFalse(cost1.isLt(cost3));
+ Assert.assertFalse(cost3.isLt(cost1));
+ Assert.assertTrue(cost3.isLe(cost1));
+ }
+
+ @Test
+ public void testComparisonOfUnboundedCost() {
+ BeamCostModel cost1 = BeamCostModel.FACTORY.makeCost(0, 10);
+ BeamCostModel cost2 = BeamCostModel.FACTORY.makeCost(0, 1);
+ BeamCostModel cost3 = BeamCostModel.FACTORY.makeCost(0, 10);
+ Assert.assertTrue(cost2.isLt(cost1));
+ Assert.assertFalse(cost1.isLt(cost2));
+
+ Assert.assertTrue(cost1.equals(cost3));
+ Assert.assertFalse(cost1.isLt(cost3));
+ Assert.assertFalse(cost3.isLt(cost1));
+ Assert.assertTrue(cost3.isLe(cost1));
+ }
+
+ @Test
+ public void testEffectOfRateVsRowCount() {
+ BeamCostModel boundedCost = BeamCostModel.FACTORY.makeCost(10, 0);
+ BeamCostModel unboundedCost = BeamCostModel.FACTORY.makeCost(0, 10);
+
+ Assert.assertTrue(boundedCost.isLt(unboundedCost));
+ Assert.assertTrue(boundedCost.isLe(unboundedCost));
+ Assert.assertFalse(unboundedCost.isLe(boundedCost));
+ }
+
+ @Test
+ public void testComparisonInfiniteVsInfinite() {
+ BeamCostModel inf1 = BeamCostModel.FACTORY.makeCost(Double.POSITIVE_INFINITY, 0);
+ BeamCostModel inf2 = BeamCostModel.FACTORY.makeInfiniteCost();
+
+ Assert.assertTrue(inf1.isLe(inf2));
+ Assert.assertTrue(inf2.isLe(inf1));
+ Assert.assertFalse(inf1.isLt(inf2));
+ Assert.assertFalse(inf2.isLt(inf1));
+ }
+
+ @Test
+ public void testComparisonInfiniteVsHuge() {
+ BeamCostModel inf = BeamCostModel.FACTORY.makeCost(Double.POSITIVE_INFINITY, 0);
+ BeamCostModel huge = BeamCostModel.FACTORY.makeHugeCost();
+
+ Assert.assertTrue(huge.isLe(inf));
+ Assert.assertTrue(huge.isLt(inf));
+ Assert.assertFalse(inf.isLt(huge));
+ Assert.assertFalse(inf.isLt(huge));
+ }
+
+ @Test
+ public void testHugePlusHugeIsInfinite() {
+ BeamCostModel cost =
+ BeamCostModel.FACTORY.makeHugeCost().plus(BeamCostModel.FACTORY.makeHugeCost());
+ Assert.assertTrue(cost.isInfinite());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/CalciteQueryPlannerTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/CalciteQueryPlannerTest.java
new file mode 100644
index 0000000..fdf1829
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/CalciteQueryPlannerTest.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BaseRelTest;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Tests the behavior of {@code CalciteQueryPlanner}. Note that this is not for the JDBC path. It
+ * will be the behavior of SQLTransform path.
+ */
+public class CalciteQueryPlannerTest extends BaseRelTest {
+ @Before
+ public void prepare() {
+ registerTable(
+ "medium_table",
+ TestBoundedTable.of(
+ Schema.FieldType.INT32, "unbounded_key",
+ Schema.FieldType.INT32, "large_key",
+ Schema.FieldType.INT32, "id")
+ .addRows(1, 1, 1, 1, 1, 2, 1, 1, 3, 1, 1, 4, 1, 1, 5));
+ }
+
+ @Test
+ public void testclusterCostHandlerUsesBeamCost() {
+ String sql = "select * from medium_table";
+ BeamRelNode root = env.parseQuery(sql);
+ Assert.assertTrue(
+ root.getCluster().getPlanner().getCost(root, root.getCluster().getMetadataQuery())
+ instanceof BeamCostModel);
+ }
+
+ @Test
+ public void testNonCumulativeCostMetadataHandler() {
+ String sql = "select * from medium_table";
+ BeamRelNode root = env.parseQuery(sql);
+ Assert.assertTrue(
+ root.getCluster().getMetadataQuery().getNonCumulativeCost(root) instanceof BeamCostModel);
+ Assert.assertFalse(
+ root.getCluster().getMetadataQuery().getNonCumulativeCost(root).isInfinite());
+ }
+
+ @Test
+ public void testCumulativeCostMetaDataHandler() {
+ // This handler is not our handler. It tests if the cumulative handler of Calcite works as
+ // expected.
+ String sql = "select * from medium_table";
+ BeamRelNode root = env.parseQuery(sql);
+ Assert.assertTrue(
+ root.getCluster().getMetadataQuery().getCumulativeCost(root) instanceof BeamCostModel);
+ Assert.assertFalse(root.getCluster().getMetadataQuery().getCumulativeCost(root).isInfinite());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java
new file mode 100644
index 0000000..9e442c5
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/planner/NodeStatsTest.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.planner;
+
+import org.apache.beam.sdk.extensions.sql.impl.rel.BaseRelTest;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.RelSubset;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.SingleRel;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** This tests the NodeStats Metadata handler and the estimations. */
+public class NodeStatsTest extends BaseRelTest {
+ static class UnknownRel extends SingleRel {
+ protected UnknownRel(RelOptCluster cluster, RelTraitSet traits, RelNode input) {
+ super(cluster, traits, input);
+ }
+ }
+
+ public static final TestBoundedTable ORDER_DETAILS1 =
+ TestBoundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price")
+ .addRows(1, 2, 3, 2, 3, 3, 3, 4, 5);
+
+ public static final TestBoundedTable ORDER_DETAILS2 =
+ TestBoundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price")
+ .addRows(1, 2, 3, 2, 3, 3, 3, 4, 5);
+
+ @BeforeClass
+ public static void prepare() {
+ registerTable("ORDER_DETAILS1", ORDER_DETAILS1);
+ registerTable("ORDER_DETAILS2", ORDER_DETAILS2);
+ }
+
+ @Test
+ public void testUnknownRel() {
+ String sql = " select * from ORDER_DETAILS1 ";
+ RelNode root = env.parseQuery(sql);
+ RelNode unknown = new UnknownRel(root.getCluster(), null, null);
+ NodeStats nodeStats =
+ unknown
+ .metadata(NodeStatsMetadata.class, unknown.getCluster().getMetadataQuery())
+ .getNodeStats();
+ Assert.assertTrue(nodeStats.isUnknown());
+ }
+
+ @Test
+ public void testKnownRel() {
+ String sql = " select * from ORDER_DETAILS1 ";
+ RelNode root = env.parseQuery(sql);
+ NodeStats nodeStats =
+ root.metadata(NodeStatsMetadata.class, root.getCluster().getMetadataQuery()).getNodeStats();
+ Assert.assertFalse(nodeStats.isUnknown());
+ }
+
+ @Test
+ public void testSubsetHavingBest() {
+ String sql = " select * from ORDER_DETAILS1 ";
+ RelNode root = env.parseQuery(sql);
+ root = root.getCluster().getPlanner().getRoot();
+
+ // tests if we are actually testing what we want.
+ Assert.assertTrue(root instanceof RelSubset);
+
+ NodeStats estimates = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+ Assert.assertFalse(estimates.isUnknown());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
index fafe10c..5ba74e8 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BaseRelTest.java
@@ -20,15 +20,15 @@
import java.util.HashMap;
import java.util.Map;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
/** Base class for rel test. */
public abstract class BaseRelTest {
private static Map<String, BeamSqlTable> tables = new HashMap<>();
- private static BeamSqlEnv env = BeamSqlEnv.readOnly("test", tables);
+ protected static BeamSqlEnv env = BeamSqlEnv.readOnly("test", tables);
protected static PCollection<Row> compilePipeline(String sql, Pipeline pipeline) {
return BeamSqlRelUtils.toPCollection(pipeline, env.parseQuery(sql));
@@ -37,4 +37,8 @@
protected static void registerTable(String tableName, BeamSqlTable table) {
tables.put(tableName, table);
}
+
+ protected static BeamSqlTable getTable(String tableName) {
+ return tables.get(tableName);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java
new file mode 100644
index 0000000..3ebe01e
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamAggregationRelTest.java
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.math.BigDecimal;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Tests related to {@code BeamAggregationRel}. */
+public class BeamAggregationRelTest extends BaseRelTest {
+ private static final DateTime FIRST_DATE = new DateTime(1);
+ private static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
+
+ private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
+ @BeforeClass
+ public static void prepare() {
+ registerTable(
+ "ORDER_DETAILS_BOUNDED",
+ TestBoundedTable.of(
+ Schema.FieldType.INT64, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.DECIMAL, "price")
+ .addRows(
+ 1L,
+ 1,
+ new BigDecimal(1.0),
+ 1L,
+ 1,
+ new BigDecimal(1.0),
+ 2L,
+ 2,
+ new BigDecimal(2.0),
+ 4L,
+ 4,
+ new BigDecimal(4.0),
+ 4L,
+ 4,
+ new BigDecimal(4.0)));
+
+ registerTable(
+ "ORDER_DETAILS_UNBOUNDED",
+ TestUnboundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price",
+ Schema.FieldType.DATETIME, "order_time")
+ .timestampColumnIndex(3)
+ .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(Duration.standardMinutes(1)),
+ 2,
+ 2,
+ 7,
+ SECOND_DATE,
+ 2,
+ 3,
+ 8,
+ SECOND_DATE,
+ // this late record is omitted(First window)
+ 1,
+ 3,
+ 3,
+ FIRST_DATE)
+ .addRows(
+ // this late record is omitted(Second window)
+ WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardMinutes(1)),
+ 2,
+ 3,
+ 3,
+ SECOND_DATE)
+ .setStatistics(BeamTableStatistics.createUnboundedTableStatistics(2d)));
+ }
+
+ private NodeStats getEstimateOf(String sql) {
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamAggregationRel)) {
+ root = root.getInput(0);
+ }
+
+ return BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+ }
+
+ @Test
+ public void testNodeStats() {
+ String sql = "SELECT order_id FROM ORDER_DETAILS_BOUNDED " + " GROUP BY order_id ";
+
+ NodeStats estimate = getEstimateOf(sql);
+
+ Assert.assertEquals(5d / 2, estimate.getRowCount(), 0.001);
+ Assert.assertEquals(5d / 2, estimate.getWindow(), 0.001);
+ Assert.assertEquals(0., estimate.getRate(), 0.001);
+ }
+
+ @Test
+ public void testNodeStatsEffectOfGroupSet() {
+ String sql1 = "SELECT order_id FROM ORDER_DETAILS_BOUNDED " + " GROUP BY order_id ";
+ String sql2 =
+ "SELECT order_id, site_id FROM ORDER_DETAILS_BOUNDED " + " GROUP BY order_id, site_id ";
+
+ NodeStats estimate1 = getEstimateOf(sql1);
+
+ NodeStats estimate2 = getEstimateOf(sql2);
+
+ Assert.assertTrue(estimate1.getRowCount() < estimate2.getRowCount());
+ Assert.assertTrue(estimate1.getWindow() < estimate2.getWindow());
+ }
+
+ @Test
+ public void testNodeStatsUnboundedWindow() {
+ String sql =
+ "select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS_UNBOUNDED "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)";
+ NodeStats estimate1 = getEstimateOf(sql);
+ Assert.assertEquals(1d, estimate1.getRate(), 0.01);
+ Assert.assertEquals(BeamIOSourceRel.CONSTANT_WINDOW_SIZE / 2, estimate1.getWindow(), 0.01);
+ }
+
+ @Test
+ public void testNodeStatsSlidingWindow() {
+ String sql =
+ "select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS_UNBOUNDED "
+ + " GROUP BY order_id, HOP(order_time, INTERVAL '1' SECOND,INTERVAL '3' SECOND)";
+ NodeStats estimate1 = getEstimateOf(sql);
+ Assert.assertEquals(3d, estimate1.getRate(), 0.01);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java
new file mode 100644
index 0000000..8b1c2dc
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRelTest.java
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.math.BigDecimal;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Tests related to {@code BeamCalcRel}. */
+public class BeamCalcRelTest extends BaseRelTest {
+ private static final DateTime FIRST_DATE = new DateTime(1);
+ private static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
+
+ private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
+ @BeforeClass
+ public static void prepare() {
+ registerTable(
+ "ORDER_DETAILS_BOUNDED",
+ TestBoundedTable.of(
+ Schema.FieldType.INT64, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.DECIMAL, "price")
+ .addRows(
+ 1L,
+ 1,
+ new BigDecimal(1.0),
+ 1L,
+ 1,
+ new BigDecimal(1.0),
+ 2L,
+ 2,
+ new BigDecimal(2.0),
+ 4L,
+ 4,
+ new BigDecimal(4.0),
+ 4L,
+ 4,
+ new BigDecimal(4.0)));
+
+ registerTable(
+ "ORDER_DETAILS_UNBOUNDED",
+ TestUnboundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price",
+ Schema.FieldType.DATETIME, "order_time")
+ .timestampColumnIndex(3)
+ .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(Duration.standardMinutes(1)),
+ 2,
+ 2,
+ 7,
+ SECOND_DATE,
+ 2,
+ 3,
+ 8,
+ SECOND_DATE,
+ // this late record is omitted(First window)
+ 1,
+ 3,
+ 3,
+ FIRST_DATE)
+ .addRows(
+ // this late record is omitted(Second window)
+ WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardMinutes(1)),
+ 2,
+ 3,
+ 3,
+ SECOND_DATE)
+ .setStatistics(BeamTableStatistics.createUnboundedTableStatistics(2d)));
+ }
+
+ @Test
+ public void testProjectionNodeStats() {
+ String sql = "SELECT order_id FROM ORDER_DETAILS_BOUNDED";
+
+ RelNode root = env.parseQuery(sql);
+
+ Assert.assertTrue(root instanceof BeamCalcRel);
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertEquals(5d, estimate.getRowCount(), 0.001);
+ Assert.assertEquals(5d, estimate.getWindow(), 0.001);
+ Assert.assertEquals(0., estimate.getRate(), 0.001);
+ }
+
+ @Test
+ public void testFilterNodeStats() {
+ String sql = "SELECT * FROM ORDER_DETAILS_BOUNDED where order_id=1";
+
+ RelNode root = env.parseQuery(sql);
+
+ Assert.assertTrue(root instanceof BeamCalcRel);
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertTrue(5d > estimate.getRowCount());
+ Assert.assertTrue(5d > estimate.getWindow());
+ Assert.assertEquals(0., estimate.getRate(), 0.001);
+ }
+
+ @Test
+ public void testNodeStatsConditionType() {
+ String equalSql = "SELECT * FROM ORDER_DETAILS_BOUNDED where order_id=1";
+ String geqSql = "SELECT * FROM ORDER_DETAILS_BOUNDED where order_id>=1";
+
+ RelNode equalRoot = env.parseQuery(equalSql);
+ RelNode geqRoot = env.parseQuery(geqSql);
+
+ NodeStats equalEstimate =
+ BeamSqlRelUtils.getNodeStats(equalRoot, equalRoot.getCluster().getMetadataQuery());
+ NodeStats geqEstimate =
+ BeamSqlRelUtils.getNodeStats(geqRoot, geqRoot.getCluster().getMetadataQuery());
+
+ Assert.assertTrue(geqEstimate.getRowCount() > equalEstimate.getRowCount());
+ Assert.assertTrue(geqEstimate.getWindow() > equalEstimate.getWindow());
+ }
+
+ @Test
+ public void testNodeStatsNumberOfConditions() {
+ String equalSql = "SELECT * FROM ORDER_DETAILS_BOUNDED where order_id=1";
+ String doubleEqualSql = "SELECT * FROM ORDER_DETAILS_BOUNDED WHERE order_id=1 AND site_id=2 ";
+
+ RelNode equalRoot = env.parseQuery(equalSql);
+ RelNode doubleEqualRoot = env.parseQuery(doubleEqualSql);
+
+ NodeStats equalEstimate =
+ BeamSqlRelUtils.getNodeStats(equalRoot, equalRoot.getCluster().getMetadataQuery());
+ NodeStats doubleEqualEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ doubleEqualRoot, doubleEqualRoot.getCluster().getMetadataQuery());
+
+ Assert.assertTrue(doubleEqualEstimate.getRowCount() < equalEstimate.getRowCount());
+ Assert.assertTrue(doubleEqualEstimate.getWindow() < equalEstimate.getWindow());
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java
new file mode 100644
index 0000000..6859f96
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java
@@ -0,0 +1,378 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.hamcrest.core.StringContains;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+/** Bounded + Bounded Test for {@code BeamCoGBKJoinRel}. */
+public class BeamCoGBKJoinRelBoundedVsBoundedTest extends BaseRelTest {
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ public static final TestBoundedTable ORDER_DETAILS1 =
+ TestBoundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price")
+ .addRows(1, 2, 3, 2, 3, 3, 3, 4, 5);
+
+ public static final TestBoundedTable ORDER_DETAILS2 =
+ TestBoundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price")
+ .addRows(1, 2, 3, 2, 3, 3, 3, 4, 5);
+
+ @BeforeClass
+ public static void prepare() {
+ registerTable("ORDER_DETAILS1", ORDER_DETAILS1);
+ registerTable("ORDER_DETAILS2", ORDER_DETAILS2);
+ }
+
+ @Test
+ public void testInnerJoin() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("site_id", Schema.FieldType.INT32)
+ .addField("price", Schema.FieldType.INT32)
+ .addField("order_id0", Schema.FieldType.INT32)
+ .addField("site_id0", Schema.FieldType.INT32)
+ .addField("price0", Schema.FieldType.INT32)
+ .build())
+ .addRows(2, 3, 3, 1, 2, 3)
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT * "
+ + " FROM ORDER_DETAILS1 o1 "
+ + " JOIN ORDER_DETAILS2 o2 "
+ + " on "
+ + " o1.order_id=o2.site_id ";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamCoGBKJoinRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+ NodeStats leftEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ ((BeamCoGBKJoinRel) root).getLeft(), root.getCluster().getMetadataQuery());
+ NodeStats rightEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ ((BeamCoGBKJoinRel) root).getRight(), root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertNotEquals(0d, estimate.getRowCount(), 0.001);
+ Assert.assertTrue(
+ estimate.getRowCount() < leftEstimate.getRowCount() * rightEstimate.getRowCount());
+
+ Assert.assertNotEquals(0d, estimate.getWindow(), 0.001);
+ Assert.assertTrue(estimate.getWindow() < leftEstimate.getWindow() * rightEstimate.getWindow());
+ }
+
+ @Test
+ public void testNodeStatsOfMoreConditions() {
+ String sql1 =
+ "SELECT * "
+ + " FROM ORDER_DETAILS1 o1 "
+ + " JOIN ORDER_DETAILS2 o2 "
+ + " on "
+ + " o1.order_id=o2.site_id ";
+
+ String sql2 =
+ "SELECT * "
+ + " FROM ORDER_DETAILS1 o1 "
+ + " JOIN ORDER_DETAILS2 o2 "
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ RelNode root1 = env.parseQuery(sql1);
+
+ while (!(root1 instanceof BeamCoGBKJoinRel)) {
+ root1 = root1.getInput(0);
+ }
+
+ RelNode root2 = env.parseQuery(sql2);
+
+ while (!(root2 instanceof BeamCoGBKJoinRel)) {
+ root2 = root2.getInput(0);
+ }
+
+ NodeStats estimate1 =
+ BeamSqlRelUtils.getNodeStats(root1, root1.getCluster().getMetadataQuery());
+ NodeStats estimate2 =
+ BeamSqlRelUtils.getNodeStats(root2, root1.getCluster().getMetadataQuery());
+
+ Assert.assertNotEquals(0d, estimate2.getRowCount(), 0.001);
+ // A join with two conditions should have lower estimate.
+ Assert.assertTrue(estimate2.getRowCount() < estimate1.getRowCount());
+ }
+
+ @Test
+ public void testLeftOuterJoin() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " LEFT OUTER JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ pipeline.enableAbandonedNodeEnforcement(false);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("site_id", Schema.FieldType.INT32)
+ .addField("price", Schema.FieldType.INT32)
+ .addNullableField("order_id0", Schema.FieldType.INT32)
+ .addNullableField("site_id0", Schema.FieldType.INT32)
+ .addNullableField("price0", Schema.FieldType.INT32)
+ .build())
+ .addRows(1, 2, 3, null, null, null, 2, 3, 3, 1, 2, 3, 3, 4, 5, null, null, null)
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testLeftOuterJoinWithEmptyTuplesOnRightSide() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " LEFT OUTER JOIN (SELECT * FROM ORDER_DETAILS2 WHERE FALSE) o2"
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ pipeline.enableAbandonedNodeEnforcement(false);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("site_id", Schema.FieldType.INT32)
+ .addField("price", Schema.FieldType.INT32)
+ .addNullableField("order_id0", Schema.FieldType.INT32)
+ .addNullableField("site_id0", Schema.FieldType.INT32)
+ .addNullableField("price0", Schema.FieldType.INT32)
+ .build())
+ .addRows(
+ 1, 2, 3, null, null, null, 2, 3, 3, null, null, null, 3, 4, 5, null, null, null)
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testInnerJoinWithEmptyTuplesOnRightSide() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " INNER JOIN (SELECT * FROM ORDER_DETAILS2 WHERE FALSE) o2"
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ pipeline.enableAbandonedNodeEnforcement(false);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("site_id", Schema.FieldType.INT32)
+ .addField("price", Schema.FieldType.INT32)
+ .addNullableField("order_id0", Schema.FieldType.INT32)
+ .addNullableField("site_id0", Schema.FieldType.INT32)
+ .addNullableField("price0", Schema.FieldType.INT32)
+ .build())
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testRightOuterJoin() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " RIGHT OUTER JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addNullableField("order_id", Schema.FieldType.INT32)
+ .addNullableField("site_id", Schema.FieldType.INT32)
+ .addNullableField("price", Schema.FieldType.INT32)
+ .addField("order_id0", Schema.FieldType.INT32)
+ .addField("site_id0", Schema.FieldType.INT32)
+ .addField("price0", Schema.FieldType.INT32)
+ .build())
+ .addRows(2, 3, 3, 1, 2, 3, null, null, null, 2, 3, 3, null, null, null, 3, 4, 5)
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testFullOuterJoin() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " FULL OUTER JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addNullableField("order_id", Schema.FieldType.INT32)
+ .addNullableField("site_id", Schema.FieldType.INT32)
+ .addNullableField("price", Schema.FieldType.INT32)
+ .addNullableField("order_id0", Schema.FieldType.INT32)
+ .addNullableField("site_id0", Schema.FieldType.INT32)
+ .addNullableField("price0", Schema.FieldType.INT32)
+ .build())
+ .addRows(
+ 2, 3, 3, 1, 2, 3, 1, 2, 3, null, null, null, 3, 4, 5, null, null, null, null,
+ null, null, 2, 3, 3, null, null, null, 3, 4, 5)
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testException_nonEqualJoin() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id>o2.site_id";
+
+ pipeline.enableAbandonedNodeEnforcement(false);
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ public void testException_join_condition1() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id = o2.site_id OR o1.price = o2.site_id";
+
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("Operator OR"));
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ public void testException_join_condition2() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id = o2.site_id AND o1.price > o2.site_id";
+
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("Non equi-join"));
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ public void testException_join_condition3() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id + o2.site_id = 2";
+
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("column reference"));
+ thrown.expectMessage(StringContains.containsString("struct field access"));
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ public void testException_join_condition4() throws Exception {
+ String sql =
+ "SELECT * "
+ + "FROM ORDER_DETAILS1 o1"
+ + " JOIN ORDER_DETAILS2 o2"
+ + " on "
+ + " o1.order_id + o2.site_id = 2 AND o1.price > o2.site_id";
+
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("column reference"));
+ thrown.expectMessage(StringContains.containsString("struct field access"));
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testException_crossJoin() throws Exception {
+ String sql = "SELECT * " + "FROM ORDER_DETAILS1 o1, ORDER_DETAILS2 o2";
+
+ pipeline.enableAbandonedNodeEnforcement(false);
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java
new file mode 100644
index 0000000..f310265
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.impl.transform.BeamSqlOutputToConsoleFn;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Unbounded + Unbounded Test for {@code BeamCoGBKJoinRel}. */
+public class BeamCoGBKJoinRelUnboundedVsUnboundedTest extends BaseRelTest {
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+ private static final DateTime FIRST_DATE = new DateTime(1);
+ private static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
+
+ private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
+ @BeforeClass
+ public static void prepare() {
+ registerTable(
+ "ORDER_DETAILS",
+ TestUnboundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price",
+ Schema.FieldType.DATETIME, "order_time")
+ .timestampColumnIndex(3)
+ .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(Duration.standardMinutes(1)),
+ 2,
+ 2,
+ 7,
+ SECOND_DATE,
+ 2,
+ 3,
+ 8,
+ SECOND_DATE,
+ // this late record is omitted(First window)
+ 1,
+ 3,
+ 3,
+ FIRST_DATE)
+ .addRows(
+ // this late record is omitted(Second window)
+ WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardMinutes(1)),
+ 2,
+ 3,
+ 3,
+ SECOND_DATE)
+ .setStatistics(BeamTableStatistics.createUnboundedTableStatistics(3d)));
+ }
+
+ @Test
+ public void testInnerJoin() throws Exception {
+ String sql =
+ "SELECT * FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id1", Schema.FieldType.INT32)
+ .addField("sum_site_id", Schema.FieldType.INT32)
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("sum_site_id0", Schema.FieldType.INT32)
+ .build())
+ .addRows(1, 3, 1, 3, 2, 5, 2, 5)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT * FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamCoGBKJoinRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+ NodeStats leftEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ ((BeamCoGBKJoinRel) root).getLeft(), root.getCluster().getMetadataQuery());
+ NodeStats rightEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ ((BeamCoGBKJoinRel) root).getRight(), root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRowCount(), 0.01);
+
+ Assert.assertNotEquals(0d, estimate.getRate(), 0.001);
+ Assert.assertTrue(
+ estimate.getRate()
+ < leftEstimate.getRate() * rightEstimate.getWindow()
+ + rightEstimate.getRate() * leftEstimate.getWindow());
+
+ Assert.assertNotEquals(0d, estimate.getWindow(), 0.001);
+ Assert.assertTrue(estimate.getWindow() < leftEstimate.getWindow() * rightEstimate.getWindow());
+ }
+
+ @Test
+ public void testLeftOuterJoin() throws Exception {
+ String sql =
+ "SELECT * FROM "
+ + "(select site_id as order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " LEFT OUTER JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ // 1, 1 | 1, 3
+ // 2, 2 | NULL, NULL
+ // ---- | -----
+ // 2, 2 | 2, 5
+ // 3, 3 | NULL, NULL
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id1", Schema.FieldType.INT32)
+ .addField("sum_site_id", Schema.FieldType.INT32)
+ .addNullableField("order_id", Schema.FieldType.INT32)
+ .addNullableField("sum_site_id0", Schema.FieldType.INT32)
+ .build())
+ .addRows(1, 1, 1, 3, 2, 2, null, null, 2, 2, 2, 5, 3, 3, null, null)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testRightOuterJoin() throws Exception {
+ String sql =
+ "SELECT * FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " RIGHT OUTER JOIN "
+ + "(select site_id as order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addNullableField("order_id1", Schema.FieldType.INT32)
+ .addNullableField("sum_site_id", Schema.FieldType.INT32)
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("sum_site_id0", Schema.FieldType.INT32)
+ .build())
+ .addRows(1, 3, 1, 1, null, null, 2, 2, 2, 5, 2, 2, null, null, 3, 3)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testFullOuterJoin() throws Exception {
+ String sql =
+ "SELECT * FROM "
+ + "(select price as order_id1, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY price, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " FULL OUTER JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id , TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
+ + " on "
+ + " o1.order_id1=o2.order_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ rows.apply(ParDo.of(new BeamSqlOutputToConsoleFn("hello")));
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addNullableField("order_id1", Schema.FieldType.INT32)
+ .addNullableField("sum_site_id", Schema.FieldType.INT32)
+ .addNullableField("order_id", Schema.FieldType.INT32)
+ .addNullableField("sum_site_id0", Schema.FieldType.INT32)
+ .build())
+ .addRows(
+ 1, 1, 1, 3, 6, 2, null, null, 7, 2, null, null, 8, 3, null, null, null, null, 2,
+ 5)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testWindowsMismatch() throws Exception {
+ String sql =
+ "SELECT * FROM "
+ + "(select site_id as order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY site_id, TUMBLE(order_time, INTERVAL '2' HOUR)) o1 "
+ + " LEFT OUTER JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+ pipeline.enableAbandonedNodeEnforcement(false);
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
index 5dfa38d..a0cf404 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamEnumerableConverterTest.java
@@ -21,11 +21,11 @@
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
-import com.google.common.collect.ImmutableList;
import java.math.BigDecimal;
import java.util.List;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.schemas.Schema;
@@ -39,17 +39,18 @@
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.linq4j.Enumerable;
-import org.apache.calcite.linq4j.Enumerator;
-import org.apache.calcite.plan.RelOptCluster;
-import org.apache.calcite.plan.volcano.VolcanoPlanner;
-import org.apache.calcite.prepare.RelOptTableImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.rex.RexBuilder;
-import org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.RelOptTableImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
@@ -123,7 +124,7 @@
assertEquals(Row.withSchema(schema).addValues(0L, 1L).build(), rowList.get(0));
}
- private static class FakeTable extends BaseBeamTable {
+ private static class FakeTable extends SchemaBaseBeamTable {
public FakeTable() {
super(null);
}
@@ -148,6 +149,11 @@
}));
return PDone.in(input.getPipeline());
}
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
}
@Test
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java
new file mode 100644
index 0000000..ff0d70f
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.math.BigDecimal;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Test for {@code BeamIOSourceRel}. */
+public class BeamIOSourceRelTest extends BaseRelTest {
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+
+ private static final DateTime FIRST_DATE = new DateTime(1);
+ private static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
+
+ private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
+ @BeforeClass
+ public static void prepare() {
+ registerTable(
+ "ORDER_DETAILS_BOUNDED",
+ TestBoundedTable.of(
+ Schema.FieldType.INT64, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.DECIMAL, "price")
+ .addRows(
+ 1L,
+ 1,
+ new BigDecimal(1.0),
+ 1L,
+ 1,
+ new BigDecimal(1.0),
+ 2L,
+ 2,
+ new BigDecimal(2.0),
+ 4L,
+ 4,
+ new BigDecimal(4.0),
+ 4L,
+ 4,
+ new BigDecimal(4.0)));
+
+ registerTable(
+ "ORDER_DETAILS_UNBOUNDED",
+ TestUnboundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price",
+ Schema.FieldType.DATETIME, "order_time")
+ .timestampColumnIndex(3)
+ .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(Duration.standardMinutes(1)),
+ 2,
+ 2,
+ 7,
+ SECOND_DATE,
+ 2,
+ 3,
+ 8,
+ SECOND_DATE,
+ // this late record is omitted(First window)
+ 1,
+ 3,
+ 3,
+ FIRST_DATE)
+ .addRows(
+ // this late record is omitted(Second window)
+ WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardMinutes(1)),
+ 2,
+ 3,
+ 3,
+ SECOND_DATE)
+ .setStatistics(BeamTableStatistics.createUnboundedTableStatistics(2d)));
+ }
+
+ @Test
+ public void boundedRowCount() {
+ String sql = "SELECT * FROM ORDER_DETAILS_BOUNDED";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamIOSourceRel)) {
+ root = root.getInput(0);
+ }
+
+ Assert.assertEquals(5d, root.estimateRowCount(RelMetadataQuery.instance()), 0.001);
+ }
+
+ @Test
+ public void unboundedRowCount() {
+ String sql = "SELECT * FROM ORDER_DETAILS_UNBOUNDED";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamIOSourceRel)) {
+ root = root.getInput(0);
+ }
+
+ Assert.assertEquals(2d, root.estimateRowCount(RelMetadataQuery.instance()), 0.001);
+ }
+
+ @Test
+ public void testBoundedNodeStats() {
+ String sql = "SELECT * FROM ORDER_DETAILS_BOUNDED";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamIOSourceRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertEquals(5d, estimate.getRowCount(), 0.01);
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+ Assert.assertEquals(5d, estimate.getWindow(), 0.01);
+ }
+
+ @Test
+ public void testUnboundedNodeStats() {
+ String sql = "SELECT * FROM ORDER_DETAILS_UNBOUNDED";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamIOSourceRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertEquals(0d, estimate.getRowCount(), 0.01);
+ Assert.assertEquals(2d, estimate.getRate(), 0.01);
+ Assert.assertEquals(BeamIOSourceRel.CONSTANT_WINDOW_SIZE, estimate.getWindow(), 0.01);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java
index 1bcbed4..d5acfab 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIntersectRelTest.java
@@ -19,12 +19,15 @@
import java.math.BigDecimal;
import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -121,4 +124,28 @@
pipeline.run();
}
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT order_id, site_id, price "
+ + " FROM ORDER_DETAILS1 "
+ + " INTERSECT "
+ + " SELECT order_id, site_id, price "
+ + " FROM ORDER_DETAILS2 ";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamIntersectRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertEquals(3. / 2., estimate.getRowCount(), 0.01);
+ Assert.assertEquals(3. / 2., estimate.getWindow(), 0.01);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
deleted file mode 100644
index d8e8a61..0000000
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
+++ /dev/null
@@ -1,303 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.rel;
-
-import org.apache.beam.sdk.extensions.sql.TestUtils;
-import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.hamcrest.core.StringContains;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Bounded + Bounded Test for {@code BeamJoinRel}. */
-public class BeamJoinRelBoundedVsBoundedTest extends BaseRelTest {
- @Rule public final TestPipeline pipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- public static final TestBoundedTable ORDER_DETAILS1 =
- TestBoundedTable.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.INT32, "site_id",
- Schema.FieldType.INT32, "price")
- .addRows(1, 2, 3, 2, 3, 3, 3, 4, 5);
-
- public static final TestBoundedTable ORDER_DETAILS2 =
- TestBoundedTable.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.INT32, "site_id",
- Schema.FieldType.INT32, "price")
- .addRows(1, 2, 3, 2, 3, 3, 3, 4, 5);
-
- @BeforeClass
- public static void prepare() {
- registerTable("ORDER_DETAILS1", ORDER_DETAILS1);
- registerTable("ORDER_DETAILS2", ORDER_DETAILS2);
- }
-
- @Test
- public void testInnerJoin() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows)
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id", Schema.FieldType.INT32)
- .addField("site_id", Schema.FieldType.INT32)
- .addField("price", Schema.FieldType.INT32)
- .addField("order_id0", Schema.FieldType.INT32)
- .addField("site_id0", Schema.FieldType.INT32)
- .addField("price0", Schema.FieldType.INT32)
- .build())
- .addRows(2, 3, 3, 1, 2, 3)
- .getRows());
- pipeline.run();
- }
-
- @Test
- public void testLeftOuterJoin() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " LEFT OUTER JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- pipeline.enableAbandonedNodeEnforcement(false);
- PAssert.that(rows)
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id", Schema.FieldType.INT32)
- .addField("site_id", Schema.FieldType.INT32)
- .addField("price", Schema.FieldType.INT32)
- .addNullableField("order_id0", Schema.FieldType.INT32)
- .addNullableField("site_id0", Schema.FieldType.INT32)
- .addNullableField("price0", Schema.FieldType.INT32)
- .build())
- .addRows(1, 2, 3, null, null, null, 2, 3, 3, 1, 2, 3, 3, 4, 5, null, null, null)
- .getRows());
- pipeline.run();
- }
-
- @Test
- public void testLeftOuterJoinWithEmptyTuplesOnRightSide() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " LEFT OUTER JOIN (SELECT * FROM ORDER_DETAILS2 WHERE FALSE) o2"
- + " on "
- + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- pipeline.enableAbandonedNodeEnforcement(false);
- PAssert.that(rows)
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id", Schema.FieldType.INT32)
- .addField("site_id", Schema.FieldType.INT32)
- .addField("price", Schema.FieldType.INT32)
- .addNullableField("order_id0", Schema.FieldType.INT32)
- .addNullableField("site_id0", Schema.FieldType.INT32)
- .addNullableField("price0", Schema.FieldType.INT32)
- .build())
- .addRows(
- 1, 2, 3, null, null, null, 2, 3, 3, null, null, null, 3, 4, 5, null, null, null)
- .getRows());
- pipeline.run();
- }
-
- @Test
- public void testInnerJoinWithEmptyTuplesOnRightSide() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " INNER JOIN (SELECT * FROM ORDER_DETAILS2 WHERE FALSE) o2"
- + " on "
- + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- pipeline.enableAbandonedNodeEnforcement(false);
- PAssert.that(rows)
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id", Schema.FieldType.INT32)
- .addField("site_id", Schema.FieldType.INT32)
- .addField("price", Schema.FieldType.INT32)
- .addNullableField("order_id0", Schema.FieldType.INT32)
- .addNullableField("site_id0", Schema.FieldType.INT32)
- .addNullableField("price0", Schema.FieldType.INT32)
- .build())
- .getRows());
- pipeline.run();
- }
-
- @Test
- public void testRightOuterJoin() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " RIGHT OUTER JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows)
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addNullableField("order_id", Schema.FieldType.INT32)
- .addNullableField("site_id", Schema.FieldType.INT32)
- .addNullableField("price", Schema.FieldType.INT32)
- .addField("order_id0", Schema.FieldType.INT32)
- .addField("site_id0", Schema.FieldType.INT32)
- .addField("price0", Schema.FieldType.INT32)
- .build())
- .addRows(2, 3, 3, 1, 2, 3, null, null, null, 2, 3, 3, null, null, null, 3, 4, 5)
- .getRows());
- pipeline.run();
- }
-
- @Test
- public void testFullOuterJoin() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " FULL OUTER JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id=o2.site_id AND o2.price=o1.site_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows)
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addNullableField("order_id", Schema.FieldType.INT32)
- .addNullableField("site_id", Schema.FieldType.INT32)
- .addNullableField("price", Schema.FieldType.INT32)
- .addNullableField("order_id0", Schema.FieldType.INT32)
- .addNullableField("site_id0", Schema.FieldType.INT32)
- .addNullableField("price0", Schema.FieldType.INT32)
- .build())
- .addRows(
- 2, 3, 3, 1, 2, 3, 1, 2, 3, null, null, null, 3, 4, 5, null, null, null, null,
- null, null, 2, 3, 3, null, null, null, 3, 4, 5)
- .getRows());
- pipeline.run();
- }
-
- @Test(expected = UnsupportedOperationException.class)
- public void testException_nonEqualJoin() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id>o2.site_id";
-
- pipeline.enableAbandonedNodeEnforcement(false);
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test
- public void testException_join_condition1() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id = o2.site_id OR o1.price = o2.site_id";
-
- thrown.expect(UnsupportedOperationException.class);
- thrown.expectMessage(StringContains.containsString("Operator OR"));
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test
- public void testException_join_condition2() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id = o2.site_id AND o1.price > o2.site_id";
-
- thrown.expect(UnsupportedOperationException.class);
- thrown.expectMessage(StringContains.containsString("Non equi-join"));
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test
- public void testException_join_condition3() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id + o2.site_id = 2";
-
- thrown.expect(UnsupportedOperationException.class);
- thrown.expectMessage(StringContains.containsString("column reference"));
- thrown.expectMessage(StringContains.containsString("struct field access"));
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test
- public void testException_join_condition4() throws Exception {
- String sql =
- "SELECT * "
- + "FROM ORDER_DETAILS1 o1"
- + " JOIN ORDER_DETAILS2 o2"
- + " on "
- + " o1.order_id + o2.site_id = 2 AND o1.price > o2.site_id";
-
- thrown.expect(UnsupportedOperationException.class);
- thrown.expectMessage(StringContains.containsString("column reference"));
- thrown.expectMessage(StringContains.containsString("struct field access"));
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test(expected = UnsupportedOperationException.class)
- public void testException_crossJoin() throws Exception {
- String sql = "SELECT * " + "FROM ORDER_DETAILS1 o1, ORDER_DETAILS2 o2";
-
- pipeline.enableAbandonedNodeEnforcement(false);
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
deleted file mode 100644
index 067af8a..0000000
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
+++ /dev/null
@@ -1,317 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.rel;
-
-import java.util.Arrays;
-import java.util.List;
-import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
-import org.apache.beam.sdk.extensions.sql.TestUtils;
-import org.apache.beam.sdk.extensions.sql.impl.schema.BaseBeamTable;
-import org.apache.beam.sdk.extensions.sql.impl.transform.BeamSqlOutputToConsoleFn;
-import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
-import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
-import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PBegin;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.sdk.values.Row;
-import org.joda.time.DateTime;
-import org.joda.time.Duration;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.Test;
-
-/** Unbounded + Unbounded Test for {@code BeamJoinRel}. */
-public class BeamJoinRelUnboundedVsBoundedTest extends BaseRelTest {
- @Rule public final TestPipeline pipeline = TestPipeline.create();
- public static final DateTime FIRST_DATE = new DateTime(1);
- public static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
- public static final DateTime THIRD_DATE = new DateTime(1 + 3600 * 1000 + 3600 * 1000 + 1);
- private static final Duration WINDOW_SIZE = Duration.standardHours(1);
-
- @BeforeClass
- public static void prepare() {
- registerTable(
- "ORDER_DETAILS",
- TestUnboundedTable.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.INT32, "site_id",
- Schema.FieldType.INT32, "price",
- Schema.FieldType.DATETIME, "order_time")
- .timestampColumnIndex(3)
- .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 2, FIRST_DATE)
- .addRows(
- WINDOW_SIZE.plus(Duration.standardSeconds(1)),
- 2,
- 2,
- 3,
- SECOND_DATE,
- 2,
- 3,
- 3,
- SECOND_DATE,
- // this late data is omitted
- 1,
- 2,
- 3,
- FIRST_DATE)
- .addRows(
- WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardSeconds(1)),
- 3,
- 3,
- 3,
- THIRD_DATE,
- // this late data is omitted
- 2,
- 2,
- 3,
- SECOND_DATE));
-
- registerTable(
- "ORDER_DETAILS1",
- TestBoundedTable.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.STRING, "buyer")
- .addRows(
- 1, "james",
- 2, "bond"));
-
- registerTable(
- "SITE_LKP",
- new SiteLookupTable(
- TestTableUtils.buildBeamSqlSchema(
- Schema.FieldType.INT32, "site_id",
- Schema.FieldType.STRING, "site_name")));
- }
-
- /** Test table for JOIN-AS-LOOKUP. */
- public static class SiteLookupTable extends BaseBeamTable implements BeamSqlSeekableTable {
-
- public SiteLookupTable(Schema schema) {
- super(schema);
- }
-
- @Override
- public PCollection.IsBounded isBounded() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public PCollection<Row> buildIOReader(PBegin begin) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public POutput buildIOWriter(PCollection<Row> input) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public List<Row> seekRow(Row lookupSubRow) {
- return Arrays.asList(Row.withSchema(getSchema()).addValues(1, "SITE1").build());
- }
- }
-
- @Test
- public void testInnerJoin_unboundedTableOnTheLeftSide() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " JOIN "
- + " ORDER_DETAILS1 o2 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.INT32, "sum_site_id",
- Schema.FieldType.STRING, "buyer")
- .addRows(1, 3, "james", 2, 5, "bond")
- .getStringRows());
- pipeline.run();
- }
-
- @Test
- public void testInnerJoin_boundedTableOnTheLeftSide() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + " ORDER_DETAILS1 o2 "
- + " JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.INT32, "sum_site_id",
- Schema.FieldType.STRING, "buyer")
- .addRows(1, 3, "james", 2, 5, "bond")
- .getStringRows());
- pipeline.run();
- }
-
- @Test
- public void testLeftOuterJoin() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " LEFT OUTER JOIN "
- + " ORDER_DETAILS1 o2 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
-
- rows.apply(ParDo.of(new BeamSqlOutputToConsoleFn("helloworld")));
-
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id", Schema.FieldType.INT32)
- .addField("sum_site_id", Schema.FieldType.INT32)
- .addNullableField("buyer", Schema.FieldType.STRING)
- .build())
- .addRows(1, 3, "james", 2, 5, "bond", 3, 3, null)
- .getStringRows());
- pipeline.run();
- }
-
- @Test(expected = UnsupportedOperationException.class)
- public void testLeftOuterJoinError() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + " ORDER_DETAILS1 o2 "
- + " LEFT OUTER JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " on "
- + " o1.order_id=o2.order_id";
- pipeline.enableAbandonedNodeEnforcement(false);
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test
- public void testRightOuterJoin() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + " ORDER_DETAILS1 o2 "
- + " RIGHT OUTER JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " on "
- + " o1.order_id=o2.order_id";
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id", Schema.FieldType.INT32)
- .addField("sum_site_id", Schema.FieldType.INT32)
- .addNullableField("buyer", Schema.FieldType.STRING)
- .build())
- .addRows(1, 3, "james", 2, 5, "bond", 3, 3, null)
- .getStringRows());
- pipeline.run();
- }
-
- @Test(expected = UnsupportedOperationException.class)
- public void testRightOuterJoinError() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " RIGHT OUTER JOIN "
- + " ORDER_DETAILS1 o2 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- pipeline.enableAbandonedNodeEnforcement(false);
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test(expected = UnsupportedOperationException.class)
- public void testFullOuterJoinError() throws Exception {
- String sql =
- "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
- + " ORDER_DETAILS1 o2 "
- + " FULL OUTER JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " on "
- + " o1.order_id=o2.order_id";
- pipeline.enableAbandonedNodeEnforcement(false);
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-
- @Test
- public void testJoinAsLookup() throws Exception {
- String sql =
- "SELECT o1.order_id, o2.site_name FROM "
- + " ORDER_DETAILS o1 "
- + " JOIN SITE_LKP o2 "
- + " on "
- + " o1.site_id=o2.site_id "
- + " WHERE o1.site_id=1";
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.STRING, "site_name")
- .addRows(1, "SITE1")
- .getStringRows());
- pipeline.run();
- }
-
- @Test
- public void testJoinAsLookupSwapped() throws Exception {
- String sql =
- "SELECT o1.order_id, o2.site_name FROM "
- + " SITE_LKP o2 "
- + " JOIN ORDER_DETAILS o1 "
- + " on "
- + " o1.site_id=o2.site_id "
- + " WHERE o1.site_id=1";
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.STRING, "site_name")
- .addRows(1, "SITE1")
- .getStringRows());
- pipeline.run();
- }
-}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsUnboundedTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsUnboundedTest.java
deleted file mode 100644
index 869b1b8..0000000
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsUnboundedTest.java
+++ /dev/null
@@ -1,210 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.rel;
-
-import org.apache.beam.sdk.extensions.sql.TestUtils;
-import org.apache.beam.sdk.extensions.sql.impl.transform.BeamSqlOutputToConsoleFn;
-import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.joda.time.DateTime;
-import org.joda.time.Duration;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.Test;
-
-/** Unbounded + Unbounded Test for {@code BeamJoinRel}. */
-public class BeamJoinRelUnboundedVsUnboundedTest extends BaseRelTest {
- @Rule public final TestPipeline pipeline = TestPipeline.create();
- public static final DateTime FIRST_DATE = new DateTime(1);
- public static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
-
- private static final Duration WINDOW_SIZE = Duration.standardHours(1);
-
- @BeforeClass
- public static void prepare() {
- registerTable(
- "ORDER_DETAILS",
- TestUnboundedTable.of(
- Schema.FieldType.INT32, "order_id",
- Schema.FieldType.INT32, "site_id",
- Schema.FieldType.INT32, "price",
- Schema.FieldType.DATETIME, "order_time")
- .timestampColumnIndex(3)
- .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)
- .addRows(
- WINDOW_SIZE.plus(Duration.standardMinutes(1)),
- 2,
- 2,
- 7,
- SECOND_DATE,
- 2,
- 3,
- 8,
- SECOND_DATE,
- // this late record is omitted(First window)
- 1,
- 3,
- 3,
- FIRST_DATE)
- .addRows(
- // this late record is omitted(Second window)
- WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardMinutes(1)),
- 2,
- 3,
- 3,
- SECOND_DATE));
- }
-
- @Test
- public void testInnerJoin() throws Exception {
- String sql =
- "SELECT * FROM "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id1", Schema.FieldType.INT32)
- .addField("sum_site_id", Schema.FieldType.INT32)
- .addField("order_id", Schema.FieldType.INT32)
- .addField("sum_site_id0", Schema.FieldType.INT32)
- .build())
- .addRows(1, 3, 1, 3, 2, 5, 2, 5)
- .getStringRows());
- pipeline.run();
- }
-
- @Test
- public void testLeftOuterJoin() throws Exception {
- String sql =
- "SELECT * FROM "
- + "(select site_id as order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " LEFT OUTER JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- // 1, 1 | 1, 3
- // 2, 2 | NULL, NULL
- // ---- | -----
- // 2, 2 | 2, 5
- // 3, 3 | NULL, NULL
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addField("order_id1", Schema.FieldType.INT32)
- .addField("sum_site_id", Schema.FieldType.INT32)
- .addNullableField("order_id", Schema.FieldType.INT32)
- .addNullableField("sum_site_id0", Schema.FieldType.INT32)
- .build())
- .addRows(1, 1, 1, 3, 2, 2, null, null, 2, 2, 2, 5, 3, 3, null, null)
- .getStringRows());
- pipeline.run();
- }
-
- @Test
- public void testRightOuterJoin() throws Exception {
- String sql =
- "SELECT * FROM "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " RIGHT OUTER JOIN "
- + "(select site_id as order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
- + " on "
- + " o1.order_id=o2.order_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addNullableField("order_id1", Schema.FieldType.INT32)
- .addNullableField("sum_site_id", Schema.FieldType.INT32)
- .addField("order_id", Schema.FieldType.INT32)
- .addField("sum_site_id0", Schema.FieldType.INT32)
- .build())
- .addRows(1, 3, 1, 1, null, null, 2, 2, 2, 5, 2, 2, null, null, 3, 3)
- .getStringRows());
- pipeline.run();
- }
-
- @Test
- public void testFullOuterJoin() throws Exception {
- String sql =
- "SELECT * FROM "
- + "(select price as order_id1, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY price, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
- + " FULL OUTER JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id , TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
- + " on "
- + " o1.order_id1=o2.order_id";
-
- PCollection<Row> rows = compilePipeline(sql, pipeline);
- rows.apply(ParDo.of(new BeamSqlOutputToConsoleFn("hello")));
- PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
- .containsInAnyOrder(
- TestUtils.RowsBuilder.of(
- Schema.builder()
- .addNullableField("order_id1", Schema.FieldType.INT32)
- .addNullableField("sum_site_id", Schema.FieldType.INT32)
- .addNullableField("order_id", Schema.FieldType.INT32)
- .addNullableField("sum_site_id0", Schema.FieldType.INT32)
- .build())
- .addRows(
- 1, 1, 1, 3, 6, 2, null, null, 7, 2, null, null, 8, 3, null, null, null, null, 2,
- 5)
- .getStringRows());
- pipeline.run();
- }
-
- @Test(expected = IllegalArgumentException.class)
- public void testWindowsMismatch() throws Exception {
- String sql =
- "SELECT * FROM "
- + "(select site_id as order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY site_id, TUMBLE(order_time, INTERVAL '2' HOUR)) o1 "
- + " LEFT OUTER JOIN "
- + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
- + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o2 "
- + " on "
- + " o1.order_id=o2.order_id";
- pipeline.enableAbandonedNodeEnforcement(false);
- compilePipeline(sql, pipeline);
- pipeline.run();
- }
-}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java
index 6484360..074c447 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamMinusRelTest.java
@@ -19,12 +19,19 @@
import java.math.BigDecimal;
import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -33,6 +40,11 @@
public class BeamMinusRelTest extends BaseRelTest {
@Rule public final TestPipeline pipeline = TestPipeline.create();
+ private static final DateTime FIRST_DATE = new DateTime(1);
+ private static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
+
+ private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
@BeforeClass
public static void prepare() {
registerTable(
@@ -74,10 +86,43 @@
3L,
3,
new BigDecimal(3.0)));
+
+ registerTable(
+ "ORDER_DETAILS_UNBOUNDED",
+ TestUnboundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price",
+ Schema.FieldType.DATETIME, "order_time")
+ .timestampColumnIndex(3)
+ .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 6, FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(Duration.standardMinutes(1)),
+ 2,
+ 2,
+ 7,
+ SECOND_DATE,
+ 2,
+ 3,
+ 8,
+ SECOND_DATE,
+ // this late record is omitted(First window)
+ 1,
+ 3,
+ 3,
+ FIRST_DATE)
+ .addRows(
+ // this late record is omitted(Second window)
+ WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardMinutes(1)),
+ 2,
+ 3,
+ 3,
+ SECOND_DATE)
+ .setStatistics(BeamTableStatistics.createUnboundedTableStatistics(4d)));
}
@Test
- public void testExcept() throws Exception {
+ public void testExcept() {
String sql = "";
sql +=
"SELECT order_id, site_id, price "
@@ -100,7 +145,7 @@
}
@Test
- public void testExceptAll() throws Exception {
+ public void testExceptAll() {
String sql = "";
sql +=
"SELECT order_id, site_id, price "
@@ -134,7 +179,7 @@
}
@Test
- public void testExceptRemovesDuplicates() throws Exception {
+ public void testExceptRemovesDuplicates() {
String sql = "(SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 1) EXCEPT SELECT 1";
PCollection<Row> rows = compilePipeline(sql, pipeline);
@@ -146,4 +191,52 @@
pipeline.run();
}
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT order_id, site_id, price "
+ + "FROM ORDER_DETAILS1 "
+ + " EXCEPT ALL "
+ + "SELECT order_id, site_id, price "
+ + "FROM ORDER_DETAILS2 ";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamMinusRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertEquals(5. - 3. / 2., estimate.getRowCount(), 0.01);
+ Assert.assertEquals(5. - 3. / 2., estimate.getWindow(), 0.01);
+ }
+
+ @Test
+ public void testNodeStatsEstimationUnbounded() {
+ String sql =
+ "SELECT * "
+ + "FROM "
+ + "(select order_id FROM ORDER_DETAILS_UNBOUNDED "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " EXCEPT ALL "
+ + " select order_id FROM ORDER_DETAILS_UNBOUNDED "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR) ";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamMinusRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ // note that we have group by
+ Assert.assertEquals(4d / 2 - 4d / 4, estimate.getRate(), 0.01);
+ Assert.assertEquals(0d, estimate.getRowCount(), 0.01);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java
new file mode 100644
index 0000000..39e2a73
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.impl.transform.BeamSqlOutputToConsoleFn;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestUnboundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Unbounded + Bounded Test for {@code BeamSideInputJoinRel}. */
+public class BeamSideInputJoinRelTest extends BaseRelTest {
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+ public static final DateTime FIRST_DATE = new DateTime(1);
+ public static final DateTime SECOND_DATE = new DateTime(1 + 3600 * 1000);
+ public static final DateTime THIRD_DATE = new DateTime(1 + 3600 * 1000 + 3600 * 1000 + 1);
+ private static final Duration WINDOW_SIZE = Duration.standardHours(1);
+
+ @BeforeClass
+ public static void prepare() {
+ registerUnboundedTable();
+
+ registerTable(
+ "ORDER_DETAILS1",
+ TestBoundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.STRING, "buyer")
+ .addRows(
+ 1, "james",
+ 2, "bond"));
+ }
+
+ public static void registerUnboundedTable() {
+ registerTable(
+ "ORDER_DETAILS",
+ TestUnboundedTable.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "site_id",
+ Schema.FieldType.INT32, "price",
+ Schema.FieldType.DATETIME, "order_time")
+ .timestampColumnIndex(3)
+ .addRows(Duration.ZERO, 1, 1, 1, FIRST_DATE, 1, 2, 2, FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(Duration.standardSeconds(1)),
+ 2,
+ 2,
+ 3,
+ SECOND_DATE,
+ 2,
+ 3,
+ 3,
+ SECOND_DATE,
+ // this late data is omitted
+ 1,
+ 2,
+ 3,
+ FIRST_DATE)
+ .addRows(
+ WINDOW_SIZE.plus(WINDOW_SIZE).plus(Duration.standardSeconds(1)),
+ 3,
+ 3,
+ 3,
+ THIRD_DATE,
+ // this late data is omitted
+ 2,
+ 2,
+ 3,
+ SECOND_DATE));
+ }
+
+ @Test
+ public void testInnerJoin_unboundedTableOnTheLeftSide() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " JOIN "
+ + " ORDER_DETAILS1 o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "sum_site_id",
+ Schema.FieldType.STRING, "buyer")
+ .addRows(1, 3, "james", 2, 5, "bond")
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testInnerJoin_boundedTableOnTheLeftSide() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + " ORDER_DETAILS1 o2 "
+ + " JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.INT32, "sum_site_id",
+ Schema.FieldType.STRING, "buyer")
+ .addRows(1, 3, "james", 2, 5, "bond")
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " JOIN "
+ + " ORDER_DETAILS1 o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamSideInputJoinRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+ NodeStats leftEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ ((BeamSideInputJoinRel) root).getLeft(), root.getCluster().getMetadataQuery());
+ NodeStats rightEstimate =
+ BeamSqlRelUtils.getNodeStats(
+ ((BeamSideInputJoinRel) root).getRight(), root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRowCount(), 0.01);
+
+ Assert.assertNotEquals(0d, estimate.getRate(), 0.001);
+ Assert.assertTrue(
+ estimate.getRate()
+ < leftEstimate.getRowCount() * rightEstimate.getWindow()
+ + rightEstimate.getRowCount() * leftEstimate.getWindow());
+
+ Assert.assertNotEquals(0d, estimate.getWindow(), 0.001);
+ Assert.assertTrue(estimate.getWindow() < leftEstimate.getWindow() * rightEstimate.getWindow());
+ }
+
+ @Test
+ public void testLeftOuterJoin() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " LEFT OUTER JOIN "
+ + " ORDER_DETAILS1 o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+
+ rows.apply(ParDo.of(new BeamSqlOutputToConsoleFn("helloworld")));
+
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("sum_site_id", Schema.FieldType.INT32)
+ .addNullableField("buyer", Schema.FieldType.STRING)
+ .build())
+ .addRows(1, 3, "james", 2, 5, "bond", 3, 3, null)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testLeftOuterJoinError() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + " ORDER_DETAILS1 o2 "
+ + " LEFT OUTER JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+ pipeline.enableAbandonedNodeEnforcement(false);
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ public void testRightOuterJoin() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + " ORDER_DETAILS1 o2 "
+ + " RIGHT OUTER JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.builder()
+ .addField("order_id", Schema.FieldType.INT32)
+ .addField("sum_site_id", Schema.FieldType.INT32)
+ .addNullableField("buyer", Schema.FieldType.STRING)
+ .build())
+ .addRows(1, 3, "james", 2, 5, "bond", 3, 3, null)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testRightOuterJoinError() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " RIGHT OUTER JOIN "
+ + " ORDER_DETAILS1 o2 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+
+ pipeline.enableAbandonedNodeEnforcement(false);
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test(expected = UnsupportedOperationException.class)
+ public void testFullOuterJoinError() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o1.sum_site_id, o2.buyer FROM "
+ + " ORDER_DETAILS1 o2 "
+ + " FULL OUTER JOIN "
+ + "(select order_id, sum(site_id) as sum_site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " on "
+ + " o1.order_id=o2.order_id";
+ pipeline.enableAbandonedNodeEnforcement(false);
+ compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
new file mode 100644
index 0000000..4b8dc38
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import static org.apache.beam.sdk.extensions.sql.impl.rel.BeamCoGBKJoinRelBoundedVsBoundedTest.ORDER_DETAILS1;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.BeamSqlSeekableTable;
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.SchemaBaseBeamTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.Row;
+import org.hamcrest.core.StringContains;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+public class BeamSideInputLookupJoinRelTest extends BaseRelTest {
+
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+ @Rule public ExpectedException thrown = ExpectedException.none();
+ private static final boolean nullable = true;
+
+ /** Test table for JOIN-AS-LOOKUP. */
+ public static class SiteLookupTable extends SchemaBaseBeamTable implements BeamSqlSeekableTable {
+
+ public SiteLookupTable(Schema schema) {
+ super(schema);
+ }
+
+ @Override
+ public PCollection.IsBounded isBounded() {
+ return PCollection.IsBounded.BOUNDED;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public POutput buildIOWriter(PCollection<Row> input) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public List<Row> seekRow(Row lookupSubRow) {
+ if (lookupSubRow.getInt32("site_id") == 2) {
+ return Arrays.asList(Row.withSchema(getSchema()).addValues(2, "SITE1").build());
+ }
+ return Arrays.asList(Row.nullRow(getSchema()));
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ return BeamTableStatistics.BOUNDED_UNKNOWN;
+ }
+ }
+
+ @BeforeClass
+ public static void prepare() {
+ BeamSideInputJoinRelTest.registerUnboundedTable();
+ registerTable("ORDER_DETAILS1", ORDER_DETAILS1);
+ registerTable(
+ "SITE_LKP",
+ new SiteLookupTable(
+ TestTableUtils.buildBeamSqlNullableSchema(
+ Schema.FieldType.INT32,
+ "site_id",
+ nullable,
+ Schema.FieldType.STRING,
+ "site_name",
+ nullable)));
+ }
+
+ @Test
+ public void testBoundedTableInnerJoinWithLookupTable() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + " ORDER_DETAILS1 o1 "
+ + " JOIN SITE_LKP o2 "
+ + " on "
+ + " o1.site_id=o2.site_id "
+ + " WHERE o1.site_id=2 ";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.STRING, "site_name")
+ .addRows(1, "SITE1")
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testLookupTableInnerJoinWithBoundedTable() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + " SITE_LKP o2 "
+ + " JOIN ORDER_DETAILS1 o1 "
+ + " on "
+ + " o1.site_id=o2.site_id "
+ + " WHERE o1.site_id=2 ";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.STRING, "site_name")
+ .addRows(1, "SITE1")
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testUnboundedTableInnerJoinWithLookupTable() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + "(select order_id, site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " JOIN "
+ + " SITE_LKP o2 "
+ + " on "
+ + " o1.site_id=o2.site_id"
+ + " WHERE o1.site_id=2 ";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.STRING, "site_name")
+ .addRows(1, "SITE1")
+ .addRows(2, "SITE1")
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testLookupTableInnerJoinWithUnboundedTable() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + " SITE_LKP o2 "
+ + " JOIN "
+ + "(select order_id, site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " on "
+ + " o1.site_id=o2.site_id"
+ + " WHERE o1.site_id=2 ";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "order_id",
+ Schema.FieldType.STRING, "site_name")
+ .addRows(1, "SITE1")
+ .addRows(2, "SITE1")
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testLookupTableRightOuterJoinWithBoundedTable() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + " SITE_LKP o2 "
+ + " RIGHT OUTER JOIN "
+ + " ORDER_DETAILS1 o1 "
+ + " on "
+ + " o1.site_id=o2.site_id ";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.ofNullable(
+ Schema.FieldType.INT32,
+ "order_id",
+ nullable,
+ Schema.FieldType.STRING,
+ "site_name",
+ nullable)
+ .addRows(1, "SITE1")
+ .addRows(2, null)
+ .addRows(3, null)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testUnboundedTableLeftOuterJoinWithLookupTable() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + "(select order_id, site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " LEFT OUTER JOIN "
+ + " SITE_LKP o2 "
+ + " on "
+ + " o1.site_id=o2.site_id";
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows.apply(ParDo.of(new TestUtils.BeamSqlRow2StringDoFn())))
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.ofNullable(
+ Schema.FieldType.INT32,
+ "order_id",
+ nullable,
+ Schema.FieldType.STRING,
+ "site_name",
+ nullable)
+ .addRows(1, "SITE1")
+ .addRows(2, "SITE1")
+ .addRows(1, null)
+ .addRows(2, null)
+ .addRows(3, null)
+ .getStringRows());
+ pipeline.run();
+ }
+
+ @Test
+ // Do not add a filter like "WHERE o1.order_id=2". By adding that filter, FilterJoinRule may
+ // convert "LEFT OUTER JOIN" to "INNER JOIN".
+ public void testLookupTableLeftOuterJoinWithBoundedTableError() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + " SITE_LKP o2 "
+ + " LEFT OUTER JOIN "
+ + " ORDER_DETAILS1 o1 "
+ + " on "
+ + " o1.site_id=o2.site_id ";
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("OUTER JOIN must be a non Seekable table"));
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ // Do not add a filter like "WHERE o1.order_id=2". By adding that filter, FilterJoinRule may
+ // convert "FULL OUTER JOIN" to "LEFT OUTER JOIN", which, in tis case is a valid scenario.
+ public void testUnboundedTableFullOuterJoinWithLookupTableError() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + "(select order_id, site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " FULL OUTER JOIN "
+ + " SITE_LKP o2 "
+ + " on "
+ + " o1.site_id=o2.site_id";
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("not supported"));
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+
+ @Test
+ // Do not add a filter like "WHERE o1.order_id=2". By adding that filter, FilterJoinRule may
+ // convert "RIGHT OUTER JOIN" to "INNER JOIN".
+ public void testUnboundedTableRightOuterJoinWithLookupTableError() throws Exception {
+ String sql =
+ "SELECT o1.order_id, o2.site_name FROM "
+ + "(select order_id, site_id FROM ORDER_DETAILS "
+ + " GROUP BY order_id, site_id, TUMBLE(order_time, INTERVAL '1' HOUR)) o1 "
+ + " RIGHT OUTER JOIN "
+ + " SITE_LKP o2 "
+ + " on "
+ + " o1.site_id=o2.site_id";
+ thrown.expect(UnsupportedOperationException.class);
+ thrown.expectMessage(StringContains.containsString("OUTER JOIN must be a non Seekable table"));
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java
index 3e058a2..bba4876 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRelTest.java
@@ -18,13 +18,16 @@
package org.apache.beam.sdk.extensions.sql.impl.rel;
import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
import org.joda.time.DateTime;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -102,7 +105,7 @@
}
@Test
- public void testOrderBy_basic() throws Exception {
+ public void testOrderBy_basic() {
String sql =
"INSERT INTO SUB_ORDER_RAM(order_id, site_id, price) SELECT "
+ " order_id, site_id, price "
@@ -122,7 +125,7 @@
}
@Test
- public void testOrderBy_timestamp() throws Exception {
+ public void testOrderBy_timestamp() {
String sql =
"SELECT order_id, site_id, price, order_time "
+ "FROM ORDER_DETAILS "
@@ -158,7 +161,7 @@
}
@Test
- public void testOrderBy_nullsFirst() throws Exception {
+ public void testOrderBy_nullsFirst() {
Schema schema =
Schema.builder()
.addField("order_id", Schema.FieldType.INT64)
@@ -188,7 +191,7 @@
}
@Test
- public void testOrderBy_nullsLast() throws Exception {
+ public void testOrderBy_nullsLast() {
Schema schema =
Schema.builder()
.addField("order_id", Schema.FieldType.INT64)
@@ -218,7 +221,7 @@
}
@Test
- public void testOrderBy_with_offset2() throws Exception {
+ public void testOrderBy_with_offset2() {
Schema schema = Schema.builder().addField("count_star", Schema.FieldType.INT64).build();
String sql =
@@ -232,7 +235,7 @@
}
@Test
- public void testOrderBy_with_offset() throws Exception {
+ public void testOrderBy_with_offset() {
String sql =
"INSERT INTO SUB_ORDER_RAM(order_id, site_id, price) SELECT "
+ " order_id, site_id, price "
@@ -252,7 +255,7 @@
}
@Test
- public void testOrderBy_bigFetch() throws Exception {
+ public void testOrderBy_bigFetch() {
String sql =
"INSERT INTO SUB_ORDER_RAM(order_id, site_id, price) SELECT "
+ " order_id, site_id, price "
@@ -288,4 +291,26 @@
TestPipeline pipeline = TestPipeline.create();
compilePipeline(sql, pipeline);
}
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT order_id, site_id, price, order_time "
+ + "FROM ORDER_DETAILS "
+ + "ORDER BY order_time asc limit 11";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamSortRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertEquals(10., estimate.getRowCount(), 0.01);
+ Assert.assertEquals(10., estimate.getWindow(), 0.01);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java
new file mode 100644
index 0000000..640a1df
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUncollectRelTest.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Tests for {@code BeamUncollectRel}. */
+public class BeamUncollectRelTest extends BaseRelTest {
+
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+
+ private NodeStats getEstimateOf(String sql) {
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamUncollectRel)) {
+ root = root.getInput(0);
+ }
+
+ return BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+ }
+
+ @Test
+ public void testNodeStats() {
+ NodeStats estimate =
+ getEstimateOf(
+ "SELECT * FROM UNNEST (SELECT * FROM (VALUES (ARRAY ['a', 'b', 'c']),(ARRAY ['a', 'b', 'c']))) t1");
+
+ Assert.assertEquals(4d, estimate.getRowCount(), 0.001);
+ Assert.assertEquals(4d, estimate.getWindow(), 0.001);
+ Assert.assertEquals(0., estimate.getRate(), 0.001);
+ }
+
+ @Test
+ public void testUncollectPrimitive() {
+ String sql = "SELECT * FROM unnest(ARRAY [1, 2, 3])";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(Schema.FieldType.INT32, "intField")
+ .addRows(1, 2, 3)
+ .getRows());
+ pipeline.run();
+ }
+
+ @Test
+ public void testUncollectNested() {
+ Schema rowSchema =
+ Schema.builder().addStringField("stringField").addInt32Field("intField").build();
+ List<Row> nestedRows =
+ Arrays.asList(
+ Row.withSchema(rowSchema).addValues("test1", 1).build(),
+ Row.withSchema(rowSchema).addValues("test2", 2).build());
+ registerTable(
+ "NESTED",
+ TestBoundedTable.of(
+ Schema.FieldType.STRING,
+ "user_id",
+ Schema.FieldType.array(Schema.FieldType.row(rowSchema)),
+ "nested")
+ .addRows("1", nestedRows));
+
+ String sql = "SELECT intField, stringField FROM unnest(SELECT nested from NESTED)";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.INT32, "intField",
+ Schema.FieldType.STRING, "stringField")
+ .addRows(1, "test1", 2, "test2")
+ .getRows());
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java
index b7ee1a8f..6f77751 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnionRelTest.java
@@ -19,12 +19,15 @@
import java.math.BigDecimal;
import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -99,4 +102,54 @@
.getRows());
pipeline.run();
}
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT "
+ + " order_id, site_id, price "
+ + "FROM ORDER_DETAILS "
+ + " UNION SELECT "
+ + " order_id, site_id, price "
+ + "FROM ORDER_DETAILS ";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamUnionRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertEquals(2., estimate.getRowCount(), 0.01);
+ Assert.assertEquals(2., estimate.getWindow(), 0.01);
+ }
+
+ @Test
+ public void testNodeStatsEstimationUnionAll() {
+ String sql =
+ "SELECT "
+ + " order_id, site_id, price "
+ + "FROM ORDER_DETAILS "
+ + " UNION ALL SELECT "
+ + " order_id, site_id, price "
+ + "FROM ORDER_DETAILS ";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamUnionRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertEquals(4., estimate.getRowCount(), 0.01);
+ Assert.assertEquals(4., estimate.getWindow(), 0.01);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRelTest.java
new file mode 100644
index 0000000..c103ac5
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRelTest.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rel;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Test for {@code BeamUnnestRel}. */
+public class BeamUnnestRelTest extends BaseRelTest {
+ @Rule public final TestPipeline pipeline = TestPipeline.create();
+
+ @BeforeClass
+ public static void prepare() {
+ Schema rowSchema =
+ Schema.builder().addStringField("stringField").addInt32Field("intField").build();
+ List<Row> nestedRows =
+ Arrays.asList(
+ Row.withSchema(rowSchema).addValues("test1", 1).build(),
+ Row.withSchema(rowSchema).addValues("test2", 2).build());
+ registerTable(
+ "NESTED",
+ TestBoundedTable.of(
+ Schema.FieldType.STRING,
+ "user_id",
+ Schema.FieldType.array(Schema.FieldType.row(rowSchema)),
+ "nested")
+ .addRows("1", nestedRows));
+ }
+
+ @Test
+ public void testUnnest() {
+ String sql =
+ "SELECT user_id, p.intField, p.stringField FROM NESTED as t, unnest(t.nested) as p";
+
+ PCollection<Row> rows = compilePipeline(sql, pipeline);
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ TestUtils.RowsBuilder.of(
+ Schema.FieldType.STRING, "user_id",
+ Schema.FieldType.INT32, "intField",
+ Schema.FieldType.STRING, "stringField")
+ .addRows("1", 1, "test1", "1", 2, "test2")
+ .getRows());
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java
index 0dc8e26..0787751 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamValuesRelTest.java
@@ -18,12 +18,15 @@
package org.apache.beam.sdk.extensions.sql.impl.rel;
import org.apache.beam.sdk.extensions.sql.TestUtils;
+import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -104,4 +107,25 @@
.getRows());
pipeline.run();
}
+
+ @Test
+ public void testNodeStatsEstimation() {
+ String sql =
+ "SELECT * FROM (VALUES ('value1'),('value2'),('value3'),('value4'),('value5'),"
+ + " ('value6'),('value7'),('value8'),('value9'))";
+
+ RelNode root = env.parseQuery(sql);
+
+ while (!(root instanceof BeamValuesRel)) {
+ root = root.getInput(0);
+ }
+
+ NodeStats estimate = BeamSqlRelUtils.getNodeStats(root, root.getCluster().getMetadataQuery());
+
+ Assert.assertFalse(estimate.isUnknown());
+ Assert.assertEquals(0d, estimate.getRate(), 0.01);
+
+ Assert.assertEquals(9., estimate.getRowCount(), 0.01);
+ Assert.assertEquals(9., estimate.getWindow(), 0.01);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java
new file mode 100644
index 0000000..2d0a1be
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java
@@ -0,0 +1,464 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.rule;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.DataContext;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableConvention;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.EnumerableRules;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Enumerable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.Linq4j;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollations;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Join;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.TableScan;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.JoinCommuteRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.SortProjectTransposeRule;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ScannableTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Statistic;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Statistics;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.AbstractSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.impl.AbstractTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParser;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Planner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Programs;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSets;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * This test ensures that we are reordering joins and get a plan similar to Join(large,Join(small,
+ * medium)) instead of Join(small, Join(medium,large).
+ */
+public class JoinReorderingTest {
+ private final PipelineOptions defaultPipelineOptions = PipelineOptionsFactory.create();
+
+ @Test
+ public void testTableSizes() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ createThreeTables(tableProvider);
+
+ Assert.assertEquals(
+ 1d,
+ tableProvider
+ .buildBeamSqlTable(tableProvider.getTable("small_table"))
+ .getTableStatistics(null)
+ .getRowCount(),
+ 0.01);
+
+ Assert.assertEquals(
+ 3d,
+ tableProvider
+ .buildBeamSqlTable(tableProvider.getTable("medium_table"))
+ .getTableStatistics(null)
+ .getRowCount(),
+ 0.01);
+
+ Assert.assertEquals(
+ 100d,
+ tableProvider
+ .buildBeamSqlTable(tableProvider.getTable("large_table"))
+ .getTableStatistics(null)
+ .getRowCount(),
+ 0.01);
+ }
+
+ @Test
+ public void testBeamJoinAssociationRule() throws Exception {
+ RuleSet prepareRules =
+ RuleSets.ofList(
+ SortProjectTransposeRule.INSTANCE,
+ EnumerableRules.ENUMERABLE_JOIN_RULE,
+ EnumerableRules.ENUMERABLE_PROJECT_RULE,
+ EnumerableRules.ENUMERABLE_SORT_RULE,
+ EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
+
+ String sqlQuery =
+ "select * from \"tt\".\"large_table\" as large_table "
+ + " JOIN \"tt\".\"medium_table\" as medium_table on large_table.\"medium_key\" = medium_table.\"large_key\" "
+ + " JOIN \"tt\".\"small_table\" as small_table on medium_table.\"small_key\" = small_table.\"medium_key\" ";
+
+ RelNode originalPlan = transform(sqlQuery, prepareRules);
+ RelNode optimizedPlan =
+ transform(
+ sqlQuery,
+ RuleSets.ofList(
+ ImmutableList.<RelOptRule>builder()
+ .addAll(prepareRules)
+ .add(BeamJoinAssociateRule.INSTANCE)
+ .build()));
+
+ assertTopTableInJoins(originalPlan, "small_table");
+ assertTopTableInJoins(optimizedPlan, "large_table");
+ }
+
+ @Test
+ public void testBeamJoinPushThroughJoinRuleLeft() throws Exception {
+ RuleSet prepareRules =
+ RuleSets.ofList(
+ SortProjectTransposeRule.INSTANCE,
+ EnumerableRules.ENUMERABLE_JOIN_RULE,
+ EnumerableRules.ENUMERABLE_PROJECT_RULE,
+ EnumerableRules.ENUMERABLE_SORT_RULE,
+ EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
+
+ String sqlQuery =
+ "select * from \"tt\".\"large_table\" as large_table "
+ + " JOIN \"tt\".\"medium_table\" as medium_table on large_table.\"medium_key\" = medium_table.\"large_key\" "
+ + " JOIN \"tt\".\"small_table\" as small_table on medium_table.\"small_key\" = small_table.\"medium_key\" ";
+
+ RelNode originalPlan = transform(sqlQuery, prepareRules);
+ RelNode optimizedPlan =
+ transform(
+ sqlQuery,
+ RuleSets.ofList(
+ ImmutableList.<RelOptRule>builder()
+ .addAll(prepareRules)
+ .add(BeamJoinPushThroughJoinRule.LEFT)
+ .build()));
+
+ assertTopTableInJoins(originalPlan, "small_table");
+ assertTopTableInJoins(optimizedPlan, "large_table");
+ }
+
+ @Test
+ public void testBeamJoinPushThroughJoinRuleRight() throws Exception {
+ RuleSet prepareRules =
+ RuleSets.ofList(
+ SortProjectTransposeRule.INSTANCE,
+ EnumerableRules.ENUMERABLE_JOIN_RULE,
+ EnumerableRules.ENUMERABLE_PROJECT_RULE,
+ EnumerableRules.ENUMERABLE_SORT_RULE,
+ EnumerableRules.ENUMERABLE_TABLE_SCAN_RULE);
+
+ String sqlQuery =
+ "select * from \"tt\".\"medium_table\" as medium_table "
+ + " JOIN \"tt\".\"large_table\" as large_table on large_table.\"medium_key\" = medium_table.\"large_key\" "
+ + " JOIN \"tt\".\"small_table\" as small_table on medium_table.\"small_key\" = small_table.\"medium_key\" ";
+
+ RelNode originalPlan = transform(sqlQuery, prepareRules);
+ RelNode optimizedPlan =
+ transform(
+ sqlQuery,
+ RuleSets.ofList(
+ ImmutableList.<RelOptRule>builder()
+ .addAll(prepareRules)
+ .add(BeamJoinPushThroughJoinRule.RIGHT)
+ .build()));
+
+ assertTopTableInJoins(originalPlan, "small_table");
+ assertTopTableInJoins(optimizedPlan, "large_table");
+ }
+
+ @Test
+ public void testSystemReorderingLargeMediumSmall() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ createThreeTables(tableProvider);
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ // This is Join(Join(large, medium), small) which should be converted to a join that large table
+ // is on the top.
+ BeamRelNode parsedQuery =
+ env.parseQuery(
+ "select * from large_table "
+ + " JOIN medium_table on large_table.medium_key = medium_table.large_key "
+ + " JOIN small_table on medium_table.small_key = small_table.medium_key ");
+ assertTopTableInJoins(parsedQuery, "large_table");
+ }
+
+ @Test
+ public void testSystemReorderingMediumLargeSmall() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ createThreeTables(tableProvider);
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ // This is Join(Join(medium, large), small) which should be converted to a join that large table
+ // is on the top.
+ BeamRelNode parsedQuery =
+ env.parseQuery(
+ "select * from medium_table "
+ + " JOIN large_table on large_table.medium_key = medium_table.large_key "
+ + " JOIN small_table on medium_table.small_key = small_table.medium_key ");
+ assertTopTableInJoins(parsedQuery, "large_table");
+ }
+
+ @Test
+ public void testSystemNotReorderingWithoutRules() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ createThreeTables(tableProvider);
+ List<RelOptRule> ruleSet =
+ Arrays.stream(BeamRuleSets.getRuleSets())
+ .flatMap(rules -> StreamSupport.stream(rules.spliterator(), false))
+ .filter(rule -> !(rule instanceof BeamJoinPushThroughJoinRule))
+ .filter(rule -> !(rule instanceof BeamJoinAssociateRule))
+ .filter(rule -> !(rule instanceof JoinCommuteRule))
+ .collect(Collectors.toList());
+
+ BeamSqlEnv env =
+ BeamSqlEnv.builder(tableProvider)
+ .setPipelineOptions(PipelineOptionsFactory.create())
+ .setRuleSets(new RuleSet[] {RuleSets.ofList(ruleSet)})
+ .build();
+
+ // This is Join(Join(medium, large), small) which should be converted to a join that large table
+ // is on the top.
+ BeamRelNode parsedQuery =
+ env.parseQuery(
+ "select * from medium_table "
+ + " JOIN large_table on large_table.medium_key = medium_table.large_key "
+ + " JOIN small_table on medium_table.small_key = small_table.medium_key ");
+ assertTopTableInJoins(parsedQuery, "small_table");
+ }
+
+ @Test
+ public void testSystemNotReorderingMediumSmallLarge() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ createThreeTables(tableProvider);
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ // This is a correct ordered join because large table is on the top. It should not change that.
+ BeamRelNode parsedQuery =
+ env.parseQuery(
+ "select * from medium_table "
+ + " JOIN small_table on medium_table.small_key = small_table.medium_key "
+ + " JOIN large_table on large_table.medium_key = medium_table.large_key ");
+ assertTopTableInJoins(parsedQuery, "large_table");
+ }
+
+ @Test
+ public void testSystemNotReorderingSmallMediumLarge() {
+ TestTableProvider tableProvider = new TestTableProvider();
+ createThreeTables(tableProvider);
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+
+ // This is a correct ordered join because large table is on the top. It should not change that.
+ BeamRelNode parsedQuery =
+ env.parseQuery(
+ "select * from small_table "
+ + " JOIN medium_table on medium_table.small_key = small_table.medium_key "
+ + " JOIN large_table on large_table.medium_key = medium_table.large_key ");
+ assertTopTableInJoins(parsedQuery, "large_table");
+ }
+
+ private RelNode transform(String sql, RuleSet prepareRules) throws Exception {
+ final SchemaPlus rootSchema = Frameworks.createRootSchema(true);
+ final SchemaPlus defSchema = rootSchema.add("tt", new ThreeTablesSchema());
+ final FrameworkConfig config =
+ Frameworks.newConfigBuilder()
+ .parserConfig(SqlParser.Config.DEFAULT)
+ .defaultSchema(defSchema)
+ .traitDefs(ConventionTraitDef.INSTANCE, RelCollationTraitDef.INSTANCE)
+ .programs(Programs.of(prepareRules))
+ .build();
+ Planner planner = Frameworks.getPlanner(config);
+ SqlNode parse = planner.parse(sql);
+ SqlNode validate = planner.validate(parse);
+ RelRoot planRoot = planner.rel(validate);
+ RelNode planBefore = planRoot.rel;
+ RelTraitSet desiredTraits = planBefore.getTraitSet().replace(EnumerableConvention.INSTANCE);
+ return planner.transform(0, desiredTraits, planBefore);
+ }
+
+ private void assertTopTableInJoins(RelNode parsedQuery, String expectedTableName) {
+ RelNode firstJoin = parsedQuery;
+ while (!(firstJoin instanceof Join)) {
+ firstJoin = firstJoin.getInput(0);
+ }
+
+ RelNode topRight = ((Join) firstJoin).getRight();
+ while (!(topRight instanceof Join) && !(topRight instanceof TableScan)) {
+ topRight = topRight.getInput(0);
+ }
+
+ if (topRight instanceof TableScan) {
+ Assert.assertTrue(topRight.getDescription().contains(expectedTableName));
+ } else {
+ RelNode topLeft = ((Join) firstJoin).getLeft();
+ while (!(topLeft instanceof TableScan)) {
+ topLeft = topLeft.getInput(0);
+ }
+
+ Assert.assertTrue(topLeft.getDescription().contains(expectedTableName));
+ }
+ }
+
+ private void createThreeTables(TestTableProvider tableProvider) {
+ BeamSqlEnv env = BeamSqlEnv.withTableProvider(tableProvider);
+ env.executeDdl("CREATE EXTERNAL TABLE small_table (id INTEGER, medium_key INTEGER) TYPE text");
+
+ env.executeDdl(
+ "CREATE EXTERNAL TABLE medium_table ("
+ + "id INTEGER,"
+ + "small_key INTEGER,"
+ + "large_key INTEGER"
+ + ") TYPE text");
+
+ env.executeDdl(
+ "CREATE EXTERNAL TABLE large_table ("
+ + "id INTEGER,"
+ + "medium_key INTEGER"
+ + ") TYPE text");
+
+ Row row =
+ Row.withSchema(tableProvider.getTable("small_table").getSchema()).addValues(1, 1).build();
+ tableProvider.addRows("small_table", row);
+
+ for (int i = 0; i < 3; i++) {
+ row =
+ Row.withSchema(tableProvider.getTable("medium_table").getSchema())
+ .addValues(i, 1, 2)
+ .build();
+ tableProvider.addRows("medium_table", row);
+ }
+
+ for (int i = 0; i < 100; i++) {
+ row =
+ Row.withSchema(tableProvider.getTable("large_table").getSchema()).addValues(i, 2).build();
+ tableProvider.addRows("large_table", row);
+ }
+ }
+}
+
+final class ThreeTablesSchema extends AbstractSchema {
+
+ private final ImmutableMap<String, Table> tables;
+
+ public ThreeTablesSchema() {
+ super();
+ ArrayList<Object[]> mediumData = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ mediumData.add(new Object[] {i, 1, 2});
+ }
+
+ ArrayList<Object[]> largeData = new ArrayList<>();
+ for (int i = 0; i < 100; i++) {
+ largeData.add(new Object[] {i, 2});
+ }
+
+ tables =
+ ImmutableMap.<String, Table>builder()
+ .put(
+ "small_table",
+ new PkClusteredTable(
+ factory ->
+ new RelDataTypeFactory.Builder(factory)
+ .add("id", factory.createJavaType(int.class))
+ .add("medium_key", factory.createJavaType(int.class))
+ .build(),
+ ImmutableBitSet.of(0),
+ Arrays.asList(new Object[] {1, 1}, new Object[] {2, 1})))
+ .put(
+ "medium_table",
+ new PkClusteredTable(
+ factory ->
+ new RelDataTypeFactory.Builder(factory)
+ .add("id", factory.createJavaType(int.class))
+ .add("small_key", factory.createJavaType(int.class))
+ .add("large_key", factory.createJavaType(int.class))
+ .build(),
+ ImmutableBitSet.of(0),
+ mediumData))
+ .put(
+ "large_table",
+ new PkClusteredTable(
+ factory ->
+ new RelDataTypeFactory.Builder(factory)
+ .add("id", factory.createJavaType(int.class))
+ .add("medium_key", factory.createJavaType(int.class))
+ .build(),
+ ImmutableBitSet.of(0),
+ largeData))
+ .build();
+ }
+
+ @Override
+ protected Map<String, org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table>
+ getTableMap() {
+ return tables;
+ }
+
+ /** A table sorted (ascending direction and nulls last) on the primary key. */
+ private static class PkClusteredTable extends AbstractTable implements ScannableTable {
+ private final ImmutableBitSet pkColumns;
+ private final List<Object[]> data;
+ private final java.util.function.Function<RelDataTypeFactory, RelDataType> typeBuilder;
+
+ PkClusteredTable(
+ Function<RelDataTypeFactory, RelDataType> dataTypeBuilder,
+ ImmutableBitSet pkColumns,
+ List<Object[]> data) {
+ this.data = data;
+ this.typeBuilder = dataTypeBuilder;
+ this.pkColumns = pkColumns;
+ }
+
+ @Override
+ public Statistic getStatistic() {
+ List<RelFieldCollation> collationFields = new ArrayList<>();
+ for (Integer key : pkColumns) {
+ collationFields.add(
+ new RelFieldCollation(
+ key, RelFieldCollation.Direction.ASCENDING, RelFieldCollation.NullDirection.LAST));
+ }
+ return Statistics.of(
+ data.size(),
+ ImmutableList.of(pkColumns),
+ ImmutableList.of(RelCollations.of(collationFields)));
+ }
+
+ @Override
+ public RelDataType getRowType(final RelDataTypeFactory typeFactory) {
+ return typeBuilder.apply(typeFactory);
+ }
+
+ @Override
+ public Enumerable<Object[]> scan(final DataContext root) {
+ return Linq4j.asEnumerable(data);
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java
index 6f14065..3624d127 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamSqlRowCoderTest.java
@@ -23,12 +23,11 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.testing.CoderProperties;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.joda.time.DateTime;
import org.junit.Test;
@@ -70,9 +69,7 @@
DateTime.now(),
true)
.build();
- Coder<Row> coder =
- SchemaCoder.of(
- beamSchema, SerializableFunctions.identity(), SerializableFunctions.identity());
+ Coder<Row> coder = SchemaCoder.of(beamSchema);
CoderProperties.coderDecodeEncodeEqual(coder, row);
}
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java
index 02349a5..50b6ab2 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtilsTest.java
@@ -24,11 +24,11 @@
import java.util.Map;
import java.util.stream.Collectors;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.calcite.rel.type.RelDataType;
-import org.apache.calcite.rel.type.RelDataTypeFactory;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.type.SqlTypeFactoryImpl;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
import org.junit.Before;
import org.junit.Test;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
index 22430ba..f025ee5 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlBuiltinFunctionsIntegrationTestBase.java
@@ -19,7 +19,7 @@
import static org.apache.beam.sdk.extensions.sql.utils.DateTimeUtils.parseTimestampWithUTCTimeZone;
import static org.apache.beam.sdk.extensions.sql.utils.RowAsserts.matchesScalar;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkArgument;
import static org.junit.Assert.assertTrue;
import com.google.auto.value.AutoValue;
@@ -37,6 +37,7 @@
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.schemas.Schema.TypeName;
@@ -45,14 +46,13 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterables;
import org.joda.time.DateTime;
import org.junit.Rule;
@@ -331,7 +331,7 @@
*/
public void check(Pipeline pipeline) throws Exception {
checkPTransform(pipeline);
- checkJdbc();
+ checkJdbc(pipeline.getOptions());
}
private static final Schema DUMMY_SCHEMA = Schema.builder().addBooleanField("dummy").build();
@@ -353,7 +353,7 @@
Schema.FieldType.STRING, "name")
.addRows(1, "first")));
- private void checkJdbc() throws Exception {
+ private void checkJdbc(PipelineOptions pipelineOptions) throws Exception {
// Beam SQL code is only invoked when the calling convention insists on it, so we
// have to express this as selecting from a Beam table, even though the contents are
// irrelevant.
@@ -363,7 +363,7 @@
//
// Here we create a Beam table just to force the calling convention.
TestTableProvider tableProvider = new TestTableProvider();
- Connection connection = JdbcDriver.connect(tableProvider);
+ Connection connection = JdbcDriver.connect(tableProvider, pipelineOptions);
connection
.createStatement()
.executeUpdate("CREATE EXTERNAL TABLE dummy (dummy BOOLEAN) TYPE 'test'");
@@ -390,12 +390,7 @@
public PDone expand(PBegin begin) {
PCollection<Boolean> result =
begin
- .apply(
- Create.of(DUMMY_ROW)
- .withSchema(
- DUMMY_SCHEMA,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()))
+ .apply(Create.of(DUMMY_ROW).withRowSchema(DUMMY_SCHEMA))
.apply(SqlTransform.query("SELECT " + expr))
.apply(MapElements.into(TypeDescriptors.booleans()).via(row -> row.getBoolean(0)));
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
index dd2a1db..0342cf0 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/integrationtest/BeamSqlDateFunctionsIntegrationTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.integrationtest;
-import static org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.DateTimeUtils.MILLIS_PER_DAY;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
new file mode 100644
index 0000000..b416a3f
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta;
+
+import static org.junit.Assert.assertThrows;
+
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.extensions.sql.impl.TableName;
+import org.apache.beam.sdk.extensions.sql.meta.provider.FullNameTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Test for custom table resolver and full name table provider. */
+public class CustomTableResolverTest implements Serializable {
+
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+
+ private static final Schema BASIC_SCHEMA =
+ Schema.builder().addInt32Field("id").addStringField("name").build();
+
+ /**
+ * Test table provider with custom name resolution.
+ *
+ * <p>Demonstrates how to parse table names as in normal Calcite queries syntax, e.g. {@code
+ * a.b.c.d} and convert them to its' own custom table name format {@code a_b_c_d}.
+ */
+ public static class CustomResolutionTestTableProvider extends FullNameTableProvider {
+
+ TestTableProvider delegateTableProvider;
+
+ public CustomResolutionTestTableProvider() {
+ delegateTableProvider = new TestTableProvider();
+ }
+
+ @Override
+ public Table getTable(String tableName) {
+ return delegateTableProvider.getTable(tableName);
+ }
+
+ @Override
+ public Table getTableByFullName(TableName fullTableName) {
+ // For the test we register tables with underscore instead of dots, so here we lookup the
+ // tables
+ // with those underscore.
+ String actualTableName =
+ String.join("_", fullTableName.getPath()) + "_" + fullTableName.getTableName();
+ return delegateTableProvider.getTable(actualTableName);
+ }
+
+ @Override
+ public String getTableType() {
+ return delegateTableProvider.getTableType();
+ }
+
+ @Override
+ public void createTable(Table table) {
+ delegateTableProvider.createTable(table);
+ }
+
+ public void addRows(String tableName, Row... rows) {
+ delegateTableProvider.addRows(tableName, rows);
+ }
+
+ @Override
+ public void dropTable(String tableName) {
+ delegateTableProvider.dropTable(tableName);
+ }
+
+ @Override
+ public Map<String, Table> getTables() {
+ return delegateTableProvider.getTables();
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return delegateTableProvider.buildBeamSqlTable(table);
+ }
+ }
+
+ @Test
+ public void testSimpleId() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable", row(1, "one"), row(2, "two"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testtable")
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testDefaultBuildIOReader_withEmptyParams_returnsPCollection() {
+ TestBoundedTable testTable = TestBoundedTable.of(BASIC_SCHEMA).addRows(1, "one");
+ Row expected = row(1, "one");
+
+ PCollection<Row> resultWithEmpty =
+ testTable.buildIOReader(
+ pipeline.begin(), testTable.constructFilter(ImmutableList.of()), ImmutableList.of());
+
+ PAssert.that(resultWithEmpty).containsInAnyOrder(expected);
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testDefaultBuildIOReader_withNonEmptyParams_throwsException() {
+ TestBoundedTable testTable = TestBoundedTable.of(BASIC_SCHEMA).addRows(1, "one");
+
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> testTable.buildIOReader(pipeline.begin(), () -> null, ImmutableList.of()));
+ assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ testTable.buildIOReader(
+ pipeline.begin(),
+ new DefaultTableFilter(ImmutableList.of()),
+ ImmutableList.of("one")));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testSimpleIdWithExplicitDefaultSchema() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable", row(1, "one"), row(2, "two"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider.testtable")
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testSimpleIdWithExplicitDefaultSchemaWithMultipleProviders() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable", row(1, "one"), row(2, "two"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable2").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable2", row(3, "three"), row(4, "four"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider2.testtable2")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(3, "three"), row(4, "four"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testSimpleIdWithExplicitNonDefaultSchema() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable", row(1, "one"), row(2, "two"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable2").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable2", row(3, "three"), row(4, "four"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider2.testtable2")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(3, "three"), row(4, "four"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testCompoundIdInDefaultSchema() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah", row(1, "one"), row(2, "two"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testtable.blah")
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testCompoundIdInExplicitDefaultSchema() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah", row(1, "one"), row(2, "two"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider.testtable.blah")
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testLongCompoundIdInDefaultSchema() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(1, "one"), row(2, "two"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testtable.blah.foo.bar")
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testLongCompoundIdInDefaultSchemaWithMultipleProviders() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(1, "one"), row(2, "two"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable_blah_foo_bar", row(3, "three"), row(4, "four"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testtable.blah.foo.bar")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testLongCompoundIdInExplicitDefaultSchema() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(1, "one"), row(2, "two"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider.testtable.blah.foo.bar")
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(1, "one"), row(2, "two"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testLongCompoundIdInNonDefaultSchemaSameTableNames() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(1, "one"), row(2, "two"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable_blah_foo_bar", row(3, "three"), row(4, "four"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider2.testtable.blah.foo.bar")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(3, "three"), row(4, "four"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testLongCompoundIdInNonDefaultSchemaDifferentNames() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(1, "one"), row(2, "two"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder()
+ .name("testtable2_blah2_foo2_bar2")
+ .schema(BASIC_SCHEMA)
+ .type("test")
+ .build());
+ tableProvider2.addRows("testtable2_blah2_foo2_bar2", row(3, "three"), row(4, "four"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query("SELECT id, name FROM testprovider2.testtable2.blah2.foo2.bar2")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(3, "three"), row(4, "four"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testJoinWithLongCompoundIds() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(3, "customer"), row(2, "nobody"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable_blah_foo_bar2").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable_blah_foo_bar2", row(4, "customer"), row(1, "nobody"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query(
+ "SELECT testprovider2.testtable.blah.foo.bar2.id, testtable.blah.foo.bar.name \n"
+ + "FROM \n"
+ + " testprovider2.testtable.blah.foo.bar2 \n"
+ + "JOIN \n"
+ + " testtable.blah.foo.bar \n"
+ + "USING(name)")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(4, "customer"), row(1, "nobody"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testInnerJoinWithLongCompoundIds() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(3, "customer"), row(2, "nobody"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable_blah_foo_bar2").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable_blah_foo_bar2", row(4, "customer"), row(1, "nobody"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query(
+ "SELECT testprovider2.testtable.blah.foo.bar2.id, testtable.blah.foo.bar.name \n"
+ + "FROM \n"
+ + " testprovider2.testtable.blah.foo.bar2 \n"
+ + "JOIN \n"
+ + " testtable.blah.foo.bar \n"
+ + "USING(name)")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(4, "customer"), row(1, "nobody"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testJoinWithLongCompoundIdsWithAliases() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(3, "customer"), row(2, "nobody"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable_blah_foo_bar2").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable_blah_foo_bar2", row(4, "customer"), row(1, "nobody"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query(
+ "SELECT b.id, a.name \n"
+ + "FROM \n"
+ + " testprovider2.testtable.blah.foo.bar2 AS b \n"
+ + "JOIN \n"
+ + " testtable.blah.foo.bar a\n"
+ + "USING(name)")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result).containsInAnyOrder(row(4, "customer"), row(1, "nobody"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ @Test
+ public void testUnionWithLongCompoundIds() throws Exception {
+ CustomResolutionTestTableProvider tableProvider = new CustomResolutionTestTableProvider();
+ tableProvider.createTable(
+ Table.builder().name("testtable_blah_foo_bar").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider.addRows("testtable_blah_foo_bar", row(3, "customer"), row(2, "nobody"));
+
+ CustomResolutionTestTableProvider tableProvider2 = new CustomResolutionTestTableProvider();
+ tableProvider2.createTable(
+ Table.builder().name("testtable_blah_foo_bar2").schema(BASIC_SCHEMA).type("test").build());
+ tableProvider2.addRows("testtable_blah_foo_bar2", row(4, "customer"), row(1, "nobody"));
+
+ PCollection<Row> result =
+ pipeline.apply(
+ SqlTransform.query(
+ "SELECT id, name \n"
+ + "FROM \n"
+ + " testprovider2.testtable.blah.foo.bar2 \n"
+ + "UNION \n"
+ + " SELECT id, name \n"
+ + " FROM \n"
+ + " testtable.blah.foo.bar \n")
+ .withTableProvider("testprovider2", tableProvider2)
+ .withDefaultTableProvider("testprovider", tableProvider));
+
+ PAssert.that(result)
+ .containsInAnyOrder(
+ row(4, "customer"), row(1, "nobody"), row(3, "customer"), row(2, "nobody"));
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(2));
+ }
+
+ private Row row(int id, String name) {
+ return Row.withSchema(BASIC_SCHEMA).addValues(id, name).build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java
new file mode 100644
index 0000000..60bd6f7
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.avro;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test for AvroTable. */
+@RunWith(JUnit4.class)
+public class AvroTableProviderTest {
+ @Rule public TestPipeline writePipeline = TestPipeline.create();
+ @Rule public TestPipeline readPipeline = TestPipeline.create();
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ private static final String AVRO_FIELD_NAMES = "(name VARCHAR, age BIGINT, country VARCHAR)";
+
+ private static final Schema OUTPUT_ROW_SCHEMA =
+ Schema.builder().addInt64Field("age").addStringField("country").build();
+
+ @Test
+ public void testReadAndWriteAvroTable() {
+ File destinationFile = new File(tempFolder.getRoot(), "person-info.avro");
+
+ BeamSqlEnv env = BeamSqlEnv.inMemory(new AvroTableProvider());
+ env.executeDdl(
+ String.format(
+ "CREATE EXTERNAL TABLE PersonInfo %s TYPE avro LOCATION '%s'",
+ AVRO_FIELD_NAMES, destinationFile.getAbsolutePath()));
+
+ BeamSqlRelUtils.toPCollection(
+ writePipeline,
+ env.parseQuery(
+ "INSERT INTO PersonInfo VALUES ('Alan', 22, 'England'), ('John', 42, 'USA')"));
+
+ writePipeline.run().waitUntilFinish();
+
+ PCollection<Row> rows =
+ BeamSqlRelUtils.toPCollection(
+ readPipeline, env.parseQuery("SELECT age, country FROM PersonInfo where age > 25"));
+
+ PAssert.that(rows)
+ .containsInAnyOrder(Row.withSchema(OUTPUT_ROW_SCHEMA).addValues(42L, "USA").build());
+
+ PipelineResult.State state = readPipeline.run().waitUntilFinish();
+ assertEquals(state, State.DONE);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
index 2e25626..2c00edb 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+import static org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTable.METHOD_PROPERTY;
import static org.apache.beam.sdk.extensions.sql.utils.DateTimeUtils.parseTimestampWithUTCTimeZone;
import static org.apache.beam.sdk.schemas.Schema.FieldType.BOOLEAN;
import static org.apache.beam.sdk.schemas.Schema.FieldType.BYTE;
@@ -42,16 +43,16 @@
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.io.gcp.bigquery.TestBigQuery;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
@@ -155,6 +156,150 @@
}
@Test
+ public void testSQLRead_withExport() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "' \n"
+ + "TBLPROPERTIES "
+ + "'{ "
+ + METHOD_PROPERTY
+ + ": \""
+ + Method.EXPORT.toString()
+ + "\" }'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "TIMESTAMP '2018-05-28 20:17:40.123', "
+ + "'varchar', "
+ + "'char', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery(insertStatement));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ String selectTableStatement = "SELECT * FROM TEST";
+ PCollection<Row> output =
+ BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery(selectTableStatement));
+
+ PAssert.that(output)
+ .containsInAnyOrder(
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456")));
+ PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+ assertEquals(state, State.DONE);
+ }
+
+ @Test
+ public void testSQLRead_withDirectRead() {
+ BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
+
+ String createTableStatement =
+ "CREATE EXTERNAL TABLE TEST( \n"
+ + " c_bigint BIGINT, \n"
+ + " c_tinyint TINYINT, \n"
+ + " c_smallint SMALLINT, \n"
+ + " c_integer INTEGER, \n"
+ + " c_float FLOAT, \n"
+ + " c_double DOUBLE, \n"
+ + " c_boolean BOOLEAN, \n"
+ + " c_timestamp TIMESTAMP, \n"
+ + " c_varchar VARCHAR, \n "
+ + " c_char CHAR, \n"
+ + " c_arr ARRAY<VARCHAR> \n"
+ + ") \n"
+ + "TYPE 'bigquery' \n"
+ + "LOCATION '"
+ + bigQueryTestingTypes.tableSpec()
+ + "' \n"
+ + "TBLPROPERTIES "
+ + "'{ "
+ + METHOD_PROPERTY
+ + ": \""
+ + Method.DIRECT_READ.toString()
+ + "\" }'";
+ sqlEnv.executeDdl(createTableStatement);
+
+ String insertStatement =
+ "INSERT INTO TEST VALUES ("
+ + "9223372036854775807, "
+ + "127, "
+ + "32767, "
+ + "2147483647, "
+ + "1.0, "
+ + "1.0, "
+ + "TRUE, "
+ + "TIMESTAMP '2018-05-28 20:17:40.123', "
+ + "'varchar', "
+ + "'char', "
+ + "ARRAY['123', '456']"
+ + ")";
+
+ sqlEnv.parseQuery(insertStatement);
+ BeamSqlRelUtils.toPCollection(pipeline, sqlEnv.parseQuery(insertStatement));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+
+ String selectTableStatement = "SELECT * FROM TEST";
+ PCollection<Row> output =
+ BeamSqlRelUtils.toPCollection(readPipeline, sqlEnv.parseQuery(selectTableStatement));
+
+ PAssert.that(output)
+ .containsInAnyOrder(
+ row(
+ SOURCE_SCHEMA_TWO,
+ 9223372036854775807L,
+ (byte) 127,
+ (short) 32767,
+ 2147483647,
+ (float) 1.0,
+ 1.0,
+ true,
+ parseTimestampWithUTCTimeZone("2018-05-28 20:17:40.123"),
+ "varchar",
+ "char",
+ Arrays.asList("123", "456")));
+ PipelineResult.State state = readPipeline.run().waitUntilFinish(Duration.standardMinutes(5));
+ assertEquals(state, State.DONE);
+ }
+
+ @Test
public void testSQLTypes() {
BeamSqlEnv sqlEnv = BeamSqlEnv.inMemory(new BigQueryTableProvider());
@@ -267,10 +412,7 @@
}
private PCollection<Row> createPCollection(Pipeline pipeline, Row... rows) {
- return pipeline.apply(
- Create.of(Arrays.asList(rows))
- .withSchema(
- SOURCE_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()));
+ return pipeline.apply(Create.of(Arrays.asList(rows)).withRowSchema(SOURCE_SCHEMA));
}
private Row row(Schema schema, Object... values) {
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
new file mode 100644
index 0000000..bd6d9ae
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT64;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.STRING;
+import static org.apache.beam.sdk.schemas.Schema.toSchema;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.util.stream.Stream;
+import org.apache.beam.sdk.extensions.sql.SqlTransform;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
+import org.apache.beam.sdk.io.gcp.bigquery.TestBigQuery;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration tests form writing to BigQuery with Beam SQL. */
+@RunWith(JUnit4.class)
+public class BigQueryRowCountIT {
+ private static final Schema SOURCE_SCHEMA =
+ Schema.builder().addNullableField("id", INT64).addNullableField("name", STRING).build();
+ private static final String FAKE_JOB_NAME = "testPipelineOptionInjectionFakeJobName";
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public transient TestPipeline readingPipeline = TestPipeline.create();
+ @Rule public transient TestBigQuery bigQuery = TestBigQuery.create(SOURCE_SCHEMA);
+
+ @Test
+ public void testEmptyTable() {
+ BigQueryTableProvider provider = new BigQueryTableProvider();
+ Table table = getTable("testTable", bigQuery.tableSpec());
+ BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
+ BeamTableStatistics size = sqlTable.getTableStatistics(TestPipeline.testingPipelineOptions());
+ assertNotNull(size);
+ assertEquals(0d, size.getRowCount(), 0.1);
+ }
+
+ @Test
+ public void testNonEmptyTable() {
+ BigQueryTableProvider provider = new BigQueryTableProvider();
+ Table table = getTable("testTable", bigQuery.tableSpec());
+
+ pipeline
+ .apply(
+ Create.of(
+ new TableRow().set("id", 1).set("name", "name1"),
+ new TableRow().set("id", 2).set("name", "name2"),
+ new TableRow().set("id", 3).set("name", "name3"))
+ .withCoder(TableRowJsonCoder.of()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(bigQuery.tableSpec())
+ .withSchema(
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema().setName("id").setType("INTEGER"),
+ new TableFieldSchema().setName("name").setType("STRING"))))
+ .withoutValidation());
+ pipeline.run().waitUntilFinish();
+
+ BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
+ BeamTableStatistics size1 = sqlTable.getTableStatistics(TestPipeline.testingPipelineOptions());
+
+ assertNotNull(size1);
+ assertEquals(3d, size1.getRowCount(), 0.1);
+ }
+
+ /** This tests if the pipeline options are injected in the path of SQL Transform. */
+ @Test
+ public void testPipelineOptionInjection() {
+ BigQueryTestTableProvider provider = new BigQueryTestTableProvider();
+ Table table = getTable("testTable", bigQuery.tableSpec());
+ provider.addTable("testTable", table);
+
+ pipeline
+ .apply(
+ Create.of(
+ new TableRow().set("id", 1).set("name", "name1"),
+ new TableRow().set("id", 2).set("name", "name2"),
+ new TableRow().set("id", 3).set("name", "name3"))
+ .withCoder(TableRowJsonCoder.of()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(bigQuery.tableSpec())
+ .withSchema(
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema().setName("id").setType("INTEGER"),
+ new TableFieldSchema().setName("name").setType("STRING"))))
+ .withoutValidation());
+ pipeline.run().waitUntilFinish();
+
+ // changing pipeline options
+ readingPipeline.getOptions().setJobName(FAKE_JOB_NAME);
+
+ // Reading from the table should update the statistics of bigQuery table
+ readingPipeline.apply(
+ SqlTransform.query(" select * from testTable ")
+ .withDefaultTableProvider("bigquery", provider));
+
+ readingPipeline.run().waitUntilFinish();
+
+ BigQueryTestTable sqlTable = (BigQueryTestTable) provider.buildBeamSqlTable(table);
+ assertEquals(FAKE_JOB_NAME, sqlTable.getJobName());
+ }
+
+ @Test
+ public void testFakeTable() {
+ BigQueryTableProvider provider = new BigQueryTableProvider();
+ Table table = getTable("fakeTable", "project:dataset.table");
+
+ BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
+ BeamTableStatistics size = sqlTable.getTableStatistics(TestPipeline.testingPipelineOptions());
+ assertTrue(size.isUnknown());
+ }
+
+ private static Table getTable(String name, String location) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .location(location)
+ .schema(
+ Stream.of(Schema.Field.nullable("id", INT64), Schema.Field.nullable("name", STRING))
+ .collect(toSchema()))
+ .type("bigquery")
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
index 47983e2..34d0e83 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
@@ -17,14 +17,18 @@
*/
package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+import static org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTable.METHOD_PROPERTY;
import static org.apache.beam.sdk.schemas.Schema.toSchema;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
import static org.junit.Assert.assertTrue;
+import com.alibaba.fastjson.JSON;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.schemas.Schema;
import org.junit.Test;
@@ -49,6 +53,66 @@
assertEquals("project:dataset.table", bqTable.bqLocation);
}
+ @Test
+ public void testDefaultMethod_whenPropertiesAreNotSet() {
+ Table table = fakeTable("hello");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.DEFAULT, sqlTable.method);
+ }
+
+ @Test
+ public void testSelectDefaultMethodExplicitly() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + METHOD_PROPERTY + ": " + "\"" + Method.DEFAULT.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.DEFAULT, sqlTable.method);
+ }
+
+ @Test
+ public void testSelectDirectReadMethod() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + METHOD_PROPERTY + ": " + "\"" + Method.DIRECT_READ.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.DIRECT_READ, sqlTable.method);
+ }
+
+ @Test
+ public void testSelectExportMethod() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + METHOD_PROPERTY + ": " + "\"" + Method.EXPORT.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(Method.EXPORT, sqlTable.method);
+ }
+
+ @Test
+ public void testRuntimeExceptionThrown_whenAnInvalidPropertyIsSpecified() {
+ Table table = fakeTableWithProperties("hello", "{ " + METHOD_PROPERTY + ": \"blahblah\" }");
+
+ assertThrows(
+ RuntimeException.class,
+ () -> {
+ provider.buildBeamSqlTable(table);
+ });
+ }
+
+ @Test
+ public void testRuntimeExceptionThrown_whenAPropertyOfInvalidTypeIsSpecified() {
+ Table table = fakeTableWithProperties("hello", "{ " + METHOD_PROPERTY + ": 1337 }");
+
+ assertThrows(
+ RuntimeException.class,
+ () -> {
+ provider.buildBeamSqlTable(table);
+ });
+ }
+
private static Table fakeTable(String name) {
return Table.builder()
.name(name)
@@ -62,4 +126,19 @@
.type("bigquery")
.build();
}
+
+ private static Table fakeTableWithProperties(String name, String properties) {
+ return Table.builder()
+ .name(name)
+ .comment(name + " table")
+ .location("project:dataset.table")
+ .schema(
+ Stream.of(
+ Schema.Field.nullable("id", Schema.FieldType.INT32),
+ Schema.Field.nullable("name", Schema.FieldType.STRING))
+ .collect(toSchema()))
+ .type("bigquery")
+ .properties(JSON.parseObject(properties))
+ .build();
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTable.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTable.java
new file mode 100644
index 0000000..a674e40
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTable.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
+import org.apache.beam.sdk.options.PipelineOptions;
+
+/**
+ * A BigQueryTable that keeps jobName from the pipeline options whenever row count is called. It is
+ * made for {@link BigQueryRowCountIT#testPipelineOptionInjection()}
+ */
+public class BigQueryTestTable extends BigQueryTable {
+ private String jobName = null;
+
+ BigQueryTestTable(Table table, BigQueryUtils.ConversionOptions options) {
+ super(table, options);
+ }
+
+ @Override
+ public BeamTableStatistics getTableStatistics(PipelineOptions options) {
+ jobName = options.getJobName();
+ return super.getTableStatistics(options);
+ }
+
+ String getJobName() {
+ return this.jobName;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
new file mode 100644
index 0000000..4c9b016
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.bigquery;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.MoreObjects.firstNonNull;
+
+import java.util.HashMap;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
+
+/** A test table provider for BigQueryRowCountIT. */
+public class BigQueryTestTableProvider extends BigQueryTableProvider {
+
+ private Map<String, Table> tableSpecMap;
+ private Map<String, BeamSqlTable> beamSqlTableMap;
+
+ BigQueryTestTableProvider() {
+ super();
+ tableSpecMap = new HashMap<>();
+ beamSqlTableMap = new HashMap<>();
+ }
+
+ void addTable(String name, Table table) {
+ tableSpecMap.put(name, table);
+ }
+
+ @Nullable
+ @Override
+ public Table getTable(String tableName) {
+ return tableSpecMap.get(tableName);
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ BeamSqlTable t = beamSqlTableMap.get(table.getLocation());
+ if (t != null) {
+ return t;
+ }
+
+ t =
+ new BigQueryTestTable(
+ table,
+ BigQueryUtils.ConversionOptions.builder()
+ .setTruncateTimestamps(
+ firstNonNull(table.getProperties().getBoolean("truncateTimestamps"), false)
+ ? BigQueryUtils.ConversionOptions.TruncateTimestamps.TRUNCATE
+ : BigQueryUtils.ConversionOptions.TruncateTimestamps.REJECT)
+ .build());
+ beamSqlTableMap.put(table.getLocation(), t);
+
+ return t;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
index 710a1a5..ce32464 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTableTest.java
@@ -20,7 +20,13 @@
import static java.nio.charset.StandardCharsets.UTF_8;
import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestTableUtils;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -30,11 +36,13 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.adapter.java.JavaTypeFactory;
-import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
-import org.apache.calcite.rel.type.RelDataTypeSystem;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.commons.csv.CSVFormat;
+import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
@@ -46,8 +54,101 @@
private static final Row ROW2 = Row.withSchema(genSchema()).addValues(2L, 2, 2.0).build();
+ private static Map<String, BeamSqlTable> tables = new HashMap<>();
+ protected static BeamSqlEnv env = BeamSqlEnv.readOnly("test", tables);
+
@Test
- public void testCsvRecorderDecoder() throws Exception {
+ public void testOrderedArrivalSinglePartitionRate() {
+ KafkaCSVTestTable table = getTable(1);
+ for (int i = 0; i < 100; i++) {
+ table.addRecord(KafkaTestRecord.create("key1", i + ",1,2", "topic1", 500 * i));
+ }
+
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertEquals(2d, stats.getRate(), 0.001);
+ }
+
+ @Test
+ public void testOrderedArrivalMultiplePartitionsRate() {
+ KafkaCSVTestTable table = getTable(3);
+ for (int i = 0; i < 100; i++) {
+ table.addRecord(KafkaTestRecord.create("key" + i, i + ",1,2", "topic1", 500 * i));
+ }
+
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertEquals(2d, stats.getRate(), 0.001);
+ }
+
+ @Test
+ public void testOnePartitionAheadRate() {
+ KafkaCSVTestTable table = getTable(3);
+ for (int i = 0; i < 100; i++) {
+ table.addRecord(KafkaTestRecord.create("1", i + ",1,2", "topic1", 1000 * i));
+ table.addRecord(KafkaTestRecord.create("2", i + ",1,2", "topic1", 500 * i));
+ }
+
+ table.setNumberOfRecordsForRate(20);
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertEquals(1d, stats.getRate(), 0.001);
+ }
+
+ @Test
+ public void testLateRecords() {
+ KafkaCSVTestTable table = getTable(3);
+
+ table.addRecord(KafkaTestRecord.create("1", 132 + ",1,2", "topic1", 1000));
+ for (int i = 0; i < 98; i++) {
+ table.addRecord(KafkaTestRecord.create("1", i + ",1,2", "topic1", 500));
+ }
+ table.addRecord(KafkaTestRecord.create("1", 133 + ",1,2", "topic1", 2000));
+
+ table.setNumberOfRecordsForRate(200);
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertEquals(1d, stats.getRate(), 0.001);
+ }
+
+ @Test
+ public void testAllLate() {
+ KafkaCSVTestTable table = getTable(3);
+
+ table.addRecord(KafkaTestRecord.create("1", 132 + ",1,2", "topic1", 1000));
+ for (int i = 0; i < 98; i++) {
+ table.addRecord(KafkaTestRecord.create("1", i + ",1,2", "topic1", 500));
+ }
+
+ table.setNumberOfRecordsForRate(200);
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertTrue(stats.isUnknown());
+ }
+
+ @Test
+ public void testEmptyPartitionsRate() {
+ KafkaCSVTestTable table = getTable(3);
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertTrue(stats.isUnknown());
+ }
+
+ @Test
+ public void allTheRecordsSameTimeRate() {
+ KafkaCSVTestTable table = getTable(3);
+ for (int i = 0; i < 100; i++) {
+ table.addRecord(KafkaTestRecord.create("key" + i, i + ",1,2", "topic1", 1000));
+ }
+ BeamTableStatistics stats = table.getTableStatistics(null);
+ Assert.assertTrue(stats.isUnknown());
+ }
+
+ private static class PrintDoFn extends DoFn<Row, Row> {
+
+ @ProcessElement
+ public void process(ProcessContext c) {
+ System.out.println("we are here");
+ System.out.println(c.element().getValues());
+ }
+ }
+
+ @Test
+ public void testCsvRecorderDecoder() {
PCollection<Row> result =
pipeline
.apply(Create.of("1,\"1\",1.0", "2,2,2.0"))
@@ -60,7 +161,7 @@
}
@Test
- public void testCsvRecorderEncoder() throws Exception {
+ public void testCsvRecorderEncoder() {
PCollection<Row> result =
pipeline
.apply(Create.of(ROW1, ROW2))
@@ -90,4 +191,17 @@
ctx.output(KV.of(new byte[] {}, ctx.element().getBytes(UTF_8)));
}
}
+
+ private KafkaCSVTestTable getTable(int numberOfPartitions) {
+ return new KafkaCSVTestTable(
+ TestTableUtils.buildBeamSqlSchema(
+ Schema.FieldType.INT32,
+ "order_id",
+ Schema.FieldType.INT32,
+ "site_id",
+ Schema.FieldType.INT32,
+ "price"),
+ ImmutableList.of("topic1", "topic2"),
+ numberOfPartitions);
+ }
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTableIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTableIT.java
new file mode 100644
index 0000000..b6da8db
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTableIT.java
@@ -0,0 +1,294 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.kafka;
+
+import static org.apache.beam.sdk.schemas.Schema.FieldType.INT32;
+import static org.apache.beam.sdk.schemas.Schema.toSchema;
+
+import com.alibaba.fastjson.JSON;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+import javax.annotation.Nullable;
+import org.apache.beam.runners.direct.DirectOptions;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.Table;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.Validation;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.state.BagState;
+import org.apache.beam.sdk.state.StateSpec;
+import org.apache.beam.sdk.state.StateSpecs;
+import org.apache.beam.sdk.state.ValueState;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * This is an integration test for KafkaCSVTable. There should be a kafka server running and the
+ * address should be passed to it. (https://issues.apache.org/jira/projects/BEAM/issues/BEAM-7523)
+ */
+@Ignore("https://issues.apache.org/jira/projects/BEAM/issues/BEAM-7523")
+public class KafkaCSVTableIT {
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ private static final Schema TEST_TABLE_SCHEMA =
+ Schema.builder()
+ .addNullableField("order_id", Schema.FieldType.INT32)
+ .addNullableField("member_id", Schema.FieldType.INT32)
+ .addNullableField("item_name", Schema.FieldType.INT32)
+ .build();
+
+ @BeforeClass
+ public static void prepare() {
+ PipelineOptionsFactory.register(KafkaOptions.class);
+ }
+
+ @Test
+ @SuppressWarnings("FutureReturnValueIgnored")
+ public void testFake2() throws BeamKafkaTable.NoEstimationException {
+ KafkaOptions kafkaOptions = pipeline.getOptions().as(KafkaOptions.class);
+ Table table =
+ Table.builder()
+ .name("kafka_table")
+ .comment("kafka" + " table")
+ .location("")
+ .schema(
+ Stream.of(
+ Schema.Field.nullable("order_id", INT32),
+ Schema.Field.nullable("member_id", INT32),
+ Schema.Field.nullable("item_name", INT32))
+ .collect(toSchema()))
+ .type("kafka")
+ .properties(JSON.parseObject(getKafkaPropertiesString(kafkaOptions)))
+ .build();
+ BeamKafkaTable kafkaTable = (BeamKafkaTable) new KafkaTableProvider().buildBeamSqlTable(table);
+ produceSomeRecordsWithDelay(100, 20);
+ double rate1 = kafkaTable.computeRate(20);
+ produceSomeRecordsWithDelay(100, 10);
+ double rate2 = kafkaTable.computeRate(20);
+ Assert.assertTrue(rate2 > rate1);
+ }
+
+ private String getKafkaPropertiesString(KafkaOptions kafkaOptions) {
+ return "{ \"bootstrap.servers\" : \""
+ + kafkaOptions.getKafkaBootstrapServerAddress()
+ + "\",\"topics\":[\""
+ + kafkaOptions.getKafkaTopic()
+ + "\"] }";
+ }
+
+ static final transient Map<Long, Boolean> FLAG = new ConcurrentHashMap<>();
+
+ @Test
+ public void testFake() throws InterruptedException {
+ KafkaOptions kafkaOptions = pipeline.getOptions().as(KafkaOptions.class);
+ pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+ String createTableString =
+ "CREATE EXTERNAL TABLE kafka_table(\n"
+ + "order_id INTEGER, \n"
+ + "member_id INTEGER, \n"
+ + "item_name INTEGER \n"
+ + ") \n"
+ + "TYPE 'kafka' \n"
+ + "LOCATION '"
+ + "'\n"
+ + "TBLPROPERTIES '"
+ + getKafkaPropertiesString(kafkaOptions)
+ + "'";
+ TableProvider tb = new KafkaTableProvider();
+ BeamSqlEnv env = BeamSqlEnv.inMemory(tb);
+
+ env.executeDdl(createTableString);
+
+ PCollection<Row> queryOutput =
+ BeamSqlRelUtils.toPCollection(pipeline, env.parseQuery("SELECT * FROM kafka_table"));
+
+ queryOutput
+ .apply(ParDo.of(new FakeKvPair()))
+ .apply(
+ "waitForSuccess",
+ ParDo.of(
+ new StreamAssertEqual(
+ ImmutableSet.of(
+ row(TEST_TABLE_SCHEMA, 0, 1, 0),
+ row(TEST_TABLE_SCHEMA, 1, 2, 1),
+ row(TEST_TABLE_SCHEMA, 2, 3, 2)))));
+ queryOutput.apply(logRecords(""));
+ pipeline.run();
+ TimeUnit.MILLISECONDS.sleep(3000);
+ produceSomeRecords(3);
+
+ for (int i = 0; i < 200; i++) {
+ if (FLAG.getOrDefault(pipeline.getOptions().getOptionsId(), false)) {
+ return;
+ }
+ TimeUnit.MILLISECONDS.sleep(60);
+ }
+ Assert.fail();
+ }
+
+ private static MapElements<Row, Void> logRecords(String suffix) {
+ return MapElements.via(
+ new SimpleFunction<Row, Void>() {
+ @Override
+ public @Nullable Void apply(Row input) {
+ System.out.println(input.getValues() + suffix);
+ return null;
+ }
+ });
+ }
+
+ /** This is made because DoFn with states should get KV as input. */
+ public static class FakeKvPair extends DoFn<Row, KV<String, Row>> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ c.output(KV.of("fake_key", c.element()));
+ }
+ }
+
+ /** This DoFn will set a flag if all the elements are seen. */
+ public static class StreamAssertEqual extends DoFn<KV<String, Row>, Void> {
+ private final Set<Row> expected;
+
+ StreamAssertEqual(Set<Row> expected) {
+ super();
+ this.expected = expected;
+ }
+
+ @DoFn.StateId("seenValues")
+ private final StateSpec<BagState<Row>> seenRows = StateSpecs.bag();
+
+ @StateId("count")
+ private final StateSpec<ValueState<Integer>> countState = StateSpecs.value();
+
+ @ProcessElement
+ public void process(
+ ProcessContext context,
+ @StateId("seenValues") BagState<Row> seenValues,
+ @StateId("count") ValueState<Integer> countState) {
+ // I don't think doing this will be safe in parallel
+ int count = MoreObjects.firstNonNull(countState.read(), 0);
+ count = count + 1;
+ countState.write(count);
+ seenValues.add(context.element().getValue());
+
+ if (count >= expected.size()) {
+ if (StreamSupport.stream(seenValues.read().spliterator(), false)
+ .collect(Collectors.toSet())
+ .containsAll(expected)) {
+ System.out.println("in second if");
+ FLAG.put(context.getPipelineOptions().getOptionsId(), true);
+ }
+ }
+ }
+ }
+
+ private Row row(Schema schema, Object... values) {
+ return Row.withSchema(schema).addValues(values).build();
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void produceSomeRecords(int num) {
+ Producer<String, String> producer = new KafkaProducer<String, String>(producerProps());
+ String topicName = pipeline.getOptions().as(KafkaOptions.class).getKafkaTopic();
+ for (int i = 0; i < num; i++) {
+ producer.send(
+ new ProducerRecord<String, String>(
+ topicName, "k" + i, i + "," + ((i % 3) + 1) + "," + i));
+ }
+ producer.flush();
+ producer.close();
+ }
+
+ @SuppressWarnings("FutureReturnValueIgnored")
+ private void produceSomeRecordsWithDelay(int num, int delayMilis) {
+ Producer<String, String> producer = new KafkaProducer<String, String>(producerProps());
+ String topicName = pipeline.getOptions().as(KafkaOptions.class).getKafkaTopic();
+ for (int i = 0; i < num; i++) {
+ producer.send(
+ new ProducerRecord<String, String>(
+ topicName, "k" + i, i + "," + ((i % 3) + 1) + "," + i));
+ try {
+ TimeUnit.MILLISECONDS.sleep(delayMilis);
+ } catch (InterruptedException e) {
+ throw new RuntimeException("Could not wait for producing", e);
+ }
+ }
+ producer.flush();
+ producer.close();
+ }
+
+ private Properties producerProps() {
+ KafkaOptions options = pipeline.getOptions().as(KafkaOptions.class);
+ Properties props = new Properties();
+ props.put("bootstrap.servers", options.getKafkaBootstrapServerAddress());
+ props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
+ props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
+ props.put("buffer.memory", 33554432);
+ props.put("acks", "all");
+ props.put("request.required.acks", "1");
+ props.put("retries", 0);
+ props.put("linger.ms", 1);
+ return props;
+ }
+
+ /** Pipeline options specific for this test. */
+ public interface KafkaOptions extends PipelineOptions {
+
+ @Description("Kafka server address")
+ @Validation.Required
+ @Default.String("localhost:9092")
+ String getKafkaBootstrapServerAddress();
+
+ void setKafkaBootstrapServerAddress(String address);
+
+ @Description("Kafka topic")
+ @Validation.Required
+ @Default.String("test")
+ String getKafkaTopic();
+
+ void setKafkaTopic(String topic);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTestTable.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTestTable.java
new file mode 100644
index 0000000..749adea
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaCSVTestTable.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.kafka;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.io.kafka.KafkaIO;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.record.TimestampType;
+
+/** This is a MockKafkaCSVTestTable. It will use a Mock Consumer. */
+public class KafkaCSVTestTable extends BeamKafkaCSVTable {
+ private int partitionsPerTopic;
+ private List<KafkaTestRecord> records;
+ private static final String TIMESTAMP_TYPE_CONFIG = "test.timestamp.type";
+
+ public KafkaCSVTestTable(Schema beamSchema, List<String> topics, int partitionsPerTopic) {
+ super(beamSchema, "server:123", topics);
+ this.partitionsPerTopic = partitionsPerTopic;
+ this.records = new ArrayList<>();
+ }
+
+ @Override
+ KafkaIO.Read<byte[], byte[]> createKafkaRead() {
+ return super.createKafkaRead().withConsumerFactoryFn(this::mkMockConsumer);
+ }
+
+ public void addRecord(KafkaTestRecord record) {
+ records.add(record);
+ }
+
+ @Override
+ double computeRate(int numberOfRecords) throws NoEstimationException {
+ return super.computeRate(mkMockConsumer(new HashMap<>()), numberOfRecords);
+ }
+
+ public void setNumberOfRecordsForRate(int numberOfRecordsForRate) {
+ this.numberOfRecordsForRate = numberOfRecordsForRate;
+ }
+
+ private MockConsumer<byte[], byte[]> mkMockConsumer(Map<String, Object> config) {
+ OffsetResetStrategy offsetResetStrategy = OffsetResetStrategy.EARLIEST;
+ final Map<TopicPartition, List<ConsumerRecord<byte[], byte[]>>> kafkaRecords = new HashMap<>();
+ Map<String, List<PartitionInfo>> partitionInfoMap = new HashMap<>();
+ Map<String, List<TopicPartition>> partitionMap = new HashMap<>();
+
+ // Create Topic Paritions
+ for (String topic : this.getTopics()) {
+ List<PartitionInfo> partIds = new ArrayList<>(partitionsPerTopic);
+ List<TopicPartition> topicParitions = new ArrayList<>(partitionsPerTopic);
+ for (int i = 0; i < partitionsPerTopic; i++) {
+ TopicPartition tp = new TopicPartition(topic, i);
+ topicParitions.add(tp);
+ partIds.add(new PartitionInfo(topic, i, null, null, null));
+ kafkaRecords.put(tp, new ArrayList<>());
+ }
+ partitionInfoMap.put(topic, partIds);
+ partitionMap.put(topic, topicParitions);
+ }
+
+ TimestampType timestampType =
+ TimestampType.forName(
+ (String)
+ config.getOrDefault(
+ TIMESTAMP_TYPE_CONFIG, TimestampType.LOG_APPEND_TIME.toString()));
+
+ for (KafkaTestRecord record : this.records) {
+ int partitionIndex = record.getKey().hashCode() % partitionsPerTopic;
+ TopicPartition tp = partitionMap.get(record.getTopic()).get(partitionIndex);
+ byte[] key = record.getKey().getBytes(UTF_8);
+ byte[] value = record.getValue().getBytes(UTF_8);
+ kafkaRecords
+ .get(tp)
+ .add(
+ new ConsumerRecord<>(
+ tp.topic(),
+ tp.partition(),
+ kafkaRecords.get(tp).size(),
+ record.getTimeStamp(),
+ timestampType,
+ 0,
+ key.length,
+ value.length,
+ key,
+ value));
+ }
+
+ // This is updated when reader assigns partitions.
+ final AtomicReference<List<TopicPartition>> assignedPartitions =
+ new AtomicReference<>(Collections.<TopicPartition>emptyList());
+ final MockConsumer<byte[], byte[]> consumer =
+ new MockConsumer<byte[], byte[]>(offsetResetStrategy) {
+ @Override
+ public synchronized void assign(final Collection<TopicPartition> assigned) {
+ Collection<TopicPartition> realPartitions =
+ assigned.stream()
+ .map(part -> partitionMap.get(part.topic()).get(part.partition()))
+ .collect(Collectors.toList());
+ super.assign(realPartitions);
+ assignedPartitions.set(ImmutableList.copyOf(realPartitions));
+ for (TopicPartition tp : realPartitions) {
+ updateBeginningOffsets(ImmutableMap.of(tp, 0L));
+ updateEndOffsets(ImmutableMap.of(tp, (long) kafkaRecords.get(tp).size()));
+ }
+ }
+ // Override offsetsForTimes() in order to look up the offsets by timestamp.
+ @Override
+ public synchronized Map<TopicPartition, OffsetAndTimestamp> offsetsForTimes(
+ Map<TopicPartition, Long> timestampsToSearch) {
+ return timestampsToSearch.entrySet().stream()
+ .map(
+ e -> {
+ // In test scope, timestamp == offset. ????
+ long maxOffset = kafkaRecords.get(e.getKey()).size();
+ long offset = e.getValue();
+ OffsetAndTimestamp value =
+ (offset >= maxOffset) ? null : new OffsetAndTimestamp(offset, offset);
+ return new AbstractMap.SimpleEntry<>(e.getKey(), value);
+ })
+ .collect(
+ Collectors.toMap(
+ AbstractMap.SimpleEntry::getKey, AbstractMap.SimpleEntry::getValue));
+ }
+ };
+
+ for (String topic : getTopics()) {
+ consumer.updatePartitions(topic, partitionInfoMap.get(topic));
+ }
+
+ Runnable recordEnqueueTask =
+ new Runnable() {
+ @Override
+ public void run() {
+ // add all the records with offset >= current partition position.
+ int recordsAdded = 0;
+ for (TopicPartition tp : assignedPartitions.get()) {
+ long curPos = consumer.position(tp);
+ for (ConsumerRecord<byte[], byte[]> r : kafkaRecords.get(tp)) {
+ if (r.offset() >= curPos) {
+ consumer.addRecord(r);
+ recordsAdded++;
+ }
+ }
+ }
+ if (recordsAdded == 0) {
+ if (config.get("inject.error.at.eof") != null) {
+ consumer.setException(new KafkaException("Injected error in consumer.poll()"));
+ }
+ // MockConsumer.poll(timeout) does not actually wait even when there aren't any
+ // records.
+ // Add a small wait here in order to avoid busy looping in the reader.
+ Uninterruptibles.sleepUninterruptibly(10, TimeUnit.MILLISECONDS);
+ }
+ consumer.schedulePollTask(this);
+ }
+ };
+
+ consumer.schedulePollTask(recordEnqueueTask);
+
+ return consumer;
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
index 077cbe6..ea9ec82 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderTest.java
@@ -25,10 +25,10 @@
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
import org.junit.Test;
/** UnitTest for {@link KafkaTableProvider}. */
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTestRecord.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTestRecord.java
new file mode 100644
index 0000000..015ac8b
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTestRecord.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.kafka;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+
+/** This class is created because Kafka Consumer Records are not serializable. */
+@AutoValue
+public abstract class KafkaTestRecord implements Serializable {
+
+ public abstract String getKey();
+
+ public abstract String getValue();
+
+ public abstract String getTopic();
+
+ public abstract long getTimeStamp();
+
+ public static KafkaTestRecord create(
+ String newKey, String newValue, String newTopic, long newTimeStamp) {
+ return new AutoValue_KafkaTestRecord(newKey, newValue, newTopic, newTimeStamp);
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordToRowTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordToRowTest.java
new file mode 100644
index 0000000..b24e745
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordToRowTest.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
+
+import java.io.Serializable;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** Unit tests for {@link GenericRecordReadConverter}. */
+public class GenericRecordToRowTest implements Serializable {
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+
+ org.apache.beam.sdk.schemas.Schema payloadSchema =
+ org.apache.beam.sdk.schemas.Schema.builder()
+ .addField("name", org.apache.beam.sdk.schemas.Schema.FieldType.STRING)
+ .addField("favorite_number", org.apache.beam.sdk.schemas.Schema.FieldType.INT32)
+ .addField("favorite_color", org.apache.beam.sdk.schemas.Schema.FieldType.STRING)
+ .addField("price", org.apache.beam.sdk.schemas.Schema.FieldType.DOUBLE)
+ .build();
+
+ @Test
+ public void testConvertsGenericRecordToRow() {
+ String schemaString =
+ "{\"namespace\": \"example.avro\",\n"
+ + " \"type\": \"record\",\n"
+ + " \"name\": \"User\",\n"
+ + " \"fields\": [\n"
+ + " {\"name\": \"name\", \"type\": \"string\"},\n"
+ + " {\"name\": \"favorite_number\", \"type\": \"int\"},\n"
+ + " {\"name\": \"favorite_color\", \"type\": \"string\"},\n"
+ + " {\"name\": \"price\", \"type\": \"double\"}\n"
+ + " ]\n"
+ + "}";
+ Schema schema = (new Schema.Parser()).parse(schemaString);
+
+ GenericRecord before = new GenericData.Record(schema);
+ before.put("name", "Bob");
+ before.put("favorite_number", 256);
+ before.put("favorite_color", "red");
+ before.put("price", 2.4);
+
+ AvroCoder<GenericRecord> coder = AvroCoder.of(schema);
+
+ PCollection<Row> rows =
+ pipeline
+ .apply("create PCollection<GenericRecord>", Create.of(before).withCoder(coder))
+ .apply(
+ "convert", GenericRecordReadConverter.builder().beamSchema(payloadSchema).build());
+
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ Row.withSchema(payloadSchema).addValues("Bob", 256, "red", 2.4).build());
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableReadTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableReadTest.java
new file mode 100644
index 0000000..efa70a1
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableReadTest.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.parquet;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Test for ParquetTable. */
+@RunWith(JUnit4.class)
+public class ParquetTableReadTest {
+ private static final Logger LOG = LoggerFactory.getLogger(ParquetTableReadTest.class);
+
+ @Rule public TestPipeline pipeline = TestPipeline.create();
+ @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private static final String SQL_PARQUET_FIELD =
+ "(name VARCHAR, favorite_color VARCHAR, favorite_numbers ARRAY<INTEGER>)";
+
+ private static final Schema PARQUET_SCHEMA =
+ Schema.builder()
+ .addField("name", Schema.FieldType.STRING)
+ .addNullableField("favorite_color", Schema.FieldType.STRING)
+ .addArrayField("favorite_numbers", Schema.FieldType.INT32)
+ .build();
+
+ private String extractParquetFile(String fileName) throws IOException {
+ InputStream inputStream = getClass().getResourceAsStream("/" + fileName);
+ File root = temporaryFolder.getRoot();
+ Path tempFilePath = new File(root, fileName).toPath();
+ Files.copy(inputStream, tempFilePath);
+ return tempFilePath.toString();
+ }
+
+ @Test
+ public void testReadParquet() throws IOException {
+ String parquetPath = extractParquetFile("users.parquet");
+
+ BeamSqlEnv env = BeamSqlEnv.inMemory(new ParquetTableProvider());
+ env.executeDdl(
+ String.format(
+ "CREATE EXTERNAL TABLE users %s TYPE parquet LOCATION '%s'",
+ SQL_PARQUET_FIELD, parquetPath));
+
+ PCollection<Row> rows =
+ BeamSqlRelUtils.toPCollection(
+ pipeline, env.parseQuery("SELECT name, favorite_color, favorite_numbers FROM users"));
+
+ PAssert.that(rows)
+ .containsInAnyOrder(
+ Row.withSchema(PARQUET_SCHEMA)
+ .addValues("Alyssa", null, Arrays.asList(3, 9, 15, 20))
+ .build(),
+ Row.withSchema(PARQUET_SCHEMA).addValues("Ben", "red", Arrays.asList()).build());
+
+ pipeline.run();
+ }
+}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
index e8fa135..2896003 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
@@ -53,15 +53,14 @@
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.SchemaCoder;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteConnection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Ignore;
@@ -137,8 +136,7 @@
queryOutput.apply(
"waitForSuccess",
resultSignal.signalSuccessWhen(
- SchemaCoder.of(
- PAYLOAD_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()),
+ SchemaCoder.of(PAYLOAD_SCHEMA),
observedRows ->
observedRows.equals(
ImmutableSet.of(
@@ -209,8 +207,7 @@
queryOutput.apply(
"waitForSuccess",
resultSignal.signalSuccessWhen(
- SchemaCoder.of(
- PAYLOAD_SCHEMA, SerializableFunctions.identity(), SerializableFunctions.identity()),
+ SchemaCoder.of(PAYLOAD_SCHEMA),
observedRows ->
observedRows.equals(
ImmutableSet.of(
@@ -255,6 +252,7 @@
}
@Test
+ @Ignore("https://jira.apache.org/jira/browse/BEAM-7582")
public void testSQLLimit() throws Exception {
String createTableString =
"CREATE EXTERNAL TABLE message (\n"
@@ -348,7 +346,7 @@
inMemoryMetaStore.registerProvider(tableProvider);
}
- JdbcConnection connection = JdbcDriver.connect(inMemoryMetaStore);
+ JdbcConnection connection = JdbcDriver.connect(inMemoryMetaStore, options);
connection.setPipelineOptionsMap(argsMap);
return connection;
}
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
index 5bd6aaf..e7f5fc7 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonTableProviderTest.java
@@ -22,7 +22,7 @@
import static org.junit.Assert.assertEquals;
import com.alibaba.fastjson.JSON;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.schemas.Schema;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java
index 467d13f..370c214 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubMessageToRowTest.java
@@ -22,7 +22,7 @@
import static org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.VARCHAR;
import static org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubMessageToRow.DLQ_TAG;
import static org.apache.beam.sdk.extensions.sql.meta.provider.pubsub.PubsubMessageToRow.MAIN_TAG;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.size;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterables.size;
import static org.junit.Assert.assertEquals;
import java.io.Serializable;
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
import org.joda.time.DateTime;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java
index ed393de..172bdb1 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProviderTest.java
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Charsets;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
index e2bbf26..92234b4 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/store/InMemoryMetaStoreTest.java
@@ -26,7 +26,7 @@
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Stream;
-import org.apache.beam.sdk.extensions.sql.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
import org.apache.beam.sdk.extensions.sql.meta.Table;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.sdk.extensions.sql.meta.provider.text.TextTableProvider;
diff --git a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java
index dd19d20..abbb4df 100644
--- a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java
+++ b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/utils/RowAsserts.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.Iterables;
/** Contain helpers to assert {@link Row}s. */
public class RowAsserts {
diff --git a/sdks/java/extensions/sql/src/test/resources/users.parquet b/sdks/java/extensions/sql/src/test/resources/users.parquet
new file mode 100644
index 0000000..aa52733
--- /dev/null
+++ b/sdks/java/extensions/sql/src/test/resources/users.parquet
Binary files differ
diff --git a/sdks/java/extensions/sql/zetasql/build.gradle b/sdks/java/extensions/sql/zetasql/build.gradle
new file mode 100644
index 0000000..21f60d1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/build.gradle
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'org.apache.beam.module'
+}
+
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.sql.zetasql')
+
+description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL :: ZetaSQL"
+ext.summary = "ZetaSQL to Calcite translator"
+
+def zetasql_version = "2019.09.1"
+
+dependencies {
+ compile project(":sdks:java:core")
+ compile project(":sdks:java:extensions:sql")
+ compile library.java.vendored_calcite_1_20_0
+ compile library.java.grpc_all
+ compile library.java.protobuf_java
+ compile library.java.protobuf_java_util
+ compile "com.google.api.grpc:proto-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
+ compile "com.google.api.grpc:grpc-google-common-protos:1.12.0" // Interfaces with ZetaSQL use this
+ compile "com.google.zetasql:zetasql-jni-channel:$zetasql_version"
+ compile "com.google.zetasql:zetasql-client:$zetasql_version"
+ compile "com.google.zetasql:zetasql-types:$zetasql_version"
+ testCompile library.java.vendored_calcite_1_20_0
+ testCompile library.java.vendored_guava_26_0_jre
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.quickcheck_core
+ testRuntimeClasspath library.java.slf4j_jdk14
+}
+
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
new file mode 100644
index 0000000..9f6b33b
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.lang.reflect.Method;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Types;
+
+/** BeamBuiltinMethods. */
+public class BeamBuiltinMethods {
+ public static final Method STARTS_WITH_METHOD =
+ Types.lookupMethod(StringFunctions.class, "startsWith", String.class, String.class);
+
+ public static final Method ENDS_WITH_METHOD =
+ Types.lookupMethod(StringFunctions.class, "endsWith", String.class, String.class);
+
+ public static final Method LIKE_METHOD =
+ Types.lookupMethod(StringFunctions.class, "like", String.class, String.class);
+
+ public static final Method CONCAT_METHOD =
+ Types.lookupMethod(
+ StringFunctions.class,
+ "concat",
+ String.class,
+ String.class,
+ String.class,
+ String.class,
+ String.class);
+
+ public static final Method REPLACE_METHOD =
+ Types.lookupMethod(
+ StringFunctions.class, "replace", String.class, String.class, String.class);
+
+ public static final Method TRIM_METHOD =
+ Types.lookupMethod(StringFunctions.class, "trim", String.class, String.class);
+
+ public static final Method LTRIM_METHOD =
+ Types.lookupMethod(StringFunctions.class, "ltrim", String.class, String.class);
+
+ public static final Method RTRIM_METHOD =
+ Types.lookupMethod(StringFunctions.class, "rtrim", String.class, String.class);
+
+ public static final Method SUBSTR_METHOD =
+ Types.lookupMethod(StringFunctions.class, "substr", String.class, long.class, long.class);
+
+ public static final Method REVERSE_METHOD =
+ Types.lookupMethod(StringFunctions.class, "reverse", String.class);
+
+ public static final Method CHAR_LENGTH_METHOD =
+ Types.lookupMethod(StringFunctions.class, "charLength", String.class);
+
+ public static final Method TIMESTAMP_METHOD =
+ Types.lookupMethod(TimestampFunctions.class, "timestamp", String.class, String.class);
+
+ public static final Method DATE_METHOD =
+ Types.lookupMethod(DateFunctions.class, "date", Integer.class, Integer.class, Integer.class);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
new file mode 100644
index 0000000..bc2229e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.io.UnsupportedEncodingException;
+import org.joda.time.DateTime;
+
+/** BeamCodegenUtils. */
+public class BeamCodegenUtils {
+ // convert bytes to String in UTF8 encoding.
+ public static String toStringUTF8(byte[] bytes) {
+ try {
+ return new String(bytes, "UTF8");
+ } catch (UnsupportedEncodingException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public static String toStringTimestamp(long timestamp) {
+ return DateTimeUtils.formatTimestampWithTimeZone(new DateTime(timestamp));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
new file mode 100644
index 0000000..c75d033
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.TimeZone;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+
+/** DateFunctions. */
+public class DateFunctions {
+ public DateTime date(Integer year, Integer month, Integer day) {
+ return DateTimeUtils.parseDate(
+ String.join("-", year.toString(), month.toString(), day.toString()));
+ }
+
+ public DateTime date(DateTime ts) {
+ return date(ts, "UTC");
+ }
+
+ public DateTime date(DateTime ts, String timezone) {
+ return ts.withZoneRetainFields(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
new file mode 100644
index 0000000..5f90efb
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static com.google.zetasql.CivilTimeEncoder.decodePacked64TimeNanos;
+import static com.google.zetasql.CivilTimeEncoder.encodePacked64TimeNanos;
+
+import com.google.zetasql.Value;
+import io.grpc.Status;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.DateString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.TimeString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.LocalTime;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+/** DateTimeUtils. */
+public class DateTimeUtils {
+ public static final Long MILLIS_PER_DAY = 86400000L;
+ private static final Long MICROS_PER_MILLI = 1000L;
+
+ @SuppressWarnings("unchecked")
+ private enum TimestampPatterns {
+ TIMESTAMP_PATTERN,
+ TIMESTAMP_PATTERN_SUBSECOND,
+ TIMESTAMP_PATTERN_T,
+ TIMESTAMP_PATTERN_SUBSECOND_T,
+ }
+
+ @SuppressWarnings("unchecked")
+ private static final ImmutableMap<Enum, DateTimeFormatter> TIMESTAMP_PATTERN_WITHOUT_TZ =
+ ImmutableMap.of(
+ TimestampPatterns.TIMESTAMP_PATTERN, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND,
+ DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS"),
+ TimestampPatterns.TIMESTAMP_PATTERN_T, DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T,
+ DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSS"));
+
+ @SuppressWarnings("unchecked")
+ private static final ImmutableMap<Enum, DateTimeFormatter> TIMESTAMP_PATTERN_WITH_TZ =
+ ImmutableMap.of(
+ TimestampPatterns.TIMESTAMP_PATTERN, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ssZZ"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND,
+ DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZZ"),
+ TimestampPatterns.TIMESTAMP_PATTERN_T,
+ DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ssZZ"),
+ TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T,
+ DateTimeFormat.forPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZZ"));
+
+ public static DateTimeFormatter findDateTimePattern(String str) {
+ if (str.indexOf('+') == -1) {
+ return findDateTimePattern(str, TIMESTAMP_PATTERN_WITHOUT_TZ);
+ } else {
+ return findDateTimePattern(str, TIMESTAMP_PATTERN_WITH_TZ);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ public static DateTimeFormatter findDateTimePattern(
+ String str, ImmutableMap<Enum, DateTimeFormatter> patternMap) {
+ if (str.indexOf('.') == -1) {
+ if (str.indexOf('T') == -1) {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN);
+ } else {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_T);
+ }
+ } else {
+ if (str.indexOf('T') == -1) {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND);
+ } else {
+ return patternMap.get(TimestampPatterns.TIMESTAMP_PATTERN_SUBSECOND_T);
+ }
+ }
+ }
+
+ // https://cloud.google.com/bigquery/docs/reference/standard-sql/migrating-from-legacy-sql#timestamp_differences
+ // 0001-01-01 00:00:00 to 9999-12-31 23:59:59.999999 UTC.
+ // -62135596800000000 to 253402300799999999
+ @SuppressWarnings("GoodTime")
+ public static final Long MIN_UNIX_MILLIS = -62135596800000L;
+
+ @SuppressWarnings("GoodTime")
+ public static final Long MAX_UNIX_MILLIS = 253402300799999L;
+
+ public static DateTime parseTimestampWithUTCTimeZone(String str) {
+ return findDateTimePattern(str).withZoneUTC().parseDateTime(str);
+ }
+
+ @SuppressWarnings("unused")
+ public static DateTime parseTimestampWithLocalTimeZone(String str) {
+ return findDateTimePattern(str).withZone(DateTimeZone.getDefault()).parseDateTime(str);
+ }
+
+ public static DateTime parseTimestampWithTimeZone(String str) {
+ // for example, accept "1990-10-20 13:24:01+0730"
+ if (str.indexOf('.') == -1) {
+ return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ssZ").parseDateTime(str);
+ } else {
+ return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZ").parseDateTime(str);
+ }
+ }
+
+ public static String formatTimestampWithTimeZone(DateTime dt) {
+ String resultWithoutZone;
+ if (dt.getMillisOfSecond() == 0) {
+ resultWithoutZone = dt.toString(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"));
+ } else {
+ resultWithoutZone = dt.toString(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS"));
+ }
+
+ // ZetaSQL expects a 2-digit timezone offset (-05) if the minute part is zero, and it expects
+ // a 4-digit timezone with a colon (-07:52) if the minute part is non-zero. None of the
+ // variations on z,Z,ZZ,.. do this for us so we have to do it manually here.
+ String zone = dt.toString(DateTimeFormat.forPattern("ZZ"));
+ List<String> zoneParts = Lists.newArrayList(Splitter.on(':').limit(2).split(zone));
+ if (zoneParts.size() == 2 && zoneParts.get(1).equals("00")) {
+ zone = zoneParts.get(0);
+ }
+
+ return resultWithoutZone + zone;
+ }
+
+ @SuppressWarnings("unused")
+ public static DateTime parseTimestampWithoutTimeZone(String str) {
+ return DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss").parseDateTime(str);
+ }
+
+ public static DateTime parseDate(String str) {
+ return DateTimeFormat.forPattern("yyyy-MM-dd").withZoneUTC().parseDateTime(str);
+ }
+
+ public static DateTime parseTime(String str) {
+ // DateTimeFormat does not parse "08:10:10" for pattern "HH:mm:ss.SSS". In this case, '.' must
+ // appear.
+ if (str.indexOf('.') == -1) {
+ return DateTimeFormat.forPattern("HH:mm:ss").withZoneUTC().parseDateTime(str);
+ } else {
+ return DateTimeFormat.forPattern("HH:mm:ss.SSS").withZoneUTC().parseDateTime(str);
+ }
+ }
+
+ @SuppressWarnings(
+ "Value with nanoseconds will be truncated to milliseconds in decodePacked64TimeNanos.")
+ public static TimeString convertTimeValueToTimeString(Value value) {
+ LocalTime localTime = decodePacked64TimeNanos(value.getTimeValue());
+ return TimeString.fromMillisOfDay(localTime.getMillisOfDay());
+ }
+
+ // dates are represented as an int32 value, indicating the offset
+ // in days from the epoch 1970-01-01. ZetaSQL dates are not timezone aware,
+ // and do not correspond to any particular 24 hour period.
+ public static DateString convertDateValueToDateString(Value value) {
+ return DateString.fromDaysSinceEpoch(value.getDateValue());
+ }
+
+ public static Value parseDateToValue(String dateString) {
+ DateTime dateTime = parseDate(dateString);
+ return Value.createDateValue((int) (dateTime.getMillis() / MILLIS_PER_DAY));
+ }
+
+ public static Value parseTimeToValue(String timeString) {
+ DateTime dateTime = parseTime(timeString);
+ return Value.createTimeValue(
+ encodePacked64TimeNanos(LocalTime.fromMillisOfDay(dateTime.getMillisOfDay())));
+ }
+
+ public static Value parseTimestampWithTZToValue(String timestampString) {
+ DateTime dateTime = parseTimestampWithTimeZone(timestampString);
+ // convert from micros.
+ // TODO: how to handle overflow.
+ return Value.createTimestampValueFromUnixMicros(
+ LongMath.checkedMultiply(dateTime.getMillis(), MICROS_PER_MILLI));
+ }
+
+ private static void safeCheckSubMillisPrecision(long micros) {
+ long subMilliPrecision = micros % 1000L;
+ if (subMilliPrecision != 0) {
+ throw new IllegalArgumentException(
+ String.format(
+ "%s has sub-millisecond precision, which Beam ZetaSQL does"
+ + " not currently support.",
+ micros));
+ }
+ }
+
+ @SuppressWarnings("GoodTime")
+ public static long safeMicrosToMillis(long micros) {
+ safeCheckSubMillisPrecision(micros);
+ return micros / 1000L;
+ }
+
+ /**
+ * This function validates that Long representation of timestamp is compatible with ZetaSQL
+ * timestamp values range.
+ *
+ * <p>Invoked via reflection. @see SqlOperators
+ *
+ * @param ts Timestamp to validate.
+ * @return Unchanged timestamp sent for validation.
+ */
+ @SuppressWarnings("GoodTime")
+ public static Long validateTimestamp(Long ts) {
+ if (ts == null) {
+ return null;
+ }
+
+ if ((ts < MIN_UNIX_MILLIS) || (ts > MAX_UNIX_MILLIS)) {
+ throw Status.OUT_OF_RANGE
+ .withDescription("Timestamp is out of valid range.")
+ .asRuntimeException();
+ }
+
+ return ts;
+ }
+
+ /**
+ * This function validates that interval is compatible with ZetaSQL timestamp values range.
+ *
+ * <p>ZetaSQL validates that if we represent interval in milliseconds, it will fit into Long.
+ *
+ * <p>In case of SECOND or smaller time unit, it converts timestamp to microseconds, so we need to
+ * convert those to microsecond and verify that we do not cause overflow.
+ *
+ * <p>Invoked via reflection. @see SqlOperators
+ *
+ * @param arg Argument for the interval.
+ * @param unit Time unit used in this interval.
+ * @return Argument for the interval.
+ */
+ @SuppressWarnings("GoodTime")
+ public static Long validateTimeInterval(Long arg, TimeUnit unit) {
+ if (arg == null) {
+ return null;
+ }
+
+ // multiplier to convert to milli or microseconds.
+ long multiplier = unit.multiplier.longValue();
+ switch (unit) {
+ case SECOND:
+ case MILLISECOND:
+ multiplier *= 1000L; // Change multiplier from milliseconds to microseconds.
+ break;
+ default:
+ break;
+ }
+
+ if ((arg > Long.MAX_VALUE / multiplier) || (arg < Long.MIN_VALUE / multiplier)) {
+ throw Status.OUT_OF_RANGE
+ .withDescription("Interval is out of valid range")
+ .asRuntimeException();
+ }
+
+ return arg;
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
new file mode 100644
index 0000000..428c360
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.zetasql.Table;
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOutputColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithEntry;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.SimpleTableWithPath;
+
+/** QueryTrait. */
+public class QueryTrait {
+ public Map<String, ResolvedWithEntry> withEntries = new HashMap<>();
+
+ public Map<ResolvedColumn, String> outputColumnMap = new HashMap<>();
+
+ public Map<Long, SimpleTableWithPath> resolvedTables = new HashMap<>();
+
+ // TODO: move query parameter map to QueryTrait.
+
+ public void addOutputColumnList(List<ResolvedOutputColumn> outputColumnList) {
+ outputColumnList.forEach(
+ column -> {
+ outputColumnMap.put(column.getColumn(), column.getName());
+ });
+ }
+
+ /** Store a table together with its full path for repeated resolutions. */
+ public void addResolvedTable(SimpleTableWithPath tableWithPath) {
+ // table ids are autoincremted in SimpleTable
+ resolvedTables.put(tableWithPath.getTable().getId(), tableWithPath);
+ }
+
+ /** True if the table was resolved using the Calcite schema. */
+ public boolean isTableResolved(Table table) {
+ return resolvedTables.containsKey(table.getId());
+ }
+
+ /** Returns a full table path (exlucding top-level schema) for a given ZetaSQL Table. */
+ public List<String> getTablePath(Table table) {
+ checkArgument(
+ isTableResolved(table),
+ "Attempting to get a path of an unresolved table. Resolve and add the table first: %s",
+ table.getFullName());
+ return resolvedTables.get(table.getId()).getPath();
+ }
+
+ public List<String> retrieveFieldNames(List<ResolvedColumn> resolvedColumnList) {
+ return resolvedColumnList.stream().map(this::resolveAlias).collect(Collectors.toList());
+ }
+
+ public String resolveAlias(ResolvedColumn resolvedColumn) {
+ return this.outputColumnMap.getOrDefault(resolvedColumn, resolvedColumn.getName());
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
new file mode 100644
index 0000000..3a4a4e2
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_CREATE_FUNCTION_STMT;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_QUERY_STMT;
+import static org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable.ZETASQL_BUILTIN_FUNCTION_WHITELIST;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils.toZetaType;
+
+import com.google.zetasql.Analyzer;
+import com.google.zetasql.AnalyzerOptions;
+import com.google.zetasql.Function;
+import com.google.zetasql.SimpleCatalog;
+import com.google.zetasql.Value;
+import com.google.zetasql.ZetaSQLBuiltinFunctionOptions;
+import com.google.zetasql.ZetaSQLFunctions.FunctionEnums.Mode;
+import com.google.zetasql.ZetaSQLOptions.ErrorMessageMode;
+import com.google.zetasql.ZetaSQLOptions.LanguageFeature;
+import com.google.zetasql.ZetaSQLOptions.ProductMode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedCreateFunctionStmt;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution.SimpleTableWithPath;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Adapter for {@link Analyzer} to simplify the API for parsing the query and resolving the AST. */
+class SqlAnalyzer {
+ static final String PRE_DEFINED_WINDOW_FUNCTIONS = "pre_defined_window_functions";
+
+ private static final ImmutableList<String> FUNCTION_LIST =
+ ImmutableList.of(
+ // TODO: support optional function argument (for window_offset).
+ "CREATE FUNCTION TUMBLE(ts TIMESTAMP, window_size STRING) AS (1);",
+ "CREATE FUNCTION TUMBLE_START(window_size STRING) AS (1);",
+ "CREATE FUNCTION TUMBLE_END(window_size STRING) AS (1);",
+ "CREATE FUNCTION HOP(ts TIMESTAMP, emit_frequency STRING, window_size STRING) AS (1);",
+ "CREATE FUNCTION HOP_START(emit_frequency STRING, window_size STRING) AS (1);",
+ "CREATE FUNCTION HOP_END(emit_frequency STRING, window_size STRING) AS (1);",
+ "CREATE FUNCTION SESSION(ts TIMESTAMP, session_gap STRING) AS (1);",
+ "CREATE FUNCTION SESSION_START(session_gap STRING) AS (1);",
+ "CREATE FUNCTION SESSION_END(session_gap STRING) AS (1);");
+
+ private final Builder builder;
+
+ private SqlAnalyzer(Builder builder) {
+ this.builder = builder;
+ }
+
+ /** Static factory method to create the builder with query parameters. */
+ static Builder withQueryParams(Map<String, Value> params) {
+ return new Builder().withQueryParams(ImmutableMap.copyOf(params));
+ }
+
+ /**
+ * Accepts the SQL string, returns the resolved AST.
+ *
+ * <p>Initializes query parameters, populates the catalog based on Calcite's schema and table name
+ * resolution strategy set in the context.
+ */
+ ResolvedStatement analyze(String sql) {
+ AnalyzerOptions options = initAnalyzerOptions(builder.queryParams);
+ List<List<String>> tables = Analyzer.extractTableNamesFromStatement(sql);
+ SimpleCatalog catalog =
+ createPopulatedCatalog(builder.topLevelSchema.getName(), options, tables);
+
+ return Analyzer.analyzeStatement(sql, options, catalog);
+ }
+
+ static AnalyzerOptions initAnalyzerOptions() {
+ AnalyzerOptions options = new AnalyzerOptions();
+ options.setErrorMessageMode(ErrorMessageMode.ERROR_MESSAGE_MULTI_LINE_WITH_CARET);
+ // +00:00 UTC offset
+ options.setDefaultTimezone("UTC");
+ options.getLanguageOptions().setProductMode(ProductMode.PRODUCT_EXTERNAL);
+ options
+ .getLanguageOptions()
+ .setEnabledLanguageFeatures(
+ new HashSet<>(
+ Arrays.asList(
+ LanguageFeature.FEATURE_DISALLOW_GROUP_BY_FLOAT,
+ LanguageFeature.FEATURE_V_1_2_CIVIL_TIME,
+ LanguageFeature.FEATURE_V_1_1_SELECT_STAR_EXCEPT_REPLACE)));
+
+ options
+ .getLanguageOptions()
+ .setSupportedStatementKinds(
+ ImmutableSet.of(RESOLVED_QUERY_STMT, RESOLVED_CREATE_FUNCTION_STMT));
+
+ return options;
+ }
+
+ private static AnalyzerOptions initAnalyzerOptions(Map<String, Value> queryParams) {
+ AnalyzerOptions options = initAnalyzerOptions();
+
+ for (Map.Entry<String, Value> entry : queryParams.entrySet()) {
+ options.addQueryParameter(entry.getKey(), entry.getValue().getType());
+ }
+
+ return options;
+ }
+
+ /**
+ * Creates a SimpleCatalog which represents the top-level schema, populates it with tables,
+ * built-in functions.
+ */
+ private SimpleCatalog createPopulatedCatalog(
+ String catalogName, AnalyzerOptions options, List<List<String>> tables) {
+
+ SimpleCatalog catalog = new SimpleCatalog(catalogName);
+ addBuiltinFunctionsToCatalog(catalog, options);
+
+ tables.forEach(table -> addTableToLeafCatalog(builder.queryTrait, catalog, table));
+
+ return catalog;
+ }
+
+ private void addBuiltinFunctionsToCatalog(SimpleCatalog catalog, AnalyzerOptions options) {
+
+ // Enable ZetaSQL builtin functions.
+ ZetaSQLBuiltinFunctionOptions zetasqlBuiltinFunctionOptions =
+ new ZetaSQLBuiltinFunctionOptions(options.getLanguageOptions());
+
+ ZETASQL_BUILTIN_FUNCTION_WHITELIST.forEach(
+ zetasqlBuiltinFunctionOptions::includeFunctionSignatureId);
+
+ catalog.addZetaSQLFunctions(zetasqlBuiltinFunctionOptions);
+
+ FUNCTION_LIST.stream()
+ .map(func -> (ResolvedCreateFunctionStmt) Analyzer.analyzeStatement(func, options, catalog))
+ .map(
+ resolvedFunc ->
+ new Function(
+ String.join(".", resolvedFunc.getNamePath()),
+ PRE_DEFINED_WINDOW_FUNCTIONS,
+ Mode.SCALAR,
+ ImmutableList.of(resolvedFunc.getSignature())))
+ .forEach(catalog::addFunction);
+ }
+
+ /**
+ * Assume last element in tablePath is a table name, and everything before is catalogs. So the
+ * logic is to create nested catalogs until the last level, then add a table at the last level.
+ *
+ * <p>Table schema is extracted from Calcite schema based on the table name resultion strategy,
+ * e.g. either by drilling down the schema.getSubschema() path or joining the table name with dots
+ * to construct a single compound identifier (e.g. Data Catalog use case).
+ */
+ private void addTableToLeafCatalog(
+ QueryTrait trait, SimpleCatalog topLevelCatalog, List<String> tablePath) {
+
+ SimpleCatalog leafCatalog = createNestedCatalogs(topLevelCatalog, tablePath);
+
+ org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table calciteTable =
+ TableResolution.resolveCalciteTable(
+ builder.calciteContext, builder.topLevelSchema, tablePath);
+
+ if (calciteTable == null) {
+ throw new RuntimeException(
+ "Wasn't able to find resolve the path "
+ + tablePath
+ + " in "
+ + builder.topLevelSchema.getName());
+ }
+
+ RelDataType rowType = calciteTable.getRowType(builder.typeFactory);
+
+ SimpleTableWithPath tableWithPath =
+ SimpleTableWithPath.of(builder.topLevelSchema.getName(), tablePath);
+ trait.addResolvedTable(tableWithPath);
+
+ addFieldsToTable(tableWithPath, rowType);
+ leafCatalog.addSimpleTable(tableWithPath.getTable());
+ }
+
+ private void addFieldsToTable(SimpleTableWithPath tableWithPath, RelDataType rowType) {
+ for (RelDataTypeField field : rowType.getFieldList()) {
+ tableWithPath.getTable().addSimpleColumn(field.getName(), toZetaType(field.getType()));
+ }
+ }
+
+ /** For table path like a.b.c we assume c is the table and a.b are the nested catalogs/schemas. */
+ private SimpleCatalog createNestedCatalogs(SimpleCatalog catalog, List<String> tablePath) {
+ SimpleCatalog currentCatalog = catalog;
+ for (int i = 0; i < tablePath.size() - 1; i++) {
+ String nextCatalogName = tablePath.get(i);
+
+ Optional<SimpleCatalog> existing = tryGetExisting(currentCatalog, nextCatalogName);
+
+ currentCatalog =
+ existing.isPresent() ? existing.get() : addNewCatalog(currentCatalog, nextCatalogName);
+ }
+ return currentCatalog;
+ }
+
+ private Optional<SimpleCatalog> tryGetExisting(
+ SimpleCatalog currentCatalog, String nextCatalogName) {
+ return currentCatalog.getCatalogList().stream()
+ .filter(c -> nextCatalogName.equals(c.getFullName()))
+ .findFirst();
+ }
+
+ private SimpleCatalog addNewCatalog(SimpleCatalog currentCatalog, String nextCatalogName) {
+ SimpleCatalog nextCatalog = new SimpleCatalog(nextCatalogName);
+ currentCatalog.addSimpleCatalog(nextCatalog);
+ return nextCatalog;
+ }
+
+ /** Builder for SqlAnalyzer. */
+ static class Builder {
+
+ private Map<String, Value> queryParams;
+ private QueryTrait queryTrait;
+ private Context calciteContext;
+ private SchemaPlus topLevelSchema;
+ private JavaTypeFactory typeFactory;
+
+ private Builder() {}
+
+ /** Query parameters. */
+ Builder withQueryParams(Map<String, Value> params) {
+ this.queryParams = ImmutableMap.copyOf(params);
+ return this;
+ }
+
+ /** QueryTrait, has parsing-time configuration for rel conversion. */
+ Builder withQueryTrait(QueryTrait trait) {
+ this.queryTrait = trait;
+ return this;
+ }
+
+ /** Current top-level schema. */
+ Builder withTopLevelSchema(SchemaPlus schema) {
+ this.topLevelSchema = schema;
+ return this;
+ }
+
+ /** Calcite parsing context, can have name resolution and other configuration. */
+ Builder withCalciteContext(Context context) {
+ this.calciteContext = context;
+ return this;
+ }
+
+ /**
+ * Current type factory.
+ *
+ * <p>Used to convert field types in schemas.
+ */
+ Builder withTypeFactory(JavaTypeFactory typeFactory) {
+ this.typeFactory = typeFactory;
+ return this;
+ }
+
+ /** Returns the parsed and resolved query. */
+ ResolvedStatement analyze(String sql) {
+ return new SqlAnalyzer(this).analyze(sql);
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
new file mode 100644
index 0000000..79d99b0
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/**
+ * Rewrites $case_with_value calls as $case_no_value calls.
+ *
+ * <p>Turns:
+ *
+ * <pre><code>CASE x
+ * WHEN w1 THEN t1
+ * WHEN w2 THEN t2
+ * ELSE e
+ * END</code></pre>
+ *
+ * <p>into:
+ *
+ * <pre><code>CASE
+ * WHEN x == w1 THEN t1
+ * WHEN x == w2 THEN t2
+ * ELSE expr
+ * END</code></pre>
+ *
+ * <p>Note that the ELSE statement is actually optional, but we don't need to worry about that here
+ * because the ZetaSQL analyzer populates the ELSE argument as a NULL literal if it's not specified.
+ */
+public class SqlCaseWithValueOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() % 2 == 0 && !operands.isEmpty(),
+ "$case_with_value should have an even number of arguments greater than 0 in function call"
+ + " (The value operand, the else operand, and paired when/then operands).");
+ SqlOperator op = SqlStdOperatorTable.CASE;
+
+ List<RexNode> newOperands = new ArrayList<>();
+ RexNode value = operands.get(0);
+
+ for (int i = 1; i < operands.size() - 2; i += 2) {
+ RexNode when = operands.get(i);
+ RexNode then = operands.get(i + 1);
+ newOperands.add(
+ rexBuilder.makeCall(SqlStdOperatorTable.EQUALS, ImmutableList.of(value, when)));
+ newOperands.add(then);
+ }
+
+ RexNode elseOperand = Iterables.getLast(operands);
+ newOperands.add(elseOperand);
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
new file mode 100644
index 0000000..58743f6
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Rewrites COALESCE calls as CASE ($case_no_value) calls.
+ *
+ * <p>Turns <code>COALESCE(a, b, c)</code> into:
+ *
+ * <pre><code>CASE
+ * WHEN a IS NOT NULL THEN a
+ * WHEN b IS NOT NULL THEN b
+ * ELSE c
+ * END</code></pre>
+ *
+ * <p>There is also a special case for the single-argument case: <code>COALESCE(a)</code> becomes
+ * just <code>a</code>.
+ */
+public class SqlCoalesceOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() >= 1, "COALESCE should have at least one argument in function call.");
+
+ // No need for a case operator if there's only one operand
+ if (operands.size() == 1) {
+ return operands.get(0);
+ }
+
+ SqlOperator op = SqlStdOperatorTable.CASE;
+
+ List<RexNode> newOperands = new ArrayList<>();
+ for (RexNode operand : Util.skipLast(operands)) {
+ newOperands.add(
+ rexBuilder.makeCall(SqlStdOperatorTable.IS_NOT_NULL, ImmutableList.of(operand)));
+ newOperands.add(operand);
+ }
+ newOperands.add(Util.last(operands));
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
new file mode 100644
index 0000000..129031e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Rewrites EXTRACT calls by swapping first two arguments to fit for calcite SqlExtractOperator. */
+public class SqlExtractTimestampOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() == 2,
+ "EXTRACT should have two arguments in function call. AT TIME ZONE not supported.");
+
+ SqlOperator op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get("$extract");
+
+ ImmutableList.Builder<RexNode> newOperands =
+ ImmutableList.<RexNode>builder().add(operands.get(1)).add(operands.get(0));
+
+ for (int i = 2; i < operands.size(); ++i) {
+ newOperands.add(operands.get(i));
+ }
+
+ return rexBuilder.makeCall(op, newOperands.build());
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
new file mode 100644
index 0000000..039797d
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Rewrites IFNULL calls as CASE ($case_no_value) calls.
+ *
+ * <p>Turns <code>IFNULL(expr, null_result)</code> into: <code><pre>CASE
+ * WHEN expr IS NULL THEN null_result
+ * ELSE expr
+ * END</pre></code>
+ */
+public class SqlIfNullOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() == 2, "IFNULL should have two arguments in function call.");
+
+ SqlOperator op = SqlStdOperatorTable.CASE;
+ List<RexNode> newOperands =
+ ImmutableList.of(
+ rexBuilder.makeCall(SqlStdOperatorTable.IS_NULL, ImmutableList.of(operands.get(0))),
+ operands.get(1),
+ operands.get(0));
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
new file mode 100644
index 0000000..382a5ca
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/**
+ * Rewrites NULLIF calls as CASE ($case_no_value) calls.
+ *
+ * <p>Turns <code>NULLIF(expression, expression_to_match)</code> into: <code><pre>CASE
+ * WHEN expression == expression_to_match THEN NULL
+ * ELSE expression
+ * END</pre></code>
+ */
+public class SqlNullIfOperatorRewriter implements SqlOperatorRewriter {
+ @Override
+ public RexNode apply(RexBuilder rexBuilder, List<RexNode> operands) {
+ Preconditions.checkArgument(
+ operands.size() == 2, "NULLIF should have two arguments in function call.");
+
+ SqlOperator op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get("$case_no_value");
+ List<RexNode> newOperands =
+ ImmutableList.of(
+ rexBuilder.makeCall(
+ SqlStdOperatorTable.EQUALS, ImmutableList.of(operands.get(0), operands.get(1))),
+ rexBuilder.makeNullLiteral(operands.get(1).getType()),
+ operands.get(0));
+
+ return rexBuilder.makeCall(op, newOperands);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
new file mode 100644
index 0000000..949632c
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Interface for rewriting calls a specific ZetaSQL operator. */
+public interface SqlOperatorRewriter {
+ /**
+ * Create and return a new {@link RexNode} that represents a call to this operator with the
+ * specified operands.
+ *
+ * @param rexBuilder A {@link RexBuilder} instance to use for creating new {@link RexNode}s
+ * @param operands The original list of {@link RexNode} operands passed to this operator call
+ * @return The created RexNode
+ */
+ RexNode apply(RexBuilder rexBuilder, List<RexNode> operands);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
new file mode 100644
index 0000000..9f0e948
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.impl.ScalarFunctionImpl;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ScalarFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.FamilyOperandTypeChecker;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.InferTypes;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.OperandTypes;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlReturnTypeInference;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlUserDefinedFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+
+/**
+ * A separate SqlOperators table for those functions that do not exist or not compatible with
+ * Calcite. Most of functions within this class is copied from Calcite.
+ */
+public class SqlOperators {
+ public static final RelDataType TIMESTAMP_WITH_NULLABILITY =
+ createSqlType(SqlTypeName.TIMESTAMP, true);
+ public static final RelDataType OTHER = createSqlType(SqlTypeName.OTHER, false);
+ public static final RelDataType TIMESTAMP = createSqlType(SqlTypeName.TIMESTAMP, false);
+ public static final RelDataType BIGINT = createSqlType(SqlTypeName.BIGINT, false);
+
+ public static SqlUserDefinedFunction createUdfOperator(
+ String name,
+ Class<?> methodClass,
+ String methodName,
+ SqlReturnTypeInference returnTypeInference,
+ List<RelDataType> paramTypes) {
+ return new SqlUserDefinedFunction(
+ new SqlIdentifier(name, SqlParserPos.ZERO),
+ returnTypeInference,
+ null,
+ null,
+ paramTypes,
+ ScalarFunctionImpl.create(methodClass, methodName));
+ }
+
+ // Helper function to create SqlUserDefinedFunction based on a function name and a method.
+ // SqlUserDefinedFunction will be able to pass through Calcite codegen and get proper function
+ // called.
+ public static SqlUserDefinedFunction createUdfOperator(String name, Method method) {
+ Function function = ScalarFunctionImpl.create(method);
+ final RelDataTypeFactory typeFactory = createTypeFactory();
+
+ List<RelDataType> argTypes = new ArrayList<>();
+ List<SqlTypeFamily> typeFamilies = new ArrayList<>();
+ for (FunctionParameter o : function.getParameters()) {
+ final RelDataType type = o.getType(typeFactory);
+ argTypes.add(type);
+ typeFamilies.add(Util.first(type.getSqlTypeName().getFamily(), SqlTypeFamily.ANY));
+ }
+
+ final FamilyOperandTypeChecker typeChecker =
+ OperandTypes.family(typeFamilies, i -> function.getParameters().get(i).isOptional());
+
+ final List<RelDataType> paramTypes = toSql(typeFactory, argTypes);
+
+ return new SqlUserDefinedFunction(
+ new SqlIdentifier(name, SqlParserPos.ZERO),
+ infer((ScalarFunction) function),
+ InferTypes.explicit(argTypes),
+ typeChecker,
+ paramTypes,
+ function);
+ }
+
+ private static RelDataType createSqlType(SqlTypeName typeName, boolean withNullability) {
+ final RelDataTypeFactory typeFactory = createTypeFactory();
+ RelDataType type = typeFactory.createSqlType(typeName);
+ if (withNullability) {
+ type = typeFactory.createTypeWithNullability(type, true);
+ }
+ return type;
+ }
+
+ private static RelDataTypeFactory createTypeFactory() {
+ return new SqlTypeFactoryImpl(BeamRelDataTypeSystem.INSTANCE);
+ }
+
+ private static SqlReturnTypeInference infer(final ScalarFunction function) {
+ return opBinding -> {
+ final RelDataTypeFactory typeFactory = opBinding.getTypeFactory();
+ final RelDataType type;
+ if (function instanceof ScalarFunctionImpl) {
+ type = ((ScalarFunctionImpl) function).getReturnType(typeFactory, opBinding);
+ } else {
+ type = function.getReturnType(typeFactory);
+ }
+ return toSql(typeFactory, type);
+ };
+ }
+
+ private static List<RelDataType> toSql(
+ final RelDataTypeFactory typeFactory, List<RelDataType> types) {
+ return Lists.transform(types, type -> toSql(typeFactory, type));
+ }
+
+ private static RelDataType toSql(RelDataTypeFactory typeFactory, RelDataType type) {
+ if (type instanceof RelDataTypeFactoryImpl.JavaType
+ && ((RelDataTypeFactoryImpl.JavaType) type).getJavaClass() == Object.class) {
+ return typeFactory.createTypeWithNullability(
+ typeFactory.createSqlType(SqlTypeName.ANY), true);
+ }
+ return JavaTypeFactoryImpl.toSql(typeFactory, type);
+ }
+
+ private static final RelDataType BIGINT_WITH_NULLABILITY =
+ createSqlType(SqlTypeName.BIGINT, true);
+
+ public static final SqlOperator START_WITHS =
+ createUdfOperator("STARTS_WITH", BeamBuiltinMethods.STARTS_WITH_METHOD);
+
+ public static final SqlOperator CONCAT =
+ createUdfOperator("CONCAT", BeamBuiltinMethods.CONCAT_METHOD);
+
+ public static final SqlOperator REPLACE =
+ createUdfOperator("REPLACE", BeamBuiltinMethods.REPLACE_METHOD);
+
+ public static final SqlOperator TRIM = createUdfOperator("TRIM", BeamBuiltinMethods.TRIM_METHOD);
+
+ public static final SqlOperator LTRIM =
+ createUdfOperator("LTRIM", BeamBuiltinMethods.LTRIM_METHOD);
+
+ public static final SqlOperator RTRIM =
+ createUdfOperator("RTRIM", BeamBuiltinMethods.RTRIM_METHOD);
+
+ public static final SqlOperator SUBSTR =
+ createUdfOperator("SUBSTR", BeamBuiltinMethods.SUBSTR_METHOD);
+
+ public static final SqlOperator REVERSE =
+ createUdfOperator("REVERSE", BeamBuiltinMethods.REVERSE_METHOD);
+
+ public static final SqlOperator CHAR_LENGTH =
+ createUdfOperator("CHAR_LENGTH", BeamBuiltinMethods.CHAR_LENGTH_METHOD);
+
+ public static final SqlOperator ENDS_WITH =
+ createUdfOperator("ENDS_WITH", BeamBuiltinMethods.ENDS_WITH_METHOD);
+
+ public static final SqlOperator LIKE = createUdfOperator("LIKE", BeamBuiltinMethods.LIKE_METHOD);
+
+ public static final SqlOperator VALIDATE_TIMESTAMP =
+ createUdfOperator(
+ "validateTimestamp",
+ DateTimeUtils.class,
+ "validateTimestamp",
+ x -> TIMESTAMP_WITH_NULLABILITY,
+ ImmutableList.of(TIMESTAMP));
+
+ public static final SqlOperator VALIDATE_TIME_INTERVAL =
+ createUdfOperator(
+ "validateIntervalArgument",
+ DateTimeUtils.class,
+ "validateTimeInterval",
+ x -> BIGINT_WITH_NULLABILITY,
+ ImmutableList.of(BIGINT, OTHER));
+
+ public static final SqlOperator TIMESTAMP_OP =
+ createUdfOperator("TIMESTAMP", BeamBuiltinMethods.TIMESTAMP_METHOD);
+
+ public static final SqlOperator DATE_OP =
+ createUdfOperator("DATE", BeamBuiltinMethods.DATE_METHOD);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
new file mode 100644
index 0000000..3079100
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.ZetaSQLFunction.FunctionSignatureId;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** SqlStdOperatorMappingTable. */
+@Internal
+public class SqlStdOperatorMappingTable {
+ static final List<FunctionSignatureId> ZETASQL_BUILTIN_FUNCTION_WHITELIST =
+ Arrays.asList(
+ FunctionSignatureId.FN_AND,
+ FunctionSignatureId.FN_OR,
+ FunctionSignatureId.FN_NOT,
+ FunctionSignatureId.FN_MULTIPLY_DOUBLE,
+ FunctionSignatureId.FN_MULTIPLY_INT64,
+ FunctionSignatureId.FN_MULTIPLY_NUMERIC,
+ FunctionSignatureId.FN_DIVIDE_DOUBLE,
+ FunctionSignatureId.FN_DIVIDE_NUMERIC,
+ FunctionSignatureId.FN_ADD_DOUBLE,
+ FunctionSignatureId.FN_ADD_INT64,
+ FunctionSignatureId.FN_ADD_NUMERIC,
+ FunctionSignatureId.FN_SUBTRACT_DOUBLE,
+ FunctionSignatureId.FN_SUBTRACT_INT64,
+ FunctionSignatureId.FN_SUBTRACT_NUMERIC,
+ FunctionSignatureId.FN_UNARY_MINUS_INT64,
+ FunctionSignatureId.FN_UNARY_MINUS_DOUBLE,
+ FunctionSignatureId.FN_UNARY_MINUS_NUMERIC,
+ FunctionSignatureId.FN_GREATER,
+ FunctionSignatureId.FN_GREATER_OR_EQUAL,
+ FunctionSignatureId.FN_LESS,
+ FunctionSignatureId.FN_LESS_OR_EQUAL,
+ FunctionSignatureId.FN_EQUAL,
+ FunctionSignatureId.FN_NOT_EQUAL,
+ FunctionSignatureId.FN_IS_NULL,
+ FunctionSignatureId.FN_IS_TRUE,
+ FunctionSignatureId.FN_IS_FALSE,
+ FunctionSignatureId.FN_STARTS_WITH_STRING,
+ FunctionSignatureId.FN_SUBSTR_STRING,
+ FunctionSignatureId.FN_TRIM_STRING,
+ FunctionSignatureId.FN_LTRIM_STRING,
+ FunctionSignatureId.FN_RTRIM_STRING,
+ FunctionSignatureId.FN_REPLACE_STRING,
+ FunctionSignatureId.FN_CONCAT_STRING,
+ FunctionSignatureId.FN_COUNT_STAR,
+ FunctionSignatureId.FN_COUNT,
+ FunctionSignatureId.FN_MAX,
+ FunctionSignatureId.FN_MIN,
+ FunctionSignatureId.FN_AVG_DOUBLE,
+ FunctionSignatureId.FN_AVG_INT64,
+ FunctionSignatureId.FN_AVG_NUMERIC,
+ FunctionSignatureId.FN_SUM_DOUBLE,
+ FunctionSignatureId.FN_SUM_INT64,
+ FunctionSignatureId.FN_SUM_NUMERIC,
+ FunctionSignatureId.FN_MOD_INT64,
+ FunctionSignatureId.FN_MOD_NUMERIC,
+ FunctionSignatureId.FN_CASE_NO_VALUE,
+ FunctionSignatureId.FN_CASE_WITH_VALUE,
+ FunctionSignatureId.FN_TIMESTAMP_ADD,
+ // TODO: FunctionSignatureId.FN_TIMESTAMP_SUB,
+ FunctionSignatureId.FN_FLOOR_DOUBLE,
+ FunctionSignatureId.FN_FLOOR_NUMERIC,
+ FunctionSignatureId.FN_CEIL_DOUBLE,
+ FunctionSignatureId.FN_CEIL_NUMERIC,
+ FunctionSignatureId.FN_REVERSE_STRING,
+ FunctionSignatureId.FN_CHAR_LENGTH_STRING,
+ FunctionSignatureId.FN_ENDS_WITH_STRING,
+ FunctionSignatureId.FN_STRING_LIKE,
+ FunctionSignatureId.FN_COALESCE,
+ FunctionSignatureId.FN_IF,
+ FunctionSignatureId.FN_IFNULL,
+ FunctionSignatureId.FN_NULLIF,
+ FunctionSignatureId.FN_EXTRACT_FROM_DATE,
+ FunctionSignatureId.FN_EXTRACT_FROM_DATETIME,
+ FunctionSignatureId.FN_EXTRACT_FROM_TIME,
+ FunctionSignatureId.FN_EXTRACT_FROM_TIMESTAMP,
+ FunctionSignatureId.FN_TIMESTAMP_FROM_STRING,
+ FunctionSignatureId.FN_TIMESTAMP_FROM_DATE,
+ // TODO: FunctionSignatureId.FN_TIMESTAMP_FROM_DATETIME
+ FunctionSignatureId.FN_DATE_FROM_YEAR_MONTH_DAY
+ // TODO: FunctionSignatureId.FN_DATE_FROM_TIMESTAMP
+ );
+
+ // todo: Some of operators defined here are later overridden in ZetaSQLPlannerImpl.
+ // We should remove them from this table and add generic way to provide custom
+ // implementation. (Ex.: timestamp_add)
+ public static final ImmutableMap<String, SqlOperator> ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR =
+ ImmutableMap.<String, SqlOperator>builder()
+ // grouped window function
+ .put("TUMBLE", SqlStdOperatorTable.TUMBLE)
+ .put("HOP", SqlStdOperatorTable.HOP)
+ .put("SESSION", SqlStdOperatorTable.SESSION)
+
+ // built-in logical operator
+ .put("$and", SqlStdOperatorTable.AND)
+ .put("$or", SqlStdOperatorTable.OR)
+ .put("$not", SqlStdOperatorTable.NOT)
+
+ // built-in comparison operator
+ .put("$equal", SqlStdOperatorTable.EQUALS)
+ .put("$not_equal", SqlStdOperatorTable.NOT_EQUALS)
+ .put("$greater", SqlStdOperatorTable.GREATER_THAN)
+ .put("$greater_or_equal", SqlStdOperatorTable.GREATER_THAN_OR_EQUAL)
+ .put("$less", SqlStdOperatorTable.LESS_THAN)
+ .put("$less_or_equal", SqlStdOperatorTable.LESS_THAN_OR_EQUAL)
+ .put("$like", SqlOperators.LIKE)
+ // .put("$in", SqlStdOperatorTable.IN)
+ // .put("$between", SqlStdOperatorTable.BETWEEN)
+ .put("$is_null", SqlStdOperatorTable.IS_NULL)
+ .put("$is_true", SqlStdOperatorTable.IS_TRUE)
+ .put("$is_false", SqlStdOperatorTable.IS_FALSE)
+
+ // +, -, *, /
+ .put("$add", SqlStdOperatorTable.PLUS)
+ .put("$subtract", SqlStdOperatorTable.MINUS)
+ .put("$multiply", SqlStdOperatorTable.MULTIPLY)
+ .put("$unary_minus", SqlStdOperatorTable.UNARY_MINUS)
+ .put("$divide", SqlStdOperatorTable.DIVIDE)
+
+ // built-in string function
+ .put("concat", SqlOperators.CONCAT)
+ // .put("lower", SqlStdOperatorTable.LOWER)
+ // .put("upper", SqlStdOperatorTable.UPPER)
+ .put("substr", SqlOperators.SUBSTR)
+ .put("trim", SqlOperators.TRIM)
+ .put("replace", SqlOperators.REPLACE)
+ .put("char_length", SqlOperators.CHAR_LENGTH)
+
+ // string function UDFs
+ // .put("strpos", )
+ // .put("length", )
+ // tells Calcite codegen that starts_with function is a udf.
+ .put("starts_with", SqlOperators.START_WITHS)
+ .put("ends_with", SqlOperators.ENDS_WITH)
+ .put("ltrim", SqlOperators.LTRIM)
+ .put("rtrim", SqlOperators.RTRIM)
+ // .put("regexp_match",)
+ // .put("regexp_extract",)
+ // .put("regexp_replace",)
+ // .put("regexp_extract_all",)
+ // .put("byte_length",)
+ // .put("format",)
+ // .put("split",)
+ // .put("regexp_contains", )
+ // .put("normalize",)
+ // .put("to_base32",)
+ // .put("to_base64",)
+ // .put("to_hex",)
+ // .put("from_base64",)
+ // .put("from_base32",)
+ // .put("from_hex",)
+ // .put("to_code_points")
+ // .put("code_points_to_string")
+ // .put("lpad", )
+ // .put("rpad", )
+ // .put("repeat", )
+ .put("reverse", SqlOperators.REVERSE)
+
+ // built-in aggregate function
+ .put("$count_star", SqlStdOperatorTable.COUNT)
+ // TODO: add support to all aggregate functions.
+ .put("max", SqlStdOperatorTable.MAX)
+ .put("min", SqlStdOperatorTable.MIN)
+ .put("avg", SqlStdOperatorTable.AVG)
+ .put("sum", SqlStdOperatorTable.SUM)
+ // .put("any_value", SqlStdOperatorTable.ANY_VALUE)
+ .put("count", SqlStdOperatorTable.COUNT)
+
+ // aggregate UDF
+ // .put("array_agg", )
+ // .put("array_concat_agg")
+ // .put("string_agg")
+ // .put("bit_and")
+ // .put("bit_or")
+ // .put("bit_xor")
+ // .put("logical_and")
+ // .put("logical_or")
+
+ // built-in statistical aggregate function
+ // .put("covar_pop", SqlStdOperatorTable.COVAR_POP)
+ // .put("covar_samp", SqlStdOperatorTable.COVAR_SAMP)
+ // .put("stddev_pop", SqlStdOperatorTable.STDDEV_POP)
+ // .put("stddev_samp", SqlStdOperatorTable.STDDEV_SAMP)
+ // .put("var_pop", SqlStdOperatorTable.VAR_POP)
+ // .put("var_samp", SqlStdOperatorTable.VAR_SAMP)
+
+ // statistical aggregate UDF
+ // .put("corr", )
+
+ // built-in approximate aggregate function
+ // .put("approx_count_distinct", SqlStdOperatorTable.APPROX_COUNT_DISTINCT)
+
+ // approximate aggregate UDF
+ // .put("approx_quantiles", )
+ // .put("approx_top_sum")
+
+ // HLL++ UDF
+ // hll_count.merge
+ // hll_count.extract
+ // hll_count.init
+ // hll_count.merge_partial
+
+ // CAST
+ // CAST operator does not go through lookup table.
+ // .put("cast", SqlStdOperatorTable.CAST)
+
+ // built-in math functions
+ // .put("math", SqlStdOperatorTable.ABS)
+ // .put("sign", SqlStdOperatorTable.SIGN)
+ // .put("round", SqlStdOperatorTable.ROUND)
+ .put("ceil", SqlStdOperatorTable.CEIL)
+ .put("floor", SqlStdOperatorTable.FLOOR)
+ .put("mod", SqlStdOperatorTable.MOD)
+ // .put("sqrt", SqlStdOperatorTable.SQRT)
+ // .put("exp", SqlStdOperatorTable.EXP)
+ // .put("ln and log", SqlStdOperatorTable.LN)
+ // .put("log10", SqlStdOperatorTable.LOG10)
+ // .put("cos", SqlStdOperatorTable.COS)
+ // .put("acos", SqlStdOperatorTable.ACOS)
+ // .put("sin", SqlStdOperatorTable.SIN)
+ // .put("asin", SqlStdOperatorTable.ASIN)
+ // .put("tan", SqlStdOperatorTable.TAN)
+ // .put("atan", SqlStdOperatorTable.ATAN)
+ // .put("atan2", SqlStdOperatorTable.ATAN2)
+ // .put("abs", SqlStdOperatorTable.ABS)
+ // .put("pow", SqlStdOperatorTable.POWER)
+ // .put("div", SqlStdOperatorTable.DIVIDE)
+ // .put("trunc", SqlStdOperatorTable.TRUNCATE)
+
+ // math UDF
+ // .put("is_inf",)
+ // .put("is_nan",)
+ // .put("ieee_divide")
+ // .put("safe_add")
+ // .put("safe_divide")
+ // .put("safe_subtract")
+ // .put("safe_multiply")
+ // .put("safe_negate")
+ // .put("greatest")
+ // .put("least")
+ // .put("log")
+ // .put("cosh")
+ // .put("acosh")
+ // .put("sinh")
+ // .put("asinh")
+ // .put("tanh")
+ // .put("atanh")
+
+ // Analytic functions
+ // .put("dense_rank", SqlStdOperatorTable.DENSE_RANK)
+ // .put("rank", SqlStdOperatorTable.RANK)
+ // .put("row_number", SqlStdOperatorTable.ROW_NUMBER)
+ // .put("percent_rank", SqlStdOperatorTable.PERCENT_RANK)
+ // .put("cume_dist", SqlStdOperatorTable.CUME_DIST)
+ // .put("ntile", SqlStdOperatorTable.NTILE)
+ // .put("lead", SqlStdOperatorTable.LEAD)
+ // .put("lag", SqlStdOperatorTable.LAG)
+ // .put("first_value", SqlStdOperatorTable.FIRST_VALUE)
+ // .put("last_value", SqlStdOperatorTable.LAST_VALUE)
+ // .put("nth_value", SqlStdOperatorTable.NTH_VALUE)
+
+ // .put("percentile_cont", )
+ // .put("percentile_disc",)
+
+ // misc functions
+ // .put("fingerprint")
+ // .put("fingerprint2011")
+
+ // hash functions
+ // .put("md5")
+ // .put("sha1")
+ // .put("sha256")
+ // .put("sha512")
+
+ // date functions
+ // .put("date_add", SqlStdOperatorTable.DATETIME_PLUS)
+ // .put("date_sub", SqlStdOperatorTable.MINUS_DATE)
+ .put("date", SqlOperators.DATE_OP)
+
+ // time functions
+ // .put("time_add", SqlStdOperatorTable.DATETIME_PLUS)
+ // .put("time_sub", SqlStdOperatorTable.MINUS_DATE)
+
+ // timestamp functions
+ .put(
+ "timestamp_add",
+ SqlStdOperatorTable.DATETIME_PLUS) // overridden in ZetaSQLPlannerImpl
+ // .put("timestamp_sub", SqlStdOperatorTable.MINUS_DATE)
+ .put("$extract", SqlStdOperatorTable.EXTRACT)
+ .put("timestamp", SqlOperators.TIMESTAMP_OP)
+
+ // other functions
+ // .put("session_user", SqlStdOperatorTable.SESSION_USER)
+ // .put("bit_cast_to_int32")
+ // .put("bit_cast_to_int64")
+ // .put("bit_cast_to_uint32")
+ // .put("bit_cast_to_uint64")
+ // .put("countif", )
+
+ // case operator
+ .put("$case_no_value", SqlStdOperatorTable.CASE)
+
+ // if operator - IF(cond, pos, neg) can actually be mapped directly to `CASE WHEN cond
+ // THEN pos ELSE neg`
+ .put("if", SqlStdOperatorTable.CASE)
+
+ // $case_no_value specializations
+ // all of these operators can have their operands adjusted to achieve the same thing with
+ // a call to $case_with_value
+ .put("$case_with_value", SqlStdOperatorTable.CASE)
+ .put("coalesce", SqlStdOperatorTable.CASE)
+ .put("ifnull", SqlStdOperatorTable.CASE)
+ .put("nullif", SqlStdOperatorTable.CASE)
+ .build();
+
+ // argument one and two should compose of a interval.
+ public static final ImmutableSet<String> FUNCTION_FAMILY_DATE_ADD =
+ ImmutableSet.of(
+ "date_add",
+ "date_sub",
+ "datetime_add",
+ "datetime_sub",
+ "time_add",
+ "time_sub",
+ "timestamp_add",
+ "timestamp_sub");
+
+ public static final ImmutableMap<String, SqlOperatorRewriter>
+ ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER =
+ ImmutableMap.<String, SqlOperatorRewriter>builder()
+ .put("$case_with_value", new SqlCaseWithValueOperatorRewriter())
+ .put("coalesce", new SqlCoalesceOperatorRewriter())
+ .put("ifnull", new SqlIfNullOperatorRewriter())
+ .put("nullif", new SqlNullIfOperatorRewriter())
+ .put("$extract", new SqlExtractTimestampOperatorRewriter())
+ .build();
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
new file mode 100644
index 0000000..c126370
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.regex.Pattern;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions;
+
+/** StringFunctions. */
+public class StringFunctions {
+ public static final String SUBSTR_PARAMETER_EXCEED_INTEGER =
+ "SUBSTR function only allows: "
+ + Integer.MIN_VALUE
+ + " <= position or length <= "
+ + Integer.MAX_VALUE;
+
+ @Strict
+ public static Boolean startsWith(String str1, String str2) {
+ return str1.startsWith(str2);
+ }
+
+ @Strict
+ public static Boolean endsWith(String str1, String str2) {
+ return str1.endsWith(str2);
+ }
+
+ @Strict
+ public static String concat(String arg) {
+ return arg;
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2) {
+ return concatIfNotIncludeNull(arg1, arg2);
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2, String arg3) {
+ return concatIfNotIncludeNull(arg1, arg2, arg3);
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2, String arg3, String arg4) {
+ return concatIfNotIncludeNull(arg1, arg2, arg3, arg4);
+ }
+
+ @Strict
+ public static String concat(String arg1, String arg2, String arg3, String arg4, String arg5) {
+ return concatIfNotIncludeNull(arg1, arg2, arg3, arg4, arg5);
+ }
+
+ @Strict
+ private static String concatIfNotIncludeNull(String... args) {
+ return String.join("", args);
+ }
+
+ // https://jira.apache.org/jira/browse/CALCITE-2889
+ // public static String concat(String... args) {
+ // StringBuilder stringBuilder = new StringBuilder();
+ // for (String arg : args) {
+ // stringBuilder.append(arg);
+ // }
+ // return stringBuilder.toString();
+ // }
+
+ @Strict
+ public static String replace(String origin, String target, String replacement) {
+ // Java's string.replace behaves differently when target = "". When target = "",
+ // Java's replace function replace every character in origin with replacement,
+ // while origin value should not be changed is expected in SQL.
+ if (target.length() == 0) {
+ return origin;
+ }
+
+ return origin.replace(target, replacement);
+ }
+
+ public static String trim(String str) {
+ return trim(str, " ");
+ }
+
+ @Strict
+ public static String trim(String str, String seek) {
+ return SqlFunctions.trim(true, true, seek, str, false);
+ }
+
+ public static String ltrim(String str) {
+ return ltrim(str, " ");
+ }
+
+ @Strict
+ public static String ltrim(String str, String seek) {
+ return SqlFunctions.trim(true, false, seek, str, false);
+ }
+
+ public static String rtrim(String str) {
+ return rtrim(str, " ");
+ }
+
+ @Strict
+ public static String rtrim(String str, String seek) {
+ return SqlFunctions.trim(false, true, seek, str, false);
+ }
+
+ public static String substr(String str, long from, long len) {
+ if (from > Integer.MAX_VALUE
+ || len > Integer.MAX_VALUE
+ || from < Integer.MIN_VALUE
+ || len < Integer.MIN_VALUE) {
+ throw new RuntimeException(SUBSTR_PARAMETER_EXCEED_INTEGER);
+ }
+ return SqlFunctions.substring(str, (int) from, (int) len);
+ }
+
+ @Strict
+ public static String reverse(String str) {
+ return new StringBuilder(str).reverse().toString();
+ }
+
+ @Strict
+ public static Long charLength(String str) {
+ return (long) str.length();
+ }
+
+ // ZetaSQL's LIKE statement does not support the ESCAPE clause. Instead it
+ // always uses \ as an escape character.
+ @Strict
+ public static Boolean like(String s, String pattern) {
+ String regex = sqlToRegexLike(pattern, '\\');
+ return Pattern.matches(regex, s);
+ }
+
+ private static final String JAVA_REGEX_SPECIALS = "[]()|^-+*?{}$\\.";
+
+ /**
+ * Translates a SQL LIKE pattern to Java regex pattern. Modified from Apache Calcite's
+ * Like.sqlToRegexLike
+ */
+ private static String sqlToRegexLike(String sqlPattern, char escapeChar) {
+ int i;
+ final int len = sqlPattern.length();
+ final StringBuilder javaPattern = new StringBuilder(len + len);
+ for (i = 0; i < len; i++) {
+ char c = sqlPattern.charAt(i);
+ if (c == escapeChar) {
+ if (i == (sqlPattern.length() - 1)) {
+ throw new RuntimeException("LIKE pattern ends with a backslash");
+ }
+ char nextChar = sqlPattern.charAt(++i);
+ if (JAVA_REGEX_SPECIALS.indexOf(nextChar) >= 0) {
+ javaPattern.append('\\');
+ }
+ javaPattern.append(nextChar);
+ } else if (c == '_') {
+ javaPattern.append('.');
+ } else if (c == '%') {
+ javaPattern.append("(?s:.*)");
+ } else {
+ if (JAVA_REGEX_SPECIALS.indexOf(c) >= 0) {
+ javaPattern.append('\\');
+ }
+ javaPattern.append(c);
+ }
+ }
+ return javaPattern.toString();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
new file mode 100644
index 0000000..a982d93
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.SimpleTable;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/** Utility methods to resolve a table, given a top-level Calcite schema and a table path. */
+public class TableResolution {
+
+ /**
+ * Returns Calcite Table by consulting the schema.
+ *
+ * <p>The way the schema is queried is defined by the name resolution strategey implemented by a
+ * TableResolver and stored as a TableResolutionContext in the context.
+ *
+ * <p>If no custom table resolution logic is provided, default one is used, which is: drill down
+ * the getSubschema() path until the second-to-last path element. We expect the path to be a table
+ * path, so the last element should be a valid table id, we don't expect anything else there.
+ *
+ * <p>This resembles a default Calcite planner strategy. One difference is that Calcite doesn't
+ * assume the last element is a table and will continue to call getSubschema(), making it
+ * impossible for a table provider to understand the context.
+ */
+ public static Table resolveCalciteTable(
+ Context context, SchemaPlus schemaPlus, List<String> tablePath) {
+ TableResolutionContext tableResolutionContext = context.unwrap(TableResolutionContext.class);
+ TableResolver tableResolver = getTableResolver(tableResolutionContext, schemaPlus.getName());
+ return tableResolver.resolveCalciteTable(schemaPlus, tablePath);
+ }
+
+ static TableResolver getTableResolver(
+ TableResolutionContext tableResolutionContext, String schemaName) {
+ if (tableResolutionContext == null
+ || !tableResolutionContext.hasCustomResolutionFor(schemaName)) {
+ return TableResolver.DEFAULT_ASSUME_LEAF_IS_TABLE;
+ }
+
+ return tableResolutionContext.getTableResolver(schemaName);
+ }
+
+ /**
+ * Data class to store simple table, its full path (excluding top-level schema), and top-level
+ * schema.
+ */
+ static class SimpleTableWithPath {
+
+ SimpleTable table;
+ List<String> path;
+ String topLevelSchema;
+
+ static SimpleTableWithPath of(String topLevelSchema, List<String> path) {
+ SimpleTableWithPath tableWithPath = new SimpleTableWithPath();
+ tableWithPath.table = new SimpleTable(Iterables.getLast(path));
+ tableWithPath.path = path;
+ tableWithPath.topLevelSchema = topLevelSchema;
+ return tableWithPath;
+ }
+
+ SimpleTable getTable() {
+ return table;
+ }
+
+ List<String> getPath() {
+ return path;
+ }
+
+ String getTopLevelSchema() {
+ return topLevelSchema;
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
new file mode 100644
index 0000000..3aed2c1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.Map;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.codehaus.commons.nullanalysis.Nullable;
+
+/**
+ * Calcite parser context to pass the configuration down to planner and rules so that we can
+ * configure custom name resolution.
+ */
+class TableResolutionContext implements Context {
+
+ /** Table resolvers, associating top-level schema to a custom name resolution logic. */
+ private final Map<String, TableResolver> resolvers;
+
+ /** Assigns a custom table resolver to the given schema. */
+ static TableResolutionContext of(String topLevelSchema, TableResolver resolver) {
+ return new TableResolutionContext(ImmutableMap.of(topLevelSchema, resolver));
+ }
+
+ /**
+ * Uses the resolution logic that joins the table path into a single compound identifier and then
+ * queries the schema once, instead of drilling down into subschemas.
+ */
+ static TableResolutionContext joinCompoundIds(String topLevelSchema) {
+ return of(topLevelSchema, TableResolver.JOIN_INTO_COMPOUND_ID);
+ }
+
+ TableResolutionContext with(String topLevelSchema, TableResolver resolver) {
+ return new TableResolutionContext(
+ ImmutableMap.<String, TableResolver>builder()
+ .putAll(this.resolvers)
+ .put(topLevelSchema, resolver)
+ .build());
+ }
+
+ boolean hasCustomResolutionFor(String schemaName) {
+ return resolvers.containsKey(schemaName);
+ }
+
+ @Nullable
+ TableResolver getTableResolver(String schemaName) {
+ return resolvers.get(schemaName);
+ }
+
+ private TableResolutionContext(Map<String, TableResolver> resolvers) {
+ this.resolvers = resolvers;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> T unwrap(Class<T> c) {
+ return c.isAssignableFrom(TableResolutionContext.class) ? (T) this : null;
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
new file mode 100644
index 0000000..b7e516e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+
+/** An interface to implement a custom resolution strategy. */
+interface TableResolver {
+
+ TableResolver DEFAULT_ASSUME_LEAF_IS_TABLE = TableResolverImpl::assumeLeafIsTable;
+ TableResolver JOIN_INTO_COMPOUND_ID = TableResolverImpl::joinIntoCompoundId;
+
+ /**
+ * Returns a resolved table given a table path.
+ *
+ * <p>Returns null if table is not found.
+ */
+ Table resolveCalciteTable(Schema calciteSchema, List<String> tablePath);
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
new file mode 100644
index 0000000..ad2f7ce
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/** A couple of implementations of TableResolver. */
+class TableResolverImpl {
+
+ /**
+ * Uses the logic similar to Calcite's EmptyScope.resolve_(...) except assumes the last element in
+ * the table path is a table name (which is assumed by ZetaSQL API getTableNames()).
+ *
+ * <p>This is the default.
+ *
+ * <p>I.e. drills down into schema.getSubschema() until the second last element of the table path,
+ * then calls schema.getTable().
+ */
+ static Table assumeLeafIsTable(Schema schema, List<String> tablePath) {
+ Schema subSchema = schema;
+
+ // subSchema.getSubschema() for all except last
+ for (int i = 0; i < tablePath.size() - 1; i++) {
+ subSchema = subSchema.getSubSchema(tablePath.get(i));
+ }
+
+ // for the final one call getTable()
+ return subSchema.getTable(Iterables.getLast(tablePath));
+ }
+
+ /**
+ * Joins the table name parts into a single ZetaSQL-compatible compound identifier, then calls
+ * schema.getTable().
+ *
+ * <p>This is the input expected, for example, by Data Catalog.
+ *
+ * <p>Escapes slashes, backticks, quotes, for details see {@link
+ * ZetaSqlIdUtils#escapeAndJoin(List)}.
+ */
+ static Table joinIntoCompoundId(Schema schema, List<String> tablePath) {
+ return schema.getTable(ZetaSqlIdUtils.escapeAndJoin(tablePath));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
new file mode 100644
index 0000000..9ad0a65
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.DateType;
+import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.TimeType;
+import org.apache.beam.sdk.extensions.sql.meta.provider.test.TestBoundedTable;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** TestInput. */
+class TestInput {
+ public static final FieldType DATE = FieldType.logicalType(new DateType());
+ public static final FieldType TIME = FieldType.logicalType(new TimeType());
+
+ public static final TestBoundedTable BASIC_TABLE_ONE =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("Key")
+ .addStringField("Value")
+ .addDateTimeField("ts")
+ .build())
+ .addRows(
+ 14L,
+ "KeyValue234",
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:06"),
+ 15L,
+ "KeyValue235",
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:07"));
+
+ public static final TestBoundedTable BASIC_TABLE_TWO =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("RowKey")
+ .addStringField("Value")
+ .addDateTimeField("ts")
+ .build())
+ .addRows(
+ 15L,
+ "BigTable235",
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:07"),
+ 16L,
+ "BigTable236",
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:08"));
+
+ public static final TestBoundedTable BASIC_TABLE_THREE =
+ TestBoundedTable.of(Schema.builder().addInt64Field("ColId").addStringField("Value").build())
+ .addRows(15L, "Spanner235", 16L, "Spanner236", 17L, "Spanner237");
+
+ public static final TestBoundedTable AGGREGATE_TABLE_ONE =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("Key")
+ .addInt64Field("Key2")
+ .addInt64Field("f_int_1")
+ .addStringField("f_str_1")
+ .addDoubleField("f_double_1")
+ .build())
+ .addRows(1L, 10L, 1L, "1", 1.0)
+ .addRows(1L, 11L, 2L, "2", 2.0)
+ .addRows(2L, 11L, 3L, "3", 3.0)
+ .addRows(2L, 11L, 4L, "4", 4.0)
+ .addRows(2L, 12L, 5L, "5", 5.0)
+ .addRows(3L, 13L, 6L, "6", 6.0)
+ .addRows(3L, 13L, 7L, "7", 7.0);
+
+ public static final TestBoundedTable AGGREGATE_TABLE_TWO =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("Key")
+ .addInt64Field("Key2")
+ .addInt64Field("f_int_1")
+ .addStringField("f_str_1")
+ .build())
+ .addRows(1L, 10L, 1L, "1")
+ .addRows(2L, 11L, 3L, "3")
+ .addRows(2L, 11L, 4L, "4")
+ .addRows(2L, 12L, 5L, "5")
+ .addRows(2L, 13L, 6L, "6")
+ .addRows(3L, 13L, 7L, "7");
+
+ public static final TestBoundedTable TABLE_ALL_TYPES =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("row_id")
+ .addBooleanField("bool_col")
+ .addInt64Field("int64_col")
+ .addDoubleField("double_col")
+ .addStringField("str_col")
+ .addByteArrayField("bytes_col")
+ .build())
+ .addRows(1L, true, -1L, 0.125d, "1", stringToBytes("1"))
+ .addRows(2L, false, -2L, Math.pow(0.1, 324.0), "2", stringToBytes("2"))
+ .addRows(3L, true, -3L, 0.375d, "3", stringToBytes("3"))
+ .addRows(4L, false, -4L, 0.5d, "4", stringToBytes("4"))
+ .addRows(5L, false, -5L, 0.5d, "5", stringToBytes("5"));
+
+ public static final TestBoundedTable TABLE_ALL_TYPES_2 =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("row_id")
+ .addBooleanField("bool_col")
+ .addInt64Field("int64_col")
+ .addDoubleField("double_col")
+ .addStringField("str_col")
+ .addByteArrayField("bytes_col")
+ .build())
+ .addRows(6L, true, -6L, 0.125d, "6", stringToBytes("6"))
+ .addRows(7L, false, -7L, Math.pow(0.1, 324.0), "7", stringToBytes("7"))
+ .addRows(8L, true, -8L, 0.375d, "8", stringToBytes("8"))
+ .addRows(9L, false, -9L, 0.5d, "9", stringToBytes("9"))
+ .addRows(10L, false, -10L, 0.5d, "10", stringToBytes("10"));
+
+ public static final TestBoundedTable TIMESTAMP_TABLE_ONE =
+ TestBoundedTable.of(Schema.builder().addDateTimeField("ts").addInt64Field("value").build())
+ .addRows(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:06"),
+ 3L,
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:07"),
+ 4L,
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:08"),
+ 6L,
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:09"),
+ 7L);
+
+ public static final TestBoundedTable TIMESTAMP_TABLE_TWO =
+ TestBoundedTable.of(Schema.builder().addDateTimeField("ts").addInt64Field("value").build())
+ .addRows(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:06"),
+ 3L,
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:07"),
+ 4L,
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:12"),
+ 6L,
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-01 21:26:13"),
+ 7L);
+
+ public static final TestBoundedTable TIME_TABLE =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addNullableField("f_date", DATE)
+ .addNullableField("f_time", TIME)
+ .addNullableField("f_timestamp", FieldType.DATETIME)
+ .addNullableField("f_timestamp_with_time_zone", FieldType.DATETIME)
+ .build())
+ .addRows(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-07-11 00:00:00"),
+ DateTimeUtils.parseTimestampWithUTCTimeZone("1970-01-01 12:33:59.348"),
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2018-12-20 23:59:59.999"),
+ DateTimeUtils.parseTimestampWithTimeZone("2018-12-10 10:38:59-1000"));
+
+ public static final TestBoundedTable TABLE_ALL_NULL =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addNullableField("primary_key", FieldType.INT64)
+ .addNullableField("bool_val", FieldType.BOOLEAN)
+ .addNullableField("double_val", FieldType.DOUBLE)
+ .addNullableField("int64_val", FieldType.INT64)
+ .addNullableField("str_val", FieldType.STRING)
+ .build())
+ .addRows(1L, null, null, null, null);
+
+ public static final Schema TABLE_WITH_STRUCT_ROW_SCHEMA =
+ Schema.builder().addInt32Field("struct_col_int").addStringField("struct_col_str").build();
+
+ public static final TestBoundedTable TABLE_WITH_STRUCT =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addField("id", FieldType.INT64)
+ .addField("struct_col", FieldType.row(TABLE_WITH_STRUCT_ROW_SCHEMA))
+ .build())
+ .addRows(
+ 1L,
+ Row.withSchema(TABLE_WITH_STRUCT_ROW_SCHEMA).addValues(16, "row_one").build(),
+ 2L,
+ Row.withSchema(TABLE_WITH_STRUCT_ROW_SCHEMA).addValues(17, "row_two").build());
+
+ public static final TestBoundedTable TABLE_WITH_STRUCT_TIMESTAMP_STRING =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addField("struct_col", FieldType.row(TABLE_WITH_STRUCT_ROW_SCHEMA))
+ .build())
+ .addRows(
+ Row.withSchema(TABLE_WITH_STRUCT_ROW_SCHEMA)
+ .addValues(3, "2019-01-15 13:21:03")
+ .build());
+
+ private static final Schema structSchema =
+ Schema.builder().addInt64Field("row_id").addStringField("data").build();
+ private static final Schema structTableSchema =
+ Schema.builder().addRowField("rowCol", structSchema).build();
+
+ public static final TestBoundedTable TABLE_WITH_STRUCT_TWO =
+ TestBoundedTable.of(structTableSchema)
+ .addRows(Row.withSchema(structSchema).addValues(1L, "data1").build())
+ .addRows(Row.withSchema(structSchema).addValues(2L, "data2").build())
+ .addRows(Row.withSchema(structSchema).addValues(3L, "data2").build())
+ .addRows(Row.withSchema(structSchema).addValues(3L, "data3").build());
+
+ public static final TestBoundedTable TABLE_WITH_ARRAY =
+ TestBoundedTable.of(Schema.builder().addArrayField("array_col", FieldType.STRING).build())
+ .addRows(Arrays.asList("1", "2", "3"), ImmutableList.of());
+
+ public static final TestBoundedTable TABLE_WITH_ARRAY_FOR_UNNEST =
+ TestBoundedTable.of(
+ Schema.builder()
+ .addInt64Field("int_col")
+ .addArrayField("int_array_col", FieldType.INT64)
+ .build())
+ .addRows(14L, Arrays.asList(14L, 18L))
+ .addRows(18L, Arrays.asList(22L, 24L));
+
+ public static final TestBoundedTable TABLE_FOR_CASE_WHEN =
+ TestBoundedTable.of(
+ Schema.builder().addInt64Field("f_int").addStringField("f_string").build())
+ .addRows(1L, "20181018");
+
+ public static final TestBoundedTable TABLE_EMPTY =
+ TestBoundedTable.of(Schema.builder().addInt64Field("ColId").addStringField("Value").build());
+
+ private static final Schema TABLE_WTH_MAP_SCHEMA =
+ Schema.builder()
+ .addMapField("map_field", FieldType.STRING, FieldType.STRING)
+ .addRowField("row_field", structSchema)
+ .build();
+ public static final TestBoundedTable TABLE_WITH_MAP =
+ TestBoundedTable.of(TABLE_WTH_MAP_SCHEMA)
+ .addRows(
+ ImmutableMap.of("MAP_KEY_1", "MAP_VALUE_1"),
+ Row.withSchema(structSchema).addValues(1L, "data1").build());
+
+ public static byte[] stringToBytes(String s) {
+ return s.getBytes(StandardCharsets.UTF_8);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
new file mode 100644
index 0000000..9bbbb9b
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import java.util.TimeZone;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+
+/** TimestampFunctions. */
+public class TimestampFunctions {
+ public static DateTime timestamp(String timestampStr) {
+ return timestamp(timestampStr, "UTC");
+ }
+
+ @Strict
+ public static DateTime timestamp(String timestampStr, String timezone) {
+ return DateTimeUtils.findDateTimePattern(timestampStr)
+ .withZone(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)))
+ .parseDateTime(timestampStr);
+ }
+
+ @Strict
+ public static DateTime timestamp(Integer numOfDays) {
+ return timestamp(numOfDays, "UTC");
+ }
+
+ @Strict
+ public static DateTime timestamp(Integer numOfDays, String timezone) {
+ return new DateTime((long) numOfDays * DateTimeUtils.MILLIS_PER_DAY, DateTimeZone.UTC)
+ .withZoneRetainFields(DateTimeZone.forTimeZone(TimeZone.getTimeZone(timezone)));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
new file mode 100644
index 0000000..0967f2e
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BOOL;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BYTES;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DATE;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DOUBLE;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_FLOAT;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT32;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT64;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_NUMERIC;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_STRING;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIME;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIMESTAMP;
+import static java.util.stream.Collectors.toList;
+
+import com.google.zetasql.ArrayType;
+import com.google.zetasql.StructType;
+import com.google.zetasql.StructType.StructField;
+import com.google.zetasql.Type;
+import com.google.zetasql.TypeFactory;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** Utility to convert types from Calcite Schema types. */
+@Internal
+public class TypeUtils {
+
+ private static final ImmutableMap<SqlTypeName, Type> CALCITE_TO_ZETA_SIMPLE_TYPES =
+ ImmutableMap.<SqlTypeName, Type>builder()
+ .put(SqlTypeName.BIGINT, TypeFactory.createSimpleType(TYPE_INT64))
+ .put(SqlTypeName.INTEGER, TypeFactory.createSimpleType(TYPE_INT32))
+ .put(SqlTypeName.VARCHAR, TypeFactory.createSimpleType(TYPE_STRING))
+ .put(SqlTypeName.BOOLEAN, TypeFactory.createSimpleType(TYPE_BOOL))
+ .put(SqlTypeName.FLOAT, TypeFactory.createSimpleType(TYPE_FLOAT))
+ .put(SqlTypeName.DOUBLE, TypeFactory.createSimpleType(TYPE_DOUBLE))
+ .put(SqlTypeName.VARBINARY, TypeFactory.createSimpleType(TYPE_BYTES))
+ .put(SqlTypeName.TIMESTAMP, TypeFactory.createSimpleType(TYPE_TIMESTAMP))
+ .put(SqlTypeName.DATE, TypeFactory.createSimpleType(TYPE_DATE))
+ .put(SqlTypeName.TIME, TypeFactory.createSimpleType(TYPE_TIME))
+ .build();
+
+ private static final ImmutableMap<TypeKind, Function<RexBuilder, RelDataType>>
+ ZETA_TO_CALCITE_SIMPLE_TYPES =
+ ImmutableMap.<TypeKind, Function<RexBuilder, RelDataType>>builder()
+ .put(TYPE_NUMERIC, relDataTypeFactory(SqlTypeName.DECIMAL))
+ .put(TYPE_INT32, relDataTypeFactory(SqlTypeName.INTEGER))
+ .put(TYPE_INT64, relDataTypeFactory(SqlTypeName.BIGINT))
+ .put(TYPE_FLOAT, relDataTypeFactory(SqlTypeName.FLOAT))
+ .put(TYPE_DOUBLE, relDataTypeFactory(SqlTypeName.DOUBLE))
+ .put(TYPE_STRING, relDataTypeFactory(SqlTypeName.VARCHAR))
+ .put(TYPE_BOOL, relDataTypeFactory(SqlTypeName.BOOLEAN))
+ .put(TYPE_BYTES, relDataTypeFactory(SqlTypeName.VARBINARY))
+ .put(TYPE_DATE, relDataTypeFactory(SqlTypeName.DATE))
+ .put(TYPE_TIME, relDataTypeFactory(SqlTypeName.TIME))
+ // TODO: handle timestamp with time zone.
+ .put(TYPE_TIMESTAMP, relDataTypeFactory(SqlTypeName.TIMESTAMP))
+ .build();
+
+ /** Returns a type matching the corresponding Calcite type. */
+ static Type toZetaType(RelDataType calciteType) {
+
+ if (CALCITE_TO_ZETA_SIMPLE_TYPES.containsKey(calciteType.getSqlTypeName())) {
+ return CALCITE_TO_ZETA_SIMPLE_TYPES.get(calciteType.getSqlTypeName());
+ }
+
+ switch (calciteType.getSqlTypeName()) {
+ case ARRAY:
+ return TypeFactory.createArrayType(toZetaType(calciteType.getComponentType()));
+ case MAP:
+
+ // it is ok to return a simple type for MAP because MAP only exists in pubsub table which
+ // used to save table attribute.
+ // TODO: have a better way to handle MAP given the fact that ZetaSQL has no MAP type.
+ return TypeFactory.createSimpleType(TypeKind.TYPE_STRING);
+ case ROW:
+ List<StructField> structFields =
+ calciteType.getFieldList().stream()
+ .map(f -> new StructField(f.getName(), toZetaType(f.getType())))
+ .collect(toList());
+
+ return TypeFactory.createStructType(structFields);
+ default:
+ throw new RuntimeException("Unsupported RelDataType: " + calciteType);
+ }
+ }
+
+ public static RelDataType toRelDataType(RexBuilder rexBuilder, Type type, boolean isNullable) {
+ if (type.getKind().equals(TypeKind.TYPE_ARRAY)) {
+ return toArrayRelDataType(rexBuilder, type.asArray(), isNullable);
+ } else if (type.getKind().equals(TypeKind.TYPE_STRUCT)) {
+ return toStructRelDataType(rexBuilder, type.asStruct(), isNullable);
+ } else {
+ // TODO: Check type's nullability?
+ return toSimpleRelDataType(type.getKind(), rexBuilder, isNullable);
+ }
+ }
+
+ public static RelDataType toArrayRelDataType(
+ RexBuilder rexBuilder, ArrayType arrayType, boolean isNullable) {
+ // -1 cardinality means unlimited array size.
+ // TODO: is unlimited array size right for general case?
+ // TODO: whether isNullable should be ArrayType's nullablity (not its element type's?)
+ return rexBuilder
+ .getTypeFactory()
+ .createArrayType(toRelDataType(rexBuilder, arrayType.getElementType(), isNullable), -1);
+ }
+
+ private static RelDataType toStructRelDataType(
+ RexBuilder rexBuilder, StructType structType, boolean isNullable) {
+
+ List<StructField> fields = structType.getFieldList();
+ List<String> fieldNames = fields.stream().map(StructField::getName).collect(toList());
+ List<RelDataType> fieldTypes =
+ fields.stream()
+ .map(f -> toRelDataType(rexBuilder, f.getType(), isNullable))
+ .collect(toList());
+
+ return rexBuilder.getTypeFactory().createStructType(fieldTypes, fieldNames);
+ }
+
+ // TODO: convert TIMESTAMP with/without TIMEZONE and DATETIME.
+ public static RelDataType toSimpleRelDataType(TypeKind kind, RexBuilder rexBuilder) {
+ return toSimpleRelDataType(kind, rexBuilder, true);
+ }
+
+ public static RelDataType toSimpleRelDataType(
+ TypeKind kind, RexBuilder rexBuilder, boolean isNullable) {
+ if (!ZETA_TO_CALCITE_SIMPLE_TYPES.containsKey(kind)) {
+ throw new RuntimeException("Unsupported column type: " + kind);
+ }
+
+ RelDataType relDataType = ZETA_TO_CALCITE_SIMPLE_TYPES.get(kind).apply(rexBuilder);
+ return nullable(rexBuilder, relDataType, isNullable);
+ }
+
+ private static RelDataType nullable(RexBuilder r, RelDataType relDataType, boolean isNullable) {
+ return r.getTypeFactory().createTypeWithNullability(relDataType, isNullable);
+ }
+
+ private static Function<RexBuilder, RelDataType> relDataTypeFactory(SqlTypeName typeName) {
+ return (RexBuilder r) -> r.getTypeFactory().createSqlType(typeName);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
new file mode 100644
index 0000000..b0e8e56
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexImpTable.createImplementor;
+
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.CallImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.NotNullImplementor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.NullPolicy;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexImpTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.enumerable.RexToLixTranslator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expression;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.tree.Expressions;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.FunctionParameter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.ImplementableFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIdentifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlUserDefinedFunction;
+
+/** ZetaSQLCastFunctionImpl. */
+public class ZetaSQLCastFunctionImpl implements Function, ImplementableFunction {
+ public static final SqlUserDefinedFunction ZETASQL_CAST_OP =
+ new SqlUserDefinedFunction(
+ new SqlIdentifier("CAST", SqlParserPos.ZERO),
+ null,
+ null,
+ null,
+ null,
+ new ZetaSQLCastFunctionImpl());
+
+ @Override
+ public CallImplementor getImplementor() {
+ return createImplementor(new ZetaSQLCastCallNotNullImplementor(), NullPolicy.STRICT, false);
+ }
+
+ @Override
+ public List<FunctionParameter> getParameters() {
+ return null;
+ }
+
+ private static class ZetaSQLCastCallNotNullImplementor implements NotNullImplementor {
+
+ @Override
+ public Expression implement(
+ RexToLixTranslator rexToLixTranslator, RexCall rexCall, List<Expression> list) {
+ if (rexCall.getOperands().size() != 1 || list.size() != 1) {
+ throw new RuntimeException("CAST should have one operand.");
+ }
+ SqlTypeName toType = rexCall.getType().getSqlTypeName();
+ SqlTypeName fromType = rexCall.getOperands().get(0).getType().getSqlTypeName();
+
+ Expression translatedOperand = list.get(0);
+ Expression convertedOperand;
+ // CAST(BYTES AS STRING) - BINARY to VARCHAR in Calcite
+ if (fromType == SqlTypeName.BINARY && toType == SqlTypeName.VARCHAR) {
+ // operand is literal, which is bytes wrapped in ByteString.
+ // this piece of code is same as
+ // BeamCodegenUtils.toStringUTF8(ByeString.getBytes());
+ convertedOperand =
+ Expressions.call(
+ BeamCodegenUtils.class,
+ "toStringUTF8",
+ Expressions.call(translatedOperand, "getBytes"));
+ } else if (fromType == SqlTypeName.VARBINARY && toType == SqlTypeName.VARCHAR) {
+ // translatedOperand is a byte[]
+ // this piece of code is same as
+ // BeamCodegenUtils.toStringUTF8(byte[]);
+ convertedOperand =
+ Expressions.call(BeamCodegenUtils.class, "toStringUTF8", translatedOperand);
+ } else if (fromType == SqlTypeName.BOOLEAN && toType == SqlTypeName.BIGINT) {
+ convertedOperand =
+ Expressions.condition(
+ translatedOperand,
+ Expressions.constant(1L, Long.class),
+ Expressions.constant(0L, Long.class));
+ } else if (fromType == SqlTypeName.BIGINT && toType == SqlTypeName.BOOLEAN) {
+ convertedOperand = Expressions.notEqual(translatedOperand, Expressions.constant(0));
+ } else if (fromType == SqlTypeName.TIMESTAMP && toType == SqlTypeName.VARCHAR) {
+ convertedOperand =
+ Expressions.call(BeamCodegenUtils.class, "toStringTimestamp", translatedOperand);
+ } else {
+ throw new RuntimeException("Unsupported CAST: " + fromType.name() + " to " + toType.name());
+ }
+
+ // If operand is nullable, wrap in a null check
+ if (rexCall.getOperands().get(0).getType().isNullable()) {
+ convertedOperand =
+ Expressions.condition(
+ Expressions.equal(translatedOperand, RexImpTable.NULL_EXPR),
+ RexImpTable.NULL_EXPR,
+ convertedOperand);
+ }
+
+ return convertedOperand;
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
new file mode 100644
index 0000000..5afdcd4
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.LanguageOptions;
+import com.google.zetasql.Value;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedStatement;
+import java.io.Reader;
+import java.util.Map;
+import java.util.logging.Logger;
+import org.apache.beam.sdk.extensions.sql.zetasql.translation.ConversionContext;
+import org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter;
+import org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.metadata.CachingRelMetadataProvider;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexExecutor;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParseException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Planner;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Program;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Pair;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+
+/** ZetaSQLPlannerImpl. */
+public class ZetaSQLPlannerImpl implements Planner {
+ private static final Logger logger = Logger.getLogger(ZetaSQLPlannerImpl.class.getName());
+
+ private final SchemaPlus defaultSchemaPlus;
+
+ // variables that are used in Calcite's planner.
+ private final FrameworkConfig config;
+ private RelOptPlanner planner;
+ private JavaTypeFactory typeFactory;
+ private final RexExecutor executor;
+ private RelOptCluster cluster;
+ private final ImmutableList<Program> programs;
+ private ExpressionConverter expressionConverter;
+
+ private static final long ONE_SECOND_IN_MILLIS = 1000L;
+ private static final long ONE_MINUTE_IN_MILLIS = 60L * ONE_SECOND_IN_MILLIS;
+ private static final long ONE_HOUR_IN_MILLIS = 60L * ONE_MINUTE_IN_MILLIS;
+ private static final long ONE_DAY_IN_MILLIS = 24L * ONE_HOUR_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_MONTH_IN_MILLIS = 30L * ONE_DAY_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_YEAR_IN_MILLIS = 365L * ONE_DAY_IN_MILLIS;
+
+ public ZetaSQLPlannerImpl(FrameworkConfig config) {
+ this.config = config;
+ this.executor = config.getExecutor();
+ this.programs = config.getPrograms();
+
+ Frameworks.withPlanner(
+ (cluster, relOptSchema, rootSchema) -> {
+ Util.discard(rootSchema); // use our own defaultSchema
+ typeFactory = (JavaTypeFactory) cluster.getTypeFactory();
+ planner = cluster.getPlanner();
+ planner.setExecutor(executor);
+ return null;
+ },
+ config);
+
+ this.defaultSchemaPlus = config.getDefaultSchema();
+ }
+
+ @Override
+ public SqlNode parse(String s) throws SqlParseException {
+ return null;
+ }
+
+ @Override
+ public SqlNode parse(Reader reader) throws SqlParseException {
+ return null;
+ }
+
+ @Override
+ public SqlNode validate(SqlNode sqlNode) throws ValidationException {
+ return null;
+ }
+
+ @Override
+ public Pair<SqlNode, RelDataType> validateAndGetType(SqlNode sqlNode) throws ValidationException {
+ throw new RuntimeException("validateAndGetType(SqlNode) is not implemented.");
+ }
+
+ @Override
+ public RelRoot rel(SqlNode sqlNode) throws RelConversionException {
+ return null;
+ }
+
+ public RelRoot rel(String sql, Map<String, Value> params) {
+ this.cluster = RelOptCluster.create(planner, new RexBuilder(typeFactory));
+ this.expressionConverter = new ExpressionConverter(cluster, params);
+
+ QueryTrait trait = new QueryTrait();
+
+ ResolvedStatement statement =
+ SqlAnalyzer.withQueryParams(params)
+ .withQueryTrait(trait)
+ .withCalciteContext(config.getContext())
+ .withTopLevelSchema(defaultSchemaPlus)
+ .withTypeFactory((JavaTypeFactory) cluster.getTypeFactory())
+ .analyze(sql);
+
+ if (!(statement instanceof ResolvedQueryStmt)) {
+ throw new UnsupportedOperationException(
+ "Unsupported query statement type: " + sql.getClass().getSimpleName());
+ }
+
+ ConversionContext context = ConversionContext.of(config, expressionConverter, cluster, trait);
+
+ RelNode convertedNode =
+ QueryStatementConverter.convertRootQuery(context, (ResolvedQueryStmt) statement);
+ return RelRoot.of(convertedNode, SqlKind.ALL);
+ }
+
+ @Override
+ public RelNode convert(SqlNode sqlNode) {
+ throw new RuntimeException("convert(SqlNode) is not implemented.");
+ }
+
+ @Override
+ public RelDataTypeFactory getTypeFactory() {
+ throw new RuntimeException("getTypeFactory() is not implemented.");
+ }
+
+ @Override
+ public RelNode transform(int i, RelTraitSet relTraitSet, RelNode relNode)
+ throws RelConversionException {
+ relNode
+ .getCluster()
+ .setMetadataProvider(
+ new CachingRelMetadataProvider(
+ relNode.getCluster().getMetadataProvider(), relNode.getCluster().getPlanner()));
+ Program program = programs.get(i);
+ return program.run(planner, relNode, relTraitSet, ImmutableList.of(), ImmutableList.of());
+ }
+
+ @Override
+ public void reset() {
+ throw new RuntimeException("reset() is not implemented.");
+ }
+
+ @Override
+ public void close() {
+ // no-op
+ }
+
+ @Override
+ public RelTraitSet getEmptyTraitSet() {
+ throw new RuntimeException("getEmptyTraitSet() is not implemented.");
+ }
+
+ public static LanguageOptions getLanguageOptions() {
+ return SqlAnalyzer.initAnalyzerOptions().getLanguageOptions();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
new file mode 100644
index 0000000..3730857
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import com.google.zetasql.Value;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.ParseException;
+import org.apache.beam.sdk.extensions.sql.impl.QueryPlanner;
+import org.apache.beam.sdk.extensions.sql.impl.SqlConversionException;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParser;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserImplFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.util.ChainedSqlOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RelConversionException;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** ZetaSQLQueryPlanner. */
+public class ZetaSQLQueryPlanner implements QueryPlanner {
+ private final ZetaSQLPlannerImpl plannerImpl;
+
+ public ZetaSQLQueryPlanner(FrameworkConfig config) {
+ plannerImpl = new ZetaSQLPlannerImpl(config);
+ }
+
+ public ZetaSQLQueryPlanner(JdbcConnection jdbcConnection, RuleSet[] ruleSets) {
+ plannerImpl = new ZetaSQLPlannerImpl(defaultConfig(jdbcConnection, ruleSets));
+ }
+
+ @Override
+ public BeamRelNode convertToBeamRel(String sqlStatement)
+ throws ParseException, SqlConversionException {
+ try {
+ return parseQuery(sqlStatement);
+ } catch (RelConversionException e) {
+ throw new SqlConversionException(e.getCause());
+ }
+ }
+
+ @Override
+ public SqlNode parse(String sqlStatement) throws ParseException {
+ return null;
+ }
+
+ public BeamRelNode convertToBeamRel(String sqlStatement, Map<String, Value> queryParams)
+ throws ParseException, SqlConversionException {
+ try {
+ return parseQuery(sqlStatement, queryParams);
+ } catch (RelConversionException e) {
+ throw new SqlConversionException(e.getCause());
+ }
+ }
+
+ public BeamRelNode parseQuery(String sql) throws RelConversionException {
+ return parseQuery(sql, Collections.emptyMap());
+ }
+
+ public BeamRelNode parseQuery(String sql, Map<String, Value> queryParams)
+ throws RelConversionException {
+ RelRoot root = plannerImpl.rel(sql, queryParams);
+ RelTraitSet desiredTraits =
+ root.rel
+ .getTraitSet()
+ .replace(BeamLogicalConvention.INSTANCE)
+ .replace(root.collation)
+ .simplify();
+ BeamRelNode beamRelNode = (BeamRelNode) plannerImpl.transform(0, desiredTraits, root.rel);
+ return beamRelNode;
+ }
+
+ private FrameworkConfig defaultConfig(JdbcConnection connection, RuleSet[] ruleSets) {
+ final CalciteConnectionConfig config = connection.config();
+ final SqlParser.ConfigBuilder parserConfig =
+ SqlParser.configBuilder()
+ .setQuotedCasing(config.quotedCasing())
+ .setUnquotedCasing(config.unquotedCasing())
+ .setQuoting(config.quoting())
+ .setConformance(config.conformance())
+ .setCaseSensitive(config.caseSensitive());
+ final SqlParserImplFactory parserFactory =
+ config.parserFactory(SqlParserImplFactory.class, null);
+ if (parserFactory != null) {
+ parserConfig.setParserFactory(parserFactory);
+ }
+
+ final SchemaPlus schema = connection.getRootSchema();
+ final SchemaPlus defaultSchema = connection.getCurrentSchemaPlus();
+
+ final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ final CalciteCatalogReader catalogReader =
+ new CalciteCatalogReader(
+ CalciteSchema.from(schema),
+ ImmutableList.of(defaultSchema.getName()),
+ connection.getTypeFactory(),
+ connection.config());
+ final SqlOperatorTable opTab0 =
+ connection.config().fun(SqlOperatorTable.class, SqlStdOperatorTable.instance());
+
+ return Frameworks.newConfigBuilder()
+ .parserConfig(parserConfig.build())
+ .defaultSchema(defaultSchema)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(connection.config()))
+ .ruleSets(ruleSets)
+ .costFactory(null)
+ .typeSystem(connection.getTypeFactory().getTypeSystem())
+ .operatorTable(ChainedSqlOperatorTable.of(opTab0, catalogReader))
+ .build();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
new file mode 100644
index 0000000..8529241
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static java.util.stream.Collectors.joining;
+
+import java.util.List;
+import java.util.regex.Pattern;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** Utils to work with ZetaSQL-compatible IDs. */
+class ZetaSqlIdUtils {
+
+ /**
+ * Some special characters we explcitily handle.
+ *
+ * <p>Everything else is ignored, e.g. tabs, newlines, etc.
+ */
+ private static final Pattern SPECIAL_CHARS_ESCAPE =
+ Pattern.compile(
+ "(?<SpecialChar>["
+ + "\\\\" // slash
+ + "`" // backtick
+ + "'" // single quote
+ + "\"" // double quote
+ + "?" // question mark
+ + "])");
+
+ private static final ImmutableMap<String, String> WHITESPACES =
+ ImmutableMap.of(
+ "\n", "\\\\n",
+ "\t", "\\\\t",
+ "\r", "\\\\r",
+ "\f", "\\\\f");
+
+ private static final Pattern SIMPLE_ID = Pattern.compile("[A-Za-z_][A-Za-z_0-9]*");
+
+ /**
+ * Joins parts into a single compound ZetaSQL identifier.
+ *
+ * <p>Escapes backticks, slashes, double and single quotes, doesn't handle other special
+ * characters for now.
+ */
+ static String escapeAndJoin(List<String> parts) {
+ String escaped =
+ parts.stream()
+ .map(ZetaSqlIdUtils::escapeSpecialChars)
+ .map(ZetaSqlIdUtils::replaceWhitespaces)
+ .map(ZetaSqlIdUtils::backtickIfNeeded)
+ .collect(joining("."));
+ return escaped;
+ }
+
+ private static String escapeSpecialChars(String str) {
+ return SPECIAL_CHARS_ESCAPE.matcher(str).replaceAll("\\\\${SpecialChar}");
+ }
+
+ private static String replaceWhitespaces(String s) {
+ String result = s;
+ for (String whitespace : WHITESPACES.keySet()) {
+ result = result.replaceAll(whitespace, WHITESPACES.get(whitespace));
+ }
+ return result;
+ }
+
+ private static String backtickIfNeeded(String s) {
+ return SIMPLE_ID.matcher(s).matches() ? s : ("`" + s + "`");
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
new file mode 100644
index 0000000..51f4348
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** ZetaSQL Dialect package. */
+package org.apache.beam.sdk.extensions.sql.zetasql;
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
new file mode 100644
index 0000000..0339592
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_CAST;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_COLUMN_REF;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_GET_STRUCT_FIELD;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils.toSimpleRelDataType;
+
+import com.google.zetasql.FunctionSignature;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateFunctionCall;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedComputedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedExpr;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.AggregateCall;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlAggFunction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts aggregate calls. */
+class AggregateScanConverter extends RelConverter<ResolvedAggregateScan> {
+ private static final String AVG_ILLEGAL_LONG_INPUT_TYPE =
+ "AVG(LONG) is not supported. You might want to use AVG(CAST(expression AS DOUBLE).";
+
+ AggregateScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedAggregateScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedAggregateScan zetaNode, List<RelNode> inputs) {
+ RelNode input = convertAggregateScanInputScanToLogicalProject(zetaNode, inputs.get(0));
+
+ // Calcite LogicalAggregate's GroupSet is indexes of group fields starting from 0.
+ int groupFieldsListSize = zetaNode.getGroupByList().size();
+ ImmutableBitSet groupSet;
+ if (groupFieldsListSize != 0) {
+ groupSet =
+ ImmutableBitSet.of(
+ IntStream.rangeClosed(0, groupFieldsListSize - 1)
+ .boxed()
+ .collect(Collectors.toList()));
+ } else {
+ groupSet = ImmutableBitSet.of();
+ }
+
+ // TODO: add support for indicator
+
+ List<AggregateCall> aggregateCalls;
+ if (zetaNode.getAggregateList().isEmpty()) {
+ aggregateCalls = ImmutableList.of();
+ } else {
+ aggregateCalls = new ArrayList<>();
+ // For aggregate calls, their input ref follow after GROUP BY input ref.
+ int columnRefoff = groupFieldsListSize;
+ for (ResolvedComputedColumn computedColumn : zetaNode.getAggregateList()) {
+ aggregateCalls.add(convertAggCall(computedColumn, columnRefoff));
+ columnRefoff++;
+ }
+ }
+
+ LogicalAggregate logicalAggregate =
+ new LogicalAggregate(
+ getCluster(),
+ input.getTraitSet(),
+ input,
+ false,
+ groupSet,
+ ImmutableList.of(groupSet),
+ aggregateCalls);
+
+ return logicalAggregate;
+ }
+
+ private RelNode convertAggregateScanInputScanToLogicalProject(
+ ResolvedAggregateScan node, RelNode input) {
+ // AggregateScan's input is the source of data (e.g. TableScan), which is different from the
+ // design of CalciteSQL, in which the LogicalAggregate's input is a LogicalProject, whose input
+ // is a LogicalTableScan. When AggregateScan's input is WithRefScan, the WithRefScan is
+ // ebullient to a LogicalTableScan. So it's still required to build another LogicalProject as
+ // the input of LogicalAggregate.
+ List<RexNode> projects = new ArrayList<>();
+ List<String> fieldNames = new ArrayList<>();
+
+ // LogicalProject has a list of expr, which including UDF in GROUP BY clause for
+ // LogicalAggregate.
+ for (ResolvedComputedColumn computedColumn : node.getGroupByList()) {
+ projects.add(
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ computedColumn.getExpr(),
+ node.getInputScan().getColumnList(),
+ input.getRowType().getFieldList()));
+ fieldNames.add(getTrait().resolveAlias(computedColumn.getColumn()));
+ }
+
+ // LogicalProject should also include columns used by aggregate functions. These columns should
+ // follow after GROUP BY columns.
+ // TODO: remove duplicate columns in projects.
+ for (ResolvedComputedColumn resolvedComputedColumn : node.getAggregateList()) {
+ // Should create Calcite's RexInputRef from ResolvedColumn from ResolvedComputedColumn.
+ // TODO: handle aggregate function with more than one argument and handle OVER
+ // TODO: is there is general way for column reference tracking and deduplication for
+ // aggregation?
+ ResolvedAggregateFunctionCall aggregateFunctionCall =
+ ((ResolvedAggregateFunctionCall) resolvedComputedColumn.getExpr());
+ if (aggregateFunctionCall.getArgumentList() != null
+ && aggregateFunctionCall.getArgumentList().size() == 1) {
+ ResolvedExpr resolvedExpr = aggregateFunctionCall.getArgumentList().get(0);
+
+ // TODO: assume aggregate function's input is either a ColumnRef or a cast(ColumnRef).
+ // TODO: user might use multiple CAST so we need to handle this rare case.
+ projects.add(
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ resolvedExpr,
+ node.getInputScan().getColumnList(),
+ input.getRowType().getFieldList()));
+ fieldNames.add(getTrait().resolveAlias(resolvedComputedColumn.getColumn()));
+ } else if (aggregateFunctionCall.getArgumentList() != null
+ && aggregateFunctionCall.getArgumentList().size() > 1) {
+ throw new RuntimeException(
+ aggregateFunctionCall.getFunction().getName() + " has more than one argument.");
+ }
+ }
+
+ return LogicalProject.create(input, projects, fieldNames);
+ }
+
+ private AggregateCall convertAggCall(ResolvedComputedColumn computedColumn, int columnRefOff) {
+ ResolvedAggregateFunctionCall aggregateFunctionCall =
+ (ResolvedAggregateFunctionCall) computedColumn.getExpr();
+
+ // Reject AVG(INT64)
+ if (aggregateFunctionCall.getFunction().getName().equals("avg")) {
+ FunctionSignature signature = aggregateFunctionCall.getSignature();
+ if (signature
+ .getFunctionArgumentList()
+ .get(0)
+ .getType()
+ .getKind()
+ .equals(TypeKind.TYPE_INT64)) {
+ throw new RuntimeException(AVG_ILLEGAL_LONG_INPUT_TYPE);
+ }
+ }
+
+ // Reject aggregation DISTINCT
+ if (aggregateFunctionCall.getDistinct()) {
+ throw new RuntimeException(
+ "Does not support "
+ + aggregateFunctionCall.getFunction().getSqlName()
+ + " DISTINCT. 'SELECT DISTINCT' syntax could be used to deduplicate before"
+ + " aggregation.");
+ }
+
+ SqlAggFunction sqlAggFunction =
+ (SqlAggFunction)
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ aggregateFunctionCall.getFunction().getName());
+ if (sqlAggFunction == null) {
+ throw new RuntimeException(
+ "Does not support ZetaSQL aggregate function: "
+ + aggregateFunctionCall.getFunction().getName());
+ }
+
+ List<Integer> argList = new ArrayList<>();
+ for (ResolvedExpr expr :
+ ((ResolvedAggregateFunctionCall) computedColumn.getExpr()).getArgumentList()) {
+ // Throw an error if aggregate function's input isn't either a ColumnRef or a cast(ColumnRef).
+ // TODO: is there a general way to handle aggregation calls conversion?
+ if (expr.nodeKind() == RESOLVED_CAST
+ || expr.nodeKind() == RESOLVED_COLUMN_REF
+ || expr.nodeKind() == RESOLVED_GET_STRUCT_FIELD) {
+ argList.add(columnRefOff);
+ } else {
+ throw new RuntimeException(
+ "Aggregate function only accepts Column Reference or CAST(Column Reference) as its"
+ + " input.");
+ }
+ }
+
+ // TODO: there should be a general way to decide if a return type of a aggcall is nullable.
+ RelDataType returnType;
+ if (sqlAggFunction.equals(SqlStdOperatorTable.ANY_VALUE)) {
+ returnType =
+ toSimpleRelDataType(
+ computedColumn.getColumn().getType().getKind(), getCluster().getRexBuilder(), true);
+ } else {
+ returnType =
+ toSimpleRelDataType(
+ computedColumn.getColumn().getType().getKind(), getCluster().getRexBuilder(), false);
+ }
+
+ String aggName = getTrait().resolveAlias(computedColumn.getColumn());
+ return AggregateCall.create(sqlAggFunction, false, false, argList, -1, returnType, aggName);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
new file mode 100644
index 0000000..f445006
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedArrayScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedColumnRef;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Converts array scan that represents join of an uncollect(array_field) to uncollect. */
+class ArrayScanToJoinConverter extends RelConverter<ResolvedArrayScan> {
+
+ ArrayScanToJoinConverter(ConversionContext context) {
+ super(context);
+ }
+
+ /** This is the case of {@code table [LEFT|INNER] JOIN UNNEST(table.array_field) on join_expr}. */
+ @Override
+ public boolean canConvert(ResolvedArrayScan zetaNode) {
+ return zetaNode.getInputScan() != null && zetaNode.getJoinExpr() != null;
+ }
+
+ /** Left input is converted from input scan. */
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedArrayScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ /** Returns a LogicJoin. */
+ @Override
+ public RelNode convert(ResolvedArrayScan zetaNode, List<RelNode> inputs) {
+ List<RexNode> projects = new ArrayList<>();
+
+ RelNode leftInput = inputs.get(0);
+
+ ResolvedColumnRef columnRef = (ResolvedColumnRef) zetaNode.getArrayExpr();
+ CorrelationId correlationId = getCluster().createCorrel();
+ getCluster().getQuery().mapCorrel(correlationId.getName(), leftInput);
+ String columnName =
+ String.format(
+ "%s%s",
+ zetaNode.getElementColumn().getTableName(), zetaNode.getElementColumn().getName());
+
+ projects.add(
+ getCluster()
+ .getRexBuilder()
+ .makeFieldAccess(
+ getCluster().getRexBuilder().makeCorrel(leftInput.getRowType(), correlationId),
+ getExpressionConverter()
+ .indexOfProjectionColumnRef(
+ columnRef.getColumn().getId(), zetaNode.getInputScan().getColumnList())));
+
+ RelNode projectNode =
+ LogicalProject.create(
+ LogicalValues.createOneRow(getCluster()), projects, ImmutableList.of(columnName));
+
+ // Create an UnCollect
+ // TODO: how to handle ordinality.
+ Uncollect uncollectNode = Uncollect.create(projectNode.getTraitSet(), projectNode, false);
+ // The InputRef should only be 0 because Uncollect has only one field.
+ RelNode rightInput =
+ LogicalProject.create(
+ uncollectNode,
+ ImmutableList.of(getCluster().getRexBuilder().makeInputRef(uncollectNode, 0)),
+ ImmutableList.of(columnName));
+
+ // Join condition should be a RexNode converted from join_expr.
+ RexNode condition =
+ getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getJoinExpr());
+ JoinRelType joinRelType = zetaNode.getIsOuter() ? JoinRelType.LEFT : JoinRelType.INNER;
+
+ return LogicalJoin.create(leftInput, rightInput, condition, ImmutableSet.of(), joinRelType);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
new file mode 100644
index 0000000..ef336cd
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedArrayScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLiteral;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts array scan that represents an array literal to uncollect. */
+class ArrayScanToUncollectConverter extends RelConverter<ResolvedArrayScan> {
+
+ ArrayScanToUncollectConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedArrayScan zetaNode) {
+ return zetaNode.getInputScan() == null;
+ }
+
+ @Override
+ public RelNode convert(ResolvedArrayScan zetaNode, List<RelNode> inputs) {
+ RexNode arrayLiteralExpression =
+ getExpressionConverter().convertResolvedLiteral((ResolvedLiteral) zetaNode.getArrayExpr());
+
+ String fieldName =
+ String.format(
+ "%s%s",
+ zetaNode.getElementColumn().getTableName(), zetaNode.getElementColumn().getName());
+
+ RelNode projectNode =
+ LogicalProject.create(
+ LogicalValues.createOneRow(getCluster()),
+ Collections.singletonList(arrayLiteralExpression),
+ ImmutableList.of(fieldName));
+
+ // TODO: how to handle ordinarily.
+ return Uncollect.create(projectNode.getTraitSet(), projectNode, false);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
new file mode 100644
index 0000000..1133574
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import org.apache.beam.sdk.extensions.sql.zetasql.QueryTrait;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+
+/** Conversion context, some rules need this data to convert the nodes. */
+public class ConversionContext {
+ private final FrameworkConfig config;
+ private final ExpressionConverter expressionConverter;
+ private final RelOptCluster cluster;
+ private final QueryTrait trait;
+
+ public static ConversionContext of(
+ FrameworkConfig config,
+ ExpressionConverter expressionConverter,
+ RelOptCluster cluster,
+ QueryTrait trait) {
+ return new ConversionContext(config, expressionConverter, cluster, trait);
+ }
+
+ private ConversionContext(
+ FrameworkConfig config,
+ ExpressionConverter expressionConverter,
+ RelOptCluster cluster,
+ QueryTrait trait) {
+ this.config = config;
+ this.expressionConverter = expressionConverter;
+ this.cluster = cluster;
+ this.trait = trait;
+ }
+
+ FrameworkConfig getConfig() {
+ return config;
+ }
+
+ ExpressionConverter getExpressionConverter() {
+ return expressionConverter;
+ }
+
+ RelOptCluster cluster() {
+ return cluster;
+ }
+
+ QueryTrait getTrait() {
+ return trait;
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
new file mode 100644
index 0000000..652aabd
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
@@ -0,0 +1,1019 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_FUNCTION_CALL;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BOOL;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_BYTES;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DOUBLE;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_INT64;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_STRING;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_TIMESTAMP;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.convertDateValueToDateString;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.convertTimeValueToTimeString;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.safeMicrosToMillis;
+import static org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable.FUNCTION_FAMILY_DATE_ADD;
+import static org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLCastFunctionImpl.ZETASQL_CAST_OP;
+
+import com.google.common.base.Ascii;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.zetasql.ArrayType;
+import com.google.zetasql.Type;
+import com.google.zetasql.Value;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.functions.ZetaSQLDateTime.DateTimestampPart;
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedAggregateScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedCast;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedColumnRef;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedComputedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedExpr;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedFunctionCall;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedGetStructField;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLiteral;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedParameter;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan;
+import io.grpc.Status;
+import java.math.BigDecimal;
+import java.text.MessageFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.annotations.Internal;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlOperatorRewriter;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlOperators;
+import org.apache.beam.sdk.extensions.sql.zetasql.SqlStdOperatorMappingTable;
+import org.apache.beam.sdk.extensions.sql.zetasql.TypeUtils;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.ByteString;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.TimeUnit;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.util.TimeUnitRange;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexBuilder;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlIntervalQualifier;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlOperator;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.TimestampString;
+
+/**
+ * Extracts expressions (function calls, field accesses) from the resolve query nodes, converts them
+ * to RexNodes.
+ */
+@Internal
+public class ExpressionConverter {
+
+ private static final String PRE_DEFINED_WINDOW_FUNCTIONS = "pre_defined_window_functions";
+
+ // Constants of pre-defined functions.
+ private static final String WINDOW_START = "_START";
+ private static final String WINDOW_END = "_END";
+ private static final String FIXED_WINDOW = "TUMBLE";
+ private static final String FIXED_WINDOW_START = FIXED_WINDOW + WINDOW_START;
+ private static final String FIXED_WINDOW_END = FIXED_WINDOW + WINDOW_END;
+ private static final String SLIDING_WINDOW = "HOP";
+ private static final String SLIDING_WINDOW_START = SLIDING_WINDOW + WINDOW_START;
+ private static final String SLIDING_WINDOW_END = SLIDING_WINDOW + WINDOW_END;
+ private static final String SESSION_WINDOW = "SESSION";
+ private static final String SESSION_WINDOW_START = SESSION_WINDOW + WINDOW_START;
+ private static final String SESSION_WINDOW_END = SESSION_WINDOW + WINDOW_END;
+
+ private static final ImmutableMap<String, String> WINDOW_START_END_TO_WINDOW_MAP =
+ ImmutableMap.<String, String>builder()
+ .put(FIXED_WINDOW_START, FIXED_WINDOW)
+ .put(FIXED_WINDOW_END, FIXED_WINDOW)
+ .put(SLIDING_WINDOW_START, SLIDING_WINDOW)
+ .put(SLIDING_WINDOW_END, SLIDING_WINDOW)
+ .put(SESSION_WINDOW_START, SESSION_WINDOW)
+ .put(SESSION_WINDOW_END, SESSION_WINDOW)
+ .build();
+
+ private static final ImmutableSet<String> WINDOW_START_END_FUNCTION_SET =
+ ImmutableSet.of(
+ FIXED_WINDOW_START,
+ FIXED_WINDOW_END,
+ SLIDING_WINDOW_START,
+ SLIDING_WINDOW_END,
+ SESSION_WINDOW_START,
+ SESSION_WINDOW_END);
+
+ private static final ImmutableMap<TypeKind, ImmutableSet<TypeKind>> UNSUPPORTED_CASTING =
+ ImmutableMap.<TypeKind, ImmutableSet<TypeKind>>builder()
+ .put(TYPE_INT64, ImmutableSet.of(TYPE_DOUBLE))
+ .put(TYPE_BOOL, ImmutableSet.of(TYPE_STRING))
+ .put(TYPE_STRING, ImmutableSet.of(TYPE_BOOL, TYPE_DOUBLE))
+ .build();
+
+ private static final ImmutableMap<Integer, TimeUnit> TIME_UNIT_CASTING_MAP =
+ ImmutableMap.<Integer, TimeUnit>builder()
+ .put(DateTimestampPart.YEAR.getNumber(), TimeUnit.YEAR)
+ .put(DateTimestampPart.MONTH.getNumber(), TimeUnit.MONTH)
+ .put(DateTimestampPart.DAY.getNumber(), TimeUnit.DAY)
+ .put(DateTimestampPart.DAYOFWEEK.getNumber(), TimeUnit.DOW)
+ .put(DateTimestampPart.DAYOFYEAR.getNumber(), TimeUnit.DOY)
+ .put(DateTimestampPart.QUARTER.getNumber(), TimeUnit.QUARTER)
+ .put(DateTimestampPart.HOUR.getNumber(), TimeUnit.HOUR)
+ .put(DateTimestampPart.MINUTE.getNumber(), TimeUnit.MINUTE)
+ .put(DateTimestampPart.SECOND.getNumber(), TimeUnit.SECOND)
+ .put(DateTimestampPart.MILLISECOND.getNumber(), TimeUnit.MILLISECOND)
+ .put(DateTimestampPart.MICROSECOND.getNumber(), TimeUnit.MICROSECOND)
+ .put(DateTimestampPart.NANOSECOND.getNumber(), TimeUnit.NANOSECOND)
+ .put(DateTimestampPart.ISOYEAR.getNumber(), TimeUnit.ISOYEAR)
+ .put(DateTimestampPart.ISOWEEK.getNumber(), TimeUnit.WEEK)
+ .build();
+
+ private static final ImmutableSet<String> DATE_PART_UNITS_TO_MILLIS =
+ ImmutableSet.of("DAY", "HOUR", "MINUTE", "SECOND");
+ private static final ImmutableSet<String> DATE_PART_UNITS_TO_MONTHS = ImmutableSet.of("YEAR");
+
+ private static final long ONE_SECOND_IN_MILLIS = 1000L;
+ private static final long ONE_MINUTE_IN_MILLIS = 60L * ONE_SECOND_IN_MILLIS;
+ private static final long ONE_HOUR_IN_MILLIS = 60L * ONE_MINUTE_IN_MILLIS;
+ private static final long ONE_DAY_IN_MILLIS = 24L * ONE_HOUR_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_MONTH_IN_MILLIS = 30L * ONE_DAY_IN_MILLIS;
+
+ @SuppressWarnings("unused")
+ private static final long ONE_YEAR_IN_MILLIS = 365L * ONE_DAY_IN_MILLIS;
+
+ // Constants of error messages.
+ private static final String INTERVAL_DATE_PART_MSG =
+ "YEAR, QUARTER, MONTH, WEEK, DAY, HOUR, MINUTE, SECOND, MILLISECOND";
+ private static final String INTERVAL_FORMAT_MSG =
+ "INTERVAL should be set as a STRING in the specific format: \"INTERVAL int64 date_part\"."
+ + " The date_part includes: "
+ + INTERVAL_DATE_PART_MSG;
+
+ private final RelOptCluster cluster;
+ private final Map<String, Value> queryParams;
+
+ public ExpressionConverter(RelOptCluster cluster, Map<String, Value> params) {
+ this.cluster = cluster;
+ this.queryParams = params;
+ }
+
+ /** Extract expressions from a project scan node. */
+ public List<RexNode> retrieveRexNode(ResolvedProjectScan node, List<RelDataTypeField> fieldList) {
+ List<RexNode> ret = new ArrayList<>();
+
+ for (ResolvedColumn column : node.getColumnList()) {
+ int index = -1;
+ if ((index = indexOfResolvedColumnInExprList(node.getExprList(), column)) != -1) {
+ ResolvedComputedColumn computedColumn = node.getExprList().get(index);
+ int windowFieldIndex = -1;
+ if (computedColumn.getExpr().nodeKind() == RESOLVED_FUNCTION_CALL) {
+ String functionName =
+ ((ResolvedFunctionCall) computedColumn.getExpr()).getFunction().getName();
+ if (WINDOW_START_END_FUNCTION_SET.contains(functionName)) {
+ ResolvedAggregateScan resolvedAggregateScan =
+ (ResolvedAggregateScan) node.getInputScan();
+ windowFieldIndex =
+ indexOfWindowField(
+ resolvedAggregateScan.getGroupByList(),
+ resolvedAggregateScan.getColumnList(),
+ WINDOW_START_END_TO_WINDOW_MAP.get(functionName));
+ }
+ }
+ ret.add(
+ convertRexNodeFromComputedColumnWithFieldList(
+ computedColumn, node.getInputScan().getColumnList(), fieldList, windowFieldIndex));
+ } else {
+ // ResolvedColumn is not a expression, which means it has to be an input column reference.
+ index = indexOfProjectionColumnRef(column.getId(), node.getInputScan().getColumnList());
+ if (index < 0 || index >= node.getInputScan().getColumnList().size()) {
+ throw new RuntimeException(
+ String.format("Cannot find %s in fieldList %s", column, fieldList));
+ }
+
+ ret.add(rexBuilder().makeInputRef(fieldList.get(index).getType(), index));
+ }
+ }
+ return ret;
+ }
+
+ /** Extract expressions from order by scan node. */
+ public List<RexNode> retrieveRexNodeFromOrderByScan(
+ RelOptCluster cluster, ResolvedOrderByScan node, List<RelDataTypeField> fieldList) {
+ final RexBuilder rexBuilder = cluster.getRexBuilder();
+ List<RexNode> ret = new ArrayList<>();
+
+ for (ResolvedColumn column : node.getColumnList()) {
+ int index = indexOfProjectionColumnRef(column.getId(), node.getInputScan().getColumnList());
+ ret.add(rexBuilder.makeInputRef(fieldList.get(index).getType(), index));
+ }
+
+ return ret;
+ }
+
+ private static int indexOfResolvedColumnInExprList(
+ ImmutableList<ResolvedComputedColumn> exprList, ResolvedColumn column) {
+ if (exprList == null || exprList.isEmpty()) {
+ return -1;
+ }
+
+ for (int i = 0; i < exprList.size(); i++) {
+ ResolvedComputedColumn computedColumn = exprList.get(i);
+ if (computedColumn.getColumn().equals(column)) {
+ return i;
+ }
+ }
+
+ return -1;
+ }
+
+ private static int indexOfWindowField(
+ List<ResolvedComputedColumn> groupByList, List<ResolvedColumn> columnList, String windowFn) {
+ for (ResolvedComputedColumn groupByComputedColumn : groupByList) {
+ if (groupByComputedColumn.getExpr().nodeKind() == RESOLVED_FUNCTION_CALL) {
+ ResolvedFunctionCall functionCall = (ResolvedFunctionCall) groupByComputedColumn.getExpr();
+ if (functionCall.getFunction().getName().equals(windowFn)) {
+ int ret =
+ indexOfResolvedColumnInColumnList(columnList, groupByComputedColumn.getColumn());
+ if (ret == -1) {
+ throw new RuntimeException("Cannot find " + windowFn + " in " + groupByList);
+ } else {
+ return ret;
+ }
+ }
+ }
+ }
+
+ throw new RuntimeException("Cannot find " + windowFn + " in " + groupByList);
+ }
+
+ private static int indexOfResolvedColumnInColumnList(
+ List<ResolvedColumn> columnList, ResolvedColumn column) {
+ if (columnList == null || columnList.isEmpty()) {
+ return -1;
+ }
+
+ for (int i = 0; i < columnList.size(); i++) {
+ if (columnList.get(i).equals(column)) {
+ return i;
+ }
+ }
+
+ return -1;
+ }
+
+ /** Create a RexNode for a corresponding resolved expression node. */
+ public RexNode convertRexNodeFromResolvedExpr(
+ ResolvedExpr expr, List<ResolvedColumn> columnList, List<RelDataTypeField> fieldList) {
+ if (columnList == null || fieldList == null) {
+ return convertRexNodeFromResolvedExpr(expr);
+ }
+
+ RexNode ret;
+
+ switch (expr.nodeKind()) {
+ case RESOLVED_LITERAL:
+ ret = convertResolvedLiteral((ResolvedLiteral) expr);
+ break;
+ case RESOLVED_COLUMN_REF:
+ ret = convertResolvedColumnRef((ResolvedColumnRef) expr, columnList, fieldList);
+ break;
+ case RESOLVED_FUNCTION_CALL:
+ ret = convertResolvedFunctionCall((ResolvedFunctionCall) expr, columnList, fieldList);
+ break;
+ case RESOLVED_CAST:
+ ret = convertResolvedCast((ResolvedCast) expr, columnList, fieldList);
+ break;
+ case RESOLVED_PARAMETER:
+ ret = convertResolvedParameter((ResolvedParameter) expr);
+ break;
+ case RESOLVED_GET_STRUCT_FIELD:
+ ret =
+ convertResolvedStructFieldAccess((ResolvedGetStructField) expr, columnList, fieldList);
+ break;
+ default:
+ ret = convertRexNodeFromResolvedExpr(expr);
+ }
+
+ return ret;
+ }
+
+ /** Create a RexNode for a corresponding resolved expression. */
+ public RexNode convertRexNodeFromResolvedExpr(ResolvedExpr expr) {
+ RexNode ret;
+
+ switch (expr.nodeKind()) {
+ case RESOLVED_LITERAL:
+ ret = convertResolvedLiteral((ResolvedLiteral) expr);
+ break;
+ case RESOLVED_COLUMN_REF:
+ ret = convertResolvedColumnRef((ResolvedColumnRef) expr);
+ break;
+ case RESOLVED_FUNCTION_CALL:
+ // TODO: is there a better way to shared code for different cases of
+ // convertResolvedFunctionCall than passing into two nulls?
+ ret = convertResolvedFunctionCall((ResolvedFunctionCall) expr, null, null);
+ break;
+ case RESOLVED_CAST:
+ ret = convertResolvedCast((ResolvedCast) expr, null, null);
+ break;
+ case RESOLVED_PARAMETER:
+ ret = convertResolvedParameter((ResolvedParameter) expr);
+ break;
+ case RESOLVED_GET_STRUCT_FIELD:
+ ret = convertResolvedStructFieldAccess((ResolvedGetStructField) expr);
+ break;
+ case RESOLVED_SUBQUERY_EXPR:
+ throw new IllegalArgumentException("Does not support sub-queries");
+ default:
+ throw new RuntimeException("Does not support expr node kind " + expr.nodeKind());
+ }
+
+ return ret;
+ }
+
+ /** Extract the RexNode from expression with ref scan. */
+ public RexNode convertRexNodeFromResolvedExprWithRefScan(
+ ResolvedExpr expr,
+ List<ResolvedColumn> refScanLeftColumnList,
+ List<RelDataTypeField> leftFieldList,
+ List<ResolvedColumn> originalLeftColumnList,
+ List<ResolvedColumn> refScanRightColumnList,
+ List<RelDataTypeField> rightFieldList,
+ List<ResolvedColumn> originalRightColumnList) {
+ RexNode ret;
+
+ switch (expr.nodeKind()) {
+ case RESOLVED_LITERAL:
+ ret = convertResolvedLiteral((ResolvedLiteral) expr);
+ break;
+ case RESOLVED_COLUMN_REF:
+ ResolvedColumnRef columnRef = (ResolvedColumnRef) expr;
+ // first look for column ref on the left side
+ ret =
+ convertRexNodeFromResolvedColumnRefWithRefScan(
+ columnRef, refScanLeftColumnList, originalLeftColumnList, leftFieldList);
+
+ // if not found there look on the right
+ if (ret == null) {
+ ret =
+ convertRexNodeFromResolvedColumnRefWithRefScan(
+ columnRef, refScanRightColumnList, originalRightColumnList, rightFieldList);
+ }
+
+ break;
+ case RESOLVED_FUNCTION_CALL:
+ // JOIN only support equal join.
+ ResolvedFunctionCall resolvedFunctionCall = (ResolvedFunctionCall) expr;
+ List<RexNode> operands = new ArrayList<>();
+
+ for (ResolvedExpr resolvedExpr : resolvedFunctionCall.getArgumentList()) {
+ operands.add(
+ convertRexNodeFromResolvedExprWithRefScan(
+ resolvedExpr,
+ refScanLeftColumnList,
+ leftFieldList,
+ originalLeftColumnList,
+ refScanRightColumnList,
+ rightFieldList,
+ originalRightColumnList));
+ }
+
+ SqlOperator op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ resolvedFunctionCall.getFunction().getName());
+ ret = rexBuilder().makeCall(op, operands);
+ break;
+ case RESOLVED_CAST:
+ ResolvedCast resolvedCast = (ResolvedCast) expr;
+ RexNode operand =
+ convertRexNodeFromResolvedExprWithRefScan(
+ resolvedCast.getExpr(),
+ refScanLeftColumnList,
+ leftFieldList,
+ originalLeftColumnList,
+ refScanRightColumnList,
+ rightFieldList,
+ originalRightColumnList);
+
+ TypeKind fromType = resolvedCast.getExpr().getType().getKind();
+ TypeKind toType = resolvedCast.getType().getKind();
+ isCastingSupported(fromType, toType);
+
+ RelDataType outputType =
+ TypeUtils.toSimpleRelDataType(toType, rexBuilder(), operand.getType().isNullable());
+
+ if (isZetaSQLCast(fromType, toType)) {
+ ret = rexBuilder().makeCall(outputType, ZETASQL_CAST_OP, ImmutableList.of(operand));
+ } else {
+ ret = rexBuilder().makeCast(outputType, operand);
+ }
+ break;
+ default:
+ throw new RuntimeException("Does not support expr node kind " + expr.nodeKind());
+ }
+
+ return ret;
+ }
+
+ private RexNode convertRexNodeFromComputedColumnWithFieldList(
+ ResolvedComputedColumn column,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList,
+ int windowFieldIndex) {
+ if (column.getExpr().nodeKind() != RESOLVED_FUNCTION_CALL) {
+ return convertRexNodeFromResolvedExpr(column.getExpr(), columnList, fieldList);
+ }
+
+ ResolvedFunctionCall functionCall = (ResolvedFunctionCall) column.getExpr();
+
+ // TODO: is there any other illegal case?
+ if (functionCall.getFunction().getName().equals(FIXED_WINDOW)
+ || functionCall.getFunction().getName().equals(SLIDING_WINDOW)
+ || functionCall.getFunction().getName().equals(SESSION_WINDOW)) {
+ throw new RuntimeException(
+ functionCall.getFunction().getName() + " shouldn't appear in SELECT exprlist.");
+ }
+
+ if (!functionCall.getFunction().getGroup().equals(PRE_DEFINED_WINDOW_FUNCTIONS)) {
+ // non-window function should still go through normal FunctionCall conversion process.
+ return convertRexNodeFromResolvedExpr(column.getExpr(), columnList, fieldList);
+ }
+
+ // ONLY window_start and window_end should arrive here.
+ // TODO: Have extra verification here to make sure window start/end functions have the same
+ // parameter with window function.
+ List<RexNode> operands = new ArrayList<>();
+ switch (functionCall.getFunction().getName()) {
+ case FIXED_WINDOW_START:
+ case SLIDING_WINDOW_START:
+ case SESSION_WINDOW_START:
+ // TODO: in Calcite implementation, session window's start is equal to end. Need to fix it
+ // in Calcite.
+ case SESSION_WINDOW_END:
+ return rexBuilder()
+ .makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex);
+ case FIXED_WINDOW_END:
+ // WINDOW END is a function call
+ operands.add(
+ rexBuilder().makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex));
+ // TODO: check window_end 's duration is the same as it's aggregate window.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(0)));
+ return rexBuilder().makeCall(SqlStdOperatorTable.PLUS, operands);
+ case SLIDING_WINDOW_END:
+ operands.add(
+ rexBuilder().makeInputRef(fieldList.get(windowFieldIndex).getType(), windowFieldIndex));
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(1)));
+ return rexBuilder().makeCall(SqlStdOperatorTable.PLUS, operands);
+ default:
+ throw new RuntimeException(
+ "Does not support window start/end: " + functionCall.getFunction().getName());
+ }
+ }
+
+ /** Convert a resolved literal to a RexNode. */
+ public RexNode convertResolvedLiteral(ResolvedLiteral resolvedLiteral) {
+ TypeKind kind = resolvedLiteral.getType().getKind();
+ RexNode ret;
+ switch (kind) {
+ case TYPE_BOOL:
+ case TYPE_INT32:
+ case TYPE_INT64:
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE:
+ case TYPE_STRING:
+ case TYPE_TIMESTAMP:
+ case TYPE_DATE:
+ case TYPE_TIME:
+ // case TYPE_DATETIME:
+ case TYPE_BYTES:
+ case TYPE_ARRAY:
+ case TYPE_STRUCT:
+ case TYPE_ENUM:
+ ret = convertValueToRexNode(resolvedLiteral.getType(), resolvedLiteral.getValue());
+ break;
+ default:
+ throw new RuntimeException(
+ MessageFormat.format(
+ "Unsupported ResolvedLiteral type: {0}, kind: {1}, value: {2}, class: {3}",
+ resolvedLiteral.getType().typeName(),
+ kind,
+ resolvedLiteral.getValue(),
+ resolvedLiteral.getClass()));
+ }
+
+ return ret;
+ }
+
+ private RexNode convertValueToRexNode(Type type, Value value) {
+ RexNode ret;
+ switch (type.getKind()) {
+ case TYPE_BOOL:
+ case TYPE_INT32:
+ case TYPE_INT64:
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE:
+ case TYPE_STRING:
+ case TYPE_TIMESTAMP:
+ case TYPE_DATE:
+ case TYPE_TIME:
+ // case TYPE_DATETIME:
+ case TYPE_BYTES:
+ ret = convertSimpleValueToRexNode(type.getKind(), value);
+ break;
+ case TYPE_ARRAY:
+ ret = convertArrayValueToRexNode(type.asArray(), value);
+ break;
+ case TYPE_ENUM:
+ ret = convertEnumToRexNode(type, value);
+ break;
+ default:
+ // TODO: convert struct literal.
+ throw new RuntimeException(
+ "Unsupported ResolvedLiteral kind: " + type.getKind() + " type: " + type.typeName());
+ }
+
+ return ret;
+ }
+
+ private RexNode convertSimpleValueToRexNode(TypeKind kind, Value value) {
+ if (value.isNull()) {
+ return rexBuilder().makeNullLiteral(TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ }
+
+ RexNode ret;
+ switch (kind) {
+ case TYPE_BOOL:
+ ret = rexBuilder().makeLiteral(value.getBoolValue());
+ break;
+ case TYPE_INT32:
+ ret =
+ rexBuilder()
+ .makeExactLiteral(
+ new BigDecimal(value.getInt32Value()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_INT64:
+ ret =
+ rexBuilder()
+ .makeExactLiteral(
+ new BigDecimal(value.getInt64Value()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_FLOAT:
+ ret =
+ rexBuilder()
+ .makeApproxLiteral(
+ new BigDecimal(value.getFloatValue()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_DOUBLE:
+ ret =
+ rexBuilder()
+ .makeApproxLiteral(
+ new BigDecimal(value.getDoubleValue()),
+ TypeUtils.toSimpleRelDataType(kind, rexBuilder()));
+ break;
+ case TYPE_STRING:
+ // has to allow CAST because Calcite create CHAR type first and does a CAST to VARCHAR.
+ // If not allow cast, rexBuilder() will only build a literal with CHAR type.
+ ret =
+ rexBuilder()
+ .makeLiteral(
+ value.getStringValue(), typeFactory().createSqlType(SqlTypeName.VARCHAR), true);
+ break;
+ case TYPE_TIMESTAMP:
+ ret =
+ rexBuilder()
+ .makeTimestampLiteral(
+ TimestampString.fromMillisSinceEpoch(
+ safeMicrosToMillis(value.getTimestampUnixMicros())),
+ typeFactory().getTypeSystem().getMaxPrecision(SqlTypeName.TIMESTAMP));
+ break;
+ case TYPE_DATE:
+ ret = rexBuilder().makeDateLiteral(convertDateValueToDateString(value));
+ break;
+ case TYPE_TIME:
+ RelDataType timeType =
+ typeFactory()
+ .createSqlType(
+ SqlTypeName.TIME,
+ typeFactory().getTypeSystem().getMaxPrecision(SqlTypeName.TIME));
+ // TODO: Doing micro to mills truncation, need to throw exception.
+ ret = rexBuilder().makeLiteral(convertTimeValueToTimeString(value), timeType, false);
+ break;
+ case TYPE_BYTES:
+ ret = rexBuilder().makeBinaryLiteral(new ByteString(value.getBytesValue().toByteArray()));
+ break;
+ default:
+ throw new RuntimeException("Unsupported column type: " + kind);
+ }
+
+ return ret;
+ }
+
+ private RexNode convertArrayValueToRexNode(ArrayType arrayType, Value value) {
+ if (value.isNull()) {
+ // TODO: should the nullable be false for a array?
+ return rexBuilder()
+ .makeNullLiteral(TypeUtils.toArrayRelDataType(rexBuilder(), arrayType, false));
+ }
+
+ List<RexNode> operands = new ArrayList<>();
+ for (Value v : value.getElementList()) {
+ operands.add(convertValueToRexNode(arrayType.getElementType(), v));
+ }
+ return rexBuilder().makeCall(SqlStdOperatorTable.ARRAY_VALUE_CONSTRUCTOR, operands);
+ }
+
+ private RexNode convertEnumToRexNode(Type type, Value value) {
+ if (type.typeName().equals("`zetasql.functions.DateTimestampPart`")) {
+ return convertTimeUnitRangeEnumToRexNode(type, value);
+ } else {
+ throw new RuntimeException(
+ MessageFormat.format(
+ "Unsupported enum. Kind: {0} Type: {1}", type.getKind(), type.typeName()));
+ }
+ }
+
+ private RexNode convertTimeUnitRangeEnumToRexNode(Type type, Value value) {
+ TimeUnit mappedUnit = TIME_UNIT_CASTING_MAP.get(value.getEnumValue());
+ if (mappedUnit == null) {
+ throw new RuntimeException(
+ MessageFormat.format(
+ "Unsupported enum value. Kind: {0} Type: {1} Value: {2} EnumName: {3}",
+ type.getKind(), type.typeName(), value.getEnumName(), value.getEnumValue()));
+ }
+
+ TimeUnitRange mappedRange = TimeUnitRange.of(mappedUnit, null);
+ return rexBuilder().makeFlag(mappedRange);
+ }
+
+ private RexNode convertResolvedColumnRef(
+ ResolvedColumnRef columnRef,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ int index = indexOfProjectionColumnRef(columnRef.getColumn().getId(), columnList);
+ if (index < 0 || index >= columnList.size()) {
+ throw new RuntimeException(
+ String.format("Cannot find %s in fieldList %s", columnRef.getColumn(), fieldList));
+ }
+ return rexBuilder().makeInputRef(fieldList.get(index).getType(), index);
+ }
+
+ private RexNode convertResolvedColumnRef(ResolvedColumnRef columnRef) {
+ // TODO: id - 1 might be only correct if the columns read from TableScan.
+ // What if the columns come from other scans (which means their id are not indexed from 0),
+ // and what if there are some mis-order?
+ // TODO: can join key be NULL?
+ return rexBuilder()
+ .makeInputRef(
+ TypeUtils.toRelDataType(rexBuilder(), columnRef.getType(), false),
+ (int) columnRef.getColumn().getId() - 1);
+ }
+
+ /** Return an index of the projection column reference. */
+ public int indexOfProjectionColumnRef(long colId, List<ResolvedColumn> columnList) {
+ int ret = -1;
+ for (int i = 0; i < columnList.size(); i++) {
+ if (columnList.get(i).getId() == colId) {
+ ret = i;
+ break;
+ }
+ }
+
+ return ret;
+ }
+
+ private RexNode convertResolvedFunctionCall(
+ ResolvedFunctionCall functionCall,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ RexNode ret;
+ SqlOperator op;
+ List<RexNode> operands = new ArrayList<>();
+
+ if (functionCall.getFunction().getGroup().equals(PRE_DEFINED_WINDOW_FUNCTIONS)) {
+ switch (functionCall.getFunction().getName()) {
+ case FIXED_WINDOW:
+ case SESSION_WINDOW:
+ op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ functionCall.getFunction().getName());
+ // TODO: check size and type of window function argument list.
+ // Add ts column reference to operands.
+ operands.add(
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(0), columnList, fieldList));
+ // Add fixed window size or session window gap to operands.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(1)));
+ break;
+ case SLIDING_WINDOW:
+ op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ SLIDING_WINDOW);
+ // Add ts column reference to operands.
+ operands.add(
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(0), columnList, fieldList));
+ // add sliding window emit frequency to operands.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(1)));
+ // add sliding window size to operands.
+ operands.add(
+ convertIntervalToRexIntervalLiteral(
+ (ResolvedLiteral) functionCall.getArgumentList().get(2)));
+ break;
+ default:
+ throw new RuntimeException("Only support TUMBLE, HOP AND SESSION functions right now.");
+ }
+ } else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
+ op =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
+ functionCall.getFunction().getName());
+
+ if (op == null) {
+ throw new RuntimeException(
+ "Does not support ZetaSQL function: " + functionCall.getFunction().getName());
+ }
+
+ // There are different processes to handle argument conversion because INTERVAL is not a
+ // type in ZetaSQL.
+ if (FUNCTION_FAMILY_DATE_ADD.contains(functionCall.getFunction().getName())) {
+ return convertTimestampAddFunction(functionCall, columnList, fieldList);
+ } else {
+ for (ResolvedExpr expr : functionCall.getArgumentList()) {
+ operands.add(convertRexNodeFromResolvedExpr(expr, columnList, fieldList));
+ }
+ }
+ } else {
+ throw new RuntimeException(
+ "Does not support function group: " + functionCall.getFunction().getGroup());
+ }
+
+ SqlOperatorRewriter rewriter =
+ SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(
+ functionCall.getFunction().getName());
+
+ if (rewriter != null) {
+ ret = rewriter.apply(rexBuilder(), operands);
+ } else {
+ ret = rexBuilder().makeCall(op, operands);
+ }
+ return ret;
+ }
+
+ private RexNode convertTimestampAddFunction(
+ ResolvedFunctionCall functionCall,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+
+ TimeUnit unit =
+ TIME_UNIT_CASTING_MAP.get(
+ ((ResolvedLiteral) functionCall.getArgumentList().get(2)).getValue().getEnumValue());
+
+ if ((unit == TimeUnit.MICROSECOND) || (unit == TimeUnit.NANOSECOND)) {
+ throw Status.UNIMPLEMENTED
+ .withDescription("Micro and Nanoseconds are not supported by Beam ZetaSQL")
+ .asRuntimeException();
+ }
+
+ SqlIntervalQualifier qualifier = new SqlIntervalQualifier(unit, null, SqlParserPos.ZERO);
+
+ RexNode intervalArgumentNode =
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(1), columnList, fieldList);
+
+ RexNode validatedIntervalArgument =
+ rexBuilder()
+ .makeCall(
+ SqlOperators.VALIDATE_TIME_INTERVAL,
+ intervalArgumentNode,
+ rexBuilder().makeFlag(unit));
+
+ RexNode intervalNode =
+ rexBuilder()
+ .makeCall(
+ SqlStdOperatorTable.MULTIPLY,
+ rexBuilder().makeIntervalLiteral(unit.multiplier, qualifier),
+ validatedIntervalArgument);
+
+ RexNode timestampNode =
+ convertRexNodeFromResolvedExpr(
+ functionCall.getArgumentList().get(0), columnList, fieldList);
+
+ RexNode dateTimePlusResult =
+ rexBuilder().makeCall(SqlStdOperatorTable.DATETIME_PLUS, timestampNode, intervalNode);
+
+ RexNode validatedTimestampResult =
+ rexBuilder().makeCall(SqlOperators.VALIDATE_TIMESTAMP, dateTimePlusResult);
+
+ return validatedTimestampResult;
+ }
+
+ private RexNode convertIntervalToRexIntervalLiteral(ResolvedLiteral resolvedLiteral) {
+ if (resolvedLiteral.getType().getKind() != TYPE_STRING) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
+ }
+
+ String valStr = resolvedLiteral.getValue().getStringValue();
+ List<String> stringList =
+ Arrays.stream(valStr.split(" ")).filter(s -> !s.isEmpty()).collect(Collectors.toList());
+
+ if (stringList.size() != 3) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
+ }
+
+ if (!Ascii.toUpperCase(stringList.get(0)).equals("INTERVAL")) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG);
+ }
+
+ long intervalValue;
+ try {
+ intervalValue = Long.parseLong(stringList.get(1));
+ } catch (NumberFormatException e) {
+ throw new IllegalArgumentException(INTERVAL_FORMAT_MSG, e);
+ }
+
+ String intervalDatepart = Ascii.toUpperCase(stringList.get(2));
+ return createCalciteIntervalRexLiteral(intervalValue, intervalDatepart);
+ }
+
+ private RexLiteral createCalciteIntervalRexLiteral(long intervalValue, String intervalTimeUnit) {
+ SqlIntervalQualifier sqlIntervalQualifier =
+ convertIntervalDatepartToSqlIntervalQualifier(intervalTimeUnit);
+ BigDecimal decimalValue;
+ if (DATE_PART_UNITS_TO_MILLIS.contains(intervalTimeUnit)) {
+ decimalValue = convertIntervalValueToMillis(sqlIntervalQualifier, intervalValue);
+ } else if (DATE_PART_UNITS_TO_MONTHS.contains(intervalTimeUnit)) {
+ decimalValue = new BigDecimal(intervalValue * 12);
+ } else {
+ decimalValue = new BigDecimal(intervalValue);
+ }
+ return rexBuilder().makeIntervalLiteral(decimalValue, sqlIntervalQualifier);
+ }
+
+ private static BigDecimal convertIntervalValueToMillis(
+ SqlIntervalQualifier qualifier, long value) {
+ switch (qualifier.typeName()) {
+ case INTERVAL_DAY:
+ return new BigDecimal(value * ONE_DAY_IN_MILLIS);
+ case INTERVAL_HOUR:
+ return new BigDecimal(value * ONE_HOUR_IN_MILLIS);
+ case INTERVAL_MINUTE:
+ return new BigDecimal(value * ONE_MINUTE_IN_MILLIS);
+ case INTERVAL_SECOND:
+ return new BigDecimal(value * ONE_SECOND_IN_MILLIS);
+ default:
+ throw new IllegalArgumentException(qualifier.typeName().toString());
+ }
+ }
+
+ private static SqlIntervalQualifier convertIntervalDatepartToSqlIntervalQualifier(
+ String datePart) {
+ switch (datePart) {
+ case "YEAR":
+ return new SqlIntervalQualifier(TimeUnit.YEAR, null, SqlParserPos.ZERO);
+ case "MONTH":
+ return new SqlIntervalQualifier(TimeUnit.MONTH, null, SqlParserPos.ZERO);
+ case "DAY":
+ return new SqlIntervalQualifier(TimeUnit.DAY, null, SqlParserPos.ZERO);
+ case "HOUR":
+ return new SqlIntervalQualifier(TimeUnit.HOUR, null, SqlParserPos.ZERO);
+ case "MINUTE":
+ return new SqlIntervalQualifier(TimeUnit.MINUTE, null, SqlParserPos.ZERO);
+ case "SECOND":
+ return new SqlIntervalQualifier(TimeUnit.SECOND, null, SqlParserPos.ZERO);
+ case "WEEK":
+ return new SqlIntervalQualifier(TimeUnit.WEEK, null, SqlParserPos.ZERO);
+ case "QUARTER":
+ return new SqlIntervalQualifier(TimeUnit.QUARTER, null, SqlParserPos.ZERO);
+ case "MILLISECOND":
+ return new SqlIntervalQualifier(TimeUnit.MILLISECOND, null, SqlParserPos.ZERO);
+ default:
+ throw new RuntimeException(
+ String.format(
+ "Received an undefined INTERVAL unit: %s. Please specify unit from the following"
+ + " list: %s.",
+ datePart, INTERVAL_DATE_PART_MSG));
+ }
+ }
+
+ private RexNode convertResolvedCast(
+ ResolvedCast resolvedCast,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ TypeKind fromType = resolvedCast.getExpr().getType().getKind();
+ TypeKind toType = resolvedCast.getType().getKind();
+ isCastingSupported(fromType, toType);
+
+ RexNode inputNode =
+ convertRexNodeFromResolvedExpr(resolvedCast.getExpr(), columnList, fieldList);
+ // nullability of the output type should match that of the input node's type
+ RelDataType outputType =
+ TypeUtils.toSimpleRelDataType(
+ resolvedCast.getType().getKind(), rexBuilder(), inputNode.getType().isNullable());
+
+ if (isZetaSQLCast(fromType, toType)) {
+ return rexBuilder().makeCall(outputType, ZETASQL_CAST_OP, ImmutableList.of(inputNode));
+ } else {
+ return rexBuilder().makeCast(outputType, inputNode);
+ }
+ }
+
+ private static void isCastingSupported(TypeKind fromType, TypeKind toType) {
+ if (UNSUPPORTED_CASTING.containsKey(toType)
+ && UNSUPPORTED_CASTING.get(toType).contains(fromType)) {
+ throw new IllegalArgumentException(
+ "Does not support CAST(" + fromType + " AS " + toType + ")");
+ }
+ }
+
+ private static boolean isZetaSQLCast(TypeKind fromType, TypeKind toType) {
+ // TODO: Structure ZETASQL_CAST_OP so that we don't have to repeat the supported types
+ // here
+ return (fromType.equals(TYPE_BYTES) && toType.equals(TYPE_STRING))
+ || (fromType.equals(TYPE_INT64) && toType.equals(TYPE_BOOL))
+ || (fromType.equals(TYPE_BOOL) && toType.equals(TYPE_INT64))
+ || (fromType.equals(TYPE_TIMESTAMP) && toType.equals(TYPE_STRING));
+ }
+
+ private RexNode convertRexNodeFromResolvedColumnRefWithRefScan(
+ ResolvedColumnRef columnRef,
+ List<ResolvedColumn> refScanColumnList,
+ List<ResolvedColumn> originalColumnList,
+ List<RelDataTypeField> fieldList) {
+
+ for (int i = 0; i < refScanColumnList.size(); i++) {
+ if (refScanColumnList.get(i).getId() == columnRef.getColumn().getId()) {
+ boolean nullable = fieldList.get(i).getType().isNullable();
+ int off = (int) originalColumnList.get(i).getId() - 1;
+ return rexBuilder()
+ .makeInputRef(
+ TypeUtils.toSimpleRelDataType(
+ columnRef.getType().getKind(), rexBuilder(), nullable),
+ off);
+ }
+ }
+
+ return null;
+ }
+
+ private RexNode convertResolvedParameter(ResolvedParameter parameter) {
+ assert parameter.getType().equals(queryParams.get(parameter.getName()).getType());
+ return convertValueToRexNode(
+ queryParams.get(parameter.getName()).getType(), queryParams.get(parameter.getName()));
+ }
+
+ private RexNode convertResolvedStructFieldAccess(ResolvedGetStructField resolvedGetStructField) {
+ return rexBuilder()
+ .makeFieldAccess(
+ convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr()),
+ (int) resolvedGetStructField.getFieldIdx());
+ }
+
+ private RexNode convertResolvedStructFieldAccess(
+ ResolvedGetStructField resolvedGetStructField,
+ List<ResolvedColumn> columnList,
+ List<RelDataTypeField> fieldList) {
+ return rexBuilder()
+ .makeFieldAccess(
+ convertRexNodeFromResolvedExpr(resolvedGetStructField.getExpr(), columnList, fieldList),
+ (int) resolvedGetStructField.getFieldIdx());
+ }
+
+ private RexBuilder rexBuilder() {
+ return cluster.getRexBuilder();
+ }
+
+ private RelDataTypeFactory typeFactory() {
+ return cluster.getTypeFactory();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
new file mode 100644
index 0000000..1a04ca1
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedFilterScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Converts filter. */
+class FilterScanConverter extends RelConverter<ResolvedFilterScan> {
+
+ FilterScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedFilterScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedFilterScan zetaNode, List<RelNode> inputs) {
+ RelNode input = inputs.get(0);
+ RexNode condition =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getFilterExpr(),
+ zetaNode.getInputScan().getColumnList(),
+ input.getRowType().getFieldList());
+
+ return LogicalFilter.create(input, condition);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
new file mode 100644
index 0000000..d0819f3
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedJoinScanEnums.JoinType;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedJoinScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Converts joins if neither side of the join is a WithRefScan. */
+class JoinScanConverter extends RelConverter<ResolvedJoinScan> {
+
+ private static final ImmutableMap<JoinType, JoinRelType> JOIN_TYPES =
+ ImmutableMap.of(
+ JoinType.INNER,
+ JoinRelType.INNER,
+ JoinType.FULL,
+ JoinRelType.FULL,
+ JoinType.LEFT,
+ JoinRelType.LEFT,
+ JoinType.RIGHT,
+ JoinRelType.RIGHT);
+
+ JoinScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedJoinScan zetaNode) {
+ return !(zetaNode.getLeftScan() instanceof ResolvedWithRefScan)
+ && !(zetaNode.getRightScan() instanceof ResolvedWithRefScan);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedJoinScan zetaNode) {
+ return ImmutableList.of(zetaNode.getLeftScan(), zetaNode.getRightScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedJoinScan zetaNode, List<RelNode> inputs) {
+ RelNode convertedLeftInput = inputs.get(0);
+ RelNode convertedRightInput = inputs.get(1);
+
+ List<ResolvedColumn> combinedZetaFieldsList =
+ ImmutableList.<ResolvedColumn>builder()
+ .addAll(zetaNode.getLeftScan().getColumnList())
+ .addAll(zetaNode.getRightScan().getColumnList())
+ .build();
+
+ List<RelDataTypeField> combinedCalciteFieldsList =
+ ImmutableList.<RelDataTypeField>builder()
+ .addAll(convertedLeftInput.getRowType().getFieldList())
+ .addAll(convertedRightInput.getRowType().getFieldList())
+ .build();
+
+ RexNode condition =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getJoinExpr(), combinedZetaFieldsList, combinedCalciteFieldsList);
+
+ return LogicalJoin.create(
+ convertedLeftInput,
+ convertedRightInput,
+ condition,
+ ImmutableSet.of(),
+ convertResolvedJoinType(zetaNode.getJoinType()));
+ }
+
+ static JoinRelType convertResolvedJoinType(JoinType joinType) {
+ if (!JOIN_TYPES.containsKey(joinType)) {
+ throw new UnsupportedOperationException("JOIN type: " + joinType + " is unsupported.");
+ }
+
+ return JOIN_TYPES.get(joinType);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
new file mode 100644
index 0000000..fee0124
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static org.apache.beam.sdk.extensions.sql.zetasql.translation.JoinScanConverter.convertResolvedJoinType;
+
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedJoinScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+
+/** Converts joins where at least one of the inputs is a WITH subquery. */
+class JoinScanWithRefConverter extends RelConverter<ResolvedJoinScan> {
+
+ JoinScanWithRefConverter(ConversionContext context) {
+ super(context);
+ }
+
+ /** This is a special logic due to re-indexed column reference in WithScan. */
+ @Override
+ public boolean canConvert(ResolvedJoinScan zetaNode) {
+ return zetaNode.getLeftScan() instanceof ResolvedWithRefScan
+ || zetaNode.getRightScan() instanceof ResolvedWithRefScan;
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedJoinScan zetaNode) {
+ return ImmutableList.of(zetaNode.getLeftScan(), zetaNode.getRightScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedJoinScan zetaNode, List<RelNode> inputs) {
+ RelNode calciteLeftInput = inputs.get(0);
+ RelNode calciteRightInput = inputs.get(1);
+
+ List<ResolvedColumn> zetaLeftColumnList = getColumnsForScan(zetaNode.getLeftScan());
+ List<ResolvedColumn> zetaRightColumnList = getColumnsForScan(zetaNode.getRightScan());
+
+ RexNode condition =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExprWithRefScan(
+ zetaNode.getJoinExpr(),
+ zetaNode.getLeftScan().getColumnList(),
+ calciteLeftInput.getRowType().getFieldList(),
+ zetaLeftColumnList,
+ zetaNode.getRightScan().getColumnList(),
+ calciteRightInput.getRowType().getFieldList(),
+ zetaRightColumnList);
+
+ return LogicalJoin.create(
+ calciteLeftInput,
+ calciteRightInput,
+ condition,
+ ImmutableSet.of(),
+ convertResolvedJoinType(zetaNode.getJoinType()));
+ }
+
+ /**
+ * WithRefScan doesn't have columns in it, it only references a WITH query by name, we have to
+ * look up the actual query node in the context by that name.
+ *
+ * <p>The context has a map of WITH queries populated when the inputs to this JOIN are parsed.
+ */
+ private List<ResolvedColumn> getColumnsForScan(ResolvedScan resolvedScan) {
+ return resolvedScan instanceof ResolvedWithRefScan
+ ? getTrait()
+ .withEntries
+ .get(((ResolvedWithRefScan) resolvedScan).getWithQueryName())
+ .getWithSubquery()
+ .getColumnList()
+ : resolvedScan.getColumnList();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
new file mode 100644
index 0000000..0be8e2c
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLimitOffsetScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollations;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts LIMIT without ORDER BY. */
+class LimitOffsetScanToLimitConverter extends RelConverter<ResolvedLimitOffsetScan> {
+
+ LimitOffsetScanToLimitConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedLimitOffsetScan zetaNode) {
+ return !(zetaNode.getInputScan() instanceof ResolvedOrderByScan);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedLimitOffsetScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedLimitOffsetScan zetaNode, List<RelNode> inputs) {
+ RelNode input = inputs.get(0);
+ RelCollation relCollation = RelCollations.of(ImmutableList.of());
+ RexNode offset =
+ zetaNode.getOffset() == null
+ ? null
+ : getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getOffset());
+ RexNode fetch =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getLimit(), zetaNode.getColumnList(), input.getRowType().getFieldList());
+ return LogicalSort.create(input, relCollation, offset, fetch);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
new file mode 100644
index 0000000..2492088
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation.Direction.ASCENDING;
+import static org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation.Direction.DESCENDING;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedLimitOffsetScan;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByItem;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelCollationImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelFieldCollation.Direction;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Converts ORDER BY LIMIT. */
+class LimitOffsetScanToOrderByLimitConverter extends RelConverter<ResolvedLimitOffsetScan> {
+
+ LimitOffsetScanToOrderByLimitConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedLimitOffsetScan zetaNode) {
+ return zetaNode.getInputScan() instanceof ResolvedOrderByScan;
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedLimitOffsetScan zetaNode) {
+ // The immediate input is the ORDER BY scan which we don't support,
+ // but we can handle the ORDER BY LIMIT if we know the underlying projection, for example.
+ return Collections.singletonList(
+ ((ResolvedOrderByScan) zetaNode.getInputScan()).getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedLimitOffsetScan zetaNode, List<RelNode> inputs) {
+ ResolvedOrderByScan inputOrderByScan = (ResolvedOrderByScan) zetaNode.getInputScan();
+ RelNode input = convertOrderByScanToLogicalScan(inputOrderByScan, inputs.get(0));
+ RelCollation relCollation = getRelCollation(inputOrderByScan);
+
+ RexNode offset =
+ zetaNode.getOffset() == null
+ ? null
+ : getExpressionConverter().convertRexNodeFromResolvedExpr(zetaNode.getOffset());
+ RexNode fetch =
+ getExpressionConverter()
+ .convertRexNodeFromResolvedExpr(
+ zetaNode.getLimit(), zetaNode.getColumnList(), input.getRowType().getFieldList());
+
+ return LogicalSort.create(input, relCollation, offset, fetch);
+ }
+
+ /** Collation is a sort order, as in ORDER BY DESCENDING/ASCENDING. */
+ private static RelCollation getRelCollation(ResolvedOrderByScan node) {
+ List<RelFieldCollation> fieldCollations =
+ node.getOrderByItemList().stream()
+ .map(LimitOffsetScanToOrderByLimitConverter::orderByItemToFieldCollation)
+ .collect(toList());
+ return RelCollationImpl.of(fieldCollations);
+ }
+
+ private static RelFieldCollation orderByItemToFieldCollation(ResolvedOrderByItem item) {
+ // TODO: might need a column ref mapping here.
+ Direction sortDirection = item.getIsDescending() ? DESCENDING : ASCENDING;
+ int fieldIndex = (int) item.getColumnRef().getColumn().getId();
+ return new RelFieldCollation(fieldIndex, sortDirection);
+ }
+
+ private RelNode convertOrderByScanToLogicalScan(ResolvedOrderByScan node, RelNode input) {
+ List<RexNode> projects =
+ getExpressionConverter()
+ .retrieveRexNodeFromOrderByScan(getCluster(), node, input.getRowType().getFieldList());
+ List<String> fieldNames = getTrait().retrieveFieldNames(node.getColumnList());
+
+ return LogicalProject.create(input, projects, fieldNames);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
new file mode 100644
index 0000000..878b2b2
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+
+/**
+ * Always throws exception, represents the case when order by is used without limit.
+ *
+ * <p>Order by limit is a special case that is handled in {@link LimitOffsetScanToLimitConverter}.
+ */
+class OrderByScanUnsupportedConverter extends RelConverter<ResolvedOrderByScan> {
+
+ OrderByScanUnsupportedConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public RelNode convert(ResolvedOrderByScan zetaNode, List<RelNode> inputs) {
+ throw new UnsupportedOperationException("ORDER BY without a LIMIT is not supported.");
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
new file mode 100644
index 0000000..d19b765
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/** Converts projection. */
+class ProjectScanConverter extends RelConverter<ResolvedProjectScan> {
+
+ ProjectScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedProjectScan zetaNode) {
+ return Collections.singletonList(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedProjectScan zetaNode, List<RelNode> inputs) {
+ RelNode input = inputs.get(0);
+
+ List<RexNode> projects =
+ getExpressionConverter().retrieveRexNode(zetaNode, input.getRowType().getFieldList());
+ List<String> fieldNames = getTrait().retrieveFieldNames(zetaNode.getColumnList());
+ return LogicalProject.create(input, projects, fieldNames);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
new file mode 100644
index 0000000..5513482
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_AGGREGATE_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_ARRAY_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_FILTER_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_JOIN_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_LIMIT_OFFSET_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_ORDER_BY_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_PROJECT_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_SET_OPERATION_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_SINGLE_ROW_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_TABLE_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_WITH_REF_SCAN;
+import static com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind.RESOLVED_WITH_SCAN;
+import static java.util.stream.Collectors.toList;
+
+import com.google.zetasql.ZetaSQLResolvedNodeKind.ResolvedNodeKind;
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedQueryStmt;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
+
+/**
+ * Converts a resolved Zeta SQL query represented by a tree to corresponding Calcite representation.
+ */
+public class QueryStatementConverter extends RelConverter<ResolvedQueryStmt> {
+
+ /** Conversion rules, multimap from node kind to conversion rule. */
+ private final ImmutableMultimap<ResolvedNodeKind, RelConverter> rules;
+
+ public static RelNode convertRootQuery(ConversionContext context, ResolvedQueryStmt query) {
+ return new QueryStatementConverter(context).convert(query, Collections.emptyList());
+ }
+
+ private QueryStatementConverter(ConversionContext context) {
+ super(context);
+ this.rules =
+ ImmutableMultimap.<ResolvedNodeKind, RelConverter>builder()
+ .put(RESOLVED_AGGREGATE_SCAN, new AggregateScanConverter(context))
+ .put(RESOLVED_ARRAY_SCAN, new ArrayScanToJoinConverter(context))
+ .put(RESOLVED_ARRAY_SCAN, new ArrayScanToUncollectConverter(context))
+ .put(RESOLVED_FILTER_SCAN, new FilterScanConverter(context))
+ .put(RESOLVED_JOIN_SCAN, new JoinScanConverter(context))
+ .put(RESOLVED_JOIN_SCAN, new JoinScanWithRefConverter(context))
+ .put(RESOLVED_LIMIT_OFFSET_SCAN, new LimitOffsetScanToLimitConverter(context))
+ .put(RESOLVED_LIMIT_OFFSET_SCAN, new LimitOffsetScanToOrderByLimitConverter(context))
+ .put(RESOLVED_ORDER_BY_SCAN, new OrderByScanUnsupportedConverter(context))
+ .put(RESOLVED_PROJECT_SCAN, new ProjectScanConverter(context))
+ .put(RESOLVED_SET_OPERATION_SCAN, new SetOperationScanConverter(context))
+ .put(RESOLVED_SINGLE_ROW_SCAN, new SingleRowScanConverter(context))
+ .put(RESOLVED_TABLE_SCAN, new TableScanConverter(context))
+ .put(RESOLVED_WITH_REF_SCAN, new WithRefScanConverter(context))
+ .put(RESOLVED_WITH_SCAN, new WithScanConverter(context))
+ .build();
+ }
+
+ @Override
+ public RelNode convert(ResolvedQueryStmt zetaNode, List<RelNode> inputs) {
+ if (zetaNode.getIsValueTable()) {
+ throw new UnsupportedOperationException("Value Tables are not supported");
+ }
+
+ getTrait().addOutputColumnList(zetaNode.getOutputColumnList());
+
+ return convertNode(zetaNode.getQuery());
+ }
+
+ /**
+ * Convert node.
+ *
+ * <p>Finds a matching rule, uses the rule to extract inputs from the node, then converts the
+ * inputs (recursively), then converts the node using the converted inputs.
+ */
+ private RelNode convertNode(ResolvedNode zetaNode) {
+ RelConverter nodeConverter = getConverterRule(zetaNode);
+ List<ResolvedNode> inputs = nodeConverter.getInputs(zetaNode);
+ List<RelNode> convertedInputs = inputs.stream().map(this::convertNode).collect(toList());
+ return nodeConverter.convert(zetaNode, convertedInputs);
+ }
+
+ private RelConverter getConverterRule(ResolvedNode zetaNode) {
+ if (!rules.containsKey(zetaNode.nodeKind())) {
+ throw new UnsupportedOperationException(
+ String.format("Conversion of %s is not supported", zetaNode.nodeKind()));
+ }
+
+ return rules.get(zetaNode.nodeKind()).stream()
+ .filter(relConverter -> relConverter.canConvert(zetaNode))
+ .findFirst()
+ .orElseThrow(
+ () ->
+ new UnsupportedOperationException(
+ String.format("Cannot find a conversion rule for: %s", zetaNode)));
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
new file mode 100644
index 0000000..2b1b722
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import java.util.List;
+import org.apache.beam.sdk.extensions.sql.zetasql.QueryTrait;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** A rule that converts Zeta SQL resolved relational node to corresponding Calcite rel node. */
+abstract class RelConverter<T extends ResolvedNode> {
+
+ /**
+ * Conversion context, contains things like FrameworkConfig, QueryTrait and other state used
+ * during conversion.
+ */
+ protected ConversionContext context;
+
+ RelConverter(ConversionContext context) {
+ this.context = context;
+ }
+
+ /** Whether this rule can handle the conversion of the specific node. */
+ public boolean canConvert(T zetaNode) {
+ return true;
+ }
+
+ /** Extract Zeta SQL resolved nodes that correspond to the inputs of the current node. */
+ public List<ResolvedNode> getInputs(T zetaNode) {
+ return ImmutableList.of();
+ }
+
+ /**
+ * Converts given Zeta SQL node to corresponding Calcite node.
+ *
+ * <p>{@code inputs} are node inputs that have already been converter to Calcite versions. They
+ * correspond to the nodes in {@link #getInputs(ResolvedNode)}.
+ */
+ public abstract RelNode convert(T zetaNode, List<RelNode> inputs);
+
+ protected RelOptCluster getCluster() {
+ return context.cluster();
+ }
+
+ protected FrameworkConfig getConfig() {
+ return context.getConfig();
+ }
+
+ protected ExpressionConverter getExpressionConverter() {
+ return context.getExpressionConverter();
+ }
+
+ protected QueryTrait getTrait() {
+ return context.getTrait();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
new file mode 100644
index 0000000..375021b
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.EXCEPT_ALL;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.EXCEPT_DISTINCT;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.INTERSECT_ALL;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.INTERSECT_DISTINCT;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.UNION_ALL;
+import static com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType.UNION_DISTINCT;
+import static java.util.stream.Collectors.toList;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSetOperationItem;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSetOperationScan;
+import com.google.zetasql.resolvedast.ResolvedSetOperationScanEnums.SetOperationType;
+import java.util.List;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalIntersect;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalMinus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalUnion;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/** Converts set operations. */
+class SetOperationScanConverter extends RelConverter<ResolvedSetOperationScan> {
+ private enum Type {
+ DISTINCT,
+ ALL
+ }
+
+ private static final ImmutableMap<SetOperationType, Function<List<RelNode>, RelNode>>
+ SET_OPERATION_FACTORIES =
+ ImmutableMap.<SetOperationType, Function<List<RelNode>, RelNode>>builder()
+ .put(UNION_ALL, createFactoryFor(LogicalUnion::create, Type.ALL))
+ .put(UNION_DISTINCT, createFactoryFor(LogicalUnion::create, Type.DISTINCT))
+ .put(INTERSECT_ALL, createFactoryFor(LogicalIntersect::create, Type.ALL))
+ .put(INTERSECT_DISTINCT, createFactoryFor(LogicalIntersect::create, Type.DISTINCT))
+ .put(EXCEPT_ALL, createFactoryFor(LogicalMinus::create, Type.ALL))
+ .put(EXCEPT_DISTINCT, createFactoryFor(LogicalMinus::create, Type.DISTINCT))
+ .build();
+
+ /**
+ * A little closure to wrap the invocation of the factory method (e.g. LogicalUnion::create) for
+ * the set operation node.
+ */
+ private static Function<List<RelNode>, RelNode> createFactoryFor(
+ BiFunction<List<RelNode>, Boolean, RelNode> setOperationFactory, Type type) {
+ return (List<RelNode> inputs) -> createRel(setOperationFactory, type == Type.ALL, inputs);
+ }
+
+ SetOperationScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedSetOperationScan zetaNode) {
+ return zetaNode.getInputItemList().stream()
+ .map(ResolvedSetOperationItem::getScan)
+ .collect(toList());
+ }
+
+ @Override
+ public RelNode convert(ResolvedSetOperationScan zetaNode, List<RelNode> inputs) {
+ if (!SET_OPERATION_FACTORIES.containsKey(zetaNode.getOpType())) {
+ throw new UnsupportedOperationException(
+ "Operation " + zetaNode.getOpType() + " is unsupported");
+ }
+
+ return SET_OPERATION_FACTORIES.get(zetaNode.getOpType()).apply(inputs);
+ }
+
+ /** Beam set operations rel expects two inputs, so we are constructing a binary tree here. */
+ private static RelNode createRel(
+ BiFunction<List<RelNode>, Boolean, RelNode> factory, boolean all, List<RelNode> inputs) {
+ return inputs.stream()
+ .skip(2)
+ .reduce(
+ // start with creating a set node for two first inputs
+ invokeFactory(factory, inputs.get(0), inputs.get(1), all),
+ // create another operation node with previous op node and the next input
+ (setOpNode, nextInput) -> invokeFactory(factory, setOpNode, nextInput, all));
+ }
+
+ /**
+ * Creates a set operation rel with two inputs.
+ *
+ * <p>Factory is, for example, LogicalUnion::create.
+ */
+ private static RelNode invokeFactory(
+ BiFunction<List<RelNode>, Boolean, RelNode> factory,
+ RelNode input1,
+ RelNode input2,
+ boolean all) {
+ return factory.apply(ImmutableList.of(input1, input2), all);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
new file mode 100644
index 0000000..4721b33
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedSingleRowScan;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalValues;
+
+/** Converts a single row value. */
+class SingleRowScanConverter extends RelConverter<ResolvedSingleRowScan> {
+
+ SingleRowScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedSingleRowScan zetaNode) {
+ return zetaNode.getColumnList() == null || zetaNode.getColumnList().isEmpty();
+ }
+
+ @Override
+ public RelNode convert(ResolvedSingleRowScan zetaNode, List<RelNode> inputs) {
+ return LogicalValues.createOneRow(getCluster());
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
new file mode 100644
index 0000000..2f0c1e6
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_DATETIME;
+import static com.google.zetasql.ZetaSQLType.TypeKind.TYPE_NUMERIC;
+import static org.apache.beam.vendor.calcite.v1_20_0.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.resolvedast.ResolvedColumn;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedTableScan;
+import java.util.List;
+import java.util.Properties;
+import org.apache.beam.sdk.extensions.sql.zetasql.TableResolution;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.config.CalciteConnectionConfigImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptTable;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.RelOptTableImpl;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelRoot;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.type.RelDataType;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.Table;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.TranslatableTable;
+
+/** Converts table scan. */
+class TableScanConverter extends RelConverter<ResolvedTableScan> {
+
+ private static final ImmutableSet<TypeKind> UNSUPPORTED_DATA_TYPES =
+ ImmutableSet.of(TYPE_DATETIME, TYPE_NUMERIC);
+
+ TableScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public RelNode convert(ResolvedTableScan zetaNode, List<RelNode> inputs) {
+ checkTableScanSchema(zetaNode.getColumnList());
+
+ List<String> tablePath = getTablePath(zetaNode.getTable());
+
+ SchemaPlus defaultSchemaPlus = getConfig().getDefaultSchema();
+ // TODO: reject incorrect top-level schema
+
+ Table calciteTable =
+ TableResolution.resolveCalciteTable(getConfig().getContext(), defaultSchemaPlus, tablePath);
+
+ // we already resolved the table before passing the query to Analyzer, so it should be there
+ checkNotNull(
+ calciteTable,
+ "Unable to resolve the table path %s in schema %s",
+ tablePath,
+ defaultSchemaPlus.getName());
+
+ String defaultSchemaName = defaultSchemaPlus.getName();
+
+ final CalciteCatalogReader catalogReader =
+ new CalciteCatalogReader(
+ CalciteSchema.from(defaultSchemaPlus),
+ ImmutableList.of(defaultSchemaName),
+ getCluster().getTypeFactory(),
+ new CalciteConnectionConfigImpl(new Properties()));
+
+ RelOptTableImpl relOptTable =
+ RelOptTableImpl.create(
+ catalogReader,
+ calciteTable.getRowType(getCluster().getTypeFactory()),
+ calciteTable,
+ ImmutableList.<String>builder().add(defaultSchemaName).addAll(tablePath).build());
+
+ if (calciteTable instanceof TranslatableTable) {
+ return ((TranslatableTable) calciteTable).toRel(createToRelContext(), relOptTable);
+ } else {
+ throw new RuntimeException("Does not support non TranslatableTable type table!");
+ }
+ }
+
+ private List<String> getTablePath(com.google.zetasql.Table table) {
+ if (!getTrait().isTableResolved(table)) {
+ throw new RuntimeException(
+ "Unexpected table found when converting to Calcite rel node: " + table);
+ }
+
+ return getTrait().getTablePath(table);
+ }
+
+ private RelOptTable.ToRelContext createToRelContext() {
+ return new RelOptTable.ToRelContext() {
+ @Override
+ public RelRoot expandView(
+ RelDataType relDataType, String s, List<String> list, List<String> list1) {
+ throw new UnsupportedOperationException("This RelContext does not support expandView");
+ }
+
+ @Override
+ public RelOptCluster getCluster() {
+ return TableScanConverter.this.getCluster();
+ }
+ };
+ }
+
+ private void checkTableScanSchema(List<ResolvedColumn> columnList) {
+ if (columnList != null) {
+ for (ResolvedColumn resolvedColumn : columnList) {
+ if (UNSUPPORTED_DATA_TYPES.contains(resolvedColumn.getType().getKind())) {
+ throw new IllegalArgumentException(
+ "Does not support " + UNSUPPORTED_DATA_TYPES + " types in source tables");
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
new file mode 100644
index 0000000..6dceeef
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithRefScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+
+/** Converts a call-site reference to a named WITH subquery. */
+class WithRefScanConverter extends RelConverter<ResolvedWithRefScan> {
+
+ WithRefScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedWithRefScan zetaNode) {
+ // WithRefScan contains only a name of a WITH query,
+ // but to actually convert it to the node we need to get the resolved node representation
+ // of the query. Here we take it from the trait, where it was persisted previously
+ // in WithScanConverter that actually parses the WITH query part.
+ //
+ // This query node returned from here will be converted by some other converter,
+ // (e.g. if the WITH query root is a projection it will go through ProjectScanConverter)
+ // and will reach the convert() method below as an already converted rel node.
+ return Collections.singletonList(
+ getTrait().withEntries.get(zetaNode.getWithQueryName()).getWithSubquery());
+ }
+
+ @Override
+ public RelNode convert(ResolvedWithRefScan zetaNode, List<RelNode> inputs) {
+ // Here the actual WITH query body has already been converted by, e.g. a ProjectScnaConverter,
+ // so to resolve the reference we just return that converter rel node.
+ return inputs.get(0);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
new file mode 100644
index 0000000..7159356
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedWithScan;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+
+/** Converts a named WITH. */
+class WithScanConverter extends RelConverter<ResolvedWithScan> {
+
+ WithScanConverter(ConversionContext context) {
+ super(context);
+ }
+
+ @Override
+ public List<ResolvedNode> getInputs(ResolvedWithScan zetaNode) {
+ // We must persist the named WITH queries nodes,
+ // so that when they are referenced by name (e.g. in FROM/JOIN), we can
+ // resolve them. We need this because the nodes that represent the references (WithRefScan)
+ // only contain the names of the queries, so we need to keep this map for resolution of the
+ // names.
+ zetaNode
+ .getWithEntryList()
+ .forEach(withEntry -> getTrait().withEntries.put(withEntry.getWithQueryName(), withEntry));
+
+ // Returning the body of the query, it is something like ProjectScan that will be converted
+ // by ProjectScanConverter before it reaches the convert() method below.
+ return Collections.singletonList(zetaNode.getQuery());
+ }
+
+ @Override
+ public RelNode convert(ResolvedWithScan zetaNode, List<RelNode> inputs) {
+ // The body of the WITH query is already converted at this point so we just
+ // return it, nothing else is needed.
+ return inputs.get(0);
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
new file mode 100644
index 0000000..d01c656
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** Conversion logic between ZetaSQL resolved query nodes and Calcite rel nodes. */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
diff --git a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
new file mode 100644
index 0000000..17eb24d
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
@@ -0,0 +1,342 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT;
+
+import java.util.List;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for identifiers. */
+@RunWith(JUnit4.class)
+public class JoinCompoundIdentifiersTest {
+
+ private static final Long TWO_MINUTES = 2L;
+ private static final String DEFAULT_SCHEMA = "beam";
+ private static final String FULL_ON_ID =
+ "a.`b-\\`c`.d.`httz://d.e-f.g:233333/blah\\?yes=1&true=false`";
+ private static final String TABLE_WITH_STRUCTS_ID = "a.`table:com`.`..::with-struct::..`";
+
+ private static final TableProvider TEST_TABLES =
+ new ReadOnlyTableProvider(
+ "test_table_provider",
+ ImmutableMap.<String, BeamSqlTable>builder()
+ .put("KeyValue", BASIC_TABLE_ONE)
+ .put("a.b", BASIC_TABLE_ONE)
+ .put("c.d.e", BASIC_TABLE_ONE)
+ .put("c.d.f", BASIC_TABLE_TWO)
+ .put("c.g.e", BASIC_TABLE_TWO)
+ .put("weird.`\\n\\t\\r\\f`", BASIC_TABLE_ONE)
+ .put("a.`b-\\`c`.d", BASIC_TABLE_TWO)
+ .put(FULL_ON_ID, BASIC_TABLE_TWO)
+ .put(TABLE_WITH_STRUCTS_ID, TABLE_WITH_STRUCT)
+ .build());
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testComplexTableName() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT Key FROM a.b");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testComplexTableName3Levels() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT Key FROM c.d.e");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testOnePartWithBackticks() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT RowKey FROM a.`b-\\`c`.d");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testNewLinesAndOtherWhitespace() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(pipeline, cfg, "SELECT Key FROM weird.`\\n\\t\\r\\f`");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testFullOnWithBackticks() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT RowKey FROM " + FULL_ON_ID);
+
+ PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testJoinWithFullOnWithBackticks() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.RowKey FROM "
+ + FULL_ON_ID
+ + " AS t1 \n"
+ + " INNER JOIN a.`b-\\`c`.d t2 on t1.RowKey = t2.RowKey");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(16L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithAliasedComplexTableName() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT t.Key FROM a.b AS t");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithAliasedComplexTableName3Levels() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT t.Key FROM c.d.e AS t");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithUnaliasedComplexTableName() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT b.Key FROM a.b");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithUnaliasedComplexTableName3Levels() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT e.Key FROM c.d.e");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithUnaliasedComplexTableName3Levels2() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result = applySqlTransform(pipeline, cfg, "SELECT e.Key FROM c.d.e");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithJoinOfAliasedComplexTableNames() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.Key FROM a.b AS t1 INNER JOIN c.d.e AS t2 ON t1.Key = t2.Key");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testJoinTwoTablesWithLastPartIdDifferent() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.Key FROM c.d.e AS t1 INNER JOIN c.d.f AS t2 ON t1.Key = t2.RowKey");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testJoinTwoTablesWithMiddlePartIdDifferent() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t1.Key FROM c.d.e AS t1 INNER JOIN c.g.e AS t2 ON t1.Key = t2.RowKey");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedFieldAccessWithJoinOfUnaliasedComplexTableNames() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(pipeline, cfg, "SELECT b.Key FROM a.b INNER JOIN c.d.e ON b.Key = e.Key");
+
+ PAssert.that(result).containsInAnyOrder(singleValue(14L), singleValue(15L));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testStructFieldAccess() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT struct_col.struct_col_str FROM a.`table:com`.`..::with-struct::..`");
+
+ PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testQualifiedStructFieldAccess() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT `..::with-struct::..`.struct_col.struct_col_str \n"
+ + " FROM a.`table:com`.`..::with-struct::..`");
+
+ PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @Test
+ public void testAliasedStructFieldAccess() throws Exception {
+ FrameworkConfig cfg = initializeCalcite();
+
+ PCollection<Row> result =
+ applySqlTransform(
+ pipeline,
+ cfg,
+ "SELECT t.struct_col.struct_col_str FROM " + TABLE_WITH_STRUCTS_ID + " t");
+
+ PAssert.that(result).containsInAnyOrder(singleValue("row_one"), singleValue("row_two"));
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(TWO_MINUTES));
+ }
+
+ @SuppressWarnings("unxchecked")
+ private static FrameworkConfig initializeCalcite() {
+ JdbcConnection jdbcConnection =
+ JdbcDriver.connect(TEST_TABLES, PipelineOptionsFactory.create());
+ SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
+ List<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ Object[] contexts =
+ ImmutableList.of(
+ Contexts.of(jdbcConnection.config()),
+ TableResolutionContext.joinCompoundIds(DEFAULT_SCHEMA))
+ .toArray();
+
+ return Frameworks.newConfigBuilder()
+ .defaultSchema(defaultSchemaPlus)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(contexts))
+ .ruleSets(BeamRuleSets.getRuleSets())
+ .costFactory(null)
+ .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
+ .build();
+ }
+
+ private PCollection<Row> applySqlTransform(
+ Pipeline pipeline, FrameworkConfig config, String query) throws Exception {
+
+ BeamRelNode beamRelNode = new ZetaSQLQueryPlanner(config).parseQuery(query);
+ return BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ }
+
+ private Row singleValue(long value) {
+ return Row.withSchema(singleLongField()).addValue(value).build();
+ }
+
+ private Row singleValue(String value) {
+ return Row.withSchema(singleStringField()).addValue(value).build();
+ }
+
+ private Schema singleLongField() {
+ return Schema.builder().addInt64Field("field1").build();
+ }
+
+ private Schema singleStringField() {
+ return Schema.builder().addStringField("field1").build();
+ }
+}
diff --git a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
new file mode 100644
index 0000000..70c185f
--- /dev/null
+++ b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -0,0 +1,3787 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql;
+
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseDate;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseDateToValue;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTime;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimeToValue;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithTZToValue;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithTimeZone;
+import static org.apache.beam.sdk.extensions.sql.zetasql.DateTimeUtils.parseTimestampWithUTCTimeZone;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.AGGREGATE_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.AGGREGATE_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_THREE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.BASIC_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_NULL;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_TYPES;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_ALL_TYPES_2;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_EMPTY;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_FOR_CASE_WHEN;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_ARRAY;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_ARRAY_FOR_UNNEST;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_MAP;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT_TIMESTAMP_STRING;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TABLE_WITH_STRUCT_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIMESTAMP_TABLE_ONE;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIMESTAMP_TABLE_TWO;
+import static org.apache.beam.sdk.extensions.sql.zetasql.TestInput.TIME_TABLE;
+import static org.apache.beam.sdk.schemas.Schema.FieldType.DATETIME;
+
+import com.google.protobuf.ByteString;
+import com.google.zetasql.SqlException;
+import com.google.zetasql.StructType.StructField;
+import com.google.zetasql.TypeFactory;
+import com.google.zetasql.Value;
+import com.google.zetasql.ZetaSQLType.TypeKind;
+import com.google.zetasql.ZetaSQLValue.ValueProto;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcConnection;
+import org.apache.beam.sdk.extensions.sql.impl.JdbcDriver;
+import org.apache.beam.sdk.extensions.sql.impl.planner.BeamRuleSets;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTable;
+import org.apache.beam.sdk.extensions.sql.meta.provider.ReadOnlyTableProvider;
+import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.schemas.Schema.FieldType;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Context;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Contexts;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelTraitDef;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.schema.SchemaPlus;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.FrameworkConfig;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.DateTime;
+import org.joda.time.Duration;
+import org.joda.time.chrono.ISOChronology;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** ZetaSQLDialectSpecTest. */
+@RunWith(JUnit4.class)
+public class ZetaSQLDialectSpecTest {
+ private static final Long PIPELINE_EXECUTION_WAITTIME_MINUTES = 2L;
+
+ private FrameworkConfig config;
+
+ private TableProvider tableProvider;
+
+ @Rule public transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Before
+ public void setUp() {
+ initializeBeamTableProvider();
+ initializeCalciteEnvironment();
+ }
+
+ @Test
+ public void testSimpleSelect() {
+ String sql =
+ "SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addDateTimeField("field2")
+ .addStringField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ1() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_BOOL))
+ .put("p1", Value.createBoolValue(true))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Does not support inf/-inf/nan in double/float literals because double/float literals are"
+ + " converted to BigDecimal in Calcite codegen.")
+ public void testEQ2() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createDoubleValue(0))
+ .put("p1", Value.createDoubleValue(Double.POSITIVE_INFINITY))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addBooleanField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ3() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_DOUBLE))
+ .put("p1", Value.createDoubleValue(3.14))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ4() {
+ String sql = "SELECT @p0 = @p1 AS ColA";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createBytesValue(ByteString.copyFromUtf8("hello")))
+ .put("p1", Value.createBytesValue(ByteString.copyFromUtf8("hello")))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEQ5() {
+ String sql = "SELECT b'hello' = b'hello' AS ColA";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIsNotNull1() {
+ String sql = "SELECT @p0 IS NOT NULL AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIsNotNull2() {
+ String sql = "SELECT @p0 IS NOT NULL AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createNullValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support struct literal.")
+ public void testIsNotNull3() {
+ String sql = "SELECT @p0 IS NOT NULL AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createNullValue(
+ TypeFactory.createStructType(
+ Arrays.asList(
+ new StructField(
+ "a", TypeFactory.createSimpleType(TypeKind.TYPE_STRING))))));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfBasic() {
+ String sql = "SELECT IF(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createBoolValue(true),
+ "p1",
+ Value.createInt64Value(1),
+ "p2",
+ Value.createInt64Value(2));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.INT64).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(1L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCoalesceBasic() {
+ String sql = "SELECT COALESCE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1",
+ Value.createStringValue("yay"),
+ "p2",
+ Value.createStringValue("nay"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("yay").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCoalesceSingleArgument() {
+ String sql = "SELECT COALESCE(@p0) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_INT64));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.array(FieldType.INT64)).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCoalesceNullArray() {
+ String sql = "SELECT COALESCE(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createNullValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))),
+ "p1",
+ Value.createNullValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64))));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.array(FieldType.INT64)).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNullIfCoercion() {
+ String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createInt64Value(3L),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_DOUBLE));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.DOUBLE).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(3.0).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Struct literals are not currently supported")
+ public void testCoalesceNullStruct() {
+ String sql = "SELECT COALESCE(NULL, STRUCT(\"a\" AS s, -33 AS i))";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema innerSchema =
+ Schema.of(Field.of("s", FieldType.STRING), Field.of("i", FieldType.INT64));
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.row(innerSchema)).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValue(Row.withSchema(innerSchema).addValues("a", -33).build())
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfTimestamp() {
+ String sql = "SELECT IF(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createBoolValue(false),
+ "p1",
+ Value.createTimestampValueFromUnixMicros(0),
+ "p2",
+ Value.createTimestampValueFromUnixMicros(
+ DateTime.parse("2019-01-01T00:00:00Z").getMillis() * 1000));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", DATETIME).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(DateTime.parse("2019-01-01T00:00:00Z")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("$make_array is not implemented")
+ public void testMakeArray() {
+ String sql = "SELECT [s3, s1, s2] FROM (SELECT \"foo\" AS s1, \"bar\" AS s2, \"baz\" AS s3);";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addNullableField("field1", FieldType.array(FieldType.STRING)).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(ImmutableList.of("baz", "foo", "bar")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNullIfPositive() {
+ String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("null"), "p1", Value.createStringValue("null"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNullIfNegative() {
+ String sql = "SELECT NULLIF(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("foo"), "p1", Value.createStringValue("null"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("foo").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfNullPositive() {
+ String sql = "SELECT IFNULL(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("foo"), "p1", Value.createStringValue("default"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("foo").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIfNullNegative() {
+ String sql = "SELECT IFNULL(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1",
+ Value.createStringValue("yay"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("yay").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Throws IndexOutOfBoundsException")
+ public void testConstructEmptyArrayLiteral() {
+ String sql = "SELECT @p0 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createArrayValue(
+ TypeFactory.createArrayType(TypeFactory.createSimpleType(TypeKind.TYPE_INT64)),
+ ImmutableList.of()));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addArrayField("field1", FieldType.INT64).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValue(ImmutableList.of()).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testLike1() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("ab%"), "p1", Value.createStringValue("ab\\%"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLikeNullPattern() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createStringValue("ab%"),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testLikeAllowsEscapingNonSpecialCharacter() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue("ab"), "p1", Value.createStringValue("\\ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testLikeAllowsEscapingBackslash() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("a\\c"), "p1", Value.createStringValue("a\\\\c"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Currently non UTF-8 values are coerced to UTF-8")
+ public void testThrowsErrorForNonUTF8() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ byte[] bytes = {(byte) 0xe8, (byte) 0xb0};
+ Value bad =
+ Value.deserialize(
+ TypeFactory.createSimpleType(TypeKind.TYPE_STRING),
+ ValueProto.newBuilder().setStringValueBytes(ByteString.copyFrom(bytes)).build());
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue("abc"), "p1", bad);
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ // TODO: message should be a constant on ZetaSQLPlannerImpl
+ thrown.expectMessage("invalid UTF-8");
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ @Test
+ @Ignore("Does not support BYTES for like")
+ public void testLikeBytes() {
+ String sql = "SELECT @p0 LIKE @p1 AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createBytesValue(ByteString.copyFromUtf8("abcd")),
+ "p1",
+ Value.createBytesValue(ByteString.copyFromUtf8("__%")));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testMod() {
+ String sql = "SELECT MOD(4, 2)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(0L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSimpleUnionAll() {
+ String sql =
+ "SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING) "
+ + " UNION ALL "
+ + " SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addDateTimeField("field2")
+ .addStringField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testThreeWayUnionAll() {
+ String sql = "SELECT a FROM (SELECT 1 a UNION ALL SELECT 2 UNION ALL SELECT 3)";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L).build(),
+ Row.withSchema(schema).addValues(2L).build(),
+ Row.withSchema(schema).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSimpleUnionDISTINCT() {
+ String sql =
+ "SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING) "
+ + " UNION DISTINCT "
+ + " SELECT CAST (1243 as INT64), "
+ + "CAST ('2018-09-15 12:59:59.000000+00' as TIMESTAMP), "
+ + "CAST ('string' as STRING);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addDateTimeField("field2")
+ .addStringField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1243L,
+ new DateTime(2018, 9, 15, 12, 59, 59, ISOChronology.getInstanceUTC()),
+ "string")
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLInnerJoin() {
+ String sql =
+ "SELECT t1.Key "
+ + "FROM KeyValue AS t1"
+ + " INNER JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey AND t1.ts = t2.ts";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ // JOIN USING(col) is equivalent to JOIN on left.col = right.col.
+ public void testZetaSQLInnerJoinWithUsing() {
+ String sql = "SELECT t1.Key " + "FROM KeyValue AS t1" + " INNER JOIN BigTable AS t2 USING(ts)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ // testing ordering of the JOIN conditions.
+ public void testZetaSQLInnerJoinTwo() {
+ String sql =
+ "SELECT t2.RowKey "
+ + "FROM KeyValue AS t1"
+ + " INNER JOIN BigTable AS t2"
+ + " on "
+ + " t2.RowKey = t1.Key AND t2.ts = t1.ts";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLLeftOuterJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " LEFT JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schemaOne =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addNullableField("field4", FieldType.INT64)
+ .addNullableField("field5", FieldType.STRING)
+ .addNullableField("field6", DATETIME)
+ .build();
+
+ final Schema schemaTwo =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schemaOne)
+ .addValues(
+ 14L,
+ "KeyValue234",
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ null,
+ null,
+ null)
+ .build(),
+ Row.withSchema(schemaTwo)
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLRightOuterJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " RIGHT JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schemaOne =
+ Schema.builder()
+ .addNullableField("field1", FieldType.INT64)
+ .addNullableField("field2", FieldType.STRING)
+ .addNullableField("field3", DATETIME)
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ final Schema schemaTwo =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schemaOne)
+ .addValues(
+ null,
+ null,
+ null,
+ 16L,
+ "BigTable236",
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schemaTwo)
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLFullOuterJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " FULL JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schemaOne =
+ Schema.builder()
+ .addNullableField("field1", FieldType.INT64)
+ .addNullableField("field2", FieldType.STRING)
+ .addNullableField("field3", DATETIME)
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ final Schema schemaTwo =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addInt64Field("field4")
+ .addStringField("field5")
+ .addDateTimeField("field6")
+ .build();
+
+ final Schema schemaThree =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .addNullableField("field4", FieldType.INT64)
+ .addNullableField("field5", FieldType.STRING)
+ .addNullableField("field6", DATETIME)
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schemaOne)
+ .addValues(
+ null,
+ null,
+ null,
+ 16L,
+ "BigTable236",
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schemaTwo)
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schemaThree)
+ .addValues(
+ 14L,
+ "KeyValue234",
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ null,
+ null,
+ null)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("BeamSQL only supports equal join")
+ public void testZetaSQLFullOuterJoinTwo() {
+ String sql =
+ "SELECT * "
+ + "FROM KeyValue AS t1"
+ + " FULL JOIN BigTable AS t2"
+ + " on "
+ + " t1.Key + t2.RowKey = 30";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLThreeWayInnerJoin() {
+ String sql =
+ "SELECT t3.Value, t2.Value, t1.Value, t1.Key, t3.ColId FROM KeyValue as t1 "
+ + "JOIN BigTable as t2 "
+ + "ON (t1.Key = t2.RowKey) "
+ + "JOIN Spanner as t3 "
+ + "ON (t3.ColId = t1.Key)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addStringField("t3.Value")
+ .addStringField("t2.Value")
+ .addStringField("t1.Value")
+ .addInt64Field("t1.Key")
+ .addInt64Field("t3.ColId")
+ .build())
+ .addValues("Spanner235", "BigTable235", "KeyValue235", 15L, 15L)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLTableJoinOnItselfWithFiltering() {
+ String sql =
+ "SELECT * FROM Spanner as t1 "
+ + "JOIN Spanner as t2 "
+ + "ON (t1.ColId = t2.ColId) WHERE t1.ColId = 17";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addInt64Field("field3")
+ .addStringField("field4")
+ .build())
+ .addValues(17L, "Spanner237", 17L, "Spanner237")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromSelect() {
+ String sql = "SELECT * FROM (SELECT \"apple\" AS fruit, \"carrot\" AS vegetable);";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder().addStringField("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("apple", "carrot").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(2, outputSchema.getFieldCount());
+ Assert.assertEquals("fruit", outputSchema.getField(0).getName());
+ Assert.assertEquals("vegetable", outputSchema.getField(1).getName());
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTable() {
+ String sql = "SELECT Key, Value FROM KeyValue;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableLimit() {
+ String sql = "SELECT Key, Value FROM KeyValue LIMIT 2;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableLimit0() {
+ String sql = "SELECT Key, Value FROM KeyValue LIMIT 0;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableLimitOffset() {
+ String sql =
+ "SELECT COUNT(a) FROM (\n"
+ + "SELECT a FROM (SELECT 1 a UNION ALL SELECT 2 UNION ALL SELECT 3) LIMIT 3 OFFSET 1);";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // There is really no order for a PCollection, so this query does not test
+ // ORDER BY but just a test to see if ORDER BY LIMIT can work.
+ @Test
+ public void testZetaSQLSelectFromTableOrderByLimit() {
+ String sql = "SELECT Key, Value FROM KeyValue ORDER BY Key DESC LIMIT 2;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableOrderBy() {
+ String sql = "SELECT Key, Value FROM KeyValue ORDER BY Key DESC;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("ORDER BY without a LIMIT is not supported.");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableWithStructType2() {
+ String sql =
+ "SELECT table_with_struct.struct_col.struct_col_str FROM table_with_struct WHERE id = 1;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue("row_one").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInFilter() {
+ String sql =
+ "SELECT table_with_struct.id FROM table_with_struct WHERE"
+ + " table_with_struct.struct_col.struct_col_str = 'row_one';";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue(1L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInCast() {
+ String sql =
+ "SELECT CAST(table_with_struct.id AS STRING) FROM table_with_struct WHERE"
+ + " table_with_struct.struct_col.struct_col_str = 'row_one';";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValue("1").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInCast2() {
+ String sql =
+ "SELECT CAST(A.struct_col.struct_col_str AS TIMESTAMP) FROM table_with_struct_ts_string AS"
+ + " A";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValue(parseTimestampWithUTCTimeZone("2019-01-15 13:21:03"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInTumble() {
+ String sql =
+ "SELECT TUMBLE_START('INTERVAL 1 MINUTE') FROM table_with_struct_ts_string AS A GROUP BY "
+ + "TUMBLE(CAST(A.struct_col.struct_col_str AS TIMESTAMP), 'INTERVAL 1 MINUTE')";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValue(parseTimestampWithUTCTimeZone("2019-01-15 13:21:00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInGroupBy() {
+ String sql = "SELECT rowCol.row_id, COUNT(*) FROM table_with_struct_two GROUP BY rowCol.row_id";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 1L).build(),
+ Row.withSchema(schema).addValues(2L, 1L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLStructFieldAccessInGroupBy2() {
+ String sql =
+ "SELECT rowCol.data, MAX(rowCol.row_id), MIN(rowCol.row_id) FROM table_with_struct_two"
+ + " GROUP BY rowCol.data";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema =
+ Schema.builder()
+ .addStringField("field1")
+ .addInt64Field("field2")
+ .addInt64Field("field3")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("data1", 1L, 1L).build(),
+ Row.withSchema(schema).addValues("data2", 3L, 2L).build(),
+ Row.withSchema(schema).addValues("data3", 3L, 3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectFromTableWithArrayType() {
+ String sql = "SELECT array_col FROM table_with_array;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addArrayField("field", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(Arrays.asList("1", "2", "3")).build(),
+ Row.withSchema(schema).addValue(ImmutableList.of()).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLSelectStarFromTable() {
+ String sql = "SELECT * FROM BigTable;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addDateTimeField("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 16L,
+ "BigTable236",
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFiltering() {
+ String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder().addInt64Field("field1").addStringField("field2").build())
+ .addValues(14L, "KeyValue234")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFilteringTwo() {
+ String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14 AND Value = 'non-existing';";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFilteringThree() {
+ String sql = "SELECT Key, Value FROM KeyValue WHERE Key = 14 OR Key = 15;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLCountOnAColumn() {
+ String sql = "SELECT COUNT(Key) FROM KeyValue";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLAggDistinct() {
+ String sql = "SELECT Key, COUNT(DISTINCT Value) FROM KeyValue GROUP BY Key";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("Does not support COUNT DISTINCT");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg() {
+ String sql = "SELECT Key, COUNT(*) FROM KeyValue GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, 1L).build(),
+ Row.withSchema(schema).addValues(15L, 1L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias1() {
+ String sql = "SELECT Key, COUNT(*) AS count_col FROM KeyValue GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(2, outputSchema.getFieldCount());
+ Assert.assertEquals("Key", outputSchema.getField(0).getName());
+ Assert.assertEquals("count_col", outputSchema.getField(1).getName());
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias2() {
+ String sql =
+ "SELECT Key AS k1, (count_col + 1) AS k2 FROM (SELECT Key, COUNT(*) AS count_col FROM"
+ + " KeyValue GROUP BY Key)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(2, outputSchema.getFieldCount());
+ Assert.assertEquals("k1", outputSchema.getField(0).getName());
+ Assert.assertEquals("k2", outputSchema.getField(1).getName());
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias3() {
+ String sql = "SELECT Key AS v1, Value AS v2, ts AS v3 FROM KeyValue";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(3, outputSchema.getFieldCount());
+ Assert.assertEquals("v1", outputSchema.getField(0).getName());
+ Assert.assertEquals("v2", outputSchema.getField(1).getName());
+ Assert.assertEquals("v3", outputSchema.getField(2).getName());
+ }
+
+ @Test
+ public void testZetaSQLColumnAlias4() {
+ String sql = "SELECT CAST(123 AS INT64) AS cast_col";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+
+ Schema outputSchema = stream.getSchema();
+ Assert.assertEquals(1, outputSchema.getFieldCount());
+ Assert.assertEquals("cast_col", outputSchema.getField(0).getName());
+ }
+
+ @Test
+ public void testZetaSQLAmbiguousAlias() {
+ String sql = "SELECT row_id as ID, int64_col as ID FROM table_all_types GROUP BY ID;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+ thrown.expectMessage(
+ "Name ID in GROUP BY clause is ambiguous; it may refer to multiple columns in the"
+ + " SELECT-list [at 1:68]");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testZetaSQLAggWithOrdinalReference() {
+ String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY 1";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 3L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLAggWithAliasReference() {
+ String sql = "SELECT Key AS K, COUNT(*) FROM aggregate_test_table GROUP BY K";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 3L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg2() {
+ String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 3L).build(),
+ Row.withSchema(schema).addValues(3L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg3() {
+ String sql = "SELECT Key, Key2, COUNT(*) FROM aggregate_test_table GROUP BY Key2, Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field3")
+ .addInt64Field("field2")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 1L).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 1L).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg4() {
+ String sql =
+ "SELECT Key, Key2, MAX(f_int_1), MIN(f_int_1), SUM(f_int_1), SUM(f_double_1) "
+ + "FROM aggregate_test_table GROUP BY Key2, Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field3")
+ .addInt64Field("field2")
+ .addInt64Field("field4")
+ .addInt64Field("field5")
+ .addDoubleField("field6")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L, 1L, 1L, 1.0).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 2L, 2L, 2L, 2.0).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 4L, 3L, 7L, 7.0).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 5L, 5L, 5L, 5.0).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 7L, 6L, 13L, 13.0).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicAgg5() {
+ String sql =
+ "SELECT Key, Key2, AVG(CAST(f_int_1 AS FLOAT64)), AVG(f_double_1) "
+ + "FROM aggregate_test_table GROUP BY Key2, Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field2")
+ .addDoubleField("field3")
+ .addDoubleField("field4")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1.0, 1.0).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 2.0, 2.0).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 3.5, 3.5).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 5.0, 5.0).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 6.5, 6.5).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Calcite infers return type of AVG(int64) as BIGINT while ZetaSQL requires it as either"
+ + " NUMERIC or DOUBLE/FLOAT64")
+ public void testZetaSQLTestAVG() {
+ String sql = "SELECT Key, AVG(f_int_1)" + "FROM aggregate_test_table GROUP BY Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field2")
+ .addInt64Field("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 6L).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 6L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLGroupByExprInSelect() {
+ String sql = "SELECT int64_col + 1 FROM table_all_types GROUP BY int64_col + 1;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(0L).build(),
+ Row.withSchema(schema).addValue(-1L).build(),
+ Row.withSchema(schema).addValue(-2L).build(),
+ Row.withSchema(schema).addValue(-3L).build(),
+ Row.withSchema(schema).addValue(-4L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLGroupByAndFiltering() {
+ String sql = "SELECT int64_col FROM table_all_types WHERE int64_col = 1 GROUP BY int64_col;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLGroupByAndFilteringOnNonGroupByColumn() {
+ String sql = "SELECT int64_col FROM table_all_types WHERE double_col = 0.5 GROUP BY int64_col;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(-5L).build(),
+ Row.withSchema(schema).addValue(-4L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicHaving() {
+ String sql = "SELECT Key, COUNT(*) FROM aggregate_test_table GROUP BY Key HAVING COUNT(*) > 2";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(2L, 3L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicFixedWindowing() {
+ String sql =
+ "SELECT "
+ + "COUNT(*) as field_count, "
+ + "TUMBLE_START(\"INTERVAL 1 SECOND\") as window_start, "
+ + "TUMBLE_END(\"INTERVAL 1 SECOND\") as window_end "
+ + "FROM KeyValue "
+ + "GROUP BY TUMBLE(ts, \"INTERVAL 1 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_start")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicSlidingWindowing() {
+ String sql =
+ "SELECT "
+ + "COUNT(*) as field_count, "
+ + "HOP_START(\"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\") as window_start, "
+ + "HOP_END(\"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\") as window_end "
+ + "FROM window_test_table "
+ + "GROUP BY HOP(ts, \"INTERVAL 1 SECOND\", \"INTERVAL 2 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_star")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 9, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 5, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 10, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 9, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 11, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLBasicSessionWindowing() {
+ String sql =
+ "SELECT "
+ + "COUNT(*) as field_count, "
+ + "SESSION_START(\"INTERVAL 3 SECOND\") as window_start, "
+ + "SESSION_END(\"INTERVAL 3 SECOND\") as window_end "
+ + "FROM window_test_table_two "
+ + "GROUP BY SESSION(ts, \"INTERVAL 3 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_star")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // Test nested selection
+ @Test
+ public void testZetaSQLNestedQueryOne() {
+ String sql =
+ "SELECT a.Value, a.Key FROM (SELECT Key, Value FROM KeyValue WHERE Key = 14 OR Key = 15)"
+ + " as a;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // Test selection, filtering and aggregation combined query.
+ @Test
+ public void testZetaSQLNestedQueryTwo() {
+ String sql =
+ "SELECT a.Key, a.Key2, COUNT(*) FROM "
+ + " (SELECT * FROM aggregate_test_table WHERE Key != 10) as a "
+ + " GROUP BY a.Key2, a.Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addInt64Field("field3")
+ .addInt64Field("field2")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L, 10L, 1L).build(),
+ Row.withSchema(schema).addValues(1L, 11L, 1L).build(),
+ Row.withSchema(schema).addValues(2L, 11L, 2L).build(),
+ Row.withSchema(schema).addValues(2L, 12L, 1L).build(),
+ Row.withSchema(schema).addValues(3L, 13L, 2L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // test selection and join combined query
+ @Test
+ public void testZetaSQLNestedQueryThree() {
+ String sql =
+ "SELECT * FROM (SELECT * FROM KeyValue) AS t1 INNER JOIN (SELECT * FROM BigTable) AS t2 on"
+ + " t1.Key = t2.RowKey";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addInt64Field("Key")
+ .addStringField("Value")
+ .addDateTimeField("ts")
+ .addInt64Field("RowKey")
+ .addStringField("Value2")
+ .addDateTimeField("ts2")
+ .build())
+ .addValues(
+ 15L,
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ 15L,
+ "BigTable235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testZetaSQLNestedQueryFour() {
+ String sql =
+ "SELECT t1.Value, TUMBLE_START('INTERVAL 1 SECOND') AS period_start, MIN(t2.Value) as"
+ + " min_v FROM KeyValue AS t1 INNER JOIN BigTable AS t2 on t1.Key = t2.RowKey GROUP BY"
+ + " t1.Value, TUMBLE(t2.ts, 'INTERVAL 1 SECOND')";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addStringField("value")
+ .addDateTimeField("min_v")
+ .addStringField("period_start")
+ .build())
+ .addValues(
+ "KeyValue235",
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ "BigTable235")
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ // Test nested select with out of order columns.
+ @Test
+ public void testZetaSQLNestedQueryFive() {
+ String sql =
+ "SELECT a.Value, a.Key FROM (SELECT Value, Key FROM KeyValue WHERE Key = 14 OR Key = 15)"
+ + " as a;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support DATE_ADD and TIME_ADD.")
+ public void testDateAndTimeAddSub() {
+ String sql =
+ "SELECT "
+ + "DATE_ADD(DATE '2008-12-25', INTERVAL 5 DAY), "
+ + "TIME_ADD(TIME '13:24:30', INTERVAL 3 HOUR)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addDateTimeField("f_date_plus")
+ .addDateTimeField("f_time_plus")
+ .build())
+ .addValues(parseDate("2008-12-30"), parseTime("16:24:30"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimestampAddSub() {
+ String sql =
+ "SELECT "
+ + "TIMESTAMP_ADD(TIMESTAMP '2008-12-25 15:30:00 UTC', INTERVAL 10 MINUTE), "
+ + "TIMESTAMP_ADD(TIMESTAMP '2008-12-25 15:30:00+07:30', INTERVAL 10 MINUTE)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(
+ Schema.builder()
+ .addDateTimeField("f_timestamp_plus")
+ .addDateTimeField("f_timestamp_with_time_zone_plus")
+ .build())
+ .addValues(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2008-12-25 15:40:00"),
+ parseTimestampWithTimeZone("2008-12-25 15:40:00+0730"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeZone() {
+ String sql = "SELECT TIMESTAMP '2018-12-10 10:38:59-10:00'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addDateTimeField("f_timestamp_with_time_zone").build())
+ .addValues(parseTimestampWithTimeZone("2018-12-10 10:38:59-1000"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testDistinct() {
+ String sql = "SELECT DISTINCT Key2 FROM aggregate_test_table";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addInt64Field("Key2").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(10L).build(),
+ Row.withSchema(schema).addValues(11L).build(),
+ Row.withSchema(schema).addValues(12L).build(),
+ Row.withSchema(schema).addValues(13L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testDistinctOnNull() {
+ String sql = "SELECT DISTINCT str_val FROM all_null_table";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addNullableField("str_val", FieldType.DOUBLE).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("BeamSQL does not support ANY_VALUE")
+ public void testAnyValue() {
+ String sql = "SELECT ANY_VALUE(double_val) FROM all_null_table";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addNullableField("double_val", FieldType.DOUBLE).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectNULL() {
+ String sql = "SELECT NULL";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema schema = Schema.builder().addNullableField("long_val", FieldType.INT64).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryOne() {
+ String sql =
+ "With T1 AS (SELECT * FROM KeyValue), T2 AS (SELECT * FROM BigTable) SELECT T2.RowKey FROM"
+ + " T1 INNER JOIN T2 on T1.Key = T2.RowKey;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addInt64Field("field1").build())
+ .addValues(15L)
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryTwo() {
+ String sql =
+ "WITH T1 AS (SELECT Key, COUNT(*) as value FROM KeyValue GROUP BY Key) SELECT T1.Key,"
+ + " T1.value FROM T1";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, 1L).build(),
+ Row.withSchema(schema).addValues(15L, 1L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryThree() {
+ String sql =
+ "WITH T1 as (SELECT Value, Key FROM KeyValue WHERE Key = 14 OR Key = 15) SELECT T1.Value,"
+ + " T1.Key FROM T1;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryFour() {
+ String sql =
+ "WITH T1 as (SELECT Value, Key FROM KeyValue) SELECT T1.Value, T1.Key FROM T1 WHERE T1.Key"
+ + " = 14 OR T1.Key = 15;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234", 14L).build(),
+ Row.withSchema(schema).addValues("KeyValue235", 15L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQueryFive() {
+ String sql =
+ "WITH T1 AS (SELECT * FROM KeyValue) SELECT T1.Key, COUNT(*) FROM T1 GROUP BY T1.Key";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addInt64Field("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, 1L).build(),
+ Row.withSchema(schema).addValues(15L, 1L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQuerySix() {
+ String sql =
+ "WITH T1 AS (SELECT * FROM window_test_table_two) SELECT "
+ + "COUNT(*) as field_count, "
+ + "SESSION_START(\"INTERVAL 3 SECOND\") as window_start, "
+ + "SESSION_END(\"INTERVAL 3 SECOND\") as window_end "
+ + "FROM T1 "
+ + "GROUP BY SESSION(ts, \"INTERVAL 3 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_star")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 12, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 2L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testWithQuerySeven() {
+ String sql =
+ "WITH T1 AS (SELECT * FROM KeyValue) SELECT "
+ + "COUNT(*) as field_count, "
+ + "TUMBLE_START(\"INTERVAL 1 SECOND\") as window_start, "
+ + "TUMBLE_END(\"INTERVAL 1 SECOND\") as window_end "
+ + "FROM T1 "
+ + "GROUP BY TUMBLE(ts, \"INTERVAL 1 SECOND\");";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("count_start")
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 8, ISOChronology.getInstanceUTC()))
+ .build(),
+ Row.withSchema(schema)
+ .addValues(
+ 1L,
+ new DateTime(2018, 7, 1, 21, 26, 6, ISOChronology.getInstanceUTC()),
+ new DateTime(2018, 7, 1, 21, 26, 7, ISOChronology.getInstanceUTC()))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testUNNESTLiteral() {
+ String sql = "SELECT * FROM UNNEST(ARRAY<STRING>['foo', 'bar']);";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ Schema schema = Schema.builder().addStringField("str_field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("foo").build(),
+ Row.withSchema(schema).addValues("bar").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testNamedUNNESTLiteral() {
+ String sql = "SELECT *, T1 FROM UNNEST(ARRAY<STRING>['foo', 'bar']) AS T1";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ Schema schema =
+ Schema.builder().addStringField("str_field").addStringField("str2_field").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("foo", "foo").build(),
+ Row.withSchema(schema).addValues("bar", "bar").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Seeing exception in Beam, need further investigation on the cause of this failed query.")
+ public void testNamedUNNESTJoin() {
+ String sql =
+ "SELECT * "
+ + "FROM table_with_array_for_unnest AS t1"
+ + " LEFT JOIN UNNEST(t1.int_array_col) AS t2"
+ + " on "
+ + " t1.int_col = t2";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream).containsInAnyOrder();
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseNoValue() {
+ String sql = "SELECT CASE WHEN 1 > 2 THEN 'not possible' ELSE 'seems right' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValue() {
+ String sql = "SELECT CASE 1 WHEN 2 THEN 'not possible' ELSE 'seems right' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValueMultipleCases() {
+ String sql =
+ "SELECT CASE 2 WHEN 1 THEN 'not possible' WHEN 2 THEN 'seems right' ELSE 'also not"
+ + " possible' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValueNoElse() {
+ String sql = "SELECT CASE 2 WHEN 1 THEN 'not possible' WHEN 2 THEN 'seems right' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addStringField("str_field").build())
+ .addValue("seems right")
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseNoValueNoElseNoMatch() {
+ String sql = "SELECT CASE WHEN 'abc' = '123' THEN 'not possible' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addNullableField("str_field", FieldType.STRING).build())
+ .addValue(null)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCaseWithValueNoElseNoMatch() {
+ String sql = "SELECT CASE 2 WHEN 1 THEN 'not possible' END";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addNullableField("str_field", FieldType.STRING).build())
+ .addValue(null)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Codegen generates code that Janino cannot compile, need further investigation on root"
+ + " cause.")
+ public void testCastToDateWithCase() {
+ String sql =
+ "SELECT f_int, \n"
+ + "CASE WHEN CHAR_LENGTH(TRIM(f_string)) = 8 \n"
+ + " THEN CAST (CONCAT(\n"
+ + " SUBSTR(TRIM(f_string), 0, 4) \n"
+ + " , '-' \n"
+ + " , SUBSTR(TRIM(f_string), 4, 2) \n"
+ + " , '-' \n"
+ + " , SUBSTR(TRIM(f_string), 6, 2)) AS DATE)\n"
+ + " ELSE NULL\n"
+ + "END \n"
+ + "FROM table_for_case_when";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType =
+ Schema.builder().addInt32Field("f_int").addNullableField("f_date", DATETIME).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1, parseDate("2018-10-18")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIntersectAll() {
+ String sql =
+ "SELECT Key FROM aggregate_test_table "
+ + "INTERSECT ALL "
+ + "SELECT Key FROM aggregate_test_table_two";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testIntersectDistinct() {
+ String sql =
+ "SELECT Key FROM aggregate_test_table "
+ + "INTERSECT DISTINCT "
+ + "SELECT Key FROM aggregate_test_table_two";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1L).build(),
+ Row.withSchema(resultType).addValues(2L).build(),
+ Row.withSchema(resultType).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testExceptAll() {
+ String sql =
+ "SELECT Key FROM aggregate_test_table "
+ + "EXCEPT ALL "
+ + "SELECT Key FROM aggregate_test_table_two";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema resultType = Schema.builder().addInt64Field("field").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(resultType).addValues(1L).build(),
+ Row.withSchema(resultType).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectFromEmptyTable() {
+ String sql = "SELECT * FROM table_empty;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ PAssert.that(stream).containsInAnyOrder();
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testStartsWithString() {
+ String sql = "SELECT STARTS_WITH('string1', 'stri')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(true).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testStartsWithString2() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createStringValue(""))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testStartsWithString3() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEndsWithString() {
+ String sql = "SELECT STARTS_WITH('string1', 'ng0')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(false).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEndsWithString2() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createStringValue(""))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testEndsWithString3() {
+ String sql = "SELECT STARTS_WITH(@p0, @p1)";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .put("p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.BOOLEAN).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Boolean) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support DateTime literal.")
+ public void testDateTimeLiteral() {
+ String sql = "SELECT DATETIME '2018-01-01 05:30:00.334'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage("Unsupported ResolvedLiteral type: DATETIME");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testTimeStampLiteral() {
+ String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00+00'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithUTCTimeZone("2016-12-25 05:30:00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeStampLiteralWithoutTimeZone() {
+ String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithUTCTimeZone("2016-12-25 05:30:00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeStampLiteralWithNonUTCTimeZone() {
+ String sql = "SELECT TIMESTAMP '2016-12-25 05:30:00+05'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithTimeZone("2016-12-25 05:30:00+05"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithOneParameters() {
+ String sql = "SELECT concat('abc')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abc").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithTwoParameters() {
+ String sql = "SELECT concat('abc', 'def')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abcdef").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithThreeParameters() {
+ String sql = "SELECT concat('abc', 'def', 'xyz')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abcdefxyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithFourParameters() {
+ String sql = "SELECT concat('abc', 'def', ' ', 'xyz')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithFiveParameters() {
+ String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyzkkk").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore(
+ "Calcite codegen does not support UDF with ... args."
+ + " See:https://jira.apache.org/jira/browse/CALCITE-2889")
+ public void testConcatWithSixParameters() {
+ String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk', 'ttt')";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues("abcdef xyzkkkttt").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithNull1() {
+ String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createStringValue(""),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatWithNull2() {
+ String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1",
+ Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testConcatParameterQuery() {
+ String sql = "SELECT CONCAT(@p0, @p1) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(""), "p1", Value.createStringValue("A"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("A").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace1() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue(""),
+ "p1", Value.createStringValue(""),
+ "p2", Value.createStringValue("a"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace2() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abc"),
+ "p1", Value.createStringValue(""),
+ "p2", Value.createStringValue("xyz"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abc").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace3() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue(""),
+ "p1", Value.createStringValue(""),
+ "p2", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testReplace4() {
+ String sql = "SELECT REPLACE(@p0, @p1, @p2) AS ColA";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p2", Value.createStringValue(""));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTrim1() {
+ String sql = "SELECT trim(@p0)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(" a b c "));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("a b c").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTrim2() {
+ String sql = "SELECT trim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("xyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTrim3() {
+ String sql = "SELECT trim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLTrim1() {
+ String sql = "SELECT ltrim(@p0)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(" a b c "));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("a b c ").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLTrim2() {
+ String sql = "SELECT ltrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("xyzab").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testLTrim3() {
+ String sql = "SELECT ltrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testRTrim1() {
+ String sql = "SELECT rtrim(@p0)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createStringValue(" a b c "));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(" a b c").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testRTrim2() {
+ String sql = "SELECT rtrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abxyzab"), "p1", Value.createStringValue("ab"));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("abxyz").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testRTrim3() {
+ String sql = "SELECT rtrim(@p0, @p1)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING),
+ "p1", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field1", FieldType.STRING).build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((String) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testCastBytesToString1() {
+ String sql = "SELECT CAST(@p0 AS STRING)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createBytesValue(ByteString.copyFromUtf8("`")));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("`").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastBytesToString2() {
+ String sql = "SELECT CAST(b'b' AS STRING)";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("b").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testCastBytesToStringFromTable() {
+ String sql = "SELECT CAST(bytes_col AS STRING) FROM table_all_types";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("1").build(),
+ Row.withSchema(schema).addValues("2").build(),
+ Row.withSchema(schema).addValues("3").build(),
+ Row.withSchema(schema).addValues("4").build(),
+ Row.withSchema(schema).addValues("5").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastStringToTS() {
+ String sql = "SELECT CAST('2019-01-15 13:21:03' AS TIMESTAMP)";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field_1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithUTCTimeZone("2019-01-15 13:21:03"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastStringToString() {
+ String sql = "SELECT CAST(@p0 AS STRING)";
+ ImmutableMap<String, Value> params = ImmutableMap.of("p0", Value.createStringValue(""));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCastStringToInt64() {
+ String sql = "SELECT CAST(@p0 AS INT64)";
+ ImmutableMap<String, Value> params = ImmutableMap.of("p0", Value.createStringValue("123"));
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(123L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectConstant() {
+ String sql = "SELECT 'hi'";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("hi").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support DATE_ADD.")
+ public void testDateAddWithParameter() {
+ String sql =
+ "SELECT "
+ + "DATE_ADD(@p0, INTERVAL @p1 DAY), "
+ + "DATE_ADD(@p2, INTERVAL @p3 DAY), "
+ + "DATE_ADD(@p4, INTERVAL @p5 YEAR), "
+ + "DATE_ADD(@p6, INTERVAL @p7 DAY), "
+ + "DATE_ADD(@p8, INTERVAL @p9 MONTH)";
+ // Value
+ ImmutableMap<String, Value> params =
+ ImmutableMap.<String, Value>builder()
+ .put("p0", Value.createDateValue(0)) // 1970-01-01
+ .put("p1", Value.createInt64Value(2L))
+ .put("p2", parseDateToValue("2019-01-01"))
+ .put("p3", Value.createInt64Value(2L))
+ .put("p4", Value.createSimpleNullValue(TypeKind.TYPE_DATE))
+ .put("p5", Value.createInt64Value(1L))
+ .put("p6", parseDateToValue("2000-02-29"))
+ .put("p7", Value.createInt64Value(-365L))
+ .put("p8", parseDateToValue("1999-03-31"))
+ .put("p9", Value.createInt64Value(-1L))
+ .build();
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addDateTimeField("field1")
+ .addDateTimeField("field2")
+ .addNullableField("field3", DATETIME)
+ .addDateTimeField("field4")
+ .addDateTimeField("field5")
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ parseDate("1970-01-03"),
+ parseDate("2019-01-03"),
+ null,
+ parseDate("1999-03-01"),
+ parseDate("1999-02-28"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Does not support TIME_ADD.")
+ public void testTimeAddWithParameter() {
+ String sql = "SELECT TIME_ADD(@p0, INTERVAL @p1 SECOND)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", parseTimeToValue("12:13:14.123"),
+ "p1", Value.createInt64Value(1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues(parseTime("12:13:15.123")).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("")
+ public void testTimestampAddWithParameter() {
+ String sql = "SELECT TIMESTAMP_ADD(@p0, INTERVAL @p1 MILLISECOND)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", parseTimestampWithTZToValue("2001-01-01 00:00:00+00"),
+ "p1", Value.createInt64Value(1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithTimeZone("2001-01-01 00:00:00.001+00"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testTimeStampAddWithParameter() {
+ String sql = "SELECT TIMESTAMP_ADD(@p0, INTERVAL @p1 MINUTE)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", parseTimestampWithTZToValue("2008-12-25 15:30:00+07:30"),
+ "p1", Value.createInt64Value(10L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addDateTimeField("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(parseTimestampWithTimeZone("2008-12-25 15:40:00+07:30"))
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectFromTableWithMap() {
+ String sql = "SELECT row_field FROM table_with_map";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ Schema rowSchema = Schema.builder().addInt64Field("row_id").addStringField("data").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(Schema.builder().addRowField("row_field", rowSchema).build())
+ .addValues(Row.withSchema(rowSchema).addValues(1L, "data1").build())
+ .build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSubQuery() {
+ String sql = "select sum(Key) from KeyValue\n" + "group by (select Key)";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage("Does not support sub-queries");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testSubstr() {
+ String sql = "SELECT substr(@p0, @p1, @p2)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abc"),
+ "p1", Value.createInt64Value(-2L),
+ "p2", Value.createInt64Value(1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field1").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("b").build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSubstrWithLargeValueExpectException() {
+ String sql = "SELECT substr(@p0, @p1, @p2)";
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of(
+ "p0", Value.createStringValue("abc"),
+ "p1", Value.createInt64Value(Integer.MAX_VALUE + 1L),
+ "p2", Value.createInt64Value(Integer.MIN_VALUE - 1L));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+ thrown.expect(RuntimeException.class);
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectAll() {
+ String sql = "SELECT ALL Key, Value FROM KeyValue;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(14L, "KeyValue234").build(),
+ Row.withSchema(schema).addValues(15L, "KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectDistinct() {
+ String sql = "SELECT DISTINCT Key FROM aggregate_test_table;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(1L).build(),
+ Row.withSchema(schema).addValues(2L).build(),
+ Row.withSchema(schema).addValues(3L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("Bytes cannot be in UNION ALL")
+ public void testSelectDistinct2() {
+ String sql =
+ "SELECT DISTINCT val.BYTES\n"
+ + "from (select b\"BYTES\" BYTES union all\n"
+ + " select b\"bytes\" union all\n"
+ + " select b\"ByTeS\") val";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addByteArrayField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("BYTES".getBytes(StandardCharsets.UTF_8)).build(),
+ Row.withSchema(schema).addValues("ByTeS".getBytes(StandardCharsets.UTF_8)).build(),
+ Row.withSchema(schema).addValues("bytes".getBytes(StandardCharsets.UTF_8)).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectBytes() {
+ String sql = "SELECT b\"ByTes\"";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addByteArrayField("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("ByTes".getBytes(StandardCharsets.UTF_8)).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectExcept() {
+ String sql = "SELECT * EXCEPT (Key, ts) FROM KeyValue;";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues("KeyValue234").build(),
+ Row.withSchema(schema).addValues("KeyValue235").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSelectReplace() {
+ String sql =
+ "WITH orders AS\n"
+ + " (SELECT 5 as order_id,\n"
+ + " \"sprocket\" as item_name,\n"
+ + " 200 as quantity)\n"
+ + "SELECT * REPLACE (\"widget\" AS item_name)\n"
+ + "FROM orders";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addInt64Field("field1")
+ .addStringField("field2")
+ .addInt64Field("field3")
+ .build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues(5L, "widget", 200L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testUnionAllBasic() {
+ String sql =
+ "SELECT row_id FROM table_all_types UNION ALL SELECT row_id FROM table_all_types_2";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field1").build();
+
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValue(1L).build(),
+ Row.withSchema(schema).addValue(2L).build(),
+ Row.withSchema(schema).addValue(3L).build(),
+ Row.withSchema(schema).addValue(4L).build(),
+ Row.withSchema(schema).addValue(5L).build(),
+ Row.withSchema(schema).addValue(6L).build(),
+ Row.withSchema(schema).addValue(7L).build(),
+ Row.withSchema(schema).addValue(8L).build(),
+ Row.withSchema(schema).addValue(9L).build(),
+ Row.withSchema(schema).addValue(10L).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testAVGWithLongInput() {
+ String sql = "SELECT AVG(f_int_1) FROM aggregate_test_table;";
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ thrown.expect(RuntimeException.class);
+ thrown.expectMessage(
+ "AVG(LONG) is not supported. You might want to use AVG(CAST(expression AS DOUBLE).");
+ zetaSQLQueryPlanner.convertToBeamRel(sql);
+ }
+
+ @Test
+ public void testReverseString() {
+ String sql = "SELECT REVERSE('abc');";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addStringField("field2").build();
+
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues("cba").build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCharLength() {
+ String sql = "SELECT CHAR_LENGTH('abc');";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addInt64Field("field").build();
+ PAssert.that(stream).containsInAnyOrder(Row.withSchema(schema).addValues(3L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testCharLengthNull() {
+ String sql = "SELECT CHAR_LENGTH(@p0);";
+
+ ImmutableMap<String, Value> params =
+ ImmutableMap.of("p0", Value.createSimpleNullValue(TypeKind.TYPE_STRING));
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema = Schema.builder().addNullableField("field", FieldType.INT64).build();
+ PAssert.that(stream)
+ .containsInAnyOrder(Row.withSchema(schema).addValues((Object) null).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testExtractTimestampThrowsOnMicrosecondNotSupported() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2000-01-01 00:11:22.345678+00' as timestamp\n"
+ + ")\n"
+ + "SELECT\n"
+ + " timestamp,\n"
+ + " EXTRACT(ISOYEAR FROM timestamp) AS isoyear,\n"
+ + " EXTRACT(YEAR FROM timestamp) AS year,\n"
+ + " EXTRACT(ISOWEEK FROM timestamp) AS week,\n"
+ + " EXTRACT(MINUTE FROM timestamp) AS minute\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ thrown.expect(IllegalArgumentException.class);
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ /** Only sample scenarios are covered here. Excessive testing is done via Compliance tests. */
+ @Test
+ @Ignore("ZetaSQL does not support EnumType to IdentifierLiteral")
+ public void testExtractTimestamp() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2005-01-03 12:34:56' AS timestamp UNION ALL\n"
+ + " SELECT TIMESTAMP '2017-05-26'\n"
+ + ")\n"
+ + "SELECT\n"
+ + " timestamp,\n"
+ + " EXTRACT(ISOYEAR FROM timestamp) AS isoyear,\n"
+ + " EXTRACT(YEAR FROM timestamp) AS year,\n"
+ + " EXTRACT(ISOWEEK FROM timestamp) AS week,\n"
+ + " EXTRACT(MINUTE FROM timestamp) AS minute\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addDateTimeField("ts")
+ .addField("isoyear", FieldType.INT64)
+ .addField("year", FieldType.INT64)
+ .addField("week", FieldType.INT64)
+ .addField("minute", FieldType.INT64)
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema)
+ .addValues(
+ DateTimeUtils.parseTimestampWithUTCTimeZone("2005-01-03 12:34:56"),
+ 2005L,
+ 2005L,
+ 1L,
+ 34L)
+ .build(),
+ Row.withSchema(schema)
+ .addValues(parseDate("2017-05-26"), 2017L, 2017L, 21L, 0L)
+ .build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ @Ignore("ZetaSQL does not support EnumType to IdentifierLiteral")
+ public void testExtractTimestampAtTimeZoneThrowsBecauseNotSupported() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2017-05-26' AS timestamp\n"
+ + ")\n"
+ + "SELECT\n"
+ + " timestamp,\n"
+ + " EXTRACT(HOUR FROM timestamp AT TIME ZONE 'America/Vancouver') AS hour,\n"
+ + " EXTRACT(DAY FROM timestamp AT TIME ZONE 'America/Vancouver') AS day\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ thrown.expect(IllegalArgumentException.class);
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ @Test
+ @Ignore("")
+ public void testExtractDateFromTimestampThrowsBecauseNotSupported() {
+ String sql =
+ "WITH Timestamps AS (\n"
+ + " SELECT TIMESTAMP '2017-05-26' AS ts\n"
+ + ")\n"
+ + "SELECT\n"
+ + " ts,\n"
+ + " EXTRACT(DATE FROM ts) AS dt\n"
+ + "FROM Timestamps\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ thrown.expect(SqlException.class);
+ zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ }
+
+ @Test
+ public void testIsNullTrueFalse() {
+ String sql =
+ "WITH Src AS (\n"
+ + " SELECT NULL as data UNION ALL\n"
+ + " SELECT TRUE UNION ALL\n"
+ + " SELECT FALSE\n"
+ + ")\n"
+ + "SELECT\n"
+ + " data IS NULL as isnull,\n"
+ + " data IS NOT NULL as isnotnull,\n"
+ + " data IS TRUE as istrue,\n"
+ + " data IS NOT TRUE as isnottrue,\n"
+ + " data IS FALSE as isfalse,\n"
+ + " data IS NOT FALSE as isnotfalse\n"
+ + "FROM Src\n";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ ImmutableMap<String, Value> params = ImmutableMap.of();
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql, params);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ final Schema schema =
+ Schema.builder()
+ .addField("isnull", FieldType.BOOLEAN)
+ .addField("isnotnull", FieldType.BOOLEAN)
+ .addField("istrue", FieldType.BOOLEAN)
+ .addField("isnottrue", FieldType.BOOLEAN)
+ .addField("isfalse", FieldType.BOOLEAN)
+ .addField("isnotfalse", FieldType.BOOLEAN)
+ .build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(schema).addValues(true, false, false, true, false, true).build(),
+ Row.withSchema(schema).addValues(false, true, true, false, false, true).build(),
+ Row.withSchema(schema).addValues(false, true, false, true, true, false).build());
+
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
+ public void testSimpleTableName() {
+ String sql = "SELECT Key FROM KeyValue";
+
+ ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+ BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+ PCollection<Row> stream = BeamSqlRelUtils.toPCollection(pipeline, beamRelNode);
+
+ Schema singleField = Schema.builder().addInt64Field("field1").build();
+ PAssert.that(stream)
+ .containsInAnyOrder(
+ Row.withSchema(singleField).addValues(14L).build(),
+ Row.withSchema(singleField).addValues(15L).build());
+ pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ private void initializeCalciteEnvironment() {
+ initializeCalciteEnvironmentWithContext();
+ }
+
+ private void initializeCalciteEnvironmentWithContext(Context... extraContext) {
+ JdbcConnection jdbcConnection =
+ JdbcDriver.connect(tableProvider, PipelineOptionsFactory.create());
+ SchemaPlus defaultSchemaPlus = jdbcConnection.getCurrentSchemaPlus();
+ final ImmutableList<RelTraitDef> traitDefs = ImmutableList.of(ConventionTraitDef.INSTANCE);
+
+ Object[] contexts =
+ ImmutableList.<Context>builder()
+ .add(Contexts.of(jdbcConnection.config()))
+ .add(extraContext)
+ .build()
+ .toArray();
+
+ this.config =
+ Frameworks.newConfigBuilder()
+ .defaultSchema(defaultSchemaPlus)
+ .traitDefs(traitDefs)
+ .context(Contexts.of(contexts))
+ .ruleSets(BeamRuleSets.getRuleSets())
+ .costFactory(null)
+ .typeSystem(jdbcConnection.getTypeFactory().getTypeSystem())
+ .build();
+ }
+
+ private void initializeBeamTableProvider() {
+ Map<String, BeamSqlTable> testBoundedTableMap = new HashMap<>();
+ testBoundedTableMap.put("KeyValue", BASIC_TABLE_ONE);
+ testBoundedTableMap.put("BigTable", BASIC_TABLE_TWO);
+ testBoundedTableMap.put("Spanner", BASIC_TABLE_THREE);
+ testBoundedTableMap.put("aggregate_test_table", AGGREGATE_TABLE_ONE);
+ testBoundedTableMap.put("window_test_table", TIMESTAMP_TABLE_ONE);
+ testBoundedTableMap.put("window_test_table_two", TIMESTAMP_TABLE_TWO);
+ testBoundedTableMap.put("time_test_table", TIME_TABLE);
+ testBoundedTableMap.put("all_null_table", TABLE_ALL_NULL);
+ testBoundedTableMap.put("table_with_struct", TABLE_WITH_STRUCT);
+ testBoundedTableMap.put("table_with_struct_two", TABLE_WITH_STRUCT_TWO);
+ testBoundedTableMap.put("table_with_array", TABLE_WITH_ARRAY);
+ testBoundedTableMap.put("table_with_array_for_unnest", TABLE_WITH_ARRAY_FOR_UNNEST);
+ testBoundedTableMap.put("table_for_case_when", TABLE_FOR_CASE_WHEN);
+ testBoundedTableMap.put("aggregate_test_table_two", AGGREGATE_TABLE_TWO);
+ testBoundedTableMap.put("table_empty", TABLE_EMPTY);
+ testBoundedTableMap.put("table_all_types", TABLE_ALL_TYPES);
+ testBoundedTableMap.put("table_all_types_2", TABLE_ALL_TYPES_2);
+ testBoundedTableMap.put("table_with_map", TABLE_WITH_MAP);
+ testBoundedTableMap.put("table_with_struct_ts_string", TABLE_WITH_STRUCT_TIMESTAMP_STRING);
+
+ tableProvider = new ReadOnlyTableProvider("test_table_provider", testBoundedTableMap);
+ }
+}
diff --git a/sdks/java/extensions/zetasketch/build.gradle b/sdks/java/extensions/zetasketch/build.gradle
new file mode 100644
index 0000000..30e8bc8
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/build.gradle
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.extensions.zetasketch')
+
+description = "Apache Beam :: SDKs :: Java :: Extensions :: ZetaSketch"
+
+evaluationDependsOn(":runners:google-cloud-dataflow-java:worker:legacy-worker")
+
+def zetasketch_version = "0.1.0"
+
+dependencies {
+ compile library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile "com.google.zetasketch:zetasketch:$zetasketch_version"
+ testCompile library.java.junit
+ testCompile project(":sdks:java:io:google-cloud-platform")
+ testRuntimeOnly library.java.slf4j_simple
+ testRuntimeOnly project(":runners:direct-java")
+ testRuntimeOnly project(":runners:google-cloud-dataflow-java")
+}
+
+/**
+ * Integration tests running on Dataflow with BigQuery.
+ */
+task integrationTest(type: Test) {
+ group = "Verification"
+
+ dependsOn ":runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar"
+ def dataflowWorkerJar = project.findProperty('dataflowWorkerJar') ?: project(":runners:google-cloud-dataflow-java:worker:legacy-worker").shadowJar.archivePath
+
+ def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
+ def gcpTempRoot = project.findProperty('gcpTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests'
+
+ // Set workerHarnessContainerImage to empty to make Dataflow pick up the non-versioned container
+ // image, which handles a staged worker jar.
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+ "--runner=TestDataflowRunner",
+ "--project=${gcpProject}",
+ "--tempRoot=${gcpTempRoot}",
+ "--dataflowWorkerJar=${dataflowWorkerJar}",
+ "--workerHarnessContainerImage=",
+ ])
+
+ // Disable Gradle cache: these ITs interact with live service that should always be considered "out of date"
+ outputs.upToDateWhen { false }
+
+ include '**/*IT.class'
+ maxParallelForks 4
+ classpath = sourceSets.test.runtimeClasspath
+ testClassesDirs = sourceSets.test.output.classesDirs
+}
+
+task postCommit {
+ group = "Verification"
+ description = "Integration tests of sketch compatibility between Dataflow and BigQuery."
+ dependsOn integrationTest
+}
diff --git a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java
new file mode 100644
index 0000000..5a975da
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java
@@ -0,0 +1,411 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
+
+import com.google.zetasketch.HyperLogLogPlusPlus;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@code PTransform}s to compute HyperLogLogPlusPlus (HLL++) sketches on data streams based on the
+ * <a href="https://github.com/google/zetasketch">ZetaSketch</a> implementation.
+ *
+ * <p>HLL++ is an algorithm implemented by Google that estimates the count of distinct elements in a
+ * data stream. HLL++ requires significantly less memory than the linear memory needed for exact
+ * computation, at the cost of a small error. Cardinalities of arbitrary breakdowns can be computed
+ * using the HLL++ sketch. See this <a
+ * href="http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/40671.pdf">published
+ * paper</a> for details about the algorithm.
+ *
+ * <p>HLL++ functions are also supported in <a
+ * href="https://cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions">Google Cloud
+ * BigQuery</a>. The {@code HllCount PTransform}s provided here produce and consume sketches
+ * compatible with BigQuery.
+ *
+ * <p>For detailed design of this class, see https://s.apache.org/hll-in-beam.
+ *
+ * <h3>Examples</h3>
+ *
+ * <h4>Example 1: Create long-type sketch for a {@code PCollection<Long>} and specify precision</h4>
+ *
+ * <pre>{@code
+ * PCollection<Long> input = ...;
+ * int p = ...;
+ * PCollection<byte[]> sketch = input.apply(HllCount.Init.forLongs().withPrecision(p).globally());
+ * }</pre>
+ *
+ * <h4>Example 2: Create bytes-type sketch for a {@code PCollection<KV<String, byte[]>>}</h4>
+ *
+ * <pre>{@code
+ * PCollection<KV<String, byte[]>> input = ...;
+ * PCollection<KV<String, byte[]>> sketch = input.apply(HllCount.Init.forBytes().perKey());
+ * }</pre>
+ *
+ * <h4>Example 3: Merge existing sketches in a {@code PCollection<byte[]>} into a new one</h4>
+ *
+ * <pre>{@code
+ * PCollection<byte[]> sketches = ...;
+ * PCollection<byte[]> mergedSketch = sketches.apply(HllCount.MergePartial.globally());
+ * }</pre>
+ *
+ * <h4>Example 4: Estimates the count of distinct elements in a {@code PCollection<String>}</h4>
+ *
+ * <pre>{@code
+ * PCollection<String> input = ...;
+ * PCollection<Long> countDistinct =
+ * input.apply(HllCount.Init.forStrings().globally()).apply(HllCount.Extract.globally());
+ * }</pre>
+ *
+ * Note: Currently HllCount does not work on FnAPI workers. See <a
+ * href="https://issues.apache.org/jira/browse/BEAM-7879">Jira ticket [BEAM-7879]</a>.
+ */
+@Experimental
+public final class HllCount {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HllCount.class);
+
+ /**
+ * The minimum {@code precision} value you can set in {@link Init.Builder#withPrecision(int)} is
+ * {@value}.
+ */
+ public static final int MINIMUM_PRECISION = HyperLogLogPlusPlus.MINIMUM_PRECISION;
+
+ /**
+ * The maximum {@code precision} value you can set in {@link Init.Builder#withPrecision(int)} is
+ * {@value}.
+ */
+ public static final int MAXIMUM_PRECISION = HyperLogLogPlusPlus.MAXIMUM_PRECISION;
+
+ /**
+ * The default {@code precision} value used in {@link Init.Builder#withPrecision(int)} is
+ * {@value}.
+ */
+ public static final int DEFAULT_PRECISION = HyperLogLogPlusPlus.DEFAULT_NORMAL_PRECISION;
+
+ // Cannot be instantiated. This class is intended to be a namespace only.
+ private HllCount() {}
+
+ /**
+ * Provides {@code PTransform}s to aggregate inputs into HLL++ sketches. The four supported input
+ * types are {@code Integer}, {@code Long}, {@code String}, and {@code byte[]}.
+ *
+ * <p>Sketches are represented using the {@code byte[]} type. Sketches of the same type can be
+ * merged into a new sketch using {@link HllCount.MergePartial}. Estimated count of distinct
+ * elements can be extracted from sketches using {@link HllCount.Extract}.
+ *
+ * <p>An "empty sketch" represented by an byte array of length 0 is returned if the input {@code
+ * PCollection} is empty.
+ *
+ * <p>Corresponds to the {@code HLL_COUNT.INIT(input [, precision])} function in <a
+ * href="https://cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions">BigQuery</a>.
+ */
+ public static final class Init {
+
+ // Cannot be instantiated. This class is intended to be a namespace only.
+ private Init() {}
+
+ /**
+ * Returns a {@link Builder} for a {@code HllCount.Init} combining {@code PTransform} that
+ * computes integer-type HLL++ sketches. Call {@link Builder#globally()} or {@link
+ * Builder#perKey()} on the returning {@link Builder} to finalize the {@code PTransform}.
+ *
+ * <p>Calling {@link Builder#globally()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<Integer>} and returns a {@code PCollection<byte[]>} which consists of the
+ * integer-type HLL++ sketch computed from the elements in the input {@code PCollection}.
+ *
+ * <p>Calling {@link Builder#perKey()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<KV<K, Integer>>} and returns a {@code PCollection<KV<K, byte[]>>} which consists
+ * of the per-key integer-type HLL++ sketch computed from the values matching each key in the
+ * input {@code PCollection}.
+ *
+ * <p>Integer-type sketches cannot be merged with sketches of other types.
+ */
+ public static Builder<Integer> forIntegers() {
+ return new Builder<>(HllCountInitFn.forInteger());
+ }
+
+ /**
+ * Returns a {@link Builder} for a {@code HllCount.Init} combining {@code PTransform} that
+ * computes long-type HLL++ sketches. Call {@link Builder#globally()} or {@link
+ * Builder#perKey()} on the returning {@link Builder} to finalize the {@code PTransform}.
+ *
+ * <p>Calling {@link Builder#globally()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<Long>} and returns a {@code PCollection<byte[]>} which consists of the long-type
+ * HLL++ sketch computed from the elements in the input {@code PCollection}.
+ *
+ * <p>Calling {@link Builder#perKey()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<KV<K, Long>>} and returns a {@code PCollection<KV<K, byte[]>>} which consists of
+ * the per-key long-type HLL++ sketch computed from the values matching each key in the input
+ * {@code PCollection}.
+ *
+ * <p>Long-type sketches cannot be merged with sketches of other types.
+ */
+ public static Builder<Long> forLongs() {
+ return new Builder<>(HllCountInitFn.forLong());
+ }
+
+ /**
+ * Returns a {@link Builder} for a {@code HllCount.Init} combining {@code PTransform} that
+ * computes string-type HLL++ sketches. Call {@link Builder#globally()} or {@link
+ * Builder#perKey()} on the returning {@link Builder} to finalize the {@code PTransform}.
+ *
+ * <p>Calling {@link Builder#globally()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<String>} and returns a {@code PCollection<byte[]>} which consists of the
+ * string-type HLL++ sketch computed from the elements in the input {@code PCollection}.
+ *
+ * <p>Calling {@link Builder#perKey()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<KV<K, String>>} and returns a {@code PCollection<KV<K, byte[]>>} which consists
+ * of the per-key string-type HLL++ sketch computed from the values matching each key in the
+ * input {@code PCollection}.
+ *
+ * <p>String-type sketches cannot be merged with sketches of other types.
+ */
+ public static Builder<String> forStrings() {
+ return new Builder<>(HllCountInitFn.forString());
+ }
+
+ /**
+ * Returns a {@link Builder} for a {@code HllCount.Init} combining {@code PTransform} that
+ * computes bytes-type HLL++ sketches. Call {@link Builder#globally()} or {@link
+ * Builder#perKey()} on the returning {@link Builder} to finalize the {@code PTransform}.
+ *
+ * <p>Calling {@link Builder#globally()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<byte[]>} and returns a {@code PCollection<byte[]>} which consists of the
+ * bytes-type HLL++ sketch computed from the elements in the input {@code PCollection}.
+ *
+ * <p>Calling {@link Builder#perKey()} returns a {@code PTransform} that takes an input {@code
+ * PCollection<KV<K, byte[]>>} and returns a {@code PCollection<KV<K, byte[]>>} which consists
+ * of the per-key bytes-type HLL++ sketch computed from the values matching each key in the
+ * input {@code PCollection}.
+ *
+ * <p>Bytes-type sketches cannot be merged with sketches of other types.
+ */
+ public static Builder<byte[]> forBytes() {
+ return new Builder<>(HllCountInitFn.forBytes());
+ }
+
+ /**
+ * Builder for the {@code HllCount.Init} combining {@code PTransform}.
+ *
+ * <p>Call {@link #withPrecision(int)} to customize the {@code precision} parameter of the
+ * sketch.
+ *
+ * <p>Call {@link #globally()} or {@link #perKey()} to finalize the {@code PTransform}.
+ *
+ * @param <InputT> element type or value type in {@code KV}s of the input {@code PCollection} to
+ * the {@code PTransform} being built
+ */
+ public static final class Builder<InputT> {
+
+ private final HllCountInitFn<InputT, ?> initFn;
+
+ private Builder(HllCountInitFn<InputT, ?> initFn) {
+ this.initFn = initFn;
+ }
+
+ /**
+ * Explicitly set the {@code precision} parameter used to compute HLL++ sketch.
+ *
+ * <p>Valid range is between {@link #MINIMUM_PRECISION} and {@link #MAXIMUM_PRECISION}. If
+ * this method is not called, {@link #DEFAULT_PRECISION} will be used. Sketches computed using
+ * different {@code precision}s cannot be merged together.
+ *
+ * @param precision the {@code precision} parameter used to compute HLL++ sketch
+ */
+ public Builder<InputT> withPrecision(int precision) {
+ initFn.setPrecision(precision);
+ return this;
+ }
+
+ /**
+ * Returns a {@link Combine.Globally} {@code PTransform} that takes an input {@code
+ * PCollection<InputT>} and returns a {@code PCollection<byte[]>} which consists of the HLL++
+ * sketch computed from the elements in the input {@code PCollection}.
+ *
+ * <p>Returns a singleton {@code PCollection} with an "empty sketch" (byte array of length 0)
+ * if the input {@code PCollection} is empty.
+ */
+ public Combine.Globally<InputT, byte[]> globally() {
+ return Combine.globally(initFn);
+ }
+
+ /**
+ * Returns a {@link Combine.PerKey} {@code PTransform} that takes an input {@code
+ * PCollection<KV<K, InputT>>} and returns a {@code PCollection<KV<K, byte[]>>} which consists
+ * of the per-key HLL++ sketch computed from the values matching each key in the input {@code
+ * PCollection}.
+ */
+ public <K> Combine.PerKey<K, InputT, byte[]> perKey() {
+ return Combine.perKey(initFn);
+ }
+ }
+ }
+
+ /**
+ * Provides {@code PTransform}s to merge HLL++ sketches into a new sketch.
+ *
+ * <p>Only sketches of the same type can be merged together. If incompatible sketches are
+ * provided, a runtime error will occur.
+ *
+ * <p>If sketches of different {@code precision}s are merged, the merged sketch will get the
+ * minimum precision encountered among all the input sketches.
+ *
+ * <p>An "empty sketch" represented by an byte array of length 0 is returned if the input {@code
+ * PCollection} is empty.
+ *
+ * <p>Corresponds to the {@code HLL_COUNT.MERGE_PARTIAL(sketch)} function in <a
+ * href="https://cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions">BigQuery</a>.
+ */
+ public static final class MergePartial {
+
+ // Cannot be instantiated. This class is intended to be a namespace only.
+ private MergePartial() {}
+
+ /**
+ * Returns a {@link Combine.Globally} {@code PTransform} that takes an input {@code
+ * PCollection<byte[]>} of HLL++ sketches and returns a {@code PCollection<byte[]>} of a new
+ * sketch merged from the input sketches.
+ *
+ * <p>Only sketches of the same type can be merged together. If incompatible sketches are
+ * provided, a runtime error will occur.
+ *
+ * <p>If sketches of different {@code precision}s are merged, the merged sketch will get the
+ * minimum precision encountered among all the input sketches.
+ *
+ * <p>Returns a singleton {@code PCollection} with an "empty sketch" (byte array of length 0) if
+ * the input {@code PCollection} is empty.
+ */
+ public static Combine.Globally<byte[], byte[]> globally() {
+ return Combine.globally(HllCountMergePartialFn.create());
+ }
+
+ /**
+ * Returns a {@link Combine.PerKey} {@code PTransform} that takes an input {@code
+ * PCollection<KV<K, byte[]>>} of (key, HLL++ sketch) pairs and returns a {@code
+ * PCollection<KV<K, byte[]>>} of (key, new sketch merged from the input sketches under the
+ * key).
+ *
+ * <p>If sketches of different {@code precision}s are merged, the merged sketch will get the
+ * minimum precision encountered among all the input sketches.
+ *
+ * <p>Only sketches of the same type can be merged together. If incompatible sketches are
+ * provided, a runtime error will occur.
+ */
+ public static <K> Combine.PerKey<K, byte[], byte[]> perKey() {
+ return Combine.perKey(HllCountMergePartialFn.create());
+ }
+ }
+
+ /**
+ * Provides {@code PTransform}s to extract the estimated count of distinct elements (as {@code
+ * Long}s) from each HLL++ sketch.
+ *
+ * <p>When extracting from an "empty sketch" represented by an byte array of length 0, the result
+ * returned is 0.
+ *
+ * <p>Corresponds to the {@code HLL_COUNT.EXTRACT(sketch)} function in <a
+ * href="https://cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions">BigQuery</a>.
+ */
+ public static final class Extract {
+
+ // Cannot be instantiated. This class is intended to be a namespace only.
+ private Extract() {}
+
+ /**
+ * Returns a {@code PTransform} that takes an input {@code PCollection<byte[]>} of HLL++
+ * sketches and returns a {@code PCollection<Long>} of the estimated count of distinct elements
+ * extracted from each sketch.
+ *
+ * <p>Returns 0 if the input element is an "empty sketch" (byte array of length 0).
+ */
+ public static PTransform<PCollection<byte[]>, PCollection<Long>> globally() {
+ return new Globally();
+ }
+
+ /**
+ * Returns a {@code PTransform} that takes an input {@code PCollection<KV<K, byte[]>>} of (key,
+ * HLL++ sketch) pairs and returns a {@code PCollection<KV<K, Long>>} of (key, estimated count
+ * of distinct elements extracted from each sketch).
+ */
+ public static <K> PTransform<PCollection<KV<K, byte[]>>, PCollection<KV<K, Long>>> perKey() {
+ return new PerKey<K>();
+ }
+
+ private static final class Globally extends PTransform<PCollection<byte[]>, PCollection<Long>> {
+
+ @Override
+ public PCollection<Long> expand(PCollection<byte[]> input) {
+ return input.apply(
+ ParDo.of(
+ new DoFn<byte[], Long>() {
+ @ProcessElement
+ public void processElement(
+ @Element byte[] sketch, OutputReceiver<Long> receiver) {
+ if (sketch == null) {
+ LOG.warn(
+ "Received a null and treated it as an empty sketch. "
+ + "Consider replacing nulls with empty byte arrays (byte[0]) "
+ + "in upstream transforms for better space-efficiency and safety.");
+ receiver.output(0L);
+ } else if (sketch.length == 0) {
+ receiver.output(0L);
+ } else {
+ receiver.output(HyperLogLogPlusPlus.forProto(sketch).result());
+ }
+ }
+ }));
+ }
+ }
+
+ private static final class PerKey<K>
+ extends PTransform<PCollection<KV<K, byte[]>>, PCollection<KV<K, Long>>> {
+
+ @Override
+ public PCollection<KV<K, Long>> expand(PCollection<KV<K, byte[]>> input) {
+ return input.apply(
+ ParDo.of(
+ new DoFn<KV<K, byte[]>, KV<K, Long>>() {
+ @ProcessElement
+ public void processElement(
+ @Element KV<K, byte[]> kv, OutputReceiver<KV<K, Long>> receiver) {
+ byte[] sketch = kv.getValue();
+ if (sketch == null) {
+ LOG.warn(
+ "Received a null and treated it as an empty sketch. "
+ + "Consider replacing nulls with empty byte arrays (byte[0]) "
+ + "in upstream transforms for better space-efficiency and safety.");
+ receiver.output(KV.of(kv.getKey(), 0L));
+ } else if (sketch.length == 0) {
+ receiver.output(KV.of(kv.getKey(), 0L));
+ } else {
+ receiver.output(
+ KV.of(kv.getKey(), HyperLogLogPlusPlus.forProto(sketch).result()));
+ }
+ }
+ }));
+ }
+ }
+ }
+}
diff --git a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountInitFn.java b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountInitFn.java
new file mode 100644
index 0000000..ebcd665
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountInitFn.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.zetasketch.HyperLogLogPlusPlus;
+import com.google.zetasketch.shaded.com.google.protobuf.ByteString;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.transforms.Combine;
+
+/**
+ * {@link Combine.CombineFn} for the {@link HllCount.Init} combiner.
+ *
+ * @param <InputT> type of input values to the function (Integer, Long, String, or byte[])
+ * @param <HllT> type of the HLL++ sketch to compute (Integer, Long, String, or ByteString)
+ */
+abstract class HllCountInitFn<InputT, HllT>
+ extends Combine.CombineFn<InputT, HyperLogLogPlusPlus<HllT>, byte[]> {
+
+ // Ideally, this would be a final field set at construction time via the builder. However, that
+ // not only requires adding an extra type enum to HllCount.Init.Builder to cache the type
+ // information, but also makes it hard to make the Builder generic with input type T (requires
+ // lots of type casting when constructing the transform).
+ private int precision = HllCount.DEFAULT_PRECISION;
+
+ int getPrecision() {
+ return precision;
+ }
+
+ // This function is only intended to be called from within a builder of HllCount.Init during
+ // pipeline construction time. Calling it at pipeline execution time has no effect, and the
+ // updates will be discarded.
+ void setPrecision(int precision) {
+ checkArgument(
+ precision >= HllCount.MINIMUM_PRECISION && precision <= HllCount.MAXIMUM_PRECISION,
+ "Invalid precision: %s. Valid range is [%s, %s].",
+ precision,
+ HllCount.MINIMUM_PRECISION,
+ HllCount.MAXIMUM_PRECISION);
+ this.precision = precision;
+ }
+
+ // The result of an empty aggregation is represented by an byte[] of length 0, because we cannot
+ // create sketches without knowing the type of its input data and because it's more compact.
+ // An empty byte[] can be encoded by the default ByteArrayCoder, and is more space-efficient and
+ // safer than using null.
+ // As opposed to returning an empty PCollection, it allows us to return 0 when extracting from the
+ // sketch.
+ @Override
+ public byte[] defaultValue() {
+ return new byte[0];
+ }
+
+ @Override
+ public Coder<HyperLogLogPlusPlus<HllT>> getAccumulatorCoder(
+ CoderRegistry registry, Coder<InputT> inputCoder) {
+ return HyperLogLogPlusPlusCoder.of();
+ }
+
+ @Override
+ public HyperLogLogPlusPlus<HllT> mergeAccumulators(
+ Iterable<HyperLogLogPlusPlus<HllT>> accumulators) {
+ HyperLogLogPlusPlus<HllT> merged = createAccumulator();
+ for (HyperLogLogPlusPlus<HllT> accumulator : accumulators) {
+ merged.merge(accumulator);
+ }
+ return merged;
+ }
+
+ @Override
+ public byte[] extractOutput(HyperLogLogPlusPlus<HllT> accumulator) {
+ return accumulator.serializeToByteArray();
+ }
+
+ static HllCountInitFn<Integer, Integer> forInteger() {
+ return new ForInteger();
+ }
+
+ static HllCountInitFn<Long, Long> forLong() {
+ return new ForLong();
+ }
+
+ static HllCountInitFn<String, String> forString() {
+ return new ForString();
+ }
+
+ static HllCountInitFn<byte[], ByteString> forBytes() {
+ return new ForBytes();
+ }
+
+ private static class ForInteger extends HllCountInitFn<Integer, Integer> {
+
+ @Override
+ public HyperLogLogPlusPlus<Integer> createAccumulator() {
+ return new HyperLogLogPlusPlus.Builder().normalPrecision(getPrecision()).buildForIntegers();
+ }
+
+ @Override
+ public HyperLogLogPlusPlus<Integer> addInput(
+ HyperLogLogPlusPlus<Integer> accumulator, Integer input) {
+ accumulator.add(input.intValue());
+ return accumulator;
+ }
+ }
+
+ private static class ForLong extends HllCountInitFn<Long, Long> {
+
+ @Override
+ public HyperLogLogPlusPlus<Long> createAccumulator() {
+ return new HyperLogLogPlusPlus.Builder().normalPrecision(getPrecision()).buildForLongs();
+ }
+
+ @Override
+ public HyperLogLogPlusPlus<Long> addInput(HyperLogLogPlusPlus<Long> accumulator, Long input) {
+ accumulator.add(input.longValue());
+ return accumulator;
+ }
+ }
+
+ private static class ForString extends HllCountInitFn<String, String> {
+
+ @Override
+ public HyperLogLogPlusPlus<String> createAccumulator() {
+ return new HyperLogLogPlusPlus.Builder().normalPrecision(getPrecision()).buildForStrings();
+ }
+
+ @Override
+ public HyperLogLogPlusPlus<String> addInput(
+ HyperLogLogPlusPlus<String> accumulator, String input) {
+ accumulator.add(input);
+ return accumulator;
+ }
+ }
+
+ private static class ForBytes extends HllCountInitFn<byte[], ByteString> {
+
+ @Override
+ public HyperLogLogPlusPlus<ByteString> createAccumulator() {
+ return new HyperLogLogPlusPlus.Builder().normalPrecision(getPrecision()).buildForBytes();
+ }
+
+ @Override
+ public HyperLogLogPlusPlus<ByteString> addInput(
+ HyperLogLogPlusPlus<ByteString> accumulator, byte[] input) {
+ accumulator.add(input);
+ return accumulator;
+ }
+ }
+}
diff --git a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountMergePartialFn.java b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountMergePartialFn.java
new file mode 100644
index 0000000..0a07160
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountMergePartialFn.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
+
+import com.google.zetasketch.HyperLogLogPlusPlus;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link Combine.CombineFn} for the {@link HllCount.MergePartial} combiner.
+ *
+ * @param <HllT> type of the HLL++ sketch to be merged
+ */
+class HllCountMergePartialFn<HllT>
+ extends Combine.CombineFn<byte[], HyperLogLogPlusPlus<HllT>, byte[]> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HllCountMergePartialFn.class);
+
+ // Call HllCountMergePartialFn.create() to instantiate
+ private HllCountMergePartialFn() {}
+
+ static HllCountMergePartialFn<?> create() {
+ return new HllCountMergePartialFn();
+ }
+
+ @Override
+ public Coder<HyperLogLogPlusPlus<HllT>> getAccumulatorCoder(
+ CoderRegistry registry, Coder<byte[]> inputCoder) {
+ // Use null to represent the "identity element" of the merge operation.
+ return NullableCoder.of(HyperLogLogPlusPlusCoder.of());
+ }
+
+ @Nullable
+ @Override
+ public HyperLogLogPlusPlus<HllT> createAccumulator() {
+ // Cannot create a sketch corresponding to an empty data set, because we do not know the sketch
+ // type and precision. So use null to represent the "identity element" of the merge operation.
+ return null;
+ }
+
+ @Nullable
+ @Override
+ public HyperLogLogPlusPlus<HllT> addInput(
+ @Nullable HyperLogLogPlusPlus<HllT> accumulator, byte[] input) {
+ if (input == null) {
+ LOG.warn(
+ "Received a null and treated it as an empty sketch. "
+ + "Consider replacing nulls with empty byte arrays (byte[0]) "
+ + "in upstream transforms for better space-efficiency and safety.");
+ return accumulator;
+ } else if (input.length == 0) {
+ return accumulator;
+ } else if (accumulator == null) {
+ @SuppressWarnings("unchecked")
+ HyperLogLogPlusPlus<HllT> result =
+ (HyperLogLogPlusPlus<HllT>) HyperLogLogPlusPlus.forProto(input);
+ return result;
+ } else {
+ accumulator.merge(input);
+ return accumulator;
+ }
+ }
+
+ @Nullable
+ @Override
+ public HyperLogLogPlusPlus<HllT> mergeAccumulators(
+ Iterable<HyperLogLogPlusPlus<HllT>> accumulators) {
+ HyperLogLogPlusPlus<HllT> merged = createAccumulator();
+ for (HyperLogLogPlusPlus<HllT> accumulator : accumulators) {
+ if (accumulator == null) {
+ continue;
+ }
+ if (merged == null) {
+ @SuppressWarnings("unchecked")
+ HyperLogLogPlusPlus<HllT> clonedAccumulator =
+ (HyperLogLogPlusPlus<HllT>)
+ HyperLogLogPlusPlus.forProto(accumulator.serializeToProto());
+ // Cannot set merged to accumulator directly because we shouldn't mutate accumulator
+ merged = clonedAccumulator;
+ } else {
+ merged.merge(accumulator);
+ }
+ }
+ return merged;
+ }
+
+ @Override
+ public byte[] extractOutput(@Nullable HyperLogLogPlusPlus<HllT> accumulator) {
+ if (accumulator == null) {
+ return new byte[0];
+ } else {
+ return accumulator.serializeToByteArray();
+ }
+ }
+}
diff --git a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HyperLogLogPlusPlusCoder.java b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HyperLogLogPlusPlusCoder.java
new file mode 100644
index 0000000..ad1a897
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HyperLogLogPlusPlusCoder.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.zetasketch.HyperLogLogPlusPlus;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+
+/** Coder for the {@link HyperLogLogPlusPlus} class with generic type parameter {@code T}. */
+class HyperLogLogPlusPlusCoder<T> extends AtomicCoder<HyperLogLogPlusPlus<T>> {
+
+ private static final HyperLogLogPlusPlusCoder<?> INSTANCE = new HyperLogLogPlusPlusCoder<>();
+
+ private static final Coder<byte[]> BYTE_ARRAY_CODER = ByteArrayCoder.of();
+
+ // Generic singleton factory pattern; the coder works for all HyperLogLogPlusPlus objects at
+ // runtime regardless of type T
+ @SuppressWarnings("unchecked")
+ static <T> HyperLogLogPlusPlusCoder<T> of() {
+ return (HyperLogLogPlusPlusCoder<T>) INSTANCE;
+ }
+
+ @Override
+ public void encode(HyperLogLogPlusPlus<T> value, OutputStream outStream) throws IOException {
+ checkNotNull(value, new CoderException("Cannot encode a null HyperLogLogPlusPlus value."));
+ BYTE_ARRAY_CODER.encode(value.serializeToByteArray(), outStream);
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public HyperLogLogPlusPlus<T> decode(InputStream inStream) throws IOException {
+ return (HyperLogLogPlusPlus<T>) HyperLogLogPlusPlus.forProto(BYTE_ARRAY_CODER.decode(inStream));
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {
+ BYTE_ARRAY_CODER.verifyDeterministic();
+ }
+
+ // TODO: check if we can know the sketch size without serializing it
+ // If so, isRegisterByteSizeObserverCheap() and registerByteSizeObserver() can be overridden
+}
diff --git a/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/package-info.java b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/package-info.java
new file mode 100644
index 0000000..e0cd4e0
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/package-info.java
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * {@code PTransform}s to compute statistical sketches on data streams based on the <a
+ * href="https://github.com/google/zetasketch">ZetaSketch</a> implementation.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
diff --git a/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java
new file mode 100644
index 0000000..3f7927d
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
+
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableReference;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
+import org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
+import org.apache.beam.sdk.options.ApplicationNameOptions;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Integration tests for HLL++ sketch compatibility between Beam and BigQuery. The tests verifies
+ * that HLL++ sketches created in Beam can be processed by BigQuery, and vice versa.
+ */
+@RunWith(JUnit4.class)
+public class BigQueryHllSketchCompatibilityIT {
+
+ private static final String APP_NAME;
+ private static final String PROJECT_ID;
+ private static final String DATASET_ID;
+ private static final BigqueryClient BIGQUERY_CLIENT;
+
+ private static final List<String> TEST_DATA =
+ Arrays.asList("Apple", "Orange", "Banana", "Orange");
+
+ // Data Table: used by testReadSketchFromBigQuery())
+ // Schema: only one STRING field named "data".
+ // Content: prepopulated with 4 rows: "Apple", "Orange", "Banana", "Orange"
+ private static final String DATA_TABLE_ID = "hll_data";
+ private static final String DATA_FIELD_NAME = "data";
+ private static final String DATA_FIELD_TYPE = "STRING";
+ private static final String QUERY_RESULT_FIELD_NAME = "sketch";
+ private static final Long EXPECTED_COUNT = 3L;
+
+ // Sketch Table: used by testWriteSketchToBigQuery()
+ // Schema: only one BYTES field named "sketch".
+ // Content: will be overridden by the sketch computed by the test pipeline each time the test runs
+ private static final String SKETCH_TABLE_ID = "hll_sketch";
+ private static final String SKETCH_FIELD_NAME = "sketch";
+ private static final String SKETCH_FIELD_TYPE = "BYTES";
+ // SHA-1 hash of string "[3]", the string representation of a row that has only one field 3 in it
+ private static final String EXPECTED_CHECKSUM = "f1e31df9806ce94c5bdbbfff9608324930f4d3f1";
+
+ static {
+ ApplicationNameOptions options =
+ TestPipeline.testingPipelineOptions().as(ApplicationNameOptions.class);
+ APP_NAME = options.getAppName();
+ PROJECT_ID = options.as(GcpOptions.class).getProject();
+ DATASET_ID = String.format("zetasketch_%tY_%<tm_%<td_%<tH_%<tM_%<tS_%<tL", new Date());
+ BIGQUERY_CLIENT = BigqueryClient.getClient(APP_NAME);
+ }
+
+ @BeforeClass
+ public static void prepareDatasetAndDataTable() throws Exception {
+ BIGQUERY_CLIENT.createNewDataset(PROJECT_ID, DATASET_ID);
+
+ // Create Data Table
+ TableSchema dataTableSchema =
+ new TableSchema()
+ .setFields(
+ Collections.singletonList(
+ new TableFieldSchema().setName(DATA_FIELD_NAME).setType(DATA_FIELD_TYPE)));
+ Table dataTable =
+ new Table()
+ .setSchema(dataTableSchema)
+ .setTableReference(
+ new TableReference()
+ .setProjectId(PROJECT_ID)
+ .setDatasetId(DATASET_ID)
+ .setTableId(DATA_TABLE_ID));
+ BIGQUERY_CLIENT.createNewTable(PROJECT_ID, DATASET_ID, dataTable);
+
+ // Prepopulate test data to Data Table
+ List<Map<String, Object>> rows =
+ TEST_DATA.stream()
+ .map(v -> Collections.singletonMap(DATA_FIELD_NAME, (Object) v))
+ .collect(Collectors.toList());
+ BIGQUERY_CLIENT.insertDataToTable(PROJECT_ID, DATASET_ID, DATA_TABLE_ID, rows);
+ }
+
+ @AfterClass
+ public static void deleteDataset() throws Exception {
+ BIGQUERY_CLIENT.deleteDataset(PROJECT_ID, DATASET_ID);
+ }
+
+ /**
+ * Test that HLL++ sketch computed in BigQuery can be processed by Beam. Hll sketch is computed by
+ * {@code HLL_COUNT.INIT} in BigQuery and read into Beam; the test verifies that we can run {@link
+ * HllCount.MergePartial} and {@link HllCount.Extract} on the sketch in Beam to get the correct
+ * estimated count.
+ */
+ @Test
+ public void testReadSketchFromBigQuery() {
+ String tableSpec = String.format("%s.%s", DATASET_ID, DATA_TABLE_ID);
+ String query =
+ String.format(
+ "SELECT HLL_COUNT.INIT(%s) AS %s FROM %s",
+ DATA_FIELD_NAME, QUERY_RESULT_FIELD_NAME, tableSpec);
+ SerializableFunction<SchemaAndRecord, byte[]> parseQueryResultToByteArray =
+ (SchemaAndRecord schemaAndRecord) ->
+ // BigQuery BYTES type corresponds to Java java.nio.ByteBuffer type
+ ((ByteBuffer) schemaAndRecord.getRecord().get(QUERY_RESULT_FIELD_NAME)).array();
+
+ TestPipelineOptions options =
+ TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);
+
+ Pipeline p = Pipeline.create(options);
+ PCollection<Long> result =
+ p.apply(
+ BigQueryIO.read(parseQueryResultToByteArray)
+ .fromQuery(query)
+ .usingStandardSql()
+ .withMethod(Method.DIRECT_READ)
+ .withCoder(ByteArrayCoder.of()))
+ .apply(HllCount.MergePartial.globally()) // no-op, only for testing MergePartial
+ .apply(HllCount.Extract.globally());
+ PAssert.thatSingleton(result).isEqualTo(EXPECTED_COUNT);
+ p.run().waitUntilFinish();
+ }
+
+ /**
+ * Test that HLL++ sketch computed in Beam can be processed by BigQuery. Hll sketch is computed by
+ * {@link HllCount.Init} in Beam and written to BigQuery; the test verifies that we can run {@code
+ * HLL_COUNT.EXTRACT()} on the sketch in BigQuery to get the correct estimated count.
+ */
+ @Test
+ public void testWriteSketchToBigQuery() {
+ String tableSpec = String.format("%s.%s", DATASET_ID, SKETCH_TABLE_ID);
+ String query =
+ String.format("SELECT HLL_COUNT.EXTRACT(%s) FROM %s", SKETCH_FIELD_NAME, tableSpec);
+ TableSchema tableSchema =
+ new TableSchema()
+ .setFields(
+ Collections.singletonList(
+ new TableFieldSchema().setName(SKETCH_FIELD_NAME).setType(SKETCH_FIELD_TYPE)));
+
+ TestPipelineOptions options =
+ TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);
+ // After the pipeline finishes, BigqueryMatcher will send a query to retrieve the estimated
+ // count and verifies its correctness using checksum.
+ options.setOnSuccessMatcher(
+ BigqueryMatcher.createUsingStandardSql(APP_NAME, PROJECT_ID, query, EXPECTED_CHECKSUM));
+
+ Pipeline p = Pipeline.create(options);
+ p.apply(Create.of(TEST_DATA))
+ .apply(HllCount.Init.forStrings().globally())
+ .apply(
+ BigQueryIO.<byte[]>write()
+ .to(tableSpec)
+ .withSchema(tableSchema)
+ .withFormatFunction(sketch -> new TableRow().set(SKETCH_FIELD_NAME, sketch))
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+ p.run().waitUntilFinish();
+ }
+}
diff --git a/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
new file mode 100644
index 0000000..4ef3e6a
--- /dev/null
+++ b/sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
@@ -0,0 +1,487 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.zetasketch;
+
+import com.google.zetasketch.HyperLogLogPlusPlus;
+import com.google.zetasketch.shaded.com.google.protobuf.ByteString;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.Pipeline.PipelineExecutionException;
+import org.apache.beam.sdk.testing.NeedsRunner;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link HllCount}. */
+@RunWith(JUnit4.class)
+public class HllCountTest {
+
+ @Rule public final transient TestPipeline p = TestPipeline.create();
+ @Rule public transient ExpectedException thrown = ExpectedException.none();
+
+ private static final byte[] EMPTY_SKETCH = new byte[0];
+
+ // Integer
+ private static final List<Integer> INTS1 = Arrays.asList(1, 2, 3, 3, 1, 4);
+ private static final byte[] INTS1_SKETCH;
+ private static final Long INTS1_ESTIMATE;
+
+ static {
+ HyperLogLogPlusPlus<Integer> hll = new HyperLogLogPlusPlus.Builder().buildForIntegers();
+ INTS1.forEach(hll::add);
+ INTS1_SKETCH = hll.serializeToByteArray();
+ INTS1_ESTIMATE = hll.longResult();
+ }
+
+ private static final List<Integer> INTS2 = Arrays.asList(3, 3, 3, 3);
+ private static final byte[] INTS2_SKETCH;
+ private static final Long INTS2_ESTIMATE;
+
+ static {
+ HyperLogLogPlusPlus<Integer> hll = new HyperLogLogPlusPlus.Builder().buildForIntegers();
+ INTS2.forEach(hll::add);
+ INTS2_SKETCH = hll.serializeToByteArray();
+ INTS2_ESTIMATE = hll.longResult();
+ }
+
+ private static final byte[] INTS1_INTS2_SKETCH;
+
+ static {
+ HyperLogLogPlusPlus<?> hll = HyperLogLogPlusPlus.forProto(INTS1_SKETCH);
+ hll.merge(INTS2_SKETCH);
+ INTS1_INTS2_SKETCH = hll.serializeToByteArray();
+ }
+
+ // Long
+ private static final List<Long> LONGS = Collections.singletonList(1L);
+ private static final byte[] LONGS_SKETCH;
+
+ static {
+ HyperLogLogPlusPlus<Long> hll = new HyperLogLogPlusPlus.Builder().buildForLongs();
+ LONGS.forEach(hll::add);
+ LONGS_SKETCH = hll.serializeToByteArray();
+ }
+
+ private static final byte[] LONGS_SKETCH_OF_EMPTY_SET;
+
+ static {
+ HyperLogLogPlusPlus<Long> hll = new HyperLogLogPlusPlus.Builder().buildForLongs();
+ LONGS_SKETCH_OF_EMPTY_SET = hll.serializeToByteArray();
+ }
+
+ // String
+ private static final List<String> STRINGS = Arrays.asList("s1", "s2", "s1", "s2");
+ private static final byte[] STRINGS_SKETCH;
+
+ static {
+ HyperLogLogPlusPlus<String> hll = new HyperLogLogPlusPlus.Builder().buildForStrings();
+ STRINGS.forEach(hll::add);
+ STRINGS_SKETCH = hll.serializeToByteArray();
+ }
+
+ private static final int TEST_PRECISION = 20;
+ private static final byte[] STRINGS_SKETCH_TEST_PRECISION;
+
+ static {
+ HyperLogLogPlusPlus<String> hll =
+ new HyperLogLogPlusPlus.Builder().normalPrecision(TEST_PRECISION).buildForStrings();
+ STRINGS.forEach(hll::add);
+ STRINGS_SKETCH_TEST_PRECISION = hll.serializeToByteArray();
+ }
+
+ // Bytes
+ private static final byte[] BYTES0 = {(byte) 0x1, (byte) 0xa};
+ private static final byte[] BYTES1 = {(byte) 0xf};
+ private static final List<byte[]> BYTES = Arrays.asList(BYTES0, BYTES1);
+ private static final byte[] BYTES_SKETCH;
+
+ static {
+ HyperLogLogPlusPlus<ByteString> hll = new HyperLogLogPlusPlus.Builder().buildForBytes();
+ BYTES.forEach(hll::add);
+ BYTES_SKETCH = hll.serializeToByteArray();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForIntegersGlobally() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(INTS1)).apply(HllCount.Init.forIntegers().globally());
+
+ PAssert.thatSingleton(result).isEqualTo(INTS1_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForLongsGlobally() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(LONGS)).apply(HllCount.Init.forLongs().globally());
+
+ PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForLongsGloballyForEmptyInput() {
+ PCollection<byte[]> result =
+ p.apply(Create.empty(TypeDescriptor.of(Long.class)))
+ .apply(HllCount.Init.forLongs().globally());
+
+ PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForStringsGlobally() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(STRINGS)).apply(HllCount.Init.forStrings().globally());
+
+ PAssert.thatSingleton(result).isEqualTo(STRINGS_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForStringsGloballyWithPrecision() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(STRINGS))
+ .apply(HllCount.Init.forStrings().withPrecision(TEST_PRECISION).globally());
+
+ PAssert.thatSingleton(result).isEqualTo(STRINGS_SKETCH_TEST_PRECISION);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForStringsGloballyWithInvalidPrecision() {
+ thrown.expect(IllegalArgumentException.class);
+ p.apply(Create.of(STRINGS)).apply(HllCount.Init.forStrings().withPrecision(0).globally());
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForBytesGlobally() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(BYTES)).apply(HllCount.Init.forBytes().globally());
+
+ PAssert.thatSingleton(result).isEqualTo(BYTES_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForIntegersPerKey() {
+ List<KV<String, Integer>> input = new ArrayList<>();
+ INTS1.forEach(i -> input.add(KV.of("k1", i)));
+ INTS1.forEach(i -> input.add(KV.of("k2", i)));
+ INTS2.forEach(i -> input.add(KV.of("k1", i)));
+ PCollection<KV<String, byte[]>> result =
+ p.apply(Create.of(input)).apply(HllCount.Init.forIntegers().perKey());
+
+ PAssert.that(result)
+ .containsInAnyOrder(
+ Arrays.asList(KV.of("k1", INTS1_INTS2_SKETCH), KV.of("k2", INTS1_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForLongsPerKey() {
+ List<KV<String, Long>> input = new ArrayList<>();
+ LONGS.forEach(l -> input.add(KV.of("k", l)));
+ PCollection<KV<String, byte[]>> result =
+ p.apply(Create.of(input)).apply(HllCount.Init.forLongs().perKey());
+
+ PAssert.that(result).containsInAnyOrder(Collections.singletonList(KV.of("k", LONGS_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForStringsPerKey() {
+ List<KV<String, String>> input = new ArrayList<>();
+ STRINGS.forEach(s -> input.add(KV.of("k", s)));
+ PCollection<KV<String, byte[]>> result =
+ p.apply(Create.of(input)).apply(HllCount.Init.forStrings().perKey());
+
+ PAssert.that(result).containsInAnyOrder(Collections.singletonList(KV.of("k", STRINGS_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForStringsPerKeyWithPrecision() {
+ List<KV<String, String>> input = new ArrayList<>();
+ STRINGS.forEach(s -> input.add(KV.of("k", s)));
+ PCollection<KV<String, byte[]>> result =
+ p.apply(Create.of(input))
+ .apply(HllCount.Init.forStrings().withPrecision(TEST_PRECISION).perKey());
+
+ PAssert.that(result)
+ .containsInAnyOrder(Collections.singletonList(KV.of("k", STRINGS_SKETCH_TEST_PRECISION)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForStringsPerKeyWithInvalidPrecision() {
+ List<KV<String, String>> input = new ArrayList<>();
+ STRINGS.forEach(s -> input.add(KV.of("k", s)));
+ thrown.expect(IllegalArgumentException.class);
+ p.apply(Create.of(input)).apply(HllCount.Init.forStrings().withPrecision(0).perKey());
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testInitForBytesPerKey() {
+ List<KV<String, byte[]>> input = new ArrayList<>();
+ BYTES.forEach(bs -> input.add(KV.of("k", bs)));
+ PCollection<KV<String, byte[]>> result =
+ p.apply(Create.of(input)).apply(HllCount.Init.forBytes().perKey());
+
+ PAssert.that(result).containsInAnyOrder(Collections.singletonList(KV.of("k", BYTES_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGlobally() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(INTS1_SKETCH, INTS2_SKETCH)).apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(INTS1_INTS2_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForEmptyInput() {
+ PCollection<byte[]> result =
+ p.apply(Create.empty(TypeDescriptor.of(byte[].class)))
+ .apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForSingletonInput() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(LONGS_SKETCH)).apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForSingletonInputEmptySketch() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForMergeWithEmptySketch() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(LONGS_SKETCH, EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForMergeMultipleEmptySketches() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(EMPTY_SKETCH, EMPTY_SKETCH)).apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(EMPTY_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForMergeWithSketchOfEmptySet() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(LONGS_SKETCH, LONGS_SKETCH_OF_EMPTY_SET))
+ .apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForMergeEmptySketchWithSketchOfEmptySet() {
+ PCollection<byte[]> result =
+ p.apply(Create.of(EMPTY_SKETCH, LONGS_SKETCH_OF_EMPTY_SET))
+ .apply(HllCount.MergePartial.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(LONGS_SKETCH_OF_EMPTY_SET);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialGloballyForIncompatibleSketches() {
+ p.apply(Create.of(INTS1_SKETCH, STRINGS_SKETCH)).apply(HllCount.MergePartial.globally());
+
+ thrown.expect(PipelineExecutionException.class);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialPerKey() {
+ PCollection<KV<String, byte[]>> result =
+ p.apply(
+ Create.of(
+ KV.of("k1", INTS1_SKETCH),
+ KV.of("k2", STRINGS_SKETCH),
+ KV.of("k1", INTS2_SKETCH)))
+ .apply(HllCount.MergePartial.perKey());
+
+ PAssert.that(result)
+ .containsInAnyOrder(
+ Arrays.asList(KV.of("k1", INTS1_INTS2_SKETCH), KV.of("k2", STRINGS_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialPerKeyForMergeWithEmptySketch() {
+ PCollection<KV<String, byte[]>> result =
+ p.apply(
+ Create.of(
+ KV.of("k1", INTS1_SKETCH),
+ KV.of("k2", EMPTY_SKETCH),
+ KV.of("k1", EMPTY_SKETCH)))
+ .apply(HllCount.MergePartial.perKey());
+
+ PAssert.that(result)
+ .containsInAnyOrder(Arrays.asList(KV.of("k1", INTS1_SKETCH), KV.of("k2", EMPTY_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialPerKeyForMergeMultipleEmptySketches() {
+ PCollection<KV<String, byte[]>> result =
+ p.apply(
+ Create.of(
+ KV.of("k1", EMPTY_SKETCH),
+ KV.of("k2", STRINGS_SKETCH),
+ KV.of("k1", EMPTY_SKETCH)))
+ .apply(HllCount.MergePartial.perKey());
+
+ PAssert.that(result)
+ .containsInAnyOrder(Arrays.asList(KV.of("k1", EMPTY_SKETCH), KV.of("k2", STRINGS_SKETCH)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testMergePartialPerKeyForIncompatibleSketches() {
+ p.apply(
+ Create.of(
+ KV.of("k1", LONGS_SKETCH), KV.of("k2", STRINGS_SKETCH), KV.of("k1", BYTES_SKETCH)))
+ .apply(HllCount.MergePartial.perKey());
+
+ thrown.expect(PipelineExecutionException.class);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testExtractGlobally() {
+ PCollection<Long> result =
+ p.apply(Create.of(INTS1_SKETCH, INTS2_SKETCH)).apply(HllCount.Extract.globally());
+
+ PAssert.that(result).containsInAnyOrder(INTS1_ESTIMATE, INTS2_ESTIMATE);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testExtractGloballyForEmptySketch() {
+ PCollection<Long> result = p.apply(Create.of(EMPTY_SKETCH)).apply(HllCount.Extract.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(0L);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testExtractGloballyForSketchOfEmptySet() {
+ PCollection<Long> result =
+ p.apply(Create.of(LONGS_SKETCH_OF_EMPTY_SET)).apply(HllCount.Extract.globally());
+
+ PAssert.thatSingleton(result).isEqualTo(0L);
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testExtractPerKey() {
+ PCollection<KV<String, Long>> result =
+ p.apply(Create.of(KV.of("k", INTS1_SKETCH), KV.of("k", INTS2_SKETCH)))
+ .apply(HllCount.Extract.perKey());
+
+ PAssert.that(result)
+ .containsInAnyOrder(Arrays.asList(KV.of("k", INTS1_ESTIMATE), KV.of("k", INTS2_ESTIMATE)));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testExtractPerKeyForEmptySketch() {
+ PCollection<KV<String, Long>> result =
+ p.apply(Create.of(KV.of("k", EMPTY_SKETCH))).apply(HllCount.Extract.perKey());
+
+ PAssert.thatSingleton(result).isEqualTo(KV.of("k", 0L));
+ p.run();
+ }
+
+ @Test
+ @Category(NeedsRunner.class)
+ public void testExtractPerKeyForSketchOfEmptySet() {
+ PCollection<KV<String, Long>> result =
+ p.apply(Create.of(KV.of("k", LONGS_SKETCH_OF_EMPTY_SET))).apply(HllCount.Extract.perKey());
+
+ PAssert.thatSingleton(result).isEqualTo(KV.of("k", 0L));
+ p.run();
+ }
+}
diff --git a/sdks/java/fn-execution/build.gradle b/sdks/java/fn-execution/build.gradle
index eb7da2b..ea46cff 100644
--- a/sdks/java/fn-execution/build.gradle
+++ b/sdks/java/fn-execution/build.gradle
@@ -17,20 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.fn')
description = "Apache Beam :: SDKs :: Java :: Fn Execution"
ext.summary = """Contains code shared across the Beam Java SDK Harness and Java Runners to execute using
the Beam Portability Framework."""
dependencies {
- shadow project(path: ":model:pipeline", configuration: "shadow")
- shadow project(path: ":model:fn-execution", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.vendored_grpc_1_13_1
- shadow library.java.vendored_guava_20_0
- shadow library.java.slf4j_api
- shadow library.java.joda_time
+ compile project(path: ":model:pipeline", configuration: "shadow")
+ compile project(path: ":model:fn-execution", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_grpc_1_21_0
+ compile library.java.vendored_guava_26_0_jre
+ compile library.java.slf4j_api
+ compile library.java.joda_time
provided library.java.junit
testCompile library.java.junit
testCompile library.java.hamcrest_core
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java
new file mode 100644
index 0000000..221ebe7
--- /dev/null
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.fn;
+
+import org.apache.beam.sdk.harness.JvmInitializer;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+
+/** Helpers for executing {@link JvmInitializer} implementations. */
+public class JvmInitializers {
+ /**
+ * Finds all registered implementations of JvmInitializer and executes their {@code onStartup}
+ * methods. Should be called in worker harness implementations at the very beginning of their main
+ * method.
+ */
+ public static void runOnStartup() {
+ for (JvmInitializer initializer : ReflectHelpers.loadServicesOrdered(JvmInitializer.class)) {
+ initializer.onStartup();
+ }
+ }
+
+ /**
+ * Finds all registered implementations of JvmInitializer and executes their {@code
+ * beforeProcessing} methods. Should be called in worker harness implementations after
+ * initialization but before beginning to process any data.
+ *
+ * @param options The pipeline options passed to the worker.
+ */
+ public static void runBeforeProcessing(PipelineOptions options) {
+ for (JvmInitializer initializer : ReflectHelpers.loadServicesOrdered(JvmInitializer.class)) {
+ initializer.beforeProcessing(options);
+ }
+ }
+}
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
index 2392576..bc6da0e 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
@@ -20,14 +20,14 @@
import java.net.SocketAddress;
import java.util.List;
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ClientInterceptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.netty.NettyChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.EpollDomainSocketChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.EpollEventLoopGroup;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.EpollSocketChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.unix.DomainSocketAddress;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ClientInterceptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.netty.NettyChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.EpollDomainSocketChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.EpollEventLoopGroup;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.EpollSocketChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.unix.DomainSocketAddress;
/** A Factory which creates an underlying {@link ManagedChannel} implementation. */
public abstract class ManagedChannelFactory {
@@ -36,7 +36,7 @@
}
public static ManagedChannelFactory createEpoll() {
- org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.Epoll.ensureAvailability();
+ org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.Epoll.ensureAvailability();
return new Epoll();
}
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java
index cccef91..c77e1bc 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.sdk.fn.channel;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.File;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.unix.DomainSocketAddress;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.net.HostAndPort;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.unix.DomainSocketAddress;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.net.HostAndPort;
/** Creates a {@link SocketAddress} based upon a supplied string. */
public class SocketAddressFactory {
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
index 72c613b..02460bf 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
@@ -20,10 +20,9 @@
import java.io.IOException;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -42,8 +41,7 @@
* <p>TODO: Handle outputting elements that are zero bytes by outputting a single byte as a marker,
* detect on the input side that no bytes were read and force reading a single byte.
*/
-public class BeamFnDataBufferingOutboundObserver<T>
- implements CloseableFnDataReceiver<WindowedValue<T>> {
+public class BeamFnDataBufferingOutboundObserver<T> implements CloseableFnDataReceiver<T> {
// TODO: Consider moving this constant out of this class
public static final String BEAM_FN_API_DATA_BUFFER_LIMIT = "beam_fn_api_data_buffer_limit=";
@VisibleForTesting static final int DEFAULT_BUFFER_LIMIT_BYTES = 1_000_000;
@@ -52,7 +50,7 @@
public static <T> BeamFnDataBufferingOutboundObserver<T> forLocation(
LogicalEndpoint endpoint,
- Coder<WindowedValue<T>> coder,
+ Coder<T> coder,
StreamObserver<BeamFnApi.Elements> outboundObserver) {
return forLocationWithBufferLimit(
DEFAULT_BUFFER_LIMIT_BYTES, endpoint, coder, outboundObserver);
@@ -61,7 +59,7 @@
public static <T> BeamFnDataBufferingOutboundObserver<T> forLocationWithBufferLimit(
int bufferLimit,
LogicalEndpoint endpoint,
- Coder<WindowedValue<T>> coder,
+ Coder<T> coder,
StreamObserver<BeamFnApi.Elements> outboundObserver) {
return new BeamFnDataBufferingOutboundObserver<>(
bufferLimit, endpoint, coder, outboundObserver);
@@ -71,7 +69,7 @@
private long counter;
private boolean closed;
private final int bufferLimit;
- private final Coder<WindowedValue<T>> coder;
+ private final Coder<T> coder;
private final LogicalEndpoint outputLocation;
private final StreamObserver<BeamFnApi.Elements> outboundObserver;
private final ByteString.Output bufferedElements;
@@ -79,7 +77,7 @@
private BeamFnDataBufferingOutboundObserver(
int bufferLimit,
LogicalEndpoint outputLocation,
- Coder<WindowedValue<T>> coder,
+ Coder<T> coder,
StreamObserver<BeamFnApi.Elements> outboundObserver) {
this.bufferLimit = bufferLimit;
this.outputLocation = outputLocation;
@@ -99,14 +97,14 @@
// This will add an empty data block representing the end of stream.
elements
.addDataBuilder()
- .setInstructionReference(outputLocation.getInstructionId())
- .setTarget(outputLocation.getTarget());
+ .setInstructionId(outputLocation.getInstructionId())
+ .setTransformId(outputLocation.getTransformId());
LOG.debug(
"Closing stream for instruction {} and "
- + "target {} having transmitted {} values {} bytes",
+ + "transform {} having transmitted {} values {} bytes",
outputLocation.getInstructionId(),
- outputLocation.getTarget(),
+ outputLocation.getTransformId(),
counter,
byteCounter);
outboundObserver.onNext(elements.build());
@@ -120,7 +118,7 @@
}
@Override
- public void accept(WindowedValue<T> t) throws IOException {
+ public void accept(T t) throws IOException {
if (closed) {
throw new IllegalStateException("Already closed.");
}
@@ -139,8 +137,8 @@
elements
.addDataBuilder()
- .setInstructionReference(outputLocation.getInstructionId())
- .setTarget(outputLocation.getTarget())
+ .setInstructionId(outputLocation.getInstructionId())
+ .setTransformId(outputLocation.getTransformId())
.setData(bufferedElements.toByteString());
byteCounter += bufferedElements.size();
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
index 002fa13..7ed83df 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
@@ -28,11 +28,11 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements.Data;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -127,8 +127,7 @@
public void onNext(BeamFnApi.Elements value) {
for (BeamFnApi.Elements.Data data : value.getDataList()) {
try {
- LogicalEndpoint key =
- LogicalEndpoint.of(data.getInstructionReference(), data.getTarget());
+ LogicalEndpoint key = LogicalEndpoint.of(data.getInstructionId(), data.getTransformId());
CompletableFuture<Consumer<BeamFnApi.Elements.Data>> consumer = receiverFuture(key);
if (!consumer.isDone()) {
LOG.debug(
@@ -146,16 +145,16 @@
*/
} catch (ExecutionException | InterruptedException e) {
LOG.error(
- "Client interrupted during handling of data for instruction {} and target {}",
- data.getInstructionReference(),
- data.getTarget(),
+ "Client interrupted during handling of data for instruction {} and transform {}",
+ data.getInstructionId(),
+ data.getTransformId(),
e);
outboundObserver.onError(e);
} catch (RuntimeException e) {
LOG.error(
- "Client failed to handle data for instruction {} and target {}",
- data.getInstructionReference(),
- data.getTarget(),
+ "Client failed to handle data for instruction {} and transform {}",
+ data.getInstructionId(),
+ data.getTransformId(),
e);
outboundObserver.onError(e);
}
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
index 1be0fa9..2ed5539 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
@@ -21,7 +21,6 @@
import java.util.function.Consumer;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.util.WindowedValue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -34,21 +33,19 @@
private static final Logger LOG = LoggerFactory.getLogger(BeamFnDataInboundObserver.class);
public static <T> BeamFnDataInboundObserver<T> forConsumer(
- Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>> receiver) {
+ Coder<T> coder, FnDataReceiver<T> receiver) {
return new BeamFnDataInboundObserver<>(
coder, receiver, CompletableFutureInboundDataClient.create());
}
- private final FnDataReceiver<WindowedValue<T>> consumer;
- private final Coder<WindowedValue<T>> coder;
+ private final FnDataReceiver<T> consumer;
+ private final Coder<T> coder;
private final InboundDataClient readFuture;
private long byteCounter;
private long counter;
public BeamFnDataInboundObserver(
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> consumer,
- InboundDataClient readFuture) {
+ Coder<T> coder, FnDataReceiver<T> consumer, InboundDataClient readFuture) {
this.coder = coder;
this.consumer = consumer;
this.readFuture = readFuture;
@@ -64,9 +61,9 @@
if (t.getData().isEmpty()) {
LOG.debug(
"Closing stream for instruction {} and "
- + "target {} having consumed {} values {} bytes",
- t.getInstructionReference(),
- t.getTarget(),
+ + "transform {} having consumed {} values {} bytes",
+ t.getInstructionId(),
+ t.getTransformId(),
counter,
byteCounter);
readFuture.complete();
@@ -77,7 +74,7 @@
InputStream inputStream = t.getData().newInput();
while (inputStream.available() > 0) {
counter += 1;
- WindowedValue<T> value = coder.decode(inputStream);
+ T value = coder.decode(inputStream);
consumer.accept(value);
}
} catch (Exception e) {
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java
index 19c8f35..2adbfa9 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java
@@ -22,17 +22,17 @@
/**
* A logical endpoint is a pair of an instruction ID corresponding to the {@link
- * BeamFnApi.ProcessBundleRequest} and the {@link BeamFnApi.Target} within the processing graph.
- * This enables the same Data Service or Data Client to be re-used across multiple bundles.
+ * BeamFnApi.ProcessBundleRequest} and the transform within the processing graph. This enables the
+ * same Data Service or Data Client to be re-used across multiple bundles.
*/
@AutoValue
public abstract class LogicalEndpoint {
public abstract String getInstructionId();
- public abstract BeamFnApi.Target getTarget();
+ public abstract String getTransformId();
- public static LogicalEndpoint of(String instructionId, BeamFnApi.Target target) {
- return new AutoValue_LogicalEndpoint(instructionId, target);
+ public static LogicalEndpoint of(String instructionId, String transformId) {
+ return new AutoValue_LogicalEndpoint(instructionId, transformId);
}
}
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java
index 3de440a..9568b90 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.sdk.fn.data;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.RemoteGrpcPort;
import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* An execution-time only {@link PTransform} which represents an SDK harness reading from a {@link
@@ -32,7 +32,7 @@
*/
@AutoValue
public abstract class RemoteGrpcPortRead {
- public static final String URN = "urn:org.apache.beam:source:runner:0.1";
+ public static final String URN = "beam:source:runner:0.1";
private static final String LOCAL_OUTPUT_ID = "local_output";
public static RemoteGrpcPortRead readFromPort(RemoteGrpcPort port, String outputPCollectionId) {
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java
index 8ad8adc..b1c7604 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.fn.data;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.RemoteGrpcPort;
import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* An execution-time only {@link PTransform} which represents a write from within an SDK harness to
@@ -33,7 +33,7 @@
*/
@AutoValue
public abstract class RemoteGrpcPortWrite {
- public static final String URN = "urn:org.apache.beam:sink:runner:0.1";
+ public static final String URN = "beam:sink:runner:0.1";
private static final String LOCAL_INPUT_ID = "local_input";
/**
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java
index addeb68..88428af 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java
@@ -18,9 +18,8 @@
package org.apache.beam.sdk.fn.splittabledofn;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.sdk.transforms.splittabledofn.Backlog;
-import org.apache.beam.sdk.transforms.splittabledofn.Backlogs;
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.transforms.splittabledofn.Sizes;
/** Support utilities for interacting with {@link RestrictionTracker RestrictionTrackers}. */
public class RestrictionTrackers {
@@ -83,39 +82,18 @@
* RestrictionTracker}.
*/
@ThreadSafe
- private static class RestrictionTrackerObserverWithBacklog<RestrictionT, PositionT>
- extends RestrictionTrackerObserver<RestrictionT, PositionT> implements Backlogs.HasBacklog {
+ private static class RestrictionTrackerObserverWithSize<RestrictionT, PositionT>
+ extends RestrictionTrackerObserver<RestrictionT, PositionT> implements Sizes.HasSize {
- protected RestrictionTrackerObserverWithBacklog(
+ protected RestrictionTrackerObserverWithSize(
RestrictionTracker<RestrictionT, PositionT> delegate,
ClaimObserver<PositionT> claimObserver) {
super(delegate, claimObserver);
}
@Override
- public synchronized Backlog getBacklog() {
- return ((Backlogs.HasBacklog) delegate).getBacklog();
- }
- }
-
- /**
- * A {@link RestrictionTracker} which forwards all calls to the delegate partitioned backlog
- * reporting {@link RestrictionTracker}.
- */
- @ThreadSafe
- private static class RestrictionTrackerObserverWithPartitionedBacklog<RestrictionT, PositionT>
- extends RestrictionTrackerObserverWithBacklog<RestrictionT, PositionT>
- implements Backlogs.HasPartitionedBacklog {
-
- protected RestrictionTrackerObserverWithPartitionedBacklog(
- RestrictionTracker<RestrictionT, PositionT> delegate,
- ClaimObserver<PositionT> claimObserver) {
- super(delegate, claimObserver);
- }
-
- @Override
- public synchronized byte[] getBacklogPartition() {
- return ((Backlogs.HasPartitionedBacklog) delegate).getBacklogPartition();
+ public synchronized double getSize() {
+ return ((Sizes.HasSize) delegate).getSize();
}
}
@@ -126,11 +104,8 @@
public static <RestrictionT, PositionT> RestrictionTracker<RestrictionT, PositionT> observe(
RestrictionTracker<RestrictionT, PositionT> restrictionTracker,
ClaimObserver<PositionT> claimObserver) {
- if (restrictionTracker instanceof Backlogs.HasPartitionedBacklog) {
- return new RestrictionTrackerObserverWithPartitionedBacklog<>(
- restrictionTracker, claimObserver);
- } else if (restrictionTracker instanceof Backlogs.HasBacklog) {
- return new RestrictionTrackerObserverWithBacklog<>(restrictionTracker, claimObserver);
+ if (restrictionTracker instanceof Sizes.HasSize) {
+ return new RestrictionTrackerObserverWithSize<>(restrictionTracker, claimObserver);
} else {
return new RestrictionTrackerObserver<>(restrictionTracker, claimObserver);
}
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java
index a49e957..da7505d 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.fn.stream;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.concurrent.CancellationException;
import java.util.concurrent.ExecutionException;
@@ -27,9 +27,9 @@
import java.util.concurrent.Phaser;
import java.util.concurrent.TimeUnit;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A thread safe {@link StreamObserver} which uses a bounded queue to pass elements to a processing
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java
index 0fe35ee..3134ea4 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.fn.stream;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
@@ -27,9 +27,9 @@
import java.util.NoSuchElementException;
import java.util.concurrent.BlockingQueue;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingInputStream;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingInputStream;
/**
* {@link #inbound(Iterator)} treats multiple {@link ByteString}s as a single input stream and
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
index 3fca8da..dce3452 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
@@ -21,8 +21,8 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import javax.annotation.concurrent.ThreadSafe;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java
index 1314748..a25985d 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.fn.stream;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientResponseObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientResponseObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A {@link ClientResponseObserver} which delegates all {@link StreamObserver} calls.
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
index c912df5..dde456b 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.fn.stream;
import java.util.concurrent.ExecutorService;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* Creates factories which determine an underlying {@link StreamObserver} implementation to use in
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java
index 374007f..c960d96 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.fn.stream;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* A {@link StreamObserver} which provides synchronous access access to an underlying {@link
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/InProcessManagedChannelFactory.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/InProcessManagedChannelFactory.java
index 63a3e8d..a4c99a1 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/InProcessManagedChannelFactory.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/InProcessManagedChannelFactory.java
@@ -19,8 +19,8 @@
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
/**
* A {@link ManagedChannelFactory} that uses in-process channels.
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestExecutors.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestExecutors.java
index 71882c5..4e46390 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestExecutors.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestExecutors.java
@@ -20,7 +20,7 @@
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ForwardingExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ForwardingExecutorService;
import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java
index 4976774..b76997e 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java
@@ -19,8 +19,8 @@
import java.util.function.Consumer;
import java.util.function.Supplier;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/** Utility methods which enable testing of {@link StreamObserver}s. */
public class TestStreams {
diff --git a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java
index 5af6fcc..94d3400 100644
--- a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java
+++ b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java
@@ -25,8 +25,8 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.util.VarInt;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
import org.joda.time.Instant;
/**
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java
new file mode 100644
index 0000000..a20b3a9
--- /dev/null
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.fn;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.harness.JvmInitializer;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link JvmInitializers}. */
+@RunWith(JUnit4.class)
+public final class JvmInitializersTest {
+
+ private static Boolean onStartupRan;
+ private static Boolean beforeProcessingRan;
+ private static PipelineOptions receivedOptions;
+
+ /** Test initializer implementation. Methods simply produce observable side effects. */
+ @AutoService(JvmInitializer.class)
+ public static class TestInitializer implements JvmInitializer {
+ @Override
+ public void onStartup() {
+ onStartupRan = true;
+ }
+
+ @Override
+ public void beforeProcessing(PipelineOptions options) {
+ beforeProcessingRan = true;
+ receivedOptions = options;
+ }
+ }
+
+ @Before
+ public void setUp() {
+ onStartupRan = false;
+ beforeProcessingRan = false;
+ receivedOptions = null;
+ }
+
+ @Test
+ public void runOnStartup_runsInitializers() {
+ JvmInitializers.runOnStartup();
+
+ assertTrue(onStartupRan);
+ }
+
+ @Test
+ public void runBeforeProcessing_runsInitializersWithOptions() {
+ PipelineOptions options = TestPipeline.testingPipelineOptions();
+
+ JvmInitializers.runBeforeProcessing(options);
+
+ assertTrue(beforeProcessingRan);
+ assertEquals(options, receivedOptions);
+ }
+}
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java
index cae36a7..3e60697 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java
@@ -21,7 +21,7 @@
import static org.junit.Assume.assumeTrue;
import org.apache.beam.model.pipeline.v1.Endpoints;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
@@ -45,7 +45,7 @@
@Test
public void testEpollHostPortChannel() {
- assumeTrue(org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.Epoll.isAvailable());
+ assumeTrue(org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.Epoll.isAvailable());
Endpoints.ApiServiceDescriptor apiServiceDescriptor =
Endpoints.ApiServiceDescriptor.newBuilder().setUrl("localhost:123").build();
ManagedChannel channel =
@@ -56,7 +56,7 @@
@Test
public void testEpollDomainSocketChannel() throws Exception {
- assumeTrue(org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.epoll.Epoll.isAvailable());
+ assumeTrue(org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.epoll.Epoll.isAvailable());
Endpoints.ApiServiceDescriptor apiServiceDescriptor =
Endpoints.ApiServiceDescriptor.newBuilder()
.setUrl("unix://" + tmpFolder.newFile().getAbsolutePath())
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java
index 384a705..0107a7b 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java
@@ -23,7 +23,7 @@
import java.io.File;
import java.net.InetSocketAddress;
import java.net.SocketAddress;
-import org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.unix.DomainSocketAddress;
+import org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.unix.DomainSocketAddress;
import org.hamcrest.Matchers;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
index cd33935..cb1752c 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
@@ -30,14 +30,13 @@
import java.util.function.Consumer;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.LengthPrefixCoder;
import org.apache.beam.sdk.fn.test.TestStreams;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -45,10 +44,7 @@
/** Tests for {@link BeamFnDataBufferingOutboundObserver}. */
@RunWith(JUnit4.class)
public class BeamFnDataBufferingOutboundObserverTest {
- private static final LogicalEndpoint OUTPUT_LOCATION =
- LogicalEndpoint.of(
- "777L",
- Target.newBuilder().setPrimitiveTransformReference("555L").setName("Test").build());
+ private static final LogicalEndpoint OUTPUT_LOCATION = LogicalEndpoint.of("777L", "555L");
private static final Coder<WindowedValue<byte[]>> CODER =
LengthPrefixCoder.of(WindowedValue.getValueOnlyCoder(ByteArrayCoder.of()));
@@ -144,8 +140,8 @@
BeamFnApi.Elements.newBuilder(messageWithData(new byte[1]))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setTarget(OUTPUT_LOCATION.getTarget()))
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId()))
.build(),
Iterables.get(values, 1));
}
@@ -158,8 +154,8 @@
return BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setTarget(OUTPUT_LOCATION.getTarget())
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId())
.setData(output.toByteString()))
.build();
}
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
index 7f08d78..bf1b1d3 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
@@ -31,35 +31,29 @@
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Test;
/** Tests for {@link BeamFnDataGrpcMultiplexer}. */
public class BeamFnDataGrpcMultiplexerTest {
private static final Endpoints.ApiServiceDescriptor DESCRIPTOR =
Endpoints.ApiServiceDescriptor.newBuilder().setUrl("test").build();
- private static final LogicalEndpoint OUTPUT_LOCATION =
- LogicalEndpoint.of(
- "777L",
- BeamFnApi.Target.newBuilder()
- .setName("name")
- .setPrimitiveTransformReference("888L")
- .build());
+ private static final LogicalEndpoint OUTPUT_LOCATION = LogicalEndpoint.of("777L", "888L");
private static final BeamFnApi.Elements ELEMENTS =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setTarget(OUTPUT_LOCATION.getTarget())
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId())
.setData(ByteString.copyFrom(new byte[1])))
.build();
private static final BeamFnApi.Elements TERMINAL_ELEMENTS =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(OUTPUT_LOCATION.getInstructionId())
- .setTarget(OUTPUT_LOCATION.getTarget()))
+ .setInstructionId(OUTPUT_LOCATION.getInstructionId())
+ .setTransformId(OUTPUT_LOCATION.getTransformId()))
.build();
@Test
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java
index 2baca01..c1b2175 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.model.pipeline.v1.Endpoints.OAuth2ClientCredentialsGrant;
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java
index c818999..c4be16b 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java
@@ -24,7 +24,7 @@
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.model.pipeline.v1.Endpoints.OAuth2ClientCredentialsGrant;
import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java
index c3bb289..8f2b5de 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java
@@ -25,9 +25,8 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.fn.splittabledofn.RestrictionTrackers.ClaimObserver;
-import org.apache.beam.sdk.transforms.splittabledofn.Backlog;
-import org.apache.beam.sdk.transforms.splittabledofn.Backlogs;
import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
+import org.apache.beam.sdk.transforms.splittabledofn.Sizes;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -85,12 +84,12 @@
assertThat(positionsObserved, contains("goodClaim", "badClaim"));
}
- private static class RestrictionTrackerWithBacklog extends RestrictionTracker<Object, Object>
- implements Backlogs.HasBacklog {
+ private static class RestrictionTrackerWithSize extends RestrictionTracker<Object, Object>
+ implements Sizes.HasSize {
@Override
- public Backlog getBacklog() {
- return null;
+ public double getSize() {
+ return 1;
}
@Override
@@ -112,45 +111,10 @@
public void checkDone() throws IllegalStateException {}
}
- private static class RestrictionTrackerWithBacklogPartitionedBacklog
- extends RestrictionTracker<Object, Object> implements Backlogs.HasPartitionedBacklog {
-
- @Override
- public Backlog getBacklog() {
- return null;
- }
-
- @Override
- public boolean tryClaim(Object position) {
- return false;
- }
-
- @Override
- public Object currentRestriction() {
- return null;
- }
-
- @Override
- public Object checkpoint() {
- return null;
- }
-
- @Override
- public void checkDone() throws IllegalStateException {}
-
- @Override
- public byte[] getBacklogPartition() {
- return null;
- }
- }
-
@Test
public void testClaimObserversMaintainBacklogInterfaces() {
- RestrictionTracker hasBacklog =
- RestrictionTrackers.observe(new RestrictionTrackerWithBacklog(), null);
- assertThat(hasBacklog, instanceOf(Backlogs.HasBacklog.class));
- RestrictionTracker hasPartitionedBacklog =
- RestrictionTrackers.observe(new RestrictionTrackerWithBacklogPartitionedBacklog(), null);
- assertThat(hasPartitionedBacklog, instanceOf(Backlogs.HasPartitionedBacklog.class));
+ RestrictionTracker hasSize =
+ RestrictionTrackers.observe(new RestrictionTrackerWithSize(), null);
+ assertThat(hasSize, instanceOf(Sizes.HasSize.class));
}
}
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserverTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserverTest.java
index d8c01c9..4fe4ebf 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserverTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserverTest.java
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.fn.test.TestExecutors;
import org.apache.beam.sdk.fn.test.TestExecutors.TestExecutorService;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java
index d3250b1..3e66d50 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java
@@ -41,11 +41,11 @@
import org.apache.beam.sdk.fn.stream.DataStreams.DataStreamDecoder;
import org.apache.beam.sdk.fn.stream.DataStreams.ElementDelimitedOutputStream;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.SettableFuture;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DirectStreamObserverTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DirectStreamObserverTest.java
index a940ca8..2edc93c 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DirectStreamObserverTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DirectStreamObserverTest.java
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.fn.test.TestExecutors;
import org.apache.beam.sdk.fn.test.TestExecutors.TestExecutorService;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java
index d2ed19e..97fc2da 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java
@@ -21,9 +21,9 @@
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientResponseObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientResponseObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java
index ead1f30..60cd8b0 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java
@@ -22,8 +22,8 @@
import static org.junit.Assert.assertThat;
import java.util.concurrent.Executors;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java
index 5086478..18d6896 100644
--- a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java
+++ b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java
@@ -19,7 +19,7 @@
import org.apache.beam.sdk.fn.windowing.EncodedBoundedWindow.Coder;
import org.apache.beam.sdk.testing.CoderProperties;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/harness/build.gradle b/sdks/java/harness/build.gradle
index 3c107e8..51c65bb 100644
--- a/sdks/java/harness/build.gradle
+++ b/sdks/java/harness/build.gradle
@@ -21,15 +21,16 @@
// We specifically enumerate all the projects that we depend on since
// the list is used in both defining the included set for the uber jar
// and also the set of project level dependencies.
-def dependOnProjects = [":model:pipeline", ":model:fn-execution", ":sdks:java:core",
- ":sdks:java:fn-execution",
+def dependOnShadedProjects = [":model:pipeline", ":model:fn-execution", ":sdks:java:core"]
+def dependOnProjects = [":sdks:java:fn-execution",
":sdks:java:extensions:google-cloud-platform-core",
":runners:core-java", ":runners:core-construction-java"]
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.fn.harness',
validateShadowJar: false,
testShadowJar: true,
- shadowClosure: DEFAULT_SHADOW_CLOSURE <<
+ shadowClosure:
// Create an uber jar without repackaging for the SDK harness
// TODO: We have been releasing this in the past, consider not
// releasing it since its typically bad practice to release 'all'
@@ -49,15 +50,19 @@
ext.summary = "This contains the SDK Fn Harness for Beam Java"
dependencies {
- dependOnProjects.each {
+ dependOnShadedProjects.each {
compile project(path: it, configuration: "shadow")
}
+ dependOnProjects.each {
+ compile project(it)
+ }
compile library.java.jackson_databind
- shadow library.java.vendored_guava_20_0
+ shadow library.java.vendored_guava_26_0_jre
shadowTest library.java.powermock
+ shadowTest library.java.powermock_mockito
compile library.java.joda_time
compile library.java.slf4j_api
- compile library.java.vendored_grpc_1_13_1
+ compile library.java.vendored_grpc_1_21_0
provided library.java.error_prone_annotations
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
index d377be9..b9d4106 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -33,9 +33,9 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
/** The Java SDK Harness implementation of the {@link Window.Assign} primitive. */
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
index e44b54b..48dbcf1 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.fn.data.RemoteGrpcPortRead;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -89,11 +89,6 @@
BundleSplitListener splitListener)
throws IOException {
- BeamFnApi.Target target =
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference(pTransformId)
- .setName(getOnlyElement(pTransform.getOutputsMap().keySet()))
- .build();
RunnerApi.Coder coderSpec;
if (RemoteGrpcPortRead.fromPTransform(pTransform).getPort().getCoderId().isEmpty()) {
LOG.error(
@@ -113,9 +108,9 @@
BeamFnDataReadRunner<OutputT> runner =
new BeamFnDataReadRunner<>(
+ pTransformId,
pTransform,
processBundleInstructionId,
- target,
coderSpec,
coders,
beamFnDataClient,
@@ -126,27 +121,27 @@
}
}
+ private final String pTransformId;
private final Endpoints.ApiServiceDescriptor apiServiceDescriptor;
private final FnDataReceiver<WindowedValue<OutputT>> consumer;
private final Supplier<String> processBundleInstructionIdSupplier;
private final BeamFnDataClient beamFnDataClient;
private final Coder<WindowedValue<OutputT>> coder;
- private final BeamFnApi.Target inputTarget;
private InboundDataClient readFuture;
BeamFnDataReadRunner(
+ String pTransformId,
RunnerApi.PTransform grpcReadNode,
Supplier<String> processBundleInstructionIdSupplier,
- BeamFnApi.Target inputTarget,
RunnerApi.Coder coderSpec,
Map<String, RunnerApi.Coder> coders,
BeamFnDataClient beamFnDataClient,
FnDataReceiver<WindowedValue<OutputT>> consumer)
throws IOException {
+ this.pTransformId = pTransformId;
RemoteGrpcPort port = RemoteGrpcPortRead.fromPTransform(grpcReadNode).getPort();
this.apiServiceDescriptor = port.getApiServiceDescriptor();
- this.inputTarget = inputTarget;
this.processBundleInstructionIdSupplier = processBundleInstructionIdSupplier;
this.beamFnDataClient = beamFnDataClient;
this.consumer = consumer;
@@ -170,16 +165,16 @@
this.readFuture =
beamFnDataClient.receive(
apiServiceDescriptor,
- LogicalEndpoint.of(processBundleInstructionIdSupplier.get(), inputTarget),
+ LogicalEndpoint.of(processBundleInstructionIdSupplier.get(), pTransformId),
coder,
consumer);
}
public void blockTillReadFinishes() throws Exception {
LOG.debug(
- "Waiting for process bundle instruction {} and target {} to close.",
+ "Waiting for process bundle instruction {} and transform {} to close.",
processBundleInstructionIdSupplier.get(),
- inputTarget);
+ pTransformId);
readFuture.awaitCompletion();
}
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
index 5e0ebb7..9d2e352 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.fn.data.RemoteGrpcPortWrite;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -88,11 +88,6 @@
PTransformFunctionRegistry finishFunctionRegistry,
BundleSplitListener splitListener)
throws IOException {
- BeamFnApi.Target target =
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference(pTransformId)
- .setName(getOnlyElement(pTransform.getInputsMap().keySet()))
- .build();
RunnerApi.Coder coderSpec;
if (RemoteGrpcPortWrite.fromPTransform(pTransform).getPort().getCoderId().isEmpty()) {
LOG.error(
@@ -106,7 +101,12 @@
}
BeamFnDataWriteRunner<InputT> runner =
new BeamFnDataWriteRunner<>(
- pTransform, processBundleInstructionId, target, coderSpec, coders, beamFnDataClient);
+ pTransformId,
+ pTransform,
+ processBundleInstructionId,
+ coderSpec,
+ coders,
+ beamFnDataClient);
startFunctionRegistry.register(pTransformId, runner::registerForOutput);
pCollectionConsumerRegistry.register(
getOnlyElement(pTransform.getInputsMap().values()),
@@ -119,7 +119,7 @@
}
private final Endpoints.ApiServiceDescriptor apiServiceDescriptor;
- private final BeamFnApi.Target outputTarget;
+ private final String pTransformId;
private final Coder<WindowedValue<InputT>> coder;
private final BeamFnDataClient beamFnDataClientFactory;
private final Supplier<String> processBundleInstructionIdSupplier;
@@ -127,18 +127,18 @@
private CloseableFnDataReceiver<WindowedValue<InputT>> consumer;
BeamFnDataWriteRunner(
+ String pTransformId,
RunnerApi.PTransform remoteWriteNode,
Supplier<String> processBundleInstructionIdSupplier,
- BeamFnApi.Target outputTarget,
RunnerApi.Coder coderSpec,
Map<String, RunnerApi.Coder> coders,
BeamFnDataClient beamFnDataClientFactory)
throws IOException {
+ this.pTransformId = pTransformId;
RemoteGrpcPort port = RemoteGrpcPortWrite.fromPTransform(remoteWriteNode).getPort();
this.apiServiceDescriptor = port.getApiServiceDescriptor();
this.beamFnDataClientFactory = beamFnDataClientFactory;
this.processBundleInstructionIdSupplier = processBundleInstructionIdSupplier;
- this.outputTarget = outputTarget;
RehydratedComponents components =
RehydratedComponents.forComponents(Components.newBuilder().putAllCoders(coders).build());
@@ -159,7 +159,7 @@
consumer =
beamFnDataClientFactory.send(
apiServiceDescriptor,
- LogicalEndpoint.of(processBundleInstructionIdSupplier.get(), outputTarget),
+ LogicalEndpoint.of(processBundleInstructionIdSupplier.get(), pTransformId),
coder);
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
index 811681d..6342e1d 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
@@ -41,9 +41,9 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.InvalidProtocolBufferException;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* A runner which creates {@link Reader}s for each {@link BoundedSource} sent as an input and
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
index ee196e1..71a807c 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
@@ -43,9 +43,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Executes different components of Combine PTransforms. */
public class CombineRunners {
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/DoFnPTransformRunnerFactory.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/DoFnPTransformRunnerFactory.java
index 847473a..d548fc5 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/DoFnPTransformRunnerFactory.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/DoFnPTransformRunnerFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.Map;
@@ -53,12 +53,12 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/** A {@link PTransformRunnerFactory} for transforms invoking a {@link DoFn}. */
abstract class DoFnPTransformRunnerFactory<
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FlattenRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FlattenRunner.java
index 7346872..020a8e8 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FlattenRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FlattenRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Executes flatten PTransforms. */
public class FlattenRunner<InputT> {
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
index 78f035b..6148336 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -59,8 +59,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.DateTimeUtils;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -115,6 +115,8 @@
private DoFnSchemaInformation doFnSchemaInformation;
+ private Map<String, PCollectionView<?>> sideInputMapping;
+
FnApiDoFnRunner(Context<InputT, OutputT> context) {
this.context = context;
@@ -122,6 +124,7 @@
(Collection<FnDataReceiver<WindowedValue<OutputT>>>)
(Collection) context.localNameToConsumer.get(context.mainOutputTag.getId());
this.doFnSchemaInformation = ParDoTranslation.getSchemaInformation(context.parDoPayload);
+ this.sideInputMapping = ParDoTranslation.getSideInputMapping(context.parDoPayload);
this.doFnInvoker = DoFnInvokers.invokerFor(context.doFn);
this.doFnInvoker.invokeSetup();
@@ -397,6 +400,11 @@
}
@Override
+ public Object sideInput(String tagId) {
+ return sideInput(sideInputMapping.get(tagId));
+ }
+
+ @Override
public Object schemaElement(int index) {
SerializableFunction converter = doFnSchemaInformation.getElementConverters().get(index);
return converter.apply(element());
@@ -581,6 +589,11 @@
}
@Override
+ public InputT sideInput(String tagId) {
+ throw new UnsupportedOperationException("SideInput parameters are not supported.");
+ }
+
+ @Override
public Object schemaElement(int index) {
throw new UnsupportedOperationException("Element parameters are not supported.");
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
index 708b669..252d902 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
@@ -19,6 +19,8 @@
import java.util.EnumMap;
import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.function.Function;
import org.apache.beam.fn.harness.control.AddHarnessIdInterceptor;
import org.apache.beam.fn.harness.control.BeamFnControlClient;
import org.apache.beam.fn.harness.control.ProcessBundleHandler;
@@ -35,14 +37,16 @@
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.JvmInitializers;
import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.function.ThrowingFunction;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -70,33 +74,52 @@
private static final String PIPELINE_OPTIONS = "PIPELINE_OPTIONS";
private static final Logger LOG = LoggerFactory.getLogger(FnHarness.class);
- private static Endpoints.ApiServiceDescriptor getApiServiceDescriptor(String env)
+ private static Endpoints.ApiServiceDescriptor getApiServiceDescriptor(String descriptor)
throws TextFormat.ParseException {
Endpoints.ApiServiceDescriptor.Builder apiServiceDescriptorBuilder =
Endpoints.ApiServiceDescriptor.newBuilder();
- TextFormat.merge(System.getenv(env), apiServiceDescriptorBuilder);
+ TextFormat.merge(descriptor, apiServiceDescriptorBuilder);
return apiServiceDescriptorBuilder.build();
}
public static void main(String[] args) throws Exception {
- System.out.format("SDK Fn Harness started%n");
- System.out.format("Harness ID %s%n", System.getenv(HARNESS_ID));
- System.out.format("Logging location %s%n", System.getenv(LOGGING_API_SERVICE_DESCRIPTOR));
- System.out.format("Control location %s%n", System.getenv(CONTROL_API_SERVICE_DESCRIPTOR));
- System.out.format("Pipeline options %s%n", System.getenv(PIPELINE_OPTIONS));
+ main(System::getenv);
+ }
- String id = System.getenv(HARNESS_ID);
- PipelineOptions options = PipelineOptionsTranslation.fromJson(System.getenv(PIPELINE_OPTIONS));
+ @VisibleForTesting
+ public static void main(Function<String, String> environmentVarGetter) throws Exception {
+ JvmInitializers.runOnStartup();
+ System.out.format("SDK Fn Harness started%n");
+ System.out.format("Harness ID %s%n", environmentVarGetter.apply(HARNESS_ID));
+ System.out.format(
+ "Logging location %s%n", environmentVarGetter.apply(LOGGING_API_SERVICE_DESCRIPTOR));
+ System.out.format(
+ "Control location %s%n", environmentVarGetter.apply(CONTROL_API_SERVICE_DESCRIPTOR));
+ System.out.format("Pipeline options %s%n", environmentVarGetter.apply(PIPELINE_OPTIONS));
+
+ String id = environmentVarGetter.apply(HARNESS_ID);
+ PipelineOptions options =
+ PipelineOptionsTranslation.fromJson(environmentVarGetter.apply(PIPELINE_OPTIONS));
Endpoints.ApiServiceDescriptor loggingApiServiceDescriptor =
- getApiServiceDescriptor(LOGGING_API_SERVICE_DESCRIPTOR);
+ getApiServiceDescriptor(environmentVarGetter.apply(LOGGING_API_SERVICE_DESCRIPTOR));
Endpoints.ApiServiceDescriptor controlApiServiceDescriptor =
- getApiServiceDescriptor(CONTROL_API_SERVICE_DESCRIPTOR);
+ getApiServiceDescriptor(environmentVarGetter.apply(CONTROL_API_SERVICE_DESCRIPTOR));
main(id, options, loggingApiServiceDescriptor, controlApiServiceDescriptor);
}
+ /**
+ * Run a FnHarness with the given id and options that attaches to the specified logging and
+ * control API service descriptors.
+ *
+ * @param id Harness ID
+ * @param options The options for this pipeline
+ * @param loggingApiServiceDescriptor
+ * @param controlApiServiceDescriptor
+ * @throws Exception
+ */
public static void main(
String id,
PipelineOptions options,
@@ -123,6 +146,18 @@
outboundObserverFactory);
}
+ /**
+ * Run a FnHarness with the given id and options that attaches to the specified logging and
+ * control API service descriptors using the given channel factory and outbound observer factory.
+ *
+ * @param id Harness ID
+ * @param options The options for this pipeline
+ * @param loggingApiServiceDescriptor
+ * @param controlApiServiceDescriptor
+ * @param channelFactory
+ * @param outboundObserverFactory
+ * @throws Exception
+ */
public static void main(
String id,
PipelineOptions options,
@@ -132,6 +167,7 @@
OutboundObserverFactory outboundObserverFactory)
throws Exception {
IdGenerator idGenerator = IdGenerators.decrementingLongs();
+ ExecutorService executorService = options.as(GcsOptions.class).getExecutorService();
// The logging client variable is not used per se, but during its lifetime (until close()) it
// intercepts logging and sends it to the logging service.
try (BeamFnLoggingClient logging =
@@ -166,10 +202,13 @@
new BeamFnControlClient(
id, controlApiServiceDescriptor, channelFactory, outboundObserverFactory, handlers);
+ JvmInitializers.runBeforeProcessing(options);
+
LOG.info("Entering instruction processing loop");
- control.processInstructionRequests(options.as(GcsOptions.class).getExecutorService());
+ control.processInstructionRequests(executorService);
} finally {
System.out.println("Shutting SDK harness down.");
+ executorService.shutdown();
}
}
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/MapFnRunners.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/MapFnRunners.java
index ade8b07..4345d29 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/MapFnRunners.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/MapFnRunners.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables.getOnlyElement;
import java.io.IOException;
import java.util.Map;
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.function.ThrowingFunction;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* Utilities to create {@code PTransformRunners} which execute simple map functions.
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PrecombineGroupingTable.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PrecombineGroupingTable.java
index 523bf3b..48a55bf 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PrecombineGroupingTable.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/PrecombineGroupingTable.java
@@ -32,9 +32,9 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
import org.joda.time.Instant;
/** Static utility methods that provide {@link GroupingTable} implementations. */
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
index 0b13408..ec2875e 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
import java.io.IOException;
@@ -49,11 +49,11 @@
import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.util.Timestamps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.util.Timestamps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -231,13 +231,13 @@
}
BundleApplication primaryApplication =
BundleApplication.newBuilder()
- .setPtransformId(context.ptransformId)
+ .setTransformId(context.ptransformId)
.setInputId(mainInputId)
.setElement(primaryBytes.toByteString())
.build();
BundleApplication residualApplication =
BundleApplication.newBuilder()
- .setPtransformId(context.ptransformId)
+ .setTransformId(context.ptransformId)
.setInputId(mainInputId)
.setElement(residualBytes.toByteString())
.build();
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
index 396fa1e..7fd71da 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMappingFnRunner.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Represents mapping of main input window onto side input window.
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java
index 904018d..d97f8de 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/WindowMergingFnRunner.java
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.transforms.windowing.WindowFn.MergeContext;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* Merges windows using a {@link org.apache.beam.sdk.transforms.windowing.WindowFn}.
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
index 04da2d7..7fec44e 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.fn.harness.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ClientInterceptor;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Metadata.Key;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.MetadataUtils;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ClientInterceptor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Metadata.Key;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.MetadataUtils;
/** A {@link ClientInterceptor} that attaches a provided SDK Harness ID to outgoing messages. */
public class AddHarnessIdInterceptor {
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java
index 9804aa7..a6a0211 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BeamFnControlClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import java.util.EnumMap;
import java.util.Objects;
@@ -32,9 +32,9 @@
import org.apache.beam.sdk.fn.channel.ManagedChannelFactory;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.function.ThrowingFunction;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
index 038c45b..8b1c5fd 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
@@ -59,16 +59,16 @@
import org.apache.beam.sdk.function.ThrowingRunnable;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.util.common.ReflectHelpers;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Message;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.TextFormat;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.SetMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.SetMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -83,8 +83,8 @@
public class ProcessBundleHandler {
// TODO: What should the initial set of URNs be?
- private static final String DATA_INPUT_URN = "urn:org.apache.beam:source:runner:0.1";
- public static final String JAVA_SOURCE_URN = "urn:org.apache.beam:source:java:0.1";
+ private static final String DATA_INPUT_URN = "beam:source:runner:0.1";
+ public static final String JAVA_SOURCE_URN = "beam:source:java:0.1";
private static final Logger LOG = LoggerFactory.getLogger(ProcessBundleHandler.class);
private static final Map<String, PTransformRunnerFactory> REGISTERED_RUNNER_FACTORIES;
@@ -220,7 +220,7 @@
// process() calls will execute on this thread when queueingClient.drainAndBlock() is called.
QueueingBeamFnDataClient queueingClient = new QueueingBeamFnDataClient(this.beamFnDataClient);
- String bundleId = request.getProcessBundle().getProcessBundleDescriptorReference();
+ String bundleId = request.getProcessBundle().getProcessBundleDescriptorId();
BeamFnApi.ProcessBundleDescriptor bundleDescriptor =
(BeamFnApi.ProcessBundleDescriptor) fnApiRegistry.apply(bundleId);
@@ -264,13 +264,13 @@
// Reset primaries and accumulate residuals.
Multimap<String, BundleApplication> newPrimaries = ArrayListMultimap.create();
for (BundleApplication primary : primaries) {
- newPrimaries.put(primary.getPtransformId(), primary);
+ newPrimaries.put(primary.getTransformId(), primary);
}
allPrimaries.clear();
allPrimaries.putAll(newPrimaries);
for (DelayedBundleApplication residual : residuals) {
- allResiduals.put(residual.getApplication().getPtransformId(), residual);
+ allResiduals.put(residual.getApplication().getTransformId(), residual);
}
};
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/RegisterHandler.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/RegisterHandler.java
index 2ac4f1b..6a02c7d 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/RegisterHandler.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/RegisterHandler.java
@@ -25,7 +25,7 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.RegisterResponse;
import org.apache.beam.model.pipeline.v1.RunnerApi;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Message;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataClient.java
index 5f06f89..26b37f4 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataClient.java
@@ -24,7 +24,6 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
-import org.apache.beam.sdk.util.WindowedValue;
/**
* The {@link BeamFnDataClient} is able to forward inbound elements to a {@link FnDataReceiver} and
@@ -45,8 +44,8 @@
<T> InboundDataClient receive(
ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> receiver);
+ Coder<T> coder,
+ FnDataReceiver<T> receiver);
/**
* Creates a {@link CloseableFnDataReceiver} using the provided instruction id and target.
@@ -57,8 +56,8 @@
*
* <p>The returned closeable receiver is not thread safe.
*/
- <T> CloseableFnDataReceiver<WindowedValue<T>> send(
+ <T> CloseableFnDataReceiver<T> send(
Endpoints.ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint outputLocation,
- Coder<WindowedValue<T>> coder);
+ Coder<T> coder);
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java
index 1677ad7..375c9af 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClient.java
@@ -37,8 +37,7 @@
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -78,12 +77,12 @@
public <T> InboundDataClient receive(
ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> consumer) {
+ Coder<T> coder,
+ FnDataReceiver<T> consumer) {
LOG.debug(
- "Registering consumer for instruction {} and target {}",
+ "Registering consumer for instruction {} and transform {}",
inputLocation.getInstructionId(),
- inputLocation.getTarget());
+ inputLocation.getTransformId());
BeamFnDataGrpcMultiplexer client = getClientFor(apiServiceDescriptor);
BeamFnDataInboundObserver<T> inboundObserver =
@@ -103,16 +102,16 @@
* <p>The returned closeable consumer is not thread safe.
*/
@Override
- public <T> CloseableFnDataReceiver<WindowedValue<T>> send(
+ public <T> CloseableFnDataReceiver<T> send(
Endpoints.ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint outputLocation,
- Coder<WindowedValue<T>> coder) {
+ Coder<T> coder) {
BeamFnDataGrpcMultiplexer client = getClientFor(apiServiceDescriptor);
LOG.debug(
- "Creating output consumer for instruction {} and target {}",
+ "Creating output consumer for instruction {} and transform {}",
outputLocation.getInstructionId(),
- outputLocation.getTarget());
+ outputLocation.getTransformId());
Optional<Integer> bufferLimit = getBufferLimit(options);
if (bufferLimit.isPresent()) {
return BeamFnDataBufferingOutboundObserver.forLocationWithBufferLimit(
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiver.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiver.java
index 86fa3b3..65f75b0 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiver.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiver.java
@@ -19,7 +19,7 @@
import java.util.Collection;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* A {@link FnDataReceiver} which forwards all received inputs to a collection of {@link
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java
index 0655d9e..0bbdd67 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistry.java
@@ -32,8 +32,8 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.metrics.MetricsEnvironment;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ListMultimap;
/**
* The {@code PCollectionConsumerRegistry} is used to maintain a collection of consuming
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
index ebcb903..01cf0c7 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
@@ -29,7 +29,6 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.fn.data.LogicalEndpoint;
-import org.apache.beam.sdk.util.WindowedValue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -57,12 +56,12 @@
public <T> InboundDataClient receive(
ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint inputLocation,
- Coder<WindowedValue<T>> coder,
- FnDataReceiver<WindowedValue<T>> consumer) {
+ Coder<T> coder,
+ FnDataReceiver<T> consumer) {
LOG.debug(
- "Registering consumer for instruction {} and target {}",
+ "Registering consumer for instruction {} and transform {}",
inputLocation.getInstructionId(),
- inputLocation.getTarget());
+ inputLocation.getTransformId());
QueueingFnDataReceiver<T> queueingConsumer = new QueueingFnDataReceiver<T>(consumer);
InboundDataClient inboundDataClient =
@@ -73,7 +72,7 @@
return inboundDataClient;
}
- // Returns true if all the InboundDataClients have finished or cancelled and no WindowedValues
+ // Returns true if all the InboundDataClients have finished or cancelled and no values
// remain on the queue.
private boolean allDone() {
for (InboundDataClient inboundDataClient : inboundDataClients.keySet()) {
@@ -88,10 +87,10 @@
}
/**
- * Drains the internal queue of this class, by waiting for all WindowedValues to be passed to
- * their consumers. The thread which wishes to process() the elements should call this method, as
- * this will cause the consumers to invoke element processing. All receive() and send() calls must
- * be made prior to calling drainAndBlock, in order to properly terminate.
+ * Drains the internal queue of this class, by waiting for all values to be passed to their
+ * consumers. The thread which wishes to process() the elements should call this method, as this
+ * will cause the consumers to invoke element processing. All receive() and send() calls must be
+ * made prior to calling drainAndBlock, in order to properly terminate.
*
* <p>All {@link InboundDataClient}s will be failed if processing throws an exception.
*
@@ -117,7 +116,7 @@
}
}
} catch (Exception e) {
- LOG.error("Client failed to dequeue and process WindowedValue", e);
+ LOG.error("Client failed to dequeue and process the value", e);
for (InboundDataClient inboundDataClient : inboundDataClients.keySet()) {
inboundDataClient.fail(e);
}
@@ -127,14 +126,14 @@
}
@Override
- public <T> CloseableFnDataReceiver<WindowedValue<T>> send(
+ public <T> CloseableFnDataReceiver<T> send(
Endpoints.ApiServiceDescriptor apiServiceDescriptor,
LogicalEndpoint outputLocation,
- Coder<WindowedValue<T>> coder) {
+ Coder<T> coder) {
LOG.debug(
- "Creating output consumer for instruction {} and target {}",
+ "Creating output consumer for instruction {} and transform {}",
outputLocation.getInstructionId(),
- outputLocation.getTarget());
+ outputLocation.getTransformId());
return this.mainClient.send(apiServiceDescriptor, outputLocation, coder);
}
@@ -146,11 +145,11 @@
* {@link QueueingBeamFnDataClient#drainAndBlock} is responsible for processing values from the
* queue.
*/
- public class QueueingFnDataReceiver<T> implements FnDataReceiver<WindowedValue<T>> {
- private final FnDataReceiver<WindowedValue<T>> consumer;
+ public class QueueingFnDataReceiver<T> implements FnDataReceiver<T> {
+ private final FnDataReceiver<T> consumer;
public InboundDataClient inboundDataClient;
- public QueueingFnDataReceiver(FnDataReceiver<WindowedValue<T>> consumer) {
+ public QueueingFnDataReceiver(FnDataReceiver<T> consumer) {
this.consumer = consumer;
}
@@ -159,7 +158,7 @@
* data arrives via the QueueingBeamFnDataClient's mainClient.
*/
@Override
- public void accept(WindowedValue<T> value) throws Exception {
+ public void accept(T value) throws Exception {
try {
ConsumerAndData offering = new ConsumerAndData(this.consumer, value);
while (!queue.offer(offering, 200, TimeUnit.MILLISECONDS)) {
@@ -169,7 +168,7 @@
}
}
} catch (Exception e) {
- LOG.error("Failed to insert WindowedValue into the queue", e);
+ LOG.error("Failed to insert the value into the queue", e);
inboundDataClient.fail(e);
throw e;
}
@@ -177,10 +176,10 @@
}
static class ConsumerAndData<T> {
- public FnDataReceiver<WindowedValue<T>> consumer;
- public WindowedValue<T> data;
+ public FnDataReceiver<T> consumer;
+ public T data;
- public ConsumerAndData(FnDataReceiver<WindowedValue<T>> receiver, WindowedValue<T> data) {
+ public ConsumerAndData(FnDataReceiver<T> receiver, T data) {
this.consumer = receiver;
this.data = data;
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
index 78590c2..1941a10 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness.logging;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import java.util.ArrayList;
import java.util.Collection;
@@ -46,14 +46,14 @@
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.SdkHarnessOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Timestamp;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientResponseObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Timestamp;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientResponseObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Configures {@link java.util.logging} to send all {@link LogRecord}s via the Beam Fn Logging API.
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java
index 1dafe1b..b3e6f64 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BagUserState.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness.state;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collections;
@@ -27,8 +27,8 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.stream.DataStreams;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* An implementation of a bag user state that utilizes the Beam Fn State API to fetch, clear and
@@ -63,10 +63,10 @@
StateRequest.Builder requestBuilder = StateRequest.newBuilder();
requestBuilder
- .setInstructionReference(instructionId)
+ .setInstructionId(instructionId)
.getStateKeyBuilder()
.getBagUserStateBuilder()
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setUserStateId(stateId)
.setWindow(encodedWindow)
.setKey(encodedKey);
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCache.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCache.java
index 8726823..1619c22 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCache.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCache.java
@@ -31,8 +31,8 @@
import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
index 08eb90f..26b0dfa 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.fn.harness.state;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.ArrayList;
@@ -54,9 +54,9 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Provides access to side inputs and state via a {@link BeamFnStateClient}. */
public class FnApiStateAccessor implements SideInputReader, StateBinder {
@@ -104,7 +104,11 @@
ByteString.Output encodedKeyOut = ByteString.newOutput();
try {
- ((Coder) keyCoder).encode(((KV<?, ?>) element.getValue()).getKey(), encodedKeyOut);
+ ((Coder) keyCoder)
+ .encode(
+ ((KV<?, ?>) element.getValue()).getKey(),
+ encodedKeyOut,
+ Coder.Context.NESTED);
} catch (IOException e) {
throw new IllegalStateException(e);
}
@@ -164,7 +168,7 @@
StateKey.Builder cacheKeyBuilder = StateKey.newBuilder();
cacheKeyBuilder
.getMultimapSideInputBuilder()
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setSideInputId(tag.getId())
.setWindow(encodedWindow);
return (T)
@@ -448,7 +452,7 @@
.getBagUserStateBuilder()
.setWindow(encodedCurrentWindowSupplier.get())
.setKey(encodedCurrentKeySupplier.get())
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setUserStateId(stateId);
return builder.build();
}
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterable.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterable.java
index c4cc3af..a20abbb 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterable.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterable.java
@@ -21,7 +21,7 @@
import java.util.Iterator;
import java.util.List;
import java.util.NoSuchElementException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/**
* Converts an iterator to an iterable lazily loading values from the underlying iterator and
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java
index 7eda61b..fff60e6 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/MultimapSideInput.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.fn.stream.DataStreams;
import org.apache.beam.sdk.transforms.Materializations.MultimapView;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
/**
* An implementation of a multimap side input that utilizes the Beam Fn State API to fetch values.
@@ -67,10 +67,10 @@
}
StateRequest.Builder requestBuilder = StateRequest.newBuilder();
requestBuilder
- .setInstructionReference(instructionId)
+ .setInstructionId(instructionId)
.getStateKeyBuilder()
.getMultimapSideInputBuilder()
- .setPtransformId(ptransformId)
+ .setTransformId(ptransformId)
.setSideInputId(sideInputId)
.setWindow(encodedWindow)
.setKey(output.toByteString());
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java
index b0291e1..1ebadb5 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateFetchingIterators.java
@@ -24,8 +24,8 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateGetRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
/**
* Adapters which convert a a logical series of chunks using continuation tokens over the Beam Fn
diff --git a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactories.java b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactories.java
index fb87ce6..15b28f6 100644
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactories.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactories.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.options.ExperimentalOptions;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
/**
* Uses {@link PipelineOptions} to configure which underlying {@link StreamObserver} implementation
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java
index 0ab1e19..fc5e797 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java
@@ -51,8 +51,8 @@
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataReadRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataReadRunnerTest.java
index 447ee3b..1807810 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataReadRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataReadRunnerTest.java
@@ -62,11 +62,11 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.hamcrest.collection.IsMapContaining;
import org.junit.Before;
import org.junit.Rule;
@@ -111,8 +111,7 @@
}
}
- private static final BeamFnApi.Target INPUT_TARGET =
- BeamFnApi.Target.newBuilder().setPrimitiveTransformReference("1").setName("out").build();
+ private static final String INPUT_TRANSFORM_ID = "1";
@Rule public TestExecutorService executor = TestExecutors.from(Executors::newCachedThreadPool);
@Mock private BeamFnDataClient mockBeamFnDataClient;
@@ -175,13 +174,7 @@
verify(mockBeamFnDataClient)
.receive(
eq(PORT_SPEC.getApiServiceDescriptor()),
- eq(
- LogicalEndpoint.of(
- bundleId,
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference("pTransformId")
- .setName(Iterables.getOnlyElement(pTransform.getOutputsMap().keySet()))
- .build())),
+ eq(LogicalEndpoint.of(bundleId, pTransformId)),
eq(CODER),
consumerCaptor.capture());
@@ -211,9 +204,9 @@
AtomicReference<String> bundleId = new AtomicReference<>("0");
BeamFnDataReadRunner<String> readRunner =
new BeamFnDataReadRunner<>(
+ INPUT_TRANSFORM_ID,
RemoteGrpcPortRead.readFromPort(PORT_SPEC, "localOutput").toPTransform(),
bundleId::get,
- INPUT_TARGET,
CODER_SPEC,
COMPONENTS.getCodersMap(),
mockBeamFnDataClient,
@@ -225,7 +218,7 @@
verify(mockBeamFnDataClient)
.receive(
eq(PORT_SPEC.getApiServiceDescriptor()),
- eq(LogicalEndpoint.of(bundleId.get(), INPUT_TARGET)),
+ eq(LogicalEndpoint.of(bundleId.get(), INPUT_TRANSFORM_ID)),
eq(CODER),
consumerCaptor.capture());
@@ -258,7 +251,7 @@
verify(mockBeamFnDataClient)
.receive(
eq(PORT_SPEC.getApiServiceDescriptor()),
- eq(LogicalEndpoint.of(bundleId.get(), INPUT_TARGET)),
+ eq(LogicalEndpoint.of(bundleId.get(), INPUT_TRANSFORM_ID)),
eq(CODER),
consumerCaptor.capture());
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataWriteRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataWriteRunnerTest.java
index c2bc9d3..0c04aab 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataWriteRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BeamFnDataWriteRunnerTest.java
@@ -57,9 +57,9 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.collection.IsMapContaining;
import org.junit.Before;
import org.junit.Test;
@@ -103,8 +103,7 @@
}
}
- private static final BeamFnApi.Target OUTPUT_TARGET =
- BeamFnApi.Target.newBuilder().setPrimitiveTransformReference("1").setName("out").build();
+ private static final String TRANSFORM_ID = "1";
@Mock private BeamFnDataClient mockBeamFnDataClient;
@@ -136,7 +135,7 @@
PipelineOptionsFactory.create(),
mockBeamFnDataClient,
null /* beamFnStateClient */,
- "ptransformId",
+ TRANSFORM_ID,
pTransform,
Suppliers.ofInstance(bundleId)::get,
ImmutableMap.of(
@@ -176,15 +175,7 @@
verify(mockBeamFnDataClient)
.send(
eq(PORT_SPEC.getApiServiceDescriptor()),
- eq(
- LogicalEndpoint.of(
- bundleId,
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference("ptransformId")
- // The local input name is arbitrary, so use whatever the
- // RemoteGrpcPortWrite uses
- .setName(Iterables.getOnlyElement(pTransform.getInputsMap().keySet()))
- .build())),
+ eq(LogicalEndpoint.of(bundleId, TRANSFORM_ID)),
eq(WIRE_CODER));
assertThat(consumers.keySet(), containsInAnyOrder(localInputId));
@@ -209,9 +200,9 @@
AtomicReference<String> bundleId = new AtomicReference<>("0");
BeamFnDataWriteRunner<String> writeRunner =
new BeamFnDataWriteRunner<>(
+ TRANSFORM_ID,
RemoteGrpcPortWrite.writeToPort("myWrite", PORT_SPEC).toPTransform(),
bundleId::get,
- OUTPUT_TARGET,
WIRE_CODER_SPEC,
COMPONENTS.getCodersMap(),
mockBeamFnDataClient);
@@ -222,7 +213,7 @@
verify(mockBeamFnDataClient)
.send(
eq(PORT_SPEC.getApiServiceDescriptor()),
- eq(LogicalEndpoint.of(bundleId.get(), OUTPUT_TARGET)),
+ eq(LogicalEndpoint.of(bundleId.get(), TRANSFORM_ID)),
eq(WIRE_CODER));
writeRunner.consume(valueInGlobalWindow("ABC"));
@@ -241,7 +232,7 @@
verify(mockBeamFnDataClient)
.send(
eq(PORT_SPEC.getApiServiceDescriptor()),
- eq(LogicalEndpoint.of(bundleId.get(), OUTPUT_TARGET)),
+ eq(LogicalEndpoint.of(bundleId.get(), TRANSFORM_ID)),
eq(WIRE_CODER));
writeRunner.consume(valueInGlobalWindow("GHI"));
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BoundedSourceRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BoundedSourceRunnerTest.java
index 68f6bc6..e5252d6 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BoundedSourceRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/BoundedSourceRunnerTest.java
@@ -43,10 +43,10 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.Matchers;
import org.hamcrest.collection.IsMapContaining;
import org.junit.Test;
@@ -57,7 +57,7 @@
@RunWith(JUnit4.class)
public class BoundedSourceRunnerTest {
- public static final String URN = "urn:org.apache.beam:source:java:0.1";
+ public static final String URN = "beam:source:java:0.1";
@Test
public void testRunReadLoopWithMultipleSources() throws Exception {
@@ -144,7 +144,7 @@
RunnerApi.FunctionSpec functionSpec =
RunnerApi.FunctionSpec.newBuilder()
- .setUrn("urn:org.apache.beam:source:java:0.1")
+ .setUrn("beam:source:java:0.1")
.setPayload(
ByteString.copyFrom(SerializableUtils.serializeToByteArray(CountingSource.upTo(3))))
.build();
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/CombineRunnersTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/CombineRunnersTest.java
index e4fd038..f7adb6c 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/CombineRunnersTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/CombineRunnersTest.java
@@ -48,7 +48,7 @@
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FlattenRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FlattenRunnerTest.java
index ae321f8..cb43c44 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FlattenRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FlattenRunnerTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java
index db4751f..7a14b38 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnApiDoFnRunnerTest.java
@@ -89,10 +89,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.hamcrest.collection.IsMapContaining;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -111,7 +111,7 @@
private static final Logger LOG = LoggerFactory.getLogger(FnApiDoFnRunnerTest.class);
- public static final String TEST_PTRANSFORM_ID = "pTransformId";
+ public static final String TEST_TRANSFORM_ID = "pTransformId";
private static class ConcatCombineFn extends CombineFn<String, String, String> {
@Override
@@ -177,7 +177,7 @@
PCollection<KV<String, String>> valuePCollection =
p.apply(Create.of(KV.of("unused", "unused")));
PCollection<String> outputPCollection =
- valuePCollection.apply(TEST_PTRANSFORM_ID, ParDo.of(new TestStatefulDoFn()));
+ valuePCollection.apply(TEST_TRANSFORM_ID, ParDo.of(new TestStatefulDoFn()));
SdkComponents sdkComponents = SdkComponents.create(p.getOptions());
RunnerApi.Pipeline pProto = PipelineTranslation.toProto(p, sdkComponents);
@@ -187,10 +187,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto
- .getComponents()
- .getTransformsOrThrow(TEST_PTRANSFORM_ID)
- .getSubtransforms(0));
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0));
FakeBeamFnStateClient fakeClient =
new FakeBeamFnStateClient(
@@ -206,7 +203,7 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
outputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) mainOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -220,7 +217,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -282,7 +279,7 @@
return StateKey.newBuilder()
.setBagUserState(
StateKey.BagUserState.newBuilder()
- .setPtransformId(TEST_PTRANSFORM_ID)
+ .setTransformId(TEST_TRANSFORM_ID)
.setUserStateId(userStateId)
.setKey(encode(key))
.setWindow(
@@ -334,7 +331,7 @@
TupleTag<String> additionalOutput = new TupleTag<String>("additional") {};
PCollectionTuple outputPCollection =
valuePCollection.apply(
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
ParDo.of(
new TestSideInputDoFn(
defaultSingletonSideInputView,
@@ -354,7 +351,7 @@
sdkComponents.registerPCollection(outputPCollection.get(additionalOutput));
RunnerApi.PTransform pTransform =
- pProto.getComponents().getTransformsOrThrow(TEST_PTRANSFORM_ID);
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID);
ImmutableMap<StateKey, ByteString> stateData =
ImmutableMap.of(
@@ -373,11 +370,11 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
outputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) mainOutputValues::add);
consumers.register(
additionalPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) additionalOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -391,7 +388,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -478,7 +475,7 @@
valuePCollection.apply(View.asIterable());
PCollection<Iterable<String>> outputPCollection =
valuePCollection.apply(
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
ParDo.of(new TestSideInputIsAccessibleForDownstreamCallersDoFn(iterableSideInputView))
.withSideInputs(iterableSideInputView));
@@ -491,10 +488,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto
- .getComponents()
- .getTransformsOrThrow(TEST_PTRANSFORM_ID)
- .getSubtransforms(0));
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0));
ImmutableMap<StateKey, ByteString> stateData =
ImmutableMap.of(
@@ -514,7 +508,7 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
Iterables.getOnlyElement(pTransform.getOutputsMap().values()),
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<Iterable<String>>>) mainOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -528,7 +522,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -587,7 +581,7 @@
valuePCollection.apply(View.asIterable());
PCollection<Iterable<String>> outputPCollection =
valuePCollection.apply(
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
ParDo.of(new TestSideInputIsAccessibleForDownstreamCallersDoFn(iterableSideInputView))
.withSideInputs(iterableSideInputView));
@@ -600,10 +594,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto
- .getComponents()
- .getTransformsOrThrow(TEST_PTRANSFORM_ID)
- .getSubtransforms(0));
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0));
ImmutableMap<StateKey, ByteString> stateData =
ImmutableMap.of(
@@ -623,7 +614,7 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
Iterables.getOnlyElement(pTransform.getOutputsMap().values()),
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<Iterable<String>>>) mainOutputValues::add);
PTransformFunctionRegistry startFunctionRegistry =
new PTransformFunctionRegistry(
@@ -637,7 +628,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
pProto.getComponents().getPcollectionsMap(),
@@ -686,7 +677,7 @@
.setLabel(
MonitoringInfoConstants.Labels.NAME,
TestSideInputIsAccessibleForDownstreamCallersDoFn.USER_COUNTER_NAME);
- builder.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, TEST_PTRANSFORM_ID);
+ builder.setLabel(MonitoringInfoConstants.Labels.PTRANSFORM, TEST_TRANSFORM_ID);
builder.setInt64Value(2);
expected.add(builder.build());
@@ -756,7 +747,7 @@
PCollection<KV<String, String>> valuePCollection =
p.apply(Create.of(KV.of("unused", "unused")));
PCollection<String> outputPCollection =
- valuePCollection.apply(TEST_PTRANSFORM_ID, ParDo.of(new TestTimerfulDoFn()));
+ valuePCollection.apply(TEST_TRANSFORM_ID, ParDo.of(new TestTimerfulDoFn()));
SdkComponents sdkComponents = SdkComponents.create();
sdkComponents.registerEnvironment(Environment.getDefaultInstance());
@@ -776,7 +767,7 @@
pProto
.getComponents()
.getTransformsOrThrow(
- pProto.getComponents().getTransformsOrThrow(TEST_PTRANSFORM_ID).getSubtransforms(0))
+ pProto.getComponents().getTransformsOrThrow(TEST_TRANSFORM_ID).getSubtransforms(0))
.toBuilder()
// We need to re-write the "output" PCollections that a runner would have inserted
// on the way to a output sink.
@@ -800,16 +791,16 @@
metricsContainerRegistry, mock(ExecutionStateTracker.class));
consumers.register(
outputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver) (FnDataReceiver<WindowedValue<String>>) mainOutputValues::add);
consumers.register(
eventTimerOutputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver)
(FnDataReceiver<WindowedValue<KV<String, Timer>>>) eventTimerOutputValues::add);
consumers.register(
processingTimerOutputPCollectionId,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
(FnDataReceiver)
(FnDataReceiver<WindowedValue<KV<String, Timer>>>) processingTimerOutputValues::add);
@@ -825,7 +816,7 @@
PipelineOptionsFactory.create(),
null /* beamFnDataClient */,
fakeClient,
- TEST_PTRANSFORM_ID,
+ TEST_TRANSFORM_ID,
pTransform,
Suppliers.ofInstance("57L")::get,
ImmutableMap.<String, RunnerApi.PCollection>builder()
@@ -967,7 +958,7 @@
return StateKey.newBuilder()
.setMultimapSideInput(
StateKey.MultimapSideInput.newBuilder()
- .setPtransformId(TEST_PTRANSFORM_ID)
+ .setTransformId(TEST_TRANSFORM_ID)
.setSideInputId(sideInputId)
.setKey(key)
.setWindow(windowKey))
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnHarnessTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnHarnessTest.java
index 48f7213..eaf0b07 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnHarnessTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/FnHarnessTest.java
@@ -17,12 +17,18 @@
*/
package org.apache.beam.fn.harness;
-import static org.hamcrest.Matchers.contains;
-import static org.junit.Assert.assertThat;
+import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.atLeastOnce;
+import static org.mockito.Mockito.inOrder;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import com.google.auto.service.AutoService;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CountDownLatch;
+import java.util.function.Consumer;
+import java.util.function.Function;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse;
@@ -30,17 +36,22 @@
import org.apache.beam.model.fnexecution.v1.BeamFnControlGrpc;
import org.apache.beam.model.fnexecution.v1.BeamFnLoggingGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
import org.apache.beam.sdk.fn.test.TestStreams;
+import org.apache.beam.sdk.harness.JvmInitializer;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.TextFormat;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.mockito.InOrder;
+import org.mockito.Mock;
/** Tests for {@link FnHarness}. */
@RunWith(JUnit4.class)
@@ -56,13 +67,39 @@
.setRegister(BeamFnApi.RegisterResponse.getDefaultInstance())
.build();
+ private static @Mock Runnable onStartupMock = mock(Runnable.class);
+ private static @Mock Consumer<PipelineOptions> beforeProcessingMock = mock(Consumer.class);
+
+ /**
+ * Fake JvmInitializer that simply forwards calls to mocked functions so that they can be observed
+ * in tests.
+ */
+ @AutoService(JvmInitializer.class)
+ public static class FnHarnessTestInitializer implements JvmInitializer {
+ @Override
+ public void onStartup() {
+ onStartupMock.run();
+ }
+
+ @Override
+ public void beforeProcessing(PipelineOptions options) {
+ beforeProcessingMock.accept(options);
+ }
+ }
+
@Test(timeout = 10 * 1000)
@SuppressWarnings("FutureReturnValueIgnored") // failure will cause test to timeout.
public void testLaunchFnHarnessAndTeardownCleanly() throws Exception {
+ Function<String, String> environmentVariableMock = mock(Function.class);
+
PipelineOptions options = PipelineOptionsFactory.create();
+ when(environmentVariableMock.apply("HARNESS_ID")).thenReturn("id");
+ when(environmentVariableMock.apply("PIPELINE_OPTIONS"))
+ .thenReturn(PipelineOptionsTranslation.toJson(options));
+
List<BeamFnApi.LogEntry> logEntries = new ArrayList<>();
- List<BeamFnApi.InstructionResponse> instructionResponses = new ArrayList<>();
+ List<BeamFnApi.InstructionResponse> instructionResponses = mock(List.class);
BeamFnLoggingGrpc.BeamFnLoggingImplBase loggingService =
new BeamFnLoggingGrpc.BeamFnLoggingImplBase() {
@@ -108,6 +145,7 @@
try {
Server controlServer = ServerBuilder.forPort(0).addService(controlService).build();
controlServer.start();
+
try {
Endpoints.ApiServiceDescriptor loggingDescriptor =
Endpoints.ApiServiceDescriptor.newBuilder()
@@ -118,13 +156,26 @@
.setUrl("localhost:" + controlServer.getPort())
.build();
- FnHarness.main("id", options, loggingDescriptor, controlDescriptor);
- assertThat(instructionResponses, contains(INSTRUCTION_RESPONSE));
+ when(environmentVariableMock.apply("LOGGING_API_SERVICE_DESCRIPTOR"))
+ .thenReturn(TextFormat.printToString(loggingDescriptor));
+ when(environmentVariableMock.apply("CONTROL_API_SERVICE_DESCRIPTOR"))
+ .thenReturn(TextFormat.printToString(controlDescriptor));
+
+ FnHarness.main(environmentVariableMock);
} finally {
controlServer.shutdownNow();
}
} finally {
loggingServer.shutdownNow();
}
+
+ // Verify that we first run onStartup functions before even reading the environment, and that
+ // we then call beforeProcessing functions before executing instructions.
+ InOrder inOrder =
+ inOrder(onStartupMock, beforeProcessingMock, environmentVariableMock, instructionResponses);
+ inOrder.verify(onStartupMock).run();
+ inOrder.verify(environmentVariableMock, atLeastOnce()).apply(any());
+ inOrder.verify(beforeProcessingMock).accept(any());
+ inOrder.verify(instructionResponses).add(INSTRUCTION_RESPONSE);
}
}
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/MapFnRunnersTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/MapFnRunnersTest.java
index 09a255f..c408ee5 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/MapFnRunnersTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/MapFnRunnersTest.java
@@ -41,8 +41,8 @@
import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/WindowMergingFnRunnerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/WindowMergingFnRunnerTest.java
index 35e9aa5..b936cfb 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/WindowMergingFnRunnerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/WindowMergingFnRunnerTest.java
@@ -33,9 +33,9 @@
import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Test;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/BeamFnControlClientTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/BeamFnControlClientTest.java
index f30cc6a..36a4779 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/BeamFnControlClientTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/BeamFnControlClientTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness.control;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.not;
import static org.hamcrest.Matchers.nullValue;
@@ -43,11 +43,11 @@
import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
import org.apache.beam.sdk.fn.test.TestStreams;
import org.apache.beam.sdk.function.ThrowingFunction;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
index 0514191..54c1d1e 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/ProcessBundleHandlerTest.java
@@ -51,9 +51,9 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Message;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Message;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -69,8 +69,8 @@
/** Tests for {@link ProcessBundleHandler}. */
@RunWith(JUnit4.class)
public class ProcessBundleHandlerTest {
- private static final String DATA_INPUT_URN = "urn:org.apache.beam:source:runner:0.1";
- private static final String DATA_OUTPUT_URN = "urn:org.apache.beam:sink:runner:0.1";
+ private static final String DATA_INPUT_URN = "beam:source:runner:0.1";
+ private static final String DATA_OUTPUT_URN = "beam:sink:runner:0.1";
@Rule public ExpectedException thrown = ExpectedException.none();
@@ -143,8 +143,7 @@
BeamFnApi.InstructionRequest.newBuilder()
.setInstructionId("999L")
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
// Processing of transforms is performed in reverse order.
@@ -197,8 +196,7 @@
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
@@ -245,8 +243,7 @@
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
@@ -293,8 +290,7 @@
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
@@ -331,7 +327,7 @@
// Simulate sleeping which introduces a race which most of the time requires
// the ProcessBundleHandler to block.
Uninterruptibles.sleepUninterruptibly(500, TimeUnit.MILLISECONDS);
- switch (stateRequestBuilder.getInstructionReference()) {
+ switch (stateRequestBuilder.getInstructionId()) {
case "SUCCESS":
completableFuture.complete(StateResponse.getDefaultInstance());
break;
@@ -378,18 +374,15 @@
private void doStateCalls(BeamFnStateClient beamFnStateClient) {
beamFnStateClient.handle(
- StateRequest.newBuilder().setInstructionReference("SUCCESS"),
- successfulResponse);
+ StateRequest.newBuilder().setInstructionId("SUCCESS"), successfulResponse);
beamFnStateClient.handle(
- StateRequest.newBuilder().setInstructionReference("FAIL"),
- unsuccessfulResponse);
+ StateRequest.newBuilder().setInstructionId("FAIL"), unsuccessfulResponse);
}
}));
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
assertTrue(successfulResponse.isDone());
@@ -442,15 +435,14 @@
thrown.expect(IllegalStateException.class);
thrown.expectMessage("State API calls are unsupported");
beamFnStateClient.handle(
- StateRequest.newBuilder().setInstructionReference("SUCCESS"),
+ StateRequest.newBuilder().setInstructionId("SUCCESS"),
new CompletableFuture<>());
}
}));
handler.processBundle(
BeamFnApi.InstructionRequest.newBuilder()
.setProcessBundle(
- BeamFnApi.ProcessBundleRequest.newBuilder()
- .setProcessBundleDescriptorReference("1L"))
+ BeamFnApi.ProcessBundleRequest.newBuilder().setProcessBundleDescriptorId("1L"))
.build());
}
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/RegisterHandlerTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/RegisterHandlerTest.java
index ffa01ef..e8014aa 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/RegisterHandlerTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/RegisterHandlerTest.java
@@ -48,11 +48,8 @@
"10L",
RunnerApi.Coder.newBuilder()
.setSpec(
- RunnerApi.SdkFunctionSpec.newBuilder()
- .setSpec(
- RunnerApi.FunctionSpec.newBuilder()
- .setUrn("urn:10L")
- .build())
+ RunnerApi.FunctionSpec.newBuilder()
+ .setUrn("testUrn1")
.build())
.build())
.build())
@@ -63,11 +60,8 @@
"20L",
RunnerApi.Coder.newBuilder()
.setSpec(
- RunnerApi.SdkFunctionSpec.newBuilder()
- .setSpec(
- RunnerApi.FunctionSpec.newBuilder()
- .setUrn("urn:20L")
- .build())
+ RunnerApi.FunctionSpec.newBuilder()
+ .setUrn("testUrn2")
.build())
.build())
.build())
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java
index 849c78c..6ebd961 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataGrpcClientTest.java
@@ -33,7 +33,6 @@
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.fnexecution.v1.BeamFnDataGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.sdk.coders.Coder;
@@ -47,13 +46,13 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -64,18 +63,9 @@
private static final Coder<WindowedValue<String>> CODER =
LengthPrefixCoder.of(
WindowedValue.getFullCoder(StringUtf8Coder.of(), GlobalWindow.Coder.INSTANCE));
- private static final LogicalEndpoint ENDPOINT_A =
- LogicalEndpoint.of(
- "12L",
- Target.newBuilder().setPrimitiveTransformReference("34L").setName("targetA").build());
+ private static final LogicalEndpoint ENDPOINT_A = LogicalEndpoint.of("12L", "34L");
- private static final LogicalEndpoint ENDPOINT_B =
- LogicalEndpoint.of(
- "56L",
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference("78L")
- .setName("targetB")
- .build());
+ private static final LogicalEndpoint ENDPOINT_B = LogicalEndpoint.of("56L", "78L");
private static final BeamFnApi.Elements ELEMENTS_A_1;
private static final BeamFnApi.Elements ELEMENTS_A_2;
@@ -87,8 +77,8 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setTarget(ENDPOINT_A.getTarget())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("ABC")))
.concat(
@@ -99,22 +89,22 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setTarget(ENDPOINT_A.getTarget())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(
encodeToByteArray(CODER, valueInGlobalWindow("GHI")))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setTarget(ENDPOINT_A.getTarget()))
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId()))
.build();
ELEMENTS_B_1 =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setTarget(ENDPOINT_B.getTarget())
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("JKL")))
.concat(
@@ -122,8 +112,8 @@
encodeToByteArray(CODER, valueInGlobalWindow("MNO"))))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setTarget(ENDPOINT_B.getTarget()))
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId()))
.build();
} catch (Exception e) {
throw new ExceptionInInitializerError(e);
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java
index 2338b1f..aa45df6 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/BeamFnDataInboundObserverTest.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.fn.data.InboundDataClient;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -55,7 +55,7 @@
public void testDecodingElements() throws Exception {
Collection<WindowedValue<String>> values = new ArrayList<>();
InboundDataClient readFuture = CompletableFutureInboundDataClient.create();
- BeamFnDataInboundObserver<String> observer =
+ BeamFnDataInboundObserver<WindowedValue<String>> observer =
new BeamFnDataInboundObserver<>(CODER, values::add, readFuture);
// Test decoding multiple messages
@@ -79,7 +79,7 @@
@Test
public void testConsumptionFailureCompletesReadFutureAndDiscardsMessages() throws Exception {
InboundDataClient readClient = CompletableFutureInboundDataClient.create();
- BeamFnDataInboundObserver<String> observer =
+ BeamFnDataInboundObserver<WindowedValue<String>> observer =
new BeamFnDataInboundObserver<>(CODER, this::throwOnDefValue, readClient);
assertFalse(readClient.isDone());
@@ -100,12 +100,7 @@
private BeamFnApi.Elements.Data dataWith(String... values) throws Exception {
BeamFnApi.Elements.Data.Builder builder =
- BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference("777L")
- .setTarget(
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference("999L")
- .setName("Test"));
+ BeamFnApi.Elements.Data.newBuilder().setInstructionId("777L").setTransformId("999L");
ByteString.Output output = ByteString.newOutput();
for (String value : values) {
CODER.encode(valueInGlobalWindow(value), output);
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/ElementCountFnDataReceiverTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/ElementCountFnDataReceiverTest.java
index 316caa7..ace16c6 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/ElementCountFnDataReceiverTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/ElementCountFnDataReceiverTest.java
@@ -92,7 +92,7 @@
verify(consumer, times(1)).accept(element);
// Verify that static scopedMetricsContainer is called with unbound container.
- PowerMockito.verifyStatic(times(1));
+ PowerMockito.verifyStatic(MetricsEnvironment.class, times(1));
MetricsEnvironment.scopedMetricsContainer(metricsContainerRegistry.getUnboundContainer());
}
}
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiverTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiverTest.java
index 4951dac..f3e21c2 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiverTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiverTest.java
@@ -26,7 +26,7 @@
import java.util.List;
import java.util.Set;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistryTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistryTest.java
index 2b72fe9..e8b377b 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistryTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PCollectionConsumerRegistryTest.java
@@ -122,11 +122,11 @@
wrapperConsumer.accept(element);
// Verify that static scopedMetricsContainer is called with pTransformA's container.
- PowerMockito.verifyStatic(times(1));
+ PowerMockito.verifyStatic(MetricsEnvironment.class, times(1));
MetricsEnvironment.scopedMetricsContainer(metricsContainerRegistry.getContainer("pTransformA"));
// Verify that static scopedMetricsContainer is called with pTransformB's container.
- PowerMockito.verifyStatic(times(1));
+ PowerMockito.verifyStatic(MetricsEnvironment.class, times(1));
MetricsEnvironment.scopedMetricsContainer(metricsContainerRegistry.getContainer("pTransformB"));
}
}
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PTransformFunctionRegistryTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PTransformFunctionRegistryTest.java
index 08b4160..b60ebef 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PTransformFunctionRegistryTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PTransformFunctionRegistryTest.java
@@ -74,11 +74,11 @@
}
// Verify that static scopedMetricsContainer is called with pTransformA's container.
- PowerMockito.verifyStatic(times(1));
+ PowerMockito.verifyStatic(MetricsEnvironment.class, times(1));
MetricsEnvironment.scopedMetricsContainer(metricsContainerRegistry.getContainer("pTransformA"));
// Verify that static scopedMetricsContainer is called with pTransformB's container.
- PowerMockito.verifyStatic(times(1));
+ PowerMockito.verifyStatic(MetricsEnvironment.class, times(1));
MetricsEnvironment.scopedMetricsContainer(metricsContainerRegistry.getContainer("pTransformB"));
}
}
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java
index f786a7c..8bcacfa 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClientTest.java
@@ -33,7 +33,6 @@
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.model.fnexecution.v1.BeamFnApi;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.fnexecution.v1.BeamFnDataGrpc;
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.sdk.coders.Coder;
@@ -48,13 +47,13 @@
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -73,18 +72,9 @@
private static final Coder<WindowedValue<String>> CODER =
LengthPrefixCoder.of(
WindowedValue.getFullCoder(StringUtf8Coder.of(), GlobalWindow.Coder.INSTANCE));
- private static final LogicalEndpoint ENDPOINT_A =
- LogicalEndpoint.of(
- "12L",
- Target.newBuilder().setPrimitiveTransformReference("34L").setName("targetA").build());
+ private static final LogicalEndpoint ENDPOINT_A = LogicalEndpoint.of("12L", "34L");
- private static final LogicalEndpoint ENDPOINT_B =
- LogicalEndpoint.of(
- "56L",
- BeamFnApi.Target.newBuilder()
- .setPrimitiveTransformReference("78L")
- .setName("targetB")
- .build());
+ private static final LogicalEndpoint ENDPOINT_B = LogicalEndpoint.of("56L", "78L");
private static final BeamFnApi.Elements ELEMENTS_A_1;
private static final BeamFnApi.Elements ELEMENTS_A_2;
@@ -96,8 +86,8 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setTarget(ENDPOINT_A.getTarget())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("ABC")))
.concat(
@@ -108,22 +98,22 @@
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setTarget(ENDPOINT_A.getTarget())
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId())
.setData(
ByteString.copyFrom(
encodeToByteArray(CODER, valueInGlobalWindow("GHI")))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_A.getInstructionId())
- .setTarget(ENDPOINT_A.getTarget()))
+ .setInstructionId(ENDPOINT_A.getInstructionId())
+ .setTransformId(ENDPOINT_A.getTransformId()))
.build();
ELEMENTS_B_1 =
BeamFnApi.Elements.newBuilder()
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setTarget(ENDPOINT_B.getTarget())
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId())
.setData(
ByteString.copyFrom(encodeToByteArray(CODER, valueInGlobalWindow("JKL")))
.concat(
@@ -131,8 +121,8 @@
encodeToByteArray(CODER, valueInGlobalWindow("MNO"))))))
.addData(
BeamFnApi.Elements.Data.newBuilder()
- .setInstructionReference(ENDPOINT_B.getInstructionId())
- .setTarget(ENDPOINT_B.getTarget()))
+ .setInstructionId(ENDPOINT_B.getInstructionId())
+ .setTransformId(ENDPOINT_B.getTransformId()))
.build();
} catch (Exception e) {
throw new ExceptionInInitializerError(e);
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClientTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClientTest.java
index 448c7dc..e3a4266 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClientTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClientTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.fn.harness.logging;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables.getStackTraceAsString;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables.getStackTraceAsString;
import static org.hamcrest.Matchers.contains;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
@@ -37,14 +37,14 @@
import org.apache.beam.model.pipeline.v1.Endpoints;
import org.apache.beam.sdk.fn.test.TestStreams;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.Timestamp;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.Timestamp;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java
index 1592ec4..5b01c0f 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BagUserStateTest.java
@@ -25,9 +25,9 @@
import java.io.IOException;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateKey;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -119,7 +119,7 @@
return StateKey.newBuilder()
.setBagUserState(
StateKey.BagUserState.newBuilder()
- .setPtransformId("ptransformId")
+ .setTransformId("ptransformId")
.setUserStateId("stateId")
.setWindow(ByteString.copyFromUtf8("encodedWindow"))
.setKey(encode(id)))
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java
index c966c71..e1feac1 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/BeamFnStateGrpcClientCacheTest.java
@@ -36,15 +36,15 @@
import org.apache.beam.sdk.fn.IdGenerators;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.fn.test.TestStreams;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ManagedChannel;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Server;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessChannelBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.inprocess.InProcessServerBuilder;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ManagedChannel;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessChannelBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.inprocess.InProcessServerBuilder;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
@@ -123,8 +123,8 @@
CompletableFuture<StateResponse> successfulResponse = new CompletableFuture<>();
CompletableFuture<StateResponse> unsuccessfulResponse = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), successfulResponse);
- client.handle(StateRequest.newBuilder().setInstructionReference(FAIL), unsuccessfulResponse);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), successfulResponse);
+ client.handle(StateRequest.newBuilder().setInstructionId(FAIL), unsuccessfulResponse);
// Wait for the client to connect.
StreamObserver<StateResponse> outboundServerObserver = outboundServerObservers.take();
@@ -150,7 +150,7 @@
BeamFnStateClient client = clientCache.forApiServiceDescriptor(apiServiceDescriptor);
CompletableFuture<StateResponse> inflight = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), inflight);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), inflight);
// Wait for the client to connect.
StreamObserver<StateResponse> outboundServerObserver = outboundServerObservers.take();
@@ -167,7 +167,7 @@
// Send a response after the client will have received an error.
CompletableFuture<StateResponse> late = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), late);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), late);
try {
inflight.get();
@@ -182,7 +182,7 @@
BeamFnStateClient client = clientCache.forApiServiceDescriptor(apiServiceDescriptor);
CompletableFuture<StateResponse> inflight = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), inflight);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), inflight);
// Wait for the client to connect.
StreamObserver<StateResponse> outboundServerObserver = outboundServerObservers.take();
@@ -198,7 +198,7 @@
// Send a response after the client will have received an error.
CompletableFuture<StateResponse> late = new CompletableFuture<>();
- client.handle(StateRequest.newBuilder().setInstructionReference(SUCCESS), late);
+ client.handle(StateRequest.newBuilder().setInstructionId(SUCCESS), late);
try {
inflight.get();
@@ -210,7 +210,7 @@
private void handleServerRequest(
StreamObserver<StateResponse> outboundObserver, StateRequest value) {
- switch (value.getInstructionReference()) {
+ switch (value.getInstructionId()) {
case SUCCESS:
outboundObserver.onNext(StateResponse.newBuilder().setId(value.getId()).build());
return;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java
index f123c0d..7762e66 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/FakeBeamFnStateClient.java
@@ -32,7 +32,7 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest.RequestCase;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
/** A fake implementation of a {@link BeamFnStateClient} to aid with testing. */
public class FakeBeamFnStateClient implements BeamFnStateClient {
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterableTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterableTest.java
index 78cb6ff..7597128 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterableTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/LazyCachingIteratorToIterableTest.java
@@ -25,8 +25,8 @@
import java.util.Iterator;
import java.util.NoSuchElementException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java
index c9fe4ab..9705267 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/MultimapSideInputTest.java
@@ -22,9 +22,9 @@
import java.io.IOException;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateKey;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -62,7 +62,7 @@
return StateKey.newBuilder()
.setMultimapSideInput(
StateKey.MultimapSideInput.newBuilder()
- .setPtransformId("ptransformId")
+ .setTransformId("ptransformId")
.setSideInputId("sideInputId")
.setWindow(ByteString.copyFromUtf8("encodedWindow"))
.setKey(encode(id)))
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/StateFetchingIteratorsTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/StateFetchingIteratorsTest.java
index d30aa31..630627d 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/StateFetchingIteratorsTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/StateFetchingIteratorsTest.java
@@ -24,8 +24,8 @@
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateGetResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactoriesTest.java b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactoriesTest.java
index d4b877f..d8f5872 100644
--- a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactoriesTest.java
+++ b/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/stream/HarnessStreamObserverFactoriesTest.java
@@ -25,8 +25,8 @@
import org.apache.beam.sdk.fn.stream.DirectStreamObserver;
import org.apache.beam.sdk.fn.stream.ForwardingClientResponseObserver;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.CallStreamObserver;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.CallStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/io/amazon-web-services/build.gradle b/sdks/java/io/amazon-web-services/build.gradle
index c39ca4d..ca88447 100644
--- a/sdks/java/io/amazon-web-services/build.gradle
+++ b/sdks/java/io/amazon-web-services/build.gradle
@@ -19,37 +19,38 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.aws')
description = "Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services"
ext.summary = "IO library to read and write Amazon Web Services services from Beam."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.aws_java_sdk_cloudwatch
- shadow library.java.aws_java_sdk_core
- shadow library.java.aws_java_sdk_s3
- shadow library.java.aws_java_sdk_sns
- shadow library.java.aws_java_sdk_sqs
- shadow "commons-lang:commons-lang:2.6"
- shadow library.java.jackson_core
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
- shadow library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.aws_java_sdk_cloudwatch
+ compile library.java.aws_java_sdk_core
+ compile library.java.aws_java_sdk_dynamodb
+ compile library.java.aws_java_sdk_s3
+ compile library.java.aws_java_sdk_sns
+ compile library.java.aws_java_sdk_sqs
+ compile "commons-lang:commons-lang:2.6"
+ compile library.java.jackson_core
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
+ compile library.java.slf4j_api
runtime 'commons-codec:commons-codec:1.9'
runtime "org.apache.httpcomponents:httpclient:4.5.6"
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile "io.findify:s3mock_2.12:0.2.4"
- shadowTest library.java.guava_testlib
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.mockito_core
- shadowTest library.java.junit
- shadowTest group: 'org.elasticmq', name: 'elasticmq-rest-sqs_2.12', version: '0.14.1'
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.mockito_core
+ testCompile library.java.junit
+ testCompile 'org.elasticmq:elasticmq-rest-sqs_2.12:0.14.1'
+ testCompile 'org.testcontainers:localstack:1.11.2'
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
test {
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java
new file mode 100644
index 0000000..4bdf8b5
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+
+/** A {@link Coder} that serializes and deserializes the {@link AttributeValue} objects. */
+public class AttributeValueCoder extends AtomicCoder<AttributeValue> {
+
+ /** Data type of each value type in AttributeValue object. */
+ private enum AttributeValueType {
+ s, // for String
+ n, // for Number
+ b, // for Byte
+ sS, // for List of String
+ nS, // for List of Number
+ bS, // for List of Byte
+ m, // for Map of String and AttributeValue
+ l, // for list of AttributeValue
+ bOOL, // for Boolean
+ nULLValue, // for null
+ }
+
+ private static final AttributeValueCoder INSTANCE = new AttributeValueCoder();
+
+ private static final ListCoder<String> LIST_STRING_CODER = ListCoder.of(StringUtf8Coder.of());
+ private static final ListCoder<byte[]> LIST_BYTE_CODER = ListCoder.of(ByteArrayCoder.of());
+
+ private static final ListCoder<AttributeValue> LIST_ATTRIBUTE_CODER =
+ ListCoder.of(AttributeValueCoder.of());
+ private static final MapCoder<String, AttributeValue> MAP_ATTRIBUTE_CODER =
+ MapCoder.of(StringUtf8Coder.of(), AttributeValueCoder.of());
+
+ private AttributeValueCoder() {}
+
+ public static AttributeValueCoder of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(AttributeValue value, OutputStream outStream) throws IOException {
+
+ if (value.getS() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.s.toString(), outStream);
+ StringUtf8Coder.of().encode(value.getS(), outStream);
+ } else if (value.getN() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.n.toString(), outStream);
+ StringUtf8Coder.of().encode(value.getN(), outStream);
+ } else if (value.getBOOL() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.bOOL.toString(), outStream);
+ BooleanCoder.of().encode(value.getBOOL(), outStream);
+ } else if (value.getB() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.b.toString(), outStream);
+ ByteArrayCoder.of().encode(convertToByteArray(value.getB()), outStream);
+ } else if (value.getSS() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.sS.toString(), outStream);
+ LIST_STRING_CODER.encode(value.getSS(), outStream);
+ } else if (value.getNS() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.nS.toString(), outStream);
+ LIST_STRING_CODER.encode(value.getNS(), outStream);
+ } else if (value.getBS() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.bS.toString(), outStream);
+ LIST_BYTE_CODER.encode(convertToListByteArray(value.getBS()), outStream);
+ } else if (value.getL() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.l.toString(), outStream);
+ LIST_ATTRIBUTE_CODER.encode(value.getL(), outStream);
+ } else if (value.getM() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.m.toString(), outStream);
+ MAP_ATTRIBUTE_CODER.encode(value.getM(), outStream);
+ } else if (value.getNULL() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.nULLValue.toString(), outStream);
+ BooleanCoder.of().encode(value.getNULL(), outStream);
+ } else {
+ throw new CoderException("Unknown Type");
+ }
+ }
+
+ @Override
+ public AttributeValue decode(InputStream inStream) throws IOException {
+ AttributeValue attrValue = new AttributeValue();
+
+ String type = StringUtf8Coder.of().decode(inStream);
+ AttributeValueType attrType = AttributeValueType.valueOf(type);
+
+ switch (attrType) {
+ case s:
+ attrValue.setS(StringUtf8Coder.of().decode(inStream));
+ break;
+ case n:
+ attrValue.setN(StringUtf8Coder.of().decode(inStream));
+ break;
+ case bOOL:
+ attrValue.setBOOL(BooleanCoder.of().decode(inStream));
+ break;
+ case b:
+ attrValue.setB(ByteBuffer.wrap(ByteArrayCoder.of().decode(inStream)));
+ break;
+ case sS:
+ attrValue.setSS(LIST_STRING_CODER.decode(inStream));
+ break;
+ case nS:
+ attrValue.setNS(LIST_STRING_CODER.decode(inStream));
+ break;
+ case bS:
+ attrValue.setBS(convertToListByteBuffer(LIST_BYTE_CODER.decode(inStream)));
+ break;
+ case l:
+ attrValue.setL(LIST_ATTRIBUTE_CODER.decode(inStream));
+ break;
+ case m:
+ attrValue.setM(MAP_ATTRIBUTE_CODER.decode(inStream));
+ break;
+ case nULLValue:
+ attrValue.setNULL(BooleanCoder.of().decode(inStream));
+ break;
+ default:
+ throw new CoderException("Unknown Type");
+ }
+
+ return attrValue;
+ }
+
+ private List<byte[]> convertToListByteArray(List<ByteBuffer> listByteBuffer) {
+ return listByteBuffer.stream().map(this::convertToByteArray).collect(Collectors.toList());
+ }
+
+ private byte[] convertToByteArray(ByteBuffer buffer) {
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.get(bytes);
+ buffer.position(buffer.position() - bytes.length);
+ return bytes;
+ }
+
+ private List<ByteBuffer> convertToListByteBuffer(List<byte[]> listByteArr) {
+ return listByteArr.stream().map(ByteBuffer::wrap).collect(Collectors.toList());
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderProviderRegistrar.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderProviderRegistrar.java
new file mode 100644
index 0000000..ff3f633
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderProviderRegistrar.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import com.google.auto.service.AutoService;
+import java.util.List;
+import org.apache.beam.sdk.coders.CoderProvider;
+import org.apache.beam.sdk.coders.CoderProviderRegistrar;
+import org.apache.beam.sdk.coders.CoderProviders;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** A {@link CoderProviderRegistrar} for standard types used with {@link DynamoDBIO}. */
+@AutoService(CoderProviderRegistrar.class)
+public class AttributeValueCoderProviderRegistrar implements CoderProviderRegistrar {
+ @Override
+ public List<CoderProvider> getCoderProviders() {
+ return ImmutableList.of(
+ CoderProviders.forCoder(TypeDescriptor.of(AttributeValue.class), AttributeValueCoder.of()));
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProvider.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProvider.java
new file mode 100644
index 0000000..8d1c267
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProvider.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import java.io.Serializable;
+
+/**
+ * Provides instances of AWS clients.
+ *
+ * <p>Please note, that any instance of {@link AwsClientsProvider} must be {@link Serializable} to
+ * ensure it can be sent to worker machines.
+ */
+public interface AwsClientsProvider extends Serializable {
+ AmazonCloudWatch getCloudWatchClient();
+
+ AmazonDynamoDB createDynamoDB();
+}
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/BasicDynamoDBProvider.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/BasicDynamoDBProvider.java
new file mode 100644
index 0000000..36c54b1
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/BasicDynamoDBProvider.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.cloudwatch.AmazonCloudWatchClientBuilder;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import javax.annotation.Nullable;
+
+/** Basic implementation of {@link AwsClientsProvider} used by default in {@link DynamoDBIO}. */
+public class BasicDynamoDBProvider implements AwsClientsProvider {
+ private final String accessKey;
+ private final String secretKey;
+ private final Regions region;
+ @Nullable private final String serviceEndpoint;
+
+ BasicDynamoDBProvider(
+ String accessKey, String secretKey, Regions region, @Nullable String serviceEndpoint) {
+ checkArgument(accessKey != null, "accessKey can not be null");
+ checkArgument(secretKey != null, "secretKey can not be null");
+ checkArgument(region != null, "region can not be null");
+ this.accessKey = accessKey;
+ this.secretKey = secretKey;
+ this.region = region;
+ this.serviceEndpoint = serviceEndpoint;
+ }
+
+ private AWSCredentialsProvider getCredentialsProvider() {
+ return new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey));
+ }
+
+ @Override
+ public AmazonCloudWatch getCloudWatchClient() {
+ AmazonCloudWatchClientBuilder clientBuilder =
+ AmazonCloudWatchClientBuilder.standard().withCredentials(getCredentialsProvider());
+ if (serviceEndpoint == null) {
+ clientBuilder.withRegion(region);
+ } else {
+ clientBuilder.withEndpointConfiguration(
+ new AwsClientBuilder.EndpointConfiguration(serviceEndpoint, region.getName()));
+ }
+ return clientBuilder.build();
+ }
+
+ @Override
+ public AmazonDynamoDB createDynamoDB() {
+ return AmazonDynamoDBClientBuilder.standard()
+ .withCredentials(getCredentialsProvider())
+ .withRegion(region)
+ .build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIO.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIO.java
new file mode 100644
index 0000000..b9c65b2
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIO.java
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.amazonaws.regions.Regions;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException;
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import com.amazonaws.services.dynamodbv2.model.BatchWriteItemRequest;
+import com.amazonaws.services.dynamodbv2.model.ScanRequest;
+import com.amazonaws.services.dynamodbv2.model.ScanResult;
+import com.amazonaws.services.dynamodbv2.model.WriteRequest;
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.http.HttpStatus;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * {@link PTransform}s to read/write from/to <a href="https://aws.amazon.com/dynamodb/">Amazon
+ * DynamoDB</a>.
+ *
+ * <h3>Writing to DynamoDB</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<T> data = ...;
+ * data.apply(
+ * DynamoDBIO.<WriteRequest>write()
+ * .withWriteRequestMapperFn(
+ * (SerializableFunction<T, KV<String, WriteRequest>>)
+ * //Transforming your T data into KV<String, WriteRequest>
+ * t -> KV.of(tableName, writeRequest))
+ * .withRetryConfiguration(
+ * DynamoDBIO.RetryConfiguration.create(5, Duration.standardMinutes(1)))
+ * .withAwsClientsProvider(new BasicDynamoDbProvider(accessKey, secretKey, region));
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>Retry configuration
+ * <li>Specify AwsClientsProvider. You can pass on the default one BasicDynamoDbProvider
+ * <li>Mapper function with a table name to map or transform your object into KV<tableName,
+ * writeRequest>
+ * </ul>
+ *
+ * <h3>Reading from DynamoDB</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<List<Map<String, AttributeValue>>> output =
+ * pipeline.apply(
+ * DynamoDBIO.<List<Map<String, AttributeValue>>>read()
+ * .withAwsClientsProvider(new BasicDynamoDBProvider(accessKey, secretKey, region))
+ * .withScanRequestFn(
+ * (SerializableFunction<Void, ScanRequest>)
+ * input -> new ScanRequest(tableName).withTotalSegments(1))
+ * .items());
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>Specify AwsClientsProvider. You can pass on the default one BasicDynamoDBProvider
+ * <li>ScanRequestFn, which you build a ScanRequest object with at least table name and total
+ * number of segment. Note This number should base on the number of your workers
+ * </ul>
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class DynamoDBIO {
+ public static <T> Read<T> read() {
+ return new AutoValue_DynamoDBIO_Read.Builder().build();
+ }
+
+ public static <T> Write<T> write() {
+ return new AutoValue_DynamoDBIO_Write.Builder().build();
+ }
+
+ /** Read data from DynamoDB and return ScanResult. */
+ @AutoValue
+ public abstract static class Read<T> extends PTransform<PBegin, PCollection<T>> {
+ @Nullable
+ abstract AwsClientsProvider getAwsClientsProvider();
+
+ @Nullable
+ abstract SerializableFunction<Void, ScanRequest> getScanRequestFn();
+
+ @Nullable
+ abstract Integer getSegmentId();
+
+ @Nullable
+ abstract SerializableFunction<ScanResult, T> getScanResultMapperFn();
+
+ @Nullable
+ abstract Coder<T> getCoder();
+
+ abstract Builder<T> toBuilder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setAwsClientsProvider(AwsClientsProvider awsClientsProvider);
+
+ abstract Builder<T> setScanRequestFn(SerializableFunction<Void, ScanRequest> fn);
+
+ abstract Builder<T> setSegmentId(Integer segmentId);
+
+ abstract Builder<T> setScanResultMapperFn(
+ SerializableFunction<ScanResult, T> scanResultMapperFn);
+
+ abstract Builder<T> setCoder(Coder<T> coder);
+
+ abstract Read<T> build();
+ }
+
+ public Read<T> withAwsClientsProvider(AwsClientsProvider awsClientsProvider) {
+ return toBuilder().setAwsClientsProvider(awsClientsProvider).build();
+ }
+
+ public Read<T> withAwsClientsProvider(
+ String awsAccessKey, String awsSecretKey, Regions region, String serviceEndpoint) {
+ return withAwsClientsProvider(
+ new BasicDynamoDBProvider(awsAccessKey, awsSecretKey, region, serviceEndpoint));
+ }
+
+ public Read<T> withAwsClientsProvider(
+ String awsAccessKey, String awsSecretKey, Regions region) {
+ return withAwsClientsProvider(awsAccessKey, awsSecretKey, region, null);
+ }
+
+ /**
+ * Can't pass ScanRequest object directly from client since this object is not full
+ * serializable.
+ */
+ public Read<T> withScanRequestFn(SerializableFunction<Void, ScanRequest> fn) {
+ return toBuilder().setScanRequestFn(fn).build();
+ }
+
+ private Read<T> withSegmentId(Integer segmentId) {
+ checkArgument(segmentId != null, "segmentId can not be null");
+ return toBuilder().setSegmentId(segmentId).build();
+ }
+
+ public Read<T> withScanResultMapperFn(SerializableFunction<ScanResult, T> scanResultMapperFn) {
+ checkArgument(scanResultMapperFn != null, "scanResultMapper can not be null");
+ return toBuilder().setScanResultMapperFn(scanResultMapperFn).build();
+ }
+
+ public Read<List<Map<String, AttributeValue>>> items() {
+ return withScanResultMapperFn(new DynamoDBIO.Read.ItemsMapper())
+ .withCoder(ListCoder.of(MapCoder.of(StringUtf8Coder.of(), AttributeValueCoder.of())));
+ }
+
+ public Read<T> withCoder(Coder<T> coder) {
+ checkArgument(coder != null, "coder can not be null");
+ return toBuilder().setCoder(coder).build();
+ }
+
+ @Override
+ public PCollection<T> expand(PBegin input) {
+ checkArgument((getScanRequestFn() != null), "withScanRequestFn() is required");
+ checkArgument((getAwsClientsProvider() != null), "withAwsClientsProvider() is required");
+ ScanRequest scanRequest = getScanRequestFn().apply(null);
+ checkArgument(
+ (scanRequest.getTotalSegments() != null && scanRequest.getTotalSegments() > 0),
+ "TotalSegments is required with withScanRequestFn() and greater zero");
+
+ PCollection<Read<T>> splits =
+ (PCollection<Read<T>>)
+ input.apply("Create", Create.of(this)).apply("Split", ParDo.of(new SplitFn()));
+ splits.setCoder(SerializableCoder.of(new TypeDescriptor<Read<T>>() {}));
+
+ PCollection<T> output =
+ (PCollection<T>)
+ splits
+ .apply("Reshuffle", Reshuffle.viaRandomKey())
+ .apply("Read", ParDo.of(new ReadFn()));
+ output.setCoder(getCoder());
+ return output;
+ }
+
+ /** A {@link DoFn} to split {@link Read} elements by segment id. */
+ private static class SplitFn<T> extends DoFn<Read<T>, Read<T>> {
+ @ProcessElement
+ public void processElement(@Element Read<T> spec, OutputReceiver<Read<T>> out) {
+ ScanRequest scanRequest = spec.getScanRequestFn().apply(null);
+ for (int i = 0; i < scanRequest.getTotalSegments(); i++) {
+ out.output(spec.withSegmentId(i));
+ }
+ }
+ }
+
+ /** A {@link DoFn} executing the ScanRequest to read from DynamoDB. */
+ private static class ReadFn<T> extends DoFn<Read<T>, T> {
+ @ProcessElement
+ public void processElement(@Element Read<T> spec, OutputReceiver<T> out) {
+ AmazonDynamoDB client = spec.getAwsClientsProvider().createDynamoDB();
+ ScanRequest scanRequest = spec.getScanRequestFn().apply(null);
+ scanRequest.setSegment(spec.getSegmentId());
+ ScanResult scanResult = client.scan(scanRequest);
+ out.output(spec.getScanResultMapperFn().apply(scanResult));
+ }
+ }
+
+ static final class ItemsMapper<T>
+ implements SerializableFunction<ScanResult, List<Map<String, AttributeValue>>> {
+ @Override
+ public List<Map<String, AttributeValue>> apply(@Nullable ScanResult scanResult) {
+ if (scanResult == null) {
+ return Collections.emptyList();
+ }
+ return scanResult.getItems();
+ }
+ }
+ }
+
+ /**
+ * A POJO encapsulating a configuration for retry behavior when issuing requests to DynamoDB. A
+ * retry will be attempted until the maxAttempts or maxDuration is exceeded, whichever comes
+ * first, for any of the following exceptions:
+ *
+ * <ul>
+ * <li>{@link IOException}
+ * </ul>
+ */
+ @AutoValue
+ public abstract static class RetryConfiguration implements Serializable {
+ @VisibleForTesting
+ static final RetryPredicate DEFAULT_RETRY_PREDICATE = new DefaultRetryPredicate();
+
+ abstract int getMaxAttempts();
+
+ abstract Duration getMaxDuration();
+
+ abstract DynamoDBIO.RetryConfiguration.RetryPredicate getRetryPredicate();
+
+ abstract DynamoDBIO.RetryConfiguration.Builder builder();
+
+ public static DynamoDBIO.RetryConfiguration create(int maxAttempts, Duration maxDuration) {
+ checkArgument(maxAttempts > 0, "maxAttempts should be greater than 0");
+ checkArgument(
+ maxDuration != null && maxDuration.isLongerThan(Duration.ZERO),
+ "maxDuration should be greater than 0");
+ return new AutoValue_DynamoDBIO_RetryConfiguration.Builder()
+ .setMaxAttempts(maxAttempts)
+ .setMaxDuration(maxDuration)
+ .setRetryPredicate(DEFAULT_RETRY_PREDICATE)
+ .build();
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract DynamoDBIO.RetryConfiguration.Builder setMaxAttempts(int maxAttempts);
+
+ abstract DynamoDBIO.RetryConfiguration.Builder setMaxDuration(Duration maxDuration);
+
+ abstract DynamoDBIO.RetryConfiguration.Builder setRetryPredicate(
+ RetryPredicate retryPredicate);
+
+ abstract DynamoDBIO.RetryConfiguration build();
+ }
+
+ /**
+ * An interface used to control if we retry the BatchWriteItemRequest call when a {@link
+ * Throwable} occurs. If {@link RetryPredicate#test(Object)} returns true, {@link Write} tries
+ * to resend the requests to the DynamoDB server if the {@link RetryConfiguration} permits it.
+ */
+ @FunctionalInterface
+ interface RetryPredicate extends Predicate<Throwable>, Serializable {}
+
+ private static class DefaultRetryPredicate implements RetryPredicate {
+ private static final ImmutableSet<Integer> ELIGIBLE_CODES =
+ ImmutableSet.of(HttpStatus.SC_SERVICE_UNAVAILABLE);
+
+ @Override
+ public boolean test(Throwable throwable) {
+ return (throwable instanceof IOException
+ || (throwable instanceof AmazonDynamoDBException)
+ || (throwable instanceof AmazonDynamoDBException
+ && ELIGIBLE_CODES.contains(((AmazonDynamoDBException) throwable).getStatusCode())));
+ }
+ }
+ }
+
+ /** Write a PCollection<T> data into DynamoDB. */
+ @AutoValue
+ public abstract static class Write<T> extends PTransform<PCollection<T>, PCollection<Void>> {
+
+ @Nullable
+ abstract AwsClientsProvider getAwsClientsProvider();
+
+ @Nullable
+ abstract RetryConfiguration getRetryConfiguration();
+
+ @Nullable
+ abstract SerializableFunction<T, KV<String, WriteRequest>> getWriteItemMapperFn();
+
+ abstract Builder<T> builder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setAwsClientsProvider(AwsClientsProvider awsClientsProvider);
+
+ abstract Builder<T> setRetryConfiguration(RetryConfiguration retryConfiguration);
+
+ abstract Builder<T> setWriteItemMapperFn(
+ SerializableFunction<T, KV<String, WriteRequest>> writeItemMapperFn);
+
+ abstract Write<T> build();
+ }
+
+ public Write<T> withAwsClientsProvider(AwsClientsProvider awsClientsProvider) {
+ return builder().setAwsClientsProvider(awsClientsProvider).build();
+ }
+
+ public Write<T> withAwsClientsProvider(
+ String awsAccessKey, String awsSecretKey, Regions region, String serviceEndpoint) {
+ return withAwsClientsProvider(
+ new BasicDynamoDBProvider(awsAccessKey, awsSecretKey, region, serviceEndpoint));
+ }
+
+ public Write<T> withAwsClientsProvider(
+ String awsAccessKey, String awsSecretKey, Regions region) {
+ return withAwsClientsProvider(awsAccessKey, awsSecretKey, region, null);
+ }
+
+ /**
+ * Provides configuration to retry a failed request to publish a set of records to DynamoDB.
+ * Users should consider that retrying might compound the underlying problem which caused the
+ * initial failure. Users should also be aware that once retrying is exhausted the error is
+ * surfaced to the runner which <em>may</em> then opt to retry the current partition in entirety
+ * or abort if the max number of retries of the runner is completed. Retrying uses an
+ * exponential backoff algorithm, with minimum backoff of 5 seconds and then surfacing the error
+ * once the maximum number of retries or maximum configuration duration is exceeded.
+ *
+ * <p>Example use:
+ *
+ * <pre>{@code
+ * DynamoDBIO.write()
+ * .withRetryConfiguration(DynamoDBIO.RetryConfiguration.create(5, Duration.standardMinutes(1))
+ * ...
+ * }</pre>
+ *
+ * @param retryConfiguration the rules which govern the retry behavior
+ * @return the {@link DynamoDBIO.Write} with retrying configured
+ */
+ public Write<T> withRetryConfiguration(RetryConfiguration retryConfiguration) {
+ checkArgument(retryConfiguration != null, "retryConfiguration is required");
+ return builder().setRetryConfiguration(retryConfiguration).build();
+ }
+
+ public Write<T> withWriteRequestMapperFn(
+ SerializableFunction<T, KV<String, WriteRequest>> writeItemMapperFn) {
+ return builder().setWriteItemMapperFn(writeItemMapperFn).build();
+ }
+
+ @Override
+ public PCollection<Void> expand(PCollection<T> input) {
+ return input.apply(ParDo.of(new WriteFn<>(this)));
+ }
+
+ static class WriteFn<T> extends DoFn<T, Void> {
+ @VisibleForTesting
+ static final String RETRY_ATTEMPT_LOG = "Error writing to DynamoDB. Retry attempt[%d]";
+
+ private static final Duration RETRY_INITIAL_BACKOFF = Duration.standardSeconds(5);
+ private transient FluentBackoff retryBackoff; // defaults to no retries
+ private static final Logger LOG = LoggerFactory.getLogger(WriteFn.class);
+ private static final Counter DYNAMO_DB_WRITE_FAILURES =
+ Metrics.counter(WriteFn.class, "DynamoDB_Write_Failures");
+
+ private static final int BATCH_SIZE = 25;
+ private transient AmazonDynamoDB client;
+ private final DynamoDBIO.Write spec;
+ private List<KV<String, WriteRequest>> batch;
+
+ WriteFn(DynamoDBIO.Write spec) {
+ this.spec = spec;
+ }
+
+ @Setup
+ public void setup() {
+ client = spec.getAwsClientsProvider().createDynamoDB();
+ retryBackoff =
+ FluentBackoff.DEFAULT
+ .withMaxRetries(0) // default to no retrying
+ .withInitialBackoff(RETRY_INITIAL_BACKOFF);
+ if (spec.getRetryConfiguration() != null) {
+ retryBackoff =
+ retryBackoff
+ .withMaxRetries(spec.getRetryConfiguration().getMaxAttempts() - 1)
+ .withMaxCumulativeBackoff(spec.getRetryConfiguration().getMaxDuration());
+ }
+ }
+
+ @StartBundle
+ public void startBundle(StartBundleContext context) {
+ batch = new ArrayList<>();
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context) throws Exception {
+ final KV<String, WriteRequest> writeRequest =
+ (KV<String, WriteRequest>) spec.getWriteItemMapperFn().apply(context.element());
+ batch.add(writeRequest);
+ if (batch.size() >= BATCH_SIZE) {
+ flushBatch();
+ }
+ }
+
+ @FinishBundle
+ public void finishBundle(FinishBundleContext context) throws Exception {
+ flushBatch();
+ }
+
+ private void flushBatch() throws IOException, InterruptedException {
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ try {
+ // Since each element is a KV<tableName, writeRequest> in the batch, we need to group them
+ // by tableName
+ Map<String, List<WriteRequest>> mapTableRequest =
+ batch.stream()
+ .collect(
+ Collectors.groupingBy(
+ KV::getKey, Collectors.mapping(KV::getValue, Collectors.toList())));
+
+ BatchWriteItemRequest batchRequest = new BatchWriteItemRequest();
+ mapTableRequest
+ .entrySet()
+ .forEach(
+ entry -> batchRequest.addRequestItemsEntry(entry.getKey(), entry.getValue()));
+
+ Sleeper sleeper = Sleeper.DEFAULT;
+ BackOff backoff = retryBackoff.backoff();
+ int attempt = 0;
+ while (true) {
+ attempt++;
+ try {
+ client.batchWriteItem(batchRequest);
+ break;
+ } catch (Exception ex) {
+ // Fail right away if there is no retry configuration
+ if (spec.getRetryConfiguration() == null
+ || !spec.getRetryConfiguration().getRetryPredicate().test(ex)) {
+ DYNAMO_DB_WRITE_FAILURES.inc();
+ LOG.info(
+ "Unable to write batch items {} due to {} ",
+ batchRequest.getRequestItems().entrySet(),
+ ex);
+ throw new IOException("Error writing to DynamoDB (no attempt made to retry)", ex);
+ }
+
+ if (!BackOffUtils.next(sleeper, backoff)) {
+ throw new IOException(
+ String.format(
+ "Error writing to DynamoDB after %d attempt(s). No more attempts allowed",
+ attempt),
+ ex);
+ } else {
+ // Note: this used in test cases to verify behavior
+ LOG.warn(String.format(RETRY_ATTEMPT_LOG, attempt), ex);
+ }
+ }
+ }
+ } finally {
+ batch.clear();
+ }
+ }
+
+ @Teardown
+ public void tearDown() {
+ if (client != null) {
+ client.shutdown();
+ client = null;
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/package-info.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/package-info.java
new file mode 100644
index 0000000..0a7ea55
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/package-info.java
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/** Defines IO connectors for Amazon Web Services DynamoDB. */
+package org.apache.beam.sdk.io.aws.dynamodb;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
index a1ac65e..79f9db0 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
@@ -51,7 +51,7 @@
import java.io.IOException;
import java.lang.reflect.Field;
import java.util.Map;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/**
* A Jackson {@link Module} that registers a {@link JsonSerializer} and {@link JsonDeserializer} for
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsPipelineOptionsRegistrar.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsPipelineOptionsRegistrar.java
index f106ade..3cee219 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsPipelineOptionsRegistrar.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsPipelineOptionsRegistrar.java
@@ -20,7 +20,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A registrar containing the default AWS options. */
@AutoService(PipelineOptionsRegistrar.class)
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/DefaultS3ClientBuilderFactory.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/DefaultS3ClientBuilderFactory.java
index 8416304..7ed6f6b 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/DefaultS3ClientBuilderFactory.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/DefaultS3ClientBuilderFactory.java
@@ -21,7 +21,7 @@
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.apache.beam.sdk.io.aws.options.S3ClientBuilderFactory;
import org.apache.beam.sdk.io.aws.options.S3Options;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
index 276d64b..b31c6a1 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.aws.s3;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.amazonaws.AmazonClientException;
import com.amazonaws.services.s3.AmazonS3;
@@ -68,18 +68,18 @@
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.util.InstanceBuilder;
import org.apache.beam.sdk.util.MoreFutures;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemRegistrar.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemRegistrar.java
index 55bbf09..7b4b0ce 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemRegistrar.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemRegistrar.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.aws.s3;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.service.AutoService;
import javax.annotation.Nonnull;
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.io.FileSystemRegistrar;
import org.apache.beam.sdk.io.aws.options.S3Options;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link AutoService} registrar for the {@link S3FileSystem}. */
@AutoService(FileSystemRegistrar.class)
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
index ae969c5..44a7820 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ReadableSeekableByteChannel.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.aws.s3;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.AmazonClientException;
import com.amazonaws.services.s3.AmazonS3;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ResourceId.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ResourceId.java
index 090693e..a4b2cc2 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ResourceId.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3ResourceId.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.aws.s3;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Date;
import java.util.Objects;
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.io.fs.ResolveOptions;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
class S3ResourceId implements ResourceId {
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannel.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannel.java
index f162843..ac0bb3a 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannel.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannel.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.aws.s3;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.AmazonClientException;
import com.amazonaws.services.s3.AmazonS3;
@@ -41,7 +41,7 @@
import java.util.List;
import org.apache.beam.sdk.io.aws.options.S3Options;
import org.apache.beam.sdk.io.aws.options.S3Options.S3UploadBufferSizeBytesFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** A writable S3 object, as a {@link WritableByteChannel}. */
class S3WritableByteChannel implements WritableByteChannel {
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
index 5a0e510..1f57473 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/BasicSnsProvider.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.aws.sns;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java
index 93af78b..8c2a29a 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoder.java
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** Custom Coder for handling publish result. */
public class PublishResultCoder extends Coder<PublishResult> implements Serializable {
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsCoderProviderRegistrar.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsCoderProviderRegistrar.java
index 6080fc3..3a17cdd 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsCoderProviderRegistrar.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsCoderProviderRegistrar.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.CoderProviderRegistrar;
import org.apache.beam.sdk.coders.CoderProviders;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A {@link CoderProviderRegistrar} for standard types used with {@link SnsIO}. */
@AutoService(CoderProviderRegistrar.class)
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
index 449dd1e..45bcc81 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/SnsIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.aws.sns;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.sns.AmazonSNS;
@@ -44,8 +44,8 @@
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.apache.http.HttpStatus;
import org.joda.time.Duration;
import org.slf4j.Logger;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java
index ae8d84c..ba57df4 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java
@@ -23,8 +23,8 @@
import java.util.List;
import java.util.Optional;
import org.apache.beam.sdk.io.UnboundedSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
class SqsCheckpointMark implements UnboundedSource.CheckpointMark, Serializable {
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java
index 8f66a08..2412671 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.aws.sqs;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
diff --git a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedSource.java b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedSource.java
index 560da8a..52b1107 100644
--- a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedSource.java
+++ b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedSource.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.io.aws.sqs.SqsIO.Read;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
class SqsUnboundedSource extends UnboundedSource<Message, SqsCheckpointMark> {
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderTest.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderTest.java
new file mode 100644
index 0000000..ade41c4
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderTest.java
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Assert;
+import org.junit.Test;
+
+/** Unit test cases for each type of AttributeValue to test encoding and decoding. */
+public class AttributeValueCoderTest {
+
+ @Test
+ public void shouldPassForStringType() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setS("testing");
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForNumberType() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setN("123");
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForBooleanType() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setBOOL(false);
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForByteArray() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setB(ByteBuffer.wrap("hello".getBytes(StandardCharsets.UTF_8)));
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForListOfString() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setSS(ImmutableList.of("foo", "bar"));
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForOneListOfNumber() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setNS(ImmutableList.of("123", "456"));
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForOneListOfByteArray() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setBS(
+ ImmutableList.of(
+ ByteBuffer.wrap("mylistbyte1".getBytes(StandardCharsets.UTF_8)),
+ ByteBuffer.wrap("mylistbyte2".getBytes(StandardCharsets.UTF_8))));
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForListType() throws IOException {
+ AttributeValue expected = new AttributeValue();
+
+ List<AttributeValue> listAttr = new ArrayList<>();
+ listAttr.add(new AttributeValue("innerMapValue1"));
+ listAttr.add(new AttributeValue().withN("8976234"));
+
+ expected.setL(listAttr);
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForMapType() throws IOException {
+ AttributeValue expected = new AttributeValue();
+
+ Map<String, AttributeValue> attrMap = new HashMap<>();
+ attrMap.put("innerMapAttr1", new AttributeValue("innerMapValue1"));
+ attrMap.put(
+ "innerMapAttr2",
+ new AttributeValue().withB(ByteBuffer.wrap("8976234".getBytes(StandardCharsets.UTF_8))));
+
+ expected.setM(attrMap);
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForNullType() throws IOException {
+ AttributeValue expected = new AttributeValue();
+ expected.setNULL(true);
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProviderMock.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProviderMock.java
new file mode 100644
index 0000000..dfcf302
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProviderMock.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import org.mockito.Mockito;
+
+/** Mocking AwsClientProvider. */
+public class AwsClientsProviderMock implements AwsClientsProvider {
+
+ private static AwsClientsProviderMock instance = new AwsClientsProviderMock();
+ private static AmazonDynamoDB db;
+
+ private AwsClientsProviderMock() {}
+
+ public static AwsClientsProviderMock of(AmazonDynamoDB dynamoDB) {
+ db = dynamoDB;
+ return instance;
+ }
+
+ @Override
+ public AmazonCloudWatch getCloudWatchClient() {
+ return Mockito.mock(AmazonCloudWatch.class);
+ }
+
+ @Override
+ public AmazonDynamoDB createDynamoDB() {
+ return db;
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java
new file mode 100644
index 0000000..7491536
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException;
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import com.amazonaws.services.dynamodbv2.model.BatchWriteItemRequest;
+import com.amazonaws.services.dynamodbv2.model.ScanRequest;
+import com.amazonaws.services.dynamodbv2.model.WriteRequest;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.testing.ExpectedLogs;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.mockito.Mockito;
+
+/** Test Coverage for the IO. */
+@Ignore("[BEAM-7794] DynamoDBIOTest is blocking forever")
+public class DynamoDBIOTest implements Serializable {
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public final transient ExpectedLogs expectedLogs = ExpectedLogs.none(DynamoDBIO.class);
+
+ private static final String tableName = "TaskA";
+ private static final int numOfItems = 10;
+
+ private static List<Map<String, AttributeValue>> expected;
+
+ @BeforeClass
+ public static void setup() {
+ DynamoDBIOTestHelper.startServerClient();
+ DynamoDBIOTestHelper.createTestTable(tableName);
+ expected = DynamoDBIOTestHelper.generateTestData(tableName, numOfItems);
+ }
+
+ @AfterClass
+ public static void destroy() {
+ DynamoDBIOTestHelper.stopServerClient(tableName);
+ }
+
+ // Test cases for Reader.
+ @Test
+ public void testReadScanResult() {
+ PCollection<List<Map<String, AttributeValue>>> actual =
+ pipeline.apply(
+ DynamoDBIO.<List<Map<String, AttributeValue>>>read()
+ .withAwsClientsProvider(
+ AwsClientsProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient()))
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input -> new ScanRequest(tableName).withTotalSegments(1))
+ .items());
+ PAssert.that(actual).containsInAnyOrder(expected);
+ pipeline.run().waitUntilFinish();
+ }
+
+ // Test cases for Reader's arguments.
+ @Test
+ public void testMissingScanRequestFn() {
+ thrown.expectMessage("withScanRequestFn() is required");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withAwsClientsProvider(
+ AwsClientsProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("withScanRequestFn() is required");
+ } catch (IllegalArgumentException ex) {
+ assertEquals("withScanRequestFn() is required", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testMissingAwsClientsProvider() {
+ thrown.expectMessage("withAwsClientsProvider() is required");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input -> new ScanRequest(tableName).withTotalSegments(3)));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("withAwsClientsProvider() is required");
+ } catch (IllegalArgumentException ex) {
+ assertEquals("withAwsClientsProvider() is required", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testMissingTotalSegments() {
+ thrown.expectMessage("TotalSegments is required with withScanRequestFn()");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>) input -> new ScanRequest(tableName))
+ .withAwsClientsProvider(
+ AwsClientsProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("TotalSegments is required with withScanRequestFn()");
+ } catch (IllegalArgumentException ex) {
+ assertEquals("TotalSegments is required with withScanRequestFn()", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testNegativeTotalSegments() {
+ thrown.expectMessage("TotalSegments is required with withScanRequestFn() and greater zero");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input -> new ScanRequest(tableName).withTotalSegments(-1))
+ .withAwsClientsProvider(
+ AwsClientsProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("withTotalSegments() is expected and greater than zero");
+ } catch (IllegalArgumentException ex) {
+ assertEquals(
+ "TotalSegments is required with withScanRequestFn() and greater zero", ex.getMessage());
+ }
+ }
+
+ // Test cases for Writer.
+ @Test
+ public void testWriteDataToDynamo() {
+ final List<WriteRequest> writeRequests = DynamoDBIOTestHelper.generateWriteRequests(numOfItems);
+
+ final PCollection<Void> output =
+ pipeline
+ .apply(Create.of(writeRequests))
+ .apply(
+ DynamoDBIO.<WriteRequest>write()
+ .withWriteRequestMapperFn(
+ (SerializableFunction<WriteRequest, KV<String, WriteRequest>>)
+ writeRequest -> KV.of(tableName, writeRequest))
+ .withRetryConfiguration(
+ DynamoDBIO.RetryConfiguration.create(5, Duration.standardMinutes(1)))
+ .withAwsClientsProvider(
+ AwsClientsProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+
+ final PCollection<Long> publishedResultsSize = output.apply(Count.globally());
+ PAssert.that(publishedResultsSize).containsInAnyOrder(0L);
+
+ pipeline.run().waitUntilFinish();
+ }
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testRetries() throws Throwable {
+ thrown.expectMessage("Error writing to DynamoDB");
+
+ final List<WriteRequest> writeRequests = DynamoDBIOTestHelper.generateWriteRequests(numOfItems);
+
+ AmazonDynamoDB amazonDynamoDBMock = Mockito.mock(AmazonDynamoDB.class);
+ Mockito.when(amazonDynamoDBMock.batchWriteItem(Mockito.any(BatchWriteItemRequest.class)))
+ .thenThrow(new AmazonDynamoDBException("Service unavailable"));
+
+ pipeline
+ .apply(Create.of(writeRequests))
+ .apply(
+ DynamoDBIO.<WriteRequest>write()
+ .withWriteRequestMapperFn(
+ (SerializableFunction<WriteRequest, KV<String, WriteRequest>>)
+ writeRequest -> KV.of(tableName, writeRequest))
+ .withRetryConfiguration(
+ DynamoDBIO.RetryConfiguration.create(4, Duration.standardSeconds(10)))
+ .withAwsClientsProvider(AwsClientsProviderMock.of(amazonDynamoDBMock)));
+
+ try {
+ pipeline.run().waitUntilFinish();
+ } catch (final Pipeline.PipelineExecutionException e) {
+ // check 3 retries were initiated by inspecting the log before passing on the exception
+ expectedLogs.verifyWarn(String.format(DynamoDBIO.Write.WriteFn.RETRY_ATTEMPT_LOG, 1));
+ expectedLogs.verifyWarn(String.format(DynamoDBIO.Write.WriteFn.RETRY_ATTEMPT_LOG, 2));
+ expectedLogs.verifyWarn(String.format(DynamoDBIO.Write.WriteFn.RETRY_ATTEMPT_LOG, 3));
+ throw e.getCause();
+ }
+ fail("Pipeline is expected to fail because we were unable to write to DynamoDB.");
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java
new file mode 100644
index 0000000..feedc99
--- /dev/null
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.dynamodb;
+
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
+import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
+import com.amazonaws.services.dynamodbv2.model.AttributeDefinition;
+import com.amazonaws.services.dynamodbv2.model.AttributeValue;
+import com.amazonaws.services.dynamodbv2.model.BatchWriteItemRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
+import com.amazonaws.services.dynamodbv2.model.CreateTableResult;
+import com.amazonaws.services.dynamodbv2.model.KeySchemaElement;
+import com.amazonaws.services.dynamodbv2.model.KeyType;
+import com.amazonaws.services.dynamodbv2.model.ListTablesResult;
+import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
+import com.amazonaws.services.dynamodbv2.model.PutRequest;
+import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType;
+import com.amazonaws.services.dynamodbv2.model.ScanRequest;
+import com.amazonaws.services.dynamodbv2.model.ScanResult;
+import com.amazonaws.services.dynamodbv2.model.TableDescription;
+import com.amazonaws.services.dynamodbv2.model.WriteRequest;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.testcontainers.containers.localstack.LocalStackContainer;
+
+/** A utility to generate test table and data for {@link DynamoDBIOTest}. */
+class DynamoDBIOTestHelper implements Serializable {
+
+ @Rule
+ private static LocalStackContainer localStackContainer =
+ new LocalStackContainer().withServices(LocalStackContainer.Service.DYNAMODB);
+
+ private static AmazonDynamoDB dynamoDBClient;
+
+ static final String ATTR_NAME_1 = "hashKey1";
+ static final String ATTR_NAME_2 = "rangeKey2";
+
+ static void startServerClient() {
+ localStackContainer.start();
+
+ if (dynamoDBClient == null) {
+ dynamoDBClient =
+ AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(
+ localStackContainer.getEndpointConfiguration(
+ LocalStackContainer.Service.DYNAMODB))
+ .withCredentials(localStackContainer.getDefaultCredentialsProvider())
+ .build();
+ }
+ }
+
+ static void stopServerClient(String tableName) {
+ if (dynamoDBClient != null) {
+ dynamoDBClient.deleteTable(tableName);
+ dynamoDBClient.shutdown();
+ }
+ localStackContainer.stop();
+ }
+
+ static AmazonDynamoDB getDynamoDBClient() {
+ // Note: each test case got to have their own dynamo client obj, can't be shared
+ // Otherwise will run into connection pool issue
+ return AmazonDynamoDBClientBuilder.standard()
+ .withEndpointConfiguration(
+ localStackContainer.getEndpointConfiguration(LocalStackContainer.Service.DYNAMODB))
+ .withCredentials(localStackContainer.getDefaultCredentialsProvider())
+ .build();
+ }
+
+ static List<Map<String, AttributeValue>> generateTestData(String tableName, int numOfItems) {
+ BatchWriteItemRequest batchWriteItemRequest =
+ generateBatchWriteItemRequest(tableName, numOfItems);
+
+ dynamoDBClient.batchWriteItem(batchWriteItemRequest);
+ ScanResult scanResult = dynamoDBClient.scan(new ScanRequest().withTableName(tableName));
+
+ List<Map<String, AttributeValue>> items = scanResult.getItems();
+ Assert.assertEquals(numOfItems, items.size());
+ return items;
+ }
+
+ static BatchWriteItemRequest generateBatchWriteItemRequest(String tableName, int numOfItems) {
+ BatchWriteItemRequest batchWriteItemRequest = new BatchWriteItemRequest();
+ batchWriteItemRequest.addRequestItemsEntry(tableName, generateWriteRequests(numOfItems));
+ return batchWriteItemRequest;
+ }
+
+ static List<WriteRequest> generateWriteRequests(int numOfItem) {
+ List<WriteRequest> writeRequests = new ArrayList<>();
+ for (int i = 1; i <= numOfItem; i++) {
+ WriteRequest writeRequest = new WriteRequest();
+ writeRequest.setPutRequest(generatePutRequest("hashKeyDataStr_" + i, "1000" + i));
+ writeRequests.add(writeRequest);
+ }
+ return writeRequests;
+ }
+
+ private static PutRequest generatePutRequest(String hashKeyData, String rangeKeyData) {
+ PutRequest putRequest = new PutRequest();
+ putRequest.addItemEntry(ATTR_NAME_1, new AttributeValue(hashKeyData));
+ putRequest.addItemEntry(ATTR_NAME_2, new AttributeValue().withN(rangeKeyData));
+ return putRequest;
+ }
+
+ static void createTestTable(String tableName) {
+ CreateTableResult res = createDynamoTable(tableName);
+
+ TableDescription tableDesc = res.getTableDescription();
+
+ Assert.assertEquals(tableName, tableDesc.getTableName());
+ Assert.assertTrue(tableDesc.getKeySchema().toString().contains(ATTR_NAME_1));
+ Assert.assertTrue(tableDesc.getKeySchema().toString().contains(ATTR_NAME_2));
+
+ Assert.assertEquals(
+ tableDesc.getProvisionedThroughput().getReadCapacityUnits(), Long.valueOf(1000));
+ Assert.assertEquals(
+ tableDesc.getProvisionedThroughput().getWriteCapacityUnits(), Long.valueOf(1000));
+ Assert.assertEquals("ACTIVE", tableDesc.getTableStatus());
+ Assert.assertEquals(
+ "arn:aws:dynamodb:us-east-1:000000000000:table/" + tableName, tableDesc.getTableArn());
+
+ ListTablesResult tables = dynamoDBClient.listTables();
+ Assert.assertEquals(1, tables.getTableNames().size());
+ }
+
+ private static CreateTableResult createDynamoTable(String tableName) {
+
+ ImmutableList<AttributeDefinition> attributeDefinitions =
+ ImmutableList.of(
+ new AttributeDefinition(ATTR_NAME_1, ScalarAttributeType.S),
+ new AttributeDefinition(ATTR_NAME_2, ScalarAttributeType.N));
+
+ ImmutableList<KeySchemaElement> ks =
+ ImmutableList.of(
+ new KeySchemaElement(ATTR_NAME_1, KeyType.HASH),
+ new KeySchemaElement(ATTR_NAME_2, KeyType.RANGE));
+
+ ProvisionedThroughput provisionedthroughput = new ProvisionedThroughput(1000L, 1000L);
+ CreateTableRequest request =
+ new CreateTableRequest()
+ .withTableName(tableName)
+ .withAttributeDefinitions(attributeDefinitions)
+ .withKeySchema(ks)
+ .withProvisionedThroughput(provisionedthroughput);
+
+ return dynamoDBClient.createTable(request);
+ }
+}
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/MatchResultMatcher.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/MatchResultMatcher.java
index c0b3105..2766a64 100644
--- a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/MatchResultMatcher.java
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/MatchResultMatcher.java
@@ -17,14 +17,14 @@
*/
package org.apache.beam.sdk.io.aws.s3;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.List;
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.BaseMatcher;
import org.hamcrest.Description;
import org.hamcrest.Matcher;
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemTest.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemTest.java
index 9f2650e..892ea48 100644
--- a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemTest.java
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemTest.java
@@ -23,15 +23,14 @@
import static org.apache.beam.sdk.io.aws.s3.S3TestUtils.s3OptionsWithCustomEndpointAndPathStyleAccessEnabled;
import static org.apache.beam.sdk.io.aws.s3.S3TestUtils.s3OptionsWithSSECustomerKey;
import static org.apache.beam.sdk.io.fs.CreateOptions.StandardCreateOptions.builder;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.hamcrest.Matchers.contains;
-import static org.hamcrest.Matchers.notNullValue;
-import static org.hamcrest.Matchers.nullValue;
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.argThat;
import static org.mockito.Matchers.anyObject;
-import static org.mockito.Matchers.argThat;
import static org.mockito.Matchers.notNull;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
@@ -71,7 +70,7 @@
import java.util.List;
import org.apache.beam.sdk.io.aws.options.S3Options;
import org.apache.beam.sdk.io.fs.MatchResult;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -180,8 +179,7 @@
s3FileSystem.copy(sourcePath, destinationPath);
- verify(s3FileSystem.getAmazonS3Client(), times(1))
- .copyObject(argThat(notNullValue(CopyObjectRequest.class)));
+ verify(s3FileSystem.getAmazonS3Client(), times(1)).copyObject(any(CopyObjectRequest.class));
// we simulate a big object >= 5GB so it takes the multiPart path
objectMetadata.setContentLength(5_368_709_120L);
@@ -194,8 +192,7 @@
// ignore failing unmocked path, this is covered by testMultipartCopy test
}
- verify(s3FileSystem.getAmazonS3Client(), never())
- .copyObject(argThat(nullValue(CopyObjectRequest.class)));
+ verify(s3FileSystem.getAmazonS3Client(), never()).copyObject(null);
}
@Test
@@ -222,9 +219,7 @@
destinationPath.getKey());
copyObjectRequest.setSourceSSECustomerKey(options.getSSECustomerKey());
copyObjectRequest.setDestinationSSECustomerKey(options.getSSECustomerKey());
- when(s3FileSystem
- .getAmazonS3Client()
- .copyObject(argThat(notNullValue(CopyObjectRequest.class))))
+ when(s3FileSystem.getAmazonS3Client().copyObject(any(CopyObjectRequest.class)))
.thenReturn(copyObjectResult);
assertEquals(
getSSECustomerKeyMd5(options),
@@ -233,8 +228,7 @@
ObjectMetadata sourceS3ObjectMetadata = new ObjectMetadata();
s3FileSystem.atomicCopy(sourcePath, destinationPath, sourceS3ObjectMetadata);
- verify(s3FileSystem.getAmazonS3Client(), times(2))
- .copyObject(argThat(notNullValue(CopyObjectRequest.class)));
+ verify(s3FileSystem.getAmazonS3Client(), times(2)).copyObject(any(CopyObjectRequest.class));
}
@Test
@@ -257,7 +251,7 @@
}
when(s3FileSystem
.getAmazonS3Client()
- .initiateMultipartUpload(argThat(notNullValue(InitiateMultipartUploadRequest.class))))
+ .initiateMultipartUpload(any(InitiateMultipartUploadRequest.class)))
.thenReturn(initiateMultipartUploadResult);
assertEquals(
getSSECustomerKeyMd5(options),
@@ -290,7 +284,7 @@
}
CopyPartRequest copyPartRequest = new CopyPartRequest();
copyPartRequest.setSourceSSECustomerKey(options.getSSECustomerKey());
- when(s3FileSystem.getAmazonS3Client().copyPart(argThat(notNullValue(CopyPartRequest.class))))
+ when(s3FileSystem.getAmazonS3Client().copyPart(any(CopyPartRequest.class)))
.thenReturn(copyPartResult1)
.thenReturn(copyPartResult2);
assertEquals(
@@ -300,7 +294,7 @@
s3FileSystem.multipartCopy(sourcePath, destinationPath, sourceObjectMetadata);
verify(s3FileSystem.getAmazonS3Client(), times(1))
- .completeMultipartUpload(argThat(notNullValue(CompleteMultipartUploadRequest.class)));
+ .completeMultipartUpload(any(CompleteMultipartUploadRequest.class));
}
@Test
@@ -323,7 +317,7 @@
// Should require 6 calls to delete 2500 objects in each of 2 buckets.
verify(s3FileSystem.getAmazonS3Client(), times(6))
- .deleteObjects(argThat(notNullValue(DeleteObjectsRequest.class)));
+ .deleteObjects(any(DeleteObjectsRequest.class));
}
@Test
@@ -460,7 +454,8 @@
MatchResultMatcher.create(MatchResult.Status.ERROR, new IOException(exception)));
}
- static class ListObjectsV2RequestArgumentMatches extends ArgumentMatcher<ListObjectsV2Request> {
+ static class ListObjectsV2RequestArgumentMatches
+ implements ArgumentMatcher<ListObjectsV2Request> {
private final ListObjectsV2Request expected;
ListObjectsV2RequestArgumentMatches(ListObjectsV2Request expected) {
@@ -468,7 +463,7 @@
}
@Override
- public boolean matches(Object argument) {
+ public boolean matches(ListObjectsV2Request argument) {
if (argument instanceof ListObjectsV2Request) {
ListObjectsV2Request actual = (ListObjectsV2Request) argument;
return expected.getBucketName().equals(actual.getBucketName())
@@ -737,7 +732,7 @@
/** A mockito argument matcher to implement equality on GetObjectMetadataRequest. */
private static class GetObjectMetadataRequestMatcher
- extends ArgumentMatcher<GetObjectMetadataRequest> {
+ implements ArgumentMatcher<GetObjectMetadataRequest> {
private final GetObjectMetadataRequest expected;
GetObjectMetadataRequestMatcher(GetObjectMetadataRequest expected) {
@@ -745,7 +740,7 @@
}
@Override
- public boolean matches(Object obj) {
+ public boolean matches(GetObjectMetadataRequest obj) {
if (!(obj instanceof GetObjectMetadataRequest)) {
return false;
}
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannelTest.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannelTest.java
index 647e951..d79f6b2 100644
--- a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannelTest.java
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3WritableByteChannelTest.java
@@ -24,11 +24,10 @@
import static org.apache.beam.sdk.io.aws.s3.S3TestUtils.s3OptionsWithSSEAwsKeyManagementParams;
import static org.apache.beam.sdk.io.aws.s3.S3TestUtils.s3OptionsWithSSECustomerKey;
import static org.apache.beam.sdk.io.aws.s3.S3WritableByteChannel.atMostOne;
-import static org.hamcrest.Matchers.notNullValue;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
-import static org.mockito.Matchers.argThat;
+import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Matchers.notNull;
import static org.mockito.Mockito.RETURNS_SMART_NULLS;
import static org.mockito.Mockito.doReturn;
@@ -89,7 +88,7 @@
}
doReturn(initiateMultipartUploadResult)
.when(mockAmazonS3)
- .initiateMultipartUpload(argThat(notNullValue(InitiateMultipartUploadRequest.class)));
+ .initiateMultipartUpload(any(InitiateMultipartUploadRequest.class));
InitiateMultipartUploadResult mockInitiateMultipartUploadResult =
mockAmazonS3.initiateMultipartUpload(
@@ -103,7 +102,7 @@
if (getSSECustomerKeyMd5(options) != null) {
result.setSSECustomerKeyMd5(getSSECustomerKeyMd5(options));
}
- doReturn(result).when(mockAmazonS3).uploadPart(argThat(notNullValue(UploadPartRequest.class)));
+ doReturn(result).when(mockAmazonS3).uploadPart(any(UploadPartRequest.class));
UploadPartResult mockUploadPartResult = mockAmazonS3.uploadPart(new UploadPartRequest());
assertEquals(getSSECustomerKeyMd5(options), mockUploadPartResult.getSSECustomerKeyMd5());
@@ -124,7 +123,7 @@
new CompleteMultipartUploadResult();
doReturn(completeMultipartUploadResult)
.when(mockAmazonS3)
- .completeMultipartUpload(argThat(notNullValue(CompleteMultipartUploadRequest.class)));
+ .completeMultipartUpload(any(CompleteMultipartUploadRequest.class));
channel.close();
diff --git a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/SnsIOTest.java b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/SnsIOTest.java
index b921c38..1fb5283 100644
--- a/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/SnsIOTest.java
+++ b/sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/SnsIOTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
diff --git a/sdks/java/io/amazon-web-services2/build.gradle b/sdks/java/io/amazon-web-services2/build.gradle
new file mode 100644
index 0000000..eb33c56
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/build.gradle
@@ -0,0 +1,53 @@
+import groovy.json.JsonOutput
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.aws2')
+
+description = "Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services 2"
+ext.summary = "IO library to read and write Amazon Web Services services from Beam."
+
+dependencies {
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.aws_java_sdk2_apache_client
+ compile library.java.aws_java_sdk2_auth
+ compile library.java.aws_java_sdk2_cloudwatch
+ compile library.java.aws_java_sdk2_dynamodb
+ compile library.java.aws_java_sdk2_sdk_core
+ compile library.java.aws_java_sdk2_sns
+ compile library.java.jackson_core
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
+ compile library.java.slf4j_api
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.junit
+ testCompile 'org.testcontainers:testcontainers:1.11.3'
+ testRuntimeOnly library.java.slf4j_jdk14
+ testRuntimeOnly project(":runners:direct-java")
+}
+
+test {
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+ '--region=us-west-2',
+ '--awsCredentialsProvider={"@type": "StaticCredentialsProvider", "accessKeyId": "key_id_value", "secretAccessKey": "secret_value"}'
+ ])
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoder.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoder.java
new file mode 100644
index 0000000..b39fff6
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoder.java
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.BooleanCoder;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+/** A {@link Coder} that serializes and deserializes the {@link AttributeValue} objects. */
+public class AttributeValueCoder extends AtomicCoder<AttributeValue> {
+
+ /** Data type of each value type in AttributeValue object. */
+ private enum AttributeValueType {
+ s, // for String
+ n, // for Number
+ b, // for Byte
+ ss, // for List of String
+ ns, // for List of Number
+ bs, // for List of Byte
+ m, // for Map of String and AttributeValue
+ l, // for list of AttributeValue
+ bool, // for Boolean
+ nul, // for null
+ }
+
+ private static final AttributeValueCoder INSTANCE = new AttributeValueCoder();
+
+ private static final ListCoder<String> LIST_STRING_CODER = ListCoder.of(StringUtf8Coder.of());
+ private static final ListCoder<byte[]> LIST_BYTE_CODER = ListCoder.of(ByteArrayCoder.of());
+
+ private static final ListCoder<AttributeValue> LIST_ATTRIBUTE_CODER =
+ ListCoder.of(AttributeValueCoder.of());
+ private static final MapCoder<String, AttributeValue> MAP_ATTRIBUTE_CODER =
+ MapCoder.of(StringUtf8Coder.of(), AttributeValueCoder.of());
+
+ private AttributeValueCoder() {}
+
+ public static AttributeValueCoder of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(AttributeValue value, OutputStream outStream) throws IOException {
+ if (value.s() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.s.toString(), outStream);
+ StringUtf8Coder.of().encode(value.s(), outStream);
+ } else if (value.n() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.n.toString(), outStream);
+ StringUtf8Coder.of().encode(value.n(), outStream);
+ } else if (value.bool() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.bool.toString(), outStream);
+ BooleanCoder.of().encode(value.bool(), outStream);
+ } else if (value.b() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.b.toString(), outStream);
+ ByteArrayCoder.of().encode(value.b().asByteArray(), outStream);
+ } else if (value.ss() != null && value.ss().size() > 0) {
+ StringUtf8Coder.of().encode(AttributeValueType.ss.toString(), outStream);
+ LIST_STRING_CODER.encode(value.ss(), outStream);
+ } else if (value.ns() != null && value.ns().size() > 0) {
+ StringUtf8Coder.of().encode(AttributeValueType.ns.toString(), outStream);
+ LIST_STRING_CODER.encode(value.ns(), outStream);
+ } else if (value.bs() != null && value.bs().size() > 0) {
+ StringUtf8Coder.of().encode(AttributeValueType.bs.toString(), outStream);
+ LIST_BYTE_CODER.encode(convertToListByteArray(value.bs()), outStream);
+ } else if (value.l() != null && value.l().size() > 0) {
+ StringUtf8Coder.of().encode(AttributeValueType.l.toString(), outStream);
+ LIST_ATTRIBUTE_CODER.encode(value.l(), outStream);
+ } else if (value.m() != null && value.m().size() > 0) {
+ StringUtf8Coder.of().encode(AttributeValueType.m.toString(), outStream);
+ MAP_ATTRIBUTE_CODER.encode(value.m(), outStream);
+ } else if (value.nul() != null) {
+ StringUtf8Coder.of().encode(AttributeValueType.nul.toString(), outStream);
+ BooleanCoder.of().encode(value.nul(), outStream);
+ } else {
+ throw new CoderException("Unknown Type");
+ }
+ }
+
+ @Override
+ public AttributeValue decode(InputStream inStream) throws IOException {
+ AttributeValue.Builder attrBuilder = AttributeValue.builder();
+
+ String type = StringUtf8Coder.of().decode(inStream);
+ AttributeValueType attrType = AttributeValueType.valueOf(type);
+
+ switch (attrType) {
+ case s:
+ attrBuilder.s(StringUtf8Coder.of().decode(inStream));
+ break;
+ case n:
+ attrBuilder.n(StringUtf8Coder.of().decode(inStream));
+ break;
+ case bool:
+ attrBuilder.bool(BooleanCoder.of().decode(inStream));
+ break;
+ case b:
+ attrBuilder.b(SdkBytes.fromByteArray(ByteArrayCoder.of().decode(inStream)));
+ break;
+ case ss:
+ attrBuilder.ss(LIST_STRING_CODER.decode(inStream));
+ break;
+ case ns:
+ attrBuilder.ns(LIST_STRING_CODER.decode(inStream));
+ break;
+ case bs:
+ attrBuilder.bs((convertToListByteBuffer(LIST_BYTE_CODER.decode(inStream))));
+ break;
+ case l:
+ attrBuilder.l(LIST_ATTRIBUTE_CODER.decode(inStream));
+ break;
+ case m:
+ attrBuilder.m(MAP_ATTRIBUTE_CODER.decode(inStream));
+ break;
+ case nul:
+ attrBuilder.nul(BooleanCoder.of().decode(inStream));
+ break;
+ default:
+ throw new CoderException("Unknown Type");
+ }
+
+ return attrBuilder.build();
+ }
+
+ private List<byte[]> convertToListByteArray(List<SdkBytes> listSdkByte) {
+ return listSdkByte.stream().map(SdkBytes::asByteArray).collect(Collectors.toList());
+ }
+
+ private List<SdkBytes> convertToListByteBuffer(List<byte[]> listByteArr) {
+ return listByteArr.stream().map(SdkBytes::fromByteArray).collect(Collectors.toList());
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProvider.java
new file mode 100644
index 0000000..1f1c8d0
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProvider.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.net.URI;
+import javax.annotation.Nullable;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClientBuilder;
+
+/** Basic implementation of {@link DynamoDbClientProvider} used by default in {@link DynamoDBIO}. */
+public class BasicDynamoDbClientProvider implements DynamoDbClientProvider {
+ private final AwsCredentialsProvider awsCredentialsProvider;
+ private final String region;
+ @Nullable private final URI serviceEndpoint;
+
+ BasicDynamoDbClientProvider(
+ AwsCredentialsProvider awsCredentialsProvider, String region, @Nullable URI serviceEndpoint) {
+ checkArgument(awsCredentialsProvider != null, "awsCredentialsProvider can not be null");
+ checkArgument(region != null, "region can not be null");
+ this.awsCredentialsProvider = awsCredentialsProvider;
+ this.region = region;
+ this.serviceEndpoint = serviceEndpoint;
+ }
+
+ @Override
+ public DynamoDbClient getDynamoDbClient() {
+ DynamoDbClientBuilder builder =
+ DynamoDbClient.builder()
+ .credentialsProvider(awsCredentialsProvider)
+ .region(Region.of(region));
+
+ if (serviceEndpoint != null) {
+ builder.endpointOverride(serviceEndpoint);
+ }
+
+ return builder.build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIO.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIO.java
new file mode 100644
index 0000000..f422b48
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIO.java
@@ -0,0 +1,532 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Reshuffle;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.http.HttpStatus;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.ScanRequest;
+import software.amazon.awssdk.services.dynamodb.model.ScanResponse;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/**
+ * {@link PTransform}s to read/write from/to <a href="https://aws.amazon.com/dynamodb/">Amazon
+ * DynamoDB</a>.
+ *
+ * <h3>Reading from DynamoDB</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<List<Map<String, AttributeValue>>> output =
+ * pipeline.apply(
+ * DynamoDBIO.<List<Map<String, AttributeValue>>>read()
+ * .withDynamoDbClientProvider(new BasicDynamoDbClientProvider(dynamoDbClientProvider, region))
+ * .withScanRequestFn(
+ * (SerializableFunction<Void, ScanRequest>)
+ * input -> new ScanRequest(tableName).withTotalSegments(1))
+ * .items());
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>Specify DynamoDbClientProvider. You can pass on the default one BasicDynamoDbClientProvider
+ * <li>ScanRequestFn, which you build a ScanRequest object with at least table name and total
+ * number of segment. Note This number should base on the number of your workers
+ * </ul>
+ *
+ * {@link PTransform}s to read/write from/to <a
+ * href="https://aws.amazon.com/dynamodb/">DynamoDB</a>.
+ *
+ * <h3>Writing to DynamoDB</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<T> data = ...;
+ * data.apply(
+ * DynamoDBIO.<WriteRequest>write()
+ * .withWriteRequestMapperFn(
+ * (SerializableFunction<T, KV<String, WriteRequest>>)
+ * //Transforming your T data into KV<String, WriteRequest>
+ * t -> KV.of(tableName, writeRequest))
+ * .withRetryConfiguration(
+ * DynamoDBIO.RetryConfiguration.create(5, Duration.standardMinutes(1)))
+ * .withDynamoDbClientProvider(new BasicDynamoDbClientProvider(dynamoDbClientProvider, region));
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>Retry configuration
+ * <li>Specify DynamoDbClientProvider. You can pass on the default one BasicDynamoDbClientProvider
+ * <li>Mapper function with a table name to map or transform your object into KV<tableName,
+ * writeRequest>
+ * </ul>
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class DynamoDBIO {
+ public static <T> Read<T> read() {
+ return new AutoValue_DynamoDBIO_Read.Builder().build();
+ }
+
+ public static <T> Write<T> write() {
+ return new AutoValue_DynamoDBIO_Write.Builder().build();
+ }
+
+ /** Read data from DynamoDB and return ScanResult. */
+ @AutoValue
+ public abstract static class Read<T> extends PTransform<PBegin, PCollection<T>> {
+ @Nullable
+ abstract DynamoDbClientProvider getDynamoDbClientProvider();
+
+ @Nullable
+ abstract SerializableFunction<Void, ScanRequest> getScanRequestFn();
+
+ @Nullable
+ abstract Integer getSegmentId();
+
+ @Nullable
+ abstract SerializableFunction<ScanResponse, T> getScanResponseMapperFn();
+
+ @Nullable
+ abstract Coder<T> getCoder();
+
+ abstract Builder<T> toBuilder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setDynamoDbClientProvider(DynamoDbClientProvider dynamoDbClientProvider);
+
+ abstract Builder<T> setScanRequestFn(SerializableFunction<Void, ScanRequest> fn);
+
+ abstract Builder<T> setSegmentId(Integer segmentId);
+
+ abstract Builder<T> setScanResponseMapperFn(
+ SerializableFunction<ScanResponse, T> scanResponseMapperFn);
+
+ abstract Builder<T> setCoder(Coder<T> coder);
+
+ abstract Read<T> build();
+ }
+
+ public Read<T> withDynamoDbClientProvider(DynamoDbClientProvider dynamoDbClientProvider) {
+ return toBuilder().setDynamoDbClientProvider(dynamoDbClientProvider).build();
+ }
+
+ public Read<T> withDynamoDbClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region, URI serviceEndpoint) {
+ return withDynamoDbClientProvider(
+ new BasicDynamoDbClientProvider(credentialsProvider, region, serviceEndpoint));
+ }
+
+ public Read<T> withDynamoDbClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region) {
+ return withDynamoDbClientProvider(credentialsProvider, region, null);
+ }
+
+ /**
+ * Can't pass ScanRequest object directly from client since this object is not full
+ * serializable.
+ */
+ public Read<T> withScanRequestFn(SerializableFunction<Void, ScanRequest> fn) {
+ return toBuilder().setScanRequestFn(fn).build();
+ }
+
+ private Read<T> withSegmentId(Integer segmentId) {
+ checkArgument(segmentId != null, "segmentId can not be null");
+ return toBuilder().setSegmentId(segmentId).build();
+ }
+
+ public Read<T> withScanResponseMapperFn(
+ SerializableFunction<ScanResponse, T> scanResultMapperFn) {
+ checkArgument(scanResultMapperFn != null, "scanResultMapper can not be null");
+ return toBuilder().setScanResponseMapperFn(scanResultMapperFn).build();
+ }
+
+ public Read<List<Map<String, AttributeValue>>> items() {
+ return withScanResponseMapperFn(new ItemsMapper())
+ .withCoder(ListCoder.of(MapCoder.of(StringUtf8Coder.of(), AttributeValueCoder.of())));
+ }
+
+ public Read<T> withCoder(Coder<T> coder) {
+ checkArgument(coder != null, "coder can not be null");
+ return toBuilder().setCoder(coder).build();
+ }
+
+ @Override
+ public PCollection<T> expand(PBegin input) {
+ checkArgument((getScanRequestFn() != null), "withScanRequestFn() is required");
+ checkArgument(
+ (getDynamoDbClientProvider() != null), "withDynamoDbClientProvider() is required");
+ ScanRequest scanRequest = getScanRequestFn().apply(null);
+ checkArgument(
+ (scanRequest.totalSegments() != null && scanRequest.totalSegments() > 0),
+ "TotalSegments is required with withScanRequestFn() and greater zero");
+
+ PCollection<Read<T>> splits =
+ (PCollection<Read<T>>)
+ input.apply("Create", Create.of(this)).apply("Split", ParDo.of(new SplitFn()));
+ splits.setCoder(SerializableCoder.of(new TypeDescriptor<Read<T>>() {}));
+
+ PCollection<T> output =
+ (PCollection<T>)
+ splits
+ .apply("Reshuffle", Reshuffle.viaRandomKey())
+ .apply("Read", ParDo.of(new ReadFn()));
+ output.setCoder(getCoder());
+ return output;
+ }
+
+ /** A {@link DoFn} to split {@link Read} elements by segment id. */
+ private static class SplitFn<T> extends DoFn<Read<T>, Read<T>> {
+ @ProcessElement
+ public void processElement(@Element Read<T> spec, OutputReceiver<Read<T>> out) {
+ ScanRequest scanRequest = spec.getScanRequestFn().apply(null);
+ for (int i = 0; i < scanRequest.totalSegments(); i++) {
+ out.output(spec.withSegmentId(i));
+ }
+ }
+ }
+
+ /** A {@link DoFn} executing the ScanRequest to read from DynamoDb. */
+ private static class ReadFn<T> extends DoFn<Read<T>, T> {
+ @ProcessElement
+ public void processElement(@Element Read<T> spec, OutputReceiver<T> out) {
+ DynamoDbClient client = spec.getDynamoDbClientProvider().getDynamoDbClient();
+
+ ScanRequest scanRequest = spec.getScanRequestFn().apply(null);
+ ScanRequest scanRequestWithSegment =
+ scanRequest.toBuilder().segment(spec.getSegmentId()).build();
+
+ ScanResponse scanResponse = client.scan(scanRequestWithSegment);
+ out.output(spec.getScanResponseMapperFn().apply(scanResponse));
+ }
+ }
+
+ static final class ItemsMapper<T>
+ implements SerializableFunction<ScanResponse, List<Map<String, AttributeValue>>> {
+ @Override
+ public List<Map<String, AttributeValue>> apply(@Nullable ScanResponse scanResponse) {
+ if (scanResponse == null) {
+ return Collections.emptyList();
+ }
+ return scanResponse.items();
+ }
+ }
+ }
+
+ /**
+ * A POJO encapsulating a configuration for retry behavior when issuing requests to DynamoDB. A
+ * retry will be attempted until the maxAttempts or maxDuration is exceeded, whichever comes
+ * first, for any of the following exceptions:
+ *
+ * <ul>
+ * <li>{@link IOException}
+ * </ul>
+ */
+ @AutoValue
+ public abstract static class RetryConfiguration implements Serializable {
+ @VisibleForTesting
+ static final RetryPredicate DEFAULT_RETRY_PREDICATE = new DefaultRetryPredicate();
+
+ abstract int getMaxAttempts();
+
+ abstract Duration getMaxDuration();
+
+ abstract RetryPredicate getRetryPredicate();
+
+ abstract Builder toBuilder();
+
+ public static Builder builder() {
+ return new AutoValue_DynamoDBIO_RetryConfiguration.Builder();
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract Builder setMaxAttempts(int maxAttempts);
+
+ abstract Builder setMaxDuration(Duration maxDuration);
+
+ abstract Builder setRetryPredicate(RetryPredicate retryPredicate);
+
+ abstract RetryConfiguration build();
+ }
+
+ /**
+ * An interface used to control if we retry the BatchWriteItemRequest call when a {@link
+ * Throwable} occurs. If {@link RetryPredicate#test(Object)} returns true, {@link Write} tries
+ * to resend the requests to the DynamoDB server if the {@link RetryConfiguration} permits it.
+ */
+ @FunctionalInterface
+ interface RetryPredicate extends Predicate<Throwable>, Serializable {}
+
+ private static class DefaultRetryPredicate implements RetryPredicate {
+ private static final ImmutableSet<Integer> ELIGIBLE_CODES =
+ ImmutableSet.of(HttpStatus.SC_SERVICE_UNAVAILABLE);
+
+ @Override
+ public boolean test(Throwable throwable) {
+ return (throwable instanceof IOException
+ || (throwable instanceof DynamoDbException)
+ || (throwable instanceof DynamoDbException
+ && ELIGIBLE_CODES.contains(((DynamoDbException) throwable).statusCode())));
+ }
+ }
+ }
+
+ /** Write a PCollection<T> data into DynamoDB. */
+ @AutoValue
+ public abstract static class Write<T> extends PTransform<PCollection<T>, PCollection<Void>> {
+
+ @Nullable
+ abstract DynamoDbClientProvider getDynamoDbClientProvider();
+
+ @Nullable
+ abstract RetryConfiguration getRetryConfiguration();
+
+ @Nullable
+ abstract SerializableFunction<T, KV<String, WriteRequest>> getWriteItemMapperFn();
+
+ abstract Builder<T> toBuilder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setDynamoDbClientProvider(DynamoDbClientProvider dynamoDbClientProvider);
+
+ abstract Builder<T> setRetryConfiguration(RetryConfiguration retryConfiguration);
+
+ abstract Builder<T> setWriteItemMapperFn(
+ SerializableFunction<T, KV<String, WriteRequest>> writeItemMapperFn);
+
+ abstract Write<T> build();
+ }
+
+ public Write<T> withDynamoDbClientProvider(DynamoDbClientProvider dynamoDbClientProvider) {
+ return toBuilder().setDynamoDbClientProvider(dynamoDbClientProvider).build();
+ }
+
+ public Write<T> withDynamoDbClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region, URI serviceEndpoint) {
+ return withDynamoDbClientProvider(
+ new BasicDynamoDbClientProvider(credentialsProvider, region, serviceEndpoint));
+ }
+
+ public Write<T> withDynamoDbClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region) {
+ return withDynamoDbClientProvider(credentialsProvider, region, null);
+ }
+
+ /**
+ * Provides configuration to retry a failed request to publish a set of records to DynamoDb.
+ * Users should consider that retrying might compound the underlying problem which caused the
+ * initial failure. Users should also be aware that once retrying is exhausted the error is
+ * surfaced to the runner which <em>may</em> then opt to retry the current partition in entirety
+ * or abort if the max number of retries of the runner is completed. Retrying uses an
+ * exponential backoff algorithm, with minimum backoff of 5 seconds and then surfacing the error
+ * once the maximum number of retries or maximum configuration duration is exceeded.
+ *
+ * <p>Example use:
+ *
+ * <pre>{@code
+ * DynamoDBIO.write()
+ * .withRetryConfiguration(DynamoDBIO.RetryConfiguration.create(5, Duration.standardMinutes(1))
+ * ...
+ * }</pre>
+ *
+ * @param retryConfiguration the rules which govern the retry behavior
+ * @return the {@link Write} with retrying configured
+ */
+ public Write<T> withRetryConfiguration(RetryConfiguration retryConfiguration) {
+ checkArgument(retryConfiguration != null, "retryConfiguration is required");
+ return toBuilder().setRetryConfiguration(retryConfiguration).build();
+ }
+
+ public Write<T> withWriteRequestMapperFn(
+ SerializableFunction<T, KV<String, WriteRequest>> writeItemMapperFn) {
+ return toBuilder().setWriteItemMapperFn(writeItemMapperFn).build();
+ }
+
+ @Override
+ public PCollection<Void> expand(PCollection<T> input) {
+ return input.apply(ParDo.of(new WriteFn<>(this)));
+ }
+
+ static class WriteFn<T> extends DoFn<T, Void> {
+ @VisibleForTesting
+ static final String RETRY_ATTEMPT_LOG = "Error writing to DynamoDB. Retry attempt[%d]";
+
+ private static final Duration RETRY_INITIAL_BACKOFF = Duration.standardSeconds(5);
+ private transient FluentBackoff retryBackoff; // defaults to no retries
+ private static final Logger LOG = LoggerFactory.getLogger(WriteFn.class);
+ private static final Counter DYNAMO_DB_WRITE_FAILURES =
+ Metrics.counter(WriteFn.class, "DynamoDB_Write_Failures");
+
+ private static final int BATCH_SIZE = 25;
+ private transient DynamoDbClient client;
+ private final Write spec;
+ private List<KV<String, WriteRequest>> batch;
+
+ WriteFn(Write spec) {
+ this.spec = spec;
+ }
+
+ @Setup
+ public void setup() {
+ client = spec.getDynamoDbClientProvider().getDynamoDbClient();
+ retryBackoff =
+ FluentBackoff.DEFAULT
+ .withMaxRetries(0) // default to no retrying
+ .withInitialBackoff(RETRY_INITIAL_BACKOFF);
+ if (spec.getRetryConfiguration() != null) {
+ retryBackoff =
+ retryBackoff
+ .withMaxRetries(spec.getRetryConfiguration().getMaxAttempts() - 1)
+ .withMaxCumulativeBackoff(spec.getRetryConfiguration().getMaxDuration());
+ }
+ }
+
+ @StartBundle
+ public void startBundle(StartBundleContext context) {
+ batch = new ArrayList<>();
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context) throws Exception {
+ final KV<String, WriteRequest> writeRequest =
+ (KV<String, WriteRequest>) spec.getWriteItemMapperFn().apply(context.element());
+ batch.add(writeRequest);
+ if (batch.size() >= BATCH_SIZE) {
+ flushBatch();
+ }
+ }
+
+ @FinishBundle
+ public void finishBundle(FinishBundleContext context) throws Exception {
+ flushBatch();
+ }
+
+ private void flushBatch() throws IOException, InterruptedException {
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ try {
+ // Since each element is a KV<tableName, writeRequest> in the batch, we need to group them
+ // by tableName
+ Map<String, List<WriteRequest>> mapTableRequest =
+ batch.stream()
+ .collect(
+ Collectors.groupingBy(
+ KV::getKey, Collectors.mapping(KV::getValue, Collectors.toList())));
+
+ BatchWriteItemRequest batchRequest =
+ BatchWriteItemRequest.builder().requestItems(mapTableRequest).build();
+
+ Sleeper sleeper = Sleeper.DEFAULT;
+ BackOff backoff = retryBackoff.backoff();
+ int attempt = 0;
+ while (true) {
+ attempt++;
+ try {
+ client.batchWriteItem(batchRequest);
+ break;
+ } catch (Exception ex) {
+ // Fail right away if there is no retry configuration
+ if (spec.getRetryConfiguration() == null
+ || !spec.getRetryConfiguration().getRetryPredicate().test(ex)) {
+ DYNAMO_DB_WRITE_FAILURES.inc();
+ LOG.info(
+ "Unable to write batch items {} due to {} ",
+ batchRequest.requestItems().entrySet(),
+ ex);
+ throw new IOException("Error writing to DynamoDB (no attempt made to retry)", ex);
+ }
+
+ if (!BackOffUtils.next(sleeper, backoff)) {
+ throw new IOException(
+ String.format(
+ "Error writing to DynamoDB after %d attempt(s). No more attempts allowed",
+ attempt),
+ ex);
+ } else {
+ // Note: this used in test cases to verify behavior
+ LOG.warn(String.format(RETRY_ATTEMPT_LOG, attempt), ex);
+ }
+ }
+ }
+ } finally {
+ batch.clear();
+ }
+ }
+
+ @Teardown
+ public void tearDown() {
+ if (client != null) {
+ client.close();
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProvider.java
new file mode 100644
index 0000000..153fe4e
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProvider.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import java.io.Serializable;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+
+/**
+ * Provides instances of DynamoDB clients.
+ *
+ * <p>Please note, that any instance of {@link DynamoDbClientProvider} must be {@link Serializable}
+ * to ensure it can be sent to worker machines.
+ */
+public interface DynamoDbClientProvider extends Serializable {
+ DynamoDbClient getDynamoDbClient();
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/package-info.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/package-info.java
new file mode 100644
index 0000000..a2d1361
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/package-info.java
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/** Defines IO connectors for Amazon Web Services DynamoDB. */
+package org.apache.beam.sdk.io.aws2.dynamodb;
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java
new file mode 100644
index 0000000..4adad67
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.options;
+
+import com.fasterxml.jackson.annotation.JsonTypeInfo;
+import com.fasterxml.jackson.core.JsonGenerator;
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.JsonDeserializer;
+import com.fasterxml.jackson.databind.JsonSerializer;
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.SerializerProvider;
+import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.fasterxml.jackson.databind.jsontype.TypeDeserializer;
+import com.fasterxml.jackson.databind.jsontype.TypeSerializer;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import com.google.auto.service.AutoService;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Map;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider;
+import software.amazon.awssdk.http.apache.ProxyConfiguration;
+
+/**
+ * A Jackson {@link Module} that registers a {@link JsonSerializer} and {@link JsonDeserializer} for
+ * {@link AwsCredentialsProvider} and some subclasses. The serialized form is a JSON map.
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+@AutoService(Module.class)
+public class AwsModule extends SimpleModule {
+ private static final String ACCESS_KEY_ID = "accessKeyId";
+ private static final String SECRET_ACCESS_KEY = "secretAccessKey";
+
+ public AwsModule() {
+ super("AwsModule");
+ setMixInAnnotation(AwsCredentialsProvider.class, AwsCredentialsProviderMixin.class);
+ setMixInAnnotation(ProxyConfiguration.class, ProxyConfigurationMixin.class);
+ }
+
+ /** A mixin to add Jackson annotations to {@link AwsCredentialsProvider}. */
+ @JsonDeserialize(using = AwsCredentialsProviderDeserializer.class)
+ @JsonSerialize(using = AWSCredentialsProviderSerializer.class)
+ @JsonTypeInfo(use = JsonTypeInfo.Id.NAME)
+ private static class AwsCredentialsProviderMixin {}
+
+ private static class AwsCredentialsProviderDeserializer
+ extends JsonDeserializer<AwsCredentialsProvider> {
+
+ @Override
+ public AwsCredentialsProvider deserialize(JsonParser jsonParser, DeserializationContext context)
+ throws IOException {
+ return context.readValue(jsonParser, AwsCredentialsProvider.class);
+ }
+
+ @Override
+ public AwsCredentialsProvider deserializeWithType(
+ JsonParser jsonParser, DeserializationContext context, TypeDeserializer typeDeserializer)
+ throws IOException {
+ Map<String, String> asMap =
+ jsonParser.readValueAs(new TypeReference<Map<String, String>>() {});
+
+ String typeNameKey = typeDeserializer.getPropertyName();
+ String typeName = asMap.get(typeNameKey);
+ if (typeName == null) {
+ throw new IOException(
+ String.format("AWS credentials provider type name key '%s' not found", typeNameKey));
+ }
+
+ if (typeName.equals(StaticCredentialsProvider.class.getSimpleName())) {
+ return StaticCredentialsProvider.create(
+ AwsBasicCredentials.create(asMap.get(ACCESS_KEY_ID), asMap.get(SECRET_ACCESS_KEY)));
+ } else if (typeName.equals(DefaultCredentialsProvider.class.getSimpleName())) {
+ return DefaultCredentialsProvider.create();
+ } else if (typeName.equals(EnvironmentVariableCredentialsProvider.class.getSimpleName())) {
+ return EnvironmentVariableCredentialsProvider.create();
+ } else if (typeName.equals(SystemPropertyCredentialsProvider.class.getSimpleName())) {
+ return SystemPropertyCredentialsProvider.create();
+ } else if (typeName.equals(ProfileCredentialsProvider.class.getSimpleName())) {
+ return ProfileCredentialsProvider.create();
+ } else if (typeName.equals(ContainerCredentialsProvider.class.getSimpleName())) {
+ return ContainerCredentialsProvider.builder().build();
+ } else {
+ throw new IOException(
+ String.format("AWS credential provider type '%s' is not supported", typeName));
+ }
+ }
+ }
+
+ private static class AWSCredentialsProviderSerializer
+ extends JsonSerializer<AwsCredentialsProvider> {
+ // These providers are singletons, so don't require any serialization, other than type.
+ private static final ImmutableSet<Object> SINGLETON_CREDENTIAL_PROVIDERS =
+ ImmutableSet.of(
+ DefaultCredentialsProvider.class,
+ EnvironmentVariableCredentialsProvider.class,
+ SystemPropertyCredentialsProvider.class,
+ ProfileCredentialsProvider.class,
+ ContainerCredentialsProvider.class);
+
+ @Override
+ public void serialize(
+ AwsCredentialsProvider credentialsProvider,
+ JsonGenerator jsonGenerator,
+ SerializerProvider serializer)
+ throws IOException {
+ serializer.defaultSerializeValue(credentialsProvider, jsonGenerator);
+ }
+
+ @Override
+ public void serializeWithType(
+ AwsCredentialsProvider credentialsProvider,
+ JsonGenerator jsonGenerator,
+ SerializerProvider serializer,
+ TypeSerializer typeSerializer)
+ throws IOException {
+ typeSerializer.writeTypePrefixForObject(credentialsProvider, jsonGenerator);
+ if (credentialsProvider.getClass().equals(StaticCredentialsProvider.class)) {
+ jsonGenerator.writeStringField(
+ ACCESS_KEY_ID, credentialsProvider.resolveCredentials().accessKeyId());
+ jsonGenerator.writeStringField(
+ SECRET_ACCESS_KEY, credentialsProvider.resolveCredentials().secretAccessKey());
+ } else if (!SINGLETON_CREDENTIAL_PROVIDERS.contains(credentialsProvider.getClass())) {
+ throw new IllegalArgumentException(
+ "Unsupported AWS credentials provider type " + credentialsProvider.getClass());
+ }
+ typeSerializer.writeTypeSuffixForObject(credentialsProvider, jsonGenerator);
+ }
+ }
+
+ /** A mixin to add Jackson annotations to {@link ProxyConfiguration}. */
+ @JsonDeserialize(using = ProxyConfigurationDeserializer.class)
+ @JsonSerialize(using = ProxyConfigurationSerializer.class)
+ private static class ProxyConfigurationMixin {}
+
+ private static class ProxyConfigurationDeserializer extends JsonDeserializer<ProxyConfiguration> {
+ @Override
+ public ProxyConfiguration deserialize(JsonParser jsonParser, DeserializationContext context)
+ throws IOException {
+ Map<String, String> asMap =
+ jsonParser.readValueAs(new TypeReference<Map<String, String>>() {});
+ return ProxyConfiguration.builder()
+ .endpoint(URI.create(asMap.get("endpoint")))
+ .username(asMap.get("username"))
+ .password(asMap.get("password"))
+ .useSystemPropertyValues(Boolean.valueOf(asMap.get("useSystemPropertyValues")))
+ .build();
+ }
+ }
+
+ private static class ProxyConfigurationSerializer extends JsonSerializer<ProxyConfiguration> {
+ @Override
+ public void serialize(
+ ProxyConfiguration proxyConfiguration,
+ JsonGenerator jsonGenerator,
+ SerializerProvider serializer)
+ throws IOException {
+ // proxyConfiguration.endpoint() is private so we have to build it manually.
+ final String endpoint =
+ proxyConfiguration.scheme()
+ + "://"
+ + proxyConfiguration.host()
+ + ":"
+ + proxyConfiguration.port();
+ jsonGenerator.writeStartObject();
+ jsonGenerator.writeStringField("endpoint", endpoint);
+ jsonGenerator.writeStringField("username", proxyConfiguration.username());
+ jsonGenerator.writeStringField("password", proxyConfiguration.password());
+ jsonGenerator.writeEndObject();
+ }
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsOptions.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsOptions.java
new file mode 100644
index 0000000..fcbec34
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsOptions.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.options;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.DefaultValueFactory;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.Validation;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
+import software.amazon.awssdk.http.apache.ProxyConfiguration;
+
+/**
+ * Options used to configure Amazon Web Services specific options such as credentials and region.
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public interface AwsOptions extends PipelineOptions {
+
+ /** AWS region used by the AWS client. */
+ @Description("AWS region used by the AWS client")
+ @Validation.Required
+ String getRegion();
+
+ void setRegion(String value);
+
+ /** The AWS service endpoint used by the AWS client. */
+ @Description("AWS service endpoint used by the AWS client")
+ String getEndpoint();
+
+ void setEndpoint(String value);
+
+ /**
+ * The credential instance that should be used to authenticate against AWS services. The option
+ * value must contain a "@type" field and an AWS Credentials Provider class as the field value.
+ * Refer to {@link DefaultCredentialsProvider} Javadoc for usage help.
+ *
+ * <p>For example, to specify the AWS key ID and secret, specify the following: <code>
+ * {"@type" : "AWSStaticCredentialsProvider", "awsAccessKeyId" : "key_id_value",
+ * "awsSecretKey" : "secret_value"}
+ * </code>
+ */
+ @Description(
+ "The credential instance that should be used to authenticate "
+ + "against AWS services. The option value must contain \"@type\" field "
+ + "and an AWS Credentials Provider class name as the field value. "
+ + "Refer to DefaultAWSCredentialsProviderChain Javadoc for usage help. "
+ + "For example, to specify the AWS key ID and secret, specify the following: "
+ + "{\"@type\": \"StaticCredentialsProvider\", "
+ + "\"accessKeyId\":\"<key_id>\", \"secretAccessKey\":\"<secret_key>\"}")
+ @Default.InstanceFactory(AwsUserCredentialsFactory.class)
+ AwsCredentialsProvider getAwsCredentialsProvider();
+
+ void setAwsCredentialsProvider(AwsCredentialsProvider value);
+
+ /** Attempts to load AWS credentials. */
+ class AwsUserCredentialsFactory implements DefaultValueFactory<AwsCredentialsProvider> {
+ @Override
+ public AwsCredentialsProvider create(PipelineOptions options) {
+ return DefaultCredentialsProvider.create();
+ }
+ }
+
+ /**
+ * The client configuration instance that should be used to configure AWS service clients. Please
+ * note that the configuration deserialization only allows one to specify proxy settings.
+ *
+ * <p>For example, to specify the proxy endpoint, username and password, specify the following:
+ * <code>
+ * --proxyConfiguration={
+ * "endpoint": "http://hostname:port",
+ * "username": "username",
+ * "password": "password"
+ * }
+ * </code>
+ */
+ @Description(
+ "The proxy configuration instance that should be used to configure AWS service "
+ + "clients. Please note that the configuration deserialization only allows one to specify "
+ + "proxy settings. For example, to specify the proxy endpoint, username and password, "
+ + "specify the following: --proxyConfiguration={\"endpoint\":\"http://hostname:port\", \"username\":\"username\", \"password\":\"password\"}")
+ ProxyConfiguration getProxyConfiguration();
+
+ void setProxyConfiguration(ProxyConfiguration value);
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsPipelineOptionsRegistrar.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsPipelineOptionsRegistrar.java
new file mode 100644
index 0000000..4384b4c
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsPipelineOptionsRegistrar.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.options;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** A registrar containing the default AWS options. */
+@AutoService(PipelineOptionsRegistrar.class)
+public class AwsPipelineOptionsRegistrar implements PipelineOptionsRegistrar {
+
+ @Override
+ public Iterable<Class<? extends PipelineOptions>> getPipelineOptions() {
+ return ImmutableList.<Class<? extends PipelineOptions>>builder().add(AwsOptions.class).build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/package-info.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/package-info.java
new file mode 100644
index 0000000..b9adbae
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/package-info.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/**
+ * Defines {@link org.apache.beam.sdk.options.PipelineOptions} for configuring pipeline execution
+ * for Amazon Web Services components.
+ */
+package org.apache.beam.sdk.io.aws2.options;
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
new file mode 100644
index 0000000..c2f3aa8
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.net.URI;
+import javax.annotation.Nullable;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.SnsClientBuilder;
+
+/** Basic implementation of {@link SnsClientProvider} used by default in {@link SnsIO}. */
+class BasicSnsClientProvider implements SnsClientProvider {
+ private final AwsCredentialsProvider awsCredentialsProvider;
+ private final String region;
+ @Nullable private final URI serviceEndpoint;
+
+ BasicSnsClientProvider(
+ AwsCredentialsProvider awsCredentialsProvider, String region, @Nullable URI serviceEndpoint) {
+ checkArgument(awsCredentialsProvider != null, "awsCredentialsProvider can not be null");
+ checkArgument(region != null, "region can not be null");
+ this.awsCredentialsProvider = awsCredentialsProvider;
+ this.region = region;
+ this.serviceEndpoint = serviceEndpoint;
+ }
+
+ @Override
+ public SnsClient getSnsClient() {
+ SnsClientBuilder builder =
+ SnsClient.builder().credentialsProvider(awsCredentialsProvider).region(Region.of(region));
+
+ if (serviceEndpoint != null) {
+ builder.endpointOverride(serviceEndpoint);
+ }
+
+ return builder.build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
new file mode 100644
index 0000000..7faf797
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Custom Coder for handling publish result. */
+public class PublishResponseCoder extends AtomicCoder<PublishResponse> implements Serializable {
+ private static final PublishResponseCoder INSTANCE = new PublishResponseCoder();
+
+ private PublishResponseCoder() {}
+
+ static PublishResponseCoder of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(PublishResponse value, OutputStream outStream) throws IOException {
+ StringUtf8Coder.of().encode(value.messageId(), outStream);
+ }
+
+ @Override
+ public PublishResponse decode(InputStream inStream) throws IOException {
+ final String messageId = StringUtf8Coder.of().decode(inStream);
+ return PublishResponse.builder().messageId(messageId).build();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
new file mode 100644
index 0000000..b1a3af2
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.io.Serializable;
+import software.amazon.awssdk.services.sns.SnsClient;
+
+/**
+ * Provides instances of DynamoDB clients.
+ *
+ * <p>Please note, that any instance of {@link SnsClientProvider} must be {@link Serializable} to
+ * ensure it can be sent to worker machines.
+ */
+public interface SnsClientProvider extends Serializable {
+ SnsClient getSnsClient();
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
new file mode 100644
index 0000000..f87e70e
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import com.google.auto.service.AutoService;
+import java.util.List;
+import org.apache.beam.sdk.coders.CoderProvider;
+import org.apache.beam.sdk.coders.CoderProviderRegistrar;
+import org.apache.beam.sdk.coders.CoderProviders;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** A {@link CoderProviderRegistrar} for standard types used with {@link SnsIO}. */
+@AutoService(CoderProviderRegistrar.class)
+public class SnsCoderProviderRegistrar implements CoderProviderRegistrar {
+ @Override
+ public List<CoderProvider> getCoderProviders() {
+ return ImmutableList.of(
+ CoderProviders.forCoder(
+ TypeDescriptor.of(PublishResponse.class), PublishResponseCoder.of()));
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
new file mode 100644
index 0000000..e9a3187
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URI;
+import java.util.function.Predicate;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.metrics.Counter;
+import org.apache.beam.sdk.metrics.Metrics;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.http.HttpStatus;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.InternalErrorException;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/**
+ * {@link PTransform}s for writing to <a href="https://aws.amazon.com/sns/">SNS</a>.
+ *
+ * <h3>Writing to SNS</h3>
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * PCollection<String> data = ...;
+ *
+ * data.apply(SnsIO.<String>write()
+ * .withPublishRequestFn(m -> PublishRequest.builder().topicArn("topicArn").message(m).build())
+ * .withTopicArn("topicArn")
+ * .withRetryConfiguration(
+ * SnsIO.RetryConfiguration.create(
+ * 4, org.joda.time.Duration.standardSeconds(10)))
+ * .withSnsClientProvider(new BasicSnsClientProvider(awsCredentialsProvider, region));
+ * }</pre>
+ *
+ * <p>As a client, you need to provide at least the following things:
+ *
+ * <ul>
+ * <li>SNS topic arn you're going to publish to
+ * <li>Retry Configuration
+ * <li>AwsCredentialsProvider, which you can pass on to BasicSnsClientProvider
+ * <li>publishRequestFn, a function to convert your message into PublishRequest
+ * </ul>
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public final class SnsIO {
+
+ // Write data tp SNS
+ public static <T> Write<T> write() {
+ return new AutoValue_SnsIO_Write.Builder().build();
+ }
+
+ /**
+ * A POJO encapsulating a configuration for retry behavior when issuing requests to SNS. A retry
+ * will be attempted until the maxAttempts or maxDuration is exceeded, whichever comes first, for
+ * any of the following exceptions:
+ *
+ * <ul>
+ * <li>{@link IOException}
+ * </ul>
+ */
+ @AutoValue
+ public abstract static class RetryConfiguration implements Serializable {
+ @VisibleForTesting
+ static final RetryPredicate DEFAULT_RETRY_PREDICATE = new DefaultRetryPredicate();
+
+ abstract int getMaxAttempts();
+
+ abstract Duration getMaxDuration();
+
+ abstract RetryPredicate getRetryPredicate();
+
+ abstract Builder builder();
+
+ public static RetryConfiguration create(int maxAttempts, Duration maxDuration) {
+ checkArgument(maxAttempts > 0, "maxAttempts should be greater than 0");
+ checkArgument(
+ maxDuration != null && maxDuration.isLongerThan(Duration.ZERO),
+ "maxDuration should be greater than 0");
+ return new AutoValue_SnsIO_RetryConfiguration.Builder()
+ .setMaxAttempts(maxAttempts)
+ .setMaxDuration(maxDuration)
+ .setRetryPredicate(DEFAULT_RETRY_PREDICATE)
+ .build();
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract Builder setMaxAttempts(int maxAttempts);
+
+ abstract Builder setMaxDuration(Duration maxDuration);
+
+ abstract Builder setRetryPredicate(RetryPredicate retryPredicate);
+
+ abstract RetryConfiguration build();
+ }
+
+ /**
+ * An interface used to control if we retry the SNS Publish call when a {@link Throwable}
+ * occurs. If {@link RetryPredicate#test(Object)} returns true, {@link Write} tries to resend
+ * the requests to the Solr server if the {@link RetryConfiguration} permits it.
+ */
+ @FunctionalInterface
+ interface RetryPredicate extends Predicate<Throwable>, Serializable {}
+
+ private static class DefaultRetryPredicate implements RetryPredicate {
+ private static final ImmutableSet<Integer> ELIGIBLE_CODES =
+ ImmutableSet.of(HttpStatus.SC_SERVICE_UNAVAILABLE);
+
+ @Override
+ public boolean test(Throwable throwable) {
+ return (throwable instanceof IOException
+ || (throwable instanceof InternalErrorException)
+ || (throwable instanceof InternalErrorException
+ && ELIGIBLE_CODES.contains(((InternalErrorException) throwable).statusCode())));
+ }
+ }
+ }
+
+ /** Implementation of {@link #write}. */
+ @AutoValue
+ public abstract static class Write<T>
+ extends PTransform<PCollection<T>, PCollection<PublishResponse>> {
+ @Nullable
+ abstract String getTopicArn();
+
+ @Nullable
+ abstract SerializableFunction<T, PublishRequest> getPublishRequestFn();
+
+ @Nullable
+ abstract SnsClientProvider getSnsClientProvider();
+
+ @Nullable
+ abstract RetryConfiguration getRetryConfiguration();
+
+ abstract Builder<T> builder();
+
+ @AutoValue.Builder
+ abstract static class Builder<T> {
+
+ abstract Builder<T> setTopicArn(String topicArn);
+
+ abstract Builder<T> setPublishRequestFn(
+ SerializableFunction<T, PublishRequest> publishRequestFn);
+
+ abstract Builder<T> setSnsClientProvider(SnsClientProvider snsClientProvider);
+
+ abstract Builder<T> setRetryConfiguration(RetryConfiguration retryConfiguration);
+
+ abstract Write<T> build();
+ }
+
+ /**
+ * Specify the SNS topic which will be used for writing, this name is mandatory.
+ *
+ * @param topicArn topicArn
+ */
+ public Write<T> withTopicArn(String topicArn) {
+ return builder().setTopicArn(topicArn).build();
+ }
+
+ /**
+ * Specify a function for converting a message into PublishRequest object, this function is
+ * mandatory.
+ *
+ * @param publishRequestFn publishRequestFn
+ */
+ public Write<T> withPublishRequestFn(SerializableFunction<T, PublishRequest> publishRequestFn) {
+ return builder().setPublishRequestFn(publishRequestFn).build();
+ }
+
+ /**
+ * Allows to specify custom {@link SnsClientProvider}. {@link SnsClientProvider} creates new
+ * {@link SnsClient} which is later used for writing to a SNS topic.
+ */
+ public Write<T> withSnsClientProvider(SnsClientProvider awsClientsProvider) {
+ return builder().setSnsClientProvider(awsClientsProvider).build();
+ }
+
+ /**
+ * Specify {@link AwsCredentialsProvider} and region to be used to write to SNS. If you need
+ * more sophisticated credential protocol, then you should look at {@link
+ * Write#withSnsClientProvider(SnsClientProvider)}.
+ */
+ public Write<T> withSnsClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region) {
+ return withSnsClientProvider(credentialsProvider, region, null);
+ }
+
+ /**
+ * Specify {@link AwsCredentialsProvider} and region to be used to write to SNS. If you need
+ * more sophisticated credential protocol, then you should look at {@link
+ * Write#withSnsClientProvider(SnsClientProvider)}.
+ *
+ * <p>The {@code serviceEndpoint} sets an alternative service host. This is useful to execute
+ * the tests with Kinesis service emulator.
+ */
+ public Write<T> withSnsClientProvider(
+ AwsCredentialsProvider credentialsProvider, String region, URI serviceEndpoint) {
+ return withSnsClientProvider(
+ new BasicSnsClientProvider(credentialsProvider, region, serviceEndpoint));
+ }
+
+ /**
+ * Provides configuration to retry a failed request to publish a message to SNS. Users should
+ * consider that retrying might compound the underlying problem which caused the initial
+ * failure. Users should also be aware that once retrying is exhausted the error is surfaced to
+ * the runner which <em>may</em> then opt to retry the current partition in entirety or abort if
+ * the max number of retries of the runner is completed. Retrying uses an exponential backoff
+ * algorithm, with minimum backoff of 5 seconds and then surfacing the error once the maximum
+ * number of retries or maximum configuration duration is exceeded.
+ *
+ * <p>Example use:
+ *
+ * <pre>{@code
+ * SnsIO.write()
+ * .withRetryConfiguration(SnsIO.RetryConfiguration.create(5, Duration.standardMinutes(1))
+ * ...
+ * }</pre>
+ *
+ * @param retryConfiguration the rules which govern the retry behavior
+ * @return the {@link Write} with retrying configured
+ */
+ public Write<T> withRetryConfiguration(RetryConfiguration retryConfiguration) {
+ checkArgument(retryConfiguration != null, "retryConfiguration is required");
+ return builder().setRetryConfiguration(retryConfiguration).build();
+ }
+
+ private static boolean isTopicExists(SnsClient client, String topicArn) {
+ try {
+ GetTopicAttributesRequest getTopicAttributesRequest =
+ GetTopicAttributesRequest.builder().topicArn(topicArn).build();
+ GetTopicAttributesResponse topicAttributesResponse =
+ client.getTopicAttributes(getTopicAttributesRequest);
+ return topicAttributesResponse != null
+ && topicAttributesResponse.sdkHttpResponse().statusCode() == 200;
+ } catch (Exception e) {
+ throw e;
+ }
+ }
+
+ @Override
+ public PCollection<PublishResponse> expand(PCollection<T> input) {
+ checkArgument(getTopicArn() != null, "withTopicArn() is required");
+ checkArgument(getPublishRequestFn() != null, "withPublishRequestFn() is required");
+ checkArgument(getSnsClientProvider() != null, "withSnsClientProvider() is required");
+ checkArgument(
+ isTopicExists(getSnsClientProvider().getSnsClient(), getTopicArn()),
+ "Topic arn %s does not exist",
+ getTopicArn());
+
+ return input.apply(ParDo.of(new SnsWriterFn<>(this)));
+ }
+
+ static class SnsWriterFn<T> extends DoFn<T, PublishResponse> {
+ @VisibleForTesting
+ static final String RETRY_ATTEMPT_LOG = "Error writing to SNS. Retry attempt[%d]";
+
+ private static final Duration RETRY_INITIAL_BACKOFF = Duration.standardSeconds(5);
+ private transient FluentBackoff retryBackoff; // defaults to no retries
+ private static final Logger LOG = LoggerFactory.getLogger(SnsWriterFn.class);
+ private static final Counter SNS_WRITE_FAILURES =
+ Metrics.counter(SnsWriterFn.class, "SNS_Write_Failures");
+
+ private final Write spec;
+ private transient SnsClient producer;
+
+ SnsWriterFn(Write spec) {
+ this.spec = spec;
+ }
+
+ @Setup
+ public void setup() throws Exception {
+ // Initialize SnsPublisher
+ producer = spec.getSnsClientProvider().getSnsClient();
+
+ retryBackoff =
+ FluentBackoff.DEFAULT
+ .withMaxRetries(0) // default to no retrying
+ .withInitialBackoff(RETRY_INITIAL_BACKOFF);
+ if (spec.getRetryConfiguration() != null) {
+ retryBackoff =
+ retryBackoff
+ .withMaxRetries(spec.getRetryConfiguration().getMaxAttempts() - 1)
+ .withMaxCumulativeBackoff(spec.getRetryConfiguration().getMaxDuration());
+ }
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext context) throws Exception {
+ PublishRequest request =
+ (PublishRequest) spec.getPublishRequestFn().apply(context.element());
+ Sleeper sleeper = Sleeper.DEFAULT;
+ BackOff backoff = retryBackoff.backoff();
+ int attempt = 0;
+ while (true) {
+ attempt++;
+ try {
+ PublishResponse pr = producer.publish(request);
+ context.output(pr);
+ break;
+ } catch (Exception ex) {
+ // Fail right away if there is no retry configuration
+ if (spec.getRetryConfiguration() == null
+ || !spec.getRetryConfiguration().getRetryPredicate().test(ex)) {
+ SNS_WRITE_FAILURES.inc();
+ LOG.info("Unable to publish message {} due to {} ", request.message(), ex);
+ throw new IOException("Error writing to SNS (no attempt made to retry)", ex);
+ }
+
+ if (!BackOffUtils.next(sleeper, backoff)) {
+ throw new IOException(
+ String.format(
+ "Error writing to SNS after %d attempt(s). No more attempts allowed",
+ attempt),
+ ex);
+ } else {
+ // Note: this used in test cases to verify behavior
+ LOG.warn(String.format(RETRY_ATTEMPT_LOG, attempt), ex);
+ }
+ }
+ }
+ }
+
+ @Teardown
+ public void tearDown() {
+ if (producer != null) {
+ producer.close();
+ producer = null;
+ }
+ }
+ }
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
new file mode 100644
index 0000000..76f0c84
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/** Defines IO connectors for Amazon Web Services SNS. */
+package org.apache.beam.sdk.io.aws2.sns;
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoderTest.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoderTest.java
new file mode 100644
index 0000000..8f27aae
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoderTest.java
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Assert;
+import org.junit.Test;
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+/** Unit test cases for each type of AttributeValue to test encoding and decoding. */
+public class AttributeValueCoderTest {
+
+ @Test
+ public void shouldPassForStringType() throws IOException {
+ AttributeValue expected = AttributeValue.builder().s("test").build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForNumberType() throws IOException {
+ AttributeValue expected = AttributeValue.builder().n("123").build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForBooleanType() throws IOException {
+ AttributeValue expected = AttributeValue.builder().bool(false).build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForByteArray() throws IOException {
+ AttributeValue expected =
+ AttributeValue.builder()
+ .b(SdkBytes.fromByteArray("hello".getBytes(StandardCharsets.UTF_8)))
+ .build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForListOfString() throws IOException {
+ AttributeValue expected = AttributeValue.builder().ss(ImmutableList.of("foo", "bar")).build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForOneListOfNumber() throws IOException {
+ AttributeValue expected = AttributeValue.builder().ns(ImmutableList.of("123", "456")).build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForOneListOfByteArray() throws IOException {
+ AttributeValue expected =
+ AttributeValue.builder()
+ .bs(
+ ImmutableList.of(
+ SdkBytes.fromByteArray("mylistbyte1".getBytes(StandardCharsets.UTF_8)),
+ SdkBytes.fromByteArray(("mylistbyte2".getBytes(StandardCharsets.UTF_8)))))
+ .build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForListType() throws IOException {
+ List<AttributeValue> listAttr = new ArrayList<>();
+ listAttr.add(AttributeValue.builder().s("innerMapValue1").build());
+ listAttr.add(AttributeValue.builder().n("8976234").build());
+
+ AttributeValue expected = AttributeValue.builder().l(listAttr).build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForMapType() throws IOException {
+
+ Map<String, AttributeValue> attrMap = new HashMap<>();
+ attrMap.put("innerMapAttr1", AttributeValue.builder().s("innerMapValue1").build());
+ attrMap.put(
+ "innerMapAttr2",
+ AttributeValue.builder()
+ .b(SdkBytes.fromByteArray("8976234".getBytes(StandardCharsets.UTF_8)))
+ .build());
+
+ AttributeValue expected = AttributeValue.builder().m(attrMap).build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+
+ @Test
+ public void shouldPassForNullType() throws IOException {
+ AttributeValue expected = AttributeValue.builder().nul(true).build();
+
+ AttributeValueCoder coder = AttributeValueCoder.of();
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ coder.encode(expected, output);
+
+ ByteArrayInputStream in = new ByteArrayInputStream(output.toByteArray());
+
+ AttributeValue actual = coder.decode(in);
+
+ Assert.assertEquals(expected, actual);
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java
new file mode 100644
index 0000000..a1be352
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import static org.apache.beam.sdk.io.aws2.dynamodb.DynamoDBIO.RetryConfiguration.DEFAULT_RETRY_PREDICATE;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.testing.ExpectedLogs;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionTuple;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.mockito.Mockito;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ScanRequest;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/** Test Coverage for the IO. */
+@Ignore("[BEAM-7794] DynamoDBIOTest is blocking forever")
+public class DynamoDBIOTest implements Serializable {
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+ @Rule public final transient ExpectedLogs expectedLogs = ExpectedLogs.none(DynamoDBIO.class);
+
+ private static final String tableName = "TaskA";
+ private static final int numOfItems = 10;
+
+ @BeforeClass
+ public static void setup() {
+ DynamoDBIOTestHelper.startServerClient();
+ }
+
+ @AfterClass
+ public static void destroy() {
+ DynamoDBIOTestHelper.stopServerClient(tableName);
+ }
+
+ @Before
+ public void createTable() {
+ DynamoDBIOTestHelper.createTestTable(tableName);
+ }
+
+ @After
+ public void cleanTable() {
+ DynamoDBIOTestHelper.deleteTestTable(tableName);
+ }
+
+ // Test cases for Reader.
+ @Test
+ public void testReaderOneSegment() {
+ List<Map<String, AttributeValue>> expected =
+ DynamoDBIOTestHelper.generateTestData(tableName, numOfItems);
+
+ PCollection<List<Map<String, AttributeValue>>> actual =
+ pipeline.apply(
+ DynamoDBIO.<List<Map<String, AttributeValue>>>read()
+ .withDynamoDbClientProvider(
+ DynamoDbClientProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient()))
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input ->
+ ScanRequest.builder().tableName(tableName).totalSegments(1).build())
+ .items());
+ PAssert.that(actual).containsInAnyOrder(expected);
+ pipeline.run().waitUntilFinish();
+ }
+
+ @Test
+ public void testReaderThreeSegments() {
+ TupleTag<List<Map<String, AttributeValue>>> outputTag = new TupleTag<>();
+ PCollectionTuple writeOutput =
+ pipeline
+ .apply(
+ DynamoDBIO.<List<Map<String, AttributeValue>>>read()
+ .withDynamoDbClientProvider(
+ DynamoDbClientProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient()))
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input ->
+ ScanRequest.builder().tableName(tableName).totalSegments(3).build())
+ .items())
+ .apply(
+ ParDo.of(
+ new DoFn<
+ List<Map<String, AttributeValue>>,
+ List<Map<String, AttributeValue>>>() {
+ @ProcessElement
+ public void processElement(
+ @Element List<Map<String, AttributeValue>> input,
+ OutputReceiver<List<Map<String, AttributeValue>>> out) {
+ out.output(input);
+ }
+ })
+ .withOutputTags(outputTag, TupleTagList.empty()));
+
+ final PCollection<Long> resultSetCount = writeOutput.get(outputTag).apply(Count.globally());
+ // Since we don't know what item will fall into what segment, so assert 3 result set returned
+ PAssert.that(resultSetCount).containsInAnyOrder(ImmutableList.of(3L));
+ pipeline.run().waitUntilFinish();
+ }
+
+ // Test cases for Reader's arguments.
+ @Test
+ public void testMissingScanRequestFn() {
+ thrown.expectMessage("withScanRequestFn() is required");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withDynamoDbClientProvider(
+ DynamoDbClientProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("withScanRequestFn() is required");
+ } catch (IllegalArgumentException ex) {
+ assertEquals("withScanRequestFn() is required", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testMissingDynamoDbClientProvider() {
+ thrown.expectMessage("withDynamoDbClientProvider() is required");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input -> ScanRequest.builder().tableName(tableName).totalSegments(3).build()));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("withDynamoDbClientProvider() is required");
+ } catch (IllegalArgumentException ex) {
+ assertEquals("withDynamoDbClientProvider() is required", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testMissingTotalSegments() {
+ thrown.expectMessage("TotalSegments is required with withScanRequestFn()");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input -> ScanRequest.builder().tableName(tableName).build())
+ .withDynamoDbClientProvider(
+ DynamoDbClientProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("TotalSegments is required with withScanRequestFn()");
+ } catch (IllegalArgumentException ex) {
+ assertEquals("TotalSegments is required with withScanRequestFn()", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testNegativeTotalSegments() {
+ thrown.expectMessage("TotalSegments is required with withScanRequestFn() and greater zero");
+ pipeline.apply(
+ DynamoDBIO.read()
+ .withScanRequestFn(
+ (SerializableFunction<Void, ScanRequest>)
+ input -> ScanRequest.builder().tableName(tableName).totalSegments(-1).build())
+ .withDynamoDbClientProvider(
+ DynamoDbClientProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+ try {
+ pipeline.run().waitUntilFinish();
+ fail("withTotalSegments() is expected and greater than zero");
+ } catch (IllegalArgumentException ex) {
+ assertEquals(
+ "TotalSegments is required with withScanRequestFn() and greater zero", ex.getMessage());
+ }
+ }
+
+ @Test
+ public void testWriteDataToDynamo() {
+ List<KV<String, Integer>> items =
+ ImmutableList.of(KV.of("test1", 111), KV.of("test2", 222), KV.of("test3", 333));
+
+ final PCollection<Void> output =
+ pipeline
+ .apply(Create.of(items))
+ .apply(
+ DynamoDBIO.<KV<String, Integer>>write()
+ .withWriteRequestMapperFn(
+ (SerializableFunction<KV<String, Integer>, KV<String, WriteRequest>>)
+ entry -> {
+ Map<String, AttributeValue> putRequest =
+ ImmutableMap.of(
+ "hashKey1",
+ AttributeValue.builder().s(entry.getKey()).build(),
+ "rangeKey2",
+ AttributeValue.builder()
+ .n(entry.getValue().toString())
+ .build());
+
+ WriteRequest writeRequest =
+ WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(putRequest).build())
+ .build();
+ return KV.of(tableName, writeRequest);
+ })
+ .withRetryConfiguration(
+ DynamoDBIO.RetryConfiguration.builder()
+ .setMaxAttempts(5)
+ .setMaxDuration(Duration.standardMinutes(1))
+ .setRetryPredicate(DEFAULT_RETRY_PREDICATE)
+ .build())
+ .withDynamoDbClientProvider(
+ DynamoDbClientProviderMock.of(DynamoDBIOTestHelper.getDynamoDBClient())));
+
+ final PCollection<Long> publishedResultsSize = output.apply(Count.globally());
+ PAssert.that(publishedResultsSize).containsInAnyOrder(0L);
+
+ pipeline.run().waitUntilFinish();
+
+ // Make sure data written to the table are in the table.
+ int actualItemCount = DynamoDBIOTestHelper.readDataFromTable(tableName).size();
+ assertEquals(3, actualItemCount);
+ }
+
+ @Rule public transient ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testRetries() throws Throwable {
+ thrown.expectMessage("Error writing to DynamoDB");
+
+ List<KV<String, Integer>> items =
+ ImmutableList.of(KV.of("test1", 111), KV.of("test2", 222), KV.of("test3", 333));
+
+ DynamoDbClient amazonDynamoDBMock = Mockito.mock(DynamoDbClient.class);
+ Mockito.when(amazonDynamoDBMock.batchWriteItem(Mockito.any(BatchWriteItemRequest.class)))
+ .thenThrow(DynamoDbException.builder().message("Service unavailable").build());
+
+ pipeline
+ .apply(Create.of(items))
+ .apply(
+ DynamoDBIO.<KV<String, Integer>>write()
+ .withWriteRequestMapperFn(
+ (SerializableFunction<KV<String, Integer>, KV<String, WriteRequest>>)
+ entry -> {
+ Map<String, AttributeValue> putRequest =
+ ImmutableMap.of(
+ "hashKey1", AttributeValue.builder().s(entry.getKey()).build(),
+ "rangeKey2",
+ AttributeValue.builder()
+ .n(entry.getValue().toString())
+ .build());
+
+ WriteRequest writeRequest =
+ WriteRequest.builder()
+ .putRequest(PutRequest.builder().item(putRequest).build())
+ .build();
+ return KV.of(tableName, writeRequest);
+ })
+ .withRetryConfiguration(
+ DynamoDBIO.RetryConfiguration.builder()
+ .setMaxAttempts(4)
+ .setMaxDuration(Duration.standardSeconds(10))
+ .setRetryPredicate(DEFAULT_RETRY_PREDICATE)
+ .build())
+ .withDynamoDbClientProvider(DynamoDbClientProviderMock.of(amazonDynamoDBMock)));
+
+ try {
+ pipeline.run().waitUntilFinish();
+ } catch (final Pipeline.PipelineExecutionException e) {
+ // check 3 retries were initiated by inspecting the log before passing on the exception
+ expectedLogs.verifyWarn(String.format(DynamoDBIO.Write.WriteFn.RETRY_ATTEMPT_LOG, 1));
+ expectedLogs.verifyWarn(String.format(DynamoDBIO.Write.WriteFn.RETRY_ATTEMPT_LOG, 2));
+ expectedLogs.verifyWarn(String.format(DynamoDBIO.Write.WriteFn.RETRY_ATTEMPT_LOG, 3));
+ throw e.getCause();
+ }
+ fail("Pipeline is expected to fail because we were unable to write to DynamoDb.");
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java
new file mode 100644
index 0000000..546f56f
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import java.io.Serializable;
+import java.net.InetAddress;
+import java.net.URI;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.testcontainers.containers.GenericContainer;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+import software.amazon.awssdk.services.dynamodb.model.AttributeDefinition;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+import software.amazon.awssdk.services.dynamodb.model.BatchWriteItemRequest;
+import software.amazon.awssdk.services.dynamodb.model.CreateTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.CreateTableResponse;
+import software.amazon.awssdk.services.dynamodb.model.DeleteTableRequest;
+import software.amazon.awssdk.services.dynamodb.model.KeySchemaElement;
+import software.amazon.awssdk.services.dynamodb.model.KeyType;
+import software.amazon.awssdk.services.dynamodb.model.ListTablesResponse;
+import software.amazon.awssdk.services.dynamodb.model.ProvisionedThroughput;
+import software.amazon.awssdk.services.dynamodb.model.PutRequest;
+import software.amazon.awssdk.services.dynamodb.model.ScalarAttributeType;
+import software.amazon.awssdk.services.dynamodb.model.ScanRequest;
+import software.amazon.awssdk.services.dynamodb.model.ScanResponse;
+import software.amazon.awssdk.services.dynamodb.model.TableDescription;
+import software.amazon.awssdk.services.dynamodb.model.TableStatus;
+import software.amazon.awssdk.services.dynamodb.model.WriteRequest;
+
+/** A utility to generate test table and data for {@link DynamoDBIOTest}. */
+class DynamoDBIOTestHelper implements Serializable {
+
+ @Rule
+ public static GenericContainer dynamoContainer =
+ new GenericContainer<>("amazon/dynamodb-local:latest").withExposedPorts(8000);
+
+ private static DynamoDbClient dynamoDBClient;
+
+ static final String ATTR_NAME_1 = "hashKey1";
+ static final String ATTR_NAME_2 = "rangeKey2";
+
+ static void startServerClient() {
+ dynamoContainer.start();
+
+ if (dynamoDBClient == null) {
+ dynamoDBClient = getDynamoDBClient();
+ }
+ }
+
+ static void stopServerClient(String tableName) {
+ if (dynamoDBClient != null) {
+ dynamoDBClient.close();
+ }
+ dynamoContainer.stop();
+ }
+
+ static DynamoDbClient getDynamoDBClient() {
+ // Note: each test case got to have their own dynamodb client obj, can't be shared
+ // Otherwise will run into connection pool issue
+ return DynamoDbClient.builder()
+ .endpointOverride(URI.create(getContainerEndpoint()))
+ .region(Region.US_EAST_1)
+ .credentialsProvider(
+ StaticCredentialsProvider.create(AwsBasicCredentials.create("accessKey", "secretKey")))
+ .build();
+ }
+
+ static List<Map<String, AttributeValue>> generateTestData(String tableName, int numOfItems) {
+ BatchWriteItemRequest batchWriteItemRequest =
+ generateBatchWriteItemRequest(tableName, numOfItems);
+
+ dynamoDBClient.batchWriteItem(batchWriteItemRequest);
+ ScanResponse scanResult =
+ dynamoDBClient.scan(ScanRequest.builder().tableName(tableName).build());
+
+ List<Map<String, AttributeValue>> items = scanResult.items();
+ Assert.assertEquals(numOfItems, items.size());
+ return items;
+ }
+
+ static BatchWriteItemRequest generateBatchWriteItemRequest(String tableName, int numOfItems) {
+ BatchWriteItemRequest batchWriteItemRequest =
+ BatchWriteItemRequest.builder()
+ .requestItems(ImmutableMap.of(tableName, generateWriteRequests(numOfItems)))
+ .build();
+ return batchWriteItemRequest;
+ }
+
+ static List<WriteRequest> generateWriteRequests(int numOfItem) {
+ List<WriteRequest> writeRequests = new ArrayList<>();
+ for (int i = 1; i <= numOfItem; i++) {
+ WriteRequest writeRequest =
+ WriteRequest.builder()
+ .putRequest(generatePutRequest("hashKeyDataStr_" + i, "1000" + i))
+ .build();
+ writeRequests.add(writeRequest);
+ }
+ return writeRequests;
+ }
+
+ private static PutRequest generatePutRequest(String hashKeyData, String rangeKeyData) {
+ ImmutableMap<String, AttributeValue> attrValueMap =
+ ImmutableMap.of(
+ ATTR_NAME_1, AttributeValue.builder().s(hashKeyData).build(),
+ ATTR_NAME_2, AttributeValue.builder().n(rangeKeyData).build());
+
+ PutRequest.Builder putRequestBuilder = PutRequest.builder();
+ putRequestBuilder.item(attrValueMap);
+ return putRequestBuilder.build();
+ }
+
+ static List<Map<String, AttributeValue>> readDataFromTable(String tableName) {
+ ScanRequest scanRequest = ScanRequest.builder().tableName(tableName).build();
+ ScanResponse scanResponse = dynamoDBClient.scan(scanRequest);
+ return scanResponse.items();
+ }
+
+ static void deleteTestTable(String tableName) {
+ DeleteTableRequest request = DeleteTableRequest.builder().tableName(tableName).build();
+ dynamoDBClient.deleteTable(request);
+ }
+
+ static void createTestTable(String tableName) {
+ CreateTableResponse res = createDynamoTable(tableName);
+
+ TableDescription tableDesc = res.tableDescription();
+
+ Assert.assertEquals(tableName, tableDesc.tableName());
+ Assert.assertTrue(tableDesc.keySchema().toString().contains(ATTR_NAME_1));
+ Assert.assertTrue(tableDesc.keySchema().toString().contains(ATTR_NAME_2));
+
+ Assert.assertEquals(tableDesc.provisionedThroughput().readCapacityUnits(), Long.valueOf(1000));
+ Assert.assertEquals(tableDesc.provisionedThroughput().writeCapacityUnits(), Long.valueOf(1000));
+ Assert.assertEquals(TableStatus.ACTIVE, tableDesc.tableStatus());
+ Assert.assertEquals(
+ "arn:aws:dynamodb:ddblocal:000000000000:table/" + tableName, tableDesc.tableArn());
+
+ ListTablesResponse tables = dynamoDBClient.listTables();
+ Assert.assertEquals(1, tables.tableNames().size());
+ }
+
+ private static CreateTableResponse createDynamoTable(String tableName) {
+
+ ImmutableList<AttributeDefinition> attributeDefinitions =
+ ImmutableList.of(
+ AttributeDefinition.builder()
+ .attributeName(ATTR_NAME_1)
+ .attributeType(ScalarAttributeType.S)
+ .build(),
+ AttributeDefinition.builder()
+ .attributeName(ATTR_NAME_2)
+ .attributeType(ScalarAttributeType.N)
+ .build());
+
+ ImmutableList<KeySchemaElement> ks =
+ ImmutableList.of(
+ KeySchemaElement.builder().attributeName(ATTR_NAME_1).keyType(KeyType.HASH).build(),
+ KeySchemaElement.builder().attributeName(ATTR_NAME_2).keyType(KeyType.RANGE).build());
+
+ ProvisionedThroughput provisionedthroughput =
+ ProvisionedThroughput.builder().readCapacityUnits(1000L).writeCapacityUnits(1000L).build();
+ CreateTableRequest request =
+ CreateTableRequest.builder()
+ .tableName(tableName)
+ .attributeDefinitions(attributeDefinitions)
+ .keySchema(ks)
+ .provisionedThroughput(provisionedthroughput)
+ .build();
+
+ return dynamoDBClient.createTable(request);
+ }
+
+ // This helper function is copied from localstack
+ private static String getContainerEndpoint() {
+ final String address = dynamoContainer.getContainerIpAddress();
+ String ipAddress = address;
+ try {
+ ipAddress = InetAddress.getByName(address).getHostAddress();
+ } catch (UnknownHostException ignored) {
+ }
+ ipAddress = ipAddress + ".nip.io";
+ while (true) {
+ try {
+ //noinspection ResultOfMethodCallIgnored
+ InetAddress.getAllByName(ipAddress);
+ break;
+ } catch (UnknownHostException ignored) {
+ }
+ }
+ return "http://" + ipAddress + ":" + dynamoContainer.getFirstMappedPort();
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProviderMock.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProviderMock.java
new file mode 100644
index 0000000..a017966
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProviderMock.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.dynamodb;
+
+import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
+
+/** Mocking AwsClientProvider. */
+public class DynamoDbClientProviderMock implements DynamoDbClientProvider {
+
+ private static DynamoDbClientProviderMock instance = new DynamoDbClientProviderMock();
+ private static DynamoDbClient db;
+
+ private DynamoDbClientProviderMock() {}
+
+ public static DynamoDbClientProviderMock of(DynamoDbClient dynamoDB) {
+ db = dynamoDB;
+ return instance;
+ }
+
+ @Override
+ public DynamoDbClient getDynamoDbClient() {
+ return db;
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsModuleTest.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsModuleTest.java
new file mode 100644
index 0000000..4fbd357
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsModuleTest.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.options;
+
+import static org.hamcrest.Matchers.hasItem;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.net.URI;
+import java.util.List;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+import org.hamcrest.Matchers;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
+import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
+import software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider;
+import software.amazon.awssdk.http.apache.ProxyConfiguration;
+
+/** Tests {@link AwsModule}. */
+@RunWith(JUnit4.class)
+public class AwsModuleTest {
+ private final ObjectMapper objectMapper = new ObjectMapper().registerModule(new AwsModule());
+
+ @Test
+ public void testObjectMapperIsAbleToFindModule() {
+ List<Module> modules = ObjectMapper.findModules(ReflectHelpers.findClassLoader());
+ assertThat(modules, hasItem(Matchers.instanceOf(AwsModule.class)));
+ }
+
+ @Test
+ public void testStaticCredentialsProviderSerializationDeserialization() throws Exception {
+ AwsCredentialsProvider credentialsProvider =
+ StaticCredentialsProvider.create(AwsBasicCredentials.create("key-id", "secret-key"));
+ String serializedCredentialsProvider = objectMapper.writeValueAsString(credentialsProvider);
+ AwsCredentialsProvider deserializedCredentialsProvider =
+ objectMapper.readValue(serializedCredentialsProvider, AwsCredentialsProvider.class);
+ assertEquals(credentialsProvider.getClass(), deserializedCredentialsProvider.getClass());
+ assertEquals(
+ credentialsProvider.resolveCredentials().accessKeyId(),
+ deserializedCredentialsProvider.resolveCredentials().accessKeyId());
+ assertEquals(
+ credentialsProvider.resolveCredentials().secretAccessKey(),
+ deserializedCredentialsProvider.resolveCredentials().secretAccessKey());
+ }
+
+ @Test
+ public void testAwsCredentialsProviderSerializationDeserialization() throws Exception {
+ AwsCredentialsProvider credentialsProvider = DefaultCredentialsProvider.create();
+ String serializedCredentialsProvider = objectMapper.writeValueAsString(credentialsProvider);
+ AwsCredentialsProvider deserializedCredentialsProvider =
+ objectMapper.readValue(serializedCredentialsProvider, DefaultCredentialsProvider.class);
+ assertEquals(credentialsProvider.getClass(), deserializedCredentialsProvider.getClass());
+
+ credentialsProvider = EnvironmentVariableCredentialsProvider.create();
+ serializedCredentialsProvider = objectMapper.writeValueAsString(credentialsProvider);
+ deserializedCredentialsProvider =
+ objectMapper.readValue(serializedCredentialsProvider, AwsCredentialsProvider.class);
+ assertEquals(credentialsProvider.getClass(), deserializedCredentialsProvider.getClass());
+
+ credentialsProvider = SystemPropertyCredentialsProvider.create();
+ serializedCredentialsProvider = objectMapper.writeValueAsString(credentialsProvider);
+ deserializedCredentialsProvider =
+ objectMapper.readValue(serializedCredentialsProvider, AwsCredentialsProvider.class);
+ assertEquals(credentialsProvider.getClass(), deserializedCredentialsProvider.getClass());
+
+ credentialsProvider = ProfileCredentialsProvider.create();
+ serializedCredentialsProvider = objectMapper.writeValueAsString(credentialsProvider);
+ deserializedCredentialsProvider =
+ objectMapper.readValue(serializedCredentialsProvider, AwsCredentialsProvider.class);
+ assertEquals(credentialsProvider.getClass(), deserializedCredentialsProvider.getClass());
+
+ credentialsProvider = ContainerCredentialsProvider.builder().build();
+ serializedCredentialsProvider = objectMapper.writeValueAsString(credentialsProvider);
+ deserializedCredentialsProvider =
+ objectMapper.readValue(serializedCredentialsProvider, AwsCredentialsProvider.class);
+ assertEquals(credentialsProvider.getClass(), deserializedCredentialsProvider.getClass());
+ }
+
+ @Test
+ public void testProxyConfigurationSerializationDeserialization() throws Exception {
+ ProxyConfiguration proxyConfiguration =
+ ProxyConfiguration.builder()
+ .endpoint(URI.create("http://localhost:8080"))
+ .username("username")
+ .password("password")
+ .build();
+ String valueAsJson = objectMapper.writeValueAsString(proxyConfiguration);
+ ProxyConfiguration deserializedProxyConfiguration =
+ objectMapper.readValue(valueAsJson, ProxyConfiguration.class);
+ assertEquals("localhost", deserializedProxyConfiguration.host());
+ assertEquals(8080, deserializedProxyConfiguration.port());
+ assertEquals("username", deserializedProxyConfiguration.username());
+ assertEquals("password", deserializedProxyConfiguration.password());
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
new file mode 100644
index 0000000..0557b64
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import org.mockito.Mockito;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.InternalErrorException;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Mock class to test a failed publish of a msg. */
+public class SnsClientMockErrors implements SnsClient {
+
+ @Override
+ public PublishResponse publish(PublishRequest publishRequest) {
+ throw InternalErrorException.builder().message("Service unavailable").build();
+ }
+
+ @Override
+ public GetTopicAttributesResponse getTopicAttributes(
+ GetTopicAttributesRequest topicAttributesRequest) {
+ GetTopicAttributesResponse response = Mockito.mock(GetTopicAttributesResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+
+ return response;
+ }
+
+ @Override
+ public String serviceName() {
+ return null;
+ }
+
+ @Override
+ public void close() {}
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
new file mode 100644
index 0000000..ca49b1f
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import java.util.HashMap;
+import java.util.UUID;
+import org.mockito.Mockito;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.sns.SnsClient;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesRequest;
+import software.amazon.awssdk.services.sns.model.GetTopicAttributesResponse;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+// import static org.mockito.BDDMockito.given;
+
+/** Mock class to test a successful publish of a msg. */
+public class SnsClientMockSuccess implements SnsClient {
+
+ @Override
+ public PublishResponse publish(PublishRequest publishRequest) {
+ PublishResponse response = Mockito.mock(PublishResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.headers()).thenReturn(new HashMap<>());
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+ Mockito.when(response.messageId()).thenReturn(UUID.randomUUID().toString());
+
+ return response;
+ }
+
+ @Override
+ public GetTopicAttributesResponse getTopicAttributes(
+ GetTopicAttributesRequest topicAttributesRequest) {
+ GetTopicAttributesResponse response = Mockito.mock(GetTopicAttributesResponse.class);
+ SdkHttpResponse metadata = Mockito.mock(SdkHttpResponse.class);
+
+ Mockito.when(metadata.statusCode()).thenReturn(200);
+ Mockito.when(response.sdkHttpResponse()).thenReturn(metadata);
+
+ return response;
+ }
+
+ @Override
+ public String serviceName() {
+ return null;
+ }
+
+ @Override
+ public void close() {}
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
new file mode 100644
index 0000000..e9372dc
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws2.sns;
+
+import static org.junit.Assert.fail;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.testing.ExpectedLogs;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import software.amazon.awssdk.services.sns.model.PublishRequest;
+import software.amazon.awssdk.services.sns.model.PublishResponse;
+
+/** Tests to verify writes to Sns. */
+@RunWith(JUnit4.class)
+public class SnsIOTest implements Serializable {
+
+ private static final String topicArn = "arn:aws:sns:us-west-2:5880:topic-FMFEHJ47NRFO";
+
+ @Rule public TestPipeline p = TestPipeline.create();
+ @Rule public final transient ExpectedLogs expectedLogs = ExpectedLogs.none(SnsIO.class);
+
+ private static PublishRequest createSampleMessage(String message) {
+ return PublishRequest.builder().topicArn(topicArn).message(message).build();
+ }
+
+ @Test
+ public void testDataWritesToSNS() {
+ ImmutableList<String> input = ImmutableList.of("message1", "message2");
+
+ final PCollection<PublishResponse> results =
+ p.apply(Create.of(input))
+ .apply(
+ SnsIO.<String>write()
+ .withPublishRequestFn(SnsIOTest::createSampleMessage)
+ .withTopicArn(topicArn)
+ .withRetryConfiguration(
+ SnsIO.RetryConfiguration.create(
+ 5, org.joda.time.Duration.standardMinutes(1)))
+ .withSnsClientProvider(SnsClientMockSuccess::new));
+
+ final PCollection<Long> publishedResultsSize = results.apply(Count.globally());
+ PAssert.that(publishedResultsSize).containsInAnyOrder(ImmutableList.of(2L));
+ p.run().waitUntilFinish();
+ }
+
+ @Rule public ExpectedException thrown = ExpectedException.none();
+
+ @Test
+ public void testRetries() throws Throwable {
+ thrown.expectMessage("Error writing to SNS");
+
+ ImmutableList<String> input = ImmutableList.of("message1", "message2");
+
+ p.apply(Create.of(input))
+ .apply(
+ SnsIO.<String>write()
+ .withPublishRequestFn(SnsIOTest::createSampleMessage)
+ .withTopicArn(topicArn)
+ .withRetryConfiguration(
+ SnsIO.RetryConfiguration.create(4, org.joda.time.Duration.standardSeconds(10)))
+ .withSnsClientProvider(SnsClientMockErrors::new));
+
+ try {
+ p.run();
+ } catch (final Pipeline.PipelineExecutionException e) {
+ // check 3 retries were initiated by inspecting the log before passing on the exception
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 1));
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 2));
+ expectedLogs.verifyWarn(String.format(SnsIO.Write.SnsWriterFn.RETRY_ATTEMPT_LOG, 3));
+ throw e.getCause();
+ }
+ fail("Pipeline is expected to fail because we were unable to write to SNS.");
+ }
+}
diff --git a/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
new file mode 100644
index 0000000..1f0955d
--- /dev/null
+++ b/sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
@@ -0,0 +1 @@
+mock-maker-inline
diff --git a/sdks/java/io/amqp/build.gradle b/sdks/java/io/amqp/build.gradle
index f74033b..a4c35a3 100644
--- a/sdks/java/io/amqp/build.gradle
+++ b/sdks/java/io/amqp/build.gradle
@@ -17,16 +17,16 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.amqp')
description = "Apache Beam :: SDKs :: Java :: IO :: AMQP"
ext.summary = "IO to read and write using AMQP 1.0 protocol (http://www.amqp.org)."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.joda_time
- shadow "org.apache.qpid:proton-j:0.13.1"
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.joda_time
+ compile "org.apache.qpid:proton-j:0.13.1"
testCompile library.java.slf4j_api
testCompile library.java.junit
testCompile library.java.hamcrest_core
@@ -35,5 +35,5 @@
testCompile library.java.activemq_amqp
testCompile library.java.activemq_junit
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java b/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java
index a757ae1..67a0b2c 100644
--- a/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java
+++ b/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.amqp;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.qpid.proton.message.Message;
import org.apache.qpid.proton.messenger.Messenger;
import org.apache.qpid.proton.messenger.Tracker;
diff --git a/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java b/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java
index d399117..9377f1e 100644
--- a/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java
+++ b/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoder.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.CustomCoder;
import org.apache.beam.sdk.util.VarInt;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
import org.apache.qpid.proton.message.Message;
/** A coder for AMQP message. */
diff --git a/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java b/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java
index 950b6d7..eb2fbbe 100644
--- a/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java
+++ b/sdks/java/io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.coders.CoderProviderRegistrar;
import org.apache.beam.sdk.coders.CoderProviders;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.qpid.proton.message.Message;
/** A {@link CoderProviderRegistrar} for standard types used with {@link AmqpIO}. */
diff --git a/sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java b/sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java
index 32a6d87..6c5c039 100644
--- a/sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java
+++ b/sdks/java/io/amqp/src/test/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderTest.java
@@ -22,7 +22,7 @@
import java.util.Collections;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.qpid.proton.amqp.messaging.AmqpValue;
import org.apache.qpid.proton.message.Message;
import org.junit.Rule;
diff --git a/sdks/java/io/bigquery-io-perf-tests/build.gradle b/sdks/java/io/bigquery-io-perf-tests/build.gradle
new file mode 100644
index 0000000..ce27468
--- /dev/null
+++ b/sdks/java/io/bigquery-io-perf-tests/build.gradle
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(exportJavadoc: false, publish: false)
+provideIntegrationTestingDependencies()
+enableJavaPerformanceTesting()
+
+description = "Apache Beam :: SDKs :: Java :: Google BigQuery IO Performance tests"
+ext.summary = "Performance tests for Google BigQuery IO sources and sinks"
+
+dependencies {
+ compile library.java.google_api_services_bigquery
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:io:google-cloud-platform", configuration: "testRuntime")
+ testCompile project(":sdks:java:io:synthetic")
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.jaxb_api
+}
diff --git a/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java b/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java
new file mode 100644
index 0000000..79472bb
--- /dev/null
+++ b/sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.bigqueryioperftests;
+
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import com.google.cloud.Timestamp;
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.BigQueryOptions;
+import com.google.cloud.bigquery.TableId;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.UUID;
+import java.util.function.Function;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.common.IOITHelper;
+import org.apache.beam.sdk.io.common.IOTestPipelineOptions;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
+import org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource;
+import org.apache.beam.sdk.io.synthetic.SyntheticOptions;
+import org.apache.beam.sdk.io.synthetic.SyntheticSourceOptions;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.Validation;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.testutils.NamedTestResult;
+import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
+import org.apache.beam.sdk.testutils.metrics.MetricsReader;
+import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * A performance test of {@link org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO}.
+ *
+ * <p>Usage:
+ *
+ * <pre>
+ * ./gradlew integrationTest -p sdks/java/io/gcp/bigquery -DintegrationTestPipelineOptions='[
+ * "--testBigQueryDataset=test-dataset",
+ * "--testBigQueryTable=test-table",
+ * "--metricsBigQueryDataset=metrics-dataset",
+ * "--metricsBigQueryTable=metrics-table",
+ * "--writeMethod=FILE_LOADS",
+ * "--sourceOptions={"numRecords":"1000", "keySize":1, valueSize:"1024"}
+ * }"]'
+ * --tests org.apache.beam.sdk.io.gcp.bigQuery.BigQueryIOIT
+ * -DintegrationTestRunner=direct
+ * </pre>
+ */
+@RunWith(JUnit4.class)
+public class BigQueryIOIT {
+ private static final String NAMESPACE = BigQueryIOIT.class.getName();
+ private static final String TEST_ID = UUID.randomUUID().toString();
+ private static final String TEST_TIMESTAMP = Timestamp.now().toString();
+ private static final String READ_TIME_METRIC_NAME = "read_time";
+ private static final String WRITE_TIME_METRIC_NAME = "write_time";
+ private static String metricsBigQueryTable;
+ private static String metricsBigQueryDataset;
+ private static String testBigQueryDataset;
+ private static String testBigQueryTable;
+ private static SyntheticSourceOptions sourceOptions;
+ private static String tableQualifier;
+ private static String tempRoot;
+ private static BigQueryPerfTestOptions options;
+
+ @BeforeClass
+ public static void setup() throws IOException {
+ options = IOITHelper.readIOTestPipelineOptions(BigQueryPerfTestOptions.class);
+ tempRoot = options.getTempRoot();
+ sourceOptions =
+ SyntheticOptions.fromJsonString(options.getSourceOptions(), SyntheticSourceOptions.class);
+ metricsBigQueryDataset = options.getMetricsBigQueryDataset();
+ metricsBigQueryTable = options.getMetricsBigQueryTable();
+ testBigQueryDataset = options.getTestBigQueryDataset();
+ testBigQueryTable = options.getTestBigQueryTable();
+ BigQueryOptions bigQueryOptions = BigQueryOptions.newBuilder().build();
+ tableQualifier =
+ String.format(
+ "%s:%s.%s", bigQueryOptions.getProjectId(), testBigQueryDataset, testBigQueryTable);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ BigQueryOptions options = BigQueryOptions.newBuilder().build();
+ BigQuery client = options.getService();
+ TableId tableId = TableId.of(options.getProjectId(), testBigQueryDataset, testBigQueryTable);
+ client.delete(tableId);
+ }
+
+ @Test
+ public void testWriteThenRead() {
+ testWrite();
+ testRead();
+ }
+
+ private void testWrite() {
+ Pipeline pipeline = Pipeline.create(options);
+
+ BigQueryIO.Write.Method method = BigQueryIO.Write.Method.valueOf(options.getWriteMethod());
+ pipeline
+ .apply("Read from source", Read.from(new SyntheticBoundedSource(sourceOptions)))
+ .apply("Gather time", ParDo.of(new TimeMonitor<>(NAMESPACE, WRITE_TIME_METRIC_NAME)))
+ .apply("Map records", ParDo.of(new MapKVToV()))
+ .apply(
+ "Write to BQ",
+ BigQueryIO.<byte[]>write()
+ .to(tableQualifier)
+ .withFormatFunction(
+ input -> {
+ TableRow tableRow = new TableRow();
+ tableRow.set("data", input);
+ return tableRow;
+ })
+ .withCustomGcsTempLocation(ValueProvider.StaticValueProvider.of(tempRoot))
+ .withMethod(method)
+ .withSchema(
+ new TableSchema()
+ .setFields(
+ Collections.singletonList(
+ new TableFieldSchema().setName("data").setType("BYTES")))));
+
+ PipelineResult pipelineResult = pipeline.run();
+ pipelineResult.waitUntilFinish();
+ extractAndPublishTime(pipelineResult, WRITE_TIME_METRIC_NAME);
+ }
+
+ private void testRead() {
+ Pipeline pipeline = Pipeline.create(options);
+ pipeline
+ .apply("Read from BQ", BigQueryIO.readTableRows().from(tableQualifier))
+ .apply("Gather time", ParDo.of(new TimeMonitor<>(NAMESPACE, READ_TIME_METRIC_NAME)));
+ PipelineResult result = pipeline.run();
+ result.waitUntilFinish();
+ extractAndPublishTime(result, READ_TIME_METRIC_NAME);
+ }
+
+ private void extractAndPublishTime(PipelineResult pipelineResult, String writeTimeMetricName) {
+ NamedTestResult metricResult =
+ getMetricSupplier(writeTimeMetricName).apply(new MetricsReader(pipelineResult, NAMESPACE));
+ IOITMetrics.publish(
+ TEST_ID,
+ TEST_TIMESTAMP,
+ metricsBigQueryDataset,
+ metricsBigQueryTable,
+ Collections.singletonList(metricResult));
+ }
+
+ private static Function<MetricsReader, NamedTestResult> getMetricSupplier(String metricName) {
+ return reader -> {
+ long startTime = reader.getStartTimeMetric(metricName);
+ long endTime = reader.getEndTimeMetric(metricName);
+ return NamedTestResult.create(
+ TEST_ID, TEST_TIMESTAMP, metricName, (endTime - startTime) / 1e3);
+ };
+ }
+
+ /** Options for this io performance test. */
+ public interface BigQueryPerfTestOptions extends IOTestPipelineOptions {
+ @Description("Synthetic source options")
+ @Validation.Required
+ String getSourceOptions();
+
+ void setSourceOptions(String value);
+
+ @Description("BQ dataset for the test data")
+ String getTestBigQueryDataset();
+
+ void setTestBigQueryDataset(String dataset);
+
+ @Description("BQ table for test data")
+ String getTestBigQueryTable();
+
+ void setTestBigQueryTable(String table);
+
+ @Description("BQ dataset for the metrics data")
+ String getMetricsBigQueryDataset();
+
+ void setMetricsBigQueryDataset(String dataset);
+
+ @Description("BQ table for metrics data")
+ String getMetricsBigQueryTable();
+
+ void setMetricsBigQueryTable(String table);
+
+ @Description("Should test use streaming writes or batch loads to BQ")
+ String getWriteMethod();
+
+ void setWriteMethod(String value);
+ }
+
+ private static class MapKVToV extends DoFn<KV<byte[], byte[]>, byte[]> {
+ @ProcessElement
+ public void process(ProcessContext context) {
+ context.output(context.element().getValue());
+ }
+ }
+}
diff --git a/sdks/java/io/cassandra/build.gradle b/sdks/java/io/cassandra/build.gradle
index db3896f..ca3015c4 100644
--- a/sdks/java/io/cassandra/build.gradle
+++ b/sdks/java/io/cassandra/build.gradle
@@ -19,23 +19,7 @@
plugins { id 'org.apache.beam.module' }
// Do not relocate guava to avoid issues with Cassandra's version.
-applyJavaNature(
- shadowClosure: {
- dependencies {
- // is default, but when omitted the default action is to include all runtime deps
- include(dependency(project.library.java.guava))
-
- // hack: now exclude the only thing that was included
- exclude(dependency(project.library.java.guava))
-
- }
- },
- shadowJarValidationExcludes: [
- "org/apache/beam/**",
- "com/google/common/**",
- "com/google/thirdparty/**"
- ]
-)
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.cassandra')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -46,12 +30,12 @@
def achilles_version = "6.0.2"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow library.java.cassandra_driver_core
- shadow library.java.cassandra_driver_mapping
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.slf4j_api
+ compile library.java.cassandra_driver_core
+ compile library.java.cassandra_driver_mapping
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
@@ -61,5 +45,5 @@
testCompile group: 'info.archinnov', name: 'achilles-junit', version: "$achilles_version"
testCompile library.java.jackson_jaxb_annotations
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java b/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
index f9e4080..8368982 100644
--- a/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
+++ b/sdks/java/io/cassandra/src/main/java/org/apache/beam/sdk/io/cassandra/CassandraIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.cassandra;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.ColumnMetadata;
@@ -57,9 +57,9 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java b/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
index 3d2b01f..f3577a3 100644
--- a/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
+++ b/sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
@@ -71,14 +71,15 @@
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListeningExecutorService;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors;
import org.apache.cassandra.service.StorageServiceMBean;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.ClassRule;
+import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
@@ -89,6 +90,7 @@
/** Tests of {@link CassandraIO}. */
@RunWith(JUnit4.class)
+@Ignore("Ignore until https://issues.apache.org/jira/browse/BEAM-8025 is resolved")
public class CassandraIOTest implements Serializable {
private static final long NUM_ROWS = 20L;
private static final String CASSANDRA_KEYSPACE = "beam_ks";
diff --git a/sdks/java/io/clickhouse/build.gradle b/sdks/java/io/clickhouse/build.gradle
index 062d197..ee8d382 100644
--- a/sdks/java/io/clickhouse/build.gradle
+++ b/sdks/java/io/clickhouse/build.gradle
@@ -21,6 +21,7 @@
id 'ca.coglinc.javacc'
}
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.io.clickhouse',
// javacc generated code produces lint warnings
disableLintWarnings: ['dep-ann']
)
@@ -50,14 +51,14 @@
dependencies {
javacc "net.java.dev.javacc:javacc:4.0"
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.joda_time
- shadow "ru.yandex.clickhouse:clickhouse-jdbc:$clickhouse_jdbc_version"
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.joda_time
+ compile "ru.yandex.clickhouse:clickhouse-jdbc:$clickhouse_jdbc_version"
testCompile library.java.slf4j_api
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile "org.testcontainers:clickhouse:$testcontainers_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java b/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java
index d37c3ef4..5d7d79f 100644
--- a/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java
+++ b/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java
@@ -45,9 +45,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseWriter.java b/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseWriter.java
index 32c95a3..73db353 100644
--- a/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseWriter.java
+++ b/sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseWriter.java
@@ -22,8 +22,8 @@
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.io.clickhouse.TableSchema.ColumnType;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.joda.time.Days;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
diff --git a/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java b/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
index edf29a5..8c39688 100644
--- a/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
+++ b/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.util.BackOffUtils;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.Sleeper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.joda.time.Duration;
import org.junit.AfterClass;
import org.junit.BeforeClass;
diff --git a/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/TableSchemaTest.java b/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/TableSchemaTest.java
index 5039b4b9..31610f0 100644
--- a/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/TableSchemaTest.java
+++ b/sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/TableSchemaTest.java
@@ -22,7 +22,7 @@
import java.util.Map;
import org.apache.beam.sdk.io.clickhouse.TableSchema.ColumnType;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
/** Tests for {@link TableSchema}. */
diff --git a/sdks/java/io/common/build.gradle b/sdks/java/io/common/build.gradle
index f137578..a17e2b7 100644
--- a/sdks/java/io/common/build.gradle
+++ b/sdks/java/io/common/build.gradle
@@ -17,14 +17,14 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.common')
description = "Apache Beam :: SDKs :: Java :: IO :: Common"
ext.summary = "Code used by all Beam IOs"
dependencies {
- shadowTest library.java.vendored_guava_20_0
+ testCompile library.java.vendored_guava_26_0_jre
testCompile library.java.junit
testCompile library.java.postgres
- shadowTest project(path: ":sdks:java:core", configuration: "shadow")
+ testCompile project(path: ":sdks:java:core", configuration: "shadow")
}
diff --git a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
index 6d610f2..c324c4d 100644
--- a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
+++ b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
@@ -47,6 +47,15 @@
}
}
+ public static void createTableForRowWithSchema(DataSource dataSource, String tableName)
+ throws SQLException {
+ try (Connection connection = dataSource.getConnection()) {
+ try (Statement statement = connection.createStatement()) {
+ statement.execute(String.format("create table %s (name VARCHAR(500), id INT)", tableName));
+ }
+ }
+ }
+
public static void deleteTable(DataSource dataSource, String tableName) throws SQLException {
if (tableName != null) {
try (Connection connection = dataSource.getConnection();
@@ -69,4 +78,13 @@
options.getPostgresPort(),
options.getPostgresDatabaseName());
}
+
+ public static void createTableWithStatement(DataSource dataSource, String stmt)
+ throws SQLException {
+ try (Connection connection = dataSource.getConnection()) {
+ try (Statement statement = connection.createStatement()) {
+ statement.execute(stmt);
+ }
+ }
+ }
}
diff --git a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/HashingFn.java b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/HashingFn.java
index 4ce5b55..f20ef98 100644
--- a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/HashingFn.java
+++ b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/HashingFn.java
@@ -28,9 +28,9 @@
import org.apache.beam.sdk.coders.CoderRegistry;
import org.apache.beam.sdk.coders.SerializableCoder;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashCode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
/** Custom Function for Hashing. The combiner is combineUnordered, and accumulator is a HashCode. */
public class HashingFn extends CombineFn<String, HashingFn.Accum, String> {
diff --git a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
index 0efb43b..d14eacb 100644
--- a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
+++ b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOITHelper.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.io.common;
import java.util.Map;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.PipelineOptionsValidator;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -41,11 +42,10 @@
return hash;
}
- public static <T extends IOTestPipelineOptions> T readIOTestPipelineOptions(
- Class<T> optionsType) {
+ public static <T extends PipelineOptions> T readIOTestPipelineOptions(Class<T> optionsType) {
PipelineOptionsFactory.register(optionsType);
- IOTestPipelineOptions options = TestPipeline.testingPipelineOptions().as(optionsType);
+ PipelineOptions options = TestPipeline.testingPipelineOptions().as(optionsType);
return PipelineOptionsValidator.validate(optionsType, options);
}
diff --git a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
index 7cbd980..f8e86ca 100644
--- a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
+++ b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/TestRow.java
@@ -24,7 +24,7 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** Used to pass values around within test pipelines. */
@AutoValue
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle
index 05578ec..0788f89 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle
@@ -17,8 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-io-elasticsearch-tests-2'
-applyJavaNature()
+applyJavaNature(
+ publish: false,
+ archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-2'
+)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -30,10 +32,10 @@
def elastic_search_version = "2.4.1"
dependencies {
- testCompile project(path: ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common", configuration: "testRuntime")
testCompile project(path: ":sdks:java:core", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:elasticsearch", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ testCompile project(":sdks:java:io:elasticsearch")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.slf4j_api
testCompile "net.java.dev.jna:jna:$jna_version"
testCompile "org.apache.logging.log4j:log4j-api:$log4j_version"
@@ -43,8 +45,8 @@
testCompile library.java.commons_io_1x
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:5.6.3"
- shadowTest library.java.vendored_guava_20_0
+ testCompile library.java.vendored_guava_26_0_jre
testCompile "org.elasticsearch:elasticsearch:$elastic_search_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
index 9eccb7f..c0ea12c 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
@@ -79,8 +79,8 @@
@AfterClass
public static void afterClass() throws Exception {
- ElasticSearchIOTestUtils.deleteIndex(writeConnectionConfiguration, restClient);
- ElasticSearchIOTestUtils.deleteIndex(updateConnectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(writeConnectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(updateConnectionConfiguration, restClient);
restClient.close();
}
@@ -130,7 +130,7 @@
*/
@Test
public void testWritePartialUpdate() throws Exception {
- ElasticSearchIOTestUtils.copyIndex(
+ ElasticsearchIOTestUtils.copyIndex(
restClient,
readConnectionConfiguration.getIndex(),
updateConnectionConfiguration.getIndex());
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
index 19bda80..85419c6 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
@@ -116,7 +116,7 @@
@Before
public void before() throws Exception {
- ElasticSearchIOTestUtils.deleteIndex(connectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(connectionConfiguration, restClient);
}
@Test
@@ -131,9 +131,15 @@
}
@Test
- public void testReadWithQuery() throws Exception {
+ public void testReadWithQueryString() throws Exception {
elasticsearchIOTestCommon.setPipeline(pipeline);
- elasticsearchIOTestCommon.testReadWithQuery();
+ elasticsearchIOTestCommon.testReadWithQueryString();
+ }
+
+ @Test
+ public void testReadWithQueryValueProvider() throws Exception {
+ elasticsearchIOTestCommon.setPipeline(pipeline);
+ elasticsearchIOTestCommon.testReadWithQueryValueProvider();
}
@Test
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
index 025470c..543c0db 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
@@ -17,8 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-io-elasticsearch-tests-5'
-applyJavaNature()
+applyJavaNature(
+ publish: false,
+ archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-5'
+)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -44,15 +46,15 @@
}
dependencies {
- testCompile project(path: ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common", configuration: "testRuntime")
testCompile "org.elasticsearch.test:framework:$elastic_search_version"
testCompile "org.elasticsearch.plugin:transport-netty4-client:$elastic_search_version"
testCompile "com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.5.0"
testCompile "org.elasticsearch:elasticsearch:$elastic_search_version"
testCompile project(path: ":sdks:java:core", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:elasticsearch", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ testCompile project(":sdks:java:io:elasticsearch")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile "org.apache.logging.log4j:log4j-core:$log4j_version"
testCompile "org.apache.logging.log4j:log4j-api:$log4j_version"
testCompile library.java.slf4j_api
@@ -63,5 +65,5 @@
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:$elastic_search_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
index 8fdb444..cf52282 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
@@ -82,8 +82,8 @@
@AfterClass
public static void afterClass() throws Exception {
- ElasticSearchIOTestUtils.deleteIndex(writeConnectionConfiguration, restClient);
- ElasticSearchIOTestUtils.deleteIndex(updateConnectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(writeConnectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(updateConnectionConfiguration, restClient);
restClient.close();
}
@@ -130,11 +130,11 @@
/**
* This test verifies volume partial updates of Elasticsearch. The test dataset index is cloned
* and then a new field is added to each document using a partial update. The test then asserts
- * the updates where appied.
+ * the updates were applied.
*/
@Test
public void testWritePartialUpdate() throws Exception {
- ElasticSearchIOTestUtils.copyIndex(
+ ElasticsearchIOTestUtils.copyIndex(
restClient,
readConnectionConfiguration.getIndex(),
updateConnectionConfiguration.getIndex());
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
index 05686cd..d809cfd 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
@@ -122,12 +122,21 @@
}
@Test
- public void testReadWithQuery() throws Exception {
+ public void testReadWithQueryString() throws Exception {
// need to create the index using the helper method (not create it at first insertion)
// for the indexSettings() to be run
createIndex(getEsIndex());
elasticsearchIOTestCommon.setPipeline(pipeline);
- elasticsearchIOTestCommon.testReadWithQuery();
+ elasticsearchIOTestCommon.testReadWithQueryString();
+ }
+
+ @Test
+ public void testReadWithQueryValueProvider() throws Exception {
+ // need to create the index using the helper method (not create it at first insertion)
+ // for the indexSettings() to be run
+ createIndex(getEsIndex());
+ elasticsearchIOTestCommon.setPipeline(pipeline);
+ elasticsearchIOTestCommon.testReadWithQueryValueProvider();
}
@Test
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle
index 0b269ff..93b59b5 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle
@@ -17,8 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-io-elasticsearch-tests-6'
-applyJavaNature()
+applyJavaNature(
+ publish: false,
+ archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-6'
+)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -44,15 +46,15 @@
}
dependencies {
- testCompile project(path: ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common", configuration: "testRuntime")
testCompile "org.elasticsearch.test:framework:$elastic_search_version"
testCompile "org.elasticsearch.plugin:transport-netty4-client:$elastic_search_version"
testCompile "com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.5.2"
testCompile "org.elasticsearch:elasticsearch:$elastic_search_version"
testCompile project(path: ":sdks:java:core", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:elasticsearch", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ testCompile project(":sdks:java:io:elasticsearch")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile "org.apache.logging.log4j:log4j-core:$log4j_version"
testCompile "org.apache.logging.log4j:log4j-api:$log4j_version"
testCompile library.java.slf4j_api
@@ -63,5 +65,5 @@
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:$elastic_search_version"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
index b89bac4..9629440 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
@@ -82,8 +82,8 @@
@AfterClass
public static void afterClass() throws Exception {
- ElasticSearchIOTestUtils.deleteIndex(writeConnectionConfiguration, restClient);
- ElasticSearchIOTestUtils.deleteIndex(updateConnectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(writeConnectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.deleteIndex(updateConnectionConfiguration, restClient);
restClient.close();
}
@@ -130,11 +130,11 @@
/**
* This test verifies volume partial updates of Elasticsearch. The test dataset index is cloned
* and then a new field is added to each document using a partial update. The test then asserts
- * the updates where appied.
+ * the updates were applied.
*/
@Test
public void testWritePartialUpdate() throws Exception {
- ElasticSearchIOTestUtils.copyIndex(
+ ElasticsearchIOTestUtils.copyIndex(
restClient,
readConnectionConfiguration.getIndex(),
updateConnectionConfiguration.getIndex());
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
index 6638b7d..84696e5 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
@@ -122,12 +122,21 @@
}
@Test
- public void testReadWithQuery() throws Exception {
+ public void testReadWithQueryString() throws Exception {
// need to create the index using the helper method (not create it at first insertion)
// for the indexSettings() to be run
createIndex(getEsIndex());
elasticsearchIOTestCommon.setPipeline(pipeline);
- elasticsearchIOTestCommon.testReadWithQuery();
+ elasticsearchIOTestCommon.testReadWithQueryString();
+ }
+
+ @Test
+ public void testReadWithQueryValueProvider() throws Exception {
+ // need to create the index using the helper method (not create it at first insertion)
+ // for the indexSettings() to be run
+ createIndex(getEsIndex());
+ elasticsearchIOTestCommon.setPipeline(pipeline);
+ elasticsearchIOTestCommon.testReadWithQueryValueProvider();
}
@Test
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle
index 29374fe..168fc5b 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle
@@ -17,8 +17,10 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-io-elasticsearch-tests-common'
-applyJavaNature()
+applyJavaNature(
+ publish: false,
+ archivesBaseName: 'beam-sdks-java-io-elasticsearch-tests-common'
+)
description = "Apache Beam :: SDKs :: Java :: IO :: Elasticsearch-Tests :: Common"
ext.summary = "Common test classes for ElasticsearchIO"
@@ -34,8 +36,8 @@
testCompile "org.apache.httpcomponents:httpclient:4.5.6"
testCompile project(path: ":sdks:java:core", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:elasticsearch", configuration: "shadow")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ testCompile project(":sdks:java:io:elasticsearch")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.slf4j_api
testCompile "net.java.dev.jna:jna:$jna_version"
testCompile "org.apache.logging.log4j:log4j-api:$log4j_version"
@@ -46,5 +48,5 @@
testCompile library.java.junit
testCompile "org.elasticsearch.client:elasticsearch-rest-client:6.4.0"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
deleted file mode 100644
index eacbbc5..0000000
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
+++ /dev/null
@@ -1,262 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.elasticsearch;
-
-import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.parseResponse;
-
-import com.fasterxml.jackson.databind.JsonNode;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-import org.apache.http.HttpEntity;
-import org.apache.http.entity.ContentType;
-import org.apache.http.nio.entity.NStringEntity;
-import org.elasticsearch.client.Request;
-import org.elasticsearch.client.Response;
-import org.elasticsearch.client.RestClient;
-
-/** Test utilities to use with {@link ElasticsearchIO}. */
-class ElasticSearchIOTestUtils {
- static final String[] FAMOUS_SCIENTISTS = {
- "Einstein",
- "Darwin",
- "Copernicus",
- "Pasteur",
- "Curie",
- "Faraday",
- "Newton",
- "Bohr",
- "Galilei",
- "Maxwell"
- };
- static final int NUM_SCIENTISTS = FAMOUS_SCIENTISTS.length;
-
- /** Enumeration that specifies whether to insert malformed documents. */
- public enum InjectionMode {
- INJECT_SOME_INVALID_DOCS,
- DO_NOT_INJECT_INVALID_DOCS
- }
-
- /** Deletes the given index synchronously. */
- static void deleteIndex(ConnectionConfiguration connectionConfiguration, RestClient restClient)
- throws IOException {
- deleteIndex(restClient, connectionConfiguration.getIndex());
- }
-
- private static void closeIndex(RestClient restClient, String index) throws IOException {
- restClient.performRequest("POST", String.format("/%s/_close", index));
- }
-
- private static void deleteIndex(RestClient restClient, String index) throws IOException {
- try {
- closeIndex(restClient, index);
- restClient.performRequest(
- "DELETE", String.format("/%s", index), Collections.singletonMap("refresh", "wait_for"));
- } catch (IOException e) {
- // it is fine to ignore this expression as deleteIndex occurs in @before,
- // so when the first tests is run, the index does not exist yet
- if (!e.getMessage().contains("index_not_found_exception")) {
- throw e;
- }
- }
- }
-
- /**
- * Synchronously deletes the target if it exists and then (re)creates it as a copy of source
- * synchronously.
- */
- static void copyIndex(RestClient restClient, String source, String target) throws IOException {
- deleteIndex(restClient, target);
- HttpEntity entity =
- new NStringEntity(
- String.format(
- "{\"source\" : { \"index\" : \"%s\" }, \"dest\" : { \"index\" : \"%s\" } }",
- source, target),
- ContentType.APPLICATION_JSON);
- restClient.performRequest(
- "POST", "/_reindex", Collections.singletonMap("refresh", "wait_for"), entity);
- }
-
- /** Inserts the given number of test documents into Elasticsearch. */
- static void insertTestDocuments(
- ConnectionConfiguration connectionConfiguration, long numDocs, RestClient restClient)
- throws IOException {
- List<String> data =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
- StringBuilder bulkRequest = new StringBuilder();
- int i = 0;
- for (String document : data) {
- bulkRequest.append(
- String.format(
- "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n%s%n",
- connectionConfiguration.getIndex(),
- connectionConfiguration.getType(),
- i++,
- document));
- }
- String endPoint =
- String.format(
- "/%s/%s/_bulk", connectionConfiguration.getIndex(), connectionConfiguration.getType());
- HttpEntity requestBody =
- new NStringEntity(bulkRequest.toString(), ContentType.APPLICATION_JSON);
- Response response =
- restClient.performRequest(
- "POST", endPoint, Collections.singletonMap("refresh", "wait_for"), requestBody);
- ElasticsearchIO.checkForErrors(
- response.getEntity(), ElasticsearchIO.getBackendVersion(connectionConfiguration), false);
- }
- /**
- * Forces a refresh of the given index to make recently inserted documents available for search
- * using the index and type named in the connectionConfiguration.
- *
- * @param connectionConfiguration providing the index and type
- * @param restClient To use for issuing queries
- * @return The number of docs in the index
- * @throws IOException On error communicating with Elasticsearch
- */
- static long refreshIndexAndGetCurrentNumDocs(
- ConnectionConfiguration connectionConfiguration, RestClient restClient) throws IOException {
- return refreshIndexAndGetCurrentNumDocs(
- restClient, connectionConfiguration.getIndex(), connectionConfiguration.getType());
- }
-
- /**
- * Forces a refresh of the given index to make recently inserted documents available for search.
- *
- * @param restClient To use for issuing queries
- * @param index The Elasticsearch index
- * @param type The Elasticsearch type
- * @return The number of docs in the index
- * @throws IOException On error communicating with Elasticsearch
- */
- static long refreshIndexAndGetCurrentNumDocs(RestClient restClient, String index, String type)
- throws IOException {
- long result = 0;
- try {
- String endPoint = String.format("/%s/_refresh", index);
- restClient.performRequest("POST", endPoint);
-
- endPoint = String.format("/%s/%s/_search", index, type);
- Response response = restClient.performRequest("GET", endPoint);
- JsonNode searchResult = ElasticsearchIO.parseResponse(response.getEntity());
- result = searchResult.path("hits").path("total").asLong();
- } catch (IOException e) {
- // it is fine to ignore bellow exceptions because in testWriteWithBatchSize* sometimes,
- // we call upgrade before any doc have been written
- // (when there are fewer docs processed than batchSize).
- // In that cases index/type has not been created (created upon first doc insertion)
- if (!e.getMessage().contains("index_not_found_exception")) {
- throw e;
- }
- }
- return result;
- }
-
- /**
- * Generates a list of test documents for insertion.
- *
- * @param numDocs Number of docs to generate
- * @param injectionMode {@link InjectionMode} that specifies whether to insert malformed documents
- * @return the list of json String representing the documents
- */
- static List<String> createDocuments(long numDocs, InjectionMode injectionMode) {
-
- ArrayList<String> data = new ArrayList<>();
- for (int i = 0; i < numDocs; i++) {
- int index = i % FAMOUS_SCIENTISTS.length;
- // insert 2 malformed documents
- if (InjectionMode.INJECT_SOME_INVALID_DOCS.equals(injectionMode) && (i == 6 || i == 7)) {
- data.add(String.format("{\"scientist\";\"%s\", \"id\":%s}", FAMOUS_SCIENTISTS[index], i));
- } else {
- data.add(String.format("{\"scientist\":\"%s\", \"id\":%s}", FAMOUS_SCIENTISTS[index], i));
- }
- }
- return data;
- }
-
- /**
- * Executes a query for the named scientist and returns the count from the result.
- *
- * @param connectionConfiguration Specifies the index and type
- * @param restClient To use to execute the call
- * @param scientistName The scientist to query for
- * @return The cound of documents found
- * @throws IOException On error talking to Elasticsearch
- */
- static int countByScientistName(
- ConnectionConfiguration connectionConfiguration, RestClient restClient, String scientistName)
- throws IOException {
- return countByMatch(connectionConfiguration, restClient, "scientist", scientistName);
- }
-
- /**
- * Executes a match query for given field/value and returns the count of results.
- *
- * @param connectionConfiguration Specifies the index and type
- * @param restClient To use to execute the call
- * @param field The field to query
- * @param value The value to match
- * @return The count of documents in the search result
- * @throws IOException On error communicating with Elasticsearch
- */
- static int countByMatch(
- ConnectionConfiguration connectionConfiguration,
- RestClient restClient,
- String field,
- String value)
- throws IOException {
- String requestBody =
- "{\n"
- + " \"query\" : {\"match\": {\n"
- + " \""
- + field
- + "\": \""
- + value
- + "\"\n"
- + " }}\n"
- + "}\n";
- String endPoint =
- String.format(
- "/%s/%s/_search",
- connectionConfiguration.getIndex(), connectionConfiguration.getType());
- HttpEntity httpEntity = new NStringEntity(requestBody, ContentType.APPLICATION_JSON);
- Response response =
- restClient.performRequest("GET", endPoint, Collections.emptyMap(), httpEntity);
- JsonNode searchResult = parseResponse(response.getEntity());
- return searchResult.path("hits").path("total").asInt();
- }
-
- public static void setIndexMapping(
- ConnectionConfiguration connectionConfiguration, RestClient restClient) throws IOException {
- String endpoint = String.format("/%s", connectionConfiguration.getIndex());
- String requestString =
- String.format(
- "{\"mappings\":{\"%s\":{\"properties\":{\"age\":{\"type\":\"long\"},"
- + " \"scientist\":{\"type\":\"%s\"}, \"id\":{\"type\":\"long\"}}}}}",
- connectionConfiguration.getType(),
- getBackendVersion(connectionConfiguration) == 2 ? "string" : "text");
- HttpEntity requestBody = new NStringEntity(requestString, ContentType.APPLICATION_JSON);
- Request request = new Request("PUT", endpoint);
- request.setEntity(requestBody);
- restClient.performRequest(request);
- }
-}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOITCommon.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOITCommon.java
index 6ef38bd..76b88f8 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOITCommon.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOITCommon.java
@@ -99,7 +99,7 @@
ConnectionConfiguration connectionConfiguration =
getConnectionConfiguration(options, IndexMode.READ);
try (RestClient restClient = connectionConfiguration.createClient()) {
- ElasticSearchIOTestUtils.insertTestDocuments(
+ ElasticsearchIOTestUtils.insertTestDocuments(
connectionConfiguration, NUM_DOCS_ITESTS, restClient);
}
}
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
index 112e0e2..386a518 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java
@@ -17,16 +17,16 @@
*/
package org.apache.beam.sdk.io.elasticsearch;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticSearchIOTestUtils.FAMOUS_SCIENTISTS;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticSearchIOTestUtils.NUM_SCIENTISTS;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticSearchIOTestUtils.countByMatch;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticSearchIOTestUtils.countByScientistName;
-import static org.apache.beam.sdk.io.elasticsearch.ElasticSearchIOTestUtils.refreshIndexAndGetCurrentNumDocs;
import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.BoundedElasticsearchSource;
import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration;
import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Read;
import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DEFAULT_RETRY_PREDICATE;
import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.Write;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.FAMOUS_SCIENTISTS;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.NUM_SCIENTISTS;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.countByMatch;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.countByScientistName;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTestUtils.refreshIndexAndGetCurrentNumDocs;
import static org.apache.beam.sdk.testing.SourceTestUtils.readFromSource;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.lessThan;
@@ -45,11 +45,13 @@
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
+import java.util.function.BiFunction;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.DefaultRetryPredicate;
import org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.RetryConfiguration.RetryPredicate;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.SourceTestUtils;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -129,7 +131,7 @@
void testSplit(final int desiredBundleSizeBytes) throws Exception {
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
}
PipelineOptions options = PipelineOptionsFactory.create();
Read read = ElasticsearchIO.read().withConnectionConfiguration(connectionConfiguration);
@@ -166,7 +168,7 @@
void testSizes() throws Exception {
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
}
PipelineOptions options = PipelineOptionsFactory.create();
Read read = ElasticsearchIO.read().withConnectionConfiguration(connectionConfiguration);
@@ -181,7 +183,7 @@
void testRead() throws Exception {
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
}
PCollection<String> output =
@@ -196,9 +198,19 @@
pipeline.run();
}
- void testReadWithQuery() throws Exception {
+ void testReadWithQueryString() throws Exception {
+ testReadWithQueryInternal(Read::withQuery);
+ }
+
+ void testReadWithQueryValueProvider() throws Exception {
+ testReadWithQueryInternal(
+ (read, query) -> read.withQuery(ValueProvider.StaticValueProvider.of(query)));
+ }
+
+ private void testReadWithQueryInternal(BiFunction<Read, String, Read> queryConfigurer)
+ throws IOException {
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
}
String query =
@@ -212,11 +224,12 @@
+ " }\n"
+ "}";
- PCollection<String> output =
- pipeline.apply(
- ElasticsearchIO.read()
- .withConnectionConfiguration(connectionConfiguration)
- .withQuery(query));
+ Read read = ElasticsearchIO.read().withConnectionConfiguration(connectionConfiguration);
+
+ read = queryConfigurer.apply(read, query);
+
+ PCollection<String> output = pipeline.apply(read);
+
PAssert.thatSingleton(output.apply("Count", Count.globally()))
.isEqualTo(numDocs / NUM_SCIENTISTS);
pipeline.run();
@@ -225,7 +238,7 @@
/** Test reading metadata by reading back the id of a document after writing it. */
void testReadWithMetadata() throws Exception {
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, 1, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, 1, restClient);
}
PCollection<String> output =
@@ -248,8 +261,8 @@
.withConnectionConfiguration(connectionConfiguration)
.withMaxBatchSize(BATCH_SIZE);
List<String> input =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.INJECT_SOME_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.INJECT_SOME_INVALID_DOCS);
expectedException.expect(isA(IOException.class));
expectedException.expectMessage(
new CustomMatcher<String>("RegExp matcher") {
@@ -286,8 +299,8 @@
// so we test the Writer as a DoFn outside of a runner.
try (DoFnTester<String, Void> fnTester = DoFnTester.of(new Write.WriteFn(write))) {
List<String> input =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
long numDocsProcessed = 0;
long numDocsInserted = 0;
for (String document : input) {
@@ -327,8 +340,8 @@
// so we test the Writer as a DoFn outside of a runner.
try (DoFnTester<String, Void> fnTester = DoFnTester.of(new Write.WriteFn(write))) {
List<String> input =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
long numDocsProcessed = 0;
long sizeProcessed = 0;
long numDocsInserted = 0;
@@ -383,8 +396,8 @@
*/
void testWriteWithIdFn() throws Exception {
List<String> data =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
pipeline
.apply(Create.of(data))
.apply(
@@ -413,8 +426,8 @@
long adjustedNumDocs = docsPerScientist * FAMOUS_SCIENTISTS.length;
List<String> data =
- ElasticSearchIOTestUtils.createDocuments(
- adjustedNumDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ adjustedNumDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
pipeline
.apply(Create.of(data))
.apply(
@@ -459,8 +472,8 @@
long adjustedNumDocs = (numDocs & 1) == 0 ? numDocs : numDocs + 1;
List<String> data =
- ElasticSearchIOTestUtils.createDocuments(
- adjustedNumDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ adjustedNumDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
pipeline
.apply(Create.of(data))
.apply(
@@ -485,8 +498,8 @@
*/
void testWriteWithFullAddressing() throws Exception {
List<String> data =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
pipeline
.apply(Create.of(data))
.apply(
@@ -514,7 +527,7 @@
*/
void testWritePartialUpdate() throws Exception {
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
}
// defensive coding to ensure our initial state is as expected
@@ -553,10 +566,10 @@
/** Tests partial updates with errors by adding some invalid info to test set. */
void testWritePartialUpdateWithErrors() throws Exception {
// put a mapping to simulate error of insertion
- ElasticSearchIOTestUtils.setIndexMapping(connectionConfiguration, restClient);
+ ElasticsearchIOTestUtils.setIndexMapping(connectionConfiguration, restClient);
if (!useAsITests) {
- ElasticSearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
+ ElasticsearchIOTestUtils.insertTestDocuments(connectionConfiguration, numDocs, restClient);
}
// try to partial update a document with an incompatible date format for the age to generate
@@ -660,8 +673,8 @@
private void executeWriteTest(ElasticsearchIO.Write write) throws Exception {
List<String> data =
- ElasticSearchIOTestUtils.createDocuments(
- numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
pipeline.apply(Create.of(data)).apply(write);
pipeline.run();
diff --git a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestUtils.java b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestUtils.java
new file mode 100644
index 0000000..ee9e47c
--- /dev/null
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestUtils.java
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.elasticsearch;
+
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.ConnectionConfiguration;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.getBackendVersion;
+import static org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.parseResponse;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.ContentType;
+import org.apache.http.nio.entity.NStringEntity;
+import org.elasticsearch.client.Request;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+
+/** Test utilities to use with {@link ElasticsearchIO}. */
+class ElasticsearchIOTestUtils {
+ static final String[] FAMOUS_SCIENTISTS = {
+ "Einstein",
+ "Darwin",
+ "Copernicus",
+ "Pasteur",
+ "Curie",
+ "Faraday",
+ "Newton",
+ "Bohr",
+ "Galilei",
+ "Maxwell"
+ };
+ static final int NUM_SCIENTISTS = FAMOUS_SCIENTISTS.length;
+
+ /** Enumeration that specifies whether to insert malformed documents. */
+ public enum InjectionMode {
+ INJECT_SOME_INVALID_DOCS,
+ DO_NOT_INJECT_INVALID_DOCS
+ }
+
+ /** Deletes the given index synchronously. */
+ static void deleteIndex(ConnectionConfiguration connectionConfiguration, RestClient restClient)
+ throws IOException {
+ deleteIndex(restClient, connectionConfiguration.getIndex());
+ }
+
+ private static void closeIndex(RestClient restClient, String index) throws IOException {
+ restClient.performRequest("POST", String.format("/%s/_close", index));
+ }
+
+ private static void deleteIndex(RestClient restClient, String index) throws IOException {
+ try {
+ closeIndex(restClient, index);
+ restClient.performRequest(
+ "DELETE", String.format("/%s", index), Collections.singletonMap("refresh", "wait_for"));
+ } catch (IOException e) {
+ // it is fine to ignore this expression as deleteIndex occurs in @before,
+ // so when the first tests is run, the index does not exist yet
+ if (!e.getMessage().contains("index_not_found_exception")) {
+ throw e;
+ }
+ }
+ }
+
+ /**
+ * Synchronously deletes the target if it exists and then (re)creates it as a copy of source
+ * synchronously.
+ */
+ static void copyIndex(RestClient restClient, String source, String target) throws IOException {
+ deleteIndex(restClient, target);
+ HttpEntity entity =
+ new NStringEntity(
+ String.format(
+ "{\"source\" : { \"index\" : \"%s\" }, \"dest\" : { \"index\" : \"%s\" } }",
+ source, target),
+ ContentType.APPLICATION_JSON);
+ restClient.performRequest(
+ "POST", "/_reindex", Collections.singletonMap("refresh", "wait_for"), entity);
+ }
+
+ /** Inserts the given number of test documents into Elasticsearch. */
+ static void insertTestDocuments(
+ ConnectionConfiguration connectionConfiguration, long numDocs, RestClient restClient)
+ throws IOException {
+ List<String> data =
+ ElasticsearchIOTestUtils.createDocuments(
+ numDocs, ElasticsearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
+ StringBuilder bulkRequest = new StringBuilder();
+ int i = 0;
+ for (String document : data) {
+ bulkRequest.append(
+ String.format(
+ "{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\", \"_id\" : \"%s\" } }%n%s%n",
+ connectionConfiguration.getIndex(),
+ connectionConfiguration.getType(),
+ i++,
+ document));
+ }
+ String endPoint =
+ String.format(
+ "/%s/%s/_bulk", connectionConfiguration.getIndex(), connectionConfiguration.getType());
+ HttpEntity requestBody =
+ new NStringEntity(bulkRequest.toString(), ContentType.APPLICATION_JSON);
+ Response response =
+ restClient.performRequest(
+ "POST", endPoint, Collections.singletonMap("refresh", "wait_for"), requestBody);
+ ElasticsearchIO.checkForErrors(
+ response.getEntity(), ElasticsearchIO.getBackendVersion(connectionConfiguration), false);
+ }
+ /**
+ * Forces a refresh of the given index to make recently inserted documents available for search
+ * using the index and type named in the connectionConfiguration.
+ *
+ * @param connectionConfiguration providing the index and type
+ * @param restClient To use for issuing queries
+ * @return The number of docs in the index
+ * @throws IOException On error communicating with Elasticsearch
+ */
+ static long refreshIndexAndGetCurrentNumDocs(
+ ConnectionConfiguration connectionConfiguration, RestClient restClient) throws IOException {
+ return refreshIndexAndGetCurrentNumDocs(
+ restClient, connectionConfiguration.getIndex(), connectionConfiguration.getType());
+ }
+
+ /**
+ * Forces a refresh of the given index to make recently inserted documents available for search.
+ *
+ * @param restClient To use for issuing queries
+ * @param index The Elasticsearch index
+ * @param type The Elasticsearch type
+ * @return The number of docs in the index
+ * @throws IOException On error communicating with Elasticsearch
+ */
+ static long refreshIndexAndGetCurrentNumDocs(RestClient restClient, String index, String type)
+ throws IOException {
+ long result = 0;
+ try {
+ String endPoint = String.format("/%s/_refresh", index);
+ restClient.performRequest("POST", endPoint);
+
+ endPoint = String.format("/%s/%s/_search", index, type);
+ Response response = restClient.performRequest("GET", endPoint);
+ JsonNode searchResult = ElasticsearchIO.parseResponse(response.getEntity());
+ result = searchResult.path("hits").path("total").asLong();
+ } catch (IOException e) {
+ // it is fine to ignore bellow exceptions because in testWriteWithBatchSize* sometimes,
+ // we call upgrade before any doc have been written
+ // (when there are fewer docs processed than batchSize).
+ // In that cases index/type has not been created (created upon first doc insertion)
+ if (!e.getMessage().contains("index_not_found_exception")) {
+ throw e;
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Generates a list of test documents for insertion.
+ *
+ * @param numDocs Number of docs to generate
+ * @param injectionMode {@link InjectionMode} that specifies whether to insert malformed documents
+ * @return the list of json String representing the documents
+ */
+ static List<String> createDocuments(long numDocs, InjectionMode injectionMode) {
+
+ ArrayList<String> data = new ArrayList<>();
+ for (int i = 0; i < numDocs; i++) {
+ int index = i % FAMOUS_SCIENTISTS.length;
+ // insert 2 malformed documents
+ if (InjectionMode.INJECT_SOME_INVALID_DOCS.equals(injectionMode) && (i == 6 || i == 7)) {
+ data.add(String.format("{\"scientist\";\"%s\", \"id\":%s}", FAMOUS_SCIENTISTS[index], i));
+ } else {
+ data.add(String.format("{\"scientist\":\"%s\", \"id\":%s}", FAMOUS_SCIENTISTS[index], i));
+ }
+ }
+ return data;
+ }
+
+ /**
+ * Executes a query for the named scientist and returns the count from the result.
+ *
+ * @param connectionConfiguration Specifies the index and type
+ * @param restClient To use to execute the call
+ * @param scientistName The scientist to query for
+ * @return The cound of documents found
+ * @throws IOException On error talking to Elasticsearch
+ */
+ static int countByScientistName(
+ ConnectionConfiguration connectionConfiguration, RestClient restClient, String scientistName)
+ throws IOException {
+ return countByMatch(connectionConfiguration, restClient, "scientist", scientistName);
+ }
+
+ /**
+ * Executes a match query for given field/value and returns the count of results.
+ *
+ * @param connectionConfiguration Specifies the index and type
+ * @param restClient To use to execute the call
+ * @param field The field to query
+ * @param value The value to match
+ * @return The count of documents in the search result
+ * @throws IOException On error communicating with Elasticsearch
+ */
+ static int countByMatch(
+ ConnectionConfiguration connectionConfiguration,
+ RestClient restClient,
+ String field,
+ String value)
+ throws IOException {
+ String requestBody =
+ "{\n"
+ + " \"query\" : {\"match\": {\n"
+ + " \""
+ + field
+ + "\": \""
+ + value
+ + "\"\n"
+ + " }}\n"
+ + "}\n";
+ String endPoint =
+ String.format(
+ "/%s/%s/_search",
+ connectionConfiguration.getIndex(), connectionConfiguration.getType());
+ HttpEntity httpEntity = new NStringEntity(requestBody, ContentType.APPLICATION_JSON);
+ Response response =
+ restClient.performRequest("GET", endPoint, Collections.emptyMap(), httpEntity);
+ JsonNode searchResult = parseResponse(response.getEntity());
+ return searchResult.path("hits").path("total").asInt();
+ }
+
+ public static void setIndexMapping(
+ ConnectionConfiguration connectionConfiguration, RestClient restClient) throws IOException {
+ String endpoint = String.format("/%s", connectionConfiguration.getIndex());
+ String requestString =
+ String.format(
+ "{\"mappings\":{\"%s\":{\"properties\":{\"age\":{\"type\":\"long\"},"
+ + " \"scientist\":{\"type\":\"%s\"}, \"id\":{\"type\":\"long\"}}}}}",
+ connectionConfiguration.getType(),
+ getBackendVersion(connectionConfiguration) == 2 ? "string" : "text");
+ HttpEntity requestBody = new NStringEntity(requestString, ContentType.APPLICATION_JSON);
+ Request request = new Request("PUT", endpoint);
+ request.setEntity(requestBody);
+ restClient.performRequest(request);
+ }
+}
diff --git a/sdks/java/io/elasticsearch/build.gradle b/sdks/java/io/elasticsearch/build.gradle
index 196222c..6eca559 100644
--- a/sdks/java/io/elasticsearch/build.gradle
+++ b/sdks/java/io/elasticsearch/build.gradle
@@ -17,20 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.elasticsearch')
description = "Apache Beam :: SDKs :: Java :: IO :: Elasticsearch"
ext.summary = "IO to read and write on Elasticsearch"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.jackson_databind
- shadow library.java.jackson_annotations
- shadow "org.elasticsearch.client:elasticsearch-rest-client:6.4.0"
- shadow "org.apache.httpcomponents:httpasyncclient:4.1.4"
- shadow "org.apache.httpcomponents:httpcore-nio:4.4.10"
- shadow "org.apache.httpcomponents:httpcore:4.4.10"
- shadow "org.apache.httpcomponents:httpclient:4.5.6"
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.jackson_databind
+ compile library.java.jackson_annotations
+ compile "org.elasticsearch.client:elasticsearch-rest-client:6.4.0"
+ compile "org.apache.httpcomponents:httpasyncclient:4.1.4"
+ compile "org.apache.httpcomponents:httpcore-nio:4.4.10"
+ compile "org.apache.httpcomponents:httpcore:4.4.10"
+ compile "org.apache.httpcomponents:httpclient:4.5.6"
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
}
diff --git a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
index ec688fb..b038e51 100644
--- a/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
+++ b/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.elasticsearch;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
@@ -51,6 +51,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
@@ -63,7 +64,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
@@ -454,7 +455,7 @@
abstract ConnectionConfiguration getConnectionConfiguration();
@Nullable
- abstract String getQuery();
+ abstract ValueProvider<String> getQuery();
abstract boolean isWithMetadata();
@@ -468,7 +469,7 @@
abstract static class Builder {
abstract Builder setConnectionConfiguration(ConnectionConfiguration connectionConfiguration);
- abstract Builder setQuery(String query);
+ abstract Builder setQuery(ValueProvider<String> query);
abstract Builder setWithMetadata(boolean withMetadata);
@@ -502,6 +503,20 @@
public Read withQuery(String query) {
checkArgument(query != null, "query can not be null");
checkArgument(!query.isEmpty(), "query can not be empty");
+ return withQuery(ValueProvider.StaticValueProvider.of(query));
+ }
+
+ /**
+ * Provide a {@link ValueProvider} that provides the query used while reading from
+ * Elasticsearch. This is useful for cases when the query must be dynamic.
+ *
+ * @param query the query. See <a
+ * href="https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl.html">Query
+ * DSL</a>
+ * @return a {@link PTransform} reading data from Elasticsearch.
+ */
+ public Read withQuery(ValueProvider<String> query) {
+ checkArgument(query != null, "query can not be null");
return builder().setQuery(query).build();
}
@@ -726,7 +741,7 @@
public boolean start() throws IOException {
restClient = source.spec.getConnectionConfiguration().createClient();
- String query = source.spec.getQuery();
+ String query = source.spec.getQuery() != null ? source.spec.getQuery().get() : null;
if (query == null) {
query = "{\"query\": { \"match_all\": {} }}";
}
@@ -1013,7 +1028,7 @@
* docs (like Elasticsearch bulk size advice). See
* https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html Depending on the
* execution engine, size of bundles may vary, this sets the maximum size. Change this if you
- * need to have smaller ElasticSearch bulks.
+ * need to have smaller Elasticsearch bulks.
*
* @param batchSize maximum batch size in number of documents
* @return the {@link Write} with connection batch size set
@@ -1029,7 +1044,7 @@
* (like Elasticsearch bulk size advice). See
* https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html Depending on the
* execution engine, size of bundles may vary, this sets the maximum size. Change this if you
- * need to have smaller ElasticSearch bulks.
+ * need to have smaller Elasticsearch bulks.
*
* @param batchSizeBytes maximum batch size in bytes
* @return the {@link Write} with connection batch size in bytes set
diff --git a/sdks/java/io/file-based-io-tests/build.gradle b/sdks/java/io/file-based-io-tests/build.gradle
index 26c10a5..845a9a8 100644
--- a/sdks/java/io/file-based-io-tests/build.gradle
+++ b/sdks/java/io/file-based-io-tests/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, publish: false)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -25,13 +25,12 @@
ext.summary = "Integration tests for reading/writing using file-based sources/sinks."
dependencies {
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:io:common", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:io:xml", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:io:parquet", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:testing:test-utils", configuration: "shadowTest")
- shadowTest library.java.guava
- shadowTest library.java.junit
- shadowTest library.java.hamcrest_core
- shadowTest library.java.jaxb_api
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:io:xml", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:io:parquet", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
+ testCompile library.java.junit
+ testCompile library.java.hamcrest_core
+ testCompile library.java.jaxb_api
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
index c038a3a..925a7c3 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java
@@ -42,8 +42,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -122,8 +120,6 @@
.apply("Produce Avro records", ParDo.of(new DeterministicallyConstructAvroRecordsFn()))
.setCoder(AvroCoder.of(AVRO_SCHEMA))
.apply("Collect start time", ParDo.of(new TimeMonitor<>(AVRO_NAMESPACE, "writeStart")))
- .apply("Collect byte count", ParDo.of(new ByteMonitor<>(AVRO_NAMESPACE, "byteCount")))
- .apply("Collect item count", ParDo.of(new CountMonitor<>(AVRO_NAMESPACE, "itemCount")))
.apply(
"Write Avro records to files",
AvroIO.writeGenericRecords(AVRO_SCHEMA)
@@ -196,18 +192,6 @@
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
- suppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("byteCount");
- return NamedTestResult.create(uuid, timestamp, "byte_count", totalBytes);
- });
-
- suppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("itemCount");
- return NamedTestResult.create(uuid, timestamp, "element_count", totalBytes);
- });
-
return suppliers;
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
index fc3b863..788292f 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/common/FileBasedIOITHelper.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
/** Contains helper methods for file based IO Integration tests. */
public class FileBasedIOITHelper {
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java
index cc000ae..3ee675d 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOIT.java
@@ -40,8 +40,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -139,11 +137,6 @@
.apply(
"Gather read start time",
ParDo.of(new TimeMonitor<>(PARQUET_NAMESPACE, "readStart")))
- .apply(
- "Collect byte count", ParDo.of(new ByteMonitor<>(PARQUET_NAMESPACE, "byteCount")))
- .apply(
- "Collect element count",
- ParDo.of(new CountMonitor<>(PARQUET_NAMESPACE, "itemCount")))
.apply("Read parquet files", ParquetIO.readFiles(SCHEMA))
.apply(
"Gather read end time", ParDo.of(new TimeMonitor<>(PARQUET_NAMESPACE, "readEnd")))
@@ -204,18 +197,6 @@
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
- metricSuppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("byteCount");
- return NamedTestResult.create(uuid, timestamp, "byte_count", totalBytes);
- });
-
- metricSuppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("itemCount");
- return NamedTestResult.create(uuid, timestamp, "item_count", totalBytes);
- });
-
return metricSuppliers;
}
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
index cdcbc7d..33c13d9 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/text/TextIOIT.java
@@ -39,8 +39,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -121,11 +119,6 @@
.apply(
"Collect write start time",
ParDo.of(new TimeMonitor<>(FILEIOIT_NAMESPACE, "startTime")))
- .apply(
- "Collect byte count", ParDo.of(new ByteMonitor<>(FILEIOIT_NAMESPACE, "byteCount")))
- .apply(
- "Collect element count",
- ParDo.of(new CountMonitor<>(FILEIOIT_NAMESPACE, "itemCount")))
.apply("Write content to files", write)
.getPerDestinationOutputFilenames()
.apply(Values.create())
@@ -197,18 +190,6 @@
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
- metricSuppliers.add(
- (metricsReader -> {
- double totalBytes = metricsReader.getCounterMetric("byteCount");
- return NamedTestResult.create(uuid, timestamp, "byte_count", totalBytes);
- }));
-
- metricSuppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("itemCount");
- return NamedTestResult.create(uuid, timestamp, "item_count", totalBytes);
- });
-
if (gatherGcsPerformanceMetrics) {
metricSuppliers.add(
reader -> {
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java
index 87fac05..fbd1af4 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/tfrecord/TFRecordIOIT.java
@@ -39,8 +39,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -124,13 +122,9 @@
.apply(
"Record time before writing",
ParDo.of(new TimeMonitor<>(TFRECORD_NAMESPACE, "writeTime")))
- .apply("Collect byte count", ParDo.of(new ByteMonitor<>(TFRECORD_NAMESPACE, "byteCount")))
- .apply(
- "Collect element count", ParDo.of(new CountMonitor<>(TFRECORD_NAMESPACE, "itemCount")))
.apply("Write content to files", writeTransform);
- PipelineResult writeResult = writePipeline.run();
- writeResult.waitUntilFinish();
+ writePipeline.run().waitUntilFinish();
String filenamePattern = createFilenamePattern();
PCollection<String> consolidatedHashcode =
@@ -152,31 +146,34 @@
"Delete test files",
ParDo.of(new DeleteFileFn())
.withSideInputs(consolidatedHashcode.apply(View.asSingleton())));
- PipelineResult readResult = readPipeline.run();
- readResult.waitUntilFinish();
- collectAndPublishMetrics(readResult, writeResult);
+ PipelineResult result = readPipeline.run();
+ result.waitUntilFinish();
+ collectAndPublishMetrics(result);
}
- private void collectAndPublishMetrics(PipelineResult readResult, PipelineResult writeResult) {
+ private void collectAndPublishMetrics(PipelineResult result) {
String uuid = UUID.randomUUID().toString();
String timestamp = Timestamp.now().toString();
- Set<Function<MetricsReader, NamedTestResult>> readMetricSuppliers =
- getReadMetricSuppliers(uuid, timestamp);
- new IOITMetrics(readMetricSuppliers, readResult, TFRECORD_NAMESPACE, uuid, timestamp)
- .publish(bigQueryDataset, bigQueryTable);
-
- Set<Function<MetricsReader, NamedTestResult>> writeMetricSuppliers =
- getWriteMetricSuppliers(uuid, timestamp);
- new IOITMetrics(writeMetricSuppliers, writeResult, TFRECORD_NAMESPACE, uuid, timestamp)
+ Set<Function<MetricsReader, NamedTestResult>> metricSuppliers =
+ fillMetricSuppliers(uuid, timestamp);
+ new IOITMetrics(metricSuppliers, result, TFRECORD_NAMESPACE, uuid, timestamp)
.publish(bigQueryDataset, bigQueryTable);
}
- private Set<Function<MetricsReader, NamedTestResult>> getWriteMetricSuppliers(
+ private Set<Function<MetricsReader, NamedTestResult>> fillMetricSuppliers(
String uuid, String timestamp) {
Set<Function<MetricsReader, NamedTestResult>> suppliers = new HashSet<>();
suppliers.add(
reader -> {
+ long writeStart = reader.getStartTimeMetric("writeTime");
+ long writeEnd = reader.getEndTimeMetric("writeTime");
+ double writeTime = (writeEnd - writeStart) / 1e3;
+ return NamedTestResult.create(uuid, timestamp, "write_time", writeTime);
+ });
+
+ suppliers.add(
+ reader -> {
long readStart = reader.getStartTimeMetric("readTime");
long readEnd = reader.getEndTimeMetric("readTime");
double readTime = (readEnd - readStart) / 1e3;
@@ -185,27 +182,10 @@
suppliers.add(
reader -> {
- double totalBytes = reader.getCounterMetric("byteCount");
- return NamedTestResult.create(uuid, timestamp, "byte_count", totalBytes);
- });
-
- suppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("itemCount");
- return NamedTestResult.create(uuid, timestamp, "item_count", totalBytes);
- });
- return suppliers;
- }
-
- private Set<Function<MetricsReader, NamedTestResult>> getReadMetricSuppliers(
- String uuid, String timestamp) {
- Set<Function<MetricsReader, NamedTestResult>> suppliers = new HashSet<>();
- suppliers.add(
- reader -> {
long writeStart = reader.getStartTimeMetric("writeTime");
- long writeEnd = reader.getEndTimeMetric("writeTime");
- double writeTime = (writeEnd - writeStart) / 1e3;
- return NamedTestResult.create(uuid, timestamp, "write_time", writeTime);
+ long readEnd = reader.getEndTimeMetric("readTime");
+ double runTime = (readEnd - writeStart) / 1e3;
+ return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
return suppliers;
diff --git a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java
index 67ce13a..3ce31fa 100644
--- a/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java
+++ b/sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/xml/XmlIOIT.java
@@ -42,8 +42,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -54,7 +52,7 @@
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.View;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
@@ -131,11 +129,6 @@
"Gather write start time",
ParDo.of(new TimeMonitor<>(XMLIOIT_NAMESPACE, "writeStart")))
.apply(
- "Gather byte count", ParDo.of(new ByteMonitor<>(XMLIOIT_NAMESPACE, "byte_count")))
- .apply(
- "Gather element count",
- ParDo.of(new CountMonitor<>(XMLIOIT_NAMESPACE, "item_count")))
- .apply(
"Write xml files",
FileIO.<Bird>write()
.via(XmlIO.sink(Bird.class).withRootElement("birds").withCharset(charset))
@@ -218,19 +211,6 @@
double runTime = (readEnd - writeStart) / 1e3;
return NamedTestResult.create(uuid, timestamp, "run_time", runTime);
});
-
- suppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("byteCount");
- return NamedTestResult.create(uuid, timestamp, "byte_count", totalBytes);
- });
-
- suppliers.add(
- reader -> {
- double totalBytes = reader.getCounterMetric("itemCount");
- return NamedTestResult.create(uuid, timestamp, "item_count", totalBytes);
- });
-
return suppliers;
}
diff --git a/sdks/java/io/google-cloud-platform/build.gradle b/sdks/java/io/google-cloud-platform/build.gradle
index 57ffe21..0a9b8a9 100644
--- a/sdks/java/io/google-cloud-platform/build.gradle
+++ b/sdks/java/io/google-cloud-platform/build.gradle
@@ -20,65 +20,61 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.io.gcp',
enableSpotbugs: false,
- // Override the default shading configuration to exclude everything since
- // Bigtable needs to expose Guava types.
- shadowClosure: {
- dependencies {
- exclude(dependency(".*:.*"))
- }
- })
+)
description = "Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform"
ext.summary = "IO library to read and write Google Cloud Platform systems from Beam."
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:protobuf", configuration: "shadow")
- shadow library.java.guava
- shadow library.java.jackson_databind
- shadow library.java.grpc_core
- shadow library.java.google_api_services_bigquery
- shadow library.java.gax_grpc
- shadow library.java.google_cloud_core_grpc
- shadow library.java.google_api_services_pubsub
- shadow library.java.grpc_google_cloud_pubsub_v1
- shadow library.java.proto_google_cloud_pubsub_v1
- shadow library.java.bigdataoss_util
- shadow library.java.datastore_v1_proto_client
- shadow library.java.datastore_v1_protos
- shadow library.java.grpc_auth
- shadow library.java.grpc_netty
- shadow library.java.netty_handler
- shadow library.java.grpc_stub
- shadow library.java.joda_time
- shadow library.java.google_cloud_bigquery_storage
- shadow library.java.google_cloud_bigquery_storage_proto
- shadow library.java.google_cloud_core
- shadow library.java.google_cloud_spanner
- shadow library.java.bigtable_protos
- shadow library.java.bigtable_client_core
- shadow library.java.google_api_client
- shadow library.java.google_http_client
- shadow library.java.google_http_client_jackson2
- shadow library.java.google_auth_library_credentials
- shadow library.java.google_auth_library_oauth2_http
- shadow library.java.slf4j_api
- shadow library.java.protobuf_java
- shadow library.java.avro
- shadow library.java.proto_google_cloud_spanner_admin_database_v1
- shadow library.java.proto_google_common_protos
- shadow library.java.grpc_all
- shadow library.java.netty_tcnative_boringssl_static
- shadowTest project(path: ":sdks:java:core", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadowTest")
- shadowTest project(path: ":runners:direct-java", configuration: "shadow")
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
- shadowTest library.java.mockito_core
- shadowTest library.java.junit
- shadowTest library.java.powermock
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:extensions:protobuf")
+ compile library.java.avro
+ compile library.java.bigdataoss_util
+ compile library.java.gax_grpc
+ compile library.java.google_api_client
+ compile library.java.google_api_services_bigquery
+ compile library.java.google_api_services_pubsub
+ compile library.java.google_auth_library_credentials
+ compile library.java.google_auth_library_oauth2_http
+ compile library.java.google_cloud_bigquery_storage
+ compile library.java.google_cloud_bigtable_client_core
+ compile library.java.google_cloud_core
+ compile library.java.google_cloud_core_grpc
+ compile library.java.google_cloud_datastore_v1_proto_client
+ compile library.java.google_cloud_spanner
+ compile library.java.google_http_client
+ compile library.java.google_http_client_jackson2
+ compile library.java.grpc_all
+ compile library.java.grpc_auth
+ compile library.java.grpc_core
+ compile library.java.grpc_netty
+ compile library.java.grpc_stub
+ compile library.java.grpc_google_cloud_pubsub_v1
+ compile library.java.guava
+ compile library.java.jackson_databind
+ compile library.java.joda_time
+ compile library.java.netty_handler
+ compile library.java.netty_tcnative_boringssl_static
+ compile library.java.proto_google_cloud_bigquery_storage_v1beta1
+ compile library.java.proto_google_cloud_bigtable_v2
+ compile library.java.proto_google_cloud_datastore_v1
+ compile library.java.proto_google_cloud_pubsub_v1
+ compile library.java.proto_google_cloud_spanner_admin_database_v1
+ compile library.java.proto_google_common_protos
+ compile library.java.protobuf_java
+ compile library.java.slf4j_api
+ testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntime")
+ testCompile project(path: ":runners:direct-java", configuration: "shadow")
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.powermock
+ testCompile library.java.powermock_mockito
testRuntimeOnly library.java.slf4j_jdk14
}
@@ -104,17 +100,18 @@
exclude '**/BigQueryIOStorageReadIT.class'
exclude '**/BigQueryIOStorageReadTableRowIT.class'
exclude '**/BigQueryToTableIT.class'
- exclude '**/*KmsKeyIT.class'
maxParallelForks 4
classpath = sourceSets.test.runtimeClasspath
testClassesDirs = sourceSets.test.output.classesDirs
- useJUnit { }
+ useJUnit {
+ excludeCategories "org.apache.beam.sdk.testing.UsesKms"
+ }
}
task integrationTestKms(type: Test) {
group = "Verification"
def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
- def gcpTempRoot = project.findProperty('gcpTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests'
+ def gcpTempRoot = project.findProperty('gcpTempRootKms') ?: 'gs://temp-storage-for-end-to-end-tests-cmek'
def dataflowKmsKey = project.findProperty('dataflowKmsKey') ?: "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test"
systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
"--runner=DirectRunner",
@@ -126,11 +123,13 @@
// Disable Gradle cache: these ITs interact with live service that should always be considered "out of date"
outputs.upToDateWhen { false }
- include '**/*KmsKeyIT.class'
+ include '**/*IT.class'
maxParallelForks 4
classpath = sourceSets.test.runtimeClasspath
testClassesDirs = sourceSets.test.output.classesDirs
- useJUnit { }
+ useJUnit {
+ includeCategories "org.apache.beam.sdk.testing.UsesKms"
+ }
}
task postCommit {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
index 6f3bc0a..0616c40 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.io.gcp.bigquery;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.resolveTempLocation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.bigquery.model.TableRow;
import java.util.List;
@@ -67,9 +67,9 @@
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -250,9 +250,9 @@
final PCollectionView<String> loadJobIdPrefixView = createLoadJobIdPrefixView(p);
final PCollectionView<String> tempFilePrefixView =
createTempFilePrefixView(p, loadJobIdPrefixView);
- // The user-supplied triggeringDuration is often chosen to to control how many BigQuery load
+ // The user-supplied triggeringDuration is often chosen to control how many BigQuery load
// jobs are generated, to prevent going over BigQuery's daily quota for load jobs. If this
- // is set to a large value, currently we have to buffer all the data unti the trigger fires.
+ // is set to a large value, currently we have to buffer all the data until the trigger fires.
// Instead we ensure that the files are written if a threshold number of records are ready.
// We use only the user-supplied trigger on the actual BigQuery load. This allows us to
// offload the data to the filesystem.
@@ -552,6 +552,17 @@
ShardedKeyCoder.of(NullableCoder.of(destinationCoder)),
ListCoder.of(StringUtf8Coder.of()));
+ // If the final destination table exists already (and we're appending to it), then the temp
+ // tables must exactly match schema, partitioning, etc. Wrap the DynamicDestinations object
+ // with one that makes this happen.
+ @SuppressWarnings("unchecked")
+ DynamicDestinations<?, DestinationT> destinations = dynamicDestinations;
+ if (createDisposition.equals(CreateDisposition.CREATE_IF_NEEDED)
+ || createDisposition.equals(CreateDisposition.CREATE_NEVER)) {
+ destinations =
+ DynamicDestinationsHelpers.matchTableDynamicDestinations(destinations, bigQueryServices);
+ }
+
// If WriteBundlesToFiles produced more than DEFAULT_MAX_FILES_PER_PARTITION files or
// DEFAULT_MAX_BYTES_PER_PARTITION bytes, then
// the import needs to be split into multiple partitions, and those partitions will be
@@ -570,7 +581,7 @@
WriteDisposition.WRITE_EMPTY,
CreateDisposition.CREATE_IF_NEEDED,
sideInputs,
- dynamicDestinations,
+ destinations,
loadJobProjectId,
maxRetryJobs,
ignoreUnknownValues,
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
index 52fd54d..d425a96 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
@@ -21,10 +21,10 @@
import static java.time.temporal.ChronoField.MINUTE_OF_HOUR;
import static java.time.temporal.ChronoField.NANO_OF_SECOND;
import static java.time.temporal.ChronoField.SECOND_OF_MINUTE;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.firstNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verify;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verifyNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.firstNonNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verify;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verifyNotNull;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
@@ -44,10 +44,10 @@
import org.apache.avro.Schema.Field;
import org.apache.avro.Schema.Type;
import org.apache.avro.generic.GenericRecord;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
@@ -65,7 +65,7 @@
* <p>Some BigQuery types are duplicated here since slightly different Avro records are produced
* when exporting data in Avro format and when reading data directly using the read API.
*/
- public static final ImmutableMultimap<String, Type> BIG_QUERY_TO_AVRO_TYPES =
+ static final ImmutableMultimap<String, Type> BIG_QUERY_TO_AVRO_TYPES =
ImmutableMultimap.<String, Type>builder()
.put("STRING", Type.STRING)
.put("GEOGRAPHY", Type.STRING)
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryCoderProviderRegistrar.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryCoderProviderRegistrar.java
index e6be7c2..e646d65 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryCoderProviderRegistrar.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryCoderProviderRegistrar.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.CoderProviderRegistrar;
import org.apache.beam.sdk.coders.CoderProviders;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A {@link CoderProviderRegistrar} for standard types used with {@link BigQueryIO}. */
@AutoService(CoderProviderRegistrar.class)
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
index 82bb5ad..6fd3f95 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.BackOff;
import com.google.api.client.util.BackOffUtils;
@@ -25,12 +25,14 @@
import com.google.api.services.bigquery.model.Dataset;
import com.google.api.services.bigquery.model.Job;
import com.google.api.services.bigquery.model.JobStatus;
+import com.google.api.services.bigquery.model.Table;
import com.google.api.services.bigquery.model.TableReference;
import com.google.api.services.bigquery.model.TableSchema;
import com.google.api.services.bigquery.model.TimePartitioning;
import com.google.cloud.bigquery.storage.v1beta1.TableReferenceProto;
import com.google.cloud.hadoop.util.ApiErrorExtractor;
import java.io.IOException;
+import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@@ -45,9 +47,9 @@
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -469,8 +471,7 @@
}
}
- @VisibleForTesting
- static String toJsonString(Object item) {
+ public static String toJsonString(Object item) {
if (item == null) {
return null;
}
@@ -483,8 +484,7 @@
}
}
- @VisibleForTesting
- static <T> T fromJsonString(String json, Class<T> clazz) {
+ public static <T> T fromJsonString(String json, Class<T> clazz) {
if (json == null) {
return null;
}
@@ -541,6 +541,23 @@
}
}
+ /**
+ * It returns the number of rows for a given table.
+ *
+ * @return The number of rows in the table or null if it cannot get any estimate.
+ */
+ @Nullable
+ public static BigInteger getNumRows(BigQueryOptions options, TableReference tableRef)
+ throws InterruptedException, IOException {
+
+ DatasetService datasetService = new BigQueryServicesImpl().getDatasetService(options);
+ Table table = datasetService.getTable(tableRef);
+ if (table == null) {
+ return null;
+ }
+ return table.getNumRows();
+ }
+
static String getDatasetLocation(
DatasetService datasetService, String projectId, String datasetId) {
Dataset dataset;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index a66c903..06bf8c1 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -21,15 +21,18 @@
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createTempTableReference;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.getExtractJobId;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.resolveTempLocation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.json.JsonFactory;
+import com.google.api.services.bigquery.model.Clustering;
import com.google.api.services.bigquery.model.Job;
import com.google.api.services.bigquery.model.JobConfigurationQuery;
import com.google.api.services.bigquery.model.JobReference;
import com.google.api.services.bigquery.model.JobStatistics;
import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableCell;
+import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableReference;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
@@ -83,6 +86,7 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
+import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.MapElements;
@@ -100,17 +104,18 @@
import org.apache.beam.sdk.values.PCollection.IsBounded;
import org.apache.beam.sdk.values.PCollectionTuple;
import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -140,17 +145,99 @@
* <li>[{@code dataset_id}].[{@code table_id}]
* </ul>
*
+ * <h3>BigQuery Concepts</h3>
+ *
+ * <p>Tables have rows ({@link TableRow}) and each row has cells ({@link TableCell}). A table has a
+ * schema ({@link TableSchema}), which in turn describes the schema of each cell ({@link
+ * TableFieldSchema}). The terms field and cell are used interchangeably.
+ *
+ * <p>{@link TableSchema}: describes the schema (types and order) for values in each row. It has one
+ * attribute, 'fields', which is list of {@link TableFieldSchema} objects.
+ *
+ * <p>{@link TableFieldSchema}: describes the schema (type, name) for one field. It has several
+ * attributes, including 'name' and 'type'. Common values for the type attribute are: 'STRING',
+ * 'INTEGER', 'FLOAT', 'BOOLEAN', 'NUMERIC', 'GEOGRAPHY'. All possible values are described at: <a
+ * href="https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types">
+ * https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types</a>
+ *
+ * <p>{@link TableRow}: Holds all values in a table row. Has one attribute, 'f', which is a list of
+ * {@link TableCell} instances.
+ *
+ * <p>{@link TableCell}: Holds the value for one cell (or field). Has one attribute, 'v', which is
+ * the value of the table cell.
+ *
+ * <p>As of Beam 2.7.0, the NUMERIC data type is supported. This data type supports high-precision
+ * decimal numbers (precision of 38 digits, scale of 9 digits). The GEOGRAPHY data type works with
+ * Well-Known Text (See <a href="https://en.wikipedia.org/wiki/Well-known_text">
+ * https://en.wikipedia.org/wiki/Well-known_text</a>) format for reading and writing to BigQuery.
+ * BigQuery IO requires values of BYTES datatype to be encoded using base64 encoding when writing to
+ * BigQuery. When bytes are read from BigQuery they are returned as base64-encoded strings.
+ *
* <h3>Reading</h3>
*
* <p>Reading from BigQuery is supported by {@link #read(SerializableFunction)}, which parses
* records in <a href="https://cloud.google.com/bigquery/data-formats#avro_format">AVRO format</a>
- * into a custom type using a specified parse function, and by {@link #readTableRows} which parses
- * them into {@link TableRow}, which may be more convenient but has lower performance.
+ * into a custom type (see the table below for type conversion) using a specified parse function,
+ * and by {@link #readTableRows} which parses them into {@link TableRow}, which may be more
+ * convenient but has lower performance.
*
* <p>Both functions support reading either from a table or from the result of a query, via {@link
* TypedRead#from(String)} and {@link TypedRead#fromQuery} respectively. Exactly one of these must
* be specified.
*
+ * <p>If you are reading from an authorized view wih {@link TypedRead#fromQuery}, you need to use
+ * {@link TypedRead#withQueryLocation(String)} to set the location of the BigQuery job. Otherwise,
+ * Beam will ty to determine that location by reading the metadata of the dataset that contains the
+ * underlying tables. With authorized views, that will result in a 403 error and the query will not
+ * be resolved.
+ *
+ * <p><b>Type Conversion Table</b>
+ *
+ * <table border="1" cellspacing="1">
+ * <tr>
+ * <td> <b>BigQuery standard SQL type</b> </td> <td> <b>Avro type</b> </td> <td> <b>Java type</b> </td>
+ * </tr>
+ * <tr>
+ * <td> BOOLEAN </td> <td> boolean </td> <td> Boolean </td>
+ * </tr>
+ * <tr>
+ * <td> INT64 </td> <td> long </td> <td> Long </td>
+ * </tr>
+ * <tr>
+ * <td> FLOAT64 </td> <td> double </td> <td> Double </td>
+ * </tr>
+ * <tr>
+ * <td> BYTES </td> <td> bytes </td> <td> java.nio.ByteBuffer </td>
+ * </tr>
+ * <tr>
+ * <td> STRING </td> <td> string </td> <td> CharSequence </td>
+ * </tr>
+ * <tr>
+ * <td> DATE </td> <td> int </td> <td> Integer </td>
+ * </tr>
+ * <tr>
+ * <td> DATETIME </td> <td> string </td> <td> CharSequence </td>
+ * </tr>
+ * <tr>
+ * <td> TIMESTAMP </td> <td> long </td> <td> Long </td>
+ * </tr>
+ * <tr>
+ * <td> TIME </td> <td> long </td> <td> Long </td>
+ * </tr>
+ * <tr>
+ * <td> NUMERIC </td> <td> bytes </td> <td> java.nio.ByteBuffer </td>
+ * </tr>
+ * <tr>
+ * <td> GEOGRAPHY </td> <td> string </td> <td> CharSequence </td>
+ * </tr>
+ * <tr>
+ * <td> ARRAY </td> <td> array </td> <td> java.util.Collection </td>
+ * </tr>
+ * <tr>
+ * <td> STRUCT </td> <td> record </td> <td> org.apache.avro.generic.GenericRecord </td>
+ * </tr>
+ * </table>
+ *
* <p><b>Example: Reading rows of a table as {@link TableRow}.</b>
*
* <pre>{@code
@@ -186,7 +273,7 @@
* <p>Users can optionally specify a query priority using {@link TypedRead#withQueryPriority(
* TypedRead.QueryPriority)} and a geographic location where the query will be executed using {@link
* TypedRead#withQueryLocation(String)}. Query location must be specified for jobs that are not
- * executed in US or EU. See <a
+ * executed in US or EU, or if you are reading from an authorized view. See <a
* href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query">BigQuery Jobs:
* query</a>.
*
@@ -405,6 +492,14 @@
return read(new TableRowParser()).withCoder(TableRowJsonCoder.of());
}
+ /** Like {@link #readTableRows()} but with {@link Schema} support. */
+ public static TypedRead<TableRow> readTableRowsWithSchema() {
+ return read(new TableRowParser())
+ .withCoder(TableRowJsonCoder.of())
+ .withBeamRowConverters(
+ BigQueryUtils.tableRowToBeamRow(), BigQueryUtils.tableRowFromBeamRow());
+ }
+
/**
* Reads from a BigQuery table or query and returns a {@link PCollection} with one element per
* each row of the table or query result, parsed from the BigQuery AVRO format using the specified
@@ -593,6 +688,12 @@
DIRECT_READ,
}
+ interface ToBeamRowFunction<T>
+ extends SerializableFunction<Schema, SerializableFunction<T, Row>> {}
+
+ interface FromBeamRowFunction<T>
+ extends SerializableFunction<Schema, SerializableFunction<Row, T>> {}
+
abstract Builder<T> toBuilder();
@AutoValue.Builder
@@ -618,9 +719,20 @@
@Experimental(Experimental.Kind.SOURCE_SINK)
abstract Builder<T> setMethod(Method method);
+ /**
+ * @deprecated Use {@link #setSelectedFields(ValueProvider)} and {@link
+ * #setRowRestriction(ValueProvider)} instead.
+ */
+ @Deprecated
@Experimental(Experimental.Kind.SOURCE_SINK)
abstract Builder<T> setReadOptions(TableReadOptions readOptions);
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ abstract Builder<T> setSelectedFields(ValueProvider<List<String>> selectedFields);
+
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ abstract Builder<T> setRowRestriction(ValueProvider<String> rowRestriction);
+
abstract TypedRead<T> build();
abstract Builder<T> setParseFn(SerializableFunction<SchemaAndRecord, T> parseFn);
@@ -628,6 +740,12 @@
abstract Builder<T> setCoder(Coder<T> coder);
abstract Builder<T> setKmsKey(String kmsKey);
+
+ @Experimental(Experimental.Kind.SCHEMAS)
+ abstract Builder<T> setToBeamRowFn(ToBeamRowFunction<T> toRowFn);
+
+ @Experimental(Experimental.Kind.SCHEMAS)
+ abstract Builder<T> setFromBeamRowFn(FromBeamRowFunction<T> fromRowFn);
}
@Nullable
@@ -659,16 +777,34 @@
@Experimental(Experimental.Kind.SOURCE_SINK)
abstract Method getMethod();
+ /** @deprecated Use {@link #getSelectedFields()} and {@link #getRowRestriction()} instead. */
+ @Deprecated
@Experimental(Experimental.Kind.SOURCE_SINK)
@Nullable
abstract TableReadOptions getReadOptions();
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ @Nullable
+ abstract ValueProvider<List<String>> getSelectedFields();
+
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ @Nullable
+ abstract ValueProvider<String> getRowRestriction();
+
@Nullable
abstract Coder<T> getCoder();
@Nullable
abstract String getKmsKey();
+ @Nullable
+ @Experimental(Experimental.Kind.SCHEMAS)
+ abstract ToBeamRowFunction<T> getToBeamRowFn();
+
+ @Nullable
+ @Experimental(Experimental.Kind.SCHEMAS)
+ abstract FromBeamRowFunction<T> getFromBeamRowFn();
+
/**
* An enumeration type for the priority of a query.
*
@@ -709,27 +845,22 @@
}
}
- private BigQuerySourceBase<T> createSource(String jobUuid, Coder<T> coder) {
- BigQuerySourceBase<T> source;
+ private BigQuerySourceDef createSourceDef() {
+ BigQuerySourceDef sourceDef;
if (getQuery() == null) {
- source =
- BigQueryTableSource.create(
- jobUuid, getTableProvider(), getBigQueryServices(), coder, getParseFn());
+ sourceDef = BigQueryTableSourceDef.create(getBigQueryServices(), getTableProvider());
} else {
- source =
- BigQueryQuerySource.create(
- jobUuid,
+ sourceDef =
+ BigQueryQuerySourceDef.create(
+ getBigQueryServices(),
getQuery(),
getFlattenResults(),
getUseLegacySql(),
- getBigQueryServices(),
- coder,
- getParseFn(),
MoreObjects.firstNonNull(getQueryPriority(), QueryPriority.BATCH),
getQueryLocation(),
getKmsKey());
}
- return source;
+ return sourceDef;
}
private BigQueryStorageQuerySource<T> createStorageQuerySource(
@@ -840,6 +971,12 @@
}
checkArgument(getParseFn() != null, "A parseFn is required");
+ // if both toRowFn and fromRowFn values are set, enable Beam schema support
+ boolean beamSchemaEnabled = false;
+ if (getToBeamRowFn() != null && getFromBeamRowFn() != null) {
+ beamSchemaEnabled = true;
+ }
+
Pipeline p = input.getPipeline();
final Coder<T> coder = inferCoder(p.getCoderRegistry());
@@ -852,6 +989,17 @@
"Invalid BigQueryIO.Read: Specifies table read options, "
+ "which only applies when using Method.DIRECT_READ");
+ checkArgument(
+ getSelectedFields() == null,
+ "Invalid BigQueryIO.Read: Specifies selected fields, "
+ + "which only applies when using Method.DIRECT_READ");
+
+ checkArgument(
+ getRowRestriction() == null,
+ "Invalid BigQueryIO.Read: Specifies row restriction, "
+ + "which only applies when using Method.DIRECT_READ");
+
+ final BigQuerySourceDef sourceDef = createSourceDef();
final PCollectionView<String> jobIdTokenView;
PCollection<String> jobIdTokenCollection;
PCollection<T> rows;
@@ -862,7 +1010,10 @@
p.apply("TriggerIdCreation", Create.of(staticJobUuid))
.apply("ViewId", View.asSingleton());
// Apply the traditional Source model.
- rows = p.apply(org.apache.beam.sdk.io.Read.from(createSource(staticJobUuid, coder)));
+ rows =
+ p.apply(
+ org.apache.beam.sdk.io.Read.from(
+ sourceDef.toSource(staticJobUuid, coder, getParseFn())));
} else {
// Create a singleton job ID token at execution time.
jobIdTokenCollection =
@@ -888,7 +1039,8 @@
@ProcessElement
public void processElement(ProcessContext c) throws Exception {
String jobUuid = c.element();
- BigQuerySourceBase<T> source = createSource(jobUuid, coder);
+ BigQuerySourceBase<T> source =
+ sourceDef.toSource(jobUuid, coder, getParseFn());
BigQueryOptions options =
c.getPipelineOptions().as(BigQueryOptions.class);
ExtractResult res = source.extractFiles(options);
@@ -919,7 +1071,8 @@
BigQueryHelpers.fromJsonString(
c.sideInput(schemaView), TableSchema.class);
String jobUuid = c.sideInput(jobIdTokenView);
- BigQuerySourceBase<T> source = createSource(jobUuid, coder);
+ BigQuerySourceBase<T> source =
+ sourceDef.toSource(jobUuid, coder, getParseFn());
List<BoundedSource<T>> sources =
source.createSources(
ImmutableList.of(
@@ -966,7 +1119,18 @@
}
}
};
- return rows.apply(new PassThroughThenCleanup<>(cleanupOperation, jobIdTokenView));
+
+ rows = rows.apply(new PassThroughThenCleanup<>(cleanupOperation, jobIdTokenView));
+
+ if (beamSchemaEnabled) {
+ BigQueryOptions bqOptions = p.getOptions().as(BigQueryOptions.class);
+ Schema beamSchema = sourceDef.getBeamSchema(bqOptions);
+ SerializableFunction<T, Row> toBeamRow = getToBeamRowFn().apply(beamSchema);
+ SerializableFunction<Row, T> fromBeamRow = getFromBeamRowFn().apply(beamSchema);
+
+ rows.setSchema(beamSchema, toBeamRow, fromBeamRow);
+ }
+ return rows;
}
private PCollection<T> expandForDirectRead(PBegin input, Coder<T> outputCoder) {
@@ -979,6 +1143,8 @@
BigQueryStorageTableSource.create(
tableProvider,
getReadOptions(),
+ getSelectedFields(),
+ getRowRestriction(),
getParseFn(),
outputCoder,
getBigQueryServices())));
@@ -989,6 +1155,16 @@
"Invalid BigQueryIO.Read: Specifies table read options, "
+ "which only applies when reading from a table");
+ checkArgument(
+ getSelectedFields() == null,
+ "Invalid BigQueryIO.Read: Specifies selected fields, "
+ + "which only applies when reading from a table");
+
+ checkArgument(
+ getRowRestriction() == null,
+ "Invalid BigQueryIO.Read: Specifies row restriction, "
+ + "which only applies when reading from a table");
+
//
// N.B. All of the code below exists because the BigQuery storage API can't (yet) read from
// all anonymous tables, so we need the job ID to reason about the name of the destination
@@ -1173,6 +1349,16 @@
getJsonTableRef() == null && getQuery() == null, "from() or fromQuery() already called");
}
+ private void ensureReadOptionsNotSet() {
+ checkState(getReadOptions() == null, "withReadOptions() already called");
+ }
+
+ private void ensureReadOptionsFieldsNotSet() {
+ checkState(
+ getSelectedFields() == null && getRowRestriction() == null,
+ "setSelectedFields() or setRowRestriction already called");
+ }
+
/** See {@link Read#getTableProvider()}. */
@Nullable
public ValueProvider<TableReference> getTableProvider() {
@@ -1201,6 +1387,17 @@
return toBuilder().setKmsKey(kmsKey).build();
}
+ /**
+ * Sets the functions to convert elements to/from {@link Row} objects.
+ *
+ * <p>Setting these conversion functions is necessary to enable {@link Schema} support.
+ */
+ @Experimental(Experimental.Kind.SCHEMAS)
+ public TypedRead<T> withBeamRowConverters(
+ ToBeamRowFunction<T> toRowFn, FromBeamRowFunction<T> fromRowFn) {
+ return toBuilder().setToBeamRowFn(toRowFn).setFromBeamRowFn(fromRowFn).build();
+ }
+
/** See {@link Read#from(String)}. */
public TypedRead<T> from(String tableSpec) {
return from(StaticValueProvider.of(tableSpec));
@@ -1257,8 +1454,9 @@
* BigQuery geographic location where the query <a
* href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs">job</a> will be
* executed. If not specified, Beam tries to determine the location by examining the tables
- * referenced by the query. Location must be specified for queries not executed in US or EU. See
- * <a href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query">BigQuery Jobs:
+ * referenced by the query. Location must be specified for queries not executed in US or EU, or
+ * when you are reading from an authorized view. See <a
+ * href="https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query">BigQuery Jobs:
* query</a>.
*/
public TypedRead<T> withQueryLocation(String location) {
@@ -1271,12 +1469,54 @@
return toBuilder().setMethod(method).build();
}
- /** Read options, including a list of selected columns and push-down SQL filter text. */
+ /**
+ * @deprecated Use {@link #withSelectedFields(List)} and {@link #withRowRestriction(String)}
+ * instead.
+ */
+ @Deprecated
@Experimental(Experimental.Kind.SOURCE_SINK)
public TypedRead<T> withReadOptions(TableReadOptions readOptions) {
+ ensureReadOptionsFieldsNotSet();
return toBuilder().setReadOptions(readOptions).build();
}
+ /** See {@link #withSelectedFields(ValueProvider)}. */
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ public TypedRead<T> withSelectedFields(List<String> selectedFields) {
+ return withSelectedFields(StaticValueProvider.of(selectedFields));
+ }
+
+ /**
+ * Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in
+ * the order specified. If no value is specified, then all fields are returned.
+ *
+ * <p>Requires {@link Method#DIRECT_READ}. Not compatible with {@link #fromQuery(String)}.
+ */
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ public TypedRead<T> withSelectedFields(ValueProvider<List<String>> selectedFields) {
+ ensureReadOptionsNotSet();
+ return toBuilder().setSelectedFields(selectedFields).build();
+ }
+
+ /** See {@link #withRowRestriction(ValueProvider)}. */
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ public TypedRead<T> withRowRestriction(String rowRestriction) {
+ return withRowRestriction(StaticValueProvider.of(rowRestriction));
+ }
+
+ /**
+ * Read only rows which match the specified filter, which must be a SQL expression compatible
+ * with <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/">Google standard
+ * SQL</a>. If no value is specified, then all rows are returned.
+ *
+ * <p>Requires {@link Method#DIRECT_READ}. Not compatible with {@link #fromQuery(String)}.
+ */
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ public TypedRead<T> withRowRestriction(ValueProvider<String> rowRestriction) {
+ ensureReadOptionsNotSet();
+ return toBuilder().setRowRestriction(rowRestriction).build();
+ }
+
@Experimental(Experimental.Kind.SOURCE_SINK)
public TypedRead<T> withTemplateCompatibility() {
return toBuilder().setWithTemplateCompatibility(true).build();
@@ -1441,6 +1681,9 @@
@Nullable
abstract ValueProvider<String> getJsonTimePartitioning();
+ @Nullable
+ abstract Clustering getClustering();
+
abstract CreateDisposition getCreateDisposition();
abstract WriteDisposition getWriteDisposition();
@@ -1510,6 +1753,8 @@
abstract Builder<T> setJsonTimePartitioning(ValueProvider<String> jsonTimePartitioning);
+ abstract Builder<T> setClustering(Clustering clustering);
+
abstract Builder<T> setCreateDisposition(CreateDisposition createDisposition);
abstract Builder<T> setWriteDisposition(WriteDisposition writeDisposition);
@@ -1644,6 +1889,10 @@
/**
* Writes to table specified by the specified table function. The table is a function of {@link
* ValueInSingleWindow}, so can be determined by the value or by the window.
+ *
+ * <p>If the function produces destinations configured with clustering fields, ensure that
+ * {@link #withClustering()} is also set so that the clustering configurations get properly
+ * encoded and decoded.
*/
public Write<T> to(
SerializableFunction<ValueInSingleWindow<T>, TableDestination> tableFunction) {
@@ -1651,7 +1900,13 @@
return toBuilder().setTableFunction(tableFunction).build();
}
- /** Writes to the table and schema specified by the {@link DynamicDestinations} object. */
+ /**
+ * Writes to the table and schema specified by the {@link DynamicDestinations} object.
+ *
+ * <p>If any of the returned destinations are configured with clustering fields, ensure that the
+ * passed {@link DynamicDestinations} object returns {@link TableDestinationCoderV3} when {@link
+ * DynamicDestinations#getDestinationCoder()} is called.
+ */
public Write<T> to(DynamicDestinations<T, ?> dynamicDestinations) {
checkArgument(dynamicDestinations != null, "dynamicDestinations can not be null");
return toBuilder().setDynamicDestinations(dynamicDestinations).build();
@@ -1710,7 +1965,7 @@
* Allows newly created tables to include a {@link TimePartitioning} class. Can only be used
* when writing to a single table. If {@link #to(SerializableFunction)} or {@link
* #to(DynamicDestinations)} is used to write dynamic tables, time partitioning can be directly
- * in the returned {@link TableDestination}.
+ * set in the returned {@link TableDestination}.
*/
public Write<T> withTimePartitioning(TimePartitioning partitioning) {
checkArgument(partitioning != null, "partitioning can not be null");
@@ -1734,6 +1989,34 @@
return toBuilder().setJsonTimePartitioning(partitioning).build();
}
+ /**
+ * Specifies the clustering fields to use when writing to a single output table. Can only be
+ * used when {@link#withTimePartitioning(TimePartitioning)} is set. If {@link
+ * #to(SerializableFunction)} or {@link #to(DynamicDestinations)} is used to write to dynamic
+ * tables, the fields here will be ignored; call {@link #withClustering()} instead.
+ */
+ public Write<T> withClustering(Clustering clustering) {
+ checkArgument(clustering != null, "clustering can not be null");
+ return toBuilder().setClustering(clustering).build();
+ }
+
+ /**
+ * Allows writing to clustered tables when {@link #to(SerializableFunction)} or {@link
+ * #to(DynamicDestinations)} is used. The returned {@link TableDestination} objects should
+ * specify the time partitioning and clustering fields per table. If writing to a single table,
+ * use {@link #withClustering(Clustering)} instead to pass a {@link Clustering} instance that
+ * specifies the static clustering fields to use.
+ *
+ * <p>Setting this option enables use of {@link TableDestinationCoderV3} which encodes
+ * clustering information. The updated coder is compatible with non-clustered tables, so can be
+ * freely set for newly deployed pipelines, but note that pipelines using an older coder must be
+ * drained before setting this option, since {@link TableDestinationCoderV3} will not be able to
+ * read state written with a previous version.
+ */
+ public Write<T> withClustering() {
+ return toBuilder().setClustering(new Clustering()).build();
+ }
+
/** Specifies whether the table should be created if it does not exist. */
public Write<T> withCreateDisposition(CreateDisposition createDisposition) {
checkArgument(createDisposition != null, "createDisposition can not be null");
@@ -1887,8 +2170,17 @@
return toBuilder().setBigQueryServices(testServices).build();
}
- @VisibleForTesting
- Write<T> withMaxFilesPerBundle(int maxFilesPerBundle) {
+ /**
+ * Control how many files will be written concurrently by a single worker when using BigQuery
+ * load jobs before spilling to a shuffle. When data comes into this transform, it is written to
+ * one file per destination per worker. When there are more files than maxFilesPerBundle
+ * (DEFAULT: 20), the data is shuffled (i.e. Grouped By Destination), and written to files
+ * one-by-one-per-worker. This flag sets the maximum number of files that a single worker can
+ * write concurrently before shuffling the data. This flag should be used with caution. Setting
+ * a high number can increase the memory pressure on workers, and setting a low number can make
+ * a pipeline slower (due to the need to shuffle data).
+ */
+ public Write<T> withMaxFilesPerBundle(int maxFilesPerBundle) {
checkArgument(
maxFilesPerBundle > 0, "maxFilesPerBundle must be > 0, but was: %s", maxFilesPerBundle);
return toBuilder().setMaxFilesPerBundle(maxFilesPerBundle).build();
@@ -2005,6 +2297,11 @@
"The supplied getTableFunction object can directly set TimePartitioning."
+ " There is no need to call BigQueryIO.Write.withTimePartitioning.");
}
+ if (getClustering() != null && getClustering().getFields() != null) {
+ checkArgument(
+ getJsonTimePartitioning() != null,
+ "Clustering fields can only be set when TimePartitioning is set.");
+ }
DynamicDestinations<T, ?> dynamicDestinations = getDynamicDestinations();
if (dynamicDestinations == null) {
@@ -2013,7 +2310,8 @@
DynamicDestinationsHelpers.ConstantTableDestinations.fromJsonTableRef(
getJsonTableRef(), getTableDescription());
} else if (getTableFunction() != null) {
- dynamicDestinations = new TableFunctionDestinations<>(getTableFunction());
+ dynamicDestinations =
+ new TableFunctionDestinations<>(getTableFunction(), getClustering() != null);
}
// Wrap with a DynamicDestinations class that will provide a schema. There might be no
@@ -2034,7 +2332,8 @@
dynamicDestinations =
new ConstantTimePartitioningDestinations<>(
(DynamicDestinations<T, TableDestination>) dynamicDestinations,
- getJsonTimePartitioning());
+ getJsonTimePartitioning(),
+ StaticValueProvider.of(BigQueryHelpers.toJsonString(getClustering())));
}
}
return expandTyped(input, dynamicDestinations);
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
index 375cc4f..e8c1e29 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
@@ -17,23 +17,13 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createJobIdToken;
-import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createTempTableReference;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-
-import com.google.api.services.bigquery.model.JobStatistics;
import com.google.api.services.bigquery.model.TableReference;
import java.io.IOException;
-import java.io.ObjectInputStream;
-import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.QueryPriority;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.DatasetService;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -45,115 +35,44 @@
static <T> BigQueryQuerySource<T> create(
String stepUuid,
- ValueProvider<String> query,
- Boolean flattenResults,
- Boolean useLegacySql,
+ BigQueryQuerySourceDef queryDef,
BigQueryServices bqServices,
Coder<T> coder,
- SerializableFunction<SchemaAndRecord, T> parseFn,
- QueryPriority priority,
- String location,
- String kmsKey) {
- return new BigQueryQuerySource<>(
- stepUuid,
- query,
- flattenResults,
- useLegacySql,
- bqServices,
- coder,
- parseFn,
- priority,
- location,
- kmsKey);
+ SerializableFunction<SchemaAndRecord, T> parseFn) {
+ return new BigQueryQuerySource<>(stepUuid, queryDef, bqServices, coder, parseFn);
}
- private final ValueProvider<String> query;
- private final Boolean flattenResults;
- private final Boolean useLegacySql;
- private final QueryPriority priority;
- private final String location;
- private final String kmsKey;
-
- private transient AtomicReference<JobStatistics> dryRunJobStats;
+ private final BigQueryQuerySourceDef queryDef;
private BigQueryQuerySource(
String stepUuid,
- ValueProvider<String> query,
- Boolean flattenResults,
- Boolean useLegacySql,
+ BigQueryQuerySourceDef queryDef,
BigQueryServices bqServices,
Coder<T> coder,
- SerializableFunction<SchemaAndRecord, T> parseFn,
- QueryPriority priority,
- String location,
- String kmsKey) {
+ SerializableFunction<SchemaAndRecord, T> parseFn) {
super(stepUuid, bqServices, coder, parseFn);
- this.query = checkNotNull(query, "query");
- this.flattenResults = checkNotNull(flattenResults, "flattenResults");
- this.useLegacySql = checkNotNull(useLegacySql, "useLegacySql");
- this.priority = priority;
- this.location = location;
- this.kmsKey = kmsKey;
- dryRunJobStats = new AtomicReference<>();
- }
-
- /**
- * Since the query helper reference is declared as transient, neither the AtomicReference nor the
- * structure it refers to are persisted across serialization boundaries. The code below is
- * resilient to the QueryHelper object disappearing in between method calls, but the reference
- * object must be recreated at deserialization time.
- */
- private void readObject(ObjectInputStream in) throws ClassNotFoundException, IOException {
- in.defaultReadObject();
- dryRunJobStats = new AtomicReference<>();
+ this.queryDef = queryDef;
}
@Override
public long getEstimatedSizeBytes(PipelineOptions options) throws Exception {
- return BigQueryQueryHelper.dryRunQueryIfNeeded(
- bqServices,
- options.as(BigQueryOptions.class),
- dryRunJobStats,
- query.get(),
- flattenResults,
- useLegacySql,
- location)
- .getQuery()
- .getTotalBytesProcessed();
+ return queryDef.getEstimatedSizeBytes(options.as(BigQueryOptions.class));
}
@Override
protected TableReference getTableToExtract(BigQueryOptions bqOptions)
throws IOException, InterruptedException {
- return BigQueryQueryHelper.executeQuery(
- bqServices,
- bqOptions,
- dryRunJobStats,
- stepUuid,
- query.get(),
- flattenResults,
- useLegacySql,
- priority,
- location,
- kmsKey);
+ return queryDef.getTableReference(bqOptions, stepUuid);
}
@Override
protected void cleanupTempResource(BigQueryOptions bqOptions) throws Exception {
- TableReference tableToRemove =
- createTempTableReference(
- bqOptions.getProject(), createJobIdToken(bqOptions.getJobName(), stepUuid));
-
- DatasetService tableService = bqServices.getDatasetService(bqOptions);
- LOG.info("Deleting temporary table with query results {}", tableToRemove);
- tableService.deleteTable(tableToRemove);
- LOG.info("Deleting temporary dataset with query results {}", tableToRemove.getDatasetId());
- tableService.deleteDataset(tableToRemove.getProjectId(), tableToRemove.getDatasetId());
+ queryDef.cleanupTempResource(bqOptions, stepUuid);
}
@Override
public void populateDisplayData(DisplayData.Builder builder) {
super.populateDisplayData(builder);
- builder.add(DisplayData.item("query", query));
+ builder.add(DisplayData.item("query", queryDef.getQuery()));
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java
new file mode 100644
index 0000000..6efc502
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createJobIdToken;
+import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createTempTableReference;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.api.services.bigquery.model.JobStatistics;
+import com.google.api.services.bigquery.model.TableReference;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BigQueryQuerySourceDef implements BigQuerySourceDef {
+ private static final Logger LOG = LoggerFactory.getLogger(BigQueryQuerySourceDef.class);
+
+ private final BigQueryServices bqServices;
+ private final ValueProvider<String> query;
+ private final Boolean flattenResults;
+ private final Boolean useLegacySql;
+ private final BigQueryIO.TypedRead.QueryPriority priority;
+ private final String location;
+ private final String kmsKey;
+
+ private transient AtomicReference<JobStatistics> dryRunJobStats;
+
+ static BigQueryQuerySourceDef create(
+ BigQueryServices bqServices,
+ ValueProvider<String> query,
+ Boolean flattenResults,
+ Boolean useLegacySql,
+ BigQueryIO.TypedRead.QueryPriority priority,
+ String location,
+ String kmsKey) {
+ return new BigQueryQuerySourceDef(
+ bqServices, query, flattenResults, useLegacySql, priority, location, kmsKey);
+ }
+
+ private BigQueryQuerySourceDef(
+ BigQueryServices bqServices,
+ ValueProvider<String> query,
+ Boolean flattenResults,
+ Boolean useLegacySql,
+ BigQueryIO.TypedRead.QueryPriority priority,
+ String location,
+ String kmsKey) {
+ this.query = checkNotNull(query, "query");
+ this.flattenResults = checkNotNull(flattenResults, "flattenResults");
+ this.useLegacySql = checkNotNull(useLegacySql, "useLegacySql");
+ this.bqServices = bqServices;
+ this.priority = priority;
+ this.location = location;
+ this.kmsKey = kmsKey;
+ dryRunJobStats = new AtomicReference<>();
+ }
+
+ /**
+ * Since the query helper reference is declared as transient, neither the AtomicReference nor the
+ * structure it refers to are persisted across serialization boundaries. The code below is
+ * resilient to the QueryHelper object disappearing in between method calls, but the reference
+ * object must be recreated at deserialization time.
+ */
+ private void readObject(ObjectInputStream in) throws ClassNotFoundException, IOException {
+ in.defaultReadObject();
+ dryRunJobStats = new AtomicReference<>();
+ }
+
+ long getEstimatedSizeBytes(BigQueryOptions bqOptions) throws Exception {
+ return BigQueryQueryHelper.dryRunQueryIfNeeded(
+ bqServices,
+ bqOptions,
+ dryRunJobStats,
+ query.get(),
+ flattenResults,
+ useLegacySql,
+ location)
+ .getQuery()
+ .getTotalBytesProcessed();
+ }
+
+ TableReference getTableReference(BigQueryOptions bqOptions, String stepUuid)
+ throws IOException, InterruptedException {
+ return BigQueryQueryHelper.executeQuery(
+ bqServices,
+ bqOptions,
+ dryRunJobStats,
+ stepUuid,
+ query.get(),
+ flattenResults,
+ useLegacySql,
+ priority,
+ location,
+ kmsKey);
+ }
+
+ void cleanupTempResource(BigQueryOptions bqOptions, String stepUuid) throws Exception {
+ TableReference tableToRemove =
+ createTempTableReference(
+ bqOptions.getProject(), createJobIdToken(bqOptions.getJobName(), stepUuid));
+
+ BigQueryServices.DatasetService tableService = bqServices.getDatasetService(bqOptions);
+ LOG.info("Deleting temporary table with query results {}", tableToRemove);
+ tableService.deleteTable(tableToRemove);
+ LOG.info("Deleting temporary dataset with query results {}", tableToRemove.getDatasetId());
+ tableService.deleteDataset(tableToRemove.getProjectId(), tableToRemove.getDatasetId());
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public <T> BigQuerySourceBase<T> toSource(
+ String stepUuid, Coder<T> coder, SerializableFunction<SchemaAndRecord, T> parseFn) {
+ return BigQueryQuerySource.create(stepUuid, this, bqServices, coder, parseFn);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public Schema getBeamSchema(BigQueryOptions bqOptions) {
+ try {
+ JobStatistics stats =
+ BigQueryQueryHelper.dryRunQueryIfNeeded(
+ bqServices,
+ bqOptions,
+ dryRunJobStats,
+ query.get(),
+ flattenResults,
+ useLegacySql,
+ location);
+ TableSchema tableSchema = stats.getQuery().getSchema();
+ return BigQueryUtils.fromTableSchema(tableSchema);
+ } catch (IOException | InterruptedException | NullPointerException e) {
+ throw new BigQuerySchemaRetrievalException(
+ "Exception while trying to retrieve schema of query", e);
+ }
+ }
+
+ ValueProvider<String> getQuery() {
+ return query;
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaRetrievalException.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaRetrievalException.java
new file mode 100644
index 0000000..2736e56
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaRetrievalException.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+/** Exception to signal that BigQuery schema retrieval failed. */
+public class BigQuerySchemaRetrievalException extends RuntimeException {
+ BigQuerySchemaRetrievalException(String message, Throwable cause) {
+ super(message, cause);
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
index e883137..ecd4a85 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
@@ -32,6 +32,8 @@
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsRequest;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsResponse;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamRequest;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamResponse;
import java.io.IOException;
import java.io.Serializable;
import java.util.List;
@@ -53,7 +55,7 @@
StorageClient getStorageClient(BigQueryOptions bqOptions) throws IOException;
/** An interface for the Cloud BigQuery load service. */
- interface JobService {
+ public interface JobService {
/** Start a BigQuery load job. */
void startLoadJob(JobReference jobRef, JobConfigurationLoad loadConfig)
throws InterruptedException, IOException;
@@ -89,7 +91,7 @@
}
/** An interface to get, create and delete Cloud BigQuery datasets and tables. */
- interface DatasetService {
+ public interface DatasetService {
/**
* Gets the specified {@link Table} resource by table ID.
*
@@ -164,6 +166,19 @@
throws IOException, InterruptedException;
}
+ /**
+ * Container for reading data from streaming endpoints.
+ *
+ * <p>An implementation does not need to be thread-safe.
+ */
+ interface BigQueryServerStream<T> extends Iterable<T>, Serializable {
+ /**
+ * Cancels the stream, releasing any client- and server-side resources. This method may be
+ * called multiple times and from any thread.
+ */
+ void cancel();
+ }
+
/** An interface representing a client object for making calls to the BigQuery Storage API. */
@Experimental(Experimental.Kind.SOURCE_SINK)
interface StorageClient extends AutoCloseable {
@@ -171,7 +186,9 @@
ReadSession createReadSession(CreateReadSessionRequest request);
/** Read rows in the context of a specific read stream. */
- Iterable<ReadRowsResponse> readRows(ReadRowsRequest request);
+ BigQueryServerStream<ReadRowsResponse> readRows(ReadRowsRequest request);
+
+ SplitReadStreamResponse splitReadStream(SplitReadStreamRequest request);
/**
* Close the client object.
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
index d01add4..147a862 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.googleapis.services.AbstractGoogleClientRequest;
@@ -29,6 +29,7 @@
import com.google.api.gax.core.FixedCredentialsProvider;
import com.google.api.gax.rpc.FixedHeaderProvider;
import com.google.api.gax.rpc.HeaderProvider;
+import com.google.api.gax.rpc.ServerStream;
import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.Bigquery.Tables;
import com.google.api.services.bigquery.model.Dataset;
@@ -56,11 +57,14 @@
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsRequest;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsResponse;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamRequest;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamResponse;
import com.google.cloud.hadoop.util.ApiErrorExtractor;
import com.google.cloud.hadoop.util.ChainingHttpRequestInitializer;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
+import java.util.Iterator;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
@@ -83,8 +87,8 @@
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.ReleaseInfo;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -984,6 +988,25 @@
return builder.build();
}
+ static class BigQueryServerStreamImpl<T> implements BigQueryServerStream<T> {
+
+ private final ServerStream serverStream;
+
+ public BigQueryServerStreamImpl(ServerStream serverStream) {
+ this.serverStream = serverStream;
+ }
+
+ @Override
+ public Iterator<T> iterator() {
+ return serverStream.iterator();
+ }
+
+ @Override
+ public void cancel() {
+ serverStream.cancel();
+ }
+ }
+
@Experimental(Experimental.Kind.SOURCE_SINK)
static class StorageClientImpl implements StorageClient {
@@ -1011,8 +1034,13 @@
}
@Override
- public Iterable<ReadRowsResponse> readRows(ReadRowsRequest request) {
- return client.readRowsCallable().call(request);
+ public BigQueryServerStream<ReadRowsResponse> readRows(ReadRowsRequest request) {
+ return new BigQueryServerStreamImpl(client.readRowsCallable().call(request));
+ }
+
+ @Override
+ public SplitReadStreamResponse splitReadStream(SplitReadStreamRequest request) {
+ return client.splitReadStream(request);
}
@Override
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
index 14c9bf3..8125761 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java
@@ -21,7 +21,7 @@
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.createJobIdToken;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.getExtractJobId;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.resolveTempLocation;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.bigquery.model.Job;
import com.google.api.services.bigquery.model.JobConfigurationExtract;
@@ -43,11 +43,11 @@
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.JobService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Function;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Function;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java
new file mode 100644
index 0000000..0f3de1d
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+
+/**
+ * Represents a source used for {@link BigQueryIO#read(SerializableFunction)}. Currently this could
+ * be either a table or a query. Direct read sources are not yet supported.
+ */
+interface BigQuerySourceDef extends Serializable {
+ /**
+ * Convert this source definition into a concrete source implementation.
+ *
+ * @param stepUuid Job UUID
+ * @param coder Coder
+ * @param parseFn Parse function
+ * @param <T> Type of the resulting PCollection
+ * @return An implementation of {@link BigQuerySourceBase}
+ */
+ <T> BigQuerySourceBase<T> toSource(
+ String stepUuid, Coder<T> coder, SerializableFunction<SchemaAndRecord, T> parseFn);
+
+ /**
+ * Extract the Beam {@link Schema} corresponding to this source.
+ *
+ * @param bqOptions BigQueryOptions
+ * @return Beam schema of the source
+ * @throws BigQuerySchemaRetrievalException if schema retrieval fails
+ */
+ @Experimental(Experimental.Kind.SCHEMAS)
+ Schema getBeamSchema(BigQueryOptions bqOptions);
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageQuerySource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageQuerySource.java
index 9112e56..579ddcd 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageQuerySource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageQuerySource.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.bigquery.model.JobStatistics;
import com.google.api.services.bigquery.model.Table;
@@ -83,7 +83,7 @@
SerializableFunction<SchemaAndRecord, T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices) {
- super(null, parseFn, outputCoder, bqServices);
+ super(null, null, null, parseFn, outputCoder, bqServices);
this.stepUuid = checkNotNull(stepUuid, "stepUuid");
this.queryProvider = checkNotNull(queryProvider, "queryProvider");
this.flattenResults = checkNotNull(flattenResults, "flattenResults");
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
index ccd1f12..2c7725c 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
@@ -17,13 +17,15 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.bigquery.model.Table;
import com.google.cloud.bigquery.storage.v1beta1.ReadOptions.TableReadOptions;
import com.google.cloud.bigquery.storage.v1beta1.Storage.CreateReadSessionRequest;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession;
import com.google.cloud.bigquery.storage.v1beta1.Storage.Stream;
+import com.google.protobuf.UnknownFieldSet;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nullable;
@@ -32,9 +34,12 @@
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.StorageClient;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* A base class for {@link BoundedSource} implementations which read from BigQuery using the
@@ -43,6 +48,8 @@
@Experimental(Experimental.Kind.SOURCE_SINK)
abstract class BigQueryStorageSourceBase<T> extends BoundedSource<T> {
+ private static final Logger LOGGER = LoggerFactory.getLogger(BigQueryStorageSourceBase.class);
+
/**
* The maximum number of streams which will be requested when creating a read session, regardless
* of the desired bundle size.
@@ -57,16 +64,26 @@
private static final int MIN_SPLIT_COUNT = 10;
protected final TableReadOptions tableReadOptions;
+ protected final ValueProvider<List<String>> selectedFieldsProvider;
+ protected final ValueProvider<String> rowRestrictionProvider;
protected final SerializableFunction<SchemaAndRecord, T> parseFn;
protected final Coder<T> outputCoder;
protected final BigQueryServices bqServices;
BigQueryStorageSourceBase(
@Nullable TableReadOptions tableReadOptions,
+ @Nullable ValueProvider<List<String>> selectedFieldsProvider,
+ @Nullable ValueProvider<String> rowRestrictionProvider,
SerializableFunction<SchemaAndRecord, T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices) {
+ checkArgument(
+ tableReadOptions == null
+ || (selectedFieldsProvider == null && rowRestrictionProvider == null),
+ "tableReadOptions is mutually exclusive with selectedFieldsProvider and rowRestrictionProvider");
this.tableReadOptions = tableReadOptions;
+ this.selectedFieldsProvider = selectedFieldsProvider;
+ this.rowRestrictionProvider = rowRestrictionProvider;
this.parseFn = checkNotNull(parseFn, "parseFn");
this.outputCoder = checkNotNull(outputCoder, "outputCoder");
this.bqServices = checkNotNull(bqServices, "bqServices");
@@ -100,15 +117,35 @@
CreateReadSessionRequest.newBuilder()
.setParent("projects/" + bqOptions.getProject())
.setTableReference(BigQueryHelpers.toTableRefProto(targetTable.getTableReference()))
- .setRequestedStreams(streamCount);
+ .setRequestedStreams(streamCount)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build());
- if (tableReadOptions != null) {
+ if (selectedFieldsProvider != null || rowRestrictionProvider != null) {
+ TableReadOptions.Builder builder = TableReadOptions.newBuilder();
+ if (selectedFieldsProvider != null) {
+ builder.addAllSelectedFields(selectedFieldsProvider.get());
+ }
+ if (rowRestrictionProvider != null) {
+ builder.setRowRestriction(rowRestrictionProvider.get());
+ }
+ requestBuilder.setReadOptions(builder);
+ } else if (tableReadOptions != null) {
requestBuilder.setReadOptions(tableReadOptions);
}
ReadSession readSession;
try (StorageClient client = bqServices.getStorageClient(bqOptions)) {
- readSession = client.createReadSession(requestBuilder.build());
+ CreateReadSessionRequest request = requestBuilder.build();
+ readSession = client.createReadSession(request);
+ LOGGER.info(
+ "Sent BigQuery Storage API CreateReadSession request '{}'; received response '{}'.",
+ request,
+ readSession);
}
if (readSession.getStreamsList().isEmpty()) {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
index ed3995e..d9c3888 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
@@ -19,14 +19,18 @@
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.fromJsonString;
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.toJsonString;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import com.google.api.gax.rpc.FailedPreconditionException;
import com.google.api.services.bigquery.model.TableSchema;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsRequest;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsResponse;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamRequest;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamResponse;
import com.google.cloud.bigquery.storage.v1beta1.Storage.Stream;
import com.google.cloud.bigquery.storage.v1beta1.Storage.StreamPosition;
+import com.google.protobuf.UnknownFieldSet;
import java.io.IOException;
import java.util.Iterator;
import java.util.List;
@@ -39,16 +43,24 @@
import org.apache.avro.io.DecoderFactory;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.io.OffsetBasedSource;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.BigQueryServerStream;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.StorageClient;
+import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** A {@link org.apache.beam.sdk.io.Source} representing a single stream in a read session. */
@Experimental(Experimental.Kind.SOURCE_SINK)
-public class BigQueryStorageStreamSource<T> extends OffsetBasedSource<T> {
+public class BigQueryStorageStreamSource<T> extends BoundedSource<T> {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(BigQueryStorageStreamSource.class);
public static <T> BigQueryStorageStreamSource<T> create(
ReadSession readSession,
@@ -60,15 +72,21 @@
return new BigQueryStorageStreamSource<>(
readSession,
stream,
- 0L,
- Long.MAX_VALUE,
- 1L,
toJsonString(checkNotNull(tableSchema, "tableSchema")),
parseFn,
outputCoder,
bqServices);
}
+ /**
+ * Creates a new source with the same properties as this one, except with a different {@link
+ * Stream}.
+ */
+ public BigQueryStorageStreamSource<T> fromExisting(Stream newStream) {
+ return new BigQueryStorageStreamSource(
+ readSession, newStream, jsonTableSchema, parseFn, outputCoder, bqServices);
+ }
+
private final ReadSession readSession;
private final Stream stream;
private final String jsonTableSchema;
@@ -79,14 +97,10 @@
private BigQueryStorageStreamSource(
ReadSession readSession,
Stream stream,
- long startOffset,
- long stopOffset,
- long minBundleSize,
String jsonTableSchema,
SerializableFunction<SchemaAndRecord, T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices) {
- super(startOffset, stopOffset, minBundleSize);
this.readSession = checkNotNull(readSession, "readSession");
this.stream = checkNotNull(stream, "stream");
this.jsonTableSchema = checkNotNull(jsonTableSchema, "jsonTableSchema");
@@ -119,7 +133,7 @@
}
@Override
- public List<? extends OffsetBasedSource<T>> split(
+ public List<? extends BoundedSource<T>> split(
long desiredBundleSizeBytes, PipelineOptions options) {
// A stream source can't be split without reading from it due to server-side liquid sharding.
// TODO: Implement dynamic work rebalancing.
@@ -127,52 +141,58 @@
}
@Override
- public long getMaxEndOffset(PipelineOptions options) {
- // This method should never be called given the overrides above.
- throw new UnsupportedOperationException("Not implemented");
- }
-
- @Override
- public OffsetBasedSource<T> createSourceForSubrange(long start, long end) {
- // This method should never be called given the overrides above.
- throw new UnsupportedOperationException("Not implemented");
- }
-
- @Override
public BigQueryStorageStreamReader<T> createReader(PipelineOptions options) throws IOException {
return new BigQueryStorageStreamReader<>(this, options.as(BigQueryOptions.class));
}
+ @Override
+ public String toString() {
+ return stream.toString();
+ }
+
/** A {@link org.apache.beam.sdk.io.Source.Reader} which reads records from a stream. */
@Experimental(Experimental.Kind.SOURCE_SINK)
- public static class BigQueryStorageStreamReader<T> extends OffsetBasedReader<T> {
+ public static class BigQueryStorageStreamReader<T> extends BoundedSource.BoundedReader<T> {
private final DatumReader<GenericRecord> datumReader;
private final SerializableFunction<SchemaAndRecord, T> parseFn;
private final StorageClient storageClient;
private final TableSchema tableSchema;
+ private BigQueryStorageStreamSource<T> source;
+ private BigQueryServerStream<ReadRowsResponse> responseStream;
private Iterator<ReadRowsResponse> responseIterator;
private BinaryDecoder decoder;
private GenericRecord record;
private T current;
private long currentOffset;
+ // Values used for progress reporting.
+ private double fractionConsumed;
+ private double fractionConsumedFromPreviousResponse;
+ private double fractionConsumedFromCurrentResponse;
+ private long rowsReadFromCurrentResponse;
+ private long totalRowCountFromCurrentResponse;
+
private BigQueryStorageStreamReader(
BigQueryStorageStreamSource<T> source, BigQueryOptions options) throws IOException {
- super(source);
+ this.source = source;
this.datumReader =
new GenericDatumReader<>(
new Schema.Parser().parse(source.readSession.getAvroSchema().getSchema()));
this.parseFn = source.parseFn;
this.storageClient = source.bqServices.getStorageClient(options);
this.tableSchema = fromJsonString(source.jsonTableSchema, TableSchema.class);
+ this.fractionConsumed = 0d;
+ this.fractionConsumedFromPreviousResponse = 0d;
+ this.fractionConsumedFromCurrentResponse = 0d;
+ this.rowsReadFromCurrentResponse = 0L;
+ this.totalRowCountFromCurrentResponse = 0L;
}
@Override
- protected boolean startImpl() throws IOException {
+ public synchronized boolean start() throws IOException {
BigQueryStorageStreamSource<T> source = getCurrentSource();
- currentOffset = source.getStartOffset();
ReadRowsRequest request =
ReadRowsRequest.newBuilder()
@@ -180,32 +200,69 @@
StreamPosition.newBuilder().setStream(source.stream).setOffset(currentOffset))
.build();
- responseIterator = storageClient.readRows(request).iterator();
+ responseStream = storageClient.readRows(request);
+ responseIterator = responseStream.iterator();
+ LOGGER.info("Started BigQuery Storage API read from stream {}.", source.stream.getName());
return readNextRecord();
}
@Override
- protected boolean advanceImpl() throws IOException {
+ public synchronized boolean advance() throws IOException {
currentOffset++;
return readNextRecord();
}
- private boolean readNextRecord() throws IOException {
+ private synchronized boolean readNextRecord() throws IOException {
while (decoder == null || decoder.isEnd()) {
if (!responseIterator.hasNext()) {
+ fractionConsumed = 1d;
return false;
}
- ReadRowsResponse nextResponse = responseIterator.next();
-
+ fractionConsumedFromPreviousResponse = fractionConsumedFromCurrentResponse;
+ ReadRowsResponse currentResponse = responseIterator.next();
decoder =
DecoderFactory.get()
.binaryDecoder(
- nextResponse.getAvroRows().getSerializedBinaryRows().toByteArray(), decoder);
+ currentResponse.getAvroRows().getSerializedBinaryRows().toByteArray(), decoder);
+
+ // Since we now have a new response, reset the row counter for the current response.
+ rowsReadFromCurrentResponse = 0L;
+
+ totalRowCountFromCurrentResponse = currentResponse.getAvroRows().getRowCount();
+ fractionConsumedFromCurrentResponse = getFractionConsumed(currentResponse);
+
+ Preconditions.checkArgument(
+ totalRowCountFromCurrentResponse > 0L,
+ "Row count from current response (%s) must be greater than one.",
+ totalRowCountFromCurrentResponse);
+ Preconditions.checkArgument(
+ 0f <= fractionConsumedFromCurrentResponse && fractionConsumedFromCurrentResponse <= 1f,
+ "Fraction consumed from current response (%s) is not in the range [0.0, 1.0].",
+ fractionConsumedFromCurrentResponse);
+ Preconditions.checkArgument(
+ fractionConsumedFromPreviousResponse < fractionConsumedFromCurrentResponse,
+ "Fraction consumed from previous response (%s) is not less than fraction consumed "
+ + "from current response (%s).",
+ fractionConsumedFromPreviousResponse,
+ fractionConsumedFromCurrentResponse);
}
record = datumReader.read(record, decoder);
current = parseFn.apply(new SchemaAndRecord(record, tableSchema));
+
+ // Updates the fraction consumed value. This value is calculated by interpolating between
+ // the fraction consumed value from the previous server response (or zero if we're consuming
+ // the first response) and the fractional value in the current response based on how many of
+ // the rows in the current response have been consumed.
+ rowsReadFromCurrentResponse++;
+ fractionConsumed =
+ fractionConsumedFromPreviousResponse
+ + (fractionConsumedFromCurrentResponse - fractionConsumedFromPreviousResponse)
+ * rowsReadFromCurrentResponse
+ * 1.0
+ / totalRowCountFromCurrentResponse;
+
return true;
}
@@ -215,23 +272,134 @@
}
@Override
- protected long getCurrentOffset() throws NoSuchElementException {
- return currentOffset;
- }
-
- @Override
- public void close() {
+ public synchronized void close() {
storageClient.close();
}
@Override
public synchronized BigQueryStorageStreamSource<T> getCurrentSource() {
- return (BigQueryStorageStreamSource<T>) super.getCurrentSource();
+ return source;
}
@Override
- public boolean allowsDynamicSplitting() {
- return false;
+ public BoundedSource<T> splitAtFraction(double fraction) {
+ Metrics.counter(BigQueryStorageStreamReader.class, "split-at-fraction-calls").inc();
+ LOGGER.debug(
+ "Received BigQuery Storage API split request for stream {} at fraction {}.",
+ source.stream.getName(),
+ fraction);
+
+ SplitReadStreamRequest splitRequest =
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(source.stream)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits((float) fraction))
+ .build())
+ .build())
+ .build();
+ SplitReadStreamResponse splitResponse = storageClient.splitReadStream(splitRequest);
+
+ if (!splitResponse.hasPrimaryStream() || !splitResponse.hasRemainderStream()) {
+ // No more splits are possible!
+ Metrics.counter(
+ BigQueryStorageStreamReader.class,
+ "split-at-fraction-calls-failed-due-to-impossible-split-point")
+ .inc();
+ LOGGER.info(
+ "BigQuery Storage API stream {} cannot be split at {}.",
+ source.stream.getName(),
+ fraction);
+ return null;
+ }
+
+ // We may be able to split this source. Before continuing, we pause the reader thread and
+ // replace its current source with the primary stream iff the reader has not moved past
+ // the split point.
+ synchronized (this) {
+ BigQueryServerStream<ReadRowsResponse> newResponseStream;
+ Iterator<ReadRowsResponse> newResponseIterator;
+ try {
+ newResponseStream =
+ storageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(
+ StreamPosition.newBuilder()
+ .setStream(splitResponse.getPrimaryStream())
+ .setOffset(currentOffset + 1))
+ .build());
+ newResponseIterator = newResponseStream.iterator();
+ newResponseIterator.hasNext();
+ } catch (FailedPreconditionException e) {
+ // The current source has already moved past the split point, so this split attempt
+ // is unsuccessful.
+ Metrics.counter(
+ BigQueryStorageStreamReader.class,
+ "split-at-fraction-calls-failed-due-to-bad-split-point")
+ .inc();
+ LOGGER.info(
+ "BigQuery Storage API split of stream {} abandoned because the primary stream is to "
+ + "the left of the split fraction {}.",
+ source.stream.getName(),
+ fraction);
+ return null;
+ } catch (Exception e) {
+ Metrics.counter(
+ BigQueryStorageStreamReader.class,
+ "split-at-fraction-calls-failed-due-to-other-reasons")
+ .inc();
+ LOGGER.error("BigQuery Storage API stream split failed.", e);
+ return null;
+ }
+
+ // Cancels the parent stream before replacing it with the primary stream.
+ responseStream.cancel();
+ source = source.fromExisting(splitResponse.getPrimaryStream());
+ responseStream = newResponseStream;
+ responseIterator = newResponseIterator;
+
+ // N.B.: Once #readNextRecord is called, this line has the effect of using the fraction
+ // consumed value at split time as the fraction consumed value of the previous response,
+ // leading to a better interpolation window start. Unfortunately, this is not the best value
+ // as it will lead to a significant speed up in the fraction consumed values while the first
+ // post-split response is being processed. In the future, if the server returns the start
+ // and end fraction consumed values in each response, then these interpolations will be
+ // easier to perform as state from the previous response will not need to be maintained.
+ fractionConsumedFromCurrentResponse = fractionConsumed;
+
+ decoder = null;
+ }
+
+ Metrics.counter(BigQueryStorageStreamReader.class, "split-at-fraction-calls-successful")
+ .inc();
+ LOGGER.info(
+ "Successfully split BigQuery Storage API stream at {}. Split response: {}",
+ fraction,
+ splitResponse);
+ return source.fromExisting(splitResponse.getRemainderStream());
+ }
+
+ @Override
+ public synchronized Double getFractionConsumed() {
+ return fractionConsumed;
+ }
+
+ private static float getFractionConsumed(ReadRowsResponse response) {
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use getFractionConsumed().
+ List<Integer> fractionConsumedField =
+ response.getStatus().getUnknownFields().getField(2).getFixed32List();
+ if (fractionConsumedField.isEmpty()) {
+ Metrics.counter(BigQueryStorageStreamReader.class, "fraction-consumed-not-set").inc();
+ return 0f;
+ }
+
+ return Float.intBitsToFloat(Iterables.getOnlyElement(fractionConsumedField));
}
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageTableSource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageTableSource.java
index e48acf7..709d07f 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageTableSource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageTableSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.bigquery.model.Table;
import com.google.api.services.bigquery.model.TableReference;
@@ -34,7 +34,7 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -47,11 +47,19 @@
public static <T> BigQueryStorageTableSource<T> create(
ValueProvider<TableReference> tableRefProvider,
@Nullable TableReadOptions readOptions,
+ @Nullable ValueProvider<List<String>> selectedFields,
+ @Nullable ValueProvider<String> rowRestriction,
SerializableFunction<SchemaAndRecord, T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices) {
return new BigQueryStorageTableSource<>(
- tableRefProvider, readOptions, parseFn, outputCoder, bqServices);
+ tableRefProvider,
+ readOptions,
+ selectedFields,
+ rowRestriction,
+ parseFn,
+ outputCoder,
+ bqServices);
}
private final ValueProvider<TableReference> tableReferenceProvider;
@@ -61,10 +69,12 @@
private BigQueryStorageTableSource(
ValueProvider<TableReference> tableRefProvider,
@Nullable TableReadOptions readOptions,
+ @Nullable ValueProvider<List<String>> selectedFields,
+ @Nullable ValueProvider<String> rowRestriction,
SerializableFunction<SchemaAndRecord, T> parseFn,
Coder<T> outputCoder,
BigQueryServices bqServices) {
- super(readOptions, parseFn, outputCoder, bqServices);
+ super(readOptions, selectedFields, rowRestriction, parseFn, outputCoder, bqServices);
this.tableReferenceProvider = checkNotNull(tableRefProvider, "tableRefProvider");
cachedTable = new AtomicReference<>();
}
@@ -113,7 +123,9 @@
}
private List<String> getSelectedFields() {
- if (tableReadOptions != null && !tableReadOptions.getSelectedFieldsList().isEmpty()) {
+ if (selectedFieldsProvider != null) {
+ return selectedFieldsProvider.get();
+ } else if (tableReadOptions != null && !tableReadOptions.getSelectedFieldsList().isEmpty()) {
return tableReadOptions.getSelectedFieldsList();
}
return null;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java
index f8ea5e1..87168ea 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSource.java
@@ -17,22 +17,15 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
-
import com.google.api.services.bigquery.model.Table;
import com.google.api.services.bigquery.model.TableReference;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicReference;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.TableRefToJson;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.ValueProvider;
-import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,69 +36,41 @@
static <T> BigQueryTableSource<T> create(
String stepUuid,
- ValueProvider<TableReference> table,
+ BigQueryTableSourceDef tableDef,
BigQueryServices bqServices,
Coder<T> coder,
SerializableFunction<SchemaAndRecord, T> parseFn) {
- return new BigQueryTableSource<>(stepUuid, table, bqServices, coder, parseFn);
+ return new BigQueryTableSource<>(stepUuid, tableDef, bqServices, coder, parseFn);
}
- private final ValueProvider<String> jsonTable;
+ private final BigQueryTableSourceDef tableDef;
private final AtomicReference<Long> tableSizeBytes;
private BigQueryTableSource(
String stepUuid,
- ValueProvider<TableReference> table,
+ BigQueryTableSourceDef tableDef,
BigQueryServices bqServices,
Coder<T> coder,
SerializableFunction<SchemaAndRecord, T> parseFn) {
super(stepUuid, bqServices, coder, parseFn);
- this.jsonTable = NestedValueProvider.of(checkNotNull(table, "table"), new TableRefToJson());
+ this.tableDef = tableDef;
this.tableSizeBytes = new AtomicReference<>();
}
@Override
protected TableReference getTableToExtract(BigQueryOptions bqOptions) throws IOException {
- TableReference tableReference =
- BigQueryIO.JSON_FACTORY.fromString(jsonTable.get(), TableReference.class);
- return setDefaultProjectIfAbsent(bqOptions, tableReference);
- }
-
- /**
- * Sets the {@link TableReference#projectId} of the provided table reference to the id of the
- * default project if the table reference does not have a project ID specified.
- */
- private TableReference setDefaultProjectIfAbsent(
- BigQueryOptions bqOptions, TableReference tableReference) {
- if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
- checkState(
- !Strings.isNullOrEmpty(bqOptions.getProject()),
- "No project ID set in %s or %s, cannot construct a complete %s",
- TableReference.class.getSimpleName(),
- BigQueryOptions.class.getSimpleName(),
- TableReference.class.getSimpleName());
- LOG.info(
- "Project ID not set in {}. Using default project from {}.",
- TableReference.class.getSimpleName(),
- BigQueryOptions.class.getSimpleName());
- tableReference.setProjectId(bqOptions.getProject());
- }
- return tableReference;
+ return tableDef.getTableReference(bqOptions);
}
@Override
public synchronized long getEstimatedSizeBytes(PipelineOptions options) throws Exception {
if (tableSizeBytes.get() == null) {
- TableReference table =
- setDefaultProjectIfAbsent(
- options.as(BigQueryOptions.class),
- BigQueryIO.JSON_FACTORY.fromString(jsonTable.get(), TableReference.class));
-
- Table tableRef =
- bqServices.getDatasetService(options.as(BigQueryOptions.class)).getTable(table);
- Long numBytes = tableRef.getNumBytes();
- if (tableRef.getStreamingBuffer() != null) {
- numBytes += tableRef.getStreamingBuffer().getEstimatedBytes().longValue();
+ BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
+ TableReference tableRef = tableDef.getTableReference(bqOptions);
+ Table table = bqServices.getDatasetService(bqOptions).getTable(tableRef);
+ Long numBytes = table.getNumBytes();
+ if (table.getStreamingBuffer() != null) {
+ numBytes += table.getStreamingBuffer().getEstimatedBytes().longValue();
}
tableSizeBytes.compareAndSet(null, numBytes);
@@ -121,6 +86,6 @@
@Override
public void populateDisplayData(DisplayData.Builder builder) {
super.populateDisplayData(builder);
- builder.add(DisplayData.item("table", jsonTable));
+ builder.add(DisplayData.item("table", tableDef.getJsonTable()));
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSourceDef.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSourceDef.java
new file mode 100644
index 0000000..aa6a71f
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSourceDef.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
+
+import com.google.api.services.bigquery.model.TableReference;
+import com.google.api.services.bigquery.model.TableSchema;
+import java.io.IOException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class BigQueryTableSourceDef implements BigQuerySourceDef {
+ private static final Logger LOG = LoggerFactory.getLogger(BigQueryTableSourceDef.class);
+
+ private final BigQueryServices bqServices;
+ private final ValueProvider<String> jsonTable;
+
+ static BigQueryTableSourceDef create(
+ BigQueryServices bqServices, ValueProvider<TableReference> table) {
+ ValueProvider<String> jsonTable =
+ ValueProvider.NestedValueProvider.of(
+ checkNotNull(table, "table"), new BigQueryHelpers.TableRefToJson());
+ return new BigQueryTableSourceDef(bqServices, jsonTable);
+ }
+
+ private BigQueryTableSourceDef(BigQueryServices bqServices, ValueProvider<String> jsonTable) {
+ this.bqServices = bqServices;
+ this.jsonTable = jsonTable;
+ }
+
+ TableReference getTableReference(BigQueryOptions bqOptions) throws IOException {
+ TableReference tableReference =
+ BigQueryIO.JSON_FACTORY.fromString(jsonTable.get(), TableReference.class);
+ return setDefaultProjectIfAbsent(bqOptions, tableReference);
+ }
+
+ /**
+ * Sets the {@link TableReference#projectId} of the provided table reference to the id of the
+ * default project if the table reference does not have a project ID specified.
+ */
+ private TableReference setDefaultProjectIfAbsent(
+ BigQueryOptions bqOptions, TableReference tableReference) {
+ if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
+ checkState(
+ !Strings.isNullOrEmpty(bqOptions.getProject()),
+ "No project ID set in %s or %s, cannot construct a complete %s",
+ TableReference.class.getSimpleName(),
+ BigQueryOptions.class.getSimpleName(),
+ TableReference.class.getSimpleName());
+ LOG.info(
+ "Project ID not set in {}. Using default project from {}.",
+ TableReference.class.getSimpleName(),
+ BigQueryOptions.class.getSimpleName());
+ tableReference.setProjectId(bqOptions.getProject());
+ }
+ return tableReference;
+ }
+
+ ValueProvider<String> getJsonTable() {
+ return jsonTable;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public <T> BigQuerySourceBase<T> toSource(
+ String stepUuid, Coder<T> coder, SerializableFunction<SchemaAndRecord, T> parseFn) {
+ return BigQueryTableSource.create(stepUuid, this, bqServices, coder, parseFn);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public Schema getBeamSchema(BigQueryOptions bqOptions) {
+ try {
+ TableReference tableRef = getTableReference(bqOptions);
+ TableSchema tableSchema =
+ bqServices.getDatasetService(bqOptions).getTable(tableRef).getSchema();
+ return BigQueryUtils.fromTableSchema(tableSchema);
+ } catch (IOException | InterruptedException | NullPointerException e) {
+ throw new BigQuerySchemaRetrievalException("Exception while trying to retrieve schema", e);
+ }
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
index bd1fda3..04a92ec0d8 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
@@ -27,14 +27,16 @@
import com.google.auto.value.AutoValue;
import java.io.Serializable;
import java.math.BigDecimal;
-import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.function.Function;
import java.util.stream.IntStream;
+import org.apache.avro.generic.GenericData;
import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.sdk.schemas.LogicalTypes;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.Field;
import org.apache.beam.sdk.schemas.Schema.FieldType;
@@ -42,11 +44,12 @@
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.SerializableFunctions;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.joda.time.Instant;
import org.joda.time.ReadableInstant;
import org.joda.time.chrono.ISOChronology;
@@ -88,6 +91,51 @@
}
}
+ private static final DateTimeFormatter BIGQUERY_TIMESTAMP_PRINTER;
+
+ /**
+ * Native BigQuery formatter for it's timestamp format, depending on the milliseconds stored in
+ * the column, the milli second part will be 6, 3 or absent. Example {@code 2019-08-16
+ * 00:52:07[.123]|[.123456] UTC}
+ */
+ private static final DateTimeFormatter BIGQUERY_TIMESTAMP_PARSER;
+
+ static {
+ DateTimeFormatter dateTimePart =
+ new DateTimeFormatterBuilder()
+ .appendYear(4, 4)
+ .appendLiteral('-')
+ .appendMonthOfYear(2)
+ .appendLiteral('-')
+ .appendDayOfMonth(2)
+ .appendLiteral(' ')
+ .appendHourOfDay(2)
+ .appendLiteral(':')
+ .appendMinuteOfHour(2)
+ .appendLiteral(':')
+ .appendSecondOfMinute(2)
+ .toFormatter()
+ .withZoneUTC();
+ BIGQUERY_TIMESTAMP_PARSER =
+ new DateTimeFormatterBuilder()
+ .append(dateTimePart)
+ .appendOptional(
+ new DateTimeFormatterBuilder()
+ .appendLiteral('.')
+ .appendFractionOfSecond(3, 6)
+ .toParser())
+ .appendLiteral(" UTC")
+ .toFormatter()
+ .withZoneUTC();
+ BIGQUERY_TIMESTAMP_PRINTER =
+ new DateTimeFormatterBuilder()
+ .append(dateTimePart)
+ .appendLiteral('.')
+ .appendFractionOfSecond(3, 3)
+ .appendLiteral(" UTC")
+ .toFormatter();
+ }
+
private static final Map<TypeName, StandardSQLTypeName> BEAM_TO_BIGQUERY_TYPE_MAPPING =
ImmutableMap.<TypeName, StandardSQLTypeName>builder()
.put(TypeName.BYTE, StandardSQLTypeName.INT64)
@@ -118,9 +166,18 @@
.put(TypeName.STRING, str -> str)
.put(
TypeName.DATETIME,
- str ->
- new DateTime(
- (long) (Double.parseDouble(str) * 1000), ISOChronology.getInstanceUTC()))
+ str -> {
+ if (str == null || str.length() == 0) {
+ return null;
+ }
+ if (str.endsWith("UTC")) {
+ return BIGQUERY_TIMESTAMP_PARSER.parseDateTime(str).toDateTime(DateTimeZone.UTC);
+ } else {
+ return new DateTime(
+ (long) (Double.parseDouble(str) * 1000), ISOChronology.getInstanceUTC());
+ }
+ })
+ .put(TypeName.BYTES, str -> BaseEncoding.base64().decode(str))
.build();
// TODO: BigQuery code should not be relying on Calcite metadata fields. If so, this belongs
@@ -130,7 +187,7 @@
.put("SqlDateType", StandardSQLTypeName.DATE)
.put("SqlTimeType", StandardSQLTypeName.TIME)
.put("SqlTimeWithLocalTzType", StandardSQLTypeName.TIME)
- .put("SqlTimestampWithLocalTzType", StandardSQLTypeName.TIMESTAMP)
+ .put("SqlTimestampWithLocalTzType", StandardSQLTypeName.DATETIME)
.put("SqlCharType", StandardSQLTypeName.STRING)
.build();
@@ -149,6 +206,79 @@
return BEAM_TO_BIGQUERY_TYPE_MAPPING.get(fieldType.getTypeName());
}
+ /**
+ * Get the Beam {@link FieldType} from a BigQuery type name.
+ *
+ * <p>Supports both standard and legacy SQL types.
+ *
+ * @param typeName Name of the type
+ * @param nestedFields Nested fields for the given type (eg. RECORD type)
+ * @return Corresponding Beam {@link FieldType}
+ */
+ private static FieldType fromTableFieldSchemaType(
+ String typeName, List<TableFieldSchema> nestedFields) {
+ switch (typeName) {
+ case "STRING":
+ return FieldType.STRING;
+ case "BYTES":
+ return FieldType.BYTES;
+ case "INT64":
+ case "INTEGER":
+ return FieldType.INT64;
+ case "FLOAT64":
+ case "FLOAT":
+ return FieldType.DOUBLE;
+ case "BOOL":
+ case "BOOLEAN":
+ return FieldType.BOOLEAN;
+ case "TIMESTAMP":
+ return FieldType.DATETIME;
+ case "TIME":
+ return FieldType.logicalType(
+ new LogicalTypes.PassThroughLogicalType<Instant>(
+ "SqlTimeType", "", FieldType.DATETIME) {});
+ case "DATE":
+ return FieldType.logicalType(
+ new LogicalTypes.PassThroughLogicalType<Instant>(
+ "SqlDateType", "", FieldType.DATETIME) {});
+ case "DATETIME":
+ return FieldType.logicalType(
+ new LogicalTypes.PassThroughLogicalType<Instant>(
+ "SqlTimestampWithLocalTzType", "", FieldType.DATETIME) {});
+ case "STRUCT":
+ case "RECORD":
+ Schema rowSchema = fromTableFieldSchema(nestedFields);
+ return FieldType.row(rowSchema);
+ default:
+ throw new UnsupportedOperationException(
+ "Converting BigQuery type " + typeName + " to Beam type is unsupported");
+ }
+ }
+
+ private static Schema fromTableFieldSchema(List<TableFieldSchema> tableFieldSchemas) {
+ Schema.Builder schemaBuilder = Schema.builder();
+ for (TableFieldSchema tableFieldSchema : tableFieldSchemas) {
+ FieldType fieldType =
+ fromTableFieldSchemaType(tableFieldSchema.getType(), tableFieldSchema.getFields());
+
+ Optional<Mode> fieldMode = Optional.ofNullable(tableFieldSchema.getMode()).map(Mode::valueOf);
+ if (fieldMode.filter(m -> m == Mode.REPEATED).isPresent()) {
+ fieldType = FieldType.array(fieldType);
+ }
+
+ // if the mode is not defined or if it is set to NULLABLE, then the field is nullable
+ boolean nullable =
+ !fieldMode.isPresent() || fieldMode.filter(m -> m == Mode.NULLABLE).isPresent();
+ Field field = Field.of(tableFieldSchema.getName(), fieldType).withNullable(nullable);
+ if (tableFieldSchema.getDescription() != null
+ && !"".equals(tableFieldSchema.getDescription())) {
+ field = field.withDescription(tableFieldSchema.getDescription());
+ }
+ schemaBuilder.addField(field);
+ }
+ return schemaBuilder.build();
+ }
+
private static List<TableFieldSchema> toTableFieldSchema(Schema schema) {
List<TableFieldSchema> fields = new ArrayList<>(schema.getFieldCount());
for (Field schemaField : schema.getFields()) {
@@ -188,6 +318,31 @@
return new TableSchema().setFields(toTableFieldSchema(schema));
}
+ /** Convert a BigQuery {@link TableSchema} to a Beam {@link Schema}. */
+ public static Schema fromTableSchema(TableSchema tableSchema) {
+ return fromTableFieldSchema(tableSchema.getFields());
+ }
+
+ /** Convert a list of BigQuery {@link TableFieldSchema} to Avro {@link org.apache.avro.Schema}. */
+ public static org.apache.avro.Schema toGenericAvroSchema(
+ String schemaName, List<TableFieldSchema> fieldSchemas) {
+ return BigQueryAvroUtils.toGenericAvroSchema(schemaName, fieldSchemas);
+ }
+
+ private static final BigQueryIO.TypedRead.ToBeamRowFunction<TableRow>
+ TABLE_ROW_TO_BEAM_ROW_FUNCTION = beamSchema -> (TableRow tr) -> toBeamRow(beamSchema, tr);
+
+ public static final BigQueryIO.TypedRead.ToBeamRowFunction<TableRow> tableRowToBeamRow() {
+ return TABLE_ROW_TO_BEAM_ROW_FUNCTION;
+ }
+
+ private static final BigQueryIO.TypedRead.FromBeamRowFunction<TableRow>
+ TABLE_ROW_FROM_BEAM_ROW_FUNCTION = ignored -> BigQueryUtils::toTableRow;
+
+ public static final BigQueryIO.TypedRead.FromBeamRowFunction<TableRow> tableRowFromBeamRow() {
+ return TABLE_ROW_FROM_BEAM_ROW_FUNCTION;
+ }
+
private static final SerializableFunction<Row, TableRow> ROW_TO_TABLE_ROW =
new ToTableRow(SerializableFunctions.identity());
@@ -219,7 +374,15 @@
public static Row toBeamRow(GenericRecord record, Schema schema, ConversionOptions options) {
List<Object> valuesInOrder =
schema.getFields().stream()
- .map(field -> convertAvroFormat(field, record.get(field.getName()), options))
+ .map(
+ field -> {
+ try {
+ return convertAvroFormat(field.getType(), record.get(field.getName()), options);
+ } catch (Exception cause) {
+ throw new IllegalArgumentException(
+ "Error converting field " + field + ": " + cause.getMessage(), cause);
+ }
+ })
.collect(toList());
return Row.withSchema(schema).addValues(valuesInOrder).build();
@@ -258,11 +421,9 @@
return toTableRow((Row) fieldValue);
case DATETIME:
- DateTimeFormatter patternFormat =
- new DateTimeFormatterBuilder()
- .appendPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZZ")
- .toFormatter();
- return ((Instant) fieldValue).toDateTime().toString(patternFormat);
+ return ((Instant) fieldValue)
+ .toDateTime(DateTimeZone.UTC)
+ .toString(BIGQUERY_TIMESTAMP_PRINTER);
case INT16:
case INT32:
@@ -277,10 +438,7 @@
return fieldValue.toString();
case BYTES:
- ByteBuffer byteBuffer = (ByteBuffer) fieldValue;
- byte[] bytes = new byte[byteBuffer.limit()];
- byteBuffer.get(bytes);
- return BaseEncoding.base64().encode(bytes);
+ return BaseEncoding.base64().encode((byte[]) fieldValue);
default:
return fieldValue;
@@ -288,6 +446,36 @@
}
/**
+ * Tries to convert a JSON {@link TableRow} from BigQuery into a Beam {@link Row}.
+ *
+ * <p>Only supports basic types and arrays. Doesn't support date types or structs.
+ */
+ public static Row toBeamRow(Schema rowSchema, TableRow jsonBqRow) {
+ // TODO deprecate toBeamRow(Schema, TableSchema, TableRow) function in favour of this function.
+ // This function attempts to convert TableRows without having access to the
+ // corresponding TableSchema because:
+ // 1. TableSchema contains redundant information already available in the Schema object.
+ // 2. TableSchema objects are not serializable and are therefore harder to propagate through a
+ // pipeline.
+ return rowSchema.getFields().stream()
+ .map(field -> toBeamRowFieldValue(field, jsonBqRow.get(field.getName())))
+ .collect(toRow(rowSchema));
+ }
+
+ private static Object toBeamRowFieldValue(Field field, Object bqValue) {
+ if (bqValue == null) {
+ if (field.getType().getNullable()) {
+ return null;
+ } else {
+ throw new IllegalArgumentException(
+ "Received null value for non-nullable field " + field.getName());
+ }
+ }
+
+ return toBeamValue(field.getType(), bqValue);
+ }
+
+ /**
* Tries to parse the JSON {@link TableRow} from BigQuery.
*
* <p>Only supports basic types and arrays. Doesn't support date types.
@@ -325,6 +513,12 @@
.collect(toList());
}
+ if (jsonBQValue instanceof Map) {
+ TableRow tr = new TableRow();
+ tr.putAll((Map<String, Object>) jsonBQValue);
+ return toBeamRow(fieldType.getRowSchema(), tr);
+ }
+
throw new UnsupportedOperationException(
"Converting BigQuery type '"
+ jsonBQValue.getClass()
@@ -344,8 +538,15 @@
* Beam field.
*/
public static Object convertAvroFormat(
- Field beamField, Object avroValue, BigQueryUtils.ConversionOptions options) {
- TypeName beamFieldTypeName = beamField.getType().getTypeName();
+ FieldType beamFieldType, Object avroValue, BigQueryUtils.ConversionOptions options) {
+ TypeName beamFieldTypeName = beamFieldType.getTypeName();
+ if (avroValue == null) {
+ if (beamFieldType.getNullable()) {
+ return null;
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s not nullable", beamFieldType));
+ }
+ }
switch (beamFieldTypeName) {
case INT16:
case INT32:
@@ -370,9 +571,9 @@
case STRING:
return convertAvroPrimitiveTypes(beamFieldTypeName, avroValue);
case ARRAY:
- return convertAvroArray(beamField, avroValue);
+ return convertAvroArray(beamFieldType, avroValue, options);
case LOGICAL_TYPE:
- String identifier = beamField.getType().getLogicalType().getIdentifier();
+ String identifier = beamFieldType.getLogicalType().getIdentifier();
if (SQL_DATE_TIME_TYPES.contains(identifier)) {
switch (options.getTruncateTimestamps()) {
case TRUNCATE:
@@ -389,6 +590,13 @@
} else {
throw new RuntimeException("Unknown logical type " + identifier);
}
+ case ROW:
+ Schema rowSchema = beamFieldType.getRowSchema();
+ if (rowSchema == null) {
+ throw new IllegalArgumentException("Nested ROW missing row schema");
+ }
+ GenericData.Record record = (GenericData.Record) avroValue;
+ return toBeamRow(record, rowSchema, options);
case DECIMAL:
throw new RuntimeException("Does not support converting DECIMAL type value");
case MAP:
@@ -418,14 +626,14 @@
return new Instant((long) value / 1000);
}
- private static Object convertAvroArray(Field beamField, Object value) {
+ private static Object convertAvroArray(
+ FieldType beamField, Object value, BigQueryUtils.ConversionOptions options) {
// Check whether the type of array element is equal.
List<Object> values = (List<Object>) value;
List<Object> ret = new ArrayList();
+ FieldType collectionElement = beamField.getCollectionElementType();
for (Object v : values) {
- ret.add(
- convertAvroPrimitiveTypes(
- beamField.getType().getCollectionElementType().getTypeName(), v));
+ ret.add(convertAvroFormat(collectionElement, v, options));
}
return (Object) ret;
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java
index 1ed2c36..69722cc 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.bigquery.model.EncryptionConfiguration;
import com.google.api.services.bigquery.model.Table;
@@ -36,10 +36,10 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* Creates any tables needed before performing streaming writes to the tables. This is a side-effect
@@ -130,6 +130,16 @@
dynamicDestinations,
tableDestination,
destination);
+ boolean destinationCoderSupportsClustering =
+ !(dynamicDestinations.getDestinationCoder() instanceof TableDestinationCoderV2);
+ checkArgument(
+ tableDestination.getClustering() == null || destinationCoderSupportsClustering,
+ "DynamicDestinations.getTable() may only return destinations with clustering configured"
+ + " if a destination coder is supplied that supports clustering, but %s is configured"
+ + " to use TableDestinationCoderV2. Set withClustering() on BigQueryIO.write() and, "
+ + " if you provided a custom DynamicDestinations instance, override"
+ + " getDestinationCoder() to return TableDestinationCoderV3.",
+ dynamicDestinations);
TableReference tableReference = tableDestination.getTableReference().clone();
if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
tableReference.setProjectId(
@@ -185,6 +195,9 @@
.setDescription(tableDestination.getTableDescription());
if (tableDestination.getTimePartitioning() != null) {
table.setTimePartitioning(tableDestination.getTimePartitioning());
+ if (tableDestination.getClustering() != null) {
+ table.setClustering(tableDestination.getClustering());
+ }
}
if (kmsKey != null) {
table.setEncryptionConfiguration(new EncryptionConfiguration().setKmsKeyName(kmsKey));
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
index ea3d435..1000f33 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.io.gcp.bigquery;
import static org.apache.beam.sdk.values.TypeDescriptors.extractFromTypeParameters;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.bigquery.model.TableSchema;
import java.io.Serializable;
@@ -27,12 +27,13 @@
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
/**
* This class provides the most general way of specifying dynamic BigQuery table destinations.
@@ -78,6 +79,7 @@
}
@Nullable private transient SideInputAccessor sideInputAccessor;
+ @Nullable private transient PipelineOptions options;
static class SideInputAccessorViaProcessContext implements SideInputAccessor {
private DoFn<?, ?>.ProcessContext processContext;
@@ -92,6 +94,12 @@
}
}
+ /** Get the current PipelineOptions if set. */
+ @Nullable
+ PipelineOptions getPipelineOptions() {
+ return options;
+ }
+
/**
* Specifies that this object needs access to one or more side inputs. This side inputs must be
* globally windowed, as they will be accessed from the global window.
@@ -114,6 +122,7 @@
void setSideInputAccessorFromProcessContext(DoFn<?, ?>.ProcessContext context) {
this.sideInputAccessor = new SideInputAccessorViaProcessContext(context);
+ this.options = context.getPipelineOptions();
}
/**
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java
index d006220..b58d18d 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java
@@ -17,27 +17,41 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableReference;
import com.google.api.services.bigquery.model.TableSchema;
+import java.io.IOException;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.beam.sdk.coders.CannotProvideCoderException;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.JsonTableRefToTableSpec;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** Contains some useful helper instances of {@link DynamicDestinations}. */
class DynamicDestinationsHelpers {
+ private static final Logger LOG = LoggerFactory.getLogger(DynamicDestinationsHelpers.class);
+
/** Always returns a constant table destination. */
static class ConstantTableDestinations<T> extends DynamicDestinations<T, TableDestination> {
private final ValueProvider<String> tableSpec;
@@ -82,13 +96,16 @@
}
}
- /** Returns a tables based on a user-supplied function. */
+ /** Returns tables based on a user-supplied function. */
static class TableFunctionDestinations<T> extends DynamicDestinations<T, TableDestination> {
private final SerializableFunction<ValueInSingleWindow<T>, TableDestination> tableFunction;
+ private final boolean clusteringEnabled;
TableFunctionDestinations(
- SerializableFunction<ValueInSingleWindow<T>, TableDestination> tableFunction) {
+ SerializableFunction<ValueInSingleWindow<T>, TableDestination> tableFunction,
+ boolean clusteringEnabled) {
this.tableFunction = tableFunction;
+ this.clusteringEnabled = clusteringEnabled;
}
@Override
@@ -114,7 +131,11 @@
@Override
public Coder<TableDestination> getDestinationCoder() {
- return TableDestinationCoderV2.of();
+ if (clusteringEnabled) {
+ return TableDestinationCoderV3.of();
+ } else {
+ return TableDestinationCoderV2.of();
+ }
}
}
@@ -154,6 +175,10 @@
@Override
Coder<DestinationT> getDestinationCoderWithDefault(CoderRegistry registry)
throws CannotProvideCoderException {
+ Coder<DestinationT> destinationCoder = getDestinationCoder();
+ if (destinationCoder != null) {
+ return destinationCoder;
+ }
return inner.getDestinationCoderWithDefault(registry);
}
@@ -206,16 +231,19 @@
extends DelegatingDynamicDestinations<T, TableDestination> {
@Nullable private final ValueProvider<String> jsonTimePartitioning;
+ @Nullable private final ValueProvider<String> jsonClustering;
ConstantTimePartitioningDestinations(
DynamicDestinations<T, TableDestination> inner,
- ValueProvider<String> jsonTimePartitioning) {
+ ValueProvider<String> jsonTimePartitioning,
+ ValueProvider<String> jsonClustering) {
super(inner);
checkArgument(jsonTimePartitioning != null, "jsonTimePartitioning provider can not be null");
if (jsonTimePartitioning.isAccessible()) {
checkArgument(jsonTimePartitioning.get() != null, "jsonTimePartitioning can not be null");
}
this.jsonTimePartitioning = jsonTimePartitioning;
+ this.jsonClustering = jsonClustering;
}
@Override
@@ -224,20 +252,31 @@
String partitioning = this.jsonTimePartitioning.get();
checkArgument(partitioning != null, "jsonTimePartitioning can not be null");
return new TableDestination(
- destination.getTableSpec(), destination.getTableDescription(), partitioning);
+ destination.getTableSpec(),
+ destination.getTableDescription(),
+ partitioning,
+ Optional.ofNullable(jsonClustering).map(ValueProvider::get).orElse(null));
}
@Override
public Coder<TableDestination> getDestinationCoder() {
- return TableDestinationCoderV2.of();
+ if (jsonClustering != null) {
+ return TableDestinationCoderV3.of();
+ } else {
+ return TableDestinationCoderV2.of();
+ }
}
@Override
public String toString() {
- return MoreObjects.toStringHelper(this)
- .add("inner", inner)
- .add("jsonTimePartitioning", jsonTimePartitioning)
- .toString();
+ MoreObjects.ToStringHelper helper =
+ MoreObjects.toStringHelper(this)
+ .add("inner", inner)
+ .add("jsonTimePartitioning", jsonTimePartitioning);
+ if (jsonClustering != null) {
+ helper.add("jsonClustering", jsonClustering);
+ }
+ return helper.toString();
}
}
@@ -285,4 +324,81 @@
.toString();
}
}
+
+ static <T, DestinationT> DynamicDestinations<T, DestinationT> matchTableDynamicDestinations(
+ DynamicDestinations<T, DestinationT> inner, BigQueryServices bqServices) {
+ return new MatchTableDynamicDestinations<>(inner, bqServices);
+ }
+
+ static class MatchTableDynamicDestinations<T, DestinationT>
+ extends DelegatingDynamicDestinations<T, DestinationT> {
+ private final BigQueryServices bqServices;
+
+ private MatchTableDynamicDestinations(
+ DynamicDestinations<T, DestinationT> inner, BigQueryServices bqServices) {
+ super(inner);
+ this.bqServices = bqServices;
+ }
+
+ private Table getBigQueryTable(TableReference tableReference) {
+ BackOff backoff =
+ BackOffAdapter.toGcpBackOff(
+ FluentBackoff.DEFAULT
+ .withMaxRetries(3)
+ .withInitialBackoff(Duration.standardSeconds(1))
+ .withMaxBackoff(Duration.standardSeconds(2))
+ .backoff());
+ try {
+ do {
+ try {
+ BigQueryOptions bqOptions = getPipelineOptions().as(BigQueryOptions.class);
+ return bqServices.getDatasetService(bqOptions).getTable(tableReference);
+ } catch (InterruptedException | IOException e) {
+ LOG.info("Failed to get BigQuery table " + tableReference);
+ }
+ } while (nextBackOff(Sleeper.DEFAULT, backoff));
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ /** Identical to {@link BackOffUtils#next} but without checked IOException. */
+ private static boolean nextBackOff(Sleeper sleeper, BackOff backoff)
+ throws InterruptedException {
+ try {
+ return BackOffUtils.next(sleeper, backoff);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /** Returns a {@link TableDestination} object for the destination. May not return null. */
+ @Override
+ public TableDestination getTable(DestinationT destination) {
+ TableDestination wrappedDestination = super.getTable(destination);
+ Table existingTable = getBigQueryTable(wrappedDestination.getTableReference());
+
+ if (existingTable == null) {
+ return wrappedDestination;
+ } else {
+ return new TableDestination(
+ wrappedDestination.getTableSpec(),
+ existingTable.getDescription(),
+ existingTable.getTimePartitioning());
+ }
+ }
+
+ /** Returns the table schema for the destination. May not return null. */
+ @Override
+ public TableSchema getSchema(DestinationT destination) {
+ TableDestination wrappedDestination = super.getTable(destination);
+ Table existingTable = getBigQueryTable(wrappedDestination.getTableReference());
+ if (existingTable == null) {
+ return super.getSchema(destination);
+ } else {
+ return existingTable.getSchema();
+ }
+ }
+ }
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.java
index 97f390e..7e82f63 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.java
@@ -21,7 +21,7 @@
import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
import java.io.Serializable;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
/** A retry policy for streaming BigQuery inserts. */
public abstract class InsertRetryPolicy implements Serializable {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PassThroughThenCleanup.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PassThroughThenCleanup.java
index 6f368dc..8710b6d 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PassThroughThenCleanup.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PassThroughThenCleanup.java
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* A {@link PTransform} that invokes {@link CleanupOperation} after the input {@link PCollection}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PrepareWrite.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PrepareWrite.java
index aebaae1..716ce68 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PrepareWrite.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PrepareWrite.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.bigquery.model.TableRow;
import java.io.IOException;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java
index 155320f..f56cf01 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java
@@ -34,8 +34,8 @@
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
/** Implementation of DoFn to perform streaming BigQuery write. */
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java
index 4b605ed..08e9ce1 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
+import com.google.api.services.bigquery.model.Clustering;
import com.google.api.services.bigquery.model.TableReference;
import com.google.api.services.bigquery.model.TimePartitioning;
import java.io.Serializable;
@@ -29,13 +30,14 @@
private final String tableSpec;
@Nullable private final String tableDescription;
@Nullable private final String jsonTimePartitioning;
+ @Nullable private final String jsonClustering;
public TableDestination(String tableSpec, @Nullable String tableDescription) {
- this(tableSpec, tableDescription, (String) null);
+ this(tableSpec, tableDescription, (String) null, (String) null);
}
public TableDestination(TableReference tableReference, @Nullable String tableDescription) {
- this(tableReference, tableDescription, (String) null);
+ this(tableReference, tableDescription, (String) null, (String) null);
}
public TableDestination(
@@ -45,7 +47,8 @@
this(
BigQueryHelpers.toTableSpec(tableReference),
tableDescription,
- timePartitioning != null ? BigQueryHelpers.toJsonString(timePartitioning) : null);
+ timePartitioning != null ? BigQueryHelpers.toJsonString(timePartitioning) : null,
+ (String) null);
}
public TableDestination(
@@ -53,25 +56,64 @@
this(
tableSpec,
tableDescription,
- timePartitioning != null ? BigQueryHelpers.toJsonString(timePartitioning) : null);
+ timePartitioning != null ? BigQueryHelpers.toJsonString(timePartitioning) : null,
+ (String) null);
+ }
+
+ public TableDestination(
+ String tableSpec,
+ @Nullable String tableDescription,
+ TimePartitioning timePartitioning,
+ Clustering clustering) {
+ this(
+ tableSpec,
+ tableDescription,
+ timePartitioning != null ? BigQueryHelpers.toJsonString(timePartitioning) : null,
+ clustering != null ? BigQueryHelpers.toJsonString(clustering) : null);
+ }
+
+ public TableDestination(
+ String tableSpec, @Nullable String tableDescription, @Nullable String jsonTimePartitioning) {
+ this(tableSpec, tableDescription, jsonTimePartitioning, (String) null);
}
public TableDestination(
TableReference tableReference,
@Nullable String tableDescription,
@Nullable String jsonTimePartitioning) {
- this(BigQueryHelpers.toTableSpec(tableReference), tableDescription, jsonTimePartitioning);
+ this(
+ BigQueryHelpers.toTableSpec(tableReference),
+ tableDescription,
+ jsonTimePartitioning,
+ (String) null);
}
public TableDestination(
- String tableSpec, @Nullable String tableDescription, @Nullable String jsonTimePartitioning) {
+ TableReference tableReference,
+ @Nullable String tableDescription,
+ @Nullable String jsonTimePartitioning,
+ @Nullable String jsonClustering) {
+ this(
+ BigQueryHelpers.toTableSpec(tableReference),
+ tableDescription,
+ jsonTimePartitioning,
+ jsonClustering);
+ }
+
+ public TableDestination(
+ String tableSpec,
+ @Nullable String tableDescription,
+ @Nullable String jsonTimePartitioning,
+ @Nullable String jsonClustering) {
this.tableSpec = tableSpec;
this.tableDescription = tableDescription;
this.jsonTimePartitioning = jsonTimePartitioning;
+ this.jsonClustering = jsonClustering;
}
public TableDestination withTableReference(TableReference tableReference) {
- return new TableDestination(tableReference, tableDescription, jsonTimePartitioning);
+ return new TableDestination(
+ tableReference, tableDescription, jsonTimePartitioning, jsonClustering);
}
public String getTableSpec() {
@@ -94,6 +136,18 @@
}
}
+ public String getJsonClustering() {
+ return jsonClustering;
+ }
+
+ public Clustering getClustering() {
+ if (jsonClustering == null) {
+ return null;
+ } else {
+ return BigQueryHelpers.fromJsonString(jsonClustering, Clustering.class);
+ }
+ }
+
@Nullable
public String getTableDescription() {
return tableDescription;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV2.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV2.java
index 28202e1..2fee354 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV2.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV2.java
@@ -26,9 +26,10 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
/**
- * A {@link Coder} for {@link TableDestination} that includes time partitioning information. This is
- * a new coder (instead of extending the old {@link TableDestinationCoder}) for compatibility
- * reasons. The old coder is kept around for the same compatibility reasons.
+ * A {@link Coder} for {@link TableDestination} that includes time partitioning information. It is
+ * the default coder for {@link TableDestination} used by {@link BigQueryIO} and does not extend the
+ * old {@link TableDestinationCoder}) for compatibility reasons. The old coder is kept around for
+ * the same compatibility reasons.
*/
public class TableDestinationCoderV2 extends AtomicCoder<TableDestination> {
private static final TableDestinationCoderV2 INSTANCE = new TableDestinationCoderV2();
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV3.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV3.java
new file mode 100644
index 0000000..36d16d2
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV3.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+
+/**
+ * A {@link Coder} for {@link TableDestination} that includes time partitioning and clustering
+ * information. Users must opt in to this version of the coder by setting one of the clustering
+ * options on {@link BigQueryIO.Write}, otherwise {@link TableDestinationCoderV2} will be used and
+ * clustering information will be discarded.
+ */
+public class TableDestinationCoderV3 extends AtomicCoder<TableDestination> {
+ private static final TableDestinationCoderV3 INSTANCE = new TableDestinationCoderV3();
+ private static final Coder<String> timePartitioningCoder = NullableCoder.of(StringUtf8Coder.of());
+ private static final Coder<String> clusteringCoder = NullableCoder.of(StringUtf8Coder.of());
+
+ public static TableDestinationCoderV3 of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(TableDestination value, OutputStream outStream) throws IOException {
+ TableDestinationCoder.of().encode(value, outStream);
+ timePartitioningCoder.encode(value.getJsonTimePartitioning(), outStream);
+ clusteringCoder.encode(value.getJsonClustering(), outStream);
+ }
+
+ @Override
+ public TableDestination decode(InputStream inStream) throws IOException {
+ TableDestination destination = TableDestinationCoder.of().decode(inStream);
+ String jsonTimePartitioning = timePartitioningCoder.decode(inStream);
+ String jsonClustering = clusteringCoder.decode(inStream);
+ return new TableDestination(
+ destination.getTableSpec(),
+ destination.getTableDescription(),
+ jsonTimePartitioning,
+ jsonClustering);
+ }
+
+ @Override
+ public void verifyDeterministic() throws NonDeterministicException {}
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowInfoCoder.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowInfoCoder.java
index f905454..6f382fe 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowInfoCoder.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowInfoCoder.java
@@ -24,7 +24,7 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** Defines a coder for {@link TableRowInfo} objects. */
@VisibleForTesting
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java
index 9953cfe..b02a5ea 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowWriter.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.bigquery.model.TableRow;
import java.io.IOException;
@@ -30,7 +30,7 @@
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.fs.ResourceId;
import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.CountingOutputStream;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.CountingOutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java
index d2de0e1..46f86af 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.ShardedKey;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* Fn that tags each table row with a unique id and destination table. To avoid calling
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java
index 6da8899..4b4a97e 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TestBigQuery.java
@@ -42,7 +42,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matcher;
import org.joda.time.DateTime;
import org.joda.time.Duration;
@@ -189,6 +189,10 @@
table.getTableReference().getTableId());
}
+ public TableReference tableReference() {
+ return table.getTableReference();
+ }
+
/**
* Loads rows from BigQuery into {@link Row Rows} with given {@link Schema}.
*
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java
index 54eec78..0d83938 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.services.bigquery.model.TableRow;
import java.io.IOException;
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* Writes each bundle of {@link TableRow} elements out to separate file using {@link
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java
index 52204f5..0b44827 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java
@@ -26,8 +26,8 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/**
* Partitions temporary files based on number of files and file sizes. Output key is a pair of
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java
index 4baf8fb..9761cf4 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java
@@ -36,9 +36,9 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.java
index 58af43f..b625eb8 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.bigquery.model.TableRow;
import java.util.Map;
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** The result of a {@link BigQueryIO.Write} transform. */
public final class WriteResult implements POutput {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
index 8a20eb7..dbe0962 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
@@ -17,8 +17,9 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import com.google.api.services.bigquery.model.Clustering;
import com.google.api.services.bigquery.model.EncryptionConfiguration;
import com.google.api.services.bigquery.model.JobConfigurationLoad;
import com.google.api.services.bigquery.model.JobReference;
@@ -59,10 +60,10 @@
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -165,6 +166,16 @@
+ "but %s returned null for destination %s",
dynamicDestinations,
destination);
+ boolean destinationCoderSupportsClustering =
+ !(dynamicDestinations.getDestinationCoder() instanceof TableDestinationCoderV2);
+ checkArgument(
+ tableDestination.getClustering() == null || destinationCoderSupportsClustering,
+ "DynamicDestinations.getTable() may only return destinations with clustering configured"
+ + " if a destination coder is supplied that supports clustering, but %s is configured"
+ + " to use TableDestinationCoderV2. Set withClustering() on BigQueryIO.write() and, "
+ + " if you provided a custom DynamicDestinations instance, override"
+ + " getDestinationCoder() to return TableDestinationCoderV3.",
+ dynamicDestinations);
TableReference tableReference = tableDestination.getTableReference();
if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
tableReference.setProjectId(c.getPipelineOptions().as(BigQueryOptions.class).getProject());
@@ -203,6 +214,7 @@
jobIdPrefix,
tableReference,
tableDestination.getTimePartitioning(),
+ tableDestination.getClustering(),
tableSchema,
partitionFiles,
writeDisposition,
@@ -327,6 +339,7 @@
String jobIdPrefix,
TableReference ref,
TimePartitioning timePartitioning,
+ Clustering clustering,
@Nullable TableSchema schema,
List<String> gcsUris,
WriteDisposition writeDisposition,
@@ -342,6 +355,10 @@
.setIgnoreUnknownValues(ignoreUnknownValues);
if (timePartitioning != null) {
loadConfig.setTimePartitioning(timePartitioning);
+ // only set clustering if timePartitioning is set
+ if (clustering != null) {
+ loadConfig.setClustering(clustering);
+ }
}
if (kmsKey != null) {
loadConfig.setDestinationEncryptionConfiguration(
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
index dc38708..42bcb99 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigtable;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import com.google.cloud.bigtable.config.BigtableOptions;
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
/** Configuration for a Cloud Bigtable client. */
@AutoValue
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 306a997..9a17e9c 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.gcp.bigtable;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import com.google.bigtable.v2.Mutation;
@@ -34,6 +34,7 @@
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
import java.util.concurrent.ConcurrentLinkedQueue;
import javax.annotation.Nullable;
@@ -53,15 +54,17 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.ToStringHelper;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.ToStringHelper;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -131,6 +134,33 @@
* .withTableId("table"));
* }</pre>
*
+ * <p>Optionally, BigtableIO.write() may be configured to emit {@link BigtableWriteResult} elements
+ * after each group of inputs is written to Bigtable. These can be used to then trigger user code
+ * after writes have completed. See {@link org.apache.beam.sdk.transforms.Wait} for details on the
+ * windowing requirements of the signal and input PCollections.
+ *
+ * <pre>{@code
+ * // See Wait.on
+ * PCollection<KV<ByteString, Iterable<Mutation>>> data = ...;
+ *
+ * PCollection<BigtableWriteResult> writeResults =
+ * data.apply("write",
+ * BigtableIO.write()
+ * .withProjectId("project")
+ * .withInstanceId("instance")
+ * .withTableId("table"))
+ * .withWriteResults();
+ *
+ * // The windowing of `moreData` must be compatible with `data`, see {@link org.apache.beam.sdk.transforms.Wait#on}
+ * // for details.
+ * PCollection<...> moreData = ...;
+ *
+ * moreData
+ * .apply("wait for writes", Wait.on(writeResults))
+ * .apply("do something", ParDo.of(...))
+ *
+ * }</pre>
+ *
* <h3>Experimental</h3>
*
* <p>This connector for Cloud Bigtable is considered experimental and may break or receive
@@ -660,118 +690,180 @@
return toBuilder().setBigtableConfig(config.withBigtableService(bigtableService)).build();
}
+ /**
+ * Returns a {@link BigtableIO.WriteWithResults} that will emit a {@link BigtableWriteResult}
+ * for each batch of rows written.
+ */
+ @Experimental
+ public WriteWithResults withWriteResults() {
+ return new WriteWithResults(getBigtableConfig());
+ }
+
@Override
public PDone expand(PCollection<KV<ByteString, Iterable<Mutation>>> input) {
- getBigtableConfig().validate();
-
- input.apply(ParDo.of(new BigtableWriterFn(getBigtableConfig())));
+ input.apply(withWriteResults());
return PDone.in(input.getPipeline());
}
@Override
public void validate(PipelineOptions options) {
- validateTableExists(getBigtableConfig(), options);
+ withWriteResults().validate(options);
}
@Override
public void populateDisplayData(DisplayData.Builder builder) {
- super.populateDisplayData(builder);
- getBigtableConfig().populateDisplayData(builder);
+ withWriteResults().populateDisplayData(builder);
}
@Override
public String toString() {
return MoreObjects.toStringHelper(Write.class).add("config", getBigtableConfig()).toString();
}
+ }
- private class BigtableWriterFn extends DoFn<KV<ByteString, Iterable<Mutation>>, Void> {
+ /**
+ * A {@link PTransform} that writes to Google Cloud Bigtable and emits a {@link
+ * BigtableWriteResult} for each batch written. See the class-level Javadoc on {@link BigtableIO}
+ * for more information.
+ *
+ * @see BigtableIO
+ */
+ @Experimental(Experimental.Kind.SOURCE_SINK)
+ public static class WriteWithResults
+ extends PTransform<
+ PCollection<KV<ByteString, Iterable<Mutation>>>, PCollection<BigtableWriteResult>> {
- public BigtableWriterFn(BigtableConfig bigtableConfig) {
- this.config = bigtableConfig;
- this.failures = new ConcurrentLinkedQueue<>();
+ private final BigtableConfig bigtableConfig;
+
+ WriteWithResults(BigtableConfig bigtableConfig) {
+ this.bigtableConfig = bigtableConfig;
+ }
+
+ @Override
+ public PCollection<BigtableWriteResult> expand(
+ PCollection<KV<ByteString, Iterable<Mutation>>> input) {
+ bigtableConfig.validate();
+
+ return input.apply(ParDo.of(new BigtableWriterFn(bigtableConfig)));
+ }
+
+ @Override
+ public void validate(PipelineOptions options) {
+ validateTableExists(bigtableConfig, options);
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ bigtableConfig.populateDisplayData(builder);
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(WriteWithResults.class)
+ .add("config", bigtableConfig)
+ .toString();
+ }
+ }
+
+ private static class BigtableWriterFn
+ extends DoFn<KV<ByteString, Iterable<Mutation>>, BigtableWriteResult> {
+
+ BigtableWriterFn(BigtableConfig bigtableConfig) {
+ this.config = bigtableConfig;
+ this.failures = new ConcurrentLinkedQueue<>();
+ }
+
+ @StartBundle
+ public void startBundle(StartBundleContext c) throws IOException {
+ if (bigtableWriter == null) {
+ bigtableWriter =
+ config
+ .getBigtableService(c.getPipelineOptions())
+ .openForWriting(config.getTableId().get());
+ }
+ recordsWritten = 0;
+ this.seenWindows = Maps.newHashMapWithExpectedSize(1);
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext c, BoundedWindow window) throws Exception {
+ checkForFailures();
+ bigtableWriter
+ .writeRecord(c.element())
+ .whenComplete(
+ (mutationResult, exception) -> {
+ if (exception != null) {
+ failures.add(new BigtableWriteException(c.element(), exception));
+ }
+ });
+ ++recordsWritten;
+ seenWindows.compute(window, (key, count) -> (count != null ? count : 0) + 1);
+ }
+
+ @FinishBundle
+ public void finishBundle(FinishBundleContext c) throws Exception {
+ bigtableWriter.flush();
+ checkForFailures();
+ LOG.debug("Wrote {} records", recordsWritten);
+
+ for (Map.Entry<BoundedWindow, Long> entry : seenWindows.entrySet()) {
+ c.output(
+ BigtableWriteResult.create(entry.getValue()),
+ entry.getKey().maxTimestamp(),
+ entry.getKey());
+ }
+ }
+
+ @Teardown
+ public void tearDown() throws Exception {
+ if (bigtableWriter != null) {
+ bigtableWriter.close();
+ bigtableWriter = null;
+ }
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ config.populateDisplayData(builder);
+ }
+
+ ///////////////////////////////////////////////////////////////////////////////
+ private final BigtableConfig config;
+ private BigtableService.Writer bigtableWriter;
+ private long recordsWritten;
+ private final ConcurrentLinkedQueue<BigtableWriteException> failures;
+ private Map<BoundedWindow, Long> seenWindows;
+
+ /** If any write has asynchronously failed, fail the bundle with a useful error. */
+ private void checkForFailures() throws IOException {
+ // Note that this function is never called by multiple threads and is the only place that
+ // we remove from failures, so this code is safe.
+ if (failures.isEmpty()) {
+ return;
}
- @StartBundle
- public void startBundle(StartBundleContext c) throws IOException {
- if (bigtableWriter == null) {
- bigtableWriter =
- config
- .getBigtableService(c.getPipelineOptions())
- .openForWriting(config.getTableId().get());
+ StringBuilder logEntry = new StringBuilder();
+ int i = 0;
+ List<BigtableWriteException> suppressed = Lists.newArrayList();
+ for (; i < 10 && !failures.isEmpty(); ++i) {
+ BigtableWriteException exc = failures.remove();
+ logEntry.append("\n").append(exc.getMessage());
+ if (exc.getCause() != null) {
+ logEntry.append(": ").append(exc.getCause().getMessage());
}
- recordsWritten = 0;
+ suppressed.add(exc);
}
-
- @ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- checkForFailures();
- bigtableWriter
- .writeRecord(c.element())
- .whenComplete(
- (mutationResult, exception) -> {
- if (exception != null) {
- failures.add(new BigtableWriteException(c.element(), exception));
- }
- });
- ++recordsWritten;
+ String message =
+ String.format(
+ "At least %d errors occurred writing to Bigtable. First %d errors: %s",
+ i + failures.size(), i, logEntry.toString());
+ LOG.error(message);
+ IOException exception = new IOException(message);
+ for (BigtableWriteException e : suppressed) {
+ exception.addSuppressed(e);
}
-
- @FinishBundle
- public void finishBundle() throws Exception {
- bigtableWriter.flush();
- checkForFailures();
- LOG.debug("Wrote {} records", recordsWritten);
- }
-
- @Teardown
- public void tearDown() throws Exception {
- if (bigtableWriter != null) {
- bigtableWriter.close();
- bigtableWriter = null;
- }
- }
-
- @Override
- public void populateDisplayData(DisplayData.Builder builder) {
- builder.delegate(Write.this);
- }
-
- ///////////////////////////////////////////////////////////////////////////////
- private final BigtableConfig config;
- private BigtableService.Writer bigtableWriter;
- private long recordsWritten;
- private final ConcurrentLinkedQueue<BigtableWriteException> failures;
-
- /** If any write has asynchronously failed, fail the bundle with a useful error. */
- private void checkForFailures() throws IOException {
- // Note that this function is never called by multiple threads and is the only place that
- // we remove from failures, so this code is safe.
- if (failures.isEmpty()) {
- return;
- }
-
- StringBuilder logEntry = new StringBuilder();
- int i = 0;
- List<BigtableWriteException> suppressed = Lists.newArrayList();
- for (; i < 10 && !failures.isEmpty(); ++i) {
- BigtableWriteException exc = failures.remove();
- logEntry.append("\n").append(exc.getMessage());
- if (exc.getCause() != null) {
- logEntry.append(": ").append(exc.getCause().getMessage());
- }
- suppressed.add(exc);
- }
- String message =
- String.format(
- "At least %d errors occurred writing to Bigtable. First %d errors: %s",
- i + failures.size(), i, logEntry.toString());
- LOG.error(message);
- IOException exception = new IOException(message);
- for (BigtableWriteException e : suppressed) {
- exception.addSuppressed(e);
- }
- throw exception;
- }
+ throw exception;
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
index 767faff..9cd47aa 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.sdk.io.gcp.bigtable;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.MoreExecutors.directExecutor;
+
import com.google.bigtable.admin.v2.GetTableRequest;
import com.google.bigtable.v2.MutateRowResponse;
import com.google.bigtable.v2.MutateRowsRequest;
@@ -43,11 +45,11 @@
import org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.BigtableSource;
import org.apache.beam.sdk.io.range.ByteKeyRange;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Closer;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.FutureCallback;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Futures;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Closer;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.FutureCallback;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Futures;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -239,7 +241,8 @@
public void onFailure(Throwable throwable) {
result.completeExceptionally(throwable);
}
- });
+ },
+ directExecutor());
return result;
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResult.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResult.java
new file mode 100644
index 0000000..d1a6bfd
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResult.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigtable;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.coders.DefaultCoder;
+
+/**
+ * The result of writing a batch of rows to Bigtable. Rows are written to bigtable in batches (based
+ * on the runner-chosen bundle size). Once each batch finishes, a single {@link BigtableWriteResult}
+ * is emitted.
+ */
+@DefaultCoder(BigtableWriteResultCoder.class)
+@AutoValue
+public abstract class BigtableWriteResult {
+ public static BigtableWriteResult create(long rowsWritten) {
+ return new AutoValue_BigtableWriteResult(rowsWritten);
+ }
+
+ /** The number of rows written in this batch. */
+ public abstract long getRowsWritten();
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResultCoder.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResultCoder.java
new file mode 100644
index 0000000..57949b4
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResultCoder.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigtable;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import org.apache.beam.sdk.coders.AtomicCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CoderProvider;
+import org.apache.beam.sdk.coders.CoderProviders;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/** A coder for {@link BigtableWriteResult}. */
+public class BigtableWriteResultCoder extends AtomicCoder<BigtableWriteResult> {
+ private static final Coder<Long> LONG_CODER = VarLongCoder.of();
+ private static final BigtableWriteResultCoder INSTANCE = new BigtableWriteResultCoder();
+
+ public static CoderProvider getCoderProvider() {
+ return CoderProviders.forCoder(
+ TypeDescriptor.of(BigtableWriteResult.class), BigtableWriteResultCoder.of());
+ }
+
+ public static BigtableWriteResultCoder of() {
+ return INSTANCE;
+ }
+
+ @Override
+ public void encode(BigtableWriteResult value, OutputStream outStream)
+ throws CoderException, IOException {
+ LONG_CODER.encode(value.getRowsWritten(), outStream);
+ }
+
+ @Override
+ public BigtableWriteResult decode(InputStream inStream) throws CoderException, IOException {
+ return BigtableWriteResult.create(LONG_CODER.decode(inStream));
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/VendoredListenableFutureAdapter.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/VendoredListenableFutureAdapter.java
index c736083..d7e0ae4 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/VendoredListenableFutureAdapter.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/VendoredListenableFutureAdapter.java
@@ -21,7 +21,7 @@
import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListenableFuture;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ListenableFuture;
/** Adapts {@link ListenableFuture} from bigtable-client-core to vendored guava. */
class VendoredListenableFutureAdapter<V> implements ListenableFuture<V> {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/common/GcpIoPipelineOptionsRegistrar.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/common/GcpIoPipelineOptionsRegistrar.java
index aff2676..90c0310 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/common/GcpIoPipelineOptionsRegistrar.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/common/GcpIoPipelineOptionsRegistrar.java
@@ -22,7 +22,7 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubOptions;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A registrar containing the default GCP options. */
@AutoService(PipelineOptionsRegistrar.class)
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/AdaptiveThrottler.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/AdaptiveThrottler.java
index 4c50b3a..c722512 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/AdaptiveThrottler.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/AdaptiveThrottler.java
@@ -20,7 +20,7 @@
import java.util.Random;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.util.MovingFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* An implementation of client-side adaptive throttling. See
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index d35fc74..acc40e1 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -26,9 +26,9 @@
import static com.google.datastore.v1.client.DatastoreHelper.makeOrder;
import static com.google.datastore.v1.client.DatastoreHelper.makeUpsert;
import static com.google.datastore.v1.client.DatastoreHelper.makeValue;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verify;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verify;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.auth.Credentials;
@@ -91,11 +91,11 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
index 8a6235c..07d6da6 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.DateTime;
import java.io.Closeable;
@@ -29,9 +29,9 @@
import java.util.Map;
import java.util.concurrent.ThreadLocalRandom;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
/** An (abstract) helper class for talking to Pubsub via an underlying transport. */
public abstract class PubsubClient implements Closeable {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubCoderProviderRegistrar.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubCoderProviderRegistrar.java
index 6ed6ed3..c10c60d 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubCoderProviderRegistrar.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubCoderProviderRegistrar.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.coders.CoderProviderRegistrar;
import org.apache.beam.sdk.coders.CoderProviders;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A {@link CoderProviderRegistrar} for standard types used with {@link PubsubIO}. */
@AutoService(CoderProviderRegistrar.class)
@@ -32,6 +32,11 @@
public List<CoderProvider> getCoderProviders() {
return ImmutableList.of(
CoderProviders.forCoder(
- TypeDescriptor.of(PubsubMessage.class), PubsubMessageWithAttributesCoder.of()));
+ TypeDescriptor.of(PubsubMessage.class), PubsubMessageWithAttributesCoder.of()),
+ CoderProviders.forCoder(
+ TypeDescriptor.of(PubsubMessage.class), PubsubMessageWithMessageIdCoder.of()),
+ CoderProviders.forCoder(
+ TypeDescriptor.of(PubsubMessage.class),
+ PubsubMessageWithAttributesAndMessageIdCoder.of()));
}
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java
index 8417f9f..d15dadb 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auth.Credentials;
import com.google.protobuf.ByteString;
@@ -58,9 +58,9 @@
import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A helper class for talking to Pubsub via grpc.
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
index 2fbbfc9..4f745ea 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.Clock;
import com.google.auto.value.AutoValue;
@@ -28,6 +28,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.annotation.Nullable;
@@ -64,9 +65,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -440,6 +441,7 @@
private static <T> Read<T> read() {
return new AutoValue_PubsubIO_Read.Builder<T>()
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.build();
}
@@ -455,6 +457,23 @@
.setCoder(PubsubMessagePayloadOnlyCoder.of())
.setParseFn(new IdentityMessageFn())
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
+ .build();
+ }
+
+ /**
+ * Returns A {@link PTransform} that continuously reads from a Google Cloud Pub/Sub stream. The
+ * messages will only contain a {@link PubsubMessage#getPayload() payload} with the {@link
+ * PubsubMessage#getMessageId() messageId} from PubSub, but no {@link
+ * PubsubMessage#getAttributeMap() attributes}.
+ */
+ public static Read<PubsubMessage> readMessagesWithMessageId() {
+ return new AutoValue_PubsubIO_Read.Builder<PubsubMessage>()
+ .setPubsubClientFactory(FACTORY)
+ .setCoder(PubsubMessageWithMessageIdCoder.of())
+ .setParseFn(new IdentityMessageFn())
+ .setNeedsAttributes(false)
+ .setNeedsMessageId(true)
.build();
}
@@ -469,6 +488,23 @@
.setCoder(PubsubMessageWithAttributesCoder.of())
.setParseFn(new IdentityMessageFn())
.setNeedsAttributes(true)
+ .setNeedsMessageId(false)
+ .build();
+ }
+
+ /**
+ * Returns A {@link PTransform} that continuously reads from a Google Cloud Pub/Sub stream. The
+ * messages will contain both a {@link PubsubMessage#getPayload() payload} and {@link
+ * PubsubMessage#getAttributeMap() attributes}, along with the {@link PubsubMessage#getMessageId()
+ * messageId} from PubSub.
+ */
+ public static Read<PubsubMessage> readMessagesWithAttributesAndMessageId() {
+ return new AutoValue_PubsubIO_Read.Builder<PubsubMessage>()
+ .setPubsubClientFactory(FACTORY)
+ .setCoder(PubsubMessageWithAttributesAndMessageIdCoder.of())
+ .setParseFn(new IdentityMessageFn())
+ .setNeedsAttributes(true)
+ .setNeedsMessageId(true)
.build();
}
@@ -479,6 +515,7 @@
public static Read<String> readStrings() {
return new AutoValue_PubsubIO_Read.Builder<String>()
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setCoder(StringUtf8Coder.of())
.setParseFn(new ParsePayloadAsUtf8())
@@ -496,6 +533,7 @@
ProtoCoder<T> coder = ProtoCoder.of(messageClass);
return new AutoValue_PubsubIO_Read.Builder<T>()
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setCoder(coder)
.setParseFn(new ParsePayloadUsingCoder<>(coder))
@@ -513,6 +551,7 @@
AvroCoder<T> coder = AvroCoder.of(clazz);
return new AutoValue_PubsubIO_Read.Builder<T>()
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setCoder(coder)
.setParseFn(new ParsePayloadUsingCoder<>(coder))
@@ -520,6 +559,21 @@
}
/**
+ * Returns A {@link PTransform} that continuously reads from a Google Cloud Pub/Sub stream,
+ * mapping each {@link PubsubMessage} into type T using the supplied parse function and coder.
+ */
+ public static <T> Read<T> readMessagesWithCoderAndParseFn(
+ Coder<T> coder, SimpleFunction<PubsubMessage, T> parseFn) {
+ return new AutoValue_PubsubIO_Read.Builder<T>()
+ .setNeedsAttributes(false)
+ .setNeedsMessageId(false)
+ .setPubsubClientFactory(FACTORY)
+ .setCoder(coder)
+ .setParseFn(parseFn)
+ .build();
+ }
+
+ /**
* Returns a {@link PTransform} that continuously reads binary encoded Avro messages into the Avro
* {@link GenericRecord} type.
*
@@ -532,6 +586,7 @@
AvroCoder<GenericRecord> coder = AvroCoder.of(GenericRecord.class, avroSchema);
return new AutoValue_PubsubIO_Read.Builder<GenericRecord>()
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setBeamSchema(schema)
.setToRowFn(AvroUtils.getToRowFunction(GenericRecord.class, avroSchema))
@@ -557,6 +612,7 @@
Schema schema = AvroUtils.getSchema(clazz, null);
return new AutoValue_PubsubIO_Read.Builder<T>()
.setNeedsAttributes(false)
+ .setNeedsMessageId(false)
.setPubsubClientFactory(FACTORY)
.setBeamSchema(schema)
.setToRowFn(AvroUtils.getToRowFunction(clazz, avroSchema))
@@ -609,6 +665,9 @@
abstract ValueProvider<PubsubTopic> getTopicProvider();
@Nullable
+ abstract PubsubClient.PubsubClientFactory getPubsubClientFactory();
+
+ @Nullable
abstract ValueProvider<PubsubSubscription> getSubscriptionProvider();
/** The name of the message attribute to read timestamps from. */
@@ -636,19 +695,21 @@
@Nullable
abstract SerializableFunction<Row, T> getFromRowFn();
- abstract PubsubClient.PubsubClientFactory getPubsubClientFactory();
-
@Nullable
abstract Clock getClock();
abstract boolean getNeedsAttributes();
+ abstract boolean getNeedsMessageId();
+
abstract Builder<T> toBuilder();
@AutoValue.Builder
abstract static class Builder<T> {
abstract Builder<T> setTopicProvider(ValueProvider<PubsubTopic> topic);
+ abstract Builder<T> setPubsubClientFactory(PubsubClient.PubsubClientFactory clientFactory);
+
abstract Builder<T> setSubscriptionProvider(ValueProvider<PubsubSubscription> subscription);
abstract Builder<T> setTimestampAttribute(String timestampAttribute);
@@ -667,7 +728,7 @@
abstract Builder<T> setNeedsAttributes(boolean needsAttributes);
- abstract Builder<T> setPubsubClientFactory(PubsubClient.PubsubClientFactory clientFactory);
+ abstract Builder<T> setNeedsMessageId(boolean needsMessageId);
abstract Builder<T> setClock(@Nullable Clock clock);
@@ -726,6 +787,16 @@
}
/**
+ * The default client to write to Pub/Sub is the {@link PubsubJsonClient}, created by the {@link
+ * PubsubJsonClient.PubsubJsonClientFactory}. This function allows to change the Pub/Sub client
+ * by providing another {@link PubsubClient.PubsubClientFactory} like the {@link
+ * PubsubGrpcClientFactory}.
+ */
+ public Read<T> withClientFactory(PubsubClient.PubsubClientFactory factory) {
+ return toBuilder().setPubsubClientFactory(factory).build();
+ }
+
+ /**
* When reading from Cloud Pub/Sub where record timestamps are provided as Pub/Sub message
* attributes, specifies the name of the attribute that contains the timestamp.
*
@@ -776,22 +847,12 @@
* output type T must be registered or set on the output via {@link
* PCollection#setCoder(Coder)}.
*/
- private Read<T> withCoderAndParseFn(Coder<T> coder, SimpleFunction<PubsubMessage, T> parseFn) {
+ public Read<T> withCoderAndParseFn(Coder<T> coder, SimpleFunction<PubsubMessage, T> parseFn) {
return toBuilder().setCoder(coder).setParseFn(parseFn).build();
}
@VisibleForTesting
/**
- * Set's the PubsubClientFactory.
- *
- * <p>Only for use by unit tests.
- */
- Read<T> withClientFactory(PubsubClient.PubsubClientFactory clientFactory) {
- return toBuilder().setPubsubClientFactory(clientFactory).build();
- }
-
- @VisibleForTesting
- /**
* Set's the internal Clock.
*
* <p>Only for use by unit tests.
@@ -824,13 +885,14 @@
PubsubUnboundedSource source =
new PubsubUnboundedSource(
getClock(),
- getPubsubClientFactory(),
+ Optional.ofNullable(getPubsubClientFactory()).orElse(FACTORY),
null /* always get project from runtime PipelineOptions */,
topicPath,
subscriptionPath,
getTimestampAttribute(),
getIdAttribute(),
- getNeedsAttributes());
+ getNeedsAttributes(),
+ getNeedsMessageId());
PCollection<T> read = input.apply(source).apply(MapElements.via(getParseFn()));
return (getBeamSchema() != null)
? read.setSchema(getBeamSchema(), getToRowFn(), getFromRowFn())
@@ -856,12 +918,20 @@
/** Implementation of {@link #write}. */
@AutoValue
public abstract static class Write<T> extends PTransform<PCollection<T>, PDone> {
- private static final int MAX_PUBLISH_BATCH_BYTE_SIZE_DEFAULT = 10 * 1024 * 1024;
+ /**
+ * Max batch byte size. Messages are base64 encoded which encodes each set of three bytes into
+ * four bytes.
+ */
+ private static final int MAX_PUBLISH_BATCH_BYTE_SIZE_DEFAULT = ((10 * 1024 * 1024) / 4) * 3;
+
private static final int MAX_PUBLISH_BATCH_SIZE = 100;
@Nullable
abstract ValueProvider<PubsubTopic> getTopicProvider();
+ @Nullable
+ abstract PubsubClient.PubsubClientFactory getPubsubClientFactory();
+
/** the batch size for bulk submissions to pubsub. */
@Nullable
abstract Integer getMaxBatchSize();
@@ -888,6 +958,8 @@
abstract static class Builder<T> {
abstract Builder<T> setTopicProvider(ValueProvider<PubsubTopic> topicProvider);
+ abstract Builder<T> setPubsubClientFactory(PubsubClient.PubsubClientFactory factory);
+
abstract Builder<T> setMaxBatchSize(Integer batchSize);
abstract Builder<T> setMaxBatchBytesSize(Integer maxBatchBytesSize);
@@ -919,6 +991,16 @@
}
/**
+ * The default client to write to Pub/Sub is the {@link PubsubJsonClient}, created by the {@link
+ * PubsubJsonClient.PubsubJsonClientFactory}. This function allows to change the Pub/Sub client
+ * by providing another {@link PubsubClient.PubsubClientFactory} like the {@link
+ * PubsubGrpcClientFactory}.
+ */
+ public Write<T> withClientFactory(PubsubClient.PubsubClientFactory factory) {
+ return toBuilder().setPubsubClientFactory(factory).build();
+ }
+
+ /**
* Writes to Pub/Sub are batched to efficiently send data. The value of the attribute will be a
* number representing the number of Pub/Sub messages to queue before sending off the bulk
* request. For example, if given 1000 the write sink will wait until 1000 messages have been
@@ -996,7 +1078,7 @@
.apply(MapElements.via(getFormatFn()))
.apply(
new PubsubUnboundedSink(
- FACTORY,
+ Optional.ofNullable(getPubsubClientFactory()).orElse(FACTORY),
NestedValueProvider.of(getTopicProvider(), new TopicPathTranslator()),
getTimestampAttribute(),
getIdAttribute(),
@@ -1046,8 +1128,10 @@
// NOTE: idAttribute is ignored.
this.pubsubClient =
- FACTORY.newClient(
- getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
+ Optional.ofNullable(getPubsubClientFactory())
+ .orElse(FACTORY)
+ .newClient(
+ getTimestampAttribute(), null, c.getPipelineOptions().as(PubsubOptions.class));
}
@ProcessElement
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java
index 5e423fb..11cb0d6 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.services.pubsub.Pubsub;
@@ -47,9 +47,9 @@
import javax.annotation.Nullable;
import org.apache.beam.sdk.extensions.gcp.util.RetryHttpRequestInitializer;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** A Pubsub client using JSON transport. */
public class PubsubJsonClient extends PubsubClient {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java
index db6a92a..b437c0a 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessage.java
@@ -17,23 +17,31 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.util.Map;
import javax.annotation.Nullable;
/**
- * Class representing a Pub/Sub message. Each message contains a single message payload and a map of
- * attached attributes.
+ * Class representing a Pub/Sub message. Each message contains a single message payload, a map of
+ * attached attributes, and a message id.
*/
public class PubsubMessage {
private byte[] message;
private Map<String, String> attributes;
+ private String messageId;
public PubsubMessage(byte[] payload, Map<String, String> attributes) {
this.message = payload;
this.attributes = attributes;
+ this.messageId = null;
+ }
+
+ public PubsubMessage(byte[] payload, Map<String, String> attributes, String messageId) {
+ this.message = payload;
+ this.attributes = attributes;
+ this.messageId = messageId;
}
/** Returns the main PubSub message. */
@@ -52,4 +60,10 @@
public Map<String, String> getAttributeMap() {
return attributes;
}
+
+ /** Returns the messageId of the message populated by Cloud Pub/Sub. */
+ @Nullable
+ public String getMessageId() {
+ return messageId;
+ }
}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoder.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoder.java
index 49bfc05..74494fd 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoder.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoder.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CustomCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** A coder for PubsubMessage treating the raw bytes being decoded as the message's payload. */
public class PubsubMessagePayloadOnlyCoder extends CustomCoder<PubsubMessage> {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoder.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoder.java
new file mode 100644
index 0000000..377bfc6
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoder.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Map;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.NullableCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.values.TypeDescriptor;
+
+/** A coder for PubsubMessage including attributes and the message id from the PubSub server. */
+public class PubsubMessageWithAttributesAndMessageIdCoder extends CustomCoder<PubsubMessage> {
+ // A message's payload cannot be null
+ private static final Coder<byte[]> PAYLOAD_CODER = ByteArrayCoder.of();
+ // A message's attributes can be null.
+ private static final Coder<Map<String, String>> ATTRIBUTES_CODER =
+ NullableCoder.of(MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()));
+ // A message's messageId cannot be null
+ private static final Coder<String> MESSAGE_ID_CODER = StringUtf8Coder.of();
+
+ public static Coder<PubsubMessage> of(TypeDescriptor<PubsubMessage> ignored) {
+ return of();
+ }
+
+ public static PubsubMessageWithAttributesAndMessageIdCoder of() {
+ return new PubsubMessageWithAttributesAndMessageIdCoder();
+ }
+
+ @Override
+ public void encode(PubsubMessage value, OutputStream outStream) throws IOException {
+ PAYLOAD_CODER.encode(value.getPayload(), outStream);
+ ATTRIBUTES_CODER.encode(value.getAttributeMap(), outStream);
+ MESSAGE_ID_CODER.encode(value.getMessageId(), outStream);
+ }
+
+ @Override
+ public PubsubMessage decode(InputStream inStream) throws IOException {
+ byte[] payload = PAYLOAD_CODER.decode(inStream);
+ Map<String, String> attributes = ATTRIBUTES_CODER.decode(inStream);
+ String messageId = MESSAGE_ID_CODER.decode(inStream);
+ return new PubsubMessage(payload, attributes, messageId);
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoder.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoder.java
new file mode 100644
index 0000000..f38e14d
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoder.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/**
+ * A coder for PubsubMessage treating the raw bytes being decoded as the message's payload, with the
+ * message id from the PubSub server.
+ */
+public class PubsubMessageWithMessageIdCoder extends CustomCoder<PubsubMessage> {
+ private static final Coder<byte[]> PAYLOAD_CODER = ByteArrayCoder.of();
+ // A message's messageId cannot be null
+ private static final Coder<String> MESSAGE_ID_CODER = StringUtf8Coder.of();
+
+ public static PubsubMessageWithMessageIdCoder of() {
+ return new PubsubMessageWithMessageIdCoder();
+ }
+
+ @Override
+ public void encode(PubsubMessage value, OutputStream outStream) throws IOException {
+ PAYLOAD_CODER.encode(value.getPayload(), outStream);
+ MESSAGE_ID_CODER.encode(value.getMessageId(), outStream);
+ }
+
+ @Override
+ public PubsubMessage decode(InputStream inStream) throws IOException {
+ byte[] payload = PAYLOAD_CODER.decode(inStream);
+ String messageId = MESSAGE_ID_CODER.decode(inStream);
+ return new PubsubMessage(payload, ImmutableMap.of(), messageId);
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
index 8ee5ed9..6b20b56 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.Clock;
import java.io.Closeable;
@@ -30,15 +30,17 @@
import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
/**
* A (partial) implementation of {@link PubsubClient} for use by unit tests. Only suitable for
* testing {@link #publish}, {@link #pull}, {@link #acknowledge} and {@link #modifyAckDeadline}
* methods. Relies on statics to mimic the Pubsub service, though we try to hide that.
*/
-class PubsubTestClient extends PubsubClient implements Serializable {
+@Experimental
+public class PubsubTestClient extends PubsubClient implements Serializable {
/**
* Mimic the state of the simulated Pubsub 'service'.
*
@@ -94,7 +96,7 @@
* Return a factory for testing publishers. Only one factory may be in-flight at a time. The
* factory must be closed when the test is complete, at which point final validation will occur.
*/
- static PubsubTestClientFactory createFactoryForPublish(
+ public static PubsubTestClientFactory createFactoryForPublish(
final TopicPath expectedTopic,
final Iterable<OutgoingMessage> expectedOutgoingMessages,
final Iterable<OutgoingMessage> failingOutgoingMessages) {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSink.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSink.java
index abccd2c..1258d0b 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSink.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSink.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
@@ -60,8 +60,8 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
index 9dc446f..d8abfe1 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.util.Clock;
import java.io.IOException;
@@ -71,9 +71,9 @@
import org.apache.beam.sdk.util.MovingFunction;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -826,9 +826,9 @@
}
/**
- * BLOCKING Return {@literal true} if a Pubsub messaage is available, {@literal false} if none
- * is available at this time or we are over-subscribed. May BLOCK while extending ACKs or
- * fetching available messages. Will not block waiting for messages.
+ * BLOCKING Return {@literal true} if a Pubsub message is available, {@literal false} if none is
+ * available at this time or we are over-subscribed. May BLOCK while extending ACKs or fetching
+ * available messages. Will not block waiting for messages.
*/
@Override
public boolean advance() throws IOException {
@@ -884,7 +884,7 @@
if (current == null) {
throw new NoSuchElementException();
}
- return new PubsubMessage(current.elementBytes, current.attributes);
+ return new PubsubMessage(current.elementBytes, current.attributes, current.recordId);
}
@Override
@@ -1083,9 +1083,15 @@
@Override
public Coder<PubsubMessage> getOutputCoder() {
- return outer.getNeedsAttributes()
- ? PubsubMessageWithAttributesCoder.of()
- : PubsubMessagePayloadOnlyCoder.of();
+ if (outer.getNeedsMessageId()) {
+ return outer.getNeedsAttributes()
+ ? PubsubMessageWithAttributesAndMessageIdCoder.of()
+ : PubsubMessageWithMessageIdCoder.of();
+ } else {
+ return outer.getNeedsAttributes()
+ ? PubsubMessageWithAttributesCoder.of()
+ : PubsubMessagePayloadOnlyCoder.of();
+ }
}
@Override
@@ -1188,6 +1194,9 @@
/** Whether this source should load the attributes of the PubsubMessage, or only the payload. */
private final boolean needsAttributes;
+ /** Whether this source should include the messageId from PubSub. */
+ private final boolean needsMessageId;
+
@VisibleForTesting
PubsubUnboundedSource(
Clock clock,
@@ -1197,7 +1206,8 @@
@Nullable ValueProvider<SubscriptionPath> subscription,
@Nullable String timestampAttribute,
@Nullable String idAttribute,
- boolean needsAttributes) {
+ boolean needsAttributes,
+ boolean needsMessageId) {
checkArgument(
(topic == null) != (subscription == null),
"Exactly one of topic and subscription must be given");
@@ -1209,6 +1219,7 @@
this.timestampAttribute = timestampAttribute;
this.idAttribute = idAttribute;
this.needsAttributes = needsAttributes;
+ this.needsMessageId = needsMessageId;
}
/** Construct an unbounded source to consume from the Pubsub {@code subscription}. */
@@ -1228,7 +1239,52 @@
subscription,
timestampAttribute,
idAttribute,
- needsAttributes);
+ needsAttributes,
+ false);
+ }
+
+ /** Construct an unbounded source to consume from the Pubsub {@code subscription}. */
+ public PubsubUnboundedSource(
+ Clock clock,
+ PubsubClientFactory pubsubFactory,
+ @Nullable ValueProvider<ProjectPath> project,
+ @Nullable ValueProvider<TopicPath> topic,
+ @Nullable ValueProvider<SubscriptionPath> subscription,
+ @Nullable String timestampAttribute,
+ @Nullable String idAttribute,
+ boolean needsAttributes) {
+ this(
+ clock,
+ pubsubFactory,
+ project,
+ topic,
+ subscription,
+ timestampAttribute,
+ idAttribute,
+ needsAttributes,
+ false);
+ }
+
+ /** Construct an unbounded source to consume from the Pubsub {@code subscription}. */
+ public PubsubUnboundedSource(
+ PubsubClientFactory pubsubFactory,
+ @Nullable ValueProvider<ProjectPath> project,
+ @Nullable ValueProvider<TopicPath> topic,
+ @Nullable ValueProvider<SubscriptionPath> subscription,
+ @Nullable String timestampAttribute,
+ @Nullable String idAttribute,
+ boolean needsAttributes,
+ boolean needsMessageId) {
+ this(
+ null,
+ pubsubFactory,
+ project,
+ topic,
+ subscription,
+ timestampAttribute,
+ idAttribute,
+ needsAttributes,
+ needsMessageId);
}
/** Get the project path. */
@@ -1277,6 +1333,10 @@
return needsAttributes;
}
+ public boolean getNeedsMessageId() {
+ return needsMessageId;
+ }
+
@Override
public PCollection<PubsubMessage> expand(PBegin input) {
return input
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
index 28c1ab1..f4f8b18 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
@@ -20,7 +20,7 @@
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.util.stream.Collectors.toList;
import static org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.createTopicName;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import io.grpc.Status;
import io.grpc.StatusRuntimeException;
@@ -50,9 +50,9 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Suppliers;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.joda.time.DateTime;
import org.joda.time.Duration;
import org.junit.rules.TestRule;
@@ -336,7 +336,7 @@
* Stateful {@link DoFn} which caches the elements it sees and checks whether they satisfy the
* predicate.
*
- * <p>When predicate is satisfied outputs "SUCCESS". If predicate throws execption, outputs
+ * <p>When predicate is satisfied outputs "SUCCESS". If predicate throws exception, outputs
* "FAILURE".
*/
static class StatefulPredicateCheck<T> extends DoFn<KV<String, ? extends T>, String> {
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/BatchSpannerRead.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/BatchSpannerRead.java
index e08822e..218d1a4 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/BatchSpannerRead.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/BatchSpannerRead.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.Reshuffle;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/**
* This transform reads from Cloud Spanner using the {@link com.google.cloud.spanner.BatchClient}.
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationCellCounter.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationCellCounter.java
index 3828b54..0ac7bc9 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationCellCounter.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationCellCounter.java
@@ -22,7 +22,7 @@
import com.google.cloud.spanner.KeySet;
import com.google.cloud.spanner.Mutation;
import com.google.cloud.spanner.Mutation.Op;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
final class MutationCellCounter {
// Prevent construction.
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroup.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroup.java
index 44ae704..6d63e56 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroup.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroup.java
@@ -22,8 +22,8 @@
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* A bundle of mutations that must be submitted atomically.
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
index e6efcca..4edbfa6 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.io.gcp.spanner;
import com.google.cloud.spanner.Mutation;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
final class MutationUtils {
private MutationUtils() {}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/NaiveSpannerRead.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/NaiveSpannerRead.java
index 73ea23f..5d4583f 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/NaiveSpannerRead.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/NaiveSpannerRead.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** A naive version of Spanner read that doesn't use the Batch API. */
@VisibleForTesting
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java
index ee94ed5..59e581b 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.io.gcp.spanner;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.math.RoundingMode;
import java.util.ArrayList;
import java.util.Arrays;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.math.LongMath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Longs;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedInteger;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.math.LongMath;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Longs;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedInteger;
/**
* This module provides routines for encoding a sequence of typed entities into a byte array. The
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java
index 834a22e..a668bf3 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerConfig.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.spanner;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.api.gax.rpc.FixedHeaderProvider;
import com.google.auto.value.AutoValue;
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** Configuration for a Cloud Spanner client. */
@AutoValue
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
index d2369b8..a46853c 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
@@ -20,8 +20,8 @@
import static org.apache.beam.sdk.io.gcp.spanner.MutationUtils.isPointDelete;
import static org.apache.beam.sdk.io.gcp.spanner.SpannerIO.WriteGrouped.decode;
import static org.apache.beam.sdk.io.gcp.spanner.SpannerIO.WriteGrouped.encode;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import com.google.cloud.ServiceFactory;
@@ -70,10 +70,10 @@
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java
index 075ecc9..e589c2b 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java
@@ -21,12 +21,12 @@
import com.google.cloud.spanner.Type;
import java.io.Serializable;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
/** Encapsulates Cloud Spanner Schema. */
@AutoValue
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteResult.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteResult.java
index 77587fe..d387aab 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteResult.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteResult.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.values.POutput;
import org.apache.beam.sdk.values.PValue;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* The results of a {@link SpannerIO#write()} transform.
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
index f86d704..8389ca5 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryClient.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.gcp.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
@@ -63,8 +63,8 @@
import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -164,7 +164,13 @@
@Nonnull
public QueryResponse queryWithRetries(String query, String projectId)
throws IOException, InterruptedException {
- return queryWithRetries(query, projectId, false);
+ return queryWithRetries(query, projectId, false, false);
+ }
+
+ @Nonnull
+ public QueryResponse queryWithRetriesUsingStandardSql(String query, String projectId)
+ throws IOException, InterruptedException {
+ return queryWithRetries(query, projectId, false, true);
}
@Nullable
@@ -328,10 +334,21 @@
@Nonnull
public QueryResponse queryWithRetries(String query, String projectId, boolean typed)
throws IOException, InterruptedException {
+ return queryWithRetries(query, projectId, typed, false);
+ }
+
+ @Nonnull
+ private QueryResponse queryWithRetries(
+ String query, String projectId, boolean typed, boolean useStandardSql)
+ throws IOException, InterruptedException {
Sleeper sleeper = Sleeper.DEFAULT;
BackOff backoff = BackOffAdapter.toGcpBackOff(BACKOFF_FACTORY.backoff());
IOException lastException = null;
- QueryRequest bqQueryRequest = new QueryRequest().setQuery(query).setTimeoutMs(QUERY_TIMEOUT_MS);
+ QueryRequest bqQueryRequest =
+ new QueryRequest()
+ .setQuery(query)
+ .setTimeoutMs(QUERY_TIMEOUT_MS)
+ .setUseLegacySql(!useStandardSql);
do {
if (lastException != null) {
LOG.warn("Retrying query ({}) after exception", bqQueryRequest.getQuery(), lastException);
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
index 6fdcebc..5312e23 100644
--- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.testing;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.api.services.bigquery.model.QueryResponse;
import com.google.api.services.bigquery.model.TableCell;
@@ -33,10 +33,10 @@
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.testing.SerializableMatcher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashCode;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashCode;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.hamcrest.Description;
import org.hamcrest.TypeSafeMatcher;
import org.slf4j.Logger;
@@ -63,6 +63,7 @@
private final String projectId;
private final String query;
+ private final boolean usingStandardSql;
private final String expectedChecksum;
private String actualChecksum;
private transient QueryResponse response;
@@ -70,6 +71,15 @@
public BigqueryMatcher(
String applicationName, String projectId, String query, String expectedChecksum) {
+ this(applicationName, projectId, query, false, expectedChecksum);
+ }
+
+ private BigqueryMatcher(
+ String applicationName,
+ String projectId,
+ String query,
+ boolean usingStandardSql,
+ String expectedChecksum) {
validateArgument("applicationName", applicationName);
validateArgument("projectId", projectId);
validateArgument("query", query);
@@ -77,10 +87,16 @@
this.projectId = projectId;
this.query = query;
+ this.usingStandardSql = usingStandardSql;
this.expectedChecksum = expectedChecksum;
this.bigqueryClient = BigqueryClient.getClient(applicationName);
}
+ public static BigqueryMatcher createUsingStandardSql(
+ String applicationName, String projectId, String query, String expectedChecksum) {
+ return new BigqueryMatcher(applicationName, projectId, query, true, expectedChecksum);
+ }
+
@Override
protected boolean matchesSafely(PipelineResult pipelineResult) {
LOG.info("Verifying Bigquery data");
@@ -88,7 +104,11 @@
// execute query
LOG.debug("Executing query: {}", query);
try {
- response = bigqueryClient.queryWithRetries(query, this.projectId);
+ if (usingStandardSql) {
+ response = bigqueryClient.queryWithRetriesUsingStandardSql(query, this.projectId);
+ } else {
+ response = bigqueryClient.queryWithRetries(query, this.projectId);
+ }
} catch (IOException | InterruptedException e) {
if (e instanceof InterruptedIOException) {
Thread.currentThread().interrupt();
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeBigQueryServices.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeBigQueryServices.java
new file mode 100644
index 0000000..10e8faf
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeBigQueryServices.java
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.testing;
+
+import com.google.api.client.util.Base64;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableRow;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices;
+import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** A fake implementation of BigQuery's query service.. */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class FakeBigQueryServices implements BigQueryServices {
+ private JobService jobService;
+ private DatasetService datasetService;
+ private StorageClient storageClient;
+
+ public FakeBigQueryServices withJobService(JobService jobService) {
+ this.jobService = jobService;
+ return this;
+ }
+
+ public FakeBigQueryServices withDatasetService(FakeDatasetService datasetService) {
+ this.datasetService = datasetService;
+ return this;
+ }
+
+ public FakeBigQueryServices withStorageClient(StorageClient storageClient) {
+ this.storageClient = storageClient;
+ return this;
+ }
+
+ @Override
+ public JobService getJobService(BigQueryOptions bqOptions) {
+ return jobService;
+ }
+
+ @Override
+ public DatasetService getDatasetService(BigQueryOptions bqOptions) {
+ return datasetService;
+ }
+
+ @Override
+ public StorageClient getStorageClient(BigQueryOptions bqOptions) {
+ return storageClient;
+ }
+
+ /**
+ * An implementation of {@link BigQueryServerStream} which takes a {@link List} as the {@link
+ * Iterable} to simulate a server stream. {@link #FakeBigQueryServerStream} is a no-op.
+ */
+ public static class FakeBigQueryServerStream<T> implements BigQueryServerStream<T> {
+
+ private final List<T> items;
+
+ public FakeBigQueryServerStream(List<T> items) {
+ this.items = items;
+ }
+
+ @Override
+ public Iterator<T> iterator() {
+ return items.iterator();
+ }
+
+ @Override
+ public void cancel() {}
+ }
+
+ public static String encodeQueryResult(Table table) throws IOException {
+ return encodeQueryResult(table, ImmutableList.of());
+ }
+
+ public static String encodeQueryResult(Table table, List<TableRow> rows) throws IOException {
+ KvCoder<String, List<TableRow>> coder =
+ KvCoder.of(StringUtf8Coder.of(), ListCoder.of(TableRowJsonCoder.of()));
+ KV<String, List<TableRow>> kv = KV.of(BigQueryHelpers.toJsonString(table), rows);
+ ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
+ coder.encode(kv, outputStream);
+ return Base64.encodeBase64String(outputStream.toByteArray());
+ }
+
+ public static KV<Table, List<TableRow>> decodeQueryResult(String queryResult) throws IOException {
+ KvCoder<String, List<TableRow>> coder =
+ KvCoder.of(StringUtf8Coder.of(), ListCoder.of(TableRowJsonCoder.of()));
+ ByteArrayInputStream inputStream = new ByteArrayInputStream(Base64.decodeBase64(queryResult));
+ KV<String, List<TableRow>> kv = coder.decode(inputStream);
+ Table table = BigQueryHelpers.fromJsonString(kv.getKey(), Table.class);
+ List<TableRow> rows = kv.getValue();
+ rows.forEach(FakeBigQueryServices::convertNumbers);
+ return KV.of(table, rows);
+ }
+
+ // Longs tend to get converted back to Integers due to JSON serialization. Convert them back.
+ public static TableRow convertNumbers(TableRow tableRow) {
+ for (TableRow.Entry entry : tableRow.entrySet()) {
+ if (entry.getValue() instanceof Integer) {
+ entry.setValue(Long.valueOf((Integer) entry.getValue()));
+ }
+ }
+ return tableRow;
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeDatasetService.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeDatasetService.java
new file mode 100644
index 0000000..7916513
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeDatasetService.java
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.testing;
+
+import static org.junit.Assert.assertEquals;
+
+import com.google.api.client.googleapis.json.GoogleJsonResponseException;
+import com.google.api.client.http.HttpHeaders;
+import com.google.api.services.bigquery.model.Dataset;
+import com.google.api.services.bigquery.model.DatasetReference;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
+import com.google.api.services.bigquery.model.TableReference;
+import com.google.api.services.bigquery.model.TableRow;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.regex.Pattern;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.DatasetService;
+import org.apache.beam.sdk.io.gcp.bigquery.ErrorContainer;
+import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy;
+import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy.Context;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+
+/** A fake dataset service that can be serialized, for use in testReadFromTable. */
+public class FakeDatasetService implements DatasetService, Serializable {
+ // Table information must be static, as each ParDo will get a separate instance of
+ // FakeDatasetServices, and they must all modify the same storage.
+ static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table<
+ String, String, Map<String, TableContainer>>
+ tables;
+
+ Map<String, List<String>> insertErrors = Maps.newHashMap();
+
+ public static void setUp() {
+ tables = HashBasedTable.create();
+ FakeJobService.setUp();
+ }
+
+ @Override
+ public Table getTable(TableReference tableRef) throws InterruptedException, IOException {
+ return getTable(tableRef, null);
+ }
+
+ @Override
+ public Table getTable(TableReference tableRef, @Nullable List<String> selectedFields)
+ throws InterruptedException, IOException {
+ synchronized (tables) {
+ Map<String, TableContainer> dataset =
+ tables.get(tableRef.getProjectId(), tableRef.getDatasetId());
+ if (dataset == null) {
+ throwNotFound(
+ "Tried to get a dataset %s:%s, but no such dataset was set",
+ tableRef.getProjectId(), tableRef.getDatasetId());
+ }
+ TableContainer tableContainer = dataset.get(tableRef.getTableId());
+ return tableContainer == null ? null : tableContainer.getTable();
+ }
+ }
+
+ public List<TableRow> getAllRows(String projectId, String datasetId, String tableId)
+ throws InterruptedException, IOException {
+ synchronized (tables) {
+ return getTableContainer(projectId, datasetId, tableId).getRows();
+ }
+ }
+
+ private TableContainer getTableContainer(String projectId, String datasetId, String tableId)
+ throws InterruptedException, IOException {
+ synchronized (tables) {
+ Map<String, TableContainer> dataset = tables.get(projectId, datasetId);
+ if (dataset == null) {
+ throwNotFound(
+ "Tried to get a dataset %s:%s, but no such dataset was set", projectId, datasetId);
+ }
+ TableContainer tableContainer = dataset.get(tableId);
+ if (tableContainer == null) {
+ throwNotFound(
+ "Tried to get a table %s:%s.%s, but no such table was set",
+ projectId, datasetId, tableId);
+ }
+ return tableContainer;
+ }
+ }
+
+ @Override
+ public void deleteTable(TableReference tableRef) throws IOException, InterruptedException {
+ validateWholeTableReference(tableRef);
+ synchronized (tables) {
+ Map<String, TableContainer> dataset =
+ tables.get(tableRef.getProjectId(), tableRef.getDatasetId());
+ if (dataset == null) {
+ throwNotFound(
+ "Tried to get a dataset %s:%s, but no such table was set",
+ tableRef.getProjectId(), tableRef.getDatasetId());
+ }
+ dataset.remove(tableRef.getTableId());
+ }
+ }
+
+ /**
+ * Validates a table reference for whole-table operations, such as create/delete/patch. Such
+ * operations do not support partition decorators.
+ */
+ private static void validateWholeTableReference(TableReference tableReference)
+ throws IOException {
+ final Pattern tableRegexp = Pattern.compile("[-\\w]{1,1024}");
+ if (!tableRegexp.matcher(tableReference.getTableId()).matches()) {
+ throw new IOException(
+ String.format(
+ "invalid table ID %s. Table IDs must be alphanumeric "
+ + "(plus underscores) and must be at most 1024 characters long. Also, table"
+ + " decorators cannot be used.",
+ tableReference.getTableId()));
+ }
+ }
+
+ @Override
+ public void createTable(Table table) throws IOException {
+ TableReference tableReference = table.getTableReference();
+ validateWholeTableReference(tableReference);
+ synchronized (tables) {
+ Map<String, TableContainer> dataset =
+ tables.get(tableReference.getProjectId(), tableReference.getDatasetId());
+ if (dataset == null) {
+ throwNotFound(
+ "Tried to get a dataset %s:%s, but no such table was set",
+ tableReference.getProjectId(), tableReference.getDatasetId());
+ }
+ dataset.computeIfAbsent(tableReference.getTableId(), k -> new TableContainer(table));
+ }
+ }
+
+ @Override
+ public boolean isTableEmpty(TableReference tableRef) throws IOException, InterruptedException {
+ Long numBytes = getTable(tableRef).getNumBytes();
+ return numBytes == null || numBytes == 0L;
+ }
+
+ @Override
+ public Dataset getDataset(String projectId, String datasetId)
+ throws IOException, InterruptedException {
+ synchronized (tables) {
+ Map<String, TableContainer> dataset = tables.get(projectId, datasetId);
+ if (dataset == null) {
+ throwNotFound(
+ "Tried to get a dataset %s:%s, but no such table was set", projectId, datasetId);
+ }
+ return new Dataset()
+ .setDatasetReference(
+ new DatasetReference().setDatasetId(datasetId).setProjectId(projectId));
+ }
+ }
+
+ @Override
+ public void createDataset(
+ String projectId,
+ String datasetId,
+ String location,
+ String description,
+ Long defaultTableExpirationMs /* ignored */)
+ throws IOException, InterruptedException {
+ synchronized (tables) {
+ Map<String, TableContainer> dataset = tables.get(projectId, datasetId);
+ if (dataset == null) {
+ dataset = new HashMap<>();
+ tables.put(projectId, datasetId, dataset);
+ }
+ }
+ }
+
+ @Override
+ public void deleteDataset(String projectId, String datasetId)
+ throws IOException, InterruptedException {
+ synchronized (tables) {
+ tables.remove(projectId, datasetId);
+ }
+ }
+
+ public long insertAll(
+ TableReference ref, List<TableRow> rowList, @Nullable List<String> insertIdList)
+ throws IOException, InterruptedException {
+ List<ValueInSingleWindow<TableRow>> windowedRows = Lists.newArrayList();
+ for (TableRow row : rowList) {
+ windowedRows.add(
+ ValueInSingleWindow.of(
+ row,
+ GlobalWindow.TIMESTAMP_MAX_VALUE,
+ GlobalWindow.INSTANCE,
+ PaneInfo.ON_TIME_AND_ONLY_FIRING));
+ }
+ return insertAll(
+ ref, windowedRows, insertIdList, InsertRetryPolicy.alwaysRetry(), null, null, false, false);
+ }
+
+ @Override
+ public <T> long insertAll(
+ TableReference ref,
+ List<ValueInSingleWindow<TableRow>> rowList,
+ @Nullable List<String> insertIdList,
+ InsertRetryPolicy retryPolicy,
+ List<ValueInSingleWindow<T>> failedInserts,
+ ErrorContainer<T> errorContainer,
+ boolean skipInvalidRows,
+ boolean ignoreUnknownValues)
+ throws IOException, InterruptedException {
+ Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> insertErrors = getInsertErrors();
+ synchronized (tables) {
+ if (insertIdList != null) {
+ assertEquals(rowList.size(), insertIdList.size());
+ } else {
+ insertIdList = Lists.newArrayListWithExpectedSize(rowList.size());
+ for (int i = 0; i < rowList.size(); ++i) {
+ insertIdList.add(Integer.toString(ThreadLocalRandom.current().nextInt()));
+ }
+ }
+
+ long dataSize = 0;
+ TableContainer tableContainer =
+ getTableContainer(
+ ref.getProjectId(),
+ ref.getDatasetId(),
+ BigQueryHelpers.stripPartitionDecorator(ref.getTableId()));
+ for (int i = 0; i < rowList.size(); ++i) {
+ TableRow row = rowList.get(i).getValue();
+ List<TableDataInsertAllResponse.InsertErrors> allErrors = insertErrors.get(row);
+ boolean shouldInsert = true;
+ if (allErrors != null) {
+ for (TableDataInsertAllResponse.InsertErrors errors : allErrors) {
+ if (!retryPolicy.shouldRetry(new Context(errors))) {
+ shouldInsert = false;
+ }
+ }
+ }
+ if (shouldInsert) {
+ dataSize += tableContainer.addRow(row, insertIdList.get(i));
+ } else {
+ errorContainer.add(
+ failedInserts, allErrors.get(allErrors.size() - 1), ref, rowList.get(i));
+ }
+ }
+ return dataSize;
+ }
+ }
+
+ @Override
+ public Table patchTableDescription(
+ TableReference tableReference, @Nullable String tableDescription)
+ throws IOException, InterruptedException {
+ validateWholeTableReference(tableReference);
+ synchronized (tables) {
+ TableContainer tableContainer =
+ getTableContainer(
+ tableReference.getProjectId(),
+ tableReference.getDatasetId(),
+ tableReference.getTableId());
+ tableContainer.getTable().setDescription(tableDescription);
+ return tableContainer.getTable();
+ }
+ }
+
+ /**
+ * Cause a given {@link TableRow} object to fail when it's inserted. The errors link the list will
+ * be returned on subsequent retries, and the insert will succeed when the errors run out.
+ */
+ public void failOnInsert(
+ Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> insertErrors) {
+ synchronized (tables) {
+ for (Map.Entry<TableRow, List<TableDataInsertAllResponse.InsertErrors>> entry :
+ insertErrors.entrySet()) {
+ List<String> errorStrings = Lists.newArrayList();
+ for (TableDataInsertAllResponse.InsertErrors errors : entry.getValue()) {
+ errorStrings.add(BigQueryHelpers.toJsonString(errors));
+ }
+ this.insertErrors.put(BigQueryHelpers.toJsonString(entry.getKey()), errorStrings);
+ }
+ }
+ }
+
+ Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> getInsertErrors() {
+ Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> parsedInsertErrors =
+ Maps.newHashMap();
+ synchronized (tables) {
+ for (Map.Entry<String, List<String>> entry : this.insertErrors.entrySet()) {
+ TableRow tableRow = BigQueryHelpers.fromJsonString(entry.getKey(), TableRow.class);
+ List<TableDataInsertAllResponse.InsertErrors> allErrors = Lists.newArrayList();
+ for (String errorsString : entry.getValue()) {
+ allErrors.add(
+ BigQueryHelpers.fromJsonString(
+ errorsString, TableDataInsertAllResponse.InsertErrors.class));
+ }
+ parsedInsertErrors.put(tableRow, allErrors);
+ }
+ }
+ return parsedInsertErrors;
+ }
+
+ void throwNotFound(String format, Object... args) throws IOException {
+ throw new IOException(
+ String.format(format, args),
+ new GoogleJsonResponseException.Builder(404, String.format(format, args), new HttpHeaders())
+ .build());
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java
new file mode 100644
index 0000000..9729f78
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.testing;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.util.BackOff;
+import com.google.api.client.util.BackOffUtils;
+import com.google.api.client.util.Sleeper;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.ErrorProto;
+import com.google.api.services.bigquery.model.Job;
+import com.google.api.services.bigquery.model.JobConfiguration;
+import com.google.api.services.bigquery.model.JobConfigurationExtract;
+import com.google.api.services.bigquery.model.JobConfigurationLoad;
+import com.google.api.services.bigquery.model.JobConfigurationQuery;
+import com.google.api.services.bigquery.model.JobConfigurationTableCopy;
+import com.google.api.services.bigquery.model.JobReference;
+import com.google.api.services.bigquery.model.JobStatistics;
+import com.google.api.services.bigquery.model.JobStatistics4;
+import com.google.api.services.bigquery.model.JobStatus;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableReference;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import com.google.api.services.bigquery.model.TimePartitioning;
+import java.io.BufferedReader;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.Serializable;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.GenericRecordBuilder;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.Coder.Context;
+import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
+import org.apache.beam.sdk.extensions.gcp.util.Transport;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResourceId;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.JobService;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils;
+import org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.MimeTypes;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.HashBasedTable;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
+
+/** A fake implementation of BigQuery's job service. */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class FakeJobService implements JobService, Serializable {
+ private static final JsonFactory JSON_FACTORY = Transport.getJsonFactory();
+ // Whenever a job is started, the first 2 calls to GetJob will report the job as pending,
+ // the next 2 will return the job as running, and only then will the job report as done.
+ private static final int GET_JOBS_TRANSITION_INTERVAL = 2;
+
+ // The number of times to simulate a failure and trigger a retry.
+ private int numFailuresExpected;
+ private int numFailures = 0;
+
+ private final FakeDatasetService datasetService;
+
+ private static class JobInfo {
+ Job job;
+ int getJobCount = 0;
+
+ JobInfo(Job job) {
+ this.job = job;
+ }
+ }
+
+ private static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table<
+ String, String, JobInfo>
+ allJobs;
+ private static int numExtractJobCalls;
+
+ private static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table<
+ String, String, List<ResourceId>>
+ filesForLoadJobs;
+ private static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Table<
+ String, String, JobStatistics>
+ dryRunQueryResults;
+
+ public FakeJobService() {
+ this(0);
+ }
+
+ public FakeJobService(int numFailures) {
+ this.datasetService = new FakeDatasetService();
+ this.numFailuresExpected = numFailures;
+ }
+
+ public void setNumFailuresExpected(int numFailuresExpected) {
+ this.numFailuresExpected = numFailuresExpected;
+ }
+
+ public static void setUp() {
+ allJobs = HashBasedTable.create();
+ numExtractJobCalls = 0;
+ filesForLoadJobs = HashBasedTable.create();
+ dryRunQueryResults = HashBasedTable.create();
+ }
+
+ @Override
+ public void startLoadJob(JobReference jobRef, JobConfigurationLoad loadConfig)
+ throws IOException {
+ synchronized (allJobs) {
+ verifyUniqueJobId(jobRef.getJobId());
+ Job job = new Job();
+ job.setJobReference(jobRef);
+ job.setConfiguration(new JobConfiguration().setLoad(loadConfig));
+ job.setKind(" bigquery#job");
+ job.setStatus(new JobStatus().setState("PENDING"));
+
+ // Copy the files to a new location for import, as the temporary files will be deleted by
+ // the caller.
+ if (loadConfig.getSourceUris().size() > 0) {
+ ImmutableList.Builder<ResourceId> sourceFiles = ImmutableList.builder();
+ ImmutableList.Builder<ResourceId> loadFiles = ImmutableList.builder();
+ for (String filename : loadConfig.getSourceUris()) {
+ sourceFiles.add(FileSystems.matchNewResource(filename, false /* isDirectory */));
+ loadFiles.add(
+ FileSystems.matchNewResource(
+ filename + ThreadLocalRandom.current().nextInt(), false /* isDirectory */));
+ }
+
+ FileSystems.copy(sourceFiles.build(), loadFiles.build());
+ filesForLoadJobs.put(jobRef.getProjectId(), jobRef.getJobId(), loadFiles.build());
+ }
+
+ allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
+ }
+ }
+
+ @Override
+ public void startExtractJob(JobReference jobRef, JobConfigurationExtract extractConfig)
+ throws IOException {
+ checkArgument(
+ "AVRO".equals(extractConfig.getDestinationFormat()), "Only extract to AVRO is supported");
+ synchronized (allJobs) {
+ verifyUniqueJobId(jobRef.getJobId());
+ ++numExtractJobCalls;
+
+ Job job = new Job();
+ job.setJobReference(jobRef);
+ job.setConfiguration(new JobConfiguration().setExtract(extractConfig));
+ job.setKind(" bigquery#job");
+ job.setStatus(new JobStatus().setState("PENDING"));
+ allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
+ }
+ }
+
+ public int getNumExtractJobCalls() {
+ synchronized (allJobs) {
+ return numExtractJobCalls;
+ }
+ }
+
+ @Override
+ public void startQueryJob(JobReference jobRef, JobConfigurationQuery query) {
+ synchronized (allJobs) {
+ Job job = new Job();
+ job.setJobReference(jobRef);
+ job.setConfiguration(new JobConfiguration().setQuery(query));
+ job.setKind(" bigquery#job");
+ job.setStatus(new JobStatus().setState("PENDING"));
+ allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
+ }
+ }
+
+ @Override
+ public void startCopyJob(JobReference jobRef, JobConfigurationTableCopy copyConfig)
+ throws IOException {
+ synchronized (allJobs) {
+ verifyUniqueJobId(jobRef.getJobId());
+ Job job = new Job();
+ job.setJobReference(jobRef);
+ job.setConfiguration(new JobConfiguration().setCopy(copyConfig));
+ job.setKind(" bigquery#job");
+ job.setStatus(new JobStatus().setState("PENDING"));
+ allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
+ }
+ }
+
+ @Override
+ public Job pollJob(JobReference jobRef, int maxAttempts) throws InterruptedException {
+ BackOff backoff =
+ BackOffAdapter.toGcpBackOff(
+ FluentBackoff.DEFAULT
+ .withMaxRetries(maxAttempts)
+ .withInitialBackoff(Duration.millis(10))
+ .withMaxBackoff(Duration.standardSeconds(1))
+ .backoff());
+ Sleeper sleeper = Sleeper.DEFAULT;
+ try {
+ do {
+ Job job = getJob(jobRef);
+ if (job != null) {
+ JobStatus status = job.getStatus();
+ if (status != null
+ && ("DONE".equals(status.getState()) || "FAILED".equals(status.getState()))) {
+ return job;
+ }
+ }
+ } while (BackOffUtils.next(sleeper, backoff));
+ } catch (IOException e) {
+ return null;
+ }
+ return null;
+ }
+
+ public void expectDryRunQuery(String projectId, String query, JobStatistics result) {
+ synchronized (dryRunQueryResults) {
+ dryRunQueryResults.put(projectId, query, result);
+ }
+ }
+
+ @Override
+ public JobStatistics dryRunQuery(String projectId, JobConfigurationQuery query, String location) {
+ synchronized (dryRunQueryResults) {
+ JobStatistics result = dryRunQueryResults.get(projectId, query.getQuery());
+ if (result != null) {
+ return result;
+ }
+ }
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public Job getJob(JobReference jobRef) {
+ try {
+ synchronized (allJobs) {
+ JobInfo job = allJobs.get(jobRef.getProjectId(), jobRef.getJobId());
+ if (job == null) {
+ return null;
+ }
+ try {
+ ++job.getJobCount;
+ if (!"FAILED".equals(job.job.getStatus().getState())) {
+ if (numFailures < numFailuresExpected) {
+ ++numFailures;
+ throw new Exception("Failure number " + numFailures);
+ }
+
+ if (job.getJobCount == GET_JOBS_TRANSITION_INTERVAL + 1) {
+ job.job.getStatus().setState("RUNNING");
+ } else if (job.getJobCount == 2 * GET_JOBS_TRANSITION_INTERVAL + 1) {
+ job.job.setStatus(runJob(job.job));
+ }
+ }
+ } catch (Exception e) {
+ job.job
+ .getStatus()
+ .setState("FAILED")
+ .setErrorResult(
+ new ErrorProto()
+ .setMessage(
+ String.format(
+ "Job %s failed: %s", job.job.getConfiguration(), e.toString())));
+ List<ResourceId> sourceFiles =
+ filesForLoadJobs.get(jobRef.getProjectId(), jobRef.getJobId());
+ FileSystems.delete(sourceFiles);
+ }
+ return JSON_FACTORY.fromString(JSON_FACTORY.toString(job.job), Job.class);
+ }
+ } catch (IOException e) {
+ return null;
+ }
+ }
+
+ private void verifyUniqueJobId(String jobId) throws IOException {
+ if (allJobs.containsColumn(jobId)) {
+ throw new IOException("Duplicate job id " + jobId);
+ }
+ }
+
+ private JobStatus runJob(Job job) throws InterruptedException, IOException {
+ if (job.getConfiguration().getLoad() != null) {
+ return runLoadJob(job.getJobReference(), job.getConfiguration().getLoad());
+ } else if (job.getConfiguration().getCopy() != null) {
+ return runCopyJob(job.getConfiguration().getCopy());
+ } else if (job.getConfiguration().getExtract() != null) {
+ return runExtractJob(job, job.getConfiguration().getExtract());
+ } else if (job.getConfiguration().getQuery() != null) {
+ return runQueryJob(job.getConfiguration().getQuery());
+ }
+ return new JobStatus().setState("DONE");
+ }
+
+ private boolean validateDispositions(
+ Table table, CreateDisposition createDisposition, WriteDisposition writeDisposition)
+ throws InterruptedException, IOException {
+ if (table == null) {
+ if (createDisposition == CreateDisposition.CREATE_NEVER) {
+ return false;
+ }
+ } else if (writeDisposition == WriteDisposition.WRITE_TRUNCATE) {
+ datasetService.deleteTable(table.getTableReference());
+ } else if (writeDisposition == WriteDisposition.WRITE_EMPTY) {
+ List<TableRow> allRows =
+ datasetService.getAllRows(
+ table.getTableReference().getProjectId(),
+ table.getTableReference().getDatasetId(),
+ table.getTableReference().getTableId());
+ if (!allRows.isEmpty()) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private JobStatus runLoadJob(JobReference jobRef, JobConfigurationLoad load)
+ throws InterruptedException, IOException {
+ TableReference destination = load.getDestinationTable();
+ TableSchema schema = load.getSchema();
+ checkArgument(schema != null, "No schema specified");
+ List<ResourceId> sourceFiles = filesForLoadJobs.get(jobRef.getProjectId(), jobRef.getJobId());
+ WriteDisposition writeDisposition = WriteDisposition.valueOf(load.getWriteDisposition());
+ CreateDisposition createDisposition = CreateDisposition.valueOf(load.getCreateDisposition());
+ checkArgument("NEWLINE_DELIMITED_JSON".equals(load.getSourceFormat()));
+ Table existingTable = datasetService.getTable(destination);
+ if (!validateDispositions(existingTable, createDisposition, writeDisposition)) {
+ return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
+ }
+ if (existingTable == null) {
+ TableReference strippedDestination =
+ destination
+ .clone()
+ .setTableId(BigQueryHelpers.stripPartitionDecorator(destination.getTableId()));
+ existingTable = new Table().setTableReference(strippedDestination).setSchema(schema);
+ if (load.getTimePartitioning() != null) {
+ existingTable = existingTable.setTimePartitioning(load.getTimePartitioning());
+ }
+ if (load.getClustering() != null) {
+ existingTable = existingTable.setClustering(load.getClustering());
+ }
+ datasetService.createTable(existingTable);
+ }
+
+ List<TableRow> rows = Lists.newArrayList();
+ for (ResourceId filename : sourceFiles) {
+ rows.addAll(readRows(filename.toString()));
+ }
+ datasetService.insertAll(destination, rows, null);
+ FileSystems.delete(sourceFiles);
+ return new JobStatus().setState("DONE");
+ }
+
+ private JobStatus runCopyJob(JobConfigurationTableCopy copy)
+ throws InterruptedException, IOException {
+ List<TableReference> sources = copy.getSourceTables();
+ TableReference destination = copy.getDestinationTable();
+ WriteDisposition writeDisposition = WriteDisposition.valueOf(copy.getWriteDisposition());
+ CreateDisposition createDisposition = CreateDisposition.valueOf(copy.getCreateDisposition());
+ Table existingTable = datasetService.getTable(destination);
+ if (!validateDispositions(existingTable, createDisposition, writeDisposition)) {
+ return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
+ }
+ TimePartitioning partitioning = null;
+ Clustering clustering = null;
+ TableSchema schema = null;
+ boolean first = true;
+ List<TableRow> allRows = Lists.newArrayList();
+ for (TableReference source : sources) {
+ Table table = checkNotNull(datasetService.getTable(source));
+ if (!first) {
+ if (!Objects.equals(partitioning, table.getTimePartitioning())) {
+ return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
+ }
+ if (!Objects.equals(clustering, table.getClustering())) {
+ return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
+ }
+ if (!Objects.equals(schema, table.getSchema())) {
+ return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
+ }
+ }
+ partitioning = table.getTimePartitioning();
+ clustering = table.getClustering();
+ schema = table.getSchema();
+ first = false;
+ allRows.addAll(
+ datasetService.getAllRows(
+ source.getProjectId(), source.getDatasetId(), source.getTableId()));
+ }
+ datasetService.createTable(
+ new Table()
+ .setTableReference(destination)
+ .setSchema(schema)
+ .setTimePartitioning(partitioning)
+ .setClustering(clustering)
+ .setEncryptionConfiguration(copy.getDestinationEncryptionConfiguration()));
+ datasetService.insertAll(destination, allRows, null);
+ return new JobStatus().setState("DONE");
+ }
+
+ private JobStatus runExtractJob(Job job, JobConfigurationExtract extract)
+ throws InterruptedException, IOException {
+ TableReference sourceTable = extract.getSourceTable();
+
+ List<TableRow> rows =
+ datasetService.getAllRows(
+ sourceTable.getProjectId(), sourceTable.getDatasetId(), sourceTable.getTableId());
+ TableSchema schema = datasetService.getTable(sourceTable).getSchema();
+ List<Long> destinationFileCounts = Lists.newArrayList();
+ for (String destination : extract.getDestinationUris()) {
+ destinationFileCounts.add(writeRows(sourceTable.getTableId(), rows, schema, destination));
+ }
+ job.setStatistics(
+ new JobStatistics()
+ .setExtract(new JobStatistics4().setDestinationUriFileCounts(destinationFileCounts)));
+ return new JobStatus().setState("DONE");
+ }
+
+ private JobStatus runQueryJob(JobConfigurationQuery query)
+ throws IOException, InterruptedException {
+ KV<Table, List<TableRow>> result = FakeBigQueryServices.decodeQueryResult(query.getQuery());
+ datasetService.createTable(result.getKey().setTableReference(query.getDestinationTable()));
+ datasetService.insertAll(query.getDestinationTable(), result.getValue(), null);
+ return new JobStatus().setState("DONE");
+ }
+
+ private List<TableRow> readRows(String filename) throws IOException {
+ Coder<TableRow> coder = TableRowJsonCoder.of();
+ List<TableRow> tableRows = Lists.newArrayList();
+ try (BufferedReader reader =
+ Files.newBufferedReader(Paths.get(filename), StandardCharsets.UTF_8)) {
+ String line;
+ while ((line = reader.readLine()) != null) {
+ TableRow tableRow =
+ coder.decode(
+ new ByteArrayInputStream(line.getBytes(StandardCharsets.UTF_8)), Context.OUTER);
+ tableRows.add(tableRow);
+ }
+ }
+ return tableRows;
+ }
+
+ private long writeRows(
+ String tableId, List<TableRow> rows, TableSchema schema, String destinationPattern)
+ throws IOException {
+ Schema avroSchema = BigQueryUtils.toGenericAvroSchema(tableId, schema.getFields());
+ List<TableRow> rowsToWrite = Lists.newArrayList();
+ int shard = 0;
+ for (TableRow row : rows) {
+ rowsToWrite.add(row);
+ if (rowsToWrite.size() == 5) {
+ writeRowsHelper(rowsToWrite, avroSchema, destinationPattern, shard++);
+ rowsToWrite.clear();
+ }
+ }
+ if (!rowsToWrite.isEmpty()) {
+ writeRowsHelper(rowsToWrite, avroSchema, destinationPattern, shard++);
+ }
+ return shard;
+ }
+
+ private void writeRowsHelper(
+ List<TableRow> rows, Schema avroSchema, String destinationPattern, int shard) {
+ String filename = destinationPattern.replace("*", String.format("%012d", shard));
+ try (WritableByteChannel channel =
+ FileSystems.create(
+ FileSystems.matchNewResource(filename, false /* isDirectory */), MimeTypes.BINARY);
+ DataFileWriter<GenericRecord> tableRowWriter =
+ new DataFileWriter<>(new GenericDatumWriter<GenericRecord>(avroSchema))
+ .create(avroSchema, Channels.newOutputStream(channel))) {
+ for (Map<String, Object> record : rows) {
+ GenericRecordBuilder genericRecordBuilder = new GenericRecordBuilder(avroSchema);
+ for (Map.Entry<String, Object> field : record.entrySet()) {
+ genericRecordBuilder.set(field.getKey(), field.getValue());
+ }
+ tableRowWriter.append(genericRecordBuilder.build());
+ }
+ } catch (IOException e) {
+ throw new IllegalStateException(
+ String.format("Could not create destination for extract job %s", filename), e);
+ }
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java
new file mode 100644
index 0000000..cc7ad4f
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.testing;
+
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableRow;
+import java.util.ArrayList;
+import java.util.List;
+
+/** Encapsulates a BigQuery Table, and it's contents. */
+class TableContainer {
+ Table table;
+ List<TableRow> rows;
+ List<String> ids;
+ Long sizeBytes;
+
+ TableContainer(Table table) {
+ this.table = table;
+
+ this.rows = new ArrayList<>();
+ this.ids = new ArrayList<>();
+ this.sizeBytes = 0L;
+ }
+
+ long addRow(TableRow row, String id) {
+ rows.add(row);
+ ids.add(id);
+ long rowSize = row.toString().length();
+ Long tableSize = table.getNumBytes();
+ if (tableSize == null) {
+ table.setNumBytes(rowSize);
+ } else {
+ table.setNumBytes(tableSize + rowSize);
+ }
+ return rowSize;
+ }
+
+ Table getTable() {
+ return table;
+ }
+
+ List<TableRow> getRows() {
+ return rows;
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/com/google/cloud/spanner/FakeBatchTransactionId.java b/sdks/java/io/google-cloud-platform/src/test/java/com/google/cloud/spanner/FakeBatchTransactionId.java
index e01a656..e524e2d 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/com/google/cloud/spanner/FakeBatchTransactionId.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/com/google/cloud/spanner/FakeBatchTransactionId.java
@@ -19,7 +19,7 @@
import com.google.cloud.Timestamp;
import com.google.protobuf.ByteString;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/**
* A fake {@link BatchTransactionId} object.
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
index fa4ca25..50f6548 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
import org.apache.beam.sdk.util.ApiSurface;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.hamcrest.Matcher;
import org.hamcrest.Matchers;
import org.junit.Test;
@@ -39,7 +39,7 @@
@Test
public void testGcpApiSurface() throws Exception {
- final Package thisPackage = getClass().getPackage();
+ final Package thisPackage = this.getClass().getPackage();
final ClassLoader thisClassLoader = getClass().getClassLoader();
final ApiSurface apiSurface =
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
index c645ee0..aeeab06 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java
@@ -41,9 +41,9 @@
import org.apache.avro.util.Utf8;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.DefaultCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
index d630799..ec0addc 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpersTest.java
@@ -41,7 +41,8 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.junit.Assert;
import org.junit.Rule;
@@ -49,7 +50,6 @@
import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.testng.collections.Sets;
/** Tests for {@link BigQueryHelpers}. */
@RunWith(JUnit4.class)
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java
index a2d9748..cdbc5f6 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.testing.TestPipelineOptions;
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
index 883bbdd..3edd6e3 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOReadTest.java
@@ -48,9 +48,14 @@
import org.apache.beam.sdk.extensions.protobuf.ProtoCoder;
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.QueryPriority;
+import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices;
+import org.apache.beam.sdk.io.gcp.testing.FakeDatasetService;
+import org.apache.beam.sdk.io.gcp.testing.FakeJobService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.transforms.Select;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.SourceTestUtils;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -65,9 +70,10 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -405,6 +411,65 @@
}
@Test
+ public void testReadTableWithSchema() throws IOException, InterruptedException {
+ // setup
+ Table someTable = new Table();
+ someTable.setSchema(
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema().setName("name").setType("STRING"),
+ new TableFieldSchema().setName("number").setType("INTEGER"))));
+ someTable.setTableReference(
+ new TableReference()
+ .setProjectId("non-executing-project")
+ .setDatasetId("schema_dataset")
+ .setTableId("schema_table"));
+ someTable.setNumBytes(1024L * 1024L);
+ FakeDatasetService fakeDatasetService = new FakeDatasetService();
+ fakeDatasetService.createDataset("non-executing-project", "schema_dataset", "", "", null);
+ fakeDatasetService.createTable(someTable);
+
+ List<TableRow> records =
+ Lists.newArrayList(
+ new TableRow().set("name", "a").set("number", 1L),
+ new TableRow().set("name", "b").set("number", 2L),
+ new TableRow().set("name", "c").set("number", 3L));
+
+ fakeDatasetService.insertAll(someTable.getTableReference(), records, null);
+
+ FakeBigQueryServices fakeBqServices =
+ new FakeBigQueryServices()
+ .withJobService(new FakeJobService())
+ .withDatasetService(fakeDatasetService);
+
+ // test
+ BigQueryIO.TypedRead<TableRow> read =
+ BigQueryIO.readTableRowsWithSchema()
+ .from("non-executing-project:schema_dataset.schema_table")
+ .withTestServices(fakeBqServices)
+ .withoutValidation();
+
+ PCollection<TableRow> bqRows = p.apply(read);
+
+ Schema expectedSchema =
+ Schema.of(
+ Schema.Field.of("name", Schema.FieldType.STRING).withNullable(true),
+ Schema.Field.of("number", Schema.FieldType.INT64).withNullable(true));
+ assertEquals(expectedSchema, bqRows.getSchema());
+
+ PCollection<Row> output = bqRows.apply(Select.fieldNames("name", "number"));
+ PAssert.that(output)
+ .containsInAnyOrder(
+ ImmutableList.of(
+ Row.withSchema(expectedSchema).addValues("a", 1L).build(),
+ Row.withSchema(expectedSchema).addValues("b", 2L).build(),
+ Row.withSchema(expectedSchema).addValues("c", 3L).build()));
+
+ p.run();
+ }
+
+ @Test
public void testBuildSourceDisplayDataTable() {
String tableSpec = "project:dataset.tableid";
@@ -509,12 +574,8 @@
String stepUuid = "testStepUuid";
BoundedSource<TableRow> bqSource =
- BigQueryTableSource.create(
- stepUuid,
- ValueProvider.StaticValueProvider.of(table),
- fakeBqServices,
- TableRowJsonCoder.of(),
- BigQueryIO.TableRowParser.INSTANCE);
+ BigQueryTableSourceDef.create(fakeBqServices, ValueProvider.StaticValueProvider.of(table))
+ .toSource(stepUuid, TableRowJsonCoder.of(), BigQueryIO.TableRowParser.INSTANCE);
PipelineOptions options = PipelineOptionsFactory.create();
options.setTempLocation(testFolder.getRoot().getAbsolutePath());
@@ -562,12 +623,8 @@
String stepUuid = "testStepUuid";
BoundedSource<TableRow> bqSource =
- BigQueryTableSource.create(
- stepUuid,
- ValueProvider.StaticValueProvider.of(table),
- fakeBqServices,
- TableRowJsonCoder.of(),
- BigQueryIO.TableRowParser.INSTANCE);
+ BigQueryTableSourceDef.create(fakeBqServices, ValueProvider.StaticValueProvider.of(table))
+ .toSource(stepUuid, TableRowJsonCoder.of(), BigQueryIO.TableRowParser.INSTANCE);
PipelineOptions options = PipelineOptionsFactory.create();
assertEquals(108, bqSource.getEstimatedSizeBytes(options));
@@ -600,12 +657,8 @@
String stepUuid = "testStepUuid";
BoundedSource<TableRow> bqSource =
- BigQueryTableSource.create(
- stepUuid,
- ValueProvider.StaticValueProvider.of(table),
- fakeBqServices,
- TableRowJsonCoder.of(),
- BigQueryIO.TableRowParser.INSTANCE);
+ BigQueryTableSourceDef.create(fakeBqServices, ValueProvider.StaticValueProvider.of(table))
+ .toSource(stepUuid, TableRowJsonCoder.of(), BigQueryIO.TableRowParser.INSTANCE);
PipelineOptions options = PipelineOptionsFactory.create();
assertEquals(118, bqSource.getEstimatedSizeBytes(options));
@@ -621,18 +674,16 @@
bqOptions.setProject("project");
String stepUuid = "testStepUuid";
- BigQueryQuerySource<TableRow> bqSource =
- BigQueryQuerySource.create(
- stepUuid,
- ValueProvider.StaticValueProvider.of(queryString),
- true /* flattenResults */,
- true /* useLegacySql */,
- fakeBqServices,
- TableRowJsonCoder.of(),
- BigQueryIO.TableRowParser.INSTANCE,
- QueryPriority.BATCH,
- null,
- null);
+ BigQuerySourceBase<TableRow> bqSource =
+ BigQueryQuerySourceDef.create(
+ fakeBqServices,
+ ValueProvider.StaticValueProvider.of(queryString),
+ true, /* flattenResults */
+ true, /* useLegacySql */
+ QueryPriority.BATCH,
+ null,
+ null)
+ .toSource(stepUuid, TableRowJsonCoder.of(), BigQueryIO.TableRowParser.INSTANCE);
fakeJobService.expectDryRunQuery(
bqOptions.getProject(),
@@ -697,17 +748,15 @@
.setReferencedTables(ImmutableList.of(sourceTableRef, tempTableReference))));
BoundedSource<TableRow> bqSource =
- BigQueryQuerySource.create(
- stepUuid,
- ValueProvider.StaticValueProvider.of(encodedQuery),
- true /* flattenResults */,
- true /* useLegacySql */,
- fakeBqServices,
- TableRowJsonCoder.of(),
- BigQueryIO.TableRowParser.INSTANCE,
- QueryPriority.BATCH,
- null,
- null);
+ BigQueryQuerySourceDef.create(
+ fakeBqServices,
+ ValueProvider.StaticValueProvider.of(encodedQuery),
+ true /* flattenResults */,
+ true /* useLegacySql */,
+ QueryPriority.BATCH,
+ null,
+ null)
+ .toSource(stepUuid, TableRowJsonCoder.of(), BigQueryIO.TableRowParser.INSTANCE);
options.setTempLocation(testFolder.getRoot().getAbsolutePath());
@@ -764,17 +813,15 @@
.setReferencedTables(ImmutableList.of())));
BoundedSource<TableRow> bqSource =
- BigQueryQuerySource.create(
- stepUuid,
- ValueProvider.StaticValueProvider.of(encodedQuery),
- true /* flattenResults */,
- true /* useLegacySql */,
- fakeBqServices,
- TableRowJsonCoder.of(),
- BigQueryIO.TableRowParser.INSTANCE,
- QueryPriority.BATCH,
- null,
- null);
+ BigQueryQuerySourceDef.create(
+ fakeBqServices,
+ ValueProvider.StaticValueProvider.of(encodedQuery),
+ true /* flattenResults */,
+ true /* useLegacySql */,
+ QueryPriority.BATCH,
+ null,
+ null)
+ .toSource(stepUuid, TableRowJsonCoder.of(), BigQueryIO.TableRowParser.INSTANCE);
options.setTempLocation(testFolder.getRoot().getAbsolutePath());
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java
index d619382..fa29f7b 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryIT.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryTest.java
index 847aa9a..c208d17 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryTest.java
@@ -23,13 +23,13 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.hasItem;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import static org.mockito.Mockito.withSettings;
-import static org.testng.Assert.assertFalse;
import com.google.api.services.bigquery.model.JobStatistics;
import com.google.api.services.bigquery.model.JobStatistics2;
@@ -47,7 +47,9 @@
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession;
import com.google.cloud.bigquery.storage.v1beta1.Storage.Stream;
import com.google.cloud.bigquery.storage.v1beta1.Storage.StreamPosition;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.StreamStatus;
import com.google.protobuf.ByteString;
+import com.google.protobuf.UnknownFieldSet;
import java.io.ByteArrayOutputStream;
import java.util.Collection;
import java.util.List;
@@ -68,6 +70,10 @@
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.QueryPriority;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.StorageClient;
+import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices;
+import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices.FakeBigQueryServerStream;
+import org.apache.beam.sdk.io.gcp.testing.FakeDatasetService;
+import org.apache.beam.sdk.io.gcp.testing.FakeJobService;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
@@ -76,8 +82,8 @@
import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -236,6 +242,29 @@
}
@Test
+ public void testBuildQueryBasedSourceWithSelectedFields() throws Exception {
+ TypedRead<TableRow> typedRead =
+ getDefaultTypedRead().withSelectedFields(Lists.newArrayList("a"));
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage(
+ "Invalid BigQueryIO.Read: Specifies selected fields, "
+ + "which only applies when reading from a table");
+ p.apply(typedRead);
+ p.run();
+ }
+
+ @Test
+ public void testBuildQueryBasedSourceWithRowRestriction() throws Exception {
+ TypedRead<TableRow> typedRead = getDefaultTypedRead().withRowRestriction("a > 5");
+ thrown.expect(IllegalArgumentException.class);
+ thrown.expectMessage(
+ "Invalid BigQueryIO.Read: Specifies row restriction, "
+ + "which only applies when reading from a table");
+ p.apply(typedRead);
+ p.run();
+ }
+
+ @Test
public void testDisplayData() throws Exception {
TypedRead<TableRow> typedRead = getDefaultTypedRead();
DisplayData displayData = DisplayData.from(typedRead);
@@ -358,6 +387,12 @@
.setParent("projects/" + options.getProject())
.setTableReference(BigQueryHelpers.toTableRefProto(tempTableReference))
.setRequestedStreams(requestedStreamCount)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession.Builder builder = ReadSession.newBuilder();
@@ -428,6 +463,12 @@
.setParent("projects/" + options.getProject())
.setTableReference(BigQueryHelpers.toTableRefProto(tempTableReference))
.setRequestedStreams(1024)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession.Builder builder = ReadSession.newBuilder();
@@ -487,7 +528,8 @@
private static final EncoderFactory ENCODER_FACTORY = EncoderFactory.get();
private static ReadRowsResponse createResponse(
- Schema schema, Collection<GenericRecord> genericRecords) throws Exception {
+ Schema schema, Collection<GenericRecord> genericRecords, double fractionConsumed)
+ throws Exception {
GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<>(schema);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
Encoder binaryEncoder = ENCODER_FACTORY.binaryEncoder(outputStream, null);
@@ -502,6 +544,19 @@
AvroRows.newBuilder()
.setSerializedBinaryRows(ByteString.copyFrom(outputStream.toByteArray()))
.setRowCount(genericRecords.size()))
+ .setStatus(
+ StreamStatus.newBuilder()
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFractionConsumed().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(
+ java.lang.Float.floatToIntBits((float) fractionConsumed))
+ .build())
+ .build()))
.build();
}
@@ -561,6 +616,12 @@
.setParent("projects/" + options.getProject())
.setTableReference(BigQueryHelpers.toTableRefProto(tempTableReference))
.setRequestedStreams(10)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession emptyReadSession = ReadSession.newBuilder().build();
@@ -674,8 +735,8 @@
List<ReadRowsResponse> readRowsResponses =
Lists.newArrayList(
- createResponse(AVRO_SCHEMA, records.subList(0, 2)),
- createResponse(AVRO_SCHEMA, records.subList(2, 4)));
+ createResponse(AVRO_SCHEMA, records.subList(0, 2), 0.500),
+ createResponse(AVRO_SCHEMA, records.subList(2, 4), 0.875));
//
// Note that since the temporary table name is generated by the pipeline, we can't match the
@@ -685,7 +746,8 @@
StorageClient fakeStorageClient = mock(StorageClient.class, withSettings().serializable());
when(fakeStorageClient.createReadSession(any())).thenReturn(readSession);
- when(fakeStorageClient.readRows(expectedReadRowsRequest)).thenReturn(readRowsResponses);
+ when(fakeStorageClient.readRows(expectedReadRowsRequest))
+ .thenReturn(new FakeBigQueryServerStream<>(readRowsResponses));
BigQueryIO.TypedRead<KV<String, Long>> typedRead =
BigQueryIO.read(new ParseKeyValue())
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadIT.java
index d565cb5..3b61b65 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadIT.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
index 3de15f3..e6f8eeb 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
@@ -23,12 +23,15 @@
import static org.hamcrest.Matchers.hasItem;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import static org.mockito.Mockito.withSettings;
+import com.google.api.gax.grpc.GrpcStatusCode;
+import com.google.api.gax.rpc.FailedPreconditionException;
import com.google.api.services.bigquery.model.Streamingbuffer;
import com.google.api.services.bigquery.model.Table;
import com.google.api.services.bigquery.model.TableFieldSchema;
@@ -42,9 +45,16 @@
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsRequest;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadRowsResponse;
import com.google.cloud.bigquery.storage.v1beta1.Storage.ReadSession;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamRequest;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.SplitReadStreamResponse;
import com.google.cloud.bigquery.storage.v1beta1.Storage.Stream;
import com.google.cloud.bigquery.storage.v1beta1.Storage.StreamPosition;
+import com.google.cloud.bigquery.storage.v1beta1.Storage.StreamStatus;
import com.google.protobuf.ByteString;
+import com.google.protobuf.UnknownFieldSet;
+import io.grpc.Status;
+import io.grpc.Status.Code;
+import io.grpc.StatusRuntimeException;
import java.io.ByteArrayOutputStream;
import java.math.BigInteger;
import java.util.ArrayList;
@@ -67,8 +77,12 @@
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.StorageClient;
+import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices;
+import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices.FakeBigQueryServerStream;
+import org.apache.beam.sdk.io.gcp.testing.FakeDatasetService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.SerializableFunction;
@@ -76,8 +90,8 @@
import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -92,6 +106,7 @@
/** Tests for {@link BigQueryIO#readTableRows() using {@link Method#DIRECT_READ}}. */
@RunWith(JUnit4.class)
public class BigQueryIOStorageReadTest {
+
private transient PipelineOptions options;
private transient TemporaryFolder testFolder = new TemporaryFolder();
private transient TestPipeline p;
@@ -239,6 +254,34 @@
}
@Test
+ public void testBuildSourceWithReadOptionsAndSelectedFields() {
+ thrown.expect(IllegalStateException.class);
+ thrown.expectMessage("withReadOptions() already called");
+ p.apply(
+ "ReadMyTable",
+ BigQueryIO.read(new TableRowParser())
+ .withCoder(TableRowJsonCoder.of())
+ .withMethod(Method.DIRECT_READ)
+ .from("foo.com:project:dataset.table")
+ .withReadOptions(TableReadOptions.newBuilder().build())
+ .withSelectedFields(Lists.newArrayList("field1")));
+ }
+
+ @Test
+ public void testBuildSourceWithReadOptionsAndRowRestriction() {
+ thrown.expect(IllegalStateException.class);
+ thrown.expectMessage("withReadOptions() already called");
+ p.apply(
+ "ReadMyTable",
+ BigQueryIO.read(new TableRowParser())
+ .withCoder(TableRowJsonCoder.of())
+ .withMethod(Method.DIRECT_READ)
+ .from("foo.com:project:dataset.table")
+ .withReadOptions(TableReadOptions.newBuilder().build())
+ .withRowRestriction("field > 1"));
+ }
+
+ @Test
public void testDisplayData() {
String tableSpec = "foo.com:project:dataset.table";
BigQueryIO.TypedRead<TableRow> typedRead =
@@ -313,6 +356,8 @@
BigQueryStorageTableSource.create(
ValueProvider.StaticValueProvider.of(tableRef),
null,
+ null,
+ null,
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices().withDatasetService(fakeDatasetService));
@@ -331,6 +376,8 @@
BigQueryStorageTableSource.create(
ValueProvider.StaticValueProvider.of(BigQueryHelpers.parseTableSpec("dataset.table")),
null,
+ null,
+ null,
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices().withDatasetService(fakeDatasetService));
@@ -370,6 +417,12 @@
.setParent("projects/project-id")
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(streamCount)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession.Builder builder = ReadSession.newBuilder();
@@ -384,6 +437,8 @@
BigQueryStorageTableSource.create(
ValueProvider.StaticValueProvider.of(tableRef),
null,
+ null,
+ null,
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices()
@@ -425,6 +480,12 @@
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(10)
.setReadOptions(readOptions)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession.Builder builder = ReadSession.newBuilder();
@@ -439,6 +500,71 @@
BigQueryStorageTableSource.create(
ValueProvider.StaticValueProvider.of(tableRef),
readOptions,
+ null,
+ null,
+ new TableRowParser(),
+ TableRowJsonCoder.of(),
+ new FakeBigQueryServices()
+ .withDatasetService(fakeDatasetService)
+ .withStorageClient(fakeStorageClient));
+
+ List<? extends BoundedSource<TableRow>> sources = tableSource.split(10L, options);
+ assertEquals(10L, sources.size());
+ }
+
+ @Test
+ public void testTableSourceInitialSplit_WithSelectedFieldsAndRowRestriction() throws Exception {
+ fakeDatasetService.createDataset("foo.com:project", "dataset", "", "", null);
+ TableReference tableRef = BigQueryHelpers.parseTableSpec("foo.com:project:dataset.table");
+
+ Table table =
+ new Table()
+ .setTableReference(tableRef)
+ .setNumBytes(100L)
+ .setSchema(
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema().setName("name").setType("STRING"),
+ new TableFieldSchema().setName("number").setType("INTEGER"))));
+
+ fakeDatasetService.createTable(table);
+
+ TableReadOptions readOptions =
+ TableReadOptions.newBuilder()
+ .addSelectedFields("name")
+ .addSelectedFields("number")
+ .setRowRestriction("number > 5")
+ .build();
+
+ CreateReadSessionRequest expectedRequest =
+ CreateReadSessionRequest.newBuilder()
+ .setParent("projects/project-id")
+ .setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
+ .setRequestedStreams(10)
+ .setReadOptions(readOptions)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
+ .build();
+
+ ReadSession.Builder builder = ReadSession.newBuilder();
+ for (int i = 0; i < 10; i++) {
+ builder.addStreams(Stream.newBuilder().setName("stream-" + i));
+ }
+
+ StorageClient fakeStorageClient = mock(StorageClient.class);
+ when(fakeStorageClient.createReadSession(expectedRequest)).thenReturn(builder.build());
+
+ BigQueryStorageTableSource<TableRow> tableSource =
+ BigQueryStorageTableSource.create(
+ ValueProvider.StaticValueProvider.of(tableRef),
+ null,
+ StaticValueProvider.of(Lists.newArrayList("name", "number")),
+ StaticValueProvider.of("number > 5"),
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices()
@@ -467,6 +593,12 @@
.setParent("projects/project-id")
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(1024)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession.Builder builder = ReadSession.newBuilder();
@@ -481,6 +613,8 @@
BigQueryStorageTableSource.create(
ValueProvider.StaticValueProvider.of(BigQueryHelpers.parseTableSpec("dataset.table")),
null,
+ null,
+ null,
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices()
@@ -509,6 +643,12 @@
.setParent("projects/project-id")
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(1024)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession emptyReadSession = ReadSession.newBuilder().build();
@@ -519,6 +659,8 @@
BigQueryStorageTableSource.create(
ValueProvider.StaticValueProvider.of(tableRef),
null,
+ null,
+ null,
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices()
@@ -536,6 +678,8 @@
ValueProvider.StaticValueProvider.of(
BigQueryHelpers.parseTableSpec("foo.com:project:dataset.table")),
null,
+ null,
+ null,
new TableRowParser(),
TableRowJsonCoder.of(),
new FakeBigQueryServices().withDatasetService(fakeDatasetService));
@@ -574,7 +718,8 @@
private static final EncoderFactory ENCODER_FACTORY = EncoderFactory.get();
private static ReadRowsResponse createResponse(
- Schema schema, Collection<GenericRecord> genericRecords) throws Exception {
+ Schema schema, Collection<GenericRecord> genericRecords, double fractionConsumed)
+ throws Exception {
GenericDatumWriter<GenericRecord> writer = new GenericDatumWriter<>(schema);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
Encoder binaryEncoder = ENCODER_FACTORY.binaryEncoder(outputStream, null);
@@ -589,6 +734,19 @@
AvroRows.newBuilder()
.setSerializedBinaryRows(ByteString.copyFrom(outputStream.toByteArray()))
.setRowCount(genericRecords.size()))
+ .setStatus(
+ StreamStatus.newBuilder()
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFractionConsumed().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(
+ java.lang.Float.floatToIntBits((float) fractionConsumed))
+ .build())
+ .build()))
.build();
}
@@ -623,40 +781,6 @@
}
@Test
- public void testStreamSourceGetMaxEndOffset() throws Exception {
-
- BigQueryStorageStreamSource<TableRow> streamSource =
- BigQueryStorageStreamSource.create(
- ReadSession.getDefaultInstance(),
- Stream.getDefaultInstance(),
- TABLE_SCHEMA,
- new TableRowParser(),
- TableRowJsonCoder.of(),
- new FakeBigQueryServices());
-
- thrown.expect(UnsupportedOperationException.class);
- thrown.expectMessage("Not implemented");
- streamSource.getMaxEndOffset(options);
- }
-
- @Test
- public void testStreamSourceCreateSouceForSubrange() throws Exception {
-
- BigQueryStorageStreamSource<TableRow> streamSource =
- BigQueryStorageStreamSource.create(
- ReadSession.getDefaultInstance(),
- Stream.getDefaultInstance(),
- TABLE_SCHEMA,
- new TableRowParser(),
- TableRowJsonCoder.of(),
- new FakeBigQueryServices());
-
- thrown.expect(UnsupportedOperationException.class);
- thrown.expectMessage("Not implemented");
- streamSource.createSourceForSubrange(0, 0);
- }
-
- @Test
public void testReadFromStreamSource() throws Exception {
ReadSession readSession =
@@ -680,11 +804,12 @@
List<ReadRowsResponse> responses =
Lists.newArrayList(
- createResponse(AVRO_SCHEMA, records.subList(0, 2)),
- createResponse(AVRO_SCHEMA, records.subList(2, 3)));
+ createResponse(AVRO_SCHEMA, records.subList(0, 2), 0.50),
+ createResponse(AVRO_SCHEMA, records.subList(2, 3), 0.75));
StorageClient fakeStorageClient = mock(StorageClient.class);
- when(fakeStorageClient.readRows(expectedRequest)).thenReturn(responses);
+ when(fakeStorageClient.readRows(expectedRequest))
+ .thenReturn(new FakeBigQueryServerStream<>(responses));
BigQueryStorageStreamSource<TableRow> streamSource =
BigQueryStorageStreamSource.create(
@@ -707,8 +832,7 @@
}
@Test
- public void testStreamSourceSplitAtFraction() throws Exception {
-
+ public void testFractionConsumed() throws Exception {
ReadSession readSession =
ReadSession.newBuilder()
.setName("readSession")
@@ -726,15 +850,23 @@
Lists.newArrayList(
createRecord("A", 1, AVRO_SCHEMA),
createRecord("B", 2, AVRO_SCHEMA),
- createRecord("C", 3, AVRO_SCHEMA));
+ createRecord("C", 3, AVRO_SCHEMA),
+ createRecord("D", 4, AVRO_SCHEMA),
+ createRecord("E", 5, AVRO_SCHEMA),
+ createRecord("F", 6, AVRO_SCHEMA),
+ createRecord("G", 7, AVRO_SCHEMA));
List<ReadRowsResponse> responses =
Lists.newArrayList(
- createResponse(AVRO_SCHEMA, records.subList(0, 2)),
- createResponse(AVRO_SCHEMA, records.subList(2, 3)));
+ // N.B.: All floating point numbers used in this test can be represented without
+ // a loss of precision.
+ createResponse(AVRO_SCHEMA, records.subList(0, 2), 0.250),
+ createResponse(AVRO_SCHEMA, records.subList(2, 4), 0.500),
+ createResponse(AVRO_SCHEMA, records.subList(4, 7), 0.875));
StorageClient fakeStorageClient = mock(StorageClient.class);
- when(fakeStorageClient.readRows(expectedRequest)).thenReturn(responses);
+ when(fakeStorageClient.readRows(expectedRequest))
+ .thenReturn(new FakeBigQueryServerStream<>(responses));
BigQueryStorageStreamSource<TableRow> streamSource =
BigQueryStorageStreamSource.create(
@@ -745,13 +877,574 @@
TableRowJsonCoder.of(),
new FakeBigQueryServices().withStorageClient(fakeStorageClient));
+ List<TableRow> rows = new ArrayList<>();
BoundedReader<TableRow> reader = streamSource.createReader(options);
- reader.start();
- assertNull(reader.splitAtFraction(0.5));
+
+ // Before call to BoundedReader#start, fraction consumed must be zero.
+ assertEquals(Double.valueOf(0.000), reader.getFractionConsumed());
+
+ assertTrue(reader.start()); // Reads A.
+ assertEquals(Double.valueOf(0.125), reader.getFractionConsumed());
+ assertTrue(reader.advance()); // Reads B.
+ assertEquals(Double.valueOf(0.250), reader.getFractionConsumed());
+
+ assertTrue(reader.advance()); // Reads C.
+ assertEquals(Double.valueOf(0.375), reader.getFractionConsumed());
+ assertTrue(reader.advance()); // Reads D.
+ assertEquals(Double.valueOf(0.500), reader.getFractionConsumed());
+
+ assertTrue(reader.advance()); // Reads E.
+ assertEquals(Double.valueOf(0.625), reader.getFractionConsumed());
+ assertTrue(reader.advance()); // Reads F.
+ assertEquals(Double.valueOf(0.750), reader.getFractionConsumed());
+ assertTrue(reader.advance()); // Reads G.
+ assertEquals(Double.valueOf(0.875), reader.getFractionConsumed());
+
+ assertFalse(reader.advance()); // Reaches the end.
+
+ // We are done with the stream, so we should report 100% consumption.
+ assertEquals(Double.valueOf(1.00), reader.getFractionConsumed());
+ }
+
+ @Test
+ public void testFractionConsumedWithSplit() throws Exception {
+ ReadSession readSession =
+ ReadSession.newBuilder()
+ .setName("readSession")
+ .setAvroSchema(AvroSchema.newBuilder().setSchema(AVRO_SCHEMA_STRING))
+ .build();
+
+ Stream parentStream = Stream.newBuilder().setName("stream").build();
+
+ ReadRowsRequest expectedRequest =
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(parentStream))
+ .build();
+
+ List<GenericRecord> records =
+ Lists.newArrayList(
+ createRecord("A", 1, AVRO_SCHEMA),
+ createRecord("B", 2, AVRO_SCHEMA),
+ createRecord("C", 3, AVRO_SCHEMA),
+ createRecord("D", 4, AVRO_SCHEMA),
+ createRecord("E", 5, AVRO_SCHEMA),
+ createRecord("F", 6, AVRO_SCHEMA),
+ createRecord("G", 7, AVRO_SCHEMA));
+
+ List<ReadRowsResponse> parentResponses =
+ Lists.newArrayList(
+ // N.B.: All floating point numbers used in this test can be represented without
+ // a loss of precision.
+ createResponse(AVRO_SCHEMA, records.subList(0, 2), 0.250),
+ createResponse(AVRO_SCHEMA, records.subList(2, 4), 0.500),
+ createResponse(AVRO_SCHEMA, records.subList(4, 7), 0.875));
+
+ StorageClient fakeStorageClient = mock(StorageClient.class);
+ when(fakeStorageClient.readRows(expectedRequest))
+ .thenReturn(new FakeBigQueryServerStream<>(parentResponses));
+
+ when(fakeStorageClient.splitReadStream(
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(parentStream)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits(0.5f))
+ .build())
+ .build())
+ .build()))
+ .thenReturn(
+ SplitReadStreamResponse.newBuilder()
+ .setPrimaryStream(Stream.newBuilder().setName("primary"))
+ .setRemainderStream(Stream.newBuilder().setName("residual"))
+ .build());
+
+ List<ReadRowsResponse> primaryResponses =
+ Lists.newArrayList(
+ createResponse(AVRO_SCHEMA, records.subList(1, 3), 0.500),
+ createResponse(AVRO_SCHEMA, records.subList(3, 4), 0.875));
+
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(
+ StreamPosition.newBuilder()
+ .setStream(Stream.newBuilder().setName("primary"))
+ .setOffset(1))
+ .build()))
+ .thenReturn(new FakeBigQueryServerStream<>(primaryResponses));
+
+ BigQueryStorageStreamSource<TableRow> streamSource =
+ BigQueryStorageStreamSource.create(
+ readSession,
+ parentStream,
+ TABLE_SCHEMA,
+ new TableRowParser(),
+ TableRowJsonCoder.of(),
+ new FakeBigQueryServices().withStorageClient(fakeStorageClient));
+
+ List<TableRow> rows = new ArrayList<>();
+ BoundedReader<TableRow> reader = streamSource.createReader(options);
+
+ // Before call to BoundedReader#start, fraction consumed must be zero.
+ assertEquals(Double.valueOf(0.0000), reader.getFractionConsumed());
+
+ assertTrue(reader.start()); // Reads A.
+ assertEquals(Double.valueOf(0.1250), reader.getFractionConsumed());
+
+ reader.splitAtFraction(0.5f);
+ assertEquals(Double.valueOf(0.1250), reader.getFractionConsumed());
+
+ assertTrue(reader.advance()); // Reads B.
+
+ // Once the split has completed but no new rows have been read, the consumed value is at the
+ // last calculated value of 0.125. For the first response of the primary stream, the progress
+ // report interpolation is done between the progress before split and the progress from the
+ // first response of the primary stream. In this case, the value is:
+ //
+ // 0.125 + (0.5 - 0.125) * 1.0 / 2
+ //
+ assertEquals(Double.valueOf(0.3125), reader.getFractionConsumed());
+
+ assertTrue(reader.advance()); // Reads C.
+ assertEquals(Double.valueOf(0.5000), reader.getFractionConsumed());
+
+ assertTrue(reader.advance()); // Reads D.
+ assertEquals(Double.valueOf(0.8750), reader.getFractionConsumed());
+
+ assertFalse(reader.advance());
+ assertEquals(Double.valueOf(1.0000), reader.getFractionConsumed());
+ }
+
+ @Test
+ public void testStreamSourceSplitAtFractionSucceeds() throws Exception {
+ Stream parentStream = Stream.newBuilder().setName("parent").build();
+
+ List<ReadRowsResponse> parentResponses =
+ Lists.newArrayList(
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("A", 1, AVRO_SCHEMA), createRecord("B", 2, AVRO_SCHEMA)),
+ 0.25),
+ createResponse(
+ AVRO_SCHEMA, Lists.newArrayList(createRecord("C", 3, AVRO_SCHEMA)), 0.50),
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("D", 4, AVRO_SCHEMA), createRecord("E", 5, AVRO_SCHEMA)),
+ 0.75));
+
+ StorageClient fakeStorageClient = mock(StorageClient.class);
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(parentStream))
+ .build()))
+ .thenReturn(new FakeBigQueryServerStream<>(parentResponses));
+
+ // Mocks the split call.
+ when(fakeStorageClient.splitReadStream(
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(parentStream)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits(0.5f))
+ .build())
+ .build())
+ .build()))
+ .thenReturn(
+ SplitReadStreamResponse.newBuilder()
+ .setPrimaryStream(Stream.newBuilder().setName("primary"))
+ .setRemainderStream(Stream.newBuilder().setName("residual"))
+ .build());
+
+ // Mocks the ReadRows calls expected on the primary and residual streams.
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(
+ StreamPosition.newBuilder()
+ .setStream(Stream.newBuilder().setName("primary"))
+ // This test will read rows 0 and 1 from the parent before calling split,
+ // so we expect the primary read to start at offset 2.
+ .setOffset(2))
+ .build()))
+ .thenReturn(new FakeBigQueryServerStream<>(parentResponses.subList(1, 2)));
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(
+ StreamPosition.newBuilder()
+ .setStream(Stream.newBuilder().setName("residual"))
+ .setOffset(0))
+ .build()))
+ .thenReturn(
+ new FakeBigQueryServerStream<>(parentResponses.subList(2, parentResponses.size())));
+
+ BigQueryStorageStreamSource<TableRow> streamSource =
+ BigQueryStorageStreamSource.create(
+ ReadSession.newBuilder()
+ .setName("readSession")
+ .setAvroSchema(AvroSchema.newBuilder().setSchema(AVRO_SCHEMA_STRING))
+ .build(),
+ parentStream,
+ TABLE_SCHEMA,
+ new TableRowParser(),
+ TableRowJsonCoder.of(),
+ new FakeBigQueryServices().withStorageClient(fakeStorageClient));
+
+ // Read a few records from the parent stream and ensure that records are returned in the
+ // prescribed order.
+ BoundedReader<TableRow> parent = streamSource.createReader(options);
+ assertTrue(parent.start());
+ assertEquals("A", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("B", parent.getCurrent().get("name"));
+
+ // Now split the stream, and ensure that the "parent" reader has been replaced with the
+ // primary stream and that the returned source points to the residual stream.
+ BoundedReader<TableRow> primary = parent;
+ BoundedSource<TableRow> residualSource = parent.splitAtFraction(0.5);
+ assertNotNull(residualSource);
+ BoundedReader<TableRow> residual = residualSource.createReader(options);
+
+ assertTrue(primary.advance());
+ assertEquals("C", primary.getCurrent().get("name"));
+ assertFalse(primary.advance());
+
+ assertTrue(residual.start());
+ assertEquals("D", residual.getCurrent().get("name"));
+ assertTrue(residual.advance());
+ assertEquals("E", residual.getCurrent().get("name"));
+ assertFalse(residual.advance());
+ }
+
+ @Test
+ public void testStreamSourceSplitAtFractionRepeated() throws Exception {
+ List<Stream> streams =
+ Lists.newArrayList(
+ Stream.newBuilder().setName("stream1").build(),
+ Stream.newBuilder().setName("stream2").build(),
+ Stream.newBuilder().setName("stream3").build());
+
+ StorageClient fakeStorageClient = mock(StorageClient.class);
+
+ // Mock the initial ReadRows call.
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(streams.get(0)))
+ .build()))
+ .thenReturn(
+ new FakeBigQueryServerStream<>(
+ Lists.newArrayList(
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("A", 1, AVRO_SCHEMA), createRecord("B", 2, AVRO_SCHEMA)),
+ 0.25),
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("C", 3, AVRO_SCHEMA), createRecord("D", 4, AVRO_SCHEMA)),
+ 0.50),
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("E", 5, AVRO_SCHEMA), createRecord("F", 6, AVRO_SCHEMA)),
+ 0.75))));
+
+ // Mock the first SplitReadStream call.
+ when(fakeStorageClient.splitReadStream(
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(streams.get(0))
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits(0.83f))
+ .build())
+ .build())
+ .build()))
+ .thenReturn(
+ SplitReadStreamResponse.newBuilder()
+ .setPrimaryStream(streams.get(1))
+ .setRemainderStream(Stream.newBuilder().setName("ignored"))
+ .build());
+
+ // Mock the second ReadRows call.
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(streams.get(1)).setOffset(1))
+ .build()))
+ .thenReturn(
+ new FakeBigQueryServerStream<>(
+ Lists.newArrayList(
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("B", 2, AVRO_SCHEMA), createRecord("C", 3, AVRO_SCHEMA)),
+ 0.50),
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("D", 4, AVRO_SCHEMA), createRecord("E", 5, AVRO_SCHEMA)),
+ 0.75))));
+
+ // Mock the second SplitReadStream call.
+ when(fakeStorageClient.splitReadStream(
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(streams.get(1))
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits(0.75f))
+ .build())
+ .build())
+ .build()))
+ .thenReturn(
+ SplitReadStreamResponse.newBuilder()
+ .setPrimaryStream(streams.get(2))
+ .setRemainderStream(Stream.newBuilder().setName("ignored"))
+ .build());
+
+ // Mock the third ReadRows call.
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(streams.get(2)).setOffset(2))
+ .build()))
+ .thenReturn(
+ new FakeBigQueryServerStream<>(
+ Lists.newArrayList(
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("C", 3, AVRO_SCHEMA), createRecord("D", 4, AVRO_SCHEMA)),
+ 0.90))));
+
+ BoundedSource<TableRow> source =
+ BigQueryStorageStreamSource.create(
+ ReadSession.newBuilder()
+ .setName("readSession")
+ .setAvroSchema(AvroSchema.newBuilder().setSchema(AVRO_SCHEMA_STRING))
+ .build(),
+ streams.get(0),
+ TABLE_SCHEMA,
+ new TableRowParser(),
+ TableRowJsonCoder.of(),
+ new FakeBigQueryServices().withStorageClient(fakeStorageClient));
+
+ BoundedReader<TableRow> reader = source.createReader(options);
+ assertTrue(reader.start());
+ assertEquals("A", reader.getCurrent().get("name"));
+
+ BoundedSource<TableRow> residualSource = reader.splitAtFraction(0.83f);
+ assertNotNull(residualSource);
+ assertEquals("A", reader.getCurrent().get("name"));
+
+ assertTrue(reader.advance());
+ assertEquals("B", reader.getCurrent().get("name"));
+
+ residualSource = reader.splitAtFraction(0.75f);
+ assertNotNull(residualSource);
+ assertEquals("B", reader.getCurrent().get("name"));
+
+ assertTrue(reader.advance());
+ assertEquals("C", reader.getCurrent().get("name"));
+ assertTrue(reader.advance());
+ assertEquals("D", reader.getCurrent().get("name"));
+ assertFalse(reader.advance());
+ }
+
+ @Test
+ public void testStreamSourceSplitAtFractionFailsWhenSplitIsNotPossible() throws Exception {
+ Stream parentStream = Stream.newBuilder().setName("parent").build();
+
+ List<ReadRowsResponse> parentResponses =
+ Lists.newArrayList(
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("A", 1, AVRO_SCHEMA), createRecord("B", 2, AVRO_SCHEMA)),
+ 0.25),
+ createResponse(
+ AVRO_SCHEMA, Lists.newArrayList(createRecord("C", 3, AVRO_SCHEMA)), 0.50),
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("D", 4, AVRO_SCHEMA), createRecord("E", 5, AVRO_SCHEMA)),
+ 0.75));
+
+ StorageClient fakeStorageClient = mock(StorageClient.class);
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(parentStream))
+ .build()))
+ .thenReturn(new FakeBigQueryServerStream<>(parentResponses));
+
+ // Mocks the split call. A response without a primary_stream and remainder_stream means
+ // that the split is not possible.
+ when(fakeStorageClient.splitReadStream(
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(parentStream)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits(0.5f))
+ .build())
+ .build())
+ .build()))
+ .thenReturn(SplitReadStreamResponse.getDefaultInstance());
+
+ BigQueryStorageStreamSource<TableRow> streamSource =
+ BigQueryStorageStreamSource.create(
+ ReadSession.newBuilder()
+ .setName("readSession")
+ .setAvroSchema(AvroSchema.newBuilder().setSchema(AVRO_SCHEMA_STRING))
+ .build(),
+ parentStream,
+ TABLE_SCHEMA,
+ new TableRowParser(),
+ TableRowJsonCoder.of(),
+ new FakeBigQueryServices().withStorageClient(fakeStorageClient));
+
+ // Read a few records from the parent stream and ensure that records are returned in the
+ // prescribed order.
+ BoundedReader<TableRow> parent = streamSource.createReader(options);
+ assertTrue(parent.start());
+ assertEquals("A", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("B", parent.getCurrent().get("name"));
+
+ assertNull(parent.splitAtFraction(0.5));
+
+ // Verify that the parent source still works okay even after an unsuccessful split attempt.
+ assertTrue(parent.advance());
+ assertEquals("C", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("D", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("E", parent.getCurrent().get("name"));
+ assertFalse(parent.advance());
+ }
+
+ @Test
+ public void testStreamSourceSplitAtFractionFailsWhenParentIsPastSplitPoint() throws Exception {
+ Stream parentStream = Stream.newBuilder().setName("parent").build();
+
+ List<ReadRowsResponse> parentResponses =
+ Lists.newArrayList(
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("A", 1, AVRO_SCHEMA), createRecord("B", 2, AVRO_SCHEMA)),
+ 0.25),
+ createResponse(
+ AVRO_SCHEMA, Lists.newArrayList(createRecord("C", 3, AVRO_SCHEMA)), 0.50),
+ createResponse(
+ AVRO_SCHEMA,
+ Lists.newArrayList(
+ createRecord("D", 4, AVRO_SCHEMA), createRecord("E", 5, AVRO_SCHEMA)),
+ 0.75));
+
+ StorageClient fakeStorageClient = mock(StorageClient.class);
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(StreamPosition.newBuilder().setStream(parentStream))
+ .build()))
+ .thenReturn(new FakeBigQueryServerStream<>(parentResponses));
+
+ // Mocks the split call. A response without a primary_stream and remainder_stream means
+ // that the split is not possible.
+ // Mocks the split call.
+ when(fakeStorageClient.splitReadStream(
+ SplitReadStreamRequest.newBuilder()
+ .setOriginalStream(parentStream)
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setFraction().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(
+ 2,
+ UnknownFieldSet.Field.newBuilder()
+ .addFixed32(java.lang.Float.floatToIntBits(0.5f))
+ .build())
+ .build())
+ .build()))
+ .thenReturn(
+ SplitReadStreamResponse.newBuilder()
+ .setPrimaryStream(Stream.newBuilder().setName("primary"))
+ .setRemainderStream(Stream.newBuilder().setName("residual"))
+ .build());
+
+ // Mocks the ReadRows calls expected on the primary and residual streams.
+ when(fakeStorageClient.readRows(
+ ReadRowsRequest.newBuilder()
+ .setReadPosition(
+ StreamPosition.newBuilder()
+ .setStream(Stream.newBuilder().setName("primary"))
+ // This test will read rows 0 and 1 from the parent before calling split,
+ // so we expect the primary read to start at offset 2.
+ .setOffset(2))
+ .build()))
+ .thenThrow(
+ new FailedPreconditionException(
+ "Given row offset is invalid for stream.",
+ new StatusRuntimeException(Status.FAILED_PRECONDITION),
+ GrpcStatusCode.of(Code.FAILED_PRECONDITION),
+ /* retryable = */ false));
+
+ BigQueryStorageStreamSource<TableRow> streamSource =
+ BigQueryStorageStreamSource.create(
+ ReadSession.newBuilder()
+ .setName("readSession")
+ .setAvroSchema(AvroSchema.newBuilder().setSchema(AVRO_SCHEMA_STRING))
+ .build(),
+ parentStream,
+ TABLE_SCHEMA,
+ new TableRowParser(),
+ TableRowJsonCoder.of(),
+ new FakeBigQueryServices().withStorageClient(fakeStorageClient));
+
+ // Read a few records from the parent stream and ensure that records are returned in the
+ // prescribed order.
+ BoundedReader<TableRow> parent = streamSource.createReader(options);
+ assertTrue(parent.start());
+ assertEquals("A", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("B", parent.getCurrent().get("name"));
+
+ assertNull(parent.splitAtFraction(0.5));
+
+ // Verify that the parent source still works okay even after an unsuccessful split attempt.
+ assertTrue(parent.advance());
+ assertEquals("C", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("D", parent.getCurrent().get("name"));
+ assertTrue(parent.advance());
+ assertEquals("E", parent.getCurrent().get("name"));
+ assertFalse(parent.advance());
}
private static final class ParseKeyValue
implements SerializableFunction<SchemaAndRecord, KV<String, Long>> {
+
@Override
public KV<String, Long> apply(SchemaAndRecord input) {
return KV.of(
@@ -774,6 +1467,14 @@
.setParent("projects/project-id")
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(10)
+ .setReadOptions(
+ TableReadOptions.newBuilder().addSelectedFields("name").addSelectedFields("number"))
+ // TODO(aryann): Once we rebuild the generated client code, we should change this to
+ // use setShardingStrategy().
+ .setUnknownFields(
+ UnknownFieldSet.newBuilder()
+ .addField(7, UnknownFieldSet.Field.newBuilder().addVarint(2).build())
+ .build())
.build();
ReadSession readSession =
@@ -798,19 +1499,21 @@
List<ReadRowsResponse> readRowsResponses =
Lists.newArrayList(
- createResponse(AVRO_SCHEMA, records.subList(0, 2)),
- createResponse(AVRO_SCHEMA, records.subList(2, 4)));
+ createResponse(AVRO_SCHEMA, records.subList(0, 2), 0.50),
+ createResponse(AVRO_SCHEMA, records.subList(2, 4), 0.75));
StorageClient fakeStorageClient = mock(StorageClient.class, withSettings().serializable());
when(fakeStorageClient.createReadSession(expectedCreateReadSessionRequest))
.thenReturn(readSession);
- when(fakeStorageClient.readRows(expectedReadRowsRequest)).thenReturn(readRowsResponses);
+ when(fakeStorageClient.readRows(expectedReadRowsRequest))
+ .thenReturn(new FakeBigQueryServerStream<>(readRowsResponses));
PCollection<KV<String, Long>> output =
p.apply(
BigQueryIO.read(new ParseKeyValue())
.from("foo.com:project:dataset.table")
.withMethod(Method.DIRECT_READ)
+ .withSelectedFields(p.newProvider(Lists.newArrayList("name", "number")))
.withTestServices(
new FakeBigQueryServices()
.withDatasetService(fakeDatasetService)
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
index a2ea13b..cd0312d 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
@@ -19,9 +19,9 @@
import static org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.toJsonString;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
@@ -34,6 +34,7 @@
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
+import com.google.api.services.bigquery.model.Clustering;
import com.google.api.services.bigquery.model.ErrorProto;
import com.google.api.services.bigquery.model.Table;
import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
@@ -61,7 +62,11 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method;
+import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices;
+import org.apache.beam.sdk.io.gcp.testing.FakeDatasetService;
+import org.apache.beam.sdk.io.gcp.testing.FakeJobService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.ValueProvider;
import org.apache.beam.sdk.schemas.JavaFieldSchema;
@@ -94,13 +99,13 @@
import org.apache.beam.sdk.values.ShardedKey;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.hamcrest.Matchers;
import org.joda.time.Duration;
import org.joda.time.Instant;
@@ -354,6 +359,62 @@
}
}
+ void testTimePartitioningClustering(
+ BigQueryIO.Write.Method insertMethod, boolean enablePartitioning, boolean enableClustering)
+ throws Exception {
+ TableRow row1 = new TableRow().set("date", "2018-01-01").set("number", "1");
+ TableRow row2 = new TableRow().set("date", "2018-01-02").set("number", "2");
+
+ TimePartitioning timePartitioning = new TimePartitioning().setType("DAY").setField("date");
+ Clustering clustering = new Clustering().setFields(ImmutableList.of("date"));
+ TableSchema schema =
+ new TableSchema()
+ .setFields(
+ ImmutableList.of(
+ new TableFieldSchema()
+ .setName("date")
+ .setType("DATE")
+ .setName("number")
+ .setType("INTEGER")));
+
+ Write<TableRow> writeTransform =
+ BigQueryIO.writeTableRows()
+ .to("project-id:dataset-id.table-id")
+ .withTestServices(fakeBqServices)
+ .withMethod(insertMethod)
+ .withSchema(schema)
+ .withoutValidation();
+
+ if (enablePartitioning) {
+ writeTransform = writeTransform.withTimePartitioning(timePartitioning);
+ }
+ if (enableClustering) {
+ writeTransform = writeTransform.withClustering(clustering);
+ }
+
+ p.apply(Create.of(row1, row2)).apply(writeTransform);
+ p.run();
+ Table table =
+ fakeDatasetService.getTable(
+ BigQueryHelpers.parseTableSpec("project-id:dataset-id.table-id"));
+
+ assertEquals(schema, table.getSchema());
+ if (enablePartitioning) {
+ assertEquals(timePartitioning, table.getTimePartitioning());
+ }
+ if (enableClustering) {
+ assertEquals(clustering, table.getClustering());
+ }
+ }
+
+ void testTimePartitioning(BigQueryIO.Write.Method insertMethod) throws Exception {
+ testTimePartitioningClustering(insertMethod, true, false);
+ }
+
+ void testClustering(BigQueryIO.Write.Method insertMethod) throws Exception {
+ testTimePartitioningClustering(insertMethod, true, true);
+ }
+
@Test
public void testTimePartitioningStreamingInserts() throws Exception {
testTimePartitioning(BigQueryIO.Write.Method.STREAMING_INSERTS);
@@ -364,31 +425,63 @@
testTimePartitioning(BigQueryIO.Write.Method.FILE_LOADS);
}
- public void testTimePartitioning(BigQueryIO.Write.Method insertMethod) throws Exception {
- TableRow row1 = new TableRow().set("name", "a").set("number", "1");
- TableRow row2 = new TableRow().set("name", "b").set("number", "2");
+ @Test
+ public void testClusteringStreamingInserts() throws Exception {
+ testClustering(BigQueryIO.Write.Method.STREAMING_INSERTS);
+ }
- TimePartitioning timePartitioning =
- new TimePartitioning().setType("DAY").setExpirationMs(1000L);
+ @Test
+ public void testClusteringBatchLoads() throws Exception {
+ testClustering(BigQueryIO.Write.Method.FILE_LOADS);
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void testClusteringThrowsWithoutPartitioning() throws Exception {
+ p.enableAbandonedNodeEnforcement(false);
+ testTimePartitioningClustering(Method.STREAMING_INSERTS, false, true);
+ }
+
+ @Test
+ public void testClusteringTableFunction() throws Exception {
+ TableRow row1 = new TableRow().set("date", "2018-01-01").set("number", "1");
+ TableRow row2 = new TableRow().set("date", "2018-01-02").set("number", "2");
+
+ TimePartitioning timePartitioning = new TimePartitioning().setType("DAY").setField("date");
+ Clustering clustering = new Clustering().setFields(ImmutableList.of("date"));
TableSchema schema =
new TableSchema()
.setFields(
- ImmutableList.of(new TableFieldSchema().setName("number").setType("INTEGER")));
+ ImmutableList.of(
+ new TableFieldSchema()
+ .setName("date")
+ .setType("DATE")
+ .setName("number")
+ .setType("INTEGER")));
p.apply(Create.of(row1, row2))
.apply(
BigQueryIO.writeTableRows()
- .to("project-id:dataset-id.table-id")
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) -> {
+ String tableSpec =
+ "project-id:dataset-id.table-" + vsw.getValue().get("number");
+ return new TableDestination(
+ tableSpec,
+ null,
+ new TimePartitioning().setType("DAY").setField("date"),
+ new Clustering().setFields(ImmutableList.of("date")));
+ })
.withTestServices(fakeBqServices)
- .withMethod(insertMethod)
+ .withMethod(BigQueryIO.Write.Method.FILE_LOADS)
.withSchema(schema)
- .withTimePartitioning(timePartitioning)
+ .withClustering()
.withoutValidation());
p.run();
Table table =
fakeDatasetService.getTable(
- BigQueryHelpers.parseTableSpec("project-id:dataset-id.table-id"));
+ BigQueryHelpers.parseTableSpec("project-id:dataset-id.table-1"));
assertEquals(schema, table.getSchema());
assertEquals(timePartitioning, table.getTimePartitioning());
+ assertEquals(clustering, table.getClustering());
}
@Test
@@ -967,7 +1060,7 @@
assertEquals(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED, write.getCreateDisposition());
assertEquals(BigQueryIO.Write.WriteDisposition.WRITE_EMPTY, write.getWriteDisposition());
assertEquals(null, write.getTableDescription());
- assertEquals(true, write.getValidate());
+ assertTrue(write.getValidate());
assertFalse(write.withoutValidation().getValidate());
TableSchema schema = new TableSchema();
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java
index 38ac61a..50610f3 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java
@@ -30,10 +30,12 @@
import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.sdk.testing.UsesKms;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.slf4j.Logger;
@@ -46,6 +48,7 @@
* used.
*/
@RunWith(JUnit4.class)
+@Category(UsesKms.class)
public class BigQueryKmsKeyIT {
private static final Logger LOG = LoggerFactory.getLogger(BigQueryKmsKeyIT.class);
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
index 29c9dcb..fb88bb4 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.gcp.bigquery;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verifyNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verifyNotNull;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.instanceOf;
import static org.hamcrest.Matchers.is;
@@ -77,8 +77,8 @@
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
import org.junit.Before;
import org.junit.Rule;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTimePartitioningClusteringIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTimePartitioningClusteringIT.java
new file mode 100644
index 0000000..0289a4d
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTimePartitioningClusteringIT.java
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.bigquery;
+
+import com.google.api.services.bigquery.Bigquery;
+import com.google.api.services.bigquery.model.Clustering;
+import com.google.api.services.bigquery.model.Table;
+import com.google.api.services.bigquery.model.TableFieldSchema;
+import com.google.api.services.bigquery.model.TableRow;
+import com.google.api.services.bigquery.model.TableSchema;
+import com.google.api.services.bigquery.model.TimePartitioning;
+import java.util.Arrays;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.gcp.testing.BigqueryClient;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.ExperimentalOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.ValueInSingleWindow;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Integration test that partitions and clusters sample data in BigQuery. */
+@RunWith(JUnit4.class)
+public class BigQueryTimePartitioningClusteringIT {
+ private static final String WEATHER_SAMPLES_TABLE =
+ "clouddataflow-readonly:samples.weather_stations";
+ private static final String DATASET_NAME = "BigQueryTimePartitioningIT";
+ private static final TimePartitioning TIME_PARTITIONING =
+ new TimePartitioning().setField("date").setType("DAY");
+ private static final Clustering CLUSTERING =
+ new Clustering().setFields(Arrays.asList("station_number"));
+ private static final TableSchema SCHEMA =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ new TableFieldSchema().setName("station_number").setType("INTEGER"),
+ new TableFieldSchema().setName("date").setType("DATE")));
+
+ private Bigquery bqClient;
+ private BigQueryClusteringITOptions options;
+
+ @Before
+ public void setUp() {
+ PipelineOptionsFactory.register(BigQueryClusteringITOptions.class);
+ options = TestPipeline.testingPipelineOptions().as(BigQueryClusteringITOptions.class);
+ options.setTempLocation(options.getTempRoot() + "/temp-it/");
+ bqClient = BigqueryClient.getNewBigquerryClient(options.getAppName());
+ }
+
+ /** Customized PipelineOptions for BigQueryClustering Integration Test. */
+ public interface BigQueryClusteringITOptions
+ extends TestPipelineOptions, ExperimentalOptions, BigQueryOptions {
+ @Description("Table to read from, specified as " + "<project_id>:<dataset_id>.<table_id>")
+ @Default.String(WEATHER_SAMPLES_TABLE)
+ String getBqcInput();
+
+ void setBqcInput(String value);
+ }
+
+ static class KeepStationNumberAndConvertDate extends DoFn<TableRow, TableRow> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ String day = (String) c.element().get("day");
+ String month = (String) c.element().get("month");
+ String year = (String) c.element().get("year");
+
+ TableRow row = new TableRow();
+ row.set("station_number", c.element().get("station_number"));
+ row.set("date", String.format("%s-%s-%s", year, month, day));
+ c.output(row);
+ }
+ }
+
+ static class ClusteredDestinations extends DynamicDestinations<TableRow, TableDestination> {
+ private final String tableName;
+
+ public ClusteredDestinations(String tableName) {
+ this.tableName = tableName;
+ }
+
+ @Nullable
+ @Override
+ public Coder<TableDestination> getDestinationCoder() {
+ return TableDestinationCoderV3.of();
+ }
+
+ @Override
+ public TableDestination getDestination(ValueInSingleWindow<TableRow> element) {
+ return new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName), null, TIME_PARTITIONING, CLUSTERING);
+ }
+
+ @Override
+ public TableDestination getTable(TableDestination destination) {
+ return destination;
+ }
+
+ @Override
+ public TableSchema getSchema(TableDestination destination) {
+ return SCHEMA;
+ }
+ }
+
+ @Test
+ public void testE2EBigQueryTimePartitioning() throws Exception {
+ String tableName = "weather_stations_time_partitioned_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(String.format("%s.%s", DATASET_NAME, tableName))
+ .withTimePartitioning(TIME_PARTITIONING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ bqClient = BigqueryClient.getNewBigquerryClient(options.getAppName());
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(table.getTimePartitioning(), TIME_PARTITIONING);
+ }
+
+ @Test
+ public void testE2EBigQueryClustering() throws Exception {
+ String tableName = "weather_stations_clustered_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(String.format("%s.%s", DATASET_NAME, tableName))
+ .withTimePartitioning(TIME_PARTITIONING)
+ .withClustering(CLUSTERING)
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(table.getClustering(), CLUSTERING);
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringTableFunction() throws Exception {
+ String tableName = "weather_stations_clustered_table_function_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(
+ (ValueInSingleWindow<TableRow> vsw) ->
+ new TableDestination(
+ String.format("%s.%s", DATASET_NAME, tableName),
+ null,
+ TIME_PARTITIONING,
+ CLUSTERING))
+ .withClustering()
+ .withSchema(SCHEMA)
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(table.getClustering(), CLUSTERING);
+ }
+
+ @Test
+ public void testE2EBigQueryClusteringDynamicDestinations() throws Exception {
+ String tableName =
+ "weather_stations_clustered_dynamic_destinations_" + System.currentTimeMillis();
+
+ Pipeline p = Pipeline.create(options);
+
+ p.apply(BigQueryIO.readTableRows().from(options.getBqcInput()))
+ .apply(ParDo.of(new KeepStationNumberAndConvertDate()))
+ .apply(
+ BigQueryIO.writeTableRows()
+ .to(new ClusteredDestinations(tableName))
+ .withClustering()
+ .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
+ .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
+
+ p.run().waitUntilFinish();
+
+ Table table = bqClient.tables().get(options.getProject(), DATASET_NAME, tableName).execute();
+
+ Assert.assertEquals(table.getClustering(), CLUSTERING);
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryToTableIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryToTableIT.java
index 596108d..79266d7 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryToTableIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryToTableIT.java
@@ -53,8 +53,8 @@
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.util.FluentBackoff;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.junit.AfterClass;
import org.junit.BeforeClass;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
index a23a3ec..a215635 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
@@ -24,6 +24,7 @@
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.nullValue;
import static org.hamcrest.collection.IsMapContaining.hasEntry;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
import static org.junit.Assert.assertThrows;
@@ -31,12 +32,18 @@
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;
import java.util.Arrays;
+import java.util.Base64;
+import java.util.Collections;
import java.util.List;
+import org.apache.avro.generic.GenericData;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.ConversionOptions.TruncateTimestamps;
import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
import org.apache.beam.sdk.values.Row;
import org.joda.time.DateTime;
import org.joda.time.Instant;
+import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.ISODateTimeFormat;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -49,8 +56,12 @@
.addNullableField("id", Schema.FieldType.INT64)
.addNullableField("value", Schema.FieldType.DOUBLE)
.addNullableField("name", Schema.FieldType.STRING)
- .addNullableField("timestamp", Schema.FieldType.DATETIME)
+ .addNullableField("timestamp_variant1", Schema.FieldType.DATETIME)
+ .addNullableField("timestamp_variant2", Schema.FieldType.DATETIME)
+ .addNullableField("timestamp_variant3", Schema.FieldType.DATETIME)
+ .addNullableField("timestamp_variant4", Schema.FieldType.DATETIME)
.addNullableField("valid", Schema.FieldType.BOOLEAN)
+ .addNullableField("binary", Schema.FieldType.BYTES)
.build();
private static final Schema ARRAY_TYPE =
@@ -71,39 +82,195 @@
private static final TableFieldSchema NAME =
new TableFieldSchema().setName("name").setType(StandardSQLTypeName.STRING.toString());
- private static final TableFieldSchema TIMESTAMP =
- new TableFieldSchema().setName("timestamp").setType(StandardSQLTypeName.TIMESTAMP.toString());
+ private static final TableFieldSchema TIMESTAMP_VARIANT1 =
+ new TableFieldSchema()
+ .setName("timestamp_variant1")
+ .setType(StandardSQLTypeName.TIMESTAMP.toString());
+ private static final TableFieldSchema TIMESTAMP_VARIANT2 =
+ new TableFieldSchema()
+ .setName("timestamp_variant2")
+ .setType(StandardSQLTypeName.TIMESTAMP.toString());
+ private static final TableFieldSchema TIMESTAMP_VARIANT3 =
+ new TableFieldSchema()
+ .setName("timestamp_variant3")
+ .setType(StandardSQLTypeName.TIMESTAMP.toString());
+ private static final TableFieldSchema TIMESTAMP_VARIANT4 =
+ new TableFieldSchema()
+ .setName("timestamp_variant4")
+ .setType(StandardSQLTypeName.TIMESTAMP.toString());
private static final TableFieldSchema VALID =
new TableFieldSchema().setName("valid").setType(StandardSQLTypeName.BOOL.toString());
+ private static final TableFieldSchema BINARY =
+ new TableFieldSchema().setName("binary").setType(StandardSQLTypeName.BYTES.toString());
+
private static final TableFieldSchema IDS =
new TableFieldSchema()
.setName("ids")
.setType(StandardSQLTypeName.INT64.toString())
.setMode(Mode.REPEATED.toString());
+ private static final TableFieldSchema ROW =
+ new TableFieldSchema()
+ .setName("row")
+ .setType(StandardSQLTypeName.STRUCT.toString())
+ .setMode(Mode.NULLABLE.toString())
+ .setFields(
+ Arrays.asList(
+ ID,
+ VALUE,
+ NAME,
+ TIMESTAMP_VARIANT1,
+ TIMESTAMP_VARIANT2,
+ TIMESTAMP_VARIANT3,
+ TIMESTAMP_VARIANT4,
+ VALID,
+ BINARY));
+
+ private static final TableFieldSchema ROWS =
+ new TableFieldSchema()
+ .setName("rows")
+ .setType(StandardSQLTypeName.STRUCT.toString())
+ .setMode(Mode.REPEATED.toString())
+ .setFields(
+ Arrays.asList(
+ ID,
+ VALUE,
+ NAME,
+ TIMESTAMP_VARIANT1,
+ TIMESTAMP_VARIANT2,
+ TIMESTAMP_VARIANT3,
+ TIMESTAMP_VARIANT4,
+ VALID,
+ BINARY));
+
+ // Make sure that chosen BYTES test value is the same after a full base64 round trip.
private static final Row FLAT_ROW =
Row.withSchema(FLAT_TYPE)
- .addValues(123L, 123.456, "test", new DateTime(123456), false)
+ .addValues(
+ 123L,
+ 123.456,
+ "test",
+ ISODateTimeFormat.dateHourMinuteSecondFraction()
+ .withZoneUTC()
+ .parseDateTime("2019-08-16T13:52:07.000"),
+ ISODateTimeFormat.dateHourMinuteSecondFraction()
+ .withZoneUTC()
+ .parseDateTime("2019-08-17T14:52:07.123"),
+ ISODateTimeFormat.dateHourMinuteSecondFraction()
+ .withZoneUTC()
+ .parseDateTime("2019-08-18T15:52:07.123"),
+ new DateTime(123456),
+ false,
+ Base64.getDecoder().decode("ABCD1234"))
.build();
+ private static final TableRow BQ_FLAT_ROW =
+ new TableRow()
+ .set("id", "123")
+ .set("value", "123.456")
+ .set("name", "test")
+ .set("timestamp_variant1", "2019-08-16 13:52:07 UTC")
+ .set("timestamp_variant2", "2019-08-17 14:52:07.123 UTC")
+ // we'll loose precession, but it's something BigQuery can output!
+ .set("timestamp_variant3", "2019-08-18 15:52:07.123456 UTC")
+ .set(
+ "timestamp_variant4",
+ String.valueOf(
+ new DateTime(123456L, ISOChronology.getInstanceUTC()).getMillis() / 1000.0D))
+ .set("valid", "false")
+ .set("binary", "ABCD1234");
+
private static final Row NULL_FLAT_ROW =
- Row.withSchema(FLAT_TYPE).addValues(null, null, null, null, null).build();
+ Row.withSchema(FLAT_TYPE)
+ .addValues(null, null, null, null, null, null, null, null, null)
+ .build();
+
+ private static final TableRow BQ_NULL_FLAT_ROW =
+ new TableRow()
+ .set("id", null)
+ .set("value", null)
+ .set("name", null)
+ .set("timestamp_variant1", null)
+ .set("timestamp_variant2", null)
+ .set("timestamp_variant3", null)
+ .set("timestamp_variant4", null)
+ .set("valid", null)
+ .set("binary", null);
private static final Row ARRAY_ROW =
Row.withSchema(ARRAY_TYPE).addValues((Object) Arrays.asList(123L, 124L)).build();
+ private static final TableRow BQ_ARRAY_ROW =
+ new TableRow()
+ .set(
+ "ids",
+ Arrays.asList(
+ Collections.singletonMap("v", "123"), Collections.singletonMap("v", "124")));
+
private static final Row ROW_ROW = Row.withSchema(ROW_TYPE).addValues(FLAT_ROW).build();
+ private static final TableRow BQ_ROW_ROW = new TableRow().set("row", BQ_FLAT_ROW);
+
private static final Row ARRAY_ROW_ROW =
Row.withSchema(ARRAY_ROW_TYPE).addValues((Object) Arrays.asList(FLAT_ROW)).build();
+ private static final TableRow BQ_ARRAY_ROW_ROW =
+ new TableRow()
+ .set("rows", Collections.singletonList(Collections.singletonMap("v", BQ_FLAT_ROW)));
+
+ private static final TableSchema BQ_FLAT_TYPE =
+ new TableSchema()
+ .setFields(
+ Arrays.asList(
+ ID,
+ VALUE,
+ NAME,
+ TIMESTAMP_VARIANT1,
+ TIMESTAMP_VARIANT2,
+ TIMESTAMP_VARIANT3,
+ TIMESTAMP_VARIANT4,
+ VALID,
+ BINARY));
+
+ private static final TableSchema BQ_ARRAY_TYPE = new TableSchema().setFields(Arrays.asList(IDS));
+
+ private static final TableSchema BQ_ROW_TYPE = new TableSchema().setFields(Arrays.asList(ROW));
+
+ private static final TableSchema BQ_ARRAY_ROW_TYPE =
+ new TableSchema().setFields(Arrays.asList(ROWS));
+
+ private static final Schema AVRO_FLAT_TYPE =
+ Schema.builder()
+ .addNullableField("id", Schema.FieldType.INT64)
+ .addNullableField("value", Schema.FieldType.DOUBLE)
+ .addNullableField("name", Schema.FieldType.STRING)
+ .addNullableField("valid", Schema.FieldType.BOOLEAN)
+ .build();
+
+ private static final Schema AVRO_ARRAY_TYPE =
+ Schema.builder().addArrayField("rows", Schema.FieldType.row(AVRO_FLAT_TYPE)).build();
+
+ private static final Schema AVRO_ARRAY_ARRAY_TYPE =
+ Schema.builder().addArrayField("array_rows", Schema.FieldType.row(AVRO_ARRAY_TYPE)).build();
+
@Test
public void testToTableSchema_flat() {
TableSchema schema = toTableSchema(FLAT_TYPE);
- assertThat(schema.getFields(), containsInAnyOrder(ID, VALUE, NAME, TIMESTAMP, VALID));
+ assertThat(
+ schema.getFields(),
+ containsInAnyOrder(
+ ID,
+ VALUE,
+ NAME,
+ TIMESTAMP_VARIANT1,
+ TIMESTAMP_VARIANT2,
+ TIMESTAMP_VARIANT3,
+ TIMESTAMP_VARIANT4,
+ VALID,
+ BINARY));
}
@Test
@@ -122,7 +289,18 @@
assertThat(field.getName(), equalTo("row"));
assertThat(field.getType(), equalTo(StandardSQLTypeName.STRUCT.toString()));
assertThat(field.getMode(), nullValue());
- assertThat(field.getFields(), containsInAnyOrder(ID, VALUE, NAME, TIMESTAMP, VALID));
+ assertThat(
+ field.getFields(),
+ containsInAnyOrder(
+ ID,
+ VALUE,
+ NAME,
+ TIMESTAMP_VARIANT1,
+ TIMESTAMP_VARIANT2,
+ TIMESTAMP_VARIANT3,
+ TIMESTAMP_VARIANT4,
+ VALID,
+ BINARY));
}
@Test
@@ -134,18 +312,31 @@
assertThat(field.getName(), equalTo("rows"));
assertThat(field.getType(), equalTo(StandardSQLTypeName.STRUCT.toString()));
assertThat(field.getMode(), equalTo(Mode.REPEATED.toString()));
- assertThat(field.getFields(), containsInAnyOrder(ID, VALUE, NAME, TIMESTAMP, VALID));
+ assertThat(
+ field.getFields(),
+ containsInAnyOrder(
+ ID,
+ VALUE,
+ NAME,
+ TIMESTAMP_VARIANT1,
+ TIMESTAMP_VARIANT2,
+ TIMESTAMP_VARIANT3,
+ TIMESTAMP_VARIANT4,
+ VALID,
+ BINARY));
}
@Test
public void testToTableRow_flat() {
TableRow row = toTableRow().apply(FLAT_ROW);
+ System.out.println(row);
- assertThat(row.size(), equalTo(5));
+ assertThat(row.size(), equalTo(9));
assertThat(row, hasEntry("id", "123"));
assertThat(row, hasEntry("value", "123.456"));
assertThat(row, hasEntry("name", "test"));
assertThat(row, hasEntry("valid", "false"));
+ assertThat(row, hasEntry("binary", "ABCD1234"));
}
@Test
@@ -162,11 +353,15 @@
assertThat(row.size(), equalTo(1));
row = (TableRow) row.get("row");
- assertThat(row.size(), equalTo(5));
+ assertThat(row.size(), equalTo(9));
assertThat(row, hasEntry("id", "123"));
assertThat(row, hasEntry("value", "123.456"));
+ assertThat(row, hasEntry("value", "123.456"));
+ assertThat(row, hasEntry("value", "123.456"));
+ assertThat(row, hasEntry("value", "123.456"));
assertThat(row, hasEntry("name", "test"));
assertThat(row, hasEntry("valid", "false"));
+ assertThat(row, hasEntry("binary", "ABCD1234"));
}
@Test
@@ -175,23 +370,28 @@
assertThat(row.size(), equalTo(1));
row = ((List<TableRow>) row.get("rows")).get(0);
- assertThat(row.size(), equalTo(5));
+ assertThat(row.size(), equalTo(9));
assertThat(row, hasEntry("id", "123"));
assertThat(row, hasEntry("value", "123.456"));
assertThat(row, hasEntry("name", "test"));
assertThat(row, hasEntry("valid", "false"));
+ assertThat(row, hasEntry("binary", "ABCD1234"));
}
@Test
public void testToTableRow_null_row() {
TableRow row = toTableRow().apply(NULL_FLAT_ROW);
- assertThat(row.size(), equalTo(5));
+ assertThat(row.size(), equalTo(9));
assertThat(row, hasEntry("id", null));
assertThat(row, hasEntry("value", null));
assertThat(row, hasEntry("name", null));
- assertThat(row, hasEntry("timestamp", null));
+ assertThat(row, hasEntry("timestamp_variant1", null));
+ assertThat(row, hasEntry("timestamp_variant2", null));
+ assertThat(row, hasEntry("timestamp_variant3", null));
+ assertThat(row, hasEntry("timestamp_variant4", null));
assertThat(row, hasEntry("valid", null));
+ assertThat(row, hasEntry("binary", null));
}
private static final BigQueryUtils.ConversionOptions TRUNCATE_OPTIONS =
@@ -211,7 +411,9 @@
IllegalArgumentException.class,
() ->
BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME), 1000000001L, REJECT_OPTIONS));
+ Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
+ 1000000001L,
+ REJECT_OPTIONS));
}
@Test
@@ -219,7 +421,9 @@
long millis = 123456789L;
assertThat(
BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME), millis * 1000, REJECT_OPTIONS),
+ Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
+ millis * 1000,
+ REJECT_OPTIONS),
equalTo(new Instant(millis)));
}
@@ -228,7 +432,7 @@
long millis = 123456789L;
assertThat(
BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME),
+ Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
millis * 1000 + 123,
TRUNCATE_OPTIONS),
equalTo(new Instant(millis)));
@@ -241,7 +445,8 @@
IllegalArgumentException.class,
() ->
BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.logicalType(new FakeSqlTimeType())),
+ Schema.Field.of("dummy", Schema.FieldType.logicalType(new FakeSqlTimeType()))
+ .getType(),
1000000001L,
REJECT_OPTIONS));
}
@@ -251,7 +456,7 @@
long millis = 123456789L;
assertThat(
BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.logicalType(new FakeSqlTimeType())),
+ Schema.Field.of("dummy", Schema.FieldType.logicalType(new FakeSqlTimeType())).getType(),
millis * 1000,
REJECT_OPTIONS),
equalTo(new Instant(millis)));
@@ -262,7 +467,7 @@
long millis = 123456789L;
assertThat(
BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.logicalType(new FakeSqlTimeType())),
+ Schema.Field.of("dummy", Schema.FieldType.logicalType(new FakeSqlTimeType())).getType(),
millis * 1000 + 123,
TRUNCATE_OPTIONS),
equalTo(new Instant(millis)));
@@ -290,4 +495,104 @@
return base.getMillis();
}
}
+
+ @Test
+ public void testFromTableSchema_flat() {
+ Schema beamSchema = BigQueryUtils.fromTableSchema(BQ_FLAT_TYPE);
+ assertEquals(FLAT_TYPE, beamSchema);
+ }
+
+ @Test
+ public void testFromTableSchema_array() {
+ Schema beamSchema = BigQueryUtils.fromTableSchema(BQ_ARRAY_TYPE);
+ assertEquals(ARRAY_TYPE, beamSchema);
+ }
+
+ @Test
+ public void testFromTableSchema_row() {
+ Schema beamSchema = BigQueryUtils.fromTableSchema(BQ_ROW_TYPE);
+ assertEquals(ROW_TYPE, beamSchema);
+ }
+
+ @Test
+ public void testFromTableSchema_array_row() {
+ Schema beamSchema = BigQueryUtils.fromTableSchema(BQ_ARRAY_ROW_TYPE);
+ assertEquals(ARRAY_ROW_TYPE, beamSchema);
+ }
+
+ @Test
+ public void testToBeamRow_flat() {
+ Row beamRow = BigQueryUtils.toBeamRow(FLAT_TYPE, BQ_FLAT_ROW);
+ assertEquals(FLAT_ROW, beamRow);
+ }
+
+ @Test
+ public void testToBeamRow_null() {
+ Row beamRow = BigQueryUtils.toBeamRow(FLAT_TYPE, BQ_NULL_FLAT_ROW);
+ assertEquals(NULL_FLAT_ROW, beamRow);
+ }
+
+ @Test
+ public void testToBeamRow_array() {
+ Row beamRow = BigQueryUtils.toBeamRow(ARRAY_TYPE, BQ_ARRAY_ROW);
+ assertEquals(ARRAY_ROW, beamRow);
+ }
+
+ @Test
+ public void testToBeamRow_row() {
+ Row beamRow = BigQueryUtils.toBeamRow(ROW_TYPE, BQ_ROW_ROW);
+ assertEquals(ROW_ROW, beamRow);
+ }
+
+ @Test
+ public void testToBeamRow_array_row() {
+ Row beamRow = BigQueryUtils.toBeamRow(ARRAY_ROW_TYPE, BQ_ARRAY_ROW_ROW);
+ assertEquals(ARRAY_ROW_ROW, beamRow);
+ }
+
+ @Test
+ public void testToBeamRow_avro_array_row() {
+ Row flatRowExpected =
+ Row.withSchema(AVRO_FLAT_TYPE).addValues(123L, 123.456, "test", false).build();
+ Row expected =
+ Row.withSchema(AVRO_ARRAY_TYPE).addValues((Object) Arrays.asList(flatRowExpected)).build();
+ GenericData.Record record = new GenericData.Record(AvroUtils.toAvroSchema(AVRO_ARRAY_TYPE));
+ GenericData.Record flat = new GenericData.Record(AvroUtils.toAvroSchema(AVRO_FLAT_TYPE));
+ flat.put("id", 123L);
+ flat.put("value", 123.456);
+ flat.put("name", "test");
+ flat.put("valid", false);
+ record.put("rows", Arrays.asList(flat));
+ Row beamRow =
+ BigQueryUtils.toBeamRow(
+ record, AVRO_ARRAY_TYPE, BigQueryUtils.ConversionOptions.builder().build());
+ assertEquals(expected, beamRow);
+ }
+
+ @Test
+ public void testToBeamRow_avro_array_array_row() {
+ Row flatRowExpected =
+ Row.withSchema(AVRO_FLAT_TYPE).addValues(123L, 123.456, "test", false).build();
+ Row arrayRowExpected =
+ Row.withSchema(AVRO_ARRAY_TYPE).addValues((Object) Arrays.asList(flatRowExpected)).build();
+ Row expected =
+ Row.withSchema(AVRO_ARRAY_ARRAY_TYPE)
+ .addValues((Object) Arrays.asList(arrayRowExpected))
+ .build();
+ GenericData.Record arrayRecord =
+ new GenericData.Record(AvroUtils.toAvroSchema(AVRO_ARRAY_TYPE));
+ GenericData.Record flat = new GenericData.Record(AvroUtils.toAvroSchema(AVRO_FLAT_TYPE));
+ GenericData.Record record =
+ new GenericData.Record(AvroUtils.toAvroSchema(AVRO_ARRAY_ARRAY_TYPE));
+ flat.put("id", 123L);
+ flat.put("value", 123.456);
+ flat.put("name", "test");
+ flat.put("valid", false);
+ arrayRecord.put("rows", Arrays.asList(flat));
+ record.put("array_rows", Arrays.asList(arrayRecord));
+ Row beamRow =
+ BigQueryUtils.toBeamRow(
+ record, AVRO_ARRAY_ARRAY_TYPE, BigQueryUtils.ConversionOptions.builder().build());
+ assertEquals(expected, beamRow);
+ }
}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeBigQueryServices.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeBigQueryServices.java
deleted file mode 100644
index b10f960..0000000
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeBigQueryServices.java
+++ /dev/null
@@ -1,104 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.gcp.bigquery;
-
-import com.google.api.client.util.Base64;
-import com.google.api.services.bigquery.model.Table;
-import com.google.api.services.bigquery.model.TableRow;
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.util.List;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.ListCoder;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-
-/** A fake implementation of BigQuery's query service.. */
-@Experimental(Experimental.Kind.SOURCE_SINK)
-public class FakeBigQueryServices implements BigQueryServices {
- private JobService jobService;
- private DatasetService datasetService;
- private StorageClient storageClient;
-
- public FakeBigQueryServices withJobService(JobService jobService) {
- this.jobService = jobService;
- return this;
- }
-
- public FakeBigQueryServices withDatasetService(FakeDatasetService datasetService) {
- this.datasetService = datasetService;
- return this;
- }
-
- public FakeBigQueryServices withStorageClient(StorageClient storageClient) {
- this.storageClient = storageClient;
- return this;
- }
-
- @Override
- public JobService getJobService(BigQueryOptions bqOptions) {
- return jobService;
- }
-
- @Override
- public DatasetService getDatasetService(BigQueryOptions bqOptions) {
- return datasetService;
- }
-
- @Override
- public StorageClient getStorageClient(BigQueryOptions bqOptions) {
- return storageClient;
- }
-
- static String encodeQueryResult(Table table) throws IOException {
- return encodeQueryResult(table, ImmutableList.of());
- }
-
- static String encodeQueryResult(Table table, List<TableRow> rows) throws IOException {
- KvCoder<String, List<TableRow>> coder =
- KvCoder.of(StringUtf8Coder.of(), ListCoder.of(TableRowJsonCoder.of()));
- KV<String, List<TableRow>> kv = KV.of(BigQueryHelpers.toJsonString(table), rows);
- ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
- coder.encode(kv, outputStream);
- return Base64.encodeBase64String(outputStream.toByteArray());
- }
-
- static KV<Table, List<TableRow>> decodeQueryResult(String queryResult) throws IOException {
- KvCoder<String, List<TableRow>> coder =
- KvCoder.of(StringUtf8Coder.of(), ListCoder.of(TableRowJsonCoder.of()));
- ByteArrayInputStream inputStream = new ByteArrayInputStream(Base64.decodeBase64(queryResult));
- KV<String, List<TableRow>> kv = coder.decode(inputStream);
- Table table = BigQueryHelpers.fromJsonString(kv.getKey(), Table.class);
- List<TableRow> rows = kv.getValue();
- rows.forEach(FakeBigQueryServices::convertNumbers);
- return KV.of(table, rows);
- }
-
- // Longs tend to get converted back to Integers due to JSON serialization. Convert them back.
- static TableRow convertNumbers(TableRow tableRow) {
- for (TableRow.Entry entry : tableRow.entrySet()) {
- if (entry.getValue() instanceof Integer) {
- entry.setValue(Long.valueOf((Integer) entry.getValue()));
- }
- }
- return tableRow;
- }
-}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeDatasetService.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeDatasetService.java
deleted file mode 100644
index 3edb288..0000000
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeDatasetService.java
+++ /dev/null
@@ -1,326 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.gcp.bigquery;
-
-import static org.junit.Assert.assertEquals;
-
-import com.google.api.client.googleapis.json.GoogleJsonResponseException;
-import com.google.api.client.http.HttpHeaders;
-import com.google.api.services.bigquery.model.Dataset;
-import com.google.api.services.bigquery.model.DatasetReference;
-import com.google.api.services.bigquery.model.Table;
-import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
-import com.google.api.services.bigquery.model.TableReference;
-import com.google.api.services.bigquery.model.TableRow;
-import java.io.IOException;
-import java.io.Serializable;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ThreadLocalRandom;
-import java.util.regex.Pattern;
-import javax.annotation.Nullable;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.DatasetService;
-import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy.Context;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.values.ValueInSingleWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-
-/** A fake dataset service that can be serialized, for use in testReadFromTable. */
-public class FakeDatasetService implements DatasetService, Serializable {
- // Table information must be static, as each ParDo will get a separate instance of
- // FakeDatasetServices, and they must all modify the same storage.
- static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table<
- String, String, Map<String, TableContainer>>
- tables;
-
- Map<String, List<String>> insertErrors = Maps.newHashMap();
-
- public static void setUp() {
- tables = HashBasedTable.create();
- FakeJobService.setUp();
- }
-
- @Override
- public Table getTable(TableReference tableRef) throws InterruptedException, IOException {
- return getTable(tableRef, null);
- }
-
- @Override
- public Table getTable(TableReference tableRef, @Nullable List<String> selectedFields)
- throws InterruptedException, IOException {
- synchronized (tables) {
- Map<String, TableContainer> dataset =
- tables.get(tableRef.getProjectId(), tableRef.getDatasetId());
- if (dataset == null) {
- throwNotFound(
- "Tried to get a dataset %s:%s, but no such dataset was set",
- tableRef.getProjectId(), tableRef.getDatasetId());
- }
- TableContainer tableContainer = dataset.get(tableRef.getTableId());
- return tableContainer == null ? null : tableContainer.getTable();
- }
- }
-
- public List<TableRow> getAllRows(String projectId, String datasetId, String tableId)
- throws InterruptedException, IOException {
- synchronized (tables) {
- return getTableContainer(projectId, datasetId, tableId).getRows();
- }
- }
-
- private TableContainer getTableContainer(String projectId, String datasetId, String tableId)
- throws InterruptedException, IOException {
- synchronized (tables) {
- Map<String, TableContainer> dataset = tables.get(projectId, datasetId);
- if (dataset == null) {
- throwNotFound(
- "Tried to get a dataset %s:%s, but no such dataset was set", projectId, datasetId);
- }
- TableContainer tableContainer = dataset.get(tableId);
- if (tableContainer == null) {
- throwNotFound(
- "Tried to get a table %s:%s.%s, but no such table was set",
- projectId, datasetId, tableId);
- }
- return tableContainer;
- }
- }
-
- @Override
- public void deleteTable(TableReference tableRef) throws IOException, InterruptedException {
- validateWholeTableReference(tableRef);
- synchronized (tables) {
- Map<String, TableContainer> dataset =
- tables.get(tableRef.getProjectId(), tableRef.getDatasetId());
- if (dataset == null) {
- throwNotFound(
- "Tried to get a dataset %s:%s, but no such table was set",
- tableRef.getProjectId(), tableRef.getDatasetId());
- }
- dataset.remove(tableRef.getTableId());
- }
- }
-
- /**
- * Validates a table reference for whole-table operations, such as create/delete/patch. Such
- * operations do not support partition decorators.
- */
- private static void validateWholeTableReference(TableReference tableReference)
- throws IOException {
- final Pattern tableRegexp = Pattern.compile("[-\\w]{1,1024}");
- if (!tableRegexp.matcher(tableReference.getTableId()).matches()) {
- throw new IOException(
- String.format(
- "invalid table ID %s. Table IDs must be alphanumeric "
- + "(plus underscores) and must be at most 1024 characters long. Also, table"
- + " decorators cannot be used.",
- tableReference.getTableId()));
- }
- }
-
- @Override
- public void createTable(Table table) throws IOException {
- TableReference tableReference = table.getTableReference();
- validateWholeTableReference(tableReference);
- synchronized (tables) {
- Map<String, TableContainer> dataset =
- tables.get(tableReference.getProjectId(), tableReference.getDatasetId());
- if (dataset == null) {
- throwNotFound(
- "Tried to get a dataset %s:%s, but no such table was set",
- tableReference.getProjectId(), tableReference.getDatasetId());
- }
- dataset.computeIfAbsent(tableReference.getTableId(), k -> new TableContainer(table));
- }
- }
-
- @Override
- public boolean isTableEmpty(TableReference tableRef) throws IOException, InterruptedException {
- Long numBytes = getTable(tableRef).getNumBytes();
- return numBytes == null || numBytes == 0L;
- }
-
- @Override
- public Dataset getDataset(String projectId, String datasetId)
- throws IOException, InterruptedException {
- synchronized (tables) {
- Map<String, TableContainer> dataset = tables.get(projectId, datasetId);
- if (dataset == null) {
- throwNotFound(
- "Tried to get a dataset %s:%s, but no such table was set", projectId, datasetId);
- }
- return new Dataset()
- .setDatasetReference(
- new DatasetReference().setDatasetId(datasetId).setProjectId(projectId));
- }
- }
-
- @Override
- public void createDataset(
- String projectId,
- String datasetId,
- String location,
- String description,
- Long defaultTableExpirationMs /* ignored */)
- throws IOException, InterruptedException {
- synchronized (tables) {
- Map<String, TableContainer> dataset = tables.get(projectId, datasetId);
- if (dataset == null) {
- dataset = new HashMap<>();
- tables.put(projectId, datasetId, dataset);
- }
- }
- }
-
- @Override
- public void deleteDataset(String projectId, String datasetId)
- throws IOException, InterruptedException {
- synchronized (tables) {
- tables.remove(projectId, datasetId);
- }
- }
-
- public long insertAll(
- TableReference ref, List<TableRow> rowList, @Nullable List<String> insertIdList)
- throws IOException, InterruptedException {
- List<ValueInSingleWindow<TableRow>> windowedRows = Lists.newArrayList();
- for (TableRow row : rowList) {
- windowedRows.add(
- ValueInSingleWindow.of(
- row,
- GlobalWindow.TIMESTAMP_MAX_VALUE,
- GlobalWindow.INSTANCE,
- PaneInfo.ON_TIME_AND_ONLY_FIRING));
- }
- return insertAll(
- ref, windowedRows, insertIdList, InsertRetryPolicy.alwaysRetry(), null, null, false, false);
- }
-
- @Override
- public <T> long insertAll(
- TableReference ref,
- List<ValueInSingleWindow<TableRow>> rowList,
- @Nullable List<String> insertIdList,
- InsertRetryPolicy retryPolicy,
- List<ValueInSingleWindow<T>> failedInserts,
- ErrorContainer<T> errorContainer,
- boolean skipInvalidRows,
- boolean ignoreUnknownValues)
- throws IOException, InterruptedException {
- Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> insertErrors = getInsertErrors();
- synchronized (tables) {
- if (insertIdList != null) {
- assertEquals(rowList.size(), insertIdList.size());
- } else {
- insertIdList = Lists.newArrayListWithExpectedSize(rowList.size());
- for (int i = 0; i < rowList.size(); ++i) {
- insertIdList.add(Integer.toString(ThreadLocalRandom.current().nextInt()));
- }
- }
-
- long dataSize = 0;
- TableContainer tableContainer =
- getTableContainer(
- ref.getProjectId(),
- ref.getDatasetId(),
- BigQueryHelpers.stripPartitionDecorator(ref.getTableId()));
- for (int i = 0; i < rowList.size(); ++i) {
- TableRow row = rowList.get(i).getValue();
- List<TableDataInsertAllResponse.InsertErrors> allErrors = insertErrors.get(row);
- boolean shouldInsert = true;
- if (allErrors != null) {
- for (TableDataInsertAllResponse.InsertErrors errors : allErrors) {
- if (!retryPolicy.shouldRetry(new Context(errors))) {
- shouldInsert = false;
- }
- }
- }
- if (shouldInsert) {
- dataSize += tableContainer.addRow(row, insertIdList.get(i));
- } else {
- errorContainer.add(
- failedInserts, allErrors.get(allErrors.size() - 1), ref, rowList.get(i));
- }
- }
- return dataSize;
- }
- }
-
- @Override
- public Table patchTableDescription(
- TableReference tableReference, @Nullable String tableDescription)
- throws IOException, InterruptedException {
- validateWholeTableReference(tableReference);
- synchronized (tables) {
- TableContainer tableContainer =
- getTableContainer(
- tableReference.getProjectId(),
- tableReference.getDatasetId(),
- tableReference.getTableId());
- tableContainer.getTable().setDescription(tableDescription);
- return tableContainer.getTable();
- }
- }
-
- /**
- * Cause a given {@link TableRow} object to fail when it's inserted. The errors link the list will
- * be returned on subsequent retries, and the insert will succeed when the errors run out.
- */
- public void failOnInsert(
- Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> insertErrors) {
- synchronized (tables) {
- for (Map.Entry<TableRow, List<TableDataInsertAllResponse.InsertErrors>> entry :
- insertErrors.entrySet()) {
- List<String> errorStrings = Lists.newArrayList();
- for (TableDataInsertAllResponse.InsertErrors errors : entry.getValue()) {
- errorStrings.add(BigQueryHelpers.toJsonString(errors));
- }
- this.insertErrors.put(BigQueryHelpers.toJsonString(entry.getKey()), errorStrings);
- }
- }
- }
-
- Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> getInsertErrors() {
- Map<TableRow, List<TableDataInsertAllResponse.InsertErrors>> parsedInsertErrors =
- Maps.newHashMap();
- synchronized (tables) {
- for (Map.Entry<String, List<String>> entry : this.insertErrors.entrySet()) {
- TableRow tableRow = BigQueryHelpers.fromJsonString(entry.getKey(), TableRow.class);
- List<TableDataInsertAllResponse.InsertErrors> allErrors = Lists.newArrayList();
- for (String errorsString : entry.getValue()) {
- allErrors.add(
- BigQueryHelpers.fromJsonString(
- errorsString, TableDataInsertAllResponse.InsertErrors.class));
- }
- parsedInsertErrors.put(tableRow, allErrors);
- }
- }
- return parsedInsertErrors;
- }
-
- void throwNotFound(String format, Object... args) throws IOException {
- throw new IOException(
- String.format(format, args),
- new GoogleJsonResponseException.Builder(404, String.format(format, args), new HttpHeaders())
- .build());
- }
-}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
deleted file mode 100644
index 436dcdb..0000000
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
+++ /dev/null
@@ -1,499 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.gcp.bigquery;
-
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-
-import com.google.api.client.json.JsonFactory;
-import com.google.api.client.util.BackOff;
-import com.google.api.client.util.BackOffUtils;
-import com.google.api.client.util.Sleeper;
-import com.google.api.services.bigquery.model.ErrorProto;
-import com.google.api.services.bigquery.model.Job;
-import com.google.api.services.bigquery.model.JobConfiguration;
-import com.google.api.services.bigquery.model.JobConfigurationExtract;
-import com.google.api.services.bigquery.model.JobConfigurationLoad;
-import com.google.api.services.bigquery.model.JobConfigurationQuery;
-import com.google.api.services.bigquery.model.JobConfigurationTableCopy;
-import com.google.api.services.bigquery.model.JobReference;
-import com.google.api.services.bigquery.model.JobStatistics;
-import com.google.api.services.bigquery.model.JobStatistics4;
-import com.google.api.services.bigquery.model.JobStatus;
-import com.google.api.services.bigquery.model.Table;
-import com.google.api.services.bigquery.model.TableReference;
-import com.google.api.services.bigquery.model.TableRow;
-import com.google.api.services.bigquery.model.TableSchema;
-import com.google.api.services.bigquery.model.TimePartitioning;
-import java.io.BufferedReader;
-import java.io.ByteArrayInputStream;
-import java.io.IOException;
-import java.io.Serializable;
-import java.nio.channels.Channels;
-import java.nio.channels.WritableByteChannel;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Paths;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.concurrent.ThreadLocalRandom;
-import org.apache.avro.Schema;
-import org.apache.avro.file.DataFileWriter;
-import org.apache.avro.generic.GenericDatumWriter;
-import org.apache.avro.generic.GenericRecord;
-import org.apache.avro.generic.GenericRecordBuilder;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.Coder.Context;
-import org.apache.beam.sdk.extensions.gcp.util.BackOffAdapter;
-import org.apache.beam.sdk.extensions.gcp.util.Transport;
-import org.apache.beam.sdk.io.FileSystems;
-import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition;
-import org.apache.beam.sdk.io.gcp.bigquery.BigQueryServices.JobService;
-import org.apache.beam.sdk.util.FluentBackoff;
-import org.apache.beam.sdk.util.MimeTypes;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.HashBasedTable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.joda.time.Duration;
-
-/** A fake implementation of BigQuery's job service. */
-@Experimental(Experimental.Kind.SOURCE_SINK)
-public class FakeJobService implements JobService, Serializable {
- private static final JsonFactory JSON_FACTORY = Transport.getJsonFactory();
- // Whenever a job is started, the first 2 calls to GetJob will report the job as pending,
- // the next 2 will return the job as running, and only then will the job report as done.
- private static final int GET_JOBS_TRANSITION_INTERVAL = 2;
-
- // The number of times to simulate a failure and trigger a retry.
- private int numFailuresExpected;
- private int numFailures = 0;
-
- private final FakeDatasetService datasetService;
-
- private static class JobInfo {
- Job job;
- int getJobCount = 0;
-
- JobInfo(Job job) {
- this.job = job;
- }
- }
-
- private static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table<
- String, String, JobInfo>
- allJobs;
- private static int numExtractJobCalls;
-
- private static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table<
- String, String, List<ResourceId>>
- filesForLoadJobs;
- private static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Table<
- String, String, JobStatistics>
- dryRunQueryResults;
-
- public FakeJobService() {
- this(0);
- }
-
- public FakeJobService(int numFailures) {
- this.datasetService = new FakeDatasetService();
- this.numFailuresExpected = numFailures;
- }
-
- public void setNumFailuresExpected(int numFailuresExpected) {
- this.numFailuresExpected = numFailuresExpected;
- }
-
- public static void setUp() {
- allJobs = HashBasedTable.create();
- numExtractJobCalls = 0;
- filesForLoadJobs = HashBasedTable.create();
- dryRunQueryResults = HashBasedTable.create();
- }
-
- @Override
- public void startLoadJob(JobReference jobRef, JobConfigurationLoad loadConfig)
- throws IOException {
- synchronized (allJobs) {
- verifyUniqueJobId(jobRef.getJobId());
- Job job = new Job();
- job.setJobReference(jobRef);
- job.setConfiguration(new JobConfiguration().setLoad(loadConfig));
- job.setKind(" bigquery#job");
- job.setStatus(new JobStatus().setState("PENDING"));
-
- // Copy the files to a new location for import, as the temporary files will be deleted by
- // the caller.
- if (loadConfig.getSourceUris().size() > 0) {
- ImmutableList.Builder<ResourceId> sourceFiles = ImmutableList.builder();
- ImmutableList.Builder<ResourceId> loadFiles = ImmutableList.builder();
- for (String filename : loadConfig.getSourceUris()) {
- sourceFiles.add(FileSystems.matchNewResource(filename, false /* isDirectory */));
- loadFiles.add(
- FileSystems.matchNewResource(
- filename + ThreadLocalRandom.current().nextInt(), false /* isDirectory */));
- }
-
- FileSystems.copy(sourceFiles.build(), loadFiles.build());
- filesForLoadJobs.put(jobRef.getProjectId(), jobRef.getJobId(), loadFiles.build());
- }
-
- allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
- }
- }
-
- @Override
- public void startExtractJob(JobReference jobRef, JobConfigurationExtract extractConfig)
- throws IOException {
- checkArgument(
- "AVRO".equals(extractConfig.getDestinationFormat()), "Only extract to AVRO is supported");
- synchronized (allJobs) {
- verifyUniqueJobId(jobRef.getJobId());
- ++numExtractJobCalls;
-
- Job job = new Job();
- job.setJobReference(jobRef);
- job.setConfiguration(new JobConfiguration().setExtract(extractConfig));
- job.setKind(" bigquery#job");
- job.setStatus(new JobStatus().setState("PENDING"));
- allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
- }
- }
-
- public int getNumExtractJobCalls() {
- synchronized (allJobs) {
- return numExtractJobCalls;
- }
- }
-
- @Override
- public void startQueryJob(JobReference jobRef, JobConfigurationQuery query) {
- synchronized (allJobs) {
- Job job = new Job();
- job.setJobReference(jobRef);
- job.setConfiguration(new JobConfiguration().setQuery(query));
- job.setKind(" bigquery#job");
- job.setStatus(new JobStatus().setState("PENDING"));
- allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
- }
- }
-
- @Override
- public void startCopyJob(JobReference jobRef, JobConfigurationTableCopy copyConfig)
- throws IOException {
- synchronized (allJobs) {
- verifyUniqueJobId(jobRef.getJobId());
- Job job = new Job();
- job.setJobReference(jobRef);
- job.setConfiguration(new JobConfiguration().setCopy(copyConfig));
- job.setKind(" bigquery#job");
- job.setStatus(new JobStatus().setState("PENDING"));
- allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
- }
- }
-
- @Override
- public Job pollJob(JobReference jobRef, int maxAttempts) throws InterruptedException {
- BackOff backoff =
- BackOffAdapter.toGcpBackOff(
- FluentBackoff.DEFAULT
- .withMaxRetries(maxAttempts)
- .withInitialBackoff(Duration.millis(10))
- .withMaxBackoff(Duration.standardSeconds(1))
- .backoff());
- Sleeper sleeper = Sleeper.DEFAULT;
- try {
- do {
- Job job = getJob(jobRef);
- if (job != null) {
- JobStatus status = job.getStatus();
- if (status != null
- && ("DONE".equals(status.getState()) || "FAILED".equals(status.getState()))) {
- return job;
- }
- }
- } while (BackOffUtils.next(sleeper, backoff));
- } catch (IOException e) {
- return null;
- }
- return null;
- }
-
- public void expectDryRunQuery(String projectId, String query, JobStatistics result) {
- synchronized (dryRunQueryResults) {
- dryRunQueryResults.put(projectId, query, result);
- }
- }
-
- @Override
- public JobStatistics dryRunQuery(String projectId, JobConfigurationQuery query, String location) {
- synchronized (dryRunQueryResults) {
- JobStatistics result = dryRunQueryResults.get(projectId, query.getQuery());
- if (result != null) {
- return result;
- }
- }
- throw new UnsupportedOperationException();
- }
-
- @Override
- public Job getJob(JobReference jobRef) {
- try {
- synchronized (allJobs) {
- JobInfo job = allJobs.get(jobRef.getProjectId(), jobRef.getJobId());
- if (job == null) {
- return null;
- }
- try {
- ++job.getJobCount;
- if (!"FAILED".equals(job.job.getStatus().getState())) {
- if (numFailures < numFailuresExpected) {
- ++numFailures;
- throw new Exception("Failure number " + numFailures);
- }
-
- if (job.getJobCount == GET_JOBS_TRANSITION_INTERVAL + 1) {
- job.job.getStatus().setState("RUNNING");
- } else if (job.getJobCount == 2 * GET_JOBS_TRANSITION_INTERVAL + 1) {
- job.job.setStatus(runJob(job.job));
- }
- }
- } catch (Exception e) {
- job.job
- .getStatus()
- .setState("FAILED")
- .setErrorResult(
- new ErrorProto()
- .setMessage(
- String.format(
- "Job %s failed: %s", job.job.getConfiguration(), e.toString())));
- List<ResourceId> sourceFiles =
- filesForLoadJobs.get(jobRef.getProjectId(), jobRef.getJobId());
- FileSystems.delete(sourceFiles);
- }
- return JSON_FACTORY.fromString(JSON_FACTORY.toString(job.job), Job.class);
- }
- } catch (IOException e) {
- return null;
- }
- }
-
- private void verifyUniqueJobId(String jobId) throws IOException {
- if (allJobs.containsColumn(jobId)) {
- throw new IOException("Duplicate job id " + jobId);
- }
- }
-
- private JobStatus runJob(Job job) throws InterruptedException, IOException {
- if (job.getConfiguration().getLoad() != null) {
- return runLoadJob(job.getJobReference(), job.getConfiguration().getLoad());
- } else if (job.getConfiguration().getCopy() != null) {
- return runCopyJob(job.getConfiguration().getCopy());
- } else if (job.getConfiguration().getExtract() != null) {
- return runExtractJob(job, job.getConfiguration().getExtract());
- } else if (job.getConfiguration().getQuery() != null) {
- return runQueryJob(job.getConfiguration().getQuery());
- }
- return new JobStatus().setState("DONE");
- }
-
- private boolean validateDispositions(
- Table table, CreateDisposition createDisposition, WriteDisposition writeDisposition)
- throws InterruptedException, IOException {
- if (table == null) {
- if (createDisposition == CreateDisposition.CREATE_NEVER) {
- return false;
- }
- } else if (writeDisposition == WriteDisposition.WRITE_TRUNCATE) {
- datasetService.deleteTable(table.getTableReference());
- } else if (writeDisposition == WriteDisposition.WRITE_EMPTY) {
- List<TableRow> allRows =
- datasetService.getAllRows(
- table.getTableReference().getProjectId(),
- table.getTableReference().getDatasetId(),
- table.getTableReference().getTableId());
- if (!allRows.isEmpty()) {
- return false;
- }
- }
- return true;
- }
-
- private JobStatus runLoadJob(JobReference jobRef, JobConfigurationLoad load)
- throws InterruptedException, IOException {
- TableReference destination = load.getDestinationTable();
- TableSchema schema = load.getSchema();
- checkArgument(schema != null, "No schema specified");
- List<ResourceId> sourceFiles = filesForLoadJobs.get(jobRef.getProjectId(), jobRef.getJobId());
- WriteDisposition writeDisposition = WriteDisposition.valueOf(load.getWriteDisposition());
- CreateDisposition createDisposition = CreateDisposition.valueOf(load.getCreateDisposition());
- checkArgument("NEWLINE_DELIMITED_JSON".equals(load.getSourceFormat()));
- Table existingTable = datasetService.getTable(destination);
- if (!validateDispositions(existingTable, createDisposition, writeDisposition)) {
- return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
- }
- if (existingTable == null) {
- TableReference strippedDestination =
- destination
- .clone()
- .setTableId(BigQueryHelpers.stripPartitionDecorator(destination.getTableId()));
- existingTable = new Table().setTableReference(strippedDestination).setSchema(schema);
- if (load.getTimePartitioning() != null) {
- existingTable = existingTable.setTimePartitioning(load.getTimePartitioning());
- }
- datasetService.createTable(existingTable);
- }
-
- List<TableRow> rows = Lists.newArrayList();
- for (ResourceId filename : sourceFiles) {
- rows.addAll(readRows(filename.toString()));
- }
- datasetService.insertAll(destination, rows, null);
- FileSystems.delete(sourceFiles);
- return new JobStatus().setState("DONE");
- }
-
- private JobStatus runCopyJob(JobConfigurationTableCopy copy)
- throws InterruptedException, IOException {
- List<TableReference> sources = copy.getSourceTables();
- TableReference destination = copy.getDestinationTable();
- WriteDisposition writeDisposition = WriteDisposition.valueOf(copy.getWriteDisposition());
- CreateDisposition createDisposition = CreateDisposition.valueOf(copy.getCreateDisposition());
- Table existingTable = datasetService.getTable(destination);
- if (!validateDispositions(existingTable, createDisposition, writeDisposition)) {
- return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
- }
- TimePartitioning partitioning = null;
- TableSchema schema = null;
- boolean first = true;
- List<TableRow> allRows = Lists.newArrayList();
- for (TableReference source : sources) {
- Table table = checkNotNull(datasetService.getTable(source));
- if (!first) {
- if (!Objects.equals(partitioning, table.getTimePartitioning())) {
- return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
- }
- if (!Objects.equals(schema, table.getSchema())) {
- return new JobStatus().setState("FAILED").setErrorResult(new ErrorProto());
- }
- }
- partitioning = table.getTimePartitioning();
- schema = table.getSchema();
- first = false;
- allRows.addAll(
- datasetService.getAllRows(
- source.getProjectId(), source.getDatasetId(), source.getTableId()));
- }
- datasetService.createTable(
- new Table()
- .setTableReference(destination)
- .setSchema(schema)
- .setTimePartitioning(partitioning)
- .setEncryptionConfiguration(copy.getDestinationEncryptionConfiguration()));
- datasetService.insertAll(destination, allRows, null);
- return new JobStatus().setState("DONE");
- }
-
- private JobStatus runExtractJob(Job job, JobConfigurationExtract extract)
- throws InterruptedException, IOException {
- TableReference sourceTable = extract.getSourceTable();
-
- List<TableRow> rows =
- datasetService.getAllRows(
- sourceTable.getProjectId(), sourceTable.getDatasetId(), sourceTable.getTableId());
- TableSchema schema = datasetService.getTable(sourceTable).getSchema();
- List<Long> destinationFileCounts = Lists.newArrayList();
- for (String destination : extract.getDestinationUris()) {
- destinationFileCounts.add(writeRows(sourceTable.getTableId(), rows, schema, destination));
- }
- job.setStatistics(
- new JobStatistics()
- .setExtract(new JobStatistics4().setDestinationUriFileCounts(destinationFileCounts)));
- return new JobStatus().setState("DONE");
- }
-
- private JobStatus runQueryJob(JobConfigurationQuery query)
- throws IOException, InterruptedException {
- KV<Table, List<TableRow>> result = FakeBigQueryServices.decodeQueryResult(query.getQuery());
- datasetService.createTable(result.getKey().setTableReference(query.getDestinationTable()));
- datasetService.insertAll(query.getDestinationTable(), result.getValue(), null);
- return new JobStatus().setState("DONE");
- }
-
- private List<TableRow> readRows(String filename) throws IOException {
- Coder<TableRow> coder = TableRowJsonCoder.of();
- List<TableRow> tableRows = Lists.newArrayList();
- try (BufferedReader reader =
- Files.newBufferedReader(Paths.get(filename), StandardCharsets.UTF_8)) {
- String line;
- while ((line = reader.readLine()) != null) {
- TableRow tableRow =
- coder.decode(
- new ByteArrayInputStream(line.getBytes(StandardCharsets.UTF_8)), Context.OUTER);
- tableRows.add(tableRow);
- }
- }
- return tableRows;
- }
-
- private long writeRows(
- String tableId, List<TableRow> rows, TableSchema schema, String destinationPattern)
- throws IOException {
- Schema avroSchema = BigQueryAvroUtils.toGenericAvroSchema(tableId, schema.getFields());
- List<TableRow> rowsToWrite = Lists.newArrayList();
- int shard = 0;
- for (TableRow row : rows) {
- rowsToWrite.add(row);
- if (rowsToWrite.size() == 5) {
- writeRowsHelper(rowsToWrite, avroSchema, destinationPattern, shard++);
- rowsToWrite.clear();
- }
- }
- if (!rowsToWrite.isEmpty()) {
- writeRowsHelper(rowsToWrite, avroSchema, destinationPattern, shard++);
- }
- return shard;
- }
-
- private void writeRowsHelper(
- List<TableRow> rows, Schema avroSchema, String destinationPattern, int shard) {
- String filename = destinationPattern.replace("*", String.format("%012d", shard));
- try (WritableByteChannel channel =
- FileSystems.create(
- FileSystems.matchNewResource(filename, false /* isDirectory */), MimeTypes.BINARY);
- DataFileWriter<GenericRecord> tableRowWriter =
- new DataFileWriter<>(new GenericDatumWriter<GenericRecord>(avroSchema))
- .create(avroSchema, Channels.newOutputStream(channel))) {
- for (Map<String, Object> record : rows) {
- GenericRecordBuilder genericRecordBuilder = new GenericRecordBuilder(avroSchema);
- for (Map.Entry<String, Object> field : record.entrySet()) {
- genericRecordBuilder.set(field.getKey(), field.getValue());
- }
- tableRowWriter.append(genericRecordBuilder.build());
- }
- } catch (IOException e) {
- throw new IllegalStateException(
- String.format("Could not create destination for extract job %s", filename), e);
- }
- }
-}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicyTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicyTest.java
index 144e6d4..3b4ebaf 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicyTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicyTest.java
@@ -25,7 +25,7 @@
import java.util.List;
import java.util.concurrent.ThreadLocalRandom;
import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy.Context;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
deleted file mode 100644
index 18dbba3..0000000
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableContainer.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.gcp.bigquery;
-
-import com.google.api.services.bigquery.model.Table;
-import com.google.api.services.bigquery.model.TableRow;
-import java.util.ArrayList;
-import java.util.List;
-
-/** Encapsulates a BigQuery Table, and it's contents. */
-class TableContainer {
- Table table;
- List<TableRow> rows;
- List<String> ids;
- Long sizeBytes;
-
- TableContainer(Table table) {
- this.table = table;
-
- this.rows = new ArrayList<>();
- this.ids = new ArrayList<>();
- this.sizeBytes = 0L;
- }
-
- long addRow(TableRow row, String id) {
- rows.add(row);
- ids.add(id);
- long rowSize = row.toString().length();
- Long tableSize = table.getNumBytes();
- if (tableSize == null) {
- table.setNumBytes(rowSize);
- } else {
- table.setNumBytes(tableSize + rowSize);
- }
- return rowSize;
- }
-
- Table getTable() {
- return table;
- }
-
- List<TableRow> getRows() {
- return rows;
- }
-}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfigTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfigTest.java
index f79b907..0a3d4e4 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfigTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfigTest.java
@@ -117,7 +117,7 @@
@Test
public void testWithValidate() {
- assertEquals(true, config.withValidate(true).getValidate());
+ assertTrue(config.withValidate(true).getValidate());
}
@Test
@@ -208,7 +208,7 @@
assertEquals(PROJECT_ID.get(), service.getBigtableOptions().getProjectId());
assertEquals(INSTANCE_ID.get(), service.getBigtableOptions().getInstanceId());
- assertEquals(true, service.getBigtableOptions().getBulkOptions().useBulkApi());
+ assertTrue(service.getBigtableOptions().getBulkOptions().useBulkApi());
}
@Test
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
index e995e7a..e3817fb 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
@@ -25,9 +25,9 @@
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasKey;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasLabel;
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasValue;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Verify.verifyNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Verify.verifyNotNull;
import static org.hamcrest.Matchers.allOf;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.greaterThan;
@@ -37,6 +37,7 @@
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
import com.google.bigtable.v2.Cell;
import com.google.bigtable.v2.Column;
@@ -75,6 +76,8 @@
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.extensions.gcp.auth.TestCredential;
import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
import org.apache.beam.sdk.extensions.protobuf.ByteStringCoder;
@@ -90,19 +93,30 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipeline.PipelineRunMissingException;
+import org.apache.beam.sdk.testing.TestStream;
import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.Wait;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matchers;
import org.hamcrest.collection.IsIterableContainingInAnyOrder;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -296,11 +310,12 @@
@Test
public void testWriteValidationFailsMissingInstanceId() {
- BigtableIO.Write write =
+ BigtableIO.WriteWithResults write =
BigtableIO.write()
.withTableId("table")
.withProjectId("project")
- .withBigtableOptions(new BigtableOptions.Builder().build());
+ .withBigtableOptions(new BigtableOptions.Builder().build())
+ .withWriteResults();
thrown.expect(IllegalArgumentException.class);
@@ -309,11 +324,12 @@
@Test
public void testWriteValidationFailsMissingProjectId() {
- BigtableIO.Write write =
+ BigtableIO.WriteWithResults write =
BigtableIO.write()
.withTableId("table")
.withInstanceId("instance")
- .withBigtableOptions(new BigtableOptions.Builder().build());
+ .withBigtableOptions(new BigtableOptions.Builder().build())
+ .withWriteResults();
thrown.expect(IllegalArgumentException.class);
@@ -322,10 +338,11 @@
@Test
public void testWriteValidationFailsMissingInstanceIdAndProjectId() {
- BigtableIO.Write write =
+ BigtableIO.WriteWithResults write =
BigtableIO.write()
.withTableId("table")
- .withBigtableOptions(new BigtableOptions.Builder().build());
+ .withBigtableOptions(new BigtableOptions.Builder().build())
+ .withWriteResults();
thrown.expect(IllegalArgumentException.class);
@@ -334,7 +351,7 @@
@Test
public void testWriteValidationFailsMissingOptionsAndInstanceAndProject() {
- BigtableIO.Write write = BigtableIO.write().withTableId("table");
+ BigtableIO.WriteWithResults write = BigtableIO.write().withTableId("table").withWriteResults();
thrown.expect(IllegalArgumentException.class);
@@ -1158,6 +1175,125 @@
assertEquals(ByteString.copyFromUtf8(value), rows.get(ByteString.copyFromUtf8(key)));
}
+ /** Tests that at least one result is emitted per element written in the global window. */
+ @Test
+ public void testWritingEmitsResultsWhenDoneInGlobalWindow() throws Exception {
+ final String table = "table";
+ final String key = "key";
+ final String value = "value";
+
+ service.createTable(table);
+
+ PCollection<BigtableWriteResult> results =
+ p.apply("single row", Create.of(makeWrite(key, value)).withCoder(bigtableCoder))
+ .apply("write", defaultWrite.withTableId(table).withWriteResults());
+ PAssert.that(results)
+ .inWindow(GlobalWindow.INSTANCE)
+ .containsInAnyOrder(BigtableWriteResult.create(1));
+
+ p.run();
+ }
+
+ /**
+ * Tests that the outputs of the Bigtable writer are correctly windowed, and can be used in a
+ * Wait.on transform as the trigger.
+ */
+ @Test
+ public void testWritingAndWaitingOnResults() throws Exception {
+ final String table = "table";
+ final String key = "key";
+ final String value = "value";
+
+ service.createTable(table);
+
+ Instant elementTimestamp = Instant.parse("2019-06-10T00:00:00");
+ Duration windowDuration = Duration.standardMinutes(1);
+
+ TestStream<KV<ByteString, Iterable<Mutation>>> writeInputs =
+ TestStream.create(bigtableCoder)
+ .advanceWatermarkTo(elementTimestamp)
+ .addElements(makeWrite(key, value))
+ .advanceWatermarkToInfinity();
+
+ TestStream<String> testInputs =
+ TestStream.create(StringUtf8Coder.of())
+ .advanceWatermarkTo(elementTimestamp)
+ .addElements("done")
+ .advanceWatermarkToInfinity();
+
+ PCollection<BigtableWriteResult> writes =
+ p.apply("rows", writeInputs)
+ .apply(
+ "window rows",
+ Window.<KV<ByteString, Iterable<Mutation>>>into(FixedWindows.of(windowDuration))
+ .withAllowedLateness(Duration.ZERO))
+ .apply("write", defaultWrite.withTableId(table).withWriteResults());
+
+ PCollection<String> inputs =
+ p.apply("inputs", testInputs)
+ .apply("window inputs", Window.into(FixedWindows.of(windowDuration)))
+ .apply("wait", Wait.on(writes));
+
+ BoundedWindow expectedWindow = new IntervalWindow(elementTimestamp, windowDuration);
+
+ PAssert.that(inputs).inWindow(expectedWindow).containsInAnyOrder("done");
+
+ p.run();
+ }
+
+ /**
+ * A DoFn used to generate N outputs, where N is the input. Used to generate bundles of >= 1
+ * element.
+ */
+ private static class WriteGeneratorDoFn
+ extends DoFn<Integer, KV<ByteString, Iterable<Mutation>>> {
+ @ProcessElement
+ public void processElement(ProcessContext ctx) {
+ for (int i = 0; i < ctx.element(); i++) {
+ ctx.output(makeWrite("key", "value"));
+ }
+ }
+ }
+
+ /** Tests that at least one result is emitted per element written in each window. */
+ @Test
+ public void testWritingEmitsResultsWhenDoneInFixedWindow() throws Exception {
+ final String table = "table";
+ final String key = "key";
+ final String value = "value";
+
+ service.createTable(table);
+
+ Instant elementTimestamp = Instant.parse("2019-06-10T00:00:00");
+ Duration windowDuration = Duration.standardMinutes(1);
+
+ TestStream<Integer> input =
+ TestStream.create(VarIntCoder.of())
+ .advanceWatermarkTo(elementTimestamp)
+ .addElements(1)
+ .advanceWatermarkTo(elementTimestamp.plus(windowDuration))
+ .addElements(2)
+ .advanceWatermarkToInfinity();
+
+ BoundedWindow expectedFirstWindow = new IntervalWindow(elementTimestamp, windowDuration);
+ BoundedWindow expectedSecondWindow =
+ new IntervalWindow(elementTimestamp.plus(windowDuration), windowDuration);
+
+ PCollection<BigtableWriteResult> results =
+ p.apply("rows", input)
+ .apply("window", Window.into(FixedWindows.of(windowDuration)))
+ .apply("expand", ParDo.of(new WriteGeneratorDoFn()))
+ .apply("write", defaultWrite.withTableId(table).withWriteResults());
+ PAssert.that(results)
+ .inWindow(expectedFirstWindow)
+ .containsInAnyOrder(BigtableWriteResult.create(1));
+ PAssert.that(results)
+ .inWindow(expectedSecondWindow)
+ .containsInAnyOrder(BigtableWriteResult.create(2));
+
+ p.run();
+ }
+
/** Tests that when writing to a non-existent table, the write fails. */
@Test
public void testWritingFailsTableDoesNotExist() throws Exception {
@@ -1287,7 +1423,7 @@
BigtableIO.Write write = BigtableIO.write().withBigtableOptions(optionsBuilder.build());
BigtableOptions options = write.getBigtableOptions();
- assertEquals(true, options.getBulkOptions().useBulkApi());
+ assertTrue(options.getBulkOptions().useBulkApi());
assertEquals(maxInflightRpcs, options.getBulkOptions().getMaxInflightRpcs());
assertEquals(initialBackoffMillis, options.getRetryOptions().getInitialBackoffMillis());
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java
index b580cc3..69be079 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java
@@ -47,7 +47,7 @@
import org.apache.beam.sdk.io.range.ByteKeyRange;
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
index f5424af..3ddc500 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteIT.java
@@ -49,7 +49,7 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.hamcrest.Matchers;
import org.junit.After;
import org.junit.Before;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
index 20b0564..de18790 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
@@ -137,8 +137,7 @@
DatastoreV1.Read initialRead =
DatastoreIO.v1().read().withProjectId(PROJECT_ID).withQuery(QUERY).withNamespace(NAMESPACE);
- when(mockDatastoreFactory.getDatastore(
- any(PipelineOptions.class), any(String.class), any(String.class)))
+ when(mockDatastoreFactory.getDatastore(any(PipelineOptions.class), any(String.class), any()))
.thenReturn(mockDatastore);
when(mockDatastoreFactory.getQuerySplitter()).thenReturn(mockQuerySplitter);
}
@@ -402,8 +401,8 @@
Entity entity = Entity.newBuilder().setKey(key).build();
UpsertFn upsertFn = new UpsertFn();
- Mutation exceptedMutation = makeUpsert(entity).build();
- assertEquals(upsertFn.apply(entity), exceptedMutation);
+ Mutation expectedMutation = makeUpsert(entity).build();
+ assertEquals(expectedMutation, upsertFn.apply(entity));
}
/** Test that entities with incomplete keys cannot be deleted. */
@@ -426,8 +425,8 @@
Entity entity = Entity.newBuilder().setKey(key).build();
DeleteEntityFn deleteEntityFn = new DeleteEntityFn();
- Mutation exceptedMutation = makeDelete(entity.getKey()).build();
- assertEquals(deleteEntityFn.apply(entity), exceptedMutation);
+ Mutation expectedMutation = makeDelete(entity.getKey()).build();
+ assertEquals(expectedMutation, deleteEntityFn.apply(entity));
}
/** Test that incomplete keys cannot be deleted. */
@@ -448,8 +447,8 @@
Key key = makeKey("bird", "finch").build();
DeleteKeyFn deleteKeyFn = new DeleteKeyFn();
- Mutation exceptedMutation = makeDelete(key).build();
- assertEquals(deleteKeyFn.apply(key), exceptedMutation);
+ Mutation expectedMutation = makeDelete(key).build();
+ assertEquals(expectedMutation, deleteKeyFn.apply(key));
}
@Test
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClientTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClientTest.java
index fd99d29..4161122 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClientTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClientTest.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.ProjectPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.SubscriptionPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClientTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClientTest.java
index 5a1ec94..7c53170 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClientTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClientTest.java
@@ -45,9 +45,9 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.OutgoingMessage;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.SubscriptionPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
index 5979373..65b89a7 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java
@@ -31,6 +31,8 @@
import com.google.api.client.util.Clock;
import java.io.IOException;
import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
import java.util.List;
import java.util.Objects;
import java.util.Set;
@@ -38,10 +40,12 @@
import java.util.stream.Collectors;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.reflect.AvroSchema;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.io.AvroGeneratedUser;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.IncomingMessage;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.SubscriptionPath;
@@ -52,12 +56,16 @@
import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
import org.junit.After;
import org.junit.Rule;
import org.junit.Test;
@@ -67,7 +75,6 @@
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.junit.runners.model.Statement;
-import org.testng.collections.Lists;
/** Tests for PubsubIO Read and Write transforms. */
@RunWith(JUnit4.class)
@@ -244,6 +251,23 @@
}
@Test
+ public void testReadWithPubsubGrpcClientFactory() {
+ String topic = "projects/project/topics/topic";
+ PubsubIO.Read<String> read =
+ PubsubIO.readStrings()
+ .fromTopic(StaticValueProvider.of(topic))
+ .withClientFactory(PubsubGrpcClient.FACTORY)
+ .withTimestampAttribute("myTimestamp")
+ .withIdAttribute("myId");
+
+ DisplayData displayData = DisplayData.from(read);
+
+ assertThat(displayData, hasDisplayItem("topic", topic));
+ assertThat(displayData, hasDisplayItem("timestampAttribute", "myTimestamp"));
+ assertThat(displayData, hasDisplayItem("idAttribute", "myId"));
+ }
+
+ @Test
public void testWriteDisplayData() {
String topic = "projects/project/topics/topic";
PubsubIO.Write<?> write =
@@ -275,11 +299,15 @@
int intField;
String stringField;
+ @AvroSchema("{\"type\": \"long\", \"logicalType\": \"timestamp-millis\"}")
+ public DateTime timestamp;
+
public GenericClass() {}
- public GenericClass(int intField, String stringField) {
+ public GenericClass(int intField, String stringField, DateTime timestamp) {
this.intField = intField;
this.stringField = stringField;
+ this.timestamp = timestamp;
}
@Override
@@ -287,12 +315,13 @@
return MoreObjects.toStringHelper(getClass())
.add("intField", intField)
.add("stringField", stringField)
+ .add("timestamp", timestamp)
.toString();
}
@Override
public int hashCode() {
- return Objects.hash(intField, stringField);
+ return Objects.hash(intField, stringField, timestamp);
}
@Override
@@ -301,7 +330,9 @@
return false;
}
GenericClass o = (GenericClass) other;
- return Objects.equals(intField, o.intField) && Objects.equals(stringField, o.stringField);
+ return Objects.equals(intField, o.intField)
+ && Objects.equals(stringField, o.stringField)
+ && Objects.equals(timestamp, o.timestamp);
}
}
@@ -405,7 +436,11 @@
public void testAvroPojo() {
AvroCoder<GenericClass> coder = AvroCoder.of(GenericClass.class);
List<GenericClass> inputs =
- Lists.newArrayList(new GenericClass(1, "foo"), new GenericClass(2, "bar"));
+ Lists.newArrayList(
+ new GenericClass(
+ 1, "foo", new DateTime().withDate(2019, 10, 1).withZone(DateTimeZone.UTC)),
+ new GenericClass(
+ 2, "bar", new DateTime().withDate(1986, 10, 1).withZone(DateTimeZone.UTC)));
setupTestClient(inputs, coder);
PCollection<GenericClass> read =
readPipeline.apply(
@@ -435,4 +470,50 @@
PAssert.that(read).containsInAnyOrder(inputs);
readPipeline.run();
}
+
+ @Test
+ public void testWriteWithPubsubGrpcClientFactory() {
+ String topic = "projects/project/topics/topic";
+ PubsubIO.Write<?> write =
+ PubsubIO.writeStrings()
+ .to(topic)
+ .withClientFactory(PubsubGrpcClient.FACTORY)
+ .withTimestampAttribute("myTimestamp")
+ .withIdAttribute("myId");
+
+ DisplayData displayData = DisplayData.from(write);
+
+ assertThat(displayData, hasDisplayItem("topic", topic));
+ assertThat(displayData, hasDisplayItem("timestampAttribute", "myTimestamp"));
+ assertThat(displayData, hasDisplayItem("idAttribute", "myId"));
+ }
+
+ static class StringPayloadParseFn extends SimpleFunction<PubsubMessage, String> {
+ @Override
+ public String apply(PubsubMessage input) {
+ return new String(input.getPayload(), StandardCharsets.UTF_8);
+ }
+ }
+
+ @Test
+ public void testReadMessagesWithCoderAndParseFn() {
+ Coder<PubsubMessage> coder = PubsubMessagePayloadOnlyCoder.of();
+ List<PubsubMessage> inputs =
+ ImmutableList.of(
+ new PubsubMessage("foo".getBytes(StandardCharsets.UTF_8), new HashMap<>()),
+ new PubsubMessage("bar".getBytes(StandardCharsets.UTF_8), new HashMap<>()));
+ setupTestClient(inputs, coder);
+
+ PCollection<String> read =
+ readPipeline.apply(
+ PubsubIO.readMessagesWithCoderAndParseFn(
+ StringUtf8Coder.of(), new StringPayloadParseFn())
+ .fromSubscription(SUBSCRIPTION.getPath())
+ .withClock(CLOCK)
+ .withClientFactory(clientFactory));
+
+ List<String> outputs = ImmutableList.of("foo", "bar");
+ PAssert.that(read).containsInAnyOrder(outputs);
+ readPipeline.run();
+ }
}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClientTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClientTest.java
index 573e212..f7fc0f3 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClientTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubJsonClientTest.java
@@ -45,8 +45,8 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.ProjectPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.SubscriptionPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoderTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoderTest.java
new file mode 100644
index 0000000..b132814
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoderTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.nio.charset.StandardCharsets;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link PubsubMessagePayloadOnlyCoder}. */
+@RunWith(JUnit4.class)
+public class PubsubMessagePayloadOnlyCoderTest {
+
+ private static final String DATA = "testData";
+ private static final Coder<PubsubMessage> TEST_CODER = PubsubMessagePayloadOnlyCoder.of();
+ private static final PubsubMessage TEST_VALUE =
+ new PubsubMessage(DATA.getBytes(StandardCharsets.UTF_8), null);
+
+ @Test
+ public void testValueEncodable() throws Exception {
+ SerializableUtils.ensureSerializableByCoder(TEST_CODER, TEST_VALUE, "error");
+ }
+
+ @Test
+ public void testCoderDecodeEncodeEqual() throws Exception {
+ CoderProperties.structuralValueDecodeEncodeEqual(TEST_CODER, TEST_VALUE);
+ }
+
+ @Test
+ public void testEncodedTypeDescriptor() throws Exception {
+ TypeDescriptor<PubsubMessage> typeDescriptor = new TypeDescriptor<PubsubMessage>() {};
+ assertThat(TEST_CODER.getEncodedTypeDescriptor(), equalTo(typeDescriptor));
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoderTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoderTest.java
new file mode 100644
index 0000000..5a061ad
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoderTest.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link PubsubMessageWithAttributesAndMessageIdCoder}. */
+@RunWith(JUnit4.class)
+public class PubsubMessageWithAttributesAndMessageIdCoderTest {
+
+ private static final String DATA = "testData";
+ private static final String MESSAGE_ID = "testMessageId";
+ private static final Map<String, String> ATTRIBUTES =
+ new ImmutableMap.Builder<String, String>().put("1", "hello").build();
+ private static final Coder<PubsubMessage> TEST_CODER =
+ PubsubMessageWithAttributesAndMessageIdCoder.of();
+ private static final PubsubMessage TEST_VALUE =
+ new PubsubMessage(DATA.getBytes(StandardCharsets.UTF_8), ATTRIBUTES, MESSAGE_ID);
+
+ @Test
+ public void testValueEncodable() throws Exception {
+ SerializableUtils.ensureSerializableByCoder(TEST_CODER, TEST_VALUE, "error");
+ }
+
+ @Test
+ public void testCoderDecodeEncodeEqual() throws Exception {
+ CoderProperties.structuralValueDecodeEncodeEqual(TEST_CODER, TEST_VALUE);
+ }
+
+ @Test
+ public void testEncodedTypeDescriptor() throws Exception {
+ TypeDescriptor<PubsubMessage> typeDescriptor = new TypeDescriptor<PubsubMessage>() {};
+ assertThat(TEST_CODER.getEncodedTypeDescriptor(), equalTo(typeDescriptor));
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesCoderTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesCoderTest.java
new file mode 100644
index 0000000..eb33fd3
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesCoderTest.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link PubsubMessageWithAttributesCoder}. */
+@RunWith(JUnit4.class)
+public class PubsubMessageWithAttributesCoderTest {
+
+ private static final String DATA = "testData";
+ private static final Map<String, String> ATTRIBUTES =
+ new ImmutableMap.Builder<String, String>().put("1", "hello").build();
+ private static final Coder<PubsubMessage> TEST_CODER = PubsubMessageWithAttributesCoder.of();
+ private static final PubsubMessage TEST_VALUE =
+ new PubsubMessage(DATA.getBytes(StandardCharsets.UTF_8), ATTRIBUTES);
+
+ @Test
+ public void testValueEncodable() throws Exception {
+ SerializableUtils.ensureSerializableByCoder(TEST_CODER, TEST_VALUE, "error");
+ }
+
+ @Test
+ public void testCoderDecodeEncodeEqual() throws Exception {
+ CoderProperties.structuralValueDecodeEncodeEqual(TEST_CODER, TEST_VALUE);
+ }
+
+ @Test
+ public void testEncodedTypeDescriptor() throws Exception {
+ TypeDescriptor<PubsubMessage> typeDescriptor = new TypeDescriptor<PubsubMessage>() {};
+ assertThat(TEST_CODER.getEncodedTypeDescriptor(), equalTo(typeDescriptor));
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoderTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoderTest.java
new file mode 100644
index 0000000..586b536
--- /dev/null
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoderTest.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.gcp.pubsub;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.nio.charset.StandardCharsets;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.testing.CoderProperties;
+import org.apache.beam.sdk.util.SerializableUtils;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link PubsubMessageWithMessageIdCoder}. */
+@RunWith(JUnit4.class)
+public class PubsubMessageWithMessageIdCoderTest {
+
+ private static final String DATA = "testData";
+ private static final String MESSAGE_ID = "testMessageId";
+ private static final Coder<PubsubMessage> TEST_CODER = PubsubMessageWithMessageIdCoder.of();
+ private static final PubsubMessage TEST_VALUE =
+ new PubsubMessage(DATA.getBytes(StandardCharsets.UTF_8), null, MESSAGE_ID);
+
+ @Test
+ public void testValueEncodable() throws Exception {
+ SerializableUtils.ensureSerializableByCoder(TEST_CODER, TEST_VALUE, "error");
+ }
+
+ @Test
+ public void testCoderDecodeEncodeEqual() throws Exception {
+ CoderProperties.structuralValueDecodeEncodeEqual(TEST_CODER, TEST_VALUE);
+ }
+
+ @Test
+ public void testEncodedTypeDescriptor() throws Exception {
+ TypeDescriptor<PubsubMessage> typeDescriptor = new TypeDescriptor<PubsubMessage>() {};
+ assertThat(TEST_CODER.getEncodedTypeDescriptor(), equalTo(typeDescriptor));
+ }
+}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java
index c392aca..9856606 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java
@@ -17,20 +17,26 @@
*/
package org.apache.beam.sdk.io.gcp.pubsub;
+import java.util.Set;
import org.apache.beam.runners.direct.DirectOptions;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Supplier;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Supplier;
import org.joda.time.Duration;
import org.junit.Rule;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** Integration test for PubsubIO. */
@RunWith(JUnit4.class)
public class PubsubReadIT {
+ private static final Logger LOG = LoggerFactory.getLogger(PubsubReadIT.class);
@Rule public transient TestPubsubSignal signal = TestPubsubSignal.create();
@Rule public transient TestPipeline pipeline = TestPipeline.create();
@@ -61,4 +67,45 @@
// noop
}
}
+
+ @Test
+ public void testReadPubsubMessageId() throws Exception {
+ // The pipeline will never terminate on its own
+ pipeline.getOptions().as(DirectOptions.class).setBlockOnRun(false);
+
+ PCollection<PubsubMessage> messages =
+ pipeline.apply(
+ PubsubIO.readMessagesWithAttributesAndMessageId()
+ .fromTopic("projects/pubsub-public-data/topics/taxirides-realtime"));
+
+ messages.apply(
+ "isMessageIdNonNull",
+ signal.signalSuccessWhen(messages.getCoder(), new NonEmptyMessageIdCheck()));
+
+ Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
+ pipeline.apply(signal.signalStart());
+ PipelineResult job = pipeline.run();
+ start.get();
+
+ signal.waitForSuccess(Duration.standardMinutes(1));
+ // A runner may not support cancel
+ try {
+ job.cancel();
+ } catch (UnsupportedOperationException exc) {
+ // noop
+ }
+ }
+
+ private static class NonEmptyMessageIdCheck
+ implements SerializableFunction<Set<PubsubMessage>, Boolean> {
+ @Override
+ public Boolean apply(Set<PubsubMessage> input) {
+ for (PubsubMessage message : input) {
+ if (Strings.isNullOrEmpty(message.getMessageId())) {
+ return false;
+ }
+ }
+ return true;
+ }
+ }
}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClientTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClientTest.java
index 510acca..2b698f0 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClientTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubTestClientTest.java
@@ -29,9 +29,9 @@
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.SubscriptionPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubClient.TopicPath;
import org.apache.beam.sdk.io.gcp.pubsub.PubsubTestClient.PubsubTestClientFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java
index 25de4be..f588e05 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java
@@ -33,9 +33,9 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Rule;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSourceTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSourceTest.java
index fd0f017..b2dacf0 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSourceTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSourceTest.java
@@ -51,7 +51,7 @@
import org.apache.beam.sdk.testing.CoderProperties;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.After;
import org.junit.Rule;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationKeyEncoderTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationKeyEncoderTest.java
index 3a0c0e2..0e8c79b 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationKeyEncoderTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationKeyEncoderTest.java
@@ -28,7 +28,7 @@
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCodeTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCodeTest.java
index 16ba020..522ff78 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCodeTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/OrderedCodeTest.java
@@ -25,11 +25,11 @@
import com.google.auto.value.AutoValue;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.Bytes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedBytes;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.primitives.UnsignedInteger;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.Bytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedBytes;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.primitives.UnsignedInteger;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/ReadSpannerSchemaTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/ReadSpannerSchemaTest.java
index 0d136b1..6e8a91d 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/ReadSpannerSchemaTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/ReadSpannerSchemaTest.java
@@ -85,7 +85,7 @@
new ArgumentMatcher<Statement>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(Statement argument) {
if (!(argument instanceof Statement)) {
return false;
}
@@ -107,7 +107,7 @@
new ArgumentMatcher<Statement>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(Statement argument) {
if (!(argument instanceof Statement)) {
return false;
}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOWriteTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOWriteTest.java
index d5553f5..83f68ef 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOWriteTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIOWriteTest.java
@@ -62,10 +62,9 @@
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.hamcrest.Description;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.junit.Before;
import org.junit.Rule;
@@ -161,7 +160,7 @@
new ArgumentMatcher<Statement>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(Statement argument) {
if (!(argument instanceof Statement)) {
return false;
}
@@ -183,7 +182,7 @@
new ArgumentMatcher<Statement>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(Statement argument) {
if (!(argument instanceof Statement)) {
return false;
}
@@ -710,7 +709,7 @@
new ArgumentMatcher<Iterable<Mutation>>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(Iterable<Mutation> argument) {
if (!(argument instanceof Iterable)) {
return false;
}
@@ -719,8 +718,8 @@
}
@Override
- public void describeTo(Description description) {
- description.appendText("Iterable must match ").appendValue(mutations);
+ public String toString() {
+ return "Iterable must match " + mutations;
}
});
}
@@ -730,13 +729,13 @@
new ArgumentMatcher<Iterable<Mutation>>() {
@Override
- public boolean matches(Object argument) {
+ public boolean matches(Iterable<Mutation> argument) {
return argument instanceof Iterable && Iterables.size((Iterable<?>) argument) == size;
}
@Override
- public void describeTo(Description description) {
- description.appendText("The size of the iterable must equal ").appendValue(size);
+ public String toString() {
+ return "The size of the iterable must equal " + size;
}
});
}
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
index 9151394..5b2fc82 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
@@ -45,9 +45,9 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.Wait;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicate;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Predicates;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicate;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Predicates;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
import org.hamcrest.TypeSafeMatcher;
import org.junit.After;
import org.junit.Before;
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/storage/GcsKmsKeyIT.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/storage/GcsKmsKeyIT.java
index 1733334..9f092e7 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/storage/GcsKmsKeyIT.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/storage/GcsKmsKeyIT.java
@@ -18,9 +18,8 @@
package org.apache.beam.sdk.io.gcp.storage;
import static org.hamcrest.CoreMatchers.equalTo;
-import static org.hamcrest.CoreMatchers.notNullValue;
-import static org.hamcrest.CoreMatchers.startsWith;
import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertNotNull;
import java.io.IOException;
import java.util.Collections;
@@ -37,13 +36,13 @@
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.io.fs.ResolveOptions.StandardResolveOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.FileChecksumMatcher;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.junit.BeforeClass;
+import org.apache.beam.sdk.testing.UsesKms;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
@@ -53,27 +52,25 @@
/** Integration test for GCS CMEK support. */
@RunWith(JUnit4.class)
+@Category(UsesKms.class)
public class GcsKmsKeyIT {
private static final String INPUT_FILE = "gs://dataflow-samples/shakespeare/kinglear.txt";
private static final String EXPECTED_CHECKSUM = "b9778bfac7fa8b934e42a322ef4bd4706b538fd0";
- @BeforeClass
- public static void setup() {
- PipelineOptionsFactory.register(TestPipelineOptions.class);
- }
-
/**
- * Tests writing to gcpTempLocation with --dataflowKmsKey set on the command line. Verifies that
+ * Tests writing to tempLocation with --dataflowKmsKey set on the command line. Verifies that
* resulting output uses specified key and is readable. Does not verify any temporary files.
+ *
+ * <p>This test verifies that GCS file copies work with CMEK-enabled files.
*/
@Test
public void testGcsWriteWithKmsKey() {
TestPipelineOptions options =
TestPipeline.testingPipelineOptions().as(TestPipelineOptions.class);
+ assertNotNull(options.getTempRoot());
+ options.setTempLocation(options.getTempRoot() + "/testGcsWriteWithKmsKey");
GcsOptions gcsOptions = options.as(GcsOptions.class);
- final String expectedKmsKey = gcsOptions.getDataflowKmsKey();
- assertThat(expectedKmsKey, notNullValue());
ResourceId filenamePrefix =
FileSystems.matchNewResource(gcsOptions.getGcpTempLocation(), true)
@@ -100,11 +97,7 @@
for (Metadata metadata : matchResult.metadata()) {
String kmsKey =
gcsUtil.getObject(GcsPath.fromUri(metadata.resourceId().toString())).getKmsKeyName();
- // Returned kmsKey should have a version suffix.
- assertThat(
- metadata.resourceId().toString(),
- kmsKey,
- startsWith(expectedKmsKey + "/cryptoKeyVersions/"));
+ assertNotNull(kmsKey);
}
} catch (IOException e) {
throw new AssertionError(e);
diff --git a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcherTest.java b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcherTest.java
index 55060df..89ebdb5 100644
--- a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcherTest.java
+++ b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcherTest.java
@@ -27,7 +27,7 @@
import com.google.api.services.bigquery.model.TableRow;
import java.math.BigInteger;
import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/io/hadoop-common/build.gradle b/sdks/java/io/hadoop-common/build.gradle
index c77235e..08f60c6 100644
--- a/sdks/java/io/hadoop-common/build.gradle
+++ b/sdks/java/io/hadoop-common/build.gradle
@@ -17,13 +17,13 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hadoop.common')
description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop Common"
ext.summary = "Library to add shared Hadoop classes among Beam IOs."
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
provided library.java.hadoop_client
provided library.java.hadoop_common
provided library.java.hadoop_mapreduce_client_core
diff --git a/sdks/java/io/hadoop-file-system/build.gradle b/sdks/java/io/hadoop-file-system/build.gradle
index a4e5c70..8ebdc93 100644
--- a/sdks/java/io/hadoop-file-system/build.gradle
+++ b/sdks/java/io/hadoop-file-system/build.gradle
@@ -17,22 +17,21 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hdfs')
description = "Apache Beam :: SDKs :: Java :: IO :: Hadoop File System"
ext.summary = "Library to read and write Hadoop/HDFS file formats from Beam."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.jackson_core
- shadow library.java.jackson_databind
- shadow library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.jackson_core
+ compile library.java.jackson_databind
+ compile library.java.slf4j_api
provided library.java.hadoop_client
provided library.java.hadoop_common
provided library.java.hadoop_mapreduce_client_core
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile library.java.guava_testlib
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.mockito_core
@@ -40,5 +39,5 @@
testCompile library.java.hadoop_minicluster
testCompile library.java.hadoop_hdfs_tests
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java
index 72e653b..a589759 100644
--- a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java
+++ b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystem.java
@@ -37,8 +37,8 @@
import org.apache.beam.sdk.io.fs.MatchResult;
import org.apache.beam.sdk.io.fs.MatchResult.Metadata;
import org.apache.beam.sdk.io.fs.MatchResult.Status;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSInputStream;
@@ -74,14 +74,19 @@
* </ul>
*/
class HadoopFileSystem extends FileSystem<HadoopResourceId> {
+
private static final Logger LOG = LoggerFactory.getLogger(HadoopFileSystem.class);
@VisibleForTesting static final String LOG_CREATE_DIRECTORY = "Creating directory %s";
@VisibleForTesting static final String LOG_DELETING_EXISTING_FILE = "Deleting existing file %s";
- @VisibleForTesting final org.apache.hadoop.fs.FileSystem fileSystem;
- HadoopFileSystem(Configuration configuration) throws IOException {
- this.fileSystem = org.apache.hadoop.fs.FileSystem.newInstance(configuration);
+ private final String scheme;
+
+ @VisibleForTesting final Configuration configuration;
+
+ HadoopFileSystem(String scheme, Configuration configuration) {
+ this.scheme = scheme;
+ this.configuration = configuration;
}
@Override
@@ -89,23 +94,22 @@
ImmutableList.Builder<MatchResult> resultsBuilder = ImmutableList.builder();
for (String spec : specs) {
try {
- Set<Metadata> metadata = new HashSet<>();
-
- FileStatus[] fileStatuses = fileSystem.globStatus(new Path(spec));
- if (fileStatuses != null) {
- for (FileStatus fileStatus : fileStatuses) {
- if (fileStatus.isFile()) {
+ final Set<Metadata> metadata = new HashSet<>();
+ if (spec.contains("**")) {
+ // recursive glob
+ int index = spec.indexOf("**");
+ metadata.addAll(
+ matchRecursiveGlob(spec.substring(0, index + 1), spec.substring(index + 1)));
+ } else {
+ // normal glob
+ final Path path = new Path(spec);
+ final FileStatus[] fileStatuses = path.getFileSystem(configuration).globStatus(path);
+ if (fileStatuses != null) {
+ for (FileStatus fileStatus : fileStatuses) {
metadata.add(toMetadata(fileStatus));
}
}
}
-
- if (spec.contains("**")) {
- int index = spec.indexOf("**");
- metadata.addAll(
- matchRecursiveGlob(spec.substring(0, index + 1), spec.substring(index + 1)));
- }
-
if (metadata.isEmpty()) {
resultsBuilder.add(MatchResult.create(Status.NOT_FOUND, Collections.emptyList()));
} else {
@@ -120,10 +124,11 @@
private Set<Metadata> matchRecursiveGlob(String directorySpec, String fileSpec)
throws IOException {
+ final org.apache.hadoop.fs.FileSystem fs = new Path(directorySpec).getFileSystem(configuration);
Set<Metadata> metadata = new HashSet<>();
if (directorySpec.contains("*")) {
// An abstract directory with a wildcard is converted to concrete directories to search.
- FileStatus[] directoryStatuses = fileSystem.globStatus(new Path(directorySpec));
+ FileStatus[] directoryStatuses = fs.globStatus(new Path(directorySpec));
for (FileStatus directoryStatus : directoryStatuses) {
if (directoryStatus.isDirectory()) {
metadata.addAll(
@@ -132,7 +137,7 @@
}
} else {
// A concrete directory is searched.
- FileStatus[] fileStatuses = fileSystem.globStatus(new Path(directorySpec + "/" + fileSpec));
+ FileStatus[] fileStatuses = fs.globStatus(new Path(directorySpec + "/" + fileSpec));
for (FileStatus fileStatus : fileStatuses) {
if (fileStatus.isFile()) {
metadata.add(toMetadata(fileStatus));
@@ -140,7 +145,7 @@
}
// All sub-directories of a concrete directory are searched.
- FileStatus[] directoryStatuses = fileSystem.globStatus(new Path(directorySpec + "/*"));
+ FileStatus[] directoryStatuses = fs.globStatus(new Path(directorySpec + "/*"));
for (FileStatus directoryStatus : directoryStatuses) {
if (directoryStatus.isDirectory()) {
metadata.addAll(
@@ -173,19 +178,24 @@
@Override
protected WritableByteChannel create(HadoopResourceId resourceId, CreateOptions createOptions)
throws IOException {
- return Channels.newChannel(fileSystem.create(resourceId.toPath()));
+ return Channels.newChannel(
+ resourceId.toPath().getFileSystem(configuration).create(resourceId.toPath()));
}
@Override
protected ReadableByteChannel open(HadoopResourceId resourceId) throws IOException {
- FileStatus fileStatus = fileSystem.getFileStatus(resourceId.toPath());
- return new HadoopSeekableByteChannel(fileStatus, fileSystem.open(resourceId.toPath()));
+ final org.apache.hadoop.fs.FileSystem fs = resourceId.toPath().getFileSystem(configuration);
+ final FileStatus fileStatus = fs.getFileStatus(resourceId.toPath());
+ return new HadoopSeekableByteChannel(fileStatus, fs.open(resourceId.toPath()));
}
@Override
protected void copy(List<HadoopResourceId> srcResourceIds, List<HadoopResourceId> destResourceIds)
throws IOException {
for (int i = 0; i < srcResourceIds.size(); ++i) {
+ // this enforces src and dest file systems to match
+ final org.apache.hadoop.fs.FileSystem fs =
+ srcResourceIds.get(i).toPath().getFileSystem(configuration);
// Unfortunately HDFS FileSystems don't support a native copy operation so we are forced
// to use the inefficient implementation found in FileUtil which copies all the bytes through
// the local machine.
@@ -194,15 +204,15 @@
// implementing it. The DFSFileSystem implemented concat by deleting the srcs after which
// is not what we want. Also, all the other FileSystem implementations I saw threw
// UnsupportedOperationException within concat.
- boolean success =
+ final boolean success =
FileUtil.copy(
- fileSystem,
+ fs,
srcResourceIds.get(i).toPath(),
- fileSystem,
+ fs,
destResourceIds.get(i).toPath(),
false,
true,
- fileSystem.getConf());
+ fs.getConf());
if (!success) {
// Defensive coding as this should not happen in practice
throw new IOException(
@@ -238,40 +248,45 @@
throws IOException {
for (int i = 0; i < srcResourceIds.size(); ++i) {
- Path src = srcResourceIds.get(i).toPath();
- Path dest = destResourceIds.get(i).toPath();
+ final Path srcPath = srcResourceIds.get(i).toPath();
+ final Path destPath = destResourceIds.get(i).toPath();
+
+ // this enforces src and dest file systems to match
+ final org.apache.hadoop.fs.FileSystem fs = srcPath.getFileSystem(configuration);
// rename in HDFS requires the target directory to exist or silently fails (BEAM-4861)
- mkdirs(dest);
+ mkdirs(destPath);
- boolean success = fileSystem.rename(src, dest);
+ boolean success = fs.rename(srcPath, destPath);
// If the failure was due to the file already existing, delete and retry (BEAM-5036).
// This should be the exceptional case, so handle here rather than incur the overhead of
// testing first
- if (!success && fileSystem.exists(src) && fileSystem.exists(dest)) {
+ if (!success && fs.exists(srcPath) && fs.exists(destPath)) {
LOG.debug(
- String.format(LOG_DELETING_EXISTING_FILE, Path.getPathWithoutSchemeAndAuthority(dest)));
- fileSystem.delete(dest, false); // not recursive
- success = fileSystem.rename(src, dest);
+ String.format(
+ LOG_DELETING_EXISTING_FILE, Path.getPathWithoutSchemeAndAuthority(destPath)));
+ fs.delete(destPath, false); // not recursive
+ success = fs.rename(srcPath, destPath);
}
if (!success) {
- if (!fileSystem.exists(src)) {
+ if (!fs.exists(srcPath)) {
throw new FileNotFoundException(
- String.format("Unable to rename resource %s to %s as source not found.", src, dest));
+ String.format(
+ "Unable to rename resource %s to %s as source not found.", srcPath, destPath));
- } else if (fileSystem.exists(dest)) {
+ } else if (fs.exists(destPath)) {
throw new FileAlreadyExistsException(
String.format(
"Unable to rename resource %s to %s as destination already exists and couldn't be deleted.",
- src, dest));
+ srcPath, destPath));
} else {
throw new IOException(
String.format(
"Unable to rename resource %s to %s. No further information provided by underlying filesystem.",
- src, dest));
+ srcPath, destPath));
}
}
}
@@ -279,13 +294,13 @@
/** Ensures that the target directory exists for the given filePath. */
private void mkdirs(Path filePath) throws IOException {
- Path targetDirectory = filePath.getParent();
- if (!fileSystem.exists(targetDirectory)) {
+ final org.apache.hadoop.fs.FileSystem fs = filePath.getFileSystem(configuration);
+ final Path targetDirectory = filePath.getParent();
+ if (!fs.exists(targetDirectory)) {
LOG.debug(
String.format(
LOG_CREATE_DIRECTORY, Path.getPathWithoutSchemeAndAuthority(targetDirectory)));
- boolean success = fileSystem.mkdirs(targetDirectory);
- if (!success) {
+ if (!fs.mkdirs(targetDirectory)) {
throw new IOException(
String.format(
"Unable to create target directory %s. No further information provided by underlying filesystem.",
@@ -298,7 +313,8 @@
protected void delete(Collection<HadoopResourceId> resourceIds) throws IOException {
for (HadoopResourceId resourceId : resourceIds) {
// ignore response as issues are surfaced with exception
- fileSystem.delete(resourceId.toPath(), false);
+ final Path resourcePath = resourceId.toPath();
+ resourcePath.getFileSystem(configuration).delete(resourceId.toPath(), false);
}
}
@@ -315,7 +331,7 @@
@Override
protected String getScheme() {
- return fileSystem.getScheme();
+ return scheme;
}
/** An adapter around {@link FSDataInputStream} that implements {@link SeekableByteChannel}. */
diff --git a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java
index 05e7bc0..66646a8 100644
--- a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java
+++ b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java
@@ -27,10 +27,10 @@
import org.apache.beam.sdk.options.DefaultValueFactory;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.slf4j.Logger;
diff --git a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrar.java b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrar.java
index 0e73d50..01c528e 100644
--- a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrar.java
+++ b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrar.java
@@ -20,7 +20,7 @@
import com.google.auto.service.AutoService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/** {@link AutoService} registrar for {@link HadoopFileSystemOptions}. */
@AutoService(PipelineOptionsRegistrar.class)
diff --git a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java
index e2593f7..9d16a41 100644
--- a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java
+++ b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.java
@@ -17,48 +17,71 @@
*/
package org.apache.beam.sdk.io.hdfs;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.service.AutoService;
-import java.io.IOException;
+import java.net.URI;
+import java.util.Arrays;
import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
+import java.util.Set;
import javax.annotation.Nonnull;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.io.FileSystem;
import org.apache.beam.sdk.io.FileSystemRegistrar;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
/** {@link AutoService} registrar for the {@link HadoopFileSystem}. */
@AutoService(FileSystemRegistrar.class)
@Experimental(Kind.FILESYSTEM)
public class HadoopFileSystemRegistrar implements FileSystemRegistrar {
+ private static final List<String> HA_SCHEMES = Arrays.asList("hdfs", "webhdfs");
+
@Override
public Iterable<FileSystem> fromOptions(@Nonnull PipelineOptions options) {
- List<Configuration> configurations =
+ final List<Configuration> configurations =
options.as(HadoopFileSystemOptions.class).getHdfsConfiguration();
if (configurations == null) {
- configurations = Collections.emptyList();
+ // nothing to register
+ return Collections.emptyList();
}
checkArgument(
- configurations.size() <= 1,
+ configurations.size() == 1,
String.format(
"The %s currently only supports at most a single Hadoop configuration.",
HadoopFileSystemRegistrar.class.getSimpleName()));
- ImmutableList.Builder<FileSystem> builder = ImmutableList.builder();
- for (Configuration configuration : configurations) {
- try {
- builder.add(new HadoopFileSystem(configuration));
- } catch (IOException e) {
- throw new IllegalArgumentException(
- String.format(
- "Failed to construct Hadoop filesystem with configuration %s", configuration),
- e);
+ final ImmutableList.Builder<FileSystem> builder = ImmutableList.builder();
+ final Set<String> registeredSchemes = new HashSet<>();
+
+ // this will only do zero or one loop
+ final Configuration configuration = Iterables.getOnlyElement(configurations);
+ final String defaultFs = configuration.get(org.apache.hadoop.fs.FileSystem.FS_DEFAULT_NAME_KEY);
+ if (defaultFs != null && !defaultFs.isEmpty()) {
+ final String scheme =
+ Objects.requireNonNull(
+ URI.create(defaultFs).getScheme(),
+ String.format(
+ "Empty scheme for %s value.",
+ org.apache.hadoop.fs.FileSystem.FS_DEFAULT_NAME_KEY));
+ builder.add(new HadoopFileSystem(scheme, configuration));
+ registeredSchemes.add(scheme);
+ }
+ final String nameServices = configuration.get(DFSConfigKeys.DFS_NAMESERVICES);
+ if (nameServices != null && !nameServices.isEmpty()) {
+ // we can register schemes that are support by HA cluster
+ for (String scheme : HA_SCHEMES) {
+ if (!registeredSchemes.contains(scheme)) {
+ builder.add(new HadoopFileSystem(scheme, configuration));
+ }
}
}
return builder.build();
diff --git a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
index e0e6889..4e5f6e2 100644
--- a/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
+++ b/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopResourceId.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.hdfs;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.net.URI;
import java.util.Objects;
diff --git a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrarTest.java b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrarTest.java
index 72cfdcc..844a58a 100644
--- a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrarTest.java
+++ b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsRegistrarTest.java
@@ -22,7 +22,7 @@
import java.util.ServiceLoader;
import org.apache.beam.sdk.options.PipelineOptionsRegistrar;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsTest.java b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsTest.java
index 37f357d..8a590b3 100644
--- a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsTest.java
+++ b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptionsTest.java
@@ -30,9 +30,9 @@
import java.util.List;
import java.util.Map;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Maps;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Files;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Maps;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Files;
import org.apache.hadoop.conf.Configuration;
import org.hamcrest.Matchers;
import org.junit.Rule;
diff --git a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrarTest.java b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrarTest.java
index dff70f1..52e67c7 100644
--- a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrarTest.java
+++ b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrarTest.java
@@ -25,9 +25,9 @@
import org.apache.beam.sdk.io.FileSystem;
import org.apache.beam.sdk.io.FileSystemRegistrar;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.junit.After;
diff --git a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemTest.java b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemTest.java
index f8ef812..f9464f9 100644
--- a/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemTest.java
+++ b/sdks/java/io/hadoop-file-system/src/test/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemTest.java
@@ -34,7 +34,9 @@
import java.nio.channels.WritableByteChannel;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
+import java.util.Collections;
import java.util.List;
+import java.util.Objects;
import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.io.fs.CreateOptions.StandardCreateOptions;
@@ -46,10 +48,11 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.util.MimeTypes;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.junit.After;
import org.junit.Before;
@@ -80,7 +83,8 @@
MiniDFSCluster.Builder builder = new MiniDFSCluster.Builder(configuration);
hdfsCluster = builder.build();
hdfsClusterBaseUri = new URI(configuration.get("fs.defaultFS") + "/");
- fileSystem = new HadoopFileSystem(configuration);
+ fileSystem =
+ new HadoopFileSystem(Objects.requireNonNull(hdfsClusterBaseUri).getScheme(), configuration);
}
@After
@@ -200,6 +204,26 @@
}
@Test
+ public void testMatchDirectory() throws Exception {
+ create("dir/file", "data".getBytes(StandardCharsets.UTF_8));
+ final MatchResult matchResult =
+ Iterables.getOnlyElement(
+ fileSystem.match(Collections.singletonList(testPath("dir").toString())));
+ assertThat(
+ matchResult,
+ equalTo(
+ MatchResult.create(
+ Status.OK,
+ ImmutableList.of(
+ Metadata.builder()
+ .setResourceId(testPath("dir"))
+ .setIsReadSeekEfficient(true)
+ .setSizeBytes(0L)
+ .setLastModifiedMillis(lastModified("dir"))
+ .build()))));
+ }
+
+ @Test
public void testMatchForNonExistentFile() throws Exception {
create("testFileAA", "testDataAA".getBytes(StandardCharsets.UTF_8));
create("testFileBB", "testDataBB".getBytes(StandardCharsets.UTF_8));
@@ -446,7 +470,7 @@
HadoopFileSystemOptions options =
TestPipeline.testingPipelineOptions().as(HadoopFileSystemOptions.class);
- options.setHdfsConfiguration(ImmutableList.of(fileSystem.fileSystem.getConf()));
+ options.setHdfsConfiguration(ImmutableList.of(fileSystem.configuration));
FileSystems.setDefaultPipelineOptions(options);
PCollection<String> pc = p.apply(TextIO.read().from(testPath("testFile*").toString()));
PAssert.that(pc).containsInAnyOrder("testDataA", "testDataB", "testDataC");
@@ -473,8 +497,9 @@
}
private long lastModified(String relativePath) throws Exception {
- return fileSystem
- .fileSystem
+ final Path testPath = testPath(relativePath).toPath();
+ return testPath
+ .getFileSystem(fileSystem.configuration)
.getFileStatus(testPath(relativePath).toPath())
.getModificationTime();
}
diff --git a/sdks/java/io/hadoop-format/build.gradle b/sdks/java/io/hadoop-format/build.gradle
index 9e7dba7..20dba8d 100644
--- a/sdks/java/io/hadoop-format/build.gradle
+++ b/sdks/java/io/hadoop-format/build.gradle
@@ -19,7 +19,7 @@
import groovy.json.JsonOutput
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hadoop.format')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -44,18 +44,18 @@
}
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.vendored_guava_20_0
- shadow library.java.slf4j_api
- shadow project(path: ":sdks:java:io:hadoop-common", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_guava_26_0_jre
+ compile library.java.slf4j_api
+ compile project(":sdks:java:io:hadoop-common")
provided library.java.hadoop_common
provided library.java.hadoop_hdfs
provided library.java.hadoop_mapreduce_client_core
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:testing:test-utils", configuration: "shadowTest")
- testCompile project(path: ":sdks:java:io:jdbc", configuration: "shadow")
- testCompile project(path: ":examples:java", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
+ testCompile project(":sdks:java:io:jdbc")
+ testCompile project(path: ":examples:java", configuration: "testRuntime")
testCompile "org.elasticsearch.plugin:transport-netty4-client:$elastic_search_version"
testCompile "org.elasticsearch.client:transport:$elastic_search_version"
@@ -82,15 +82,15 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
- shadow library.java.commons_io_2x
+ testRuntimeOnly project(":runners:direct-java")
+ compile library.java.commons_io_2x
- delegate.add("sparkRunner", project(path: ":sdks:java:io:hadoop-format", configuration: "shadow"))
- delegate.add("sparkRunner", project(path: ":sdks:java:io:hadoop-format", configuration: "shadowTest"))
+ delegate.add("sparkRunner", project(":sdks:java:io:hadoop-format"))
+ delegate.add("sparkRunner", project(path: ":sdks:java:io:hadoop-format", configuration: "testRuntime"))
- sparkRunner project(path: ":examples:java", configuration: "shadowTest")
- sparkRunner project(path: ":runners:spark", configuration: "shadow")
- sparkRunner project(path: ":sdks:java:io:hadoop-file-system", configuration: "shadow")
+ sparkRunner project(path: ":examples:java", configuration: "testRuntime")
+ sparkRunner project(":runners:spark")
+ sparkRunner project(":sdks:java:io:hadoop-file-system")
sparkRunner library.java.spark_streaming
sparkRunner library.java.spark_core
}
diff --git a/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java b/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java
index 60f60b2..4353868 100644
--- a/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java
+++ b/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HDFSSynchronization.java
@@ -20,7 +20,7 @@
import java.io.IOException;
import java.io.Serializable;
import java.util.Random;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
diff --git a/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java b/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
index b39ec80..3095421 100644
--- a/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
+++ b/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.io.hadoop.format;
import static java.util.Objects.requireNonNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import java.io.DataInputStream;
@@ -75,9 +75,9 @@
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.WindowingStrategy;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AtomicDouble;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AtomicDouble;
import org.apache.hadoop.conf.Configurable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.ObjectWritable;
diff --git a/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormats.java b/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormats.java
index 4386fb4..a095c8a 100644
--- a/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormats.java
+++ b/sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormats.java
@@ -20,7 +20,7 @@
import java.lang.reflect.InvocationTargetException;
import java.util.UUID;
import javax.annotation.Nullable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.JobID;
import org.apache.hadoop.mapreduce.MRJobConfig;
diff --git a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/EmployeeInputFormat.java b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/EmployeeInputFormat.java
index 8f5ca55..93679f9 100644
--- a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/EmployeeInputFormat.java
+++ b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/EmployeeInputFormat.java
@@ -23,7 +23,7 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.InputFormat;
diff --git a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticTest.java b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticTest.java
index 7341f71..ef7c83a 100644
--- a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticTest.java
+++ b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOElasticTest.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.commons.io.FileUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
diff --git a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOIT.java b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOIT.java
index 4e2f371..9933aee 100644
--- a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOIT.java
+++ b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOIT.java
@@ -37,8 +37,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -172,8 +170,6 @@
.apply("Produce db rows", ParDo.of(new TestRow.DeterministicallyConstructTestRowFn()))
.apply("Prevent fusion before writing", Reshuffle.viaRandomKey())
.apply("Collect write time", ParDo.of(new TimeMonitor<>(NAMESPACE, "write_time")))
- .apply("Count bytes", ParDo.of(new ByteMonitor<>(NAMESPACE, "byte_count")))
- .apply("Count items", ParDo.of(new CountMonitor<>(NAMESPACE, "item_count")))
.apply("Construct rows for DBOutputFormat", ParDo.of(new ConstructDBOutputFormatRowFn()))
.apply(
"Write using Hadoop OutputFormat",
@@ -231,16 +227,6 @@
return NamedTestResult.create(
uuid, timestamp, "write_time", (writeEnd - writeStart) / 1e3);
});
- suppliers.add(
- reader -> {
- long byteCount = reader.getCounterMetric("byte_count");
- return NamedTestResult.create(uuid, timestamp, "byte_count", byteCount);
- });
- suppliers.add(
- reader -> {
- long itemCount = reader.getCounterMetric("item_count");
- return NamedTestResult.create(uuid, timestamp, "item_count", itemCount);
- });
return suppliers;
}
diff --git a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java
index 7eacab6..aca7fe0 100644
--- a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java
+++ b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java
@@ -24,7 +24,9 @@
import static org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
import java.io.IOException;
import java.util.ArrayList;
@@ -483,9 +485,7 @@
InputFormat mockInputFormat = Mockito.mock(EmployeeInputFormat.class);
EmployeeRecordReader mockReader = Mockito.mock(EmployeeRecordReader.class);
- Mockito.when(
- mockInputFormat.createRecordReader(
- Mockito.any(InputSplit.class), Mockito.any(TaskAttemptContext.class)))
+ Mockito.when(mockInputFormat.createRecordReader(Mockito.any(), Mockito.any()))
.thenReturn(mockReader);
Mockito.when(mockReader.nextKeyValue()).thenReturn(false);
InputSplit mockInputSplit = Mockito.mock(NewObjectsEmployeeInputSplit.class);
@@ -499,7 +499,7 @@
new SerializableSplit(mockInputSplit));
boundedSource.setInputFormatObj(mockInputFormat);
BoundedReader<KV<Text, Employee>> reader = boundedSource.createReader(p.getOptions());
- assertEquals(false, reader.start());
+ assertFalse(reader.start());
assertEquals(Double.valueOf(1), reader.getFractionConsumed());
reader.close();
}
@@ -532,7 +532,7 @@
// When start is not called, getFractionConsumed() should return 0.
assertEquals(Double.valueOf(0), reader.getFractionConsumed());
boolean start = reader.start();
- assertEquals(true, start);
+ assertTrue(start);
if (start) {
elements.add(reader.getCurrent());
boolean advance = reader.advance();
@@ -541,7 +541,7 @@
assertEquals(
Double.valueOf(++recordsRead / TestEmployeeDataSet.NUMBER_OF_RECORDS_IN_EACH_SPLIT),
reader.getFractionConsumed());
- assertEquals(true, advance);
+ assertTrue(advance);
while (advance) {
elements.add(reader.getCurrent());
advance = reader.advance();
@@ -566,9 +566,7 @@
public void testGetFractionConsumedForBadProgressValue() throws Exception {
InputFormat<Text, Employee> mockInputFormat = Mockito.mock(EmployeeInputFormat.class);
EmployeeRecordReader mockReader = Mockito.mock(EmployeeRecordReader.class);
- Mockito.when(
- mockInputFormat.createRecordReader(
- Mockito.any(InputSplit.class), Mockito.any(TaskAttemptContext.class)))
+ Mockito.when(mockInputFormat.createRecordReader(Mockito.any(), Mockito.any()))
.thenReturn(mockReader);
Mockito.when(mockReader.nextKeyValue()).thenReturn(true);
// Set to a bad value , not in range of 0 to 1
@@ -586,11 +584,11 @@
BoundedReader<KV<Text, Employee>> reader = boundedSource.createReader(p.getOptions());
assertEquals(Double.valueOf(0), reader.getFractionConsumed());
boolean start = reader.start();
- assertEquals(true, start);
+ assertTrue(start);
if (start) {
boolean advance = reader.advance();
assertEquals(null, reader.getFractionConsumed());
- assertEquals(true, advance);
+ assertTrue(advance);
if (advance) {
advance = reader.advance();
assertEquals(null, reader.getFractionConsumed());
@@ -782,7 +780,7 @@
hifSource.createInputFormatInstance();
ConfigurableEmployeeInputFormat inputFormatObj =
(ConfigurableEmployeeInputFormat) hifSource.getInputFormat();
- assertEquals(true, inputFormatObj.isConfSet);
+ assertTrue(inputFormatObj.isConfSet);
}
}
diff --git a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/ReuseObjectsEmployeeInputFormat.java b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/ReuseObjectsEmployeeInputFormat.java
index 1e7a8e2..2e20102 100644
--- a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/ReuseObjectsEmployeeInputFormat.java
+++ b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/ReuseObjectsEmployeeInputFormat.java
@@ -23,7 +23,7 @@
import java.util.ArrayList;
import java.util.List;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.InputFormat;
diff --git a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/TestEmployeeDataSet.java b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/TestEmployeeDataSet.java
index 6e19f17..45a9703 100644
--- a/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/TestEmployeeDataSet.java
+++ b/sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/TestEmployeeDataSet.java
@@ -21,7 +21,7 @@
import java.util.List;
import java.util.stream.Collectors;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.hadoop.io.Text;
/**
diff --git a/sdks/java/io/hbase/build.gradle b/sdks/java/io/hbase/build.gradle
index 4588a9f..882eb5e 100644
--- a/sdks/java/io/hbase/build.gradle
+++ b/sdks/java/io/hbase/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hbase')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -37,12 +37,12 @@
def hbase_version = "1.2.6"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:hadoop-common", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow "org.apache.hbase:hbase-shaded-client:$hbase_version"
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:io:hadoop-common")
+ compile library.java.slf4j_api
+ compile "org.apache.hbase:hbase-shaded-client:$hbase_version"
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.commons_lang3
testCompile library.java.junit
@@ -64,6 +64,6 @@
}
testCompile "org.apache.hbase:hbase-hadoop-compat:$hbase_version:tests"
testCompile "org.apache.hbase:hbase-hadoop2-compat:$hbase_version:tests"
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java b/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java
index 77048a1..e10eaab 100644
--- a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java
+++ b/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseCoderProviderRegistrar.java
@@ -23,7 +23,7 @@
import org.apache.beam.sdk.coders.CoderProviderRegistrar;
import org.apache.beam.sdk.coders.CoderProviders;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.hadoop.hbase.client.Result;
/** A {@link CoderProviderRegistrar} for standard types used with {@link HBaseIO}. */
diff --git a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java b/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java
index 2a2fc13..5ed97e7 100644
--- a/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java
+++ b/sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.hbase;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.ArrayList;
diff --git a/sdks/java/io/hcatalog/build.gradle b/sdks/java/io/hcatalog/build.gradle
index d48f18a..da0ee24 100644
--- a/sdks/java/io/hcatalog/build.gradle
+++ b/sdks/java/io/hcatalog/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.hcatalog')
description = "Apache Beam :: SDKs :: Java :: IO :: HCatalog"
ext.summary = "IO to read and write for HCatalog source."
@@ -41,10 +41,10 @@
evaluationDependsOn(":sdks:java:io:common")
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:hadoop-common", configuration: "shadow")
- shadow library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:io:hadoop-common")
+ compile library.java.slf4j_api
// Hive bundles without repackaging Jackson which is why we redeclare it here so that it appears
// on the compile/test/runtime classpath before Hive.
provided library.java.jackson_annotations
@@ -52,7 +52,6 @@
provided library.java.jackson_databind
// Calcite (a dependency of Hive) bundles without repackaging Guava which is why we redeclare it
// here so that it appears on the compile/test/runtime classpath before Calcite.
- provided library.java.guava
provided library.java.hadoop_common
provided "org.apache.hive:hive-exec:$hive_version"
provided(group: "org.apache.hive.hcatalog", name: "hive-hcatalog-core", version: hive_version) {
@@ -68,6 +67,6 @@
testCompile "org.apache.hive:hive-exec:$hive_version"
testCompile "org.apache.hive:hive-common:$hive_version"
testCompile "org.apache.hive:hive-cli:$hive_version"
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogBeamSchema.java b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogBeamSchema.java
index 9aaad6d..4d07397 100644
--- a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogBeamSchema.java
+++ b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogBeamSchema.java
@@ -22,8 +22,8 @@
import java.util.Map;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Optional;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Optional;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
import org.apache.hadoop.hive.metastore.IMetaStoreClient;
diff --git a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
index 73518f6..ba4173d 100644
--- a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
+++ b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.hcatalog;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.util.ArrayList;
@@ -25,7 +25,6 @@
import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
import java.util.NoSuchElementException;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
@@ -33,15 +32,17 @@
import org.apache.beam.sdk.io.BoundedSource;
import org.apache.beam.sdk.io.hadoop.WritableCoder;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Watch;
+import org.apache.beam.sdk.transforms.Watch.Growth.TerminationCondition;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.hadoop.conf.Configuration;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.ql.metadata.Table;
@@ -58,6 +59,7 @@
import org.apache.hive.hcatalog.data.transfer.ReaderContext;
import org.apache.hive.hcatalog.data.transfer.WriteEntity;
import org.apache.hive.hcatalog.data.transfer.WriterContext;
+import org.joda.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -69,7 +71,7 @@
* <p>HCatalog source supports reading of HCatRecord from a HCatalog managed source, for eg. Hive.
*
* <p>To configure a HCatalog source, you must specify a metastore URI and a table name. Other
- * optional parameters are database & filter For instance:
+ * optional parameters are database & filter. For instance:
*
* <pre>{@code
* Map<String, String> configProperties = new HashMap<>();
@@ -83,13 +85,27 @@
* .withFilter(filterString) //optional, may be specified if the table is partitioned
* }</pre>
*
+ * <p>HCatalog source supports reading of HCatRecord in an unbounded mode. When run in an unbounded
+ * mode, HCatalogIO will continuously poll for new partitions and read that data. If provided with a
+ * termination condition, it will stop reading data after the condition is met.
+ *
+ * <pre>{@code
+ * pipeline
+ * .apply(HCatalogIO.read()
+ * .withConfigProperties(configProperties)
+ * .withDatabase("default") //optional, assumes default if none specified
+ * .withTable("employee")
+ * .withPollingInterval(Duration.millis(15000)) // poll for new partitions every 15 seconds
+ * .withTerminationCondition(Watch.Growth.afterTotalOf(Duration.millis(60000)))) //optional
+ * }</pre>
+ *
* <h3>Writing using HCatalog</h3>
*
* <p>HCatalog sink supports writing of HCatRecord to a HCatalog managed source, for eg. Hive.
*
* <p>To configure a HCatalog sink, you must specify a metastore URI and a table name. Other
- * optional parameters are database, partition & batchsize The destination table should exist
- * beforehand, the transform does not create a new table if it does not exist For instance:
+ * optional parameters are database, partition & batchsize. The destination table should exist
+ * beforehand, the transform does not create a new table if it does not exist. For instance:
*
* <pre>{@code
* Map<String, String> configProperties = new HashMap<>();
@@ -120,7 +136,10 @@
/** Read data from Hive. */
public static Read read() {
- return new AutoValue_HCatalogIO_Read.Builder().setDatabase(DEFAULT_DATABASE).build();
+ return new AutoValue_HCatalogIO_Read.Builder()
+ .setDatabase(DEFAULT_DATABASE)
+ .setPartitionCols(new ArrayList<>())
+ .build();
}
private HCatalogIO() {}
@@ -129,6 +148,7 @@
@VisibleForTesting
@AutoValue
public abstract static class Read extends PTransform<PBegin, PCollection<HCatRecord>> {
+
@Nullable
abstract Map<String, String> getConfigProperties();
@@ -147,6 +167,15 @@
@Nullable
abstract Integer getSplitId();
+ @Nullable
+ abstract Duration getPollingInterval();
+
+ @Nullable
+ abstract List<String> getPartitionCols();
+
+ @Nullable
+ abstract TerminationCondition<Read, ?> getTerminationCondition();
+
abstract Builder toBuilder();
@AutoValue.Builder
@@ -163,6 +192,12 @@
abstract Builder setContext(ReaderContext context);
+ abstract Builder setPollingInterval(Duration pollingInterval);
+
+ abstract Builder setPartitionCols(List<String> partitionCols);
+
+ abstract Builder setTerminationCondition(TerminationCondition<Read, ?> terminationCondition);
+
abstract Read build();
}
@@ -186,6 +221,28 @@
return toBuilder().setFilter(filter).build();
}
+ /**
+ * If specified, polling for new partitions will happen at this periodicity. The returned
+ * PCollection will be unbounded. However if a withTerminationCondition is set along with
+ * pollingInterval, polling will stop after the termination condition has been met.
+ */
+ public Read withPollingInterval(Duration pollingInterval) {
+ return toBuilder().setPollingInterval(pollingInterval).build();
+ }
+
+ /** Set the names of the columns that are partitions. */
+ public Read withPartitionCols(List<String> partitionCols) {
+ return toBuilder().setPartitionCols(partitionCols).build();
+ }
+
+ /**
+ * If specified, the poll function will stop polling after the termination condition has been
+ * satisfied.
+ */
+ public Read withTerminationCondition(TerminationCondition<Read, ?> terminationCondition) {
+ return toBuilder().setTerminationCondition(terminationCondition).build();
+ }
+
Read withSplitId(int splitId) {
checkArgument(splitId >= 0, "Invalid split id-" + splitId);
return toBuilder().setSplitId(splitId).build();
@@ -196,11 +253,27 @@
}
@Override
+ @SuppressWarnings("deprecation")
public PCollection<HCatRecord> expand(PBegin input) {
checkArgument(getTable() != null, "withTable() is required");
checkArgument(getConfigProperties() != null, "withConfigProperties() is required");
-
- return input.apply(org.apache.beam.sdk.io.Read.from(new BoundedHCatalogSource(this)));
+ Watch.Growth<Read, Integer, Integer> growthFn;
+ if (getPollingInterval() != null) {
+ growthFn = Watch.growthOf(new PartitionPollerFn()).withPollInterval(getPollingInterval());
+ if (getTerminationCondition() != null) {
+ growthFn = growthFn.withTerminationPerInput(getTerminationCondition());
+ }
+ return input
+ .apply("ConvertToReadRequest", Create.of(this))
+ .apply("WatchForNewPartitions", growthFn)
+ .apply("PartitionReader", ParDo.of(new PartitionReaderFn(getConfigProperties())));
+ } else {
+ // Treat as Bounded
+ checkArgument(
+ getTerminationCondition() == null,
+ "withTerminationCondition() is not required when using in bounded reads mode");
+ return input.apply(org.apache.beam.sdk.io.Read.from(new BoundedHCatalogSource(this)));
+ }
}
@Override
@@ -244,14 +317,10 @@
*/
@Override
public long getEstimatedSizeBytes(PipelineOptions pipelineOptions) throws Exception {
- Configuration conf = new Configuration();
- for (Entry<String, String> entry : spec.getConfigProperties().entrySet()) {
- conf.set(entry.getKey(), entry.getValue());
- }
IMetaStoreClient client = null;
try {
- HiveConf hiveConf = HCatUtil.getHiveConf(conf);
- client = HCatUtil.getHiveMetastoreClient(hiveConf);
+ HiveConf hiveConf = HCatalogUtils.createHiveConf(spec);
+ client = HCatalogUtils.createMetaStoreClient(hiveConf);
Table table = HCatUtil.getTable(client, spec.getDatabase(), spec.getTable());
return StatsUtils.getFileSizeForTable(hiveConf, table);
} finally {
diff --git a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogUtils.java b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogUtils.java
new file mode 100644
index 0000000..bf3638e
--- /dev/null
+++ b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogUtils.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.hcatalog;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.io.hcatalog.HCatalogIO.Read;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.stats.StatsUtils;
+import org.apache.hive.hcatalog.common.HCatUtil;
+
+/** Utility classes to enable meta store conf/client creation. */
+public class HCatalogUtils {
+
+ private static final int DESIRED_BUNDLE_SIZE_BYTES = 134217728; // 128 MB
+
+ static IMetaStoreClient createMetaStoreClient(Configuration conf)
+ throws IOException, MetaException {
+ final HiveConf hiveConf = HCatUtil.getHiveConf(conf);
+ return HCatUtil.getHiveMetastoreClient(hiveConf);
+ }
+
+ static HiveConf createHiveConf(Read readRequest) throws IOException {
+ Configuration conf = createConfiguration(readRequest.getConfigProperties());
+ return HCatUtil.getHiveConf(conf);
+ }
+
+ static int getSplitCount(Read readRequest, Partition partitionToRead) throws Exception {
+ int desiredSplitCount = 1;
+ long estimatedSizeBytes = getFileSizeForPartition(readRequest, partitionToRead);
+ if (estimatedSizeBytes > 0) {
+ desiredSplitCount = (int) Math.ceil((double) estimatedSizeBytes / DESIRED_BUNDLE_SIZE_BYTES);
+ }
+ return desiredSplitCount;
+ }
+
+ static Configuration createConfiguration(Map<String, String> configProperties) {
+ Configuration conf = new Configuration();
+ for (Map.Entry<String, String> entry : configProperties.entrySet()) {
+ conf.set(entry.getKey(), entry.getValue());
+ }
+ return conf;
+ }
+
+ private static long getFileSizeForPartition(Read readRequest, Partition partitionToRead)
+ throws Exception {
+ IMetaStoreClient client = null;
+ try {
+ HiveConf hiveConf = HCatalogUtils.createHiveConf(readRequest);
+ client = HCatalogUtils.createMetaStoreClient(hiveConf);
+ List<org.apache.hadoop.hive.ql.metadata.Partition> p = new ArrayList<>();
+ Table table = HCatUtil.getTable(client, readRequest.getDatabase(), readRequest.getTable());
+ final org.apache.hadoop.hive.ql.metadata.Partition partition =
+ new org.apache.hadoop.hive.ql.metadata.Partition(table, partitionToRead);
+ p.add(partition);
+ final List<Long> fileSizeForPartitions = StatsUtils.getFileSizeForPartitions(hiveConf, p);
+ return fileSizeForPartitions.get(0);
+ } finally {
+ // IMetaStoreClient is not AutoCloseable, closing it manually
+ if (client != null) {
+ client.close();
+ }
+ }
+ }
+}
diff --git a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionPollerFn.java b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionPollerFn.java
new file mode 100644
index 0000000..2e40710
--- /dev/null
+++ b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionPollerFn.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.hcatalog;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.io.hcatalog.HCatalogIO.Read;
+import org.apache.beam.sdk.transforms.Watch.Growth.PollFn;
+import org.apache.beam.sdk.transforms.Watch.Growth.PollResult;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.joda.time.Instant;
+
+/** Return the list of current partitions present. */
+class PartitionPollerFn extends PollFn<Read, Integer> {
+ private transient IMetaStoreClient metaStoreClient;
+
+ @Override
+ public PollResult<Integer> apply(Read element, Context c) throws Exception {
+ final Configuration conf = HCatalogUtils.createConfiguration(element.getConfigProperties());
+ metaStoreClient = HCatalogUtils.createMetaStoreClient(conf);
+ final Instant now = Instant.now();
+ final PollResult<Integer> pollResult =
+ PollResult.incomplete(now, getPartitionIndices(element)).withWatermark(now);
+ if (metaStoreClient != null) {
+ metaStoreClient.close();
+ }
+ return pollResult;
+ }
+
+ private List<Integer> getPartitionIndices(Read read) throws Exception {
+ return IntStream.range(
+ 0,
+ metaStoreClient
+ .listPartitions(read.getDatabase(), read.getTable(), Short.MAX_VALUE)
+ .size())
+ .boxed()
+ .collect(Collectors.toList());
+ }
+}
diff --git a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionReaderFn.java b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionReaderFn.java
new file mode 100644
index 0000000..fe69417
--- /dev/null
+++ b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionReaderFn.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.hcatalog;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.io.hcatalog.HCatalogIO.Read;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.values.KV;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hive.hcatalog.common.HCatConstants;
+import org.apache.hive.hcatalog.data.HCatRecord;
+import org.apache.hive.hcatalog.data.transfer.DataTransferFactory;
+import org.apache.hive.hcatalog.data.transfer.HCatReader;
+import org.apache.hive.hcatalog.data.transfer.ReadEntity;
+import org.apache.hive.hcatalog.data.transfer.ReaderContext;
+
+/** Reads partition at a given index. */
+class PartitionReaderFn extends DoFn<KV<Read, Integer>, HCatRecord> {
+ private transient IMetaStoreClient metaStoreClient;
+ private Map<String, String> configProperties;
+
+ public PartitionReaderFn(Map<String, String> configProperties) {
+ this.configProperties = configProperties;
+ }
+
+ private ReaderContext getReaderContext(Read readRequest, Integer partitionIndexToRead)
+ throws Exception {
+ final List<Partition> partitions =
+ metaStoreClient.listPartitions(
+ readRequest.getDatabase(), readRequest.getTable(), Short.MAX_VALUE);
+ final Partition partition = partitions.get(partitionIndexToRead);
+ checkArgument(
+ partition != null, "Unable to find a partition to read at index " + partitionIndexToRead);
+
+ final int desiredSplitCount = HCatalogUtils.getSplitCount(readRequest, partition);
+ final List<String> values = partition.getValues();
+ final List<String> partitionCols = readRequest.getPartitionCols();
+ checkArgument(
+ values.size() == partitionCols.size(),
+ "Number of input partitions should be equal to the values of list partition values.");
+
+ List<String> filter = new ArrayList<>();
+ for (int i = 0; i < partitionCols.size(); i++) {
+ filter.add(partitionCols.get(i) + "=" + "'" + values.get(i) + "'");
+ }
+ final String filterString = String.join(" and ", filter);
+
+ ReadEntity entity =
+ new ReadEntity.Builder()
+ .withDatabase(readRequest.getDatabase())
+ .withFilter(filterString)
+ .withTable(readRequest.getTable())
+ .build();
+ // pass the 'desired' split count as an hint to the API
+ Map<String, String> configProps = new HashMap<>(readRequest.getConfigProperties());
+ configProps.put(
+ HCatConstants.HCAT_DESIRED_PARTITION_NUM_SPLITS, String.valueOf(desiredSplitCount));
+ return DataTransferFactory.getHCatReader(entity, configProps).prepareRead();
+ }
+
+ @ProcessElement
+ public void processElement(ProcessContext c) throws Exception {
+ final Read readRequest = c.element().getKey();
+ final Integer partitionIndexToRead = c.element().getValue();
+ ReaderContext readerContext = getReaderContext(readRequest, partitionIndexToRead);
+ for (int i = 0; i < readerContext.numSplits(); i++) {
+ HCatReader reader = DataTransferFactory.getHCatReader(readerContext, i);
+ Iterator<HCatRecord> hcatIterator = reader.read();
+ while (hcatIterator.hasNext()) {
+ final HCatRecord record = hcatIterator.next();
+ c.output(record);
+ }
+ }
+ }
+
+ @Setup
+ public void setup() throws Exception {
+ final Configuration conf = HCatalogUtils.createConfiguration(configProperties);
+ metaStoreClient = HCatalogUtils.createMetaStoreClient(conf);
+ }
+
+ @Teardown
+ public void teardown() {
+ if (metaStoreClient != null) {
+ metaStoreClient.close();
+ }
+ }
+}
diff --git a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/SchemaUtils.java b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/SchemaUtils.java
index c0761ed..04e536f 100644
--- a/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/SchemaUtils.java
+++ b/sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/SchemaUtils.java
@@ -23,7 +23,7 @@
import java.util.Map;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.Schema.FieldType;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.serde.serdeConstants;
diff --git a/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOIT.java b/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOIT.java
index 930ffb6..60d9258 100644
--- a/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOIT.java
+++ b/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOIT.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.hive.hcatalog.data.HCatRecord;
import org.junit.After;
import org.junit.BeforeClass;
diff --git a/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOTest.java b/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOTest.java
index da631a3..786984b 100644
--- a/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOTest.java
+++ b/sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOTest.java
@@ -40,8 +40,12 @@
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.hadoop.WritableCoder;
import org.apache.beam.sdk.io.hcatalog.HCatalogIO.BoundedHCatalogSource;
import org.apache.beam.sdk.io.hcatalog.test.EmbeddedMetastoreService;
import org.apache.beam.sdk.options.PipelineOptions;
@@ -52,11 +56,16 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Watch;
import org.apache.beam.sdk.util.UserCodeException;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.ql.CommandNeedRetryException;
+import org.apache.hive.hcatalog.data.DefaultHCatRecord;
import org.apache.hive.hcatalog.data.HCatRecord;
import org.apache.hive.hcatalog.data.transfer.ReaderContext;
+import org.joda.time.Duration;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
@@ -96,6 +105,9 @@
prepareTestData();
} else if (description.getAnnotation(NeedsEmptyTestTables.class) != null) {
reCreateTestTable();
+ } else if (description.getAnnotation(NeedsEmptyTestTablesForUnboundedReads.class)
+ != null) {
+ reCreateTestTableForUnboundedReads();
}
base.evaluate();
}
@@ -110,6 +122,11 @@
@Target({ElementType.METHOD})
private @interface NeedsTestData {}
+ /** Use this annotation to setup complete test data(table populated with unbounded records). */
+ @Retention(RetentionPolicy.RUNTIME)
+ @Target({ElementType.METHOD})
+ private @interface NeedsEmptyTestTablesForUnboundedReads {}
+
/** Use this annotation to setup test tables alone(empty tables, no records are populated). */
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
@@ -163,6 +180,56 @@
readAfterWritePipeline.run();
}
+ private Map<String, String> getPartitions() {
+ Map<String, String> partitions = new HashMap<>();
+ partitions.put("load_date", "2019-05-14T23:28:04.425Z");
+ partitions.put("product_type", "1");
+ return partitions;
+ }
+
+ /** Perform end-to-end test of Write-then-Read operation. */
+ @Test
+ @NeedsEmptyTestTablesForUnboundedReads
+ public void testWriteThenUnboundedReadSuccess() throws Exception {
+
+ defaultPipeline
+ .apply(Create.of(buildHCatRecords(TEST_RECORDS_COUNT)))
+ .apply(
+ HCatalogIO.write()
+ .withConfigProperties(getConfigPropertiesAsMap(service.getHiveConf()))
+ .withDatabase(TEST_DATABASE)
+ .withTable(TEST_TABLE)
+ .withPartition(getPartitions())
+ .withBatchSize(512L));
+ defaultPipeline.run();
+ final ImmutableList<String> partitions = ImmutableList.of("load_date", "product_type");
+ final PCollection<HCatRecord> data =
+ readAfterWritePipeline
+ .apply(
+ "ReadData",
+ HCatalogIO.read()
+ .withConfigProperties(getConfigPropertiesAsMap(service.getHiveConf()))
+ .withDatabase(TEST_DATABASE)
+ .withPartitionCols(partitions)
+ .withTable(TEST_TABLE)
+ .withPollingInterval(Duration.millis(15000))
+ .withTerminationCondition(Watch.Growth.afterTotalOf(Duration.millis(60000))))
+ .setCoder((Coder) WritableCoder.of(DefaultHCatRecord.class));
+
+ final PCollection<String> output =
+ data.apply(
+ ParDo.of(
+ new DoFn<HCatRecord, String>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ c.output(c.element().get(0).toString());
+ }
+ }));
+
+ PAssert.that(output).containsInAnyOrder(getExpectedRecords(TEST_RECORDS_COUNT));
+ readAfterWritePipeline.run();
+ }
+
/** Test of Write to a non-existent table. */
@Test
public void testWriteFailureTableDoesNotExist() {
@@ -276,6 +343,19 @@
service.executeQuery("create table " + TEST_TABLE + "(mycol1 string, mycol2 int)");
}
+ private void reCreateTestTableForUnboundedReads() throws CommandNeedRetryException {
+ service.executeQuery("drop table " + TEST_TABLE);
+ service.executeQuery(
+ "create table "
+ + TEST_TABLE
+ + "(mycol1 string, mycol2 int) "
+ + "partitioned by (load_date string, product_type string)");
+ service.executeQuery(
+ "ALTER TABLE "
+ + TEST_TABLE
+ + " ADD PARTITION (load_date='2019-05-14T23:28:04.425Z', product_type='1')");
+ }
+
private void prepareTestData() throws Exception {
reCreateTestTable();
insertTestData(getConfigPropertiesAsMap(service.getHiveConf()));
diff --git a/sdks/java/io/jdbc/build.gradle b/sdks/java/io/jdbc/build.gradle
index ca7bb2a..d8ce5f2 100644
--- a/sdks/java/io/jdbc/build.gradle
+++ b/sdks/java/io/jdbc/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(enableSpotbugs: false)
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.jdbc')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -25,20 +25,20 @@
ext.summary = "IO to read and write on JDBC datasource."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow "org.apache.commons:commons-dbcp2:2.6.0"
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile "org.apache.commons:commons-dbcp2:2.6.0"
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:testing:test-utils", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.slf4j_api
testCompile library.java.postgres
- testCompile group: "org.apache.derby", name: "derby", version:"10.14.2.0"
- testCompile group: "org.apache.derby", name: "derbyclient", version:"10.14.2.0"
- testCompile group: "org.apache.derby", name: "derbynet", version:"10.14.2.0"
+ testCompile "org.apache.derby:derby:10.14.2.0"
+ testCompile "org.apache.derby:derbyclient:10.14.2.0"
+ testCompile "org.apache.derby:derbynet:10.14.2.0"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/BeamSchemaInferenceException.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/BeamSchemaInferenceException.java
new file mode 100644
index 0000000..683144d
--- /dev/null
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/BeamSchemaInferenceException.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+/** Exception to signal that inferring the Beam schema from the JDBC source failed. */
+public class BeamSchemaInferenceException extends RuntimeException {
+ public BeamSchemaInferenceException(String message, Throwable cause) {
+ super(message, cause);
+ }
+}
diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 8bd4f7e..7079db6 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -17,7 +17,8 @@
*/
package org.apache.beam.sdk.io.jdbc;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.sdk.io.jdbc.SchemaUtil.checkNullabilityForFields;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -28,12 +29,20 @@
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
import java.util.function.Predicate;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
import javax.annotation.Nullable;
import javax.sql.DataSource;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.RowCoder;
import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.Filter;
@@ -54,6 +63,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.commons.dbcp2.BasicDataSource;
import org.apache.commons.dbcp2.DataSourceConnectionFactory;
import org.apache.commons.dbcp2.PoolableConnectionFactory;
@@ -188,6 +199,16 @@
.build();
}
+ /** Read Beam {@link Row}s from a JDBC data source. */
+ @Experimental(Experimental.Kind.SCHEMAS)
+ public static ReadRows readRows() {
+ return new AutoValue_JdbcIO_ReadRows.Builder()
+ .setFetchSize(DEFAULT_FETCH_SIZE)
+ .setOutputParallelization(true)
+ .setStatementPreparator(ignored -> {})
+ .build();
+ }
+
/**
* Like {@link #read}, but executes multiple instances of the query substituting each element of a
* {@link PCollection} as query parameters.
@@ -382,46 +403,6 @@
}
}
- /** Wraps a {@link DataSourceConfiguration} to provide a {@link PoolingDataSource}. */
- public static class PoolableDataSourceProvider extends BaseDataSourceProvider {
- private static SerializableFunction<Void, DataSource> instance = null;
-
- private PoolableDataSourceProvider(
- SerializableFunction<Void, DataSource> dataSourceProviderFn) {
- super(dataSourceProviderFn);
- }
-
- public static SerializableFunction<Void, DataSource> of(DataSourceConfiguration config) {
- if (instance == null) {
- instance =
- MemoizedDataSourceProvider.of(
- new PoolableDataSourceProvider(
- DataSourceProviderFromDataSourceConfiguration.of(config)));
- }
- return instance;
- }
-
- @Override
- public DataSource apply(Void input) {
- DataSource current = super.dataSourceProviderFn.apply(input);
- // wrapping the datasource as a pooling datasource
- DataSourceConnectionFactory connectionFactory = new DataSourceConnectionFactory(current);
- PoolableConnectionFactory poolableConnectionFactory =
- new PoolableConnectionFactory(connectionFactory, null);
- GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
- poolConfig.setMaxTotal(1);
- poolConfig.setMinIdle(0);
- poolConfig.setMinEvictableIdleTimeMillis(10000);
- poolConfig.setSoftMinEvictableIdleTimeMillis(30000);
- GenericObjectPool connectionPool =
- new GenericObjectPool(poolableConnectionFactory, poolConfig);
- poolableConnectionFactory.setPool(connectionPool);
- poolableConnectionFactory.setDefaultAutoCommit(false);
- poolableConnectionFactory.setDefaultReadOnly(false);
- return new PoolingDataSource(connectionPool);
- }
- }
-
/**
* An interface used by the JdbcIO Write to set the parameters of the {@link PreparedStatement}
* used to setParameters into the database.
@@ -431,14 +412,126 @@
void setParameters(PreparedStatement preparedStatement) throws Exception;
}
+ /** Implementation of {@link #readRows()}. */
+ @AutoValue
+ @Experimental(Experimental.Kind.SCHEMAS)
+ public abstract static class ReadRows extends PTransform<PBegin, PCollection<Row>> {
+ @Nullable
+ abstract SerializableFunction<Void, DataSource> getDataSourceProviderFn();
+
+ @Nullable
+ abstract ValueProvider<String> getQuery();
+
+ @Nullable
+ abstract StatementPreparator getStatementPreparator();
+
+ abstract int getFetchSize();
+
+ abstract boolean getOutputParallelization();
+
+ abstract Builder toBuilder();
+
+ @AutoValue.Builder
+ abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction<Void, DataSource> dataSourceProviderFn);
+
+ abstract Builder setQuery(ValueProvider<String> query);
+
+ abstract Builder setStatementPreparator(StatementPreparator statementPreparator);
+
+ abstract Builder setFetchSize(int fetchSize);
+
+ abstract Builder setOutputParallelization(boolean outputParallelization);
+
+ abstract ReadRows build();
+ }
+
+ public ReadRows withDataSourceProviderFn(
+ SerializableFunction<Void, DataSource> dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+ }
+
+ public ReadRows withQuery(String query) {
+ checkArgument(query != null, "query can not be null");
+ return withQuery(ValueProvider.StaticValueProvider.of(query));
+ }
+
+ public ReadRows withQuery(ValueProvider<String> query) {
+ checkArgument(query != null, "query can not be null");
+ return toBuilder().setQuery(query).build();
+ }
+
+ public ReadRows withStatementPreparator(StatementPreparator statementPreparator) {
+ checkArgument(statementPreparator != null, "statementPreparator can not be null");
+ return toBuilder().setStatementPreparator(statementPreparator).build();
+ }
+
+ /**
+ * This method is used to set the size of the data that is going to be fetched and loaded in
+ * memory per every database call. Please refer to: {@link java.sql.Statement#setFetchSize(int)}
+ * It should ONLY be used if the default value throws memory errors.
+ */
+ public ReadRows withFetchSize(int fetchSize) {
+ checkArgument(fetchSize > 0, "fetch size must be > 0");
+ return toBuilder().setFetchSize(fetchSize).build();
+ }
+
+ /**
+ * Whether to reshuffle the resulting PCollection so results are distributed to all workers. The
+ * default is to parallelize and should only be changed if this is known to be unnecessary.
+ */
+ public ReadRows withOutputParallelization(boolean outputParallelization) {
+ return toBuilder().setOutputParallelization(outputParallelization).build();
+ }
+
+ @Override
+ public PCollection<Row> expand(PBegin input) {
+ checkArgument(getQuery() != null, "withQuery() is required");
+ checkArgument(
+ (getDataSourceProviderFn() != null),
+ "withDataSourceConfiguration() or withDataSourceProviderFn() is required");
+
+ Schema schema = inferBeamSchema();
+ PCollection<Row> rows =
+ input.apply(
+ JdbcIO.<Row>read()
+ .withDataSourceProviderFn(getDataSourceProviderFn())
+ .withQuery(getQuery())
+ .withCoder(RowCoder.of(schema))
+ .withRowMapper(SchemaUtil.BeamRowMapper.of(schema))
+ .withFetchSize(getFetchSize())
+ .withOutputParallelization(getOutputParallelization())
+ .withStatementPreparator(getStatementPreparator()));
+ rows.setRowSchema(schema);
+ return rows;
+ }
+
+ private Schema inferBeamSchema() {
+ DataSource ds = getDataSourceProviderFn().apply(null);
+ try (Connection conn = ds.getConnection();
+ PreparedStatement statement =
+ conn.prepareStatement(
+ getQuery().get(), ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)) {
+ return SchemaUtil.toBeamSchema(statement.getMetaData());
+ } catch (SQLException e) {
+ throw new BeamSchemaInferenceException("Failed to infer Beam schema", e);
+ }
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ super.populateDisplayData(builder);
+ builder.add(DisplayData.item("query", getQuery()));
+ if (getDataSourceProviderFn() instanceof HasDisplayData) {
+ ((HasDisplayData) getDataSourceProviderFn()).populateDisplayData(builder);
+ }
+ }
+ }
+
/** Implementation of {@link #read}. */
@AutoValue
public abstract static class Read<T> extends PTransform<PBegin, PCollection<T>> {
- /** @deprecated It is not needed anymore. It will be removed in a future version of Beam. */
- @Deprecated
- @Nullable
- abstract DataSourceConfiguration getDataSourceConfiguration();
-
@Nullable
abstract SerializableFunction<Void, DataSource> getDataSourceProviderFn();
@@ -462,10 +555,6 @@
@AutoValue.Builder
abstract static class Builder<T> {
- /** @deprecated It is not needed anymore. It will be removed in a future version of Beam. */
- @Deprecated
- abstract Builder<T> setDataSourceConfiguration(DataSourceConfiguration config);
-
abstract Builder<T> setDataSourceProviderFn(
SerializableFunction<Void, DataSource> dataSourceProviderFn);
@@ -485,7 +574,6 @@
}
public Read<T> withDataSourceConfiguration(final DataSourceConfiguration config) {
- toBuilder().setDataSourceConfiguration(config);
return withDataSourceProviderFn(new DataSourceProviderFromDataSourceConfiguration(config));
}
@@ -550,7 +638,6 @@
.apply(Create.of((Void) null))
.apply(
JdbcIO.<Void, T>readAll()
- .withDataSourceConfiguration(getDataSourceConfiguration())
.withDataSourceProviderFn(getDataSourceProviderFn())
.withQuery(getQuery())
.withCoder(getCoder())
@@ -581,11 +668,6 @@
@AutoValue
public abstract static class ReadAll<ParameterT, OutputT>
extends PTransform<PCollection<ParameterT>, PCollection<OutputT>> {
- /** @deprecated It is not needed anymore. It will be removed in a future version of Beam. */
- @Deprecated
- @Nullable
- abstract DataSourceConfiguration getDataSourceConfiguration();
-
@Nullable
abstract SerializableFunction<Void, DataSource> getDataSourceProviderFn();
@@ -609,11 +691,6 @@
@AutoValue.Builder
abstract static class Builder<ParameterT, OutputT> {
- /** @deprecated It is not needed anymore. It will be removed in a future version of Beam. */
- @Deprecated
- abstract Builder<ParameterT, OutputT> setDataSourceConfiguration(
- DataSourceConfiguration config);
-
abstract Builder<ParameterT, OutputT> setDataSourceProviderFn(
SerializableFunction<Void, DataSource> dataSourceProviderFn);
@@ -635,7 +712,6 @@
public ReadAll<ParameterT, OutputT> withDataSourceConfiguration(
DataSourceConfiguration config) {
- toBuilder().setDataSourceConfiguration(config);
return withDataSourceProviderFn(new DataSourceProviderFromDataSourceConfiguration(config));
}
@@ -711,6 +787,16 @@
output = output.apply(new Reparallelize<>());
}
+ try {
+ TypeDescriptor<OutputT> typeDesc = getCoder().getEncodedTypeDescriptor();
+ SchemaRegistry registry = input.getPipeline().getSchemaRegistry();
+ Schema schema = registry.getSchema(typeDesc);
+ output.setSchema(
+ schema, registry.getToRowFunction(typeDesc), registry.getFromRowFunction(typeDesc));
+ } catch (NoSuchSchemaException e) {
+ // ignore
+ }
+
return output;
}
@@ -802,7 +888,7 @@
* <p>All methods in this class delegate to the appropriate method of {@link JdbcIO.WriteVoid}.
*/
public static class Write<T> extends PTransform<PCollection<T>, PDone> {
- final WriteVoid<T> inner;
+ WriteVoid<T> inner;
Write() {
this(JdbcIO.writeVoid());
@@ -846,6 +932,11 @@
return new Write(inner.withRetryStrategy(retryStrategy));
}
+ /** See {@link WriteVoid#withTable(String)}. */
+ public Write<T> withTable(String table) {
+ return new Write(inner.withTable(table));
+ }
+
/**
* Returns {@link WriteVoid} transform which can be used in {@link Wait#on(PCollection[])} to
* wait until all data is written.
@@ -870,21 +961,147 @@
inner.populateDisplayData(builder);
}
+ private boolean hasStatementAndSetter() {
+ return inner.getStatement() != null && inner.getPreparedStatementSetter() != null;
+ }
+
@Override
public PDone expand(PCollection<T> input) {
+ // fixme: validate invalid table input
+ if (input.hasSchema() && !hasStatementAndSetter()) {
+ checkArgument(
+ inner.getTable() != null, "table cannot be null if statement is not provided");
+ Schema schema = input.getSchema();
+ List<SchemaUtil.FieldWithIndex> fields = getFilteredFields(schema);
+ inner =
+ inner.withStatement(
+ JdbcUtil.generateStatement(
+ inner.getTable(),
+ fields.stream()
+ .map(SchemaUtil.FieldWithIndex::getField)
+ .collect(Collectors.toList())));
+ inner =
+ inner.withPreparedStatementSetter(
+ new AutoGeneratedPreparedStatementSetter(fields, input.getToRowFunction()));
+ }
+
inner.expand(input);
return PDone.in(input.getPipeline());
}
+
+ private List<SchemaUtil.FieldWithIndex> getFilteredFields(Schema schema) {
+ Schema tableSchema;
+
+ try (Connection connection = inner.getDataSourceProviderFn().apply(null).getConnection();
+ PreparedStatement statement =
+ connection.prepareStatement((String.format("SELECT * FROM %s", inner.getTable())))) {
+ tableSchema = SchemaUtil.toBeamSchema(statement.getMetaData());
+ statement.close();
+ } catch (SQLException e) {
+ throw new RuntimeException(
+ "Error while determining columns from table: " + inner.getTable(), e);
+ }
+
+ if (tableSchema.getFieldCount() < schema.getFieldCount()) {
+ throw new RuntimeException("Input schema has more fields than actual table.");
+ }
+
+ // filter out missing fields from output table
+ List<Schema.Field> missingFields =
+ tableSchema.getFields().stream()
+ .filter(
+ line ->
+ schema.getFields().stream()
+ .noneMatch(s -> s.getName().equalsIgnoreCase(line.getName())))
+ .collect(Collectors.toList());
+
+ // allow insert only if missing fields are nullable
+ if (checkNullabilityForFields(missingFields)) {
+ throw new RuntimeException("Non nullable fields are not allowed without schema.");
+ }
+
+ List<SchemaUtil.FieldWithIndex> tableFilteredFields =
+ tableSchema.getFields().stream()
+ .map(
+ (tableField) -> {
+ Optional<Schema.Field> optionalSchemaField =
+ schema.getFields().stream()
+ .filter((f) -> SchemaUtil.compareSchemaField(tableField, f))
+ .findFirst();
+ return (optionalSchemaField.isPresent())
+ ? SchemaUtil.FieldWithIndex.of(
+ tableField, schema.getFields().indexOf(optionalSchemaField.get()))
+ : null;
+ })
+ .filter(Objects::nonNull)
+ .collect(Collectors.toList());
+
+ if (tableFilteredFields.size() != schema.getFieldCount()) {
+ throw new RuntimeException("Provided schema doesn't match with database schema.");
+ }
+
+ return tableFilteredFields;
+ }
+
+ /**
+ * A {@link org.apache.beam.sdk.io.jdbc.JdbcIO.PreparedStatementSetter} implementation that
+ * calls related setters on prepared statement.
+ */
+ private class AutoGeneratedPreparedStatementSetter implements PreparedStatementSetter<T> {
+
+ private List<SchemaUtil.FieldWithIndex> fields;
+ private SerializableFunction<T, Row> toRowFn;
+ private List<PreparedStatementSetCaller> preparedStatementFieldSetterList = new ArrayList<>();
+
+ AutoGeneratedPreparedStatementSetter(
+ List<SchemaUtil.FieldWithIndex> fieldsWithIndex, SerializableFunction<T, Row> toRowFn) {
+ this.fields = fieldsWithIndex;
+ this.toRowFn = toRowFn;
+ populatePreparedStatementFieldSetter();
+ }
+
+ private void populatePreparedStatementFieldSetter() {
+ IntStream.range(0, fields.size())
+ .forEach(
+ (index) -> {
+ Schema.FieldType fieldType = fields.get(index).getField().getType();
+ preparedStatementFieldSetterList.add(
+ JdbcUtil.getPreparedStatementSetCaller(fieldType));
+ });
+ }
+
+ @Override
+ public void setParameters(T element, PreparedStatement preparedStatement) throws Exception {
+ Row row = (element instanceof Row) ? (Row) element : toRowFn.apply(element);
+ IntStream.range(0, fields.size())
+ .forEach(
+ (index) -> {
+ try {
+ preparedStatementFieldSetterList
+ .get(index)
+ .set(row, preparedStatement, index, fields.get(index));
+ } catch (SQLException | NullPointerException e) {
+ throw new RuntimeException("Error while setting data to preparedStatement", e);
+ }
+ });
+ }
+ }
+ }
+
+ /** Interface implemented by functions that sets prepared statement data. */
+ @FunctionalInterface
+ interface PreparedStatementSetCaller extends Serializable {
+ void set(
+ Row element,
+ PreparedStatement preparedStatement,
+ int prepareStatementIndex,
+ SchemaUtil.FieldWithIndex schemaFieldWithIndex)
+ throws SQLException;
}
/** A {@link PTransform} to write to a JDBC datasource. */
@AutoValue
public abstract static class WriteVoid<T> extends PTransform<PCollection<T>, PCollection<Void>> {
- /** @deprecated It is not needed anymore. It will be removed in a future version of Beam. */
- @Deprecated
- @Nullable
- abstract DataSourceConfiguration getDataSourceConfiguration();
-
@Nullable
abstract SerializableFunction<Void, DataSource> getDataSourceProviderFn();
@@ -899,14 +1116,13 @@
@Nullable
abstract RetryStrategy getRetryStrategy();
+ @Nullable
+ abstract String getTable();
+
abstract Builder<T> toBuilder();
@AutoValue.Builder
abstract static class Builder<T> {
- /** @deprecated It is not needed anymore. It will be removed in a future version of Beam. */
- @Deprecated
- abstract Builder<T> setDataSourceConfiguration(DataSourceConfiguration config);
-
abstract Builder<T> setDataSourceProviderFn(
SerializableFunction<Void, DataSource> dataSourceProviderFn);
@@ -918,11 +1134,12 @@
abstract Builder<T> setRetryStrategy(RetryStrategy deadlockPredicate);
+ abstract Builder<T> setTable(String table);
+
abstract WriteVoid<T> build();
}
public WriteVoid<T> withDataSourceConfiguration(DataSourceConfiguration config) {
- toBuilder().setDataSourceConfiguration(config);
return withDataSourceProviderFn(new DataSourceProviderFromDataSourceConfiguration(config));
}
@@ -963,6 +1180,11 @@
return toBuilder().setRetryStrategy(retryStrategy).build();
}
+ public WriteVoid<T> withTable(String table) {
+ checkArgument(table != null, "table name can not be null");
+ return toBuilder().setTable(table).build();
+ }
+
@Override
public PCollection<Void> expand(PCollection<T> input) {
checkArgument(getStatement() != null, "withStatement() is required");
@@ -1112,6 +1334,60 @@
}
}
+ /** Wraps a {@link DataSourceConfiguration} to provide a {@link PoolingDataSource}. */
+ public static class PoolableDataSourceProvider
+ implements SerializableFunction<Void, DataSource>, HasDisplayData {
+ private static PoolableDataSourceProvider instance;
+ private static transient DataSource source;
+ private static SerializableFunction<Void, DataSource> dataSourceProviderFn;
+
+ private PoolableDataSourceProvider(DataSourceConfiguration config) {
+ dataSourceProviderFn = DataSourceProviderFromDataSourceConfiguration.of(config);
+ }
+
+ public static synchronized SerializableFunction<Void, DataSource> of(
+ DataSourceConfiguration config) {
+ if (instance == null) {
+ instance = new PoolableDataSourceProvider(config);
+ }
+ return instance;
+ }
+
+ @Override
+ public DataSource apply(Void input) {
+ return buildDataSource(input);
+ }
+
+ static synchronized DataSource buildDataSource(Void input) {
+ if (source == null) {
+ DataSource basicSource = dataSourceProviderFn.apply(input);
+ DataSourceConnectionFactory connectionFactory =
+ new DataSourceConnectionFactory(basicSource);
+ PoolableConnectionFactory poolableConnectionFactory =
+ new PoolableConnectionFactory(connectionFactory, null);
+ GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
+ poolConfig.setMaxTotal(1);
+ poolConfig.setMinIdle(0);
+ poolConfig.setMinEvictableIdleTimeMillis(10000);
+ poolConfig.setSoftMinEvictableIdleTimeMillis(30000);
+ GenericObjectPool connectionPool =
+ new GenericObjectPool(poolableConnectionFactory, poolConfig);
+ poolableConnectionFactory.setPool(connectionPool);
+ poolableConnectionFactory.setDefaultAutoCommit(false);
+ poolableConnectionFactory.setDefaultReadOnly(false);
+ source = new PoolingDataSource(connectionPool);
+ }
+ return source;
+ }
+
+ @Override
+ public void populateDisplayData(DisplayData.Builder builder) {
+ if (dataSourceProviderFn instanceof HasDisplayData) {
+ ((HasDisplayData) dataSourceProviderFn).populateDisplayData(builder);
+ }
+ }
+ }
+
private static class DataSourceProviderFromDataSourceConfiguration
implements SerializableFunction<Void, DataSource>, HasDisplayData {
private final DataSourceConfiguration config;
@@ -1138,46 +1414,4 @@
config.populateDisplayData(builder);
}
}
-
- private abstract static class BaseDataSourceProvider
- implements SerializableFunction<Void, DataSource>, HasDisplayData {
- private final SerializableFunction<Void, DataSource> dataSourceProviderFn;
-
- BaseDataSourceProvider(SerializableFunction<Void, DataSource> dataSourceProviderFn) {
- this.dataSourceProviderFn = dataSourceProviderFn;
- }
-
- @Override
- public void populateDisplayData(DisplayData.Builder builder) {
- if (dataSourceProviderFn instanceof HasDisplayData) {
- ((HasDisplayData) dataSourceProviderFn).populateDisplayData(builder);
- }
- }
- }
-
- private static class MemoizedDataSourceProvider extends BaseDataSourceProvider {
- private static MemoizedDataSourceProvider instance = null;
- @Nullable private static DataSource datasource = null;
-
- private MemoizedDataSourceProvider(
- SerializableFunction<Void, DataSource> dataSourceProviderFn) {
- super(dataSourceProviderFn);
- }
-
- public static MemoizedDataSourceProvider of(
- SerializableFunction<Void, DataSource> dataSourceProviderFn) {
- if (instance == null) {
- instance = new MemoizedDataSourceProvider(dataSourceProviderFn);
- }
- return instance;
- }
-
- @Override
- public DataSource apply(Void input) {
- if (datasource == null) {
- datasource = super.dataSourceProviderFn.apply(null);
- }
- return datasource;
- }
- }
}
diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
new file mode 100644
index 0000000..b0cd0a5
--- /dev/null
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.sql.Date;
+import java.sql.JDBCType;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.util.Calendar;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TimeZone;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.DateTime;
+
+/** Provides utility functions for working with {@link JdbcIO}. */
+public class JdbcUtil {
+
+ /** Generates an insert statement based on {@Link Schema.Field}. * */
+ public static String generateStatement(String tableName, List<Schema.Field> fields) {
+
+ String fieldNames =
+ IntStream.range(0, fields.size())
+ .mapToObj(
+ (index) -> {
+ return fields.get(index).getName();
+ })
+ .collect(Collectors.joining(", "));
+
+ String valuePlaceholder =
+ IntStream.range(0, fields.size())
+ .mapToObj(
+ (index) -> {
+ return "?";
+ })
+ .collect(Collectors.joining(", "));
+
+ return String.format("INSERT INTO %s(%s) VALUES(%s)", tableName, fieldNames, valuePlaceholder);
+ }
+
+ /** PreparedStatementSetCaller for Schema Field types. * */
+ private static Map<Schema.TypeName, JdbcIO.PreparedStatementSetCaller> typeNamePsSetCallerMap =
+ new EnumMap<Schema.TypeName, JdbcIO.PreparedStatementSetCaller>(
+ ImmutableMap.<Schema.TypeName, JdbcIO.PreparedStatementSetCaller>builder()
+ .put(
+ Schema.TypeName.BYTE,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setByte(i + 1, element.getByte(fieldWithIndex.getIndex())))
+ .put(
+ Schema.TypeName.INT16,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setInt(i + 1, element.getInt16(fieldWithIndex.getIndex())))
+ .put(
+ Schema.TypeName.INT64,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setLong(i + 1, element.getInt64(fieldWithIndex.getIndex())))
+ .put(
+ Schema.TypeName.DECIMAL,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setBigDecimal(i + 1, element.getDecimal(fieldWithIndex.getIndex())))
+ .put(
+ Schema.TypeName.FLOAT,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setFloat(i + 1, element.getFloat(fieldWithIndex.getIndex())))
+ .put(
+ Schema.TypeName.DOUBLE,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setDouble(i + 1, element.getDouble(fieldWithIndex.getIndex())))
+ .put(
+ Schema.TypeName.DATETIME,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setTimestamp(
+ i + 1,
+ new Timestamp(
+ element.getDateTime(fieldWithIndex.getIndex()).getMillis())))
+ .put(
+ Schema.TypeName.BOOLEAN,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setBoolean(i + 1, element.getBoolean(fieldWithIndex.getIndex())))
+ .put(Schema.TypeName.BYTES, createBytesCaller())
+ .put(
+ Schema.TypeName.INT32,
+ (element, ps, i, fieldWithIndex) ->
+ ps.setInt(i + 1, element.getInt32(fieldWithIndex.getIndex())))
+ .put(Schema.TypeName.STRING, createStringCaller())
+ .build());
+
+ /** PreparedStatementSetCaller for Schema Field Logical types. * */
+ public static JdbcIO.PreparedStatementSetCaller getPreparedStatementSetCaller(
+ Schema.FieldType fieldType) {
+ switch (fieldType.getTypeName()) {
+ case ARRAY:
+ return (element, ps, i, fieldWithIndex) -> {
+ ps.setArray(
+ i + 1,
+ ps.getConnection()
+ .createArrayOf(
+ fieldType.getCollectionElementType().getTypeName().name(),
+ element.getArray(fieldWithIndex.getIndex()).toArray()));
+ };
+ case LOGICAL_TYPE:
+ {
+ String logicalTypeName = fieldType.getLogicalType().getIdentifier();
+ JDBCType jdbcType = JDBCType.valueOf(logicalTypeName);
+ switch (jdbcType) {
+ case DATE:
+ return (element, ps, i, fieldWithIndex) -> {
+ ps.setDate(
+ i + 1,
+ new Date(
+ getDateOrTimeOnly(
+ element.getDateTime(fieldWithIndex.getIndex()).toDateTime(), true)
+ .getTime()
+ .getTime()));
+ };
+ case TIME:
+ return (element, ps, i, fieldWithIndex) -> {
+ ps.setTime(
+ i + 1,
+ new Time(
+ getDateOrTimeOnly(
+ element.getDateTime(fieldWithIndex.getIndex()).toDateTime(), false)
+ .getTime()
+ .getTime()));
+ };
+ case TIMESTAMP_WITH_TIMEZONE:
+ return (element, ps, i, fieldWithIndex) -> {
+ Calendar calendar =
+ withTimestampAndTimezone(
+ element.getDateTime(fieldWithIndex.getIndex()).toDateTime());
+ ps.setTimestamp(i + 1, new Timestamp(calendar.getTime().getTime()), calendar);
+ };
+ default:
+ return getPreparedStatementSetCaller(fieldType.getLogicalType().getBaseType());
+ }
+ }
+ default:
+ {
+ if (typeNamePsSetCallerMap.containsKey(fieldType.getTypeName())) {
+ return typeNamePsSetCallerMap.get(fieldType.getTypeName());
+ } else {
+ throw new RuntimeException(
+ fieldType.getTypeName().name()
+ + " in schema is not supported while writing. Please provide statement and preparedStatementSetter");
+ }
+ }
+ }
+ }
+
+ private static JdbcIO.PreparedStatementSetCaller createBytesCaller() {
+ return (element, ps, i, fieldWithIndex) -> {
+ validateLogicalTypeLength(
+ fieldWithIndex.getField(), element.getBytes(fieldWithIndex.getIndex()).length);
+ ps.setBytes(i + 1, element.getBytes(fieldWithIndex.getIndex()));
+ };
+ }
+
+ private static JdbcIO.PreparedStatementSetCaller createStringCaller() {
+ return (element, ps, i, fieldWithIndex) -> {
+ validateLogicalTypeLength(
+ fieldWithIndex.getField(), element.getString(fieldWithIndex.getIndex()).length());
+ ps.setString(i + 1, element.getString(fieldWithIndex.getIndex()));
+ };
+ }
+
+ private static void validateLogicalTypeLength(Schema.Field field, Integer length) {
+ try {
+ if (field.getType().getTypeName().isLogicalType()
+ && !field.getType().getLogicalType().getArgument().isEmpty()) {
+ int maxLimit = Integer.parseInt(field.getType().getLogicalType().getArgument());
+ if (field.getType().getTypeName().isLogicalType() && length >= maxLimit) {
+ throw new RuntimeException(
+ String.format(
+ "Length of Schema.Field[%s] data exceeds database column capacity",
+ field.getName()));
+ }
+ }
+ } catch (NumberFormatException e) {
+ // if argument is not set or not integer then do nothing and proceed with the insertion
+ }
+ }
+
+ private static Calendar getDateOrTimeOnly(DateTime dateTime, boolean wantDateOnly) {
+ Calendar cal = Calendar.getInstance();
+ cal.setTimeZone(TimeZone.getTimeZone(dateTime.getZone().getID()));
+
+ if (wantDateOnly) { // return date only
+ cal.set(Calendar.YEAR, dateTime.getYear());
+ cal.set(Calendar.MONTH, dateTime.getMonthOfYear() - 1);
+ cal.set(Calendar.DATE, dateTime.getDayOfMonth());
+
+ cal.set(Calendar.HOUR_OF_DAY, 0);
+ cal.set(Calendar.MINUTE, 0);
+ cal.set(Calendar.SECOND, 0);
+ cal.set(Calendar.MILLISECOND, 0);
+ } else { // return time only
+ cal.set(Calendar.YEAR, 1970);
+ cal.set(Calendar.MONTH, Calendar.JANUARY);
+ cal.set(Calendar.DATE, 1);
+
+ cal.set(Calendar.HOUR_OF_DAY, dateTime.getHourOfDay());
+ cal.set(Calendar.MINUTE, dateTime.getMinuteOfHour());
+ cal.set(Calendar.SECOND, dateTime.getSecondOfMinute());
+ cal.set(Calendar.MILLISECOND, dateTime.getMillisOfSecond());
+ }
+
+ return cal;
+ }
+
+ private static Calendar withTimestampAndTimezone(DateTime dateTime) {
+ Calendar calendar = Calendar.getInstance(TimeZone.getTimeZone(dateTime.getZone().getID()));
+ calendar.setTimeInMillis(dateTime.getMillis());
+
+ return calendar;
+ }
+}
diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java
new file mode 100644
index 0000000..e8b67e6
--- /dev/null
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.math.BigDecimal;
+import java.sql.JDBCType;
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.beam.repackaged.core.org.apache.commons.lang3.StringUtils;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+
+/** Beam {@link org.apache.beam.sdk.schemas.Schema.LogicalType} implementations of JDBC types. */
+public class LogicalTypes {
+ public static final Schema.FieldType JDBC_BIT_TYPE =
+ Schema.FieldType.logicalType(
+ new org.apache.beam.sdk.schemas.LogicalTypes.PassThroughLogicalType<Boolean>(
+ JDBCType.BIT.getName(), "", Schema.FieldType.BOOLEAN) {});
+
+ public static final Schema.FieldType JDBC_DATE_TYPE =
+ Schema.FieldType.logicalType(
+ new org.apache.beam.sdk.schemas.LogicalTypes.PassThroughLogicalType<Instant>(
+ JDBCType.DATE.getName(), "", Schema.FieldType.DATETIME) {});
+
+ public static final Schema.FieldType JDBC_FLOAT_TYPE =
+ Schema.FieldType.logicalType(
+ new org.apache.beam.sdk.schemas.LogicalTypes.PassThroughLogicalType<Double>(
+ JDBCType.FLOAT.getName(), "", Schema.FieldType.DOUBLE) {});
+
+ public static final Schema.FieldType JDBC_TIME_TYPE =
+ Schema.FieldType.logicalType(
+ new org.apache.beam.sdk.schemas.LogicalTypes.PassThroughLogicalType<Instant>(
+ JDBCType.TIME.getName(), "", Schema.FieldType.DATETIME) {});
+
+ public static final Schema.FieldType JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE =
+ Schema.FieldType.logicalType(
+ new org.apache.beam.sdk.schemas.LogicalTypes.PassThroughLogicalType<Instant>(
+ JDBCType.TIMESTAMP_WITH_TIMEZONE.getName(), "", Schema.FieldType.DATETIME) {});
+
+ @VisibleForTesting
+ static Schema.FieldType fixedLengthString(JDBCType jdbcType, int length) {
+ return Schema.FieldType.logicalType(FixedLengthString.of(jdbcType.getName(), length));
+ }
+
+ @VisibleForTesting
+ static Schema.FieldType fixedLengthBytes(JDBCType jdbcType, int length) {
+ return Schema.FieldType.logicalType(FixedLengthBytes.of(jdbcType.getName(), length));
+ }
+
+ @VisibleForTesting
+ static Schema.FieldType variableLengthString(JDBCType jdbcType, int length) {
+ return Schema.FieldType.logicalType(VariableLengthString.of(jdbcType.getName(), length));
+ }
+
+ @VisibleForTesting
+ static Schema.FieldType variableLengthBytes(JDBCType jdbcType, int length) {
+ return Schema.FieldType.logicalType(VariableLengthBytes.of(jdbcType.getName(), length));
+ }
+
+ @VisibleForTesting
+ static Schema.FieldType numeric(int precision, int scale) {
+ return Schema.FieldType.logicalType(
+ FixedPrecisionNumeric.of(JDBCType.NUMERIC.getName(), precision, scale));
+ }
+
+ /** Base class for JDBC logical types. */
+ public abstract static class JdbcLogicalType<T> implements Schema.LogicalType<T, T> {
+ protected final String identifier;
+ protected final Schema.FieldType baseType;
+ protected final String argument;
+
+ protected JdbcLogicalType(String identifier, Schema.FieldType baseType, String argument) {
+ this.identifier = identifier;
+ this.baseType = baseType;
+ this.argument = argument;
+ }
+
+ @Override
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ @Override
+ public String getArgument() {
+ return argument;
+ }
+
+ @Override
+ public Schema.FieldType getBaseType() {
+ return baseType;
+ }
+
+ @Override
+ public T toBaseType(T input) {
+ return input;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof JdbcLogicalType)) {
+ return false;
+ }
+ JdbcLogicalType<?> that = (JdbcLogicalType<?>) o;
+ return Objects.equals(identifier, that.identifier)
+ && Objects.equals(baseType, that.baseType)
+ && Objects.equals(argument, that.argument);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(identifier, baseType, argument);
+ }
+ }
+
+ /** Fixed length string types such as CHAR. */
+ public static final class FixedLengthString extends JdbcLogicalType<String> {
+ private final int length;
+
+ public static FixedLengthString of(String identifier, int length) {
+ return new FixedLengthString(identifier, length);
+ }
+
+ private FixedLengthString(String identifier, int length) {
+ super(identifier, Schema.FieldType.STRING, String.valueOf(length));
+ this.length = length;
+ }
+
+ @Override
+ public String toInputType(String base) {
+ checkArgument(base == null || base.length() <= length);
+ return StringUtils.rightPad(base, length);
+ }
+ }
+
+ /** Fixed length byte types such as BINARY. */
+ public static final class FixedLengthBytes extends JdbcLogicalType<byte[]> {
+ private final int length;
+
+ public static FixedLengthBytes of(String identifier, int length) {
+ return new FixedLengthBytes(identifier, length);
+ }
+
+ private FixedLengthBytes(String identifier, int length) {
+ super(identifier, Schema.FieldType.BYTES, String.valueOf(length));
+ this.length = length;
+ }
+
+ @Override
+ public byte[] toInputType(byte[] base) {
+ checkArgument(base == null || base.length <= length);
+ if (base == null || base.length == length) {
+ return base;
+ } else {
+ return Arrays.copyOf(base, length);
+ }
+ }
+ }
+
+ /** Variable length string types such as VARCHAR and LONGVARCHAR. */
+ public static final class VariableLengthString extends JdbcLogicalType<String> {
+ private final int maxLength;
+
+ public static VariableLengthString of(String identifier, int maxLength) {
+ return new VariableLengthString(identifier, maxLength);
+ }
+
+ private VariableLengthString(String identifier, int maxLength) {
+ super(identifier, Schema.FieldType.STRING, String.valueOf(maxLength));
+ this.maxLength = maxLength;
+ }
+
+ @Override
+ public String toInputType(String base) {
+ checkArgument(base == null || base.length() <= maxLength);
+ return base;
+ }
+ }
+
+ /** Variable length bytes types such as VARBINARY and LONGVARBINARY. */
+ public static final class VariableLengthBytes extends JdbcLogicalType<byte[]> {
+ private final int maxLength;
+
+ public static VariableLengthBytes of(String identifier, int maxLength) {
+ return new VariableLengthBytes(identifier, maxLength);
+ }
+
+ private VariableLengthBytes(String identifier, int maxLength) {
+ super(identifier, Schema.FieldType.BYTES, String.valueOf(maxLength));
+ this.maxLength = maxLength;
+ }
+
+ @Override
+ public byte[] toInputType(byte[] base) {
+ checkArgument(base == null || base.length <= maxLength);
+ return base;
+ }
+ }
+
+ /** Fixed precision numeric types such as NUMERIC. */
+ public static final class FixedPrecisionNumeric extends JdbcLogicalType<BigDecimal> {
+ private final int precision;
+ private final int scale;
+
+ public static FixedPrecisionNumeric of(String identifier, int precision, int scale) {
+ return new FixedPrecisionNumeric(identifier, precision, scale);
+ }
+
+ private FixedPrecisionNumeric(String identifier, int precision, int scale) {
+ super(identifier, Schema.FieldType.DECIMAL, precision + ":" + scale);
+ this.precision = precision;
+ this.scale = scale;
+ }
+
+ @Override
+ public BigDecimal toInputType(BigDecimal base) {
+ checkArgument(base == null || (base.precision() == precision && base.scale() == scale));
+ return base;
+ }
+ }
+}
diff --git a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java
new file mode 100644
index 0000000..b0f1b54
--- /dev/null
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java
@@ -0,0 +1,411 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import static java.sql.JDBCType.BINARY;
+import static java.sql.JDBCType.CHAR;
+import static java.sql.JDBCType.LONGNVARCHAR;
+import static java.sql.JDBCType.LONGVARBINARY;
+import static java.sql.JDBCType.LONGVARCHAR;
+import static java.sql.JDBCType.NCHAR;
+import static java.sql.JDBCType.NUMERIC;
+import static java.sql.JDBCType.NVARCHAR;
+import static java.sql.JDBCType.VARBINARY;
+import static java.sql.JDBCType.VARCHAR;
+import static java.sql.JDBCType.valueOf;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import java.io.Serializable;
+import java.sql.Array;
+import java.sql.Date;
+import java.sql.JDBCType;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.time.LocalTime;
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.TimeZone;
+import java.util.function.BiFunction;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.joda.time.DateTime;
+import org.joda.time.LocalDate;
+import org.joda.time.chrono.ISOChronology;
+
+/** Provides utility functions for working with Beam {@link Schema} types. */
+class SchemaUtil {
+ /**
+ * Interface implemented by functions that extract values of different types from a JDBC
+ * ResultSet.
+ */
+ @FunctionalInterface
+ interface ResultSetFieldExtractor extends Serializable {
+ Object extract(ResultSet rs, Integer index) throws SQLException;
+ }
+
+ // ResultSetExtractors for primitive schema types (excluding arrays, structs and logical types).
+ private static final EnumMap<Schema.TypeName, ResultSetFieldExtractor>
+ RESULTSET_FIELD_EXTRACTORS =
+ new EnumMap<>(
+ ImmutableMap.<Schema.TypeName, ResultSetFieldExtractor>builder()
+ .put(Schema.TypeName.BOOLEAN, ResultSet::getBoolean)
+ .put(Schema.TypeName.BYTE, ResultSet::getByte)
+ .put(Schema.TypeName.BYTES, ResultSet::getBytes)
+ .put(Schema.TypeName.DATETIME, ResultSet::getTimestamp)
+ .put(Schema.TypeName.DECIMAL, ResultSet::getBigDecimal)
+ .put(Schema.TypeName.DOUBLE, ResultSet::getDouble)
+ .put(Schema.TypeName.FLOAT, ResultSet::getFloat)
+ .put(Schema.TypeName.INT16, ResultSet::getShort)
+ .put(Schema.TypeName.INT32, ResultSet::getInt)
+ .put(Schema.TypeName.INT64, ResultSet::getLong)
+ .put(Schema.TypeName.STRING, ResultSet::getString)
+ .build());
+
+ private static final ResultSetFieldExtractor DATE_EXTRACTOR = createDateExtractor();
+ private static final ResultSetFieldExtractor TIME_EXTRACTOR = createTimeExtractor();
+ private static final ResultSetFieldExtractor TIMESTAMP_EXTRACTOR = createTimestampExtractor();
+
+ /**
+ * Interface implemented by functions that create Beam {@link
+ * org.apache.beam.sdk.schemas.Schema.Field} corresponding to JDBC field metadata.
+ */
+ @FunctionalInterface
+ interface BeamFieldConverter extends Serializable {
+ Schema.Field create(int index, ResultSetMetaData md) throws SQLException;
+ }
+
+ private static BeamFieldConverter jdbcTypeToBeamFieldConverter(JDBCType jdbcType) {
+ switch (jdbcType) {
+ case ARRAY:
+ return beamArrayField();
+ case BIGINT:
+ return beamFieldOfType(Schema.FieldType.INT64);
+ case BINARY:
+ return beamLogicalField(BINARY.getName(), LogicalTypes.FixedLengthBytes::of);
+ case BIT:
+ return beamFieldOfType(LogicalTypes.JDBC_BIT_TYPE);
+ case BOOLEAN:
+ return beamFieldOfType(Schema.FieldType.BOOLEAN);
+ case CHAR:
+ return beamLogicalField(CHAR.getName(), LogicalTypes.FixedLengthString::of);
+ case DATE:
+ return beamFieldOfType(LogicalTypes.JDBC_DATE_TYPE);
+ case DECIMAL:
+ return beamFieldOfType(Schema.FieldType.DECIMAL);
+ case DOUBLE:
+ return beamFieldOfType(Schema.FieldType.DOUBLE);
+ case FLOAT:
+ return beamFieldOfType(LogicalTypes.JDBC_FLOAT_TYPE);
+ case INTEGER:
+ return beamFieldOfType(Schema.FieldType.INT32);
+ case LONGNVARCHAR:
+ return beamLogicalField(LONGNVARCHAR.getName(), LogicalTypes.VariableLengthString::of);
+ case LONGVARBINARY:
+ return beamLogicalField(LONGVARBINARY.getName(), LogicalTypes.VariableLengthBytes::of);
+ case LONGVARCHAR:
+ return beamLogicalField(LONGVARCHAR.getName(), LogicalTypes.VariableLengthString::of);
+ case NCHAR:
+ return beamLogicalField(NCHAR.getName(), LogicalTypes.FixedLengthString::of);
+ case NUMERIC:
+ return beamLogicalNumericField(NUMERIC.getName());
+ case NVARCHAR:
+ return beamLogicalField(NVARCHAR.getName(), LogicalTypes.VariableLengthString::of);
+ case REAL:
+ return beamFieldOfType(Schema.FieldType.FLOAT);
+ case SMALLINT:
+ return beamFieldOfType(Schema.FieldType.INT16);
+ case TIME:
+ return beamFieldOfType(LogicalTypes.JDBC_TIME_TYPE);
+ case TIMESTAMP:
+ return beamFieldOfType(Schema.FieldType.DATETIME);
+ case TIMESTAMP_WITH_TIMEZONE:
+ return beamFieldOfType(LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE);
+ case TINYINT:
+ return beamFieldOfType(Schema.FieldType.BYTE);
+ case VARBINARY:
+ return beamLogicalField(VARBINARY.getName(), LogicalTypes.VariableLengthBytes::of);
+ case VARCHAR:
+ return beamLogicalField(VARCHAR.getName(), LogicalTypes.VariableLengthString::of);
+ default:
+ throw new UnsupportedOperationException(
+ "Converting " + jdbcType + " to Beam schema type is not supported");
+ }
+ }
+
+ /** Infers the Beam {@link Schema} from {@link ResultSetMetaData}. */
+ static Schema toBeamSchema(ResultSetMetaData md) throws SQLException {
+ Schema.Builder schemaBuilder = Schema.builder();
+
+ for (int i = 1; i <= md.getColumnCount(); i++) {
+ JDBCType jdbcType = valueOf(md.getColumnType(i));
+ BeamFieldConverter fieldConverter = jdbcTypeToBeamFieldConverter(jdbcType);
+ schemaBuilder.addField(fieldConverter.create(i, md));
+ }
+
+ return schemaBuilder.build();
+ }
+
+ /** Converts a primitive JDBC field to corresponding Beam schema field. */
+ private static BeamFieldConverter beamFieldOfType(Schema.FieldType fieldType) {
+ return (index, md) -> {
+ String label = md.getColumnLabel(index);
+ return Schema.Field.of(label, fieldType)
+ .withNullable(md.isNullable(index) == ResultSetMetaData.columnNullable);
+ };
+ }
+
+ /** Converts logical types with arguments such as VARCHAR(25). */
+ private static <InputT, BaseT> BeamFieldConverter beamLogicalField(
+ String identifier,
+ BiFunction<String, Integer, Schema.LogicalType<InputT, BaseT>> constructor) {
+ return (index, md) -> {
+ int size = md.getPrecision(index);
+ Schema.FieldType fieldType =
+ Schema.FieldType.logicalType(constructor.apply(identifier, size));
+ return beamFieldOfType(fieldType).create(index, md);
+ };
+ }
+
+ /** Converts numeric fields with specified precision and scale. */
+ private static BeamFieldConverter beamLogicalNumericField(String identifier) {
+ return (index, md) -> {
+ int precision = md.getPrecision(index);
+ int scale = md.getScale(index);
+ Schema.FieldType fieldType =
+ Schema.FieldType.logicalType(
+ LogicalTypes.FixedPrecisionNumeric.of(identifier, precision, scale));
+ return beamFieldOfType(fieldType).create(index, md);
+ };
+ }
+
+ /** Converts array fields. */
+ private static BeamFieldConverter beamArrayField() {
+ return (index, md) -> {
+ JDBCType elementJdbcType = valueOf(md.getColumnTypeName(index));
+ BeamFieldConverter elementFieldConverter = jdbcTypeToBeamFieldConverter(elementJdbcType);
+
+ String label = md.getColumnLabel(index);
+ Schema.FieldType elementBeamType = elementFieldConverter.create(index, md).getType();
+ return Schema.Field.of(label, Schema.FieldType.array(elementBeamType))
+ .withNullable(md.isNullable(index) == ResultSetMetaData.columnNullable);
+ };
+ }
+
+ /** Creates a {@link ResultSetFieldExtractor} for the given type. */
+ private static ResultSetFieldExtractor createFieldExtractor(Schema.FieldType fieldType) {
+ Schema.TypeName typeName = fieldType.getTypeName();
+ switch (typeName) {
+ case ARRAY:
+ Schema.FieldType elementType = fieldType.getCollectionElementType();
+ ResultSetFieldExtractor elementExtractor = createFieldExtractor(elementType);
+ return createArrayExtractor(elementExtractor);
+ case DATETIME:
+ return TIMESTAMP_EXTRACTOR;
+ case LOGICAL_TYPE:
+ return createLogicalTypeExtractor(fieldType.getLogicalType());
+ default:
+ if (!RESULTSET_FIELD_EXTRACTORS.containsKey(typeName)) {
+ throw new UnsupportedOperationException(
+ "BeamRowMapper does not have support for fields of type " + fieldType.toString());
+ }
+ return RESULTSET_FIELD_EXTRACTORS.get(typeName);
+ }
+ }
+
+ /** Creates a {@link ResultSetFieldExtractor} for array types. */
+ private static ResultSetFieldExtractor createArrayExtractor(
+ ResultSetFieldExtractor elementExtractor) {
+ return (rs, index) -> {
+ Array arrayVal = rs.getArray(index);
+ if (arrayVal == null) {
+ return null;
+ }
+
+ List<Object> arrayElements = new ArrayList<>();
+ ResultSet arrayRs = arrayVal.getResultSet();
+ while (arrayRs.next()) {
+ arrayElements.add(elementExtractor.extract(arrayRs, 1));
+ }
+ return arrayElements;
+ };
+ }
+
+ /** Creates a {@link ResultSetFieldExtractor} for logical types. */
+ private static <InputT, BaseT> ResultSetFieldExtractor createLogicalTypeExtractor(
+ final Schema.LogicalType<InputT, BaseT> fieldType) {
+ String logicalTypeName = fieldType.getIdentifier();
+ JDBCType underlyingType = JDBCType.valueOf(logicalTypeName);
+ switch (underlyingType) {
+ case DATE:
+ return DATE_EXTRACTOR;
+ case TIME:
+ return TIME_EXTRACTOR;
+ case TIMESTAMP_WITH_TIMEZONE:
+ return TIMESTAMP_EXTRACTOR;
+ default:
+ ResultSetFieldExtractor extractor = createFieldExtractor(fieldType.getBaseType());
+ return (rs, index) -> fieldType.toInputType((BaseT) extractor.extract(rs, index));
+ }
+ }
+
+ /** Convert SQL date type to Beam DateTime. */
+ private static ResultSetFieldExtractor createDateExtractor() {
+ return (rs, i) -> {
+ Date date = rs.getDate(i, Calendar.getInstance(TimeZone.getTimeZone(ZoneOffset.UTC)));
+ if (date == null) {
+ return null;
+ }
+ ZonedDateTime zdt = ZonedDateTime.of(date.toLocalDate(), LocalTime.MIDNIGHT, ZoneOffset.UTC);
+ return new DateTime(zdt.toInstant().toEpochMilli(), ISOChronology.getInstanceUTC());
+ };
+ }
+
+ /** Convert SQL time type to Beam DateTime. */
+ private static ResultSetFieldExtractor createTimeExtractor() {
+ return (rs, i) -> {
+ Time time = rs.getTime(i, Calendar.getInstance(TimeZone.getTimeZone(ZoneOffset.UTC)));
+ if (time == null) {
+ return null;
+ }
+ return new DateTime(time.getTime(), ISOChronology.getInstanceUTC())
+ .withDate(new LocalDate(0L));
+ };
+ }
+
+ /** Convert SQL timestamp type to Beam DateTime. */
+ private static ResultSetFieldExtractor createTimestampExtractor() {
+ return (rs, i) -> {
+ Timestamp ts = rs.getTimestamp(i, Calendar.getInstance(TimeZone.getTimeZone(ZoneOffset.UTC)));
+ if (ts == null) {
+ return null;
+ }
+ return new DateTime(ts.toInstant().toEpochMilli(), ISOChronology.getInstanceUTC());
+ };
+ }
+
+ /**
+ * A {@link org.apache.beam.sdk.io.jdbc.JdbcIO.RowMapper} implementation that converts JDBC
+ * results into Beam {@link Row} objects.
+ */
+ static final class BeamRowMapper implements JdbcIO.RowMapper<Row> {
+ private final Schema schema;
+ private final List<ResultSetFieldExtractor> fieldExtractors;
+
+ public static BeamRowMapper of(Schema schema) {
+ List<ResultSetFieldExtractor> fieldExtractors =
+ IntStream.range(0, schema.getFieldCount())
+ .mapToObj(i -> createFieldExtractor(schema.getField(i).getType()))
+ .collect(Collectors.toList());
+
+ return new BeamRowMapper(schema, fieldExtractors);
+ }
+
+ private BeamRowMapper(Schema schema, List<ResultSetFieldExtractor> fieldExtractors) {
+ this.schema = schema;
+ this.fieldExtractors = fieldExtractors;
+ }
+
+ @Override
+ public Row mapRow(ResultSet rs) throws Exception {
+ Row.Builder rowBuilder = Row.withSchema(schema);
+ for (int i = 0; i < schema.getFieldCount(); i++) {
+ rowBuilder.addValue(fieldExtractors.get(i).extract(rs, i + 1));
+ }
+ return rowBuilder.build();
+ }
+ }
+
+ /**
+ * compares two fields. Does not compare nullability of field types.
+ *
+ * @param a field 1
+ * @param b field 2
+ * @return TRUE if fields are equal. Otherwise FALSE
+ */
+ public static boolean compareSchemaField(Schema.Field a, Schema.Field b) {
+ if (!a.getName().equalsIgnoreCase(b.getName())) {
+ return false;
+ }
+
+ return compareSchemaFieldType(a.getType(), b.getType());
+ }
+
+ /**
+ * checks nullability for fields.
+ *
+ * @param fields
+ * @return TRUE if any field is not nullable
+ */
+ public static boolean checkNullabilityForFields(List<Schema.Field> fields) {
+ return fields.stream().anyMatch(field -> !field.getType().getNullable());
+ }
+
+ /**
+ * compares two FieldType. Does not compare nullability.
+ *
+ * @param a FieldType 1
+ * @param b FieldType 2
+ * @return TRUE if FieldType are equal. Otherwise FALSE
+ */
+ public static boolean compareSchemaFieldType(Schema.FieldType a, Schema.FieldType b) {
+ if (a.getTypeName().equals(b.getTypeName())) {
+ return !a.getTypeName().equals(Schema.TypeName.LOGICAL_TYPE)
+ || compareSchemaFieldType(
+ a.getLogicalType().getBaseType(), b.getLogicalType().getBaseType());
+ } else if (a.getTypeName().isLogicalType()) {
+ return a.getLogicalType().getBaseType().getTypeName().equals(b.getTypeName());
+ } else if (b.getTypeName().isLogicalType()) {
+ return b.getLogicalType().getBaseType().getTypeName().equals(a.getTypeName());
+ }
+ return false;
+ }
+
+ static class FieldWithIndex implements Serializable {
+ private final Schema.Field field;
+ private final Integer index;
+
+ private FieldWithIndex(Schema.Field field, Integer index) {
+ this.field = field;
+ this.index = index;
+ }
+
+ public static FieldWithIndex of(Schema.Field field, Integer index) {
+ checkArgument(field != null);
+ checkArgument(index != null);
+ return new FieldWithIndex(field, index);
+ }
+
+ public Schema.Field getField() {
+ return field;
+ }
+
+ public Integer getIndex() {
+ return index;
+ }
+ }
+}
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
index da5f34a..311cfe0 100644
--- a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
@@ -37,8 +37,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -150,16 +148,6 @@
return NamedTestResult.create(
uuid, timestamp, "write_time", (writeEnd - writeStart) / 1e3);
});
- suppliers.add(
- reader -> {
- double byteCount = reader.getCounterMetric("byte_count");
- return NamedTestResult.create(uuid, timestamp, "byte_count", byteCount);
- });
- suppliers.add(
- reader -> {
- double itemCount = reader.getCounterMetric("item_count");
- return NamedTestResult.create(uuid, timestamp, "item_count", itemCount);
- });
return suppliers;
}
@@ -188,8 +176,6 @@
.apply(GenerateSequence.from(0).to(numberOfRows))
.apply(ParDo.of(new TestRow.DeterministicallyConstructTestRowFn()))
.apply(ParDo.of(new TimeMonitor<>(NAMESPACE, "write_time")))
- .apply(ParDo.of(new ByteMonitor<>(NAMESPACE, "byte_count")))
- .apply(ParDo.of(new CountMonitor<>(NAMESPACE, "item_count")))
.apply(
JdbcIO.<TestRow>write()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dataSource))
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
index 82df39b..fab7d35 100644
--- a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java
@@ -17,19 +17,37 @@
*/
package org.apache.beam.sdk.io.jdbc;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
import java.io.PrintWriter;
import java.io.Serializable;
import java.io.StringWriter;
+import java.math.BigDecimal;
import java.net.InetAddress;
+import java.nio.charset.Charset;
+import java.sql.Array;
import java.sql.Connection;
+import java.sql.Date;
+import java.sql.JDBCType;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
+import java.sql.Time;
+import java.sql.Timestamp;
import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Calendar;
import java.util.Collections;
+import java.util.List;
+import java.util.TimeZone;
import javax.sql.DataSource;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.SerializableCoder;
@@ -38,23 +56,32 @@
import org.apache.beam.sdk.io.common.DatabaseTestHelper;
import org.apache.beam.sdk.io.common.NetworkTestHelper;
import org.apache.beam.sdk.io.common.TestRow;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.transforms.Select;
import org.apache.beam.sdk.testing.ExpectedLogs;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.Wait;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.commons.dbcp2.PoolingDataSource;
import org.apache.derby.drda.NetworkServerControl;
import org.apache.derby.jdbc.ClientDataSource;
+import org.joda.time.DateTime;
+import org.joda.time.LocalDate;
+import org.joda.time.chrono.ISOChronology;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.ExpectedException;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import org.slf4j.Logger;
@@ -78,6 +105,8 @@
@Rule public final transient ExpectedLogs expectedLogs = ExpectedLogs.none(JdbcIO.class);
+ @Rule public transient ExpectedException thrown = ExpectedException.none();
+
@BeforeClass
public static void beforeClass() throws Exception {
port = NetworkTestHelper.getAvailableLocalPort();
@@ -273,6 +302,95 @@
}
@Test
+ public void testReadRows() {
+ SerializableFunction<Void, DataSource> dataSourceProvider = ignored -> dataSource;
+ PCollection<Row> rows =
+ pipeline.apply(
+ JdbcIO.readRows()
+ .withDataSourceProviderFn(dataSourceProvider)
+ .withQuery(String.format("select name,id from %s where name = ?", readTableName))
+ .withStatementPreparator(
+ preparedStatement ->
+ preparedStatement.setString(1, TestRow.getNameForSeed(1))));
+
+ Schema expectedSchema =
+ Schema.of(
+ Schema.Field.of("NAME", LogicalTypes.variableLengthString(JDBCType.VARCHAR, 500))
+ .withNullable(true),
+ Schema.Field.of("ID", Schema.FieldType.INT32).withNullable(true));
+
+ assertEquals(expectedSchema, rows.getSchema());
+
+ PCollection<Row> output = rows.apply(Select.fieldNames("NAME", "ID"));
+ PAssert.that(output)
+ .containsInAnyOrder(
+ ImmutableList.of(Row.withSchema(expectedSchema).addValues("Testval1", 1).build()));
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testReadRowsWithoutStatementPreparator() {
+ SerializableFunction<Void, DataSource> dataSourceProvider = ignored -> dataSource;
+ String name = TestRow.getNameForSeed(1);
+ PCollection<Row> rows =
+ pipeline.apply(
+ JdbcIO.readRows()
+ .withDataSourceProviderFn(dataSourceProvider)
+ .withQuery(
+ String.format(
+ "select name,id from %s where name = '%s'", readTableName, name)));
+
+ Schema expectedSchema =
+ Schema.of(
+ Schema.Field.of("NAME", LogicalTypes.variableLengthString(JDBCType.VARCHAR, 500))
+ .withNullable(true),
+ Schema.Field.of("ID", Schema.FieldType.INT32).withNullable(true));
+
+ assertEquals(expectedSchema, rows.getSchema());
+
+ PCollection<Row> output = rows.apply(Select.fieldNames("NAME", "ID"));
+ PAssert.that(output)
+ .containsInAnyOrder(
+ ImmutableList.of(Row.withSchema(expectedSchema).addValues(name, 1).build()));
+
+ pipeline.run();
+ }
+
+ @Test
+ public void testReadWithSchema() {
+ SerializableFunction<Void, DataSource> dataSourceProvider = ignored -> dataSource;
+ JdbcIO.RowMapper<RowWithSchema> rowMapper =
+ rs -> new RowWithSchema(rs.getString("NAME"), rs.getInt("ID"));
+ pipeline.getSchemaRegistry().registerJavaBean(RowWithSchema.class);
+
+ PCollection<RowWithSchema> rows =
+ pipeline.apply(
+ JdbcIO.<RowWithSchema>read()
+ .withDataSourceProviderFn(dataSourceProvider)
+ .withQuery(String.format("select name,id from %s where name = ?", readTableName))
+ .withRowMapper(rowMapper)
+ .withCoder(SerializableCoder.of(RowWithSchema.class))
+ .withStatementPreparator(
+ preparedStatement ->
+ preparedStatement.setString(1, TestRow.getNameForSeed(1))));
+
+ Schema expectedSchema =
+ Schema.of(
+ Schema.Field.of("name", Schema.FieldType.STRING),
+ Schema.Field.of("id", Schema.FieldType.INT32));
+
+ assertEquals(expectedSchema, rows.getSchema());
+
+ PCollection<Row> output = rows.apply(Select.fieldNames("name", "id"));
+ PAssert.that(output)
+ .containsInAnyOrder(
+ ImmutableList.of(Row.withSchema(expectedSchema).addValues("Testval1", 1).build()));
+
+ pipeline.run();
+ }
+
+ @Test
public void testWrite() throws Exception {
final long rowsToAdd = 1000L;
@@ -423,6 +541,345 @@
}
@Test
+ public void testWriteWithoutPreparedStatement() throws Exception {
+ final int rowsToAdd = 10;
+
+ Schema.Builder schemaBuilder = Schema.builder();
+ schemaBuilder.addField(Schema.Field.of("column_boolean", Schema.FieldType.BOOLEAN));
+ schemaBuilder.addField(Schema.Field.of("column_string", Schema.FieldType.STRING));
+ schemaBuilder.addField(Schema.Field.of("column_int", Schema.FieldType.INT32));
+ schemaBuilder.addField(Schema.Field.of("column_long", Schema.FieldType.INT64));
+ schemaBuilder.addField(Schema.Field.of("column_float", Schema.FieldType.FLOAT));
+ schemaBuilder.addField(Schema.Field.of("column_double", Schema.FieldType.DOUBLE));
+ schemaBuilder.addField(Schema.Field.of("column_bigdecimal", Schema.FieldType.DECIMAL));
+ schemaBuilder.addField(Schema.Field.of("column_date", LogicalTypes.JDBC_DATE_TYPE));
+ schemaBuilder.addField(Schema.Field.of("column_time", LogicalTypes.JDBC_TIME_TYPE));
+ schemaBuilder.addField(
+ Schema.Field.of("column_timestamptz", LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE));
+ schemaBuilder.addField(Schema.Field.of("column_timestamp", Schema.FieldType.DATETIME));
+ schemaBuilder.addField(Schema.Field.of("column_short", Schema.FieldType.INT16));
+ Schema schema = schemaBuilder.build();
+
+ String tableName = DatabaseTestHelper.getTestTableName("UT_WRITE_PS");
+ StringBuilder stmt = new StringBuilder("CREATE TABLE ");
+ stmt.append(tableName);
+ stmt.append(" (");
+ stmt.append("column_boolean BOOLEAN,"); // boolean
+ stmt.append("column_string VARCHAR(254),"); // String
+ stmt.append("column_int INTEGER,"); // int
+ stmt.append("column_long BIGINT,"); // long
+ stmt.append("column_float REAL,"); // float
+ stmt.append("column_double DOUBLE PRECISION,"); // double
+ stmt.append("column_bigdecimal DECIMAL(13,0),"); // BigDecimal
+ stmt.append("column_date DATE,"); // Date
+ stmt.append("column_time TIME,"); // Time
+ stmt.append("column_timestamptz TIMESTAMP,"); // Timestamp
+ stmt.append("column_timestamp TIMESTAMP,"); // Timestamp
+ stmt.append("column_short SMALLINT"); // short
+ stmt.append(" )");
+ DatabaseTestHelper.createTableWithStatement(dataSource, stmt.toString());
+ try {
+ ArrayList<Row> data = getRowsToWrite(rowsToAdd, schema);
+ pipeline
+ .apply(Create.of(data))
+ .setRowSchema(schema)
+ .apply(
+ JdbcIO.<Row>write()
+ .withDataSourceConfiguration(
+ JdbcIO.DataSourceConfiguration.create(
+ "org.apache.derby.jdbc.ClientDriver",
+ "jdbc:derby://localhost:" + port + "/target/beam"))
+ .withBatchSize(10L)
+ .withTable(tableName));
+ pipeline.run();
+ assertRowCount(tableName, rowsToAdd);
+ } finally {
+ DatabaseTestHelper.deleteTable(dataSource, tableName);
+ }
+ }
+
+ @Test
+ public void testWriteWithoutPreparedStatementWithReadRows() throws Exception {
+ SerializableFunction<Void, DataSource> dataSourceProvider = ignored -> dataSource;
+ PCollection<Row> rows =
+ pipeline.apply(
+ JdbcIO.readRows()
+ .withDataSourceProviderFn(dataSourceProvider)
+ .withQuery(String.format("select name,id from %s where name = ?", readTableName))
+ .withStatementPreparator(
+ preparedStatement ->
+ preparedStatement.setString(1, TestRow.getNameForSeed(1))));
+
+ String writeTableName = DatabaseTestHelper.getTestTableName("UT_WRITE_PS_WITH_READ_ROWS");
+ DatabaseTestHelper.createTableForRowWithSchema(dataSource, writeTableName);
+ try {
+ rows.apply(
+ JdbcIO.<Row>write()
+ .withDataSourceConfiguration(
+ JdbcIO.DataSourceConfiguration.create(
+ "org.apache.derby.jdbc.ClientDriver",
+ "jdbc:derby://localhost:" + port + "/target/beam"))
+ .withBatchSize(10L)
+ .withTable(writeTableName));
+ pipeline.run();
+ } finally {
+ DatabaseTestHelper.deleteTable(dataSource, writeTableName);
+ }
+ }
+
+ @Test
+ public void testWriteWithoutPsWithNonNullableTableField() throws Exception {
+ final int rowsToAdd = 10;
+
+ Schema.Builder schemaBuilder = Schema.builder();
+ schemaBuilder.addField(Schema.Field.of("column_boolean", Schema.FieldType.BOOLEAN));
+ schemaBuilder.addField(Schema.Field.of("column_string", Schema.FieldType.STRING));
+ Schema schema = schemaBuilder.build();
+
+ String tableName = DatabaseTestHelper.getTestTableName("UT_WRITE");
+ StringBuilder stmt = new StringBuilder("CREATE TABLE ");
+ stmt.append(tableName);
+ stmt.append(" (");
+ stmt.append("column_boolean BOOLEAN,");
+ stmt.append("column_int INTEGER NOT NULL");
+ stmt.append(" )");
+ DatabaseTestHelper.createTableWithStatement(dataSource, stmt.toString());
+ try {
+ ArrayList<Row> data = getRowsToWrite(rowsToAdd, schema);
+ pipeline
+ .apply(Create.of(data))
+ .setRowSchema(schema)
+ .apply(
+ JdbcIO.<Row>write()
+ .withDataSourceConfiguration(
+ JdbcIO.DataSourceConfiguration.create(
+ "org.apache.derby.jdbc.ClientDriver",
+ "jdbc:derby://localhost:" + port + "/target/beam"))
+ .withBatchSize(10L)
+ .withTable(tableName));
+ pipeline.run();
+ } finally {
+ DatabaseTestHelper.deleteTable(dataSource, tableName);
+ thrown.expect(RuntimeException.class);
+ }
+ }
+
+ @Test
+ public void testWriteWithoutPreparedStatementAndNonRowType() throws Exception {
+ final int rowsToAdd = 10;
+
+ String tableName = DatabaseTestHelper.getTestTableName("UT_WRITE_PS_NON_ROW");
+ DatabaseTestHelper.createTableForRowWithSchema(dataSource, tableName);
+ try {
+ List<RowWithSchema> data = getRowsWithSchemaToWrite(rowsToAdd);
+
+ pipeline
+ .apply(Create.of(data))
+ .apply(
+ JdbcIO.<RowWithSchema>write()
+ .withDataSourceConfiguration(
+ JdbcIO.DataSourceConfiguration.create(
+ "org.apache.derby.jdbc.ClientDriver",
+ "jdbc:derby://localhost:" + port + "/target/beam"))
+ .withBatchSize(10L)
+ .withTable(tableName));
+ pipeline.run();
+ assertRowCount(tableName, rowsToAdd);
+ } finally {
+ DatabaseTestHelper.deleteTable(dataSource, tableName);
+ }
+ }
+
+ @Test
+ public void testGetPreparedStatementSetCaller() throws Exception {
+
+ Schema schema =
+ Schema.builder()
+ .addField("bigint_col", Schema.FieldType.INT64)
+ .addField("binary_col", Schema.FieldType.BYTES)
+ .addField("bit_col", Schema.FieldType.BOOLEAN)
+ .addField("char_col", Schema.FieldType.STRING)
+ .addField("decimal_col", Schema.FieldType.DECIMAL)
+ .addField("double_col", Schema.FieldType.DOUBLE)
+ .addField("float_col", Schema.FieldType.FLOAT)
+ .addField("integer_col", Schema.FieldType.INT32)
+ .addField("datetime_col", Schema.FieldType.DATETIME)
+ .addField("int16_col", Schema.FieldType.INT16)
+ .addField("byte_col", Schema.FieldType.BYTE)
+ .build();
+ Row row =
+ Row.withSchema(schema)
+ .addValues(
+ 42L,
+ "binary".getBytes(Charset.forName("UTF-8")),
+ true,
+ "char",
+ BigDecimal.valueOf(25L),
+ 20.5D,
+ 15.5F,
+ 10,
+ new DateTime(),
+ (short) 5,
+ Byte.parseByte("1", 2))
+ .build();
+
+ PreparedStatement psMocked = mock(PreparedStatement.class);
+
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.INT64)
+ .set(row, psMocked, 0, SchemaUtil.FieldWithIndex.of(schema.getField(0), 0));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.BYTES)
+ .set(row, psMocked, 1, SchemaUtil.FieldWithIndex.of(schema.getField(1), 1));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.BOOLEAN)
+ .set(row, psMocked, 2, SchemaUtil.FieldWithIndex.of(schema.getField(2), 2));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.STRING)
+ .set(row, psMocked, 3, SchemaUtil.FieldWithIndex.of(schema.getField(3), 3));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.DECIMAL)
+ .set(row, psMocked, 4, SchemaUtil.FieldWithIndex.of(schema.getField(4), 4));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.DOUBLE)
+ .set(row, psMocked, 5, SchemaUtil.FieldWithIndex.of(schema.getField(5), 5));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.FLOAT)
+ .set(row, psMocked, 6, SchemaUtil.FieldWithIndex.of(schema.getField(6), 6));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.INT32)
+ .set(row, psMocked, 7, SchemaUtil.FieldWithIndex.of(schema.getField(7), 7));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.DATETIME)
+ .set(row, psMocked, 8, SchemaUtil.FieldWithIndex.of(schema.getField(8), 8));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.INT16)
+ .set(row, psMocked, 9, SchemaUtil.FieldWithIndex.of(schema.getField(9), 9));
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.BYTE)
+ .set(row, psMocked, 10, SchemaUtil.FieldWithIndex.of(schema.getField(10), 10));
+
+ verify(psMocked, times(1)).setLong(1, 42L);
+ verify(psMocked, times(1)).setBytes(2, "binary".getBytes(Charset.forName("UTF-8")));
+ verify(psMocked, times(1)).setBoolean(3, true);
+ verify(psMocked, times(1)).setString(4, "char");
+ verify(psMocked, times(1)).setBigDecimal(5, BigDecimal.valueOf(25L));
+ verify(psMocked, times(1)).setDouble(6, 20.5D);
+ verify(psMocked, times(1)).setFloat(7, 15.5F);
+ verify(psMocked, times(1)).setInt(8, 10);
+ verify(psMocked, times(1))
+ .setTimestamp(9, new Timestamp(row.getDateTime("datetime_col").getMillis()));
+ verify(psMocked, times(1)).setInt(10, (short) 5);
+ verify(psMocked, times(1)).setByte(11, Byte.parseByte("1", 2));
+ }
+
+ @Test
+ public void testGetPreparedStatementSetCallerForLogicalTypes() throws Exception {
+
+ Schema schema =
+ Schema.builder()
+ .addField("logical_date_col", LogicalTypes.JDBC_DATE_TYPE)
+ .addField("logical_time_col", LogicalTypes.JDBC_TIME_TYPE)
+ .addField("logical_time_with_tz_col", LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE)
+ .build();
+
+ long epochMilli = 1558719710000L;
+ DateTime dateTime = new DateTime(epochMilli, ISOChronology.getInstanceUTC());
+
+ Row row =
+ Row.withSchema(schema)
+ .addValues(
+ dateTime.withTimeAtStartOfDay(), dateTime.withDate(new LocalDate(0L)), dateTime)
+ .build();
+
+ PreparedStatement psMocked = mock(PreparedStatement.class);
+
+ JdbcUtil.getPreparedStatementSetCaller(LogicalTypes.JDBC_DATE_TYPE)
+ .set(row, psMocked, 0, SchemaUtil.FieldWithIndex.of(schema.getField(0), 0));
+ JdbcUtil.getPreparedStatementSetCaller(LogicalTypes.JDBC_TIME_TYPE)
+ .set(row, psMocked, 1, SchemaUtil.FieldWithIndex.of(schema.getField(1), 1));
+ JdbcUtil.getPreparedStatementSetCaller(LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE)
+ .set(row, psMocked, 2, SchemaUtil.FieldWithIndex.of(schema.getField(2), 2));
+
+ verify(psMocked, times(1)).setDate(1, new Date(row.getDateTime(0).getMillis()));
+ verify(psMocked, times(1)).setTime(2, new Time(row.getDateTime(1).getMillis()));
+
+ Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC"));
+ cal.setTimeInMillis(epochMilli);
+
+ verify(psMocked, times(1)).setTimestamp(3, new Timestamp(cal.getTime().getTime()), cal);
+ }
+
+ @Test
+ public void testGetPreparedStatementSetCallerForArray() throws Exception {
+
+ Schema schema =
+ Schema.builder()
+ .addField("string_array_col", Schema.FieldType.array(Schema.FieldType.STRING))
+ .build();
+
+ List<String> stringList = Arrays.asList("string 1", "string 2");
+
+ Row row = Row.withSchema(schema).addValues(stringList).build();
+
+ PreparedStatement psMocked = mock(PreparedStatement.class);
+ Connection connectionMocked = mock(Connection.class);
+ Array arrayMocked = mock(Array.class);
+
+ when(psMocked.getConnection()).thenReturn(connectionMocked);
+ when(connectionMocked.createArrayOf(anyString(), any())).thenReturn(arrayMocked);
+
+ JdbcUtil.getPreparedStatementSetCaller(Schema.FieldType.array(Schema.FieldType.STRING))
+ .set(row, psMocked, 0, SchemaUtil.FieldWithIndex.of(schema.getField(0), 0));
+
+ verify(psMocked, times(1)).setArray(1, arrayMocked);
+ }
+
+ private static ArrayList<Row> getRowsToWrite(long rowsToAdd, Schema schema) {
+
+ ArrayList<Row> data = new ArrayList<>();
+ for (int i = 0; i < rowsToAdd; i++) {
+ List<Object> fields = new ArrayList<>();
+
+ Row row =
+ schema.getFields().stream()
+ .map(field -> dummyFieldValue(field.getType()))
+ .collect(Row.toRow(schema));
+ data.add(row);
+ }
+ return data;
+ }
+
+ private static ArrayList<RowWithSchema> getRowsWithSchemaToWrite(long rowsToAdd) {
+
+ ArrayList<RowWithSchema> data = new ArrayList<>();
+ for (int i = 0; i < rowsToAdd; i++) {
+ data.add(new RowWithSchema("Test", i));
+ }
+ return data;
+ }
+
+ private static Object dummyFieldValue(Schema.FieldType fieldType) {
+ long epochMilli = 1558719710000L;
+ if (fieldType.equals(Schema.FieldType.STRING)) {
+ return "string value";
+ } else if (fieldType.equals(Schema.FieldType.INT32)) {
+ return 100;
+ } else if (fieldType.equals(Schema.FieldType.DOUBLE)) {
+ return 20.5D;
+ } else if (fieldType.equals(Schema.FieldType.BOOLEAN)) {
+ return Boolean.TRUE;
+ } else if (fieldType.equals(Schema.FieldType.INT16)) {
+ return Short.MAX_VALUE;
+ } else if (fieldType.equals(Schema.FieldType.INT64)) {
+ return Long.MAX_VALUE;
+ } else if (fieldType.equals(Schema.FieldType.FLOAT)) {
+ return 15.5F;
+ } else if (fieldType.equals(Schema.FieldType.DECIMAL)) {
+ return BigDecimal.ONE;
+ } else if (fieldType.equals(LogicalTypes.JDBC_DATE_TYPE)) {
+ return new DateTime(epochMilli, ISOChronology.getInstanceUTC()).withTimeAtStartOfDay();
+ } else if (fieldType.equals(LogicalTypes.JDBC_TIME_TYPE)) {
+ return new DateTime(epochMilli, ISOChronology.getInstanceUTC()).withDate(new LocalDate(0L));
+ } else if (fieldType.equals(LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE)) {
+ return new DateTime(epochMilli, ISOChronology.getInstanceUTC());
+ } else if (fieldType.equals(Schema.FieldType.DATETIME)) {
+ return new DateTime(epochMilli, ISOChronology.getInstanceUTC());
+ } else {
+ return null;
+ }
+ }
+
+ @Test
public void testWriteWithEmptyPCollection() {
pipeline
.apply(Create.empty(KvCoder.of(VarIntCoder.of(), StringUtf8Coder.of())))
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
new file mode 100644
index 0000000..a867f8e
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import static org.junit.Assert.assertEquals;
+
+import org.apache.beam.sdk.schemas.Schema;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test JdbcUtil. */
+@RunWith(JUnit4.class)
+public class JdbcUtilTest {
+
+ @Test
+ public void testGetPreparedStatementSetCaller() throws Exception {
+ Schema wantSchema =
+ Schema.builder()
+ .addField("col1", Schema.FieldType.INT64)
+ .addField("col2", Schema.FieldType.INT64)
+ .addField("col3", Schema.FieldType.INT64)
+ .build();
+
+ String generatedStmt = JdbcUtil.generateStatement("test_table", wantSchema.getFields());
+ String expectedStmt = "INSERT INTO test_table(col1, col2, col3) VALUES(?, ?, ?)";
+ assertEquals(expectedStmt, generatedStmt);
+ }
+}
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/RowWithSchema.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/RowWithSchema.java
new file mode 100644
index 0000000..a175216
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/RowWithSchema.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.schemas.JavaBeanSchema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaCreate;
+
+/** Test row. */
+@DefaultSchema(JavaBeanSchema.class)
+public class RowWithSchema implements Serializable {
+ private final String name;
+ private final int id;
+
+ @SchemaCreate
+ public RowWithSchema(String name, int id) {
+ this.name = name;
+ this.id = id;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public int getId() {
+ return id;
+ }
+}
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/SchemaUtilTest.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/SchemaUtilTest.java
new file mode 100644
index 0000000..7ec9e7b
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/SchemaUtilTest.java
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.math.BigDecimal;
+import java.nio.charset.Charset;
+import java.sql.Array;
+import java.sql.Date;
+import java.sql.JDBCType;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.sql.Time;
+import java.sql.Timestamp;
+import java.sql.Types;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.joda.time.DateTime;
+import org.joda.time.LocalDate;
+import org.joda.time.chrono.ISOChronology;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Test SchemaUtils. */
+@RunWith(JUnit4.class)
+public class SchemaUtilTest {
+ @Test
+ public void testToBeamSchema() throws SQLException {
+ ResultSetMetaData mockResultSetMetaData = mock(ResultSetMetaData.class);
+
+ ImmutableList<JdbcFieldInfo> fieldInfo =
+ ImmutableList.of(
+ JdbcFieldInfo.of("int_array_col", Types.ARRAY, JDBCType.INTEGER.getName(), false),
+ JdbcFieldInfo.of("bigint_col", Types.BIGINT),
+ JdbcFieldInfo.of("binary_col", Types.BINARY, 255),
+ JdbcFieldInfo.of("bit_col", Types.BIT),
+ JdbcFieldInfo.of("boolean_col", Types.BOOLEAN),
+ JdbcFieldInfo.of("char_col", Types.CHAR, 255),
+ JdbcFieldInfo.of("date_col", Types.DATE),
+ JdbcFieldInfo.of("decimal_col", Types.DECIMAL),
+ JdbcFieldInfo.of("double_col", Types.DOUBLE),
+ JdbcFieldInfo.of("float_col", Types.FLOAT),
+ JdbcFieldInfo.of("integer_col", Types.INTEGER),
+ JdbcFieldInfo.of("longnvarchar_col", Types.LONGNVARCHAR, 1024),
+ JdbcFieldInfo.of("longvarchar_col", Types.LONGVARCHAR, 1024),
+ JdbcFieldInfo.of("longvarbinary_col", Types.LONGVARBINARY, 1024),
+ JdbcFieldInfo.of("nchar_col", Types.NCHAR, 255),
+ JdbcFieldInfo.of("numeric_col", Types.NUMERIC, 12, 4),
+ JdbcFieldInfo.of("nvarchar_col", Types.NVARCHAR, 255),
+ JdbcFieldInfo.of("real_col", Types.REAL),
+ JdbcFieldInfo.of("smallint_col", Types.SMALLINT),
+ JdbcFieldInfo.of("time_col", Types.TIME),
+ JdbcFieldInfo.of("timestamp_col", Types.TIMESTAMP),
+ JdbcFieldInfo.of("timestamptz_col", Types.TIMESTAMP_WITH_TIMEZONE),
+ JdbcFieldInfo.of("tinyint_col", Types.TINYINT),
+ JdbcFieldInfo.of("varbinary_col", Types.VARBINARY, 255),
+ JdbcFieldInfo.of("varchar_col", Types.VARCHAR, 255));
+
+ when(mockResultSetMetaData.getColumnCount()).thenReturn(fieldInfo.size());
+ for (int i = 0; i < fieldInfo.size(); i++) {
+ JdbcFieldInfo f = fieldInfo.get(i);
+ when(mockResultSetMetaData.getColumnLabel(eq(i + 1))).thenReturn(f.columnLabel);
+ when(mockResultSetMetaData.getColumnType(eq(i + 1))).thenReturn(f.columnType);
+ when(mockResultSetMetaData.getColumnTypeName(eq(i + 1))).thenReturn(f.columnTypeName);
+ when(mockResultSetMetaData.getPrecision(eq(i + 1))).thenReturn(f.precision);
+ when(mockResultSetMetaData.getScale(eq(i + 1))).thenReturn(f.scale);
+ when(mockResultSetMetaData.isNullable(eq(i + 1)))
+ .thenReturn(
+ f.nullable ? ResultSetMetaData.columnNullable : ResultSetMetaData.columnNoNulls);
+ }
+
+ Schema wantBeamSchema =
+ Schema.builder()
+ .addArrayField("int_array_col", Schema.FieldType.INT32)
+ .addField("bigint_col", Schema.FieldType.INT64)
+ .addField("binary_col", LogicalTypes.fixedLengthBytes(JDBCType.BINARY, 255))
+ .addField("bit_col", LogicalTypes.JDBC_BIT_TYPE)
+ .addField("boolean_col", Schema.FieldType.BOOLEAN)
+ .addField("char_col", LogicalTypes.fixedLengthString(JDBCType.CHAR, 255))
+ .addField("date_col", LogicalTypes.JDBC_DATE_TYPE)
+ .addField("decimal_col", Schema.FieldType.DECIMAL)
+ .addField("double_col", Schema.FieldType.DOUBLE)
+ .addField("float_col", LogicalTypes.JDBC_FLOAT_TYPE)
+ .addField("integer_col", Schema.FieldType.INT32)
+ .addField(
+ "longnvarchar_col", LogicalTypes.variableLengthString(JDBCType.LONGNVARCHAR, 1024))
+ .addField(
+ "longvarchar_col", LogicalTypes.variableLengthString(JDBCType.LONGVARCHAR, 1024))
+ .addField(
+ "longvarbinary_col", LogicalTypes.variableLengthBytes(JDBCType.LONGVARBINARY, 1024))
+ .addField("nchar_col", LogicalTypes.fixedLengthString(JDBCType.NCHAR, 255))
+ .addField("numeric_col", LogicalTypes.numeric(12, 4))
+ .addField("nvarchar_col", LogicalTypes.variableLengthString(JDBCType.NVARCHAR, 255))
+ .addField("real_col", Schema.FieldType.FLOAT)
+ .addField("smallint_col", Schema.FieldType.INT16)
+ .addField("time_col", LogicalTypes.JDBC_TIME_TYPE)
+ .addField("timestamp_col", Schema.FieldType.DATETIME)
+ .addField("timestamptz_col", LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE)
+ .addField("tinyint_col", Schema.FieldType.BYTE)
+ .addField("varbinary_col", LogicalTypes.variableLengthBytes(JDBCType.VARBINARY, 255))
+ .addField("varchar_col", LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255))
+ .build();
+
+ Schema haveBeamSchema = SchemaUtil.toBeamSchema(mockResultSetMetaData);
+ assertEquals(wantBeamSchema, haveBeamSchema);
+ }
+
+ @Test
+ public void testBeamRowMapperArray() throws Exception {
+ ResultSet mockArrayElementsResultSet = mock(ResultSet.class);
+ when(mockArrayElementsResultSet.next()).thenReturn(true, true, true, false);
+ when(mockArrayElementsResultSet.getInt(eq(1))).thenReturn(10, 20, 30);
+
+ Array mockArray = mock(Array.class);
+ when(mockArray.getResultSet()).thenReturn(mockArrayElementsResultSet);
+
+ ResultSet mockResultSet = mock(ResultSet.class);
+ when(mockResultSet.getArray(eq(1))).thenReturn(mockArray);
+
+ Schema wantSchema =
+ Schema.builder().addField("array", Schema.FieldType.array(Schema.FieldType.INT32)).build();
+ Row wantRow =
+ Row.withSchema(wantSchema).addValues((Object) ImmutableList.of(10, 20, 30)).build();
+
+ SchemaUtil.BeamRowMapper beamRowMapper = SchemaUtil.BeamRowMapper.of(wantSchema);
+ Row haveRow = beamRowMapper.mapRow(mockResultSet);
+
+ assertEquals(wantRow, haveRow);
+ }
+
+ @Test
+ public void testBeamRowMapperPrimitiveTypes() throws Exception {
+ ResultSet mockResultSet = mock(ResultSet.class);
+ when(mockResultSet.getLong(eq(1))).thenReturn(42L);
+ when(mockResultSet.getBytes(eq(2))).thenReturn("binary".getBytes(Charset.forName("UTF-8")));
+ when(mockResultSet.getBoolean(eq(3))).thenReturn(true);
+ when(mockResultSet.getBoolean(eq(4))).thenReturn(false);
+ when(mockResultSet.getString(eq(5))).thenReturn("char");
+ when(mockResultSet.getBigDecimal(eq(6))).thenReturn(BigDecimal.valueOf(25L));
+ when(mockResultSet.getDouble(eq(7))).thenReturn(20.5D);
+ when(mockResultSet.getFloat(eq(8))).thenReturn(15.5F);
+ when(mockResultSet.getInt(eq(9))).thenReturn(10);
+ when(mockResultSet.getString(eq(10))).thenReturn("longvarchar");
+ when(mockResultSet.getBytes(eq(11)))
+ .thenReturn("longvarbinary".getBytes(Charset.forName("UTF-8")));
+ when(mockResultSet.getBigDecimal(eq(12))).thenReturn(BigDecimal.valueOf(1000L));
+ when(mockResultSet.getFloat(eq(13))).thenReturn(32F);
+ when(mockResultSet.getShort(eq(14))).thenReturn((short) 8);
+ when(mockResultSet.getShort(eq(15))).thenReturn((short) 4);
+ when(mockResultSet.getBytes(eq(16))).thenReturn("varbinary".getBytes(Charset.forName("UTF-8")));
+ when(mockResultSet.getString(eq(17))).thenReturn("varchar");
+
+ Schema wantSchema =
+ Schema.builder()
+ .addField("bigint_col", Schema.FieldType.INT64)
+ .addField("binary_col", Schema.FieldType.BYTES)
+ .addField("bit_col", Schema.FieldType.BOOLEAN)
+ .addField("boolean_col", Schema.FieldType.BOOLEAN)
+ .addField("char_col", Schema.FieldType.STRING)
+ .addField("decimal_col", Schema.FieldType.DECIMAL)
+ .addField("double_col", Schema.FieldType.DOUBLE)
+ .addField("float_col", Schema.FieldType.FLOAT)
+ .addField("integer_col", Schema.FieldType.INT32)
+ .addField("longvarchar_col", Schema.FieldType.STRING)
+ .addField("longvarbinary_col", Schema.FieldType.BYTES)
+ .addField("numeric_col", Schema.FieldType.DECIMAL)
+ .addField("real_col", Schema.FieldType.FLOAT)
+ .addField("smallint_col", Schema.FieldType.INT16)
+ .addField("tinyint_col", Schema.FieldType.INT16)
+ .addField("varbinary_col", Schema.FieldType.BYTES)
+ .addField("varchar_col", Schema.FieldType.STRING)
+ .build();
+ Row wantRow =
+ Row.withSchema(wantSchema)
+ .addValues(
+ 42L,
+ "binary".getBytes(Charset.forName("UTF-8")),
+ true,
+ false,
+ "char",
+ BigDecimal.valueOf(25L),
+ 20.5D,
+ 15.5F,
+ 10,
+ "longvarchar",
+ "longvarbinary".getBytes(Charset.forName("UTF-8")),
+ BigDecimal.valueOf(1000L),
+ 32F,
+ (short) 8,
+ (short) 4,
+ "varbinary".getBytes(Charset.forName("UTF-8")),
+ "varchar")
+ .build();
+
+ SchemaUtil.BeamRowMapper beamRowMapper = SchemaUtil.BeamRowMapper.of(wantSchema);
+ Row haveRow = beamRowMapper.mapRow(mockResultSet);
+
+ assertEquals(wantRow, haveRow);
+ }
+
+ @Test
+ public void testBeamRowMapperDateTime() throws Exception {
+ long epochMilli = 1558719710000L;
+
+ ResultSet mockResultSet = mock(ResultSet.class);
+ when(mockResultSet.getDate(eq(1), any())).thenReturn(new Date(epochMilli));
+ when(mockResultSet.getTime(eq(2), any())).thenReturn(new Time(epochMilli));
+ when(mockResultSet.getTimestamp(eq(3), any())).thenReturn(new Timestamp(epochMilli));
+ when(mockResultSet.getTimestamp(eq(4), any())).thenReturn(new Timestamp(epochMilli));
+
+ Schema wantSchema =
+ Schema.builder()
+ .addField("date_col", LogicalTypes.JDBC_DATE_TYPE)
+ .addField("time_col", LogicalTypes.JDBC_TIME_TYPE)
+ .addField("timestamptz_col", LogicalTypes.JDBC_TIMESTAMP_WITH_TIMEZONE_TYPE)
+ .addField("timestamp_col", Schema.FieldType.DATETIME)
+ .build();
+
+ DateTime wantDateTime = new DateTime(epochMilli, ISOChronology.getInstanceUTC());
+
+ Row wantRow =
+ Row.withSchema(wantSchema)
+ .addValues(
+ wantDateTime.withTimeAtStartOfDay(),
+ wantDateTime.withDate(new LocalDate(0L)),
+ wantDateTime,
+ wantDateTime)
+ .build();
+
+ SchemaUtil.BeamRowMapper beamRowMapper = SchemaUtil.BeamRowMapper.of(wantSchema);
+ Row haveRow = beamRowMapper.mapRow(mockResultSet);
+
+ assertEquals(wantRow, haveRow);
+ }
+
+ ////////////////////////////////////////////////////////////////////////////////////////
+ private static final class JdbcFieldInfo {
+ private final String columnLabel;
+ private final int columnType;
+ private final String columnTypeName;
+ private final boolean nullable;
+ private final int precision;
+ private final int scale;
+
+ private JdbcFieldInfo(
+ String columnLabel,
+ int columnType,
+ String columnTypeName,
+ boolean nullable,
+ int precision,
+ int scale) {
+ this.columnLabel = columnLabel;
+ this.columnType = columnType;
+ this.columnTypeName = columnTypeName;
+ this.nullable = nullable;
+ this.precision = precision;
+ this.scale = scale;
+ }
+
+ private static JdbcFieldInfo of(
+ String columnLabel, int columnType, String columnTypeName, boolean nullable) {
+ return new JdbcFieldInfo(columnLabel, columnType, columnTypeName, nullable, 0, 0);
+ }
+
+ private static JdbcFieldInfo of(String columnLabel, int columnType, boolean nullable) {
+ return new JdbcFieldInfo(columnLabel, columnType, null, nullable, 0, 0);
+ }
+
+ private static JdbcFieldInfo of(String columnLabel, int columnType) {
+ return new JdbcFieldInfo(columnLabel, columnType, null, false, 0, 0);
+ }
+
+ private static JdbcFieldInfo of(String columnLabel, int columnType, int precision) {
+ return new JdbcFieldInfo(columnLabel, columnType, null, false, precision, 0);
+ }
+
+ private static JdbcFieldInfo of(String columnLabel, int columnType, int precision, int scale) {
+ return new JdbcFieldInfo(columnLabel, columnType, null, false, precision, scale);
+ }
+ }
+
+ @Test
+ public void testSchemaFieldComparator() {
+ assertTrue(
+ SchemaUtil.compareSchemaField(
+ Schema.Field.of("name", Schema.FieldType.STRING),
+ Schema.Field.of("name", Schema.FieldType.STRING)));
+ assertFalse(
+ SchemaUtil.compareSchemaField(
+ Schema.Field.of("name", Schema.FieldType.STRING),
+ Schema.Field.of("anotherName", Schema.FieldType.STRING)));
+ assertFalse(
+ SchemaUtil.compareSchemaField(
+ Schema.Field.of("name", Schema.FieldType.STRING),
+ Schema.Field.of("name", Schema.FieldType.INT64)));
+ }
+
+ @Test
+ public void testSchemaFieldTypeComparator() {
+ assertTrue(SchemaUtil.compareSchemaFieldType(Schema.FieldType.STRING, Schema.FieldType.STRING));
+ assertFalse(SchemaUtil.compareSchemaFieldType(Schema.FieldType.STRING, Schema.FieldType.INT16));
+ assertTrue(
+ SchemaUtil.compareSchemaFieldType(
+ LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255),
+ LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255)));
+ assertFalse(
+ SchemaUtil.compareSchemaFieldType(
+ LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255),
+ LogicalTypes.fixedLengthBytes(JDBCType.BIT, 255)));
+ assertTrue(
+ SchemaUtil.compareSchemaFieldType(
+ Schema.FieldType.STRING, LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255)));
+ assertFalse(
+ SchemaUtil.compareSchemaFieldType(
+ Schema.FieldType.INT16, LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255)));
+ assertTrue(
+ SchemaUtil.compareSchemaFieldType(
+ LogicalTypes.variableLengthString(JDBCType.VARCHAR, 255), Schema.FieldType.STRING));
+ }
+}
diff --git a/sdks/java/io/jms/build.gradle b/sdks/java/io/jms/build.gradle
index fdb8c7e..8056106 100644
--- a/sdks/java/io/jms/build.gradle
+++ b/sdks/java/io/jms/build.gradle
@@ -17,18 +17,18 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.jms')
description = "Apache Beam :: SDKs :: Java :: IO :: JMS"
ext.summary = """IO to read and write to JMS (Java Messaging Service)
destinations (queues and topics)."""
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow "org.apache.geronimo.specs:geronimo-jms_1.1_spec:1.1.1"
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile "org.apache.geronimo.specs:geronimo-jms_1.1_spec:1.1.1"
testCompile library.java.activemq_broker
testCompile library.java.activemq_jaas
testCompile library.java.activemq_kahadb_store
@@ -37,5 +37,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
index 777eb9f..8aadf98 100644
--- a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
+++ b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.jms;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -53,7 +53,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java b/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
index b08a848..fb5534b 100644
--- a/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
+++ b/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
@@ -56,7 +56,7 @@
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
diff --git a/sdks/java/io/kafka/build.gradle b/sdks/java/io/kafka/build.gradle
index e74ae25..da8655e 100644
--- a/sdks/java/io/kafka/build.gradle
+++ b/sdks/java/io/kafka/build.gradle
@@ -17,28 +17,34 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.kafka')
+provideIntegrationTestingDependencies()
+enableJavaPerformanceTesting()
description = "Apache Beam :: SDKs :: Java :: IO :: Kafka"
ext.summary = "Library to read Kafka topics."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
// Get back to "provided" since 2.14
provided library.java.kafka_clients
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
- shadow "org.springframework:spring-expression:4.3.18.RELEASE"
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
+ compile "org.springframework:spring-expression:4.3.18.RELEASE"
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
+ testCompile project(":sdks:java:io:synthetic")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
// For testing Cross-language transforms
- testCompile project(path: ":runners:core-construction-java", configuration: "shadow")
+ testCompile project(":runners:core-construction-java")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
testCompile library.java.powermock
+ testCompile library.java.powermock_mockito
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
index 16fda1e..c0de080 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConsumerSpEL.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Collection;
import java.util.Map;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.OffsetAndTimestamp;
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelay.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelay.java
index ea7620b..7b5e1f7 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelay.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelay.java
@@ -20,7 +20,7 @@
import java.util.Optional;
import org.apache.beam.sdk.transforms.SerializableFunction;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.apache.kafka.common.TopicPartition;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
index 4a665dfa..e5f709b 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaCheckpointMark.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
/**
* Checkpoint for a {@link KafkaUnboundedReader}. Consists of Kafka topic name, partition id, and
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
index 41d69dc..6ea6baf 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaExactlyOnceSink.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
@@ -57,16 +57,16 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TimestampedValue.TimestampedValueCoder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.Cache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.RemovalCause;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.Cache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.RemovalCause;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index 8dee9bf..ca74ecb 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.service.AutoService;
import com.google.auto.value.AutoValue;
@@ -64,11 +64,10 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
@@ -359,6 +358,7 @@
abstract Builder<K, V> toBuilder();
+ @Experimental
@AutoValue.Builder
abstract static class Builder<K, V>
implements ExternalTransformBuilder<External.Configuration, PBegin, PCollection<KV<K, V>>> {
@@ -401,26 +401,22 @@
public PTransform<PBegin, PCollection<KV<K, V>>> buildExternal(
External.Configuration config) {
ImmutableList.Builder<String> listBuilder = ImmutableList.builder();
- for (byte[] topic : config.topics) {
- listBuilder.add(utf8String(topic));
+ for (String topic : config.topics) {
+ listBuilder.add(topic);
}
setTopics(listBuilder.build());
- String keyDeserializerClassName = utf8String(config.keyDeserializer);
- Class keyDeserializer = resolveClass(keyDeserializerClassName);
+ Class keyDeserializer = resolveClass(config.keyDeserializer);
setKeyDeserializer(keyDeserializer);
setKeyCoder(resolveCoder(keyDeserializer));
- String valueDeserializerClassName = utf8String(config.valueDeserializer);
- Class valueDeserializer = resolveClass(valueDeserializerClassName);
+ Class valueDeserializer = resolveClass(config.valueDeserializer);
setValueDeserializer(valueDeserializer);
setValueCoder(resolveCoder(valueDeserializer));
Map<String, Object> consumerConfig = new HashMap<>();
- for (KV<byte[], byte[]> kv : config.consumerConfig) {
- String key = utf8String(kv.getKey());
- String value = utf8String(kv.getValue());
- consumerConfig.put(key, value);
+ for (KV<String, String> kv : config.consumerConfig) {
+ consumerConfig.put(kv.getKey(), kv.getValue());
}
// Key and Value Deserializers always have to be in the config.
consumerConfig.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, keyDeserializer.getName());
@@ -464,6 +460,7 @@
* Exposes {@link KafkaIO.TypedWithoutMetadata} as an external transform for cross-language
* usage.
*/
+ @Experimental
@AutoService(ExternalTransformRegistrar.class)
public static class External implements ExternalTransformRegistrar {
@@ -478,24 +475,24 @@
public static class Configuration {
// All byte arrays are UTF-8 encoded strings
- private Iterable<KV<byte[], byte[]>> consumerConfig;
- private Iterable<byte[]> topics;
- private byte[] keyDeserializer;
- private byte[] valueDeserializer;
+ private Iterable<KV<String, String>> consumerConfig;
+ private Iterable<String> topics;
+ private String keyDeserializer;
+ private String valueDeserializer;
- public void setConsumerConfig(Iterable<KV<byte[], byte[]>> consumerConfig) {
+ public void setConsumerConfig(Iterable<KV<String, String>> consumerConfig) {
this.consumerConfig = consumerConfig;
}
- public void setTopics(Iterable<byte[]> topics) {
+ public void setTopics(Iterable<String> topics) {
this.topics = topics;
}
- public void setKeyDeserializer(byte[] keyDeserializer) {
+ public void setKeyDeserializer(String keyDeserializer) {
this.keyDeserializer = keyDeserializer;
}
- public void setValueDeserializer(byte[] valueDeserializer) {
+ public void setValueDeserializer(String valueDeserializer) {
this.valueDeserializer = valueDeserializer;
}
}
@@ -1350,6 +1347,7 @@
abstract Builder<K, V> toBuilder();
+ @Experimental
@AutoValue.Builder
abstract static class Builder<K, V>
implements ExternalTransformBuilder<External.Configuration, PCollection<KV<K, V>>, PDone> {
@@ -1362,24 +1360,21 @@
@Override
public PTransform<PCollection<KV<K, V>>, PDone> buildExternal(
External.Configuration configuration) {
- String topic = utf8String(configuration.topic);
- setTopic(topic);
+ setTopic(configuration.topic);
Map<String, Object> producerConfig = new HashMap<>();
- for (KV<byte[], byte[]> kv : configuration.producerConfig) {
- String key = utf8String(kv.getKey());
- String value = utf8String(kv.getValue());
- producerConfig.put(key, value);
+ for (KV<String, String> kv : configuration.producerConfig) {
+ producerConfig.put(kv.getKey(), kv.getValue());
}
- Class keySerializer = resolveClass(utf8String(configuration.keySerializer));
- Class valSerializer = resolveClass(utf8String(configuration.valueSerializer));
+ Class keySerializer = resolveClass(configuration.keySerializer);
+ Class valSerializer = resolveClass(configuration.valueSerializer);
WriteRecords<K, V> writeRecords =
KafkaIO.<K, V>writeRecords()
.withProducerConfigUpdates(producerConfig)
.withKeySerializer(keySerializer)
.withValueSerializer(valSerializer)
- .withTopic(topic);
+ .withTopic(configuration.topic);
setWriteRecordsTransform(writeRecords);
return build();
@@ -1387,6 +1382,7 @@
}
/** Exposes {@link KafkaIO.Write} as an external transform for cross-language usage. */
+ @Experimental
@AutoService(ExternalTransformRegistrar.class)
public static class External implements ExternalTransformRegistrar {
@@ -1401,24 +1397,24 @@
public static class Configuration {
// All byte arrays are UTF-8 encoded strings
- private Iterable<KV<byte[], byte[]>> producerConfig;
- private byte[] topic;
- private byte[] keySerializer;
- private byte[] valueSerializer;
+ private Iterable<KV<String, String>> producerConfig;
+ private String topic;
+ private String keySerializer;
+ private String valueSerializer;
- public void setProducerConfig(Iterable<KV<byte[], byte[]>> producerConfig) {
+ public void setProducerConfig(Iterable<KV<String, String>> producerConfig) {
this.producerConfig = producerConfig;
}
- public void setTopic(byte[] topic) {
+ public void setTopic(String topic) {
this.topic = topic;
}
- public void setKeySerializer(byte[] keySerializer) {
+ public void setKeySerializer(String keySerializer) {
this.keySerializer = keySerializer;
}
- public void setValueSerializer(byte[] valueSerializer) {
+ public void setValueSerializer(String valueSerializer) {
this.valueSerializer = valueSerializer;
}
}
@@ -1687,10 +1683,6 @@
String.format("Could not extract the Kafka Deserializer type from %s", deserializer));
}
- private static String utf8String(byte[] bytes) {
- return new String(bytes, Charsets.UTF_8);
- }
-
private static Class resolveClass(String className) {
try {
return Class.forName(className);
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
index b59d3fa..c2c56fb 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecord.java
@@ -20,7 +20,7 @@
import java.util.Arrays;
import javax.annotation.Nullable;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
import org.apache.kafka.common.header.Headers;
/**
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
index a0a7d94..25bd814 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
@@ -52,10 +52,10 @@
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.metrics.SourceMetrics;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.Closeables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.Closeables;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java
index fcc5605..7eb397f 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collections;
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.io.UnboundedSource;
import org.apache.beam.sdk.io.kafka.KafkaIO.Read;
import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.common.PartitionInfo;
import org.apache.kafka.common.TopicPartition;
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ProducerSpEL.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ProducerSpEL.java
index 801b3a5..0f622ea 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ProducerSpEL.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ProducerSpEL.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
diff --git a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/TimestampPolicyFactory.java b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/TimestampPolicyFactory.java
index b29d908..62b6092 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/TimestampPolicyFactory.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/TimestampPolicyFactory.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kafka;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.Serializable;
import java.util.Optional;
diff --git a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java
index 5fbe272..affff6c 100644
--- a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java
+++ b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/CustomTimestampPolicyWithLimitedDelayTest.java
@@ -26,7 +26,7 @@
import java.util.Optional;
import java.util.stream.Collectors;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.kafka.common.header.internals.RecordHeaders;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOExternalTest.java b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOExternalTest.java
index 4f2130e..a7b7f8a 100644
--- a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOExternalTest.java
+++ b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOExternalTest.java
@@ -32,27 +32,25 @@
import org.apache.beam.runners.core.construction.ReadTranslation;
import org.apache.beam.runners.core.construction.expansion.ExpansionService;
import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.Impulse;
import org.apache.beam.sdk.transforms.WithKeys;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.ByteString;
-import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Charsets;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.hamcrest.Matchers;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
-import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.reflect.Whitebox;
/** Tests for building {@link KafkaIO} externally via the ExpansionService. */
@RunWith(JUnit4.class)
@@ -76,7 +74,7 @@
"topics",
ExternalTransforms.ConfigValue.newBuilder()
.addCoderUrn("beam:coder:iterable:v1")
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(listAsBytes(topics)))
.build())
.putConfiguration(
@@ -84,20 +82,20 @@
ExternalTransforms.ConfigValue.newBuilder()
.addCoderUrn("beam:coder:iterable:v1")
.addCoderUrn("beam:coder:kv:v1")
- .addCoderUrn("beam:coder:bytes:v1")
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(mapAsBytes(consumerConfig)))
.build())
.putConfiguration(
"key_deserializer",
ExternalTransforms.ConfigValue.newBuilder()
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(encodeString(keyDeserializer)))
.build())
.putConfiguration(
"value_deserializer",
ExternalTransforms.ConfigValue.newBuilder()
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(encodeString(valueDeserializer)))
.build())
.build();
@@ -161,7 +159,7 @@
.putConfiguration(
"topic",
ExternalTransforms.ConfigValue.newBuilder()
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(encodeString(topic)))
.build())
.putConfiguration(
@@ -169,20 +167,20 @@
ExternalTransforms.ConfigValue.newBuilder()
.addCoderUrn("beam:coder:iterable:v1")
.addCoderUrn("beam:coder:kv:v1")
- .addCoderUrn("beam:coder:bytes:v1")
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(mapAsBytes(producerConfig)))
.build())
.putConfiguration(
"key_serializer",
ExternalTransforms.ConfigValue.newBuilder()
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(encodeString(keySerializer)))
.build())
.putConfiguration(
"value_serializer",
ExternalTransforms.ConfigValue.newBuilder()
- .addCoderUrn("beam:coder:bytes:v1")
+ .addCoderUrn("beam:coder:string_utf8:v1")
.setPayload(ByteString.copyFrom(encodeString(valueSerializer)))
.build())
.build();
@@ -248,37 +246,30 @@
}
private static byte[] listAsBytes(List<String> stringList) throws IOException {
- IterableCoder<byte[]> coder = IterableCoder.of(ByteArrayCoder.of());
- List<byte[]> bytesList =
- stringList.stream().map(KafkaIOExternalTest::utf8Bytes).collect(Collectors.toList());
+ IterableCoder<String> coder = IterableCoder.of(StringUtf8Coder.of());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
- coder.encode(bytesList, baos);
+ coder.encode(stringList, baos);
return baos.toByteArray();
}
private static byte[] mapAsBytes(Map<String, String> stringMap) throws IOException {
- IterableCoder<KV<byte[], byte[]>> coder =
- IterableCoder.of(KvCoder.of(ByteArrayCoder.of(), ByteArrayCoder.of()));
- List<KV<byte[], byte[]>> bytesList =
+ IterableCoder<KV<String, String>> coder =
+ IterableCoder.of(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()));
+ List<KV<String, String>> stringList =
stringMap.entrySet().stream()
- .map(kv -> KV.of(utf8Bytes(kv.getKey()), utf8Bytes(kv.getValue())))
+ .map(kv -> KV.of(kv.getKey(), kv.getValue()))
.collect(Collectors.toList());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
- coder.encode(bytesList, baos);
+ coder.encode(stringList, baos);
return baos.toByteArray();
}
private static byte[] encodeString(String str) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
- ByteArrayCoder.of().encode(utf8Bytes(str), baos);
+ StringUtf8Coder.of().encode(str, baos);
return baos.toByteArray();
}
- private static byte[] utf8Bytes(String str) {
- Preconditions.checkNotNull(str, "String must not be null.");
- return str.getBytes(Charsets.UTF_8);
- }
-
private static class TestStreamObserver<T> implements StreamObserver<T> {
private T result;
diff --git a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
new file mode 100644
index 0000000..92eb524
--- /dev/null
+++ b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.kafka;
+
+import static org.apache.beam.sdk.io.synthetic.SyntheticOptions.fromJsonString;
+
+import com.google.cloud.Timestamp;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.UUID;
+import java.util.function.BiFunction;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.common.HashingFn;
+import org.apache.beam.sdk.io.common.IOITHelper;
+import org.apache.beam.sdk.io.common.IOTestPipelineOptions;
+import org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource;
+import org.apache.beam.sdk.io.synthetic.SyntheticSourceOptions;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.options.Validation;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testutils.NamedTestResult;
+import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
+import org.apache.beam.sdk.testutils.metrics.MetricsReader;
+import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.joda.time.Duration;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * IO Integration test for {@link org.apache.beam.sdk.io.kafka.KafkaIO}.
+ *
+ * <p>{@see https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests} for
+ * more details.
+ *
+ * <p>NOTE: This test sets retention policy of the messages so that all messages are retained in the
+ * topic so that we could read them back after writing.
+ */
+@RunWith(JUnit4.class)
+public class KafkaIOIT {
+
+ private static final String READ_TIME_METRIC_NAME = "read_time";
+
+ private static final String WRITE_TIME_METRIC_NAME = "write_time";
+
+ private static final String RUN_TIME_METRIC_NAME = "run_time";
+
+ private static final String NAMESPACE = KafkaIOIT.class.getName();
+
+ private static final String TEST_ID = UUID.randomUUID().toString();
+
+ private static final String TIMESTAMP = Timestamp.now().toString();
+
+ /** Hash for 1000 uniformly distributed records with 10B keys and 90B values (100kB total). */
+ private static final String EXPECTED_HASHCODE = "4507649971ee7c51abbb446e65a5c660";
+
+ private static SyntheticSourceOptions sourceOptions;
+
+ private static Options options;
+
+ @Rule public TestPipeline writePipeline = TestPipeline.create();
+
+ @Rule public TestPipeline readPipeline = TestPipeline.create();
+
+ @BeforeClass
+ public static void setup() throws IOException {
+ options = IOITHelper.readIOTestPipelineOptions(Options.class);
+ sourceOptions = fromJsonString(options.getSourceOptions(), SyntheticSourceOptions.class);
+ }
+
+ @Test
+ public void testKafkaIOReadsAndWritesCorrectly() throws IOException {
+ writePipeline
+ .apply("Generate records", Read.from(new SyntheticBoundedSource(sourceOptions)))
+ .apply("Measure write time", ParDo.of(new TimeMonitor<>(NAMESPACE, WRITE_TIME_METRIC_NAME)))
+ .apply("Write to Kafka", writeToKafka());
+
+ PCollection<String> hashcode =
+ readPipeline
+ .apply("Read from Kafka", readFromKafka())
+ .apply(
+ "Measure read time", ParDo.of(new TimeMonitor<>(NAMESPACE, READ_TIME_METRIC_NAME)))
+ .apply("Map records to strings", MapElements.via(new MapKafkaRecordsToStrings()))
+ .apply("Calculate hashcode", Combine.globally(new HashingFn()).withoutDefaults());
+
+ PAssert.thatSingleton(hashcode).isEqualTo(EXPECTED_HASHCODE);
+
+ PipelineResult writeResult = writePipeline.run();
+ writeResult.waitUntilFinish();
+
+ PipelineResult readResult = readPipeline.run();
+ PipelineResult.State readState =
+ readResult.waitUntilFinish(Duration.standardSeconds(options.getReadTimeout()));
+ cancelIfNotTerminal(readResult, readState);
+
+ Set<NamedTestResult> metrics = readMetrics(writeResult, readResult);
+ IOITMetrics.publish(
+ TEST_ID, TIMESTAMP, options.getBigQueryDataset(), options.getBigQueryTable(), metrics);
+ }
+
+ private Set<NamedTestResult> readMetrics(PipelineResult writeResult, PipelineResult readResult) {
+ BiFunction<MetricsReader, String, NamedTestResult> supplier =
+ (reader, metricName) -> {
+ long start = reader.getStartTimeMetric(metricName);
+ long end = reader.getEndTimeMetric(metricName);
+ return NamedTestResult.create(TEST_ID, TIMESTAMP, metricName, (end - start) / 1e3);
+ };
+
+ NamedTestResult writeTime =
+ supplier.apply(new MetricsReader(writeResult, NAMESPACE), WRITE_TIME_METRIC_NAME);
+ NamedTestResult readTime =
+ supplier.apply(new MetricsReader(readResult, NAMESPACE), READ_TIME_METRIC_NAME);
+ NamedTestResult runTime =
+ NamedTestResult.create(
+ TEST_ID, TIMESTAMP, RUN_TIME_METRIC_NAME, writeTime.getValue() + readTime.getValue());
+
+ return ImmutableSet.of(readTime, writeTime, runTime);
+ }
+
+ private void cancelIfNotTerminal(PipelineResult readResult, PipelineResult.State readState)
+ throws IOException {
+ if (!readState.isTerminal()) {
+ readResult.cancel();
+ }
+ }
+
+ private KafkaIO.Write<byte[], byte[]> writeToKafka() {
+ return KafkaIO.<byte[], byte[]>write()
+ .withBootstrapServers(options.getKafkaBootstrapServerAddress())
+ .withTopic(options.getKafkaTopic())
+ .withKeySerializer(ByteArraySerializer.class)
+ .withValueSerializer(ByteArraySerializer.class);
+ }
+
+ private KafkaIO.Read<byte[], byte[]> readFromKafka() {
+ return KafkaIO.readBytes()
+ .withBootstrapServers(options.getKafkaBootstrapServerAddress())
+ .withConsumerConfigUpdates(ImmutableMap.of("auto.offset.reset", "earliest"))
+ .withTopic(options.getKafkaTopic())
+ .withMaxNumRecords(sourceOptions.numRecords);
+ }
+
+ /** Pipeline options specific for this test. */
+ public interface Options extends IOTestPipelineOptions, StreamingOptions {
+
+ @Description("Options for synthetic source.")
+ @Validation.Required
+ String getSourceOptions();
+
+ void setSourceOptions(String sourceOptions);
+
+ @Description("Kafka server address")
+ @Validation.Required
+ String getKafkaBootstrapServerAddress();
+
+ void setKafkaBootstrapServerAddress(String address);
+
+ @Description("Kafka topic")
+ @Validation.Required
+ String getKafkaTopic();
+
+ void setKafkaTopic(String topic);
+
+ @Description("Time to wait for the events to be processed by the read pipeline (in seconds)")
+ @Validation.Required
+ Integer getReadTimeout();
+
+ void setReadTimeout(Integer readTimeout);
+ }
+
+ private static class MapKafkaRecordsToStrings
+ extends SimpleFunction<KafkaRecord<byte[], byte[]>, String> {
+ @Override
+ public String apply(KafkaRecord<byte[], byte[]> input) {
+ String key = Arrays.toString(input.getKV().getKey());
+ String value = Arrays.toString(input.getKV().getValue());
+ return String.format("%s %s", key, value);
+ }
+ }
+}
diff --git a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
index b1fe273..43b5cc1 100644
--- a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
+++ b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
@@ -95,10 +95,10 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
diff --git a/sdks/java/io/kinesis/build.gradle b/sdks/java/io/kinesis/build.gradle
index da930e5..a73c770 100644
--- a/sdks/java/io/kinesis/build.gradle
+++ b/sdks/java/io/kinesis/build.gradle
@@ -17,15 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(
- // Override the default shading configuration to exclude everything since
- // AWS-KPL library has hard dependency on guava.
- shadowClosure: {
- dependencies {
- exclude(dependency(".*:.*"))
- }
- }
-)
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.kinesis')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -38,24 +30,25 @@
}
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.guava
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow library.java.jackson_dataformat_cbor
- shadow library.java.aws_java_sdk_cloudwatch
- shadow library.java.aws_java_sdk_core
- shadow library.java.aws_java_sdk_kinesis
- shadow "com.amazonaws:amazon-kinesis-client:1.10.0"
- shadow "com.amazonaws:amazon-kinesis-producer:0.12.11"
- shadow "commons-lang:commons-lang:2.6"
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile library.java.jackson_dataformat_cbor
+ compile library.java.aws_java_sdk_cloudwatch
+ compile library.java.aws_java_sdk_core
+ compile library.java.aws_java_sdk_kinesis
+ compile "com.amazonaws:amazon-kinesis-client:1.10.0"
+ compile "com.amazonaws:amazon-kinesis-producer:0.13.1"
+ compile "commons-lang:commons-lang:2.6"
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.junit
testCompile library.java.mockito_core
testCompile library.java.guava_testlib
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.powermock
+ testCompile library.java.powermock_mockito
testCompile "org.assertj:assertj-core:3.11.1"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/BasicKinesisProvider.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/BasicKinesisProvider.java
index cfc4508..6c05c3c 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/BasicKinesisProvider.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/BasicKinesisProvider.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGenerator.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGenerator.java
index 362f5d3..06a9669 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGenerator.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGenerator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.services.kinesis.model.Shard;
import java.util.Set;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
index 04c8b86..9ea3ff5 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java
@@ -17,26 +17,28 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.cloudwatch.AmazonCloudWatch;
import com.amazonaws.services.kinesis.AmazonKinesis;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
-import com.amazonaws.services.kinesis.model.DescribeStreamResult;
import com.amazonaws.services.kinesis.producer.Attempt;
import com.amazonaws.services.kinesis.producer.IKinesisProducer;
import com.amazonaws.services.kinesis.producer.KinesisProducerConfiguration;
import com.amazonaws.services.kinesis.producer.UserRecordFailedException;
import com.amazonaws.services.kinesis.producer.UserRecordResult;
import com.google.auto.value.AutoValue;
-import com.google.common.util.concurrent.FutureCallback;
-import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import java.io.IOException;
+import java.io.ObjectInputStream;
import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
import java.util.concurrent.LinkedBlockingDeque;
import javax.annotation.Nullable;
import org.apache.beam.sdk.annotations.Experimental;
@@ -47,7 +49,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -581,31 +583,43 @@
getPartitionKey() == null || (getPartitioner() == null),
"only one of either withPartitionKey() or withPartitioner() is possible");
checkArgument(getAWSClientsProvider() != null, "withAWSClientsProvider() is required");
+
input.apply(ParDo.of(new KinesisWriterFn(this)));
return PDone.in(input.getPipeline());
}
private static class KinesisWriterFn extends DoFn<byte[], Void> {
- private static final int MAX_NUM_RECORDS = 100 * 1000;
private static final int MAX_NUM_FAILURES = 10;
private final KinesisIO.Write spec;
- private transient IKinesisProducer producer;
+ private static transient IKinesisProducer producer;
private transient KinesisPartitioner partitioner;
private transient LinkedBlockingDeque<KinesisWriteException> failures;
+ private transient List<Future<UserRecordResult>> putFutures;
- public KinesisWriterFn(KinesisIO.Write spec) {
+ KinesisWriterFn(KinesisIO.Write spec) {
this.spec = spec;
+ initKinesisProducer();
}
@Setup
- public void setup() throws Exception {
- checkArgument(
- streamExists(spec.getAWSClientsProvider().getKinesisClient(), spec.getStreamName()),
- "Stream %s does not exist",
- spec.getStreamName());
+ public void setup() {
+ // Use custom partitioner if it exists
+ if (spec.getPartitioner() != null) {
+ partitioner = spec.getPartitioner();
+ }
+ }
+ @StartBundle
+ public void startBundle() {
+ putFutures = Collections.synchronizedList(new ArrayList<>());
+ /** Keep only the first {@link MAX_NUM_FAILURES} occurred exceptions */
+ failures = new LinkedBlockingDeque<>(MAX_NUM_FAILURES);
+ initKinesisProducer();
+ }
+
+ private synchronized void initKinesisProducer() {
// Init producer config
Properties props = spec.getProducerProperties();
if (props == null) {
@@ -618,14 +632,14 @@
config.setCredentialsRefreshDelay(100);
// Init Kinesis producer
- producer = spec.getAWSClientsProvider().createKinesisProducer(config);
- // Use custom partitioner if it exists
- if (spec.getPartitioner() != null) {
- partitioner = spec.getPartitioner();
+ if (producer == null) {
+ producer = spec.getAWSClientsProvider().createKinesisProducer(config);
}
+ }
- /** Keep only the first {@link MAX_NUM_FAILURES} occurred exceptions */
- failures = new LinkedBlockingDeque<>(MAX_NUM_FAILURES);
+ private void readObject(ObjectInputStream is) throws IOException, ClassNotFoundException {
+ is.defaultReadObject();
+ initKinesisProducer();
}
/**
@@ -643,13 +657,7 @@
* the KPL</a>
*/
@ProcessElement
- public void processElement(ProcessContext c) throws Exception {
- checkForFailures();
-
- // Need to avoid keeping too many futures in producer's map to prevent OOM.
- // In usual case, it should exit immediately.
- flush(MAX_NUM_RECORDS);
-
+ public void processElement(ProcessContext c) {
ByteBuffer data = ByteBuffer.wrap(c.element());
String partitionKey = spec.getPartitionKey();
String explicitHashKey = null;
@@ -662,73 +670,77 @@
ListenableFuture<UserRecordResult> f =
producer.addUserRecord(spec.getStreamName(), partitionKey, explicitHashKey, data);
- Futures.addCallback(f, new UserRecordResultFutureCallback());
+ putFutures.add(f);
}
@FinishBundle
public void finishBundle() throws Exception {
- // Flush all outstanding records, blocking call
- flushAll();
-
- checkForFailures();
- }
-
- @Teardown
- public void tearDown() throws Exception {
- if (producer != null) {
- producer.destroy();
- producer = null;
- }
+ flushBundle();
}
/**
- * Flush outstanding records until the total number will be less than required or the number
- * of retries will be exhausted. The retry timeout starts from 1 second and it doubles on
- * every iteration.
+ * Flush outstanding records until the total number of failed records will be less than 0 or
+ * the number of retries will be exhausted. The retry timeout starts from 1 second and it
+ * doubles on every iteration.
*/
- private void flush(int numMax) throws InterruptedException, IOException {
+ private void flushBundle() throws InterruptedException, ExecutionException, IOException {
int retries = spec.getRetries();
- int numOutstandingRecords = producer.getOutstandingRecordsCount();
+ int numFailedRecords;
int retryTimeout = 1000; // initial timeout, 1 sec
+ String message = "";
- while (numOutstandingRecords > numMax && retries-- > 0) {
+ do {
+ numFailedRecords = 0;
producer.flush();
+
+ // Wait for puts to finish and check the results
+ for (Future<UserRecordResult> f : putFutures) {
+ UserRecordResult result = f.get(); // this does block
+ if (!result.isSuccessful()) {
+ numFailedRecords++;
+ }
+ }
+
// wait until outstanding records will be flushed
Thread.sleep(retryTimeout);
- numOutstandingRecords = producer.getOutstandingRecordsCount();
retryTimeout *= 2; // exponential backoff
- }
+ } while (numFailedRecords > 0 && retries-- > 0);
- if (numOutstandingRecords > numMax) {
- String message =
+ if (numFailedRecords > 0) {
+ for (Future<UserRecordResult> f : putFutures) {
+ UserRecordResult result = f.get();
+ if (!result.isSuccessful()) {
+ failures.offer(
+ new KinesisWriteException(
+ "Put record was not successful.", new UserRecordFailedException(result)));
+ }
+ }
+
+ message =
String.format(
- "After [%d] retries, number of outstanding records [%d] is still greater than "
- + "required [%d].",
- spec.getRetries(), numOutstandingRecords, numMax);
+ "After [%d] retries, number of failed records [%d] is still greater than 0",
+ spec.getRetries(), numFailedRecords);
LOG.error(message);
- throw new IOException(message);
}
- }
- private void flushAll() throws InterruptedException, IOException {
- flush(0);
+ checkForFailures(message);
}
/** If any write has asynchronously failed, fail the bundle with a useful error. */
- private void checkForFailures() throws IOException {
- // Note that this function is never called by multiple threads and is the only place that
- // we remove from failures, so this code is safe.
+ private void checkForFailures(String message) throws IOException {
if (failures.isEmpty()) {
return;
}
StringBuilder logEntry = new StringBuilder();
+ logEntry.append(message).append(System.lineSeparator());
+
int i = 0;
while (!failures.isEmpty()) {
i++;
KinesisWriteException exc = failures.remove();
- logEntry.append("\n").append(exc.getMessage());
+ logEntry.append(System.lineSeparator()).append(exc.getMessage());
Throwable cause = exc.getCause();
if (cause != null) {
logEntry.append(": ").append(cause.getMessage());
@@ -738,59 +750,34 @@
((UserRecordFailedException) cause).getResult().getAttempts();
for (Attempt attempt : attempts) {
if (attempt.getErrorMessage() != null) {
- logEntry.append("\n").append(attempt.getErrorMessage());
+ logEntry.append(System.lineSeparator()).append(attempt.getErrorMessage());
}
}
}
}
}
- failures.clear();
- String message =
+ String errorMessage =
String.format(
"Some errors occurred writing to Kinesis. First %d errors: %s",
i, logEntry.toString());
- throw new IOException(message);
+ throw new IOException(errorMessage);
}
- private class UserRecordResultFutureCallback implements FutureCallback<UserRecordResult> {
-
- @Override
- public void onFailure(Throwable cause) {
- failures.offer(new KinesisWriteException(cause));
+ @Teardown
+ public void teardown() throws Exception {
+ if (producer != null && producer.getOutstandingRecordsCount() > 0) {
+ producer.flushSync();
}
-
- @Override
- public void onSuccess(UserRecordResult result) {
- if (!result.isSuccessful()) {
- failures.offer(
- new KinesisWriteException(
- "Put record was not successful.", new UserRecordFailedException(result)));
- }
- }
+ producer = null;
}
}
}
- private static boolean streamExists(AmazonKinesis client, String streamName) {
- try {
- DescribeStreamResult describeStreamResult = client.describeStream(streamName);
- return (describeStreamResult != null
- && describeStreamResult.getSdkHttpMetadata().getHttpStatusCode() == 200);
- } catch (Exception e) {
- LOG.warn("Error checking whether stream {} exists.", streamName, e);
- }
- return false;
- }
-
/** An exception that puts information about the failed record. */
static class KinesisWriteException extends IOException {
KinesisWriteException(String message, Throwable cause) {
super(message, cause);
}
-
- KinesisWriteException(Throwable cause) {
- super(cause);
- }
}
}
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
index 58b0e37..db73b99 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
@@ -17,11 +17,12 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.IOException;
import java.util.NoSuchElementException;
import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
@@ -141,13 +142,19 @@
}
/**
- * Returns total size of all records that remain in Kinesis stream after current watermark. When
- * currently processed record is not further behind than {@link #upToDateThreshold} then this
- * method returns 0.
+ * Returns total size of all records that remain in Kinesis stream after current watermark. If the
+ * watermark was not already set then it returns {@link
+ * UnboundedSource.UnboundedReader#BACKLOG_UNKNOWN}. When currently processed record is not
+ * further behind than {@link #upToDateThreshold} then this method returns 0.
*/
@Override
public long getTotalBacklogBytes() {
Instant watermark = getWatermark();
+
+ if (watermark.equals(BoundedWindow.TIMESTAMP_MIN_VALUE)) {
+ return UnboundedSource.UnboundedReader.BACKLOG_UNKNOWN;
+ }
+
if (watermark.plus(upToDateThreshold).isAfterNow()) {
return 0L;
}
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpoint.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpoint.java
index 94e08f3..5829e35 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpoint.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpoint.java
@@ -17,15 +17,15 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.newArrayList;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.partition;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.partition;
import java.io.IOException;
import java.io.Serializable;
import java.util.Iterator;
import java.util.List;
import org.apache.beam.sdk.io.UnboundedSource;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
/**
* Checkpoint representing a total progress in a set of shards in single stream. The set of shards
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java
index f0f4f5d..7785cb7 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.newArrayList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
import java.util.List;
import org.apache.beam.sdk.coders.Coder;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/RecordFilter.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/RecordFilter.java
index 8d673c5..ce8218e 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/RecordFilter.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/RecordFilter.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.newArrayList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
import java.util.List;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardCheckpoint.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardCheckpoint.java
index 032c1ca..0f5acd7 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardCheckpoint.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardCheckpoint.java
@@ -20,8 +20,8 @@
import static com.amazonaws.services.kinesis.model.ShardIteratorType.AFTER_SEQUENCE_NUMBER;
import static com.amazonaws.services.kinesis.model.ShardIteratorType.AT_SEQUENCE_NUMBER;
import static com.amazonaws.services.kinesis.model.ShardIteratorType.AT_TIMESTAMP;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.services.kinesis.clientlibrary.types.ExtendedSequenceNumber;
import com.amazonaws.services.kinesis.model.Record;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
index 8a71ce3..71a12fc 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardReadersPool.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.util.Comparator;
import java.util.List;
@@ -33,8 +33,8 @@
import java.util.concurrent.atomic.AtomicReference;
import java.util.stream.Collectors;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIterator.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIterator.java
index 5043ef0..ef219dc 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIterator.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIterator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import com.amazonaws.services.kinesis.model.ExpiredIteratorException;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
index 9903f0f..7cf4cd5 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
@@ -41,7 +41,7 @@
import java.util.Date;
import java.util.List;
import java.util.concurrent.Callable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Instant;
import org.joda.time.Minutes;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java
index 28c5c45..30bae1f 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPoint.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import com.amazonaws.services.kinesis.model.ShardIteratorType;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinder.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinder.java
index 5428568..d9e00a8 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinder.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinder.java
@@ -24,7 +24,7 @@
import java.util.List;
import java.util.Objects;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StaticCheckpointGenerator.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StaticCheckpointGenerator.java
index ea2a87f..642e6e9 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StaticCheckpointGenerator.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/StaticCheckpointGenerator.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
/** Always returns the same instance of checkpoint. */
class StaticCheckpointGenerator implements CheckpointGenerator {
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkParameters.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkParameters.java
index 82e9e2e..704e1ff 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkParameters.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkParameters.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
diff --git a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicyFactory.java b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicyFactory.java
index 68920e9..f2d47b4 100644
--- a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicyFactory.java
+++ b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicyFactory.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.io.kinesis;
import java.io.Serializable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
index dc3fdbe..6f0184d 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java
@@ -19,7 +19,7 @@
import static java.lang.Integer.parseInt;
import static java.lang.Math.min;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.transform;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.transform;
import static org.apache.commons.lang.builder.HashCodeBuilder.reflectionHashCode;
import com.amazonaws.AmazonWebServiceRequest;
@@ -96,7 +96,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.commons.lang.builder.EqualsBuilder;
import org.joda.time.Instant;
import org.mockito.Mockito;
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGeneratorTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGeneratorTest.java
index 917b5ce..352caa5 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGeneratorTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/DynamicCheckpointGeneratorTest.java
@@ -18,12 +18,12 @@
package org.apache.beam.sdk.io.kinesis;
import static org.assertj.core.api.Assertions.assertThat;
-import static org.mockito.BDDMockito.given;
+import static org.mockito.Mockito.when;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import com.amazonaws.services.kinesis.model.Shard;
import java.util.Set;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
@@ -39,15 +39,14 @@
@Test
public void shouldMapAllShardsToCheckpoints() throws Exception {
- given(shard1.getShardId()).willReturn("shard-01");
- given(shard2.getShardId()).willReturn("shard-02");
- given(shard3.getShardId()).willReturn("shard-03");
+ when(shard1.getShardId()).thenReturn("shard-01");
+ when(shard2.getShardId()).thenReturn("shard-02");
+ when(shard3.getShardId()).thenReturn("shard-03");
Set<Shard> shards = Sets.newHashSet(shard1, shard2, shard3);
StartingPoint startingPoint = new StartingPoint(InitialPositionInStream.LATEST);
- given(
- startingPointShardsFinder.findShardsAtStartingPoint(
- kinesisClient, "stream", startingPoint))
- .willReturn(shards);
+ when(startingPointShardsFinder.findShardsAtStartingPoint(
+ kinesisClient, "stream", startingPoint))
+ .thenReturn(shards);
DynamicCheckpointGenerator underTest =
new DynamicCheckpointGenerator("stream", startingPoint, startingPointShardsFinder);
@@ -58,16 +57,15 @@
@Test
public void shouldMapAllValidShardsToCheckpoints() throws Exception {
- given(shard1.getShardId()).willReturn("shard-01");
- given(shard2.getShardId()).willReturn("shard-02");
- given(shard3.getShardId()).willReturn("shard-03");
+ when(shard1.getShardId()).thenReturn("shard-01");
+ when(shard2.getShardId()).thenReturn("shard-02");
+ when(shard3.getShardId()).thenReturn("shard-03");
String streamName = "stream";
Set<Shard> shards = Sets.newHashSet(shard1, shard2);
StartingPoint startingPoint = new StartingPoint(InitialPositionInStream.LATEST);
- given(
- startingPointShardsFinder.findShardsAtStartingPoint(
- kinesisClient, "stream", startingPoint))
- .willReturn(shards);
+ when(startingPointShardsFinder.findShardsAtStartingPoint(
+ kinesisClient, "stream", startingPoint))
+ .thenReturn(shards);
DynamicCheckpointGenerator underTest =
new DynamicCheckpointGenerator(streamName, startingPoint, startingPointShardsFinder);
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java
index 01004c3..5f3a003 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java
@@ -17,18 +17,19 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.newArrayList;
-
import com.amazonaws.regions.Regions;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import java.io.Serializable;
import java.nio.charset.StandardCharsets;
-import java.util.List;
import java.util.Random;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.common.HashingFn;
+import org.apache.beam.sdk.io.common.TestRow;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
@@ -41,33 +42,43 @@
import org.junit.runners.JUnit4;
/**
- * Integration test, that writes and reads data to and from real Kinesis. You need to provide all
- * {@link KinesisTestOptions} in order to run this.
+ * Integration test, that writes and reads data to and from real Kinesis. You need to provide {@link
+ * KinesisTestOptions} in order to run this.
*/
@RunWith(JUnit4.class)
public class KinesisIOIT implements Serializable {
- public static final int NUM_RECORDS = 1000;
- public static final int NUM_SHARDS = 2;
+ private static int numberOfShards;
+ private static int numberOfRows;
- @Rule public final transient TestPipeline p = TestPipeline.create();
- @Rule public final transient TestPipeline p2 = TestPipeline.create();
+ @Rule public TestPipeline pipelineWrite = TestPipeline.create();
+ @Rule public TestPipeline pipelineRead = TestPipeline.create();
private static KinesisTestOptions options;
+ private static final Instant now = Instant.now();
@BeforeClass
public static void setup() {
PipelineOptionsFactory.register(KinesisTestOptions.class);
options = TestPipeline.testingPipelineOptions().as(KinesisTestOptions.class);
+ numberOfShards = options.getNumberOfShards();
+ numberOfRows = options.getNumberOfRecords();
}
+ /** Test which write and then read data for a Kinesis stream. */
@Test
- public void testWriteThenRead() throws Exception {
- Instant now = Instant.now();
- List<byte[]> inputData = prepareData();
+ public void testWriteThenRead() {
+ runWrite();
+ runRead();
+ }
- // Write data into stream
- p.apply(Create.of(inputData))
+ /** Write test dataset into Kinesis stream. */
+ private void runWrite() {
+ pipelineWrite
+ .apply("Generate Sequence", GenerateSequence.from(0).to((long) numberOfRows))
+ .apply("Prepare TestRows", ParDo.of(new TestRow.DeterministicallyConstructTestRowFn()))
+ .apply("Prepare Kinesis input records", ParDo.of(new ConvertToBytes()))
.apply(
+ "Write to Kinesis",
KinesisIO.write()
.withStreamName(options.getAwsKinesisStream())
.withPartitioner(new RandomPartitioner())
@@ -75,51 +86,62 @@
options.getAwsAccessKey(),
options.getAwsSecretKey(),
Regions.fromName(options.getAwsKinesisRegion())));
- p.run().waitUntilFinish();
- // Read new data from stream that was just written before
- PCollection<byte[]> output =
- p2.apply(
- KinesisIO.read()
- .withStreamName(options.getAwsKinesisStream())
- .withAWSClientsProvider(
- options.getAwsAccessKey(),
- options.getAwsSecretKey(),
- Regions.fromName(options.getAwsKinesisRegion()))
- .withMaxNumRecords(inputData.size())
- // to prevent endless running in case of error
- .withMaxReadTime(Duration.standardMinutes(5))
- .withInitialPositionInStream(InitialPositionInStream.AT_TIMESTAMP)
- .withInitialTimestampInStream(now)
- .withRequestRecordsLimit(1000))
- .apply(
- ParDo.of(
- new DoFn<KinesisRecord, byte[]>() {
-
- @ProcessElement
- public void processElement(ProcessContext c) {
- KinesisRecord record = c.element();
- byte[] data = record.getData().array();
- c.output(data);
- }
- }));
- PAssert.that(output).containsInAnyOrder(inputData);
- p2.run().waitUntilFinish();
+ pipelineWrite.run().waitUntilFinish();
}
- private List<byte[]> prepareData() {
- List<byte[]> data = newArrayList();
- for (int i = 0; i < NUM_RECORDS; i++) {
- data.add(String.valueOf(i).getBytes(StandardCharsets.UTF_8));
+ /** Read test dataset from Kinesis stream. */
+ private void runRead() {
+ PCollection<KinesisRecord> output =
+ pipelineRead.apply(
+ KinesisIO.read()
+ .withStreamName(options.getAwsKinesisStream())
+ .withAWSClientsProvider(
+ options.getAwsAccessKey(),
+ options.getAwsSecretKey(),
+ Regions.fromName(options.getAwsKinesisRegion()))
+ .withMaxNumRecords(numberOfRows)
+ // to prevent endless running in case of error
+ .withMaxReadTime(Duration.standardMinutes(10))
+ .withInitialPositionInStream(InitialPositionInStream.AT_TIMESTAMP)
+ .withInitialTimestampInStream(now)
+ .withRequestRecordsLimit(1000));
+
+ PAssert.thatSingleton(output.apply("Count All", Count.globally()))
+ .isEqualTo((long) numberOfRows);
+
+ PCollection<String> consolidatedHashcode =
+ output
+ .apply(ParDo.of(new ExtractDataValues()))
+ .apply("Hash row contents", Combine.globally(new HashingFn()).withoutDefaults());
+
+ PAssert.that(consolidatedHashcode)
+ .containsInAnyOrder(TestRow.getExpectedHashForRowCount(numberOfRows));
+
+ pipelineRead.run().waitUntilFinish();
+ }
+
+ /** Produces test rows. */
+ private static class ConvertToBytes extends DoFn<TestRow, byte[]> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ c.output(String.valueOf(c.element().name()).getBytes(StandardCharsets.UTF_8));
}
- return data;
+ }
+
+ /** Read rows from Table. */
+ private static class ExtractDataValues extends DoFn<KinesisRecord, String> {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ c.output(new String(c.element().getDataAsBytes(), StandardCharsets.UTF_8));
+ }
}
private static final class RandomPartitioner implements KinesisPartitioner {
@Override
public String getPartitionKey(byte[] value) {
Random rand = new Random();
- int n = rand.nextInt(NUM_SHARDS) + 1;
+ int n = rand.nextInt(numberOfShards) + 1;
return String.valueOf(n);
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockReadTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockReadTest.java
index 25d025b..ce30e2a 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockReadTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockReadTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.newArrayList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import java.util.List;
@@ -26,7 +26,7 @@
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.DateTime;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockWriteTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockWriteTest.java
index 7ae9cc9..f81065a 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockWriteTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockWriteTest.java
@@ -37,8 +37,8 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
@@ -131,35 +131,19 @@
}
@Test
- public void testNotExistedStream() {
- Iterable<byte[]> data = ImmutableList.of("1".getBytes(StandardCharsets.UTF_8));
- p.apply(Create.of(data))
- .apply(
- KinesisIO.write()
- .withStreamName(STREAM)
- .withPartitionKey(PARTITION_KEY)
- .withAWSClientsProvider(new FakeKinesisProvider(false)));
-
- thrown.expect(RuntimeException.class);
- p.run().waitUntilFinish();
- }
-
- @Test
public void testSetInvalidProperty() {
Properties properties = new Properties();
properties.setProperty("KinesisPort", "qwe");
- Iterable<byte[]> data = ImmutableList.of("1".getBytes(StandardCharsets.UTF_8));
- p.apply(Create.of(data))
- .apply(
- KinesisIO.write()
- .withStreamName(STREAM)
- .withPartitionKey(PARTITION_KEY)
- .withAWSClientsProvider(new FakeKinesisProvider())
- .withProducerProperties(properties));
+ KinesisIO.Write write =
+ KinesisIO.write()
+ .withStreamName(STREAM)
+ .withPartitionKey(PARTITION_KEY)
+ .withAWSClientsProvider(new FakeKinesisProvider())
+ .withProducerProperties(properties);
- thrown.expect(RuntimeException.class);
- p.run().waitUntilFinish();
+ thrown.expect(IllegalArgumentException.class);
+ write.expand(null);
}
@Test
@@ -197,7 +181,7 @@
.withStreamName(STREAM)
.withPartitionKey(PARTITION_KEY)
.withAWSClientsProvider(new FakeKinesisProvider().setFailedFlush(true))
- .withRetries(1));
+ .withRetries(2));
thrown.expect(RuntimeException.class);
p.run().waitUntilFinish();
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
index f9f03ce..17c8c1d 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
@@ -26,8 +26,10 @@
import com.google.common.util.concurrent.SettableFuture;
import java.nio.ByteBuffer;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import java.util.concurrent.ExecutionException;
+import java.util.concurrent.atomic.AtomicInteger;
import org.joda.time.DateTime;
/** Simple mock implementation of {@link IKinesisProducer} for testing. */
@@ -35,31 +37,39 @@
private boolean isFailedFlush = false;
- private List<UserRecord> addedRecords = new ArrayList<>();
+ private List<UserRecord> addedRecords = Collections.synchronizedList(new ArrayList<>());
private KinesisServiceMock kinesisService = KinesisServiceMock.getInstance();
+ private AtomicInteger seqNumber = new AtomicInteger(0);
+
public KinesisProducerMock() {}
public KinesisProducerMock(KinesisProducerConfiguration config, boolean isFailedFlush) {
this.isFailedFlush = isFailedFlush;
+ this.seqNumber.set(0);
}
@Override
public ListenableFuture<UserRecordResult> addUserRecord(
String stream, String partitionKey, ByteBuffer data) {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
public ListenableFuture<UserRecordResult> addUserRecord(UserRecord userRecord) {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
- public ListenableFuture<UserRecordResult> addUserRecord(
+ public synchronized ListenableFuture<UserRecordResult> addUserRecord(
String stream, String partitionKey, String explicitHashKey, ByteBuffer data) {
+ seqNumber.incrementAndGet();
SettableFuture<UserRecordResult> f = SettableFuture.create();
+ f.set(
+ new UserRecordResult(
+ new ArrayList<>(), String.valueOf(seqNumber.get()), explicitHashKey, !isFailedFlush));
+
if (kinesisService.getExistedStream().equals(stream)) {
addedRecords.add(new UserRecord(stream, partitionKey, explicitHashKey, data));
}
@@ -74,24 +84,24 @@
@Override
public List<Metric> getMetrics(String metricName, int windowSeconds)
throws InterruptedException, ExecutionException {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
public List<Metric> getMetrics(String metricName)
throws InterruptedException, ExecutionException {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
public List<Metric> getMetrics() throws InterruptedException, ExecutionException {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
public List<Metric> getMetrics(int windowSeconds)
throws InterruptedException, ExecutionException {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
@@ -99,16 +109,11 @@
@Override
public void flush(String stream) {
- throw new RuntimeException("Not implemented");
+ throw new UnsupportedOperationException("Not implemented");
}
@Override
- public void flush() {
- if (isFailedFlush) {
- // don't flush
- return;
- }
-
+ public synchronized void flush() {
DateTime arrival = DateTime.now();
for (int i = 0; i < addedRecords.size(); i++) {
UserRecord record = addedRecords.get(i);
@@ -120,8 +125,6 @@
@Override
public synchronized void flushSync() {
- if (getOutstandingRecordsCount() > 0) {
- flush();
- }
+ flush();
}
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpointTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpointTest.java
index ecead5b..1653daf 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpointTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderCheckpointTest.java
@@ -22,7 +22,7 @@
import java.util.Iterator;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java
index 635e20a..37528ef 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderTest.java
@@ -27,16 +27,18 @@
import java.io.IOException;
import java.util.NoSuchElementException;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
-import org.mockito.runners.MockitoJUnitRunner;
+import org.mockito.junit.MockitoJUnitRunner;
/** Tests {@link KinesisReader}. */
-@RunWith(MockitoJUnitRunner.class)
+@RunWith(MockitoJUnitRunner.Silent.class)
public class KinesisReaderTest {
@Mock private SimplifiedKinesisClient kinesis;
@@ -163,4 +165,19 @@
assertThat(backlogCachingReader.getTotalBacklogBytes()).isEqualTo(10);
assertThat(backlogCachingReader.getTotalBacklogBytes()).isEqualTo(10);
}
+
+ @Test
+ public void getTotalBacklogBytesShouldReturnBacklogUnknown()
+ throws IOException, TransientKinesisException {
+ reader.start();
+ when(kinesisSource.getStreamName()).thenReturn("stream1");
+ when(reader.getWatermark())
+ .thenReturn(BoundedWindow.TIMESTAMP_MIN_VALUE)
+ .thenReturn(Instant.now().minus(Duration.standardMinutes(1)));
+ when(kinesis.getBacklogBytes(eq("stream1"), any(Instant.class))).thenReturn(10L);
+
+ assertThat(reader.getTotalBacklogBytes())
+ .isEqualTo(UnboundedSource.UnboundedReader.BACKLOG_UNKNOWN);
+ assertThat(reader.getTotalBacklogBytes()).isEqualTo(10);
+ }
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisServiceMock.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisServiceMock.java
index 7acdb49..0508b05 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisServiceMock.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisServiceMock.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists.newArrayList;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisTestOptions.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisTestOptions.java
index 30f1f86..185f953 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisTestOptions.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisTestOptions.java
@@ -47,4 +47,16 @@
String getAwsAccessKey();
void setAwsAccessKey(String value);
+
+ @Description("Number of shards of stream")
+ @Default.Integer(2)
+ Integer getNumberOfShards();
+
+ void setNumberOfShards(Integer count);
+
+ @Description("Number of records that will be written and read by the test")
+ @Default.Integer(1000)
+ Integer getNumberOfRecords();
+
+ void setNumberOfRecords(Integer count);
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/RecordFilterTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/RecordFilterTest.java
index 06c0ab5..e17fa86 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/RecordFilterTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/RecordFilterTest.java
@@ -17,11 +17,11 @@
*/
package org.apache.beam.sdk.io.kinesis;
-import static org.mockito.BDDMockito.given;
+import static org.mockito.Mockito.when;
import java.util.Collections;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.assertj.core.api.Assertions;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -37,11 +37,11 @@
@Test
public void shouldFilterOutRecordsBeforeOrAtCheckpoint() {
- given(checkpoint.isBeforeOrAt(record1)).willReturn(false);
- given(checkpoint.isBeforeOrAt(record2)).willReturn(true);
- given(checkpoint.isBeforeOrAt(record3)).willReturn(true);
- given(checkpoint.isBeforeOrAt(record4)).willReturn(false);
- given(checkpoint.isBeforeOrAt(record5)).willReturn(true);
+ when(checkpoint.isBeforeOrAt(record1)).thenReturn(false);
+ when(checkpoint.isBeforeOrAt(record2)).thenReturn(true);
+ when(checkpoint.isBeforeOrAt(record3)).thenReturn(true);
+ when(checkpoint.isBeforeOrAt(record4)).thenReturn(false);
+ when(checkpoint.isBeforeOrAt(record5)).thenReturn(true);
List<KinesisRecord> records = Lists.newArrayList(record1, record2, record3, record4, record5);
RecordFilter underTest = new RecordFilter();
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardCheckpointTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardCheckpointTest.java
index 1f67cb0..10de1df 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardCheckpointTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardCheckpointTest.java
@@ -23,7 +23,6 @@
import static com.amazonaws.services.kinesis.model.ShardIteratorType.AT_SEQUENCE_NUMBER;
import static com.amazonaws.services.kinesis.model.ShardIteratorType.AT_TIMESTAMP;
import static org.assertj.core.api.Assertions.assertThat;
-import static org.mockito.BDDMockito.given;
import static org.mockito.Matchers.anyString;
import static org.mockito.Matchers.eq;
import static org.mockito.Matchers.isNull;
@@ -141,7 +140,7 @@
private KinesisRecord recordWith(ExtendedSequenceNumber extendedSequenceNumber) {
KinesisRecord record = mock(KinesisRecord.class);
- given(record.getExtendedSequenceNumber()).willReturn(extendedSequenceNumber);
+ when(record.getExtendedSequenceNumber()).thenReturn(extendedSequenceNumber);
return record;
}
@@ -153,7 +152,7 @@
private KinesisRecord recordWith(Instant approximateArrivalTimestamp) {
KinesisRecord record = mock(KinesisRecord.class);
- given(record.getApproximateArrivalTimestamp()).willReturn(approximateArrivalTimestamp);
+ when(record.getApproximateArrivalTimestamp()).thenReturn(approximateArrivalTimestamp);
return record;
}
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
index 194232b..125ff8c 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java
@@ -31,8 +31,8 @@
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.junit.After;
@@ -41,11 +41,11 @@
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.Mockito;
-import org.mockito.runners.MockitoJUnitRunner;
+import org.mockito.junit.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
/** Tests {@link ShardReadersPool}. */
-@RunWith(MockitoJUnitRunner.class)
+@RunWith(MockitoJUnitRunner.Silent.class)
public class ShardReadersPoolTest {
private static final int TIMEOUT_IN_MILLIS = (int) TimeUnit.SECONDS.toMillis(10);
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIteratorTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIteratorTest.java
index 2115c4f..38aedb8 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIteratorTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ShardRecordsIteratorTest.java
@@ -34,11 +34,11 @@
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.invocation.InvocationOnMock;
-import org.mockito.runners.MockitoJUnitRunner;
+import org.mockito.junit.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
/** Tests {@link ShardRecordsIterator}. */
-@RunWith(MockitoJUnitRunner.class)
+@RunWith(MockitoJUnitRunner.Silent.class)
public class ShardRecordsIteratorTest {
private static final String INITIAL_ITERATOR = "INITIAL_ITERATOR";
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClientTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClientTest.java
index f4e0c90..7858de7 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClientTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/SimplifiedKinesisClientTest.java
@@ -19,11 +19,11 @@
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.failBecauseExceptionWasNotThrown;
-import static org.mockito.BDDMockito.given;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.doAnswer;
import static org.mockito.Mockito.reset;
import static org.mockito.Mockito.verifyZeroInteractions;
+import static org.mockito.Mockito.when;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.AmazonServiceException.ErrorType;
@@ -74,14 +74,13 @@
@Test
public void shouldReturnIteratorStartingWithSequenceNumber() throws Exception {
- given(
- kinesis.getShardIterator(
- new GetShardIteratorRequest()
- .withStreamName(STREAM)
- .withShardId(SHARD_1)
- .withShardIteratorType(ShardIteratorType.AT_SEQUENCE_NUMBER)
- .withStartingSequenceNumber(SEQUENCE_NUMBER)))
- .willReturn(new GetShardIteratorResult().withShardIterator(SHARD_ITERATOR));
+ when(kinesis.getShardIterator(
+ new GetShardIteratorRequest()
+ .withStreamName(STREAM)
+ .withShardId(SHARD_1)
+ .withShardIteratorType(ShardIteratorType.AT_SEQUENCE_NUMBER)
+ .withStartingSequenceNumber(SEQUENCE_NUMBER)))
+ .thenReturn(new GetShardIteratorResult().withShardIterator(SHARD_ITERATOR));
String stream =
underTest.getShardIterator(
@@ -93,14 +92,13 @@
@Test
public void shouldReturnIteratorStartingWithTimestamp() throws Exception {
Instant timestamp = Instant.now();
- given(
- kinesis.getShardIterator(
- new GetShardIteratorRequest()
- .withStreamName(STREAM)
- .withShardId(SHARD_1)
- .withShardIteratorType(ShardIteratorType.AT_SEQUENCE_NUMBER)
- .withTimestamp(timestamp.toDate())))
- .willReturn(new GetShardIteratorResult().withShardIterator(SHARD_ITERATOR));
+ when(kinesis.getShardIterator(
+ new GetShardIteratorRequest()
+ .withStreamName(STREAM)
+ .withShardId(SHARD_1)
+ .withShardIteratorType(ShardIteratorType.AT_SEQUENCE_NUMBER)
+ .withTimestamp(timestamp.toDate())))
+ .thenReturn(new GetShardIteratorResult().withShardIterator(SHARD_ITERATOR));
String stream =
underTest.getShardIterator(
@@ -152,7 +150,7 @@
.withShardId(SHARD_1)
.withShardIteratorType(ShardIteratorType.LATEST);
- given(kinesis.getShardIterator(request)).willThrow(thrownException);
+ when(kinesis.getShardIterator(request)).thenThrow(thrownException);
try {
underTest.getShardIterator(STREAM, SHARD_1, ShardIteratorType.LATEST, null, null);
@@ -169,13 +167,13 @@
Shard shard1 = new Shard().withShardId(SHARD_1);
Shard shard2 = new Shard().withShardId(SHARD_2);
Shard shard3 = new Shard().withShardId(SHARD_3);
- given(kinesis.describeStream(STREAM, null))
- .willReturn(
+ when(kinesis.describeStream(STREAM, null))
+ .thenReturn(
new DescribeStreamResult()
.withStreamDescription(
new StreamDescription().withShards(shard1, shard2).withHasMoreShards(true)));
- given(kinesis.describeStream(STREAM, SHARD_2))
- .willReturn(
+ when(kinesis.describeStream(STREAM, SHARD_2))
+ .thenReturn(
new DescribeStreamResult()
.withStreamDescription(
new StreamDescription().withShards(shard3).withHasMoreShards(false)));
@@ -220,7 +218,7 @@
private void shouldHandleShardListingError(
Exception thrownException, Class<? extends Exception> expectedExceptionClass) {
- given(kinesis.describeStream(STREAM, null)).willThrow(thrownException);
+ when(kinesis.describeStream(STREAM, null)).thenThrow(thrownException);
try {
underTest.listShards(STREAM);
failBecauseExceptionWasNotThrown(expectedExceptionClass);
@@ -241,7 +239,7 @@
GetMetricStatisticsResult result =
new GetMetricStatisticsResult().withDatapoints(new Datapoint().withSum(1.0));
- given(cloudWatch.getMetricStatistics(metricStatisticsRequest)).willReturn(result);
+ when(cloudWatch.getMetricStatistics(metricStatisticsRequest)).thenReturn(result);
long backlogBytes = underTest.getBacklogBytes(STREAM, countSince, countTo);
@@ -262,7 +260,7 @@
new Datapoint().withSum(3.0),
new Datapoint().withSum(2.0));
- given(cloudWatch.getMetricStatistics(metricStatisticsRequest)).willReturn(result);
+ when(cloudWatch.getMetricStatistics(metricStatisticsRequest)).thenReturn(result);
long backlogBytes = underTest.getBacklogBytes(STREAM, countSince, countTo);
@@ -317,7 +315,7 @@
GetMetricStatisticsRequest metricStatisticsRequest =
underTest.createMetricStatisticsRequest(STREAM, countSince, countTo, periodTime);
- given(cloudWatch.getMetricStatistics(metricStatisticsRequest)).willThrow(thrownException);
+ when(cloudWatch.getMetricStatistics(metricStatisticsRequest)).thenThrow(thrownException);
try {
underTest.getBacklogBytes(STREAM, countSince, countTo);
failBecauseExceptionWasNotThrown(expectedExceptionClass);
diff --git a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinderTest.java b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinderTest.java
index 9147cf0..a7eb9d8 100644
--- a/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinderTest.java
+++ b/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/StartingPointShardsFinderTest.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.io.kinesis;
import static org.assertj.core.api.Assertions.assertThat;
-import static org.mockito.BDDMockito.given;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
import com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream;
import com.amazonaws.services.kinesis.clientlibrary.types.UserRecord;
@@ -27,7 +27,7 @@
import com.amazonaws.services.kinesis.model.ShardIteratorType;
import java.util.Collections;
import java.util.List;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -89,7 +89,7 @@
for (Shard shard : allShards) {
activeAtTimestamp(shard, timestampAtTheBeginning);
}
- given(kinesis.listShards(STREAM_NAME)).willReturn(allShards);
+ when(kinesis.listShards(STREAM_NAME)).thenReturn(allShards);
// when
Iterable<Shard> shardsAtStartingPoint =
@@ -118,7 +118,7 @@
activeAtTimestamp(shard09, timestampAfterShards3And4Merge);
activeAtTimestamp(shard10, timestampAfterShards3And4Merge);
- given(kinesis.listShards(STREAM_NAME)).willReturn(allShards);
+ when(kinesis.listShards(STREAM_NAME)).thenReturn(allShards);
// when
Iterable<Shard> shardsAtStartingPoint =
@@ -147,7 +147,7 @@
activeAtTimestamp(shard09, timestampAtTheEnd);
activeAtTimestamp(shard10, timestampAtTheEnd);
- given(kinesis.listShards(STREAM_NAME)).willReturn(allShards);
+ when(kinesis.listShards(STREAM_NAME)).thenReturn(allShards);
// when
Iterable<Shard> shardsAtStartingPoint =
@@ -161,7 +161,7 @@
public void shouldFindLastShardsWhenLatestStartingPointRequested() throws Exception {
// given
StartingPoint latestStartingPoint = new StartingPoint(InitialPositionInStream.LATEST);
- given(kinesis.listShards(STREAM_NAME)).willReturn(allShards);
+ when(kinesis.listShards(STREAM_NAME)).thenReturn(allShards);
// when
Iterable<Shard> shardsAtStartingPoint =
@@ -176,7 +176,7 @@
// given
StartingPoint trimHorizonStartingPoint =
new StartingPoint(InitialPositionInStream.TRIM_HORIZON);
- given(kinesis.listShards(STREAM_NAME)).willReturn(allShards);
+ when(kinesis.listShards(STREAM_NAME)).thenReturn(allShards);
// when
Iterable<Shard> shardsAtStartingPoint =
@@ -206,7 +206,7 @@
shard09,
closedShard10);
- given(kinesis.listShards(STREAM_NAME)).willReturn(shards);
+ when(kinesis.listShards(STREAM_NAME)).thenReturn(shards);
// when
underTest.findShardsAtStartingPoint(kinesis, STREAM_NAME, latestStartingPoint);
@@ -254,19 +254,17 @@
try {
String shardIterator = shardIteratorType + shard.getShardId() + "-current";
if (shardIteratorType == ShardIteratorType.AT_TIMESTAMP) {
- given(
- kinesis.getShardIterator(
- STREAM_NAME,
- shard.getShardId(),
- ShardIteratorType.AT_TIMESTAMP,
- null,
- startTimestamp))
- .willReturn(shardIterator);
+ when(kinesis.getShardIterator(
+ STREAM_NAME,
+ shard.getShardId(),
+ ShardIteratorType.AT_TIMESTAMP,
+ null,
+ startTimestamp))
+ .thenReturn(shardIterator);
} else {
- given(
- kinesis.getShardIterator(
- STREAM_NAME, shard.getShardId(), shardIteratorType, null, null))
- .willReturn(shardIterator);
+ when(kinesis.getShardIterator(
+ STREAM_NAME, shard.getShardId(), shardIteratorType, null, null))
+ .thenReturn(shardIterator);
}
GetKinesisRecordsResult result =
new GetKinesisRecordsResult(
@@ -275,7 +273,7 @@
0,
STREAM_NAME,
shard.getShardId());
- given(kinesis.getRecords(shardIterator, STREAM_NAME, shard.getShardId())).willReturn(result);
+ when(kinesis.getRecords(shardIterator, STREAM_NAME, shard.getShardId())).thenReturn(result);
} catch (TransientKinesisException e) {
throw new RuntimeException(e);
}
diff --git a/sdks/java/io/kudu/build.gradle b/sdks/java/io/kudu/build.gradle
index 4acc7cd..8619d67 100644
--- a/sdks/java/io/kudu/build.gradle
+++ b/sdks/java/io/kudu/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.kudu')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -35,16 +35,16 @@
def kudu_version = "1.9.0"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow "org.apache.kudu:kudu-client:$kudu_version"
- shadow library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile "org.apache.kudu:kudu-client:$kudu_version"
+ compile library.java.slf4j_api
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java b/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java
index 126b706..d92b098 100644
--- a/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java
+++ b/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.kudu;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.util.Collections;
@@ -42,8 +42,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
import org.apache.beam.sdk.values.TypeDescriptors;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
import org.apache.kudu.Common;
import org.apache.kudu.client.KuduException;
import org.apache.kudu.client.KuduPredicate;
diff --git a/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduServiceImpl.java b/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduServiceImpl.java
index 1f90687..4c27a35 100644
--- a/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduServiceImpl.java
+++ b/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduServiceImpl.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.kudu;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.util.List;
diff --git a/sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduTestUtils.java b/sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduTestUtils.java
index 9bd7691..7de6a86 100644
--- a/sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduTestUtils.java
+++ b/sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduTestUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.kudu;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.kudu.ColumnSchema;
import org.apache.kudu.Schema;
import org.apache.kudu.Type;
diff --git a/sdks/java/io/mongodb/build.gradle b/sdks/java/io/mongodb/build.gradle
index ad46ca9..d8ac11c 100644
--- a/sdks/java/io/mongodb/build.gradle
+++ b/sdks/java/io/mongodb/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.mongodb')
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -25,18 +25,18 @@
ext.summary = "IO to read and write on MongoDB."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow "org.mongodb:mongo-java-driver:3.9.1"
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile "org.mongodb:mongo-java-driver:3.9.1"
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:testing:test-utils", configuration: "shadowTest")
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
testCompile "de.flapdoodle.embed:de.flapdoodle.embed.mongo:2.2.0"
testCompile "de.flapdoodle.embed:de.flapdoodle.embed.process:2.1.2"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/FindQuery.java b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/FindQuery.java
index 9da9d4d..a5e8424 100644
--- a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/FindQuery.java
+++ b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/FindQuery.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.mongodb;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import com.mongodb.BasicDBObject;
diff --git a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbGridFSIO.java b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbGridFSIO.java
index bd82136..45b6791 100644
--- a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbGridFSIO.java
+++ b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbGridFSIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.mongodb;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import com.mongodb.DB;
diff --git a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
index 6f42935..4bdbbf4 100644
--- a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
+++ b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.io.mongodb;
import static org.apache.beam.sdk.io.mongodb.FindQuery.bson2BsonDocument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import com.mongodb.BasicDBObject;
@@ -52,7 +52,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.bson.BsonDocument;
import org.bson.BsonInt32;
import org.bson.BsonString;
@@ -454,14 +454,19 @@
return Collections.singletonList(this);
}
- List<String> keys = splitKeysToFilters(splitKeys);
for (String shardFilter : splitKeysToFilters(splitKeys)) {
SerializableFunction<MongoCollection<Document>, MongoCursor<Document>> queryFn =
spec.queryFn();
BsonDocument filters = bson2BsonDocument(Document.parse(shardFilter));
FindQuery findQuery = (FindQuery) queryFn;
- FindQuery queryWithFilter = findQuery.toBuilder().setFilters(filters).build();
+ final BsonDocument allFilters =
+ bson2BsonDocument(
+ findQuery.filters() != null
+ ? Filters.and(findQuery.filters(), filters)
+ : filters);
+ FindQuery queryWithFilter = findQuery.toBuilder().setFilters(allFilters).build();
+ LOG.debug("using filters: " + allFilters.toJson());
sources.add(new BoundedMongoDbSource(spec.withQueryFn(queryWithFilter)));
}
} else {
diff --git a/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDBIOIT.java b/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDBIOIT.java
index a4f28ca..39af3af 100644
--- a/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDBIOIT.java
+++ b/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDBIOIT.java
@@ -38,8 +38,6 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
-import org.apache.beam.sdk.testutils.metrics.CountMonitor;
import org.apache.beam.sdk.testutils.metrics.IOITMetrics;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
@@ -48,7 +46,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.bson.Document;
import org.junit.AfterClass;
import org.junit.BeforeClass;
@@ -145,8 +143,6 @@
.apply("Generate sequence", GenerateSequence.from(0).to(options.getNumberOfRecords()))
.apply("Produce documents", MapElements.via(new LongToDocumentFn()))
.apply("Collect write time metric", ParDo.of(new TimeMonitor<>(NAMESPACE, "write_time")))
- .apply("Collect item count", ParDo.of(new CountMonitor<>(NAMESPACE, "item_count")))
- .apply("Collect byte count", ParDo.of(new ByteMonitor<>(NAMESPACE, "byte_count")))
.apply(
"Write documents to MongoDB",
MongoDbIO.write()
@@ -201,16 +197,6 @@
return NamedTestResult.create(
uuid, timestamp, "write_time", (writeEnd - writeStart) / 1e3);
});
- suppliers.add(
- reader -> {
- long itemCount = reader.getCounterMetric("item_count");
- return NamedTestResult.create(uuid, timestamp, "item_count", itemCount);
- });
- suppliers.add(
- reader -> {
- long byteCount = reader.getCounterMetric("byte_count");
- return NamedTestResult.create(uuid, timestamp, "byte_count", byteCount);
- });
return suppliers;
}
diff --git a/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbIOTest.java b/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbIOTest.java
index 4f5f49e..7f9f0ed 100644
--- a/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbIOTest.java
+++ b/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbIOTest.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterators;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators;
import org.bson.BsonDocument;
import org.bson.BsonInt32;
import org.bson.BsonString;
diff --git a/sdks/java/io/mqtt/build.gradle b/sdks/java/io/mqtt/build.gradle
index e279522..9d7b188 100644
--- a/sdks/java/io/mqtt/build.gradle
+++ b/sdks/java/io/mqtt/build.gradle
@@ -17,19 +17,19 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.mqtt')
description = "Apache Beam :: SDKs :: Java :: IO :: MQTT"
ext.summary = "IO to read and write to a MQTT broker."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow library.java.joda_time
- shadow "org.fusesource.mqtt-client:mqtt-client:1.15"
- shadow "org.fusesource.hawtbuf:hawtbuf:1.11"
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.slf4j_api
+ compile library.java.joda_time
+ compile "org.fusesource.mqtt-client:mqtt-client:1.15"
+ compile "org.fusesource.hawtbuf:hawtbuf:1.11"
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.activemq_broker
testCompile library.java.activemq_mqtt
testCompile library.java.activemq_kahadb_store
@@ -37,5 +37,5 @@
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java b/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java
index dc7cde3..9463894 100644
--- a/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java
+++ b/sdks/java/io/mqtt/src/main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.mqtt;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
import org.fusesource.mqtt.client.BlockingConnection;
import org.fusesource.mqtt.client.MQTT;
import org.fusesource.mqtt.client.Message;
diff --git a/sdks/java/io/parquet/build.gradle b/sdks/java/io/parquet/build.gradle
index fe0f28d..7a038f3 100644
--- a/sdks/java/io/parquet/build.gradle
+++ b/sdks/java/io/parquet/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.parquet')
description = "Apache Beam :: SDKs :: Java :: IO :: Parquet"
ext.summary = "IO to read and write on Parquet storage format."
@@ -25,19 +25,19 @@
def parquet_version = "1.10.0"
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.slf4j_api
- shadow "org.apache.parquet:parquet-avro:$parquet_version"
- shadow "org.apache.parquet:parquet-common:$parquet_version"
- shadow "org.apache.parquet:parquet-hadoop:$parquet_version"
- shadow library.java.avro
- shadow library.java.hadoop_client
- shadow library.java.hadoop_common
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.slf4j_api
+ compile "org.apache.parquet:parquet-avro:$parquet_version"
+ compile "org.apache.parquet:parquet-common:$parquet_version"
+ compile "org.apache.parquet:parquet-hadoop:$parquet_version"
+ compile library.java.avro
+ compile library.java.hadoop_client
+ compile library.java.hadoop_common
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
index 0a4bdb4..fa50715 100644
--- a/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
+++ b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.parquet;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import static org.apache.parquet.hadoop.ParquetFileWriter.Mode.OVERWRITE;
import com.google.auto.value.AutoValue;
@@ -101,15 +101,18 @@
* <pre>{@code
* pipeline
* .apply(...) // PCollection<GenericRecord>
- * .apply(FileIO.<GenericRecord>
- * .write()
+ * .apply(FileIO
+ * .<GenericRecord>write()
* .via(ParquetIO.sink(SCHEMA)
- * .withCompression(CompressionCodecName.SNAPPY))
- * .to("destination/path")
+ * .withCompressionCodec(CompressionCodecName.SNAPPY))
+ * .to("destination/path"))
* }</pre>
*
* <p>This IO API is considered experimental and may break or receive backwards-incompatible changes
* in future versions of the Apache Beam SDK.
+ *
+ * @see <a href="https://beam.apache.org/documentation/io/built-in/parquet/">Beam ParquetIO
+ * documentation</a>
*/
@Experimental(Experimental.Kind.SOURCE_SINK)
public class ParquetIO {
@@ -316,6 +319,7 @@
@Override
public void flush() throws IOException {
+ // the only way to completely flush the output is to call writer.close() here
writer.close();
}
diff --git a/sdks/java/io/rabbitmq/build.gradle b/sdks/java/io/rabbitmq/build.gradle
index a890edc..52802c5 100644
--- a/sdks/java/io/rabbitmq/build.gradle
+++ b/sdks/java/io/rabbitmq/build.gradle
@@ -17,17 +17,17 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.rabbitmq')
description = "Apache Beam :: SDKs :: Java :: IO :: RabbitMQ"
ext.summary = "IO to read and write to a RabbitMQ broker."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.joda_time
- shadow "com.rabbitmq:amqp-client:4.9.3"
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.joda_time
+ compile "com.rabbitmq:amqp-client:4.9.3"
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile "org.apache.qpid:qpid-broker:0.28"
testCompile "org.apache.qpid:qpid-broker-core:0.28"
testCompile library.java.junit
@@ -35,5 +35,5 @@
testCompile library.java.hamcrest_library
testCompile library.java.slf4j_api
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
index ae152b8..fef40d2 100644
--- a/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
+++ b/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.rabbitmq;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import com.rabbitmq.client.Channel;
diff --git a/sdks/java/io/redis/build.gradle b/sdks/java/io/redis/build.gradle
index f8b4112..2626400 100644
--- a/sdks/java/io/redis/build.gradle
+++ b/sdks/java/io/redis/build.gradle
@@ -17,20 +17,20 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.redis')
description = "Apache Beam :: SDKs :: Java :: IO :: Redis"
ext.summary ="IO to read and write on a Redis keystore."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow "redis.clients:jedis:3.0.1"
- testCompile project(path: ":sdks:java:io:common", configuration: "shadowTest")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile "redis.clients:jedis:3.0.1"
+ testCompile project(path: ":sdks:java:io:common", configuration: "testRuntime")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile "com.github.kstyrc:embedded-redis:0.6"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
index 6f5ff52..14d84e9 100644
--- a/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
+++ b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisConnectionConfiguration.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.redis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.Serializable;
diff --git a/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java
index 39d1b91..9612487 100644
--- a/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java
+++ b/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.redis;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.util.List;
@@ -41,8 +41,8 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;
import redis.clients.jedis.ScanParams;
@@ -450,7 +450,13 @@
SADD,
/** Use PFADD command. Insert value in a HLL structure. Create key if it doesn't exist */
- PFADD
+ PFADD,
+
+ /** Use INCBY command. Increment counter value of a key by a given value. */
+ INCRBY,
+
+ /** Use DECRBY command. Decrement counter value of a key by given value. */
+ DECRBY,
}
@Nullable
@@ -578,6 +584,10 @@
writeUsingSaddCommand(record, expireTime);
} else if (Method.PFADD == method) {
writeUsingHLLCommand(record, expireTime);
+ } else if (Method.INCRBY == method) {
+ writeUsingIncrBy(record);
+ } else if (Method.DECRBY == method) {
+ writeUsingDecrBy(record);
}
}
@@ -630,6 +640,20 @@
pipeline.pfadd(key, value);
}
+ private void writeUsingIncrBy(KV<String, String> record) {
+ String key = record.getKey();
+ String value = record.getValue();
+ long inc = Long.parseLong(value);
+ pipeline.incrBy(key, inc);
+ }
+
+ private void writeUsingDecrBy(KV<String, String> record) {
+ String key = record.getKey();
+ String value = record.getValue();
+ long decr = Long.parseLong(value);
+ pipeline.decrBy(key, decr);
+ }
+
private void setExpireTimeWhenRequired(String key, Long expireTime) {
if (expireTime != null) {
pipeline.pexpire(key, expireTime);
diff --git a/sdks/java/io/redis/src/test/java/org/apache/beam/sdk/io/redis/RedisIOTest.java b/sdks/java/io/redis/src/test/java/org/apache/beam/sdk/io/redis/RedisIOTest.java
index b5792a3..bcb3fca 100644
--- a/sdks/java/io/redis/src/test/java/org/apache/beam/sdk/io/redis/RedisIOTest.java
+++ b/sdks/java/io/redis/src/test/java/org/apache/beam/sdk/io/redis/RedisIOTest.java
@@ -174,6 +174,37 @@
assertEquals(6, count);
}
+ @Test
+ public void testWriteUsingINCRBY() throws Exception {
+ String key = "key_incr";
+ List<String> values = Arrays.asList("0", "1", "2", "-3", "2", "4", "0", "5");
+ List<KV<String, String>> data = buildConstantKeyList(key, values);
+
+ PCollection<KV<String, String>> write = p.apply(Create.of(data));
+ write.apply(RedisIO.write().withEndpoint(REDIS_HOST, port).withMethod(Method.INCRBY));
+
+ p.run();
+
+ long count = Long.parseLong(client.get(key));
+ assertEquals(11, count);
+ }
+
+ @Test
+ public void testWriteUsingDECRBY() throws Exception {
+ String key = "key_decr";
+
+ List<String> values = Arrays.asList("-10", "1", "2", "-3", "2", "4", "0", "5");
+ List<KV<String, String>> data = buildConstantKeyList(key, values);
+
+ PCollection<KV<String, String>> write = p.apply(Create.of(data));
+ write.apply(RedisIO.write().withEndpoint(REDIS_HOST, port).withMethod(Method.DECRBY));
+
+ p.run();
+
+ long count = Long.parseLong(client.get(key));
+ assertEquals(-1, count);
+ }
+
private static List<KV<String, String>> buildConstantKeyList(String key, List<String> values) {
List<KV<String, String>> data = new ArrayList<>();
for (String value : values) {
diff --git a/sdks/java/io/solr/build.gradle b/sdks/java/io/solr/build.gradle
index 6ce544e..928d6db 100644
--- a/sdks/java/io/solr/build.gradle
+++ b/sdks/java/io/solr/build.gradle
@@ -17,19 +17,19 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.solr')
description = "Apache Beam :: SDKs :: Java :: IO :: Solr"
ext.summary = "IO to read and write from/to Solr."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.commons_compress
- shadow "org.apache.solr:solr-solrj:5.5.4"
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.commons_compress
+ compile "org.apache.solr:solr-solrj:5.5.4"
compileOnly "org.apache.httpcomponents:httpclient:4.5.6"
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
- testCompile project(path: ":sdks:java:io:common", configuration: "shadow")
+ testCompile project(":sdks:java:io:common")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile library.java.junit
@@ -38,5 +38,5 @@
testCompile "org.apache.solr:solr-core:5.5.4"
testCompile "com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.3.2"
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/AuthorizedSolrClient.java b/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/AuthorizedSolrClient.java
index 8c66fa8..70056f3 100644
--- a/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/AuthorizedSolrClient.java
+++ b/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/AuthorizedSolrClient.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.solr;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.Closeable;
import java.io.IOException;
diff --git a/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/SolrIO.java b/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/SolrIO.java
index 98c2b99..a506dba 100644
--- a/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/SolrIO.java
+++ b/sdks/java/io/solr/src/main/java/org/apache/beam/sdk/io/solr/SolrIO.java
@@ -17,9 +17,9 @@
*/
package org.apache.beam.sdk.io.solr;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -44,8 +44,8 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
import org.apache.http.client.HttpClient;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
diff --git a/sdks/java/io/solr/src/test/java/org/apache/beam/sdk/io/solr/SolrIOTest.java b/sdks/java/io/solr/src/test/java/org/apache/beam/sdk/io/solr/SolrIOTest.java
index 28592ef..09c38a2 100644
--- a/sdks/java/io/solr/src/test/java/org/apache/beam/sdk/io/solr/SolrIOTest.java
+++ b/sdks/java/io/solr/src/test/java/org/apache/beam/sdk/io/solr/SolrIOTest.java
@@ -36,7 +36,7 @@
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFnTester;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.BaseEncoding;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.BaseEncoding;
import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
diff --git a/sdks/java/io/synthetic/build.gradle b/sdks/java/io/synthetic/build.gradle
index 57019b1..52c794b 100644
--- a/sdks/java/io/synthetic/build.gradle
+++ b/sdks/java/io/synthetic/build.gradle
@@ -17,23 +17,22 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.sdk.io.synthetic')
description = "Apache Beam :: SDKs :: Java :: IO :: Synthetic"
ext.summary = "Generators of Synthetic IO for Testing."
dependencies {
compile library.java.joda_time
- shadow library.java.commons_math3
- shadow library.java.jackson_core
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
- shadow library.java.guava
+ compile library.java.commons_math3
+ compile library.java.jackson_core
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
+ compile project(path: ":sdks:java:core", configuration: "shadow")
- shadowTest library.java.vendored_guava_20_0
+ testCompile library.java.vendored_guava_26_0_jre
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedSource.java b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedSource.java
index eae04bc..a4f92b8 100644
--- a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedSource.java
+++ b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedSource.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.synthetic;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import java.io.IOException;
import java.util.List;
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
index 6b4be79..3179c1c 100644
--- a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
+++ b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.synthetic;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
@@ -32,8 +32,8 @@
import java.nio.ByteBuffer;
import java.util.Random;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.HashFunction;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.HashFunction;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.apache.commons.math3.distribution.ConstantRealDistribution;
import org.apache.commons.math3.distribution.ExponentialDistribution;
import org.apache.commons.math3.distribution.IntegerDistribution;
diff --git a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticSourceOptions.java b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticSourceOptions.java
index 20a1df2..3ae8f76 100644
--- a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticSourceOptions.java
+++ b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticSourceOptions.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.synthetic;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
diff --git a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticStep.java b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticStep.java
index 98c5531..75a4c9d 100644
--- a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticStep.java
+++ b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticStep.java
@@ -18,7 +18,7 @@
package org.apache.beam.sdk.io.synthetic;
import static org.apache.beam.sdk.io.synthetic.delay.SyntheticDelay.delay;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.fasterxml.jackson.annotation.JsonProperty;
import java.util.Random;
@@ -28,10 +28,10 @@
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.values.KV;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheBuilder;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.CacheLoader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LoadingCache;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.RateLimiter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.CacheLoader;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LoadingCache;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.RateLimiter;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/delay/SyntheticDelay.java b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/delay/SyntheticDelay.java
index 090cc12..04a6206 100644
--- a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/delay/SyntheticDelay.java
+++ b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/delay/SyntheticDelay.java
@@ -20,9 +20,9 @@
import java.util.Random;
import java.util.concurrent.TimeUnit;
import org.apache.beam.sdk.io.synthetic.SyntheticOptions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Stopwatch;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Stopwatch;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.Uninterruptibles;
import org.joda.time.Duration;
/** Utility functions used in {@link org.apache.beam.sdk.io.synthetic}. */
diff --git a/sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/SyntheticStepTest.java b/sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/SyntheticStepTest.java
index 08845af..f2eea4b 100644
--- a/sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/SyntheticStepTest.java
+++ b/sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/SyntheticStepTest.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.apache.commons.math3.distribution.ConstantRealDistribution;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/io/tika/build.gradle b/sdks/java/io/tika/build.gradle
index 9fba673..813a692 100644
--- a/sdks/java/io/tika/build.gradle
+++ b/sdks/java/io/tika/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.tika')
description = "Apache Beam :: SDKs :: Java :: IO :: Tika"
ext.summary = "Tika Input to parse files."
@@ -26,15 +26,15 @@
def bndlib_version = "1.43.0"
dependencies {
- shadow library.java.vendored_guava_20_0
+ compile library.java.vendored_guava_26_0_jre
compileOnly "biz.aQute:bndlib:$bndlib_version"
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow "org.apache.tika:tika-core:$tika_version"
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile "org.apache.tika:tika-core:$tika_version"
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testCompile "org.apache.tika:tika-parsers:$tika_version"
testCompileOnly "biz.aQute:bndlib:$bndlib_version"
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java
index 2c8c322..bb54ea8 100644
--- a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java
+++ b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java
@@ -17,16 +17,16 @@
*/
package org.apache.beam.sdk.io.tika;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.io.Serializable;
import java.util.Arrays;
import java.util.Objects;
import javax.annotation.Nullable;
import org.apache.beam.sdk.util.SerializableThrowable;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Throwables;
import org.apache.tika.metadata.Metadata;
/**
diff --git a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
index 0075190..28a316a 100644
--- a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
+++ b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.sdk.io.tika;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import com.google.auto.value.AutoValue;
import java.io.InputStream;
diff --git a/sdks/java/io/xml/build.gradle b/sdks/java/io/xml/build.gradle
index c2bada0..2cb9077 100644
--- a/sdks/java/io/xml/build.gradle
+++ b/sdks/java/io/xml/build.gradle
@@ -17,21 +17,21 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.xml')
description = "Apache Beam :: SDKs :: Java :: IO :: XML"
ext.summary = "IO to read and write XML files."
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.stax2_api
- shadow library.java.woodstox_core_asl
- shadowTest library.java.jaxb_api
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.stax2_api
+ compile library.java.woodstox_core_asl
+ testCompile library.java.jaxb_api
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile library.java.junit
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
- testRuntimeOnly project(path: ":runners:direct-java")
+ testRuntimeOnly project(":runners:direct-java")
}
diff --git a/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/JAXBCoder.java b/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/JAXBCoder.java
index 89c5aa1..fa7b419 100644
--- a/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/JAXBCoder.java
+++ b/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/JAXBCoder.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.util.EmptyOnDeserializationThreadLocal;
import org.apache.beam.sdk.util.VarInt;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.io.ByteStreams;
/**
* A coder for JAXB annotated objects. This coder uses JAXB marshalling/unmarshalling mechanisms to
diff --git a/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java b/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java
index 17fdcf4..97da11c 100644
--- a/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java
+++ b/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.io.xml;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.google.auto.value.AutoValue;
import java.io.IOException;
@@ -50,7 +50,7 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PDone;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
/** Transforms for reading and writing XML files using JAXB mappers. */
public class XmlIO {
@@ -680,6 +680,7 @@
@Override
public void flush() throws IOException {
outputStream.write(("\n</" + getRootElement() + ">").getBytes(Charset.forName(getCharset())));
+ outputStream.flush();
}
}
}
diff --git a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
index 410026a..4a35766 100644
--- a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
+++ b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/JAXBCoderTest.java
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.SerializableUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
diff --git a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlIOTest.java b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlIOTest.java
index e1e36e6..db4ccbb 100644
--- a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlIOTest.java
+++ b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlIOTest.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.transforms.Values;
import org.apache.beam.sdk.transforms.display.DisplayData;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlSourceTest.java b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlSourceTest.java
index 93bf869..ff9b392 100644
--- a/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlSourceTest.java
+++ b/sdks/java/io/xml/src/test/java/org/apache/beam/sdk/io/xml/XmlSourceTest.java
@@ -43,7 +43,7 @@
import org.apache.beam.sdk.testing.PAssert;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/javadoc/build.gradle b/sdks/java/javadoc/build.gradle
index 8d7486e..d3d4a97 100644
--- a/sdks/java/javadoc/build.gradle
+++ b/sdks/java/javadoc/build.gradle
@@ -24,7 +24,7 @@
* used as part of the beam-site source tree.
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature()
+applyJavaNature(publish: false)
description = "Apache Beam :: SDKs :: Java :: Aggregated Javadoc"
for (p in rootProject.subprojects) {
diff --git a/sdks/java/maven-archetypes/examples/build.gradle b/sdks/java/maven-archetypes/examples/build.gradle
index 2fe79a5..beec649 100644
--- a/sdks/java/maven-archetypes/examples/build.gradle
+++ b/sdks/java/maven-archetypes/examples/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.maven.archetypes.examples')
description = "Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples"
ext.summary = """A Maven Archetype to create a project containing all the
@@ -28,7 +28,6 @@
'project.version': version,
'bigquery.version': dependencies.create(project.library.java.google_api_services_bigquery).getVersion(),
'google-clients.version': dependencies.create(project.library.java.google_api_client).getVersion(),
- 'guava.version': dependencies.create(project.library.java.guava).getVersion(),
'hamcrest.version': dependencies.create(project.library.java.hamcrest_library).getVersion(),
'jackson.version': dependencies.create(project.library.java.jackson_core).getVersion(),
'joda.version': dependencies.create(project.library.java.joda_time).getVersion(),
@@ -44,6 +43,7 @@
'maven-jar-plugin.version': dependencies.create(project.library.maven.maven_jar_plugin).getVersion(),
'maven-shade-plugin.version': dependencies.create(project.library.maven.maven_shade_plugin).getVersion(),
'maven-surefire-plugin.version': dependencies.create(project.library.maven.maven_surefire_plugin).getVersion(),
+ 'flink.artifact.name': 'beam-runners-flink-'.concat(project(":runners:flink:1.8").getName()),
]
}
@@ -69,5 +69,5 @@
}
dependencies {
- shadow project(path: ":examples:java", configuration: "shadow")
+ compile project(":examples:java")
}
diff --git a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 3cc47b9..eb9efc6 100644
--- a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -31,7 +31,6 @@
<bigquery.version>@bigquery.version@</bigquery.version>
<google-clients.version>@google-clients.version@</google-clients.version>
- <guava.version>@guava.version@</guava.version>
<hamcrest.version>@hamcrest.version@</hamcrest.version>
<jackson.version>@jackson.version@</jackson.version>
<joda.version>@joda.version@</joda.version>
@@ -47,6 +46,7 @@
<hadoop.version>@hadoop.version@</hadoop.version>
<maven-surefire-plugin.version>@maven-surefire-plugin.version@</maven-surefire-plugin.version>
<nemo.version>@nemo.version@</nemo.version>
+ <flink.artifact.name>@flink.artifact.name@</flink.artifact.name>
</properties>
<repositories>
@@ -249,7 +249,10 @@
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
- <artifactId>beam-runners-flink_2.11</artifactId>
+ <!-- Please see the Flink Runner page for an up-to-date list
+ of supported Flink versions and their artifact names:
+ https://beam.apache.org/documentation/runners/flink/ -->
+ <artifactId>${flink.artifact.name}</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
@@ -359,6 +362,19 @@
</dependency>
</dependencies>
</profile>
+
+ <profile>
+ <id>jet-runner</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-runners-jet</artifactId>
+ <version>${beam.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ </dependencies>
+ </profile>
+
</profiles>
<dependencies>
@@ -439,12 +455,6 @@
<version>${joda.version}</version>
</dependency>
- <dependency>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- <version>${guava.version}</version>
- </dependency>
-
<!-- Add slf4j API frontend binding with JUL backend -->
<dependency>
<groupId>org.slf4j</groupId>
@@ -467,7 +477,7 @@
<artifactId>hamcrest-core</artifactId>
<version>${hamcrest.version}</version>
</dependency>
-
+
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
@@ -487,7 +497,7 @@
<version>${beam.version}</version>
<scope>test</scope>
</dependency>
-
+
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
diff --git a/sdks/java/maven-archetypes/starter/build.gradle b/sdks/java/maven-archetypes/starter/build.gradle
index 38eda09..850c661 100644
--- a/sdks/java/maven-archetypes/starter/build.gradle
+++ b/sdks/java/maven-archetypes/starter/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(exportJavadoc: false, automaticModuleName: 'org.apache.beam.maven.archetypes.starter')
description = "Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter"
ext.summary = """A Maven archetype to create a simple starter pipeline to
@@ -33,6 +33,6 @@
}
dependencies {
- shadow project(path: ":runners:direct-java", configuration: "shadow")
- shadow project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(path: ":runners:direct-java", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
}
diff --git a/sdks/java/testing/expansion-service/build.gradle b/sdks/java/testing/expansion-service/build.gradle
new file mode 100644
index 0000000..bfbcc23
--- /dev/null
+++ b/sdks/java/testing/expansion-service/build.gradle
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.expansion.service')
+
+description = "Apache Beam :: SDKs :: Java :: Test Expansion Service"
+ext.summary = """Testing Expansion Service used for executing cross-language transform tests."""
+
+
+dependencies {
+ compile project(path: ":runners:core-construction-java")
+ compile project(path: ":sdks:java:io:parquet")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+}
+
+task runTestExpansionService (type: JavaExec) {
+ main = "org.apache.beam.sdk.expansion.TestExpansionService"
+ classpath = sourceSets.test.runtimeClasspath
+ args = [project.findProperty("constructionService.port") ?: "8097"]
+}
+
+task buildTestExpansionServiceJar(type: ShadowJar) {
+ appendix = "testExpansionService"
+ // Use zip64 mode to avoid "Archive contains more than 65535 entries".
+ zip64 = true
+ mergeServiceFiles()
+ manifest {
+ attributes(
+ 'Main-Class': 'org.apache.beam.sdk.expansion.TestExpansionService'
+ )
+ }
+ from { project.configurations.testRuntime.collect { it.isDirectory() ? it : zipTree(it) }}
+ from sourceSets.main.output
+ from sourceSets.test.output
+}
diff --git a/sdks/java/testing/expansion-service/src/test/java/org/apache/beam/sdk/expansion/TestExpansionService.java b/sdks/java/testing/expansion-service/src/test/java/org/apache/beam/sdk/expansion/TestExpansionService.java
new file mode 100644
index 0000000..68c0e71
--- /dev/null
+++ b/sdks/java/testing/expansion-service/src/test/java/org/apache/beam/sdk/expansion/TestExpansionService.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.expansion;
+
+import com.google.auto.service.AutoService;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.beam.runners.core.construction.expansion.ExpansionService;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.io.FileIO;
+import org.apache.beam.sdk.io.parquet.ParquetIO;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.Values;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Server;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ServerBuilder;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+/**
+ * An {@link org.apache.beam.runners.core.construction.expansion.ExpansionService} useful for tests.
+ */
+public class TestExpansionService {
+
+ private static final String TEST_COUNT_URN = "beam:transforms:xlang:count";
+ private static final String TEST_FILTER_URN = "beam:transforms:xlang:filter_less_than_eq";
+ private static final String TEST_PARQUET_READ_URN = "beam:transforms:xlang:parquet_read";
+ private static final String TEST_PARQUET_WRITE_URN = "beam:transforms:xlang:parquet_write";
+
+ /** Registers a single test transformation. */
+ @AutoService(ExpansionService.ExpansionServiceRegistrar.class)
+ public static class TestTransforms implements ExpansionService.ExpansionServiceRegistrar {
+ String rawSchema =
+ "{ \"type\": \"record\", \"name\": \"testrecord\", \"fields\": "
+ + "[ {\"name\": \"name\", \"type\": \"string\"} ]}";
+
+ @Override
+ public Map<String, ExpansionService.TransformProvider> knownTransforms() {
+ Schema schema = new Schema.Parser().parse(rawSchema);
+ return ImmutableMap.of(
+ TEST_COUNT_URN, spec -> Count.perElement(),
+ TEST_FILTER_URN,
+ spec ->
+ Filter.lessThanEq(
+ // TODO(BEAM-6587): Use strings directly rather than longs.
+ (long) spec.getPayload().toStringUtf8().charAt(0)),
+ TEST_PARQUET_READ_URN,
+ spec ->
+ new PTransform<PBegin, PCollection<GenericRecord>>() {
+ @Override
+ public PCollection<GenericRecord> expand(PBegin input) {
+ return input
+ .apply(FileIO.match().filepattern(spec.getPayload().toStringUtf8()))
+ .apply(FileIO.readMatches())
+ .apply(ParquetIO.readFiles(schema))
+ .setCoder(AvroCoder.of(schema));
+ }
+ },
+ TEST_PARQUET_WRITE_URN,
+ spec ->
+ new PTransform<PCollection<GenericRecord>, PCollection<String>>() {
+ @Override
+ public PCollection<String> expand(PCollection<GenericRecord> input) {
+ return input
+ .apply(
+ FileIO.<GenericRecord>write()
+ .via(ParquetIO.sink(schema))
+ .to(spec.getPayload().toStringUtf8()))
+ .getPerDestinationOutputFilenames()
+ .apply(Values.create());
+ }
+ });
+ }
+ }
+
+ public static void main(String[] args) throws Exception {
+ int port = Integer.parseInt(args[0]);
+ System.out.println("Starting expansion service at localhost:" + port);
+ Server server = ServerBuilder.forPort(port).addService(new ExpansionService()).build();
+ server.start();
+ server.awaitTermination();
+ }
+}
diff --git a/sdks/java/testing/load-tests/build.gradle b/sdks/java/testing/load-tests/build.gradle
index 403e838..fbea1a7 100644
--- a/sdks/java/testing/load-tests/build.gradle
+++ b/sdks/java/testing/load-tests/build.gradle
@@ -17,8 +17,11 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-load-tests'
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(
+ publish: false,
+ archivesBaseName: 'beam-sdks-java-load-tests',
+ exportJavadoc: false
+)
description = "Apache Beam :: SDKs :: Java :: Load Tests"
@@ -56,18 +59,18 @@
}
dependencies {
- shadow library.java.kafka_clients
+ compile library.java.kafka_clients
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":runners:direct-java", configuration: "shadow")
- shadow project(path: ":sdks:java:io:synthetic", configuration: "shadow")
- shadow project(path: ":sdks:java:testing:test-utils", configuration: "shadow")
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadow project(path: ":sdks:java:io:kafka", configuration: "shadow")
- shadow project(path: ":sdks:java:io:kinesis", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(path: ":runners:direct-java", configuration: "shadow")
+ compile project(":sdks:java:io:synthetic")
+ compile project(":sdks:java:testing:test-utils")
+ compile project(":sdks:java:io:google-cloud-platform")
+ compile project(":sdks:java:io:kafka")
+ compile project(":sdks:java:io:kinesis")
- gradleRun project(path: project.path, configuration: "shadow")
- gradleRun project(path: runnerDependency, configuration: "shadow")
+ gradleRun project(project.path)
+ gradleRun project(runnerDependency)
// The Spark runner requires the user to provide a Spark dependency. For self-contained
// runs with the Spark runner, we can provide such a dependency. This is deliberately phrased
diff --git a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
index dfcec93..ec6b96b 100644
--- a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
+++ b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
@@ -115,8 +115,7 @@
.apply("CoGroupByKey", CoGroupByKey.create())
.apply("Ungroup and reiterate", ParDo.of(new UngroupAndReiterate(options.getIterations())))
.apply(
- "Collect total bytes",
- ParDo.of(new ByteMonitor<>(METRICS_NAMESPACE, "totalBytes.count")))
+ "Collect total bytes", ParDo.of(new ByteMonitor(METRICS_NAMESPACE, "totalBytes.count")))
.apply("Collect end time metrics", ParDo.of(runtimeMonitor));
}
diff --git a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
index 11c1579..17b0fec 100644
--- a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
+++ b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
@@ -38,7 +38,7 @@
import org.apache.beam.sdk.transforms.Top;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
/**
* Load test for {@link ParDo} operation.
@@ -124,7 +124,7 @@
ParDo.of(new TimeMonitor<>(METRICS_NAMESPACE, "runtime")))
.apply(
"Collect metrics",
- ParDo.of(new ByteMonitor<>(METRICS_NAMESPACE, "totalBytes.count")));
+ ParDo.of(new ByteMonitor(METRICS_NAMESPACE, "totalBytes.count")));
input = applyWindowing(input);
diff --git a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/GroupByKeyLoadTest.java b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/GroupByKeyLoadTest.java
index 77b7970..4a569f7 100644
--- a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/GroupByKeyLoadTest.java
+++ b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/GroupByKeyLoadTest.java
@@ -88,7 +88,7 @@
.apply("Collect start time metrics", ParDo.of(runtimeMonitor))
.apply(
"Total bytes monitor",
- ParDo.of(new ByteMonitor<>(METRICS_NAMESPACE, "totalBytes.count")));
+ ParDo.of(new ByteMonitor(METRICS_NAMESPACE, "totalBytes.count")));
input = applyWindowing(input);
diff --git a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTest.java b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTest.java
index 5b787e6..40a172c 100644
--- a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTest.java
+++ b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTest.java
@@ -47,7 +47,7 @@
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java
index 524b0a6..115089c 100644
--- a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java
+++ b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java
@@ -21,7 +21,7 @@
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.testutils.TestResult;
import org.apache.beam.sdk.testutils.metrics.MetricsReader;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** POJO that represents load test results. */
public class LoadTestResult implements TestResult {
diff --git a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/ParDoLoadTest.java b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/ParDoLoadTest.java
index 27be5f4..633e7fb 100644
--- a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/ParDoLoadTest.java
+++ b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/ParDoLoadTest.java
@@ -85,7 +85,7 @@
pipeline
.apply("Read input", readFromSource(sourceOptions))
.apply(ParDo.of(runtimeMonitor))
- .apply(ParDo.of(new ByteMonitor<>(METRICS_NAMESPACE, "totalBytes.count")));
+ .apply(ParDo.of(new ByteMonitor(METRICS_NAMESPACE, "totalBytes.count")));
for (int i = 0; i < options.getIterations(); i++) {
input =
diff --git a/sdks/java/testing/nexmark/build.gradle b/sdks/java/testing/nexmark/build.gradle
index 5344631..1fdfbed 100644
--- a/sdks/java/testing/nexmark/build.gradle
+++ b/sdks/java/testing/nexmark/build.gradle
@@ -17,8 +17,11 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-nexmark'
-applyJavaNature(testShadowJar: true, exportJavadoc: false)
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.sdk.nexmark',
+ exportJavadoc: false,
+ archivesBaseName: 'beam-sdks-java-nexmark'
+)
description = "Apache Beam :: SDKs :: Java :: Nexmark"
@@ -50,34 +53,32 @@
}
dependencies {
- shadow library.java.vendored_guava_20_0
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
- shadow project(path: ":sdks:java:extensions:sql", configuration: "shadow")
- shadow project(path: ":sdks:java:io:kafka", configuration: "shadow")
- shadow project(path: ":sdks:java:testing:test-utils", configuration: "shadow")
- shadow library.java.google_api_services_bigquery
- shadow library.java.jackson_core
- shadow library.java.jackson_annotations
- shadow library.java.jackson_databind
- shadow library.java.avro
- shadow library.java.joda_time
- shadow library.java.slf4j_api
- shadow library.java.commons_lang3
- shadow library.java.kafka_clients
- shadow project(path: ":runners:direct-java", configuration: "shadow")
+ compile library.java.vendored_guava_26_0_jre
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile project(":sdks:java:io:google-cloud-platform")
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
+ compile project(":sdks:java:extensions:sql")
+ compile project(":sdks:java:io:kafka")
+ compile project(":sdks:java:testing:test-utils")
+ compile library.java.google_api_services_bigquery
+ compile library.java.jackson_core
+ compile library.java.jackson_annotations
+ compile library.java.jackson_databind
+ compile library.java.avro
+ compile library.java.joda_time
+ compile library.java.slf4j_api
+ compile library.java.commons_lang3
+ compile library.java.kafka_clients
provided library.java.junit
provided library.java.hamcrest_core
- shadow project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadow")
- shadowTestRuntimeClasspath library.java.slf4j_jdk14
- shadowTest project(path: ":sdks:java:io:google-cloud-platform", configuration: "shadowTest")
- shadowTest project(path: ":sdks:java:testing:test-utils", configuration: "shadowTest")
+ testRuntimeClasspath library.java.slf4j_jdk14
+ testCompile project(path: ":sdks:java:io:google-cloud-platform", configuration: "testRuntime")
+ testCompile project(path: ":sdks:java:testing:test-utils", configuration: "testRuntime")
testCompile library.java.hamcrest_core
testCompile library.java.hamcrest_library
- gradleRun project(path: project.path, configuration: "shadow")
- gradleRun project(path: nexmarkRunnerDependency, configuration: "shadow")
+ gradleRun project(project.path)
+ gradleRun project(nexmarkRunnerDependency)
// The Spark runner requires the user to provide a Spark dependency. For self-contained
// runs with the Spark runner, we can provide such a dependency. This is deliberately phrased
@@ -101,7 +102,7 @@
//
// Parameters:
// -Pnexmark.runner
-// Specify a runner subproject, such as ":runners:spark" or ":runners:flink:1.5"
+// Specify a runner subproject, such as ":runners:spark" or ":runners:flink:1.8"
// Defaults to ":runners:direct-java"
//
// -Pnexmark.args
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
index f65a8d4..6039658 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
@@ -39,8 +39,8 @@
import org.apache.beam.sdk.nexmark.model.Person;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.testutils.publishing.BigQueryResultsPublisher;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
index 419d151..12109ec 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
@@ -18,8 +18,8 @@
package org.apache.beam.sdk.nexmark;
import static org.apache.beam.sdk.nexmark.NexmarkUtils.PubSubMode.COMBINED;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
@@ -93,10 +93,10 @@
import org.apache.beam.sdk.values.TimestampedValue;
import org.apache.beam.sdk.values.TupleTag;
import org.apache.beam.sdk.values.TupleTagList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
import org.apache.kafka.common.serialization.ByteArraySerializer;
import org.apache.kafka.common.serialization.LongDeserializer;
@@ -499,6 +499,7 @@
boolean running = true;
switch (state) {
case UNKNOWN:
+ case UNRECOGNIZED:
case STOPPED:
case RUNNING:
// Keep going.
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java
index 1d0e019..ca6f693 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
@@ -77,10 +77,10 @@
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Splitter;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Strings;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.hash.Hashing;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.hash.Hashing;
import org.joda.time.Duration;
import org.joda.time.Instant;
import org.slf4j.Logger;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Auction.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Auction.java
index 7d08c3c..6d4ac9d 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Auction.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Auction.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.nexmark.NexmarkUtils;
import org.apache.beam.sdk.schemas.JavaFieldSchema;
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
import org.joda.time.Instant;
/** An auction submitted by a person. */
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/AuctionBid.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/AuctionBid.java
index ff1080c..f4bc7ee 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/AuctionBid.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/AuctionBid.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.CustomCoder;
import org.apache.beam.sdk.nexmark.NexmarkUtils;
import org.apache.beam.sdk.nexmark.queries.WinningBids;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Result of {@link WinningBids} transform. */
public class AuctionBid implements KnownSize, Serializable {
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/BidsPerSession.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/BidsPerSession.java
index fdfcd35..995f3d7 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/BidsPerSession.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/BidsPerSession.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.CustomCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.nexmark.NexmarkUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Result of query 11. */
public class BidsPerSession implements KnownSize, Serializable {
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/CategoryPrice.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/CategoryPrice.java
index 5442ce1..bea201b 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/CategoryPrice.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/CategoryPrice.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.nexmark.NexmarkUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Result of Query4. */
public class CategoryPrice implements KnownSize, Serializable {
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Done.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Done.java
index b6f88c8..f730b02 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Done.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Done.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.CustomCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.nexmark.NexmarkUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Result of query 10. */
public class Done implements KnownSize, Serializable {
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java
index c2345c7..6d9d3e9 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Event.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.schemas.JavaFieldSchema;
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/**
* An event in the auction system, either a (new) {@link Person}, a (new) {@link Auction}, or a
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/IdNameReserve.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/IdNameReserve.java
index ce8f09f..165452a 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/IdNameReserve.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/IdNameReserve.java
@@ -29,7 +29,7 @@
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.nexmark.NexmarkUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Result type of Query8. */
public class IdNameReserve implements KnownSize, Serializable {
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Person.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Person.java
index 2decc70..3634278 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Person.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/Person.java
@@ -32,7 +32,7 @@
import org.apache.beam.sdk.nexmark.NexmarkUtils;
import org.apache.beam.sdk.schemas.JavaFieldSchema;
import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
import org.joda.time.Instant;
/** A person either creating an auction or making a bid. */
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/SellerPrice.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/SellerPrice.java
index ce7c6d4..1789473 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/SellerPrice.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/SellerPrice.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.coders.CustomCoder;
import org.apache.beam.sdk.coders.VarLongCoder;
import org.apache.beam.sdk.nexmark.NexmarkUtils;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Objects;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Objects;
/** Result of Query6. */
public class SellerPrice implements KnownSize, Serializable {
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
deleted file mode 100644
index eb58c3e..0000000
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
+++ /dev/null
@@ -1,83 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.nexmark.model.sql;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CustomCoder;
-import org.apache.beam.sdk.coders.RowCoder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.nexmark.model.KnownSize;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.values.Row;
-
-/**
- * {@link KnownSize} implementation to estimate the size of a {@link Row}, similar to Java model.
- * NexmarkLauncher/Queries infrastructure expects the events to be able to quickly provide the
- * estimates of their sizes.
- *
- * <p>The {@link Row} size is calculated at creation time.
- */
-public class RowSize implements KnownSize {
- private static final Coder<Long> LONG_CODER = VarLongCoder.of();
- public static final Coder<RowSize> CODER =
- new CustomCoder<RowSize>() {
- @Override
- public void encode(RowSize rowSize, OutputStream outStream) throws IOException {
-
- LONG_CODER.encode(rowSize.sizeInBytes(), outStream);
- }
-
- @Override
- public RowSize decode(InputStream inStream) throws IOException {
- return new RowSize(LONG_CODER.decode(inStream));
- }
- };
-
- public static ParDo.SingleOutput<Row, RowSize> parDo() {
- return ParDo.of(
- new DoFn<Row, RowSize>() {
- @ProcessElement
- public void processElement(ProcessContext c) {
- c.output(RowSize.of(c.element()));
- }
- });
- }
-
- public static RowSize of(Row row) {
- return new RowSize(sizeInBytes(row));
- }
-
- private static long sizeInBytes(Row row) {
- return RowCoder.estimatedSizeBytes(row);
- }
-
- private long sizeInBytes;
-
- private RowSize(long sizeInBytes) {
- this.sizeInBytes = sizeInBytes;
- }
-
- @Override
- public long sizeInBytes() {
- return sizeInBytes;
- }
-}
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoin.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoin.java
index 9a13bc3..6f96a40 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoin.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoin.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark.queries;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Map;
import org.apache.beam.sdk.nexmark.NexmarkConfiguration;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3Model.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3Model.java
index 0ec98dc..54a19d8 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3Model.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query3Model.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.nexmark.model.NameCityStateId;
import org.apache.beam.sdk.nexmark.model.Person;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.joda.time.Instant;
/** A direct implementation of {@link Query3}. */
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6.java
index 1fbd060..5570563 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query6.java
@@ -35,7 +35,7 @@
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Lists;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.joda.time.Duration;
/**
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query8Model.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query8Model.java
index 46f2685..789cc08 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query8Model.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query8Model.java
@@ -29,8 +29,8 @@
import org.apache.beam.sdk.nexmark.model.IdNameReserve;
import org.apache.beam.sdk.nexmark.model.Person;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ArrayListMultimap;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Multimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ArrayListMultimap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Multimap;
import org.joda.time.Duration;
import org.joda.time.Instant;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoin.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoin.java
index aafdc73..5d46530 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoin.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoin.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark.queries;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.Map;
import org.apache.beam.sdk.nexmark.NexmarkConfiguration;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinModel.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinModel.java
index b327141..c81f352 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinModel.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinModel.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark.queries;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import java.util.ArrayList;
import java.util.Collection;
@@ -31,8 +31,8 @@
import org.apache.beam.sdk.nexmark.model.Bid;
import org.apache.beam.sdk.nexmark.model.Event;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableSet;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Ordering;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Ordering;
import org.joda.time.Instant;
/** A direct implementation of {@link SessionSideInputJoin}. */
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/WinningBids.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/WinningBids.java
index 580baa4..2f66078 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/WinningBids.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/WinningBids.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark.queries;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import com.fasterxml.jackson.annotation.JsonCreator;
import java.io.IOException;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java
index 25da89f..806b0db 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoin.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark.queries.sql;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState;
import org.apache.beam.sdk.extensions.sql.SqlTransform;
import org.apache.beam.sdk.nexmark.NexmarkConfiguration;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5.java
index 680f64c..011132b 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5.java
@@ -33,7 +33,7 @@
import org.apache.beam.sdk.values.PInput;
import org.apache.beam.sdk.values.Row;
import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Joiner;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Joiner;
/**
* Query 5, 'Hot Items'. Which auctions have seen the most bids in the last hour (updated every
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/Generator.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/Generator.java
index 98a47b7..d3e111a 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/Generator.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/Generator.java
@@ -20,7 +20,7 @@
import static org.apache.beam.sdk.nexmark.sources.generator.model.AuctionGenerator.nextAuction;
import static org.apache.beam.sdk.nexmark.sources.generator.model.BidGenerator.nextBid;
import static org.apache.beam.sdk.nexmark.sources.generator.model.PersonGenerator.nextPerson;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
import java.io.Serializable;
import java.util.Iterator;
diff --git a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/GeneratorCheckpoint.java b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/GeneratorCheckpoint.java
index 837906b..18286b1 100644
--- a/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/GeneratorCheckpoint.java
+++ b/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/sources/generator/GeneratorCheckpoint.java
@@ -17,7 +17,7 @@
*/
package org.apache.beam.sdk.nexmark.sources.generator;
-import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.MoreObjects.toStringHelper;
+import static org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.MoreObjects.toStringHelper;
import java.io.IOException;
import java.io.InputStream;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/NexmarkUtilsTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/NexmarkUtilsTest.java
index df1927d..3efba39 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/NexmarkUtilsTest.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/NexmarkUtilsTest.java
@@ -21,7 +21,7 @@
import static org.apache.beam.sdk.nexmark.NexmarkUtils.ResourceNameMode.QUERY_AND_SALT;
import static org.apache.beam.sdk.nexmark.NexmarkUtils.ResourceNameMode.QUERY_RUNNER_AND_MODE;
import static org.apache.beam.sdk.nexmark.NexmarkUtils.ResourceNameMode.VERBATIM;
-import static org.testng.Assert.assertEquals;
+import static org.junit.Assert.assertEquals;
import java.util.Random;
import java.util.stream.Collectors;
@@ -78,13 +78,13 @@
@Test
public void testFullQueryNameAppendsLanguageIfNeeded() {
String fullName = NexmarkUtils.fullQueryName("sql", "1");
- assertEquals(fullName, "1_sql");
+ assertEquals("1_sql", fullName);
}
@Test
public void testFullQueryNameDoesntContainNullLanguage() {
String fullName = NexmarkUtils.fullQueryName(null, "1");
- assertEquals(fullName, "1");
+ assertEquals("1", fullName);
}
@Test
@@ -149,7 +149,7 @@
String version,
Class runner,
Boolean isStreaming,
- String expected) {
+ final String expected) {
NexmarkOptions options = PipelineOptionsFactory.as(NexmarkOptions.class);
options.setResourceNameMode(nameMode);
options.setBigQueryTable(baseTableName);
@@ -158,7 +158,7 @@
String tableName = NexmarkUtils.tableName(options, queryName, salt, version);
- assertEquals(tableName, expected);
+ assertEquals(expected, tableName);
}
private static class Runner extends PipelineRunner<PipelineResult> {
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
deleted file mode 100644
index 5281787..0000000
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.nexmark.model.sql;
-
-import static org.hamcrest.core.IsEqual.equalTo;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
-
-import java.math.BigDecimal;
-import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.SchemaCoder;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.TestStream;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.transforms.SerializableFunctions;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
-import org.joda.time.DateTime;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-
-/** Unit tests for {@link RowSize}. */
-public class RowSizeTest {
-
- private static final Schema ROW_TYPE =
- Schema.builder()
- .addByteField("f_tinyint")
- .addInt16Field("f_smallint")
- .addInt32Field("f_int")
- .addInt64Field("f_bigint")
- .addFloatField("f_float")
- .addDoubleField("f_double")
- .addDecimalField("f_decimal")
- .addBooleanField("f_boolean")
- .addField("f_time", CalciteUtils.TIME)
- .addField("f_date", CalciteUtils.DATE)
- .addDateTimeField("f_timestamp")
- .addField("f_char", CalciteUtils.CHAR)
- .addField("f_varchar", CalciteUtils.VARCHAR)
- .build();
-
- private static final long ROW_SIZE = 96L;
-
- private static final Row ROW =
- Row.withSchema(ROW_TYPE)
- .addValues(
- (byte) 1,
- (short) 2,
- (int) 3,
- (long) 4,
- (float) 5.12,
- (double) 6.32,
- new BigDecimal(7),
- false,
- new DateTime().withDate(2019, 03, 02),
- new DateTime(10L),
- new DateTime(11L),
- "12",
- "13")
- .build();
-
- @Rule public TestPipeline testPipeline = TestPipeline.create();
- @Rule public ExpectedException thrown = ExpectedException.none();
-
- @Test
- public void testCalculatesCorrectSize() throws Exception {
- assertEquals(ROW_SIZE, RowSize.of(ROW).sizeInBytes());
- }
-
- @Test
- public void testParDoConvertsToRecordSize() throws Exception {
- PCollection<Row> rows =
- testPipeline.apply(
- TestStream.create(
- SchemaCoder.of(
- ROW_TYPE,
- SerializableFunctions.identity(),
- SerializableFunctions.identity()))
- .addElements(ROW)
- .advanceWatermarkToInfinity());
-
- PAssert.that(rows).satisfies(new CorrectSize());
-
- testPipeline.run();
- }
-
- static class CorrectSize implements SerializableFunction<Iterable<Row>, Void> {
- @Override
- public Void apply(Iterable<Row> input) {
- RowSize recordSize = RowSize.of(Iterables.getOnlyElement(input));
- assertThat(recordSize.sizeInBytes(), equalTo(ROW_SIZE));
- return null;
- }
- }
-}
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoinTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoinTest.java
index 1d7295f..ee80ee4 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoinTest.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/BoundedSideInputJoinTest.java
@@ -39,7 +39,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinTest.java
index 2b3cef0..2842358 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinTest.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/SessionSideInputJoinTest.java
@@ -41,7 +41,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoinTest.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoinTest.java
index 3f689fb..2198973 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoinTest.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlBoundedSideInputJoinTest.java
@@ -44,7 +44,7 @@
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionList;
import org.apache.beam.sdk.values.TimestampedValue;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery2Test.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery2Test.java
index 022e54b..ff14fc6 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery2Test.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery2Test.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery3Test.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery3Test.java
index 7663660..e9eb4b3 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery3Test.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery3Test.java
@@ -27,7 +27,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5Test.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5Test.java
index d4d109a..31d48a9 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5Test.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery5Test.java
@@ -28,7 +28,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Ignore;
import org.junit.Rule;
diff --git a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery7Test.java b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery7Test.java
index 90d9eb0..8aa4256 100644
--- a/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery7Test.java
+++ b/sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/queries/sql/SqlQuery7Test.java
@@ -25,7 +25,7 @@
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import org.joda.time.Instant;
import org.junit.Rule;
import org.junit.Test;
diff --git a/sdks/java/testing/test-utils/build.gradle b/sdks/java/testing/test-utils/build.gradle
index a073b5d..45b007d 100644
--- a/sdks/java/testing/test-utils/build.gradle
+++ b/sdks/java/testing/test-utils/build.gradle
@@ -17,20 +17,23 @@
*/
plugins { id 'org.apache.beam.module' }
-archivesBaseName = 'beam-sdks-java-test-utils'
-applyJavaNature(exportJavadoc: false)
+applyJavaNature(
+ exportJavadoc: false,
+ automaticModuleName: 'org.apache.beam.sdk.testutils',
+ archivesBaseName: 'beam-sdks-java-test-utils'
+)
description = "Apache Beam :: SDKs :: Java :: Test Utils"
dependencies {
- shadow project(path: ":sdks:java:core", configuration: "shadow")
- shadow library.java.vendored_guava_20_0
- shadow library.java.google_cloud_bigquery
- shadow project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "shadow")
+ compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.vendored_guava_26_0_jre
+ compile library.java.google_cloud_bigquery
+ compile project(":sdks:java:extensions:google-cloud-platform-core")
- shadowTest library.java.junit
- shadowTest library.java.mockito_core
- shadowTest library.java.hamcrest_core
- shadowTest library.java.hamcrest_library
+ testCompile library.java.junit
+ testCompile library.java.mockito_core
+ testCompile library.java.hamcrest_core
+ testCompile library.java.hamcrest_library
testRuntimeOnly project(path: ":runners:direct-java", configuration: "shadowTest")
}
diff --git a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java
index 8967064..63be56d 100644
--- a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java
+++ b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java
@@ -19,7 +19,7 @@
import com.google.cloud.bigquery.LegacySQLTypeName;
import java.util.Map;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/**
* Represents a schema and corresponding test result. Each test may have multiple named results
diff --git a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/ByteMonitor.java b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/ByteMonitor.java
index 0e0588a..fcaaca8 100644
--- a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/ByteMonitor.java
+++ b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/ByteMonitor.java
@@ -17,10 +17,10 @@
*/
package org.apache.beam.sdk.testutils.metrics;
-import jdk.nashorn.internal.ir.debug.ObjectSizeCalculator;
import org.apache.beam.sdk.metrics.Counter;
import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.values.KV;
/**
* Monitor that records the number of bytes flowing through a PCollection.
@@ -29,7 +29,7 @@
* flew through this DoFn. Such information can be then collected and written out and queried using
* {@link org.apache.beam.sdk.testutils.metrics.MetricsReader}.
*/
-public class ByteMonitor<T> extends DoFn<T, T> {
+public class ByteMonitor extends DoFn<KV<byte[], byte[]>, KV<byte[], byte[]>> {
private Counter totalBytes;
@@ -39,7 +39,7 @@
@ProcessElement
public void processElement(ProcessContext c) {
- totalBytes.inc(ObjectSizeCalculator.getObjectSize(c.element()));
+ totalBytes.inc(c.element().getKey().length + c.element().getValue().length);
c.output(c.element());
}
}
diff --git a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/CountMonitor.java b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/CountMonitor.java
deleted file mode 100644
index a20b125..0000000
--- a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/CountMonitor.java
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.testutils.metrics;
-
-import org.apache.beam.sdk.metrics.Counter;
-import org.apache.beam.sdk.metrics.Metrics;
-import org.apache.beam.sdk.transforms.DoFn;
-
-/**
- * Monitor that records number of elements flowing through a pipeline
- *
- * <p>To use: apply a monitor in a desired place in the pipeline. This will capture how many
- * elements * flew through this DoFn. Such information can be then collected and written out and
- * queried using * {@link org.apache.beam.sdk.testutils.metrics.MetricsReader}.
- */
-public class CountMonitor<T> extends DoFn<T, T> {
- private Counter counter;
-
- public CountMonitor(String namespace, String name) {
- this.counter = Metrics.counter(namespace, name);
- }
-
- @ProcessElement
- public void processElement(ProcessContext context) {
- counter.inc();
- context.output(context.element());
- }
-}
diff --git a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/IOITMetrics.java b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/IOITMetrics.java
index ca9790f..f7b1559 100644
--- a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/IOITMetrics.java
+++ b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/IOITMetrics.java
@@ -34,8 +34,8 @@
private final Set<Function<MetricsReader, NamedTestResult>> metricSuppliers;
private final PipelineResult result;
private final String namespace;
- private String uuid;
- private String timestamp;
+ private final String uuid;
+ private final String timestamp;
public IOITMetrics(
Set<Function<MetricsReader, NamedTestResult>> metricSuppliers,
@@ -53,10 +53,21 @@
public void publish(String bigQueryDataset, String bigQueryTable) {
MetricsReader reader = new MetricsReader(result, namespace);
Collection<NamedTestResult> namedTestResults = reader.readAll(metricSuppliers);
+
+ publish(uuid, timestamp, bigQueryDataset, bigQueryTable, namedTestResults);
+ }
+
+ public static void publish(
+ String uuid,
+ String timestamp,
+ String bigQueryDataset,
+ String bigQueryTable,
+ Collection<NamedTestResult> results) {
+
if (bigQueryDataset != null && bigQueryTable != null) {
BigQueryResultsPublisher.create(bigQueryDataset, NamedTestResult.getSchema())
- .publish(namedTestResults, bigQueryTable);
+ .publish(results, bigQueryTable);
}
- ConsoleResultPublisher.publish(namedTestResults, uuid, timestamp);
+ ConsoleResultPublisher.publish(results, uuid, timestamp);
}
}
diff --git a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
index 17cef6f..986d568 100644
--- a/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
+++ b/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/MetricsReader.java
@@ -31,9 +31,9 @@
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricsFilter;
import org.apache.beam.sdk.testutils.NamedTestResult;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
import org.joda.time.Duration;
import org.slf4j.LoggerFactory;
diff --git a/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisherTest.java b/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisherTest.java
index c75cc72..07a4035 100644
--- a/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisherTest.java
+++ b/sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/publishing/BigQueryResultsPublisherTest.java
@@ -25,7 +25,7 @@
import java.util.Map;
import org.apache.beam.sdk.testutils.TestResult;
import org.apache.beam.sdk.testutils.fakes.FakeBigQueryClient;
-import org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableMap;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
diff --git a/sdks/python/apache_beam/__init__.py b/sdks/python/apache_beam/__init__.py
index e06cc69..2082250 100644
--- a/sdks/python/apache_beam/__init__.py
+++ b/sdks/python/apache_beam/__init__.py
@@ -79,11 +79,11 @@
import warnings
-if sys.version_info[0] == 3:
+if sys.version_info[0] == 2 and sys.version_info[1] == 7:
warnings.warn(
- 'Running the Apache Beam SDK on Python 3 is not yet fully supported. '
- 'You may encounter buggy behavior or missing features.')
-elif sys.version_info[0] == 2 and sys.version_info[1] == 7:
+ 'You are using Apache Beam with Python 2. '
+ 'New releases of Apache Beam will soon support Python 3 only.')
+elif sys.version_info[0] == 3:
pass
else:
raise RuntimeError(
diff --git a/sdks/python/apache_beam/coders/avro_coder.py b/sdks/python/apache_beam/coders/avro_coder.py
deleted file mode 100644
index f0bb24f..0000000
--- a/sdks/python/apache_beam/coders/avro_coder.py
+++ /dev/null
@@ -1,99 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-"""Coder for AvroRecord serialization/deserialization."""
-
-from __future__ import absolute_import
-
-import json
-from io import BytesIO
-
-from fastavro import parse_schema
-from fastavro import schemaless_reader
-from fastavro import schemaless_writer
-
-from apache_beam.coders.coder_impl import SimpleCoderImpl
-from apache_beam.coders.coders import Coder
-from apache_beam.coders.coders import FastCoder
-
-AVRO_CODER_URN = "beam:coder:avro:v1"
-
-__all__ = ['AvroCoder', 'AvroRecord']
-
-
-class AvroCoder(FastCoder):
- """A coder used for AvroRecord values."""
-
- def __init__(self, schema):
- self.schema = schema
-
- def _create_impl(self):
- return AvroCoderImpl(self.schema)
-
- def is_deterministic(self):
- # TODO: need to confirm if it's deterministic
- return False
-
- def __eq__(self, other):
- return (type(self) == type(other)
- and self.schema == other.schema)
-
- def __hash__(self):
- return hash(self.schema)
-
- def to_type_hint(self):
- return AvroRecord
-
- def to_runner_api_parameter(self, context):
- return AVRO_CODER_URN, self.schema, ()
-
- @Coder.register_urn(AVRO_CODER_URN, bytes)
- def from_runner_api_parameter(payload, unused_components, unused_context):
- return AvroCoder(payload)
-
-
-class AvroCoderImpl(SimpleCoderImpl):
- """For internal use only; no backwards-compatibility guarantees."""
-
- def __init__(self, schema):
- self.parsed_schema = parse_schema(json.loads(schema))
-
- def encode(self, value):
- assert issubclass(type(value), AvroRecord)
- with BytesIO() as buf:
- schemaless_writer(buf, self.parsed_schema, value.record)
- return buf.getvalue()
-
- def decode(self, encoded):
- with BytesIO(encoded) as buf:
- return AvroRecord(schemaless_reader(buf, self.parsed_schema))
-
-
-class AvroRecord(object):
- """Simple wrapper class for dictionary records."""
-
- def __init__(self, value):
- self.record = value
-
- def __eq__(self, other):
- return (
- issubclass(type(other), AvroRecord) and
- self.record == other.record
- )
-
- def __hash__(self):
- return hash(self.record)
diff --git a/sdks/python/apache_beam/coders/avro_coder_test.py b/sdks/python/apache_beam/coders/avro_coder_test.py
deleted file mode 100644
index 3f060a6..0000000
--- a/sdks/python/apache_beam/coders/avro_coder_test.py
+++ /dev/null
@@ -1,71 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-from __future__ import absolute_import
-
-import logging
-import unittest
-
-from apache_beam.coders.avro_coder import AvroCoder
-from apache_beam.coders.avro_coder import AvroRecord
-from apache_beam.coders.typecoders import registry as coders_registry
-
-
-class AvroTestCoder(AvroCoder):
- SCHEMA = """
- {
- "type": "record", "name": "testrecord",
- "fields": [
- {"name": "name", "type": "string"},
- {"name": "age", "type": "int"}
- ]
- }
- """
-
- def __init__(self):
- super(AvroTestCoder, self).__init__(self.SCHEMA)
-
-
-class AvroTestRecord(AvroRecord):
- pass
-
-
-coders_registry.register_coder(AvroTestRecord, AvroTestCoder)
-
-
-class CodersTest(unittest.TestCase):
-
- def test_avro_record_coder(self):
- real_coder = coders_registry.get_coder(AvroTestRecord)
- expected_coder = AvroTestCoder()
- self.assertEqual(
- real_coder.encode(
- AvroTestRecord({"name": "Daenerys targaryen", "age": 23})),
- expected_coder.encode(
- AvroTestRecord({"name": "Daenerys targaryen", "age": 23}))
- )
- self.assertEqual(
- AvroTestRecord({"name": "Jon Snow", "age": 23}),
- real_coder.decode(
- real_coder.encode(
- AvroTestRecord({"name": "Jon Snow", "age": 23}))
- )
- )
-
-
-if __name__ == '__main__':
- logging.getLogger().setLevel(logging.INFO)
- unittest.main()
diff --git a/sdks/python/apache_beam/coders/avro_record.py b/sdks/python/apache_beam/coders/avro_record.py
new file mode 100644
index 0000000..e65057b
--- /dev/null
+++ b/sdks/python/apache_beam/coders/avro_record.py
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""AvroRecord for AvroCoder."""
+
+from __future__ import absolute_import
+
+__all__ = ['AvroRecord']
+
+
+class AvroRecord(object):
+ """Simple wrapper class for dictionary records."""
+
+ def __init__(self, value):
+ self.record = value
+
+ def __eq__(self, other):
+ return (
+ issubclass(type(other), AvroRecord) and
+ self.record == other.record
+ )
+
+ def __hash__(self):
+ return hash(self.record)
diff --git a/sdks/python/apache_beam/coders/coder_impl.pxd b/sdks/python/apache_beam/coders/coder_impl.pxd
index c5ce4e8..e4b2832 100644
--- a/sdks/python/apache_beam/coders/coder_impl.pxd
+++ b/sdks/python/apache_beam/coders/coder_impl.pxd
@@ -92,6 +92,10 @@
pass
+cdef class BooleanCoderImpl(CoderImpl):
+ pass
+
+
cdef class FloatCoderImpl(StreamCoderImpl):
pass
diff --git a/sdks/python/apache_beam/coders/coder_impl.py b/sdks/python/apache_beam/coders/coder_impl.py
index 45e9b4a..14705df 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -34,13 +34,19 @@
from __future__ import absolute_import
from __future__ import division
+import json
from builtins import chr
from builtins import object
+from io import BytesIO
+from fastavro import parse_schema
+from fastavro import schemaless_reader
+from fastavro import schemaless_writer
from past.builtins import unicode as past_unicode
from past.builtins import long
from apache_beam.coders import observable
+from apache_beam.coders.avro_record import AvroRecord
from apache_beam.utils import windowed_value
from apache_beam.utils.timestamp import MAX_TIMESTAMP
from apache_beam.utils.timestamp import MIN_TIMESTAMP
@@ -440,6 +446,38 @@
return encoded
+class BooleanCoderImpl(CoderImpl):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ A coder for bool objects."""
+
+ def encode_to_stream(self, value, out, nested):
+ out.write_byte(1 if value else 0)
+
+ def decode_from_stream(self, in_stream, nested):
+ value = in_stream.read_byte()
+ if value is 0:
+ return False
+ elif value is 1:
+ return True
+ raise ValueError("Expected 0 or 1, got %s" % value)
+
+ def encode(self, value):
+ return b'\x01' if value else b'\x00'
+
+ def decode(self, encoded):
+ value = ord(encoded)
+ if value is 0:
+ return False
+ elif value is 1:
+ return True
+ raise ValueError("Expected 0 or 1, got %s" % value)
+
+ def estimate_size(self, unused_value, nested=False):
+ # Note that booleans are encoded the same way regardless of nesting.
+ return 1
+
+
class FloatCoderImpl(StreamCoderImpl):
"""For internal use only; no backwards-compatibility guarantees."""
@@ -657,6 +695,23 @@
return estimated_size, observables
+class AvroCoderImpl(SimpleCoderImpl):
+ """For internal use only; no backwards-compatibility guarantees."""
+
+ def __init__(self, schema):
+ self.parsed_schema = parse_schema(json.loads(schema))
+
+ def encode(self, value):
+ assert issubclass(type(value), AvroRecord)
+ with BytesIO() as buf:
+ schemaless_writer(buf, self.parsed_schema, value.record)
+ return buf.getvalue()
+
+ def decode(self, encoded):
+ with BytesIO(encoded) as buf:
+ return AvroRecord(schemaless_reader(buf, self.parsed_schema))
+
+
class TupleCoderImpl(AbstractComponentCoderImpl):
"""A coder for tuple objects."""
diff --git a/sdks/python/apache_beam/coders/coders.py b/sdks/python/apache_beam/coders/coders.py
index df3ac6f..35020b6 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -23,6 +23,7 @@
import base64
import sys
+import typing
from builtins import object
import google.protobuf.wrappers_pb2
@@ -30,6 +31,7 @@
from past.builtins import unicode
from apache_beam.coders import coder_impl
+from apache_beam.coders.avro_record import AvroRecord
from apache_beam.portability import common_urns
from apache_beam.portability import python_urns
from apache_beam.portability.api import beam_runner_api_pb2
@@ -56,11 +58,14 @@
import dill
-__all__ = ['Coder',
- 'BytesCoder', 'DillCoder', 'FastPrimitivesCoder', 'FloatCoder',
- 'IterableCoder', 'PickleCoder', 'ProtoCoder', 'SingletonCoder',
- 'StrUtf8Coder', 'TimestampCoder', 'TupleCoder',
- 'TupleSequenceCoder', 'VarIntCoder', 'WindowedValueCoder']
+__all__ = [
+ 'Coder',
+ 'AvroCoder', 'BooleanCoder', 'BytesCoder', 'DillCoder',
+ 'FastPrimitivesCoder', 'FloatCoder', 'IterableCoder', 'PickleCoder',
+ 'ProtoCoder', 'SingletonCoder', 'StrUtf8Coder', 'TimestampCoder',
+ 'TupleCoder', 'TupleSequenceCoder', 'VarIntCoder',
+ 'WindowedValueCoder'
+]
def serialize_coder(coder):
@@ -86,6 +91,14 @@
"""Decodes the given byte string into the corresponding object."""
raise NotImplementedError('Decode not implemented: %s.' % self)
+ def encode_nested(self, value):
+ """Uses the underlying implementation to encode in nested format."""
+ return self.get_impl().encode_nested(value)
+
+ def decode_nested(self, encoded):
+ """Uses the underlying implementation to decode in nested format."""
+ return self.get_impl().decode_nested(encoded)
+
def is_deterministic(self):
"""Whether this coder is guaranteed to encode values deterministically.
@@ -263,27 +276,24 @@
def to_runner_api(self, context):
urn, typed_param, components = self.to_runner_api_parameter(context)
return beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- environment_id=(
- context.default_environment_id() if context else None),
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=urn,
- payload=typed_param
- if isinstance(typed_param, (bytes, type(None)))
- else typed_param.SerializeToString())),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=urn,
+ payload=typed_param
+ if isinstance(typed_param, (bytes, type(None)))
+ else typed_param.SerializeToString()),
component_coder_ids=[context.coders.get_id(c) for c in components])
@classmethod
def from_runner_api(cls, coder_proto, context):
- """Converts from an SdkFunctionSpec to a Fn object.
+ """Converts from an FunctionSpec to a Fn object.
Prefer registering a urn with its parameter type and constructor.
"""
- parameter_type, constructor = cls._known_urns[coder_proto.spec.spec.urn]
+ parameter_type, constructor = cls._known_urns[coder_proto.spec.urn]
try:
return constructor(
proto_utils.parse_Bytes(
- coder_proto.spec.spec.payload, parameter_type),
+ coder_proto.spec.payload, parameter_type),
[context.coders.get_by_id(c)
for c in coder_proto.component_coder_ids],
context)
@@ -413,6 +423,26 @@
Coder.register_structured_urn(common_urns.coders.BYTES.urn, BytesCoder)
+class BooleanCoder(FastCoder):
+ def _create_impl(self):
+ return coder_impl.BooleanCoderImpl()
+
+ def is_deterministic(self):
+ return True
+
+ def to_type_hint(self):
+ return bool
+
+ def __eq__(self, other):
+ return type(self) == type(other)
+
+ def __hash__(self):
+ return hash(type(self))
+
+
+Coder.register_structured_urn(common_urns.coders.BOOL.urn, BooleanCoder)
+
+
class VarIntCoder(FastCoder):
"""Variable-length integer coder."""
@@ -600,7 +630,7 @@
return DeterministicFastPrimitivesCoder(self, step_label)
def to_type_hint(self):
- return typehints.Any
+ return typing.Any
class DillCoder(_PickleCoderBase):
@@ -634,7 +664,7 @@
return self
def to_type_hint(self):
- return typehints.Any
+ return typing.Any
class FastPrimitivesCoder(FastCoder):
@@ -659,7 +689,7 @@
return DeterministicFastPrimitivesCoder(self, step_label)
def to_type_hint(self):
- return typehints.Any
+ return typing.Any
def as_cloud_object(self, coders_context=None, is_pair_like=True):
value = super(FastCoder, self).as_cloud_object(coders_context)
@@ -789,6 +819,40 @@
return self
+AVRO_CODER_URN = "beam:coder:avro:v1"
+
+
+class AvroCoder(FastCoder):
+ """A coder used for AvroRecord values."""
+
+ def __init__(self, schema):
+ self.schema = schema
+
+ def _create_impl(self):
+ return coder_impl.AvroCoderImpl(self.schema)
+
+ def is_deterministic(self):
+ # TODO(BEAM-7903): need to confirm if it's deterministic
+ return False
+
+ def __eq__(self, other):
+ return (type(self) == type(other)
+ and self.schema == other.schema)
+
+ def __hash__(self):
+ return hash(self.schema)
+
+ def to_type_hint(self):
+ return AvroRecord
+
+ def to_runner_api_parameter(self, context):
+ return AVRO_CODER_URN, self.schema, ()
+
+ @Coder.register_urn(AVRO_CODER_URN, bytes)
+ def from_runner_api_parameter(payload, unused_components, unused_context):
+ return AvroCoder(payload)
+
+
class TupleCoder(FastCoder):
"""Coder of tuple objects."""
@@ -1190,4 +1254,4 @@
return self._proto
def to_type_hint(self):
- return typehints.Any
+ return typing.Any
diff --git a/sdks/python/apache_beam/coders/coders_test.py b/sdks/python/apache_beam/coders/coders_test.py
index c37df18..99994f1 100644
--- a/sdks/python/apache_beam/coders/coders_test.py
+++ b/sdks/python/apache_beam/coders/coders_test.py
@@ -23,6 +23,7 @@
from apache_beam.coders import proto2_coder_test_messages_pb2 as test_message
from apache_beam.coders import coders
+from apache_beam.coders.avro_record import AvroRecord
from apache_beam.coders.typecoders import registry as coders_registry
@@ -31,16 +32,16 @@
def test_basics(self):
v = ('a' * 10, 'b' * 90)
pickler = coders.PickleCoder()
- self.assertEquals(v, pickler.decode(pickler.encode(v)))
+ self.assertEqual(v, pickler.decode(pickler.encode(v)))
pickler = coders.Base64PickleCoder()
- self.assertEquals(v, pickler.decode(pickler.encode(v)))
- self.assertEquals(
+ self.assertEqual(v, pickler.decode(pickler.encode(v)))
+ self.assertEqual(
coders.Base64PickleCoder().encode(v),
base64.b64encode(coders.PickleCoder().encode(v)))
def test_equality(self):
- self.assertEquals(coders.PickleCoder(), coders.PickleCoder())
- self.assertEquals(coders.Base64PickleCoder(), coders.Base64PickleCoder())
+ self.assertEqual(coders.PickleCoder(), coders.PickleCoder())
+ self.assertEqual(coders.Base64PickleCoder(), coders.Base64PickleCoder())
self.assertNotEquals(coders.Base64PickleCoder(), coders.PickleCoder())
self.assertNotEquals(coders.Base64PickleCoder(), object())
@@ -112,6 +113,48 @@
self.assertEqual(coder.encode(mm_forward), coder.encode(mm_reverse))
+class AvroTestCoder(coders.AvroCoder):
+ SCHEMA = """
+ {
+ "type": "record", "name": "testrecord",
+ "fields": [
+ {"name": "name", "type": "string"},
+ {"name": "age", "type": "int"}
+ ]
+ }
+ """
+
+ def __init__(self):
+ super(AvroTestCoder, self).__init__(self.SCHEMA)
+
+
+class AvroTestRecord(AvroRecord):
+ pass
+
+
+coders_registry.register_coder(AvroTestRecord, AvroTestCoder)
+
+
+class AvroCoderTest(unittest.TestCase):
+
+ def test_avro_record_coder(self):
+ real_coder = coders_registry.get_coder(AvroTestRecord)
+ expected_coder = AvroTestCoder()
+ self.assertEqual(
+ real_coder.encode(
+ AvroTestRecord({"name": "Daenerys targaryen", "age": 23})),
+ expected_coder.encode(
+ AvroTestRecord({"name": "Daenerys targaryen", "age": 23}))
+ )
+ self.assertEqual(
+ AvroTestRecord({"name": "Jon Snow", "age": 23}),
+ real_coder.decode(
+ real_coder.encode(
+ AvroTestRecord({"name": "Jon Snow", "age": 23}))
+ )
+ )
+
+
class DummyClass(object):
"""A class with no registered coder."""
def __init__(self):
diff --git a/sdks/python/apache_beam/coders/coders_test_common.py b/sdks/python/apache_beam/coders/coders_test_common.py
index 94dfe8a..1b40b64 100644
--- a/sdks/python/apache_beam/coders/coders_test_common.py
+++ b/sdks/python/apache_beam/coders/coders_test_common.py
@@ -24,10 +24,9 @@
import unittest
from builtins import range
-import dill
-
from apache_beam.coders import proto2_coder_test_messages_pb2 as test_message
from apache_beam.coders import coders
+from apache_beam.internal import pickler
from apache_beam.runners import pipeline_context
from apache_beam.transforms import window
from apache_beam.transforms.window import GlobalWindow
@@ -68,6 +67,7 @@
if isinstance(c, type) and issubclass(c, coders.Coder) and
'Base' not in c.__name__)
standard -= set([coders.Coder,
+ coders.AvroCoder,
coders.DeterministicProtoCoder,
coders.FastCoder,
coders.ProtoCoder,
@@ -102,7 +102,7 @@
coder.get_impl().estimate_size(v))
self.assertEqual(coder.get_impl().get_estimated_size_and_observables(v),
(coder.get_impl().estimate_size(v), []))
- copy1 = dill.loads(dill.dumps(coder))
+ copy1 = pickler.loads(pickler.dumps(coder))
copy2 = coders.Coder.from_runner_api(coder.to_runner_api(context), context)
for v in values:
self.assertEqual(v, copy1.decode(copy2.encode(v)))
@@ -155,6 +155,9 @@
def test_bytes_coder(self):
self.check_coder(coders.BytesCoder(), b'a', b'\0', b'z' * 1000)
+ def test_bool_coder(self):
+ self.check_coder(coders.BooleanCoder(), True, False)
+
def test_varint_coder(self):
# Small ints.
self.check_coder(coders.VarIntCoder(), *range(-10, 10))
@@ -241,10 +244,11 @@
self.check_coder(
coders.TupleCoder(
(coders.TupleCoder((coders.PickleCoder(), coders.VarIntCoder())),
- coders.StrUtf8Coder())),
- ((1, 2), 'a'),
- ((-2, 5), u'a\u0101' * 100),
- ((300, 1), 'abc\0' * 5))
+ coders.StrUtf8Coder(),
+ coders.BooleanCoder())),
+ ((1, 2), 'a', True),
+ ((-2, 5), u'a\u0101' * 100, False),
+ ((300, 1), 'abc\0' * 5, True))
def test_tuple_sequence_coder(self):
int_tuple_coder = coders.TupleSequenceCoder(coders.VarIntCoder())
diff --git a/sdks/python/apache_beam/coders/observable_test.py b/sdks/python/apache_beam/coders/observable_test.py
index ce32bf0..a56a320 100644
--- a/sdks/python/apache_beam/coders/observable_test.py
+++ b/sdks/python/apache_beam/coders/observable_test.py
@@ -47,9 +47,9 @@
for _ in watched:
pass
- self.assertEquals(3, self.observed_count)
- self.assertEquals(8, self.observed_sum)
- self.assertEquals(['a1', 'a3', 'a4'], sorted(self.observed_keys))
+ self.assertEqual(3, self.observed_count)
+ self.assertEqual(8, self.observed_sum)
+ self.assertEqual(['a1', 'a3', 'a4'], sorted(self.observed_keys))
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py b/sdks/python/apache_beam/coders/standard_coders_test.py
index 437d2be..606ca81 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -22,6 +22,7 @@
import json
import logging
+import math
import os.path
import sys
import unittest
@@ -48,16 +49,27 @@
"""
if not os.path.exists(test_yaml):
raise ValueError('Could not find the test spec: %s' % test_yaml)
- for ix, spec in enumerate(yaml.load_all(open(test_yaml))):
- spec['index'] = ix
- name = spec.get('name', spec['coder']['urn'].split(':')[-2])
- yield [name, spec]
+ with open(test_yaml, 'rb') as coder_spec:
+ for ix, spec in enumerate(yaml.load_all(coder_spec)):
+ spec['index'] = ix
+ name = spec.get('name', spec['coder']['urn'].split(':')[-2])
+ yield [name, spec]
+
+
+def parse_float(s):
+ x = float(s)
+ if math.isnan(x):
+ # In Windows, float('NaN') has opposite sign from other platforms.
+ # For the purpose of this test, we just need consistency.
+ x = abs(x)
+ return x
class StandardCodersTest(unittest.TestCase):
_urn_to_json_value_parser = {
'beam:coder:bytes:v1': lambda x: x.encode('utf-8'),
+ 'beam:coder:bool:v1': lambda x: x,
'beam:coder:string_utf8:v1': lambda x: x,
'beam:coder:varint:v1': lambda x: x,
'beam:coder:kv:v1':
@@ -77,7 +89,7 @@
lambda x, payload_parser: dict(
payload=payload_parser(x['payload']),
timestamp=Timestamp(micros=x['timestamp'] * 1000)),
- 'beam:coder:double:v1': lambda x: float(x),
+ 'beam:coder:double:v1': parse_float,
}
def test_standard_coders(self):
@@ -88,7 +100,6 @@
def _run_standard_coder(self, name, spec):
def assert_equal(actual, expected):
"""Handle nan values which self.assertEqual fails on."""
- import math
if (isinstance(actual, float)
and isinstance(expected, float)
and math.isnan(actual)
@@ -122,9 +133,8 @@
component_ids = [context.coders.get_id(self.parse_coder(c))
for c in spec.get('components', ())]
context.coders.put_proto(coder_id, beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=spec['urn'], payload=spec.get('payload'))),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=spec['urn'], payload=spec.get('payload')),
component_coder_ids=component_ids))
return context.coders.get_by_id(coder_id)
diff --git a/sdks/python/apache_beam/coders/stream_test.py b/sdks/python/apache_beam/coders/stream_test.py
index ad046fb..e627ebb 100644
--- a/sdks/python/apache_beam/coders/stream_test.py
+++ b/sdks/python/apache_beam/coders/stream_test.py
@@ -41,15 +41,15 @@
out_s.write(b'xyz', True)
out_s.write(b'', True)
in_s = self.InputStream(out_s.get())
- self.assertEquals(b'abc\0\t\n', in_s.read(6))
- self.assertEquals(b'xyz', in_s.read_all(True))
- self.assertEquals(b'', in_s.read_all(True))
+ self.assertEqual(b'abc\0\t\n', in_s.read(6))
+ self.assertEqual(b'xyz', in_s.read_all(True))
+ self.assertEqual(b'', in_s.read_all(True))
def test_read_all(self):
out_s = self.OutputStream()
out_s.write(b'abc')
in_s = self.InputStream(out_s.get())
- self.assertEquals(b'abc', in_s.read_all(False))
+ self.assertEqual(b'abc', in_s.read_all(False))
def test_read_write_byte(self):
out_s = self.OutputStream()
@@ -57,9 +57,9 @@
out_s.write_byte(0)
out_s.write_byte(0xFF)
in_s = self.InputStream(out_s.get())
- self.assertEquals(1, in_s.read_byte())
- self.assertEquals(0, in_s.read_byte())
- self.assertEquals(0xFF, in_s.read_byte())
+ self.assertEqual(1, in_s.read_byte())
+ self.assertEqual(0, in_s.read_byte())
+ self.assertEqual(0xFF, in_s.read_byte())
def test_read_write_large(self):
values = range(4 * 1024)
@@ -68,7 +68,7 @@
out_s.write_bigendian_int64(v)
in_s = self.InputStream(out_s.get())
for v in values:
- self.assertEquals(v, in_s.read_bigendian_int64())
+ self.assertEqual(v, in_s.read_bigendian_int64())
def run_read_write_var_int64(self, values):
out_s = self.OutputStream()
@@ -76,7 +76,7 @@
out_s.write_var_int64(v)
in_s = self.InputStream(out_s.get())
for v in values:
- self.assertEquals(v, in_s.read_var_int64())
+ self.assertEqual(v, in_s.read_var_int64())
def test_small_var_int64(self):
self.run_read_write_var_int64(range(-10, 30))
@@ -97,7 +97,7 @@
out_s.write_bigendian_double(v)
in_s = self.InputStream(out_s.get())
for v in values:
- self.assertEquals(v, in_s.read_bigendian_double())
+ self.assertEqual(v, in_s.read_bigendian_double())
def test_read_write_bigendian_int64(self):
values = 0, 1, -1, 2**63-1, -2**63, int(2**61 * math.pi)
@@ -106,7 +106,7 @@
out_s.write_bigendian_int64(v)
in_s = self.InputStream(out_s.get())
for v in values:
- self.assertEquals(v, in_s.read_bigendian_int64())
+ self.assertEqual(v, in_s.read_bigendian_int64())
def test_read_write_bigendian_uint64(self):
values = 0, 1, 2**64-1, int(2**61 * math.pi)
@@ -115,7 +115,7 @@
out_s.write_bigendian_uint64(v)
in_s = self.InputStream(out_s.get())
for v in values:
- self.assertEquals(v, in_s.read_bigendian_uint64())
+ self.assertEqual(v, in_s.read_bigendian_uint64())
def test_read_write_bigendian_int32(self):
values = 0, 1, -1, 2**31-1, -2**31, int(2**29 * math.pi)
@@ -124,31 +124,31 @@
out_s.write_bigendian_int32(v)
in_s = self.InputStream(out_s.get())
for v in values:
- self.assertEquals(v, in_s.read_bigendian_int32())
+ self.assertEqual(v, in_s.read_bigendian_int32())
def test_byte_counting(self):
bc_s = self.ByteCountingOutputStream()
- self.assertEquals(0, bc_s.get_count())
+ self.assertEqual(0, bc_s.get_count())
bc_s.write(b'def')
- self.assertEquals(3, bc_s.get_count())
+ self.assertEqual(3, bc_s.get_count())
bc_s.write(b'')
- self.assertEquals(3, bc_s.get_count())
+ self.assertEqual(3, bc_s.get_count())
bc_s.write_byte(10)
- self.assertEquals(4, bc_s.get_count())
+ self.assertEqual(4, bc_s.get_count())
# "nested" also writes the length of the string, which should
# cause 1 extra byte to be counted.
bc_s.write(b'2345', nested=True)
- self.assertEquals(9, bc_s.get_count())
+ self.assertEqual(9, bc_s.get_count())
bc_s.write_var_int64(63)
- self.assertEquals(10, bc_s.get_count())
+ self.assertEqual(10, bc_s.get_count())
bc_s.write_bigendian_int64(42)
- self.assertEquals(18, bc_s.get_count())
+ self.assertEqual(18, bc_s.get_count())
bc_s.write_bigendian_int32(36)
- self.assertEquals(22, bc_s.get_count())
+ self.assertEqual(22, bc_s.get_count())
bc_s.write_bigendian_double(6.25)
- self.assertEquals(30, bc_s.get_count())
+ self.assertEqual(30, bc_s.get_count())
bc_s.write_bigendian_uint64(47)
- self.assertEquals(38, bc_s.get_count())
+ self.assertEqual(38, bc_s.get_count())
try:
diff --git a/sdks/python/apache_beam/coders/typecoders.py b/sdks/python/apache_beam/coders/typecoders.py
index 7afb015..6f6f322 100644
--- a/sdks/python/apache_beam/coders/typecoders.py
+++ b/sdks/python/apache_beam/coders/typecoders.py
@@ -88,6 +88,7 @@
self._register_coder_internal(int, coders.VarIntCoder)
self._register_coder_internal(float, coders.FloatCoder)
self._register_coder_internal(bytes, coders.BytesCoder)
+ self._register_coder_internal(bool, coders.BooleanCoder)
self._register_coder_internal(unicode, coders.StrUtf8Coder)
self._register_coder_internal(typehints.TupleConstraint, coders.TupleCoder)
# Default fallback coders applied in that order until the first matching
@@ -119,7 +120,8 @@
raise RuntimeError(
'Coder registry has no fallback coder. This can happen if the '
'fast_coders module could not be imported.')
- if isinstance(typehint, typehints.IterableTypeConstraint):
+ if isinstance(typehint, (typehints.IterableTypeConstraint,
+ typehints.ListConstraint)):
return coders.IterableCoder.from_type_hint(typehint, self)
elif typehint is None:
# In some old code, None is used for Any.
diff --git a/sdks/python/apache_beam/coders/typecoders_test.py b/sdks/python/apache_beam/coders/typecoders_test.py
index 64843ae..52e32fb 100644
--- a/sdks/python/apache_beam/coders/typecoders_test.py
+++ b/sdks/python/apache_beam/coders/typecoders_test.py
@@ -122,6 +122,16 @@
real_coder.encode(b'abc'), expected_coder.encode(b'abc'))
self.assertEqual(b'abc', real_coder.decode(real_coder.encode(b'abc')))
+ def test_standard_bool_coder(self):
+ real_coder = typecoders.registry.get_coder(bool)
+ expected_coder = coders.BooleanCoder()
+ self.assertEqual(
+ real_coder.encode(True), expected_coder.encode(True))
+ self.assertEqual(True, real_coder.decode(real_coder.encode(True)))
+ self.assertEqual(
+ real_coder.encode(False), expected_coder.encode(False))
+ self.assertEqual(False, real_coder.decode(real_coder.encode(False)))
+
def test_iterable_coder(self):
real_coder = typecoders.registry.get_coder(typehints.Iterable[bytes])
expected_coder = coders.IterableCoder(coders.BytesCoder())
@@ -129,6 +139,18 @@
self.assertEqual(expected_coder, real_coder)
self.assertEqual(real_coder.encode(values), expected_coder.encode(values))
+ def test_list_coder(self):
+ real_coder = typecoders.registry.get_coder(typehints.List[bytes])
+ expected_coder = coders.IterableCoder(coders.BytesCoder())
+ values = [b'abc', b'xyz']
+ self.assertEqual(expected_coder, real_coder)
+ self.assertEqual(real_coder.encode(values), expected_coder.encode(values))
+ # IterableCoder.decode() always returns a list. Its implementation,
+ # IterableCoderImpl, *can* return a non-list if it is provided a read_state
+ # object, but this is not possible using the atomic IterableCoder interface.
+ self.assertIs(list,
+ type(expected_coder.decode(expected_coder.encode(values))))
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/examples/avro_bitcoin.py b/sdks/python/apache_beam/examples/avro_bitcoin.py
index 079d9d7..f6ab89e 100644
--- a/sdks/python/apache_beam/examples/avro_bitcoin.py
+++ b/sdks/python/apache_beam/examples/avro_bitcoin.py
@@ -48,7 +48,9 @@
"""Count inputs and outputs per transaction"""
def __init__(self):
- super(BitcoinTxnCountDoFn, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(BitcoinTxnCountDoFn, self).__init__()
+ beam.DoFn.__init__(self)
self.txn_counter = Metrics.counter(self.__class__, 'txns')
self.inputs_dist = Metrics.distribution(self.__class__, 'inputs_per_txn')
self.outputs_dist = Metrics.distribution(self.__class__, 'outputs_per_txn')
diff --git a/sdks/python/apache_beam/examples/complete/autocomplete.py b/sdks/python/apache_beam/examples/complete/autocomplete.py
index d9717ba..03b8500 100644
--- a/sdks/python/apache_beam/examples/complete/autocomplete.py
+++ b/sdks/python/apache_beam/examples/complete/autocomplete.py
@@ -61,7 +61,9 @@
class TopPerPrefix(beam.PTransform):
def __init__(self, count):
- super(TopPerPrefix, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(TopPerPrefix, self).__init__()
+ beam.PTransform.__init__(self)
self._count = count
def expand(self, words):
diff --git a/sdks/python/apache_beam/examples/complete/distribopt.py b/sdks/python/apache_beam/examples/complete/distribopt.py
index a16a40b..42c7b83 100644
--- a/sdks/python/apache_beam/examples/complete/distribopt.py
+++ b/sdks/python/apache_beam/examples/complete/distribopt.py
@@ -71,7 +71,6 @@
"""Greenhouse simulation for the optimization of greenhouse parameters."""
def __init__(self, quantities):
- super(Simulator, self).__init__()
self.quantities = np.atleast_1d(quantities)
self.A = np.array([[3.0, 10, 30],
@@ -314,7 +313,7 @@
return result
-def run(argv=None):
+def run(argv=None, save_main_session=True):
parser = argparse.ArgumentParser()
parser.add_argument('--input',
dest='input',
@@ -326,7 +325,7 @@
help='Output file to write results to.')
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Parse input file
diff --git a/sdks/python/apache_beam/examples/complete/distribopt_test.py b/sdks/python/apache_beam/examples/complete/distribopt_test.py
index eb8ff53..ffdbd99 100644
--- a/sdks/python/apache_beam/examples/complete/distribopt_test.py
+++ b/sdks/python/apache_beam/examples/complete/distribopt_test.py
@@ -70,9 +70,10 @@
with patch.dict('sys.modules', modules):
from apache_beam.examples.complete import distribopt
- distribopt.run([
- '--input=%s/input.txt' % temp_folder,
- '--output', os.path.join(temp_folder, 'result')])
+ distribopt.run(
+ ['--input=%s/input.txt' % temp_folder,
+ '--output', os.path.join(temp_folder, 'result')],
+ save_main_session=False)
# Load result file and compare.
with open_shards(os.path.join(temp_folder, 'result-*-of-*')) as result_file:
diff --git a/sdks/python/apache_beam/examples/complete/estimate_pi.py b/sdks/python/apache_beam/examples/complete/estimate_pi.py
index 3323462..aa41d02 100644
--- a/sdks/python/apache_beam/examples/complete/estimate_pi.py
+++ b/sdks/python/apache_beam/examples/complete/estimate_pi.py
@@ -33,14 +33,14 @@
import random
from builtins import object
from builtins import range
+from typing import Any
+from typing import Iterable
+from typing import Tuple
import apache_beam as beam
from apache_beam.io import WriteToText
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
-from apache_beam.typehints import Any
-from apache_beam.typehints import Iterable
-from apache_beam.typehints import Tuple
@beam.typehints.with_output_types(Tuple[int, int, int])
diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats.py b/sdks/python/apache_beam/examples/complete/game/game_stats.py
index 9b5cc32..8f446e6 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats.py
@@ -106,7 +106,9 @@
The human-readable time string is not used here.
"""
def __init__(self):
- super(ParseGameEventFn, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ParseGameEventFn, self).__init__()
+ beam.DoFn.__init__(self)
self.num_parse_errors = Metrics.counter(self.__class__, 'num_parse_errors')
def process(self, elem):
@@ -130,7 +132,9 @@
extracted.
"""
def __init__(self, field):
- super(ExtractAndSumScore, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ExtractAndSumScore, self).__init__()
+ beam.PTransform.__init__(self)
self.field = field
def expand(self, pcoll):
@@ -167,7 +171,9 @@
schema: Dictionary in the format {'column_name': 'bigquery_type'}
project: Name of the Cloud project containing BigQuery table.
"""
- super(WriteToBigQuery, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(WriteToBigQuery, self).__init__()
+ beam.PTransform.__init__(self)
self.table_name = table_name
self.dataset = dataset
self.schema = schema
@@ -234,7 +240,7 @@
yield (window.end.micros - window.start.micros)//1000000
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -290,7 +296,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
# Enforce that this pipeline is always run in streaming mode
options.view_as(StandardOptions).streaming = True
diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
index cba4b00..70dafb0 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats_it_test.py
@@ -140,7 +140,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
game_stats.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
index c46559a..e0a5c47 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py
@@ -106,7 +106,9 @@
The human-readable time string is not used here.
"""
def __init__(self):
- super(ParseGameEventFn, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ParseGameEventFn, self).__init__()
+ beam.DoFn.__init__(self)
self.num_parse_errors = Metrics.counter(self.__class__, 'num_parse_errors')
def process(self, elem):
@@ -130,7 +132,9 @@
extracted.
"""
def __init__(self, field):
- super(ExtractAndSumScore, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ExtractAndSumScore, self).__init__()
+ beam.PTransform.__init__(self)
self.field = field
def expand(self, pcoll):
@@ -167,7 +171,9 @@
schema: Dictionary in the format {'column_name': 'bigquery_type'}
project: Name of the Cloud project containing BigQuery table.
"""
- super(WriteToBigQuery, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(WriteToBigQuery, self).__init__()
+ beam.PTransform.__init__(self)
self.table_name = table_name
self.dataset = dataset
self.schema = schema
@@ -190,7 +196,9 @@
# [START main]
class HourlyTeamScore(beam.PTransform):
def __init__(self, start_min, stop_min, window_duration):
- super(HourlyTeamScore, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(HourlyTeamScore, self).__init__()
+ beam.PTransform.__init__(self)
self.start_timestamp = str2timestamp(start_min)
self.stop_timestamp = str2timestamp(stop_min)
self.window_duration_in_seconds = window_duration * 60
@@ -228,7 +236,7 @@
| 'ExtractAndSumScore' >> ExtractAndSumScore('team'))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -279,7 +287,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as p:
(p # pylint: disable=expression-not-assigned
diff --git a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
index 5685132..8d86f18 100644
--- a/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py
@@ -86,7 +86,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
hourly_team_score.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board.py b/sdks/python/apache_beam/examples/complete/game/leader_board.py
index 43a599b..2288d16 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board.py
@@ -115,7 +115,9 @@
The human-readable time string is not used here.
"""
def __init__(self):
- super(ParseGameEventFn, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ParseGameEventFn, self).__init__()
+ beam.DoFn.__init__(self)
self.num_parse_errors = Metrics.counter(self.__class__, 'num_parse_errors')
def process(self, elem):
@@ -139,7 +141,9 @@
extracted.
"""
def __init__(self, field):
- super(ExtractAndSumScore, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ExtractAndSumScore, self).__init__()
+ beam.PTransform.__init__(self)
self.field = field
def expand(self, pcoll):
@@ -176,7 +180,9 @@
schema: Dictionary in the format {'column_name': 'bigquery_type'}
project: Name of the Cloud project containing BigQuery table.
"""
- super(WriteToBigQuery, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(WriteToBigQuery, self).__init__()
+ beam.PTransform.__init__(self)
self.table_name = table_name
self.dataset = dataset
self.schema = schema
@@ -204,7 +210,9 @@
default.
"""
def __init__(self, team_window_duration, allowed_lateness):
- super(CalculateTeamScores, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(CalculateTeamScores, self).__init__()
+ beam.PTransform.__init__(self)
self.team_window_duration = team_window_duration * 60
self.allowed_lateness_seconds = allowed_lateness * 60
@@ -232,7 +240,9 @@
global windowing. Get periodic updates on all users' running scores.
"""
def __init__(self, allowed_lateness):
- super(CalculateUserScores, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(CalculateUserScores, self).__init__()
+ beam.PTransform.__init__(self)
self.allowed_lateness_seconds = allowed_lateness * 60
def expand(self, pcoll):
@@ -251,7 +261,7 @@
# [END processing_time_trigger]
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the hourly_team_score pipeline."""
parser = argparse.ArgumentParser()
@@ -296,7 +306,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
# Enforce that this pipeline is always run in streaming mode
options.view_as(StandardOptions).streaming = True
diff --git a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
index 9f057fd..af2f2e6 100644
--- a/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py
@@ -149,7 +149,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
leader_board.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/game/user_score.py b/sdks/python/apache_beam/examples/complete/game/user_score.py
index 7e1a07d..74b47ba 100644
--- a/sdks/python/apache_beam/examples/complete/game/user_score.py
+++ b/sdks/python/apache_beam/examples/complete/game/user_score.py
@@ -78,7 +78,9 @@
The human-readable time string is not used here.
"""
def __init__(self):
- super(ParseGameEventFn, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ParseGameEventFn, self).__init__()
+ beam.DoFn.__init__(self)
self.num_parse_errors = Metrics.counter(self.__class__, 'num_parse_errors')
def process(self, elem):
@@ -103,7 +105,9 @@
extracted.
"""
def __init__(self, field):
- super(ExtractAndSumScore, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ExtractAndSumScore, self).__init__()
+ beam.PTransform.__init__(self)
self.field = field
def expand(self, pcoll):
@@ -123,7 +127,7 @@
# [START main]
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the user_score pipeline."""
parser = argparse.ArgumentParser()
@@ -144,7 +148,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
- options.view_as(SetupOptions).save_main_session = True
+ options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=options) as p:
def format_user_score_sums(user_score):
diff --git a/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py b/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
index a29c99e..5e5ba97 100644
--- a/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
+++ b/sdks/python/apache_beam/examples/complete/game/user_score_it_test.py
@@ -81,7 +81,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
user_score.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/complete/tfidf.py b/sdks/python/apache_beam/examples/complete/tfidf.py
index 4d99b98..77ee4c1 100644
--- a/sdks/python/apache_beam/examples/complete/tfidf.py
+++ b/sdks/python/apache_beam/examples/complete/tfidf.py
@@ -187,7 +187,7 @@
return word_to_uri_and_tfidf
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the tfidf pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument('--uris',
@@ -200,7 +200,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read documents specified by the uris command line option.
diff --git a/sdks/python/apache_beam/examples/complete/tfidf_test.py b/sdks/python/apache_beam/examples/complete/tfidf_test.py
index 2580d68..4b19269 100644
--- a/sdks/python/apache_beam/examples/complete/tfidf_test.py
+++ b/sdks/python/apache_beam/examples/complete/tfidf_test.py
@@ -77,9 +77,10 @@
self.create_file(os.path.join(temp_folder, '1.txt'), 'abc def ghi')
self.create_file(os.path.join(temp_folder, '2.txt'), 'abc def')
self.create_file(os.path.join(temp_folder, '3.txt'), 'abc')
- tfidf.run([
- '--uris=%s/*' % temp_folder,
- '--output', os.path.join(temp_folder, 'result')])
+ tfidf.run(
+ ['--uris=%s/*' % temp_folder,
+ '--output', os.path.join(temp_folder, 'result')],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(os.path.join(
diff --git a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
index dd827bc..6b04a00 100644
--- a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
+++ b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
@@ -117,7 +117,9 @@
"""Computes the top user sessions for each month."""
def __init__(self, sampling_threshold):
- super(ComputeTopSessions, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(ComputeTopSessions, self).__init__()
+ beam.PTransform.__init__(self)
self.sampling_threshold = sampling_threshold
def expand(self, pcoll):
diff --git a/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py b/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py
index 82e2869..ca5c4a5 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py
@@ -64,7 +64,9 @@
"""
def __init__(self, number, project_id=None, instance_id=None,
table_id=None):
- super(WriteToBigTable, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(WriteToBigTable, self).__init__()
+ beam.PTransform.__init__(self)
self.number = number
self.rand = random.choice(string.ascii_letters + string.digits)
self.column_family_id = 'cf1'
diff --git a/sdks/python/apache_beam/examples/cookbook/group_with_coder.py b/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
index a953fb3..2202b8c 100644
--- a/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
+++ b/sdks/python/apache_beam/examples/cookbook/group_with_coder.py
@@ -30,6 +30,7 @@
import argparse
import logging
import sys
+import typing
from builtins import object
import apache_beam as beam
@@ -38,7 +39,6 @@
from apache_beam.io import WriteToText
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
-from apache_beam.typehints import typehints
from apache_beam.typehints.decorators import with_output_types
@@ -74,13 +74,13 @@
# Annotate the get_players function so that the typehint system knows that the
# input to the CombinePerKey operation is a key-value pair of a Player object
# and an integer.
-@with_output_types(typehints.KV[Player, int])
+@with_output_types(typing.Tuple[Player, int])
def get_players(descriptor):
name, points = descriptor.split(',')
return Player(name), int(points)
-def run(args=None):
+def run(args=None, save_main_session=True):
"""Runs the workflow computing total points from a collection of matches."""
if args is None:
@@ -96,7 +96,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Register the custom coder for the Player class, so that it will be used in
diff --git a/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py b/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
index 73d7377..6f1b796 100644
--- a/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/group_with_coder_test.py
@@ -51,9 +51,10 @@
# and therefore any custom coders will be used. In our case we want to make
# sure the coder for the Player class will be used.
temp_path = self.create_temp_file(self.SAMPLE_RECORDS)
- group_with_coder.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ group_with_coder.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
@@ -71,10 +72,11 @@
# therefore any custom coders will not be used. The default coder (pickler)
# will be used instead.
temp_path = self.create_temp_file(self.SAMPLE_RECORDS)
- group_with_coder.run([
- '--no_pipeline_type_check',
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ group_with_coder.run(
+ ['--no_pipeline_type_check',
+ '--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/cookbook/mergecontacts.py b/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
index 79e7274..9bd6a96 100644
--- a/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
+++ b/sdks/python/apache_beam/examples/cookbook/mergecontacts.py
@@ -45,7 +45,7 @@
from apache_beam.testing.util import equal_to
-def run(argv=None, assert_results=None):
+def run(argv=None, assert_results=None, save_main_session=True):
parser = argparse.ArgumentParser()
parser.add_argument(
@@ -70,7 +70,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Helper: read a tab-separated key-value mapping from a text file,
diff --git a/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py b/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
index 0c2bc47..23f22bc 100644
--- a/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/mergecontacts_test.py
@@ -110,12 +110,14 @@
result_prefix = self.create_temp_file('')
- mergecontacts.run([
- '--input_email=%s' % path_email,
- '--input_phone=%s' % path_phone,
- '--input_snailmail=%s' % path_snailmail,
- '--output_tsv=%s.tsv' % result_prefix,
- '--output_stats=%s.stats' % result_prefix], assert_results=(2, 1, 3))
+ mergecontacts.run(
+ ['--input_email=%s' % path_email,
+ '--input_phone=%s' % path_phone,
+ '--input_snailmail=%s' % path_snailmail,
+ '--output_tsv=%s.tsv' % result_prefix,
+ '--output_stats=%s.stats' % result_prefix],
+ assert_results=(2, 1, 3),
+ save_main_session=False)
with open_shards('%s.tsv-*-of-*' % result_prefix) as f:
contents = f.read()
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
index e3df3a8..7896027 100644
--- a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
+++ b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
@@ -134,7 +134,7 @@
| 'format' >> beam.Map(format_result))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the workflow counting the long words and short words separately."""
parser = argparse.ArgumentParser()
@@ -148,7 +148,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
lines = p | ReadFromText(known_args.input)
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
index 6f7aa9f..afe350d 100644
--- a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo_test.py
@@ -53,9 +53,10 @@
temp_path = self.create_temp_file(self.SAMPLE_TEXT)
result_prefix = temp_path + '.result'
- multiple_output_pardo.run([
- '--input=%s*' % temp_path,
- '--output=%s' % result_prefix])
+ multiple_output_pardo.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s' % result_prefix],
+ save_main_session=False)
expected_char_count = len(''.join(self.SAMPLE_TEXT.split('\n')))
with open_shards(result_prefix + '-chars-*-of-*') as f:
diff --git a/sdks/python/apache_beam/examples/snippets/snippets.py b/sdks/python/apache_beam/examples/snippets/snippets.py
index 5e7f9bf..510e814 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets.py
@@ -33,8 +33,10 @@
from __future__ import division
import argparse
+import base64
from builtins import object
from builtins import range
+from decimal import Decimal
from past.builtins import unicode
@@ -444,7 +446,7 @@
# [END examples_wordcount_minimal_count]
# [START examples_wordcount_minimal_map]
- | beam.Map(lambda word_count: '%s: %s' % (word_count[0], word_count[1]))
+ | beam.MapTuple(lambda word, count: '%s: %s' % (word, count))
# [END examples_wordcount_minimal_map]
# [START examples_wordcount_minimal_write]
@@ -751,22 +753,22 @@
return OffsetRangeTracker(start_position, stop_position)
def read(self, range_tracker):
- for i in range(self._count):
+ for i in range(range_tracker.start_position(),
+ range_tracker.stop_position()):
if not range_tracker.try_claim(i):
return
self.records_read.inc()
yield i
- def split(self, desired_bundle_size, start_position=None,
- stop_position=None):
+ def split(self, desired_bundle_size, start_position=None, stop_position=None):
if start_position is None:
start_position = 0
if stop_position is None:
stop_position = self._count
bundle_start = start_position
- while bundle_start < self._count:
- bundle_stop = max(self._count, bundle_start + desired_bundle_size)
+ while bundle_start < stop_position:
+ bundle_stop = min(stop_position, bundle_start + desired_bundle_size)
yield iobase.SourceBundle(weight=(bundle_stop - bundle_start),
source=self,
start_position=bundle_start,
@@ -1081,6 +1083,22 @@
tableId='weather_stations')
# [END model_bigqueryio_table_spec_object]
+ # [START model_bigqueryio_data_types]
+ bigquery_data = [{
+ 'string': 'abc',
+ 'bytes': base64.b64encode(b'\xab\xac'),
+ 'integer': 5,
+ 'float': 0.5,
+ 'numeric': Decimal('5'),
+ 'boolean': True,
+ 'timestamp': '2018-12-31 12:44:31.744957 UTC',
+ 'date': '2018-12-31',
+ 'time': '12:44:31',
+ 'datetime': '2018-12-31T12:44:31',
+ 'geography': 'POINT(30 10)'
+ }]
+ # [END model_bigqueryio_data_types]
+
# [START model_bigqueryio_read_table]
max_temperatures = (
p
@@ -1352,3 +1370,52 @@
| beam.Map(lambda x: (x.metadata.path,
x.read_utf8())))
# [END FileProcessPatternAccessMetadataSnip1]
+
+
+def accessing_valueprovider_info_after_run():
+ # [START AccessingValueProviderInfoAfterRunSnip1]
+ import logging
+
+ import apache_beam as beam
+ from apache_beam.options.pipeline_options import PipelineOptions
+ from apache_beam.utils.value_provider import RuntimeValueProvider
+ from apache_beam.io import WriteToText
+
+ class MyOptions(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_value_provider_argument('--string_value', type=str)
+
+ class LogValueProvidersFn(beam.DoFn):
+ def __init__(self, string_vp):
+ self.string_vp = string_vp
+
+ # Define the DoFn that logs the ValueProvider value.
+ # The DoFn is called when creating the pipeline branch.
+ # This example logs the ValueProvider value, but
+ # you could store it by pushing it to an external database.
+ def process(self, an_int):
+ logging.info('The string_value is %s' % self.string_vp.get())
+ # Another option (where you don't need to pass the value at all) is:
+ logging.info('The string value is %s' %
+ RuntimeValueProvider.get_value('string_value', str, ''))
+
+ pipeline_options = PipelineOptions()
+ # Create pipeline.
+ p = beam.Pipeline(options=pipeline_options)
+
+ my_options = pipeline_options.view_as(MyOptions)
+ # Add a branch for logging the ValueProvider value.
+ _ = (p
+ | beam.Create([None])
+ | 'LogValueProvs' >> beam.ParDo(
+ LogValueProvidersFn(my_options.string_value)))
+
+ # The main pipeline.
+ result_pc = (p
+ | "main_pc" >> beam.Create([1, 2, 3])
+ | beam.combiners.Sum.Globally())
+
+ p.run().wait_until_finish()
+
+ # [END AccessingValueProviderInfoAfterRunSnip1]
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index ef6fce8..4b52266 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -26,6 +26,7 @@
import os
import sys
import tempfile
+import typing
import unittest
import uuid
from builtins import map
@@ -321,10 +322,10 @@
# One can assert outputs and apply them to transforms as well.
# Helps document the contract and checks it at pipeline construction time.
# [START type_hints_transform]
- T = beam.typehints.TypeVariable('T')
+ T = typing.TypeVar('T')
@beam.typehints.with_input_types(T)
- @beam.typehints.with_output_types(beam.typehints.Tuple[int, T])
+ @beam.typehints.with_output_types(typing.Tuple[int, T])
class MyTransform(beam.PTransform):
def expand(self, pcoll):
return pcoll | beam.Map(lambda x: (len(x), x))
@@ -335,7 +336,7 @@
# pylint: disable=expression-not-assigned
with self.assertRaises(typehints.TypeCheckError):
words_with_lens | beam.Map(lambda x: x).with_input_types(
- beam.typehints.Tuple[int, int])
+ typing.Tuple[int, int])
def test_runtime_checks_off(self):
# We do not run the following pipeline, as it has incorrect type
@@ -391,7 +392,7 @@
lines
| beam.Map(parse_player_and_score)
| beam.CombinePerKey(sum).with_input_types(
- beam.typehints.Tuple[Player, int]))
+ typing.Tuple[Player, int]))
# [END type_hints_deterministic_key]
assert_that(
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/__init__.py b/sdks/python/apache_beam/examples/snippets/transforms/__init__.py
new file mode 100644
index 0000000..6569e3f
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
new file mode 100644
index 0000000..6569e3f
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py
new file mode 100644
index 0000000..44b11b8
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py
@@ -0,0 +1,182 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def filter_function(test=None):
+ # [START filter_function]
+ import apache_beam as beam
+
+ def is_perennial(plant):
+ return plant['duration'] == 'perennial'
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(is_perennial)
+ | beam.Map(print)
+ )
+ # [END filter_function]
+ if test:
+ test(perennials)
+
+
+def filter_lambda(test=None):
+ # [START filter_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(
+ lambda plant: plant['duration'] == 'perennial')
+ | beam.Map(print)
+ )
+ # [END filter_lambda]
+ if test:
+ test(perennials)
+
+
+def filter_multiple_arguments(test=None):
+ # [START filter_multiple_arguments]
+ import apache_beam as beam
+
+ def has_duration(plant, duration):
+ return plant['duration'] == duration
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(has_duration, 'perennial')
+ | beam.Map(print)
+ )
+ # [END filter_multiple_arguments]
+ if test:
+ test(perennials)
+
+
+def filter_side_inputs_singleton(test=None):
+ # [START filter_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ perennial = pipeline | 'Perennial' >> beam.Create(['perennial'])
+
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(
+ lambda plant, duration: plant['duration'] == duration,
+ duration=beam.pvalue.AsSingleton(perennial),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_singleton]
+ if test:
+ test(perennials)
+
+
+def filter_side_inputs_iter(test=None):
+ # [START filter_side_inputs_iter]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ valid_durations = pipeline | 'Valid durations' >> beam.Create([
+ 'annual',
+ 'biennial',
+ 'perennial',
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'PERENNIAL'},
+ ])
+ | 'Filter valid plants' >> beam.Filter(
+ lambda plant, valid_durations: plant['duration'] in valid_durations,
+ valid_durations=beam.pvalue.AsIter(valid_durations),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_iter]
+ if test:
+ test(valid_plants)
+
+
+def filter_side_inputs_dict(test=None):
+ # [START filter_side_inputs_dict]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ keep_duration = pipeline | 'Duration filters' >> beam.Create([
+ ('annual', False),
+ ('biennial', False),
+ ('perennial', True),
+ ])
+
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter plants by duration' >> beam.Filter(
+ lambda plant, keep_duration: keep_duration[plant['duration']],
+ keep_duration=beam.pvalue.AsDict(keep_duration),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_dict]
+ if test:
+ test(perennials)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
new file mode 100644
index 0000000..02da146
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
@@ -0,0 +1,80 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.transforms.element_wise.filter import *
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.filter.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class FilterTest(unittest.TestCase):
+ def __init__(self, methodName):
+ super(FilterTest, self).__init__(methodName)
+ # [START perennials]
+ perennials = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END perennials]
+ self.perennials_test = lambda actual: \
+ assert_that(actual, equal_to(perennials))
+
+ # [START valid_plants]
+ valid_plants = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END valid_plants]
+ self.valid_plants_test = lambda actual: \
+ assert_that(actual, equal_to(valid_plants))
+
+ def test_filter_function(self):
+ filter_function(self.perennials_test)
+
+ def test_filter_lambda(self):
+ filter_lambda(self.perennials_test)
+
+ def test_filter_multiple_arguments(self):
+ filter_multiple_arguments(self.perennials_test)
+
+ def test_filter_side_inputs_singleton(self):
+ filter_side_inputs_singleton(self.perennials_test)
+
+ def test_filter_side_inputs_iter(self):
+ filter_side_inputs_iter(self.valid_plants_test)
+
+ def test_filter_side_inputs_dict(self):
+ filter_side_inputs_dict(self.perennials_test)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py
new file mode 100644
index 0000000..01c9d6b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def keys(test=None):
+ # [START keys]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ icons = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Keys' >> beam.Keys()
+ | beam.Map(print)
+ )
+ # [END keys]
+ if test:
+ test(icons)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py
new file mode 100644
index 0000000..9cb4909
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py
@@ -0,0 +1,55 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.transforms.element_wise.keys import *
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.keys.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class KeysTest(unittest.TestCase):
+ def __init__(self, methodName):
+ super(KeysTest, self).__init__(methodName)
+ # [START icons]
+ icons = [
+ '🍓',
+ '🥕',
+ '🍆',
+ '🍅',
+ '🥔',
+ ]
+ # [END icons]
+ self.icons_test = lambda actual: assert_that(actual, equal_to(icons))
+
+ def test_keys(self):
+ keys(self.icons_test)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py
new file mode 100644
index 0000000..2107fd5
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def kvswap(test=None):
+ # [START kvswap]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Key-Value swap' >> beam.KvSwap()
+ | beam.Map(print)
+ )
+ # [END kvswap]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py
new file mode 100644
index 0000000..85fa9dc
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py
@@ -0,0 +1,55 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.transforms.element_wise.kvswap import *
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.kvswap.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class KvSwapTest(unittest.TestCase):
+ def __init__(self, methodName):
+ super(KvSwapTest, self).__init__(methodName)
+ # [START plants]
+ plants = [
+ ('Strawberry', '🍓'),
+ ('Carrot', '🥕'),
+ ('Eggplant', '🍆'),
+ ('Tomato', '🍅'),
+ ('Potato', '🥔'),
+ ]
+ # [END plants]
+ self.plants_test = lambda actual: assert_that(actual, equal_to(plants))
+
+ def test_kvswap(self):
+ kvswap(self.plants_test)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py
new file mode 100644
index 0000000..9defd47
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py
@@ -0,0 +1,226 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def map_simple(test=None):
+ # [START map_simple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ' 🍓Strawberry \n',
+ ' 🥕Carrot \n',
+ ' 🍆Eggplant \n',
+ ' 🍅Tomato \n',
+ ' 🥔Potato \n',
+ ])
+ | 'Strip' >> beam.Map(str.strip)
+ | beam.Map(print)
+ )
+ # [END map_simple]
+ if test:
+ test(plants)
+
+
+def map_function(test=None):
+ # [START map_function]
+ import apache_beam as beam
+
+ def strip_header_and_newline(text):
+ return text.strip('# \n')
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(strip_header_and_newline)
+ | beam.Map(print)
+ )
+ # [END map_function]
+ if test:
+ test(plants)
+
+
+def map_lambda(test=None):
+ # [START map_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(lambda text: text.strip('# \n'))
+ | beam.Map(print)
+ )
+ # [END map_lambda]
+ if test:
+ test(plants)
+
+
+def map_multiple_arguments(test=None):
+ # [START map_multiple_arguments]
+ import apache_beam as beam
+
+ def strip(text, chars=None):
+ return text.strip(chars)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(strip, chars='# \n')
+ | beam.Map(print)
+ )
+ # [END map_multiple_arguments]
+ if test:
+ test(plants)
+
+
+def map_tuple(test=None):
+ # [START map_tuple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Format' >> beam.MapTuple(
+ lambda icon, plant: '{}{}'.format(icon, plant))
+ | beam.Map(print)
+ )
+ # [END map_tuple]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_singleton(test=None):
+ # [START map_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ chars = pipeline | 'Create chars' >> beam.Create(['# \n'])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(
+ lambda text, chars: text.strip(chars),
+ chars=beam.pvalue.AsSingleton(chars),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_singleton]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_iter(test=None):
+ # [START map_side_inputs_iter]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ chars = pipeline | 'Create chars' >> beam.Create(['#', ' ', '\n'])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(
+ lambda text, chars: text.strip(''.join(chars)),
+ chars=beam.pvalue.AsIter(chars),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_iter]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_dict(test=None):
+ # [START map_side_inputs_dict]
+ import apache_beam as beam
+
+ def replace_duration(plant, durations):
+ plant['duration'] = durations[plant['duration']]
+ return plant
+
+ with beam.Pipeline() as pipeline:
+ durations = pipeline | 'Durations' >> beam.Create([
+ (0, 'annual'),
+ (1, 'biennial'),
+ (2, 'perennial'),
+ ])
+
+ plant_details = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 1},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 0},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 2},
+ ])
+ | 'Replace duration' >> beam.Map(
+ replace_duration,
+ durations=beam.pvalue.AsDict(durations),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_dict]
+ if test:
+ test(plant_details)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py
new file mode 100644
index 0000000..5fcee8a
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py
@@ -0,0 +1,90 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import map
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_plant_details(actual):
+ # [START plant_details]
+ plant_details = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END plant_details]
+ assert_that(actual, equal_to(plant_details))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.map.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class MapTest(unittest.TestCase):
+ def test_map_simple(self):
+ map.map_simple(check_plants)
+
+ def test_map_function(self):
+ map.map_function(check_plants)
+
+ def test_map_lambda(self):
+ map.map_lambda(check_plants)
+
+ def test_map_multiple_arguments(self):
+ map.map_multiple_arguments(check_plants)
+
+ def test_map_tuple(self):
+ map.map_tuple(check_plants)
+
+ def test_map_side_inputs_singleton(self):
+ map.map_side_inputs_singleton(check_plants)
+
+ def test_map_side_inputs_iter(self):
+ map.map_side_inputs_iter(check_plants)
+
+ def test_map_side_inputs_dict(self):
+ map.map_side_inputs_dict(check_plant_details)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py
new file mode 100644
index 0000000..971e9f0
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py
@@ -0,0 +1,126 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+
+def pardo_dofn(test=None):
+ # [START pardo_dofn]
+ import apache_beam as beam
+
+ class SplitWords(beam.DoFn):
+ def __init__(self, delimiter=','):
+ self.delimiter = delimiter
+
+ def process(self, text):
+ for word in text.split(self.delimiter):
+ yield word
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.ParDo(SplitWords(','))
+ | beam.Map(print)
+ )
+ # [END pardo_dofn]
+ if test:
+ test(plants)
+
+
+def pardo_dofn_params(test=None):
+ # pylint: disable=line-too-long
+ # [START pardo_dofn_params]
+ import apache_beam as beam
+
+ class AnalyzeElement(beam.DoFn):
+ def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):
+ yield '\n'.join([
+ '# timestamp',
+ 'type(timestamp) -> ' + repr(type(timestamp)),
+ 'timestamp.micros -> ' + repr(timestamp.micros),
+ 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),
+ 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),
+ '',
+ '# window',
+ 'type(window) -> ' + repr(type(window)),
+ 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),
+ 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),
+ 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),
+ ])
+
+ with beam.Pipeline() as pipeline:
+ dofn_params = (
+ pipeline
+ | 'Create a single test element' >> beam.Create([':)'])
+ | 'Add timestamp (Spring equinox 2020)' >> beam.Map(
+ lambda elem: beam.window.TimestampedValue(elem, 1584675660))
+ | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))
+ | 'Analyze element' >> beam.ParDo(AnalyzeElement())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_params]
+ # pylint: enable=line-too-long
+ if test:
+ test(dofn_params)
+
+
+def pardo_dofn_methods(test=None):
+ # [START pardo_dofn_methods]
+ import apache_beam as beam
+
+ class DoFnMethods(beam.DoFn):
+ def __init__(self):
+ print('__init__')
+ self.window = beam.window.GlobalWindow()
+
+ def setup(self):
+ print('setup')
+
+ def start_bundle(self):
+ print('start_bundle')
+
+ def process(self, element, window=beam.DoFn.WindowParam):
+ self.window = window
+ yield '* process: ' + element
+
+ def finish_bundle(self):
+ yield beam.utils.windowed_value.WindowedValue(
+ value='* finish_bundle: 🌱🌳🌍',
+ timestamp=0,
+ windows=[self.window],
+ )
+
+ def teardown(self):
+ print('teardown')
+
+ with beam.Pipeline() as pipeline:
+ results = (
+ pipeline
+ | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])
+ | 'DoFn methods' >> beam.ParDo(DoFnMethods())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_methods]
+ if test:
+ return test(results)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py
new file mode 100644
index 0000000..a8de2d0
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py
@@ -0,0 +1,120 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import io
+import platform
+import sys
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import pardo
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_dofn_params(actual):
+ # pylint: disable=line-too-long
+ dofn_params = '\n'.join('''[START dofn_params]
+# timestamp
+type(timestamp) -> <class 'apache_beam.utils.timestamp.Timestamp'>
+timestamp.micros -> 1584675660000000
+timestamp.to_rfc3339() -> '2020-03-20T03:41:00Z'
+timestamp.to_utc_datetime() -> datetime.datetime(2020, 3, 20, 3, 41)
+
+# window
+type(window) -> <class 'apache_beam.transforms.window.IntervalWindow'>
+window.start -> Timestamp(1584675660) (2020-03-20 03:41:00)
+window.end -> Timestamp(1584675690) (2020-03-20 03:41:30)
+window.max_timestamp() -> Timestamp(1584675689.999999) (2020-03-20 03:41:29.999999)
+[END dofn_params]'''.splitlines()[1:-1])
+ # pylint: enable=line-too-long
+ assert_that(actual, equal_to([dofn_params]))
+
+
+def check_dofn_methods(actual):
+ # Return the expected stdout to check the ordering of the called methods.
+ return '''[START results]
+__init__
+setup
+start_bundle
+* process: 🍓
+* process: 🥕
+* process: 🍆
+* process: 🍅
+* process: 🥔
+* finish_bundle: 🌱🌳🌍
+teardown
+[END results]'''.splitlines()[1:-1]
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.pardo.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ParDoTest(unittest.TestCase):
+ def test_pardo_dofn(self):
+ pardo.pardo_dofn(check_plants)
+
+ # TODO: Remove this after Python 2 deprecation.
+ # https://issues.apache.org/jira/browse/BEAM-8124
+ @unittest.skipIf(sys.version_info[0] < 3 and platform.system() == 'Windows',
+ 'Python 2 on Windows uses `long` rather than `int`')
+ def test_pardo_dofn_params(self):
+ pardo.pardo_dofn_params(check_dofn_params)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch('sys.stdout', new_callable=io.StringIO)
+class ParDoStdoutTest(unittest.TestCase):
+ def test_pardo_dofn_methods(self, mock_stdout):
+ expected = pardo.pardo_dofn_methods(check_dofn_methods)
+ actual = mock_stdout.getvalue().splitlines()
+
+ # For the stdout, check the ordering of the methods, not of the elements.
+ actual_stdout = [line.split(':')[0] for line in actual]
+ expected_stdout = [line.split(':')[0] for line in expected]
+ self.assertEqual(actual_stdout, expected_stdout)
+
+ # For the elements, ignore the stdout and just make sure all elements match.
+ actual_elements = {line for line in actual if line.startswith('*')}
+ expected_elements = {line for line in expected if line.startswith('*')}
+ self.assertEqual(actual_elements, expected_elements)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
new file mode 100644
index 0000000..6f839d4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
@@ -0,0 +1,136 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def partition_function(test=None):
+ # [START partition_function]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ def by_duration(plant, num_partitions):
+ return durations.index(plant['duration'])
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(by_duration, len(durations))
+ )
+ _ = (
+ annuals
+ | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+ )
+ # [END partition_function]
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_lambda(test=None):
+ # [START partition_lambda]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(
+ lambda plant, num_partitions: durations.index(plant['duration']),
+ len(durations),
+ )
+ )
+ _ = (
+ annuals
+ | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+ )
+ # [END partition_lambda]
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_multiple_arguments(test=None):
+ # [START partition_multiple_arguments]
+ import apache_beam as beam
+ import json
+
+ def split_dataset(plant, num_partitions, ratio):
+ assert num_partitions == len(ratio)
+ bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)
+ total = 0
+ for i, part in enumerate(ratio):
+ total += part
+ if bucket < total:
+ return i
+ return len(ratio) - 1
+
+ with beam.Pipeline() as pipeline:
+ train_dataset, test_dataset = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])
+ )
+ _ = (
+ train_dataset
+ | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))
+ )
+ _ = (
+ test_dataset
+ | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))
+ )
+ # [END partition_multiple_arguments]
+ if test:
+ test(train_dataset, test_dataset)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
new file mode 100644
index 0000000..48f83da
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
@@ -0,0 +1,84 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from ..element_wise import partition
+
+
+def check_partitions(actual1, actual2, actual3):
+ # [START partitions]
+ annuals = [
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ biennials = [
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ ]
+ perennials = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END partitions]
+ assert_that(actual1, equal_to(annuals), label='assert annuals')
+ assert_that(actual2, equal_to(biennials), label='assert biennials')
+ assert_that(actual3, equal_to(perennials), label='assert perennials')
+
+
+def check_split_datasets(actual1, actual2):
+ # [START train_test]
+ train_dataset = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ test_dataset = [
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END train_test]
+ assert_that(actual1, equal_to(train_dataset), label='assert train')
+ assert_that(actual2, equal_to(test_dataset), label='assert test')
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.partition.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class PartitionTest(unittest.TestCase):
+ def test_partition_function(self):
+ partition.partition_function(check_partitions)
+
+ def test_partition_lambda(self):
+ partition.partition_lambda(check_partitions)
+
+ def test_partition_multiple_arguments(self):
+ partition.partition_multiple_arguments(check_split_datasets)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py
new file mode 100644
index 0000000..b39b534
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py
@@ -0,0 +1,236 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def regex_matches(test=None):
+ # [START regex_matches]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.matches(regex)
+ | beam.Map(print)
+ )
+ # [END regex_matches]
+ if test:
+ test(plants_matches)
+
+
+def regex_all_matches(test=None):
+ # [START regex_all_matches]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_all_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.all_matches(regex)
+ | beam.Map(print)
+ )
+ # [END regex_all_matches]
+ if test:
+ test(plants_all_matches)
+
+
+def regex_matches_kv(test=None):
+ # [START regex_matches_kv]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches_kv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.matches_kv(regex, keyGroup='icon')
+ | beam.Map(print)
+ )
+ # [END regex_matches_kv]
+ if test:
+ test(plants_matches_kv)
+
+
+def regex_find(test=None):
+ # [START regex_find]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find(regex)
+ | beam.Map(print)
+ )
+ # [END regex_find]
+ if test:
+ test(plants_matches)
+
+
+def regex_find_all(test=None):
+ # [START regex_find_all]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_find_all = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find_all(regex)
+ | beam.Map(print)
+ )
+ # [END regex_find_all]
+ if test:
+ test(plants_find_all)
+
+
+def regex_find_kv(test=None):
+ # [START regex_find_kv]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches_kv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find_kv(regex, keyGroup='icon')
+ | beam.Map(print)
+ )
+ # [END regex_find_kv]
+ if test:
+ test(plants_matches_kv)
+
+
+def regex_replace_all(test=None):
+ # [START regex_replace_all]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_replace_all = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓 : Strawberry : perennial',
+ '🥕 : Carrot : biennial',
+ '🍆\t:\tEggplant\t:\tperennial',
+ '🍅 : Tomato : annual',
+ '🥔 : Potato : perennial',
+ ])
+ | 'To CSV' >> beam.Regex.replace_all(r'\s*:\s*', ',')
+ | beam.Map(print)
+ )
+ # [END regex_replace_all]
+ if test:
+ test(plants_replace_all)
+
+
+def regex_replace_first(test=None):
+ # [START regex_replace_first]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_replace_first = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial',
+ '🍆,\tEggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ ])
+ | 'As dictionary' >> beam.Regex.replace_first(r'\s*,\s*', ': ')
+ | beam.Map(print)
+ )
+ # [END regex_replace_first]
+ if test:
+ test(plants_replace_first)
+
+
+def regex_split(test=None):
+ # [START regex_split]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_split = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓 : Strawberry : perennial',
+ '🥕 : Carrot : biennial',
+ '🍆\t:\tEggplant : perennial',
+ '🍅 : Tomato : annual',
+ '🥔 : Potato : perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.split(r'\s*:\s*')
+ | beam.Map(print)
+ )
+ # [END regex_split]
+ if test:
+ test(plants_split)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py
new file mode 100644
index 0000000..7e2bf78
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py
@@ -0,0 +1,173 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import regex
+
+
+def check_matches(actual):
+ # [START plants_matches]
+ plants_matches = [
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ ]
+ # [END plants_matches]
+ assert_that(actual, equal_to(plants_matches))
+
+
+def check_all_matches(actual):
+ # [START plants_all_matches]
+ plants_all_matches = [
+ ['🍓, Strawberry, perennial', '🍓', 'Strawberry', 'perennial'],
+ ['🥕, Carrot, biennial', '🥕', 'Carrot', 'biennial'],
+ ['🍆, Eggplant, perennial', '🍆', 'Eggplant', 'perennial'],
+ ['🍅, Tomato, annual', '🍅', 'Tomato', 'annual'],
+ ['🥔, Potato, perennial', '🥔', 'Potato', 'perennial'],
+ ]
+ # [END plants_all_matches]
+ assert_that(actual, equal_to(plants_all_matches))
+
+
+def check_matches_kv(actual):
+ # [START plants_matches_kv]
+ plants_matches_kv = [
+ ('🍓', '🍓, Strawberry, perennial'),
+ ('🥕', '🥕, Carrot, biennial'),
+ ('🍆', '🍆, Eggplant, perennial'),
+ ('🍅', '🍅, Tomato, annual'),
+ ('🥔', '🥔, Potato, perennial'),
+ ]
+ # [END plants_matches_kv]
+ assert_that(actual, equal_to(plants_matches_kv))
+
+
+def check_find_all(actual):
+ # [START plants_find_all]
+ plants_find_all = [
+ ['🍓, Strawberry, perennial'],
+ ['🥕, Carrot, biennial'],
+ ['🍆, Eggplant, perennial', '🍌, Banana, perennial'],
+ ['🍅, Tomato, annual', '🍉, Watermelon, annual'],
+ ['🥔, Potato, perennial'],
+ ]
+ # [END plants_find_all]
+ assert_that(actual, equal_to(plants_find_all))
+
+
+def check_find_kv(actual):
+ # [START plants_find_kv]
+ plants_find_all = [
+ ('🍓', '🍓, Strawberry, perennial'),
+ ('🥕', '🥕, Carrot, biennial'),
+ ('🍆', '🍆, Eggplant, perennial'),
+ ('🍌', '🍌, Banana, perennial'),
+ ('🍅', '🍅, Tomato, annual'),
+ ('🍉', '🍉, Watermelon, annual'),
+ ('🥔', '🥔, Potato, perennial'),
+ ]
+ # [END plants_find_kv]
+ assert_that(actual, equal_to(plants_find_all))
+
+
+def check_replace_all(actual):
+ # [START plants_replace_all]
+ plants_replace_all = [
+ '🍓,Strawberry,perennial',
+ '🥕,Carrot,biennial',
+ '🍆,Eggplant,perennial',
+ '🍅,Tomato,annual',
+ '🥔,Potato,perennial',
+ ]
+ # [END plants_replace_all]
+ assert_that(actual, equal_to(plants_replace_all))
+
+
+def check_replace_first(actual):
+ # [START plants_replace_first]
+ plants_replace_first = [
+ '🍓: Strawberry, perennial',
+ '🥕: Carrot, biennial',
+ '🍆: Eggplant, perennial',
+ '🍅: Tomato, annual',
+ '🥔: Potato, perennial',
+ ]
+ # [END plants_replace_first]
+ assert_that(actual, equal_to(plants_replace_first))
+
+
+def check_split(actual):
+ # [START plants_split]
+ plants_split = [
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ]
+ # [END plants_split]
+ assert_that(actual, equal_to(plants_split))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.regex.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class RegexTest(unittest.TestCase):
+ def test_matches(self):
+ regex.regex_matches(check_matches)
+
+ def test_all_matches(self):
+ regex.regex_all_matches(check_all_matches)
+
+ def test_matches_kv(self):
+ regex.regex_matches_kv(check_matches_kv)
+
+ def test_find(self):
+ regex.regex_find(check_matches)
+
+ def test_find_all(self):
+ regex.regex_find_all(check_find_all)
+
+ def test_find_kv(self):
+ regex.regex_find_kv(check_find_kv)
+
+ def test_replace_all(self):
+ regex.regex_replace_all(check_replace_all)
+
+ def test_replace_first(self):
+ regex.regex_replace_first(check_replace_first)
+
+ def test_split(self):
+ regex.regex_split(check_split)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py
new file mode 100644
index 0000000..8504ff4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def values(test=None):
+ # [START values]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Values' >> beam.Values()
+ | beam.Map(print)
+ )
+ # [END values]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py
new file mode 100644
index 0000000..b43d911
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py
@@ -0,0 +1,55 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.examples.snippets.transforms.element_wise.values import *
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.element_wise.values.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ValuesTest(unittest.TestCase):
+ def __init__(self, methodName):
+ super(ValuesTest, self).__init__(methodName)
+ # [START plants]
+ plants = [
+ 'Strawberry',
+ 'Carrot',
+ 'Eggplant',
+ 'Tomato',
+ 'Potato',
+ ]
+ # [END plants]
+ self.plants_test = lambda actual: assert_that(actual, equal_to(plants))
+
+ def test_values(self):
+ values(self.plants_test)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py
new file mode 100644
index 0000000..6569e3f
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py
new file mode 100644
index 0000000..44b11b8
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py
@@ -0,0 +1,182 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def filter_function(test=None):
+ # [START filter_function]
+ import apache_beam as beam
+
+ def is_perennial(plant):
+ return plant['duration'] == 'perennial'
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(is_perennial)
+ | beam.Map(print)
+ )
+ # [END filter_function]
+ if test:
+ test(perennials)
+
+
+def filter_lambda(test=None):
+ # [START filter_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(
+ lambda plant: plant['duration'] == 'perennial')
+ | beam.Map(print)
+ )
+ # [END filter_lambda]
+ if test:
+ test(perennials)
+
+
+def filter_multiple_arguments(test=None):
+ # [START filter_multiple_arguments]
+ import apache_beam as beam
+
+ def has_duration(plant, duration):
+ return plant['duration'] == duration
+
+ with beam.Pipeline() as pipeline:
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(has_duration, 'perennial')
+ | beam.Map(print)
+ )
+ # [END filter_multiple_arguments]
+ if test:
+ test(perennials)
+
+
+def filter_side_inputs_singleton(test=None):
+ # [START filter_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ perennial = pipeline | 'Perennial' >> beam.Create(['perennial'])
+
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter perennials' >> beam.Filter(
+ lambda plant, duration: plant['duration'] == duration,
+ duration=beam.pvalue.AsSingleton(perennial),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_singleton]
+ if test:
+ test(perennials)
+
+
+def filter_side_inputs_iter(test=None):
+ # [START filter_side_inputs_iter]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ valid_durations = pipeline | 'Valid durations' >> beam.Create([
+ 'annual',
+ 'biennial',
+ 'perennial',
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'PERENNIAL'},
+ ])
+ | 'Filter valid plants' >> beam.Filter(
+ lambda plant, valid_durations: plant['duration'] in valid_durations,
+ valid_durations=beam.pvalue.AsIter(valid_durations),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_iter]
+ if test:
+ test(valid_plants)
+
+
+def filter_side_inputs_dict(test=None):
+ # [START filter_side_inputs_dict]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ keep_duration = pipeline | 'Duration filters' >> beam.Create([
+ ('annual', False),
+ ('biennial', False),
+ ('perennial', True),
+ ])
+
+ perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Filter plants by duration' >> beam.Filter(
+ lambda plant, keep_duration: keep_duration[plant['duration']],
+ keep_duration=beam.pvalue.AsDict(keep_duration),
+ )
+ | beam.Map(print)
+ )
+ # [END filter_side_inputs_dict]
+ if test:
+ test(perennials)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
new file mode 100644
index 0000000..d989e43
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
@@ -0,0 +1,81 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import filter
+
+
+def check_perennials(actual):
+ # [START perennials]
+ perennials = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END perennials]
+ assert_that(actual, equal_to(perennials))
+
+
+def check_valid_plants(actual):
+ # [START valid_plants]
+ valid_plants = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END valid_plants]
+ assert_that(actual, equal_to(valid_plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.filter.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class FilterTest(unittest.TestCase):
+ def test_filter_function(self):
+ filter.filter_function(check_perennials)
+
+ def test_filter_lambda(self):
+ filter.filter_lambda(check_perennials)
+
+ def test_filter_multiple_arguments(self):
+ filter.filter_multiple_arguments(check_perennials)
+
+ def test_filter_side_inputs_singleton(self):
+ filter.filter_side_inputs_singleton(check_perennials)
+
+ def test_filter_side_inputs_iter(self):
+ filter.filter_side_inputs_iter(check_valid_plants)
+
+ def test_filter_side_inputs_dict(self):
+ filter.filter_side_inputs_dict(check_perennials)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
new file mode 100644
index 0000000..50ffe7a
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
@@ -0,0 +1,248 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def flatmap_simple(test=None):
+ # [START flatmap_simple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry 🥕Carrot 🍆Eggplant',
+ '🍅Tomato 🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(str.split)
+ | beam.Map(print)
+ )
+ # [END flatmap_simple]
+ if test:
+ test(plants)
+
+
+def flatmap_function(test=None):
+ # [START flatmap_function]
+ import apache_beam as beam
+
+ def split_words(text):
+ return text.split(',')
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(split_words)
+ | beam.Map(print)
+ )
+ # [END flatmap_function]
+ if test:
+ test(plants)
+
+
+def flatmap_lambda(test=None):
+ # [START flatmap_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],
+ ['🍅Tomato', '🥔Potato'],
+ ])
+ | 'Flatten lists' >> beam.FlatMap(lambda elements: elements)
+ | beam.Map(print)
+ )
+ # [END flatmap_lambda]
+ if test:
+ test(plants)
+
+
+def flatmap_generator(test=None):
+ # [START flatmap_generator]
+ import apache_beam as beam
+
+ def generate_elements(elements):
+ for element in elements:
+ yield element
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ['🍓Strawberry', '🥕Carrot', '🍆Eggplant'],
+ ['🍅Tomato', '🥔Potato'],
+ ])
+ | 'Flatten lists' >> beam.FlatMap(generate_elements)
+ | beam.Map(print)
+ )
+ # [END flatmap_generator]
+ if test:
+ test(plants)
+
+
+def flatmap_multiple_arguments(test=None):
+ # [START flatmap_multiple_arguments]
+ import apache_beam as beam
+
+ def split_words(text, delimiter=None):
+ return text.split(delimiter)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(split_words, delimiter=',')
+ | beam.Map(print)
+ )
+ # [END flatmap_multiple_arguments]
+ if test:
+ test(plants)
+
+
+def flatmap_tuple(test=None):
+ # [START flatmap_tuple]
+ import apache_beam as beam
+
+ def format_plant(icon, plant):
+ if icon:
+ yield '{}{}'.format(icon, plant)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ (None, 'Invalid'),
+ ])
+ | 'Format' >> beam.FlatMapTuple(format_plant)
+ | beam.Map(print)
+ )
+ # [END flatmap_tuple]
+ if test:
+ test(plants)
+
+
+def flatmap_side_inputs_singleton(test=None):
+ # [START flatmap_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ delimiter = pipeline | 'Create delimiter' >> beam.Create([','])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.FlatMap(
+ lambda text, delimiter: text.split(delimiter),
+ delimiter=beam.pvalue.AsSingleton(delimiter),
+ )
+ | beam.Map(print)
+ )
+ # [END flatmap_side_inputs_singleton]
+ if test:
+ test(plants)
+
+
+def flatmap_side_inputs_iter(test=None):
+ # [START flatmap_side_inputs_iter]
+ import apache_beam as beam
+
+ def normalize_and_validate_durations(plant, valid_durations):
+ plant['duration'] = plant['duration'].lower()
+ if plant['duration'] in valid_durations:
+ yield plant
+
+ with beam.Pipeline() as pipeline:
+ valid_durations = pipeline | 'Valid durations' >> beam.Create([
+ 'annual',
+ 'biennial',
+ 'perennial',
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'Perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'BIENNIAL'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'unknown'},
+ ])
+ | 'Normalize and validate durations' >> beam.FlatMap(
+ normalize_and_validate_durations,
+ valid_durations=beam.pvalue.AsIter(valid_durations),
+ )
+ | beam.Map(print)
+ )
+ # [END flatmap_side_inputs_iter]
+ if test:
+ test(valid_plants)
+
+
+def flatmap_side_inputs_dict(test=None):
+ # [START flatmap_side_inputs_dict]
+ import apache_beam as beam
+
+ def replace_duration_if_valid(plant, durations):
+ if plant['duration'] in durations:
+ plant['duration'] = durations[plant['duration']]
+ yield plant
+
+ with beam.Pipeline() as pipeline:
+ durations = pipeline | 'Durations dict' >> beam.Create([
+ (0, 'annual'),
+ (1, 'biennial'),
+ (2, 'perennial'),
+ ])
+
+ valid_plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 1},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 0},
+ {'icon': '🥔', 'name': 'Potato', 'duration': -1},
+ ])
+ | 'Replace duration if valid' >> beam.FlatMap(
+ replace_duration_if_valid,
+ durations=beam.pvalue.AsDict(durations),
+ )
+ | beam.Map(print)
+ )
+ # [END flatmap_side_inputs_dict]
+ if test:
+ test(valid_plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
new file mode 100644
index 0000000..718dcee
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
@@ -0,0 +1,92 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import flatmap
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_valid_plants(actual):
+ # [START valid_plants]
+ valid_plants = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END valid_plants]
+ assert_that(actual, equal_to(valid_plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.flatmap.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class FlatMapTest(unittest.TestCase):
+ def test_flatmap_simple(self):
+ flatmap.flatmap_simple(check_plants)
+
+ def test_flatmap_function(self):
+ flatmap.flatmap_function(check_plants)
+
+ def test_flatmap_lambda(self):
+ flatmap.flatmap_lambda(check_plants)
+
+ def test_flatmap_generator(self):
+ flatmap.flatmap_generator(check_plants)
+
+ def test_flatmap_multiple_arguments(self):
+ flatmap.flatmap_multiple_arguments(check_plants)
+
+ def test_flatmap_tuple(self):
+ flatmap.flatmap_tuple(check_plants)
+
+ def test_flatmap_side_inputs_singleton(self):
+ flatmap.flatmap_side_inputs_singleton(check_plants)
+
+ def test_flatmap_side_inputs_iter(self):
+ flatmap.flatmap_side_inputs_iter(check_valid_plants)
+
+ def test_flatmap_side_inputs_dict(self):
+ flatmap.flatmap_side_inputs_dict(check_valid_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py
new file mode 100644
index 0000000..01c9d6b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def keys(test=None):
+ # [START keys]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ icons = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Keys' >> beam.Keys()
+ | beam.Map(print)
+ )
+ # [END keys]
+ if test:
+ test(icons)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
new file mode 100644
index 0000000..780c5e4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
@@ -0,0 +1,56 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import keys
+
+
+def check_icons(actual):
+ # [START icons]
+ icons = [
+ '🍓',
+ '🥕',
+ '🍆',
+ '🍅',
+ '🥔',
+ ]
+ # [END icons]
+ assert_that(actual, equal_to(icons))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.keys.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class KeysTest(unittest.TestCase):
+ def test_keys(self):
+ keys.keys(check_icons)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py
new file mode 100644
index 0000000..2107fd5
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def kvswap(test=None):
+ # [START kvswap]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Key-Value swap' >> beam.KvSwap()
+ | beam.Map(print)
+ )
+ # [END kvswap]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
new file mode 100644
index 0000000..ea7698b
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
@@ -0,0 +1,56 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import kvswap
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ ('Strawberry', '🍓'),
+ ('Carrot', '🥕'),
+ ('Eggplant', '🍆'),
+ ('Tomato', '🍅'),
+ ('Potato', '🥔'),
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.kvswap.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class KvSwapTest(unittest.TestCase):
+ def test_kvswap(self):
+ kvswap.kvswap(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py
new file mode 100644
index 0000000..9defd47
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py
@@ -0,0 +1,226 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def map_simple(test=None):
+ # [START map_simple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ' 🍓Strawberry \n',
+ ' 🥕Carrot \n',
+ ' 🍆Eggplant \n',
+ ' 🍅Tomato \n',
+ ' 🥔Potato \n',
+ ])
+ | 'Strip' >> beam.Map(str.strip)
+ | beam.Map(print)
+ )
+ # [END map_simple]
+ if test:
+ test(plants)
+
+
+def map_function(test=None):
+ # [START map_function]
+ import apache_beam as beam
+
+ def strip_header_and_newline(text):
+ return text.strip('# \n')
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(strip_header_and_newline)
+ | beam.Map(print)
+ )
+ # [END map_function]
+ if test:
+ test(plants)
+
+
+def map_lambda(test=None):
+ # [START map_lambda]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(lambda text: text.strip('# \n'))
+ | beam.Map(print)
+ )
+ # [END map_lambda]
+ if test:
+ test(plants)
+
+
+def map_multiple_arguments(test=None):
+ # [START map_multiple_arguments]
+ import apache_beam as beam
+
+ def strip(text, chars=None):
+ return text.strip(chars)
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(strip, chars='# \n')
+ | beam.Map(print)
+ )
+ # [END map_multiple_arguments]
+ if test:
+ test(plants)
+
+
+def map_tuple(test=None):
+ # [START map_tuple]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Format' >> beam.MapTuple(
+ lambda icon, plant: '{}{}'.format(icon, plant))
+ | beam.Map(print)
+ )
+ # [END map_tuple]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_singleton(test=None):
+ # [START map_side_inputs_singleton]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ chars = pipeline | 'Create chars' >> beam.Create(['# \n'])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(
+ lambda text, chars: text.strip(chars),
+ chars=beam.pvalue.AsSingleton(chars),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_singleton]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_iter(test=None):
+ # [START map_side_inputs_iter]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ chars = pipeline | 'Create chars' >> beam.Create(['#', ' ', '\n'])
+
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '# 🍓Strawberry\n',
+ '# 🥕Carrot\n',
+ '# 🍆Eggplant\n',
+ '# 🍅Tomato\n',
+ '# 🥔Potato\n',
+ ])
+ | 'Strip header' >> beam.Map(
+ lambda text, chars: text.strip(''.join(chars)),
+ chars=beam.pvalue.AsIter(chars),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_iter]
+ if test:
+ test(plants)
+
+
+def map_side_inputs_dict(test=None):
+ # [START map_side_inputs_dict]
+ import apache_beam as beam
+
+ def replace_duration(plant, durations):
+ plant['duration'] = durations[plant['duration']]
+ return plant
+
+ with beam.Pipeline() as pipeline:
+ durations = pipeline | 'Durations' >> beam.Create([
+ (0, 'annual'),
+ (1, 'biennial'),
+ (2, 'perennial'),
+ ])
+
+ plant_details = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 2},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 1},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 2},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 0},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 2},
+ ])
+ | 'Replace duration' >> beam.Map(
+ replace_duration,
+ durations=beam.pvalue.AsDict(durations),
+ )
+ | beam.Map(print)
+ )
+ # [END map_side_inputs_dict]
+ if test:
+ test(plant_details)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
new file mode 100644
index 0000000..4186176
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
@@ -0,0 +1,90 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import map
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_plant_details(actual):
+ # [START plant_details]
+ plant_details = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END plant_details]
+ assert_that(actual, equal_to(plant_details))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.map.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class MapTest(unittest.TestCase):
+ def test_map_simple(self):
+ map.map_simple(check_plants)
+
+ def test_map_function(self):
+ map.map_function(check_plants)
+
+ def test_map_lambda(self):
+ map.map_lambda(check_plants)
+
+ def test_map_multiple_arguments(self):
+ map.map_multiple_arguments(check_plants)
+
+ def test_map_tuple(self):
+ map.map_tuple(check_plants)
+
+ def test_map_side_inputs_singleton(self):
+ map.map_side_inputs_singleton(check_plants)
+
+ def test_map_side_inputs_iter(self):
+ map.map_side_inputs_iter(check_plants)
+
+ def test_map_side_inputs_dict(self):
+ map.map_side_inputs_dict(check_plant_details)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
new file mode 100644
index 0000000..971e9f0
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
@@ -0,0 +1,126 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+
+def pardo_dofn(test=None):
+ # [START pardo_dofn]
+ import apache_beam as beam
+
+ class SplitWords(beam.DoFn):
+ def __init__(self, delimiter=','):
+ self.delimiter = delimiter
+
+ def process(self, text):
+ for word in text.split(self.delimiter):
+ yield word
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ '🍓Strawberry,🥕Carrot,🍆Eggplant',
+ '🍅Tomato,🥔Potato',
+ ])
+ | 'Split words' >> beam.ParDo(SplitWords(','))
+ | beam.Map(print)
+ )
+ # [END pardo_dofn]
+ if test:
+ test(plants)
+
+
+def pardo_dofn_params(test=None):
+ # pylint: disable=line-too-long
+ # [START pardo_dofn_params]
+ import apache_beam as beam
+
+ class AnalyzeElement(beam.DoFn):
+ def process(self, elem, timestamp=beam.DoFn.TimestampParam, window=beam.DoFn.WindowParam):
+ yield '\n'.join([
+ '# timestamp',
+ 'type(timestamp) -> ' + repr(type(timestamp)),
+ 'timestamp.micros -> ' + repr(timestamp.micros),
+ 'timestamp.to_rfc3339() -> ' + repr(timestamp.to_rfc3339()),
+ 'timestamp.to_utc_datetime() -> ' + repr(timestamp.to_utc_datetime()),
+ '',
+ '# window',
+ 'type(window) -> ' + repr(type(window)),
+ 'window.start -> {} ({})'.format(window.start, window.start.to_utc_datetime()),
+ 'window.end -> {} ({})'.format(window.end, window.end.to_utc_datetime()),
+ 'window.max_timestamp() -> {} ({})'.format(window.max_timestamp(), window.max_timestamp().to_utc_datetime()),
+ ])
+
+ with beam.Pipeline() as pipeline:
+ dofn_params = (
+ pipeline
+ | 'Create a single test element' >> beam.Create([':)'])
+ | 'Add timestamp (Spring equinox 2020)' >> beam.Map(
+ lambda elem: beam.window.TimestampedValue(elem, 1584675660))
+ | 'Fixed 30sec windows' >> beam.WindowInto(beam.window.FixedWindows(30))
+ | 'Analyze element' >> beam.ParDo(AnalyzeElement())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_params]
+ # pylint: enable=line-too-long
+ if test:
+ test(dofn_params)
+
+
+def pardo_dofn_methods(test=None):
+ # [START pardo_dofn_methods]
+ import apache_beam as beam
+
+ class DoFnMethods(beam.DoFn):
+ def __init__(self):
+ print('__init__')
+ self.window = beam.window.GlobalWindow()
+
+ def setup(self):
+ print('setup')
+
+ def start_bundle(self):
+ print('start_bundle')
+
+ def process(self, element, window=beam.DoFn.WindowParam):
+ self.window = window
+ yield '* process: ' + element
+
+ def finish_bundle(self):
+ yield beam.utils.windowed_value.WindowedValue(
+ value='* finish_bundle: 🌱🌳🌍',
+ timestamp=0,
+ windows=[self.window],
+ )
+
+ def teardown(self):
+ print('teardown')
+
+ with beam.Pipeline() as pipeline:
+ results = (
+ pipeline
+ | 'Create inputs' >> beam.Create(['🍓', '🥕', '🍆', '🍅', '🥔'])
+ | 'DoFn methods' >> beam.ParDo(DoFnMethods())
+ | beam.Map(print)
+ )
+ # [END pardo_dofn_methods]
+ if test:
+ return test(results)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
new file mode 100644
index 0000000..8507e01
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
@@ -0,0 +1,120 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+from __future__ import unicode_literals
+
+import io
+import platform
+import sys
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import pardo
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓Strawberry',
+ '🥕Carrot',
+ '🍆Eggplant',
+ '🍅Tomato',
+ '🥔Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_dofn_params(actual):
+ # pylint: disable=line-too-long
+ dofn_params = '\n'.join('''[START dofn_params]
+# timestamp
+type(timestamp) -> <class 'apache_beam.utils.timestamp.Timestamp'>
+timestamp.micros -> 1584675660000000
+timestamp.to_rfc3339() -> '2020-03-20T03:41:00Z'
+timestamp.to_utc_datetime() -> datetime.datetime(2020, 3, 20, 3, 41)
+
+# window
+type(window) -> <class 'apache_beam.transforms.window.IntervalWindow'>
+window.start -> Timestamp(1584675660) (2020-03-20 03:41:00)
+window.end -> Timestamp(1584675690) (2020-03-20 03:41:30)
+window.max_timestamp() -> Timestamp(1584675689.999999) (2020-03-20 03:41:29.999999)
+[END dofn_params]'''.splitlines()[1:-1])
+ # pylint: enable=line-too-long
+ assert_that(actual, equal_to([dofn_params]))
+
+
+def check_dofn_methods(actual):
+ # Return the expected stdout to check the ordering of the called methods.
+ return '''[START results]
+__init__
+setup
+start_bundle
+* process: 🍓
+* process: 🥕
+* process: 🍆
+* process: 🍅
+* process: 🥔
+* finish_bundle: 🌱🌳🌍
+teardown
+[END results]'''.splitlines()[1:-1]
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.pardo.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ParDoTest(unittest.TestCase):
+ def test_pardo_dofn(self):
+ pardo.pardo_dofn(check_plants)
+
+ # TODO: Remove this after Python 2 deprecation.
+ # https://issues.apache.org/jira/browse/BEAM-8124
+ @unittest.skipIf(sys.version_info[0] < 3 and platform.system() == 'Windows',
+ 'Python 2 on Windows uses `long` rather than `int`')
+ def test_pardo_dofn_params(self):
+ pardo.pardo_dofn_params(check_dofn_params)
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+@mock.patch('sys.stdout', new_callable=io.StringIO)
+class ParDoStdoutTest(unittest.TestCase):
+ def test_pardo_dofn_methods(self, mock_stdout):
+ expected = pardo.pardo_dofn_methods(check_dofn_methods)
+ actual = mock_stdout.getvalue().splitlines()
+
+ # For the stdout, check the ordering of the methods, not of the elements.
+ actual_stdout = [line.split(':')[0] for line in actual]
+ expected_stdout = [line.split(':')[0] for line in expected]
+ self.assertEqual(actual_stdout, expected_stdout)
+
+ # For the elements, ignore the stdout and just make sure all elements match.
+ actual_elements = {line for line in actual if line.startswith('*')}
+ expected_elements = {line for line in expected if line.startswith('*')}
+ self.assertEqual(actual_elements, expected_elements)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
new file mode 100644
index 0000000..6f839d4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
@@ -0,0 +1,136 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def partition_function(test=None):
+ # [START partition_function]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ def by_duration(plant, num_partitions):
+ return durations.index(plant['duration'])
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(by_duration, len(durations))
+ )
+ _ = (
+ annuals
+ | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+ )
+ # [END partition_function]
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_lambda(test=None):
+ # [START partition_lambda]
+ import apache_beam as beam
+
+ durations = ['annual', 'biennial', 'perennial']
+
+ with beam.Pipeline() as pipeline:
+ annuals, biennials, perennials = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(
+ lambda plant, num_partitions: durations.index(plant['duration']),
+ len(durations),
+ )
+ )
+ _ = (
+ annuals
+ | 'Annuals' >> beam.Map(lambda x: print('annual: ' + str(x)))
+ )
+ _ = (
+ biennials
+ | 'Biennials' >> beam.Map(lambda x: print('biennial: ' + str(x)))
+ )
+ _ = (
+ perennials
+ | 'Perennials' >> beam.Map(lambda x: print('perennial: ' + str(x)))
+ )
+ # [END partition_lambda]
+ if test:
+ test(annuals, biennials, perennials)
+
+
+def partition_multiple_arguments(test=None):
+ # [START partition_multiple_arguments]
+ import apache_beam as beam
+ import json
+
+ def split_dataset(plant, num_partitions, ratio):
+ assert num_partitions == len(ratio)
+ bucket = sum(map(ord, json.dumps(plant))) % sum(ratio)
+ total = 0
+ for i, part in enumerate(ratio):
+ total += part
+ if bucket < total:
+ return i
+ return len(ratio) - 1
+
+ with beam.Pipeline() as pipeline:
+ train_dataset, test_dataset = (
+ pipeline
+ | 'Gardening plants' >> beam.Create([
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ])
+ | 'Partition' >> beam.Partition(split_dataset, 2, ratio=[8, 2])
+ )
+ _ = (
+ train_dataset
+ | 'Train' >> beam.Map(lambda x: print('train: ' + str(x)))
+ )
+ _ = (
+ test_dataset
+ | 'Test' >> beam.Map(lambda x: print('test: ' + str(x)))
+ )
+ # [END partition_multiple_arguments]
+ if test:
+ test(train_dataset, test_dataset)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
new file mode 100644
index 0000000..0b8ae3d
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
@@ -0,0 +1,84 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import partition
+
+
+def check_partitions(actual1, actual2, actual3):
+ # [START partitions]
+ annuals = [
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ biennials = [
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ ]
+ perennials = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ # [END partitions]
+ assert_that(actual1, equal_to(annuals), label='assert annuals')
+ assert_that(actual2, equal_to(biennials), label='assert biennials')
+ assert_that(actual3, equal_to(perennials), label='assert perennials')
+
+
+def check_split_datasets(actual1, actual2):
+ # [START train_test]
+ train_dataset = [
+ {'icon': '🍓', 'name': 'Strawberry', 'duration': 'perennial'},
+ {'icon': '🥕', 'name': 'Carrot', 'duration': 'biennial'},
+ {'icon': '🥔', 'name': 'Potato', 'duration': 'perennial'},
+ ]
+ test_dataset = [
+ {'icon': '🍆', 'name': 'Eggplant', 'duration': 'perennial'},
+ {'icon': '🍅', 'name': 'Tomato', 'duration': 'annual'},
+ ]
+ # [END train_test]
+ assert_that(actual1, equal_to(train_dataset), label='assert train')
+ assert_that(actual2, equal_to(test_dataset), label='assert test')
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.partition.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class PartitionTest(unittest.TestCase):
+ def test_partition_function(self):
+ partition.partition_function(check_partitions)
+
+ def test_partition_lambda(self):
+ partition.partition_lambda(check_partitions)
+
+ def test_partition_multiple_arguments(self):
+ partition.partition_multiple_arguments(check_split_datasets)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
new file mode 100644
index 0000000..b39b534
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
@@ -0,0 +1,236 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def regex_matches(test=None):
+ # [START regex_matches]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.matches(regex)
+ | beam.Map(print)
+ )
+ # [END regex_matches]
+ if test:
+ test(plants_matches)
+
+
+def regex_all_matches(test=None):
+ # [START regex_all_matches]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_all_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.all_matches(regex)
+ | beam.Map(print)
+ )
+ # [END regex_all_matches]
+ if test:
+ test(plants_all_matches)
+
+
+def regex_matches_kv(test=None):
+ # [START regex_matches_kv]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches_kv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial ignoring trailing words',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ '# 🍌, invalid, format',
+ 'invalid, 🍉, format',
+ ])
+ | 'Parse plants' >> beam.Regex.matches_kv(regex, keyGroup='icon')
+ | beam.Map(print)
+ )
+ # [END regex_matches_kv]
+ if test:
+ test(plants_matches_kv)
+
+
+def regex_find(test=None):
+ # [START regex_find]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find(regex)
+ | beam.Map(print)
+ )
+ # [END regex_find]
+ if test:
+ test(plants_matches)
+
+
+def regex_find_all(test=None):
+ # [START regex_find_all]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_find_all = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find_all(regex)
+ | beam.Map(print)
+ )
+ # [END regex_find_all]
+ if test:
+ test(plants_find_all)
+
+
+def regex_find_kv(test=None):
+ # [START regex_find_kv]
+ import apache_beam as beam
+
+ # Matches a named group 'icon', and then two comma-separated groups.
+ regex = r'(?P<icon>[^\s,]+), *(\w+), *(\w+)'
+ with beam.Pipeline() as pipeline:
+ plants_matches_kv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '# 🍓, Strawberry, perennial',
+ '# 🥕, Carrot, biennial ignoring trailing words',
+ '# 🍆, Eggplant, perennial - 🍌, Banana, perennial',
+ '# 🍅, Tomato, annual - 🍉, Watermelon, annual',
+ '# 🥔, Potato, perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.find_kv(regex, keyGroup='icon')
+ | beam.Map(print)
+ )
+ # [END regex_find_kv]
+ if test:
+ test(plants_matches_kv)
+
+
+def regex_replace_all(test=None):
+ # [START regex_replace_all]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_replace_all = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓 : Strawberry : perennial',
+ '🥕 : Carrot : biennial',
+ '🍆\t:\tEggplant\t:\tperennial',
+ '🍅 : Tomato : annual',
+ '🥔 : Potato : perennial',
+ ])
+ | 'To CSV' >> beam.Regex.replace_all(r'\s*:\s*', ',')
+ | beam.Map(print)
+ )
+ # [END regex_replace_all]
+ if test:
+ test(plants_replace_all)
+
+
+def regex_replace_first(test=None):
+ # [START regex_replace_first]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_replace_first = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial',
+ '🍆,\tEggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ ])
+ | 'As dictionary' >> beam.Regex.replace_first(r'\s*,\s*', ': ')
+ | beam.Map(print)
+ )
+ # [END regex_replace_first]
+ if test:
+ test(plants_replace_first)
+
+
+def regex_split(test=None):
+ # [START regex_split]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_split = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ '🍓 : Strawberry : perennial',
+ '🥕 : Carrot : biennial',
+ '🍆\t:\tEggplant : perennial',
+ '🍅 : Tomato : annual',
+ '🥔 : Potato : perennial',
+ ])
+ | 'Parse plants' >> beam.Regex.split(r'\s*:\s*')
+ | beam.Map(print)
+ )
+ # [END regex_split]
+ if test:
+ test(plants_split)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
new file mode 100644
index 0000000..8312312
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
@@ -0,0 +1,173 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import regex
+
+
+def check_matches(actual):
+ # [START plants_matches]
+ plants_matches = [
+ '🍓, Strawberry, perennial',
+ '🥕, Carrot, biennial',
+ '🍆, Eggplant, perennial',
+ '🍅, Tomato, annual',
+ '🥔, Potato, perennial',
+ ]
+ # [END plants_matches]
+ assert_that(actual, equal_to(plants_matches))
+
+
+def check_all_matches(actual):
+ # [START plants_all_matches]
+ plants_all_matches = [
+ ['🍓, Strawberry, perennial', '🍓', 'Strawberry', 'perennial'],
+ ['🥕, Carrot, biennial', '🥕', 'Carrot', 'biennial'],
+ ['🍆, Eggplant, perennial', '🍆', 'Eggplant', 'perennial'],
+ ['🍅, Tomato, annual', '🍅', 'Tomato', 'annual'],
+ ['🥔, Potato, perennial', '🥔', 'Potato', 'perennial'],
+ ]
+ # [END plants_all_matches]
+ assert_that(actual, equal_to(plants_all_matches))
+
+
+def check_matches_kv(actual):
+ # [START plants_matches_kv]
+ plants_matches_kv = [
+ ('🍓', '🍓, Strawberry, perennial'),
+ ('🥕', '🥕, Carrot, biennial'),
+ ('🍆', '🍆, Eggplant, perennial'),
+ ('🍅', '🍅, Tomato, annual'),
+ ('🥔', '🥔, Potato, perennial'),
+ ]
+ # [END plants_matches_kv]
+ assert_that(actual, equal_to(plants_matches_kv))
+
+
+def check_find_all(actual):
+ # [START plants_find_all]
+ plants_find_all = [
+ ['🍓, Strawberry, perennial'],
+ ['🥕, Carrot, biennial'],
+ ['🍆, Eggplant, perennial', '🍌, Banana, perennial'],
+ ['🍅, Tomato, annual', '🍉, Watermelon, annual'],
+ ['🥔, Potato, perennial'],
+ ]
+ # [END plants_find_all]
+ assert_that(actual, equal_to(plants_find_all))
+
+
+def check_find_kv(actual):
+ # [START plants_find_kv]
+ plants_find_all = [
+ ('🍓', '🍓, Strawberry, perennial'),
+ ('🥕', '🥕, Carrot, biennial'),
+ ('🍆', '🍆, Eggplant, perennial'),
+ ('🍌', '🍌, Banana, perennial'),
+ ('🍅', '🍅, Tomato, annual'),
+ ('🍉', '🍉, Watermelon, annual'),
+ ('🥔', '🥔, Potato, perennial'),
+ ]
+ # [END plants_find_kv]
+ assert_that(actual, equal_to(plants_find_all))
+
+
+def check_replace_all(actual):
+ # [START plants_replace_all]
+ plants_replace_all = [
+ '🍓,Strawberry,perennial',
+ '🥕,Carrot,biennial',
+ '🍆,Eggplant,perennial',
+ '🍅,Tomato,annual',
+ '🥔,Potato,perennial',
+ ]
+ # [END plants_replace_all]
+ assert_that(actual, equal_to(plants_replace_all))
+
+
+def check_replace_first(actual):
+ # [START plants_replace_first]
+ plants_replace_first = [
+ '🍓: Strawberry, perennial',
+ '🥕: Carrot, biennial',
+ '🍆: Eggplant, perennial',
+ '🍅: Tomato, annual',
+ '🥔: Potato, perennial',
+ ]
+ # [END plants_replace_first]
+ assert_that(actual, equal_to(plants_replace_first))
+
+
+def check_split(actual):
+ # [START plants_split]
+ plants_split = [
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ]
+ # [END plants_split]
+ assert_that(actual, equal_to(plants_split))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.regex.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class RegexTest(unittest.TestCase):
+ def test_matches(self):
+ regex.regex_matches(check_matches)
+
+ def test_all_matches(self):
+ regex.regex_all_matches(check_all_matches)
+
+ def test_matches_kv(self):
+ regex.regex_matches_kv(check_matches_kv)
+
+ def test_find(self):
+ regex.regex_find(check_matches)
+
+ def test_find_all(self):
+ regex.regex_find_all(check_find_all)
+
+ def test_find_kv(self):
+ regex.regex_find_kv(check_find_kv)
+
+ def test_replace_all(self):
+ regex.regex_replace_all(check_replace_all)
+
+ def test_replace_first(self):
+ regex.regex_replace_first(check_replace_first)
+
+ def test_split(self):
+ regex.regex_split(check_split)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
new file mode 100644
index 0000000..1d0b7dd
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
@@ -0,0 +1,86 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def tostring_kvs(test=None):
+ # [START tostring_kvs]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'To string' >> beam.ToString.Kvs()
+ | beam.Map(print)
+ )
+ # [END tostring_kvs]
+ if test:
+ test(plants)
+
+
+def tostring_element(test=None):
+ # [START tostring_element]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plant_lists = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ])
+ | 'To string' >> beam.ToString.Element()
+ | beam.Map(print)
+ )
+ # [END tostring_element]
+ if test:
+ test(plant_lists)
+
+
+def tostring_iterables(test=None):
+ # [START tostring_iterables]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants_csv = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ['🍓', 'Strawberry', 'perennial'],
+ ['🥕', 'Carrot', 'biennial'],
+ ['🍆', 'Eggplant', 'perennial'],
+ ['🍅', 'Tomato', 'annual'],
+ ['🥔', 'Potato', 'perennial'],
+ ])
+ | 'To string' >> beam.ToString.Iterables()
+ | beam.Map(print)
+ )
+ # [END tostring_iterables]
+ if test:
+ test(plants_csv)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
new file mode 100644
index 0000000..b253ea1
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
@@ -0,0 +1,105 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import sys
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import tostring
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ '🍓,Strawberry',
+ '🥕,Carrot',
+ '🍆,Eggplant',
+ '🍅,Tomato',
+ '🥔,Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+def check_plant_lists(actual):
+ # [START plant_lists]
+ plant_lists = [
+ "['🍓', 'Strawberry', 'perennial']",
+ "['🥕', 'Carrot', 'biennial']",
+ "['🍆', 'Eggplant', 'perennial']",
+ "['🍅', 'Tomato', 'annual']",
+ "['🥔', 'Potato', 'perennial']",
+ ]
+ # [END plant_lists]
+
+ # Some unicode characters become escaped with double backslashes.
+ import apache_beam as beam
+
+ def normalize_escaping(elem):
+ # In Python 2 all utf-8 characters are escaped with double backslashes.
+ # TODO: Remove this after Python 2 deprecation.
+ # https://issues.apache.org/jira/browse/BEAM-8124
+ if sys.version_info.major == 2:
+ return elem.decode('string-escape')
+
+ # In Python 3.5 some utf-8 characters are escaped with double backslashes.
+ if '\\' in elem:
+ return bytes(elem, 'utf-8').decode('unicode-escape')
+ return elem
+ actual = actual | beam.Map(normalize_escaping)
+ assert_that(actual, equal_to(plant_lists))
+
+
+def check_plants_csv(actual):
+ # [START plants_csv]
+ plants_csv = [
+ '🍓,Strawberry,perennial',
+ '🥕,Carrot,biennial',
+ '🍆,Eggplant,perennial',
+ '🍅,Tomato,annual',
+ '🥔,Potato,perennial',
+ ]
+ # [END plants_csv]
+ assert_that(actual, equal_to(plants_csv))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.tostring.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ToStringTest(unittest.TestCase):
+ def test_tostring_kvs(self):
+ tostring.tostring_kvs(check_plants)
+
+ def test_tostring_element(self):
+ tostring.tostring_element(check_plant_lists)
+
+ def test_tostring_iterables(self):
+ tostring.tostring_iterables(check_plants_csv)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py
new file mode 100644
index 0000000..8504ff4
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py
@@ -0,0 +1,42 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def values(test=None):
+ # [START values]
+ import apache_beam as beam
+
+ with beam.Pipeline() as pipeline:
+ plants = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ ('🍓', 'Strawberry'),
+ ('🥕', 'Carrot'),
+ ('🍆', 'Eggplant'),
+ ('🍅', 'Tomato'),
+ ('🥔', 'Potato'),
+ ])
+ | 'Values' >> beam.Values()
+ | beam.Map(print)
+ )
+ # [END values]
+ if test:
+ test(plants)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
new file mode 100644
index 0000000..06abef6
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
@@ -0,0 +1,56 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import values
+
+
+def check_plants(actual):
+ # [START plants]
+ plants = [
+ 'Strawberry',
+ 'Carrot',
+ 'Eggplant',
+ 'Tomato',
+ 'Potato',
+ ]
+ # [END plants]
+ assert_that(actual, equal_to(plants))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.values.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class ValuesTest(unittest.TestCase):
+ def test_values(self):
+ values.values(check_plants)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
new file mode 100644
index 0000000..79a9c44
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
@@ -0,0 +1,128 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+
+def withtimestamps_event_time(test=None):
+ # [START withtimestamps_event_time]
+ import apache_beam as beam
+
+ class GetTimestamp(beam.DoFn):
+ def process(self, plant, timestamp=beam.DoFn.TimestampParam):
+ yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])
+
+ with beam.Pipeline() as pipeline:
+ plant_timestamps = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ {'name': 'Strawberry', 'season': 1585699200}, # April, 2020
+ {'name': 'Carrot', 'season': 1590969600}, # June, 2020
+ {'name': 'Artichoke', 'season': 1583020800}, # March, 2020
+ {'name': 'Tomato', 'season': 1588291200}, # May, 2020
+ {'name': 'Potato', 'season': 1598918400}, # September, 2020
+ ])
+ | 'With timestamps' >> beam.Map(
+ lambda plant: beam.window.TimestampedValue(plant, plant['season']))
+ | 'Get timestamp' >> beam.ParDo(GetTimestamp())
+ | beam.Map(print)
+ )
+ # [END withtimestamps_event_time]
+ if test:
+ test(plant_timestamps)
+
+
+def withtimestamps_logical_clock(test=None):
+ # [START withtimestamps_logical_clock]
+ import apache_beam as beam
+
+ class GetTimestamp(beam.DoFn):
+ def process(self, plant, timestamp=beam.DoFn.TimestampParam):
+ event_id = int(timestamp.micros / 1e6) # equivalent to seconds
+ yield '{} - {}'.format(event_id, plant['name'])
+
+ with beam.Pipeline() as pipeline:
+ plant_events = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ {'name': 'Strawberry', 'event_id': 1},
+ {'name': 'Carrot', 'event_id': 4},
+ {'name': 'Artichoke', 'event_id': 2},
+ {'name': 'Tomato', 'event_id': 3},
+ {'name': 'Potato', 'event_id': 5},
+ ])
+ | 'With timestamps' >> beam.Map(lambda plant: \
+ beam.window.TimestampedValue(plant, plant['event_id']))
+ | 'Get timestamp' >> beam.ParDo(GetTimestamp())
+ | beam.Map(print)
+ )
+ # [END withtimestamps_logical_clock]
+ if test:
+ test(plant_events)
+
+
+def withtimestamps_processing_time(test=None):
+ # [START withtimestamps_processing_time]
+ import apache_beam as beam
+ import time
+
+ class GetTimestamp(beam.DoFn):
+ def process(self, plant, timestamp=beam.DoFn.TimestampParam):
+ yield '{} - {}'.format(timestamp.to_utc_datetime(), plant['name'])
+
+ with beam.Pipeline() as pipeline:
+ plant_processing_times = (
+ pipeline
+ | 'Garden plants' >> beam.Create([
+ {'name': 'Strawberry'},
+ {'name': 'Carrot'},
+ {'name': 'Artichoke'},
+ {'name': 'Tomato'},
+ {'name': 'Potato'},
+ ])
+ | 'With timestamps' >> beam.Map(lambda plant: \
+ beam.window.TimestampedValue(plant, time.time()))
+ | 'Get timestamp' >> beam.ParDo(GetTimestamp())
+ | beam.Map(print)
+ )
+ # [END withtimestamps_processing_time]
+ if test:
+ test(plant_processing_times)
+
+
+def time_tuple2unix_time():
+ # [START time_tuple2unix_time]
+ import time
+
+ time_tuple = time.strptime('2020-03-19 20:50:00', '%Y-%m-%d %H:%M:%S')
+ unix_time = time.mktime(time_tuple)
+ # [END time_tuple2unix_time]
+ return unix_time
+
+
+def datetime2unix_time():
+ # [START datetime2unix_time]
+ import time
+ import datetime
+
+ now = datetime.datetime.now()
+ time_tuple = now.timetuple()
+ unix_time = time.mktime(time_tuple)
+ # [END datetime2unix_time]
+ return unix_time
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
new file mode 100644
index 0000000..53fa7e2
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
@@ -0,0 +1,103 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+import mock
+
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+from . import withtimestamps
+
+
+def check_plant_timestamps(actual):
+ # [START plant_timestamps]
+ plant_timestamps = [
+ '2020-04-01 00:00:00 - Strawberry',
+ '2020-06-01 00:00:00 - Carrot',
+ '2020-03-01 00:00:00 - Artichoke',
+ '2020-05-01 00:00:00 - Tomato',
+ '2020-09-01 00:00:00 - Potato',
+ ]
+ # [END plant_timestamps]
+ assert_that(actual, equal_to(plant_timestamps))
+
+
+def check_plant_events(actual):
+ # [START plant_events]
+ plant_events = [
+ '1 - Strawberry',
+ '4 - Carrot',
+ '2 - Artichoke',
+ '3 - Tomato',
+ '5 - Potato',
+ ]
+ # [END plant_events]
+ assert_that(actual, equal_to(plant_events))
+
+
+def check_plant_processing_times(actual):
+ import apache_beam as beam
+
+ # [START plant_processing_times]
+ plant_processing_times = [
+ '2020-03-20 20:12:42.145594 - Strawberry',
+ '2020-03-20 20:12:42.145827 - Carrot',
+ '2020-03-20 20:12:42.145962 - Artichoke',
+ '2020-03-20 20:12:42.146093 - Tomato',
+ '2020-03-20 20:12:42.146216 - Potato',
+ ]
+ # [END plant_processing_times]
+
+ # Since `time.time()` will always give something different, we'll
+ # simply strip the timestamp information before testing the results.
+ actual = actual | beam.Map(lambda row: row.split('-')[-1].strip())
+ expected = [row.split('-')[-1].strip() for row in plant_processing_times]
+ assert_that(actual, equal_to(expected))
+
+
+@mock.patch('apache_beam.Pipeline', TestPipeline)
+# pylint: disable=line-too-long
+@mock.patch('apache_beam.examples.snippets.transforms.elementwise.withtimestamps.print', lambda elem: elem)
+# pylint: enable=line-too-long
+class WithTimestampsTest(unittest.TestCase):
+ def test_event_time(self):
+ withtimestamps.withtimestamps_event_time(check_plant_timestamps)
+
+ def test_logical_clock(self):
+ withtimestamps.withtimestamps_logical_clock(check_plant_events)
+
+ def test_processing_time(self):
+ withtimestamps.withtimestamps_processing_time(check_plant_processing_times)
+
+ def test_time_tuple2unix_time(self):
+ unix_time = withtimestamps.time_tuple2unix_time()
+ self.assertIsInstance(unix_time, float)
+
+ def test_datetime2unix_time(self):
+ unix_time = withtimestamps.datetime2unix_time()
+ self.assertIsInstance(unix_time, float)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/snippets/util.py b/sdks/python/apache_beam/examples/snippets/util.py
new file mode 100644
index 0000000..6e6e9e0
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/util.py
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import argparse
+import shlex
+import subprocess as sp
+
+
+def parse_example(argv=None):
+ """Parse the command line arguments and return it as a string function call.
+
+ Examples:
+ python path/to/snippets.py function_name
+ python path/to/snippets.py function_name arg1
+ python path/to/snippets.py function_name arg1 arg2 ... argN
+ """
+ parser = argparse.ArgumentParser()
+ parser.add_argument('example', help='Name of the example to run.')
+ parser.add_argument('args', nargs=argparse.REMAINDER,
+ help='Arguments for example.')
+ args = parser.parse_args(argv)
+
+ # Return the example as a string representing the Python function call.
+ example_args = ', '.join([repr(arg) for arg in args.args])
+ return '{}({})'.format(args.example, example_args)
+
+
+def run_shell_commands(commands, **kwargs):
+ """Runs a list of Notebook-like shell commands.
+
+ Lines starting with `#` are ignored as comments.
+ Lines starting with `!` are run as commands.
+ Variables like `{variable}` are substituted with **kwargs.
+ """
+ for cmd in commands:
+ cmd = cmd.strip().lstrip('!').format(**kwargs)
+ sp_cmd = shlex.split(cmd, comments=True, posix=True)
+ if sp_cmd:
+ sp.call(sp_cmd)
+ yield sp_cmd
diff --git a/sdks/python/apache_beam/examples/snippets/util_test.py b/sdks/python/apache_beam/examples/snippets/util_test.py
new file mode 100644
index 0000000..a23e916
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/util_test.py
@@ -0,0 +1,67 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import unittest
+
+from mock import patch
+
+from apache_beam.examples.snippets.util import *
+
+
+class UtilTest(unittest.TestCase):
+ def test_parse_example_empty(self):
+ # python path/to/snippets.py
+ argv = []
+ with self.assertRaises(SystemExit):
+ self.assertEqual(parse_example(argv), 'example()')
+
+ def test_parse_example_no_arguments(self):
+ # python path/to/snippets.py example
+ argv = ['example']
+ self.assertEqual(parse_example(argv), 'example()')
+
+ def test_parse_example_one_argument(self):
+ # python path/to/snippets.py example A
+ argv = ['example', 'A']
+ self.assertEqual(parse_example(argv), "example('A')")
+
+ def test_parse_example_multiple_arguments(self):
+ # python path/to/snippets.py example A B "C's"
+ argv = ['example', 'A', 'B', "C's"]
+ self.assertEqual(parse_example(argv), "example('A', 'B', \"C's\")")
+
+ @patch('subprocess.call', lambda cmd: None)
+ def test_run_shell_commands(self):
+ commands = [
+ ' # this is a comment ',
+ ' ! echo this is a shell command ',
+ ' !echo {variable} ',
+ ' echo "quoted arguments work" # trailing comment ',
+ ]
+ actual = list(run_shell_commands(commands, variable='hello world'))
+ expected = [
+ ['echo', 'this', 'is', 'a', 'shell', 'command'],
+ ['echo', 'hello', 'world'],
+ ['echo', 'quoted arguments work'],
+ ]
+ self.assertEqual(actual, expected)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/examples/streaming_wordcount.py b/sdks/python/apache_beam/examples/streaming_wordcount.py
index d1ecc63..f0db06a 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount.py
@@ -33,13 +33,13 @@
from apache_beam.options.pipeline_options import StandardOptions
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Build and run the pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument(
'--output_topic', required=True,
help=('Output PubSub topic of the form '
- '"projects/<PROJECT>/topic/<TOPIC>".'))
+ '"projects/<PROJECT>/topics/<TOPIC>".'))
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument(
'--input_topic',
@@ -54,7 +54,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
diff --git a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py b/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
index fabb773..d87d0f4 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py
@@ -20,8 +20,6 @@
from __future__ import absolute_import
import logging
-import os
-import sys
import unittest
import uuid
from builtins import range
@@ -42,7 +40,7 @@
OUTPUT_SUB = 'wc_subscription_output'
DEFAULT_INPUT_NUMBERS = 500
-WAIT_UNTIL_FINISH_DURATION = 3 * 60 * 1000 # in milliseconds
+WAIT_UNTIL_FINISH_DURATION = 6 * 60 * 1000 # in milliseconds
class StreamingWordCountIT(unittest.TestCase):
@@ -66,7 +64,8 @@
self.input_topic.name)
self.output_sub = self.sub_client.create_subscription(
self.sub_client.subscription_path(self.project, OUTPUT_SUB + self.uuid),
- self.output_topic.name)
+ self.output_topic.name,
+ ack_deadline_seconds=60)
def _inject_numbers(self, topic, num_messages):
"""Inject numbers as test data to PubSub."""
@@ -80,14 +79,11 @@
test_utils.cleanup_topics(self.pub_client,
[self.input_topic, self.output_topic])
- @unittest.skipIf(sys.version[0:3] == '3.6' and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6 '
- 'TODO: BEAM-7181')
@attr('IT')
def test_streaming_wordcount_it(self):
# Build expected dataset.
- expected_msg = [('%d: 1' % num) for num in range(DEFAULT_INPUT_NUMBERS)]
+ expected_msg = [('%d: 1' % num).encode('utf-8')
+ for num in range(DEFAULT_INPUT_NUMBERS)]
# Set extra options to the pipeline for test purpose
state_verifier = PipelineStateMatcher(PipelineState.RUNNING)
@@ -107,7 +103,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
streaming_wordcount.run(
- self.test_pipeline.get_full_options_as_args(**extra_opts))
+ self.test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/wordcount.py b/sdks/python/apache_beam/examples/wordcount.py
index 0f0cefd..a8f17e3 100644
--- a/sdks/python/apache_beam/examples/wordcount.py
+++ b/sdks/python/apache_beam/examples/wordcount.py
@@ -38,6 +38,9 @@
"""Parse each line of input text into words."""
def __init__(self):
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(WordExtractingDoFn, self).__init__()
+ beam.DoFn.__init__(self)
self.words_counter = Metrics.counter(self.__class__, 'words')
self.word_lengths_counter = Metrics.counter(self.__class__, 'word_lengths')
self.word_lengths_dist = Metrics.distribution(
@@ -66,7 +69,7 @@
return words
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
parser.add_argument('--input',
@@ -82,7 +85,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
diff --git a/sdks/python/apache_beam/examples/wordcount_debugging.py b/sdks/python/apache_beam/examples/wordcount_debugging.py
index 23bb52b..389bdd6 100644
--- a/sdks/python/apache_beam/examples/wordcount_debugging.py
+++ b/sdks/python/apache_beam/examples/wordcount_debugging.py
@@ -60,7 +60,9 @@
class FilterTextFn(beam.DoFn):
"""A DoFn that filters for a specific key based on a regular expression."""
def __init__(self, pattern):
- super(FilterTextFn, self).__init__()
+ # TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.
+ # super(FilterTextFn, self).__init__()
+ beam.DoFn.__init__(self)
self.pattern = pattern
# A custom metric can track values in your pipeline as it runs. Those
# values will be available in the monitoring system of the runner used
@@ -107,7 +109,7 @@
| 'count' >> beam.Map(count_ones))
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the debugging wordcount pipeline."""
parser = argparse.ArgumentParser()
@@ -123,7 +125,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection, count the occurrences of
diff --git a/sdks/python/apache_beam/examples/wordcount_debugging_test.py b/sdks/python/apache_beam/examples/wordcount_debugging_test.py
index cf824d1..124b680 100644
--- a/sdks/python/apache_beam/examples/wordcount_debugging_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_debugging_test.py
@@ -28,7 +28,7 @@
from apache_beam.testing.util import open_shards
-class WordCountTest(unittest.TestCase):
+class WordCountDebuggingTest(unittest.TestCase):
SAMPLE_TEXT = 'xx yy Flourish\n zz Flourish Flourish stomach\n aa\n bb cc dd'
@@ -49,9 +49,10 @@
def test_basics(self):
temp_path = self.create_temp_file(self.SAMPLE_TEXT)
expected_words = [('Flourish', 3), ('stomach', 1)]
- wordcount_debugging.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount_debugging.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = self.get_results(temp_path)
diff --git a/sdks/python/apache_beam/examples/wordcount_it_test.py b/sdks/python/apache_beam/examples/wordcount_it_test.py
index 8a79443..8a9b2c5 100644
--- a/sdks/python/apache_beam/examples/wordcount_it_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_it_test.py
@@ -81,7 +81,8 @@
# Get pipeline options from command argument: --test-pipeline-options,
# and start pipeline job by calling pipeline main function.
- run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts))
+ run_wordcount(test_pipeline.get_full_options_as_args(**extra_opts),
+ save_main_session=False)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/examples/wordcount_minimal.py b/sdks/python/apache_beam/examples/wordcount_minimal.py
index ef66a38..2bfb6ec 100644
--- a/sdks/python/apache_beam/examples/wordcount_minimal.py
+++ b/sdks/python/apache_beam/examples/wordcount_minimal.py
@@ -59,7 +59,7 @@
from apache_beam.options.pipeline_options import SetupOptions
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
@@ -93,7 +93,7 @@
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection.
diff --git a/sdks/python/apache_beam/examples/wordcount_minimal_test.py b/sdks/python/apache_beam/examples/wordcount_minimal_test.py
index 011e678..9a772d5 100644
--- a/sdks/python/apache_beam/examples/wordcount_minimal_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_minimal_test.py
@@ -44,9 +44,10 @@
expected_words = collections.defaultdict(int)
for word in re.findall(r'\w+', self.SAMPLE_TEXT):
expected_words[word] += 1
- wordcount_minimal.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount_minimal.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/wordcount_test.py b/sdks/python/apache_beam/examples/wordcount_test.py
index aa131cb..84b14f2 100644
--- a/sdks/python/apache_beam/examples/wordcount_test.py
+++ b/sdks/python/apache_beam/examples/wordcount_test.py
@@ -45,9 +45,10 @@
expected_words = collections.defaultdict(int)
for word in re.findall(r'[\w\']+', self.SAMPLE_TEXT, re.UNICODE):
expected_words[word] += 1
- wordcount.run([
- '--input=%s*' % temp_path,
- '--output=%s.result' % temp_path])
+ wordcount.run(
+ ['--input=%s*' % temp_path,
+ '--output=%s.result' % temp_path],
+ save_main_session=False)
# Parse result file and compare.
results = []
with open_shards(temp_path + '.result-*-of-*') as result_file:
diff --git a/sdks/python/apache_beam/examples/wordcount_xlang.py b/sdks/python/apache_beam/examples/wordcount_xlang.py
index f4071f8..fe1994d 100644
--- a/sdks/python/apache_beam/examples/wordcount_xlang.py
+++ b/sdks/python/apache_beam/examples/wordcount_xlang.py
@@ -70,7 +70,7 @@
| 'split' >> (beam.ParDo(WordExtractingDoFn())
.with_output_types(bytes))
| 'count' >> beam.ExternalTransform(
- 'pytest:beam:transforms:count', None, EXPANSION_SERVICE_ADDR))
+ 'beam:transforms:xlang:count', None, EXPANSION_SERVICE_ADDR))
# Format the counts into a PCollection of strings.
def format_result(word_count):
@@ -116,7 +116,8 @@
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
- p.runner.init_dockerized_job_server()
+ # Preemptively start due to BEAM-6666.
+ p.runner.create_job_service(pipeline_options)
try:
server = subprocess.Popen([
diff --git a/sdks/python/apache_beam/internal/gcp/auth.py b/sdks/python/apache_beam/internal/gcp/auth.py
index 1c6d8e2..f8c1254 100644
--- a/sdks/python/apache_beam/internal/gcp/auth.py
+++ b/sdks/python/apache_beam/internal/gcp/auth.py
@@ -19,18 +19,20 @@
from __future__ import absolute_import
-import datetime
-import json
import logging
-import os
+import socket
+import threading
-from future.moves.urllib.request import Request
-from future.moves.urllib.request import urlopen
from oauth2client.client import GoogleCredentials
-from oauth2client.client import OAuth2Credentials
from apache_beam.utils import retry
+# Protect against environments where apitools library is not available.
+try:
+ from apitools.base.py.credentials_lib import GceAssertionCredentials
+except ImportError:
+ GceAssertionCredentials = None
+
# When we are running in GCE, we can authenticate with VM credentials.
is_running_in_gce = False
@@ -38,6 +40,19 @@
# information.
executing_project = None
+if GceAssertionCredentials is not None:
+ class _GceAssertionCredentials(GceAssertionCredentials):
+ """GceAssertionCredentials with retry wrapper.
+
+ For internal use only; no backwards-compatibility guarantees.
+ """
+
+ @retry.with_exponential_backoff(
+ retry_filter=retry.retry_on_server_errors_and_timeout_filter)
+ def _do_refresh_request(self, http_request):
+ return super(_GceAssertionCredentials, self)._do_refresh_request(
+ http_request)
+
def set_running_in_gce(worker_executing_project):
"""For internal use only; no backwards-compatibility guarantees.
@@ -57,77 +72,66 @@
executing_project = worker_executing_project
-class AuthenticationException(retry.PermanentException):
- pass
-
-
-class _GCEMetadataCredentials(OAuth2Credentials):
- """For internal use only; no backwards-compatibility guarantees.
-
- Credential object initialized using access token from GCE VM metadata."""
-
- def __init__(self, user_agent=None):
- """Create an instance of GCEMetadataCredentials.
-
- These credentials are generated by contacting the metadata server on a GCE
- VM instance.
-
- Args:
- user_agent: string, The HTTP User-Agent to provide for this application.
- """
- super(_GCEMetadataCredentials, self).__init__(
- None, # access_token
- None, # client_id
- None, # client_secret
- None, # refresh_token
- datetime.datetime(2010, 1, 1), # token_expiry, set to time in past.
- None, # token_uri
- user_agent)
-
- @retry.with_exponential_backoff(
- retry_filter=retry.retry_on_server_errors_and_timeout_filter)
- def _refresh(self, http_request):
- refresh_time = datetime.datetime.utcnow()
- metadata_root = os.environ.get(
- 'GCE_METADATA_ROOT', 'metadata.google.internal')
- token_url = ('http://{}/computeMetadata/v1/instance/service-accounts/'
- 'default/token').format(metadata_root)
- req = Request(token_url, headers={'Metadata-Flavor': 'Google'})
- token_data = json.loads(urlopen(req).read().decode('utf-8'))
- self.access_token = token_data['access_token']
- self.token_expiry = (refresh_time +
- datetime.timedelta(seconds=token_data['expires_in']))
-
-
def get_service_credentials():
"""For internal use only; no backwards-compatibility guarantees.
- Get credentials to access Google services."""
- user_agent = 'beam-python-sdk/1.0'
- if is_running_in_gce:
- # We are currently running as a GCE taskrunner worker.
- #
- # TODO(ccy): It's not entirely clear if these credentials are thread-safe.
- # If so, we can cache these credentials to save the overhead of creating
- # them again.
- return _GCEMetadataCredentials(user_agent=user_agent)
- else:
- client_scopes = [
- 'https://www.googleapis.com/auth/bigquery',
- 'https://www.googleapis.com/auth/cloud-platform',
- 'https://www.googleapis.com/auth/devstorage.full_control',
- 'https://www.googleapis.com/auth/userinfo.email',
- 'https://www.googleapis.com/auth/datastore'
- ]
+ Get credentials to access Google services.
- try:
- credentials = GoogleCredentials.get_application_default()
- credentials = credentials.create_scoped(client_scopes)
- logging.debug('Connecting using Google Application Default '
- 'Credentials.')
- return credentials
- except Exception as e:
- logging.warning(
- 'Unable to find default credentials to use: %s\n'
- 'Connecting anonymously.', e)
- return None
+ Returns:
+ A ``oauth2client.client.OAuth2Credentials`` object or None if credentials
+ not found. Returned object is thread-safe.
+ """
+ return _Credentials.get_service_credentials()
+
+
+class _Credentials(object):
+ _credentials_lock = threading.Lock()
+ _credentials_init = False
+ _credentials = None
+
+ @classmethod
+ def get_service_credentials(cls):
+ if cls._credentials_init:
+ return cls._credentials
+
+ with cls._credentials_lock:
+ if cls._credentials_init:
+ return cls._credentials
+
+ # apitools use urllib with the global timeout. Set it to 60 seconds
+ # to prevent network related stuckness issues.
+ if not socket.getdefaulttimeout():
+ logging.info("Setting socket default timeout to 60 seconds.")
+ socket.setdefaulttimeout(60)
+ logging.info(
+ "socket default timeout is % seconds.", socket.getdefaulttimeout())
+
+ cls._credentials = cls._get_service_credentials()
+ cls._credentials_init = True
+
+ return cls._credentials
+
+ @staticmethod
+ def _get_service_credentials():
+ if is_running_in_gce:
+ # We are currently running as a GCE taskrunner worker.
+ return _GceAssertionCredentials(user_agent='beam-python-sdk/1.0')
+ else:
+ client_scopes = [
+ 'https://www.googleapis.com/auth/bigquery',
+ 'https://www.googleapis.com/auth/cloud-platform',
+ 'https://www.googleapis.com/auth/devstorage.full_control',
+ 'https://www.googleapis.com/auth/userinfo.email',
+ 'https://www.googleapis.com/auth/datastore'
+ ]
+ try:
+ credentials = GoogleCredentials.get_application_default()
+ credentials = credentials.create_scoped(client_scopes)
+ logging.debug('Connecting using Google Application Default '
+ 'Credentials.')
+ return credentials
+ except Exception as e:
+ logging.warning(
+ 'Unable to find default credentials to use: %s\n'
+ 'Connecting anonymously.', e)
+ return None
diff --git a/sdks/python/apache_beam/internal/gcp/json_value_test.py b/sdks/python/apache_beam/internal/gcp/json_value_test.py
index e6d064a..5605d41 100644
--- a/sdks/python/apache_beam/internal/gcp/json_value_test.py
+++ b/sdks/python/apache_beam/internal/gcp/json_value_test.py
@@ -39,65 +39,65 @@
class JsonValueTest(unittest.TestCase):
def test_string_to(self):
- self.assertEquals(JsonValue(string_value='abc'), to_json_value('abc'))
+ self.assertEqual(JsonValue(string_value='abc'), to_json_value('abc'))
def test_bytes_to(self):
- self.assertEquals(JsonValue(string_value='abc'), to_json_value(b'abc'))
+ self.assertEqual(JsonValue(string_value='abc'), to_json_value(b'abc'))
def test_true_to(self):
- self.assertEquals(JsonValue(boolean_value=True), to_json_value(True))
+ self.assertEqual(JsonValue(boolean_value=True), to_json_value(True))
def test_false_to(self):
- self.assertEquals(JsonValue(boolean_value=False), to_json_value(False))
+ self.assertEqual(JsonValue(boolean_value=False), to_json_value(False))
def test_int_to(self):
- self.assertEquals(JsonValue(integer_value=14), to_json_value(14))
+ self.assertEqual(JsonValue(integer_value=14), to_json_value(14))
def test_float_to(self):
- self.assertEquals(JsonValue(double_value=2.75), to_json_value(2.75))
+ self.assertEqual(JsonValue(double_value=2.75), to_json_value(2.75))
def test_static_value_provider_to(self):
svp = StaticValueProvider(str, 'abc')
- self.assertEquals(JsonValue(string_value=svp.value), to_json_value(svp))
+ self.assertEqual(JsonValue(string_value=svp.value), to_json_value(svp))
def test_runtime_value_provider_to(self):
RuntimeValueProvider.runtime_options = None
rvp = RuntimeValueProvider('arg', 123, int)
- self.assertEquals(JsonValue(is_null=True), to_json_value(rvp))
+ self.assertEqual(JsonValue(is_null=True), to_json_value(rvp))
def test_none_to(self):
- self.assertEquals(JsonValue(is_null=True), to_json_value(None))
+ self.assertEqual(JsonValue(is_null=True), to_json_value(None))
def test_string_from(self):
- self.assertEquals('WXYZ', from_json_value(to_json_value('WXYZ')))
+ self.assertEqual('WXYZ', from_json_value(to_json_value('WXYZ')))
def test_true_from(self):
- self.assertEquals(True, from_json_value(to_json_value(True)))
+ self.assertEqual(True, from_json_value(to_json_value(True)))
def test_false_from(self):
- self.assertEquals(False, from_json_value(to_json_value(False)))
+ self.assertEqual(False, from_json_value(to_json_value(False)))
def test_int_from(self):
- self.assertEquals(-27, from_json_value(to_json_value(-27)))
+ self.assertEqual(-27, from_json_value(to_json_value(-27)))
def test_float_from(self):
- self.assertEquals(4.5, from_json_value(to_json_value(4.5)))
+ self.assertEqual(4.5, from_json_value(to_json_value(4.5)))
def test_with_type(self):
rt = from_json_value(to_json_value('abcd', with_type=True))
- self.assertEquals('http://schema.org/Text', rt['@type'])
- self.assertEquals('abcd', rt['value'])
+ self.assertEqual('http://schema.org/Text', rt['@type'])
+ self.assertEqual('abcd', rt['value'])
def test_none_from(self):
self.assertIsNone(from_json_value(to_json_value(None)))
def test_large_integer(self):
num = 1 << 35
- self.assertEquals(num, from_json_value(to_json_value(num)))
+ self.assertEqual(num, from_json_value(to_json_value(num)))
def test_long_value(self):
num = 1 << 63 - 1
- self.assertEquals(num, from_json_value(to_json_value(num)))
+ self.assertEqual(num, from_json_value(to_json_value(num)))
def test_too_long_value(self):
with self.assertRaises(TypeError):
diff --git a/sdks/python/apache_beam/internal/http_client_test.py b/sdks/python/apache_beam/internal/http_client_test.py
index 9755edf..c3c0f83 100644
--- a/sdks/python/apache_beam/internal/http_client_test.py
+++ b/sdks/python/apache_beam/internal/http_client_test.py
@@ -35,54 +35,54 @@
with mock.patch.dict(os.environ, http_proxy='http://localhost:9000'):
proxy_info = proxy_info_from_environment_var('http_proxy')
expected = ProxyInfo(3, 'localhost', 9000)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_https_with_port(self):
with mock.patch.dict(os.environ, https_proxy='https://localhost:9000'):
proxy_info = proxy_info_from_environment_var('https_proxy')
expected = ProxyInfo(3, 'localhost', 9000)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_http_without_port(self):
with mock.patch.dict(os.environ, http_proxy='http://localhost'):
proxy_info = proxy_info_from_environment_var('http_proxy')
expected = ProxyInfo(3, 'localhost', 80)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_https_without_port(self):
with mock.patch.dict(os.environ, https_proxy='https://localhost'):
proxy_info = proxy_info_from_environment_var('https_proxy')
expected = ProxyInfo(3, 'localhost', 443)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_http_without_method(self):
with mock.patch.dict(os.environ, http_proxy='localhost:8000'):
proxy_info = proxy_info_from_environment_var('http_proxy')
expected = ProxyInfo(3, 'localhost', 8000)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_https_without_method(self):
with mock.patch.dict(os.environ, https_proxy='localhost:8000'):
proxy_info = proxy_info_from_environment_var('https_proxy')
expected = ProxyInfo(3, 'localhost', 8000)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_http_without_port_without_method(self):
with mock.patch.dict(os.environ, http_proxy='localhost'):
proxy_info = proxy_info_from_environment_var('http_proxy')
expected = ProxyInfo(3, 'localhost', 80)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_https_without_port_without_method(self):
with mock.patch.dict(os.environ, https_proxy='localhost'):
proxy_info = proxy_info_from_environment_var('https_proxy')
expected = ProxyInfo(3, 'localhost', 443)
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_invalid_var(self):
proxy_info = proxy_info_from_environment_var('http_proxy_host')
expected = None
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_proxy_from_env_wrong_method_in_var_name(self):
with mock.patch.dict(os.environ, smtp_proxy='localhost'):
@@ -93,17 +93,17 @@
with mock.patch.dict(os.environ, http_proxy='smtp://localhost:8000'):
proxy_info = proxy_info_from_environment_var('http_proxy')
expected = ProxyInfo(3, 'smtp', 80) # wrong proxy info generated
- self.assertEquals(str(expected), str(proxy_info))
+ self.assertEqual(str(expected), str(proxy_info))
def test_get_new_http_proxy_info(self):
with mock.patch.dict(os.environ, http_proxy='localhost'):
http = get_new_http()
expected = ProxyInfo(3, 'localhost', 80)
- self.assertEquals(str(http.proxy_info), str(expected))
+ self.assertEqual(str(http.proxy_info), str(expected))
def test_get_new_http_timeout(self):
http = get_new_http()
- self.assertEquals(http.timeout, DEFAULT_HTTP_TIMEOUT_SECONDS)
+ self.assertEqual(http.timeout, DEFAULT_HTTP_TIMEOUT_SECONDS)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/internal/module_test.py b/sdks/python/apache_beam/internal/module_test.py
index baed965..e2fee34 100644
--- a/sdks/python/apache_beam/internal/module_test.py
+++ b/sdks/python/apache_beam/internal/module_test.py
@@ -20,6 +20,7 @@
from __future__ import absolute_import
import re
+import sys
from builtins import object
@@ -76,3 +77,12 @@
RecursiveClass.SELF_TYPE = RecursiveClass
+
+# pylint: disable=exec-used
+if sys.version_info >= (3, 7):
+ # create dataclass to be pickled
+ exec('''
+from dataclasses import dataclass
+@dataclass
+class DataClass:
+ datum: str''')
diff --git a/sdks/python/apache_beam/internal/pickler.py b/sdks/python/apache_beam/internal/pickler.py
index 29e4f4b..ec8db53 100644
--- a/sdks/python/apache_beam/internal/pickler.py
+++ b/sdks/python/apache_beam/internal/pickler.py
@@ -198,6 +198,13 @@
return old_save_module_dict(pickler, obj)
dill.dill.save_module_dict = new_save_module_dict
+ if hasattr(types, "MappingProxyType"):
+ @dill.dill.register(types.MappingProxyType)
+ def save_mappingproxy(pickler, obj):
+ # pickling mappingproxy AS IS, not as dict
+ # inspired from https://github.com/cloudpipe/cloudpickle/pull/245
+ pickler.save_reduce(types.MappingProxyType, (dict(obj),), obj=obj)
+
def _nest_dill_logging():
"""Prefix all dill logging with its depth in the callstack.
diff --git a/sdks/python/apache_beam/internal/pickler_test.py b/sdks/python/apache_beam/internal/pickler_test.py
index 79208aa..e18c726 100644
--- a/sdks/python/apache_beam/internal/pickler_test.py
+++ b/sdks/python/apache_beam/internal/pickler_test.py
@@ -19,6 +19,8 @@
from __future__ import absolute_import
+import sys
+import types
import unittest
from apache_beam.internal import module_test
@@ -28,53 +30,55 @@
class PicklerTest(unittest.TestCase):
+ NO_MAPPINGPROXYTYPE = not hasattr(types, "MappingProxyType")
+
def test_basics(self):
- self.assertEquals([1, 'a', (u'z',)], loads(dumps([1, 'a', (u'z',)])))
+ self.assertEqual([1, 'a', (u'z',)], loads(dumps([1, 'a', (u'z',)])))
fun = lambda x: 'xyz-%s' % x
- self.assertEquals('xyz-abc', loads(dumps(fun))('abc'))
+ self.assertEqual('xyz-abc', loads(dumps(fun))('abc'))
def test_lambda_with_globals(self):
"""Tests that the globals of a function are preserved."""
# The point of the test is that the lambda being called after unpickling
# relies on having the re module being loaded.
- self.assertEquals(
+ self.assertEqual(
['abc', 'def'],
loads(dumps(module_test.get_lambda_with_globals()))('abc def'))
def test_lambda_with_main_globals(self):
- self.assertEquals(unittest, loads(dumps(lambda: unittest))())
+ self.assertEqual(unittest, loads(dumps(lambda: unittest))())
def test_lambda_with_closure(self):
"""Tests that the closure of a function is preserved."""
- self.assertEquals(
+ self.assertEqual(
'closure: abc',
loads(dumps(module_test.get_lambda_with_closure('abc')))())
def test_class(self):
"""Tests that a class object is pickled correctly."""
- self.assertEquals(
+ self.assertEqual(
['abc', 'def'],
loads(dumps(module_test.Xyz))().foo('abc def'))
def test_object(self):
"""Tests that a class instance is pickled correctly."""
- self.assertEquals(
+ self.assertEqual(
['abc', 'def'],
loads(dumps(module_test.XYZ_OBJECT)).foo('abc def'))
def test_nested_class(self):
"""Tests that a nested class object is pickled correctly."""
- self.assertEquals(
+ self.assertEqual(
'X:abc',
loads(dumps(module_test.TopClass.NestedClass('abc'))).datum)
- self.assertEquals(
+ self.assertEqual(
'Y:abc',
loads(dumps(module_test.TopClass.MiddleClass.NestedClass('abc'))).datum)
def test_dynamic_class(self):
"""Tests that a nested class object is pickled correctly."""
- self.assertEquals(
+ self.assertEqual(
'Z:abc',
loads(dumps(module_test.create_class('abc'))).get())
@@ -83,8 +87,23 @@
dumps((_ for _ in range(10)))
def test_recursive_class(self):
- self.assertEquals('RecursiveClass:abc',
- loads(dumps(module_test.RecursiveClass('abc').datum)))
+ self.assertEqual('RecursiveClass:abc',
+ loads(dumps(module_test.RecursiveClass('abc').datum)))
+
+ @unittest.skipIf(NO_MAPPINGPROXYTYPE, 'test if MappingProxyType introduced')
+ def test_dump_and_load_mapping_proxy(self):
+ self.assertEqual(
+ 'def', loads(dumps(types.MappingProxyType({'abc': 'def'})))['abc'])
+ self.assertEqual(types.MappingProxyType,
+ type(loads(dumps(types.MappingProxyType({})))))
+
+ # pylint: disable=exec-used
+ @unittest.skipIf(sys.version_info < (3, 7), 'Python 3.7 or above only')
+ def test_dataclass(self):
+ exec('''
+from apache_beam.internal.module_test import DataClass
+self.assertEqual(DataClass(datum='abc'), loads(dumps(DataClass(datum='abc'))))
+ ''')
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/internal/util.py b/sdks/python/apache_beam/internal/util.py
index 8081ad4..499214f 100644
--- a/sdks/python/apache_beam/internal/util.py
+++ b/sdks/python/apache_beam/internal/util.py
@@ -62,7 +62,7 @@
return hash(type(self))
-def remove_objects_from_args(args, kwargs, pvalue_classes):
+def remove_objects_from_args(args, kwargs, pvalue_class):
"""For internal use only; no backwards-compatibility guarantees.
Replaces all objects of a given type in args/kwargs with a placeholder.
@@ -70,9 +70,8 @@
Args:
args: A list of positional arguments.
kwargs: A dictionary of keyword arguments.
- pvalue_classes: A tuple of class objects representing the types of the
- arguments that must be replaced with a placeholder value (instance of
- ArgumentPlaceholder)
+ pvalue_class: A class object representing the types of arguments that must
+ be replaced with a placeholder value (instance of ArgumentPlaceholder).
Returns:
A 3-tuple containing a modified list of positional arguments, a modified
@@ -84,11 +83,11 @@
def swapper(value):
pvals.append(value)
return ArgumentPlaceholder()
- new_args = [swapper(v) if isinstance(v, pvalue_classes) else v for v in args]
+ new_args = [swapper(v) if isinstance(v, pvalue_class) else v for v in args]
# Make sure the order in which we process the dictionary keys is predictable
# by sorting the entries first. This will be important when putting back
# PValues.
- new_kwargs = dict((k, swapper(v)) if isinstance(v, pvalue_classes) else (k, v)
+ new_kwargs = dict((k, swapper(v)) if isinstance(v, pvalue_class) else (k, v)
for k, v in sorted(kwargs.items()))
return (new_args, new_kwargs, pvals)
diff --git a/sdks/python/apache_beam/internal/util_test.py b/sdks/python/apache_beam/internal/util_test.py
index aeba8dc..c3ae191 100644
--- a/sdks/python/apache_beam/internal/util_test.py
+++ b/sdks/python/apache_beam/internal/util_test.py
@@ -30,32 +30,32 @@
def test_remove_objects_from_args(self):
args, kwargs, objs = remove_objects_from_args(
[1, 'a'], {'x': 1, 'y': 3.14}, (str, float))
- self.assertEquals([1, ArgumentPlaceholder()], args)
- self.assertEquals({'x': 1, 'y': ArgumentPlaceholder()}, kwargs)
- self.assertEquals(['a', 3.14], objs)
+ self.assertEqual([1, ArgumentPlaceholder()], args)
+ self.assertEqual({'x': 1, 'y': ArgumentPlaceholder()}, kwargs)
+ self.assertEqual(['a', 3.14], objs)
def test_remove_objects_from_args_nothing_to_remove(self):
args, kwargs, objs = remove_objects_from_args(
[1, 2], {'x': 1, 'y': 2}, (str, float))
- self.assertEquals([1, 2], args)
- self.assertEquals({'x': 1, 'y': 2}, kwargs)
- self.assertEquals([], objs)
+ self.assertEqual([1, 2], args)
+ self.assertEqual({'x': 1, 'y': 2}, kwargs)
+ self.assertEqual([], objs)
def test_insert_values_in_args(self):
values = ['a', 'b']
args = [1, ArgumentPlaceholder()]
kwargs = {'x': 1, 'y': ArgumentPlaceholder()}
args, kwargs = insert_values_in_args(args, kwargs, values)
- self.assertEquals([1, 'a'], args)
- self.assertEquals({'x': 1, 'y': 'b'}, kwargs)
+ self.assertEqual([1, 'a'], args)
+ self.assertEqual({'x': 1, 'y': 'b'}, kwargs)
def test_insert_values_in_args_nothing_to_insert(self):
values = []
args = [1, 'a']
kwargs = {'x': 1, 'y': 'b'}
args, kwargs = insert_values_in_args(args, kwargs, values)
- self.assertEquals([1, 'a'], args)
- self.assertEquals({'x': 1, 'y': 'b'}, kwargs)
+ self.assertEqual([1, 'a'], args)
+ self.assertEqual({'x': 1, 'y': 'b'}, kwargs)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/io/__init__.py b/sdks/python/apache_beam/io/__init__.py
index 6ce2645..f730bcb 100644
--- a/sdks/python/apache_beam/io/__init__.py
+++ b/sdks/python/apache_beam/io/__init__.py
@@ -14,7 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
"""A package defining several input sources and output sinks."""
# pylint: disable=wildcard-import
@@ -26,6 +25,7 @@
from apache_beam.io.iobase import Sink
from apache_beam.io.iobase import Write
from apache_beam.io.iobase import Writer
+from apache_beam.io.mongodbio import *
from apache_beam.io.parquetio import *
from apache_beam.io.textio import *
from apache_beam.io.tfrecordio import *
diff --git a/sdks/python/apache_beam/io/avroio.py b/sdks/python/apache_beam/io/avroio.py
index b2599e1..b1cf6bd 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -629,3 +629,4 @@
def close(self, writer):
writer.flush()
+ writer.fo.close()
diff --git a/sdks/python/apache_beam/io/avroio_test.py b/sdks/python/apache_beam/io/avroio_test.py
index 06c098b..049159e 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -277,13 +277,9 @@
def test_split_points(self):
num_records = 12000
- if self.use_fastavro:
- sync_interval = 16000
- file_name = self._write_data(count=num_records,
- sync_interval=sync_interval)
- else:
- sync_interval = avro.datafile.SYNC_INTERVAL
- file_name = self._write_data(count=num_records)
+ sync_interval = 16000
+ file_name = self._write_data(count=num_records,
+ sync_interval=sync_interval)
source = _create_avro_source(file_name, use_fastavro=self.use_fastavro)
@@ -348,8 +344,6 @@
self._run_avro_test(pattern, 100, True, expected_result)
def test_dynamic_work_rebalancing_exhaustive(self):
- # Adjusting block size so that we can perform a exhaustive dynamic
- # work rebalancing test that completes within an acceptable amount of time.
def compare_split_points(file_name):
source = _create_avro_source(file_name,
use_fastavro=self.use_fastavro)
@@ -358,17 +352,11 @@
assert len(splits) == 1
source_test_utils.assert_split_at_fraction_exhaustive(splits[0].source)
- if self.use_fastavro:
- file_name = self._write_data(count=200, sync_interval=2)
- compare_split_points(file_name)
- else:
- old_sync_interval = avro.datafile.SYNC_INTERVAL
- try:
- avro.datafile.SYNC_INTERVAL = 2
- file_name = self._write_data(count=5)
- compare_split_points(file_name)
- finally:
- avro.datafile.SYNC_INTERVAL = old_sync_interval
+ # Adjusting block size so that we can perform a exhaustive dynamic
+ # work rebalancing test that completes within an acceptable amount of time.
+ file_name = self._write_data(count=5, sync_interval=2)
+
+ compare_split_points(file_name)
def test_corrupted_file(self):
file_name = self._write_data()
@@ -377,9 +365,9 @@
# Corrupt the last character of the file which is also the last character of
# the last sync_marker.
- last_char_index = len(data) - 1
- corrupted_data = data[:last_char_index]
- corrupted_data += b'A' if data[last_char_index] == b'B' else b'B'
+ # https://avro.apache.org/docs/current/spec.html#Object+Container+Files
+ corrupted_data = bytearray(data)
+ corrupted_data[-1] = (corrupted_data[-1] + 1) % 256
with tempfile.NamedTemporaryFile(
delete=False, prefix=tempfile.template) as f:
f.write(corrupted_data)
@@ -387,9 +375,8 @@
source = _create_avro_source(
corrupted_file_name, use_fastavro=self.use_fastavro)
- with self.assertRaises(ValueError) as exn:
+ with self.assertRaisesRegexp(ValueError, r'expected sync marker'):
source_test_utils.read_from_source(source, None, None)
- self.assertEqual(0, exn.exception.message.find('Unexpected sync marker'))
def test_read_from_avro(self):
path = self._write_data()
@@ -490,17 +477,23 @@
directory=None,
prefix=tempfile.template,
codec='null',
- count=len(RECORDS)):
- with tempfile.NamedTemporaryFile(delete=False,
- dir=directory,
- prefix=prefix) as f:
- writer = DataFileWriter(f, DatumWriter(), self.SCHEMA, codec=codec)
- len_records = len(self.RECORDS)
- for i in range(count):
- writer.append(self.RECORDS[i % len_records])
- writer.close()
- self._temp_files.append(f.name)
- return f.name
+ count=len(RECORDS),
+ sync_interval=avro.datafile.SYNC_INTERVAL):
+ old_sync_interval = avro.datafile.SYNC_INTERVAL
+ try:
+ avro.datafile.SYNC_INTERVAL = sync_interval
+ with tempfile.NamedTemporaryFile(delete=False,
+ dir=directory,
+ prefix=prefix) as f:
+ writer = DataFileWriter(f, DatumWriter(), self.SCHEMA, codec=codec)
+ len_records = len(self.RECORDS)
+ for i in range(count):
+ writer.append(self.RECORDS[i % len_records])
+ writer.close()
+ self._temp_files.append(f.name)
+ return f.name
+ finally:
+ avro.datafile.SYNC_INTERVAL = old_sync_interval
class TestFastAvro(AvroBase, unittest.TestCase):
diff --git a/sdks/python/apache_beam/io/concat_source_test.py b/sdks/python/apache_beam/io/concat_source_test.py
index 06fb6cd..8a5b0fa 100644
--- a/sdks/python/apache_beam/io/concat_source_test.py
+++ b/sdks/python/apache_beam/io/concat_source_test.py
@@ -32,6 +32,8 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+__all__ = ['RangeSource']
+
class RangeSource(iobase.BoundedSource):
@@ -61,7 +63,7 @@
yield iobase.SourceBundle(
sub_end - sub_start,
RangeSource(sub_start, sub_end, self._split_freq),
- None, None)
+ sub_start, sub_end)
def get_range_tracker(self, start_position, end_position):
start, end = self._normalize(start_position, end_position)
@@ -145,64 +147,64 @@
for range in ranges])
range_tracker = source.get_range_tracker()
- self.assertEquals(range_tracker.position_at_fraction(0), (0, 0))
- self.assertEquals(range_tracker.position_at_fraction(.01), (0, 1))
- self.assertEquals(range_tracker.position_at_fraction(.1), (0, 4))
- self.assertEquals(range_tracker.position_at_fraction(.125), (1, 4))
- self.assertEquals(range_tracker.position_at_fraction(.2), (1, 7))
- self.assertEquals(range_tracker.position_at_fraction(.7), (2, 23))
- self.assertEquals(range_tracker.position_at_fraction(.75), (3, 24))
- self.assertEquals(range_tracker.position_at_fraction(.8), (3, 26))
- self.assertEquals(range_tracker.position_at_fraction(1), (4, None))
+ self.assertEqual(range_tracker.position_at_fraction(0), (0, 0))
+ self.assertEqual(range_tracker.position_at_fraction(.01), (0, 1))
+ self.assertEqual(range_tracker.position_at_fraction(.1), (0, 4))
+ self.assertEqual(range_tracker.position_at_fraction(.125), (1, 4))
+ self.assertEqual(range_tracker.position_at_fraction(.2), (1, 7))
+ self.assertEqual(range_tracker.position_at_fraction(.7), (2, 23))
+ self.assertEqual(range_tracker.position_at_fraction(.75), (3, 24))
+ self.assertEqual(range_tracker.position_at_fraction(.8), (3, 26))
+ self.assertEqual(range_tracker.position_at_fraction(1), (4, None))
range_tracker = source.get_range_tracker((1, None), (3, None))
- self.assertEquals(range_tracker.position_at_fraction(0), (1, 4))
- self.assertEquals(range_tracker.position_at_fraction(.01), (1, 5))
- self.assertEquals(range_tracker.position_at_fraction(.5), (1, 14))
- self.assertEquals(range_tracker.position_at_fraction(.599), (1, 16))
- self.assertEquals(range_tracker.position_at_fraction(.601), (2, 17))
- self.assertEquals(range_tracker.position_at_fraction(1), (3, None))
+ self.assertEqual(range_tracker.position_at_fraction(0), (1, 4))
+ self.assertEqual(range_tracker.position_at_fraction(.01), (1, 5))
+ self.assertEqual(range_tracker.position_at_fraction(.5), (1, 14))
+ self.assertEqual(range_tracker.position_at_fraction(.599), (1, 16))
+ self.assertEqual(range_tracker.position_at_fraction(.601), (2, 17))
+ self.assertEqual(range_tracker.position_at_fraction(1), (3, None))
def test_empty_source(self):
read_all = source_test_utils.read_from_source
empty = RangeSource(0, 0)
- self.assertEquals(read_all(ConcatSource([])), [])
- self.assertEquals(read_all(ConcatSource([empty])), [])
- self.assertEquals(read_all(ConcatSource([empty, empty])), [])
+ self.assertEqual(read_all(ConcatSource([])), [])
+ self.assertEqual(read_all(ConcatSource([empty])), [])
+ self.assertEqual(read_all(ConcatSource([empty, empty])), [])
range10 = RangeSource(0, 10)
- self.assertEquals(read_all(ConcatSource([range10]), (0, None), (0, 0)),
- [])
- self.assertEquals(read_all(ConcatSource([range10]), (0, 10), (1, None)),
- [])
- self.assertEquals(read_all(ConcatSource([range10, range10]),
- (0, 10), (1, 0)),
- [])
+ self.assertEqual(read_all(ConcatSource([range10]), (0, None), (0, 0)),
+ [])
+ self.assertEqual(read_all(ConcatSource([range10]), (0, 10), (1, None)),
+ [])
+ self.assertEqual(read_all(ConcatSource([range10, range10]),
+ (0, 10), (1, 0)),
+ [])
def test_single_source(self):
read_all = source_test_utils.read_from_source
range10 = RangeSource(0, 10)
- self.assertEquals(read_all(ConcatSource([range10])), list(range(10)))
- self.assertEquals(read_all(ConcatSource([range10]), (0, 5)),
- list(range(5, 10)))
- self.assertEquals(read_all(ConcatSource([range10]), None, (0, 5)),
- list(range(5)))
+ self.assertEqual(read_all(ConcatSource([range10])), list(range(10)))
+ self.assertEqual(read_all(ConcatSource([range10]), (0, 5)),
+ list(range(5, 10)))
+ self.assertEqual(read_all(ConcatSource([range10]), None, (0, 5)),
+ list(range(5)))
def test_source_with_empty_ranges(self):
read_all = source_test_utils.read_from_source
empty = RangeSource(0, 0)
- self.assertEquals(read_all(empty), [])
+ self.assertEqual(read_all(empty), [])
range10 = RangeSource(0, 10)
- self.assertEquals(read_all(ConcatSource([empty, empty, range10])),
- list(range(10)))
- self.assertEquals(read_all(ConcatSource([empty, range10, empty])),
- list(range(10)))
- self.assertEquals(read_all(ConcatSource([range10, empty, range10, empty])),
- list(range(10)) + list(range(10)))
+ self.assertEqual(read_all(ConcatSource([empty, empty, range10])),
+ list(range(10)))
+ self.assertEqual(read_all(ConcatSource([empty, range10, empty])),
+ list(range(10)))
+ self.assertEqual(read_all(ConcatSource([range10, empty, range10, empty])),
+ list(range(10)) + list(range(10)))
def test_source_with_empty_ranges_exhastive(self):
empty = RangeSource(0, 0)
diff --git a/sdks/python/apache_beam/io/external/generate_sequence.py b/sdks/python/apache_beam/io/external/generate_sequence.py
index 8de68ab..a17ec7b 100644
--- a/sdks/python/apache_beam/io/external/generate_sequence.py
+++ b/sdks/python/apache_beam/io/external/generate_sequence.py
@@ -17,15 +17,11 @@
from __future__ import absolute_import
-from apache_beam import ExternalTransform
-from apache_beam import pvalue
-from apache_beam.coders import VarIntCoder
-from apache_beam.portability.api.external_transforms_pb2 import ConfigValue
-from apache_beam.portability.api.external_transforms_pb2 import ExternalConfigurationPayload
-from apache_beam.transforms import ptransform
+from apache_beam.transforms.external import ExternalTransform
+from apache_beam.transforms.external import ImplicitSchemaPayloadBuilder
-class GenerateSequence(ptransform.PTransform):
+class GenerateSequence(ExternalTransform):
"""
An external PTransform which provides a bounded or unbounded stream of
integers.
@@ -46,47 +42,22 @@
Note: Runners need to support translating Read operations in order to use
this source. At the moment only the Flink Runner supports this.
+
+ Experimental; no backwards compatibility guarantees.
"""
+ URN = 'beam:external:java:generate_sequence:v1'
def __init__(self, start, stop=None,
elements_per_period=None, max_read_time=None,
- expansion_service='localhost:8097'):
- super(GenerateSequence, self).__init__()
- self._urn = 'beam:external:java:generate_sequence:v1'
- self.start = start
- self.stop = stop
- self.elements_per_period = elements_per_period
- self.max_read_time = max_read_time
- self.expansion_service = expansion_service
-
- def expand(self, pbegin):
- if not isinstance(pbegin, pvalue.PBegin):
- raise Exception("GenerateSequence must be a root transform")
-
- coder = VarIntCoder()
- coder_urn = ['beam:coder:varint:v1']
- args = {
- 'start':
- ConfigValue(
- coder_urn=coder_urn,
- payload=coder.encode(self.start))
- }
- if self.stop:
- args['stop'] = ConfigValue(
- coder_urn=coder_urn,
- payload=coder.encode(self.stop))
- if self.elements_per_period:
- args['elements_per_period'] = ConfigValue(
- coder_urn=coder_urn,
- payload=coder.encode(self.elements_per_period))
- if self.max_read_time:
- args['max_read_time'] = ConfigValue(
- coder_urn=coder_urn,
- payload=coder.encode(self.max_read_time))
-
- payload = ExternalConfigurationPayload(configuration=args)
- return pbegin.apply(
- ExternalTransform(
- self._urn,
- payload.SerializeToString(),
- self.expansion_service))
+ expansion_service=None):
+ super(GenerateSequence, self).__init__(
+ self.URN,
+ ImplicitSchemaPayloadBuilder(
+ {
+ 'start': start,
+ 'stop': stop,
+ 'elements_per_period': elements_per_period,
+ 'max_read_time': max_read_time,
+ }
+ ),
+ expansion_service)
diff --git a/sdks/python/apache_beam/io/external/generate_sequence_test.py b/sdks/python/apache_beam/io/external/generate_sequence_test.py
new file mode 100644
index 0000000..95d6fbc
--- /dev/null
+++ b/sdks/python/apache_beam/io/external/generate_sequence_test.py
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for cross-language generate sequence."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import os
+import re
+import unittest
+
+from nose.plugins.attrib import attr
+
+from apache_beam.io.external.generate_sequence import GenerateSequence
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@attr('UsesCrossLanguageTransforms')
+@unittest.skipUnless(
+ os.environ.get('EXPANSION_PORT'),
+ "EXPANSION_PORT environment var is not provided.")
+class XlangGenerateSequenceTest(unittest.TestCase):
+ def test_generate_sequence(self):
+ test_pipeline = TestPipeline()
+ port = os.environ.get('EXPANSION_PORT')
+ address = 'localhost:%s' % port
+
+ try:
+ with test_pipeline as p:
+ res = (
+ p
+ | GenerateSequence(start=1, stop=10,
+ expansion_service=address)
+ )
+
+ assert_that(res, equal_to([i for i in range(1, 10)]))
+ except RuntimeError as e:
+ if re.search(GenerateSequence.URN, str(e)):
+ print("looks like URN not implemented in expansion service, skipping.")
+ else:
+ raise e
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/external/kafka.py b/sdks/python/apache_beam/io/external/kafka.py
index b61cf2c..f824515 100644
--- a/sdks/python/apache_beam/io/external/kafka.py
+++ b/sdks/python/apache_beam/io/external/kafka.py
@@ -37,18 +37,25 @@
from __future__ import absolute_import
-from apache_beam import ExternalTransform
-from apache_beam import pvalue
-from apache_beam.coders import BytesCoder
-from apache_beam.coders import IterableCoder
-from apache_beam.coders import TupleCoder
-from apache_beam.coders.coders import LengthPrefixCoder
-from apache_beam.portability.api.external_transforms_pb2 import ConfigValue
-from apache_beam.portability.api.external_transforms_pb2 import ExternalConfigurationPayload
-from apache_beam.transforms import ptransform
+import typing
+
+from past.builtins import unicode
+
+from apache_beam.transforms.external import ExternalTransform
+from apache_beam.transforms.external import NamedTupleBasedPayloadBuilder
+
+ReadFromKafkaSchema = typing.NamedTuple(
+ 'ReadFromKafkaSchema',
+ [
+ ('consumer_config', typing.List[typing.Tuple[unicode, unicode]]),
+ ('topics', typing.List[unicode]),
+ ('key_deserializer', unicode),
+ ('value_deserializer', unicode),
+ ]
+)
-class ReadFromKafka(ptransform.PTransform):
+class ReadFromKafka(ExternalTransform):
"""
An external PTransform which reads from Kafka and returns a KV pair for
each item in the specified Kafka topics. If no Kafka Deserializer for
@@ -56,17 +63,21 @@
Note: Runners need to support translating Read operations in order to use
this source. At the moment only the Flink Runner supports this.
+
+ Experimental; no backwards compatibility guarantees.
"""
# Returns the key/value data as raw byte arrays
byte_array_deserializer = 'org.apache.kafka.common.serialization.' \
'ByteArrayDeserializer'
+ URN = 'beam:external:java:kafka:read:v1'
+
def __init__(self, consumer_config,
topics,
key_deserializer=byte_array_deserializer,
value_deserializer=byte_array_deserializer,
- expansion_service='localhost:8097'):
+ expansion_service=None):
"""
Initializes a read operation from Kafka.
@@ -86,53 +97,51 @@
serialization.ByteArrayDeserializer'.
:param expansion_service: The address (host:port) of the ExpansionService.
"""
- super(ReadFromKafka, self).__init__()
- self._urn = 'beam:external:java:kafka:read:v1'
- self.consumer_config = consumer_config
- self.topics = topics
- self.key_deserializer = key_deserializer
- self.value_deserializer = value_deserializer
- self.expansion_service = expansion_service
-
- def expand(self, pbegin):
- if not isinstance(pbegin, pvalue.PBegin):
- raise Exception("ReadFromKafka must be a root transform")
-
- args = {
- 'consumer_config':
- _encode_map(self.consumer_config),
- 'topics':
- _encode_list(self.topics),
- 'key_deserializer':
- _encode_str(self.key_deserializer),
- 'value_deserializer':
- _encode_str(self.value_deserializer),
- }
-
- payload = ExternalConfigurationPayload(configuration=args)
- return pbegin.apply(
- ExternalTransform(
- self._urn,
- payload.SerializeToString(),
- self.expansion_service))
+ super(ReadFromKafka, self).__init__(
+ self.URN,
+ NamedTupleBasedPayloadBuilder(
+ ReadFromKafkaSchema(
+ consumer_config=list(consumer_config.items()),
+ topics=topics,
+ key_deserializer=key_deserializer,
+ value_deserializer=value_deserializer,
+ )
+ ),
+ expansion_service
+ )
-class WriteToKafka(ptransform.PTransform):
+WriteToKafkaSchema = typing.NamedTuple(
+ 'WriteToKafkaSchema',
+ [
+ ('producer_config', typing.List[typing.Tuple[unicode, unicode]]),
+ ('topic', unicode),
+ ('key_serializer', unicode),
+ ('value_serializer', unicode),
+ ]
+)
+
+
+class WriteToKafka(ExternalTransform):
"""
An external PTransform which writes KV data to a specified Kafka topic.
If no Kafka Serializer for key/value is provided, then key/value are
assumed to be byte arrays.
+
+ Experimental; no backwards compatibility guarantees.
"""
# Default serializer which passes raw bytes to Kafka
byte_array_serializer = 'org.apache.kafka.common.serialization.' \
'ByteArraySerializer'
+ URN = 'beam:external:java:kafka:write:v1'
+
def __init__(self, producer_config,
topic,
key_serializer=byte_array_serializer,
value_serializer=byte_array_serializer,
- expansion_service='localhost:8097'):
+ expansion_service=None):
"""
Initializes a write operation to Kafka.
@@ -152,62 +161,15 @@
serialization.ByteArraySerializer'.
:param expansion_service: The address (host:port) of the ExpansionService.
"""
- super(WriteToKafka, self).__init__()
- self._urn = 'beam:external:java:kafka:write:v1'
- self.producer_config = producer_config
- self.topic = topic
- self.key_serializer = key_serializer
- self.value_serializer = value_serializer
- self.expansion_service = expansion_service
-
- def expand(self, pvalue):
- args = {
- 'producer_config':
- _encode_map(self.producer_config),
- 'topic':
- _encode_str(self.topic),
- 'key_serializer':
- _encode_str(self.key_serializer),
- 'value_serializer':
- _encode_str(self.value_serializer),
- }
-
- payload = ExternalConfigurationPayload(configuration=args)
- return pvalue.apply(
- ExternalTransform(
- self._urn,
- payload.SerializeToString(),
- self.expansion_service))
-
-
-def _encode_map(dict_obj):
- kv_list = [(key.encode('utf-8'), val.encode('utf-8'))
- for key, val in dict_obj.items()]
- coder = IterableCoder(TupleCoder(
- [LengthPrefixCoder(BytesCoder()), LengthPrefixCoder(BytesCoder())]))
- coder_urns = ['beam:coder:iterable:v1',
- 'beam:coder:kv:v1',
- 'beam:coder:bytes:v1',
- 'beam:coder:bytes:v1']
- return ConfigValue(
- coder_urn=coder_urns,
- payload=coder.encode(kv_list))
-
-
-def _encode_list(list_obj):
- encoded_list = [val.encode('utf-8') for val in list_obj]
- coder = IterableCoder(LengthPrefixCoder(BytesCoder()))
- coder_urns = ['beam:coder:iterable:v1',
- 'beam:coder:bytes:v1']
- return ConfigValue(
- coder_urn=coder_urns,
- payload=coder.encode(encoded_list))
-
-
-def _encode_str(str_obj):
- encoded_str = str_obj.encode('utf-8')
- coder = LengthPrefixCoder(BytesCoder())
- coder_urns = ['beam:coder:bytes:v1']
- return ConfigValue(
- coder_urn=coder_urns,
- payload=coder.encode(encoded_str))
+ super(WriteToKafka, self).__init__(
+ self.URN,
+ NamedTupleBasedPayloadBuilder(
+ WriteToKafkaSchema(
+ producer_config=list(producer_config.items()),
+ topic=topic,
+ key_serializer=key_serializer,
+ value_serializer=value_serializer,
+ )
+ ),
+ expansion_service
+ )
diff --git a/sdks/python/apache_beam/io/external/xlang_parquetio_test.py b/sdks/python/apache_beam/io/external/xlang_parquetio_test.py
new file mode 100644
index 0000000..0c948ca
--- /dev/null
+++ b/sdks/python/apache_beam/io/external/xlang_parquetio_test.py
@@ -0,0 +1,88 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for cross-language parquet io read/write."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import os
+import re
+import unittest
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.coders.avro_record import AvroRecord
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+
+PARQUET_WRITE_URN = "beam:transforms:xlang:parquet_write"
+
+
+@attr('UsesCrossLanguageTransforms')
+@unittest.skipUnless(
+ os.environ.get('EXPANSION_JAR'),
+ "EXPANSION_JAR environment variable is not set.")
+@unittest.skipUnless(
+ os.environ.get('EXPANSION_PORT'),
+ "EXPANSION_PORT environment var is not provided.")
+class XlangParquetIOTest(unittest.TestCase):
+ # TODO: add verification for the file written by external transform
+ # after fixing BEAM-7612
+ def test_write(self):
+ expansion_jar = os.environ.get('EXPANSION_JAR')
+ port = os.environ.get('EXPANSION_PORT')
+ address = 'localhost:%s' % port
+ try:
+ test_pipeline = TestPipeline()
+ test_pipeline.get_pipeline_options().view_as(
+ DebugOptions).experiments.append('jar_packages='+expansion_jar)
+ test_pipeline.not_use_test_runner_api = True
+ with test_pipeline as p:
+ _ = p \
+ | beam.Create([
+ AvroRecord({"name": "abc"}), AvroRecord({"name": "def"}),
+ AvroRecord({"name": "ghi"})]) \
+ | beam.ExternalTransform(
+ PARQUET_WRITE_URN, b'/tmp/test.parquet', address)
+ except RuntimeError as e:
+ if re.search(PARQUET_WRITE_URN, str(e)):
+ print("looks like URN not implemented in expansion service, skipping.")
+ else:
+ raise e
+
+
+class AvroTestCoder(coders.AvroCoder):
+ SCHEMA = """
+ {
+ "type": "record", "name": "testrecord",
+ "fields": [ {"name": "name", "type": "string"} ]
+ }
+ """
+
+ def __init__(self):
+ super(AvroTestCoder, self).__init__(self.SCHEMA)
+
+
+coders.registry.register_coder(AvroRecord, AvroTestCoder)
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/filebasedsink.py b/sdks/python/apache_beam/io/filebasedsink.py
index a7eb424..3e91470 100644
--- a/sdks/python/apache_beam/io/filebasedsink.py
+++ b/sdks/python/apache_beam/io/filebasedsink.py
@@ -182,7 +182,8 @@
file_name_suffix = self.file_name_suffix.get()
suffix = (
'.' + os.path.basename(file_path_prefix) + file_name_suffix)
- return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
+ writer_path = FileSystems.join(init_result, uid) + suffix
+ return FileBasedSinkWriter(self, writer_path)
@check_accessible(['file_path_prefix', 'file_name_suffix'])
def _get_final_name(self, shard_num, num_shards):
diff --git a/sdks/python/apache_beam/io/filebasedsink_test.py b/sdks/python/apache_beam/io/filebasedsink_test.py
index a934381..666bc9c 100644
--- a/sdks/python/apache_beam/io/filebasedsink_test.py
+++ b/sdks/python/apache_beam/io/filebasedsink_test.py
@@ -231,6 +231,14 @@
with self.assertRaises(ValueError):
_get_temp_dir(dir_root_path)
+ def test_temp_dir_uniqueness(self):
+ temp_path = os.path.join(self._new_tempdir(), 'unique')
+ sink = MyFileBasedSink(temp_path, coder=coders.ToStringCoder())
+ init_list = [''] * 1000
+ temp_dir_list = [sink._create_temp_dir(temp_path) for _ in init_list]
+ temp_dir_set = set(temp_dir_list)
+ self.assertEqual(len(temp_dir_list), len(temp_dir_set))
+
def test_temp_dir_gcs(self):
try:
self.run_temp_dir_check(
diff --git a/sdks/python/apache_beam/io/filebasedsource_test.py b/sdks/python/apache_beam/io/filebasedsource_test.py
index 2e9af06..5f969d8 100644
--- a/sdks/python/apache_beam/io/filebasedsource_test.py
+++ b/sdks/python/apache_beam/io/filebasedsource_test.py
@@ -67,8 +67,10 @@
start += len(line)
current = start
line = f.readline()
- while line:
- if not range_tracker.try_claim(current):
+ while range_tracker.try_claim(current):
+ # When the source is unsplittable, try_claim is not enough to determine
+ # whether the file has reached to the end.
+ if not line:
return
yield line.rstrip(b'\n')
current += len(line)
@@ -233,7 +235,7 @@
for start in [0, 10, 20]]
concat = ConcatSource(sources)
splits = [split for split in concat.split()]
- self.assertEquals(6, len(splits))
+ self.assertEqual(6, len(splits))
# Reading all splits
read_data = []
@@ -249,7 +251,7 @@
sources = [TestConcatSource.DummySource(range(start, start + 10)) for start
in [0, 10, 20]]
concat = ConcatSource(sources)
- self.assertEquals(30, concat.estimate_size())
+ self.assertEqual(30, concat.estimate_size())
class TestFileBasedSource(unittest.TestCase):
@@ -344,18 +346,18 @@
file_name, expected_data = write_data(10)
assert len(expected_data) == 10
fbs = LineSource(file_name)
- self.assertEquals(10 * 6, fbs.estimate_size())
+ self.assertEqual(10 * 6, fbs.estimate_size())
def test_estimate_size_of_pattern(self):
pattern, expected_data = write_pattern([5, 3, 10, 8, 8, 4])
assert len(expected_data) == 38
fbs = LineSource(pattern)
- self.assertEquals(38 * 6, fbs.estimate_size())
+ self.assertEqual(38 * 6, fbs.estimate_size())
pattern, expected_data = write_pattern([5, 3, 9])
assert len(expected_data) == 17
fbs = LineSource(pattern)
- self.assertEquals(17 * 6, fbs.estimate_size())
+ self.assertEqual(17 * 6, fbs.estimate_size())
def test_estimate_size_with_sampling_same_size(self):
num_files = 2 * FileBasedSource.MIN_NUMBER_OF_FILES_TO_STAT
@@ -451,7 +453,7 @@
assert len(expected_data) == 20
fbs = LineSource(pattern, splittable=False)
splits = [split for split in fbs.split(desired_bundle_size=15)]
- self.assertEquals(3, len(splits))
+ self.assertEqual(3, len(splits))
def test_source_file_unsplittable(self):
file_name, expected_data = write_data(100)
@@ -681,11 +683,11 @@
# Should simply return stop_offset - start_offset
source = SingleFileSource(
fbs, file_name='dummy_file', start_offset=0, stop_offset=100)
- self.assertEquals(100, source.estimate_size())
+ self.assertEqual(100, source.estimate_size())
source = SingleFileSource(fbs, file_name='dummy_file', start_offset=10,
stop_offset=100)
- self.assertEquals(90, source.estimate_size())
+ self.assertEqual(90, source.estimate_size())
def test_read_range_at_beginning(self):
fbs = LineSource('dummy_pattern', validate=False)
@@ -727,10 +729,10 @@
assert len(expected_data) == 10
source = SingleFileSource(fbs, file_name, 0, 10 * 6)
splits = [split for split in source.split(desired_bundle_size=100)]
- self.assertEquals(1, len(splits))
- self.assertEquals(60, splits[0].weight)
- self.assertEquals(0, splits[0].start_position)
- self.assertEquals(60, splits[0].stop_position)
+ self.assertEqual(1, len(splits))
+ self.assertEqual(60, splits[0].weight)
+ self.assertEqual(0, splits[0].start_position)
+ self.assertEqual(60, splits[0].stop_position)
range_tracker = splits[0].source.get_range_tracker(None, None)
read_data = [value for value in splits[0].source.read(range_tracker)]
@@ -743,7 +745,7 @@
assert len(expected_data) == 10
source = SingleFileSource(fbs, file_name, 0, 10 * 6)
splits = [split for split in source.split(desired_bundle_size=25)]
- self.assertEquals(3, len(splits))
+ self.assertEqual(3, len(splits))
read_data = []
for split in splits:
@@ -763,7 +765,7 @@
splits = [split for split in
source.split(desired_bundle_size=15, start_offset=10,
stop_offset=50)]
- self.assertEquals(3, len(splits))
+ self.assertEqual(3, len(splits))
read_data = []
for split in splits:
diff --git a/sdks/python/apache_beam/io/fileio.py b/sdks/python/apache_beam/io/fileio.py
index 10890ca..a1a7a58 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -23,17 +23,88 @@
metadata records, and produces a ``PCollection`` of ``ReadableFile`` objects.
These transforms currently do not support splitting by themselves.
+Writing to Files
+================
+
+The transforms in this file include ``WriteToFiles``, which allows you to write
+a ``beam.PCollection`` to files, and gives you many options to customize how to
+do this.
+
+The ``WriteToFiles`` transform supports bounded and unbounded PCollections
+(i.e. it can be used both batch and streaming pipelines). For streaming
+pipelines, it currently does not have support for multiple trigger firings
+on the same window.
+
+File Naming
+-----------
+One of the parameters received by ``WriteToFiles`` is a function specifying how
+to name the files that are written. This is a function that takes in the
+following parameters:
+
+- window
+- pane
+- shard_index
+- total_shards
+- compression
+- destination
+
+It should return a file name that is unique for a combination of these
+parameters.
+
+The default naming strategy is to name files
+in the format
+`$prefix-$start-$end-$pane-$shard-of-$numShards$suffix$compressionSuffix`,
+where:
+
+- `$prefix` is, by default, `"output"`.
+- `$start` and `$end` are the boundaries of the window for the data being
+ written. These are omitted if we're using the Global window.
+- `$pane` is the index for the number of firing for a window.
+- `$shard` and `$numShards` are the current shard number, and the total number
+ of shards for this window firing.
+- `$suffix` is, by default, an empty string, but it can be set by the user via
+ ``default_file_naming``.
+
+Dynamic Destinations
+--------------------
+If the elements in the input ``beam.PCollection`` can be partitioned into groups
+that should be treated differently (e.g. some events are to be stored as CSV,
+while some others are to be stored as Avro files), it is possible to do this
+by passing a `destination` parameter to ``WriteToFiles``. Something like the
+following::
+
+ my_pcollection | beam.io.fileio.WriteToFiles(
+ path='/my/file/path',
+ destination=lambda record: 'avro' if record['type'] == 'A' else 'csv',
+ sink=lambda dest: AvroSink() if dest == 'avro' else CsvSink(),
+ file_naming=beam.io.fileio.destination_prefix_naming())
+
+In this transform, depending on the type of a record, it will be written down to
+a destination named `'avro'`, or `'csv'`. The value returned by the
+`destination` call is then passed to the `sink` call, to determine what sort of
+sink will be used for each destination. The return type of the `destination`
+parameter can be anything, as long as elements can be grouped by it.
+
No backward compatibility guarantees. Everything in this module is experimental.
"""
from __future__ import absolute_import
+import collections
+import logging
+import random
+import uuid
+
from past.builtins import unicode
import apache_beam as beam
from apache_beam.io import filesystem
from apache_beam.io import filesystems
from apache_beam.io.filesystem import BeamIOError
+from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.options.value_provider import StaticValueProvider
+from apache_beam.options.value_provider import ValueProvider
+from apache_beam.transforms.window import GlobalWindow
from apache_beam.utils.annotations import experimental
__all__ = ['EmptyMatchTreatment',
@@ -148,10 +219,11 @@
self.metadata = metadata
def open(self, mime_type='text/plain'):
- return filesystems.FileSystems.open(self.metadata.path)
+ return filesystems.FileSystems.open(self.metadata.path,
+ mime_type=mime_type)
def read(self):
- return self.open().read()
+ return self.open('application/octet-stream').read()
def read_utf8(self):
return self.open().read().decode('utf-8')
@@ -170,3 +242,485 @@
def expand(self, pcoll):
return pcoll | beam.ParDo(_ReadMatchesFn(self._compression,
self._skip_directories))
+
+
+class FileSink(object):
+ """Specifies how to write elements to individual files in ``WriteToFiles``.
+
+ **NOTE: THIS CLASS IS EXPERIMENTAL.**
+
+ A Sink class must implement the following:
+
+ - The ``open`` method, which initializes writing to a file handler (it is not
+ responsible for opening the file handler itself).
+ - The ``write`` method, which writes an element to the file that was passed
+ in ``open``.
+ - The ``flush`` method, which flushes any buffered state. This is most often
+ called before closing a file (but not exclusively called in that
+ situation). The sink is not responsible for closing the file handler.
+ """
+
+ def open(self, fh):
+ raise NotImplementedError
+
+ def write(self, record):
+ raise NotImplementedError
+
+ def flush(self):
+ raise NotImplementedError
+
+
+@beam.typehints.with_input_types(str)
+class TextSink(FileSink):
+ """A sink that encodes utf8 elements, and writes to file handlers.
+
+ **NOTE: THIS CLASS IS EXPERIMENTAL.**
+
+ This sink simply calls file_handler.write(record.encode('utf8') + '\n') on all
+ records that come into it.
+ """
+
+ def open(self, fh):
+ self._fh = fh
+
+ def write(self, record):
+ self._fh.write(record.encode('utf8'))
+ self._fh.write(b'\n')
+
+ def flush(self):
+ self._fh.flush()
+
+
+def prefix_naming(prefix):
+ return default_file_naming(prefix)
+
+
+_DEFAULT_FILE_NAME_TEMPLATE = (
+ '{prefix}-{start}-{end}-{pane}-'
+ '{shard:05d}-{total_shards:05d}'
+ '{suffix}{compression}')
+
+
+def destination_prefix_naming():
+
+ def _inner(window, pane, shard_index, total_shards, compression, destination):
+ kwargs = {'prefix': str(destination),
+ 'start': '',
+ 'end': '',
+ 'pane': '',
+ 'shard': 0,
+ 'total_shards': 0,
+ 'suffix': '',
+ 'compression': ''}
+ if total_shards is not None and shard_index is not None:
+ kwargs['shard'] = int(shard_index)
+ kwargs['total_shards'] = int(total_shards)
+
+ if window != GlobalWindow():
+ kwargs['start'] = window.start.to_utc_datetime().isoformat()
+ kwargs['end'] = window.end.to_utc_datetime().isoformat()
+
+ # TODO(BEAM-3759): Add support for PaneInfo
+ # If the PANE is the ONLY firing in the window, we don't add it.
+ #if pane and not (pane.is_first and pane.is_last):
+ # kwargs['pane'] = pane.index
+
+ if compression:
+ kwargs['compression'] = '.%s' % compression
+
+ return _DEFAULT_FILE_NAME_TEMPLATE.format(**kwargs)
+
+ return _inner
+
+
+def default_file_naming(prefix, suffix=None):
+
+ def _inner(window, pane, shard_index, total_shards, compression, destination):
+ kwargs = {'prefix': prefix,
+ 'start': '',
+ 'end': '',
+ 'pane': '',
+ 'shard': 0,
+ 'total_shards': 0,
+ 'suffix': '',
+ 'compression': ''}
+ if total_shards is not None and shard_index is not None:
+ kwargs['shard'] = int(shard_index)
+ kwargs['total_shards'] = int(total_shards)
+
+ if window != GlobalWindow():
+ kwargs['start'] = window.start.to_utc_datetime().isoformat()
+ kwargs['end'] = window.end.to_utc_datetime().isoformat()
+
+ # TODO(pabloem): Add support for PaneInfo
+ # If the PANE is the ONLY firing in the window, we don't add it.
+ #if pane and not (pane.is_first and pane.is_last):
+ # kwargs['pane'] = pane.index
+
+ if compression:
+ kwargs['compression'] = '.%s' % compression
+ if suffix:
+ kwargs['suffix'] = suffix
+
+ return _DEFAULT_FILE_NAME_TEMPLATE.format(**kwargs)
+
+ return _inner
+
+
+_FileResult = collections.namedtuple('FileResult',
+ ['file_name',
+ 'shard_index',
+ 'total_shards',
+ 'window',
+ 'pane',
+ 'destination'])
+
+
+# Adding a class to contain PyDoc.
+class FileResult(_FileResult):
+ """A descriptor of a file that has been written."""
+ pass
+
+
+@experimental()
+class WriteToFiles(beam.PTransform):
+ """Write the incoming PCollection to a set of output files.
+
+ The incoming ``PCollection`` may be bounded or unbounded.
+
+ **Note:** For unbounded ``PCollection``s, this transform does not support
+ multiple firings per Window (due to the fact that files are named only by
+ their destination, and window, at the moment).
+ """
+
+ # We allow up to 20 different destinations to be written in a single bundle.
+ # Too many files will add memory pressure to the worker, so we let it be 20.
+ MAX_NUM_WRITERS_PER_BUNDLE = 20
+
+ DEFAULT_SHARDING = 5
+
+ def __init__(self,
+ path,
+ file_naming=None,
+ destination=None,
+ temp_directory=None,
+ sink=None,
+ shards=None,
+ output_fn=None,
+ max_writers_per_bundle=MAX_NUM_WRITERS_PER_BUNDLE):
+ """Initializes a WriteToFiles transform.
+
+ Args:
+ path (str, ValueProvider): The directory to write files into.
+ file_naming (callable): A callable that takes in a window, pane,
+ shard_index, total_shards and compression; and returns a file name.
+ destination (callable): If this argument is provided, the sink parameter
+ must also be a callable.
+ temp_directory (str, ValueProvider): To ensure atomicity in the transform,
+ the output is written into temporary files, which are written to a
+ directory that is meant to be temporary as well. Once the whole output
+ has been written, the files are moved into their final destination, and
+ given their final names. By default, the temporary directory will be
+ within the temp_location of your pipeline.
+ sink (callable, FileSink): The sink to use to write into a file. It should
+ implement the methods of a ``FileSink``. If none is provided, a
+ ``TextSink`` is used.
+ shards (int): The number of shards per destination and trigger firing.
+ max_writers_per_bundle (int): The number of writers that can be open
+ concurrently in a single worker that's processing one bundle.
+ """
+ self.path = (
+ path if isinstance(path, ValueProvider) else StaticValueProvider(str,
+ path))
+ self.file_naming_fn = file_naming or default_file_naming('output')
+ self.destination_fn = self._get_destination_fn(destination)
+ self._temp_directory = temp_directory
+ self.sink_fn = self._get_sink_fn(sink)
+ self.shards = shards or WriteToFiles.DEFAULT_SHARDING
+ self.output_fn = output_fn or (lambda x: x)
+
+ self._max_num_writers_per_bundle = max_writers_per_bundle
+
+ @staticmethod
+ def _get_sink_fn(input_sink):
+ if isinstance(input_sink, FileSink):
+ return lambda x: input_sink
+ elif callable(input_sink):
+ return input_sink
+ else:
+ return lambda x: TextSink()
+
+ @staticmethod
+ def _get_destination_fn(destination):
+ if isinstance(destination, ValueProvider):
+ return lambda elm: destination.get()
+ elif callable(destination):
+ return destination
+ else:
+ return lambda elm: destination
+
+ def expand(self, pcoll):
+ p = pcoll.pipeline
+
+ if not self._temp_directory:
+ temp_location = (
+ p.options.view_as(GoogleCloudOptions).temp_location
+ or self.path.get())
+ dir_uid = str(uuid.uuid4())
+ self._temp_directory = StaticValueProvider(
+ str,
+ filesystems.FileSystems.join(temp_location,
+ '.temp%s' % dir_uid))
+ logging.info('Added temporary directory %s', self._temp_directory.get())
+
+ output = (pcoll
+ | beam.ParDo(_WriteUnshardedRecordsFn(
+ base_path=self._temp_directory,
+ destination_fn=self.destination_fn,
+ sink_fn=self.sink_fn,
+ max_writers_per_bundle=self._max_num_writers_per_bundle))
+ .with_outputs(_WriteUnshardedRecordsFn.SPILLED_RECORDS,
+ _WriteUnshardedRecordsFn.WRITTEN_FILES))
+
+ written_files_pc = output[_WriteUnshardedRecordsFn.WRITTEN_FILES]
+ spilled_records_pc = output[_WriteUnshardedRecordsFn.SPILLED_RECORDS]
+
+ more_written_files_pc = (
+ spilled_records_pc
+ | beam.ParDo(_AppendShardedDestination(self.destination_fn,
+ self.shards))
+ | "GroupRecordsByDestinationAndShard" >> beam.GroupByKey()
+ | beam.ParDo(_WriteShardedRecordsFn(self._temp_directory,
+ self.sink_fn,
+ self.shards))
+ )
+
+ files_by_destination_pc = (
+ (written_files_pc, more_written_files_pc)
+ | beam.Flatten()
+ | beam.Map(lambda file_result: (file_result.destination, file_result))
+ | "GroupTempFilesByDestination" >> beam.GroupByKey())
+
+ # Now we should take the temporary files, and write them to the final
+ # destination, with their proper names.
+
+ file_results = (files_by_destination_pc
+ | beam.ParDo(
+ _MoveTempFilesIntoFinalDestinationFn(
+ self.path, self.file_naming_fn,
+ self._temp_directory)))
+
+ return file_results
+
+
+def _create_writer(base_path, writer_key):
+ try:
+ filesystems.FileSystems.mkdirs(base_path)
+ except IOError:
+ # Directory already exists.
+ pass
+
+ # The file name has a prefix determined by destination+window, along with
+ # a random string. This allows us to retrieve orphaned files later on.
+ file_name = '%s_%s' % (abs(hash(writer_key)), uuid.uuid4())
+ full_file_name = filesystems.FileSystems.join(base_path, file_name)
+ return full_file_name, filesystems.FileSystems.create(full_file_name)
+
+
+class _MoveTempFilesIntoFinalDestinationFn(beam.DoFn):
+
+ def __init__(self, path, file_naming_fn, temp_dir):
+ self.path = path
+ self.file_naming_fn = file_naming_fn
+ self.temporary_directory = temp_dir
+
+ def process(self,
+ element,
+ w=beam.DoFn.WindowParam):
+ destination = element[0]
+ file_results = list(element[1])
+
+ for i, r in enumerate(file_results):
+ # TODO(pabloem): Handle compression for files.
+ final_file_name = self.file_naming_fn(r.window,
+ r.pane,
+ i,
+ len(file_results),
+ '',
+ destination)
+
+ logging.info('Moving temporary file %s to dir: %s as %s. Res: %s',
+ r.file_name, self.path.get(), final_file_name, r)
+
+ final_full_path = filesystems.FileSystems.join(self.path.get(),
+ final_file_name)
+
+ # TODO(pabloem): Batch rename requests?
+ try:
+ filesystems.FileSystems.rename([r.file_name],
+ [final_full_path])
+ except BeamIOError:
+ # This error is not serious, because it may happen on a retry of the
+ # bundle. We simply log it.
+ logging.debug('File %s failed to be copied. This may be due to a bundle'
+ ' being retried.', r.file_name)
+
+ yield FileResult(final_file_name,
+ i,
+ len(file_results),
+ r.window,
+ r.pane,
+ destination)
+
+ logging.info('Cautiously removing temporary files for'
+ ' destination %s and window %s', destination, w)
+ writer_key = (destination, w)
+ self._remove_temporary_files(writer_key)
+
+ def _remove_temporary_files(self, writer_key):
+ try:
+ prefix = filesystems.FileSystems.join(
+ self.temporary_directory.get(), str(abs(hash(writer_key))))
+ match_result = filesystems.FileSystems.match(['%s*' % prefix])
+ orphaned_files = [m.path for m in match_result[0].metadata_list]
+
+ logging.debug('Deleting orphaned files: %s', orphaned_files)
+ filesystems.FileSystems.delete(orphaned_files)
+ except BeamIOError as e:
+ logging.debug('Exceptions when deleting files: %s', e)
+
+
+class _WriteShardedRecordsFn(beam.DoFn):
+
+ def __init__(self, base_path, sink_fn, shards):
+ self.base_path = base_path
+ self.sink_fn = sink_fn
+ self.shards = shards
+
+ def process(self,
+ element,
+ w=beam.DoFn.WindowParam,
+ pane=beam.DoFn.PaneInfoParam):
+ destination_and_shard = element[0]
+ destination = destination_and_shard[0]
+ shard = destination_and_shard[1]
+ records = element[1]
+
+ full_file_name, writer = _create_writer(base_path=self.base_path.get(),
+ writer_key=(destination, w))
+ sink = self.sink_fn(destination)
+ sink.open(writer)
+
+ for r in records:
+ sink.write(r)
+
+ sink.flush()
+ writer.close()
+
+ logging.info('Writing file %s for destination %s and shard %s',
+ full_file_name, destination, repr(shard))
+
+ yield FileResult(full_file_name,
+ shard_index=shard,
+ total_shards=self.shards,
+ window=w,
+ pane=pane,
+ destination=destination)
+
+
+class _AppendShardedDestination(beam.DoFn):
+
+ def __init__(self, destination, shards):
+ self.destination_fn = destination
+ self.shards = shards
+
+ # We start the shards for a single destination at an arbitrary point.
+ self._shard_counter = collections.defaultdict(
+ lambda: random.randrange(self.shards))
+
+ def _next_shard_for_destination(self, destination):
+ self._shard_counter[destination] = (
+ (self._shard_counter[destination] + 1) % self.shards)
+
+ return self._shard_counter[destination]
+
+ def process(self, record):
+ destination = self.destination_fn(record)
+ shard = self._next_shard_for_destination(destination)
+
+ yield ((destination, shard), record)
+
+
+class _WriteUnshardedRecordsFn(beam.DoFn):
+
+ SPILLED_RECORDS = 'spilled_records'
+ WRITTEN_FILES = 'written_files'
+
+ def __init__(self,
+ base_path,
+ destination_fn,
+ sink_fn,
+ max_writers_per_bundle=WriteToFiles.MAX_NUM_WRITERS_PER_BUNDLE):
+ self.base_path = base_path
+ self.destination_fn = destination_fn
+ self.sink_fn = sink_fn
+ self.max_num_writers_per_bundle = max_writers_per_bundle
+
+ def start_bundle(self):
+ self._writers_and_sinks = {}
+ self._file_names = {}
+
+ def process(self,
+ record,
+ w=beam.DoFn.WindowParam,
+ pane=beam.DoFn.PaneInfoParam):
+ destination = self.destination_fn(record)
+
+ writer, sink = self._get_or_create_writer_and_sink(destination, w)
+
+ if not writer:
+ return [beam.pvalue.TaggedOutput(self.SPILLED_RECORDS, record)]
+ else:
+ sink.write(record)
+
+ def _get_or_create_writer_and_sink(self, destination, window):
+ """Returns a tuple of writer, sink."""
+ writer_key = (destination, window)
+
+ if writer_key in self._writers_and_sinks:
+ return self._writers_and_sinks.get(writer_key)
+ elif len(self._writers_and_sinks) >= self.max_num_writers_per_bundle:
+ # The writer does not exist, and we have too many writers already.
+ return None, None
+ else:
+ # The writer does not exist, but we can still create a new one.
+ full_file_name, writer = _create_writer(base_path=self.base_path.get(),
+ writer_key=writer_key)
+ sink = self.sink_fn(destination)
+
+ sink.open(writer)
+
+ self._writers_and_sinks[writer_key] = (writer, sink)
+ self._file_names[writer_key] = full_file_name
+ return self._writers_and_sinks[writer_key]
+
+ def finish_bundle(self):
+ for key, (writer, sink) in self._writers_and_sinks.items():
+
+ sink.flush()
+ writer.close()
+
+ file_result = FileResult(self._file_names[key],
+ shard_index=-1,
+ total_shards=0,
+ window=key[1],
+ pane=None, # TODO(pabloem): get the pane info
+ destination=key[0])
+
+ yield beam.pvalue.TaggedOutput(
+ self.WRITTEN_FILES,
+ beam.transforms.window.WindowedValue(
+ file_result,
+ timestamp=key[1].start,
+ windows=[key[1]] # TODO(pabloem) HOW DO WE GET THE PANE
+ ))
diff --git a/sdks/python/apache_beam/io/fileio_test.py b/sdks/python/apache_beam/io/fileio_test.py
index 096149b..8c02dcf 100644
--- a/sdks/python/apache_beam/io/fileio_test.py
+++ b/sdks/python/apache_beam/io/fileio_test.py
@@ -21,20 +21,38 @@
import csv
import io
+import json
import logging
import os
import sys
import unittest
+import uuid
+from hamcrest.library.text import stringmatches
from nose.plugins.attrib import attr
import apache_beam as beam
from apache_beam.io import fileio
from apache_beam.io.filebasedsink_test import _TestCaseWithTempDirCleanUp
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.test_utils import compute_hash
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.testing.util import matches_all
+from apache_beam.transforms import trigger
+from apache_beam.transforms.window import FixedWindows
+from apache_beam.transforms.window import GlobalWindow
+
+
+def _get_file_reader(readable_file):
+ if sys.version_info >= (3, 0):
+ return io.TextIOWrapper(readable_file.open())
+ else:
+ return readable_file.open()
class MatchTest(_TestCaseWithTempDirCleanUp):
@@ -48,7 +66,9 @@
files.append(self._create_temp_file(dir=tempdir))
with TestPipeline() as p:
- files_pc = p | fileio.MatchFiles(tempdir) | beam.Map(lambda x: x.path)
+ files_pc = (p
+ | fileio.MatchFiles(FileSystems.join(tempdir, '*'))
+ | beam.Map(lambda x: x.path))
assert_that(files_pc, equal_to(files))
@@ -66,7 +86,7 @@
with TestPipeline() as p:
files_pc = (p
- | beam.Create(directories)
+ | beam.Create([FileSystems.join(d, '*') for d in directories])
| fileio.MatchAll()
| beam.Map(lambda x: x.path))
@@ -85,7 +105,7 @@
with TestPipeline() as p:
files_pc = (
p
- | beam.Create(directories)
+ | beam.Create([FileSystems.join(d, '*') for d in directories])
| fileio.MatchAll(fileio.EmptyMatchTreatment.DISALLOW)
| beam.Map(lambda x: x.path))
@@ -103,7 +123,7 @@
with TestPipeline() as p:
files_pc = (
p
- | beam.Create(['%s*' % d for d in directories])
+ | beam.Create([FileSystems.join(d, '*') for d in directories])
| fileio.MatchAll(fileio.EmptyMatchTreatment.ALLOW_IF_WILDCARD)
| beam.Map(lambda x: x.path))
@@ -119,7 +139,7 @@
with TestPipeline() as p:
content_pc = (p
- | beam.Create([dir])
+ | beam.Create([FileSystems.join(dir, '*')])
| fileio.MatchAll()
| fileio.ReadMatches()
| beam.FlatMap(
@@ -134,18 +154,12 @@
dir = '%s%s' % (self._new_tempdir(), os.sep)
self._create_temp_file(dir=dir, content=content)
- def get_csv_reader(readable_file):
- if sys.version_info >= (3, 0):
- return csv.reader(io.TextIOWrapper(readable_file.open()))
- else:
- return csv.reader(readable_file.open())
-
with TestPipeline() as p:
content_pc = (p
- | beam.Create([dir])
+ | beam.Create([FileSystems.join(dir, '*')])
| fileio.MatchAll()
| fileio.ReadMatches()
- | beam.FlatMap(get_csv_reader))
+ | beam.FlatMap(lambda rf: csv.reader(_get_file_reader(rf))))
assert_that(content_pc, equal_to(rows))
@@ -160,7 +174,7 @@
with TestPipeline() as p:
contents_pc = (p
- | beam.Create(files + [tempdir])
+ | beam.Create(files + ['%s/' % tempdir])
| fileio.ReadMatches()
| beam.FlatMap(
lambda x: x.read().decode('utf-8').splitlines()))
@@ -179,7 +193,7 @@
with self.assertRaises(beam.io.filesystem.BeamIOError):
with TestPipeline() as p:
_ = (p
- | beam.Create(files + [tempdir])
+ | beam.Create(files + ['%s/' % tempdir])
| fileio.ReadMatches(skip_directories=False)
| beam.Map(lambda x: x.read_utf8()))
@@ -233,6 +247,332 @@
label='Assert Checksums')
+class WriteFilesTest(_TestCaseWithTempDirCleanUp):
+
+ SIMPLE_COLLECTION = [
+ {'project': 'beam', 'foundation': 'apache'},
+ {'project': 'prometheus', 'foundation': 'cncf'},
+ {'project': 'flink', 'foundation': 'apache'},
+ {'project': 'grpc', 'foundation': 'cncf'},
+ {'project': 'spark', 'foundation': 'apache'},
+ {'project': 'kubernetes', 'foundation': 'cncf'},
+ {'project': 'spark', 'foundation': 'apache'},
+ {'project': 'knative', 'foundation': 'cncf'},
+ {'project': 'linux', 'foundation': 'linux'},
+ ]
+
+ LARGER_COLLECTION = ['{:05d}'.format(i) for i in range(200)]
+
+ CSV_HEADERS = ['project', 'foundation']
+
+ SIMPLE_COLLECTION_VALIDATION_SET = set([
+ (elm['project'], elm['foundation']) for elm in SIMPLE_COLLECTION])
+
+ class CsvSink(fileio.TextSink):
+ def __init__(self, headers):
+ self.headers = headers
+
+ def write(self, record):
+ self._fh.write(','.join([record[h] for h in self.headers]).encode('utf8'))
+ self._fh.write('\n'.encode('utf8'))
+
+ class JsonSink(fileio.TextSink):
+
+ def write(self, record):
+ self._fh.write(json.dumps(record).encode('utf8'))
+ self._fh.write('\n'.encode('utf8'))
+
+ def test_write_to_single_file_batch(self):
+
+ dir = self._new_tempdir()
+
+ with TestPipeline() as p:
+ _ = (p
+ | beam.Create(WriteFilesTest.SIMPLE_COLLECTION)
+ | "Serialize" >> beam.Map(json.dumps)
+ | beam.io.fileio.WriteToFiles(path=dir))
+
+ with TestPipeline() as p:
+ result = (p
+ | fileio.MatchFiles(FileSystems.join(dir, '*'))
+ | fileio.ReadMatches()
+ | beam.FlatMap(lambda f: f.read_utf8().strip().split('\n'))
+ | beam.Map(json.loads))
+
+ assert_that(result,
+ equal_to([row for row in self.SIMPLE_COLLECTION]))
+
+ def test_write_to_different_file_types_some_spilling(self):
+
+ dir = self._new_tempdir()
+
+ with TestPipeline() as p:
+ _ = (p
+ | beam.Create(WriteFilesTest.SIMPLE_COLLECTION)
+ | beam.io.fileio.WriteToFiles(
+ path=dir,
+ destination=lambda record: record['foundation'],
+ sink=lambda dest: (
+ WriteFilesTest.CsvSink(WriteFilesTest.CSV_HEADERS)
+ if dest == 'apache' else WriteFilesTest.JsonSink()),
+ file_naming=fileio.destination_prefix_naming(),
+ max_writers_per_bundle=1))
+
+ with TestPipeline() as p:
+ cncf_res = (p
+ | fileio.MatchFiles(FileSystems.join(dir, 'cncf*'))
+ | fileio.ReadMatches()
+ | beam.FlatMap(lambda f: f.read_utf8().strip().split('\n'))
+ | beam.Map(json.loads))
+
+ apache_res = (p
+ | "MatchApache" >> fileio.MatchFiles(
+ FileSystems.join(dir, 'apache*'))
+ | "ReadApache" >> fileio.ReadMatches()
+ | "MapApache" >> beam.FlatMap(
+ lambda rf: csv.reader(_get_file_reader(rf))))
+
+ assert_that(cncf_res,
+ equal_to([row
+ for row in self.SIMPLE_COLLECTION
+ if row['foundation'] == 'cncf']),
+ label='verifyCNCF')
+
+ assert_that(apache_res,
+ equal_to([[row['project'], row['foundation']]
+ for row in self.SIMPLE_COLLECTION
+ if row['foundation'] == 'apache']),
+ label='verifyApache')
+
+ def test_find_orphaned_files(self):
+ dir = self._new_tempdir()
+
+ write_transform = beam.io.fileio.WriteToFiles(path=dir)
+
+ def write_orphaned_file(temp_dir, writer_key):
+ temp_dir_path = FileSystems.join(dir, temp_dir)
+
+ file_prefix_dir = FileSystems.join(
+ temp_dir_path,
+ str(abs(hash(writer_key))))
+
+ file_name = '%s_%s' % (file_prefix_dir, uuid.uuid4())
+ with FileSystems.create(file_name) as f:
+ f.write(b'Hello y\'all')
+
+ return file_name
+
+ with TestPipeline() as p:
+ _ = (p
+ | beam.Create(WriteFilesTest.SIMPLE_COLLECTION)
+ | "Serialize" >> beam.Map(json.dumps)
+ | write_transform)
+
+ # Pre-create the temp directory.
+ temp_dir_path = FileSystems.mkdirs(FileSystems.join(
+ dir, write_transform._temp_directory.get()))
+ write_orphaned_file(write_transform._temp_directory.get(),
+ (None, GlobalWindow()))
+ f2 = write_orphaned_file(write_transform._temp_directory.get(),
+ ('other-dest', GlobalWindow()))
+
+ temp_dir_path = FileSystems.join(dir, write_transform._temp_directory.get())
+ leftovers = FileSystems.match(['%s%s*' % (temp_dir_path, os.sep)])
+ found_files = [m.path for m in leftovers[0].metadata_list]
+ self.assertListEqual(found_files, [f2])
+
+ def test_write_to_different_file_types(self):
+
+ dir = self._new_tempdir()
+
+ with TestPipeline() as p:
+ _ = (p
+ | beam.Create(WriteFilesTest.SIMPLE_COLLECTION)
+ | beam.io.fileio.WriteToFiles(
+ path=dir,
+ destination=lambda record: record['foundation'],
+ sink=lambda dest: (
+ WriteFilesTest.CsvSink(WriteFilesTest.CSV_HEADERS)
+ if dest == 'apache' else WriteFilesTest.JsonSink()),
+ file_naming=fileio.destination_prefix_naming()))
+
+ with TestPipeline() as p:
+ cncf_res = (p
+ | fileio.MatchFiles(FileSystems.join(dir, 'cncf*'))
+ | fileio.ReadMatches()
+ | beam.FlatMap(lambda f: f.read_utf8().strip().split('\n'))
+ | beam.Map(json.loads))
+
+ apache_res = (p
+ | "MatchApache" >> fileio.MatchFiles(
+ FileSystems.join(dir, 'apache*'))
+ | "ReadApache" >> fileio.ReadMatches()
+ | "MapApache" >> beam.FlatMap(
+ lambda rf: csv.reader(_get_file_reader(rf))))
+
+ assert_that(cncf_res,
+ equal_to([row
+ for row in self.SIMPLE_COLLECTION
+ if row['foundation'] == 'cncf']),
+ label='verifyCNCF')
+
+ assert_that(apache_res,
+ equal_to([[row['project'], row['foundation']]
+ for row in self.SIMPLE_COLLECTION
+ if row['foundation'] == 'apache']),
+ label='verifyApache')
+
+ def record_dofn(self):
+ class RecordDoFn(beam.DoFn):
+ def process(self, element):
+ WriteFilesTest.all_records.append(element)
+
+ return RecordDoFn()
+
+ def test_streaming_complex_timing(self):
+ # Use state on the TestCase class, since other references would be pickled
+ # into a closure and not have the desired side effects.
+ #
+ # TODO(BEAM-5295): Use assert_that after it works for the cases here in
+ # streaming mode.
+ WriteFilesTest.all_records = []
+
+ dir = '%s%s' % (self._new_tempdir(), os.sep)
+
+ # Setting up the input (TestStream)
+ ts = TestStream().advance_watermark_to(0)
+ for elm in WriteFilesTest.LARGER_COLLECTION:
+ timestamp = int(elm)
+
+ ts.add_elements([('key', '%s' % elm)])
+ if timestamp % 5 == 0 and timestamp != 0:
+ # TODO(BEAM-3759): Add many firings per window after getting PaneInfo.
+ ts.advance_processing_time(5)
+ ts.advance_watermark_to(timestamp)
+
+ def no_colon_file_naming(*args):
+ file_name = fileio.destination_prefix_naming()(*args)
+ return file_name.replace(':', '_')
+
+ # The pipeline that we are testing
+ options = PipelineOptions()
+ options.view_as(StandardOptions).streaming = True
+ with TestPipeline(options=options) as p:
+ res = (p
+ | ts
+ | beam.WindowInto(
+ FixedWindows(10),
+ trigger=trigger.AfterWatermark(),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING)
+ | beam.GroupByKey()
+ | beam.FlatMap(lambda x: x[1]))
+ # Triggering after 5 processing-time seconds, and on the watermark. Also
+ # discarding old elements.
+
+ _ = (res
+ | beam.io.fileio.WriteToFiles(path=dir,
+ file_naming=no_colon_file_naming,
+ max_writers_per_bundle=0)
+ | beam.Map(lambda fr: FileSystems.join(dir, fr.file_name))
+ | beam.ParDo(self.record_dofn()))
+
+ # Verification pipeline
+ with TestPipeline() as p:
+ files = (p | beam.io.fileio.MatchFiles(FileSystems.join(dir, '*')))
+
+ file_names = (files | beam.Map(lambda fm: fm.path))
+
+ file_contents = (
+ files
+ | beam.io.fileio.ReadMatches()
+ | beam.Map(lambda rf: (rf.metadata.path,
+ rf.read_utf8().strip().split('\n'))))
+
+ content = (file_contents
+ | beam.FlatMap(lambda fc: [ln.strip() for ln in fc[1]]))
+
+ assert_that(file_names, equal_to(WriteFilesTest.all_records),
+ label='AssertFilesMatch')
+ assert_that(content, matches_all(WriteFilesTest.LARGER_COLLECTION),
+ label='AssertContentsMatch')
+
+ def test_streaming_different_file_types(self):
+ dir = self._new_tempdir()
+ input = iter(WriteFilesTest.SIMPLE_COLLECTION)
+ ts = (TestStream()
+ .advance_watermark_to(0)
+ .add_elements([next(input), next(input)])
+ .advance_watermark_to(10)
+ .add_elements([next(input), next(input)])
+ .advance_watermark_to(20)
+ .add_elements([next(input), next(input)])
+ .advance_watermark_to(30)
+ .add_elements([next(input), next(input)])
+ .advance_watermark_to(40))
+
+ def no_colon_file_naming(*args):
+ file_name = fileio.destination_prefix_naming()(*args)
+ return file_name.replace(':', '_')
+
+ with TestPipeline() as p:
+ _ = (p
+ | ts
+ | beam.WindowInto(FixedWindows(10))
+ | beam.io.fileio.WriteToFiles(
+ path=dir,
+ destination=lambda record: record['foundation'],
+ sink=lambda dest: (
+ WriteFilesTest.CsvSink(WriteFilesTest.CSV_HEADERS)
+ if dest == 'apache' else WriteFilesTest.JsonSink()),
+ file_naming=no_colon_file_naming,
+ max_writers_per_bundle=0,
+ ))
+
+ with TestPipeline() as p:
+ cncf_files = (p
+ | fileio.MatchFiles(FileSystems.join(dir, 'cncf*'))
+ | "CncfFileNames" >> beam.Map(lambda fm: fm.path))
+
+ apache_files = (p
+ | "MatchApache" >> fileio.MatchFiles(
+ FileSystems.join(dir, 'apache*'))
+ | "ApacheFileNames" >> beam.Map(lambda fm: fm.path))
+
+ assert_that(cncf_files,
+ matches_all([
+ stringmatches.matches_regexp(
+ '.*cncf-1970-01-01T00_00_00-1970-01-01T00_00_10--.*'
+ ),
+ stringmatches.matches_regexp(
+ '.*cncf-1970-01-01T00_00_10-1970-01-01T00_00_20--.*'
+ ),
+ stringmatches.matches_regexp(
+ '.*cncf-1970-01-01T00_00_20-1970-01-01T00_00_30--.*'
+ ),
+ stringmatches.matches_regexp(
+ '.*cncf-1970-01-01T00_00_30-1970-01-01T00_00_40--.*'
+ )
+ ]),
+ label='verifyCNCFFiles')
+
+ assert_that(apache_files,
+ matches_all([
+ stringmatches.matches_regexp(
+ '.*apache-1970-01-01T00_00_00-1970-01-01T00_00_10--.*'
+ ),
+ stringmatches.matches_regexp(
+ '.*apache-1970-01-01T00_00_10-1970-01-01T00_00_20--.*'
+ ),
+ stringmatches.matches_regexp(
+ '.*apache-1970-01-01T00_00_20-1970-01-01T00_00_30--.*'
+ ),
+ stringmatches.matches_regexp(
+ '.*apache-1970-01-01T00_00_30-1970-01-01T00_00_40--.*'
+ )
+ ]),
+ label='verifyApacheFiles')
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/io/filesystem.py b/sdks/python/apache_beam/io/filesystem.py
index cfdf472..4df3e01 100644
--- a/sdks/python/apache_beam/io/filesystem.py
+++ b/sdks/python/apache_beam/io/filesystem.py
@@ -207,34 +207,25 @@
) < num_bytes:
# Continue reading from the underlying file object until enough bytes are
# available, or EOF is reached.
- buf = self._file.read(self._read_size)
+ if not self._decompressor.unused_data:
+ buf = self._file.read(self._read_size)
+ else:
+ # Any uncompressed data at the end of the stream of a gzip or bzip2
+ # file that is not corrupted points to a concatenated compressed
+ # file. We read concatenated files by recursively creating decompressor
+ # objects for the unused compressed data.
+ buf = self._decompressor.unused_data
+ self._initialize_decompressor()
if buf:
decompressed = self._decompressor.decompress(buf)
del buf # Free up some possibly large and no-longer-needed memory.
self._read_buffer.write(decompressed)
else:
# EOF of current stream reached.
- #
- # Any uncompressed data at the end of the stream of a gzip or bzip2
- # file that is not corrupted points to a concatenated compressed
- # file. We read concatenated files by recursively creating decompressor
- # objects for the unused compressed data.
if (self._compression_type == CompressionTypes.BZIP2 or
self._compression_type == CompressionTypes.DEFLATE or
self._compression_type == CompressionTypes.GZIP):
- if self._decompressor.unused_data != b'':
- buf = self._decompressor.unused_data
-
- if self._compression_type == CompressionTypes.BZIP2:
- self._decompressor = bz2.BZ2Decompressor()
- elif self._compression_type == CompressionTypes.DEFLATE:
- self._decompressor = zlib.decompressobj()
- else:
- self._decompressor = zlib.decompressobj(self._gzip_mask)
-
- decompressed = self._decompressor.decompress(buf)
- self._read_buffer.write(decompressed)
- continue
+ pass
else:
# Deflate, Gzip and bzip2 formats do not require flushing
# remaining data in the decompressor into the read buffer when
@@ -400,8 +391,7 @@
class FileMetadata(object):
- """Metadata about a file path that is the output of FileSystem.match
- """
+ """Metadata about a file path that is the output of FileSystem.match."""
def __init__(self, path, size_in_bytes):
assert isinstance(path, (str, unicode)) and path, "Path should be a string"
assert isinstance(size_in_bytes, (int, long)) and size_in_bytes >= 0, \
@@ -430,7 +420,7 @@
class MatchResult(object):
"""Result from the ``FileSystem`` match operation which contains the list
- of matched FileMetadata.
+ of matched ``FileMetadata``.
"""
def __init__(self, pattern, metadata_list):
self.metadata_list = metadata_list
diff --git a/sdks/python/apache_beam/io/filesystem_test.py b/sdks/python/apache_beam/io/filesystem_test.py
index b26d79d..fbc094f 100644
--- a/sdks/python/apache_beam/io/filesystem_test.py
+++ b/sdks/python/apache_beam/io/filesystem_test.py
@@ -461,6 +461,79 @@
if not line:
break
+ def test_concatenated_compressed_file(self):
+ # The test apache_beam.io.textio_test.test_read_gzip_concat
+ # does not encounter the problem in the Beam 2.13 and earlier
+ # code base because the test data is too small: the data is
+ # smaller than read_size, so it goes through logic in the code
+ # that avoids the problem in the code. So, this test sets
+ # read_size smaller and test data bigger, in order to
+ # encounter the problem. It would be difficult to test in the
+ # textio_test module, because you'd need very large test data
+ # because default read_size is 16MiB, and the ReadFromText
+ # interface does not allow you to modify the read_size.
+ import random
+ import threading
+ from six import int2byte
+ num_test_lines = 10
+ timeout = 30
+ read_size = (64<<10) # set much smaller than the line size
+ byte_table = tuple(int2byte(i) for i in range(32, 96))
+
+ def generate_random_line():
+ byte_list = list(b
+ for i in range(4096)
+ for b in random.sample(byte_table, 64)
+ )
+ byte_list.append(b'\n')
+ return b''.join(byte_list)
+
+ def create_test_file(compression_type, lines):
+ filenames = list()
+ file_name = self._create_temp_file()
+ if compression_type == CompressionTypes.BZIP2:
+ compress_factory = bz2.BZ2File
+ elif compression_type == CompressionTypes.GZIP:
+ compress_factory = gzip.open
+ else:
+ assert False, "Invalid compression type: %s" % compression_type
+ for line in lines:
+ filenames.append(self._create_temp_file())
+ with compress_factory(filenames[-1], 'wb') as f:
+ f.write(line)
+ with open(file_name, 'wb') as o:
+ for name in filenames:
+ with open(name, 'rb') as i:
+ o.write(i.read())
+ return file_name
+
+ # I remember some time ago when a job ran with a real concatenated
+ # gzip file, I got into an endless loop in the beam filesystem module.
+ # That's why I put this handler in to trap an endless loop. However,
+ # this unit test doesn't encounter an endless loop, it encounters a
+ # different error, in the Beam 2.13 and earlier implementation.
+ # So it's not strictly necessary to have this handler in this unit test.
+
+ def timeout_handler():
+ raise IOError('Exiting due to likley infinite loop logic in code.')
+
+ timer = threading.Timer(timeout, timeout_handler)
+ try:
+ test_lines = tuple(generate_random_line() for i in range(num_test_lines))
+ for compression_type in [CompressionTypes.BZIP2, CompressionTypes.GZIP]:
+ file_name = create_test_file(compression_type, test_lines)
+ timer.start()
+ with open(file_name, 'rb') as f:
+ data = CompressedFile(f, compression_type, read_size=read_size)
+ for written_line in test_lines:
+ read_line = data.readline()
+ self.assertEqual(written_line, read_line)
+ timer.cancel()
+ # Starting a new timer for the next iteration/test.
+ timer = threading.Timer(timeout, timeout_handler)
+ finally:
+ timer.cancel()
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/io/filesystemio.py b/sdks/python/apache_beam/io/filesystemio.py
index dca341d..b2d74c1 100644
--- a/sdks/python/apache_beam/io/filesystemio.py
+++ b/sdks/python/apache_beam/io/filesystemio.py
@@ -80,16 +80,21 @@
class DownloaderStream(io.RawIOBase):
"""Provides a stream interface for Downloader objects."""
- def __init__(self, downloader, mode='rb'):
+ def __init__(self,
+ downloader,
+ read_buffer_size=io.DEFAULT_BUFFER_SIZE,
+ mode='rb'):
"""Initializes the stream.
Args:
downloader: (Downloader) Filesystem dependent implementation.
+ read_buffer_size: (int) Buffer size to use during read operations.
mode: (string) Python mode attribute for this stream.
"""
self._downloader = downloader
self.mode = mode
self._position = 0
+ self._reader_buffer_size = read_buffer_size
def readinto(self, b):
"""Read up to len(b) bytes into b.
@@ -157,6 +162,16 @@
def readable(self):
return True
+ def readall(self):
+ """Read until EOF, using multiple read() call."""
+ res = []
+ while True:
+ data = self.read(self._reader_buffer_size)
+ if not data:
+ break
+ res.append(data)
+ return b''.join(res)
+
class UploaderStream(io.RawIOBase):
"""Provides a stream interface for Uploader objects."""
@@ -208,16 +223,25 @@
class PipeStream(object):
- """A class that presents a pipe connection as a readable stream."""
+ """A class that presents a pipe connection as a readable stream.
+
+ Not thread-safe.
+
+ Remembers the last ``size`` bytes read and allows rewinding the stream by that
+ amount exactly. See BEAM-6380 for more.
+ """
def __init__(self, recv_pipe):
self.conn = recv_pipe
self.closed = False
- # TODO(BEAM-6380): For debugging.
- self.last_position = None
self.position = 0
self.remaining = b''
+ # Data and position of last block streamed. Allows limited seeking backwards
+ # of stream.
+ self.last_block_position = None
+ self.last_block = b''
+
def read(self, size):
"""Read data from the wrapped pipe connection.
@@ -230,11 +254,12 @@
"""
data_list = []
bytes_read = 0
+ self.last_block_position = self.position
+
while bytes_read < size:
bytes_from_remaining = min(size - bytes_read, len(self.remaining))
data_list.append(self.remaining[0:bytes_from_remaining])
self.remaining = self.remaining[bytes_from_remaining:]
- self.last_position = self.position
self.position += bytes_from_remaining
bytes_read += bytes_from_remaining
if not self.remaining:
@@ -242,7 +267,8 @@
self.remaining = self.conn.recv_bytes()
except EOFError:
break
- return b''.join(data_list)
+ self.last_block = b''.join(data_list)
+ return self.last_block
def tell(self):
"""Tell the file's current offset.
@@ -262,11 +288,17 @@
# must have this no-op method here in that case.
if whence == os.SEEK_END and offset == 0:
return
- elif whence == os.SEEK_SET and offset == self.position:
- return
+ elif whence == os.SEEK_SET:
+ if offset == self.position:
+ return
+ elif offset == self.last_block_position and self.last_block:
+ self.position = offset
+ self.remaining = b''.join([self.last_block, self.remaining])
+ self.last_block = b''
+ return
raise NotImplementedError(
'offset: %s, whence: %s, position: %s, last: %s' % (
- offset, whence, self.position, self.last_position))
+ offset, whence, self.position, self.last_block_position))
def _check_open(self):
if self.closed:
diff --git a/sdks/python/apache_beam/io/filesystemio_test.py b/sdks/python/apache_beam/io/filesystemio_test.py
index 2177459..72e7f0d 100644
--- a/sdks/python/apache_beam/io/filesystemio_test.py
+++ b/sdks/python/apache_beam/io/filesystemio_test.py
@@ -150,7 +150,7 @@
class TestPipeStream(unittest.TestCase):
- def _read_and_verify(self, stream, expected, buffer_size):
+ def _read_and_verify(self, stream, expected, buffer_size, success):
data_list = []
bytes_read = 0
seen_last_block = False
@@ -169,6 +169,33 @@
bytes_read += len(data)
self.assertEqual(stream.tell(), bytes_read)
self.assertEqual(b''.join(data_list), expected)
+ success[0] = True
+
+ def _read_and_seek(self, stream, expected, buffer_size, success):
+ data_list = []
+ bytes_read = 0
+ while True:
+ data = stream.read(buffer_size)
+
+ # Test bad seek positions.
+ with self.assertRaises(NotImplementedError):
+ stream.seek(bytes_read + 1)
+ with self.assertRaises(NotImplementedError):
+ stream.seek(bytes_read - 1)
+
+ # Rewind stream and test that it reads back the same data again.
+ stream.seek(bytes_read)
+ data2 = stream.read(buffer_size)
+ self.assertEqual(data, data2)
+
+ if not data:
+ break
+ data_list.append(data)
+ bytes_read += len(data)
+ self.assertEqual(stream.tell(), bytes_read)
+ self.assertEqual(len(b''.join(data_list)), len(expected))
+ self.assertEqual(b''.join(data_list), expected)
+ success[0] = True
def test_pipe_stream(self):
block_sizes = list(4**i for i in range(0, 12))
@@ -178,15 +205,19 @@
buffer_sizes = [100001, 512 * 1024, 1024 * 1024]
for buffer_size in buffer_sizes:
- parent_conn, child_conn = multiprocessing.Pipe()
- stream = filesystemio.PipeStream(child_conn)
- child_thread = threading.Thread(
- target=self._read_and_verify, args=(stream, expected, buffer_size))
- child_thread.start()
- for data in data_blocks:
- parent_conn.send_bytes(data)
- parent_conn.close()
- child_thread.join()
+ for target in [self._read_and_verify, self._read_and_seek]:
+ logging.info('buffer_size=%s, target=%s' % (buffer_size, target))
+ parent_conn, child_conn = multiprocessing.Pipe()
+ stream = filesystemio.PipeStream(child_conn)
+ success = [False]
+ child_thread = threading.Thread(
+ target=target, args=(stream, expected, buffer_size, success))
+ child_thread.start()
+ for data in data_blocks:
+ parent_conn.send_bytes(data)
+ parent_conn.close()
+ child_thread.join()
+ self.assertTrue(success[0], 'error in test thread')
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py b/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py
index d4a0dfc..1edf743 100644
--- a/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py
+++ b/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py
@@ -38,7 +38,7 @@
def expand(self, pbegin):
assert isinstance(pbegin, pvalue.PBegin), (
'Input to transform must be a PBegin but found %s' % pbegin)
- return pvalue.PCollection(pbegin.pipeline)
+ return pvalue.PCollection(pbegin.pipeline, is_bounded=False)
def get_windowing(self, inputs):
return Windowing(GlobalWindows())
diff --git a/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py b/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
index caf8101..21de828 100644
--- a/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py
@@ -23,9 +23,7 @@
import base64
import datetime
import logging
-import os
import random
-import sys
import time
import unittest
@@ -47,6 +45,8 @@
except ImportError:
pass
+WAIT_UNTIL_FINISH_DURATION_MS = 15 * 60 * 1000
+
BIG_QUERY_DATASET_ID = 'python_query_to_table_'
NEW_TYPES_INPUT_TABLE = 'python_new_types_table'
NEW_TYPES_OUTPUT_SCHEMA = (
@@ -143,6 +143,7 @@
'output': self.output_table,
'output_schema': DIALECT_OUTPUT_SCHEMA,
'use_standard_sql': False,
+ 'wait_until_finish_duration': WAIT_UNTIL_FINISH_DURATION_MS,
'on_success_matcher': all_of(*pipeline_verifiers)}
options = self.test_pipeline.get_full_options_as_args(**extra_opts)
big_query_query_to_table_pipeline.run_bq_pipeline(options)
@@ -160,6 +161,7 @@
'output': self.output_table,
'output_schema': DIALECT_OUTPUT_SCHEMA,
'use_standard_sql': True,
+ 'wait_until_finish_duration': WAIT_UNTIL_FINISH_DURATION_MS,
'on_success_matcher': all_of(*pipeline_verifiers)}
options = self.test_pipeline.get_full_options_as_args(**extra_opts)
big_query_query_to_table_pipeline.run_bq_pipeline(options)
@@ -180,6 +182,7 @@
'output': self.output_table,
'output_schema': DIALECT_OUTPUT_SCHEMA,
'use_standard_sql': True,
+ 'wait_until_finish_duration': WAIT_UNTIL_FINISH_DURATION_MS,
'on_success_matcher': all_of(*pipeline_verifiers),
'kms_key': kms_key,
'native': True,
@@ -194,10 +197,6 @@
'No encryption configuration found: %s' % table)
self.assertEqual(kms_key, table.encryptionConfiguration.kmsKeyName)
- @unittest.skipIf(sys.version_info[0] == 3 and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3'
- 'TODO: BEAM-6769')
@attr('IT')
def test_big_query_new_types(self):
expected_checksum = test_utils.compute_hash(NEW_TYPES_OUTPUT_EXPECTED)
@@ -212,6 +211,7 @@
'output': self.output_table,
'output_schema': NEW_TYPES_OUTPUT_SCHEMA,
'use_standard_sql': False,
+ 'wait_until_finish_duration': WAIT_UNTIL_FINISH_DURATION_MS,
'on_success_matcher': all_of(*pipeline_verifiers)}
options = self.test_pipeline.get_full_options_as_args(**extra_opts)
big_query_query_to_table_pipeline.run_bq_pipeline(options)
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py b/sdks/python/apache_beam/io/gcp/bigquery.py
index 18df195..4f0a762 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -224,6 +224,9 @@
The GEOGRAPHY data type works with Well-Known Text (See
https://en.wikipedia.org/wiki/Well-known_text) format for reading and writing
to BigQuery.
+BigQuery IO requires values of BYTES datatype to be encoded using base64
+encoding when writing to BigQuery. When bytes are read from BigQuery they are
+returned as base64-encoded bytes.
"""
from __future__ import absolute_import
@@ -917,6 +920,7 @@
additional_bq_parameters=None,
table_side_inputs=None,
schema_side_inputs=None,
+ triggering_frequency=None,
validate=True):
"""Initialize a WriteToBigQuery transform.
@@ -1002,6 +1006,14 @@
passed to the table callable (if one is provided).
schema_side_inputs: A tuple with ``AsSideInput`` PCollections to be
passed to the schema callable (if one is provided).
+ triggering_frequency (int): Every triggering_frequency duration, a
+ BigQuery load job will be triggered for all the data written since
+ the last load job. BigQuery has limits on how many load jobs can be
+ triggered per day, so be careful not to set this duration too low, or
+ you may exceed daily quota. Often this is set to 5 or 10 minutes to
+ ensure that the project stays well under the BigQuery quota.
+ See https://cloud.google.com/bigquery/quota-policy for more information
+ about BigQuery quotas.
validate: Indicates whether to perform validation checks on
inputs. This parameter is primarily used for testing.
"""
@@ -1024,6 +1036,7 @@
self.max_file_size = max_file_size
self.max_files_per_bundle = max_files_per_bundle
self.method = method or WriteToBigQuery.Method.DEFAULT
+ self.triggering_frequency = triggering_frequency
self.insert_retry_strategy = insert_retry_strategy
self._validate = validate
@@ -1109,19 +1122,14 @@
else:
raise TypeError('Unexpected schema argument: %s.' % schema)
- def _compute_method(self, pipeline, options):
- experiments = options.view_as(DebugOptions).experiments or []
-
- # TODO(pabloem): Use a different method to determine if streaming or batch.
- streaming_pipeline = pipeline.options.view_as(StandardOptions).streaming
-
+ def _compute_method(self, experiments, is_streaming_pipeline):
# If the new BQ sink is not activated for experiment flags, then we use
# streaming inserts by default (it gets overridden in dataflow_runner.py).
if 'use_beam_bq_sink' not in experiments:
return self.Method.STREAMING_INSERTS
- elif self.method == self.Method.DEFAULT and streaming_pipeline:
+ elif self.method == self.Method.DEFAULT and is_streaming_pipeline:
return self.Method.STREAMING_INSERTS
- elif self.method == self.Method.DEFAULT and not streaming_pipeline:
+ elif self.method == self.Method.DEFAULT and not is_streaming_pipeline:
return self.Method.FILE_LOADS
else:
return self.method
@@ -1134,7 +1142,11 @@
self.table_reference.projectId = pcoll.pipeline.options.view_as(
GoogleCloudOptions).project
- method_to_use = self._compute_method(p, p.options)
+ experiments = p.options.view_as(DebugOptions).experiments or []
+ # TODO(pabloem): Use a different method to determine if streaming or batch.
+ is_streaming_pipeline = p.options.view_as(StandardOptions).streaming
+
+ method_to_use = self._compute_method(experiments, is_streaming_pipeline)
if (method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS
and self.schema == SCHEMA_AUTODETECT):
@@ -1142,7 +1154,9 @@
'inserts into BigQuery. Only for File Loads.')
if method_to_use == WriteToBigQuery.Method.STREAMING_INSERTS:
- # TODO: Support load jobs for streaming pipelines.
+ if self.triggering_frequency:
+ raise ValueError('triggering_frequency can only be used with '
+ 'FILE_LOADS method of writing to BigQuery.')
bigquery_write_fn = BigQueryWriteFn(
schema=self.schema,
batch_size=self.batch_size,
@@ -1163,16 +1177,13 @@
return {BigQueryWriteFn.FAILED_ROWS: outputs[BigQueryWriteFn.FAILED_ROWS]}
else:
- if p.options.view_as(StandardOptions).streaming:
- raise NotImplementedError(
- 'File Loads to BigQuery are only supported on Batch pipelines.')
-
from apache_beam.io.gcp import bigquery_file_loads
return pcoll | bigquery_file_loads.BigQueryBatchFileLoads(
destination=self.table_reference,
schema=self.schema,
create_disposition=self.create_disposition,
write_disposition=self.write_disposition,
+ triggering_frequency=self.triggering_frequency,
max_file_size=self.max_file_size,
max_files_per_bundle=self.max_files_per_bundle,
custom_gcs_temp_location=self.custom_gcs_temp_location,
@@ -1180,7 +1191,8 @@
table_side_inputs=self.table_side_inputs,
schema_side_inputs=self.schema_side_inputs,
additional_bq_parameters=self.additional_bq_parameters,
- validate=self._validate)
+ validate=self._validate,
+ is_streaming_pipeline=is_streaming_pipeline)
def display_data(self):
res = {}
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py b/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
index 2af9d9f..ab8242d 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
@@ -30,7 +30,6 @@
import datetime
import hashlib
-import itertools
import logging
import random
import time
@@ -44,6 +43,8 @@
from apache_beam.io.gcp import bigquery_tools
from apache_beam.options import value_provider as vp
from apache_beam.options.pipeline_options import GoogleCloudOptions
+from apache_beam.transforms import trigger
+from apache_beam.transforms.window import GlobalWindows
ONE_TERABYTE = (1 << 40)
@@ -58,6 +59,10 @@
# Big query only supports up to 10 thousand URIs for a single load job.
_MAXIMUM_SOURCE_URIS = 10*1000
+# If triggering_frequency is supplied, we will trigger the file write after
+# this many records are written.
+_FILE_TRIGGERING_RECORD_COUNT = 500000
+
def _generate_load_job_name():
datetime_component = datetime.datetime.now().strftime("%Y_%m_%d_%H%M%S")
@@ -65,22 +70,26 @@
return 'beam_load_%s_%s' % (datetime_component, random.randint(0, 100))
-def file_prefix_generator(with_validation=True):
- def _generate_file_prefix(pipeline_gcs_location):
+def file_prefix_generator(with_validation=True,
+ pipeline_gcs_location=None,
+ temp_location=None):
+ def _generate_file_prefix(unused_elm):
# If a gcs location is provided to the pipeline, then we shall use that.
# Otherwise, we shall use the temp_location from pipeline options.
- gcs_base = str(pipeline_gcs_location or
- vp.RuntimeValueProvider.get_value('temp_location', str, ''))
+ gcs_base = pipeline_gcs_location.get()
+ if not gcs_base:
+ gcs_base = temp_location
# This will fail at pipeline execution time, but will fail early, as this
# step doesn't have any dependencies (and thus will be one of the first
# stages to be run).
if with_validation and (not gcs_base or not gcs_base.startswith('gs://')):
- raise ValueError('Invalid GCS location.\n'
+ raise ValueError('Invalid GCS location: %r.\n'
'Writing to BigQuery with FILE_LOADS method requires a '
'GCS location to be provided to write files to be loaded'
' loaded into BigQuery. Please provide a GCS bucket, or '
- 'pass method="STREAMING_INSERTS" to WriteToBigQuery.')
+ 'pass method="STREAMING_INSERTS" to WriteToBigQuery.'
+ % gcs_base)
prefix_uuid = _bq_uuid()
return fs.FileSystems.join(gcs_base, 'bq_load', prefix_uuid)
@@ -189,29 +198,37 @@
destination = bigquery_tools.get_hashable_destination(element[0])
row = element[1]
- if destination in self._destination_to_file_writer:
- writer = self._destination_to_file_writer[destination]
- elif len(self._destination_to_file_writer) < self.max_files_per_bundle:
- (file_path, writer) = _make_new_file_writer(file_prefix, destination)
- self._destination_to_file_writer[destination] = writer
- yield pvalue.TaggedOutput(WriteRecordsToFile.WRITTEN_FILE_TAG,
- (element[0], file_path))
- else:
- yield pvalue.TaggedOutput(
- WriteRecordsToFile.UNWRITTEN_RECORD_TAG, element)
- return
+ if destination not in self._destination_to_file_writer:
+ if len(self._destination_to_file_writer) < self.max_files_per_bundle:
+ self._destination_to_file_writer[destination] = _make_new_file_writer(
+ file_prefix, destination)
+ else:
+ yield pvalue.TaggedOutput(
+ WriteRecordsToFile.UNWRITTEN_RECORD_TAG, element)
+ return
+
+ (file_path, writer) = self._destination_to_file_writer[destination]
# TODO(pabloem): Is it possible for this to throw exception?
writer.write(self.coder.encode(row))
writer.write(b'\n')
- if writer.tell() > self.max_file_size:
+ file_size = writer.tell()
+ if file_size > self.max_file_size:
writer.close()
self._destination_to_file_writer.pop(destination)
+ yield pvalue.TaggedOutput(WriteRecordsToFile.WRITTEN_FILE_TAG,
+ (element[0], (file_path, file_size)))
def finish_bundle(self):
- for _, writer in iteritems(self._destination_to_file_writer):
+ for destination, file_path_writer in \
+ iteritems(self._destination_to_file_writer):
+ (file_path, writer) = file_path_writer
+ file_size = writer.tell()
writer.close()
+ yield pvalue.TaggedOutput(WriteRecordsToFile.WRITTEN_FILE_TAG,
+ GlobalWindows.windowed_value(
+ (destination, (file_path, file_size))))
self._destination_to_file_writer = {}
@@ -235,19 +252,23 @@
destination = element[0]
rows = element[1]
- writer = None
+ file_path, writer = None, None
for row in rows:
if writer is None:
(file_path, writer) = _make_new_file_writer(file_prefix, destination)
- yield (destination, file_path)
writer.write(self.coder.encode(row))
writer.write(b'\n')
- if writer.tell() > self.max_file_size:
+ file_size = writer.tell()
+ if file_size > self.max_file_size:
writer.close()
- writer = None
+ yield (destination, (file_path, file_size))
+ file_path, writer = None, None
+ if writer is not None:
+ writer.close()
+ yield (destination, (file_path, file_size))
class TriggerCopyJobs(beam.DoFn):
@@ -259,16 +280,18 @@
destination tables.
This transform emits (destination, job_reference) pairs.
+
+ TODO(BEAM-7822): In file loads method of writing to BigQuery,
+ copying from temp_tables to destination_table is not atomic.
+ See: https://issues.apache.org/jira/browse/BEAM-7822
"""
def __init__(self,
create_disposition=None,
write_disposition=None,
- test_client=None,
- temporary_tables=False):
+ test_client=None):
self.create_disposition = create_disposition
self.write_disposition = write_disposition
self.test_client = test_client
- self.temporary_tables = temporary_tables
def start_bundle(self):
self.bq_wrapper = bigquery_tools.BigQueryWrapper(client=self.test_client)
@@ -277,11 +300,6 @@
destination = element[0]
job_reference = element[1]
- if not self.temporary_tables:
- # If we did not use temporary tables, then we do not need to trigger any
- # copy jobs.
- return
-
copy_to_reference = bigquery_tools.parse_table_reference(destination)
if copy_to_reference.projectId is None:
copy_to_reference.projectId = vp.RuntimeValueProvider.get_value('project',
@@ -354,8 +372,13 @@
self.bq_wrapper = bigquery_tools.BigQueryWrapper(client=self.test_client)
def process(self, element, load_job_name_prefix, *schema_side_inputs):
+ # Each load job is assumed to have files respecting these constraints:
+ # 1. Total size of all files < 15 TB (Max size for load jobs)
+ # 2. Total no. of files in a single load job < 10,000
+ # This assumption means that there will always be a single load job
+ # triggered for each partition of files.
destination = element[0]
- files = iter(element[1])
+ files = element[1]
if callable(self.schema):
schema = self.schema(destination, *schema_side_inputs)
@@ -371,47 +394,94 @@
else:
additional_parameters = self.additional_bq_parameters
- job_count = 0
- batch_of_files = list(itertools.islice(files, _MAXIMUM_SOURCE_URIS))
- while batch_of_files:
+ table_reference = bigquery_tools.parse_table_reference(destination)
+ if table_reference.projectId is None:
+ table_reference.projectId = vp.RuntimeValueProvider.get_value(
+ 'project', str, '')
+ # Load jobs for a single destination are always triggered from the same
+ # worker. This means that we can generate a deterministic numbered job id,
+ # and not need to worry.
+ destination_hash = _bq_uuid('%s:%s.%s' % (table_reference.projectId,
+ table_reference.datasetId,
+ table_reference.tableId))
+ uid = _bq_uuid()
+ job_name = '%s_%s_%s' % (
+ load_job_name_prefix, destination_hash, uid)
+ logging.debug('Load job has %s files. Job name is %s.',
+ len(files), job_name)
- table_reference = bigquery_tools.parse_table_reference(destination)
- if table_reference.projectId is None:
- table_reference.projectId = vp.RuntimeValueProvider.get_value(
- 'project', str, '')
+ if self.temporary_tables:
+ # For temporary tables, we create a new table with the name with JobId.
+ table_reference.tableId = job_name
+ yield pvalue.TaggedOutput(TriggerLoadJobs.TEMP_TABLES, table_reference)
- # Load jobs for a single destination are always triggered from the same
- # worker. This means that we can generate a deterministic numbered job id,
- # and not need to worry.
- job_name = '%s_%s_%s' % (
- load_job_name_prefix,
- _bq_uuid('%s:%s.%s' % (table_reference.projectId,
- table_reference.datasetId,
- table_reference.tableId)),
- job_count)
- logging.debug('Batch of files has %s files. Job name is %s.',
- len(batch_of_files), job_name)
+ logging.info('Triggering job %s to load data to BigQuery table %s.'
+ 'Schema: %s. Additional parameters: %s',
+ job_name, table_reference,
+ schema, additional_parameters)
+ job_reference = self.bq_wrapper.perform_load_job(
+ table_reference, files, job_name,
+ schema=schema,
+ write_disposition=self.write_disposition,
+ create_disposition=self.create_disposition,
+ additional_load_parameters=additional_parameters)
+ yield (destination, job_reference)
- if self.temporary_tables:
- # For temporary tables, we create a new table with the name with JobId.
- table_reference.tableId = job_name
- yield pvalue.TaggedOutput(TriggerLoadJobs.TEMP_TABLES, table_reference)
- logging.info('Triggering job %s to load data to BigQuery table %s.'
- 'Schema: %s. Additional parameters: %s',
- job_name, table_reference,
- schema, additional_parameters)
- job_reference = self.bq_wrapper.perform_load_job(
- table_reference, batch_of_files, job_name,
- schema=schema,
- write_disposition=self.write_disposition,
- create_disposition=self.create_disposition,
- additional_load_parameters=additional_parameters)
- yield (destination, job_reference)
+class PartitionFiles(beam.DoFn):
- # Prepare to trigger the next job
- job_count += 1
- batch_of_files = list(itertools.islice(files, _MAXIMUM_SOURCE_URIS))
+ MULTIPLE_PARTITIONS_TAG = 'MULTIPLE_PARTITIONS'
+ SINGLE_PARTITION_TAG = 'SINGLE_PARTITION'
+
+ class Partition(object):
+
+ def __init__(self, max_size, max_files, files=None, size=0):
+ self.max_size = max_size
+ self.max_files = max_files
+ self.files = files if files is not None else []
+ self.size = size
+
+ def can_accept(self, file_size, no_of_files=1):
+ if (((self.size + file_size) <= self.max_size)
+ and ((len(self.files) + no_of_files) <= self.max_files)):
+ return True
+ else:
+ return False
+
+ def add(self, file_path, file_size):
+ self.files.append(file_path)
+ self.size += file_size
+
+ def __init__(self, max_partition_size, max_files_per_partition):
+ self.max_partition_size = max_partition_size
+ self.max_files_per_partition = max_files_per_partition
+
+ def process(self, element):
+ destination = element[0]
+ files = element[1]
+ partitions = []
+
+ latest_partition = PartitionFiles.Partition(self.max_partition_size,
+ self.max_files_per_partition)
+
+ for file_path, file_size in files:
+ if latest_partition.can_accept(file_size):
+ latest_partition.add(file_path, file_size)
+ else:
+ partitions.append(latest_partition.files)
+ latest_partition = PartitionFiles.Partition(
+ self.max_partition_size,
+ self.max_files_per_partition)
+ latest_partition.add(file_path, file_size)
+ partitions.append(latest_partition.files)
+
+ if len(partitions) > 1:
+ output_tag = PartitionFiles.MULTIPLE_PARTITIONS_TAG
+ else:
+ output_tag = PartitionFiles.SINGLE_PARTITION_TAG
+
+ for partition in partitions:
+ yield pvalue.TaggedOutput(output_tag, (destination, partition))
class WaitForBQJobs(beam.DoFn):
@@ -493,74 +563,99 @@
custom_gcs_temp_location=None,
create_disposition=None,
write_disposition=None,
+ triggering_frequency=None,
coder=None,
max_file_size=None,
max_files_per_bundle=None,
+ max_partition_size=None,
+ max_files_per_partition=None,
additional_bq_parameters=None,
table_side_inputs=None,
schema_side_inputs=None,
test_client=None,
- validate=True):
+ validate=True,
+ is_streaming_pipeline=False):
self.destination = destination
self.create_disposition = create_disposition
self.write_disposition = write_disposition
+ self.triggering_frequency = triggering_frequency
self.max_file_size = max_file_size or _DEFAULT_MAX_FILE_SIZE
self.max_files_per_bundle = (max_files_per_bundle or
_DEFAULT_MAX_WRITERS_PER_BUNDLE)
- self._custom_gcs_temp_location = custom_gcs_temp_location
+ self.max_partition_size = max_partition_size or _MAXIMUM_LOAD_SIZE
+ self.max_files_per_partition = (max_files_per_partition or
+ _MAXIMUM_SOURCE_URIS)
+ if (isinstance(custom_gcs_temp_location, str)
+ or custom_gcs_temp_location is None):
+ self._custom_gcs_temp_location = vp.StaticValueProvider(
+ str, custom_gcs_temp_location or '')
+ elif isinstance(custom_gcs_temp_location, vp.ValueProvider):
+ self._custom_gcs_temp_location = custom_gcs_temp_location
+ else:
+ raise ValueError('custom_gcs_temp_location must be str or ValueProvider')
+
self.test_client = test_client
self.schema = schema
self.coder = coder or bigquery_tools.RowAsDictJsonCoder()
# If we have multiple destinations, then we will have multiple load jobs,
# thus we will need temporary tables for atomicity.
- # If the destination is a single one, we assume that we will have only one
- # job to run - and thus we avoid using temporary tables
- self.temp_tables = True if callable(destination) else False
+ self.dynamic_destinations = True if callable(destination) else False
self.additional_bq_parameters = additional_bq_parameters or {}
self.table_side_inputs = table_side_inputs or ()
self.schema_side_inputs = schema_side_inputs or ()
+ self.is_streaming_pipeline = is_streaming_pipeline
self._validate = validate
if self._validate:
self.verify()
def verify(self):
- if (isinstance(self._custom_gcs_temp_location, str) and
- not self._custom_gcs_temp_location.startswith('gs://')):
+ if (isinstance(self._custom_gcs_temp_location.get(),
+ vp.StaticValueProvider) and
+ not self._custom_gcs_temp_location.get().startswith('gs://')):
# Only fail if the custom location is provided, and it is not a GCS
# location.
- raise ValueError('Invalid GCS location.\n'
+ raise ValueError('Invalid GCS location: %r.\n'
'Writing to BigQuery with FILE_LOADS method requires a '
'GCS location to be provided to write files to be '
'loaded into BigQuery. Please provide a GCS bucket, or '
- 'pass method="STREAMING_INSERTS" to WriteToBigQuery.')
+ 'pass method="STREAMING_INSERTS" to WriteToBigQuery.'
+ % self._custom_gcs_temp_location.get())
+ if self.is_streaming_pipeline and not self.triggering_frequency:
+ raise ValueError('triggering_frequency must be specified to use file'
+ 'loads in streaming')
+ elif not self.is_streaming_pipeline and self.triggering_frequency:
+ raise ValueError('triggering_frequency can only be used with file'
+ 'loads in streaming')
- def expand(self, pcoll):
- p = pcoll.pipeline
+ def _window_fn(self):
+ """Set the correct WindowInto PTransform"""
- self._custom_gcs_temp_location = (
- self._custom_gcs_temp_location
- or p.options.view_as(GoogleCloudOptions).temp_location)
+ # The user-supplied triggering_frequency is often chosen to control how
+ # many BigQuery load jobs are triggered, to prevent going over BigQuery's
+ # daily quota for load jobs. If this is set to a large value, currently we
+ # have to buffer all the data until the trigger fires. Instead we ensure
+ # that the files are written if a threshold number of records are ready.
+ # We use only the user-supplied trigger on the actual BigQuery load.
+ # This allows us to offload the data to the filesystem.
+ if self.is_streaming_pipeline:
+ return beam.WindowInto(beam.window.GlobalWindows(),
+ trigger=trigger.Repeatedly(
+ trigger.AfterAny(
+ trigger.AfterProcessingTime(
+ self.triggering_frequency),
+ trigger.AfterCount(
+ _FILE_TRIGGERING_RECORD_COUNT))),
+ accumulation_mode=trigger.AccumulationMode\
+ .DISCARDING)
+ else:
+ return beam.WindowInto(beam.window.GlobalWindows())
- load_job_name_pcv = pvalue.AsSingleton(
- p
- | "ImpulseJobName" >> beam.Create([None])
- | beam.Map(lambda _: _generate_load_job_name()))
-
- file_prefix_pcv = pvalue.AsSingleton(
- p
- | "CreateFilePrefixView" >> beam.Create(
- [self._custom_gcs_temp_location])
- | "GenerateFilePrefix" >> beam.Map(
- file_prefix_generator(self._validate)))
-
+ def _write_files(self, destination_data_kv_pc, file_prefix_pcv):
outputs = (
- pcoll
- | "ApplyGlobalWindow" >> beam.WindowInto(beam.window.GlobalWindows())
- | "AppendDestination" >> beam.ParDo(bigquery_tools.AppendDestinationsFn(
- self.destination), *self.table_side_inputs)
+ destination_data_kv_pc
| beam.ParDo(
WriteRecordsToFile(max_files_per_bundle=self.max_files_per_bundle,
max_file_size=self.max_file_size,
@@ -585,55 +680,76 @@
| "GroupShardedRows" >> beam.GroupByKey()
| "DropShardNumber" >> beam.Map(lambda x: (x[0][0], x[1]))
| "WriteGroupedRecordsToFile" >> beam.ParDo(WriteGroupedRecordsToFile(
- coder=self.coder), file_prefix=file_prefix_pcv)
- )
+ coder=self.coder), file_prefix=file_prefix_pcv))
all_destination_file_pairs_pc = (
(destination_files_kv_pc, more_destination_files_kv_pc)
| "DestinationFilesUnion" >> beam.Flatten())
- grouped_files_pc = (
- all_destination_file_pairs_pc
- | "GroupFilesByTableDestinations" >> beam.GroupByKey())
+ if self.is_streaming_pipeline:
+ # Apply the user's trigger back before we start triggering load jobs
+ all_destination_file_pairs_pc = (
+ all_destination_file_pairs_pc
+ | "ApplyUserTrigger" >> beam.WindowInto(
+ beam.window.GlobalWindows(),
+ trigger=trigger.Repeatedly(
+ trigger.AfterAll(
+ trigger.AfterProcessingTime(self.triggering_frequency),
+ trigger.AfterCount(1))),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING))
+ return all_destination_file_pairs_pc
- # Load Jobs are triggered to temporary tables, and those are later copied to
- # the actual appropriate destination query. This ensures atomicity when only
- # some of the load jobs would fail but not other.
- # If any of them fails, then copy jobs are not triggered.
+ def _load_data(self, partitions_using_temp_tables,
+ partitions_direct_to_destination, load_job_name_pcv,
+ singleton_pc):
+ """Load data to BigQuery
+
+ Data is loaded into BigQuery in the following two ways:
+ 1. Single partition:
+ When there is a single partition of files destined to a single
+ destination, a single load job is triggered.
+ 2. Multiple partitions and/or Dynamic Destinations:
+ When there are multiple partitions of files destined for a single
+ destination or when Dynamic Destinations are used, multiple load jobs
+ need to be triggered for each partition/destination. Load Jobs are
+ triggered to temporary tables, and those are later copied to the actual
+ appropriate destination table. This ensures atomicity when only some
+ of the load jobs would fail but not other. If any of them fails, then
+ copy jobs are not triggered.
+ """
+ # Load data using temp tables
trigger_loads_outputs = (
- grouped_files_pc | beam.ParDo(
+ partitions_using_temp_tables
+ | "TriggerLoadJobsWithTempTables" >> beam.ParDo(
TriggerLoadJobs(
schema=self.schema,
write_disposition=self.write_disposition,
create_disposition=self.create_disposition,
test_client=self.test_client,
- temporary_tables=self.temp_tables,
+ temporary_tables=True,
additional_bq_parameters=self.additional_bq_parameters),
- load_job_name_pcv, *self.schema_side_inputs).with_outputs(
- TriggerLoadJobs.TEMP_TABLES, main='main')
+ load_job_name_pcv, *self.schema_side_inputs)
+ .with_outputs(TriggerLoadJobs.TEMP_TABLES, main='main')
)
- destination_job_ids_pc = trigger_loads_outputs['main']
+ temp_tables_load_job_ids_pc = trigger_loads_outputs['main']
temp_tables_pc = trigger_loads_outputs[TriggerLoadJobs.TEMP_TABLES]
destination_copy_job_ids_pc = (
- p
- | "ImpulseMonitorLoadJobs" >> beam.Create([None])
- | "WaitForLoadJobs" >> beam.ParDo(
+ singleton_pc
+ | "WaitForTempTableLoadJobs" >> beam.ParDo(
WaitForBQJobs(self.test_client),
- beam.pvalue.AsList(destination_job_ids_pc))
+ beam.pvalue.AsList(temp_tables_load_job_ids_pc))
| beam.ParDo(TriggerCopyJobs(
create_disposition=self.create_disposition,
write_disposition=self.write_disposition,
- temporary_tables=self.temp_tables,
test_client=self.test_client), load_job_name_pcv))
- finished_copy_jobs_pc = (p
- | "ImpulseMonitorCopyJobs" >> beam.Create([None])
- | "WaitForCopyJobs" >> beam.ParDo(
- WaitForBQJobs(self.test_client),
- beam.pvalue.AsList(destination_copy_job_ids_pc)
- ))
+ finished_copy_jobs_pc = (
+ singleton_pc
+ | "WaitForCopyJobs" >> beam.ParDo(
+ WaitForBQJobs(self.test_client),
+ beam.pvalue.AsList(destination_copy_job_ids_pc)))
_ = (finished_copy_jobs_pc
| "RemoveTempTables/PassTables" >> beam.FlatMap(
@@ -642,10 +758,96 @@
| "RemoveTempTables/AddUselessValue" >> beam.Map(lambda x: (x, None))
| "RemoveTempTables/DeduplicateTables" >> beam.GroupByKey()
| "RemoveTempTables/GetTableNames" >> beam.Map(lambda elm: elm[0])
- | "RemoveTempTables/Delete" >> beam.ParDo(DeleteTablesFn()))
+ | "RemoveTempTables/Delete" >> beam.ParDo(
+ DeleteTablesFn(self.test_client)))
+
+ # Load data directly to destination table
+ destination_load_job_ids_pc = (
+ partitions_direct_to_destination
+ | "TriggerLoadJobsWithoutTempTables" >> beam.ParDo(
+ TriggerLoadJobs(
+ schema=self.schema,
+ write_disposition=self.write_disposition,
+ create_disposition=self.create_disposition,
+ test_client=self.test_client,
+ temporary_tables=False,
+ additional_bq_parameters=self.additional_bq_parameters),
+ load_job_name_pcv, *self.schema_side_inputs)
+ )
+
+ _ = (
+ singleton_pc
+ | "WaitForDestinationLoadJobs" >> beam.ParDo(
+ WaitForBQJobs(self.test_client),
+ beam.pvalue.AsList(destination_load_job_ids_pc)))
+
+ destination_load_job_ids_pc = (
+ (temp_tables_load_job_ids_pc, destination_load_job_ids_pc)
+ | beam.Flatten())
+
+ return destination_load_job_ids_pc, destination_copy_job_ids_pc
+
+ def expand(self, pcoll):
+ p = pcoll.pipeline
+
+ temp_location = p.options.view_as(GoogleCloudOptions).temp_location
+
+ empty_pc = p | "ImpulseEmptyPC" >> beam.Create([])
+ singleton_pc = p | "ImpulseSingleElementPC" >> beam.Create([None])
+
+ load_job_name_pcv = pvalue.AsSingleton(
+ singleton_pc
+ | beam.Map(lambda _: _generate_load_job_name()))
+
+ file_prefix_pcv = pvalue.AsSingleton(
+ singleton_pc
+ | "GenerateFilePrefix" >> beam.Map(
+ file_prefix_generator(self._validate,
+ self._custom_gcs_temp_location,
+ temp_location)))
+
+ destination_data_kv_pc = (
+ pcoll
+ | "RewindowIntoGlobal" >> self._window_fn()
+ | "AppendDestination" >> beam.ParDo(bigquery_tools.AppendDestinationsFn(
+ self.destination), *self.table_side_inputs))
+
+ all_destination_file_pairs_pc = self._write_files(destination_data_kv_pc,
+ file_prefix_pcv)
+
+ grouped_files_pc = (
+ all_destination_file_pairs_pc
+ | "GroupFilesByTableDestinations" >> beam.GroupByKey())
+
+ partitions = (grouped_files_pc
+ | beam.ParDo(PartitionFiles(self.max_partition_size,
+ self.max_files_per_partition))
+ .with_outputs(PartitionFiles.MULTIPLE_PARTITIONS_TAG,
+ PartitionFiles.SINGLE_PARTITION_TAG))
+
+ multiple_partitions_per_destination_pc = partitions[
+ PartitionFiles.MULTIPLE_PARTITIONS_TAG]
+ single_partition_per_destination_pc = partitions[
+ PartitionFiles.SINGLE_PARTITION_TAG]
+
+ # When using dynamic destinations, elements with both single as well as
+ # multiple partitions are loaded into BigQuery using temporary tables to
+ # ensure atomicity.
+ if self.dynamic_destinations:
+ all_partitions = ((multiple_partitions_per_destination_pc,
+ single_partition_per_destination_pc)
+ | "FlattenPartitions" >> beam.Flatten())
+ destination_load_job_ids_pc, destination_copy_job_ids_pc = self.\
+ _load_data(all_partitions, empty_pc, load_job_name_pcv,
+ singleton_pc)
+ else:
+ destination_load_job_ids_pc, destination_copy_job_ids_pc = self.\
+ _load_data(multiple_partitions_per_destination_pc,
+ single_partition_per_destination_pc,
+ load_job_name_pcv, singleton_pc)
return {
- self.DESTINATION_JOBID_PAIRS: destination_job_ids_pc,
+ self.DESTINATION_JOBID_PAIRS: destination_load_job_ids_pc,
self.DESTINATION_FILE_PAIRS: all_destination_file_pairs_pc,
self.DESTINATION_COPY_JOBID_PAIRS: destination_copy_job_ids_pc,
}
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py b/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
index fb272f7..157fb2c 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
@@ -33,15 +33,23 @@
from nose.plugins.attrib import attr
import apache_beam as beam
+from apache_beam import coders
from apache_beam.io.filebasedsink_test import _TestCaseWithTempDirCleanUp
from apache_beam.io.gcp import bigquery_file_loads as bqfl
from apache_beam.io.gcp import bigquery
from apache_beam.io.gcp import bigquery_tools
from apache_beam.io.gcp.internal.clients import bigquery as bigquery_api
from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryFullResultMatcher
+from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryFullResultStreamingMatcher
+from apache_beam.runners.dataflow.test_dataflow_runner import TestDataflowRunner
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.transforms import combiners
+from apache_beam.typehints.typehints import Tuple
try:
from apitools.base.py.exceptions import HttpError
@@ -83,6 +91,23 @@
_ELEMENTS = list([json.loads(elm[1]) for elm in _DESTINATION_ELEMENT_PAIRS])
+class CustomRowCoder(coders.Coder):
+ """
+ Custom row coder that also expects strings as input data when encoding
+ """
+
+ def __init__(self):
+ self.coder = bigquery_tools.RowAsDictJsonCoder()
+
+ def encode(self, table_row):
+ if type(table_row) == str:
+ table_row = json.loads(table_row)
+ return self.coder.encode(table_row)
+
+ def decode(self, encoded_table_row):
+ return self.coder.decode(encoded_table_row)
+
+
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
class TestWriteRecordsToFile(_TestCaseWithTempDirCleanUp):
maxDiff = None
@@ -104,14 +129,14 @@
def test_files_created(self):
"""Test that the files are created and written."""
- fn = bqfl.WriteRecordsToFile()
+ fn = bqfl.WriteRecordsToFile(coder=CustomRowCoder())
self.tmpdir = self._new_tempdir()
def check_files_created(output_pcs):
dest_file_pc = output_pcs[bqfl.WriteRecordsToFile.WRITTEN_FILE_TAG]
- files = dest_file_pc | "GetFiles" >> beam.Map(lambda x: x[1])
- file_count = files | "CountFiles" >> beam.combiners.Count.Globally()
+ files = dest_file_pc | "GetFiles" >> beam.Map(lambda x: x[1][0])
+ file_count = files | "CountFiles" >> combiners.Count.Globally()
_ = files | "FilesExist" >> beam.Map(
lambda x: hamcrest_assert(os.path.exists(x), is_(True)))
@@ -133,7 +158,7 @@
file length is very small, so only a couple records fit in each file.
"""
- fn = bqfl.WriteRecordsToFile(max_file_size=50)
+ fn = bqfl.WriteRecordsToFile(max_file_size=50, coder=CustomRowCoder())
self.tmpdir = self._new_tempdir()
def check_many_files(output_pcs):
@@ -141,8 +166,8 @@
files_per_dest = (dest_file_pc
| beam.Map(lambda x: x).with_output_types(
- beam.typehints.KV[str, str])
- | beam.combiners.Count.PerKey())
+ beam.typehints.KV[str, Tuple[str, int]])
+ | combiners.Count.PerKey())
files_per_dest = (
files_per_dest
| "GetDests" >> beam.Map(
@@ -155,7 +180,7 @@
('project1:dataset1.table3', 1)]))
# Check that the files exist
- _ = dest_file_pc | beam.Map(lambda x: x[1]) | beam.Map(
+ _ = dest_file_pc | beam.Map(lambda x: x[1][0]) | beam.Map(
lambda x: hamcrest_assert(os.path.exists(x), is_(True)))
self._consume_input(fn, check_many_files)
@@ -163,12 +188,13 @@
def test_records_are_spilled(self):
"""Forces records to be written to many files.
- For each destination multiple files are necessary, and at most two files can
- be created. This forces records to be spilled to the next stage of
+ For each destination multiple files are necessary, and at most two files
+ can be created. This forces records to be spilled to the next stage of
processing.
"""
- fn = bqfl.WriteRecordsToFile(max_files_per_bundle=2)
+ fn = bqfl.WriteRecordsToFile(max_files_per_bundle=2,
+ coder=CustomRowCoder())
self.tmpdir = self._new_tempdir()
def check_many_files(output_pcs):
@@ -177,13 +203,13 @@
bqfl.WriteRecordsToFile.UNWRITTEN_RECORD_TAG]
spilled_records_count = (spilled_records_pc |
- beam.combiners.Count.Globally())
+ combiners.Count.Globally())
assert_that(spilled_records_count, equal_to([3]), label='spilled count')
files_per_dest = (dest_file_pc
| beam.Map(lambda x: x).with_output_types(
- beam.typehints.KV[str, str])
- | beam.combiners.Count.PerKey())
+ beam.typehints.KV[str, Tuple[str, int]])
+ | combiners.Count.PerKey())
files_per_dest = (
files_per_dest
| "GetDests" >> beam.Map(
@@ -197,7 +223,7 @@
label='file count')
# Check that the files exist
- _ = dest_file_pc | beam.Map(lambda x: x[1]) | beam.Map(
+ _ = dest_file_pc | beam.Map(lambda x: x[1][0]) | beam.Map(
lambda x: hamcrest_assert(os.path.exists(x), is_(True)))
self._consume_input(fn, check_many_files)
@@ -222,12 +248,12 @@
def test_files_are_created(self):
"""Test that the files are created and written."""
- fn = bqfl.WriteGroupedRecordsToFile()
+ fn = bqfl.WriteGroupedRecordsToFile(coder=CustomRowCoder())
self.tmpdir = self._new_tempdir()
def check_files_created(output_pc):
- files = output_pc | "GetFiles" >> beam.Map(lambda x: x[1])
- file_count = files | "CountFiles" >> beam.combiners.Count.Globally()
+ files = output_pc | "GetFiles" >> beam.Map(lambda x: x[1][0])
+ file_count = files | "CountFiles" >> combiners.Count.Globally()
_ = files | "FilesExist" >> beam.Map(
lambda x: hamcrest_assert(os.path.exists(x), is_(True)))
@@ -249,11 +275,12 @@
For each destination multiple files are necessary. This is because the max
file length is very small, so only a couple records fit in each file.
"""
- fn = bqfl.WriteGroupedRecordsToFile(max_file_size=50)
+ fn = bqfl.WriteGroupedRecordsToFile(max_file_size=50,
+ coder=CustomRowCoder())
self.tmpdir = self._new_tempdir()
def check_multiple_files(output_pc):
- files_per_dest = output_pc | beam.combiners.Count.PerKey()
+ files_per_dest = output_pc | combiners.Count.PerKey()
files_per_dest = (
files_per_dest
| "GetDests" >> beam.Map(
@@ -265,12 +292,76 @@
('project1:dataset1.table3', 1), ]))
# Check that the files exist
- _ = output_pc | beam.Map(lambda x: x[1]) | beam.Map(os.path.exists)
+ _ = output_pc | beam.Map(lambda x: x[1][0]) | beam.Map(os.path.exists)
self._consume_input(fn, _DESTINATION_ELEMENT_PAIRS, check_multiple_files)
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
+class TestPartitionFiles(unittest.TestCase):
+
+ _ELEMENTS = [('destination0', [('file0', 50), ('file1', 50),
+ ('file2', 50), ('file3', 50)]),
+ ('destination1', [('file0', 50), ('file1', 50)])]
+
+ def test_partition(self):
+ partition = bqfl.PartitionFiles.Partition(1000, 1)
+ self.assertEqual(partition.can_accept(50), True)
+ self.assertEqual(partition.can_accept(2000), False)
+ self.assertEqual(partition.can_accept(1000), True)
+
+ partition.add('file1', 50)
+ self.assertEqual(partition.files, ['file1'])
+ self.assertEqual(partition.size, 50)
+ self.assertEqual(partition.can_accept(50), False)
+ self.assertEqual(partition.can_accept(0), False)
+
+ def test_partition_files_dofn_file_split(self):
+ """Force partitions to split based on max_files"""
+ multiple_partitions_result = [('destination0', ['file0', 'file1']),
+ ('destination0', ['file2', 'file3'])]
+ single_partition_result = [('destination1', ['file0', 'file1'])]
+ with TestPipeline() as p:
+ destination_file_pairs = p | beam.Create(self._ELEMENTS)
+ partitioned_files = (
+ destination_file_pairs
+ | beam.ParDo(bqfl.PartitionFiles(1000, 2))
+ .with_outputs(bqfl.PartitionFiles.MULTIPLE_PARTITIONS_TAG,
+ bqfl.PartitionFiles.SINGLE_PARTITION_TAG))
+ multiple_partitions = partitioned_files[bqfl.PartitionFiles\
+ .MULTIPLE_PARTITIONS_TAG]
+ single_partition = partitioned_files[bqfl.PartitionFiles\
+ .SINGLE_PARTITION_TAG]
+
+ assert_that(multiple_partitions, equal_to(multiple_partitions_result),
+ label='CheckMultiplePartitions')
+ assert_that(single_partition, equal_to(single_partition_result),
+ label='CheckSinglePartition')
+
+ def test_partition_files_dofn_size_split(self):
+ """Force partitions to split based on max_partition_size"""
+ multiple_partitions_result = [('destination0', ['file0', 'file1', 'file2']),
+ ('destination0', ['file3'])]
+ single_partition_result = [('destination1', ['file0', 'file1'])]
+ with TestPipeline() as p:
+ destination_file_pairs = p | beam.Create(self._ELEMENTS)
+ partitioned_files = (
+ destination_file_pairs
+ | beam.ParDo(bqfl.PartitionFiles(150, 10))
+ .with_outputs(bqfl.PartitionFiles.MULTIPLE_PARTITIONS_TAG,
+ bqfl.PartitionFiles.SINGLE_PARTITION_TAG))
+ multiple_partitions = partitioned_files[bqfl.PartitionFiles\
+ .MULTIPLE_PARTITIONS_TAG]
+ single_partition = partitioned_files[bqfl.PartitionFiles\
+ .SINGLE_PARTITION_TAG]
+
+ assert_that(multiple_partitions, equal_to(multiple_partitions_result),
+ label='CheckMultiplePartitions')
+ assert_that(single_partition, equal_to(single_partition_result),
+ label='CheckSinglePartition')
+
+
+@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
class TestBigQueryFileLoads(_TestCaseWithTempDirCleanUp):
def test_records_traverse_transform_with_mocks(self):
@@ -296,7 +387,8 @@
destination,
custom_gcs_temp_location=self._new_tempdir(),
test_client=bq_client,
- validate=False)
+ validate=False,
+ coder=CustomRowCoder())
# Need to test this with the DirectRunner to avoid serializing mocks
with TestPipeline('DirectRunner') as p:
@@ -307,13 +399,13 @@
jobs = dest_job | "GetJobs" >> beam.Map(lambda x: x[1])
- files = dest_files | "GetFiles" >> beam.Map(lambda x: x[1])
+ files = dest_files | "GetFiles" >> beam.Map(lambda x: x[1][0])
destinations = (
dest_files
| "GetDests" >> beam.Map(
lambda x: (
bigquery_tools.get_hashable_destination(x[0]), x[1]))
- | "GetUniques" >> beam.combiners.Count.PerKey()
+ | "GetUniques" >> combiners.Count.PerKey()
| "GetFinalDests" >>beam.Keys())
# All files exist
@@ -321,7 +413,7 @@
lambda x: hamcrest_assert(os.path.exists(x), is_(True))))
# One file per destination
- assert_that(files | beam.combiners.Count.Globally(),
+ assert_that(files | combiners.Count.Globally(),
equal_to([1]),
label='CountFiles')
@@ -332,6 +424,76 @@
assert_that(jobs,
equal_to([job_reference]), label='CheckJobs')
+ def test_multiple_partition_files(self):
+ destination = 'project1:dataset1.table1'
+
+ job_reference = bigquery_api.JobReference()
+ job_reference.projectId = 'project1'
+ job_reference.jobId = 'job_name1'
+ result_job = mock.Mock()
+ result_job.jobReference = job_reference
+
+ mock_job = mock.Mock()
+ mock_job.status.state = 'DONE'
+ mock_job.status.errorResult = None
+ mock_job.jobReference = job_reference
+
+ bq_client = mock.Mock()
+ bq_client.jobs.Get.return_value = mock_job
+
+ bq_client.jobs.Insert.return_value = result_job
+ bq_client.tables.Delete.return_value = None
+
+ with TestPipeline('DirectRunner') as p:
+ outputs = (p
+ | beam.Create(_ELEMENTS)
+ | bqfl.BigQueryBatchFileLoads(
+ destination,
+ custom_gcs_temp_location=self._new_tempdir(),
+ test_client=bq_client,
+ validate=False,
+ coder=CustomRowCoder(),
+ max_file_size=45,
+ max_partition_size=80,
+ max_files_per_partition=2))
+
+ dest_files = outputs[
+ bqfl.BigQueryBatchFileLoads.DESTINATION_FILE_PAIRS]
+ dest_load_jobs = outputs[
+ bqfl.BigQueryBatchFileLoads.DESTINATION_JOBID_PAIRS]
+ dest_copy_jobs = outputs[
+ bqfl.BigQueryBatchFileLoads.DESTINATION_COPY_JOBID_PAIRS]
+
+ load_jobs = dest_load_jobs | "GetLoadJobs" >> beam.Map(lambda x: x[1])
+ copy_jobs = dest_copy_jobs | "GetCopyJobs" >> beam.Map(lambda x: x[1])
+
+ files = dest_files | "GetFiles" >> beam.Map(lambda x: x[1][0])
+ destinations = (
+ dest_files
+ | "GetDests" >> beam.Map(
+ lambda x: (
+ bigquery_tools.get_hashable_destination(x[0]), x[1]))
+ | "GetUniques" >> combiners.Count.PerKey()
+ | "GetFinalDests" >>beam.Keys())
+
+ # All files exist
+ _ = (files | beam.Map(
+ lambda x: hamcrest_assert(os.path.exists(x), is_(True))))
+
+ # One file per destination
+ assert_that(files | "CountFiles" >> combiners.Count.Globally(),
+ equal_to([6]),
+ label='CheckFileCount')
+
+ assert_that(destinations,
+ equal_to([destination]),
+ label='CheckDestinations')
+
+ assert_that(load_jobs | "CountLoadJobs" >> combiners.Count.Globally(),
+ equal_to([6]), label='CheckLoadJobCount')
+ assert_that(copy_jobs | "CountCopyJobs" >> combiners.Count.Globally(),
+ equal_to([6]), label='CheckCopyJobCount')
+
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
class BigQueryFileLoadsIT(unittest.TestCase):
@@ -347,6 +509,10 @@
'{"name": "foundation","type": "STRING"}]}'
)
+ BIG_QUERY_STREAMING_SCHEMA = (
+ {'fields': [{'name': 'Integr', 'type': 'INTEGER', 'mode': 'NULLABLE'}]}
+ )
+
def setUp(self):
self.test_pipeline = TestPipeline(is_integration_test=True)
self.runner_name = type(self.test_pipeline.runner).__name__
@@ -379,25 +545,25 @@
pipeline_verifiers = [
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_1,
+ query="SELECT name, language FROM %s" % output_table_1,
data=[(d['name'], d['language'])
for d in _ELEMENTS
if 'language' in d]),
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_2,
+ query="SELECT name, foundation FROM %s" % output_table_2,
data=[(d['name'], d['foundation'])
for d in _ELEMENTS
if 'foundation' in d]),
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_3,
+ query="SELECT name, language FROM %s" % output_table_3,
data=[(d['name'], d['language'])
for d in _ELEMENTS
if 'language' in d]),
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_4,
+ query="SELECT name, foundation FROM %s" % output_table_4,
data=[(d['name'], d['foundation'])
for d in _ELEMENTS
if 'foundation' in d])]
@@ -446,6 +612,50 @@
max_files_per_bundle=-1))
@attr('IT')
+ def test_bqfl_streaming(self):
+ if isinstance(self.test_pipeline.runner, TestDataflowRunner):
+ self.skipTest("TestStream is not supported on TestDataflowRunner")
+ output_table = '%s_%s' % (self.output_table, 'ints')
+ _SIZE = 100
+ schema = self.BIG_QUERY_STREAMING_SCHEMA
+ l = [{'Integr': i} for i in range(_SIZE)]
+
+ state_matcher = PipelineStateMatcher(PipelineState.RUNNING)
+ bq_matcher = BigqueryFullResultStreamingMatcher(
+ project=self.project,
+ query="SELECT Integr FROM %s"
+ % output_table,
+ data=[(i,) for i in range(100)])
+
+ args = self.test_pipeline.get_full_options_as_args(
+ on_success_matcher=all_of(state_matcher, bq_matcher),
+ experiments='use_beam_bq_sink',
+ streaming=True)
+ with beam.Pipeline(argv=args) as p:
+ stream_source = (TestStream()
+ .advance_watermark_to(0)
+ .advance_processing_time(100)
+ .add_elements(l[:_SIZE//4])
+ .advance_processing_time(100)
+ .advance_watermark_to(100)
+ .add_elements(l[_SIZE//4:2*_SIZE//4])
+ .advance_processing_time(100)
+ .advance_watermark_to(200)
+ .add_elements(l[2*_SIZE//4:3*_SIZE//4])
+ .advance_processing_time(100)
+ .advance_watermark_to(300)
+ .add_elements(l[3*_SIZE//4:])
+ .advance_processing_time(100)
+ .advance_watermark_to_infinity())
+ _ = (p
+ | stream_source
+ | bigquery.WriteToBigQuery(output_table,
+ schema=schema,
+ method=bigquery.WriteToBigQuery \
+ .Method.FILE_LOADS,
+ triggering_frequency=100))
+
+ @attr('IT')
def test_one_job_fails_all_jobs_fail(self):
# If one of the import jobs fails, then other jobs must not be performed.
@@ -466,11 +676,11 @@
pipeline_verifiers = [
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_1,
+ query="SELECT name, language FROM %s" % output_table_1,
data=[]),
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_2,
+ query="SELECT name, foundation FROM %s" % output_table_2,
data=[])]
args = self.test_pipeline.get_full_options_as_args(
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py b/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
new file mode 100644
index 0000000..246d2ce
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
@@ -0,0 +1,213 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for BigQuery sources and sinks."""
+from __future__ import absolute_import
+
+import base64
+import logging
+import random
+import time
+import unittest
+from decimal import Decimal
+
+from future.utils import iteritems
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.io.gcp.bigquery_tools import BigQueryWrapper
+from apache_beam.io.gcp.internal.clients import bigquery
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# Protect against environments where bigquery library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from apitools.base.py.exceptions import HttpError
+except ImportError:
+ HttpError = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+
+class BigQueryReadIntegrationTests(unittest.TestCase):
+ BIG_QUERY_DATASET_ID = 'python_read_table_'
+
+ def setUp(self):
+ self.test_pipeline = TestPipeline(is_integration_test=True)
+ self.runner_name = type(self.test_pipeline.runner).__name__
+ self.project = self.test_pipeline.get_option('project')
+
+ self.bigquery_client = BigQueryWrapper()
+ self.dataset_id = '%s%s%d' % (self.BIG_QUERY_DATASET_ID,
+ str(int(time.time())),
+ random.randint(0, 10000))
+ self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
+ logging.info("Created dataset %s in project %s",
+ self.dataset_id, self.project)
+
+ def tearDown(self):
+ request = bigquery.BigqueryDatasetsDeleteRequest(
+ projectId=self.project, datasetId=self.dataset_id,
+ deleteContents=True)
+ try:
+ logging.info("Deleting dataset %s in project %s",
+ self.dataset_id, self.project)
+ self.bigquery_client.client.datasets.Delete(request)
+ except HttpError:
+ logging.debug('Failed to clean up dataset %s in project %s',
+ self.dataset_id, self.project)
+
+ def create_table(self, tablename):
+ table_schema = bigquery.TableSchema()
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'number'
+ table_field.type = 'INTEGER'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'str'
+ table_field.type = 'STRING'
+ table_schema.fields.append(table_field)
+ table = bigquery.Table(
+ tableReference=bigquery.TableReference(
+ projectId=self.project,
+ datasetId=self.dataset_id,
+ tableId=tablename),
+ schema=table_schema)
+ request = bigquery.BigqueryTablesInsertRequest(
+ projectId=self.project, datasetId=self.dataset_id, table=table)
+ self.bigquery_client.client.tables.Insert(request)
+ table_data = [
+ {'number': 1, 'str': 'abc'},
+ {'number': 2, 'str': 'def'},
+ {'number': 3, 'str': u'你好'},
+ {'number': 4, 'str': u'привет'}
+ ]
+ self.bigquery_client.insert_rows(
+ self.project, self.dataset_id, tablename, table_data)
+
+ def create_table_new_types(self, table_name):
+ table_schema = bigquery.TableSchema()
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'float'
+ table_field.type = 'FLOAT'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'numeric'
+ table_field.type = 'NUMERIC'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'bytes'
+ table_field.type = 'BYTES'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'date'
+ table_field.type = 'DATE'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'time'
+ table_field.type = 'TIME'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'datetime'
+ table_field.type = 'DATETIME'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'timestamp'
+ table_field.type = 'TIMESTAMP'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'geo'
+ table_field.type = 'GEOGRAPHY'
+ table_schema.fields.append(table_field)
+ table = bigquery.Table(
+ tableReference=bigquery.TableReference(
+ projectId=self.project,
+ datasetId=self.dataset_id,
+ tableId=table_name),
+ schema=table_schema)
+ request = bigquery.BigqueryTablesInsertRequest(
+ projectId=self.project, datasetId=self.dataset_id, table=table)
+ self.bigquery_client.client.tables.Insert(request)
+ row_data = {
+ 'float': 0.33, 'numeric': Decimal('10'), 'bytes':
+ base64.b64encode(b'\xab\xac').decode('utf-8'), 'date': '3000-12-31',
+ 'time': '23:59:59', 'datetime': '2018-12-31T12:44:31',
+ 'timestamp': '2018-12-31 12:44:31.744957 UTC', 'geo': 'POINT(30 10)'
+ }
+
+ table_data = [row_data]
+ # add rows with only one key value pair and None values for all other keys
+ for key, value in iteritems(row_data):
+ table_data.append({key: value})
+
+ self.bigquery_client.insert_rows(
+ self.project, self.dataset_id, table_name, table_data)
+
+ @attr('IT')
+ def test_big_query_read(self):
+ table_name = 'python_write_table'
+ self.create_table(table_name)
+ table_id = '{}.{}'.format(self.dataset_id, table_name)
+
+ args = self.test_pipeline.get_full_options_as_args()
+
+ with beam.Pipeline(argv=args) as p:
+ result = (p | 'read' >> beam.io.Read(beam.io.BigQuerySource(
+ query='SELECT number, str FROM `%s`' % table_id,
+ use_standard_sql=True)))
+ assert_that(result, equal_to([{'number': 1, 'str': 'abc'},
+ {'number': 2, 'str': 'def'},
+ {'number': 3, 'str': u'你好'},
+ {'number': 4, 'str': u'привет'}]))
+
+ @attr('IT')
+ def test_big_query_read_new_types(self):
+ table_name = 'python_new_types'
+ self.create_table_new_types(table_name)
+ table_id = '{}.{}'.format(self.dataset_id, table_name)
+
+ args = self.test_pipeline.get_full_options_as_args()
+
+ expected_row = {
+ 'float': 0.33, 'numeric': Decimal('10'), 'bytes':
+ base64.b64encode(b'\xab\xac'), 'date': '3000-12-31',
+ 'time': '23:59:59', 'datetime': '2018-12-31T12:44:31',
+ 'timestamp': '2018-12-31 12:44:31.744957 UTC', 'geo': 'POINT(30 10)'
+ }
+
+ expected_data = [expected_row]
+
+ # add rows with only one key value pair and None values for all other keys
+ for key, value in iteritems(expected_row):
+ row = {k: None for k in expected_row}
+ row[key] = value
+ expected_data.append(row)
+
+ with beam.Pipeline(argv=args) as p:
+ result = (p | 'read' >> beam.io.Read(beam.io.BigQuerySource(
+ query='SELECT float, numeric, bytes, date, time, datetime,'
+ 'timestamp, geo FROM `%s`' % table_id,
+ use_standard_sql=True)))
+ assert_that(result, equal_to(expected_data))
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_read_perf_test.py b/sdks/python/apache_beam/io/gcp/bigquery_read_perf_test.py
new file mode 100644
index 0000000..3b98768
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/bigquery_read_perf_test.py
@@ -0,0 +1,147 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+A performance test for reading data from a BigQuery table.
+Besides of the standard options, there are options with special meaning:
+* input_dataset - BQ dataset id.
+* input_table - BQ table id.
+The table will be created and populated with data from Synthetic Source if it
+does not exist.
+* input_options - options for Synthetic Source:
+num_records - number of rows to be inserted,
+value_size - the length of a single row,
+key_size - required option, but its value has no meaning.
+
+Example test run on DataflowRunner:
+
+python setup.py nosetests \
+ --test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
+ --publish_to_big_query=true
+ --metrics_dataset=gs://...
+ --metrics_table=...
+ --input_dataset=...
+ --input_table=...
+ --input_options='{
+ \"num_records\": 1024,
+ \"key_size\": 1,
+ \"value_size\": 1024,
+ }'" \
+ --tests apache_beam.io.gcp.bigquery_read_perf_test
+"""
+
+from __future__ import absolute_import
+
+import base64
+import logging
+import os
+import unittest
+
+from apache_beam import Map
+from apache_beam import ParDo
+from apache_beam.io import BigQueryDisposition
+from apache_beam.io import BigQuerySource
+from apache_beam.io import Read
+from apache_beam.io import WriteToBigQuery
+from apache_beam.io.gcp.bigquery_tools import BigQueryWrapper
+from apache_beam.io.gcp.bigquery_tools import parse_table_schema_from_json
+from apache_beam.testing.load_tests.load_test import LoadTest
+from apache_beam.testing.load_tests.load_test_metrics_utils import CountMessages
+from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
+from apache_beam.testing.synthetic_pipeline import SyntheticSource
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.combiners import Count
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from apitools.base.py.exceptions import HttpError
+except ImportError:
+ HttpError = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+load_test_enabled = False
+if os.environ.get('LOAD_TEST_ENABLED') == 'true':
+ load_test_enabled = True
+
+
+@unittest.skipIf(not load_test_enabled, 'Enabled only for phrase triggering.')
+class BigQueryReadPerfTest(LoadTest):
+ def setUp(self):
+ super(BigQueryReadPerfTest, self).setUp()
+ self.input_dataset = self.pipeline.get_option('input_dataset')
+ self.input_table = self.pipeline.get_option('input_table')
+ self._check_for_input_data()
+
+ def tearDown(self):
+ super(BigQueryReadPerfTest, self).tearDown()
+ assert_that(self.result, equal_to([self.input_options['num_records']]))
+
+ def _check_for_input_data(self):
+ """Checks if a BQ table with input data exists and creates it if not."""
+ wrapper = BigQueryWrapper()
+ try:
+ wrapper.get_table(self.project_id, self.input_dataset, self.input_table)
+ except HttpError as exn:
+ if exn.status_code == 404:
+ self._create_input_data()
+
+ def _create_input_data(self):
+ """
+ Runs an additional pipeline which creates test data and waits for its
+ completion.
+ """
+ SCHEMA = parse_table_schema_from_json(
+ '{"fields": [{"name": "data", "type": "BYTES"}]}')
+
+ def format_record(record):
+ # Since Synthetic Source returns data as a dictionary, we should skip one
+ # of the part
+ return {'data': base64.b64encode(record[1])}
+
+ p = TestPipeline()
+ # pylint: disable=expression-not-assigned
+ (p
+ | 'Produce rows' >> Read(SyntheticSource(self.parseTestPipelineOptions()))
+ | 'Format' >> Map(format_record)
+ | 'Write to BigQuery' >> WriteToBigQuery(
+ dataset=self.input_dataset, table=self.input_table,
+ schema=SCHEMA,
+ create_disposition=BigQueryDisposition.CREATE_IF_NEEDED,
+ write_disposition=BigQueryDisposition.WRITE_EMPTY))
+ p.run().wait_until_finish()
+
+ def test(self):
+ self.result = (self.pipeline
+ | 'Read from BigQuery' >> Read(BigQuerySource(
+ dataset=self.input_dataset, table=self.input_table))
+ | 'Count messages' >> ParDo(CountMessages(
+ self.metrics_namespace))
+ | 'Measure time' >> ParDo(MeasureTime(
+ self.metrics_namespace))
+ | 'Count' >> Count.Globally())
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_test.py b/sdks/python/apache_beam/io/gcp/bigquery_test.py
index 40c5f67..2c0ef81 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_test.py
@@ -25,6 +25,7 @@
import re
import time
import unittest
+import uuid
import hamcrest as hc
import mock
@@ -34,13 +35,23 @@
from apache_beam.internal.gcp.json_value import to_json_value
from apache_beam.io.gcp import bigquery_tools
from apache_beam.io.gcp.bigquery import TableRowJsonCoder
+from apache_beam.io.gcp.bigquery import WriteToBigQuery
from apache_beam.io.gcp.bigquery_file_loads_test import _ELEMENTS
from apache_beam.io.gcp.bigquery_tools import JSON_COMPLIANCE_ERROR
from apache_beam.io.gcp.internal.clients import bigquery
+from apache_beam.io.gcp.pubsub import ReadFromPubSub
+from apache_beam.io.gcp.tests import utils
from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryFullResultMatcher
+from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryFullResultStreamingMatcher
from apache_beam.io.gcp.tests.bigquery_matcher import BigQueryTableMatcher
from apache_beam.options import value_provider
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.runners.dataflow.test_dataflow_runner import TestDataflowRunner
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing import test_utils
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms.display import DisplayData
@@ -283,7 +294,7 @@
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
-class WriteToBigQuery(unittest.TestCase):
+class TestWriteToBigQuery(unittest.TestCase):
def test_noop_schema_parsing(self):
expected_table_schema = None
@@ -479,6 +490,13 @@
class BigQueryStreamingInsertTransformIntegrationTests(unittest.TestCase):
BIG_QUERY_DATASET_ID = 'python_bq_streaming_inserts_'
+ # Prevent nose from finding and running tests that were not
+ # specified in the Gradle file.
+ # See "More tests may be found" in:
+ # https://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess
+ # /multiprocess.html#other-differences-in-test-running
+ _multiprocess_can_split_ = True
+
def setUp(self):
self.test_pipeline = TestPipeline(is_integration_test=True)
self.runner_name = type(self.test_pipeline.runner).__name__
@@ -521,13 +539,13 @@
expected_properties=additional_bq_parameters),
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_1,
+ query="SELECT name, language FROM %s" % output_table_1,
data=[(d['name'], d['language'])
for d in _ELEMENTS
if 'language' in d]),
BigqueryFullResultMatcher(
project=self.project,
- query="SELECT * FROM %s" % output_table_2,
+ query="SELECT name, language FROM %s" % output_table_2,
data=[(d['name'], d['language'])
for d in _ELEMENTS
if 'language' in d])]
@@ -556,6 +574,10 @@
@attr('IT')
def test_multiple_destinations_transform(self):
+ streaming = self.test_pipeline.options.view_as(StandardOptions).streaming
+ if streaming and isinstance(self.test_pipeline.runner, TestDataflowRunner):
+ self.skipTest("TestStream is not supported on TestDataflowRunner")
+
output_table_1 = '%s%s' % (self.output_table, 1)
output_table_2 = '%s%s' % (self.output_table, 2)
@@ -571,26 +593,52 @@
bad_record = {'language': 1, 'manguage': 2}
- pipeline_verifiers = [
- BigqueryFullResultMatcher(
- project=self.project,
- query="SELECT * FROM %s" % output_table_1,
- data=[(d['name'], d['language'])
- for d in _ELEMENTS
- if 'language' in d]),
- BigqueryFullResultMatcher(
- project=self.project,
- query="SELECT * FROM %s" % output_table_2,
- data=[(d['name'], d['foundation'])
- for d in _ELEMENTS
- if 'foundation' in d])]
+ if streaming:
+ pipeline_verifiers = [
+ PipelineStateMatcher(PipelineState.RUNNING),
+ BigqueryFullResultStreamingMatcher(
+ project=self.project,
+ query="SELECT name, language FROM %s" % output_table_1,
+ data=[(d['name'], d['language'])
+ for d in _ELEMENTS
+ if 'language' in d]),
+ BigqueryFullResultStreamingMatcher(
+ project=self.project,
+ query="SELECT name, foundation FROM %s" % output_table_2,
+ data=[(d['name'], d['foundation'])
+ for d in _ELEMENTS
+ if 'foundation' in d])]
+ else:
+ pipeline_verifiers = [
+ BigqueryFullResultMatcher(
+ project=self.project,
+ query="SELECT name, language FROM %s" % output_table_1,
+ data=[(d['name'], d['language'])
+ for d in _ELEMENTS
+ if 'language' in d]),
+ BigqueryFullResultMatcher(
+ project=self.project,
+ query="SELECT name, foundation FROM %s" % output_table_2,
+ data=[(d['name'], d['foundation'])
+ for d in _ELEMENTS
+ if 'foundation' in d])]
args = self.test_pipeline.get_full_options_as_args(
on_success_matcher=hc.all_of(*pipeline_verifiers),
experiments='use_beam_bq_sink')
with beam.Pipeline(argv=args) as p:
- input = p | beam.Create(_ELEMENTS)
+ if streaming:
+ _SIZE = len(_ELEMENTS)
+ test_stream = (TestStream()
+ .advance_watermark_to(0)
+ .add_elements(_ELEMENTS[:_SIZE//2])
+ .advance_watermark_to(100)
+ .add_elements(_ELEMENTS[_SIZE//2:])
+ .advance_watermark_to_infinity())
+ input = p | test_stream
+ else:
+ input = p | beam.Create(_ELEMENTS)
schema_table_pcv = beam.pvalue.AsDict(
p | "MakeSchemas" >> beam.Create([(full_output_table_1, schema1),
@@ -630,6 +678,95 @@
self.dataset_id, self.project)
+@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
+class PubSubBigQueryIT(unittest.TestCase):
+
+ INPUT_TOPIC = 'psit_topic_output'
+ INPUT_SUB = 'psit_subscription_input'
+
+ BIG_QUERY_DATASET_ID = 'python_pubsub_bq_'
+ SCHEMA = {'fields': [
+ {'name': 'number', 'type': 'INTEGER', 'mode': 'NULLABLE'}]}
+
+ _SIZE = 4
+
+ WAIT_UNTIL_FINISH_DURATION = 15 * 60 * 1000
+
+ def setUp(self):
+ # Set up PubSub
+ self.test_pipeline = TestPipeline(is_integration_test=True)
+ self.runner_name = type(self.test_pipeline.runner).__name__
+ self.project = self.test_pipeline.get_option('project')
+ self.uuid = str(uuid.uuid4())
+ from google.cloud import pubsub
+ self.pub_client = pubsub.PublisherClient()
+ self.input_topic = self.pub_client.create_topic(
+ self.pub_client.topic_path(self.project, self.INPUT_TOPIC + self.uuid))
+ self.sub_client = pubsub.SubscriberClient()
+ self.input_sub = self.sub_client.create_subscription(
+ self.sub_client.subscription_path(self.project,
+ self.INPUT_SUB + self.uuid),
+ self.input_topic.name)
+
+ # Set up BQ
+ self.dataset_ref = utils.create_bq_dataset(self.project,
+ self.BIG_QUERY_DATASET_ID)
+ self.output_table = "%s.output_table" % (self.dataset_ref.dataset_id)
+
+ def tearDown(self):
+ # Tear down PubSub
+ test_utils.cleanup_topics(self.pub_client,
+ [self.input_topic])
+ test_utils.cleanup_subscriptions(self.sub_client,
+ [self.input_sub])
+ # Tear down BigQuery
+ utils.delete_bq_dataset(self.project, self.dataset_ref)
+
+ def _run_pubsub_bq_pipeline(self, method, triggering_frequency=None):
+ l = [i for i in range(self._SIZE)]
+
+ matchers = [
+ PipelineStateMatcher(PipelineState.RUNNING),
+ BigqueryFullResultStreamingMatcher(
+ project=self.project,
+ query="SELECT number FROM %s" % self.output_table,
+ data=[(i,) for i in l])]
+
+ args = self.test_pipeline.get_full_options_as_args(
+ on_success_matcher=hc.all_of(*matchers),
+ wait_until_finish_duration=self.WAIT_UNTIL_FINISH_DURATION,
+ experiments='use_beam_bq_sink',
+ streaming=True)
+
+ def add_schema_info(element):
+ yield {'number': element}
+
+ messages = [str(i).encode('utf-8') for i in l]
+ for message in messages:
+ self.pub_client.publish(self.input_topic.name, message)
+
+ with beam.Pipeline(argv=args) as p:
+ mesages = (p
+ | ReadFromPubSub(subscription=self.input_sub.name)
+ | beam.ParDo(add_schema_info))
+ _ = mesages | WriteToBigQuery(
+ self.output_table,
+ schema=self.SCHEMA,
+ method=method,
+ triggering_frequency=triggering_frequency)
+
+ @attr('IT')
+ def test_streaming_inserts(self):
+ self._run_pubsub_bq_pipeline(WriteToBigQuery.Method.STREAMING_INSERTS)
+
+ @attr('IT')
+ def test_file_loads(self):
+ if isinstance(self.test_pipeline.runner, TestDataflowRunner):
+ self.skipTest('https://issuetracker.google.com/issues/118375066')
+ self._run_pubsub_bq_pipeline(WriteToBigQuery.Method.FILE_LOADS,
+ triggering_frequency=20)
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_tools.py b/sdks/python/apache_beam/io/gcp/bigquery_tools.py
index 1999838..9f30d5f 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_tools.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_tools.py
@@ -659,12 +659,21 @@
if found_table and write_disposition != BigQueryDisposition.WRITE_TRUNCATE:
return found_table
else:
- created_table = self._create_table(
- project_id=project_id,
- dataset_id=dataset_id,
- table_id=table_id,
- schema=schema or found_table.schema,
- additional_parameters=additional_create_parameters)
+ created_table = None
+ try:
+ created_table = self._create_table(
+ project_id=project_id,
+ dataset_id=dataset_id,
+ table_id=table_id,
+ schema=schema or found_table.schema,
+ additional_parameters=additional_create_parameters)
+ except HttpError as exn:
+ if exn.status_code == 409:
+ logging.debug('Skipping Creation. Table %s:%s.%s already exists.'
+ % (project_id, dataset_id, table_id))
+ created_table = self.get_table(project_id, dataset_id, table_id)
+ else:
+ raise
logging.info('Created table %s.%s.%s with schema %s. '
'Result: %s.',
project_id, dataset_id, table_id,
@@ -925,6 +934,12 @@
if self.schema is None:
self.schema = schema
for row in rows:
+ # return base64 encoded bytes as byte type on python 3
+ # which matches the behavior of Beam Java SDK
+ for i in range(len(row.f)):
+ if self.schema.fields[i].type == 'BYTES' and row.f[i].v:
+ row.f[i].v.string_value = row.f[i].v.string_value.encode('utf-8')
+
if self.row_as_dict:
yield self.client.convert_row_to_dict(row, schema)
else:
@@ -998,6 +1013,12 @@
# This code will catch this error to emit an error that explains
# to the programmer that they have used NAN/INF values.
try:
+ # on python 3 base64-encoded bytes are decoded to strings
+ # before being send to bq
+ if sys.version_info[0] > 2:
+ for field, value in iteritems(table_row):
+ if type(value) == bytes:
+ table_row[field] = value.decode('utf-8')
return json.dumps(
table_row, allow_nan=False, default=default_encoder).encode('utf-8')
except ValueError as e:
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py b/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py
index 2dd9af8..ecc0185 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_tools_test.py
@@ -154,6 +154,46 @@
new_dataset = wrapper.get_or_create_dataset('project_id', 'dataset_id')
self.assertEqual(new_dataset.datasetReference.datasetId, 'dataset_id')
+ def test_get_or_create_table(self):
+ client = mock.Mock()
+ client.tables.Insert.return_value = 'table_id'
+ client.tables.Get.side_effect = [None, 'table_id']
+ wrapper = beam.io.gcp.bigquery_tools.BigQueryWrapper(client)
+ new_table = wrapper.get_or_create_table(
+ 'project_id', 'dataset_id', 'table_id',
+ bigquery.TableSchema(fields=[
+ bigquery.TableFieldSchema(name='b', type='BOOLEAN',
+ mode='REQUIRED')]), False, False)
+ self.assertEqual(new_table, 'table_id')
+
+ def test_get_or_create_table_race_condition(self):
+ client = mock.Mock()
+ client.tables.Insert.side_effect = HttpError(
+ response={'status': '409'}, url='', content='')
+ client.tables.Get.side_effect = [None, 'table_id']
+ wrapper = beam.io.gcp.bigquery_tools.BigQueryWrapper(client)
+ new_table = wrapper.get_or_create_table(
+ 'project_id', 'dataset_id', 'table_id',
+ bigquery.TableSchema(fields=[
+ bigquery.TableFieldSchema(name='b', type='BOOLEAN',
+ mode='REQUIRED')]), False, False)
+ self.assertEqual(new_table, 'table_id')
+
+ def test_get_or_create_table_intermittent_exception(self):
+ client = mock.Mock()
+ client.tables.Insert.side_effect = [
+ HttpError(response={'status': '408'}, url='', content=''), 'table_id'
+ ]
+ client.tables.Get.side_effect = [None, 'table_id']
+ wrapper = beam.io.gcp.bigquery_tools.BigQueryWrapper(client)
+ new_table = wrapper.get_or_create_table(
+ 'project_id', 'dataset_id', 'table_id',
+ bigquery.TableSchema(fields=[
+ bigquery.TableFieldSchema(
+ name='b', type='BOOLEAN', mode='REQUIRED')
+ ]), False, False)
+ self.assertEqual(new_table, 'table_id')
+
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
class TestBigQueryReader(unittest.TestCase):
@@ -406,8 +446,8 @@
options = PipelineOptions(flags=['--project', 'myproject'])
source.pipeline_options = options
reader = source.reader()
- self.assertEquals('SELECT * FROM [myproject:mydataset.mytable];',
- reader.query)
+ self.assertEqual('SELECT * FROM [myproject:mydataset.mytable];',
+ reader.query)
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
@@ -573,7 +613,7 @@
options = PipelineOptions(flags=['--project', 'myproject'])
sink.pipeline_options = options
writer = sink.writer()
- self.assertEquals('myproject', writer.project_id)
+ self.assertEqual('myproject', writer.project_id)
@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
new file mode 100644
index 0000000..3658b9c
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
@@ -0,0 +1,274 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for BigQuery sources and sinks."""
+from __future__ import absolute_import
+
+import base64
+import datetime
+import logging
+import random
+import time
+import unittest
+from decimal import Decimal
+
+import hamcrest as hc
+import pytz
+from future.utils import iteritems
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.io.gcp.bigquery_tools import BigQueryWrapper
+from apache_beam.io.gcp.internal.clients import bigquery
+from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryFullResultMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+# Protect against environments where bigquery library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from apitools.base.py.exceptions import HttpError
+except ImportError:
+ HttpError = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+
+class BigQueryWriteIntegrationTests(unittest.TestCase):
+ BIG_QUERY_DATASET_ID = 'python_write_to_table_'
+
+ def setUp(self):
+ self.test_pipeline = TestPipeline(is_integration_test=True)
+ self.runner_name = type(self.test_pipeline.runner).__name__
+ self.project = self.test_pipeline.get_option('project')
+
+ self.bigquery_client = BigQueryWrapper()
+ self.dataset_id = '%s%s%d' % (self.BIG_QUERY_DATASET_ID,
+ str(int(time.time())),
+ random.randint(0, 10000))
+ self.bigquery_client.get_or_create_dataset(self.project, self.dataset_id)
+ logging.info("Created dataset %s in project %s",
+ self.dataset_id, self.project)
+
+ def tearDown(self):
+ request = bigquery.BigqueryDatasetsDeleteRequest(
+ projectId=self.project, datasetId=self.dataset_id,
+ deleteContents=True)
+ try:
+ logging.info("Deleting dataset %s in project %s",
+ self.dataset_id, self.project)
+ self.bigquery_client.client.datasets.Delete(request)
+ except HttpError:
+ logging.debug('Failed to clean up dataset %s in project %s',
+ self.dataset_id, self.project)
+
+ def create_table(self, table_name):
+ table_schema = bigquery.TableSchema()
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'bytes'
+ table_field.type = 'BYTES'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'date'
+ table_field.type = 'DATE'
+ table_schema.fields.append(table_field)
+ table_field = bigquery.TableFieldSchema()
+ table_field.name = 'time'
+ table_field.type = 'TIME'
+ table_schema.fields.append(table_field)
+ table = bigquery.Table(
+ tableReference=bigquery.TableReference(
+ projectId=self.project,
+ datasetId=self.dataset_id,
+ tableId=table_name),
+ schema=table_schema)
+ request = bigquery.BigqueryTablesInsertRequest(
+ projectId=self.project, datasetId=self.dataset_id, table=table)
+ self.bigquery_client.client.tables.Insert(request)
+
+ @attr('IT')
+ def test_big_query_write(self):
+ table_name = 'python_write_table'
+ table_id = '{}.{}'.format(self.dataset_id, table_name)
+
+ input_data = [
+ {'number': 1, 'str': 'abc'},
+ {'number': 2, 'str': 'def'},
+ {'number': 3, 'str': u'你好'},
+ {'number': 4, 'str': u'привет'},
+ ]
+ table_schema = {"fields": [
+ {"name": "number", "type": "INTEGER"},
+ {"name": "str", "type": "STRING"}]}
+
+ pipeline_verifiers = [
+ BigqueryFullResultMatcher(
+ project=self.project,
+ query="SELECT number, str FROM %s" % table_id,
+ data=[(1, 'abc',), (2, 'def',), (3, u'你好',), (4, u'привет',)])]
+
+ args = self.test_pipeline.get_full_options_as_args(
+ on_success_matcher=hc.all_of(*pipeline_verifiers))
+
+ with beam.Pipeline(argv=args) as p:
+ # pylint: disable=expression-not-assigned
+ (p | 'create' >> beam.Create(input_data)
+ | 'write' >> beam.io.WriteToBigQuery(
+ table_id,
+ schema=table_schema,
+ create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
+
+ @attr('IT')
+ def test_big_query_write_schema_autodetect(self):
+ if self.runner_name == 'TestDataflowRunner':
+ self.skipTest('DataflowRunner does not support schema autodetection')
+
+ table_name = 'python_write_table'
+ table_id = '{}.{}'.format(self.dataset_id, table_name)
+
+ input_data = [
+ {'number': 1, 'str': 'abc'},
+ {'number': 2, 'str': 'def'},
+ ]
+
+ pipeline_verifiers = [
+ BigqueryFullResultMatcher(
+ project=self.project,
+ query="SELECT number, str FROM %s" % table_id,
+ data=[(1, 'abc',), (2, 'def',)])]
+
+ args = self.test_pipeline.get_full_options_as_args(
+ on_success_matcher=hc.all_of(*pipeline_verifiers),
+ experiments='use_beam_bq_sink')
+
+ with beam.Pipeline(argv=args) as p:
+ # pylint: disable=expression-not-assigned
+ (p | 'create' >> beam.Create(input_data)
+ | 'write' >> beam.io.WriteToBigQuery(
+ table_id,
+ method=beam.io.WriteToBigQuery.Method.FILE_LOADS,
+ schema=beam.io.gcp.bigquery.SCHEMA_AUTODETECT,
+ create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
+
+ @attr('IT')
+ def test_big_query_write_new_types(self):
+ table_name = 'python_new_types_table'
+ table_id = '{}.{}'.format(self.dataset_id, table_name)
+
+ row_data = {
+ 'float': 0.33, 'numeric': Decimal('10'), 'bytes':
+ base64.b64encode(b'\xab\xac').decode('utf-8'), 'date': '3000-12-31',
+ 'time': '23:59:59', 'datetime': '2018-12-31T12:44:31',
+ 'timestamp': '2018-12-31 12:44:31.744957 UTC', 'geo': 'POINT(30 10)'
+ }
+
+ input_data = [row_data]
+ # add rows with only one key value pair and None values for all other keys
+ for key, value in iteritems(row_data):
+ input_data.append({key: value})
+
+ table_schema = {"fields": [
+ {"name": "float", "type": "FLOAT"},
+ {"name": "numeric", "type": "NUMERIC"},
+ {"name": "bytes", "type": "BYTES"},
+ {"name": "date", "type": "DATE"},
+ {"name": "time", "type": "TIME"},
+ {"name": "datetime", "type": "DATETIME"},
+ {"name": "timestamp", "type": "TIMESTAMP"},
+ {"name": "geo", "type": "GEOGRAPHY"}
+ ]}
+
+ expected_row = (0.33, Decimal('10'), b'\xab\xac',
+ datetime.date(3000, 12, 31), datetime.time(23, 59, 59),
+ datetime.datetime(2018, 12, 31, 12, 44, 31),
+ datetime.datetime(2018, 12, 31, 12, 44, 31, 744957,
+ tzinfo=pytz.utc), 'POINT(30 10)',
+ )
+
+ expected_data = [expected_row]
+
+ # add rows with only one key value pair and None values for all other keys
+ for i, value in enumerate(expected_row):
+ row = [None]*len(expected_row)
+ row[i] = value
+ expected_data.append(tuple(row))
+
+ pipeline_verifiers = [
+ BigqueryFullResultMatcher(
+ project=self.project,
+ query='SELECT float, numeric, bytes, date, time, datetime,'
+ 'timestamp, geo FROM %s' % table_id,
+ data=expected_data)]
+
+ args = self.test_pipeline.get_full_options_as_args(
+ on_success_matcher=hc.all_of(*pipeline_verifiers))
+
+ with beam.Pipeline(argv=args) as p:
+ # pylint: disable=expression-not-assigned
+ (p | 'create' >> beam.Create(input_data)
+ | 'write' >> beam.io.WriteToBigQuery(
+ table_id,
+ schema=table_schema,
+ create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_EMPTY))
+
+ @attr('IT')
+ def test_big_query_write_without_schema(self):
+ table_name = 'python_no_schema_table'
+ self.create_table(table_name)
+ table_id = '{}.{}'.format(self.dataset_id, table_name)
+
+ input_data = [
+ {'bytes': b'xyw', 'date': '2011-01-01', 'time': '23:59:59.999999'},
+ {'bytes': b'abc', 'date': '2000-01-01', 'time': '00:00:00'},
+ {'bytes': b'\xe4\xbd\xa0\xe5\xa5\xbd', 'date': '3000-12-31',
+ 'time': '23:59:59'},
+ {'bytes': b'\xab\xac\xad', 'date': '2000-01-01', 'time': '00:00:00'}
+ ]
+ # bigquery io expects bytes to be base64 encoded values
+ for row in input_data:
+ row['bytes'] = base64.b64encode(row['bytes'])
+
+ pipeline_verifiers = [
+ BigqueryFullResultMatcher(
+ project=self.project,
+ query="SELECT bytes, date, time FROM %s" % table_id,
+ data=[(b'xyw', datetime.date(2011, 1, 1),
+ datetime.time(23, 59, 59, 999999), ),
+ (b'abc', datetime.date(2000, 1, 1),
+ datetime.time(0, 0, 0), ),
+ (b'\xe4\xbd\xa0\xe5\xa5\xbd', datetime.date(3000, 12, 31),
+ datetime.time(23, 59, 59), ),
+ (b'\xab\xac\xad', datetime.date(2000, 1, 1),
+ datetime.time(0, 0, 0), )])]
+
+ args = self.test_pipeline.get_full_options_as_args(
+ on_success_matcher=hc.all_of(*pipeline_verifiers))
+
+ with beam.Pipeline(argv=args) as p:
+ # pylint: disable=expression-not-assigned
+ (p | 'create' >> beam.Create(input_data)
+ | 'write' >> beam.io.WriteToBigQuery(
+ table_id,
+ write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_perf_test.py b/sdks/python/apache_beam/io/gcp/bigquery_write_perf_test.py
new file mode 100644
index 0000000..c8dd304
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/bigquery_write_perf_test.py
@@ -0,0 +1,115 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+A pipeline that writes data from Synthetic Source to a BigQuery table.
+Besides of the standard options, there are options with special meaning:
+* output_dataset - BQ dataset name.
+* output_table - BQ table name. The table will be removed after test completion,
+* input_options - options for Synthetic Source:
+num_records - number of rows to be inserted,
+value_size - the length of a single row,
+key_size - required option, but its value has no meaning.
+
+Example test run on DataflowRunner:
+
+python setup.py nosetests \
+ --test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
+ --publish_to_big_query=true
+ --metrics_dataset=gs://...
+ --metrics_table=...
+ --output_dataset=...
+ --output_table=...
+ --input_options='{
+ \"num_records\": 1024,
+ \"key_size\": 1,
+ \"value_size\": 1024,
+ }'" \
+ --tests apache_beam.io.gcp.bigquery_write_perf_test
+
+This setup will result in a table of 1MB size.
+"""
+
+from __future__ import absolute_import
+
+import base64
+import logging
+import os
+import unittest
+
+from apache_beam import Map
+from apache_beam import ParDo
+from apache_beam.io import BigQueryDisposition
+from apache_beam.io import Read
+from apache_beam.io import WriteToBigQuery
+from apache_beam.io.gcp.bigquery_tools import parse_table_schema_from_json
+from apache_beam.io.gcp.tests import utils
+from apache_beam.testing.load_tests.load_test import LoadTest
+from apache_beam.testing.load_tests.load_test_metrics_utils import CountMessages
+from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
+from apache_beam.testing.synthetic_pipeline import SyntheticSource
+
+load_test_enabled = False
+if os.environ.get('LOAD_TEST_ENABLED') == 'true':
+ load_test_enabled = True
+
+
+@unittest.skipIf(not load_test_enabled, 'Enabled only for phrase triggering.')
+class BigQueryWritePerfTest(LoadTest):
+ def setUp(self):
+ super(BigQueryWritePerfTest, self).setUp()
+ self.output_dataset = self.pipeline.get_option('output_dataset')
+ self.output_table = self.pipeline.get_option('output_table')
+
+ def tearDown(self):
+ super(BigQueryWritePerfTest, self).tearDown()
+ self._cleanup_data()
+
+ def _cleanup_data(self):
+ """Removes an output BQ table."""
+ utils.delete_bq_table(self.project_id, self.output_dataset,
+ self.output_table)
+
+ def test(self):
+ SCHEMA = parse_table_schema_from_json(
+ '{"fields": [{"name": "data", "type": "BYTES"}]}')
+
+ def format_record(record):
+ # Since Synthetic Source returns data as a dictionary, we should skip one
+ # of the part
+ return {'data': base64.b64encode(record[1])}
+
+ # pylint: disable=expression-not-assigned
+ (self.pipeline
+ | 'Produce rows' >> Read(SyntheticSource(self.parseTestPipelineOptions()))
+ | 'Count messages' >> ParDo(CountMessages(self.metrics_namespace))
+ | 'Format' >> Map(format_record)
+ | 'Measure time' >> ParDo(MeasureTime(self.metrics_namespace))
+ | 'Write to BigQuery' >> WriteToBigQuery(
+ dataset=self.output_dataset, table=self.output_table,
+ schema=SCHEMA,
+ create_disposition=BigQueryDisposition.CREATE_IF_NEEDED,
+ write_disposition=BigQueryDisposition.WRITE_TRUNCATE))
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
index d7c38aa..a2bc521 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
@@ -151,7 +151,8 @@
if isinstance(exception, SocketError):
return (exception.errno == errno.ECONNRESET or
- exception.errno == errno.ETIMEDOUT)
+ exception.errno == errno.ETIMEDOUT or
+ exception.errno == errno.EPIPE)
return False
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
index c327e6a..80c66ae 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
@@ -171,11 +171,16 @@
batch_size: the number of entities returned by fake datastore in one req.
"""
- # Test for both random long ids and string ids.
- id_or_name = [True, False]
+ # Test for random long ids, string ids, and a mix of both.
+ id_or_name = [True, False, None]
for id_type in id_or_name:
- entities = fake_datastore.create_entities(num_entities, id_type)
+ if id_type is None:
+ entities = fake_datastore.create_entities(num_entities, False)
+ entities.extend(fake_datastore.create_entities(num_entities, True))
+ num_entities *= 2
+ else:
+ entities = fake_datastore.create_entities(num_entities, id_type)
mock_datastore = MagicMock()
# Assign a fake run_query method as a side_effect to the mock.
mock_datastore.run_query.side_effect = \
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
index 64596b8..f5b1157 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastore_write_it_pipeline.py
@@ -43,7 +43,6 @@
from apache_beam.io.gcp.datastore.v1new.types import Query
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -100,7 +99,6 @@
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
gcloud_options = pipeline_options.view_as(GoogleCloudOptions)
job_name = gcloud_options.job_name
kind = known_args.kind
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
index c04daa2..7ecd1fc 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py
@@ -365,7 +365,9 @@
:class:`~apache_beam.io.gcp.datastore.v1new.types.Entity` to Cloud Datastore.
Entity keys must be complete. The ``project`` field in each key must match the
- project ID passed to this transform.
+ project ID passed to this transform. If ``project`` field in entity or
+ property key is empty then it is filled with the project ID passed to this
+ transform.
"""
def __init__(self, project):
@@ -382,6 +384,8 @@
if not isinstance(element, types.Entity):
raise ValueError('apache_beam.io.gcp.datastore.v1new.datastoreio.Entity'
' expected, got: %s' % type(element))
+ if not element.key.project:
+ element.key.project = self._project
client_entity = element.to_client_entity()
if client_entity.key.is_partial:
raise ValueError('Entities to be written to Cloud Datastore must '
@@ -403,7 +407,8 @@
Datastore.
Keys must be complete. The ``project`` field in each key must match the
- project ID passed to this transform.
+ project ID passed to this transform. If ``project`` field in key is empty then
+ it is filled with the project ID passed to this transform.
"""
def __init__(self, project):
"""Initialize the `DeleteFromDatastore` transform.
@@ -420,6 +425,8 @@
if not isinstance(element, types.Key):
raise ValueError('apache_beam.io.gcp.datastore.v1new.datastoreio.Key'
' expected, got: %s' % type(element))
+ if not element.project:
+ element.project = self._project
client_key = element.to_client_key()
if client_key.is_partial:
raise ValueError('Keys to be deleted from Cloud Datastore must be '
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py
index e3975c7..79d43fe 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio_test.py
@@ -37,6 +37,7 @@
from apache_beam.io.gcp.datastore.v1new.datastoreio import DeleteFromDatastore
from apache_beam.io.gcp.datastore.v1new.datastoreio import ReadFromDatastore
from apache_beam.io.gcp.datastore.v1new.datastoreio import WriteToDatastore
+ from apache_beam.io.gcp.datastore.v1new.types import Key
from google.cloud.datastore import client
from google.cloud.datastore import entity
from google.cloud.datastore import helpers
@@ -203,6 +204,14 @@
entities = helper.create_entities(num_entities)
expected_entities = [entity.to_client_entity() for entity in entities]
+ # Infer project from write fn project arg.
+ if num_entities:
+ key = Key(['k1', 1234], project=self._PROJECT)
+ expected_key = key.to_client_key()
+ key.project = None
+ entities[0].key = key
+ expected_entities[0].key = expected_key
+
all_batch_entities = []
commit_count = [0]
self._mock_client.batch.side_effect = (
@@ -274,6 +283,13 @@
keys = [entity.key for entity in helper.create_entities(10)]
expected_keys = [key.to_client_key() for key in keys]
+ # Infer project from delete fn project arg.
+ key = Key(['k1', 1234], project=self._PROJECT)
+ expected_key = key.to_client_key()
+ key.project = None
+ keys.append(key)
+ expected_keys.append(expected_key)
+
all_batch_keys = []
self._mock_client.batch.side_effect = (
lambda: FakeBatch(all_batch_items=all_batch_keys))
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py
index c4531cc..a5e9ce3 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/helper.py
@@ -24,11 +24,13 @@
from __future__ import absolute_import
import logging
+import os
import time
import uuid
from builtins import range
from google.api_core import exceptions
+from google.cloud import environment_vars
from google.cloud.datastore import client
from apache_beam.io.gcp.datastore.v1new import types
@@ -48,7 +50,9 @@
def get_client(project, namespace):
"""Returns a Cloud Datastore client."""
_client = client.Client(project=project, namespace=namespace)
- _client.base_url = 'https://batch-datastore.googleapis.com' # BEAM-1387
+ # Avoid overwriting user setting. BEAM-7608
+ if not os.environ.get(environment_vars.GCD_HOST, None):
+ _client.base_url = 'https://batch-datastore.googleapis.com' # BEAM-1387
return _client
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py
index 4f8be83..6db4ca4 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py
@@ -26,7 +26,11 @@
from builtins import range
from builtins import round
+from past.builtins import long
+from past.builtins import unicode
+
from apache_beam.io.gcp.datastore.v1new import types
+from apache_beam.options.value_provider import ValueProvider
__all__ = ['QuerySplitterError', 'SplitNotPossibleError', 'get_splits']
@@ -104,7 +108,11 @@
raise SplitNotPossibleError('Query cannot have a limit set.')
for filter in query.filters:
- if filter[1] in ['<', '<=', '>', '>=']:
+ if isinstance(filter[1], ValueProvider):
+ filter_operator = filter[1].get()
+ else:
+ filter_operator = filter[1]
+ if filter_operator in ['<', '<=', '>', '>=']:
raise SplitNotPossibleError('Query cannot have any inequality filters.')
@@ -123,10 +131,59 @@
return scatter_query
+class IdOrName(object):
+ """Represents an ID or name of a Datastore key,
+
+ Implements sort ordering: by ID, then by name, keys with IDs before those
+ with names.
+ """
+ def __init__(self, id_or_name):
+ self.id_or_name = id_or_name
+ if isinstance(id_or_name, (str, unicode)):
+ self.id = None
+ self.name = id_or_name
+ elif isinstance(id_or_name, (int, long)):
+ self.id = id_or_name
+ self.name = None
+ else:
+ raise TypeError('Unexpected type of id_or_name: %s' % id_or_name)
+
+ def __lt__(self, other):
+ if not isinstance(other, IdOrName):
+ return super(IdOrName, self).__lt__(other)
+
+ if self.id is not None:
+ if other.id is None:
+ return True
+ else:
+ return self.id < other.id
+
+ if other.id is not None:
+ return False
+
+ return self.name < other.name
+
+ def __eq__(self, other):
+ if not isinstance(other, IdOrName):
+ return super(IdOrName, self).__eq__(other)
+ return self.id == other.id and self.name == other.name
+
+ def __hash__(self):
+ return hash((self.id, self.other))
+
+
def client_key_sort_key(client_key):
"""Key function for sorting lists of ``google.cloud.datastore.key.Key``."""
- return [client_key.project, client_key.namespace or ''] + [
- str(element) for element in client_key.flat_path]
+ sort_key = [client_key.project, client_key.namespace or '']
+ # A key path is made up of (kind, id_or_name) pairs. The last pair might be
+ # missing an id_or_name.
+ flat_path = list(client_key.flat_path)
+ while flat_path:
+ sort_key.append(flat_path.pop(0)) # kind
+ if flat_path:
+ sort_key.append(IdOrName(flat_path.pop(0)))
+
+ return sort_key
def _get_scatter_keys(client, query, num_splits):
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py
index 3e30859..7f3d1ed 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter_test.py
@@ -103,9 +103,16 @@
unused_batch_size: ignored in v1new since query results are entirely
handled by the Datastore client.
"""
- # Test for both random long ids and string ids.
- for id_or_name in [True, False]:
- client_entities = helper.create_client_entities(num_entities, id_or_name)
+ # Test for random long ids, string ids, and a mix of both.
+ for id_or_name in [True, False, None]:
+ if id_or_name is None:
+ client_entities = helper.create_client_entities(num_entities, False)
+ client_entities.extend(helper.create_client_entities(num_entities,
+ True))
+ num_entities *= 2
+ else:
+ client_entities = helper.create_client_entities(num_entities,
+ id_or_name)
mock_client = mock.MagicMock()
mock_client_query = mock.MagicMock()
@@ -154,6 +161,19 @@
if lt_key is None:
last_query_seen = True
+ def test_id_or_name(self):
+ id_ = query_splitter.IdOrName(1)
+ self.assertEqual(1, id_.id)
+ self.assertIsNone(id_.name)
+ name = query_splitter.IdOrName('1')
+ self.assertIsNone(name.id)
+ self.assertEqual('1', name.name)
+ self.assertEqual(query_splitter.IdOrName(1), query_splitter.IdOrName(1))
+ self.assertEqual(query_splitter.IdOrName('1'), query_splitter.IdOrName('1'))
+ self.assertLess(query_splitter.IdOrName(2), query_splitter.IdOrName('1'))
+ self.assertLess(query_splitter.IdOrName(1), query_splitter.IdOrName(2))
+ self.assertLess(query_splitter.IdOrName('1'), query_splitter.IdOrName('2'))
+
def test_client_key_sort_key(self):
k = key.Key('kind1', 1, project=self._PROJECT, namespace=self._NAMESPACE)
k2 = key.Key('kind2', 'a', parent=k)
@@ -165,6 +185,31 @@
keys.sort(key=query_splitter.client_key_sort_key)
self.assertEqual(expected_sort, keys)
+ def test_client_key_sort_key_ids(self):
+ k1 = key.Key('kind', 2, project=self._PROJECT)
+ k2 = key.Key('kind', 1, project=self._PROJECT)
+ keys = [k1, k2]
+ expected_sort = [k2, k1]
+ keys.sort(key=query_splitter.client_key_sort_key)
+ self.assertEqual(expected_sort, keys)
+
+ def test_client_key_sort_key_names(self):
+ k1 = key.Key('kind', '2', project=self._PROJECT)
+ k2 = key.Key('kind', '1', project=self._PROJECT)
+ keys = [k1, k2]
+ expected_sort = [k2, k1]
+ keys.sort(key=query_splitter.client_key_sort_key)
+ self.assertEqual(expected_sort, keys)
+
+ def test_client_key_sort_key_ids_vs_names(self):
+ # Keys with IDs always come before keys with names.
+ k1 = key.Key('kind', '1', project=self._PROJECT)
+ k2 = key.Key('kind', 2, project=self._PROJECT)
+ keys = [k1, k2]
+ expected_sort = [k2, k1]
+ keys.sort(key=query_splitter.client_key_sort_key)
+ self.assertEqual(expected_sort, keys)
+
# Hide base class from collection by nose.
del QuerySplitterTestBase
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py
index c80fe04..7370d97 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py
@@ -29,6 +29,8 @@
from google.cloud.datastore import key
from google.cloud.datastore import query
+from apache_beam.options.value_provider import ValueProvider
+
__all__ = ['Query', 'Key', 'Entity']
@@ -44,8 +46,11 @@
ancestor: (:class:`~apache_beam.io.gcp.datastore.v1new.types.Key`)
(Optional) key of the ancestor to which this query's results are
restricted.
- filters: (sequence of tuple[str, str, str]) Property filters applied by
- this query. The sequence is ``(property_name, operator, value)``.
+ filters: (sequence of tuple[str, str, str],
+ sequence of
+ tuple[ValueProvider(str), ValueProvider(str), ValueProvider(str)])
+ Property filters applied by this query.
+ The sequence is ``(property_name, operator, value)``.
projection: (sequence of string) fields returned as part of query results.
order: (sequence of string) field names used to order query results.
Prepend ``-`` to a field name to sort it in descending order.
@@ -75,12 +80,39 @@
ancestor_client_key = None
if self.ancestor is not None:
ancestor_client_key = self.ancestor.to_client_key()
+
+ self.filters = self._set_runtime_filters()
+
return query.Query(
client, kind=self.kind, project=self.project, namespace=self.namespace,
ancestor=ancestor_client_key, filters=self.filters,
projection=self.projection, order=self.order,
distinct_on=self.distinct_on)
+ def _set_runtime_filters(self):
+ """
+ Extracts values from ValueProviders in `self.filters` if available
+ :param filters: sequence of tuple[str, str, str] or
+ sequence of tuple[ValueProvider, ValueProvider, ValueProvider]
+ :return: tuple[str, str, str]
+ """
+ runtime_filters = []
+ if not all(len(filter_tuple) == 3 for filter_tuple in self.filters):
+ raise TypeError('%s: filters must be a sequence of tuple with length=3'
+ ' got %r instead'
+ % (self.__class__.__name__, self.filters))
+
+ for filter_type, filter_operator, filter_value in self.filters:
+ if isinstance(filter_type, ValueProvider):
+ filter_type = filter_type.get()
+ if isinstance(filter_operator, ValueProvider):
+ filter_operator = filter_operator.get()
+ if isinstance(filter_value, ValueProvider):
+ filter_value = filter_value.get()
+ runtime_filters.append((filter_type, filter_operator, filter_value))
+
+ return runtime_filters or ()
+
def clone(self):
return copy.copy(self)
@@ -140,6 +172,8 @@
return False
if self.path_elements != other.path_elements:
return False
+ if self.project != other.project:
+ return False
if self.parent is not None and other.parent is not None:
return self.parent == other.parent
@@ -181,21 +215,28 @@
@staticmethod
def from_client_entity(client_entity):
- key = Key.from_client_key(client_entity.key)
- entity = Entity(
- key, exclude_from_indexes=set(client_entity.exclude_from_indexes))
- entity.set_properties(client_entity)
- return entity
+ res = Entity(
+ Key.from_client_key(client_entity.key),
+ exclude_from_indexes=set(client_entity.exclude_from_indexes))
+ for name, value in client_entity.items():
+ if isinstance(value, key.Key):
+ value = Key.from_client_key(value)
+ res.properties[name] = value
+ return res
def to_client_entity(self):
"""
Returns a :class:`google.cloud.datastore.entity.Entity` instance that
represents this entity.
"""
- key = self.key.to_client_key()
- res = entity.Entity(key=key,
+ res = entity.Entity(key=self.key.to_client_key(),
exclude_from_indexes=tuple(self.exclude_from_indexes))
- res.update(self.properties)
+ for name, value in self.properties.items():
+ if isinstance(value, Key):
+ if not value.project:
+ value.project = self.key.project
+ value = value.to_client_key()
+ res[name] = value
return res
def __eq__(self, other):
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
index 7ba82c5..3ceeef8 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/types_test.py
@@ -30,6 +30,7 @@
from apache_beam.io.gcp.datastore.v1new.types import Entity
from apache_beam.io.gcp.datastore.v1new.types import Key
from apache_beam.io.gcp.datastore.v1new.types import Query
+ from apache_beam.options.value_provider import StaticValueProvider
# TODO(BEAM-4543): Remove TypeError once googledatastore dependency is removed.
except (ImportError, TypeError):
client = None
@@ -51,18 +52,23 @@
kc = k.to_client_key()
exclude_from_indexes = ('efi1', 'efi2')
e = Entity(k, exclude_from_indexes=exclude_from_indexes)
- e.set_properties({'efi1': 'value', 'property': 'value'})
+ ref = Key(['kind2', 1235])
+ e.set_properties({'efi1': 'value', 'property': 'value', 'ref': ref})
ec = e.to_client_entity()
self.assertEqual(kc, ec.key)
self.assertSetEqual(set(exclude_from_indexes), ec.exclude_from_indexes)
self.assertEqual('kind', ec.kind)
self.assertEqual(1234, ec.id)
+ self.assertEqual('kind2', ec['ref'].kind)
+ self.assertEqual(1235, ec['ref'].id)
+ self.assertEqual(self._PROJECT, ec['ref'].project)
def testEntityFromClientEntity(self):
k = Key(['kind', 1234], project=self._PROJECT)
exclude_from_indexes = ('efi1', 'efi2')
e = Entity(k, exclude_from_indexes=exclude_from_indexes)
- e.set_properties({'efi1': 'value', 'property': 'value'})
+ ref = Key(['kind2', 1235])
+ e.set_properties({'efi1': 'value', 'property': 'value', 'ref': ref})
efc = Entity.from_client_entity(e.to_client_entity())
self.assertEqual(e, efc)
@@ -101,6 +107,10 @@
kfc3 = Key.from_client_key(kfc2.to_client_key())
self.assertEqual(kfc2, kfc3)
+ kfc4 = Key.from_client_key(kfc2.to_client_key())
+ kfc4.project = 'other'
+ self.assertNotEqual(kfc2, kfc4)
+
def testKeyFromClientKeyNoNamespace(self):
k = Key(['k1', 1234], project=self._PROJECT)
ck = k.to_client_key()
@@ -134,6 +144,31 @@
logging.info('query: %s', q) # Test __repr__()
+ def testValueProviderFilters(self):
+ self.vp_filters = [
+ [(
+ StaticValueProvider(str, 'property_name'),
+ StaticValueProvider(str, '='),
+ StaticValueProvider(str, 'value'))],
+ [(
+ StaticValueProvider(str, 'property_name'),
+ StaticValueProvider(str, '='),
+ StaticValueProvider(str, 'value')),
+ ('property_name', '=', 'value')],
+ ]
+ self.expected_filters = [[('property_name', '=', 'value')],
+ [('property_name', '=', 'value'),
+ ('property_name', '=', 'value')],
+ ]
+
+ for vp_filter, exp_filter in zip(self.vp_filters, self.expected_filters):
+ q = Query(kind='kind', project=self._PROJECT, namespace=self._NAMESPACE,
+ filters=vp_filter)
+ cq = q._to_client_query(self._test_client)
+ self.assertEqual(exp_filter, cq.filters)
+
+ logging.info('query: %s', q) # Test __repr__()
+
def testQueryEmptyNamespace(self):
# Test that we can pass a namespace of None.
self._test_client.namespace = None
diff --git a/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py b/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
index 1ee3e5a..67e375f 100644
--- a/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
@@ -40,7 +40,6 @@
from apache_beam.io.gcp.datastore.v1.datastoreio import WriteToDatastore
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -127,7 +126,6 @@
known_args, pipeline_args = parser.parse_known_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
gcloud_options = pipeline_options.view_as(GoogleCloudOptions)
job_name = gcloud_options.job_name
kind = known_args.kind
diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py b/sdks/python/apache_beam/io/gcp/gcsio.py
index 33a94fc..dfdc29d 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio.py
@@ -115,33 +115,14 @@
class GcsIO(object):
"""Google Cloud Storage I/O client."""
- def __new__(cls, storage_client=None):
- if storage_client:
- # This path is only used for testing.
- return super(GcsIO, cls).__new__(cls)
- else:
- # Create a single storage client for each thread. We would like to avoid
- # creating more than one storage client for each thread, since each
- # initialization requires the relatively expensive step of initializing
- # credentaials.
- local_state = threading.local()
- if getattr(local_state, 'gcsio_instance', None) is None:
- credentials = auth.get_service_credentials()
- storage_client = storage.StorageV1(
- credentials=credentials,
- get_credentials=False,
- http=get_new_http(),
- response_encoding=None if sys.version_info[0] < 3 else 'utf8')
- local_state.gcsio_instance = super(GcsIO, cls).__new__(cls)
- local_state.gcsio_instance.client = storage_client
- return local_state.gcsio_instance
-
def __init__(self, storage_client=None):
- # We must do this check on storage_client because the client attribute may
- # have already been set in __new__ for the singleton case when
- # storage_client is None.
- if storage_client is not None:
- self.client = storage_client
+ if storage_client is None:
+ storage_client = storage.StorageV1(
+ credentials=auth.get_service_credentials(),
+ get_credentials=False,
+ http=get_new_http(),
+ response_encoding=None if sys.version_info[0] < 3 else 'utf8')
+ self.client = storage_client
self._rewrite_cb = None
def _set_rewrite_response_callback(self, callback):
@@ -174,7 +155,7 @@
if mode == 'r' or mode == 'rb':
downloader = GcsDownloader(self.client, filename,
buffer_size=read_buffer_size)
- return io.BufferedReader(DownloaderStream(downloader, mode=mode),
+ return io.BufferedReader(DownloaderStream(downloader, read_buffer_size=read_buffer_size, mode=mode),
buffer_size=read_buffer_size)
elif mode == 'w' or mode == 'wb':
uploader = GcsUploader(self.client, filename, mime_type)
@@ -522,7 +503,8 @@
# Initialize read buffer state.
self._download_stream = io.BytesIO()
self._downloader = transfer.Download(
- self._download_stream, auto_transfer=False, chunksize=self._buffer_size)
+ self._download_stream, auto_transfer=False, chunksize=self._buffer_size,
+ num_retries=20)
self._client.objects.Get(self._get_request, download=self._downloader)
@retry.with_exponential_backoff(
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py b/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
index 26c8655..d4e387b 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
@@ -31,7 +31,7 @@
permissions for the key specified in --kms_key_name.
To run these tests manually:
- ./gradlew :sdks:python:integrationTest \
+ ./gradlew :sdks:python:test-suites:dataflow:integrationTest \
-Dtests=apache_beam.io.gcp.gcsio_integration_test:GcsIOIntegrationTest \
-DkmsKeyName=KMS_KEY_NAME
"""
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_overrides.py b/sdks/python/apache_beam/io/gcp/gcsio_overrides.py
new file mode 100644
index 0000000..a5fc749
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/gcsio_overrides.py
@@ -0,0 +1,51 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import logging
+import math
+import time
+
+from apache_beam.metrics.metric import Metrics
+from apitools.base.py import exceptions
+from apitools.base.py import http_wrapper
+from apitools.base.py import util
+
+
+class GcsIOOverrides(object):
+ """Functions for overriding Google Cloud Storage I/O client."""
+
+ _THROTTLED_SECS = Metrics.counter('StorageV1', "cumulativeThrottlingSeconds")
+
+ @classmethod
+ def retry_func(cls, retry_args):
+ # handling GCS download throttling errors (BEAM-7424)
+ if (isinstance(retry_args.exc, exceptions.BadStatusCodeError) and
+ retry_args.exc.status_code == http_wrapper.TOO_MANY_REQUESTS):
+ logging.debug(
+ 'Caught GCS quota error (%s), retrying.', retry_args.exc.status_code)
+ else:
+ return http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
+
+ http_wrapper.RebuildHttpConnections(retry_args.http)
+ logging.debug('Retrying request to url %s after exception %s',
+ retry_args.http_request.url, retry_args.exc)
+ sleep_seconds = util.CalculateWaitForRetry(
+ retry_args.num_retries, max_wait=retry_args.max_retry_wait)
+ cls._THROTTLED_SECS.inc(math.ceil(sleep_seconds))
+ time.sleep(sleep_seconds)
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_test.py b/sdks/python/apache_beam/io/gcp/gcsio_test.py
index 8aa8cd5..8027980 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_test.py
@@ -288,6 +288,16 @@
self.client = FakeGcsClient()
self.gcs = gcsio.GcsIO(self.client)
+ def test_num_retries(self):
+ # BEAM-7424: update num_retries accordingly if storage_client is
+ # regenerated.
+ self.assertEqual(gcsio.GcsIO().client.num_retries, 20)
+
+ def test_retry_func(self):
+ # BEAM-7667: update retry_func accordingly if storage_client is
+ # regenerated.
+ self.assertIsNotNone(gcsio.GcsIO().client.retry_func)
+
def test_exists(self):
file_name = 'gs://gcsio-test/dummy_file'
file_size = 1234
@@ -305,7 +315,7 @@
self._insert_random_file(self.client, file_name, file_size)
with self.assertRaises(HttpError) as cm:
self.gcs.exists(file_name)
- self.assertEquals(400, cm.exception.status_code)
+ self.assertEqual(400, cm.exception.status_code)
def test_checksum(self):
file_name = 'gs://gcsio-test/dummy_file'
diff --git a/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py b/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py
index cb5a0c9..df890a9 100644
--- a/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py
+++ b/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py
@@ -22,6 +22,7 @@
from apitools.base.py import base_api
+from apache_beam.io.gcp.gcsio_overrides import GcsIOOverrides
from apache_beam.io.gcp.internal.clients.storage import \
storage_v1_messages as messages
@@ -52,10 +53,11 @@
super(StorageV1, self).__init__(
url, credentials=credentials,
get_credentials=get_credentials, http=http, model=model,
- log_request=log_request, log_response=log_response,
+ log_request=log_request, log_response=log_response, num_retries=20,
credentials_args=credentials_args,
default_global_params=default_global_params,
additional_http_headers=additional_http_headers,
+ retry_func=GcsIOOverrides.retry_func,
response_encoding=response_encoding)
self.bucketAccessControls = self.BucketAccessControlsService(self)
self.buckets = self.BucketsService(self)
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py b/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py
index c8a743e..2c43786 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py
@@ -63,6 +63,10 @@
PubsubMessage(b'data002', {
TIMESTAMP_ATTRIBUTE: '2018-07-11T02:02:50.149000Z',
}),
+ PubsubMessage(b'data003\xab\xac', {}),
+ PubsubMessage(b'data004\xab\xac', {
+ TIMESTAMP_ATTRIBUTE: '2018-07-11T02:02:50.149000Z',
+ })
],
'TestDataflowRunner': [
# Use ID_LABEL attribute to deduplicate messages with the same ID.
@@ -74,6 +78,12 @@
# by Beam), as a TIMESTAMP_ATTRIBUTE + '_out' attribute.
PubsubMessage(b'data002', {
TIMESTAMP_ATTRIBUTE: '2018-07-11T02:02:50.149000Z',
+ }),
+ PubsubMessage(b'data003\xab\xac', {ID_LABEL: 'foo2'}),
+ PubsubMessage(b'data003\xab\xac', {ID_LABEL: 'foo2'}),
+ PubsubMessage(b'data003\xab\xac', {ID_LABEL: 'foo2'}),
+ PubsubMessage(b'data004\xab\xac', {
+ TIMESTAMP_ATTRIBUTE: '2018-07-11T02:02:50.149000Z',
})
],
}
@@ -85,6 +95,12 @@
TIMESTAMP_ATTRIBUTE + '_out': '2018-07-11T02:02:50.149000Z',
'processed': 'IT',
}),
+ PubsubMessage(b'data003\xab\xac-seen', {'processed': 'IT'}),
+ PubsubMessage(b'data004\xab\xac-seen', {
+ TIMESTAMP_ATTRIBUTE: '2018-07-11T02:02:50.149000Z',
+ TIMESTAMP_ATTRIBUTE + '_out': '2018-07-11T02:02:50.149000Z',
+ 'processed': 'IT',
+ })
],
'TestDataflowRunner': [
PubsubMessage(b'data001-seen', {'processed': 'IT'}),
@@ -92,6 +108,11 @@
TIMESTAMP_ATTRIBUTE + '_out': '2018-07-11T02:02:50.149000Z',
'processed': 'IT',
}),
+ PubsubMessage(b'data003\xab\xac-seen', {'processed': 'IT'}),
+ PubsubMessage(b'data004\xab\xac-seen', {
+ TIMESTAMP_ATTRIBUTE + '_out': '2018-07-11T02:02:50.149000Z',
+ 'processed': 'IT',
+ })
],
}
@@ -139,8 +160,7 @@
state_verifier = PipelineStateMatcher(PipelineState.RUNNING)
expected_messages = self.EXPECTED_OUTPUT_MESSAGES[self.runner_name]
if not with_attributes:
- expected_messages = [pubsub_msg.data.decode('utf-8')
- for pubsub_msg in expected_messages]
+ expected_messages = [pubsub_msg.data for pubsub_msg in expected_messages]
if self.runner_name == 'TestDirectRunner':
strip_attributes = None
else:
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py b/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
index 6862bf8..8a8c8b4 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py
@@ -24,7 +24,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import StandardOptions
@@ -42,10 +41,7 @@
'"projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."'))
known_args, pipeline_args = parser.parse_known_args(argv)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
pipeline_options.view_as(StandardOptions).streaming = True
p = beam.Pipeline(options=pipeline_options)
runner_name = type(p.runner).__name__
diff --git a/sdks/python/apache_beam/io/gcp/pubsub_test.py b/sdks/python/apache_beam/io/gcp/pubsub_test.py
index bcc0158..b1dac60 100644
--- a/sdks/python/apache_beam/io/gcp/pubsub_test.py
+++ b/sdks/python/apache_beam/io/gcp/pubsub_test.py
@@ -21,8 +21,6 @@
from __future__ import absolute_import
import logging
-import os
-import sys
import unittest
from builtins import object
@@ -111,10 +109,6 @@
@unittest.skipIf(pubsub is None, 'GCP dependencies are not installed')
class TestReadFromPubSubOverride(unittest.TestCase):
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_expand_with_topic(self):
options = PipelineOptions([])
options.view_as(StandardOptions).streaming = True
@@ -139,10 +133,6 @@
self.assertEqual('a_topic', source.topic_name)
self.assertEqual('a_label', source.id_label)
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_expand_with_subscription(self):
options = PipelineOptions([])
options.view_as(StandardOptions).streaming = True
@@ -179,10 +169,6 @@
ReadFromPubSub('a_topic', 'a_subscription', 'a_label',
with_attributes=False, timestamp_attribute=None)
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_expand_with_other_options(self):
options = PipelineOptions([])
options.view_as(StandardOptions).streaming = True
@@ -373,6 +359,9 @@
mock_pubsub.return_value.acknowledge.assert_has_calls([
mock.call(mock.ANY, [ack_id])])
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_strings_success(self, mock_pubsub):
data = u'🤷 ¯\\_(ツ)_/¯'
data_encoded = data.encode('utf-8')
@@ -394,6 +383,9 @@
mock_pubsub.return_value.acknowledge.assert_has_calls([
mock.call(mock.ANY, [ack_id])])
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_data_success(self, mock_pubsub):
data_encoded = u'🤷 ¯\\_(ツ)_/¯'.encode('utf-8')
ack_id = 'ack_id'
@@ -412,6 +404,9 @@
mock_pubsub.return_value.acknowledge.assert_has_calls([
mock.call(mock.ANY, [ack_id])])
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_messages_timestamp_attribute_milli_success(self, mock_pubsub):
data = b'data'
attributes = {'time': '1337'}
@@ -442,6 +437,9 @@
mock_pubsub.return_value.acknowledge.assert_has_calls([
mock.call(mock.ANY, [ack_id])])
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_messages_timestamp_attribute_rfc3339_success(self, mock_pubsub):
data = b'data'
attributes = {'time': '2018-03-12T13:37:01.234567Z'}
@@ -472,6 +470,9 @@
mock_pubsub.return_value.acknowledge.assert_has_calls([
mock.call(mock.ANY, [ack_id])])
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_messages_timestamp_attribute_missing(self, mock_pubsub):
data = b'data'
attributes = {}
@@ -503,6 +504,9 @@
mock_pubsub.return_value.acknowledge.assert_has_calls([
mock.call(mock.ANY, [ack_id])])
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_messages_timestamp_attribute_fail_parse(self, mock_pubsub):
data = b'data'
attributes = {'time': '1337 unparseable'}
@@ -526,6 +530,9 @@
p.run()
mock_pubsub.return_value.acknowledge.assert_not_called()
+ mock_pubsub.return_value.api.transport.channel.close.assert_has_calls([
+ mock.call()])
+
def test_read_message_id_label_unsupported(self, unused_mock_pubsub):
# id_label is unsupported in DirectRunner.
options = PipelineOptions([])
diff --git a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
index a7c6230..c3394a1 100644
--- a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
+++ b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
@@ -20,11 +20,15 @@
from __future__ import absolute_import
import logging
+import sys
+import time
from hamcrest.core.base_matcher import BaseMatcher
from apache_beam.io.gcp import bigquery_tools
from apache_beam.testing.test_utils import compute_hash
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.testing.util import equal_to
from apache_beam.utils import retry
__all__ = ['BigqueryMatcher', 'BigQueryTableMatcher']
@@ -39,7 +43,7 @@
bigquery = None
# pylint: enable=wrong-import-order, wrong-import-position
-MAX_RETRIES = 4
+MAX_RETRIES = 5
def retry_on_http_and_value_error(exception):
@@ -74,29 +78,29 @@
self.project = project
self.query = query
self.expected_checksum = checksum
+ self.checksum = None
def _matches(self, _):
- logging.info('Start verify Bigquery data.')
- # Run query
- bigquery_client = bigquery.Client(project=self.project)
- response = self._query_with_retry(bigquery_client)
- logging.info('Read from given query (%s), total rows %d',
- self.query, len(response))
+ if self.checksum is None:
+ response = self._query_with_retry()
+ logging.info('Read from given query (%s), total rows %d',
+ self.query, len(response))
+ self.checksum = compute_hash(response)
+ logging.info('Generate checksum: %s', self.checksum)
- # Compute checksum
- self.checksum = compute_hash(response)
- logging.info('Generate checksum: %s', self.checksum)
-
- # Verify result
return self.checksum == self.expected_checksum
@retry.with_exponential_backoff(
num_retries=MAX_RETRIES,
retry_filter=retry_on_http_and_value_error)
- def _query_with_retry(self, bigquery_client):
+ def _query_with_retry(self):
"""Run Bigquery query with retry if got error http response"""
+ logging.info('Attempting to perform query %s to BQ', self.query)
+ # Create client here since it throws an exception if pickled.
+ bigquery_client = bigquery.Client(self.project)
query_job = bigquery_client.query(self.query)
- return [row.values() for row in query_job]
+ rows = query_job.result(timeout=60)
+ return [row.values() for row in rows]
def describe_to(self, description):
description \
@@ -109,7 +113,7 @@
.append_text(self.checksum)
-class BigqueryFullResultMatcher(BaseMatcher):
+class BigqueryFullResultMatcher(BigqueryMatcher):
"""Matcher that verifies Bigquery data with given query.
Fetch Bigquery data with given query, compare to the expected data.
@@ -122,37 +126,24 @@
query: The query (string) to perform.
data: List of tuples with the expected data.
"""
- if bigquery is None:
- raise ImportError(
- 'Bigquery dependencies are not installed.')
- if not query or not isinstance(query, str):
- raise ValueError(
- 'Invalid argument: query. Please use non-empty string')
-
- self.project = project
- self.query = query
- self.expected_data = [sorted(i) for i in data]
+ super(BigqueryFullResultMatcher, self).__init__(project, query,
+ 'unused_checksum')
+ self.expected_data = data
+ self.actual_data = None
def _matches(self, _):
- logging.info('Start verify Bigquery data.')
- # Run query
- bigquery_client = bigquery.Client(project=self.project)
- response = self._query_with_retry(bigquery_client)
- logging.info('Read from given query (%s), total rows %d',
- self.query, len(response))
+ if self.actual_data is None:
+ self.actual_data = self._get_query_result()
+ logging.info('Result of query is: %r', self.actual_data)
- self.actual_data = [sorted(i) for i in response]
+ try:
+ equal_to(self.expected_data)(self.actual_data)
+ return True
+ except BeamAssertException:
+ return False
- # Verify result
- return sorted(self.expected_data) == sorted(self.actual_data)
-
- @retry.with_exponential_backoff(
- num_retries=MAX_RETRIES,
- retry_filter=retry_on_http_and_value_error)
- def _query_with_retry(self, bigquery_client):
- """Run Bigquery query with retry if got error http response"""
- query_job = bigquery_client.query(self.query)
- return [row.values() for row in query_job]
+ def _get_query_result(self):
+ return self._query_with_retry()
def describe_to(self, description):
description \
@@ -165,6 +156,37 @@
.append_text(self.actual_data)
+class BigqueryFullResultStreamingMatcher(BigqueryFullResultMatcher):
+ """
+ Matcher that verifies Bigquery data with given query.
+
+ Fetch Bigquery data with given query, compare to the expected data.
+ This matcher polls BigQuery until the no. of records in BigQuery is
+ equal to the no. of records in expected data.
+ A timeout can be specified.
+ """
+
+ DEFAULT_TIMEOUT = 5*60
+
+ def __init__(self, project, query, data, timeout=DEFAULT_TIMEOUT):
+ super(BigqueryFullResultStreamingMatcher, self).__init__(
+ project, query, data)
+ self.timeout = timeout
+
+ def _get_query_result(self):
+ start_time = time.time()
+ while time.time() - start_time <= self.timeout:
+ response = self._query_with_retry()
+ if len(response) >= len(self.expected_data):
+ return response
+ logging.debug('Query result contains %d rows' % len(response))
+ time.sleep(1)
+ if sys.version_info >= (3,):
+ raise TimeoutError('Timeout exceeded for matcher.') # noqa: F821
+ else:
+ raise RuntimeError('Timeout exceeded for matcher.')
+
+
class BigQueryTableMatcher(BaseMatcher):
"""Matcher that verifies the properties of a Table in BigQuery."""
diff --git a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
index c8315db..005347b 100644
--- a/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/bigquery_matcher_test.py
@@ -20,6 +20,7 @@
from __future__ import absolute_import
import logging
+import sys
import unittest
import mock
@@ -32,7 +33,6 @@
# Protect against environments where bigquery library is not available.
# pylint: disable=wrong-import-order, wrong-import-position
try:
- # TODO: fix usage
from google.cloud import bigquery
from google.cloud.exceptions import NotFound
except ImportError:
@@ -55,7 +55,8 @@
mock_query_result[1].values.return_value = None
mock_query_result[2].values.return_value = None
- mock_bigquery.return_value.query.return_value = mock_query_result
+ mock_bigquery.return_value.query.return_value.result.return_value = (
+ mock_query_result)
matcher = bq_verifier.BigqueryMatcher(
'mock_project',
@@ -113,6 +114,27 @@
self.assertEqual(bq_verifier.MAX_RETRIES + 1, mock_query.call_count)
+@unittest.skipIf(bigquery is None, 'Bigquery dependencies are not installed.')
+@mock.patch.object(
+ bq_verifier.BigqueryFullResultStreamingMatcher,
+ '_query_with_retry')
+class BigqueryFullResultStreamingMatcher(unittest.TestCase):
+
+ def setUp(self):
+ self.timeout = 0.01
+
+ def test__get_query_result_timeout(self, mock__query_with_retry):
+ mock__query_with_retry.side_effect = lambda: []
+ matcher = bq_verifier.BigqueryFullResultStreamingMatcher(
+ 'some-project', 'some-query', [1, 2, 3], timeout=self.timeout)
+ if sys.version_info >= (3,):
+ with self.assertRaises(TimeoutError): # noqa: F821
+ matcher._get_query_result()
+ else:
+ with self.assertRaises(RuntimeError):
+ matcher._get_query_result()
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
index ba7a674..7a0b5c8 100644
--- a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
+++ b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py
@@ -103,7 +103,7 @@
for rm in response.received_messages:
msg = PubsubMessage._from_message(rm.message)
if not self.with_attributes:
- total_messages.append(msg.data.decode('utf-8'))
+ total_messages.append(msg.data)
continue
if self.strip_attributes:
diff --git a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py
index 6a58ddf..cb9fbb9 100644
--- a/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher_test.py
@@ -59,7 +59,7 @@
def test_message_matcher_success(self, mock_get_sub, unsued_mock):
self.init_matcher()
- self.pubsub_matcher.expected_msg = ['a', 'b']
+ self.pubsub_matcher.expected_msg = [b'a', b'b']
mock_sub = mock_get_sub.return_value
mock_sub.pull.side_effect = [
create_pull_response([PullResponseMessage(b'a', {})]),
@@ -121,7 +121,7 @@
def test_message_matcher_mismatch(self, mock_get_sub, unused_mock):
self.init_matcher()
- self.pubsub_matcher.expected_msg = ['a']
+ self.pubsub_matcher.expected_msg = [b'a']
mock_sub = mock_get_sub.return_value
mock_sub.pull.side_effect = [
create_pull_response([PullResponseMessage(b'c', {}),
@@ -130,7 +130,7 @@
with self.assertRaises(AssertionError) as error:
hc_assert_that(self.mock_presult, self.pubsub_matcher)
self.assertEqual(mock_sub.pull.call_count, 1)
- self.assertCountEqual(['c', 'd'], self.pubsub_matcher.messages)
+ self.assertCountEqual([b'c', b'd'], self.pubsub_matcher.messages)
self.assertTrue(
'\nExpected: Expected 1 messages.\n but: Got 2 messages.'
in str(error.exception.args[0]))
diff --git a/sdks/python/apache_beam/io/gcp/tests/utils.py b/sdks/python/apache_beam/io/gcp/tests/utils.py
index e72d917..4ed9af3 100644
--- a/sdks/python/apache_beam/io/gcp/tests/utils.py
+++ b/sdks/python/apache_beam/io/gcp/tests/utils.py
@@ -21,18 +21,20 @@
from __future__ import absolute_import
import logging
+import random
import time
from apache_beam.io import filesystems
+from apache_beam.io.gcp.pubsub import PubsubMessage
from apache_beam.utils import retry
# Protect against environments where bigquery library is not available.
try:
+ from google.api_core import exceptions as gexc
from google.cloud import bigquery
- from google.cloud.exceptions import NotFound
except ImportError:
+ gexc = None
bigquery = None
- NotFound = None
class GcpTestIOError(retry.PermanentException):
@@ -56,7 +58,9 @@
new dataset.
"""
client = bigquery.Client(project=project)
- unique_dataset_name = dataset_base_name + str(int(time.time()))
+ unique_dataset_name = '%s%s%d' % (dataset_base_name,
+ str(int(time.time())),
+ random.randint(0, 10000))
dataset_ref = client.dataset(unique_dataset_name, project=project)
dataset = bigquery.Dataset(dataset_ref)
client.create_dataset(dataset)
@@ -95,7 +99,7 @@
table_ref = client.dataset(dataset_id).table(table_id)
try:
client.delete_table(table_ref)
- except NotFound:
+ except gexc.NotFound:
raise GcpTestIOError('BigQuery table does not exist: %s' % table_ref)
@@ -110,3 +114,53 @@
"gs://mybucket/mydir/", "s3://...", ...)
"""
filesystems.FileSystems.delete([directory])
+
+
+def write_to_pubsub(pub_client,
+ topic_path,
+ messages,
+ with_attributes=False,
+ chunk_size=100,
+ delay_between_chunks=0.1):
+ for start in range(0, len(messages), chunk_size):
+ message_chunk = messages[start:start + chunk_size]
+ if with_attributes:
+ futures = [
+ pub_client.publish(topic_path, message.data, **message.attributes)
+ for message in message_chunk
+ ]
+ else:
+ futures = [
+ pub_client.publish(topic_path, message) for message in message_chunk
+ ]
+ for future in futures:
+ future.result()
+ time.sleep(delay_between_chunks)
+
+
+def read_from_pubsub(sub_client,
+ subscription_path,
+ with_attributes=False,
+ number_of_elements=None,
+ timeout=None):
+ if number_of_elements is None and timeout is None:
+ raise ValueError("Either number_of_elements or timeout must be specified.")
+ messages = []
+ start_time = time.time()
+
+ while ((number_of_elements is None or len(messages) < number_of_elements) and
+ (timeout is None or (time.time() - start_time) < timeout)):
+ try:
+ response = sub_client.pull(
+ subscription_path, max_messages=1000, retry=None, timeout=10)
+ except (gexc.RetryError, gexc.DeadlineExceeded):
+ continue
+ ack_ids = [msg.ack_id for msg in response.received_messages]
+ sub_client.acknowledge(subscription_path, ack_ids)
+ for msg in response.received_messages:
+ message = PubsubMessage._from_message(msg.message)
+ if with_attributes:
+ messages.append(message)
+ else:
+ messages.append(message.data)
+ return messages
diff --git a/sdks/python/apache_beam/io/gcp/tests/utils_test.py b/sdks/python/apache_beam/io/gcp/tests/utils_test.py
index 8af7497..c9e96d1 100644
--- a/sdks/python/apache_beam/io/gcp/tests/utils_test.py
+++ b/sdks/python/apache_beam/io/gcp/tests/utils_test.py
@@ -24,16 +24,19 @@
import mock
+from apache_beam.io.gcp.pubsub import PubsubMessage
from apache_beam.io.gcp.tests import utils
-from apache_beam.testing.test_utils import patch_retry
+from apache_beam.testing import test_utils
# Protect against environments where bigquery library is not available.
try:
+ from google.api_core import exceptions as gexc
from google.cloud import bigquery
- from google.cloud.exceptions import NotFound
+ from google.cloud import pubsub
except ImportError:
+ gexc = None
bigquery = None
- NotFound = None
+ pubsub = None
@unittest.skipIf(bigquery is None, 'Bigquery dependencies are not installed.')
@@ -41,7 +44,7 @@
class UtilsTest(unittest.TestCase):
def setUp(self):
- patch_retry(self, utils)
+ test_utils.patch_retry(self, utils)
@mock.patch.object(bigquery, 'Dataset')
def test_create_bq_dataset(self, mock_dataset, mock_client):
@@ -68,7 +71,7 @@
def test_delete_table_fails_not_found(self, mock_client):
mock_client.return_value.dataset.return_value.table.return_value = (
'table_ref')
- mock_client.return_value.delete_table.side_effect = NotFound('test')
+ mock_client.return_value.delete_table.side_effect = gexc.NotFound('test')
with self.assertRaisesRegexp(Exception, r'does not exist:.*table_ref'):
utils.delete_bq_table('unused_project',
@@ -76,6 +79,193 @@
'unused_table')
+@unittest.skipIf(pubsub is None, 'GCP dependencies are not installed')
+class PubSubUtilTest(unittest.TestCase):
+
+ def test_write_to_pubsub(self):
+ mock_pubsub = mock.Mock()
+ topic_path = "project/fakeproj/topics/faketopic"
+ data = b'data'
+ utils.write_to_pubsub(mock_pubsub, topic_path, [data])
+ mock_pubsub.publish.assert_has_calls(
+ [mock.call(topic_path, data),
+ mock.call().result()])
+
+ def test_write_to_pubsub_with_attributes(self):
+ mock_pubsub = mock.Mock()
+ topic_path = "project/fakeproj/topics/faketopic"
+ data = b'data'
+ attributes = {'key': 'value'}
+ message = PubsubMessage(data, attributes)
+ utils.write_to_pubsub(
+ mock_pubsub, topic_path, [message], with_attributes=True)
+ mock_pubsub.publish.assert_has_calls(
+ [mock.call(topic_path, data, **attributes),
+ mock.call().result()])
+
+ def test_write_to_pubsub_delay(self):
+ number_of_elements = 2
+ chunk_size = 1
+ mock_pubsub = mock.Mock()
+ topic_path = "project/fakeproj/topics/faketopic"
+ data = b'data'
+ with mock.patch('apache_beam.io.gcp.tests.utils.time') as mock_time:
+ utils.write_to_pubsub(
+ mock_pubsub,
+ topic_path, [data] * number_of_elements,
+ chunk_size=chunk_size,
+ delay_between_chunks=123)
+ mock_time.sleep.assert_called_with(123)
+ mock_pubsub.publish.assert_has_calls(
+ [mock.call(topic_path, data),
+ mock.call().result()] * number_of_elements)
+
+ def test_write_to_pubsub_many_chunks(self):
+ number_of_elements = 83
+ chunk_size = 11
+ mock_pubsub = mock.Mock()
+ topic_path = "project/fakeproj/topics/faketopic"
+ data_list = [
+ 'data {}'.format(i).encode("utf-8") for i in range(number_of_elements)
+ ]
+ utils.write_to_pubsub(
+ mock_pubsub, topic_path, data_list, chunk_size=chunk_size)
+ call_list = []
+ for start in range(0, number_of_elements, chunk_size):
+ # Publish a batch of messages
+ call_list += [
+ mock.call(topic_path, data)
+ for data in data_list[start:start + chunk_size]
+ ]
+ # Wait for those messages to be received
+ call_list += [
+ mock.call().result() for _ in data_list[start:start + chunk_size]
+ ]
+ mock_pubsub.publish.assert_has_calls(call_list)
+
+ def test_read_from_pubsub(self):
+ mock_pubsub = mock.Mock()
+ subscription_path = "project/fakeproj/subscriptions/fakesub"
+ data = b'data'
+ ack_id = 'ack_id'
+ pull_response = test_utils.create_pull_response(
+ [test_utils.PullResponseMessage(data, ack_id=ack_id)])
+ mock_pubsub.pull.return_value = pull_response
+ output = utils.read_from_pubsub(
+ mock_pubsub, subscription_path, number_of_elements=1)
+ self.assertEqual([data], output)
+ mock_pubsub.acknowledge.assert_called_once_with(subscription_path, [ack_id])
+
+ def test_read_from_pubsub_with_attributes(self):
+ mock_pubsub = mock.Mock()
+ subscription_path = "project/fakeproj/subscriptions/fakesub"
+ data = b'data'
+ ack_id = 'ack_id'
+ attributes = {'key': 'value'}
+ message = PubsubMessage(data, attributes)
+ pull_response = test_utils.create_pull_response(
+ [test_utils.PullResponseMessage(data, attributes, ack_id=ack_id)])
+ mock_pubsub.pull.return_value = pull_response
+ output = utils.read_from_pubsub(
+ mock_pubsub,
+ subscription_path,
+ with_attributes=True,
+ number_of_elements=1)
+ self.assertEqual([message], output)
+ mock_pubsub.acknowledge.assert_called_once_with(subscription_path, [ack_id])
+
+ def test_read_from_pubsub_flaky(self):
+ number_of_elements = 10
+ mock_pubsub = mock.Mock()
+ subscription_path = "project/fakeproj/subscriptions/fakesub"
+ data = b'data'
+ ack_id = 'ack_id'
+ pull_response = test_utils.create_pull_response(
+ [test_utils.PullResponseMessage(data, ack_id=ack_id)])
+
+ class FlakyPullResponse(object):
+
+ def __init__(self, pull_response):
+ self.pull_response = pull_response
+ self._state = -1
+
+ def __call__(self, *args, **kwargs):
+ self._state += 1
+ if self._state % 3 == 0:
+ raise gexc.RetryError("", "")
+ if self._state % 3 == 1:
+ raise gexc.DeadlineExceeded("")
+ if self._state % 3 == 2:
+ return self.pull_response
+
+ mock_pubsub.pull.side_effect = FlakyPullResponse(pull_response)
+ output = utils.read_from_pubsub(
+ mock_pubsub, subscription_path, number_of_elements=number_of_elements)
+ self.assertEqual([data] * number_of_elements, output)
+ self._assert_ack_ids_equal(mock_pubsub, [ack_id] * number_of_elements)
+
+ def test_read_from_pubsub_many(self):
+ response_size = 33
+ number_of_elements = 100
+ mock_pubsub = mock.Mock()
+ subscription_path = "project/fakeproj/subscriptions/fakesub"
+ data_list = [
+ 'data {}'.format(i).encode("utf-8") for i in range(number_of_elements)
+ ]
+ attributes_list = [{
+ 'key': 'value {}'.format(i)
+ } for i in range(number_of_elements)]
+ ack_ids = ['ack_id_{}'.format(i) for i in range(number_of_elements)]
+ messages = [
+ PubsubMessage(data, attributes)
+ for data, attributes in zip(data_list, attributes_list)
+ ]
+ response_messages = [
+ test_utils.PullResponseMessage(data, attributes, ack_id=ack_id)
+ for data, attributes, ack_id in zip(data_list, attributes_list, ack_ids)
+ ]
+
+ class SequentialPullResponse(object):
+
+ def __init__(self, response_messages, response_size):
+ self.response_messages = response_messages
+ self.response_size = response_size
+ self._index = 0
+
+ def __call__(self, *args, **kwargs):
+ start = self._index
+ self._index += self.response_size
+ response = test_utils.create_pull_response(
+ self.response_messages[start:start + self.response_size])
+ return response
+
+ mock_pubsub.pull.side_effect = SequentialPullResponse(
+ response_messages, response_size)
+ output = utils.read_from_pubsub(
+ mock_pubsub,
+ subscription_path,
+ with_attributes=True,
+ number_of_elements=number_of_elements)
+ self.assertEqual(messages, output)
+ self._assert_ack_ids_equal(mock_pubsub, ack_ids)
+
+ def test_read_from_pubsub_invalid_arg(self):
+ sub_client = mock.Mock()
+ subscription_path = "project/fakeproj/subscriptions/fakesub"
+ with self.assertRaisesRegexp(ValueError, "number_of_elements"):
+ utils.read_from_pubsub(sub_client, subscription_path)
+ with self.assertRaisesRegexp(ValueError, "number_of_elements"):
+ utils.read_from_pubsub(
+ sub_client, subscription_path, with_attributes=True)
+
+ def _assert_ack_ids_equal(self, mock_pubsub, ack_ids):
+ actual_ack_ids = [
+ ack_id for args_list in mock_pubsub.acknowledge.call_args_list
+ for ack_id in args_list[0][1]
+ ]
+ self.assertEqual(actual_ack_ids, ack_ids)
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/io/hadoopfilesystem_test.py b/sdks/python/apache_beam/io/hadoopfilesystem_test.py
index 2db3f25..06d9fa9 100644
--- a/sdks/python/apache_beam/io/hadoopfilesystem_test.py
+++ b/sdks/python/apache_beam/io/hadoopfilesystem_test.py
@@ -276,12 +276,12 @@
for filename in ['old_file1', 'old_file2']]
result = self.fs.match([self.tmpdir + '/'], [1])[0]
files = [f.path for f in result.metadata_list]
- self.assertEquals(len(files), 1)
+ self.assertEqual(len(files), 1)
self.assertIn(files[0], expected_files)
def test_match_file_with_zero_limit(self):
result = self.fs.match([self.tmpdir + '/'], [0])[0]
- self.assertEquals(len(result.metadata_list), 0)
+ self.assertEqual(len(result.metadata_list), 0)
def test_match_file_empty(self):
url = self.fs.join(self.tmpdir, 'nonexistent_file')
diff --git a/sdks/python/apache_beam/io/hdfs_integration_test/Dockerfile b/sdks/python/apache_beam/io/hdfs_integration_test/Dockerfile
index 6a4f307..788b8d2 100644
--- a/sdks/python/apache_beam/io/hdfs_integration_test/Dockerfile
+++ b/sdks/python/apache_beam/io/hdfs_integration_test/Dockerfile
@@ -19,8 +19,9 @@
# This image contains a Python SDK build and dependencies.
# By default it runs wordcount against a locally accessible HDFS service.
# See hdfs_integration_test.sh for example usage.
+ARG BASE_IMAGE
+FROM $BASE_IMAGE
-FROM python:2
WORKDIR /app
ENV HDFSCLI_CONFIG /app/sdks/python/apache_beam/io/hdfs_integration_test/hdfscli.cfg
RUN pip install --no-cache-dir holdup gsutil
diff --git a/sdks/python/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh b/sdks/python/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
index a2bf5f0..f9ee722 100755
--- a/sdks/python/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
+++ b/sdks/python/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
@@ -19,6 +19,14 @@
#
# Requires docker, docker-compose to be installed.
+# Usage check.
+if [[ $# != 1 ]]; then
+ printf "Usage: \n$> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh <python_version>"
+ printf "\n\tpython_version: [required] Python version used for container build and run tests."
+ printf " Use 'python:2' for Python2, 'python:3.7' for Python3.7."
+ exit 1
+fi
+
set -e -u -x
# Setup context directory.
@@ -53,6 +61,6 @@
}
trap finally EXIT
-time docker-compose ${COMPOSE_OPT} build
+time docker-compose ${COMPOSE_OPT} build --build-arg BASE_IMAGE=$1
time docker-compose ${COMPOSE_OPT} up --exit-code-from test \
--abort-on-container-exit --force-recreate
diff --git a/sdks/python/apache_beam/io/iobase.py b/sdks/python/apache_beam/io/iobase.py
index 1386d20..605c1bf 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -436,7 +436,7 @@
Returns:
the approximate fraction of positions that have been consumed by
- successful 'try_split()' and 'report_current_position()' calls, or
+ successful 'try_split()' and 'try_claim()' calls, or
0.0 if no such calls have happened.
"""
raise NotImplementedError
@@ -845,6 +845,16 @@
super(Read, self).__init__()
self.source = source
+ @staticmethod
+ def get_desired_chunk_size(total_size):
+ total_size
+ if total_size:
+ # 1MB = 1 shard, 1GB = 32 shards, 1TB = 1000 shards, 1PB = 32k shards
+ chunk_size = max(1 << 20, 1000 * int(math.sqrt(total_size)))
+ else:
+ chunk_size = 64 << 20 # 64mb
+ return chunk_size
+
def expand(self, pbegin):
from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.transforms import util
@@ -857,13 +867,8 @@
source = self.source
def split_source(unused_impulse):
- total_size = source.estimate_size()
- if total_size:
- # 1MB = 1 shard, 1GB = 32 shards, 1TB = 1000 shards, 1PB = 32k shards
- chunk_size = max(1 << 20, 1000 * int(math.sqrt(total_size)))
- else:
- chunk_size = 64 << 20 # 64mb
- return source.split(chunk_size)
+ return source.split(
+ self.get_desired_chunk_size(self.source.estimate_size()))
return (
pbegin
@@ -875,7 +880,8 @@
split.start_position, split.stop_position))))
else:
# Treat Read itself as a primitive.
- return pvalue.PCollection(self.pipeline)
+ return pvalue.PCollection(self.pipeline,
+ is_bounded=self.source.is_bounded())
def get_windowing(self, unused_inputs):
return core.Windowing(window.GlobalWindows())
@@ -1122,6 +1128,8 @@
Methods of the class ``RestrictionTracker`` including this method may get
invoked by different threads, hence must be made thread-safe, e.g. by using
a single lock object.
+
+ TODO(BEAM-7473): Remove thread safety requirements from API implementation.
"""
raise NotImplementedError
@@ -1130,6 +1138,13 @@
"""
raise NotImplementedError
+ def current_watermark(self):
+ """Returns current watermark. By default, not report watermark.
+
+ TODO(BEAM-7473): Provide synchronization guarantee by using a wrapper.
+ """
+ return None
+
def checkpoint(self):
"""Performs a checkpoint of the current restriction.
@@ -1148,6 +1163,8 @@
Methods of the class ``RestrictionTracker`` including this method may get
invoked by different threads, hence must be made thread-safe, e.g. by using
a single lock object.
+
+ TODO(BEAM-7473): Remove thread safety requirements from API implementation.
"""
raise NotImplementedError
@@ -1168,12 +1185,103 @@
invoked by different threads, hence must be made thread-safe, e.g. by using
a single lock object.
+ TODO(BEAM-7473): Remove thread safety requirements from API implementation.
+
Returns: ``True`` if current restriction has been fully processed.
Raises:
~exceptions.ValueError: if there is still any unclaimed work remaining.
"""
raise NotImplementedError
+ def try_split(self, fraction_of_remainder):
+ """Splits current restriction based on fraction_of_remainder.
+
+ If splitting the current restriction is possible, the current restriction is
+ split into a primary and residual restriction pair. This invocation updates
+ the ``current_restriction()`` to be the primary restriction effectively
+ having the current ``DoFn.process()`` execution responsible for performing
+ the work that the primary restriction represents. The residual restriction
+ will be executed in a separate ``DoFn.process()`` invocation (likely in a
+ different process). The work performed by executing the primary and residual
+ restrictions as separate ``DoFn.process()`` invocations MUST be equivalent
+ to the work performed as if this split never occurred.
+
+ The ``fraction_of_remainder`` should be used in a best effort manner to
+ choose a primary and residual restriction based upon the fraction of the
+ remaining work that the current ``DoFn.process()`` invocation is responsible
+ for. For example, if a ``DoFn.process()`` was reading a file with a
+ restriction representing the offset range [100, 200) and has processed up to
+ offset 130 with a fraction_of_remainder of 0.7, the primary and residual
+ restrictions returned would be [100, 179), [179, 200) (note: current_offset
+ + fraction_of_remainder * remaining_work = 130 + 0.7 * 70 = 179).
+
+ It is very important for pipeline scaling and end to end pipeline execution
+ that try_split is implemented well.
+
+ Args:
+ fraction_of_remainder: A hint as to the fraction of work the primary
+ restriction should represent based upon the current known remaining
+ amount of work.
+
+ Returns:
+ (primary_restriction, residual_restriction) if a split was possible,
+ otherwise returns ``None``.
+
+ ** Thread safety **
+
+ Methods of the class ``RestrictionTracker`` including this method may get
+ invoked by different threads, hence must be made thread-safe, e.g. by using
+ a single lock object.
+
+ TODO(BEAM-7473): Remove thread safety requirements from API implementation.
+ """
+ raise NotImplementedError
+
+ def try_claim(self, position):
+ """ Attempts to claim the block of work in the current restriction
+ identified by the given position.
+
+ If this succeeds, the DoFn MUST execute the entire block of work. If it
+ fails, the ``DoFn.process()`` MUST return ``None`` without performing any
+ additional work or emitting output (note that emitting output or performing
+ work from ``DoFn.process()`` is also not allowed before the first call of
+ this method).
+
+ Args:
+ position: current position that wants to be claimed.
+
+ Returns: ``True`` if the position can be claimed as current_position.
+ Otherwise, returns ``False``.
+
+ ** Thread safety **
+
+ Methods of the class ``RestrictionTracker`` including this method may get
+ invoked by different threads, hence must be made thread-safe, e.g. by using
+ a single lock object.
+
+ TODO(BEAM-7473): Remove thread safety requirements from API implementation.
+ """
+ raise NotImplementedError
+
+ def defer_remainder(self, watermark=None):
+ """ Invokes checkpoint() in an SDF.process().
+
+ TODO(BEAM-7472): Remove defer_remainder() once SDF.process() uses
+ ``ProcessContinuation``.
+
+ Args:
+ watermark
+ """
+ raise NotImplementedError
+
+ def deferred_status(self):
+ """ Returns deferred_residual with deferred_watermark.
+
+ TODO(BEAM-7472): Remove defer_status() once SDF.process() uses
+ ``ProcessContinuation``.
+ """
+ raise NotImplementedError
+
class RestrictionProgress(object):
"""Used to record the progress of a restriction.
@@ -1226,3 +1334,149 @@
def with_completed(self, completed):
return RestrictionProgress(
fraction=self._fraction, remaining=self._remaining, completed=completed)
+
+
+class _SDFBoundedSourceWrapper(ptransform.PTransform):
+ """A ``PTransform`` that uses SDF to read from a ``BoundedSource``.
+
+ NOTE: This transform can only be used with beam_fn_api enabled.
+ """
+ class _SDFBoundedSourceRestrictionTracker(RestrictionTracker):
+ """An `iobase.RestrictionTracker` implementations for wrapping BoundedSource
+ with SDF.
+
+ Delegated RangeTracker guarantees synchronization safety.
+ """
+ def __init__(self, restriction):
+ if not isinstance(restriction, SourceBundle):
+ raise ValueError('Initializing SDFBoundedSourceRestrictionTracker'
+ 'requires a SourceBundle')
+ self._delegate_range_tracker = restriction.source.get_range_tracker(
+ restriction.start_position, restriction.stop_position)
+ self._source = restriction.source
+ self._weight = restriction.weight
+
+ def current_progress(self):
+ return RestrictionProgress(
+ fraction=self._delegate_range_tracker.fraction_consumed())
+
+ def current_restriction(self):
+ start_pos = self._delegate_range_tracker.start_position()
+ stop_pos = self._delegate_range_tracker.stop_position()
+ return SourceBundle(
+ self._weight,
+ self._source,
+ start_pos,
+ stop_pos)
+
+ def start_pos(self):
+ return self._delegate_range_tracker.start_position()
+
+ def stop_pos(self):
+ return self._delegate_range_tracker.stop_position()
+
+ def try_claim(self, position):
+ return self._delegate_range_tracker.try_claim(position)
+
+ def try_split(self, fraction_of_remainder):
+ consumed_fraction = self._delegate_range_tracker.fraction_consumed()
+ fraction = (consumed_fraction +
+ (1 - consumed_fraction) * fraction_of_remainder)
+ position = self._delegate_range_tracker.position_at_fraction(fraction)
+ # Need to stash current stop_pos before splitting since
+ # range_tracker.split will update its stop_pos if splits
+ # successfully.
+ start_pos = self.start_pos()
+ stop_pos = self.stop_pos()
+ split_result = self._delegate_range_tracker.try_split(position)
+ if split_result:
+ split_pos, split_fraction = split_result
+ primary_weight = self._weight * split_fraction
+ residual_weight = self._weight - primary_weight
+ # Update self._weight to primary weight
+ self._weight = primary_weight
+ return (SourceBundle(primary_weight, self._source, start_pos,
+ split_pos),
+ SourceBundle(residual_weight, self._source, split_pos,
+ stop_pos))
+
+ def deferred_status(self):
+ return None
+
+ def current_watermark(self):
+ return None
+
+ def get_delegate_range_tracker(self):
+ return self._delegate_range_tracker
+
+ def get_tracking_source(self):
+ return self._source
+
+ class _SDFBoundedSourceRestrictionProvider(core.RestrictionProvider):
+ """A `RestrictionProvider` that is used by SDF for `BoundedSource`."""
+
+ def __init__(self, source, desired_chunk_size=None):
+ self._source = source
+ self._desired_chunk_size = desired_chunk_size
+
+ def initial_restriction(self, element):
+ # Get initial range_tracker from source
+ range_tracker = self._source.get_range_tracker(None, None)
+ return SourceBundle(None,
+ self._source,
+ range_tracker.start_position(),
+ range_tracker.stop_position())
+
+ def create_tracker(self, restriction):
+ return _SDFBoundedSourceWrapper._SDFBoundedSourceRestrictionTracker(
+ restriction)
+
+ def split(self, element, restriction):
+ # Invoke source.split to get initial splitting results.
+ source_bundles = self._source.split(self._desired_chunk_size)
+ for source_bundle in source_bundles:
+ yield source_bundle
+
+ def restriction_size(self, element, restriction):
+ return restriction.weight
+
+ def __init__(self, source):
+ if not isinstance(source, BoundedSource):
+ raise RuntimeError('SDFBoundedSourceWrapper can only wrap BoundedSource')
+ super(_SDFBoundedSourceWrapper, self).__init__()
+ self.source = source
+
+ def _create_sdf_bounded_source_dofn(self):
+ source = self.source
+ chunk_size = Read.get_desired_chunk_size(source.estimate_size())
+
+ class SDFBoundedSourceDoFn(core.DoFn):
+ def __init__(self, read_source):
+ self.source = read_source
+
+ def process(
+ self,
+ element,
+ restriction_tracker=core.DoFn.RestrictionParam(
+ _SDFBoundedSourceWrapper._SDFBoundedSourceRestrictionProvider(
+ source, chunk_size))):
+ return restriction_tracker.get_tracking_source().read(
+ restriction_tracker.get_delegate_range_tracker())
+
+ return SDFBoundedSourceDoFn(self.source)
+
+ def expand(self, pbegin):
+ return (pbegin
+ | core.Impulse()
+ | core.ParDo(self._create_sdf_bounded_source_dofn()))
+
+ def get_windowing(self, unused_inputs):
+ return core.Windowing(window.GlobalWindows())
+
+ def _infer_output_coder(self, input_type=None, input_coder=None):
+ return self.source.default_output_coder()
+
+ def display_data(self):
+ return {'source': DisplayDataItem(self.source.__class__,
+ label='Read Source'),
+ 'source_dd': self.source}
diff --git a/sdks/python/apache_beam/io/iobase_test.py b/sdks/python/apache_beam/io/iobase_test.py
new file mode 100644
index 0000000..c7d1656
--- /dev/null
+++ b/sdks/python/apache_beam/io/iobase_test.py
@@ -0,0 +1,158 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for the SDFRestrictionProvider module."""
+
+from __future__ import absolute_import
+
+import unittest
+
+from apache_beam.io.concat_source import ConcatSource
+from apache_beam.io.concat_source_test import RangeSource
+from apache_beam.io import iobase
+from apache_beam.io.iobase import SourceBundle
+
+
+class SDFBoundedSourceRestrictionProviderTest(unittest.TestCase):
+ def setUp(self):
+ self.initial_range_start = 0
+ self.initial_range_stop = 4
+ self.initial_range_source = RangeSource(self.initial_range_start,
+ self.initial_range_stop)
+ self.sdf_restriction_provider = (
+ iobase._SDFBoundedSourceWrapper._SDFBoundedSourceRestrictionProvider(
+ self.initial_range_source,
+ desired_chunk_size=2))
+
+ def test_initial_restriction(self):
+ unused_element = None
+ restriction = (
+ self.sdf_restriction_provider.initial_restriction(unused_element))
+ self.assertTrue(isinstance(restriction, SourceBundle))
+ self.assertEqual(self.initial_range_start, restriction.start_position)
+ self.assertEqual(self.initial_range_stop, restriction.stop_position)
+ self.assertTrue(isinstance(restriction.source, RangeSource))
+
+ def test_create_tracker(self):
+ expected_start = 1
+ expected_stop = 3
+ restriction = SourceBundle(expected_stop - expected_start,
+ RangeSource(1, 3),
+ expected_start,
+ expected_stop)
+ restriction_tracker = (
+ self.sdf_restriction_provider.create_tracker(restriction))
+ self.assertTrue(isinstance(restriction_tracker,
+ iobase.
+ _SDFBoundedSourceWrapper.
+ _SDFBoundedSourceRestrictionTracker))
+ self.assertEqual(expected_start, restriction_tracker.start_pos())
+ self.assertEqual(expected_stop, restriction_tracker.stop_pos())
+
+ def test_simple_source_split(self):
+ unused_element = None
+ restriction = (
+ self.sdf_restriction_provider.initial_restriction(unused_element))
+ expect_splits = [(0, 2), (2, 4)]
+ split_bundles = list(self.sdf_restriction_provider.split(unused_element,
+ restriction))
+ self.assertTrue(
+ all([isinstance(bundle, SourceBundle) for bundle in split_bundles]))
+
+ splits = ([(bundle.start_position,
+ bundle.stop_position) for bundle in split_bundles])
+ self.assertEqual(expect_splits, list(splits))
+
+ def test_concat_source_split(self):
+ unused_element = None
+ initial_concat_source = ConcatSource([self.initial_range_source])
+ sdf_concat_restriction_provider = (
+ iobase._SDFBoundedSourceWrapper._SDFBoundedSourceRestrictionProvider(
+ initial_concat_source,
+ desired_chunk_size=2))
+ restriction = (
+ self.sdf_restriction_provider.initial_restriction(unused_element))
+ expect_splits = [(0, 2), (2, 4)]
+ split_bundles = list(sdf_concat_restriction_provider.split(unused_element,
+ restriction))
+ self.assertTrue(
+ all([isinstance(bundle, SourceBundle) for bundle in split_bundles]))
+ splits = ([(bundle.start_position,
+ bundle.stop_position) for bundle in split_bundles])
+ self.assertEqual(expect_splits, list(splits))
+
+ def test_restriction_size(self):
+ unused_element = None
+ restriction = (
+ self.sdf_restriction_provider.initial_restriction(unused_element))
+ split_1, split_2 = self.sdf_restriction_provider.split(unused_element,
+ restriction)
+ split_1_size = self.sdf_restriction_provider.restriction_size(
+ unused_element, split_1)
+ split_2_size = self.sdf_restriction_provider.restriction_size(
+ unused_element, split_2)
+ self.assertEqual(2, split_1_size)
+ self.assertEqual(2, split_2_size)
+
+
+class SDFBoundedSourceRestrictionTrackerTest(unittest.TestCase):
+
+ def setUp(self):
+ self.initial_start_pos = 0
+ self.initial_stop_pos = 4
+ source_bundle = SourceBundle(
+ self.initial_stop_pos - self.initial_start_pos,
+ RangeSource(self.initial_start_pos, self.initial_stop_pos),
+ self.initial_start_pos,
+ self.initial_stop_pos)
+ self.sdf_restriction_tracker = (
+ iobase._SDFBoundedSourceWrapper._SDFBoundedSourceRestrictionTracker(
+ source_bundle))
+
+ def test_current_restriction_before_split(self):
+ _, _, actual_start, actual_stop = (
+ self.sdf_restriction_tracker.current_restriction())
+ self.assertEqual(self.initial_start_pos, actual_start)
+ self.assertEqual(self.initial_stop_pos, actual_stop)
+
+ def test_current_restriction_after_split(self):
+ fraction_of_remainder = 0.5
+ self.sdf_restriction_tracker.try_claim(1)
+ expected_restriction, _ = (
+ self.sdf_restriction_tracker.try_split(fraction_of_remainder))
+ self.assertEqual(expected_restriction,
+ self.sdf_restriction_tracker.current_restriction())
+
+ def test_try_split_at_remainder(self):
+ fraction_of_remainder = 0.4
+ expected_primary = (0, 2, 2.0)
+ expected_residual = (2, 4, 2.0)
+ self.sdf_restriction_tracker.try_claim(0)
+ actual_primary, actual_residual = (
+ self.sdf_restriction_tracker.try_split(fraction_of_remainder))
+ self.assertEqual(expected_primary, (actual_primary.start_position,
+ actual_primary.stop_position,
+ actual_primary.weight))
+ self.assertEqual(expected_residual, (actual_residual.start_position,
+ actual_residual.stop_position,
+ actual_residual.weight))
+ self.assertEqual(actual_primary.weight,
+ self.sdf_restriction_tracker._weight)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/localfilesystem_test.py b/sdks/python/apache_beam/io/localfilesystem_test.py
index e86b00a..43f64f7 100644
--- a/sdks/python/apache_beam/io/localfilesystem_test.py
+++ b/sdks/python/apache_beam/io/localfilesystem_test.py
@@ -299,8 +299,8 @@
f.write('Hello')
with open(path2, 'a') as f:
f.write('foo')
- self.assertEquals(self.fs.checksum(path1), str(5))
- self.assertEquals(self.fs.checksum(path2), str(3))
+ self.assertEqual(self.fs.checksum(path1), str(5))
+ self.assertEqual(self.fs.checksum(path2), str(3))
def make_tree(self, path, value, expected_leaf_count=None):
"""Create a file+directory structure from a simple dict-based DSL
diff --git a/sdks/python/apache_beam/io/mongodbio.py b/sdks/python/apache_beam/io/mongodbio.py
new file mode 100644
index 0000000..6004ca1
--- /dev/null
+++ b/sdks/python/apache_beam/io/mongodbio.py
@@ -0,0 +1,512 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""This module implements IO classes to read and write data on MongoDB.
+
+
+Read from MongoDB
+-----------------
+:class:`ReadFromMongoDB` is a ``PTransform`` that reads from a configured
+MongoDB source and returns a ``PCollection`` of dict representing MongoDB
+documents.
+To configure MongoDB source, the URI to connect to MongoDB server, database
+name, collection name needs to be provided.
+
+Example usage::
+
+ pipeline | ReadFromMongoDB(uri='mongodb://localhost:27017',
+ db='testdb',
+ coll='input')
+
+
+Write to MongoDB:
+-----------------
+:class:`WriteToMongoDB` is a ``PTransform`` that writes MongoDB documents to
+configured sink, and the write is conducted through a mongodb bulk_write of
+``ReplaceOne`` operations. If the document's _id field already existed in the
+MongoDB collection, it results in an overwrite, otherwise, a new document
+will be inserted.
+
+Example usage::
+
+ pipeline | WriteToMongoDB(uri='mongodb://localhost:27017',
+ db='testdb',
+ coll='output',
+ batch_size=10)
+
+
+No backward compatibility guarantees. Everything in this module is experimental.
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import struct
+
+import apache_beam as beam
+from apache_beam.io import iobase
+from apache_beam.io.range_trackers import OrderedPositionRangeTracker
+from apache_beam.transforms import DoFn
+from apache_beam.transforms import PTransform
+from apache_beam.transforms import Reshuffle
+from apache_beam.utils.annotations import experimental
+
+try:
+ # Mongodb has its own bundled bson, which is not compatible with bson pakcage.
+ # (https://github.com/py-bson/bson/issues/82). Try to import objectid and if
+ # it fails because bson package is installed, MongoDB IO will not work but at
+ # least rest of the SDK will work.
+ from bson import objectid
+
+ # pymongo also internally depends on bson.
+ from pymongo import ASCENDING
+ from pymongo import DESCENDING
+ from pymongo import MongoClient
+ from pymongo import ReplaceOne
+except ImportError:
+ objectid = None
+ logging.warning("Could not find a compatible bson package.")
+
+__all__ = ['ReadFromMongoDB', 'WriteToMongoDB']
+
+
+@experimental()
+class ReadFromMongoDB(PTransform):
+ """A ``PTransfrom`` to read MongoDB documents into a ``PCollection``.
+ """
+
+ def __init__(self,
+ uri='mongodb://localhost:27017',
+ db=None,
+ coll=None,
+ filter=None,
+ projection=None,
+ extra_client_params=None):
+ """Initialize a :class:`ReadFromMongoDB`
+
+ Args:
+ uri (str): The MongoDB connection string following the URI format
+ db (str): The MongoDB database name
+ coll (str): The MongoDB collection name
+ filter: A `bson.SON
+ <https://api.mongodb.com/python/current/api/bson/son.html>`_ object
+ specifying elements which must be present for a document to be included
+ in the result set
+ projection: A list of field names that should be returned in the result
+ set or a dict specifying the fields to include or exclude
+ extra_client_params(dict): Optional `MongoClient
+ <https://api.mongodb.com/python/current/api/pymongo/mongo_client.html>`_
+ parameters
+
+ Returns:
+ :class:`~apache_beam.transforms.ptransform.PTransform`
+
+ """
+ if extra_client_params is None:
+ extra_client_params = {}
+ if not isinstance(db, str):
+ raise ValueError('ReadFromMongDB db param must be specified as a string')
+ if not isinstance(coll, str):
+ raise ValueError('ReadFromMongDB coll param must be specified as a '
+ 'string')
+ self._mongo_source = _BoundedMongoSource(
+ uri=uri,
+ db=db,
+ coll=coll,
+ filter=filter,
+ projection=projection,
+ extra_client_params=extra_client_params)
+
+ def expand(self, pcoll):
+ return pcoll | iobase.Read(self._mongo_source)
+
+
+class _BoundedMongoSource(iobase.BoundedSource):
+ def __init__(self,
+ uri=None,
+ db=None,
+ coll=None,
+ filter=None,
+ projection=None,
+ extra_client_params=None):
+ if extra_client_params is None:
+ extra_client_params = {}
+ if filter is None:
+ filter = {}
+ self.uri = uri
+ self.db = db
+ self.coll = coll
+ self.filter = filter
+ self.projection = projection
+ self.spec = extra_client_params
+
+ def estimate_size(self):
+ with MongoClient(self.uri, **self.spec) as client:
+ return client[self.db].command('collstats', self.coll).get('size')
+
+ def split(self, desired_bundle_size, start_position=None, stop_position=None):
+ start_position, stop_position = self._replace_none_positions(
+ start_position, stop_position)
+
+ desired_bundle_size_in_mb = desired_bundle_size // 1024 // 1024
+ split_keys = self._get_split_keys(desired_bundle_size_in_mb, start_position,
+ stop_position)
+
+ bundle_start = start_position
+ for split_key_id in split_keys:
+ if bundle_start >= stop_position:
+ break
+ bundle_end = min(stop_position, split_key_id)
+ yield iobase.SourceBundle(weight=desired_bundle_size_in_mb,
+ source=self,
+ start_position=bundle_start,
+ stop_position=bundle_end)
+ bundle_start = bundle_end
+ # add range of last split_key to stop_position
+ if bundle_start < stop_position:
+ yield iobase.SourceBundle(weight=desired_bundle_size_in_mb,
+ source=self,
+ start_position=bundle_start,
+ stop_position=stop_position)
+
+ def get_range_tracker(self, start_position, stop_position):
+ start_position, stop_position = self._replace_none_positions(
+ start_position, stop_position)
+ return _ObjectIdRangeTracker(start_position, stop_position)
+
+ def read(self, range_tracker):
+ with MongoClient(self.uri, **self.spec) as client:
+ all_filters = self._merge_id_filter(range_tracker)
+ docs_cursor = client[self.db][self.coll].find(filter=all_filters)
+ for doc in docs_cursor:
+ if not range_tracker.try_claim(doc['_id']):
+ return
+ yield doc
+
+ def display_data(self):
+ res = super(_BoundedMongoSource, self).display_data()
+ res['uri'] = self.uri
+ res['database'] = self.db
+ res['collection'] = self.coll
+ res['filter'] = json.dumps(self.filter)
+ res['projection'] = str(self.projection)
+ res['mongo_client_spec'] = json.dumps(self.spec)
+ return res
+
+ def _get_split_keys(self, desired_chunk_size_in_mb, start_pos, end_pos):
+ # calls mongodb splitVector command to get document ids at split position
+ # for desired bundle size, if desired chunk size smaller than 1mb, use
+ # mongodb default split size of 1mb.
+ if desired_chunk_size_in_mb < 1:
+ desired_chunk_size_in_mb = 1
+ if start_pos >= end_pos:
+ # single document not splittable
+ return []
+ with MongoClient(self.uri, **self.spec) as client:
+ name_space = '%s.%s' % (self.db, self.coll)
+ return (client[self.db].command(
+ 'splitVector',
+ name_space,
+ keyPattern={'_id': 1}, # Ascending index
+ min={'_id': start_pos},
+ max={'_id': end_pos},
+ maxChunkSize=desired_chunk_size_in_mb)['splitKeys'])
+
+ def _merge_id_filter(self, range_tracker):
+ # Merge the default filter with refined _id field range of range_tracker.
+ # see more at https://docs.mongodb.com/manual/reference/operator/query/and/
+ all_filters = {
+ '$and': [
+ self.filter.copy(),
+ # add additional range filter to query. $gte specifies start
+ # position(inclusive) and $lt specifies the end position(exclusive),
+ # see more at
+ # https://docs.mongodb.com/manual/reference/operator/query/gte/ and
+ # https://docs.mongodb.com/manual/reference/operator/query/lt/
+ {
+ '_id': {
+ '$gte': range_tracker.start_position(),
+ '$lt': range_tracker.stop_position()
+ }
+ },
+ ]
+ }
+
+ return all_filters
+
+ def _get_head_document_id(self, sort_order):
+ with MongoClient(self.uri, **self.spec) as client:
+ cursor = client[self.db][self.coll].find(filter={}, projection=[]).sort([
+ ('_id', sort_order)
+ ]).limit(1)
+ try:
+ return cursor[0]['_id']
+ except IndexError:
+ raise ValueError('Empty Mongodb collection')
+
+ def _replace_none_positions(self, start_position, stop_position):
+ if start_position is None:
+ start_position = self._get_head_document_id(ASCENDING)
+ if stop_position is None:
+ last_doc_id = self._get_head_document_id(DESCENDING)
+ # increment last doc id binary value by 1 to make sure the last document
+ # is not excluded
+ stop_position = _ObjectIdHelper.increment_id(last_doc_id, 1)
+ return start_position, stop_position
+
+
+class _ObjectIdHelper(object):
+ """A Utility class to manipulate bson object ids."""
+
+ @classmethod
+ def id_to_int(cls, id):
+ """
+ Args:
+ id: ObjectId required for each MongoDB document _id field.
+
+ Returns: Converted integer value of ObjectId's 12 bytes binary value.
+
+ """
+ # converts object id binary to integer
+ # id object is bytes type with size of 12
+ ints = struct.unpack('>III', id.binary)
+ return (ints[0] << 64) + (ints[1] << 32) + ints[2]
+
+ @classmethod
+ def int_to_id(cls, number):
+ """
+ Args:
+ number(int): The integer value to be used to convert to ObjectId.
+
+ Returns: The ObjectId that has the 12 bytes binary converted from the
+ integer value.
+
+ """
+ # converts integer value to object id. Int value should be less than
+ # (2 ^ 96) so it can be convert to 12 bytes required by object id.
+ if number < 0 or number >= (1 << 96):
+ raise ValueError('number value must be within [0, %s)' % (1 << 96))
+ ints = [(number & 0xffffffff0000000000000000) >> 64,
+ (number & 0x00000000ffffffff00000000) >> 32,
+ number & 0x0000000000000000ffffffff]
+
+ bytes = struct.pack('>III', *ints)
+ return objectid.ObjectId(bytes)
+
+ @classmethod
+ def increment_id(cls, object_id, inc):
+ """
+ Args:
+ object_id: The ObjectId to change.
+ inc(int): The incremental int value to be added to ObjectId.
+
+ Returns:
+
+ """
+ # increment object_id binary value by inc value and return new object id.
+ id_number = _ObjectIdHelper.id_to_int(object_id)
+ new_number = id_number + inc
+ if new_number < 0 or new_number >= (1 << 96):
+ raise ValueError('invalid incremental, inc value must be within ['
+ '%s, %s)' % (0 - id_number, 1 << 96 - id_number))
+ return _ObjectIdHelper.int_to_id(new_number)
+
+
+class _ObjectIdRangeTracker(OrderedPositionRangeTracker):
+ """RangeTracker for tracking mongodb _id of bson ObjectId type."""
+
+ def position_to_fraction(self, pos, start, end):
+ pos_number = _ObjectIdHelper.id_to_int(pos)
+ start_number = _ObjectIdHelper.id_to_int(start)
+ end_number = _ObjectIdHelper.id_to_int(end)
+ return (pos_number - start_number) / (end_number - start_number)
+
+ def fraction_to_position(self, fraction, start, end):
+ start_number = _ObjectIdHelper.id_to_int(start)
+ end_number = _ObjectIdHelper.id_to_int(end)
+ total = end_number - start_number
+ pos = int(total * fraction + start_number)
+ # make sure split position is larger than start position and smaller than
+ # end position.
+ if pos <= start_number:
+ return _ObjectIdHelper.increment_id(start, 1)
+ if pos >= end_number:
+ return _ObjectIdHelper.increment_id(end, -1)
+ return _ObjectIdHelper.int_to_id(pos)
+
+
+@experimental()
+class WriteToMongoDB(PTransform):
+ """WriteToMongoDB is a ``PTransform`` that writes a ``PCollection`` of
+ mongodb document to the configured MongoDB server.
+
+ In order to make the document writes idempotent so that the bundles are
+ retry-able without creating duplicates, the PTransform added 2 transformations
+ before final write stage:
+ a ``GenerateId`` transform and a ``Reshuffle`` transform.::
+
+ -----------------------------------------------
+ Pipeline --> |GenerateId --> Reshuffle --> WriteToMongoSink|
+ -----------------------------------------------
+ (WriteToMongoDB)
+
+ The ``GenerateId`` transform adds a random and unique*_id* field to the
+ documents if they don't already have one, it uses the same format as MongoDB
+ default. The ``Reshuffle`` transform makes sure that no fusion happens between
+ ``GenerateId`` and the final write stage transform,so that the set of
+ documents and their unique IDs are not regenerated if final write step is
+ retried due to a failure. This prevents duplicate writes of the same document
+ with different unique IDs.
+
+ """
+
+ def __init__(self,
+ uri='mongodb://localhost:27017',
+ db=None,
+ coll=None,
+ batch_size=100,
+ extra_client_params=None):
+ """
+
+ Args:
+ uri (str): The MongoDB connection string following the URI format
+ db (str): The MongoDB database name
+ coll (str): The MongoDB collection name
+ batch_size(int): Number of documents per bulk_write to MongoDB,
+ default to 100
+ extra_client_params(dict): Optional `MongoClient
+ <https://api.mongodb.com/python/current/api/pymongo/mongo_client.html>`_
+ parameters as keyword arguments
+
+ Returns:
+ :class:`~apache_beam.transforms.ptransform.PTransform`
+
+ """
+ if extra_client_params is None:
+ extra_client_params = {}
+ if not isinstance(db, str):
+ raise ValueError('WriteToMongoDB db param must be specified as a string')
+ if not isinstance(coll, str):
+ raise ValueError('WriteToMongoDB coll param must be specified as a '
+ 'string')
+ self._uri = uri
+ self._db = db
+ self._coll = coll
+ self._batch_size = batch_size
+ self._spec = extra_client_params
+
+ def expand(self, pcoll):
+ return pcoll \
+ | beam.ParDo(_GenerateObjectIdFn()) \
+ | Reshuffle() \
+ | beam.ParDo(_WriteMongoFn(self._uri, self._db, self._coll,
+ self._batch_size, self._spec))
+
+
+class _GenerateObjectIdFn(DoFn):
+ def process(self, element, *args, **kwargs):
+ # if _id field already exist we keep it as it is, otherwise the ptransform
+ # generates a new _id field to achieve idempotent write to mongodb.
+ if '_id' not in element:
+ # object.ObjectId() generates a unique identifier that follows mongodb
+ # default format, if _id is not present in document, mongodb server
+ # generates it with this same function upon write. However the
+ # uniqueness of generated id may not be guaranteed if the work load are
+ # distributed across too many processes. See more on the ObjectId format
+ # https://docs.mongodb.com/manual/reference/bson-types/#objectid.
+ element['_id'] = objectid.ObjectId()
+
+ yield element
+
+
+class _WriteMongoFn(DoFn):
+ def __init__(self,
+ uri=None,
+ db=None,
+ coll=None,
+ batch_size=100,
+ extra_params=None):
+ if extra_params is None:
+ extra_params = {}
+ self.uri = uri
+ self.db = db
+ self.coll = coll
+ self.spec = extra_params
+ self.batch_size = batch_size
+ self.batch = []
+
+ def finish_bundle(self):
+ self._flush()
+
+ def process(self, element, *args, **kwargs):
+ self.batch.append(element)
+ if len(self.batch) >= self.batch_size:
+ self._flush()
+
+ def _flush(self):
+ if len(self.batch) == 0:
+ return
+ with _MongoSink(self.uri, self.db, self.coll, self.spec) as sink:
+ sink.write(self.batch)
+ self.batch = []
+
+ def display_data(self):
+ res = super(_WriteMongoFn, self).display_data()
+ res['uri'] = self.uri
+ res['database'] = self.db
+ res['collection'] = self.coll
+ res['mongo_client_params'] = json.dumps(self.spec)
+ res['batch_size'] = self.batch_size
+ return res
+
+
+class _MongoSink(object):
+ def __init__(self, uri=None, db=None, coll=None, extra_params=None):
+ if extra_params is None:
+ extra_params = {}
+ self.uri = uri
+ self.db = db
+ self.coll = coll
+ self.spec = extra_params
+ self.client = None
+
+ def write(self, documents):
+ if self.client is None:
+ self.client = MongoClient(host=self.uri, **self.spec)
+ requests = []
+ for doc in documents:
+ # match document based on _id field, if not found in current collection,
+ # insert new one, otherwise overwrite it.
+ requests.append(
+ ReplaceOne(filter={'_id': doc.get('_id', None)},
+ replacement=doc,
+ upsert=True))
+ resp = self.client[self.db][self.coll].bulk_write(requests)
+ logging.debug('BulkWrite to MongoDB result in nModified:%d, nUpserted:%d, '
+ 'nMatched:%d, Errors:%s' %
+ (resp.modified_count, resp.upserted_count, resp.matched_count,
+ resp.bulk_api_result.get('writeErrors')))
+
+ def __enter__(self):
+ if self.client is None:
+ self.client = MongoClient(host=self.uri, **self.spec)
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ if self.client is not None:
+ self.client.close()
diff --git a/sdks/python/apache_beam/io/mongodbio_it_test.py b/sdks/python/apache_beam/io/mongodbio_it_test.py
new file mode 100644
index 0000000..bfc6099
--- /dev/null
+++ b/sdks/python/apache_beam/io/mongodbio_it_test.py
@@ -0,0 +1,94 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import time
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+def run(argv=None):
+ default_db = 'beam_mongodbio_it_db'
+ default_coll = 'integration_test_%d' % time.time()
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--mongo_uri',
+ default='mongodb://localhost:27017',
+ help='mongo uri string for connection')
+ parser.add_argument('--mongo_db',
+ default=default_db,
+ help='mongo uri string for connection')
+ parser.add_argument('--mongo_coll',
+ default=default_coll,
+ help='mongo uri string for connection')
+ parser.add_argument('--num_documents',
+ default=100000,
+ help='The expected number of documents to be generated '
+ 'for write or read',
+ type=int)
+ parser.add_argument('--batch_size',
+ default=10000,
+ help=('batch size for writing to mongodb'))
+ known_args, pipeline_args = parser.parse_known_args(argv)
+
+ # Test Write to MongoDB
+ with TestPipeline(options=PipelineOptions(pipeline_args)) as p:
+ start_time = time.time()
+ logging.info('Writing %d documents to mongodb' % known_args.num_documents)
+ docs = [{
+ 'number': x,
+ 'number_mod_2': x % 2,
+ 'number_mod_3': x % 3
+ } for x in range(known_args.num_documents)]
+
+ _ = p | 'Create documents' >> beam.Create(docs) \
+ | 'WriteToMongoDB' >> beam.io.WriteToMongoDB(known_args.mongo_uri,
+ known_args.mongo_db,
+ known_args.mongo_coll,
+ known_args.batch_size)
+ elapsed = time.time() - start_time
+ logging.info('Writing %d documents to mongodb finished in %.3f seconds' %
+ (known_args.num_documents, elapsed))
+
+ # Test Read from MongoDB
+ with TestPipeline(options=PipelineOptions(pipeline_args)) as p:
+ start_time = time.time()
+ logging.info('Reading from mongodb %s:%s' %
+ (known_args.mongo_db, known_args.mongo_coll))
+ r = p | 'ReadFromMongoDB' >> \
+ beam.io.ReadFromMongoDB(known_args.mongo_uri,
+ known_args.mongo_db,
+ known_args.mongo_coll,
+ projection=['number']) \
+ | 'Map' >> beam.Map(lambda doc: doc['number'])
+ assert_that(
+ r, equal_to([number for number in range(known_args.num_documents)]))
+
+ elapsed = time.time() - start_time
+ logging.info('Read %d documents from mongodb finished in %.3f seconds' %
+ (known_args.num_documents, elapsed))
+
+
+if __name__ == "__main__":
+ logging.getLogger().setLevel(logging.INFO)
+ run()
diff --git a/sdks/python/apache_beam/io/mongodbio_test.py b/sdks/python/apache_beam/io/mongodbio_test.py
new file mode 100644
index 0000000..3f07ec1
--- /dev/null
+++ b/sdks/python/apache_beam/io/mongodbio_test.py
@@ -0,0 +1,396 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import division
+
+import datetime
+import logging
+import random
+import sys
+import unittest
+from unittest import TestCase
+
+import mock
+from bson import objectid
+from pymongo import ASCENDING
+from pymongo import ReplaceOne
+
+import apache_beam as beam
+from apache_beam.io import ReadFromMongoDB
+from apache_beam.io import WriteToMongoDB
+from apache_beam.io import source_test_utils
+from apache_beam.io.mongodbio import _BoundedMongoSource
+from apache_beam.io.mongodbio import _GenerateObjectIdFn
+from apache_beam.io.mongodbio import _MongoSink
+from apache_beam.io.mongodbio import _ObjectIdHelper
+from apache_beam.io.mongodbio import _ObjectIdRangeTracker
+from apache_beam.io.mongodbio import _WriteMongoFn
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+class _MockMongoColl(object):
+ """Fake mongodb collection cursor."""
+
+ def __init__(self, docs):
+ self.docs = docs
+
+ def _filter(self, filter):
+ match = []
+ if not filter:
+ return self
+ if '$and' not in filter or not filter['$and']:
+ return self
+ start = filter['$and'][1]['_id'].get('$gte')
+ end = filter['$and'][1]['_id'].get('$lt')
+ assert start is not None
+ assert end is not None
+ for doc in self.docs:
+ if start and doc['_id'] < start:
+ continue
+ if end and doc['_id'] >= end:
+ continue
+ match.append(doc)
+ return match
+
+ def find(self, filter=None, **kwargs):
+ return _MockMongoColl(self._filter(filter))
+
+ def sort(self, sort_items):
+ key, order = sort_items[0]
+ self.docs = sorted(self.docs,
+ key=lambda x: x[key],
+ reverse=(order != ASCENDING))
+ return self
+
+ def limit(self, num):
+ return _MockMongoColl(self.docs[0:num])
+
+ def count_documents(self, filter):
+ return len(self._filter(filter))
+
+ def __getitem__(self, index):
+ return self.docs[index]
+
+
+class _MockMongoDb(object):
+ """Fake Mongo Db."""
+
+ def __init__(self, docs):
+ self.docs = docs
+
+ def __getitem__(self, coll_name):
+ return _MockMongoColl(self.docs)
+
+ def command(self, command, *args, **kwargs):
+ if command == 'collstats':
+ return {'size': 5, 'avgSize': 1}
+ elif command == 'splitVector':
+ return self.get_split_keys(command, *args, **kwargs)
+
+ def get_split_keys(self, command, ns, min, max, maxChunkSize, **kwargs):
+ # simulate mongo db splitVector command, return split keys base on chunk
+ # size, assuming every doc is of size 1mb
+ start_id = min['_id']
+ end_id = max['_id']
+ if start_id >= end_id:
+ return []
+ start_index = 0
+ end_index = 0
+ # get split range of [min, max]
+ for doc in self.docs:
+ if doc['_id'] < start_id:
+ start_index += 1
+ if doc['_id'] <= end_id:
+ end_index += 1
+ else:
+ break
+ # Return ids of elements in the range with chunk size skip and exclude
+ # head element. For simplicity of tests every document is considered 1Mb
+ # by default.
+ return {
+ 'splitKeys':
+ [x['_id'] for x in self.docs[start_index:end_index:maxChunkSize]][1:]
+ }
+
+
+class _MockMongoClient(object):
+ def __init__(self, docs):
+ self.docs = docs
+
+ def __getitem__(self, db_name):
+ return _MockMongoDb(self.docs)
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_val, exc_tb):
+ pass
+
+
+class MongoSourceTest(unittest.TestCase):
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def setUp(self, mock_client):
+ self._ids = [
+ objectid.ObjectId.from_datetime(
+ datetime.datetime(year=2020, month=i + 1, day=i + 1))
+ for i in range(5)
+ ]
+ self._docs = [{'_id': self._ids[i], 'x': i} for i in range(len(self._ids))]
+ mock_client.return_value = _MockMongoClient(self._docs)
+
+ self.mongo_source = _BoundedMongoSource('mongodb://test', 'testdb',
+ 'testcoll')
+
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_estimate_size(self, mock_client):
+ mock_client.return_value = _MockMongoClient(self._docs)
+ self.assertEqual(self.mongo_source.estimate_size(), 5)
+
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_split(self, mock_client):
+ mock_client.return_value = _MockMongoClient(self._docs)
+ for size in [i * 1024 * 1024 for i in (1, 2, 10)]:
+ splits = list(
+ self.mongo_source.split(start_position=None,
+ stop_position=None,
+ desired_bundle_size=size))
+
+ reference_info = (self.mongo_source, None, None)
+ sources_info = ([(split.source, split.start_position, split.stop_position)
+ for split in splits])
+ source_test_utils.assert_sources_equal_reference_source(
+ reference_info, sources_info)
+
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_dynamic_work_rebalancing(self, mock_client):
+ mock_client.return_value = _MockMongoClient(self._docs)
+ splits = list(
+ self.mongo_source.split(desired_bundle_size=3000 * 1024 * 1024))
+ assert len(splits) == 1
+ source_test_utils.assert_split_at_fraction_exhaustive(
+ splits[0].source, splits[0].start_position, splits[0].stop_position)
+
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_get_range_tracker(self, mock_client):
+ mock_client.return_value = _MockMongoClient(self._docs)
+ self.assertIsInstance(self.mongo_source.get_range_tracker(None, None),
+ _ObjectIdRangeTracker)
+
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_read(self, mock_client):
+ mock_tracker = mock.MagicMock()
+ test_cases = [
+ {
+ # range covers the first(inclusive) to third(exclusive) documents
+ 'start': self._ids[0],
+ 'stop': self._ids[2],
+ 'expected': self._docs[0:2]
+ },
+ {
+ # range covers from the first to the third documents
+ 'start': _ObjectIdHelper.int_to_id(0), # smallest possible id
+ 'stop': self._ids[2],
+ 'expected': self._docs[0:2]
+ },
+ {
+ # range covers from the third to last documents
+ 'start': self._ids[2],
+ 'stop': _ObjectIdHelper.int_to_id(2**96 - 1), # largest possible id
+ 'expected': self._docs[2:]
+ },
+ {
+ # range covers all documents
+ 'start': _ObjectIdHelper.int_to_id(0),
+ 'stop': _ObjectIdHelper.int_to_id(2**96 - 1),
+ 'expected': self._docs
+ },
+ {
+ # range doesn't include any document
+ 'start': _ObjectIdHelper.increment_id(self._ids[2], 1),
+ 'stop': _ObjectIdHelper.increment_id(self._ids[3], -1),
+ 'expected': []
+ },
+ ]
+ mock_client.return_value = _MockMongoClient(self._docs)
+ for case in test_cases:
+ mock_tracker.start_position.return_value = case['start']
+ mock_tracker.stop_position.return_value = case['stop']
+ result = list(self.mongo_source.read(mock_tracker))
+ self.assertListEqual(case['expected'], result)
+
+ def test_display_data(self):
+ data = self.mongo_source.display_data()
+ self.assertTrue('uri' in data)
+ self.assertTrue('database' in data)
+ self.assertTrue('collection' in data)
+
+
+class ReadFromMongoDBTest(unittest.TestCase):
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_read_from_mongodb(self, mock_client):
+ documents = [{'_id': objectid.ObjectId(), 'x': i} for i in range(3)]
+ mock_client.return_value = _MockMongoClient(documents)
+
+ with TestPipeline() as p:
+ docs = p | 'ReadFromMongoDB' >> ReadFromMongoDB(
+ uri='mongodb://test', db='db', coll='collection')
+ assert_that(docs, equal_to(documents))
+
+
+class GenerateObjectIdFnTest(unittest.TestCase):
+ def test_process(self):
+ with TestPipeline() as p:
+ output = (p | "Create" >> beam.Create([{
+ 'x': 1
+ }, {
+ 'x': 2,
+ '_id': 123
+ }])
+ | "Generate ID" >> beam.ParDo(_GenerateObjectIdFn())
+ | "Check" >> beam.Map(lambda x: '_id' in x))
+ assert_that(output, equal_to([True] * 2))
+
+
+class WriteMongoFnTest(unittest.TestCase):
+ @mock.patch('apache_beam.io.mongodbio._MongoSink')
+ def test_process(self, mock_sink):
+ docs = [{'x': 1}, {'x': 2}, {'x': 3}]
+ with TestPipeline() as p:
+ _ = (p | "Create" >> beam.Create(docs)
+ | "Write" >> beam.ParDo(_WriteMongoFn(batch_size=2)))
+ p.run()
+
+ self.assertEqual(
+ 2, mock_sink.return_value.__enter__.return_value.write.call_count)
+
+ def test_display_data(self):
+ data = _WriteMongoFn(batch_size=10).display_data()
+ self.assertEqual(10, data['batch_size'])
+
+
+class MongoSinkTest(unittest.TestCase):
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_write(self, mock_client):
+ docs = [{'x': 1}, {'x': 2}, {'x': 3}]
+ _MongoSink(uri='test', db='test', coll='test').write(docs)
+ self.assertTrue(mock_client.return_value.__getitem__.return_value.
+ __getitem__.return_value.bulk_write.called)
+
+
+class WriteToMongoDBTest(unittest.TestCase):
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_write_to_mongodb_with_existing_id(self, mock_client):
+ id = objectid.ObjectId()
+ docs = [{'x': 1, '_id': id}]
+ expected_update = [ReplaceOne({'_id': id}, {'x': 1, '_id': id}, True, None)]
+ with TestPipeline() as p:
+ _ = (p | "Create" >> beam.Create(docs)
+ | "Write" >> WriteToMongoDB(db='test', coll='test'))
+ p.run()
+ mock_client.return_value.__getitem__.return_value.__getitem__. \
+ return_value.bulk_write.assert_called_with(expected_update)
+
+ @mock.patch('apache_beam.io.mongodbio.MongoClient')
+ def test_write_to_mongodb_with_generated_id(self, mock_client):
+ docs = [{'x': 1}]
+ expected_update = [
+ ReplaceOne({'_id': mock.ANY}, {
+ 'x': 1,
+ '_id': mock.ANY
+ }, True, None)
+ ]
+ with TestPipeline() as p:
+ _ = (p | "Create" >> beam.Create(docs)
+ | "Write" >> WriteToMongoDB(db='test', coll='test'))
+ p.run()
+ mock_client.return_value.__getitem__.return_value.__getitem__. \
+ return_value.bulk_write.assert_called_with(expected_update)
+
+
+class ObjectIdHelperTest(TestCase):
+ def test_conversion(self):
+ test_cases = [
+ (objectid.ObjectId('000000000000000000000000'), 0),
+ (objectid.ObjectId('000000000000000100000000'), 2**32),
+ (objectid.ObjectId('0000000000000000ffffffff'), 2**32 - 1),
+ (objectid.ObjectId('000000010000000000000000'), 2**64),
+ (objectid.ObjectId('00000000ffffffffffffffff'), 2**64 - 1),
+ (objectid.ObjectId('ffffffffffffffffffffffff'), 2**96 - 1),
+ ]
+ for (id, number) in test_cases:
+ self.assertEqual(id, _ObjectIdHelper.int_to_id(number))
+ self.assertEqual(number, _ObjectIdHelper.id_to_int(id))
+
+ # random tests
+ for _ in range(100):
+ id = objectid.ObjectId()
+ if sys.version_info[0] < 3:
+ number = int(id.binary.encode('hex'), 16)
+ else: # PY3
+ number = int(id.binary.hex(), 16)
+ self.assertEqual(id, _ObjectIdHelper.int_to_id(number))
+ self.assertEqual(number, _ObjectIdHelper.id_to_int(id))
+
+ def test_increment_id(self):
+ test_cases = [
+ (objectid.ObjectId('000000000000000100000000'),
+ objectid.ObjectId('0000000000000000ffffffff')),
+ (objectid.ObjectId('000000010000000000000000'),
+ objectid.ObjectId('00000000ffffffffffffffff')),
+ ]
+ for (first, second) in test_cases:
+ self.assertEqual(second, _ObjectIdHelper.increment_id(first, -1))
+ self.assertEqual(first, _ObjectIdHelper.increment_id(second, 1))
+
+ for _ in range(100):
+ id = objectid.ObjectId()
+ self.assertLess(id, _ObjectIdHelper.increment_id(id, 1))
+ self.assertGreater(id, _ObjectIdHelper.increment_id(id, -1))
+
+
+class ObjectRangeTrackerTest(TestCase):
+ def test_fraction_position_conversion(self):
+ start_int = 0
+ stop_int = 2**96 - 1
+ start = _ObjectIdHelper.int_to_id(start_int)
+ stop = _ObjectIdHelper.int_to_id(stop_int)
+ test_cases = ([start_int, stop_int, 2**32, 2**32 - 1, 2**64, 2**64 - 1] +
+ [random.randint(start_int, stop_int) for _ in range(100)])
+ tracker = _ObjectIdRangeTracker()
+ for pos in test_cases:
+ id = _ObjectIdHelper.int_to_id(pos - start_int)
+ desired_fraction = (pos - start_int) / (stop_int - start_int)
+ self.assertAlmostEqual(tracker.position_to_fraction(id, start, stop),
+ desired_fraction,
+ places=20)
+
+ convert_id = tracker.fraction_to_position(
+ (pos - start_int) / (stop_int - start_int), start, stop)
+ # due to precision loss, the convert fraction is only gonna be close to
+ # original fraction.
+ convert_fraction = tracker.position_to_fraction(convert_id, start, stop)
+
+ self.assertGreater(convert_id, start)
+ self.assertLess(convert_id, stop)
+ self.assertAlmostEqual(convert_fraction, desired_fraction, places=20)
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/parquetio.py b/sdks/python/apache_beam/io/parquetio.py
index 41c13f3..a4e894cd 100644
--- a/sdks/python/apache_beam/io/parquetio.py
+++ b/sdks/python/apache_beam/io/parquetio.py
@@ -29,8 +29,6 @@
"""
from __future__ import absolute_import
-import platform
-import sys
from functools import partial
from apache_beam.io import filebasedsink
@@ -39,19 +37,98 @@
from apache_beam.io.iobase import RangeTracker
from apache_beam.io.iobase import Read
from apache_beam.io.iobase import Write
+from apache_beam.transforms import DoFn
+from apache_beam.transforms import ParDo
from apache_beam.transforms import PTransform
-if not (platform.system() == 'Windows' and sys.version_info[0] == 2):
+try:
import pyarrow as pa
import pyarrow.parquet as pq
+except ImportError:
+ pa = None
+ pq = None
-__all__ = ['ReadFromParquet', 'ReadAllFromParquet', 'WriteToParquet']
+__all__ = ['ReadFromParquet', 'ReadAllFromParquet', 'ReadFromParquetBatched',
+ 'ReadAllFromParquetBatched', 'WriteToParquet']
+
+
+class _ArrowTableToRowDictionaries(DoFn):
+ """ A DoFn that consumes an Arrow table and yields a python dictionary for
+ each row in the table."""
+ def process(self, table):
+ num_rows = table.num_rows
+ data_items = table.to_pydict().items()
+ for n in range(num_rows):
+ row = {}
+ for column, values in data_items:
+ row[column] = values[n]
+ yield row
+
+
+class ReadFromParquetBatched(PTransform):
+ """A :class:`~apache_beam.transforms.ptransform.PTransform` for reading
+ Parquet files as a `PCollection` of `pyarrow.Table`. This `PTransform` is
+ currently experimental. No backward-compatibility guarantees."""
+
+ def __init__(self, file_pattern=None, min_bundle_size=0,
+ validate=True, columns=None):
+ """ Initializes :class:`~ReadFromParquetBatched`
+
+ An alternative to :class:`~ReadFromParquet` that yields each row group from
+ the Parquet file as a `pyarrow.Table`. These Table instances can be
+ processed directly, or converted to a pandas DataFrame for processing. For
+ more information on supported types and schema, please see the pyarrow
+ documentation.
+
+ .. testcode::
+
+ with beam.Pipeline() as p:
+ dataframes = p \\
+ | 'Read' >> beam.io.ReadFromParquetBatched('/mypath/mypqfiles*') \\
+ | 'Convert to pandas' >> beam.Map(lambda table: table.to_pandas())
+
+ .. NOTE: We're not actually interested in this error; but if we get here,
+ it means that the way of calling this transform hasn't changed.
+
+ .. testoutput::
+ :hide:
+
+ Traceback (most recent call last):
+ ...
+ IOError: No files found based on the file pattern
+
+ See also: :class:`~ReadFromParquet`.
+
+ Args:
+ file_pattern (str): the file glob to read
+ min_bundle_size (int): the minimum size in bytes, to be considered when
+ splitting the input into bundles.
+ validate (bool): flag to verify that the files exist during the pipeline
+ creation time.
+ columns (List[str]): list of columns that will be read from files.
+ A column name may be a prefix of a nested field, e.g. 'a' will select
+ 'a.b', 'a.c', and 'a.d.e'
+ """
+
+ super(ReadFromParquetBatched, self).__init__()
+ self._source = _create_parquet_source(
+ file_pattern,
+ min_bundle_size,
+ validate=validate,
+ columns=columns,
+ )
+
+ def expand(self, pvalue):
+ return pvalue.pipeline | Read(self._source)
+
+ def display_data(self):
+ return {'source_dd': self._source}
class ReadFromParquet(PTransform):
"""A :class:`~apache_beam.transforms.ptransform.PTransform` for reading
- Parquet files. This `PTransform` is currently experimental. No
- backward-compatibility guarantees."""
+ Parquet files as a `PCollection` of dictionaries. This `PTransform` is
+ currently experimental. No backward-compatibility guarantees."""
def __init__(self, file_pattern=None, min_bundle_size=0,
validate=True, columns=None):
@@ -86,8 +163,9 @@
that are of simple types will be mapped into corresponding Python types.
Records that are of complex types like list and struct will be mapped to
Python list and dictionary respectively. For more information on supported
- types and schema, please see the pyarrow document.
+ types and schema, please see the pyarrow documentation.
+ See also: :class:`~ReadFromParquetBatched`.
Args:
file_pattern (str): the file glob to read
@@ -98,29 +176,29 @@
columns (List[str]): list of columns that will be read from files.
A column name may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'
-"""
+ """
super(ReadFromParquet, self).__init__()
self._source = _create_parquet_source(
file_pattern,
min_bundle_size,
validate=validate,
- columns=columns
+ columns=columns,
)
def expand(self, pvalue):
- return pvalue.pipeline | Read(self._source)
+ return pvalue | Read(self._source) | ParDo(_ArrowTableToRowDictionaries())
def display_data(self):
return {'source_dd': self._source}
-class ReadAllFromParquet(PTransform):
+class ReadAllFromParquetBatched(PTransform):
"""A ``PTransform`` for reading ``PCollection`` of Parquet files.
Uses source ``_ParquetSource`` to read a ``PCollection`` of Parquet files or
- file patterns and produce a ``PCollection`` of Parquet records. This
- ``PTransform`` is currently experimental. No backward-compatibility
- guarantees.
+ file patterns and produce a ``PCollection`` of ``pyarrow.Table``, one for
+ each Parquet file row group. This ``PTransform`` is currently experimental.
+ No backward-compatibility guarantees.
"""
DEFAULT_DESIRED_BUNDLE_SIZE = 64 * 1024 * 1024 # 64MB
@@ -140,7 +218,7 @@
may be a prefix of a nested field, e.g. 'a' will select
'a.b', 'a.c', and 'a.d.e'
"""
- super(ReadAllFromParquet, self).__init__()
+ super(ReadAllFromParquetBatched, self).__init__()
source_from_file = partial(
_create_parquet_source,
min_bundle_size=min_bundle_size,
@@ -156,6 +234,14 @@
return pvalue | self.label >> self._read_all_files
+class ReadAllFromParquet(PTransform):
+ def __init__(self, **kwargs):
+ self._read_batches = ReadAllFromParquetBatched(**kwargs)
+
+ def expand(self, pvalue):
+ return pvalue | self._read_batches | ParDo(_ArrowTableToRowDictionaries())
+
+
def _create_parquet_source(file_pattern=None,
min_bundle_size=0,
validate=False,
@@ -165,7 +251,7 @@
file_pattern=file_pattern,
min_bundle_size=min_bundle_size,
validate=validate,
- columns=columns
+ columns=columns,
)
@@ -244,13 +330,7 @@
else:
next_block_start = range_tracker.stop_position()
- num_rows = table.num_rows
- data_items = table.to_pydict().items()
- for n in range(num_rows):
- row = {}
- for column, values in data_items:
- row[column] = values[n]
- yield row
+ yield table
class WriteToParquet(PTransform):
diff --git a/sdks/python/apache_beam/io/parquetio_it_test.py b/sdks/python/apache_beam/io/parquetio_it_test.py
index f66c05f..0b58f47 100644
--- a/sdks/python/apache_beam/io/parquetio_it_test.py
+++ b/sdks/python/apache_beam/io/parquetio_it_test.py
@@ -18,7 +18,6 @@
from __future__ import division
import logging
-import platform
import string
import sys
import unittest
@@ -41,14 +40,13 @@
from apache_beam.transforms import CombineGlobally
from apache_beam.transforms.combiners import Count
-if not (platform.system() == 'Windows' and sys.version_info[0] == 2):
+try:
import pyarrow as pa
+except ImportError:
+ pa = None
-@unittest.skipIf(
- platform.system() == 'Windows' and sys.version_info[0] == 2,
- "pyarrow doesn't support Windows Python 2."
-)
+@unittest.skipIf(pa is None, "PyArrow is not installed.")
class TestParquetIT(unittest.TestCase):
@classmethod
diff --git a/sdks/python/apache_beam/io/parquetio_test.py b/sdks/python/apache_beam/io/parquetio_test.py
index d7dd842..d9b488d 100644
--- a/sdks/python/apache_beam/io/parquetio_test.py
+++ b/sdks/python/apache_beam/io/parquetio_test.py
@@ -19,7 +19,6 @@
import json
import logging
import os
-import platform
import shutil
import sys
import tempfile
@@ -36,7 +35,9 @@
from apache_beam.io import source_test_utils
from apache_beam.io.iobase import RangeTracker
from apache_beam.io.parquetio import ReadAllFromParquet
+from apache_beam.io.parquetio import ReadAllFromParquetBatched
from apache_beam.io.parquetio import ReadFromParquet
+from apache_beam.io.parquetio import ReadFromParquetBatched
from apache_beam.io.parquetio import WriteToParquet
from apache_beam.io.parquetio import _create_parquet_sink
from apache_beam.io.parquetio import _create_parquet_source
@@ -46,16 +47,17 @@
from apache_beam.transforms.display import DisplayData
from apache_beam.transforms.display_test import DisplayDataItemMatcher
-if not (platform.system() == 'Windows' and sys.version_info[0] == 2):
+try:
import pyarrow as pa
import pyarrow.lib as pl
import pyarrow.parquet as pq
+except ImportError:
+ pa = None
+ pl = None
+ pq = None
-@unittest.skipIf(
- platform.system() == 'Windows' and sys.version_info[0] == 2,
- "pyarrow doesn't support Windows Python 2."
-)
+@unittest.skipIf(pa is None, "PyArrow is not installed.")
class TestParquet(unittest.TestCase):
@classmethod
@@ -113,6 +115,23 @@
col_list.append(column)
return col_list
+ def _records_as_arrow(self, schema=None, count=None):
+ if schema is None:
+ schema = self.SCHEMA
+
+ if count is None:
+ count = len(self.RECORDS)
+
+ len_records = len(self.RECORDS)
+ data = []
+ for i in range(count):
+ data.append(self.RECORDS[i % len_records])
+ col_data = self._record_to_columns(data, schema)
+ col_array = [
+ pa.array(c, schema.types[cn]) for cn, c in enumerate(col_data)
+ ]
+ return pa.Table.from_arrays(col_array, schema.names)
+
def _write_data(self,
directory=None,
schema=None,
@@ -120,26 +139,12 @@
row_group_size=1000,
codec='none',
count=None):
- if schema is None:
- schema = self.SCHEMA
-
if directory is None:
directory = self.temp_dir
- if count is None:
- count = len(self.RECORDS)
-
with tempfile.NamedTemporaryFile(
delete=False, dir=directory, prefix=prefix) as f:
- len_records = len(self.RECORDS)
- data = []
- for i in range(count):
- data.append(self.RECORDS[i % len_records])
- col_data = self._record_to_columns(data, schema)
- col_array = [
- pa.array(c, schema.types[cn]) for cn, c in enumerate(col_data)
- ]
- table = pa.Table.from_arrays(col_array, schema.names)
+ table = self._records_as_arrow(schema, count)
pq.write_table(
table, f, row_group_size=row_group_size, compression=codec,
use_deprecated_int96_timestamps=True
@@ -177,12 +182,12 @@
def test_read_without_splitting(self):
file_name = self._write_data()
- expected_result = self.RECORDS
+ expected_result = [self._records_as_arrow()]
self._run_parquet_test(file_name, None, None, False, expected_result)
def test_read_with_splitting(self):
file_name = self._write_data()
- expected_result = self.RECORDS
+ expected_result = [self._records_as_arrow()]
self._run_parquet_test(file_name, None, 100, True, expected_result)
def test_source_display_data(self):
@@ -205,12 +210,19 @@
ReadFromParquet(
file_name,
validate=False)
- dd = DisplayData.create_from(read)
+ read_batched = \
+ ReadFromParquetBatched(
+ file_name,
+ validate=False)
expected_items = [
DisplayDataItemMatcher('compression', 'auto'),
DisplayDataItemMatcher('file_pattern', file_name)]
- hc.assert_that(dd.items, hc.contains_inanyorder(*expected_items))
+
+ hc.assert_that(DisplayData.create_from(read).items,
+ hc.contains_inanyorder(*expected_items))
+ hc.assert_that(DisplayData.create_from(read_batched).items,
+ hc.contains_inanyorder(*expected_items))
def test_sink_display_data(self):
file_name = 'some_parquet_sink'
@@ -271,6 +283,8 @@
path = dst.name
# pylint: disable=c-extension-no-member
with self.assertRaises(pl.ArrowInvalid):
+ # Should throw an error "ArrowInvalid: Casting from timestamp[ns] to
+ # timestamp[us] would lose data"
with TestPipeline() as p:
_ = p \
| Create(self.RECORDS) \
@@ -293,6 +307,21 @@
| Map(json.dumps)
assert_that(readback, equal_to([json.dumps(r) for r in self.RECORDS]))
+ def test_batched_read(self):
+ with tempfile.NamedTemporaryFile() as dst:
+ path = dst.name
+ with TestPipeline() as p:
+ _ = p \
+ | Create(self.RECORDS) \
+ | WriteToParquet(
+ path, self.SCHEMA, num_shards=1, shard_name_template='')
+ with TestPipeline() as p:
+ # json used for stable sortability
+ readback = \
+ p \
+ | ReadFromParquetBatched(path)
+ assert_that(readback, equal_to([self._records_as_arrow()]))
+
@parameterized.expand([
param(compression_type='snappy'),
param(compression_type='gzip'),
@@ -318,18 +347,28 @@
assert_that(readback, equal_to([json.dumps(r) for r in self.RECORDS]))
def test_read_reentrant(self):
- file_name = self._write_data()
+ file_name = self._write_data(count=6, row_group_size=3)
source = _create_parquet_source(file_name)
source_test_utils.assert_reentrant_reads_succeed((source, None, None))
def test_read_without_splitting_multiple_row_group(self):
- file_name = self._write_data(count=12000)
- expected_result = self.RECORDS * 2000
+ file_name = self._write_data(count=12000, row_group_size=1000)
+ # We expect 12000 elements, split into batches of 1000 elements. Create
+ # a list of pa.Table instances to model this expecation
+ expected_result = [
+ pa.Table.from_batches([batch]) for batch in self._records_as_arrow(
+ count=12000).to_batches(chunksize=1000)
+ ]
self._run_parquet_test(file_name, None, None, False, expected_result)
def test_read_with_splitting_multiple_row_group(self):
- file_name = self._write_data(count=12000)
- expected_result = self.RECORDS * 2000
+ file_name = self._write_data(count=12000, row_group_size=1000)
+ # We expect 12000 elements, split into batches of 1000 elements. Create
+ # a list of pa.Table instances to model this expecation
+ expected_result = [
+ pa.Table.from_batches([batch]) for batch in self._records_as_arrow(
+ count=12000).to_batches(chunksize=1000)
+ ]
self._run_parquet_test(file_name, None, 10000, True, expected_result)
def test_dynamic_work_rebalancing(self):
@@ -353,7 +392,7 @@
splits = [
split for split in source.split(desired_bundle_size=1)
]
- self.assertEquals(len(splits), 1)
+ self.assertEqual(len(splits), 1)
source = _create_parquet_source(file_name, min_bundle_size=0)
splits = [
@@ -370,9 +409,11 @@
def test_int96_type_conversion(self):
file_name = self._write_data(
count=120, row_group_size=20, schema=self.SCHEMA96)
+ orig = self._records_as_arrow(count=120, schema=self.SCHEMA96)
expected_result = [
- self._convert_to_timestamped_record(x) for x in self.RECORDS
- ] * 20
+ pa.Table.from_batches([batch])
+ for batch in orig.to_batches(chunksize=20)
+ ]
self._run_parquet_test(file_name, None, None, False, expected_result)
def test_split_points(self):
@@ -396,17 +437,18 @@
# When reading records of the first group, range_tracker.split_points()
# should return (0, iobase.RangeTracker.SPLIT_POINTS_UNKNOWN)
- self.assertEquals(
- split_points_report[:10],
- [(0, RangeTracker.SPLIT_POINTS_UNKNOWN)] * 10)
-
- # When reading records of last group, range_tracker.split_points() should
- # return (3, 1)
- self.assertEquals(split_points_report[-10:], [(3, 1)] * 10)
+ self.assertEqual(
+ split_points_report,
+ [(0, RangeTracker.SPLIT_POINTS_UNKNOWN),
+ (1, RangeTracker.SPLIT_POINTS_UNKNOWN),
+ (2, RangeTracker.SPLIT_POINTS_UNKNOWN),
+ (3, 1),
+ ])
def test_selective_columns(self):
file_name = self._write_data()
- expected_result = [{'name': r['name']} for r in self.RECORDS]
+ orig = self._records_as_arrow()
+ expected_result = [pa.Table.from_arrays([orig.column('name')])]
self._run_parquet_test(file_name, ['name'], None, False, expected_result)
def test_sink_transform_multiple_row_group(self):
@@ -430,6 +472,13 @@
| ReadAllFromParquet(),
equal_to(self.RECORDS))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([path]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()]))
+
def test_read_all_from_parquet_many_single_files(self):
path1 = self._write_data()
path2 = self._write_data()
@@ -440,6 +489,12 @@
| Create([path1, path2, path3]) \
| ReadAllFromParquet(),
equal_to(self.RECORDS * 3))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([path1, path2, path3]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()] * 3))
def test_read_all_from_parquet_file_pattern(self):
file_pattern = self._write_pattern(5)
@@ -449,6 +504,12 @@
| Create([file_pattern]) \
| ReadAllFromParquet(),
equal_to(self.RECORDS * 5))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([file_pattern]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()] * 5))
def test_read_all_from_parquet_many_file_patterns(self):
file_pattern1 = self._write_pattern(5)
@@ -460,6 +521,12 @@
| Create([file_pattern1, file_pattern2, file_pattern3]) \
| ReadAllFromParquet(),
equal_to(self.RECORDS * 10))
+ with TestPipeline() as p:
+ assert_that(
+ p \
+ | Create([file_pattern1, file_pattern2, file_pattern3]) \
+ | ReadAllFromParquetBatched(),
+ equal_to([self._records_as_arrow()] * 10))
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/io/range_trackers.py b/sdks/python/apache_beam/io/range_trackers.py
index e02bea7..c46f801 100644
--- a/sdks/python/apache_beam/io/range_trackers.py
+++ b/sdks/python/apache_beam/io/range_trackers.py
@@ -61,6 +61,7 @@
self._stop_offset = end
self._last_record_start = -1
+ self._last_attempted_record_start = -1
self._offset_of_last_split_point = -1
self._lock = threading.Lock()
@@ -77,6 +78,16 @@
def last_record_start(self):
return self._last_record_start
+ @property
+ def last_attempted_record_start(self):
+ """Return current value of last_attempted_record_start.
+
+ last_attempted_record_start records a valid position that tried to be
+ claimed by calling try_claim(). This value is only updated by `try_claim()`
+ no matter `try_claim()` returns `True` or `False`.
+ """
+ return self._last_attempted_record_start
+
def _validate_record_start(self, record_start, split_point):
# This function must only be called under the lock self.lock.
if not self._lock.locked():
@@ -102,7 +113,14 @@
def try_claim(self, record_start):
with self._lock:
+ # Attempted claim should be monotonous.
+ if record_start <= self._last_attempted_record_start:
+ raise ValueError(
+ 'Trying to return a record [starting at %d] which is not greater'
+ 'than the last-attempted record [starting at %d]' %
+ (record_start, self._last_attempted_record_start))
self._validate_record_start(record_start, True)
+ self._last_attempted_record_start = record_start
if record_start >= self.stop_position():
return False
self._offset_of_last_split_point = record_start
@@ -149,17 +167,19 @@
def fraction_consumed(self):
with self._lock:
- fraction = ((1.0 * (self._last_record_start - self.start_position()) /
- (self.stop_position() - self.start_position())) if
- self.stop_position() != self.start_position() else 0.0)
-
# self.last_record_start may become larger than self.end_offset when
# reading the records since any record that starts before the first 'split
# point' at or after the defined 'stop offset' is considered to be within
# the range of the OffsetRangeTracker. Hence fraction could be > 1.
# self.last_record_start is initialized to -1, hence fraction may be < 0.
# Bounding the to range [0, 1].
- return max(0.0, min(1.0, fraction))
+ return self.position_to_fraction(self._last_record_start,
+ self.start_position(),
+ self.stop_position())
+
+ def position_to_fraction(self, pos, start, stop):
+ fraction = 1.0 * (pos - start) / (stop - start) if start != stop else 0.0
+ return max(0.0, min(1.0, fraction))
def position_at_fraction(self, fraction):
if self.stop_position() == OffsetRangeTracker.OFFSET_INFINITY:
@@ -253,13 +273,6 @@
return self.position_to_fraction(
self._last_claim, self._start_position, self._stop_position)
- def position_to_fraction(self, pos, start, end):
- """
- Converts a position `pos` betweeen `start` and `end` (inclusive) to a
- fraction between 0 and 1.
- """
- raise NotImplementedError
-
def fraction_to_position(self, fraction, start, end):
"""
Converts a fraction between 0 and 1 to a position between start and end.
diff --git a/sdks/python/apache_beam/io/range_trackers_test.py b/sdks/python/apache_beam/io/range_trackers_test.py
index c9a9c42..e80401f 100644
--- a/sdks/python/apache_beam/io/range_trackers_test.py
+++ b/sdks/python/apache_beam/io/range_trackers_test.py
@@ -45,6 +45,32 @@
self.assertTrue(tracker.try_claim(5))
self.assertFalse(tracker.try_claim(6))
+ def test_try_claim_update_last_attempt(self):
+ tracker = range_trackers.OffsetRangeTracker(1, 2)
+ self.assertTrue(tracker.try_claim(1))
+ self.assertEqual(1, tracker.last_attempted_record_start)
+
+ self.assertFalse(tracker.try_claim(3))
+ self.assertEqual(3, tracker.last_attempted_record_start)
+
+ self.assertFalse(tracker.try_claim(6))
+ self.assertEqual(6, tracker.last_attempted_record_start)
+
+ with self.assertRaises(Exception):
+ tracker.try_claim(6)
+
+ def test_set_current_position(self):
+ tracker = range_trackers.OffsetRangeTracker(0, 6)
+ self.assertTrue(tracker.try_claim(2))
+ # Cannot set current position before successful claimed pos.
+ with self.assertRaises(Exception):
+ tracker.set_current_position(1)
+
+ self.assertFalse(tracker.try_claim(10))
+ tracker.set_current_position(11)
+ self.assertEqual(10, tracker.last_attempted_record_start)
+ self.assertEqual(11, tracker.last_record_start)
+
def test_try_return_record_continuous_until_split_point(self):
tracker = range_trackers.OffsetRangeTracker(9, 18)
# Return records with gaps of 2; every 3rd record is a split point.
@@ -93,7 +119,6 @@
self.assertFalse(tracker.try_claim(150))
self.assertFalse(tracker.try_claim(151))
# Should accept non-splitpoint records starting after stop offset.
- tracker.set_current_position(135)
tracker.set_current_position(152)
tracker.set_current_position(160)
tracker.set_current_position(171)
diff --git a/sdks/python/apache_beam/io/restriction_trackers.py b/sdks/python/apache_beam/io/restriction_trackers.py
index fe60a5e..0ba5b23 100644
--- a/sdks/python/apache_beam/io/restriction_trackers.py
+++ b/sdks/python/apache_beam/io/restriction_trackers.py
@@ -66,19 +66,25 @@
yield OffsetRange(current_split_start, current_split_stop)
current_split_start = current_split_stop
+ def split_at(self, split_pos):
+ return OffsetRange(self.start, split_pos), OffsetRange(split_pos, self.stop)
+
def new_tracker(self):
return OffsetRangeTracker(self.start, self.stop)
+ def size(self):
+ return self.stop - self.start
+
class OffsetRestrictionTracker(RestrictionTracker):
"""An `iobase.RestrictionTracker` implementations for an offset range.
- Offset range is represented as a pair of integers
- [start_position, stop_position}.
+ Offset range is represented as OffsetRange.
"""
- def __init__(self, start_position, stop_position):
- self._range = OffsetRange(start_position, stop_position)
+ def __init__(self, offset_range):
+ assert isinstance(offset_range, OffsetRange)
+ self._range = offset_range
self._current_position = None
self._current_watermark = None
self._last_claim_attempt = None
@@ -98,7 +104,7 @@
def current_restriction(self):
with self._lock:
- return (self._range.start, self._range.stop)
+ return self._range
def current_watermark(self):
return self._current_watermark
@@ -125,7 +131,7 @@
return self._range.stop
def default_size(self):
- return self._range.stop - self._range.start
+ return self._range.size()
def try_claim(self, position):
with self._lock:
@@ -148,17 +154,18 @@
return False
- def try_split(self, fraction):
+ def try_split(self, fraction_of_remainder):
with self._lock:
if not self._checkpointed:
if self._current_position is None:
cur = self._range.start - 1
else:
cur = self._current_position
- split_point = cur + int(max(1, (self._range.stop - cur) * fraction))
+ split_point = (
+ cur + int(max(1, (self._range.stop - cur) * fraction_of_remainder)))
if split_point < self._range.stop:
- prev_stop, self._range.stop = self._range.stop, split_point
- return (self._range.start, split_point), (split_point, prev_stop)
+ self._range, residual_range = self._range.split_at(split_point)
+ return self._range, residual_range
# TODO(SDF): Replace all calls with try_claim(0).
def checkpoint(self):
@@ -169,10 +176,7 @@
end_position = self._range.start
else:
end_position = self._current_position + 1
-
- residual_range = (end_position, self._range.stop)
-
- self._range = OffsetRange(self._range.start, end_position)
+ self._range, residual_range = self._range.split_at(end_position)
return residual_range
def defer_remainder(self, watermark=None):
diff --git a/sdks/python/apache_beam/io/restriction_trackers_test.py b/sdks/python/apache_beam/io/restriction_trackers_test.py
index 2820426..459b039 100644
--- a/sdks/python/apache_beam/io/restriction_trackers_test.py
+++ b/sdks/python/apache_beam/io/restriction_trackers_test.py
@@ -62,99 +62,113 @@
self.assertIn(OffsetRange(35, 60), splits)
self.assertIn(OffsetRange(60, 90), splits)
+ def test_split_at(self):
+ range = OffsetRange(0, 10)
+ cur, residual = range.split_at(5)
+ self.assertEqual(cur, OffsetRange(0, 5))
+ self.assertEqual(residual, OffsetRange(5, 10))
+
class OffsetRestrictionTrackerTest(unittest.TestCase):
def test_try_claim(self):
- tracker = OffsetRestrictionTracker(100, 200)
- self.assertEqual((100, 200), tracker.current_restriction())
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
+ self.assertEqual(OffsetRange(100, 200), tracker.current_restriction())
self.assertTrue(tracker.try_claim(100))
self.assertTrue(tracker.try_claim(150))
self.assertTrue(tracker.try_claim(199))
self.assertFalse(tracker.try_claim(200))
def test_checkpoint_unstarted(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
checkpoint = tracker.checkpoint()
- self.assertEqual((100, 100), tracker.current_restriction())
- self.assertEqual((100, 200), checkpoint)
+ self.assertEqual(OffsetRange(100, 100), tracker.current_restriction())
+ self.assertEqual(OffsetRange(100, 200), checkpoint)
def test_checkpoint_just_started(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(100))
checkpoint = tracker.checkpoint()
- self.assertEqual((100, 101), tracker.current_restriction())
- self.assertEqual((101, 200), checkpoint)
+ self.assertEqual(OffsetRange(100, 101), tracker.current_restriction())
+ self.assertEqual(OffsetRange(101, 200), checkpoint)
def test_checkpoint_regular(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(105))
self.assertTrue(tracker.try_claim(110))
checkpoint = tracker.checkpoint()
- self.assertEqual((100, 111), tracker.current_restriction())
- self.assertEqual((111, 200), checkpoint)
+ self.assertEqual(OffsetRange(100, 111), tracker.current_restriction())
+ self.assertEqual(OffsetRange(111, 200), checkpoint)
def test_checkpoint_claimed_last(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(105))
self.assertTrue(tracker.try_claim(110))
self.assertTrue(tracker.try_claim(199))
checkpoint = tracker.checkpoint()
- self.assertEqual((100, 200), tracker.current_restriction())
- self.assertEqual((200, 200), checkpoint)
+ self.assertEqual(OffsetRange(100, 200), tracker.current_restriction())
+ self.assertEqual(OffsetRange(200, 200), checkpoint)
def test_checkpoint_after_failed_claim(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(105))
self.assertTrue(tracker.try_claim(110))
self.assertTrue(tracker.try_claim(160))
self.assertFalse(tracker.try_claim(240))
checkpoint = tracker.checkpoint()
- self.assertTrue((100, 161), tracker.current_restriction())
- self.assertTrue((161, 200), checkpoint)
+ self.assertTrue(OffsetRange(100, 161), tracker.current_restriction())
+ self.assertTrue(OffsetRange(161, 200), checkpoint)
def test_non_monotonic_claim(self):
with self.assertRaises(ValueError):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(105))
self.assertTrue(tracker.try_claim(110))
self.assertTrue(tracker.try_claim(103))
def test_claim_before_starting_range(self):
with self.assertRaises(ValueError):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
tracker.try_claim(90)
def test_check_done_after_try_claim_past_end_of_range(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(150))
self.assertTrue(tracker.try_claim(175))
self.assertFalse(tracker.try_claim(220))
tracker.check_done()
def test_check_done_after_try_claim_past_end_of_range(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(150))
self.assertTrue(tracker.try_claim(175))
self.assertFalse(tracker.try_claim(200))
tracker.check_done()
def test_check_done_after_try_claim_right_before_end_of_range(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(150))
self.assertTrue(tracker.try_claim(175))
self.assertTrue(tracker.try_claim(199))
tracker.check_done()
def test_check_done_when_not_done(self):
- tracker = OffsetRestrictionTracker(100, 200)
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
self.assertTrue(tracker.try_claim(150))
self.assertTrue(tracker.try_claim(175))
with self.assertRaises(ValueError):
tracker.check_done()
+ def test_try_split(self):
+ tracker = OffsetRestrictionTracker(OffsetRange(100, 200))
+ tracker.try_claim(100)
+ cur, residual = tracker.try_split(0.5)
+ self.assertEqual(OffsetRange(100, 150), cur)
+ self.assertEqual(OffsetRange(150, 200), residual)
+ self.assertEqual(cur, tracker.current_restriction())
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/io/source_test_utils_test.py b/sdks/python/apache_beam/io/source_test_utils_test.py
index 6977d09..a8c3d82 100644
--- a/sdks/python/apache_beam/io/source_test_utils_test.py
+++ b/sdks/python/apache_beam/io/source_test_utils_test.py
@@ -82,8 +82,8 @@
result2 = source_test_utils.assert_split_at_fraction_behavior(
source, 20, 0.5,
source_test_utils.ExpectedSplitOutcome.MUST_SUCCEED_AND_BE_CONSISTENT)
- self.assertEquals(result1, result2)
- self.assertEquals(100, result1[0] + result1[1])
+ self.assertEqual(result1, result2)
+ self.assertEqual(100, result1[0] + result1[1])
result3 = source_test_utils.assert_split_at_fraction_behavior(
source, 30, 0.8,
@@ -91,8 +91,8 @@
result4 = source_test_utils.assert_split_at_fraction_behavior(
source, 50, 0.8,
source_test_utils.ExpectedSplitOutcome.MUST_SUCCEED_AND_BE_CONSISTENT)
- self.assertEquals(result3, result4)
- self.assertEquals(100, result3[0] + result4[1])
+ self.assertEqual(result3, result4)
+ self.assertEqual(100, result3[0] + result4[1])
self.assertTrue(result1[0] < result3[0])
self.assertTrue(result1[1] > result3[1])
@@ -103,8 +103,8 @@
result = source_test_utils.assert_split_at_fraction_behavior(
source, 90, 0.1, source_test_utils.ExpectedSplitOutcome.MUST_FAIL)
- self.assertEquals(result[0], 100)
- self.assertEquals(result[1], -1)
+ self.assertEqual(result[0], 100)
+ self.assertEqual(result[1], -1)
with self.assertRaises(ValueError):
source_test_utils.assert_split_at_fraction_behavior(
diff --git a/sdks/python/apache_beam/io/sources_test.py b/sdks/python/apache_beam/io/sources_test.py
index 318508a..8908681 100644
--- a/sdks/python/apache_beam/io/sources_test.py
+++ b/sdks/python/apache_beam/io/sources_test.py
@@ -50,37 +50,46 @@
start -= 1
start += len(f.readline())
current = start
- for line in f:
- if not range_tracker.try_claim(current):
+ line = f.readline()
+ while range_tracker.try_claim(current):
+ if not line:
return
yield line.rstrip(b'\n')
current += len(line)
+ line = f.readline()
def split(self, desired_bundle_size, start_position=None, stop_position=None):
assert start_position is None
assert stop_position is None
- with open(self._file_name, 'rb') as f:
- f.seek(0, os.SEEK_END)
- size = f.tell()
+ size = self.estimate_size()
bundle_start = 0
while bundle_start < size:
bundle_stop = min(bundle_start + LineSource.TEST_BUNDLE_SIZE, size)
- yield iobase.SourceBundle(1, self, bundle_start, bundle_stop)
+ yield iobase.SourceBundle(bundle_stop - bundle_start,
+ self,
+ bundle_start,
+ bundle_stop)
bundle_start = bundle_stop
def get_range_tracker(self, start_position, stop_position):
if start_position is None:
start_position = 0
if stop_position is None:
- with open(self._file_name, 'rb') as f:
- f.seek(0, os.SEEK_END)
- stop_position = f.tell()
+ stop_position = self._get_file_size()
return range_trackers.OffsetRangeTracker(start_position, stop_position)
def default_output_coder(self):
return coders.BytesCoder()
+ def estimate_size(self):
+ return self._get_file_size()
+
+ def _get_file_size(self):
+ with open(self._file_name, 'rb') as f:
+ f.seek(0, os.SEEK_END)
+ return f.tell()
+
class SourcesTest(unittest.TestCase):
@@ -103,6 +112,14 @@
result = [line for line in source.read(range_tracker)]
self.assertCountEqual([b'aaaa', b'bbbb', b'cccc', b'dddd'], result)
+ self.assertTrue(range_tracker.last_attempted_record_start
+ >= range_tracker.stop_position())
+
+ def test_source_estimated_size(self):
+ file_name = self._create_temp_file(b'aaaa\n')
+
+ source = LineSource(file_name)
+ self.assertEqual(5, source.estimate_size())
def test_run_direct(self):
file_name = self._create_temp_file(b'aaaa\nbbbb\ncccc\ndddd')
diff --git a/sdks/python/apache_beam/io/tfrecordio_test.py b/sdks/python/apache_beam/io/tfrecordio_test.py
index f003c34..c0a3c2d 100644
--- a/sdks/python/apache_beam/io/tfrecordio_test.py
+++ b/sdks/python/apache_beam/io/tfrecordio_test.py
@@ -216,7 +216,7 @@
with TempDir() as temp_dir:
file_path_prefix = temp_dir.create_temp_file('result')
with TestPipeline() as p:
- input_data = ['foo', 'bar']
+ input_data = [b'foo', b'bar']
_ = p | beam.Create(input_data) | WriteToTFRecord(
file_path_prefix, compression_type=CompressionTypes.GZIP)
@@ -232,7 +232,7 @@
with TempDir() as temp_dir:
file_path_prefix = temp_dir.create_temp_file('result')
with TestPipeline() as p:
- input_data = ['foo', 'bar']
+ input_data = [b'foo', b'bar']
_ = p | beam.Create(input_data) | WriteToTFRecord(
file_path_prefix, file_name_suffix='.gz')
diff --git a/sdks/python/apache_beam/io/utils.py b/sdks/python/apache_beam/io/utils.py
index 1dfadb5..d5912ae 100644
--- a/sdks/python/apache_beam/io/utils.py
+++ b/sdks/python/apache_beam/io/utils.py
@@ -29,7 +29,6 @@
class CountingSource(iobase.BoundedSource):
-
def __init__(self, count):
self.records_read = Metrics.counter(self.__class__, 'recordsRead')
self._count = count
@@ -46,22 +45,22 @@
return OffsetRangeTracker(start_position, stop_position)
def read(self, range_tracker):
- for i in range(self._count):
+ for i in range(range_tracker.start_position(),
+ range_tracker.stop_position()):
if not range_tracker.try_claim(i):
return
self.records_read.inc()
yield i
- def split(self, desired_bundle_size, start_position=None,
- stop_position=None):
+ def split(self, desired_bundle_size, start_position=None, stop_position=None):
if start_position is None:
start_position = 0
if stop_position is None:
stop_position = self._count
bundle_start = start_position
- while bundle_start < self._count:
- bundle_stop = max(self._count, bundle_start + desired_bundle_size)
+ while bundle_start < stop_position:
+ bundle_stop = min(stop_position, bundle_start + desired_bundle_size)
yield iobase.SourceBundle(weight=(bundle_stop - bundle_start),
source=self,
start_position=bundle_start,
diff --git a/sdks/python/apache_beam/io/utils_test.py b/sdks/python/apache_beam/io/utils_test.py
new file mode 100644
index 0000000..3d571b3
--- /dev/null
+++ b/sdks/python/apache_beam/io/utils_test.py
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import unittest
+
+import mock as mock
+
+from apache_beam.io import OffsetRangeTracker
+from apache_beam.io import source_test_utils
+from apache_beam.io.utils import CountingSource
+
+
+class CountingSourceTest(unittest.TestCase):
+ def setUp(self):
+ self.source = CountingSource(10)
+
+ def test_estimate_size(self):
+ self.assertEqual(10, self.source.estimate_size())
+
+ @mock.patch('apache_beam.io.utils.OffsetRangeTracker')
+ def test_get_range_tracker(self, mock_tracker):
+ _ = self.source.get_range_tracker(None, None)
+ mock_tracker.assert_called_with(0, 10)
+ _ = self.source.get_range_tracker(3, 7)
+ mock_tracker.assert_called_with(3, 7)
+
+ def test_read(self):
+ tracker = OffsetRangeTracker(3, 6)
+ res = list(self.source.read(tracker))
+ self.assertEqual([3, 4, 5], res)
+
+ def test_split(self):
+ for size in [1, 3, 10]:
+ splits = list(self.source.split(desired_bundle_size=size))
+
+ reference_info = (self.source, None, None)
+ sources_info = ([(split.source, split.start_position, split.stop_position)
+ for split in splits])
+ source_test_utils.assert_sources_equal_reference_source(
+ reference_info, sources_info)
+
+ def test_dynamic_work_rebalancing(self):
+ splits = list(self.source.split(desired_bundle_size=20))
+ assert len(splits) == 1
+ source_test_utils.assert_split_at_fraction_exhaustive(
+ splits[0].source, splits[0].start_position, splits[0].stop_position)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/io/vcfio.py b/sdks/python/apache_beam/io/vcfio.py
index 0ce76bd..2a13bf8 100644
--- a/sdks/python/apache_beam/io/vcfio.py
+++ b/sdks/python/apache_beam/io/vcfio.py
@@ -308,13 +308,13 @@
range_tracker)
try:
self._vcf_reader = vcf.Reader(fsock=self._create_generator())
- except SyntaxError as e:
+ except SyntaxError:
# Throw the exception inside the generator to ensure file is properly
# closed (it's opened inside TextSource.read_records).
self._text_lines.throw(
ValueError('An exception was raised when reading header from VCF '
'file %s: %s' % (self._file_name,
- traceback.format_exc(e))))
+ traceback.format_exc())))
def _store_header_lines(self, header_lines):
self._header_lines = header_lines
@@ -344,12 +344,12 @@
record = next(self._vcf_reader)
return self._convert_to_variant_record(record, self._vcf_reader.infos,
self._vcf_reader.formats)
- except (LookupError, ValueError) as e:
+ except (LookupError, ValueError):
if self._allow_malformed_records:
logging.warning(
'An exception was raised when reading record from VCF file '
'%s. Invalid record was %s: %s',
- self._file_name, self._last_record, traceback.format_exc(e))
+ self._file_name, self._last_record, traceback.format_exc())
return MalformedVcfRecord(self._file_name, self._last_record)
# Throw the exception inside the generator to ensure file is properly
@@ -359,7 +359,7 @@
'file %s. Invalid record was %s: %s' % (
self._file_name,
self._last_record,
- traceback.format_exc(e))))
+ traceback.format_exc())))
def _convert_to_variant_record(self, record, infos, formats):
"""Converts the PyVCF record to a :class:`Variant` object.
diff --git a/sdks/python/apache_beam/options/pipeline_options.py b/sdks/python/apache_beam/options/pipeline_options.py
index fde1a7e..5842f7e 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -20,6 +20,7 @@
from __future__ import absolute_import
import argparse
+import json
import logging
from builtins import list
from builtins import object
@@ -405,6 +406,11 @@
default=0,
help='replay every bundle this many extra times, for profiling'
'and debugging')
+ parser.add_argument(
+ '--direct_num_workers',
+ type=int,
+ default=1,
+ help='number of parallel running workers.')
class GoogleCloudOptions(PipelineOptions):
@@ -442,13 +448,12 @@
parser.add_argument('--temp_location',
default=None,
help='GCS path for saving temporary workflow jobs.')
- # The Cloud Dataflow service does not yet honor this setting. However, once
- # service support is added then users of this SDK will be able to control
- # the region. Default is up to the Dataflow service. See
+ # The Google Compute Engine region for creating Dataflow jobs. See
# https://cloud.google.com/compute/docs/regions-zones/regions-zones for a
- # list of valid options/
+ # list of valid options. Currently defaults to us-central1, but future
+ # releases of Beam will require the user to set the region explicitly.
parser.add_argument('--region',
- default='us-central1',
+ default=None,
help='The Google Compute Engine region for creating '
'Dataflow job.')
parser.add_argument('--service_account_email',
@@ -471,7 +476,16 @@
action='store_true',
help='Update an existing streaming Cloud Dataflow job. '
'Experimental. '
- 'See https://cloud.google.com/dataflow/pipelines/'
+ 'See https://cloud.google.com/dataflow/docs/guides/'
+ 'updating-a-pipeline')
+ parser.add_argument('--transform_name_mapping',
+ default=None,
+ type=json.loads,
+ help='The transform mapping that maps the named '
+ 'transforms in your prior pipeline code to names '
+ 'in your replacement pipeline code.'
+ 'Experimental. '
+ 'See https://cloud.google.com/dataflow/docs/guides/'
'updating-a-pipeline')
parser.add_argument('--enable_streaming_engine',
default=False,
@@ -500,6 +514,15 @@
errors.append('--dataflow_job_file and --template_location '
'are mutually exclusive.')
+ if self.view_as(GoogleCloudOptions).region is None:
+ self.view_as(GoogleCloudOptions).region = 'us-central1'
+ runner = self.view_as(StandardOptions).runner
+ if runner == 'DataflowRunner' or runner == 'TestDataflowRunner':
+ logging.warning(
+ '--region not set; will default to us-central1. Future releases of '
+ 'Beam will require the user to set the region explicitly. '
+ 'https://cloud.google.com/compute/docs/regions-zones/regions-zones')
+
return errors
@@ -558,7 +581,7 @@
help=
('If and how to autoscale the workerpool.'))
parser.add_argument(
- '--worker_machine_type',
+ '--worker_machine_type', '--machine_type',
dest='machine_type',
default=None,
help=('Machine type to create Dataflow worker VMs as. See '
@@ -574,7 +597,7 @@
('Remote worker disk size, in gigabytes, or 0 to use the default size. '
'If not set, the Dataflow service will use a reasonable default.'))
parser.add_argument(
- '--worker_disk_type',
+ '--worker_disk_type', '--disk_type',
dest='disk_type',
default=None,
help=('Specifies what type of persistent disk should be used.'))
@@ -657,6 +680,16 @@
'enabled with this flag. Please sync with the owners of the runner '
'before enabling any experiments.'))
+ parser.add_argument(
+ '--number_of_worker_harness_threads',
+ type=int,
+ default=None,
+ help=
+ ('Number of threads per worker to use on the runner. If left '
+ 'unspecified, the runner will compute an appropriate number of '
+ 'threads to use. Currently only enabled for DataflowRunner when '
+ 'experiment \'use_unified_worker\' is enabled.'))
+
def add_experiment(self, experiment):
# pylint: disable=access-member-before-definition
if self.experiments is None:
@@ -785,15 +818,24 @@
"""
@classmethod
def _add_argparse_args(cls, parser):
- parser.add_argument('--job_endpoint',
- default=None,
- help=
- ('Job service endpoint to use. Should be in the form '
- 'of address and port, e.g. localhost:3000'))
+ parser.add_argument(
+ '--job_endpoint', default=None,
+ help=('Job service endpoint to use. Should be in the form of address '
+ 'and port, e.g. localhost:3000'))
+ parser.add_argument(
+ '--job-server-timeout', default=60, type=int,
+ help=('Job service request timeout in seconds. The timeout '
+ 'determines the max time the driver program will wait to '
+ 'get a response from the job server. NOTE: the timeout does not '
+ 'apply to the actual pipeline run time. The driver program can '
+ 'still wait for job completion indefinitely.'))
parser.add_argument(
'--environment_type', default=None,
help=('Set the default environment type for running '
- 'user code. Possible options are DOCKER and PROCESS.'))
+ 'user code. DOCKER (default) runs user code in a container. '
+ 'PROCESS runs user code in processes that are automatically '
+ 'started on each worker node. LOOPBACK runs user code on the '
+ 'same process that originally submitted the job.'))
parser.add_argument(
'--environment_config', default=None,
help=('Set environment configuration for running the user code.\n For '
@@ -805,9 +847,8 @@
parser.add_argument(
'--sdk_worker_parallelism', default=0,
help=('Sets the number of sdk worker processes that will run on each '
- 'worker node. Default is 0. If 0, it will be automatically set '
- 'by the runner by looking at different parameters (e.g. number '
- 'of CPU cores on the worker machine or configuration).'))
+ 'worker node. Default is 0. If 0, a value will be chosen by the '
+ 'runner.'))
parser.add_argument(
'--environment_cache_millis', default=0,
help=('Duration in milliseconds for environment cache within a job. '
diff --git a/sdks/python/apache_beam/options/pipeline_options_test.py b/sdks/python/apache_beam/options/pipeline_options_test.py
index 5c51725..fdedfc4 100644
--- a/sdks/python/apache_beam/options/pipeline_options_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_test.py
@@ -25,9 +25,11 @@
import hamcrest as hc
from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import ProfilingOptions
from apache_beam.options.pipeline_options import TypeOptions
+from apache_beam.options.pipeline_options import WorkerOptions
from apache_beam.options.value_provider import RuntimeValueProvider
from apache_beam.options.value_provider import StaticValueProvider
from apache_beam.transforms.display import DisplayData
@@ -46,6 +48,12 @@
'mock_option': None,
'mock_multi_option': None},
'display_data': [DisplayDataItemMatcher('num_workers', 5)]},
+ {'flags': ['--direct_num_workers', '5'],
+ 'expected': {'direct_num_workers': 5,
+ 'mock_flag': False,
+ 'mock_option': None,
+ 'mock_multi_option': None},
+ 'display_data': [DisplayDataItemMatcher('direct_num_workers', 5)]},
{
'flags': [
'--profile_cpu', '--profile_location', 'gs://bucket/', 'ignored'],
@@ -252,6 +260,18 @@
options = PipelineOptions(flags=[''])
self.assertEqual(options.get_all_options()['experiments'], None)
+ def test_worker_options(self):
+ options = PipelineOptions(['--machine_type', 'abc', '--disk_type', 'def'])
+ worker_options = options.view_as(WorkerOptions)
+ self.assertEqual(worker_options.machine_type, 'abc')
+ self.assertEqual(worker_options.disk_type, 'def')
+
+ options = PipelineOptions(
+ ['--worker_machine_type', 'abc', '--worker_disk_type', 'def'])
+ worker_options = options.view_as(WorkerOptions)
+ self.assertEqual(worker_options.machine_type, 'abc')
+ self.assertEqual(worker_options.disk_type, 'def')
+
def test_option_modifications_are_shared_between_views(self):
pipeline_options = PipelineOptions([
'--mock_option', 'value', '--mock_flag',
@@ -458,6 +478,11 @@
True,
debug_options.lookup_experiment('existing_experiment'))
+ def test_transform_name_mapping(self):
+ options = PipelineOptions(['--transform_name_mapping={\"from\":\"to\"}'])
+ mapping = options.view_as(GoogleCloudOptions).transform_name_mapping
+ self.assertEqual(mapping['from'], 'to')
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator.py b/sdks/python/apache_beam/options/pipeline_options_validator.py
index 373a7b9..8f7c946 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator.py
@@ -24,6 +24,8 @@
import re
from builtins import object
+from past.builtins import unicode
+
from apache_beam.internal import pickler
from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.options.pipeline_options import GoogleCloudOptions
@@ -75,6 +77,9 @@
ERR_INVALID_TEST_MATCHER_UNPICKLABLE = (
'Invalid value (%s) for option: %s. Please make sure the test matcher '
'is unpicklable.')
+ ERR_INVALID_TRANSFORM_NAME_MAPPING = (
+ 'Invalid transform name mapping format. Please make sure the mapping is '
+ 'string key-value pairs. Invalid pair: (%s:%s)')
# GCS path specific patterns.
GCS_URI = '(?P<SCHEME>[^:]+)://(?P<BUCKET>[^/]+)(/(?P<OBJECT>.*))?'
@@ -174,6 +179,16 @@
if not view.job_name:
errors.extend(self._validate_error(
'Existing job name must be provided when updating a pipeline.'))
+ if view.transform_name_mapping:
+ if not view.update or not self.options.view_as(StandardOptions).streaming:
+ errors.append('Transform name mapping option is only useful when '
+ '--update and --streaming is specified')
+ for _, (key, value) in enumerate(view.transform_name_mapping.items()):
+ if not isinstance(key, (str, unicode)) \
+ or not isinstance(value, (str, unicode)):
+ errors.extend(self._validate_error(
+ self.ERR_INVALID_TRANSFORM_NAME_MAPPING, key, value))
+ break
return errors
def validate_optional_argument_positive(self, view, arg_name):
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator_test.py b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
index 9818422..7010716 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
@@ -23,6 +23,9 @@
import unittest
from builtins import object
+from hamcrest import assert_that
+from hamcrest import contains_string
+from hamcrest import only_contains
from hamcrest.core.base_matcher import BaseMatcher
from apache_beam.internal import pickler
@@ -333,6 +336,36 @@
self.assertEqual(
self.check_errors_for_arguments(errors, case['errors']), [])
+ def test_transform_name_mapping_without_update(self):
+ options = ['--project=example:example',
+ '--staging_location=gs://foo/bar',
+ '--temp_location=gs://foo/bar',
+ '--transform_name_mapping={\"fromPardo\":\"toPardo\"}']
+
+ pipeline_options = PipelineOptions(options)
+ runner = MockRunners.DataflowRunner()
+ validator = PipelineOptionsValidator(pipeline_options, runner)
+ errors = validator.validate()
+ assert_that(errors, only_contains(
+ contains_string('Transform name mapping option is only useful when '
+ '--update and --streaming is specified')))
+
+ def test_transform_name_mapping_invalid_format(self):
+ options = ['--project=example:example',
+ '--staging_location=gs://foo/bar',
+ '--temp_location=gs://foo/bar',
+ '--update',
+ '--job_name=test',
+ '--streaming',
+ '--transform_name_mapping={\"fromPardo\":123}']
+
+ pipeline_options = PipelineOptions(options)
+ runner = MockRunners.DataflowRunner()
+ validator = PipelineOptionsValidator(pipeline_options, runner)
+ errors = validator.validate()
+ assert_that(errors, only_contains(
+ contains_string('Invalid transform name mapping format.')))
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/options/value_provider_test.py b/sdks/python/apache_beam/options/value_provider_test.py
index 0a935b6..8f530a0 100644
--- a/sdks/python/apache_beam/options/value_provider_test.py
+++ b/sdks/python/apache_beam/options/value_provider_test.py
@@ -101,7 +101,7 @@
self.assertEqual(options.vpt_vp_arg5.get(), 123)
def test_set_runtime_option(self):
- # define ValueProvider ptions, with and without default values
+ # define ValueProvider options, with and without default values
class UserDefinedOptions1(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
diff --git a/sdks/python/apache_beam/pipeline.py b/sdks/python/apache_beam/pipeline.py
index a0e8a72..5ce95d0 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -145,8 +145,8 @@
if isinstance(runner, str):
runner = create_runner(runner)
elif not isinstance(runner, PipelineRunner):
- raise TypeError('Runner must be a PipelineRunner object or the '
- 'name of a registered runner.')
+ raise TypeError('Runner %s is not a PipelineRunner object or the '
+ 'name of a registered runner.' % runner)
# Validate pipeline options
errors = PipelineOptionsValidator(self._options, runner).validate()
@@ -156,7 +156,8 @@
# set default experiments for portable runner
# (needs to occur prior to pipeline construction)
- if self._options.view_as(StandardOptions).runner == 'PortableRunner':
+ portable_runners = ['PortableRunner', 'FlinkRunner']
+ if self._options.view_as(StandardOptions).runner in portable_runners:
experiments = (self._options.view_as(DebugOptions).experiments or [])
if not 'beam_fn_api' in experiments:
experiments.append('beam_fn_api')
@@ -619,10 +620,10 @@
default_environment=default_environment)
elif default_environment is not None:
raise ValueError(
- 'Only one of context or default_environment may be specificed.')
+ 'Only one of context or default_environment may be specified.')
- # The RunnerAPI spec requires certain transforms to have KV inputs
- # (and corresponding outputs).
+ # The RunnerAPI spec requires certain transforms and side-inputs to have KV
+ # inputs (and corresponding outputs).
# Currently we only upgrade to KV pairs. If there is a need for more
# general shapes, potential conflicts will have to be resolved.
# We also only handle single-input, and (for fixing the output) single
@@ -632,16 +633,24 @@
self.visit_transform(transform_node)
def visit_transform(self, transform_node):
- if (transform_node.transform
- and transform_node.transform.runner_api_requires_keyed_input()):
+ if not transform_node.transform:
+ return
+ if transform_node.transform.runner_api_requires_keyed_input():
pcoll = transform_node.inputs[0]
pcoll.element_type = typehints.coerce_to_kv_type(
pcoll.element_type, transform_node.full_label)
if len(transform_node.outputs) == 1:
# The runner often has expectations about the output types as well.
output, = transform_node.outputs.values()
- output.element_type = transform_node.transform.infer_output_type(
- pcoll.element_type)
+ if not output.element_type:
+ output.element_type = transform_node.transform.infer_output_type(
+ pcoll.element_type
+ )
+ for side_input in transform_node.transform.side_inputs:
+ if side_input.requires_keyed_input():
+ side_input.pvalue.element_type = typehints.coerce_to_kv_type(
+ side_input.pvalue.element_type, transform_node.full_label,
+ side_input_producer=side_input.pvalue.producer.full_label)
self.visit(ForceKvInputTypes())
diff --git a/sdks/python/apache_beam/pipeline_test.py b/sdks/python/apache_beam/pipeline_test.py
index 8140858..e01e100 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -31,6 +31,7 @@
import apache_beam as beam
from apache_beam import typehints
+from apache_beam.coders import BytesCoder
from apache_beam.io import Read
from apache_beam.metrics import Metrics
from apache_beam.pipeline import Pipeline
@@ -53,6 +54,7 @@
from apache_beam.transforms import ParDo
from apache_beam.transforms import PTransform
from apache_beam.transforms import WindowInto
+from apache_beam.transforms.userstate import BagStateSpec
from apache_beam.transforms.window import SlidingWindows
from apache_beam.transforms.window import TimestampedValue
from apache_beam.utils.timestamp import MIN_TIMESTAMP
@@ -88,6 +90,16 @@
return FakeSource._Reader(self._vals)
+class FakeUnboundedSource(NativeSource):
+ """Fake unbounded source. Does not work at runtime"""
+
+ def reader(self):
+ return None
+
+ def is_bounded(self):
+ return False
+
+
class DoubleParDo(beam.PTransform):
def expand(self, input):
return input | 'Inner' >> beam.Map(lambda a: a * 2)
@@ -168,6 +180,55 @@
assert_that(pcoll4, equal_to([11, 12, 12, 12, 13]), label='pcoll4')
pipeline.run()
+ def test_maptuple_builtin(self):
+ pipeline = TestPipeline()
+ pcoll = pipeline | Create([('e1', 'e2')])
+ side1 = beam.pvalue.AsSingleton(pipeline | 'side1' >> Create(['s1']))
+ side2 = beam.pvalue.AsSingleton(pipeline | 'side2' >> Create(['s2']))
+
+ # A test function with a tuple input, an auxiliary parameter,
+ # and some side inputs.
+ fn = lambda e1, e2, t=DoFn.TimestampParam, s1=None, s2=None: (
+ e1, e2, t, s1, s2)
+ assert_that(pcoll | 'NoSides' >> beam.core.MapTuple(fn),
+ equal_to([('e1', 'e2', MIN_TIMESTAMP, None, None)]),
+ label='NoSidesCheck')
+ assert_that(pcoll | 'StaticSides' >> beam.core.MapTuple(fn, 's1', 's2'),
+ equal_to([('e1', 'e2', MIN_TIMESTAMP, 's1', 's2')]),
+ label='StaticSidesCheck')
+ assert_that(pcoll | 'DynamicSides' >> beam.core.MapTuple(fn, side1, side2),
+ equal_to([('e1', 'e2', MIN_TIMESTAMP, 's1', 's2')]),
+ label='DynamicSidesCheck')
+ assert_that(pcoll | 'MixedSides' >> beam.core.MapTuple(fn, s2=side2),
+ equal_to([('e1', 'e2', MIN_TIMESTAMP, None, 's2')]),
+ label='MixedSidesCheck')
+ pipeline.run()
+
+ def test_flatmaptuple_builtin(self):
+ pipeline = TestPipeline()
+ pcoll = pipeline | Create([('e1', 'e2')])
+ side1 = beam.pvalue.AsSingleton(pipeline | 'side1' >> Create(['s1']))
+ side2 = beam.pvalue.AsSingleton(pipeline | 'side2' >> Create(['s2']))
+
+ # A test function with a tuple input, an auxiliary parameter,
+ # and some side inputs.
+ fn = lambda e1, e2, t=DoFn.TimestampParam, s1=None, s2=None: (
+ e1, e2, t, s1, s2)
+ assert_that(pcoll | 'NoSides' >> beam.core.FlatMapTuple(fn),
+ equal_to(['e1', 'e2', MIN_TIMESTAMP, None, None]),
+ label='NoSidesCheck')
+ assert_that(pcoll | 'StaticSides' >> beam.core.FlatMapTuple(fn, 's1', 's2'),
+ equal_to(['e1', 'e2', MIN_TIMESTAMP, 's1', 's2']),
+ label='StaticSidesCheck')
+ assert_that(pcoll
+ | 'DynamicSides' >> beam.core.FlatMapTuple(fn, side1, side2),
+ equal_to(['e1', 'e2', MIN_TIMESTAMP, 's1', 's2']),
+ label='DynamicSidesCheck')
+ assert_that(pcoll | 'MixedSides' >> beam.core.FlatMapTuple(fn, s2=side2),
+ equal_to(['e1', 'e2', MIN_TIMESTAMP, None, 's2']),
+ label='MixedSidesCheck')
+ pipeline.run()
+
def test_create_singleton_pcollection(self):
pipeline = TestPipeline()
pcoll = pipeline | 'label' >> Create([[1, 2, 3]])
@@ -376,7 +437,89 @@
| 'NoOp' >> beam.Map(lambda x: x))
p.replace_all([override])
- self.assertEquals(pcoll.producer.inputs[0].element_type, expected_type)
+ self.assertEqual(pcoll.producer.inputs[0].element_type, expected_type)
+
+ def test_kv_ptransform_honor_type_hints(self):
+
+ # The return type of this DoFn cannot be inferred by the default
+ # Beam type inference
+ class StatefulDoFn(DoFn):
+ BYTES_STATE = BagStateSpec('bytes', BytesCoder())
+
+ def return_recursive(self, count):
+ if count == 0:
+ return ["some string"]
+ else:
+ self.return_recursive(count-1)
+
+ def process(self, element, counter=DoFn.StateParam(BYTES_STATE)):
+ return self.return_recursive(1)
+
+ p = TestPipeline()
+ pcoll = (p
+ | beam.Create([(1, 1), (2, 2), (3, 3)])
+ | beam.GroupByKey()
+ | beam.ParDo(StatefulDoFn()))
+ p.run()
+ self.assertEqual(pcoll.element_type, typehints.Any)
+
+ p = TestPipeline()
+ pcoll = (p
+ | beam.Create([(1, 1), (2, 2), (3, 3)])
+ | beam.GroupByKey()
+ | beam.ParDo(StatefulDoFn()).with_output_types(str))
+ p.run()
+ self.assertEqual(pcoll.element_type, str)
+
+ def test_track_pcoll_unbounded(self):
+ pipeline = TestPipeline()
+ pcoll1 = pipeline | 'read' >> Read(FakeUnboundedSource())
+ pcoll2 = pcoll1 | 'do1' >> FlatMap(lambda x: [x + 1])
+ pcoll3 = pcoll2 | 'do2' >> FlatMap(lambda x: [x + 1])
+ self.assertIs(pcoll1.is_bounded, False)
+ self.assertIs(pcoll1.is_bounded, False)
+ self.assertIs(pcoll3.is_bounded, False)
+
+ def test_track_pcoll_bounded(self):
+ pipeline = TestPipeline()
+ pcoll1 = pipeline | 'label1' >> Create([1, 2, 3])
+ pcoll2 = pcoll1 | 'do1' >> FlatMap(lambda x: [x + 1])
+ pcoll3 = pcoll2 | 'do2' >> FlatMap(lambda x: [x + 1])
+ self.assertIs(pcoll1.is_bounded, True)
+ self.assertIs(pcoll2.is_bounded, True)
+ self.assertIs(pcoll3.is_bounded, True)
+
+ def test_track_pcoll_bounded_flatten(self):
+ pipeline = TestPipeline()
+ pcoll1_a = pipeline | 'label_a' >> Create([1, 2, 3])
+ pcoll2_a = pcoll1_a | 'do_a' >> FlatMap(lambda x: [x + 1])
+
+ pcoll1_b = pipeline | 'label_b' >> Create([1, 2, 3])
+ pcoll2_b = pcoll1_b | 'do_b' >> FlatMap(lambda x: [x + 1])
+
+ merged = (pcoll2_a, pcoll2_b) | beam.Flatten()
+
+ self.assertIs(pcoll1_a.is_bounded, True)
+ self.assertIs(pcoll2_a.is_bounded, True)
+ self.assertIs(pcoll1_b.is_bounded, True)
+ self.assertIs(pcoll2_b.is_bounded, True)
+ self.assertIs(merged.is_bounded, True)
+
+ def test_track_pcoll_unbounded_flatten(self):
+ pipeline = TestPipeline()
+ pcoll1_bounded = pipeline | 'label1' >> Create([1, 2, 3])
+ pcoll2_bounded = pcoll1_bounded | 'do1' >> FlatMap(lambda x: [x + 1])
+
+ pcoll1_unbounded = pipeline | 'read' >> Read(FakeUnboundedSource())
+ pcoll2_unbounded = pcoll1_unbounded | 'do2' >> FlatMap(lambda x: [x + 1])
+
+ merged = (pcoll2_bounded, pcoll2_unbounded) | beam.Flatten()
+
+ self.assertIs(pcoll1_bounded.is_bounded, True)
+ self.assertIs(pcoll2_bounded.is_bounded, True)
+ self.assertIs(pcoll1_unbounded.is_bounded, False)
+ self.assertIs(pcoll2_unbounded.is_bounded, False)
+ self.assertIs(merged.is_bounded, False)
class DoFnTest(unittest.TestCase):
@@ -459,6 +602,28 @@
p | Create([1, 2]) | beam.Map(lambda _, t=DoFn.TimestampParam: t),
equal_to([MIN_TIMESTAMP, MIN_TIMESTAMP]))
+ def test_incomparable_default(self):
+
+ class IncomparableType(object):
+
+ def __eq__(self, other):
+ raise RuntimeError()
+
+ def __ne__(self, other):
+ raise RuntimeError()
+
+ def __hash__(self):
+ raise RuntimeError()
+
+ # Ensure that we don't use default values in a context where they must be
+ # comparable (see BEAM-8301).
+ pipeline = TestPipeline()
+ pcoll = (pipeline
+ | beam.Create([None])
+ | Map(lambda e, x=IncomparableType(): (e, type(x).__name__)))
+ assert_that(pcoll, equal_to([(None, 'IncomparableType')]))
+ pipeline.run()
+
class Bacon(PipelineOptions):
@@ -482,29 +647,29 @@
def test_flag_parsing(self):
options = Breakfast(['--slices=3', '--style=sunny side up', '--ignored'])
- self.assertEquals(3, options.slices)
- self.assertEquals('sunny side up', options.style)
+ self.assertEqual(3, options.slices)
+ self.assertEqual('sunny side up', options.style)
def test_keyword_parsing(self):
options = Breakfast(
['--slices=3', '--style=sunny side up', '--ignored'],
slices=10)
- self.assertEquals(10, options.slices)
- self.assertEquals('sunny side up', options.style)
+ self.assertEqual(10, options.slices)
+ self.assertEqual('sunny side up', options.style)
def test_attribute_setting(self):
options = Breakfast(slices=10)
- self.assertEquals(10, options.slices)
+ self.assertEqual(10, options.slices)
options.slices = 20
- self.assertEquals(20, options.slices)
+ self.assertEqual(20, options.slices)
def test_view_as(self):
generic_options = PipelineOptions(['--slices=3'])
- self.assertEquals(3, generic_options.view_as(Bacon).slices)
- self.assertEquals(3, generic_options.view_as(Breakfast).slices)
+ self.assertEqual(3, generic_options.view_as(Bacon).slices)
+ self.assertEqual(3, generic_options.view_as(Breakfast).slices)
generic_options.view_as(Breakfast).slices = 10
- self.assertEquals(10, generic_options.view_as(Bacon).slices)
+ self.assertEqual(10, generic_options.view_as(Bacon).slices)
with self.assertRaises(AttributeError):
generic_options.slices # pylint: disable=pointless-statement
@@ -514,17 +679,17 @@
def test_defaults(self):
options = Breakfast(['--slices=3'])
- self.assertEquals(3, options.slices)
- self.assertEquals('scrambled', options.style)
+ self.assertEqual(3, options.slices)
+ self.assertEqual('scrambled', options.style)
def test_dir(self):
options = Breakfast()
- self.assertEquals(
+ self.assertEqual(
set(['from_dictionary', 'get_all_options', 'slices', 'style',
'view_as', 'display_data']),
set([attr for attr in dir(options) if not attr.startswith('_') and
attr != 'next']))
- self.assertEquals(
+ self.assertEqual(
set(['from_dictionary', 'get_all_options', 'style', 'view_as',
'display_data']),
set([attr for attr in dir(options.view_as(Eggs))
@@ -545,8 +710,8 @@
p = Pipeline.from_runner_api(
Pipeline.to_runner_api(p, use_fake_coders=True), None, None)
self.assertIsNotNone(p.transforms_stack[0].parts[0].parent)
- self.assertEquals(p.transforms_stack[0].parts[0].parent,
- p.transforms_stack[0])
+ self.assertEqual(p.transforms_stack[0].parts[0].parent,
+ p.transforms_stack[0])
class DirectRunnerRetryTests(unittest.TestCase):
diff --git a/sdks/python/apache_beam/portability/python_urns.py b/sdks/python/apache_beam/portability/python_urns.py
index 980ca68..358c9b3 100644
--- a/sdks/python/apache_beam/portability/python_urns.py
+++ b/sdks/python/apache_beam/portability/python_urns.py
@@ -38,8 +38,8 @@
EMBEDDED_PYTHON = "beam:env:embedded_python:v1"
# Invoke UserFns in process, but over GRPC channels.
-# Payload: (optional) Number of worker threads, as a decimal string.
-# (Used for testing.)
+# Payload: (optional) Number of worker threads, followed by ',' and the size of
+# the state cache, as a decimal string, e.g. '2,1000'.
EMBEDDED_PYTHON_GRPC = "beam:env:embedded_python_grpc:v1"
# Instantiate SDK harness via a command line provided in the payload.
diff --git a/sdks/python/apache_beam/pvalue.py b/sdks/python/apache_beam/pvalue.py
index 34cafd0..7c9d869 100644
--- a/sdks/python/apache_beam/pvalue.py
+++ b/sdks/python/apache_beam/pvalue.py
@@ -34,7 +34,6 @@
from past.builtins import unicode
-from apache_beam import coders
from apache_beam import typehints
from apache_beam.internal import pickler
from apache_beam.portability import common_urns
@@ -64,7 +63,8 @@
(3) Has a value which is meaningful if the transform was executed.
"""
- def __init__(self, pipeline, tag=None, element_type=None, windowing=None):
+ def __init__(self, pipeline, tag=None, element_type=None, windowing=None,
+ is_bounded=True):
"""Initializes a PValue with all arguments hidden behind keyword arguments.
Args:
@@ -79,6 +79,7 @@
# generating this PValue. The field gets initialized when a transform
# gets applied.
self.producer = None
+ self.is_bounded = is_bounded
if windowing:
self._windowing = windowing
@@ -143,11 +144,21 @@
# of a closure).
return _InvalidUnpickledPCollection, ()
+ @staticmethod
+ def from_(pcoll):
+ """Create a PCollection, using another PCollection as a starting point.
+
+ Transfers relevant attributes.
+ """
+ return PCollection(pcoll.pipeline, is_bounded=pcoll.is_bounded)
+
def to_runner_api(self, context):
return beam_runner_api_pb2.PCollection(
unique_name=self._unique_name(),
coder_id=context.coder_id_from_element_type(self.element_type),
- is_bounded=beam_runner_api_pb2.IsBounded.BOUNDED,
+ is_bounded=beam_runner_api_pb2.IsBounded.BOUNDED
+ if self.is_bounded
+ else beam_runner_api_pb2.IsBounded.UNBOUNDED,
windowing_strategy_id=context.windowing_strategies.get_id(
self.windowing))
@@ -166,7 +177,8 @@
None,
element_type=context.element_type_from_coder_id(proto.coder_id),
windowing=context.windowing_strategies.get_by_id(
- proto.windowing_strategy_id))
+ proto.windowing_strategy_id),
+ is_bounded=proto.is_bounded == beam_runner_api_pb2.IsBounded.BOUNDED)
class _InvalidUnpickledPCollection(object):
@@ -322,11 +334,6 @@
self._window_mapping_fn,
lambda iterable: from_runtime_iterable(iterable, view_options))
- def _input_element_coder(self):
- return coders.WindowedValueCoder(
- coders.registry.get_coder(self.pvalue.element_type),
- window_coder=self.pvalue.windowing.windowfn.get_window_coder())
-
def to_runner_api(self, context):
return self._side_input_data().to_runner_api(context)
@@ -335,6 +342,9 @@
return _UnpickledSideInput(
SideInputData.from_runner_api(proto, context))
+ def requires_keyed_input(self):
+ return False
+
class _UnpickledSideInput(AsSideInput):
def __init__(self, side_input_data):
@@ -400,9 +410,9 @@
Wrapping a PCollection side input argument to a PTransform in this container
(e.g., data.apply('label', MyPTransform(), AsSingleton(my_side_input) )
- selects the latter behavor.
+ selects the latter behavior.
- The input PCollection must contain exactly one value per window, unless a
+ The input PCollection must contain exactly one value per window, unless a
default is given, in which case it may be empty.
"""
_NO_DEFAULT = object()
@@ -547,6 +557,9 @@
self._window_mapping_fn,
lambda x: x)
+ def requires_keyed_input(self):
+ return True
+
class EmptySideInput(object):
"""Value indicating when a singleton side input was empty.
diff --git a/sdks/python/apache_beam/runners/common.pxd b/sdks/python/apache_beam/runners/common.pxd
index 1d87507..2ffe432 100644
--- a/sdks/python/apache_beam/runners/common.pxd
+++ b/sdks/python/apache_beam/runners/common.pxd
@@ -37,6 +37,11 @@
cdef bint has_userstate_arguments
cdef object state_args_to_replace
cdef object timer_args_to_replace
+ cdef object timestamp_arg_name
+ cdef object window_arg_name
+ cdef object key_arg_name
+ cdef object restriction_provider
+ cdef object restriction_provider_arg_name
cdef class DoFnSignature(object):
@@ -88,6 +93,7 @@
cdef bint is_splittable
cdef object restriction_tracker
cdef WindowedValue current_windowed_value
+ cdef bint is_key_param_required
cdef class DoFnRunner(Receiver):
diff --git a/sdks/python/apache_beam/runners/common.py b/sdks/python/apache_beam/runners/common.py
index cae0d4c..541959a 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -44,6 +44,7 @@
from apache_beam.transforms.window import WindowFn
from apache_beam.utils.counters import Counter
from apache_beam.utils.counters import CounterName
+from apache_beam.utils.timestamp import Timestamp
from apache_beam.utils.windowed_value import WindowedValue
@@ -152,38 +153,59 @@
'a \'RestrictionProvider\'. Received %r instead.'
% obj_to_invoke)
- fullargspec = core.get_function_arguments(
- obj_to_invoke, method_name)
+ self.args, self.defaults = core.get_function_arguments(obj_to_invoke,
+ method_name)
# TODO(BEAM-5878) support kwonlyargs on Python 3.
- args = fullargspec[0]
- defaults = fullargspec[3]
-
- defaults = defaults if defaults else []
- method_value = getattr(obj_to_invoke, method_name)
- self.method_value = method_value
- self.args = args
- self.defaults = defaults
+ self.method_value = getattr(obj_to_invoke, method_name)
self.has_userstate_arguments = False
self.state_args_to_replace = {}
self.timer_args_to_replace = {}
- for kw, v in zip(args[-len(defaults):], defaults):
+ self.timestamp_arg_name = None
+ self.window_arg_name = None
+ self.key_arg_name = None
+ self.restriction_provider = None
+ self.restriction_provider_arg_name = None
+
+ for kw, v in zip(self.args[-len(self.defaults):], self.defaults):
if isinstance(v, core.DoFn.StateParam):
self.state_args_to_replace[kw] = v.state_spec
self.has_userstate_arguments = True
elif isinstance(v, core.DoFn.TimerParam):
self.timer_args_to_replace[kw] = v.timer_spec
self.has_userstate_arguments = True
+ elif core.DoFn.TimestampParam == v:
+ self.timestamp_arg_name = kw
+ elif core.DoFn.WindowParam == v:
+ self.window_arg_name = kw
+ elif core.DoFn.KeyParam == v:
+ self.key_arg_name = kw
+ elif isinstance(v, core.DoFn.RestrictionParam):
+ self.restriction_provider = v.restriction_provider
+ self.restriction_provider_arg_name = kw
- def invoke_timer_callback(self, user_state_context, key, window):
- # TODO(ccy): support WindowParam, TimestampParam and side inputs.
+ def invoke_timer_callback(self,
+ user_state_context,
+ key,
+ window,
+ timestamp):
+ # TODO(ccy): support side inputs.
+ kwargs = {}
if self.has_userstate_arguments:
- kwargs = {}
for kw, state_spec in self.state_args_to_replace.items():
kwargs[kw] = user_state_context.get_state(state_spec, key, window)
for kw, timer_spec in self.timer_args_to_replace.items():
kwargs[kw] = user_state_context.get_timer(timer_spec, key, window)
+
+ if self.timestamp_arg_name:
+ kwargs[self.timestamp_arg_name] = Timestamp(seconds=timestamp)
+ if self.window_arg_name:
+ kwargs[self.window_arg_name] = window
+ if self.key_arg_name:
+ kwargs[self.key_arg_name] = key
+
+ if kwargs:
return self.method_value(**kwargs)
else:
return self.method_value()
@@ -240,9 +262,7 @@
self.timer_methods[timer_spec] = MethodWrapper(do_fn, method.__name__)
def get_restriction_provider(self):
- result = _find_param_with_default(self.process_method,
- default_as_type=DoFn.RestrictionParam)
- return result[1].restriction_provider if result else None
+ return self.process_method.restriction_provider
def _validate(self):
self._validate_process()
@@ -273,8 +293,7 @@
userstate.validate_stateful_dofn(self.do_fn)
def is_splittable_dofn(self):
- return any([isinstance(default, DoFn.RestrictionParam) for default in
- self.process_method.defaults])
+ return self.get_restriction_provider() is not None
def is_stateful_dofn(self):
return self._is_stateful_dofn
@@ -291,6 +310,13 @@
represented by a given DoFnSignature."""
def __init__(self, output_processor, signature):
+ """
+ Initializes `DoFnInvoker`
+
+ :param output_processor: an OutputProcessor for receiving elements produced
+ by invoking functions of the DoFn.
+ :param signature: a DoFnSignature for the DoFn being invoked
+ """
self.output_processor = output_processor
self.signature = signature
self.user_state_context = None
@@ -384,7 +410,7 @@
self.output_processor.process_outputs(
WindowedValue(None, timestamp, (window,)),
self.signature.timer_methods[timer_spec].invoke_timer_callback(
- self.user_state_context, key, window))
+ self.user_state_context, key, window, timestamp))
def invoke_split(self, element, restriction):
return self.signature.split_method.method_value(element, restriction)
@@ -399,26 +425,6 @@
return self.signature.create_tracker_method.method_value(restriction)
-def _find_param_with_default(
- method, default_as_value=None, default_as_type=None):
- if ((default_as_value and default_as_type) or
- not (default_as_value or default_as_type)):
- raise ValueError(
- 'Exactly one of \'default_as_value\' and \'default_as_type\' should be '
- 'provided. Received %r and %r.' % (default_as_value, default_as_type))
-
- defaults = method.defaults
- ret = None
- for i, value in enumerate(defaults):
- if default_as_value and value == default_as_value:
- ret = (method.args[len(method.args) - len(defaults) + i], value)
- elif default_as_type and isinstance(value, default_as_type):
- index = len(method.args) - len(defaults) + i
- ret = (method.args[index], value)
-
- return ret
-
-
class SimpleInvoker(DoFnInvoker):
"""An invoker that processes elements ignoring windowing information."""
@@ -455,6 +461,7 @@
self.restriction_tracker = None
self.current_windowed_value = None
self.bundle_finalizer_param = bundle_finalizer_param
+ self.is_key_param_required = False
# Try to prepare all the arguments that can just be filled in
# without any additional work. in the process function.
@@ -467,34 +474,40 @@
input_args = input_args if input_args else []
input_kwargs = input_kwargs if input_kwargs else {}
- arguments = signature.process_method.args
- defaults = signature.process_method.defaults
+ arg_names = signature.process_method.args
# Create placeholder for element parameter of DoFn.process() method.
- self_in_args = int(signature.do_fn.is_process_bounded())
-
+ # Not to be confused with ArgumentPlaceHolder, which may be passed in
+ # input_args and is a placeholder for side-inputs.
class ArgPlaceholder(object):
def __init__(self, placeholder):
self.placeholder = placeholder
if core.DoFn.ElementParam not in default_arg_values:
- args_to_pick = len(arguments) - len(default_arg_values) - 1 - self_in_args
+ # TODO(BEAM-7867): Handle cases in which len(arg_names) ==
+ # len(default_arg_values).
+ args_to_pick = len(arg_names) - len(default_arg_values) - 1
+ # Positional argument values for process(), with placeholders for special
+ # values such as the element, timestamp, etc.
args_with_placeholders = (
[ArgPlaceholder(core.DoFn.ElementParam)] + input_args[:args_to_pick])
else:
- args_to_pick = len(arguments) - len(defaults) - self_in_args
+ args_to_pick = len(arg_names) - len(default_arg_values)
args_with_placeholders = input_args[:args_to_pick]
- # Fill the OtherPlaceholders for context, window or timestamp
+ # Fill the OtherPlaceholders for context, key, window or timestamp
remaining_args_iter = iter(input_args[args_to_pick:])
- for a, d in zip(arguments[-len(defaults):], defaults):
- if d == core.DoFn.ElementParam:
+ for a, d in zip(arg_names[-len(default_arg_values):], default_arg_values):
+ if core.DoFn.ElementParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.WindowParam:
+ elif core.DoFn.KeyParam == d:
+ self.is_key_param_required = True
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.TimestampParam:
+ elif core.DoFn.WindowParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.SideInputParam:
+ elif core.DoFn.TimestampParam == d:
+ args_with_placeholders.append(ArgPlaceholder(d))
+ elif core.DoFn.SideInputParam == d:
# If no more args are present then the value must be passed via kwarg
try:
args_with_placeholders.append(next(remaining_args_iter))
@@ -505,7 +518,7 @@
args_with_placeholders.append(ArgPlaceholder(d))
elif isinstance(d, core.DoFn.TimerParam):
args_with_placeholders.append(ArgPlaceholder(d))
- elif d == core.DoFn.BundleFinalizerParam:
+ elif isinstance(d, type) and core.DoFn.BundleFinalizerParam == d:
args_with_placeholders.append(ArgPlaceholder(d))
else:
# If no more args are present then the value must be passed via kwarg
@@ -548,9 +561,8 @@
# the upstream pair-with-restriction.
raise NotImplementedError(
'SDFs in multiply-windowed values with windowed arguments.')
- restriction_tracker_param = _find_param_with_default(
- self.signature.process_method,
- default_as_type=DoFn.RestrictionParam)[0]
+ restriction_tracker_param = (
+ self.signature.process_method.restriction_provider_arg_name)
if not restriction_tracker_param:
raise ValueError(
'A RestrictionTracker %r was provided but DoFn does not have a '
@@ -606,21 +618,22 @@
# stateful DoFn, we set during __init__ self.has_windowed_inputs to be
# True. Therefore, windows will be exploded coming into this method, and
# we can rely on the window variable being set above.
- if self.user_state_context:
+ if self.user_state_context or self.is_key_param_required:
try:
key, unused_value = windowed_value.value
except (TypeError, ValueError):
raise ValueError(
- ('Input value to a stateful DoFn must be a KV tuple; instead, '
- 'got %s.') % (windowed_value.value,))
+ ('Input value to a stateful DoFn or KeyParam must be a KV tuple; '
+ 'instead, got \'%s\'.') % (windowed_value.value,))
- # TODO(sourabhbajaj): Investigate why we can't use `is` instead of ==
for i, p in self.placeholders:
- if p == core.DoFn.ElementParam:
+ if core.DoFn.ElementParam == p:
args_for_process[i] = windowed_value.value
- elif p == core.DoFn.WindowParam:
+ elif core.DoFn.KeyParam == p:
+ args_for_process[i] = key
+ elif core.DoFn.WindowParam == p:
args_for_process[i] = window
- elif p == core.DoFn.TimestampParam:
+ elif core.DoFn.TimestampParam == p:
args_for_process[i] = windowed_value.timestamp
elif isinstance(p, core.DoFn.StateParam):
args_for_process[i] = (
@@ -628,7 +641,7 @@
elif isinstance(p, core.DoFn.TimerParam):
args_for_process[i] = (
self.user_state_context.get_timer(p.timer_spec, key, window))
- elif p == core.DoFn.BundleFinalizerParam:
+ elif core.DoFn.BundleFinalizerParam == p:
args_for_process[i] = self.bundle_finalizer_param
if additional_kwargs:
@@ -661,6 +674,10 @@
restriction_tracker = self.restriction_tracker
current_windowed_value = self.current_windowed_value
if restriction_tracker and current_windowed_value:
+ # Temporary workaround for [BEAM-7473]: get current_watermark before
+ # split, in case watermark gets advanced before getting split results.
+ # In worst case, current_watermark is always stale, which is ok.
+ current_watermark = restriction_tracker.current_watermark()
split = restriction_tracker.try_split(fraction)
if split:
primary, residual = split
@@ -674,7 +691,7 @@
None),
(self.current_windowed_value.with_value(
((element, residual), residual_size)),
- restriction_tracker.current_watermark()))
+ current_watermark))
def current_element_progress(self):
restriction_tracker = self.restriction_tracker
diff --git a/sdks/python/apache_beam/runners/common_test.py b/sdks/python/apache_beam/runners/common_test.py
index 18e2c45..9377708 100644
--- a/sdks/python/apache_beam/runners/common_test.py
+++ b/sdks/python/apache_beam/runners/common_test.py
@@ -19,7 +19,13 @@
import unittest
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.runners.common import DoFnSignature
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_stream import TestStream
+from apache_beam.transforms import trigger
+from apache_beam.transforms import window
from apache_beam.transforms.core import DoFn
@@ -56,5 +62,58 @@
DoFnSignature(MyDoFn())
+class DoFnProcessTest(unittest.TestCase):
+ # pylint: disable=expression-not-assigned
+ all_records = None
+
+ def setUp(self):
+ DoFnProcessTest.all_records = []
+
+ def record_dofn(self):
+ class RecordDoFn(DoFn):
+ def process(self, element):
+ DoFnProcessTest.all_records.append(element)
+
+ return RecordDoFn()
+
+ def test_dofn_process_keyparam(self):
+
+ class DoFnProcessWithKeyparam(DoFn):
+
+ def process(self, element, mykey=DoFn.KeyParam):
+ yield "{key}-verify".format(key=mykey)
+
+ pipeline_options = PipelineOptions()
+
+ with TestPipeline(options=pipeline_options) as p:
+ test_stream = (TestStream().advance_watermark_to(10).add_elements([1, 2]))
+ (p
+ | test_stream
+ | beam.Map(lambda x: (x, "some-value"))
+ | "window_into" >> beam.WindowInto(
+ window.FixedWindows(5),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING)
+ | beam.ParDo(DoFnProcessWithKeyparam())
+ | beam.ParDo(self.record_dofn()))
+
+ self.assertEqual(
+ ['1-verify', '2-verify'],
+ sorted(DoFnProcessTest.all_records))
+
+ def test_dofn_process_keyparam_error_no_key(self):
+ class DoFnProcessWithKeyparam(DoFn):
+
+ def process(self, element, mykey=DoFn.KeyParam):
+ yield "{key}-verify".format(key=mykey)
+
+ pipeline_options = PipelineOptions()
+ with self.assertRaises(ValueError),\
+ TestPipeline(options=pipeline_options) as p:
+ test_stream = (TestStream().advance_watermark_to(10).add_elements([1, 2]))
+ (p
+ | test_stream
+ | beam.ParDo(DoFnProcessWithKeyparam()))
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py
index a77497f..f0d6df9 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py
@@ -34,7 +34,7 @@
'dataflow_exercise_metrics_pipeline.UserMetricsDoFn')
-def common_metric_matchers():
+def metric_matchers():
"""MetricResult matchers common to all tests."""
# TODO(ajamato): Matcher for the 'metrics' step's ElementCount.
# TODO(ajamato): Matcher for the 'metrics' step's MeanByteCount.
@@ -66,54 +66,7 @@
step='metrics',
attempted=greater_than(0),
committed=greater_than(0)
- )
- ]
-
- pcoll_names = [
- 'GroupByKey/Reify-out0',
- 'GroupByKey/Read-out0',
- 'map_to_common_key-out0',
- 'GroupByKey/GroupByWindow-out0',
- 'GroupByKey/Read-out0',
- 'GroupByKey/Reify-out0'
- ]
- for name in pcoll_names:
- matchers.extend([
- MetricResultMatcher(
- name='ElementCount',
- labels={
- 'output_user_name': name,
- 'original_name': '%s-ElementCount' % name
- },
- attempted=greater_than(0),
- committed=greater_than(0)
- ),
- MetricResultMatcher(
- name='MeanByteCount',
- labels={
- 'output_user_name': name,
- 'original_name': '%s-MeanByteCount' % name
- },
- attempted=greater_than(0),
- committed=greater_than(0)
- ),
- ])
- return matchers
-
-
-def fn_api_metric_matchers():
- """MetricResult matchers with adjusted step names for the FN API DF test."""
- matchers = common_metric_matchers()
- return matchers
-
-
-def legacy_metric_matchers():
- """MetricResult matchers with adjusted step names for the legacy DF test."""
- # TODO(ajamato): Move these to the common_metric_matchers once implemented
- # in the FN API.
- matchers = common_metric_matchers()
- matchers.extend([
- # User distribution metric, legacy DF only.
+ ),
MetricResultMatcher(
name='distribution_values',
namespace=METRIC_NAMESPACE,
@@ -149,8 +102,38 @@
},
attempted=greater_than(0),
committed=greater_than(0)
- ),
- ])
+ )
+ ]
+
+ pcoll_names = [
+ 'GroupByKey/Reify-out0',
+ 'GroupByKey/Read-out0',
+ 'map_to_common_key-out0',
+ 'GroupByKey/GroupByWindow-out0',
+ 'GroupByKey/Read-out0',
+ 'GroupByKey/Reify-out0'
+ ]
+ for name in pcoll_names:
+ matchers.extend([
+ MetricResultMatcher(
+ name='ElementCount',
+ labels={
+ 'output_user_name': name,
+ 'original_name': '%s-ElementCount' % name
+ },
+ attempted=greater_than(0),
+ committed=greater_than(0)
+ ),
+ MetricResultMatcher(
+ name='MeanByteCount',
+ labels={
+ 'output_user_name': name,
+ 'original_name': '%s-MeanByteCount' % name
+ },
+ attempted=greater_than(0),
+ committed=greater_than(0)
+ ),
+ ])
return matchers
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
index c62824d..d1afbcf 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py
@@ -26,7 +26,6 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.runners.dataflow import dataflow_exercise_metrics_pipeline
from apache_beam.testing import metric_result_matchers
from apache_beam.testing.test_pipeline import TestPipeline
@@ -40,10 +39,7 @@
parser = argparse.ArgumentParser()
unused_known_args, pipeline_args = parser.parse_known_args(argv)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
return dataflow_exercise_metrics_pipeline.apply_and_run(p)
@@ -52,7 +48,7 @@
result = self.run_pipeline()
errors = metric_result_matchers.verify_all(
result.metrics().all_metrics(),
- dataflow_exercise_metrics_pipeline.legacy_metric_matchers())
+ dataflow_exercise_metrics_pipeline.metric_matchers())
self.assertFalse(errors, str(errors))
@attr('IT', 'ValidatesContainer')
@@ -60,7 +56,7 @@
result = self.run_pipeline(experiment='beam_fn_api')
errors = metric_result_matchers.verify_all(
result.metrics().all_metrics(),
- dataflow_exercise_metrics_pipeline.fn_api_metric_matchers())
+ dataflow_exercise_metrics_pipeline.metric_matchers())
self.assertFalse(errors, str(errors))
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline.py b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline.py
new file mode 100644
index 0000000..fffd413
--- /dev/null
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline.py
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A word-counting workflow."""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import time
+
+import apache_beam as beam
+from apache_beam.metrics import Metrics
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.options.pipeline_options import StandardOptions
+
+SLEEP_TIME_SECS = 1
+
+
+class StreamingUserMetricsDoFn(beam.DoFn):
+ """Generates user metrics and outputs same element."""
+
+ def __init__(self):
+ self.double_message_counter = Metrics.counter(self.__class__,
+ 'double_msg_counter_name')
+ self.msg_len_dist_metric = Metrics.distribution(
+ self.__class__, 'msg_len_dist_metric_name')
+
+ def start_bundle(self):
+ time.sleep(SLEEP_TIME_SECS)
+
+ def process(self, element):
+ """Returns the processed element and increments the metrics."""
+
+ text_line = element.strip()
+
+ self.double_message_counter.inc()
+ self.double_message_counter.inc()
+ self.msg_len_dist_metric.update(len(text_line))
+
+ logging.debug("Done processing returning element array: '%s'", element)
+
+ return [element]
+
+ def finish_bundle(self):
+ time.sleep(SLEEP_TIME_SECS)
+
+
+def run(argv=None):
+ """Given an initialized Pipeline applies transforms and runs it."""
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ '--output_topic', required=True,
+ help=('Output PubSub topic of the form '
+ '"projects/<PROJECT>/topic/<TOPIC>".'))
+ group = parser.add_mutually_exclusive_group(required=True)
+ group.add_argument(
+ '--input_topic',
+ help=('Input PubSub topic of the form '
+ '"projects/<PROJECT>/topics/<TOPIC>".'))
+ group.add_argument(
+ '--input_subscription',
+ help=('Input PubSub subscription of the form '
+ '"projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."'))
+ known_args, pipeline_args = parser.parse_known_args(argv)
+
+ pipeline_options = PipelineOptions(pipeline_args)
+ pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(StandardOptions).streaming = True
+ pipeline = beam.Pipeline(options=pipeline_options)
+
+ _ = (pipeline
+ | beam.io.ReadFromPubSub(subscription=known_args.input_subscription)
+ | 'generate_metrics' >> (beam.ParDo(StreamingUserMetricsDoFn()))
+ | 'dump_to_pub' >> beam.io.WriteToPubSub(known_args.output_topic))
+
+ result = pipeline.run()
+ result.wait_until_finish()
+ return result
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ run()
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py
new file mode 100644
index 0000000..a2b0928
--- /dev/null
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py
@@ -0,0 +1,177 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A word-counting workflow."""
+
+from __future__ import absolute_import
+
+import logging
+import unittest
+import uuid
+
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
+
+from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
+from apache_beam.runners.dataflow import dataflow_exercise_streaming_metrics_pipeline
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing import metric_result_matchers
+from apache_beam.testing import test_utils
+from apache_beam.testing.metric_result_matchers import DistributionMatcher
+from apache_beam.testing.metric_result_matchers import MetricResultMatcher
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+INPUT_TOPIC = 'exercise_streaming_metrics_topic_input'
+INPUT_SUB = 'exercise_streaming_metrics_subscription_input'
+OUTPUT_TOPIC = 'exercise_streaming_metrics_topic_output'
+OUTPUT_SUB = 'exercise_streaming_metrics_subscription_output'
+
+WAIT_UNTIL_FINISH_DURATION = 1 * 60 * 1000 # in milliseconds
+MESSAGES_TO_PUBLISH = ["message a", "message b b", "message c"]
+
+SLEEP_TIME_SECS = 1
+
+
+class ExerciseStreamingMetricsPipelineTest(unittest.TestCase):
+
+ def setUp(self):
+ """Creates all required topics and subs."""
+ self.test_pipeline = TestPipeline(is_integration_test=True)
+ self.project = self.test_pipeline.get_option('project')
+ self.uuid = str(uuid.uuid4())
+
+ # Set up PubSub environment.
+ from google.cloud import pubsub
+ self.pub_client = pubsub.PublisherClient()
+ self.input_topic_name = INPUT_TOPIC + self.uuid
+ self.input_topic = self.pub_client.create_topic(
+ self.pub_client.topic_path(self.project, self.input_topic_name))
+
+ self.output_topic_name = OUTPUT_TOPIC + self.uuid
+ self.output_topic = self.pub_client.create_topic(
+ self.pub_client.topic_path(self.project, self.output_topic_name))
+
+ self.sub_client = pubsub.SubscriberClient()
+ self.input_sub_name = INPUT_SUB + self.uuid
+ self.input_sub = self.sub_client.create_subscription(
+ self.sub_client.subscription_path(self.project, self.input_sub_name),
+ self.input_topic.name)
+ self.output_sub_name = OUTPUT_SUB + self.uuid
+ self.output_sub = self.sub_client.create_subscription(
+ self.sub_client.subscription_path(self.project, self.output_sub_name),
+ self.output_topic.name,
+ ack_deadline_seconds=60)
+
+ def _inject_words(self, topic, messages):
+ """Inject messages as test data to PubSub."""
+ logging.debug('Injecting messages to topic %s', topic.name)
+ for msg in messages:
+ self.pub_client.publish(self.input_topic.name, msg.encode('utf-8'))
+ logging.debug('Done. Injecting messages to topic %s', topic.name)
+
+ def tearDown(self):
+ """Delete all created topics and subs."""
+ test_utils.cleanup_subscriptions(self.sub_client,
+ [self.input_sub, self.output_sub])
+ test_utils.cleanup_topics(self.pub_client,
+ [self.input_topic, self.output_topic])
+
+ def run_pipeline(self):
+ # Waits for messages to appear in output topic.
+ expected_msg = [msg.encode('utf-8') for msg in MESSAGES_TO_PUBLISH]
+ pubsub_msg_verifier = PubSubMessageMatcher(self.project,
+ self.output_sub.name,
+ expected_msg,
+ timeout=600)
+
+ # Checks that pipeline initializes to RUNNING state.
+ state_verifier = PipelineStateMatcher(PipelineState.RUNNING)
+
+ extra_opts = {'wait_until_finish_duration': WAIT_UNTIL_FINISH_DURATION,
+ 'on_success_matcher': all_of(state_verifier,
+ pubsub_msg_verifier),
+ 'experiment': 'beam_fn_api',
+ 'input_subscription': self.input_sub.name,
+ 'output_topic': self.output_topic.name,
+ }
+
+ argv = self.test_pipeline.get_full_options_as_args(**extra_opts)
+ return dataflow_exercise_streaming_metrics_pipeline.run(argv)
+
+ @attr('IT', 'ValidatesRunner')
+ def test_streaming_pipeline_returns_expected_user_metrics_fnapi_it(self):
+ """
+ Runs streaming Dataflow job and verifies that user metrics are reported
+ correctly.
+ """
+ self._inject_words(self.input_topic, MESSAGES_TO_PUBLISH)
+ result = self.run_pipeline()
+
+ METRIC_NAMESPACE = \
+ ('apache_beam.runners.dataflow.'
+ 'dataflow_exercise_streaming_metrics_pipeline.StreamingUserMetricsDoFn')
+ matchers = [
+ # System metrics
+ MetricResultMatcher(
+ name='ElementCount',
+ labels={"output_user_name": "generate_metrics-out0",
+ "original_name": "generate_metrics-out0-ElementCount"},
+ attempted=len(MESSAGES_TO_PUBLISH),
+ committed=len(MESSAGES_TO_PUBLISH),
+ ),
+ MetricResultMatcher(
+ name='ElementCount',
+ labels={"output_user_name": "ReadFromPubSub/Read-out0",
+ "original_name": "ReadFromPubSub/Read-out0-ElementCount"},
+ attempted=len(MESSAGES_TO_PUBLISH),
+ committed=len(MESSAGES_TO_PUBLISH),
+ ),
+ # User Counter Metrics.
+ MetricResultMatcher(
+ name='double_msg_counter_name',
+ namespace=METRIC_NAMESPACE,
+ step='generate_metrics',
+ attempted=len(MESSAGES_TO_PUBLISH) * 2,
+ committed=len(MESSAGES_TO_PUBLISH) * 2
+ ),
+ MetricResultMatcher(
+ name='msg_len_dist_metric_name',
+ namespace=METRIC_NAMESPACE,
+ step='generate_metrics',
+ attempted=DistributionMatcher(
+ sum_value=len(''.join(MESSAGES_TO_PUBLISH)),
+ count_value=len(MESSAGES_TO_PUBLISH),
+ min_value=len(MESSAGES_TO_PUBLISH[0]),
+ max_value=len(MESSAGES_TO_PUBLISH[1])
+ ),
+ committed=DistributionMatcher(
+ sum_value=len(''.join(MESSAGES_TO_PUBLISH)),
+ count_value=len(MESSAGES_TO_PUBLISH),
+ min_value=len(MESSAGES_TO_PUBLISH[0]),
+ max_value=len(MESSAGES_TO_PUBLISH[1])
+ )
+ ),
+ ]
+
+ metrics = result.metrics().all_metrics()
+ errors = metric_result_matchers.verify_all(metrics, matchers)
+ self.assertFalse(errors, str(errors))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
index 741e944..3a07e65 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py
@@ -154,14 +154,14 @@
# Get the tentative/committed versions of every metric together.
metrics_by_name = defaultdict(lambda: {})
for metric in metrics:
- if (metric.name.name.endswith('[MIN]') or
- metric.name.name.endswith('[MAX]') or
- metric.name.name.endswith('[MEAN]') or
- metric.name.name.endswith('[COUNT]')):
+ if (metric.name.name.endswith('_MIN') or
+ metric.name.name.endswith('_MAX') or
+ metric.name.name.endswith('_MEAN') or
+ metric.name.name.endswith('_COUNT')):
# The Dataflow Service presents distribution metrics in two ways:
# One way is as a single distribution object with all its fields, and
- # another way is as four different scalar metrics labeled as [MIN],
- # [MAX], [COUNT], [MEAN].
+ # another way is as four different scalar metrics labeled as _MIN,
+ # _MAX, _COUNT_, _MEAN.
# TODO(pabloem) remove these when distributions are not being broken up
# in the service.
# The second way is only useful for the UI, and should be ignored.
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 1a037f6..ca96c18 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -25,6 +25,7 @@
import json
import logging
+import sys
import threading
import time
import traceback
@@ -94,11 +95,16 @@
# Imported here to avoid circular dependencies.
# TODO: Remove the apache_beam.pipeline dependency in CreatePTransformOverride
from apache_beam.runners.dataflow.ptransform_overrides import CreatePTransformOverride
+ from apache_beam.runners.dataflow.ptransform_overrides import ReadPTransformOverride
_PTRANSFORM_OVERRIDES = [
CreatePTransformOverride(),
]
+ _SDF_PTRANSFORM_OVERRIDES = [
+ ReadPTransformOverride(),
+ ]
+
def __init__(self, cache=None):
# Cache of CloudWorkflowStep protos generated while the runner
# "executes" a pipeline.
@@ -304,7 +310,8 @@
new_side_input.pvalue = beam.pvalue.PCollection(
pipeline,
element_type=typehints.KV[
- bytes, side_input.pvalue.element_type])
+ bytes, side_input.pvalue.element_type],
+ is_bounded=side_input.pvalue.is_bounded)
parent = transform_node.parent or pipeline._root_transform()
map_to_void_key = beam.pipeline.AppliedPTransform(
pipeline,
@@ -363,20 +370,29 @@
'please install apache_beam[gcp]')
# Convert all side inputs into a form acceptable to Dataflow.
- if apiclient._use_fnapi(options) and (
- not apiclient._use_unified_worker(options)):
+ if apiclient._use_fnapi(options):
pipeline.visit(self.side_input_visitor())
# Performing configured PTransform overrides. Note that this is currently
# done before Runner API serialization, since the new proto needs to contain
# any added PTransforms.
pipeline.replace_all(DataflowRunner._PTRANSFORM_OVERRIDES)
+ if apiclient._use_sdf_bounded_source(options):
+ pipeline.replace_all(DataflowRunner._SDF_PTRANSFORM_OVERRIDES)
+
+ use_fnapi = apiclient._use_fnapi(options)
+ from apache_beam.portability.api import beam_runner_api_pb2
+ default_environment = beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image=apiclient.get_container_image_from_options(options)
+ ).SerializeToString())
# Snapshot the pipeline in a portable proto.
self.proto_pipeline, self.proto_context = pipeline.to_runner_api(
- return_context=True)
+ return_context=True, default_environment=default_environment)
- if apiclient._use_fnapi(options):
+ if use_fnapi:
# Cross language transform require using a pipeline object constructed
# from the full pipeline proto to make sure that expanded version of
# external transforms are reflected in the Pipeline job graph.
@@ -391,7 +407,7 @@
# We need to generate a new context that maps to the new pipeline object.
self.proto_pipeline, self.proto_context = pipeline.to_runner_api(
- return_context=True)
+ return_context=True, default_environment=default_environment)
# Add setup_options for all the BeamPlugin imports
setup_options = options.view_as(SetupOptions)
@@ -431,6 +447,14 @@
else:
debug_options.add_experiment('use_staged_dataflow_worker_jar')
+ # Make Dataflow workers use FastAvro on Python 3 unless use_avro experiment
+ # is set. Note that use_avro is only interpreted by the Dataflow runner
+ # at job submission and is not interpreted by Dataflow service or workers,
+ # which by default use avro library unless use_fastavro experiment is set.
+ if sys.version_info[0] > 2 and (
+ not debug_options.lookup_experiment('use_avro')):
+ debug_options.add_experiment('use_fastavro')
+
self.job = apiclient.Job(options, self.proto_pipeline)
# Dataflow runner requires a KV type for GBK inputs, hence we enforce that
@@ -646,6 +670,9 @@
if (not isinstance(transform, beam.io.WriteToBigQuery)
or 'use_beam_bq_sink' in experiments):
return self.apply_PTransform(transform, pcoll, options)
+ if transform.schema == beam.io.gcp.bigquery.SCHEMA_AUTODETECT:
+ raise RuntimeError(
+ 'Schema auto-detection is not supported on the native sink')
standard_options = options.view_as(StandardOptions)
if standard_options.streaming:
if (transform.write_disposition ==
@@ -654,12 +681,15 @@
return self.apply_PTransform(transform, pcoll, options)
else:
from apache_beam.io.gcp.bigquery_tools import parse_table_schema_from_json
+ schema = None
+ if transform.schema:
+ schema = parse_table_schema_from_json(json.dumps(transform.schema))
return pcoll | 'WriteToBigQuery' >> beam.io.Write(
beam.io.BigQuerySink(
transform.table_reference.tableId,
transform.table_reference.datasetId,
transform.table_reference.projectId,
- parse_table_schema_from_json(json.dumps(transform.schema)),
+ schema,
transform.create_disposition,
transform.write_disposition,
kms_key=transform.kms_key))
@@ -682,7 +712,7 @@
coders.registry.verify_deterministic(
coder.key_coder(), 'GroupByKey operation "%s"' % transform.label)
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
def run_GroupByKey(self, transform_node, options):
input_tag = transform_node.inputs[0].tag
@@ -827,13 +857,31 @@
outputs = []
step.encoding = self._get_encoded_output_coder(transform_node)
+ all_output_tags = transform_proto.outputs.keys()
+
+ from apache_beam.transforms.core import RunnerAPIPTransformHolder
+ external_transform = isinstance(transform, RunnerAPIPTransformHolder)
+
+ # Some external transforms require output tags to not be modified.
+ # So we randomly select one of the output tags as the main output and
+ # leave others as side outputs. Transform execution should not change
+ # dependending on which output tag we choose as the main output here.
+ # Also, some SDKs do not work correctly if output tags are modified. So for
+ # external transforms, we leave tags unmodified.
+ main_output_tag = (
+ all_output_tags[0] if external_transform else PropertyNames.OUT)
+
+ # Python SDK uses 'None' as the tag of the main output.
+ tag_to_ignore = main_output_tag if external_transform else 'None'
+ side_output_tags = set(all_output_tags).difference({tag_to_ignore})
+
# Add the main output to the description.
outputs.append(
{PropertyNames.USER_NAME: (
'%s.%s' % (transform_node.full_label, PropertyNames.OUT)),
PropertyNames.ENCODING: step.encoding,
- PropertyNames.OUTPUT_NAME: PropertyNames.OUT})
- for side_tag in transform.output_tags:
+ PropertyNames.OUTPUT_NAME: main_output_tag})
+ for side_tag in side_output_tags:
# The assumption here is that all outputs will have the same typehint
# and coder as the main output. This is certainly the case right now
# but conceivably it could change in the future.
@@ -842,7 +890,8 @@
'%s.%s' % (transform_node.full_label, side_tag)),
PropertyNames.ENCODING: step.encoding,
PropertyNames.OUTPUT_NAME: (
- '%s_%s' % (PropertyNames.OUT, side_tag))})
+ side_tag if external_transform
+ else '%s_%s' % (PropertyNames.OUT, side_tag))})
step.add_property(PropertyNames.OUTPUT_INFO, outputs)
@@ -864,7 +913,7 @@
transform_node.inputs[0].windowing)
def apply_CombineValues(self, transform, pcoll, options):
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
def run_CombineValues(self, transform_node, options):
transform = transform_node.transform
@@ -917,7 +966,7 @@
def apply_Read(self, transform, pbegin, options):
if hasattr(transform.source, 'format'):
# Consider native Read to be a primitive for dataflow.
- return beam.pvalue.PCollection(pbegin.pipeline)
+ return beam.pvalue.PCollection.from_(pbegin)
else:
debug_options = options.view_as(DebugOptions)
if (
@@ -928,7 +977,7 @@
return self.apply_PTransform(transform, pbegin, options)
else:
# Custom Read is also a primitive for non-FnAPI on dataflow.
- return beam.pvalue.PCollection(pbegin.pipeline)
+ return beam.pvalue.PCollection.from_(pbegin)
def run_Read(self, transform_node, options):
transform = transform_node.transform
@@ -1246,9 +1295,9 @@
def _get_job_state(self):
values_enum = dataflow_api.Job.CurrentStateValueValuesEnum
- # TODO: Move this table to a another location.
- # Ordered by the enum values.
- api_jobstate_map = {
+ # Ordered by the enum values. Values that may be introduced in
+ # future versions of Dataflow API are considered UNRECOGNIZED by the SDK.
+ api_jobstate_map = defaultdict(lambda: PipelineState.UNRECOGNIZED, {
values_enum.JOB_STATE_UNKNOWN: PipelineState.UNKNOWN,
values_enum.JOB_STATE_STOPPED: PipelineState.STOPPED,
values_enum.JOB_STATE_RUNNING: PipelineState.RUNNING,
@@ -1260,7 +1309,7 @@
values_enum.JOB_STATE_DRAINED: PipelineState.DRAINED,
values_enum.JOB_STATE_PENDING: PipelineState.PENDING,
values_enum.JOB_STATE_CANCELLING: PipelineState.CANCELLING,
- }
+ })
return (api_jobstate_map[self._job.currentState] if self._job.currentState
else PipelineState.UNKNOWN)
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py b/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
index c774b61..f9c80c5 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py
@@ -20,6 +20,7 @@
from __future__ import absolute_import
import json
+import sys
import unittest
from builtins import object
from builtins import range
@@ -34,6 +35,7 @@
from apache_beam.pipeline import AppliedPTransform
from apache_beam.pipeline import Pipeline
from apache_beam.portability import common_urns
+from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.pvalue import PCollection
from apache_beam.runners import DataflowRunner
from apache_beam.runners import TestDataflowRunner
@@ -175,6 +177,23 @@
isinstance(create_runner('TestDataflowRunner'),
TestDataflowRunner))
+ def test_environment_override_translation(self):
+ self.default_properties.append('--experiments=beam_fn_api')
+ self.default_properties.append('--worker_harness_container_image=FOO')
+ remote_runner = DataflowRunner()
+ p = Pipeline(remote_runner,
+ options=PipelineOptions(self.default_properties))
+ (p | ptransform.Create([1, 2, 3]) # pylint: disable=expression-not-assigned
+ | 'Do' >> ptransform.FlatMap(lambda x: [(x, x)])
+ | ptransform.GroupByKey())
+ p.run()
+ self.assertEqual(
+ list(remote_runner.proto_pipeline.components.environments.values()),
+ [beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.DOCKER.urn,
+ payload=beam_runner_api_pb2.DockerPayload(
+ container_image='FOO').SerializeToString())])
+
def test_remote_runner_translation(self):
remote_runner = DataflowRunner()
p = Pipeline(remote_runner,
@@ -444,6 +463,30 @@
self.assertIn('beam_fn_api', experiments_for_job)
self.assertIn('use_staged_dataflow_worker_jar', experiments_for_job)
+ def test_use_fastavro_experiment_is_added_on_py3_and_onwards(self):
+ remote_runner = DataflowRunner()
+
+ p = Pipeline(remote_runner, PipelineOptions(self.default_properties))
+ p | ptransform.Create([1]) # pylint: disable=expression-not-assigned
+ p.run()
+
+ self.assertEqual(
+ sys.version_info[0] > 2,
+ remote_runner.job.options.view_as(DebugOptions).lookup_experiment(
+ 'use_fastavro', False))
+
+ def test_use_fastavro_experiment_is_not_added_when_use_avro_is_present(self):
+ remote_runner = DataflowRunner()
+ self.default_properties.append('--experiment=use_avro')
+
+ p = Pipeline(remote_runner, PipelineOptions(self.default_properties))
+ p | ptransform.Create([1]) # pylint: disable=expression-not-assigned
+ p.run()
+
+ debug_options = remote_runner.job.options.view_as(DebugOptions)
+
+ self.assertFalse(debug_options.lookup_experiment('use_fastavro', False))
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index 6458f3e..b53f1aa 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -27,10 +27,12 @@
import json
import logging
import os
+import pkg_resources
import re
import sys
import tempfile
import time
+import warnings
from datetime import datetime
from builtins import object
@@ -177,20 +179,20 @@
key='major', value=to_json_value(environment_version))])
# TODO: Use enumerated type instead of strings for job types.
if job_type.startswith('FNAPI_'):
+ self.debug_options.experiments = self.debug_options.experiments or []
+ debug_options_experiments = self.debug_options.experiments
runner_harness_override = (
get_runner_harness_container_image())
- self.debug_options.experiments = self.debug_options.experiments or []
if runner_harness_override:
- self.debug_options.experiments.append(
+ debug_options_experiments.append(
'runner_harness_container_image=' + runner_harness_override)
- # Add use_multiple_sdk_containers flag if its not already present. Do not
+ # Add use_multiple_sdk_containers flag if it's not already present. Do not
# add the flag if 'no_use_multiple_sdk_containers' is present.
# TODO: Cleanup use_multiple_sdk_containers once we deprecate Python SDK
# till version 2.4.
- debug_options_experiments = self.debug_options.experiments
if ('use_multiple_sdk_containers' not in debug_options_experiments and
'no_use_multiple_sdk_containers' not in debug_options_experiments):
- self.debug_options.experiments.append('use_multiple_sdk_containers')
+ debug_options_experiments.append('use_multiple_sdk_containers')
# FlexRS
if self.google_cloud_options.flexrs_goal == 'COST_OPTIMIZED':
self.proto.flexResourceSchedulingGoal = (
@@ -248,12 +250,11 @@
pool.network = self.worker_options.network
if self.worker_options.subnetwork:
pool.subnetwork = self.worker_options.subnetwork
- if self.worker_options.worker_harness_container_image:
- pool.workerHarnessContainerImage = (
- self.worker_options.worker_harness_container_image)
- else:
- pool.workerHarnessContainerImage = (
- get_default_container_image_for_current_sdk(job_type))
+ pool.workerHarnessContainerImage = (
+ get_container_image_from_options(options))
+ if self.debug_options.number_of_worker_harness_threads:
+ pool.numThreadsPerWorker = (
+ self.debug_options.number_of_worker_harness_threads)
if self.worker_options.use_public_ips is not None:
if self.worker_options.use_public_ips:
pool.ipConfiguration = (
@@ -269,7 +270,8 @@
disk = dataflow.Disk()
if self.local:
disk.diskType = 'local'
- # TODO(ccy): allow customization of disk.
+ if self.worker_options.disk_type:
+ disk.diskType = self.worker_options.disk_type
pool.dataDisks.append(disk)
self.proto.workerPools.append(pool)
@@ -402,7 +404,14 @@
self.proto.type = dataflow.Job.TypeValueValuesEnum.JOB_TYPE_BATCH
if self.google_cloud_options.update:
self.proto.replaceJobId = self.job_id_for_name(self.proto.name)
-
+ if self.google_cloud_options.transform_name_mapping:
+ self.proto.transformNameMapping = (
+ dataflow.Job.TransformNameMappingValue())
+ for _, (key, value) in enumerate(
+ self.google_cloud_options.transform_name_mapping.items()):
+ self.proto.transformNameMapping.additionalProperties.append(
+ dataflow.Job.TransformNameMappingValue
+ .AdditionalProperty(key=key, value=value))
# Labels.
if self.google_cloud_options.labels:
self.proto.labels = dataflow.Job.LabelsValue()
@@ -632,7 +641,8 @@
self._client.projects_locations_jobs.Update(request)
return True
- @retry.with_exponential_backoff() # Using retry defaults from utils/retry.py
+ @retry.with_exponential_backoff(
+ retry_filter=retry.retry_on_server_errors_and_notfound_filter)
def get_job(self, job_id):
"""Gets the job status for a submitted job.
@@ -661,7 +671,8 @@
response = self._client.projects_locations_jobs.Get(request)
return response
- @retry.with_exponential_backoff() # Using retry defaults from utils/retry.py
+ @retry.with_exponential_backoff(
+ retry_filter=retry.retry_on_server_errors_and_notfound_filter)
def list_messages(
self, job_id, start_time=None, end_time=None, page_token=None,
minimum_importance=None):
@@ -877,15 +888,38 @@
'use_unified_worker' in debug_options.experiments)
-def get_default_container_image_for_current_sdk(job_type):
+def _use_sdf_bounded_source(pipeline_options):
+ debug_options = pipeline_options.view_as(DebugOptions)
+ return _use_fnapi(pipeline_options) and (
+ debug_options.experiments and
+ 'use_sdf_bounded_source' in debug_options.experiments)
+
+
+def _get_container_image_tag():
+ base_version = pkg_resources.parse_version(
+ beam_version.__version__).base_version
+ if base_version != beam_version.__version__:
+ warnings.warn(
+ "A non-standard version of Beam SDK detected: %s. "
+ "Dataflow runner will use container image tag %s. "
+ "This use case is not supported." % (
+ beam_version.__version__, base_version))
+ return base_version
+
+
+def get_container_image_from_options(pipeline_options):
"""For internal use only; no backwards-compatibility guarantees.
Args:
- job_type (str): BEAM job type.
+ pipeline_options (PipelineOptions): A container for pipeline options.
Returns:
- str: Google Cloud Dataflow container image for remote execution.
+ str: Container image for remote execution.
"""
+ worker_options = pipeline_options.view_as(WorkerOptions)
+ if worker_options.worker_harness_container_image:
+ return worker_options.worker_harness_container_image
+
if sys.version_info[0] == 2:
version_suffix = ''
elif sys.version_info[0:2] == (3, 5):
@@ -898,8 +932,9 @@
raise Exception('Dataflow only supports Python versions 2 and 3.5+, got: %s'
% str(sys.version_info[0:2]))
+ use_fnapi = _use_fnapi(pipeline_options)
# TODO(tvalentyn): Use enumerated type instead of strings for job types.
- if job_type == 'FNAPI_BATCH' or job_type == 'FNAPI_STREAMING':
+ if use_fnapi:
fnapi_suffix = '-fnapi'
else:
fnapi_suffix = ''
@@ -909,27 +944,27 @@
version_suffix=version_suffix,
fnapi_suffix=fnapi_suffix)
- image_tag = _get_required_container_version(job_type)
+ image_tag = _get_required_container_version(use_fnapi)
return image_name + ':' + image_tag
-def _get_required_container_version(job_type=None):
+def _get_required_container_version(use_fnapi):
"""For internal use only; no backwards-compatibility guarantees.
Args:
- job_type (str, optional): BEAM job type. Defaults to None.
+ use_fnapi (bool): True, if pipeline is using FnAPI, False otherwise.
Returns:
str: The tag of worker container images in GCR that corresponds to
current version of the SDK.
"""
if 'dev' in beam_version.__version__:
- if job_type == 'FNAPI_BATCH' or job_type == 'FNAPI_STREAMING':
+ if use_fnapi:
return names.BEAM_FNAPI_CONTAINER_VERSION
else:
return names.BEAM_CONTAINER_VERSION
else:
- return beam_version.__version__
+ return _get_container_image_tag()
def get_runner_harness_container_image():
@@ -943,7 +978,7 @@
# Pin runner harness for released versions of the SDK.
if 'dev' not in beam_version.__version__:
return (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY + '/' + 'harness' + ':' +
- beam_version.__version__)
+ _get_container_image_tag())
# Don't pin runner harness for dev versions so that we can notice
# potential incompatibility between runner and sdk harnesses.
return None
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
index 2f65716..b548be7 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
@@ -288,6 +288,18 @@
env.proto.workerPools[0].ipConfiguration,
dataflow.WorkerPool.IpConfigurationValueValuesEnum.WORKER_IP_PRIVATE)
+ def test_number_of_worker_harness_threads(self):
+ pipeline_options = PipelineOptions(
+ ['--temp_location', 'gs://any-location/temp',
+ '--number_of_worker_harness_threads', '2'])
+ env = apiclient.Environment([],
+ pipeline_options,
+ '2.0.0',
+ FAKE_PIPELINE_URL)
+ self.assertEqual(
+ env.proto.workerPools[0].numThreadsPerWorker,
+ 2)
+
@mock.patch('apache_beam.runners.dataflow.internal.apiclient.'
'beam_version.__version__', '2.2.0')
def test_harness_override_present_in_released_sdks(self):
@@ -304,6 +316,21 @@
self.assertIn(override, env.proto.experiments)
@mock.patch('apache_beam.runners.dataflow.internal.apiclient.'
+ 'beam_version.__version__', '2.2.0.rc1')
+ def test_harness_override_uses_base_version_in_rc_releases(self):
+ pipeline_options = PipelineOptions(
+ ['--temp_location', 'gs://any-location/temp', '--streaming'])
+ override = ''.join(
+ ['runner_harness_container_image=',
+ names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY,
+ '/harness:2.2.0'])
+ env = apiclient.Environment([], #packages
+ pipeline_options,
+ '2.0.0', #any environment version
+ FAKE_PIPELINE_URL)
+ self.assertIn(override, env.proto.experiments)
+
+ @mock.patch('apache_beam.runners.dataflow.internal.apiclient.'
'beam_version.__version__', '2.2.0.dev')
def test_harness_override_absent_in_unreleased_sdk(self):
pipeline_options = PipelineOptions(
@@ -420,6 +447,57 @@
'/python%d%d:2.2.0' % (sys.version_info[0],
sys.version_info[1])))
+ @mock.patch('apache_beam.runners.dataflow.internal.apiclient.'
+ 'beam_version.__version__', '2.2.0.rc1')
+ def test_worker_harness_image_tag_matches_base_sdk_version_of_an_rc(self):
+ # streaming, fnapi pipeline.
+ pipeline_options = PipelineOptions(
+ ['--temp_location', 'gs://any-location/temp', '--streaming'])
+ env = apiclient.Environment([], #packages
+ pipeline_options,
+ '2.0.0', #any environment version
+ FAKE_PIPELINE_URL)
+ if sys.version_info[0] == 2:
+ self.assertEqual(
+ env.proto.workerPools[0].workerHarnessContainerImage,
+ (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY +
+ '/python-fnapi:2.2.0'))
+ elif sys.version_info[0:2] == (3, 5):
+ self.assertEqual(
+ env.proto.workerPools[0].workerHarnessContainerImage,
+ (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY +
+ '/python3-fnapi:2.2.0'))
+ else:
+ self.assertEqual(
+ env.proto.workerPools[0].workerHarnessContainerImage,
+ (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY +
+ '/python%d%d-fnapi:2.2.0' % (sys.version_info[0],
+ sys.version_info[1])))
+
+ # batch, legacy pipeline.
+ pipeline_options = PipelineOptions(
+ ['--temp_location', 'gs://any-location/temp'])
+ env = apiclient.Environment([], #packages
+ pipeline_options,
+ '2.0.0', #any environment version
+ FAKE_PIPELINE_URL)
+ if sys.version_info[0] == 2:
+ self.assertEqual(
+ env.proto.workerPools[0].workerHarnessContainerImage,
+ (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY +
+ '/python:2.2.0'))
+ elif sys.version_info[0:2] == (3, 5):
+ self.assertEqual(
+ env.proto.workerPools[0].workerHarnessContainerImage,
+ (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY +
+ '/python3:2.2.0'))
+ else:
+ self.assertEqual(
+ env.proto.workerPools[0].workerHarnessContainerImage,
+ (names.DATAFLOW_CONTAINER_IMAGE_REPOSITORY +
+ '/python%d%d:2.2.0' % (sys.version_info[0],
+ sys.version_info[1])))
+
def test_worker_harness_override_takes_precedence_over_sdk_defaults(self):
# streaming, fnapi pipeline.
pipeline_options = PipelineOptions(
@@ -444,6 +522,16 @@
env.proto.workerPools[0].workerHarnessContainerImage,
'some:image')
+ @mock.patch('apache_beam.runners.dataflow.internal.apiclient.Job.'
+ 'job_id_for_name', return_value='test_id')
+ def test_transform_name_mapping(self, mock_job):
+ pipeline_options = PipelineOptions(
+ ['--project', 'test_project', '--job_name', 'test_job_name',
+ '--temp_location', 'gs://test-location/temp', '--update',
+ '--transform_name_mapping', '{\"from\":\"to\"}'])
+ job = apiclient.Job(pipeline_options, FAKE_PIPELINE_URL)
+ self.assertIsNotNone(job.proto.transformNameMapping)
+
def test_labels(self):
pipeline_options = PipelineOptions(
['--project', 'test_project', '--job_name', 'test_job_name',
@@ -479,7 +567,7 @@
'--experiments', 'beam_fn_api'])
environment = apiclient.Environment(
[], pipeline_options, 1, FAKE_PIPELINE_URL)
- self.assertIn("use_multiple_sdk_containers", environment.proto.experiments)
+ self.assertIn('use_multiple_sdk_containers', environment.proto.experiments)
pipeline_options = PipelineOptions(
['--project', 'test_project', '--job_name', 'test_job_name',
@@ -488,7 +576,7 @@
'--experiments', 'use_multiple_sdk_containers'])
environment = apiclient.Environment(
[], pipeline_options, 1, FAKE_PIPELINE_URL)
- self.assertIn("use_multiple_sdk_containers", environment.proto.experiments)
+ self.assertIn('use_multiple_sdk_containers', environment.proto.experiments)
pipeline_options = PipelineOptions(
['--project', 'test_project', '--job_name', 'test_job_name',
@@ -498,7 +586,7 @@
environment = apiclient.Environment(
[], pipeline_options, 1, FAKE_PIPELINE_URL)
self.assertNotIn(
- "use_multiple_sdk_containers", environment.proto.experiments)
+ 'use_multiple_sdk_containers', environment.proto.experiments)
@mock.patch(
'apache_beam.runners.dataflow.internal.apiclient.sys.version_info',
@@ -552,6 +640,13 @@
Exception,
apiclient._verify_interpreter_version_is_supported, pipeline_options)
+ def test_get_response_encoding(self):
+ encoding = apiclient.get_response_encoding()
+ version_to_encoding = {3: 'utf8',
+ 2: None}
+
+ assert encoding == version_to_encoding[sys.version_info[0]]
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py b/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py
index 061a60b..1df3f2b 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py
@@ -64,11 +64,9 @@
self.projects_locations_jobs_messages = self.ProjectsLocationsJobsMessagesService(self)
self.projects_locations_jobs_workItems = self.ProjectsLocationsJobsWorkItemsService(self)
self.projects_locations_jobs = self.ProjectsLocationsJobsService(self)
- self.projects_locations_snapshots = self.ProjectsLocationsSnapshotsService(self)
self.projects_locations_sql = self.ProjectsLocationsSqlService(self)
self.projects_locations_templates = self.ProjectsLocationsTemplatesService(self)
self.projects_locations = self.ProjectsLocationsService(self)
- self.projects_snapshots = self.ProjectsSnapshotsService(self)
self.projects_templates = self.ProjectsTemplatesService(self)
self.projects = self.ProjectsService(self)
@@ -403,32 +401,6 @@
supports_download=False,
)
- def Snapshot(self, request, global_params=None):
- r"""Snapshot the state of a streaming job.
-
- Args:
- request: (DataflowProjectsJobsSnapshotRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (Snapshot) The response message.
- """
- config = self.GetMethodConfig('Snapshot')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- Snapshot.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'POST',
- method_id=u'dataflow.projects.jobs.snapshot',
- ordered_params=[u'projectId', u'jobId'],
- path_params=[u'jobId', u'projectId'],
- query_params=[],
- relative_path=u'v1b3/projects/{projectId}/jobs/{jobId}:snapshot',
- request_field=u'snapshotJobRequest',
- request_type_name=u'DataflowProjectsJobsSnapshotRequest',
- response_type_name=u'Snapshot',
- supports_download=False,
- )
-
def Update(self, request, global_params=None):
r"""Updates the state of an existing Cloud Dataflow job.
@@ -766,32 +738,6 @@
supports_download=False,
)
- def Snapshot(self, request, global_params=None):
- r"""Snapshot the state of a streaming job.
-
- Args:
- request: (DataflowProjectsLocationsJobsSnapshotRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (Snapshot) The response message.
- """
- config = self.GetMethodConfig('Snapshot')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- Snapshot.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'POST',
- method_id=u'dataflow.projects.locations.jobs.snapshot',
- ordered_params=[u'projectId', u'location', u'jobId'],
- path_params=[u'jobId', u'location', u'projectId'],
- query_params=[],
- relative_path=u'v1b3/projects/{projectId}/locations/{location}/jobs/{jobId}:snapshot',
- request_field=u'snapshotJobRequest',
- request_type_name=u'DataflowProjectsLocationsJobsSnapshotRequest',
- response_type_name=u'Snapshot',
- supports_download=False,
- )
-
def Update(self, request, global_params=None):
r"""Updates the state of an existing Cloud Dataflow job.
@@ -824,94 +770,6 @@
supports_download=False,
)
- class ProjectsLocationsSnapshotsService(base_api.BaseApiService):
- """Service class for the projects_locations_snapshots resource."""
-
- _NAME = u'projects_locations_snapshots'
-
- def __init__(self, client):
- super(DataflowV1b3.ProjectsLocationsSnapshotsService, self).__init__(client)
- self._upload_configs = {
- }
-
- def Delete(self, request, global_params=None):
- r"""Deletes a snapshot.
-
- Args:
- request: (DataflowProjectsLocationsSnapshotsDeleteRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (DeleteSnapshotResponse) The response message.
- """
- config = self.GetMethodConfig('Delete')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- Delete.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'DELETE',
- method_id=u'dataflow.projects.locations.snapshots.delete',
- ordered_params=[u'projectId', u'location', u'snapshotId'],
- path_params=[u'location', u'projectId', u'snapshotId'],
- query_params=[],
- relative_path=u'v1b3/projects/{projectId}/locations/{location}/snapshots/{snapshotId}',
- request_field='',
- request_type_name=u'DataflowProjectsLocationsSnapshotsDeleteRequest',
- response_type_name=u'DeleteSnapshotResponse',
- supports_download=False,
- )
-
- def Get(self, request, global_params=None):
- r"""Gets information about a snapshot.
-
- Args:
- request: (DataflowProjectsLocationsSnapshotsGetRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (Snapshot) The response message.
- """
- config = self.GetMethodConfig('Get')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- Get.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'GET',
- method_id=u'dataflow.projects.locations.snapshots.get',
- ordered_params=[u'projectId', u'location', u'snapshotId'],
- path_params=[u'location', u'projectId', u'snapshotId'],
- query_params=[],
- relative_path=u'v1b3/projects/{projectId}/locations/{location}/snapshots/{snapshotId}',
- request_field='',
- request_type_name=u'DataflowProjectsLocationsSnapshotsGetRequest',
- response_type_name=u'Snapshot',
- supports_download=False,
- )
-
- def List(self, request, global_params=None):
- r"""Lists snapshots.
-
- Args:
- request: (DataflowProjectsLocationsSnapshotsListRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (ListSnapshotsResponse) The response message.
- """
- config = self.GetMethodConfig('List')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- List.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'GET',
- method_id=u'dataflow.projects.locations.snapshots.list',
- ordered_params=[u'projectId', u'location'],
- path_params=[u'location', u'projectId'],
- query_params=[],
- relative_path=u'v1b3/projects/{projectId}/locations/{location}/snapshots',
- request_field='',
- request_type_name=u'DataflowProjectsLocationsSnapshotsListRequest',
- response_type_name=u'ListSnapshotsResponse',
- supports_download=False,
- )
-
class ProjectsLocationsSqlService(base_api.BaseApiService):
"""Service class for the projects_locations_sql resource."""
@@ -1075,68 +933,6 @@
supports_download=False,
)
- class ProjectsSnapshotsService(base_api.BaseApiService):
- """Service class for the projects_snapshots resource."""
-
- _NAME = u'projects_snapshots'
-
- def __init__(self, client):
- super(DataflowV1b3.ProjectsSnapshotsService, self).__init__(client)
- self._upload_configs = {
- }
-
- def Get(self, request, global_params=None):
- r"""Gets information about a snapshot.
-
- Args:
- request: (DataflowProjectsSnapshotsGetRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (Snapshot) The response message.
- """
- config = self.GetMethodConfig('Get')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- Get.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'GET',
- method_id=u'dataflow.projects.snapshots.get',
- ordered_params=[u'projectId', u'snapshotId'],
- path_params=[u'projectId', u'snapshotId'],
- query_params=[u'location'],
- relative_path=u'v1b3/projects/{projectId}/snapshots/{snapshotId}',
- request_field='',
- request_type_name=u'DataflowProjectsSnapshotsGetRequest',
- response_type_name=u'Snapshot',
- supports_download=False,
- )
-
- def List(self, request, global_params=None):
- r"""Lists snapshots.
-
- Args:
- request: (DataflowProjectsSnapshotsListRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (ListSnapshotsResponse) The response message.
- """
- config = self.GetMethodConfig('List')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- List.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'GET',
- method_id=u'dataflow.projects.snapshots.list',
- ordered_params=[u'projectId'],
- path_params=[u'projectId'],
- query_params=[u'location'],
- relative_path=u'v1b3/projects/{projectId}/snapshots',
- request_field='',
- request_type_name=u'DataflowProjectsSnapshotsListRequest',
- response_type_name=u'ListSnapshotsResponse',
- supports_download=False,
- )
-
class ProjectsTemplatesService(base_api.BaseApiService):
"""Service class for the projects_templates resource."""
@@ -1235,32 +1031,6 @@
self._upload_configs = {
}
- def DeleteSnapshots(self, request, global_params=None):
- r"""Deletes a snapshot.
-
- Args:
- request: (DataflowProjectsDeleteSnapshotsRequest) input message
- global_params: (StandardQueryParameters, default: None) global arguments
- Returns:
- (DeleteSnapshotResponse) The response message.
- """
- config = self.GetMethodConfig('DeleteSnapshots')
- return self._RunMethod(
- config, request, global_params=global_params)
-
- DeleteSnapshots.method_config = lambda: base_api.ApiMethodInfo(
- http_method=u'DELETE',
- method_id=u'dataflow.projects.deleteSnapshots',
- ordered_params=[u'projectId'],
- path_params=[u'projectId'],
- query_params=[u'location', u'snapshotId'],
- relative_path=u'v1b3/projects/{projectId}/snapshots',
- request_field='',
- request_type_name=u'DataflowProjectsDeleteSnapshotsRequest',
- response_type_name=u'DeleteSnapshotResponse',
- supports_download=False,
- )
-
def WorkerMessages(self, request, global_params=None):
r"""Send a worker_message to the service.
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py b/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
index 021f394..9a76e8d 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
@@ -575,21 +575,6 @@
vmInstance = _messages.StringField(2)
-class DataflowProjectsDeleteSnapshotsRequest(_messages.Message):
- r"""A DataflowProjectsDeleteSnapshotsRequest object.
-
- Fields:
- location: The location that contains this snapshot.
- projectId: The ID of the Cloud Platform project that the snapshot belongs
- to.
- snapshotId: The ID of the snapshot.
- """
-
- location = _messages.StringField(1)
- projectId = _messages.StringField(2, required=True)
- snapshotId = _messages.StringField(3)
-
-
class DataflowProjectsJobsAggregatedRequest(_messages.Message):
r"""A DataflowProjectsJobsAggregatedRequest object.
@@ -886,21 +871,6 @@
startTime = _messages.StringField(8)
-class DataflowProjectsJobsSnapshotRequest(_messages.Message):
- r"""A DataflowProjectsJobsSnapshotRequest object.
-
- Fields:
- jobId: The job to be snapshotted.
- projectId: The project which owns the job to be snapshotted.
- snapshotJobRequest: A SnapshotJobRequest resource to be passed as the
- request body.
- """
-
- jobId = _messages.StringField(1, required=True)
- projectId = _messages.StringField(2, required=True)
- snapshotJobRequest = _messages.MessageField('SnapshotJobRequest', 3)
-
-
class DataflowProjectsJobsUpdateRequest(_messages.Message):
r"""A DataflowProjectsJobsUpdateRequest object.
@@ -1193,23 +1163,6 @@
startTime = _messages.StringField(8)
-class DataflowProjectsLocationsJobsSnapshotRequest(_messages.Message):
- r"""A DataflowProjectsLocationsJobsSnapshotRequest object.
-
- Fields:
- jobId: The job to be snapshotted.
- location: The location that contains this job.
- projectId: The project which owns the job to be snapshotted.
- snapshotJobRequest: A SnapshotJobRequest resource to be passed as the
- request body.
- """
-
- jobId = _messages.StringField(1, required=True)
- location = _messages.StringField(2, required=True)
- projectId = _messages.StringField(3, required=True)
- snapshotJobRequest = _messages.MessageField('SnapshotJobRequest', 4)
-
-
class DataflowProjectsLocationsJobsUpdateRequest(_messages.Message):
r"""A DataflowProjectsLocationsJobsUpdateRequest object.
@@ -1266,48 +1219,6 @@
reportWorkItemStatusRequest = _messages.MessageField('ReportWorkItemStatusRequest', 4)
-class DataflowProjectsLocationsSnapshotsDeleteRequest(_messages.Message):
- r"""A DataflowProjectsLocationsSnapshotsDeleteRequest object.
-
- Fields:
- location: The location that contains this snapshot.
- projectId: The ID of the Cloud Platform project that the snapshot belongs
- to.
- snapshotId: The ID of the snapshot.
- """
-
- location = _messages.StringField(1, required=True)
- projectId = _messages.StringField(2, required=True)
- snapshotId = _messages.StringField(3, required=True)
-
-
-class DataflowProjectsLocationsSnapshotsGetRequest(_messages.Message):
- r"""A DataflowProjectsLocationsSnapshotsGetRequest object.
-
- Fields:
- location: The location that contains this snapshot.
- projectId: The ID of the Cloud Platform project that the snapshot belongs
- to.
- snapshotId: The ID of the snapshot.
- """
-
- location = _messages.StringField(1, required=True)
- projectId = _messages.StringField(2, required=True)
- snapshotId = _messages.StringField(3, required=True)
-
-
-class DataflowProjectsLocationsSnapshotsListRequest(_messages.Message):
- r"""A DataflowProjectsLocationsSnapshotsListRequest object.
-
- Fields:
- location: The location to list snapshots in.
- projectId: The project ID to list snapshots for.
- """
-
- location = _messages.StringField(1, required=True)
- projectId = _messages.StringField(2, required=True)
-
-
class DataflowProjectsLocationsSqlValidateRequest(_messages.Message):
r"""A DataflowProjectsLocationsSqlValidateRequest object.
@@ -1421,33 +1332,6 @@
sendWorkerMessagesRequest = _messages.MessageField('SendWorkerMessagesRequest', 3)
-class DataflowProjectsSnapshotsGetRequest(_messages.Message):
- r"""A DataflowProjectsSnapshotsGetRequest object.
-
- Fields:
- location: The location that contains this snapshot.
- projectId: The ID of the Cloud Platform project that the snapshot belongs
- to.
- snapshotId: The ID of the snapshot.
- """
-
- location = _messages.StringField(1)
- projectId = _messages.StringField(2, required=True)
- snapshotId = _messages.StringField(3, required=True)
-
-
-class DataflowProjectsSnapshotsListRequest(_messages.Message):
- r"""A DataflowProjectsSnapshotsListRequest object.
-
- Fields:
- location: The location to list snapshots in.
- projectId: The project ID to list snapshots for.
- """
-
- location = _messages.StringField(1)
- projectId = _messages.StringField(2, required=True)
-
-
class DataflowProjectsTemplatesCreateRequest(_messages.Message):
r"""A DataflowProjectsTemplatesCreateRequest object.
@@ -1548,10 +1432,6 @@
projectId = _messages.StringField(2)
-class DeleteSnapshotResponse(_messages.Message):
- r"""Response from deleting a snapshot."""
-
-
class DerivedSource(_messages.Message):
r"""Specification of one of the bundles produced as a result of splitting a
Source (e.g. when executing a SourceSplitRequest, or when splitting an
@@ -1748,6 +1628,16 @@
service are required in order to run the job.
workerPools: The worker pools. At least one "harness" worker pool must be
specified in order for the job to have workers.
+ workerRegion: The Compute Engine region
+ (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in
+ which worker processing should occur, e.g. "us-west1". Mutually
+ exclusive with worker_zone. If neither worker_region nor worker_zone is
+ specified, default to the control plane's region.
+ workerZone: The Compute Engine zone (https://cloud.google.com/compute/docs
+ /regions-zones/regions-zones) in which worker processing should occur,
+ e.g. "us-west1-a". Mutually exclusive with worker_region. If neither
+ worker_region nor worker_zone is specified, a zone in the control
+ plane's region is chosen based on available capacity.
"""
class FlexResourceSchedulingGoalValueValuesEnum(_messages.Enum):
@@ -1877,6 +1767,8 @@
userAgent = _messages.MessageField('UserAgentValue', 10)
version = _messages.MessageField('VersionValue', 11)
workerPools = _messages.MessageField('WorkerPool', 12, repeated=True)
+ workerRegion = _messages.StringField(13)
+ workerZone = _messages.StringField(14)
class ExecutionStageState(_messages.Message):
@@ -2133,6 +2025,22 @@
firstBucketOffset = _messages.IntegerField(2, variant=_messages.Variant.INT32)
+class HotKeyDetection(_messages.Message):
+ r"""Proto describing a hot key detected on a given WorkItem.
+
+ Fields:
+ hotKeyAge: The age of the hot key measured from when it was first
+ detected.
+ systemName: System-defined name of the step containing this hot key.
+ Unique across the workflow.
+ userStepName: User-provided name of the step that contains this hot key.
+ """
+
+ hotKeyAge = _messages.StringField(1)
+ systemName = _messages.StringField(2)
+ userStepName = _messages.StringField(3)
+
+
class InstructionInput(_messages.Message):
r"""An input of an instruction, as a reference to an output of a producer
instruction.
@@ -2775,11 +2683,19 @@
Messages:
ParametersValue: The runtime parameters to pass to the job.
+ TransformNameMappingValue: Only applicable when updating a pipeline. Map
+ of transform name prefixes of the job to be replaced to the
+ corresponding name prefixes of the new job.
Fields:
environment: The runtime environment for the job.
jobName: Required. The job name to use for the created job.
parameters: The runtime parameters to pass to the job.
+ transformNameMapping: Only applicable when updating a pipeline. Map of
+ transform name prefixes of the job to be replaced to the corresponding
+ name prefixes of the new job.
+ update: If set, replace the existing pipeline with the name specified by
+ jobName with this pipeline, preserving state.
"""
@encoding.MapUnrecognizedFields('additionalProperties')
@@ -2806,9 +2722,39 @@
additionalProperties = _messages.MessageField('AdditionalProperty', 1, repeated=True)
+ @encoding.MapUnrecognizedFields('additionalProperties')
+ class TransformNameMappingValue(_messages.Message):
+ r"""Only applicable when updating a pipeline. Map of transform name
+ prefixes of the job to be replaced to the corresponding name prefixes of
+ the new job.
+
+ Messages:
+ AdditionalProperty: An additional property for a
+ TransformNameMappingValue object.
+
+ Fields:
+ additionalProperties: Additional properties of type
+ TransformNameMappingValue
+ """
+
+ class AdditionalProperty(_messages.Message):
+ r"""An additional property for a TransformNameMappingValue object.
+
+ Fields:
+ key: Name of the additional property.
+ value: A string attribute.
+ """
+
+ key = _messages.StringField(1)
+ value = _messages.StringField(2)
+
+ additionalProperties = _messages.MessageField('AdditionalProperty', 1, repeated=True)
+
environment = _messages.MessageField('RuntimeEnvironment', 1)
jobName = _messages.StringField(2)
parameters = _messages.MessageField('ParametersValue', 3)
+ transformNameMapping = _messages.MessageField('TransformNameMappingValue', 4)
+ update = _messages.BooleanField(5)
class LaunchTemplateResponse(_messages.Message):
@@ -2954,16 +2900,6 @@
nextPageToken = _messages.StringField(3)
-class ListSnapshotsResponse(_messages.Message):
- r"""List of snapshots.
-
- Fields:
- snapshots: Returned snapshots.
- """
-
- snapshots = _messages.MessageField('Snapshot', 1, repeated=True)
-
-
class MapTask(_messages.Message):
r"""MapTask consists of an ordered set of instructions, each of which
describes one particular low-level operation for the worker to perform in
@@ -3621,6 +3557,9 @@
class RuntimeEnvironment(_messages.Message):
r"""The environment values to set at runtime.
+ Enums:
+ IpConfigurationValueValuesEnum: Configuration for VM IPs.
+
Messages:
AdditionalUserLabelsValue: Additional user labels to be specified for the
job. Keys and values should follow the restrictions specified in the
@@ -3635,6 +3574,10 @@
resources#restrictions) page.
bypassTempDirValidation: Whether to bypass the safety checks for the job's
temporary directory. Use with caution.
+ ipConfiguration: Configuration for VM IPs.
+ kmsKeyName: Optional. Name for the Cloud KMS key for the job. Key format
+ is: projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKey
+ s/<key>
machineType: The machine type to use for the job. Defaults to the value
from the template if not specified.
maxWorkers: The maximum number of Google Compute Engine instances to be
@@ -3649,11 +3592,35 @@
Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK".
tempLocation: The Cloud Storage path to use for temporary files. Must be a
valid Cloud Storage URL, beginning with `gs://`.
+ workerRegion: The Compute Engine region
+ (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in
+ which worker processing should occur, e.g. "us-west1". Mutually
+ exclusive with worker_zone. If neither worker_region nor worker_zone is
+ specified, default to the control plane's region.
+ workerZone: The Compute Engine zone (https://cloud.google.com/compute/docs
+ /regions-zones/regions-zones) in which worker processing should occur,
+ e.g. "us-west1-a". Mutually exclusive with worker_region. If neither
+ worker_region nor worker_zone is specified, a zone in the control
+ plane's region is chosen based on available capacity. If both
+ `worker_zone` and `zone` are set, `worker_zone` takes precedence.
zone: The Compute Engine [availability
zone](https://cloud.google.com/compute/docs/regions-zones/regions-zones)
- for launching worker instances to run your pipeline.
+ for launching worker instances to run your pipeline. In the future,
+ worker_zone will take precedence.
"""
+ class IpConfigurationValueValuesEnum(_messages.Enum):
+ r"""Configuration for VM IPs.
+
+ Values:
+ WORKER_IP_UNSPECIFIED: The configuration is unknown, or unspecified.
+ WORKER_IP_PUBLIC: Workers should have public IP addresses.
+ WORKER_IP_PRIVATE: Workers should have private IP addresses.
+ """
+ WORKER_IP_UNSPECIFIED = 0
+ WORKER_IP_PUBLIC = 1
+ WORKER_IP_PRIVATE = 2
+
@encoding.MapUnrecognizedFields('additionalProperties')
class AdditionalUserLabelsValue(_messages.Message):
r"""Additional user labels to be specified for the job. Keys and values
@@ -3686,14 +3653,18 @@
additionalExperiments = _messages.StringField(1, repeated=True)
additionalUserLabels = _messages.MessageField('AdditionalUserLabelsValue', 2)
bypassTempDirValidation = _messages.BooleanField(3)
- machineType = _messages.StringField(4)
- maxWorkers = _messages.IntegerField(5, variant=_messages.Variant.INT32)
- network = _messages.StringField(6)
- numWorkers = _messages.IntegerField(7, variant=_messages.Variant.INT32)
- serviceAccountEmail = _messages.StringField(8)
- subnetwork = _messages.StringField(9)
- tempLocation = _messages.StringField(10)
- zone = _messages.StringField(11)
+ ipConfiguration = _messages.EnumField('IpConfigurationValueValuesEnum', 4)
+ kmsKeyName = _messages.StringField(5)
+ machineType = _messages.StringField(6)
+ maxWorkers = _messages.IntegerField(7, variant=_messages.Variant.INT32)
+ network = _messages.StringField(8)
+ numWorkers = _messages.IntegerField(9, variant=_messages.Variant.INT32)
+ serviceAccountEmail = _messages.StringField(10)
+ subnetwork = _messages.StringField(11)
+ tempLocation = _messages.StringField(12)
+ workerRegion = _messages.StringField(13)
+ workerZone = _messages.StringField(14)
+ zone = _messages.StringField(15)
class SdkVersion(_messages.Message):
@@ -3962,60 +3933,6 @@
spec = _messages.MessageField('SpecValue', 2)
-class Snapshot(_messages.Message):
- r"""Represents a snapshot of a job.
-
- Enums:
- StateValueValuesEnum: State of the snapshot.
-
- Fields:
- creationTime: The time this snapshot was created.
- id: The unique ID of this snapshot.
- projectId: The project this snapshot belongs to.
- sourceJobId: The job this snapshot was created from.
- state: State of the snapshot.
- ttl: The time after which this snapshot will be automatically deleted.
- """
-
- class StateValueValuesEnum(_messages.Enum):
- r"""State of the snapshot.
-
- Values:
- UNKNOWN_SNAPSHOT_STATE: Unknown state.
- PENDING: Snapshot intent to create has been persisted, snapshotting of
- state has not yet started.
- RUNNING: Snapshotting is being performed.
- READY: Snapshot has been created and is ready to be used.
- FAILED: Snapshot failed to be created.
- DELETED: Snapshot has been deleted.
- """
- UNKNOWN_SNAPSHOT_STATE = 0
- PENDING = 1
- RUNNING = 2
- READY = 3
- FAILED = 4
- DELETED = 5
-
- creationTime = _messages.StringField(1)
- id = _messages.StringField(2)
- projectId = _messages.StringField(3)
- sourceJobId = _messages.StringField(4)
- state = _messages.EnumField('StateValueValuesEnum', 5)
- ttl = _messages.StringField(6)
-
-
-class SnapshotJobRequest(_messages.Message):
- r"""Request to create a snapshot of a job.
-
- Fields:
- location: The location that contains this job.
- ttl: TTL for the snapshot.
- """
-
- location = _messages.StringField(1)
- ttl = _messages.StringField(2)
-
-
class Source(_messages.Message):
r"""A source that records can be read and decoded from.
@@ -4465,37 +4382,10 @@
class Status(_messages.Message):
r"""The `Status` type defines a logical error model that is suitable for
different programming environments, including REST APIs and RPC APIs. It is
- used by [gRPC](https://github.com/grpc). The error model is designed to be:
- - Simple to use and understand for most users - Flexible enough to meet
- unexpected needs # Overview The `Status` message contains three pieces of
- data: error code, error message, and error details. The error code should be
- an enum value of google.rpc.Code, but it may accept additional error codes
- if needed. The error message should be a developer-facing English message
- that helps developers *understand* and *resolve* the error. If a localized
- user-facing error message is needed, put the localized message in the error
- details or localize it in the client. The optional error details may contain
- arbitrary information about the error. There is a predefined set of error
- detail types in the package `google.rpc` that can be used for common error
- conditions. # Language mapping The `Status` message is the logical
- representation of the error model, but it is not necessarily the actual wire
- format. When the `Status` message is exposed in different client libraries
- and different wire protocols, it can be mapped differently. For example, it
- will likely be mapped to some exceptions in Java, but more likely mapped to
- some error codes in C. # Other uses The error model and the `Status`
- message can be used in a variety of environments, either with or without
- APIs, to provide a consistent developer experience across different
- environments. Example uses of this error model include: - Partial errors.
- If a service needs to return partial errors to the client, it may embed
- the `Status` in the normal response to indicate the partial errors. -
- Workflow errors. A typical workflow has multiple steps. Each step may
- have a `Status` message for error reporting. - Batch operations. If a
- client uses batch request and batch response, the `Status` message
- should be used directly inside batch response, one for each error sub-
- response. - Asynchronous operations. If an API call embeds asynchronous
- operation results in its response, the status of those operations should
- be represented directly using the `Status` message. - Logging. If some
- API errors are stored in logs, the message `Status` could be used
- directly after any stripping needed for security/privacy reasons.
+ used by [gRPC](https://github.com/grpc). Each `Status` message contains
+ three pieces of data: error code, error message, and error details. You can
+ find out more about this error model and how to work with it in the [API
+ Design Guide](https://cloud.google.com/apis/design/errors).
Messages:
DetailsValueListEntry: A DetailsValueListEntry object.
@@ -4738,6 +4628,10 @@
families.
Fields:
+ commitStreamChunkSizeBytes: Chunk size for commit streams from the harness
+ to windmill.
+ getDataStreamChunkSizeBytes: Chunk size for get data streams from the
+ harness to windmill.
maxWorkItemCommitBytes: Maximum size for work item commit supported
windmill storage layer.
streamingComputationConfigs: Set of computation configuration information.
@@ -4777,11 +4671,13 @@
additionalProperties = _messages.MessageField('AdditionalProperty', 1, repeated=True)
- maxWorkItemCommitBytes = _messages.IntegerField(1)
- streamingComputationConfigs = _messages.MessageField('StreamingComputationConfig', 2, repeated=True)
- userStepToStateFamilyNameMap = _messages.MessageField('UserStepToStateFamilyNameMapValue', 3)
- windmillServiceEndpoint = _messages.StringField(4)
- windmillServicePort = _messages.IntegerField(5)
+ commitStreamChunkSizeBytes = _messages.IntegerField(1)
+ getDataStreamChunkSizeBytes = _messages.IntegerField(2)
+ maxWorkItemCommitBytes = _messages.IntegerField(3)
+ streamingComputationConfigs = _messages.MessageField('StreamingComputationConfig', 4, repeated=True)
+ userStepToStateFamilyNameMap = _messages.MessageField('UserStepToStateFamilyNameMapValue', 5)
+ windmillServiceEndpoint = _messages.StringField(6)
+ windmillServicePort = _messages.IntegerField(7)
class StreamingSetupTask(_messages.Message):
@@ -5102,6 +4998,10 @@
Fields:
harnessData: Other data returned by the service, specific to the
particular worker harness.
+ hotKeyDetection: A hot key is a symptom of poor data distribution in which
+ there are enough elements mapped to a single key to impact pipeline
+ performance. When present, this field includes metadata associated with
+ any hot key.
leaseExpireTime: Time at which the current lease will expire.
metricShortId: The short ids that workers should use in subsequent metric
updates. Workers should strive to use short ids whenever possible, but
@@ -5145,13 +5045,14 @@
additionalProperties = _messages.MessageField('AdditionalProperty', 1, repeated=True)
harnessData = _messages.MessageField('HarnessDataValue', 1)
- leaseExpireTime = _messages.StringField(2)
- metricShortId = _messages.MessageField('MetricShortId', 3, repeated=True)
- nextReportIndex = _messages.IntegerField(4)
- reportStatusInterval = _messages.StringField(5)
- splitRequest = _messages.MessageField('ApproximateSplitRequest', 6)
- suggestedStopPoint = _messages.MessageField('ApproximateProgress', 7)
- suggestedStopPosition = _messages.MessageField('Position', 8)
+ hotKeyDetection = _messages.MessageField('HotKeyDetection', 2)
+ leaseExpireTime = _messages.StringField(3)
+ metricShortId = _messages.MessageField('MetricShortId', 4, repeated=True)
+ nextReportIndex = _messages.IntegerField(5)
+ reportStatusInterval = _messages.StringField(6)
+ splitRequest = _messages.MessageField('ApproximateSplitRequest', 7)
+ suggestedStopPoint = _messages.MessageField('ApproximateProgress', 8)
+ suggestedStopPosition = _messages.MessageField('Position', 9)
class WorkItemStatus(_messages.Message):
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py
index 8c2cab7..c0445a2 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/names.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py
@@ -38,10 +38,10 @@
# Update this version to the next version whenever there is a change that will
# require changes to legacy Dataflow worker execution environment.
-BEAM_CONTAINER_VERSION = 'beam-master-20190509'
+BEAM_CONTAINER_VERSION = 'beam-master-20191010'
# Update this version to the next version whenever there is a change that
# requires changes to SDK harness container or SDK harness launcher.
-BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20190509'
+BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20191010'
# TODO(BEAM-5939): Remove these shared names once Dataflow worker is updated.
PICKLED_MAIN_SESSION_FILE = 'pickled_main_session'
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/streaming_create.py b/sdks/python/apache_beam/runners/dataflow/native_io/streaming_create.py
index 980ad24..481209e 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/streaming_create.py
+++ b/sdks/python/apache_beam/runners/dataflow/native_io/streaming_create.py
@@ -61,7 +61,7 @@
def expand(self, pbegin):
assert isinstance(pbegin, pvalue.PBegin), (
'Input to Impulse transform must be a PBegin but found %s' % pbegin)
- return pvalue.PCollection(pbegin.pipeline)
+ return pvalue.PCollection(pbegin.pipeline, is_bounded=False)
def get_windowing(self, inputs):
return Windowing(GlobalWindows())
diff --git a/sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py b/sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py
index 502ebd9..6e84c15 100644
--- a/sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py
+++ b/sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py
@@ -48,3 +48,20 @@
StreamingCreate
coder = typecoders.registry.get_coder(ptransform.get_output_type())
return StreamingCreate(ptransform.values, coder)
+
+
+class ReadPTransformOverride(PTransformOverride):
+ """A ``PTransformOverride`` for ``Read(BoundedSource)``"""
+
+ def matches(self, applied_ptransform):
+ from apache_beam.io import Read
+ from apache_beam.io.iobase import BoundedSource
+ # Only overrides Read(BoundedSource) transform
+ if isinstance(applied_ptransform.transform, Read):
+ if isinstance(applied_ptransform.transform.source, BoundedSource):
+ return True
+ return False
+
+ def get_replacement_transform(self, ptransform):
+ from apache_beam.io.iobase import _SDFBoundedSourceWrapper
+ return _SDFBoundedSourceWrapper(ptransform.source)
diff --git a/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py b/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
index 219f34b..34402b7 100644
--- a/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py
@@ -54,7 +54,7 @@
if self.result.has_job:
# TODO(markflyhigh)(BEAM-1890): Use print since Nose dosen't show logs
# in some cases.
- print('Found: %s.' % self.build_console_url(options))
+ print('Worker logs: %s' % self.build_console_url(options))
try:
self.wait_until_in_state(PipelineState.RUNNING)
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner.py b/sdks/python/apache_beam/runners/direct/direct_runner.py
index a3d2d42..7ae16a9 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner.py
@@ -26,6 +26,7 @@
import itertools
import logging
import time
+import typing
from google.protobuf import wrappers_pb2
@@ -129,12 +130,12 @@
# Type variables.
-K = typehints.TypeVariable('K')
-V = typehints.TypeVariable('V')
+K = typing.TypeVar('K')
+V = typing.TypeVar('V')
-@typehints.with_input_types(typehints.KV[K, V])
-@typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
+@typehints.with_input_types(typing.Tuple[K, V])
+@typehints.with_output_types(typing.Tuple[K, typing.Iterable[V]])
class _StreamingGroupByKeyOnly(_GroupByKeyOnly):
"""Streaming GroupByKeyOnly placeholder for overriding in DirectRunner."""
urn = "direct_runner:streaming_gbko:v0.1"
@@ -148,8 +149,8 @@
return _StreamingGroupByKeyOnly()
-@typehints.with_input_types(typehints.KV[K, typehints.Iterable[V]])
-@typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
+@typehints.with_input_types(typing.Tuple[K, typing.Iterable[V]])
+@typehints.with_output_types(typing.Tuple[K, typing.Iterable[V]])
class _StreamingGroupAlsoByWindow(_GroupAlsoByWindow):
"""Streaming GroupAlsoByWindow placeholder for overriding in DirectRunner."""
urn = "direct_runner:streaming_gabw:v0.1"
@@ -250,7 +251,7 @@
def expand(self, pvalue):
# This is handled as a native transform.
- return PCollection(self.pipeline)
+ return PCollection(self.pipeline, is_bounded=self._source.is_bounded())
class _DirectWriteToPubSubFn(DoFn):
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner_test.py b/sdks/python/apache_beam/runners/direct/direct_runner_test.py
index b60521a..66f1845 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner_test.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner_test.py
@@ -121,5 +121,13 @@
TestDirectRunner))
+class BundleBasedRunnerTest(unittest.TestCase):
+ def test_type_hints(self):
+ with test_pipeline.TestPipeline(runner='BundleBasedDirectRunner') as p:
+ _ = (p
+ | beam.Create([[]]).with_output_types(beam.typehints.List[int])
+ | beam.combiners.Count.Globally())
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/direct/direct_userstate.py b/sdks/python/apache_beam/runners/direct/direct_userstate.py
index f0fd9b8..42afaa3 100644
--- a/sdks/python/apache_beam/runners/direct/direct_userstate.py
+++ b/sdks/python/apache_beam/runners/direct/direct_userstate.py
@@ -18,8 +18,132 @@
"""Support for user state in the BundleBasedDirectRunner."""
from __future__ import absolute_import
+import itertools
+
from apache_beam.transforms import userstate
from apache_beam.transforms.trigger import _ListStateTag
+from apache_beam.transforms.trigger import _SetStateTag
+
+
+class DirectRuntimeState(userstate.RuntimeState):
+ def __init__(self, state_spec, state_tag, current_value_accessor):
+ self._state_spec = state_spec
+ self._state_tag = state_tag
+ self._current_value_accessor = current_value_accessor
+
+ @staticmethod
+ def for_spec(state_spec, state_tag, current_value_accessor):
+ if isinstance(state_spec, userstate.BagStateSpec):
+ return BagRuntimeState(state_spec, state_tag, current_value_accessor)
+ elif isinstance(state_spec, userstate.CombiningValueStateSpec):
+ return CombiningValueRuntimeState(state_spec, state_tag,
+ current_value_accessor)
+ elif isinstance(state_spec, userstate.SetStateSpec):
+ return SetRuntimeState(state_spec, state_tag, current_value_accessor)
+ else:
+ raise ValueError('Invalid state spec: %s' % state_spec)
+
+ def _encode(self, value):
+ return self._state_spec.coder.encode(value)
+
+ def _decode(self, value):
+ return self._state_spec.coder.decode(value)
+
+
+# Sentinel designating an unread value.
+UNREAD_VALUE = object()
+
+
+class BagRuntimeState(DirectRuntimeState, userstate.BagRuntimeState):
+ def __init__(self, state_spec, state_tag, current_value_accessor):
+ super(BagRuntimeState, self).__init__(
+ state_spec, state_tag, current_value_accessor)
+ self._cached_value = UNREAD_VALUE
+ self._cleared = False
+ self._new_values = []
+
+ def read(self):
+ if self._cached_value is UNREAD_VALUE:
+ self._cached_value = self._current_value_accessor()
+ if not self._cleared:
+ encoded_values = itertools.chain(self._cached_value, self._new_values)
+ else:
+ encoded_values = self._new_values
+ return (self._decode(v) for v in encoded_values)
+
+ def add(self, value):
+ self._new_values.append(self._encode(value))
+
+ def clear(self):
+ self._cleared = True
+ self._cached_value = []
+ self._new_values = []
+
+
+class SetRuntimeState(DirectRuntimeState, userstate.SetRuntimeState):
+ def __init__(self, state_spec, state_tag, current_value_accessor):
+ super(SetRuntimeState, self).__init__(
+ state_spec, state_tag, current_value_accessor)
+ self._current_accumulator = UNREAD_VALUE
+ self._modified = False
+
+ def _read_initial_value(self):
+ if self._current_accumulator is UNREAD_VALUE:
+ self._current_accumulator = {
+ self._decode(a) for a in self._current_value_accessor()
+ }
+
+ def read(self):
+ self._read_initial_value()
+ return self._current_accumulator
+
+ def add(self, value):
+ self._read_initial_value()
+ self._modified = True
+ self._current_accumulator.add(value)
+
+ def clear(self):
+ self._current_accumulator = set()
+ self._modified = True
+
+ def is_modified(self):
+ return self._modified
+
+
+class CombiningValueRuntimeState(
+ DirectRuntimeState, userstate.CombiningValueRuntimeState):
+ """Combining value state interface object passed to user code."""
+
+ def __init__(self, state_spec, state_tag, current_value_accessor):
+ super(CombiningValueRuntimeState, self).__init__(
+ state_spec, state_tag, current_value_accessor)
+ self._current_accumulator = UNREAD_VALUE
+ self._modified = False
+ self._combine_fn = state_spec.combine_fn
+
+ def _read_initial_value(self):
+ if self._current_accumulator is UNREAD_VALUE:
+ existing_accumulators = list(
+ self._decode(a) for a in self._current_value_accessor())
+ if existing_accumulators:
+ self._current_accumulator = self._combine_fn.merge_accumulators(
+ existing_accumulators)
+ else:
+ self._current_accumulator = self._combine_fn.create_accumulator()
+
+ def read(self):
+ self._read_initial_value()
+ return self._combine_fn.extract_output(self._current_accumulator)
+
+ def add(self, value):
+ self._read_initial_value()
+ self._modified = True
+ self._current_accumulator = self._combine_fn.add_input(
+ self._current_accumulator, value)
+
+ def clear(self):
+ self._modified = True
+ self._current_accumulator = self._combine_fn.create_accumulator()
class DirectUserStateContext(userstate.UserStateContext):
@@ -43,6 +167,8 @@
state_tag = _ListStateTag(state_key)
elif isinstance(state_spec, userstate.CombiningValueStateSpec):
state_tag = _ListStateTag(state_key)
+ elif isinstance(state_spec, userstate.SetStateSpec):
+ state_tag = _SetStateTag(state_key)
else:
raise ValueError('Invalid state spec: %s' % state_spec)
self.state_tags[state_spec] = state_tag
@@ -66,7 +192,7 @@
state_tag = self.state_tags[state_spec]
value_accessor = (
lambda: self._get_underlying_state(state_spec, key, window))
- self.cached_states[cache_key] = userstate.RuntimeState.for_spec(
+ self.cached_states[cache_key] = DirectRuntimeState.for_spec(
state_spec, state_tag, value_accessor)
return self.cached_states[cache_key]
@@ -93,6 +219,12 @@
state.add_state(
window, state_tag,
state_spec.coder.encode(runtime_state._current_accumulator))
+ elif isinstance(state_spec, userstate.SetStateSpec):
+ if runtime_state.is_modified():
+ state.clear_state(window, state_tag)
+ for new_value in runtime_state._current_accumulator:
+ state.add_state(
+ window, state_tag, state_spec.coder.encode(new_value))
else:
raise ValueError('Invalid state spec: %s' % state_spec)
diff --git a/sdks/python/apache_beam/runners/direct/helper_transforms.py b/sdks/python/apache_beam/runners/direct/helper_transforms.py
index 51377ba..2cdff58 100644
--- a/sdks/python/apache_beam/runners/direct/helper_transforms.py
+++ b/sdks/python/apache_beam/runners/direct/helper_transforms.py
@@ -19,6 +19,7 @@
import collections
import itertools
+import typing
import apache_beam as beam
from apache_beam import typehints
@@ -81,8 +82,8 @@
args = (typehints.Tuple[K, args[0]],) + args[1:]
hints.set_input_types(*args, **kwargs)
else:
- hints.set_input_types(typehints.Tuple[K, typehints.Any])
- hints.set_output_types(typehints.Tuple[K, typehints.Any])
+ hints.set_input_types(typehints.Tuple[K, typing.Any])
+ hints.set_output_types(typehints.Tuple[K, typing.Any])
return hints
@@ -102,7 +103,7 @@
def default_type_hints(self):
hints = self._combine_fn.get_type_hints().copy()
K = typehints.TypeVariable('K')
- hints.set_input_types(typehints.Tuple[K, typehints.Any])
+ hints.set_input_types(typehints.Tuple[K, typing.Any])
if hints.output_types:
main_output_type = hints.simple_output_type('')
hints.set_output_types(typehints.Tuple[K, main_output_type])
diff --git a/sdks/python/apache_beam/runners/direct/sdf_direct_runner.py b/sdks/python/apache_beam/runners/direct/sdf_direct_runner.py
index 6418632..3076790 100644
--- a/sdks/python/apache_beam/runners/direct/sdf_direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct/sdf_direct_runner.py
@@ -85,7 +85,7 @@
self.sdf = self._process_keyed_elements_transform.sdf
def expand(self, pcoll):
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
def new_process_fn(self, sdf):
return ProcessFn(
diff --git a/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py b/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
index 3e1e344..946ef34 100644
--- a/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
+++ b/sdks/python/apache_beam/runners/direct/sdf_direct_runner_test.py
@@ -29,6 +29,7 @@
from apache_beam import Create
from apache_beam import DoFn
from apache_beam.io import filebasedsource_test
+from apache_beam.io.restriction_trackers import OffsetRange
from apache_beam.io.restriction_trackers import OffsetRestrictionTracker
from apache_beam.pvalue import AsList
from apache_beam.pvalue import AsSingleton
@@ -45,10 +46,10 @@
def initial_restriction(self, element):
size = os.path.getsize(element)
- return (0, size)
+ return OffsetRange(0, size)
def create_tracker(self, restriction):
- return OffsetRestrictionTracker(*restriction)
+ return OffsetRestrictionTracker(restriction)
class ReadFiles(DoFn):
@@ -94,11 +95,12 @@
class ExpandStringsProvider(RestrictionProvider):
def initial_restriction(self, element):
- return (0, len(element[0]))
+ return OffsetRange(0, len(element[0]))
def create_tracker(self, restriction):
- return OffsetRestrictionTracker(restriction[0], restriction[1])
+ return OffsetRestrictionTracker(restriction)
+ # No initial split performed.
def split(self, element, restriction):
return [restriction,]
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index fad0704..e1fc3cd 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -19,10 +19,12 @@
from __future__ import absolute_import
+import atexit
import collections
import logging
import random
import time
+import typing
from builtins import object
from future.utils import iteritems
@@ -30,7 +32,6 @@
import apache_beam.io as io
from apache_beam import coders
from apache_beam import pvalue
-from apache_beam import typehints
from apache_beam.internal import pickler
from apache_beam.runners import common
from apache_beam.runners.common import DoFnRunner
@@ -376,45 +377,11 @@
self, self.bundles, unprocessed_bundles, None, {None: hold})
-class _PubSubSubscriptionWrapper(object):
- """Wrapper for managing temporary PubSub subscriptions."""
-
- def __init__(self, project, short_topic_name, short_sub_name):
- """Initialize subscription wrapper.
-
- If sub_name is None, will create a temporary subscription to topic_name.
-
- Args:
- project: GCP project name for topic and subscription. May be None.
- Required if sub_name is None.
- short_topic_name: Valid topic name without
- 'projects/{project}/topics/' prefix. May be None.
- Required if sub_name is None.
- short_sub_name: Valid subscription name without
- 'projects/{project}/subscriptions/' prefix. May be None.
- """
- from google.cloud import pubsub
- self.sub_client = pubsub.SubscriberClient()
-
- if short_sub_name is None:
- self.sub_name = self.sub_client.subscription_path(
- project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32)))
- topic_name = self.sub_client.topic_path(project, short_topic_name)
- self.sub_client.create_subscription(self.sub_name, topic_name)
- self._should_cleanup = True
- else:
- self.sub_name = self.sub_client.subscription_path(project, short_sub_name)
- self._should_cleanup = False
-
- def __del__(self):
- if self._should_cleanup:
- self.sub_client.delete_subscription(self.sub_name)
-
-
class _PubSubReadEvaluator(_TransformEvaluator):
"""TransformEvaluator for PubSub read."""
# A mapping of transform to _PubSubSubscriptionWrapper.
+ # TODO(BEAM-7750): Prevents garbage collection of pipeline instances.
_subscription_cache = {}
def __init__(self, evaluation_context, applied_ptransform,
@@ -428,16 +395,29 @@
if self.source.id_label:
raise NotImplementedError(
'DirectRunner: id_label is not supported for PubSub reads')
- self._sub_name = _PubSubReadEvaluator.get_subscription(
+ self._sub_name = self.get_subscription(
self._applied_ptransform, self.source.project, self.source.topic_name,
self.source.subscription_name)
@classmethod
- def get_subscription(cls, transform, project, topic, short_sub_name):
- if transform not in cls._subscription_cache:
- wrapper = _PubSubSubscriptionWrapper(project, topic, short_sub_name)
- cls._subscription_cache[transform] = wrapper
- return cls._subscription_cache[transform].sub_name
+ def get_subscription(cls, transform, project, short_topic_name,
+ short_sub_name):
+ from google.cloud import pubsub
+
+ if short_sub_name:
+ return pubsub.SubscriberClient.subscription_path(project, short_sub_name)
+
+ if transform in cls._subscription_cache:
+ return cls._subscription_cache[transform]
+
+ sub_client = pubsub.SubscriberClient()
+ sub_name = sub_client.subscription_path(
+ project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32)))
+ topic_name = sub_client.topic_path(project, short_topic_name)
+ sub_client.create_subscription(sub_name, topic_name)
+ atexit.register(sub_client.delete_subscription, sub_name)
+ cls._subscription_cache[transform] = sub_name
+ return cls._subscription_cache[transform]
def start_bundle(self):
pass
@@ -448,13 +428,6 @@
def _read_from_pubsub(self, timestamp_attribute):
from apache_beam.io.gcp.pubsub import PubsubMessage
from google.cloud import pubsub
- # Because of the AutoAck, we are not able to reread messages if this
- # evaluator fails with an exception before emitting a bundle. However,
- # the DirectRunner currently doesn't retry work items anyway, so the
- # pipeline would enter an inconsistent state on any error.
- sub_client = pubsub.SubscriberClient()
- response = sub_client.pull(self._sub_name, max_messages=10,
- return_immediately=True)
def _get_element(message):
parsed_message = PubsubMessage._from_message(message)
@@ -462,10 +435,10 @@
timestamp_attribute in parsed_message.attributes):
rfc3339_or_milli = parsed_message.attributes[timestamp_attribute]
try:
- timestamp = Timestamp.from_rfc3339(rfc3339_or_milli)
+ timestamp = Timestamp(micros=int(rfc3339_or_milli) * 1000)
except ValueError:
try:
- timestamp = Timestamp(micros=int(rfc3339_or_milli) * 1000)
+ timestamp = Timestamp.from_rfc3339(rfc3339_or_milli)
except ValueError as e:
raise ValueError('Bad timestamp value: %s' % e)
else:
@@ -474,10 +447,20 @@
return timestamp, parsed_message
- results = [_get_element(rm.message) for rm in response.received_messages]
- ack_ids = [rm.ack_id for rm in response.received_messages]
- if ack_ids:
- sub_client.acknowledge(self._sub_name, ack_ids)
+ # Because of the AutoAck, we are not able to reread messages if this
+ # evaluator fails with an exception before emitting a bundle. However,
+ # the DirectRunner currently doesn't retry work items anyway, so the
+ # pipeline would enter an inconsistent state on any error.
+ sub_client = pubsub.SubscriberClient()
+ try:
+ response = sub_client.pull(self._sub_name, max_messages=10,
+ return_immediately=True)
+ results = [_get_element(rm.message) for rm in response.received_messages]
+ ack_ids = [rm.ack_id for rm in response.received_messages]
+ if ack_ids:
+ sub_client.acknowledge(self._sub_name, ack_ids)
+ finally:
+ sub_client.api.transport.channel.close()
return results
@@ -599,11 +582,11 @@
self.user_timer_map = {}
if is_stateful_dofn(dofn):
kv_type_hint = self._applied_ptransform.inputs[0].element_type
- if kv_type_hint and kv_type_hint != typehints.Any:
+ if kv_type_hint and kv_type_hint != typing.Any:
coder = coders.registry.get_coder(kv_type_hint)
self.key_coder = coder.key_coder()
else:
- self.key_coder = coders.registry.get_coder(typehints.Any)
+ self.key_coder = coders.registry.get_coder(typing.Any)
self.user_state_context = DirectUserStateContext(
self._step_context, dofn, self.key_coder)
@@ -666,7 +649,7 @@
assert len(self._outputs) == 1
self.output_pcollection = list(self._outputs)[0]
- # The output type of a GroupByKey will be KV[Any, Any] or more specific.
+ # The output type of a GroupByKey will be Tuple[Any, Any] or more specific.
# TODO(BEAM-2717): Infer coders earlier.
kv_type_hint = (
self._applied_ptransform.outputs[None].element_type
@@ -756,10 +739,10 @@
assert len(self._outputs) == 1
self.output_pcollection = list(self._outputs)[0]
- # The input type of a GroupByKey will be KV[Any, Any] or more specific.
+ # The input type of a GroupByKey will be Tuple[Any, Any] or more specific.
kv_type_hint = self._applied_ptransform.inputs[0].element_type
key_type_hint = (kv_type_hint.tuple_types[0] if kv_type_hint
- else typehints.Any)
+ else typing.Any)
self.key_coder = coders.registry.get_coder(key_type_hint)
def process_element(self, element):
@@ -812,10 +795,10 @@
self.keyed_holds = {}
# The input type (which is the same as the output type) of a
- # GroupAlsoByWindow will be KV[Any, Iter[Any]] or more specific.
+ # GroupAlsoByWindow will be Tuple[Any, Iter[Any]] or more specific.
kv_type_hint = self._applied_ptransform.outputs[None].element_type
key_type_hint = (kv_type_hint.tuple_types[0] if kv_type_hint
- else typehints.Any)
+ else typing.Any)
self.key_coder = coders.registry.get_coder(key_type_hint)
def process_element(self, element):
diff --git a/sdks/python/apache_beam/runners/interactive/README.md b/sdks/python/apache_beam/runners/interactive/README.md
index 76200ba..6f187de 100644
--- a/sdks/python/apache_beam/runners/interactive/README.md
+++ b/sdks/python/apache_beam/runners/interactive/README.md
@@ -224,8 +224,8 @@
* Build the SDK container and start the local FlinkService.
```bash
- $ ./gradlew -p sdks/python/container docker
- $ ./gradlew beam-runners-flink_2.11-job-server:runShadow # Blocking
+ $ ./gradlew -p sdks/python/container/py35 docker # Optionally replace py35 with the Python version of your choice
+ $ ./gradlew :runners:flink:1.8:job-server:runShadow # Blocking
```
* Run `$ jupyter notebook` in another terminal.
diff --git a/sdks/python/apache_beam/runners/interactive/cache_manager.py b/sdks/python/apache_beam/runners/interactive/cache_manager.py
index e8816fe..20d84e3 100644
--- a/sdks/python/apache_beam/runners/interactive/cache_manager.py
+++ b/sdks/python/apache_beam/runners/interactive/cache_manager.py
@@ -28,6 +28,8 @@
import apache_beam as beam
from apache_beam import coders
from apache_beam.io import filesystems
+from apache_beam.io import textio
+from apache_beam.io import tfrecordio
from apache_beam.transforms import combiners
try: # Python 3
@@ -62,9 +64,12 @@
def read(self, *labels):
"""Return the PCollection as a list as well as the version number.
+ Args:
+ *labels: List of labels for PCollection instance.
+
Returns:
- (List[PCollection])
- (int) the version number
+ Tuple[List[Any], int]: A tuple containing a list of items in the
+ PCollection and the version number.
It is possible that the version numbers from read() and_latest_version()
are different. This usually means that the cache's been evicted (thus
@@ -81,6 +86,25 @@
"""Returns a beam.io.Sink that writes the PCollection cache."""
raise NotImplementedError
+ def save_pcoder(self, pcoder, *labels):
+ """Saves pcoder for given PCollection.
+
+ Correct reading of PCollection from Cache requires PCoder to be known.
+ This method saves desired PCoder for PCollection that will subsequently
+ be used by sink(...), source(...), and, most importantly, read(...) method.
+ The latter must be able to read a PCollection written by Beam using
+ non-Beam IO.
+
+ Args:
+ pcoder: A PCoder to be used for reading and writing a PCollection.
+ *labels: List of labels for PCollection instance.
+ """
+ raise NotImplementedError
+
+ def load_pcoder(self, *labels):
+ """Returns previously saved PCoder for reading and writing PCollection."""
+ raise NotImplementedError
+
def cleanup(self):
"""Cleans up all the PCollection caches."""
raise NotImplementedError
@@ -89,7 +113,12 @@
class FileBasedCacheManager(CacheManager):
"""Maps PCollections to local temp files for materialization."""
- def __init__(self, cache_dir=None):
+ _available_formats = {
+ 'text': (textio.ReadFromText, textio.WriteToText),
+ 'tfrecord': (tfrecordio.ReadFromTFRecord, tfrecordio.WriteToTFRecord)
+ }
+
+ def __init__(self, cache_dir=None, cache_format='text'):
if cache_dir:
self._cache_dir = filesystems.FileSystems.join(
cache_dir,
@@ -99,6 +128,25 @@
prefix='interactive-temp-', dir=os.environ.get('TEST_TMPDIR', None))
self._versions = collections.defaultdict(lambda: self._CacheVersion())
+ if cache_format not in self._available_formats:
+ raise ValueError("Unsupported cache format: '%s'." % cache_format)
+ self._reader_class, self._writer_class = self._available_formats[
+ cache_format]
+ self._default_pcoder = (
+ SafeFastPrimitivesCoder() if cache_format == 'text' else None)
+
+ # List of saved pcoders keyed by PCollection path. It is OK to keep this
+ # list in memory because once FileBasedCacheManager object is
+ # destroyed/re-created it loses the access to previously written cache
+ # objects anyways even if cache_dir already exists. In other words,
+ # it is not possible to resume execution of Beam pipeline from the
+ # saved cache if FileBasedCacheManager has been reset.
+ #
+ # However, if we are to implement better cache persistence, one needs
+ # to take care of keeping consistency between the cached PCollection
+ # and its PCoder type.
+ self._saved_pcoders = {}
+
def exists(self, *labels):
return bool(self._match(*labels))
@@ -109,29 +157,35 @@
result = self._versions["-".join(labels)].get_version(timestamp)
return result
+ def save_pcoder(self, pcoder, *labels):
+ self._saved_pcoders[self._path(*labels)] = pcoder
+
+ def load_pcoder(self, *labels):
+ return (self._default_pcoder if self._default_pcoder is not None else
+ self._saved_pcoders[self._path(*labels)])
+
def read(self, *labels):
if not self.exists(*labels):
return [], -1
- def _read_helper():
- coder = SafeFastPrimitivesCoder()
- for path in self._match(*labels):
- for line in filesystems.FileSystems.open(path):
- yield coder.decode(line.strip())
- result, version = list(_read_helper()), self._latest_version(*labels)
+ source = self.source(*labels)
+ range_tracker = source.get_range_tracker(None, None)
+ result = list(source.read(range_tracker))
+ version = self._latest_version(*labels)
return result, version
def source(self, *labels):
- return beam.io.ReadFromText(self._glob_path(*labels),
- coder=SafeFastPrimitivesCoder())._source
+ return self._reader_class(
+ self._glob_path(*labels), coder=self.load_pcoder(*labels))._source
def sink(self, *labels):
- return beam.io.WriteToText(self._path(*labels),
- coder=SafeFastPrimitivesCoder())._sink
+ return self._writer_class(
+ self._path(*labels), coder=self.load_pcoder(*labels))._sink
def cleanup(self):
if filesystems.FileSystems.exists(self._cache_dir):
filesystems.FileSystems.delete([self._cache_dir])
+ self._saved_pcoders = {}
def _glob_path(self, *labels):
return self._path(*labels) + '-*-of-*'
@@ -188,6 +242,14 @@
def expand(self, pcoll):
prefix = 'sample' if self._sample else 'full'
+
+ # We save pcoder that is necessary for proper reading of
+ # cached PCollection. _cache_manager.sink(...) call below
+ # should be using this saved pcoder.
+ self._cache_manager.save_pcoder(
+ coders.registry.get_coder(pcoll.element_type),
+ prefix, self._label)
+
if self._sample:
pcoll |= 'Sample' >> (
combiners.Sample.FixedSizeGlobally(self._sample_size)
diff --git a/sdks/python/apache_beam/runners/interactive/cache_manager_test.py b/sdks/python/apache_beam/runners/interactive/cache_manager_test.py
index 641643f..3ad81b8 100644
--- a/sdks/python/apache_beam/runners/interactive/cache_manager_test.py
+++ b/sdks/python/apache_beam/runners/interactive/cache_manager_test.py
@@ -25,14 +25,15 @@
import time
import unittest
+from apache_beam import coders
from apache_beam.io import filesystems
from apache_beam.runners.interactive import cache_manager as cache
-class FileBasedCacheManagerTest(unittest.TestCase):
+class FileBasedCacheManagerTest(object):
"""Unit test for FileBasedCacheManager.
- Note that this set of tests focuses only the the methods that interacts with
+ Note that this set of tests focuses only the methods that interacts with
the LOCAL file system. The idea is that once FileBasedCacheManager works well
with the local file system, it should work with any file system with
`apache_beam.io.filesystem` interface. Those tests that involve interactions
@@ -40,9 +41,12 @@
tested with InteractiveRunner as a part of integration tests instead.
"""
+ cache_format = None
+
def setUp(self):
self.test_dir = tempfile.mkdtemp()
- self.cache_manager = cache.FileBasedCacheManager(self.test_dir)
+ self.cache_manager = cache.FileBasedCacheManager(
+ self.test_dir, cache_format=self.cache_format)
def tearDown(self):
# The test_dir might have already been removed by cache_manager.cleanup().
@@ -61,10 +65,16 @@
time.sleep(0.1)
cache_file = cache_label + '-1-of-2'
+ labels = [prefix, cache_label]
+
+ # Usually, the pcoder will be inferred from `pcoll.element_type`
+ pcoder = coders.registry.get_coder(object)
+ self.cache_manager.save_pcoder(pcoder, *labels)
+ sink = self.cache_manager.sink(*labels)
+
with open(self.cache_manager._path(prefix, cache_file), 'wb') as f:
for line in pcoll_list:
- f.write(cache.SafeFastPrimitivesCoder().encode(line))
- f.write(b'\n')
+ sink.write_record(f, line)
def test_exists(self):
"""Test that CacheManager can correctly tell if the cache exists or not."""
@@ -163,5 +173,21 @@
self.cache_manager.is_latest_version(version, prefix, cache_label))
+class TextFileBasedCacheManagerTest(
+ FileBasedCacheManagerTest,
+ unittest.TestCase,
+):
+
+ cache_format = 'text'
+
+
+class TFRecordBasedCacheManagerTest(
+ FileBasedCacheManagerTest,
+ unittest.TestCase,
+):
+
+ cache_format = 'tfrecord'
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/display/pipeline_graph_renderer.py b/sdks/python/apache_beam/runners/interactive/display/pipeline_graph_renderer.py
index a216d19..2df5c61 100644
--- a/sdks/python/apache_beam/runners/interactive/display/pipeline_graph_renderer.py
+++ b/sdks/python/apache_beam/runners/interactive/display/pipeline_graph_renderer.py
@@ -94,7 +94,7 @@
return 'graph'
def render_pipeline_graph(self, pipeline_graph):
- return pipeline_graph._get_graph().create_svg() # pylint: disable=protected-access
+ return pipeline_graph._get_graph().create_svg().decode("utf-8") # pylint: disable=protected-access
def get_renderer(option=None):
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_beam.py b/sdks/python/apache_beam/runners/interactive/interactive_beam.py
new file mode 100644
index 0000000..a7a7584
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_beam.py
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module of Interactive Beam features that can be used in notebook.
+
+The purpose of the module is to reduce the learning curve of Interactive Beam
+users, provide a single place for importing and add sugar syntax for all
+Interactive Beam components. It gives users capability to interact with existing
+environment/session/context for Interactive Beam and visualize PCollections as
+bounded dataset. In the meantime, it hides the interactivity implementation
+from users so that users can focus on developing Beam pipeline without worrying
+about how hidden states in the interactive session are managed.
+
+Note: If you want backward-compatibility, only invoke interfaces provided by
+this module in your notebook or application code.
+"""
+from __future__ import absolute_import
+
+from apache_beam.runners.interactive import interactive_environment as ie
+
+
+def watch(watchable):
+ """Monitors a watchable.
+
+ This allows Interactive Beam to implicitly pass on the information about the
+ location of your pipeline definition.
+
+ Current implementation mainly watches for PCollection variables defined in
+ user code. A watchable can be a dictionary of variable metadata such as
+ locals(), a str name of a module, a module object or an instance of a class.
+ The variable can come from any scope even local variables in a method of a
+ class defined in a module.
+
+ Below are all valid::
+
+ watch(__main__) # if import __main__ is already invoked
+ watch('__main__') # does not require invoking import __main__ beforehand
+ watch(self) # inside a class
+ watch(SomeInstance()) # an instance of a class
+ watch(locals()) # inside a function, watching local variables within
+
+ If you write a Beam pipeline in the __main__ module directly, since the
+ __main__ module is always watched, you don't have to instruct Interactive
+ Beam. If your Beam pipeline is defined in some module other than __main__,
+ such as inside a class function or a unit test, you can watch() the scope.
+
+ For example::
+
+ class Foo(object)
+ def run_pipeline(self):
+ p = beam.Pipeline()
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ watch(locals())
+ p.run()
+ return init_pcoll
+ init_pcoll = Foo().run_pipeline()
+
+ Interactive Beam caches init_pcoll for the first run.
+
+ Then you can use::
+
+ visualize(init_pcoll)
+
+ To visualize data from init_pcoll once the pipeline is executed.
+ """
+ ie.current_env().watch(watchable)
+
+
+def visualize(pcoll):
+ """Visualizes a PCollection."""
+ # TODO(BEAM-7926)
+ pass
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_beam_test.py b/sdks/python/apache_beam/runners/interactive/interactive_beam_test.py
new file mode 100644
index 0000000..7660b1a
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_beam_test.py
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.interactive_beam."""
+from __future__ import absolute_import
+
+import importlib
+import unittest
+
+from apache_beam.runners.interactive import interactive_beam as ib
+from apache_beam.runners.interactive import interactive_environment as ie
+
+# The module name is also a variable in module.
+_module_name = 'apache_beam.runners.interactive.interactive_beam_test'
+
+
+class InteractiveBeamTest(unittest.TestCase):
+
+ def setUp(self):
+ self._var_in_class_instance = 'a var in class instance, not directly used'
+ ie.new_env()
+
+ def test_watch_main_by_default(self):
+ test_env = ie.InteractiveEnvironment()
+ # Current Interactive Beam env fetched and the test env are 2 instances.
+ self.assertNotEqual(id(ie.current_env()), id(test_env))
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_a_module_by_name(self):
+ test_env = ie.InteractiveEnvironment()
+ ib.watch(_module_name)
+ test_env.watch(_module_name)
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_a_module_by_module_object(self):
+ test_env = ie.InteractiveEnvironment()
+ module = importlib.import_module(_module_name)
+ ib.watch(module)
+ test_env.watch(module)
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_locals(self):
+ # test_env serves as local var too.
+ test_env = ie.InteractiveEnvironment()
+ ib.watch(locals())
+ test_env.watch(locals())
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+ def test_watch_class_instance(self):
+ test_env = ie.InteractiveEnvironment()
+ ib.watch(self)
+ test_env.watch(self)
+ self.assertEqual(ie.current_env().watching(), test_env.watching())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_environment.py b/sdks/python/apache_beam/runners/interactive/interactive_environment.py
new file mode 100644
index 0000000..3dee1e3
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_environment.py
@@ -0,0 +1,107 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module of the current Interactive Beam environment.
+
+For internal use only; no backwards-compatibility guarantees.
+Provides interfaces to interact with existing Interactive Beam environment.
+External Interactive Beam users please use interactive_beam module in
+application code or notebook.
+"""
+from __future__ import absolute_import
+
+import importlib
+
+_interactive_beam_env = None
+
+
+def current_env(cache_manager=None):
+ """Gets current Interactive Beam environment."""
+ global _interactive_beam_env
+ if not _interactive_beam_env:
+ _interactive_beam_env = InteractiveEnvironment(cache_manager)
+ return _interactive_beam_env
+
+
+def new_env(cache_manager=None):
+ """Creates a new Interactive Beam environment to replace current one."""
+ global _interactive_beam_env
+ _interactive_beam_env = None
+ return current_env(cache_manager)
+
+
+class InteractiveEnvironment(object):
+ """An interactive environment with cache and pipeline variable metadata.
+
+ Interactive Beam will use the watched variable information to determine if a
+ PCollection is assigned to a variable in user pipeline definition. When
+ executing the pipeline, interactivity is applied with implicit cache
+ mechanism for those PCollections if the pipeline is interactive. Users can
+ also visualize and introspect those PCollections in user code since they have
+ handles to the variables.
+ """
+
+ def __init__(self, cache_manager=None):
+ self._cache_manager = cache_manager
+ # Holds class instances, module object, string of module names.
+ self._watching_set = set()
+ # Holds variables list of (Dict[str, object]).
+ self._watching_dict_list = []
+ # Always watch __main__ module.
+ self.watch('__main__')
+
+ def watch(self, watchable):
+ """Watches a watchable.
+
+ A watchable can be a dictionary of variable metadata such as locals(), a str
+ name of a module, a module object or an instance of a class. The variable
+ can come from any scope even local. Duplicated variable naming doesn't
+ matter since they are different instances. Duplicated variables are also
+ allowed when watching.
+ """
+ if isinstance(watchable, dict):
+ self._watching_dict_list.append(watchable.items())
+ else:
+ self._watching_set.add(watchable)
+
+ def watching(self):
+ """Analyzes and returns a list of pair lists referring to variable names and
+ values from watched scopes.
+
+ Each entry in the list represents the variable defined within a watched
+ watchable. Currently, each entry holds a list of pairs. The format might
+ change in the future to hold more metadata. Duplicated pairs are allowed.
+ And multiple paris can have the same variable name as the "first" while
+ having different variable values as the "second" since variables in
+ different scopes can have the same name.
+ """
+ watching = list(self._watching_dict_list)
+ for watchable in self._watching_set:
+ if isinstance(watchable, str):
+ module = importlib.import_module(watchable)
+ watching.append(vars(module).items())
+ else:
+ watching.append(vars(watchable).items())
+ return watching
+
+ def set_cache_manager(self, cache_manager):
+ """Sets the cache manager held by current Interactive Environment."""
+ self._cache_manager = cache_manager
+
+ def cache_manager(self):
+ """Gets the cache manager held by current Interactive Environment."""
+ return self._cache_manager
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_environment_test.py b/sdks/python/apache_beam/runners/interactive/interactive_environment_test.py
new file mode 100644
index 0000000..95bb163
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/interactive_environment_test.py
@@ -0,0 +1,93 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.interactive_environment."""
+from __future__ import absolute_import
+
+import importlib
+import unittest
+
+from apache_beam.runners.interactive import interactive_environment as ie
+
+# The module name is also a variable in module.
+_module_name = 'apache_beam.runners.interactive.interactive_environment_test'
+
+
+class InteractiveEnvironmentTest(unittest.TestCase):
+
+ def setUp(self):
+ self._var_in_class_instance = 'a var in class instance'
+ ie.new_env()
+
+ def assertVariableWatched(self, variable_name, variable_val):
+ self.assertTrue(self._is_variable_watched(variable_name, variable_val))
+
+ def assertVariableNotWatched(self, variable_name, variable_val):
+ self.assertFalse(self._is_variable_watched(variable_name, variable_val))
+
+ def _is_variable_watched(self, variable_name, variable_val):
+ return any([(variable_name, variable_val) in watching for watching in
+ ie.current_env().watching()])
+
+ def _a_function_with_local_watched(self):
+ local_var_watched = 123 # pylint: disable=unused-variable
+ ie.current_env().watch(locals())
+
+ def _a_function_not_watching_local(self):
+ local_var_not_watched = 456 # pylint: disable=unused-variable
+
+ def test_watch_main_by_default(self):
+ self.assertTrue('__main__' in ie.current_env()._watching_set)
+ # __main__ module has variable __name__ with value '__main__'
+ self.assertVariableWatched('__name__', '__main__')
+
+ def test_watch_a_module_by_name(self):
+ self.assertFalse(
+ _module_name in ie.current_env()._watching_set)
+ self.assertVariableNotWatched('_module_name', _module_name)
+ ie.current_env().watch(_module_name)
+ self.assertTrue(
+ _module_name in
+ ie.current_env()._watching_set)
+ self.assertVariableWatched('_module_name', _module_name)
+
+ def test_watch_a_module_by_module_object(self):
+ module = importlib.import_module(_module_name)
+ self.assertFalse(module in ie.current_env()._watching_set)
+ self.assertVariableNotWatched('_module_name', _module_name)
+ ie.current_env().watch(module)
+ self.assertTrue(module in ie.current_env()._watching_set)
+ self.assertVariableWatched('_module_name', _module_name)
+
+ def test_watch_locals(self):
+ self.assertVariableNotWatched('local_var_watched', 123)
+ self.assertVariableNotWatched('local_var_not_watched', 456)
+ self._a_function_with_local_watched()
+ self.assertVariableWatched('local_var_watched', 123)
+ self._a_function_not_watching_local()
+ self.assertVariableNotWatched('local_var_not_watched', 456)
+
+ def test_watch_class_instance(self):
+ self.assertVariableNotWatched('_var_in_class_instance',
+ self._var_in_class_instance)
+ ie.current_env().watch(self)
+ self.assertVariableWatched('_var_in_class_instance',
+ self._var_in_class_instance)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/interactive/interactive_runner.py b/sdks/python/apache_beam/runners/interactive/interactive_runner.py
index 391f3f0..4bf125e 100644
--- a/sdks/python/apache_beam/runners/interactive/interactive_runner.py
+++ b/sdks/python/apache_beam/runners/interactive/interactive_runner.py
@@ -44,19 +44,24 @@
Allows interactively building and running Beam Python pipelines.
"""
- def __init__(self, underlying_runner=None, cache_dir=None,
+ def __init__(self,
+ underlying_runner=None,
+ cache_dir=None,
+ cache_format='text',
render_option=None):
"""Constructor of InteractiveRunner.
Args:
underlying_runner: (runner.PipelineRunner)
cache_dir: (str) the directory where PCollection caches are kept
+ cache_format: (str) the file format that should be used for saving
+ PCollection caches. Available options are 'text' and 'tfrecord'.
render_option: (str) this parameter decides how the pipeline graph is
rendered. See display.pipeline_graph_renderer for available options.
"""
self._underlying_runner = (underlying_runner
or direct_runner.DirectRunner())
- self._cache_manager = cache.FileBasedCacheManager(cache_dir)
+ self._cache_manager = cache.FileBasedCacheManager(cache_dir, cache_format)
self._renderer = pipeline_graph_renderer.get_renderer(render_option)
self._in_session = False
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py b/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
new file mode 100644
index 0000000..63664ed
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
@@ -0,0 +1,433 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Module to instrument interactivity to the given pipeline.
+
+For internal use only; no backwards-compatibility guarantees.
+This module accesses current interactive environment and analyzes given pipeline
+to transform original pipeline into a one-shot pipeline with interactivity.
+"""
+from __future__ import absolute_import
+
+import apache_beam as beam
+from apache_beam.pipeline import PipelineVisitor
+from apache_beam.runners.interactive import cache_manager as cache
+from apache_beam.runners.interactive import interactive_environment as ie
+
+READ_CACHE = "_ReadCache_"
+WRITE_CACHE = "_WriteCache_"
+
+
+class PipelineInstrument(object):
+ """A pipeline instrument for pipeline to be executed by interactive runner.
+
+ This module should never depend on underlying runner that interactive runner
+ delegates. It instruments the original instance of pipeline directly by
+ appending or replacing transforms with help of cache. It provides
+ interfaces to recover states of original pipeline. It's the interactive
+ runner's responsibility to coordinate supported underlying runners to run
+ the pipeline instrumented and recover the original pipeline states if needed.
+ """
+
+ def __init__(self, pipeline, options=None):
+ self._pipeline = pipeline
+ # The cache manager should be initiated outside of this module and outside
+ # of run_pipeline() from interactive runner so that its lifespan could cover
+ # multiple runs in the interactive environment. Owned by
+ # interactive_environment module. Not owned by this module.
+ # TODO(BEAM-7760): change the scope of cache to be owned by runner or
+ # pipeline result instances because a pipeline is not 1:1 correlated to a
+ # running job. Only complete and read-only cache is valid across multiple
+ # jobs. Other cache instances should have their own scopes. Some design
+ # change should support only runner.run(pipeline) pattern rather than
+ # pipeline.run([runner]) and a runner can only run at most one pipeline at a
+ # time. Otherwise, result returned by run() is the only 1:1 anchor.
+ self._cache_manager = ie.current_env().cache_manager()
+
+ # Invoke a round trip through the runner API. This makes sure the Pipeline
+ # proto is stable. The snapshot of pipeline will not be mutated within this
+ # module and can be used to recover original pipeline if needed.
+ self._pipeline_snap = beam.pipeline.Pipeline.from_runner_api(
+ pipeline.to_runner_api(use_fake_coders=True),
+ pipeline.runner,
+ options)
+ # Snapshot of original pipeline information.
+ (self._original_pipeline_proto,
+ self._original_context) = self._pipeline_snap.to_runner_api(
+ return_context=True, use_fake_coders=True)
+
+ # All compute-once-against-original-pipeline fields.
+ self._has_unbounded_source = has_unbounded_source(self._pipeline_snap)
+ # TODO(BEAM-7760): once cache scope changed, this is not needed to manage
+ # relationships across pipelines, runners, and jobs.
+ self._pcolls_to_pcoll_id = pcolls_to_pcoll_id(self._pipeline_snap,
+ self._original_context)
+
+ # A mapping from PCollection id to python id() value in user defined
+ # pipeline instance.
+ (self._pcoll_version_map,
+ self._cacheables) = cacheables(self.pcolls_to_pcoll_id)
+
+ # A dict from cache key to PCollection that is read from cache.
+ # If exists, caller should reuse the PCollection read. If not, caller
+ # should create new transform and track the PCollection read from cache.
+ # (Dict[str, AppliedPTransform]).
+ self._cached_pcoll_read = {}
+
+ def instrumented_pipeline_proto(self):
+ """Always returns a new instance of portable instrumented proto."""
+ return self._pipeline.to_runner_api(use_fake_coders=True)
+
+ @property
+ def has_unbounded_source(self):
+ """Checks if a given pipeline has any source that is unbounded.
+
+ The function directly checks the source transform definition instead
+ of pvalues in the pipeline. Thus manually setting is_bounded field of
+ a PCollection or switching streaming mode will not affect this
+ function's result. The result is always deterministic when the source
+ code of a pipeline is defined.
+ """
+ return self._has_unbounded_source
+
+ @property
+ def cacheables(self):
+ """Finds cacheable PCollections from the pipeline.
+
+ The function only treats the result as cacheables since there is no
+ guarantee whether the cache desired PCollection has been cached or
+ not. A PCollection desires caching when it's bound to a user defined
+ variable in source code. Otherwise, the PCollection is not reusale
+ nor introspectable which nullifying the need of cache.
+ """
+ return self._cacheables
+
+ @property
+ def pcolls_to_pcoll_id(self):
+ """Returns a dict mapping str(PCollection)s to IDs."""
+ return self._pcolls_to_pcoll_id
+
+ @property
+ def original_pipeline_proto(self):
+ """Returns the portable proto representation of the pipeline before
+ instrumentation."""
+ return self._original_pipeline_proto
+
+ @property
+ def original_pipeline(self):
+ """Returns a snapshot of the pipeline before instrumentation."""
+ return self._pipeline_snap
+
+ def instrument(self):
+ """Instruments original pipeline with cache.
+
+ For cacheable output PCollection, if cache for the key doesn't exist, do
+ _write_cache(); for cacheable input PCollection, if cache for the key
+ exists, do _read_cache(). No instrument in any other situation.
+
+ Modifies:
+ self._pipeline
+ """
+ self._preprocess()
+ cacheable_inputs = set()
+
+ class InstrumentVisitor(PipelineVisitor):
+ """Visitor utilizes cache to instrument the pipeline."""
+
+ def __init__(self, pin):
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ cacheable_inputs.update(self._pin._cacheable_inputs(transform_node))
+
+ v = InstrumentVisitor(self)
+ self._pipeline.visit(v)
+ # Create ReadCache transforms.
+ for cacheable_input in cacheable_inputs:
+ self._read_cache(cacheable_input)
+ # Replace/wire inputs w/ cached PCollections from ReadCache transforms.
+ self._replace_with_cached_inputs()
+ # Write cache for all cacheables.
+ for _, cacheable in self.cacheables.items():
+ self._write_cache(cacheable['pcoll'])
+ # TODO(BEAM-7760): prune sub graphs that doesn't need to be executed.
+
+ def _preprocess(self):
+ """Pre-processes the pipeline.
+
+ Since the pipeline instance in the class might not be the same instance
+ defined in the user code, the pre-process will figure out the relationship
+ of cacheable PCollections between these 2 instances by replacing 'pcoll'
+ fields in the cacheable dictionary with ones from the running instance.
+ """
+
+ class PreprocessVisitor(PipelineVisitor):
+
+ def __init__(self, pin):
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ for in_pcoll in transform_node.inputs:
+ self._process(in_pcoll)
+ for out_pcoll in transform_node.outputs.values():
+ self._process(out_pcoll)
+
+ def _process(self, pcoll):
+ pcoll_id = self._pin.pcolls_to_pcoll_id.get(str(pcoll), '')
+ if pcoll_id in self._pin._pcoll_version_map:
+ cacheable_key = self._pin._cacheable_key(pcoll)
+ if (cacheable_key in self._pin.cacheables and
+ self._pin.cacheables[cacheable_key]['pcoll'] != pcoll):
+ self._pin.cacheables[cacheable_key]['pcoll'] = pcoll
+
+ v = PreprocessVisitor(self)
+ self._pipeline.visit(v)
+
+ def _write_cache(self, pcoll):
+ """Caches a cacheable PCollection.
+
+ For the given PCollection, by appending sub transform part that materialize
+ the PCollection through sink into cache implementation. The cache write is
+ not immediate. It happens when the runner runs the transformed pipeline
+ and thus not usable for this run as intended. This function always writes
+ the cache for the given PCollection as long as the PCollection belongs to
+ the pipeline being instrumented and the keyed cache is absent.
+
+ Modifies:
+ self._pipeline
+ """
+ # Makes sure the pcoll belongs to the pipeline being instrumented.
+ if pcoll.pipeline is not self._pipeline:
+ return
+ # The keyed cache is always valid within this instrumentation.
+ key = self.cache_key(pcoll)
+ # Only need to write when the cache with expected key doesn't exist.
+ if not self._cache_manager.exists('full', key):
+ _ = pcoll | '{}{}'.format(WRITE_CACHE, key) >> cache.WriteCache(
+ self._cache_manager, key)
+
+ def _read_cache(self, pcoll):
+ """Reads a cached pvalue.
+
+ A noop will cause the pipeline to execute the transform as
+ it is and cache nothing from this transform for next run.
+
+ Modifies:
+ self._pipeline
+ """
+ # Makes sure the pcoll belongs to the pipeline being instrumented.
+ if pcoll.pipeline is not self._pipeline:
+ return
+ # The keyed cache is always valid within this instrumentation.
+ key = self.cache_key(pcoll)
+ # Can only read from cache when the cache with expected key exists.
+ if self._cache_manager.exists('full', key):
+ if key not in self._cached_pcoll_read:
+ # Mutates the pipeline with cache read transform attached
+ # to root of the pipeline.
+ pcoll_from_cache = (
+ self._pipeline
+ | '{}{}'.format(READ_CACHE, key) >> cache.ReadCache(
+ self._cache_manager, key))
+ self._cached_pcoll_read[key] = pcoll_from_cache
+ # else: NOOP when cache doesn't exist, just compute the original graph.
+
+ def _replace_with_cached_inputs(self):
+ """Replace PCollection inputs in the pipeline with cache if possible.
+
+ For any input PCollection, find out whether there is valid cache. If so,
+ replace the input of the AppliedPTransform with output of the
+ AppliedPtransform that sources pvalue from the cache. If there is no valid
+ cache, noop.
+ """
+
+ class ReadCacheWireVisitor(PipelineVisitor):
+ """Visitor wires cache read as inputs to replace corresponding original
+ input PCollections in pipeline.
+ """
+
+ def __init__(self, pin):
+ """Initializes with a PipelineInstrument."""
+ self._pin = pin
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if transform_node.inputs:
+ input_list = list(transform_node.inputs)
+ for i in range(len(input_list)):
+ key = self._pin.cache_key(input_list[i])
+ if key in self._pin._cached_pcoll_read:
+ input_list[i] = self._pin._cached_pcoll_read[key]
+ transform_node.inputs = tuple(input_list)
+
+ v = ReadCacheWireVisitor(self)
+ self._pipeline.visit(v)
+
+ def _cacheable_inputs(self, transform):
+ inputs = set()
+ for in_pcoll in transform.inputs:
+ if self._cacheable_key(in_pcoll) in self.cacheables:
+ inputs.add(in_pcoll)
+ return inputs
+
+ def _cacheable_key(self, pcoll):
+ """Gets the key a cacheable PCollection is tracked within the instrument."""
+ return cacheable_key(pcoll, self.pcolls_to_pcoll_id,
+ self._pcoll_version_map)
+
+ def cache_key(self, pcoll):
+ """Gets the identifier of a cacheable PCollection in cache.
+
+ If the pcoll is not a cacheable, return ''.
+ The key is what the pcoll would use as identifier if it's materialized in
+ cache. It doesn't mean that there would definitely be such cache already.
+ Also, the pcoll can come from the original user defined pipeline object or
+ an equivalent pcoll from a transformed copy of the original pipeline.
+ """
+ cacheable = self.cacheables.get(self._cacheable_key(pcoll), None)
+ if cacheable:
+ return '_'.join((cacheable['var'],
+ cacheable['version'],
+ cacheable['pcoll_id'],
+ cacheable['producer_version']))
+ return ''
+
+
+def pin(pipeline, options=None):
+ """Creates PipelineInstrument for a pipeline and its options with cache."""
+ pi = PipelineInstrument(pipeline, options)
+ pi.instrument() # Instruments the pipeline only once.
+ return pi
+
+
+def cacheables(pcolls_to_pcoll_id):
+ """Finds cache desired PCollections from the instrumented pipeline.
+
+ The function only treats the result as cacheables since whether the cache
+ desired PCollection has been cached depends on whether the pipeline has been
+ executed in current interactive environment. A PCollection desires caching
+ when it's bound to a user defined variable in source code. Otherwise, the
+ PCollection is not reusable nor introspectable which nullifies the need of
+ cache. There might be multiple pipelines defined and watched, this will
+ return for PCollections from the ones with pcolls_to_pcoll_id analyzed. The
+ check is not strict because pcoll_id is not unique across multiple pipelines.
+ Additional check needs to be done during instrument.
+ """
+ pcoll_version_map = {}
+ cacheables = {}
+ for watching in ie.current_env().watching():
+ for key, val in watching:
+ # TODO(BEAM-8288): cleanup the attribute check when py2 is not supported.
+ if hasattr(val, '__class__') and isinstance(val, beam.pvalue.PCollection):
+ cacheable = {}
+ cacheable['pcoll_id'] = pcolls_to_pcoll_id.get(str(val), None)
+ # It's highly possible that PCollection str is not unique across
+ # multiple pipelines, further check during instrument is needed.
+ if not cacheable['pcoll_id']:
+ continue
+ cacheable['var'] = key
+ cacheable['version'] = str(id(val))
+ cacheable['pcoll'] = val
+ cacheable['producer_version'] = str(id(val.producer))
+ cacheables[cacheable_key(val, pcolls_to_pcoll_id)] = cacheable
+ pcoll_version_map[cacheable['pcoll_id']] = cacheable['version']
+ return pcoll_version_map, cacheables
+
+
+def cacheable_key(pcoll, pcolls_to_pcoll_id, pcoll_version_map=None):
+ pcoll_version = str(id(pcoll))
+ pcoll_id = pcolls_to_pcoll_id.get(str(pcoll), '')
+ if pcoll_version_map:
+ original_pipeline_pcoll_version = pcoll_version_map.get(pcoll_id, None)
+ if original_pipeline_pcoll_version:
+ pcoll_version = original_pipeline_pcoll_version
+ return '_'.join((pcoll_version, pcoll_id))
+
+
+def has_unbounded_source(pipeline):
+ """Checks if a given pipeline has any source that is unbounded."""
+
+ class CheckUnboundednessVisitor(PipelineVisitor):
+ """Vsitor checks if there is any unbouned read source in the Pipeline.
+
+ Visitor visits all nodes and check is_bounded() for all sources of read
+ PTransform. As long as there is at least 1 source introduces unbounded
+ data, returns True. We don't check the is_bounded field from proto based
+ PCollection since they may not be correctly set with to_runner_api.
+ """
+
+ def __init__(self):
+ self.has_unbounded_source = False
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if (not self.has_unbounded_source and
+ isinstance(transform_node, beam.pipeline.AppliedPTransform) and
+ isinstance(transform_node.transform, beam.io.iobase.Read) and
+ not transform_node.transform.source.is_bounded()):
+ self.has_unbounded_source = True
+
+ v = CheckUnboundednessVisitor()
+ pipeline.visit(v)
+ return v.has_unbounded_source
+
+
+def pcolls_to_pcoll_id(pipeline, original_context):
+ """Returns a dict mapping PCollections string to PCollection IDs.
+
+ Using a PipelineVisitor to iterate over every node in the pipeline,
+ records the mapping from PCollections to PCollections IDs. This mapping
+ will be used to query cached PCollections.
+
+ Returns:
+ (dict from str to str) a dict mapping str(pcoll) to pcoll_id.
+ """
+
+ class PCollVisitor(PipelineVisitor):
+ """"A visitor that records input and output values to be replaced.
+
+ Input and output values that should be updated are recorded in maps
+ input_replacements and output_replacements respectively.
+
+ We cannot update input and output values while visiting since that
+ results in validation errors.
+ """
+
+ def __init__(self):
+ self.pcolls_to_pcoll_id = {}
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ for pcoll in transform_node.outputs.values():
+ self.pcolls_to_pcoll_id[str(pcoll)] = (
+ original_context.pcollections.get_id(pcoll))
+
+ v = PCollVisitor()
+ pipeline.visit(v)
+ return v.pcolls_to_pcoll_id
diff --git a/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py b/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
new file mode 100644
index 0000000..3d9a611
--- /dev/null
+++ b/sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
@@ -0,0 +1,283 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.runners.interactive.pipeline_instrument."""
+from __future__ import absolute_import
+
+import tempfile
+import time
+import unittest
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io import filesystems
+from apache_beam.pipeline import PipelineVisitor
+from apache_beam.runners.interactive import cache_manager as cache
+from apache_beam.runners.interactive import interactive_beam as ib
+from apache_beam.runners.interactive import interactive_environment as ie
+from apache_beam.runners.interactive import pipeline_instrument as instr
+from apache_beam.runners.interactive import interactive_runner
+
+# Work around nose tests using Python2 without unittest.mock module.
+try:
+ from unittest.mock import MagicMock
+except ImportError:
+ from mock import MagicMock
+
+
+class PipelineInstrumentTest(unittest.TestCase):
+
+ def setUp(self):
+ ie.new_env(cache_manager=cache.FileBasedCacheManager())
+
+ def assertPipelineEqual(self, actual_pipeline, expected_pipeline):
+ actual_pipeline_proto = actual_pipeline.to_runner_api(use_fake_coders=True)
+ expected_pipeline_proto = expected_pipeline.to_runner_api(
+ use_fake_coders=True)
+ components1 = actual_pipeline_proto.components
+ components2 = expected_pipeline_proto.components
+ self.assertEqual(len(components1.transforms), len(components2.transforms))
+ self.assertEqual(len(components1.pcollections),
+ len(components2.pcollections))
+
+ # GreatEqual instead of Equal because the pipeline_proto_to_execute could
+ # include more windowing_stratagies and coders than necessary.
+ self.assertGreaterEqual(len(components1.windowing_strategies),
+ len(components2.windowing_strategies))
+ self.assertGreaterEqual(len(components1.coders), len(components2.coders))
+ self.assertTransformEqual(actual_pipeline_proto,
+ actual_pipeline_proto.root_transform_ids[0],
+ expected_pipeline_proto,
+ expected_pipeline_proto.root_transform_ids[0])
+
+ def assertTransformEqual(self, actual_pipeline_proto, actual_transform_id,
+ expected_pipeline_proto, expected_transform_id):
+ transform_proto1 = actual_pipeline_proto.components.transforms[
+ actual_transform_id]
+ transform_proto2 = expected_pipeline_proto.components.transforms[
+ expected_transform_id]
+ self.assertEqual(transform_proto1.spec.urn, transform_proto2.spec.urn)
+ # Skipping payload checking because PTransforms of the same functionality
+ # could generate different payloads.
+ self.assertEqual(len(transform_proto1.subtransforms),
+ len(transform_proto2.subtransforms))
+ self.assertSetEqual(set(transform_proto1.inputs),
+ set(transform_proto2.inputs))
+ self.assertSetEqual(set(transform_proto1.outputs),
+ set(transform_proto2.outputs))
+
+ def test_pcolls_to_pcoll_id(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p.to_runner_api(use_fake_coders=True, return_context=True)
+ self.assertEqual(instr.pcolls_to_pcoll_id(p, ctx), {
+ str(init_pcoll): 'ref_PCollection_PCollection_1'})
+
+ def test_cacheable_key_without_version_map(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p.to_runner_api(use_fake_coders=True, return_context=True)
+ self.assertEqual(
+ instr.cacheable_key(init_pcoll, instr.pcolls_to_pcoll_id(p, ctx)),
+ str(id(init_pcoll)) + '_ref_PCollection_PCollection_1')
+
+ def test_cacheable_key_with_version_map(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+
+ # It's normal that when executing, the pipeline object is a different
+ # but equivalent instance from what user has built. The pipeline instrument
+ # should be able to identify if the original instance has changed in an
+ # interactive env while mutating the other instance for execution. The
+ # version map can be used to figure out what the PCollection instances are
+ # in the original instance and if the evaluation has changed since last
+ # execution.
+ p2 = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll_2 = p2 | 'Init Create' >> beam.Create(range(10))
+ _, ctx = p2.to_runner_api(use_fake_coders=True, return_context=True)
+
+ # The cacheable_key should use id(init_pcoll) as prefix even when
+ # init_pcoll_2 is supplied as long as the version map is given.
+ self.assertEqual(
+ instr.cacheable_key(init_pcoll_2, instr.pcolls_to_pcoll_id(p2, ctx), {
+ 'ref_PCollection_PCollection_1': str(id(init_pcoll))}),
+ str(id(init_pcoll)) + '_ref_PCollection_PCollection_1')
+
+ def test_cache_key(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ squares = init_pcoll | 'Square' >> beam.Map(lambda x: x * x)
+ cubes = init_pcoll | 'Cube' >> beam.Map(lambda x: x ** 3)
+ # Watch the local variables, i.e., the Beam pipeline defined.
+ ib.watch(locals())
+
+ pin = instr.pin(p)
+ self.assertEqual(pin.cache_key(init_pcoll), 'init_pcoll_' + str(
+ id(init_pcoll)) + '_ref_PCollection_PCollection_1_' + str(id(
+ init_pcoll.producer)))
+ self.assertEqual(pin.cache_key(squares), 'squares_' + str(
+ id(squares)) + '_ref_PCollection_PCollection_2_' + str(id(
+ squares.producer)))
+ self.assertEqual(pin.cache_key(cubes), 'cubes_' + str(
+ id(cubes)) + '_ref_PCollection_PCollection_3_' + str(id(
+ cubes.producer)))
+
+ def test_cacheables(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ squares = init_pcoll | 'Square' >> beam.Map(lambda x: x * x)
+ cubes = init_pcoll | 'Cube' >> beam.Map(lambda x: x ** 3)
+ ib.watch(locals())
+
+ pin = instr.pin(p)
+ self.assertEqual(pin.cacheables, {
+ pin._cacheable_key(init_pcoll): {
+ 'var': 'init_pcoll',
+ 'version': str(id(init_pcoll)),
+ 'pcoll_id': 'ref_PCollection_PCollection_1',
+ 'producer_version': str(id(init_pcoll.producer)),
+ 'pcoll': init_pcoll
+ },
+ pin._cacheable_key(squares): {
+ 'var': 'squares',
+ 'version': str(id(squares)),
+ 'pcoll_id': 'ref_PCollection_PCollection_2',
+ 'producer_version': str(id(squares.producer)),
+ 'pcoll': squares
+ },
+ pin._cacheable_key(cubes): {
+ 'var': 'cubes',
+ 'version': str(id(cubes)),
+ 'pcoll_id': 'ref_PCollection_PCollection_3',
+ 'producer_version': str(id(cubes.producer)),
+ 'pcoll': cubes
+ }
+ })
+
+ def test_has_unbounded_source(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ _ = p | 'ReadUnboundedSource' >> beam.io.ReadFromPubSub(
+ subscription='projects/fake-project/subscriptions/fake_sub')
+ self.assertTrue(instr.has_unbounded_source(p))
+
+ def test_not_has_unbounded_source(self):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ with tempfile.NamedTemporaryFile(delete=False) as f:
+ f.write(b'test')
+ _ = p | 'ReadBoundedSource' >> beam.io.ReadFromText(f.name)
+ self.assertFalse(instr.has_unbounded_source(p))
+
+ def _example_pipeline(self, watch=True):
+ p = beam.Pipeline(interactive_runner.InteractiveRunner())
+ # pylint: disable=range-builtin-not-iterating
+ init_pcoll = p | 'Init Create' >> beam.Create(range(10))
+ second_pcoll = init_pcoll | 'Second' >> beam.Map(lambda x: x * x)
+ if watch:
+ ib.watch(locals())
+ return (p, init_pcoll, second_pcoll)
+
+ def _mock_write_cache(self, pcoll, cache_key):
+ """Cache the PCollection where cache.WriteCache would write to."""
+ cache_path = filesystems.FileSystems.join(
+ ie.current_env().cache_manager()._cache_dir, 'full')
+ if not filesystems.FileSystems.exists(cache_path):
+ filesystems.FileSystems.mkdirs(cache_path)
+
+ # Pause for 0.1 sec, because the Jenkins test runs so fast that the file
+ # writes happen at the same timestamp.
+ time.sleep(0.1)
+
+ cache_file = cache_key + '-1-of-2'
+ labels = ['full', cache_key]
+
+ # Usually, the pcoder will be inferred from `pcoll.element_type`
+ pcoder = coders.registry.get_coder(object)
+ ie.current_env().cache_manager().save_pcoder(pcoder, *labels)
+ sink = ie.current_env().cache_manager().sink(*labels)
+
+ with open(ie.current_env().cache_manager()._path('full', cache_file),
+ 'wb') as f:
+ sink.write_record(f, pcoll)
+
+ def test_instrument_example_pipeline_to_write_cache(self):
+ # Original instance defined by user code has all variables handlers.
+ p_origin, init_pcoll, second_pcoll = self._example_pipeline()
+ # Copied instance when execution has no user defined variables.
+ p_copy, _, _ = self._example_pipeline(False)
+ # Instrument the copied pipeline.
+ pin = instr.pin(p_copy)
+ # Manually instrument original pipeline with expected pipeline transforms.
+ init_pcoll_cache_key = pin.cache_key(init_pcoll)
+ _ = init_pcoll | (
+ ('_WriteCache_' + init_pcoll_cache_key) >> cache.WriteCache(
+ ie.current_env().cache_manager(), init_pcoll_cache_key))
+ second_pcoll_cache_key = pin.cache_key(second_pcoll)
+ _ = second_pcoll | (
+ ('_WriteCache_' + second_pcoll_cache_key) >> cache.WriteCache(
+ ie.current_env().cache_manager(), second_pcoll_cache_key))
+ # The 2 pipelines should be the same now.
+ self.assertPipelineEqual(p_copy, p_origin)
+
+ def test_instrument_example_pipeline_to_read_cache(self):
+ p_origin, init_pcoll, second_pcoll = self._example_pipeline()
+ p_copy, _, _ = self._example_pipeline(False)
+
+ # Mock as if cacheable PCollections are cached.
+ init_pcoll_cache_key = 'init_pcoll_' + str(
+ id(init_pcoll)) + '_ref_PCollection_PCollection_1_' + str(id(
+ init_pcoll.producer))
+ self._mock_write_cache(init_pcoll, init_pcoll_cache_key)
+ second_pcoll_cache_key = 'second_pcoll_' + str(
+ id(second_pcoll)) + '_ref_PCollection_PCollection_2_' + str(id(
+ second_pcoll.producer))
+ self._mock_write_cache(second_pcoll, second_pcoll_cache_key)
+ ie.current_env().cache_manager().exists = MagicMock(return_value=True)
+ instr.pin(p_copy)
+
+ cached_init_pcoll = p_origin | (
+ '_ReadCache_' + init_pcoll_cache_key) >> cache.ReadCache(
+ ie.current_env().cache_manager(), init_pcoll_cache_key)
+
+ # second_pcoll is never used as input and there is no need to read cache.
+
+ class TestReadCacheWireVisitor(PipelineVisitor):
+ """Replace init_pcoll with cached_init_pcoll for all occuring inputs."""
+
+ def enter_composite_transform(self, transform_node):
+ self.visit_transform(transform_node)
+
+ def visit_transform(self, transform_node):
+ if transform_node.inputs:
+ input_list = list(transform_node.inputs)
+ for i in range(len(input_list)):
+ if input_list[i] == init_pcoll:
+ input_list[i] = cached_init_pcoll
+ transform_node.inputs = tuple(input_list)
+
+ v = TestReadCacheWireVisitor()
+ p_origin.visit(v)
+ self.assertPipelineEqual(p_origin, p_copy)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/pipeline_context.py b/sdks/python/apache_beam/runners/pipeline_context.py
index 6a18605..913dac6 100644
--- a/sdks/python/apache_beam/runners/pipeline_context.py
+++ b/sdks/python/apache_beam/runners/pipeline_context.py
@@ -31,6 +31,7 @@
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.transforms import core
+from apache_beam.typehints import native_type_compatibility
class Environment(object):
@@ -39,10 +40,10 @@
Provides consistency with how the other componentes are accessed.
"""
def __init__(self, proto):
- self._proto = proto
+ self.proto = proto
def to_runner_api(self, context):
- return self._proto
+ return self.proto
@staticmethod
def from_runner_api(proto, context):
@@ -100,6 +101,9 @@
return id
return self.put_proto(self._unique_ref(label), maybe_new_proto)
+ def get_id_to_proto_map(self):
+ return self._id_to_proto
+
def put_proto(self, id, proto):
if id in self._id_to_proto:
raise ValueError("Id '%s' is already taken." % id)
@@ -164,7 +168,8 @@
if self.use_fake_coders or coder_id not in self.coders:
return pickler.loads(coder_id)
else:
- return self.coders[coder_id].to_type_hint()
+ return native_type_compatibility.convert_to_beam_type(
+ self.coders[coder_id].to_type_hint())
@staticmethod
def from_runner_api(proto):
diff --git a/sdks/python/apache_beam/runners/portability/expansion_service_test.py b/sdks/python/apache_beam/runners/portability/expansion_service_test.py
index 9e93a64..66b0fa3 100644
--- a/sdks/python/apache_beam/runners/portability/expansion_service_test.py
+++ b/sdks/python/apache_beam/runners/portability/expansion_service_test.py
@@ -35,22 +35,40 @@
# external transform test cases. See external_test.py for details.
-@ptransform.PTransform.register_urn('count_per_element_bytes', None)
-class KV2BytesTransform(ptransform.PTransform):
+@ptransform.PTransform.register_urn('beam:transforms:xlang:count', None)
+class CountPerElementTransform(ptransform.PTransform):
def expand(self, pcoll):
return (
- pcoll
- | combine.Count.PerElement()
- | beam.Map(
- lambda x: '{}->{}'.format(x[0], x[1])).with_output_types(bytes)
+ pcoll | combine.Count.PerElement()
)
def to_runner_api_parameter(self, unused_context):
- return 'kv_to_bytes', None
+ return 'beam:transforms:xlang:count', None
@staticmethod
def from_runner_api_parameter(unused_parameter, unused_context):
- return KV2BytesTransform()
+ return CountPerElementTransform()
+
+
+@ptransform.PTransform.register_urn(
+ 'beam:transforms:xlang:filter_less_than_eq', bytes)
+class FilterLessThanTransform(ptransform.PTransform):
+ def __init__(self, payload):
+ self._payload = payload
+
+ def expand(self, pcoll):
+ return (
+ pcoll | beam.Filter(
+ lambda elem, target: elem <= target, int(ord(self._payload[0])))
+ )
+
+ def to_runner_api_parameter(self, unused_context):
+ return (
+ 'beam:transforms:xlang:filter_less_than', self._payload.encode('utf8'))
+
+ @staticmethod
+ def from_runner_api_parameter(payload, unused_context):
+ return FilterLessThanTransform(payload.decode('utf8'))
@ptransform.PTransform.register_urn('simple', None)
@@ -133,6 +151,11 @@
server = None
+def cleanup(unused_signum, unused_frame):
+ logging.info('Shutting down expansion service.')
+ server.stop(None)
+
+
def main(unused_argv):
parser = argparse.ArgumentParser()
parser.add_argument('-p', '--port',
@@ -148,19 +171,12 @@
server.start()
logging.info('Listening for expansion requests at %d', options.port)
+ signal.signal(signal.SIGTERM, cleanup)
+ signal.signal(signal.SIGINT, cleanup)
# blocking main thread forever.
signal.pause()
-def cleanup(unused_signum, unused_frame):
- logging.info('Shutting down expansion service.')
- server.stop(None)
-
-
-signal.signal(signal.SIGTERM, cleanup)
-signal.signal(signal.SIGINT, cleanup)
-
-
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
main(sys.argv)
diff --git a/sdks/python/apache_beam/runners/portability/flink_runner.py b/sdks/python/apache_beam/runners/portability/flink_runner.py
new file mode 100644
index 0000000..76c15ef
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/flink_runner.py
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A runner for executing portable pipelines on Flink."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+from apache_beam.options import pipeline_options
+from apache_beam.runners.portability import job_server
+from apache_beam.runners.portability import portable_runner
+
+PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8']
+
+
+class FlinkRunner(portable_runner.PortableRunner):
+ def default_job_server(self, options):
+ return job_server.StopOnExitJobServer(FlinkJarJobServer(options))
+
+
+class FlinkRunnerOptions(pipeline_options.PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ parser.add_argument('--flink_master_url', default='[local]')
+ parser.add_argument('--flink_version',
+ default=PUBLISHED_FLINK_VERSIONS[-1],
+ choices=PUBLISHED_FLINK_VERSIONS,
+ help='Flink version to use.')
+ parser.add_argument('--flink_job_server_jar',
+ help='Path or URL to a flink jobserver jar.')
+ parser.add_argument('--artifacts_dir', default=None)
+
+
+class FlinkJarJobServer(job_server.JavaJarJobServer):
+ def __init__(self, options):
+ super(FlinkJarJobServer, self).__init__()
+ options = options.view_as(FlinkRunnerOptions)
+ self._jar = options.flink_job_server_jar
+ self._master_url = options.flink_master_url
+ self._flink_version = options.flink_version
+ self._artifacts_dir = options.artifacts_dir
+
+ def path_to_jar(self):
+ if self._jar:
+ return self._jar
+ else:
+ return self.path_to_beam_jar(
+ 'runners:flink:%s:job-server:shadowJar' % self._flink_version)
+
+ def java_arguments(self, job_port, artifacts_dir):
+ return [
+ '--flink-master-url', self._master_url,
+ '--artifacts-dir', (self._artifacts_dir
+ if self._artifacts_dir else artifacts_dir),
+ '--job-port', job_port,
+ '--artifact-port', 0,
+ '--expansion-port', 0
+ ]
diff --git a/sdks/python/apache_beam/runners/portability/flink_runner_test.py b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
index 5e94d9e..397297b 100644
--- a/sdks/python/apache_beam/runners/portability/flink_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
@@ -30,6 +30,7 @@
import apache_beam as beam
from apache_beam import Impulse
from apache_beam import Map
+from apache_beam import Pipeline
from apache_beam.io.external.generate_sequence import GenerateSequence
from apache_beam.io.external.kafka import ReadFromKafka
from apache_beam.io.external.kafka import WriteToKafka
@@ -140,7 +141,7 @@
options = super(FlinkRunnerTest, self).create_options()
options.view_as(DebugOptions).experiments = [
'beam_fn_api'] + extra_experiments
- options._all_options['parallelism'] = 1
+ options._all_options['parallelism'] = 2
options._all_options['shutdown_sources_on_final_watermark'] = True
options.view_as(PortableOptions).environment_type = (
environment_type.upper())
@@ -231,28 +232,27 @@
def process(self, v):
self.counter.inc()
- p = self.create_pipeline()
+ options = self.create_options()
+ # Test only supports parallelism of 1
+ options._all_options['parallelism'] = 1
n = 100
-
- # pylint: disable=expression-not-assigned
- p \
- | beam.Create(list(range(n))) \
- | beam.ParDo(DoFn())
-
- result = p.run()
- result.wait_until_finish()
+ with Pipeline(self.get_runner(), options) as p:
+ # pylint: disable=expression-not-assigned
+ (p
+ | beam.Create(list(range(n)))
+ | beam.ParDo(DoFn()))
with open(self.test_metrics_path, 'r') as f:
lines = [line for line in f.readlines() if counter_name in line]
self.assertEqual(
len(lines), 1,
- msg='Expected 1 line matching "%s":\n%s' % (
+ msg='Expected 1 line matching "{}":\n{}'.format(
counter_name, '\n'.join(lines))
)
line = lines[0]
self.assertTrue(
- '%s: 100' % counter_name in line,
- msg='Failed to find expected counter %s in line %s' % (
+ '{}: {}'.format(counter_name in line, n),
+ msg='Failed to find expected counter {} in line {}'.format(
counter_name, line)
)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index 5f6e5f7..4b36af8 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -50,6 +50,7 @@
from apache_beam.options.value_provider import RuntimeValueProvider
from apache_beam.portability import common_urns
from apache_beam.portability import python_urns
+from apache_beam.portability.api import beam_artifact_api_pb2
from apache_beam.portability.api import beam_artifact_api_pb2_grpc
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_fn_api_pb2_grpc
@@ -69,6 +70,8 @@
from apache_beam.runners.worker import data_plane
from apache_beam.runners.worker import sdk_worker
from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.sdk_worker import _Future
+from apache_beam.runners.worker.statecache import StateCache
from apache_beam.transforms import trigger
from apache_beam.transforms.window import GlobalWindows
from apache_beam.utils import profiler
@@ -81,8 +84,63 @@
beam.coders.coders.GlobalWindowCoder()).get_impl().encode_nested(
beam.transforms.window.GlobalWindows.windowed_value(b''))
+# State caching is enabled in the fn_api_runner for testing, except for one
+# test which runs without state caching (FnApiRunnerTestWithDisabledCaching).
+# The cache is disabled in production for other runners.
+STATE_CACHE_SIZE = 100
+
+
+class ControlConnection(object):
+
+ _uid_counter = 0
+ _lock = threading.Lock()
+
+ def __init__(self):
+ self._push_queue = queue.Queue()
+ self._input = None
+ self._futures_by_id = dict()
+ self._read_thread = threading.Thread(
+ name='beam_control_read', target=self._read)
+ self._state = BeamFnControlServicer.UNSTARTED_STATE
+
+ def _read(self):
+ for data in self._input:
+ self._futures_by_id.pop(data.instruction_id).set(data)
+
+ def push(self, req):
+ if req == BeamFnControlServicer._DONE_MARKER:
+ self._push_queue.put(req)
+ return None
+ if not req.instruction_id:
+ with ControlConnection._lock:
+ ControlConnection._uid_counter += 1
+ req.instruction_id = 'control_%s' % ControlConnection._uid_counter
+ future = ControlFuture(req.instruction_id)
+ self._futures_by_id[req.instruction_id] = future
+ self._push_queue.put(req)
+ return future
+
+ def get_req(self):
+ return self._push_queue.get()
+
+ def set_input(self, input):
+ with ControlConnection._lock:
+ if self._input:
+ raise RuntimeError('input is already set.')
+ self._input = input
+ self._read_thread.start()
+ self._state = BeamFnControlServicer.STARTED_STATE
+
+ def close(self):
+ with ControlConnection._lock:
+ if self._state == BeamFnControlServicer.STARTED_STATE:
+ self.push(BeamFnControlServicer._DONE_MARKER)
+ self._read_thread.join()
+ self._state = BeamFnControlServicer.DONE_STATE
+
class BeamFnControlServicer(beam_fn_api_pb2_grpc.BeamFnControlServicer):
+ """Implementation of BeamFnControlServicer for clients."""
UNSTARTED_STATE = 'unstarted'
STARTED_STATE = 'started'
@@ -91,13 +149,19 @@
_DONE_MARKER = object()
def __init__(self):
- self._push_queue = queue.Queue()
- self._futures_by_id = dict()
- self._read_thread = threading.Thread(
- name='beam_control_read', target=self._read)
+ self._lock = threading.Lock()
self._uid_counter = 0
self._state = self.UNSTARTED_STATE
- self._lock = threading.Lock()
+ # following self._req_* variables are used for debugging purpose, data is
+ # added only when self._log_req is True.
+ self._req_sent = collections.defaultdict(int)
+ self._req_worker_mapping = {}
+ self._log_req = logging.getLogger().getEffectiveLevel() <= logging.DEBUG
+ self._connections_by_worker_id = collections.defaultdict(ControlConnection)
+
+ def get_conn_by_worker_id(self, worker_id):
+ with self._lock:
+ return self._connections_by_worker_id[worker_id]
def Control(self, iterator, context):
with self._lock:
@@ -105,37 +169,36 @@
return
else:
self._state = self.STARTED_STATE
- self._inputs = iterator
- # Note: We only support one client for now.
- self._read_thread.start()
+
+ worker_id = dict(context.invocation_metadata()).get('worker_id')
+ if not worker_id:
+ raise RuntimeError('All workers communicate through gRPC should have '
+ 'worker_id. Received None.')
+
+ control_conn = self.get_conn_by_worker_id(worker_id)
+ control_conn.set_input(iterator)
+
while True:
- to_push = self._push_queue.get()
+ to_push = control_conn.get_req()
if to_push is self._DONE_MARKER:
return
yield to_push
-
- def _read(self):
- for data in self._inputs:
- self._futures_by_id.pop(data.instruction_id).set(data)
-
- def push(self, item):
- if item is self._DONE_MARKER:
- future = None
- else:
- if not item.instruction_id:
- self._uid_counter += 1
- item.instruction_id = 'control_%s' % self._uid_counter
- future = ControlFuture(item.instruction_id)
- self._futures_by_id[item.instruction_id] = future
- self._push_queue.put(item)
- return future
+ if self._log_req:
+ self._req_sent[to_push.instruction_id] += 1
def done(self):
- with self._lock:
- if self._state == self.STARTED_STATE:
- self.push(self._DONE_MARKER)
- self._read_thread.join()
- self._state = self.DONE_STATE
+ self._state = self.DONE_STATE
+ logging.debug('Runner: Requests sent by runner: %s',
+ [(str(req), cnt) for req, cnt in self._req_sent.items()])
+ logging.debug('Runner: Requests multiplexing info: %s',
+ [(str(req), worker) for req, worker
+ in self._req_worker_mapping.items()])
+
+
+class _ListBuffer(list):
+ """Used to support parititioning of a list."""
+ def partition(self, n):
+ return [self[k::n] for k in range(n)]
class _GroupingBuffer(object):
@@ -164,25 +227,44 @@
value if is_trivial_windowing
else windowed_key_value.with_value(value))
- def __iter__(self):
+ def partition(self, n):
+ """ It is used to partition _GroupingBuffer to N parts. Once it is
+ partitioned, it would not be re-partitioned with diff N. Re-partition
+ is not supported now.
+ """
if not self._grouped_output:
- output_stream = create_OutputStream()
if self._windowing.is_default():
globally_window = GlobalWindows.windowed_value(None).with_value
windowed_key_values = lambda key, values: [
globally_window((key, values))]
else:
+ # TODO(pabloem, BEAM-7514): Trigger driver needs access to the clock
+ # note that this only comes through if windowing is default - but what
+ # about having multiple firings on the global window.
+ # May need to revise.
trigger_driver = trigger.create_trigger_driver(self._windowing, True)
windowed_key_values = trigger_driver.process_entire_key
coder_impl = self._post_grouped_coder.get_impl()
key_coder_impl = self._key_coder.get_impl()
- for encoded_key, windowed_values in self._table.items():
+ self._grouped_output = [[] for _ in range(n)]
+ output_stream_list = []
+ for _ in range(n):
+ output_stream_list.append(create_OutputStream())
+ for idx, (encoded_key, windowed_values) in enumerate(self._table.items()):
key = key_coder_impl.decode(encoded_key)
for wkvs in windowed_key_values(key, windowed_values):
- coder_impl.encode_to_stream(wkvs, output_stream, True)
- self._grouped_output = [output_stream.get()]
+ coder_impl.encode_to_stream(wkvs, output_stream_list[idx % n], True)
+ for ix, output_stream in enumerate(output_stream_list):
+ self._grouped_output[ix] = [output_stream.get()]
self._table = None
- return iter(self._grouped_output)
+ return self._grouped_output
+
+ def __iter__(self):
+ """ Since partition() returns a list of lists, add this __iter__ to return
+ a list to simplify code when we need to iterate through ALL elements of
+ _GroupingBuffer.
+ """
+ return itertools.chain(*self.partition(1))
class _WindowGroupingBuffer(object):
@@ -234,7 +316,8 @@
default_environment=None,
bundle_repeat=0,
use_state_iterables=False,
- provision_info=None):
+ provision_info=None,
+ progress_request_frequency=None):
"""Creates a new Fn API Runner.
Args:
@@ -244,6 +327,8 @@
use_state_iterables: Intentionally split gbk iterables over state API
(for testing)
provision_info: provisioning info to make available to workers, or None
+ progress_request_frequency: The frequency (in seconds) that the runner
+ waits before requesting progress from the SDK.
"""
super(FnApiRunner, self).__init__()
self._last_uid = -1
@@ -251,10 +336,15 @@
default_environment
or beam_runner_api_pb2.Environment(urn=python_urns.EMBEDDED_PYTHON))
self._bundle_repeat = bundle_repeat
- self._progress_frequency = None
+ self._num_workers = 1
+ self._progress_frequency = progress_request_frequency
self._profiler_factory = None
self._use_state_iterables = use_state_iterables
- self._provision_info = provision_info
+ self._provision_info = provision_info or ExtendedProvisionInfo(
+ beam_provision_api_pb2.ProvisionInfo(
+ job_id='unknown-job-id',
+ job_name='unknown-job-name',
+ retrieval_token='unused-retrieval-token'))
def _next_uid(self):
self._last_uid += 1
@@ -263,6 +353,14 @@
def run_pipeline(self, pipeline, options):
MetricsEnvironment.set_metrics_supported(False)
RuntimeValueProvider.set_runtime_options({})
+
+ # Setup "beam_fn_api" experiment options if lacked.
+ experiments = (options.view_as(pipeline_options.DebugOptions).experiments
+ or [])
+ if not 'beam_fn_api' in experiments:
+ experiments.append('beam_fn_api')
+ options.view_as(pipeline_options.DebugOptions).experiments = experiments
+
# This is sometimes needed if type checking is disabled
# to enforce that the inputs (and outputs) of GroupByKey operations
# are known to be KVs.
@@ -271,14 +369,23 @@
pipeline.visit(DataflowRunner.group_by_key_input_visitor())
self._bundle_repeat = self._bundle_repeat or options.view_as(
pipeline_options.DirectOptions).direct_runner_bundle_repeat
+ self._num_workers = options.view_as(
+ pipeline_options.DirectOptions).direct_num_workers or self._num_workers
self._profiler_factory = profiler.Profile.factory_from_options(
options.view_as(pipeline_options.ProfilingOptions))
+
+ if 'use_sdf_bounded_source' in experiments:
+ pipeline.replace_all(DataflowRunner._SDF_PTRANSFORM_OVERRIDES)
+
self._latest_run_result = self.run_via_runner_api(pipeline.to_runner_api(
default_environment=self._default_environment))
return self._latest_run_result
def run_via_runner_api(self, pipeline_proto):
- return self.run_stages(*self.create_stages(pipeline_proto))
+ stage_context, stages = self.create_stages(pipeline_proto)
+ # TODO(pabloem, BEAM-7514): Create a watermark manager (that has access to
+ # the teststream (if any), and all the stages).
+ return self.run_stages(stage_context, stages)
@contextlib.contextmanager
def maybe_profile(self):
@@ -340,6 +447,12 @@
use_state_iterables=self._use_state_iterables)
def run_stages(self, stage_context, stages):
+ """Run a list of topologically-sorted stages in batch mode.
+
+ Args:
+ stage_context (fn_api_runner_transforms.TransformContext)
+ stages (list[fn_api_runner_transforms.Stage])
+ """
worker_handler_manager = WorkerHandlerManager(
stage_context.components.environments, self._provision_info)
metrics_by_stage = {}
@@ -347,10 +460,10 @@
try:
with self.maybe_profile():
- pcoll_buffers = collections.defaultdict(list)
+ pcoll_buffers = collections.defaultdict(_ListBuffer)
for stage in stages:
- stage_results = self.run_stage(
- worker_handler_manager.get_worker_handler,
+ stage_results = self._run_stage(
+ worker_handler_manager.get_worker_handlers,
stage_context.components,
stage,
pcoll_buffers,
@@ -363,80 +476,202 @@
return RunnerResult(
runner.PipelineState.DONE, monitoring_infos_by_stage, metrics_by_stage)
- def run_stage(self,
- worker_handler_factory,
- pipeline_components,
- stage,
- pcoll_buffers,
- safe_coders):
+ def _store_side_inputs_in_state(self,
+ worker_handler,
+ context,
+ pipeline_components,
+ data_side_input,
+ pcoll_buffers,
+ safe_coders):
+ for (transform_id, tag), (buffer_id, si) in data_side_input.items():
+ _, pcoll_id = split_buffer_id(buffer_id)
+ value_coder = context.coders[safe_coders[
+ pipeline_components.pcollections[pcoll_id].coder_id]]
+ elements_by_window = _WindowGroupingBuffer(si, value_coder)
+ for element_data in pcoll_buffers[buffer_id]:
+ elements_by_window.append(element_data)
+ for key, window, elements_data in elements_by_window.encoded_items():
+ state_key = beam_fn_api_pb2.StateKey(
+ multimap_side_input=beam_fn_api_pb2.StateKey.MultimapSideInput(
+ transform_id=transform_id,
+ side_input_id=tag,
+ window=window,
+ key=key))
+ worker_handler.state.append_raw(state_key, elements_data)
+
+ def _run_bundle_multiple_times_for_testing(
+ self, worker_handler_list, process_bundle_descriptor, data_input,
+ data_output, get_input_coder_callable, cache_token_generator):
+
+ # all workers share state, so use any worker_handler.
+ worker_handler = worker_handler_list[0]
+ for k in range(self._bundle_repeat):
+ try:
+ worker_handler.state.checkpoint()
+ testing_bundle_manager = ParallelBundleManager(
+ worker_handler_list, lambda pcoll_id: [],
+ get_input_coder_callable, process_bundle_descriptor,
+ self._progress_frequency, k,
+ num_workers=self._num_workers,
+ cache_token_generator=cache_token_generator
+ )
+ testing_bundle_manager.process_bundle(data_input, data_output)
+ finally:
+ worker_handler.state.restore()
+
+ def _collect_written_timers_and_add_to_deferred_inputs(self,
+ context,
+ pipeline_components,
+ stage,
+ get_buffer_callable,
+ deferred_inputs):
+
+ for transform_id, timer_writes in stage.timer_pcollections:
+
+ # Queue any set timers as new inputs.
+ windowed_timer_coder_impl = context.coders[
+ pipeline_components.pcollections[timer_writes].coder_id].get_impl()
+ written_timers = get_buffer_callable(
+ create_buffer_id(timer_writes, kind='timers'))
+ if written_timers:
+ # Keep only the "last" timer set per key and window.
+ timers_by_key_and_window = {}
+ for elements_data in written_timers:
+ input_stream = create_InputStream(elements_data)
+ while input_stream.size() > 0:
+ windowed_key_timer = windowed_timer_coder_impl.decode_from_stream(
+ input_stream, True)
+ key, _ = windowed_key_timer.value
+ # TODO: Explode and merge windows.
+ assert len(windowed_key_timer.windows) == 1
+ timers_by_key_and_window[
+ key, windowed_key_timer.windows[0]] = windowed_key_timer
+ out = create_OutputStream()
+ for windowed_key_timer in timers_by_key_and_window.values():
+ windowed_timer_coder_impl.encode_to_stream(
+ windowed_key_timer, out, True)
+ deferred_inputs[transform_id] = _ListBuffer([out.get()])
+ written_timers[:] = []
+
+ def _add_residuals_and_channel_splits_to_deferred_inputs(
+ self, splits, get_input_coder_callable,
+ input_for_callable, last_sent, deferred_inputs):
+ prev_stops = {}
+ for split in splits:
+ for delayed_application in split.residual_roots:
+ deferred_inputs[
+ input_for_callable(
+ delayed_application.application.transform_id,
+ delayed_application.application.input_id)
+ ].append(delayed_application.application.element)
+ for channel_split in split.channel_splits:
+ coder_impl = get_input_coder_callable(channel_split.transform_id)
+ # TODO(SDF): This requires determanistic ordering of buffer iteration.
+ # TODO(SDF): The return split is in terms of indices. Ideally,
+ # a runner could map these back to actual positions to effectively
+ # describe the two "halves" of the now-split range. Even if we have
+ # to buffer each element we send (or at the very least a bit of
+ # metadata, like position, about each of them) this should be doable
+ # if they're already in memory and we are bounding the buffer size
+ # (e.g. to 10mb plus whatever is eagerly read from the SDK). In the
+ # case of non-split-points, we can either immediately replay the
+ # "non-split-position" elements or record them as we do the other
+ # delayed applications.
+
+ # Decode and recode to split the encoded buffer by element index.
+ all_elements = list(coder_impl.decode_all(b''.join(last_sent[
+ channel_split.transform_id])))
+ residual_elements = all_elements[
+ channel_split.first_residual_element : prev_stops.get(
+ channel_split.transform_id, len(all_elements)) + 1]
+ if residual_elements:
+ deferred_inputs[channel_split.transform_id].append(
+ coder_impl.encode_all(residual_elements))
+ prev_stops[
+ channel_split.transform_id] = channel_split.last_primary_element
+
+ @staticmethod
+ def _extract_stage_data_endpoints(
+ stage, pipeline_components, data_api_service_descriptor, pcoll_buffers):
+ # Returns maps of transform names to PCollection identifiers.
+ # Also mutates IO stages to point to the data ApiServiceDescriptor.
+ data_input = {}
+ data_side_input = {}
+ data_output = {}
+ for transform in stage.transforms:
+ if transform.spec.urn in (bundle_processor.DATA_INPUT_URN,
+ bundle_processor.DATA_OUTPUT_URN):
+ pcoll_id = transform.spec.payload
+ if transform.spec.urn == bundle_processor.DATA_INPUT_URN:
+ target = transform.unique_name, only_element(transform.outputs)
+ if pcoll_id == fn_api_runner_transforms.IMPULSE_BUFFER:
+ data_input[target] = _ListBuffer([ENCODED_IMPULSE_VALUE])
+ else:
+ data_input[target] = pcoll_buffers[pcoll_id]
+ coder_id = pipeline_components.pcollections[
+ only_element(transform.outputs.values())].coder_id
+ elif transform.spec.urn == bundle_processor.DATA_OUTPUT_URN:
+ target = transform.unique_name, only_element(transform.inputs)
+ data_output[target] = pcoll_id
+ coder_id = pipeline_components.pcollections[
+ only_element(transform.inputs.values())].coder_id
+ else:
+ raise NotImplementedError
+ data_spec = beam_fn_api_pb2.RemoteGrpcPort(coder_id=coder_id)
+ if data_api_service_descriptor:
+ data_spec.api_service_descriptor.url = (
+ data_api_service_descriptor.url)
+ transform.spec.payload = data_spec.SerializeToString()
+ elif transform.spec.urn in fn_api_runner_transforms.PAR_DO_URNS:
+ payload = proto_utils.parse_Bytes(
+ transform.spec.payload, beam_runner_api_pb2.ParDoPayload)
+ for tag, si in payload.side_inputs.items():
+ data_side_input[transform.unique_name, tag] = (
+ create_buffer_id(transform.inputs[tag]), si.access_pattern)
+ return data_input, data_side_input, data_output
+
+ def _run_stage(self,
+ worker_handler_factory,
+ pipeline_components,
+ stage,
+ pcoll_buffers,
+ safe_coders):
"""Run an individual stage.
Args:
worker_handler_factory: A ``callable`` that takes in an environment, and
returns a ``WorkerHandler`` class.
- pipeline_components: TODO
- stage: TODO
- pcoll_buffers: TODO
- safe_coders: TODO
+ pipeline_components (beam_runner_api_pb2.Components): TODO
+ stage (fn_api_runner_transforms.Stage)
+ pcoll_buffers (collections.defaultdict of str: list): Mapping of
+ PCollection IDs to list that functions as buffer for the
+ ``beam.PCollection``.
+ safe_coders (dict): TODO
"""
-
def iterable_state_write(values, element_coder_impl):
token = unique_name(None, 'iter').encode('ascii')
out = create_OutputStream()
for element in values:
element_coder_impl.encode_to_stream(element, out, True)
- controller.state.blocking_append(
+ worker_handler.state.append_raw(
beam_fn_api_pb2.StateKey(
runner=beam_fn_api_pb2.StateKey.Runner(key=token)),
out.get())
return token
- controller = worker_handler_factory(stage.environment)
+ worker_handler_list = worker_handler_factory(
+ stage.environment, self._num_workers)
+
+ # All worker_handlers share the same grpc server, so we can read grpc server
+ # info from any worker_handler and read from the first worker_handler.
+ worker_handler = next(iter(worker_handler_list))
context = pipeline_context.PipelineContext(
pipeline_components, iterable_state_write=iterable_state_write)
- data_api_service_descriptor = controller.data_api_service_descriptor()
-
- def extract_endpoints(stage):
- # Returns maps of transform names to PCollection identifiers.
- # Also mutates IO stages to point to the data ApiServiceDescriptor.
- data_input = {}
- data_side_input = {}
- data_output = {}
- for transform in stage.transforms:
- if transform.spec.urn in (bundle_processor.DATA_INPUT_URN,
- bundle_processor.DATA_OUTPUT_URN):
- pcoll_id = transform.spec.payload
- if transform.spec.urn == bundle_processor.DATA_INPUT_URN:
- target = transform.unique_name, only_element(transform.outputs)
- if pcoll_id == fn_api_runner_transforms.IMPULSE_BUFFER:
- data_input[target] = [ENCODED_IMPULSE_VALUE]
- else:
- data_input[target] = pcoll_buffers[pcoll_id]
- coder_id = pipeline_components.pcollections[
- only_element(transform.outputs.values())].coder_id
- elif transform.spec.urn == bundle_processor.DATA_OUTPUT_URN:
- target = transform.unique_name, only_element(transform.inputs)
- data_output[target] = pcoll_id
- coder_id = pipeline_components.pcollections[
- only_element(transform.inputs.values())].coder_id
- else:
- raise NotImplementedError
- data_spec = beam_fn_api_pb2.RemoteGrpcPort(coder_id=coder_id)
- if data_api_service_descriptor:
- data_spec.api_service_descriptor.url = (
- data_api_service_descriptor.url)
- transform.spec.payload = data_spec.SerializeToString()
- elif transform.spec.urn in fn_api_runner_transforms.PAR_DO_URNS:
- payload = proto_utils.parse_Bytes(
- transform.spec.payload, beam_runner_api_pb2.ParDoPayload)
- for tag, si in payload.side_inputs.items():
- data_side_input[transform.unique_name, tag] = (
- create_buffer_id(transform.inputs[tag]), si.access_pattern)
- return data_input, data_side_input, data_output
+ data_api_service_descriptor = worker_handler.data_api_service_descriptor()
logging.info('Running %s', stage.name)
- logging.debug(' %s', stage)
- data_input, data_side_input, data_output = extract_endpoints(stage)
+ data_input, data_side_input, data_output = self._extract_endpoints(
+ stage, pipeline_components, data_api_service_descriptor, pcoll_buffers)
process_bundle_descriptor = beam_fn_api_pb2.ProcessBundleDescriptor(
id=self._next_uid(),
@@ -448,33 +683,30 @@
pipeline_components.windowing_strategies.items()),
environments=dict(pipeline_components.environments.items()))
- if controller.state_api_service_descriptor():
+ if worker_handler.state_api_service_descriptor():
process_bundle_descriptor.state_api_service_descriptor.url = (
- controller.state_api_service_descriptor().url)
+ worker_handler.state_api_service_descriptor().url)
- # Store the required side inputs into state.
- for (transform_id, tag), (buffer_id, si) in data_side_input.items():
- _, pcoll_id = split_buffer_id(buffer_id)
- value_coder = context.coders[safe_coders[
- pipeline_components.pcollections[pcoll_id].coder_id]]
- elements_by_window = _WindowGroupingBuffer(si, value_coder)
- for element_data in pcoll_buffers[buffer_id]:
- elements_by_window.append(element_data)
- for key, window, elements_data in elements_by_window.encoded_items():
- state_key = beam_fn_api_pb2.StateKey(
- multimap_side_input=beam_fn_api_pb2.StateKey.MultimapSideInput(
- ptransform_id=transform_id,
- side_input_id=tag,
- window=window,
- key=key))
- controller.state.blocking_append(state_key, elements_data)
+ # Store the required side inputs into state so it is accessible for the
+ # worker when it runs this bundle.
+ self._store_side_inputs_in_state(worker_handler,
+ context,
+ pipeline_components,
+ data_side_input,
+ pcoll_buffers,
+ safe_coders)
def get_buffer(buffer_id):
+ """Returns the buffer for a given (operation_type, PCollection ID).
+
+ For grouping-typed operations, we produce a ``_GroupingBuffer``. For
+ others, we produce a ``_ListBuffer``.
+ """
kind, name = split_buffer_id(buffer_id)
if kind in ('materialize', 'timers'):
- if buffer_id not in pcoll_buffers:
- # Just store the data chunks for replay.
- pcoll_buffers[buffer_id] = list()
+ # If `buffer_id` is not a key in `pcoll_buffers`, it will be added by
+ # the `defaultdict`.
+ return pcoll_buffers[buffer_id]
elif kind == 'group':
# This is a grouping write, create a grouping buffer if needed.
if buffer_id not in pcoll_buffers:
@@ -505,120 +737,65 @@
).coder_id
]].get_impl()
- for k in range(self._bundle_repeat):
- try:
- controller.state.checkpoint()
- BundleManager(
- controller, lambda pcoll_id: [], get_input_coder_impl,
- process_bundle_descriptor, self._progress_frequency, k
- ).process_bundle(data_input, data_output)
- finally:
- controller.state.restore()
+ # Change cache token across bundle repeats
+ cache_token_generator = FnApiRunner.get_cache_token_generator(static=False)
- result, splits = BundleManager(
- controller, get_buffer, get_input_coder_impl, process_bundle_descriptor,
- self._progress_frequency).process_bundle(
- data_input, data_output)
+ self._run_bundle_multiple_times_for_testing(
+ worker_handler_list, process_bundle_descriptor, data_input, data_output,
+ get_input_coder_impl, cache_token_generator=cache_token_generator)
- def input_for(ptransform_id, input_id):
+ bundle_manager = ParallelBundleManager(
+ worker_handler_list, get_buffer, get_input_coder_impl,
+ process_bundle_descriptor, self._progress_frequency,
+ num_workers=self._num_workers,
+ cache_token_generator=cache_token_generator)
+
+ result, splits = bundle_manager.process_bundle(data_input, data_output)
+
+ def input_for(transform_id, input_id):
input_pcoll = process_bundle_descriptor.transforms[
- ptransform_id].inputs[input_id]
+ transform_id].inputs[input_id]
for read_id, proto in process_bundle_descriptor.transforms.items():
if (proto.spec.urn == bundle_processor.DATA_INPUT_URN
and input_pcoll in proto.outputs.values()):
- return read_id, 'out'
+ return read_id
raise RuntimeError(
- 'No IO transform feeds %s' % ptransform_id)
+ 'No IO transform feeds %s' % transform_id)
last_result = result
last_sent = data_input
while True:
- deferred_inputs = collections.defaultdict(list)
- for transform_id, timer_writes in stage.timer_pcollections:
+ deferred_inputs = collections.defaultdict(_ListBuffer)
- # Queue any set timers as new inputs.
- windowed_timer_coder_impl = context.coders[
- pipeline_components.pcollections[timer_writes].coder_id].get_impl()
- written_timers = get_buffer(
- create_buffer_id(timer_writes, kind='timers'))
- if written_timers:
- # Keep only the "last" timer set per key and window.
- timers_by_key_and_window = {}
- for elements_data in written_timers:
- input_stream = create_InputStream(elements_data)
- while input_stream.size() > 0:
- windowed_key_timer = windowed_timer_coder_impl.decode_from_stream(
- input_stream, True)
- key, _ = windowed_key_timer.value
- # TODO: Explode and merge windows.
- assert len(windowed_key_timer.windows) == 1
- timers_by_key_and_window[
- key, windowed_key_timer.windows[0]] = windowed_key_timer
- out = create_OutputStream()
- for windowed_key_timer in timers_by_key_and_window.values():
- windowed_timer_coder_impl.encode_to_stream(
- windowed_key_timer, out, True)
- deferred_inputs[transform_id, 'out'] = [out.get()]
- written_timers[:] = []
+ self._collect_written_timers_and_add_to_deferred_inputs(
+ context, pipeline_components, stage, get_buffer, deferred_inputs)
# Queue any process-initiated delayed bundle applications.
for delayed_application in last_result.process_bundle.residual_roots:
deferred_inputs[
input_for(
- delayed_application.application.ptransform_id,
+ delayed_application.application.transform_id,
delayed_application.application.input_id)
].append(delayed_application.application.element)
# Queue any runner-initiated delayed bundle applications.
- prev_stops = {}
- for split in splits:
- for delayed_application in split.residual_roots:
- deferred_inputs[
- input_for(
- delayed_application.application.ptransform_id,
- delayed_application.application.input_id)
- ].append(delayed_application.application.element)
- for channel_split in split.channel_splits:
- coder_impl = get_input_coder_impl(channel_split.ptransform_id)
- # TODO(SDF): This requires determanistic ordering of buffer iteration.
- # TODO(SDF): The return split is in terms of indices. Ideally,
- # a runner could map these back to actual positions to effectively
- # describe the two "halves" of the now-split range. Even if we have
- # to buffer each element we send (or at the very least a bit of
- # metadata, like position, about each of them) this should be doable
- # if they're already in memory and we are bounding the buffer size
- # (e.g. to 10mb plus whatever is eagerly read from the SDK). In the
- # case of non-split-points, we can either immediately replay the
- # "non-split-position" elements or record them as we do the other
- # delayed applications.
-
- # Decode and recode to split the encoded buffer by element index.
- all_elements = list(coder_impl.decode_all(b''.join(last_sent[
- channel_split.ptransform_id, channel_split.input_id])))
- residual_elements = all_elements[
- channel_split.first_residual_element : prev_stops.get(
- channel_split.ptransform_id, len(all_elements)) + 1]
- if residual_elements:
- deferred_inputs[
- channel_split.ptransform_id, channel_split.input_id].append(
- coder_impl.encode_all(residual_elements))
- prev_stops[
- channel_split.ptransform_id] = channel_split.last_primary_element
+ self._add_residuals_and_channel_splits_to_deferred_inputs(
+ splits, get_input_coder_impl, input_for, last_sent, deferred_inputs)
if deferred_inputs:
# The worker will be waiting on these inputs as well.
for other_input in data_input:
if other_input not in deferred_inputs:
- deferred_inputs[other_input] = []
+ deferred_inputs[other_input] = _ListBuffer([])
# TODO(robertwb): merge results
- last_result, splits = BundleManager(
- controller,
- get_buffer,
- get_input_coder_impl,
- process_bundle_descriptor,
- self._progress_frequency,
- True).process_bundle(deferred_inputs, data_output)
+ # We cannot split deferred_input until we include residual_roots to
+ # merged results. Without residual_roots, pipeline stops earlier and we
+ # may miss some data.
+ bundle_manager._num_workers = 1
+ bundle_manager._skip_registration = True
+ last_result, splits = bundle_manager.process_bundle(
+ deferred_inputs, data_output)
last_sent = deferred_inputs
result = beam_fn_api_pb2.InstructionResponse(
process_bundle=beam_fn_api_pb2.ProcessBundleResponse(
@@ -632,6 +809,63 @@
return result
+ @staticmethod
+ def _extract_endpoints(stage,
+ pipeline_components,
+ data_api_service_descriptor,
+ pcoll_buffers):
+ """Returns maps of transform names to PCollection identifiers.
+
+ Also mutates IO stages to point to the data ApiServiceDescriptor.
+
+ Args:
+ stage (fn_api_runner_transforms.Stage): The stage to extract endpoints
+ for.
+ pipeline_components (beam_runner_api_pb2.Components): Components of the
+ pipeline to include coders, transforms, PCollections, etc.
+ data_api_service_descriptor: A GRPC endpoint descriptor for data plane.
+ pcoll_buffers (dict): A dictionary containing buffers for PCollection
+ elements.
+ Returns:
+ A tuple of (data_input, data_side_input, data_output) dictionaries.
+ `data_input` is a dictionary mapping (transform_name, output_name) to a
+ PCollection buffer; `data_output` is a dictionary mapping
+ (transform_name, output_name) to a PCollection ID.
+ """
+ data_input = {}
+ data_side_input = {}
+ data_output = {}
+ for transform in stage.transforms:
+ if transform.spec.urn in (bundle_processor.DATA_INPUT_URN,
+ bundle_processor.DATA_OUTPUT_URN):
+ pcoll_id = transform.spec.payload
+ if transform.spec.urn == bundle_processor.DATA_INPUT_URN:
+ if pcoll_id == fn_api_runner_transforms.IMPULSE_BUFFER:
+ data_input[transform.unique_name] = _ListBuffer(
+ [ENCODED_IMPULSE_VALUE])
+ else:
+ data_input[transform.unique_name] = pcoll_buffers[pcoll_id]
+ coder_id = pipeline_components.pcollections[
+ only_element(transform.outputs.values())].coder_id
+ elif transform.spec.urn == bundle_processor.DATA_OUTPUT_URN:
+ data_output[transform.unique_name] = pcoll_id
+ coder_id = pipeline_components.pcollections[
+ only_element(transform.inputs.values())].coder_id
+ else:
+ raise NotImplementedError
+ data_spec = beam_fn_api_pb2.RemoteGrpcPort(coder_id=coder_id)
+ if data_api_service_descriptor:
+ data_spec.api_service_descriptor.url = (
+ data_api_service_descriptor.url)
+ transform.spec.payload = data_spec.SerializeToString()
+ elif transform.spec.urn in fn_api_runner_transforms.PAR_DO_URNS:
+ payload = proto_utils.parse_Bytes(
+ transform.spec.payload, beam_runner_api_pb2.ParDoPayload)
+ for tag, si in payload.side_inputs.items():
+ data_side_input[transform.unique_name, tag] = (
+ create_buffer_id(transform.inputs[tag]), si.access_pattern)
+ return data_input, data_side_input, data_output
+
# These classes are used to interact with the worker.
class StateServicer(beam_fn_api_pb2_grpc.BeamFnStateServicer):
@@ -697,7 +931,7 @@
def process_instruction_id(self, unused_instruction_id):
yield
- def blocking_get(self, state_key, continuation_token=None):
+ def get_raw(self, state_key, continuation_token=None):
with self._lock:
full_state = self._state[self._to_key(state_key)]
if self._use_continuation_tokens:
@@ -718,13 +952,22 @@
assert not continuation_token
return b''.join(full_state), None
- def blocking_append(self, state_key, data):
+ def append_raw(self, state_key, data):
with self._lock:
self._state[self._to_key(state_key)].append(data)
+ return _Future.done()
- def blocking_clear(self, state_key):
+ def clear(self, state_key):
with self._lock:
- del self._state[self._to_key(state_key)]
+ try:
+ del self._state[self._to_key(state_key)]
+ except KeyError:
+ # This may happen with the caching layer across bundles. Caching may
+ # skip this storage layer for a blocking_get(key) request. Without
+ # the caching, the state for a key would be initialized via the
+ # defaultdict that _state uses.
+ pass
+ return _Future.done()
@staticmethod
def _to_key(state_key):
@@ -736,23 +979,23 @@
def State(self, request_stream, context=None):
# Note that this eagerly mutates state, assuming any failures are fatal.
- # Thus it is safe to ignore instruction_reference.
+ # Thus it is safe to ignore instruction_id.
for request in request_stream:
request_type = request.WhichOneof('request')
if request_type == 'get':
- data, continuation_token = self._state.blocking_get(
+ data, continuation_token = self._state.get_raw(
request.state_key, request.get.continuation_token)
yield beam_fn_api_pb2.StateResponse(
id=request.id,
get=beam_fn_api_pb2.StateGetResponse(
data=data, continuation_token=continuation_token))
elif request_type == 'append':
- self._state.blocking_append(request.state_key, request.append.data)
+ self._state.append_raw(request.state_key, request.append.data)
yield beam_fn_api_pb2.StateResponse(
id=request.id,
append=beam_fn_api_pb2.StateAppendResponse())
elif request_type == 'clear':
- self._state.blocking_clear(request.state_key)
+ self._state.clear(request.state_key)
yield beam_fn_api_pb2.StateResponse(
id=request.id,
clear=beam_fn_api_pb2.StateClearResponse())
@@ -773,18 +1016,78 @@
"""Does nothing."""
pass
+ @staticmethod
+ def get_cache_token_generator(static=True):
+ """A generator for cache tokens.
+ :arg static If True, generator always returns the same cache token
+ If False, generator returns a new cache token each time
+ :return A generator which returns a cache token on next(generator)
+ """
+ def generate_token(identifier):
+ return beam_fn_api_pb2.ProcessBundleRequest.CacheToken(
+ user_state=beam_fn_api_pb2
+ .ProcessBundleRequest.CacheToken.UserState(),
+ token="cache_token_{}".format(identifier).encode("utf-8"))
+
+ class StaticGenerator(object):
+ def __init__(self):
+ self._token = generate_token(1)
+
+ def __iter__(self):
+ # pylint: disable=non-iterator-returned
+ return self
+
+ def __next__(self):
+ return self._token
+
+ class DynamicGenerator(object):
+ def __init__(self):
+ self._counter = 0
+ self._lock = threading.Lock()
+
+ def __iter__(self):
+ # pylint: disable=non-iterator-returned
+ return self
+
+ def __next__(self):
+ with self._lock:
+ self._counter += 1
+ return generate_token(self._counter)
+
+ return StaticGenerator() if static else DynamicGenerator()
+
class WorkerHandler(object):
+ """worker_handler for a worker.
+
+ It provides utilities to start / stop the worker, provision any resources for
+ it, as well as provide descriptors for the data, state and logging APIs for
+ it.
+ """
_registered_environments = {}
+ _worker_id_counter = -1
+ _lock = threading.Lock()
def __init__(
self, control_handler, data_plane_handler, state, provision_info):
+ """Initialize a WorkerHandler.
+
+ Args:
+ control_handler:
+ data_plane_handler (data_plane.DataChannel):
+ state:
+ provision_info:
+ """
self.control_handler = control_handler
self.data_plane_handler = data_plane_handler
self.state = state
self.provision_info = provision_info
+ with WorkerHandler._lock:
+ WorkerHandler._worker_id_counter += 1
+ self.worker_id = 'worker_%s' % WorkerHandler._worker_id_counter
+
def close(self):
self.stop_worker()
@@ -811,24 +1114,30 @@
return wrapper
@classmethod
- def create(cls, environment, state, provision_info):
+ def create(cls, environment, state, provision_info, grpc_server):
constructor, payload_type = cls._registered_environments[environment.urn]
return constructor(
proto_utils.parse_Bytes(environment.payload, payload_type),
state,
- provision_info)
+ provision_info,
+ grpc_server)
@WorkerHandler.register_environment(python_urns.EMBEDDED_PYTHON, None)
class EmbeddedWorkerHandler(WorkerHandler):
- """An in-memory controller for fn API control, state and data planes."""
+ """An in-memory worker_handler for fn API control, state and data planes."""
- def __init__(self, unused_payload, state, provision_info):
+ def __init__(self, unused_payload, state, provision_info,
+ unused_grpc_server=None):
super(EmbeddedWorkerHandler, self).__init__(
self, data_plane.InMemoryDataChannel(), state, provision_info)
+ self.control_conn = self
+ self.data_conn = self.data_plane_handler
self.worker = sdk_worker.SdkWorker(
sdk_worker.BundleProcessorCache(
- FnApiRunner.SingletonStateHandlerFactory(self.state),
+ FnApiRunner.SingletonStateHandlerFactory(
+ sdk_worker.CachingMaterializingStateHandler(
+ StateCache(STATE_CACHE_SIZE), state)),
data_plane.InMemoryDataChannelFactory(
self.data_plane_handler.inverse()),
{}))
@@ -838,9 +1147,7 @@
if not request.instruction_id:
self._uid_counter += 1
request.instruction_id = 'control_%s' % self._uid_counter
- logging.debug('CONTROL REQUEST %s', request)
response = self.worker.do_instruction(request)
- logging.debug('CONTROL RESPONSE %s', response)
return ControlFuture(request.instruction_id, response)
def start_worker(self):
@@ -893,16 +1200,27 @@
info=self._info)
-class GrpcWorkerHandler(WorkerHandler):
- """An grpc based controller for fn API control, state and data planes."""
+class EmptyArtifactRetrievalService(
+ beam_artifact_api_pb2_grpc.ArtifactRetrievalServiceServicer):
+
+ def GetManifest(self, request, context=None):
+ return beam_artifact_api_pb2.GetManifestResponse(
+ manifest=beam_artifact_api_pb2.Manifest())
+
+ def GetArtifact(self, request, context=None):
+ raise ValueError('No artifacts staged.')
+
+
+class GrpcServer(object):
_DEFAULT_SHUTDOWN_TIMEOUT_SECS = 5
- def __init__(self, state, provision_info):
+ def __init__(self, state, provision_info, max_workers):
self.state = state
self.provision_info = provision_info
+ self.max_workers = max_workers
self.control_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=10))
+ futures.ThreadPoolExecutor(max_workers=self.max_workers))
self.control_port = self.control_server.add_insecure_port('[::]:0')
self.control_address = 'localhost:%s' % self.control_port
@@ -912,12 +1230,12 @@
no_max_message_sizes = [("grpc.max_receive_message_length", -1),
("grpc.max_send_message_length", -1)]
self.data_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=10),
+ futures.ThreadPoolExecutor(max_workers=self.max_workers),
options=no_max_message_sizes)
self.data_port = self.data_server.add_insecure_port('[::]:0')
self.state_server = grpc.server(
- futures.ThreadPoolExecutor(max_workers=10),
+ futures.ThreadPoolExecutor(max_workers=self.max_workers),
options=no_max_message_sizes)
self.state_port = self.state_server.add_insecure_port('[::]:0')
@@ -937,13 +1255,14 @@
self.control_server)
if self.provision_info.artifact_staging_dir:
- m = beam_artifact_api_pb2_grpc
- m.add_ArtifactRetrievalServiceServicer_to_server(
- artifact_service.BeamFilesystemArtifactService(
- self.provision_info.artifact_staging_dir),
- self.control_server)
+ service = artifact_service.BeamFilesystemArtifactService(
+ self.provision_info.artifact_staging_dir)
+ else:
+ service = EmptyArtifactRetrievalService()
+ beam_artifact_api_pb2_grpc.add_ArtifactRetrievalServiceServicer_to_server(
+ service, self.control_server)
- self.data_plane_handler = data_plane.GrpcServerDataChannel()
+ self.data_plane_handler = data_plane.BeamFnDataServicer()
beam_fn_api_pb2_grpc.add_BeamFnDataServicer_to_server(
self.data_plane_handler, self.data_server)
@@ -968,21 +1287,8 @@
self.data_server.start()
self.control_server.start()
- def data_api_service_descriptor(self):
- return endpoints_pb2.ApiServiceDescriptor(
- url='localhost:%s' % self.data_port)
-
- def state_api_service_descriptor(self):
- return endpoints_pb2.ApiServiceDescriptor(
- url='localhost:%s' % self.state_port)
-
- def logging_api_service_descriptor(self):
- return endpoints_pb2.ApiServiceDescriptor(
- url='localhost:%s' % self.logging_port)
-
def close(self):
self.control_handler.done()
- self.data_plane_handler.close()
to_wait = [
self.control_server.stop(self._DEFAULT_SHUTDOWN_TIMEOUT_SECS),
self.data_server.stop(self._DEFAULT_SHUTDOWN_TIMEOUT_SECS),
@@ -991,23 +1297,64 @@
]
for w in to_wait:
w.wait()
+
+
+class GrpcWorkerHandler(WorkerHandler):
+ """An grpc based worker_handler for fn API control, state and data planes."""
+
+ def __init__(self, state, provision_info, grpc_server):
+ self._grpc_server = grpc_server
+ super(GrpcWorkerHandler, self).__init__(
+ self._grpc_server.control_handler, self._grpc_server.data_plane_handler,
+ state, provision_info)
+ self.state = state
+
+ self.control_address = self.port_from_worker(self._grpc_server.control_port)
+ self.control_conn = self._grpc_server.control_handler.get_conn_by_worker_id(
+ self.worker_id)
+
+ self.data_conn = self._grpc_server.data_plane_handler.get_conn_by_worker_id(
+ self.worker_id)
+
+ def data_api_service_descriptor(self):
+ return endpoints_pb2.ApiServiceDescriptor(
+ url=self.port_from_worker(self._grpc_server.data_port))
+
+ def state_api_service_descriptor(self):
+ return endpoints_pb2.ApiServiceDescriptor(
+ url=self.port_from_worker(self._grpc_server.state_port))
+
+ def logging_api_service_descriptor(self):
+ return endpoints_pb2.ApiServiceDescriptor(
+ url=self.port_from_worker(self._grpc_server.logging_port))
+
+ def close(self):
+ self.control_conn.close()
+ self.data_conn.close()
super(GrpcWorkerHandler, self).close()
+ def port_from_worker(self, port):
+ return '%s:%s' % (self.localhost_from_worker(), port)
+
+ def localhost_from_worker(self):
+ return 'localhost'
+
@WorkerHandler.register_environment(
common_urns.environments.EXTERNAL.urn, beam_runner_api_pb2.ExternalPayload)
class ExternalWorkerHandler(GrpcWorkerHandler):
- def __init__(self, external_payload, state, provision_info):
- super(ExternalWorkerHandler, self).__init__(state, provision_info)
+ def __init__(self, external_payload, state, provision_info, grpc_server):
+ super(ExternalWorkerHandler, self).__init__(state, provision_info,
+ grpc_server)
self._external_payload = external_payload
def start_worker(self):
stub = beam_fn_api_pb2_grpc.BeamFnExternalWorkerPoolStub(
GRPCChannelFactory.insecure_channel(
self._external_payload.endpoint.url))
- response = stub.NotifyRunnerAvailable(
- beam_fn_api_pb2.NotifyRunnerAvailableRequest(
- worker_id='worker_%s' % uuid.uuid4(),
+ response = stub.StartWorker(
+ beam_fn_api_pb2.StartWorkerRequest(
+ worker_id=self.worker_id,
control_endpoint=endpoints_pb2.ApiServiceDescriptor(
url=self.control_address),
logging_endpoint=self.logging_api_service_descriptor(),
@@ -1021,13 +1368,21 @@
@WorkerHandler.register_environment(python_urns.EMBEDDED_PYTHON_GRPC, bytes)
class EmbeddedGrpcWorkerHandler(GrpcWorkerHandler):
- def __init__(self, num_workers_payload, state, provision_info):
- super(EmbeddedGrpcWorkerHandler, self).__init__(state, provision_info)
- self._num_threads = int(num_workers_payload) if num_workers_payload else 1
+ def __init__(self, payload, state, provision_info, grpc_server):
+ super(EmbeddedGrpcWorkerHandler, self).__init__(state, provision_info,
+ grpc_server)
+ if payload:
+ num_workers, state_cache_size = payload.decode('ascii').split(',')
+ self._num_threads = int(num_workers)
+ self._state_cache_size = int(state_cache_size)
+ else:
+ self._num_threads = 1
+ self._state_cache_size = STATE_CACHE_SIZE
def start_worker(self):
self.worker = sdk_worker.SdkHarness(
- self.control_address, worker_count=self._num_threads)
+ self.control_address, worker_count=self._num_threads,
+ state_cache_size=self._state_cache_size, worker_id=self.worker_id)
self.worker_thread = threading.Thread(
name='run_worker', target=self.worker.run)
self.worker_thread.daemon = True
@@ -1037,16 +1392,22 @@
self.worker_thread.join()
+# The subprocesses module is not threadsafe on Python 2.7. Use this lock to
+# prevent concurrent calls to POpen().
+SUBPROCESS_LOCK = threading.Lock()
+
+
@WorkerHandler.register_environment(python_urns.SUBPROCESS_SDK, bytes)
class SubprocessSdkWorkerHandler(GrpcWorkerHandler):
- def __init__(self, worker_command_line, state, provision_info):
- super(SubprocessSdkWorkerHandler, self).__init__(state, provision_info)
+ def __init__(self, worker_command_line, state, provision_info, grpc_server):
+ super(SubprocessSdkWorkerHandler, self).__init__(state, provision_info,
+ grpc_server)
self._worker_command_line = worker_command_line
def start_worker(self):
from apache_beam.runners.portability import local_job_service
self.worker = local_job_service.SubprocessSdkWorker(
- self._worker_command_line, self.control_address)
+ self._worker_command_line, self.control_address, self.worker_id)
self.worker_thread = threading.Thread(
name='run_worker', target=self.worker.run)
self.worker_thread.start()
@@ -1058,84 +1419,125 @@
@WorkerHandler.register_environment(common_urns.environments.DOCKER.urn,
beam_runner_api_pb2.DockerPayload)
class DockerSdkWorkerHandler(GrpcWorkerHandler):
- def __init__(self, payload, state, provision_info):
- super(DockerSdkWorkerHandler, self).__init__(state, provision_info)
+ def __init__(self, payload, state, provision_info, grpc_server):
+ super(DockerSdkWorkerHandler, self).__init__(state, provision_info,
+ grpc_server)
self._container_image = payload.container_image
self._container_id = None
+ def localhost_from_worker(self):
+ if sys.platform == "darwin":
+ # See https://docs.docker.com/docker-for-mac/networking/
+ return 'host.docker.internal'
+ else:
+ return super(DockerSdkWorkerHandler, self).localhost_from_worker()
+
def start_worker(self):
- try:
- subprocess.check_call(['docker', 'pull', self._container_image])
- except Exception:
- logging.info('Unable to pull image %s' % self._container_image)
- self._container_id = subprocess.check_output(
- ['docker',
- 'run',
- '-d',
- # TODO: credentials
- '--network=host',
- self._container_image,
- '--id=%s' % uuid.uuid4(),
- '--logging_endpoint=%s' % self.logging_api_service_descriptor().url,
- '--control_endpoint=%s' % self.control_address,
- '--artifact_endpoint=%s' % self.control_address,
- '--provision_endpoint=%s' % self.control_address,
- ]).strip()
- while True:
- logging.info('Waiting for docker to start up...')
- status = subprocess.check_output([
- 'docker',
- 'inspect',
- '-f',
- '{{.State.Status}}',
- self._container_id]).strip()
- if status == 'running':
- break
- elif status in ('dead', 'exited'):
- subprocess.call([
+ with SUBPROCESS_LOCK:
+ try:
+ subprocess.check_call(['docker', 'pull', self._container_image])
+ except Exception:
+ logging.info('Unable to pull image %s' % self._container_image)
+ self._container_id = subprocess.check_output(
+ ['docker',
+ 'run',
+ '-d',
+ # TODO: credentials
+ '--network=host',
+ self._container_image,
+ '--id=%s' % self.worker_id,
+ '--logging_endpoint=%s' % self.logging_api_service_descriptor().url,
+ '--control_endpoint=%s' % self.control_address,
+ '--artifact_endpoint=%s' % self.control_address,
+ '--provision_endpoint=%s' % self.control_address,
+ ]).strip()
+ while True:
+ status = subprocess.check_output([
'docker',
- 'container',
- 'logs',
- self._container_id])
- raise RuntimeError('SDK failed to start.')
+ 'inspect',
+ '-f',
+ '{{.State.Status}}',
+ self._container_id]).strip()
+ logging.info('Waiting for docker to start up.Current status is %s' %
+ status)
+ if status == b'running':
+ logging.info('Docker container is running. container_id = %s, '
+ 'worker_id = %s', self._container_id, self.worker_id)
+ break
+ elif status in (b'dead', b'exited'):
+ subprocess.call([
+ 'docker',
+ 'container',
+ 'logs',
+ self._container_id])
+ raise RuntimeError('SDK failed to start. Final status is %s' % status)
time.sleep(1)
def stop_worker(self):
if self._container_id:
- subprocess.call([
- 'docker',
- 'kill',
- self._container_id])
+ with SUBPROCESS_LOCK:
+ subprocess.call([
+ 'docker',
+ 'kill',
+ self._container_id])
class WorkerHandlerManager(object):
- def __init__(self, environments, job_provision_info=None):
+ def __init__(self, environments, job_provision_info):
self._environments = environments
self._job_provision_info = job_provision_info
- self._cached_handlers = {}
+ self._cached_handlers = collections.defaultdict(list)
self._state = FnApiRunner.StateServicer() # rename?
+ self._grpc_server = None
- def get_worker_handler(self, environment_id):
+ def get_worker_handlers(self, environment_id, num_workers):
if environment_id is None:
# Any environment will do, pick one arbitrarily.
environment_id = next(iter(self._environments.keys()))
environment = self._environments[environment_id]
+ max_total_workers = num_workers * len(self._environments)
- worker_handler = self._cached_handlers.get(environment_id)
- if worker_handler is None:
- worker_handler = self._cached_handlers[
- environment_id] = WorkerHandler.create(
- environment, self._state, self._job_provision_info)
- worker_handler.start_worker()
- return worker_handler
+ # assume all environments except EMBEDDED_PYTHON use gRPC.
+ if environment.urn == python_urns.EMBEDDED_PYTHON:
+ pass # no need for a gRPC server
+ elif self._grpc_server is None:
+ self._grpc_server = GrpcServer(self._state, self._job_provision_info,
+ max_total_workers)
+ elif max_total_workers > self._grpc_server.max_workers:
+ # each gRPC server is running with fixed number of threads (
+ # max_total_workers), which is defined by the first call to
+ # get_worker_handlers(). Assumption here is a worker has a connection to a
+ # gRPC server. In case a stage tries to add more workers
+ # than the max_total_workers, some workers cannot connect to gRPC and
+ # pipeline will hang, hence raise an error here.
+ raise RuntimeError('gRPC servers are running with %s threads, we cannot '
+ 'attach %s workers.' % (self._grpc_server.max_workers,
+ max_total_workers))
+
+ worker_handler_list = self._cached_handlers[environment_id]
+ if len(worker_handler_list) < num_workers:
+ for _ in range(len(worker_handler_list), num_workers):
+ worker_handler = WorkerHandler.create(
+ environment, self._state, self._job_provision_info,
+ self._grpc_server)
+ logging.info("Created Worker handler %s for environment %s",
+ worker_handler, environment)
+ self._cached_handlers[environment_id].append(worker_handler)
+ worker_handler.start_worker()
+ return self._cached_handlers[environment_id][:num_workers]
def close_all(self):
- for controller in set(self._cached_handlers.values()):
- try:
- controller.close()
- except Exception:
- logging.error("Error closing controller %s" % controller, exc_info=True)
+ for worker_handler_list in self._cached_handlers.values():
+ for worker_handler in set(worker_handler_list):
+ try:
+ worker_handler.close()
+ except Exception:
+ logging.error("Error closing worker_handler %s" % worker_handler,
+ exc_info=True)
self._cached_handlers = {}
+ if self._grpc_server is not None:
+ self._grpc_server.close()
+ self._grpc_server = None
class ExtendedProvisionInfo(object):
@@ -1166,35 +1568,69 @@
class BundleManager(object):
+ """Manages the execution of a bundle from the runner-side.
+
+ This class receives a bundle descriptor, and performs the following tasks:
+ - Registration of the bundle with the worker.
+ - Splitting of the bundle
+ - Setting up any other bundle requirements (e.g. side inputs).
+ - Submitting the bundle to worker for execution
+ - Passing bundle input data to the worker
+ - Collecting bundle output data from the worker
+ - Finalizing the bundle.
+ """
_uid_counter = 0
+ _lock = threading.Lock()
def __init__(
- self, controller, get_buffer, get_input_coder_impl, bundle_descriptor,
- progress_frequency=None, skip_registration=False):
- self._controller = controller
+ self, worker_handler_list, get_buffer, get_input_coder_impl,
+ bundle_descriptor, progress_frequency=None, skip_registration=False,
+ cache_token_generator=FnApiRunner.get_cache_token_generator()):
+ """Set up a bundle manager.
+
+ Args:
+ worker_handler_list
+ get_buffer (Callable[[str], list])
+ get_input_coder_impl (Callable[[str], Coder])
+ bundle_descriptor (beam_fn_api_pb2.ProcessBundleDescriptor)
+ progress_frequency
+ skip_registration
+ """
+ self._worker_handler_list = worker_handler_list
self._get_buffer = get_buffer
self._get_input_coder_impl = get_input_coder_impl
self._bundle_descriptor = bundle_descriptor
self._registered = skip_registration
self._progress_frequency = progress_frequency
+ self._worker_handler = None
+ self._cache_token_generator = cache_token_generator
- def process_bundle(self, inputs, expected_outputs):
- # Unique id for the instruction processing this bundle.
- BundleManager._uid_counter += 1
- process_bundle_id = 'bundle_%s' % BundleManager._uid_counter
+ def _send_input_to_worker(self,
+ process_bundle_id,
+ read_transform_id,
+ byte_streams):
+ data_out = self._worker_handler.data_conn.output_stream(
+ process_bundle_id, read_transform_id)
+ for byte_stream in byte_streams:
+ data_out.write(byte_stream)
+ data_out.close()
- # Register the bundle descriptor, if needed.
+ def _register_bundle_descriptor(self):
if self._registered:
registration_future = None
else:
process_bundle_registration = beam_fn_api_pb2.InstructionRequest(
register=beam_fn_api_pb2.RegisterRequest(
process_bundle_descriptor=[self._bundle_descriptor]))
- registration_future = self._controller.control_handler.push(
+ registration_future = self._worker_handler.control_conn.push(
process_bundle_registration)
self._registered = True
+ return registration_future
+
+ def _select_split_manager(self):
+ """TODO(pabloem) WHAT DOES THIS DO"""
unique_names = set(
t.unique_name for t in self._bundle_descriptor.transforms.values())
for stage_name, candidate in reversed(_split_managers):
@@ -1205,104 +1641,117 @@
else:
split_manager = None
- if not split_manager:
- # Write all the input data to the channel immediately.
- for (transform_id, name), elements in inputs.items():
- data_out = self._controller.data_plane_handler.output_stream(
- process_bundle_id, beam_fn_api_pb2.Target(
- primitive_transform_reference=transform_id, name=name))
- for element_data in elements:
- data_out.write(element_data)
- data_out.close()
+ return split_manager
+ def _generate_splits_for_testing(self,
+ split_manager,
+ inputs,
+ process_bundle_id):
split_results = []
+ read_transform_id, buffer_data = only_element(inputs.items())
- # Actually start the bundle.
+ byte_stream = b''.join(buffer_data)
+ num_elements = len(list(
+ self._get_input_coder_impl(read_transform_id).decode_all(byte_stream)))
+
+ # Start the split manager in case it wants to set any breakpoints.
+ split_manager_generator = split_manager(num_elements)
+ try:
+ split_fraction = next(split_manager_generator)
+ done = False
+ except StopIteration:
+ done = True
+
+ # Send all the data.
+ self._send_input_to_worker(
+ process_bundle_id, read_transform_id, [byte_stream])
+
+ # Execute the requested splits.
+ while not done:
+ if split_fraction is None:
+ split_result = None
+ else:
+ split_request = beam_fn_api_pb2.InstructionRequest(
+ process_bundle_split=
+ beam_fn_api_pb2.ProcessBundleSplitRequest(
+ instruction_id=process_bundle_id,
+ desired_splits={
+ read_transform_id:
+ beam_fn_api_pb2.ProcessBundleSplitRequest.DesiredSplit(
+ fraction_of_remainder=split_fraction,
+ estimated_input_elements=num_elements)
+ }))
+ split_response = self._worker_handler.control_conn.push(
+ split_request).get()
+ for t in (0.05, 0.1, 0.2):
+ waiting = ('Instruction not running', 'not yet scheduled')
+ if any(msg in split_response.error for msg in waiting):
+ time.sleep(t)
+ split_response = self._worker_handler.control_conn.push(
+ split_request).get()
+ if 'Unknown process bundle' in split_response.error:
+ # It may have finished too fast.
+ split_result = None
+ elif split_response.error:
+ raise RuntimeError(split_response.error)
+ else:
+ split_result = split_response.process_bundle_split
+ split_results.append(split_result)
+ try:
+ split_fraction = split_manager_generator.send(split_result)
+ except StopIteration:
+ break
+ return split_results
+
+ def process_bundle(self, inputs, expected_outputs):
+ # Unique id for the instruction processing this bundle.
+ with BundleManager._lock:
+ BundleManager._uid_counter += 1
+ process_bundle_id = 'bundle_%s' % BundleManager._uid_counter
+ self._worker_handler = self._worker_handler_list[
+ BundleManager._uid_counter % len(self._worker_handler_list)]
+
+ # Register the bundle descriptor, if needed - noop if already registered.
+ registration_future = self._register_bundle_descriptor()
+ # Check that the bundle was successfully registered.
if registration_future and registration_future.get().error:
raise RuntimeError(registration_future.get().error)
- process_bundle = beam_fn_api_pb2.InstructionRequest(
+
+ split_manager = self._select_split_manager()
+ if not split_manager:
+ # If there is no split_manager, write all input data to the channel.
+ for transform_id, elements in inputs.items():
+ self._send_input_to_worker(
+ process_bundle_id, transform_id, elements)
+
+ # Actually start the bundle.
+ process_bundle_req = beam_fn_api_pb2.InstructionRequest(
instruction_id=process_bundle_id,
process_bundle=beam_fn_api_pb2.ProcessBundleRequest(
- process_bundle_descriptor_reference=self._bundle_descriptor.id))
- result_future = self._controller.control_handler.push(process_bundle)
+ process_bundle_descriptor_id=self._bundle_descriptor.id,
+ cache_tokens=[next(self._cache_token_generator)]))
+ result_future = self._worker_handler.control_conn.push(process_bundle_req)
+ split_results = []
with ProgressRequester(
- self._controller, process_bundle_id, self._progress_frequency):
+ self._worker_handler, process_bundle_id, self._progress_frequency):
+
if split_manager:
- (read_transform_id, name), buffer_data = only_element(inputs.items())
- num_elements = len(list(
- self._get_input_coder_impl(read_transform_id).decode_all(
- b''.join(buffer_data))))
-
- # Start the split manager in case it wants to set any breakpoints.
- split_manager_generator = split_manager(num_elements)
- try:
- split_fraction = next(split_manager_generator)
- done = False
- except StopIteration:
- done = True
-
- # Send all the data.
- data_out = self._controller.data_plane_handler.output_stream(
- process_bundle_id,
- beam_fn_api_pb2.Target(
- primitive_transform_reference=read_transform_id, name=name))
- data_out.write(b''.join(buffer_data))
- data_out.close()
-
- # Execute the requested splits.
- while not done:
- if split_fraction is None:
- split_result = None
- else:
- split_request = beam_fn_api_pb2.InstructionRequest(
- process_bundle_split=
- beam_fn_api_pb2.ProcessBundleSplitRequest(
- instruction_reference=process_bundle_id,
- desired_splits={
- read_transform_id:
- beam_fn_api_pb2.ProcessBundleSplitRequest.DesiredSplit(
- fraction_of_remainder=split_fraction,
- estimated_input_elements=num_elements)
- }))
- split_response = self._controller.control_handler.push(
- split_request).get()
- for t in (0.05, 0.1, 0.2):
- waiting = ('Instruction not running', 'not yet scheduled')
- if any(msg in split_response.error for msg in waiting):
- time.sleep(t)
- split_response = self._controller.control_handler.push(
- split_request).get()
- if 'Unknown process bundle' in split_response.error:
- # It may have finished too fast.
- split_result = None
- elif split_response.error:
- raise RuntimeError(split_response.error)
- else:
- split_result = split_response.process_bundle_split
- split_results.append(split_result)
- try:
- split_fraction = split_manager_generator.send(split_result)
- except StopIteration:
- break
+ split_results = self._generate_splits_for_testing(
+ split_manager, inputs, process_bundle_id)
# Gather all output data.
- expected_targets = [
- beam_fn_api_pb2.Target(primitive_transform_reference=transform_id,
- name=output_name)
- for (transform_id, output_name), _ in expected_outputs.items()]
- logging.debug('Gather all output data from %s.', expected_targets)
- for output in self._controller.data_plane_handler.input_elements(
+ for output in self._worker_handler.data_conn.input_elements(
process_bundle_id,
- expected_targets,
+ expected_outputs.keys(),
abort_callback=lambda: (result_future.is_done()
and result_future.get().error)):
- target_tuple = (
- output.target.primitive_transform_reference, output.target.name)
- if target_tuple in expected_outputs:
- self._get_buffer(expected_outputs[target_tuple]).append(output.data)
+ if output.transform_id in expected_outputs:
+ with BundleManager._lock:
+ self._get_buffer(
+ expected_outputs[output.transform_id]).append(output.data)
- logging.debug('Wait for the bundle to finish.')
+ logging.debug('Wait for the bundle %s to finish.' % process_bundle_id)
result = result_future.get()
if result.error:
@@ -1312,18 +1761,65 @@
finalize_request = beam_fn_api_pb2.InstructionRequest(
finalize_bundle=
beam_fn_api_pb2.FinalizeBundleRequest(
- instruction_reference=process_bundle_id
+ instruction_id=process_bundle_id
))
- self._controller.control_handler.push(
- finalize_request)
+ self._worker_handler.control_conn.push(finalize_request)
return result, split_results
+class ParallelBundleManager(BundleManager):
+
+ def __init__(
+ self, worker_handler_list, get_buffer, get_input_coder_impl,
+ bundle_descriptor, progress_frequency=None, skip_registration=False,
+ cache_token_generator=None, **kwargs):
+ super(ParallelBundleManager, self).__init__(
+ worker_handler_list, get_buffer, get_input_coder_impl,
+ bundle_descriptor, progress_frequency, skip_registration,
+ cache_token_generator=cache_token_generator)
+ self._num_workers = kwargs.pop('num_workers', 1)
+
+ def process_bundle(self, inputs, expected_outputs):
+ part_inputs = [{} for _ in range(self._num_workers)]
+ for name, input in inputs.items():
+ for ix, part in enumerate(input.partition(self._num_workers)):
+ part_inputs[ix][name] = part
+
+ merged_result = None
+ split_result_list = []
+ with futures.ThreadPoolExecutor(max_workers=self._num_workers) as executor:
+ for result, split_result in executor.map(lambda part: BundleManager(
+ self._worker_handler_list, self._get_buffer,
+ self._get_input_coder_impl, self._bundle_descriptor,
+ self._progress_frequency, self._registered,
+ cache_token_generator=self._cache_token_generator).process_bundle(
+ part, expected_outputs), part_inputs):
+
+ split_result_list += split_result
+ if merged_result is None:
+ merged_result = result
+ else:
+ merged_result = beam_fn_api_pb2.InstructionResponse(
+ process_bundle=beam_fn_api_pb2.ProcessBundleResponse(
+ monitoring_infos=monitoring_infos.consolidate(
+ itertools.chain(
+ result.process_bundle.monitoring_infos,
+ merged_result.process_bundle.monitoring_infos))),
+ error=result.error or merged_result.error)
+
+ return merged_result, split_result_list
+
+
class ProgressRequester(threading.Thread):
- def __init__(self, controller, instruction_id, frequency, callback=None):
+ """ Thread that asks SDK Worker for progress reports with a certain frequency.
+
+ A callback can be passed to call with progress updates.
+ """
+
+ def __init__(self, worker_handler, instruction_id, frequency, callback=None):
super(ProgressRequester, self).__init__()
- self._controller = controller
+ self._worker_handler = worker_handler
self._instruction_id = instruction_id
self._frequency = frequency
self._done = False
@@ -1342,11 +1838,11 @@
def run(self):
while not self._done:
try:
- progress_result = self._controller.control_handler.push(
+ progress_result = self._worker_handler.control_conn.push(
beam_fn_api_pb2.InstructionRequest(
process_bundle_progress=
beam_fn_api_pb2.ProcessBundleProgressRequest(
- instruction_reference=self._instruction_id))).get()
+ instruction_id=self._instruction_id))).get()
self._latest_progress = progress_result.process_bundle_progress
if self._callback:
self._callback(self._latest_progress)
@@ -1418,9 +1914,9 @@
def _to_metric_key(self, monitoring_info):
# Right now this assumes that all metrics have a PTRANSFORM
- ptransform_id = monitoring_info.labels['PTRANSFORM']
+ transform_id = monitoring_info.labels['PTRANSFORM']
namespace, name = monitoring_infos.parse_namespace_and_name(monitoring_info)
- return MetricKey(ptransform_id, MetricName(namespace, name))
+ return MetricKey(transform_id, MetricName(namespace, name))
def query(self, filter=None):
counters = [metrics.execution.MetricResult(k, v, v)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index a807cfa..e3620d7 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -27,11 +27,13 @@
import threading
import time
import traceback
+import typing
import unittest
import uuid
from builtins import range
-import hamcrest
+import hamcrest # pylint: disable=ungrouped-imports
+import mock
from hamcrest.core.matcher import Matcher
from hamcrest.core.string_description import StringDescription
from tenacity import retry
@@ -39,6 +41,7 @@
import apache_beam as beam
from apache_beam.io import restriction_trackers
+from apache_beam.io.concat_source_test import RangeSource
from apache_beam.metrics import monitoring_infos
from apache_beam.metrics.execution import MetricKey
from apache_beam.metrics.execution import MetricsEnvironment
@@ -47,10 +50,13 @@
from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.runners.portability import fn_api_runner
from apache_beam.runners.worker import data_plane
+from apache_beam.runners.worker import sdk_worker
from apache_beam.runners.worker import statesampler
from apache_beam.testing.synthetic_pipeline import SyntheticSDFAsSource
+from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.tools import utils
from apache_beam.transforms import userstate
from apache_beam.transforms import window
@@ -240,10 +246,6 @@
main | beam.Map(lambda a, b: (a, b), beam.pvalue.AsDict(side)),
equal_to([(None, {'a': [1]})]))
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6878')
def test_multimap_side_input(self):
with self.create_pipeline() as p:
main = p | 'main' >> beam.Create(['a', 'b'])
@@ -255,6 +257,19 @@
beam.pvalue.AsMultiMap(side)),
equal_to([('a', [1, 3]), ('b', [2])]))
+ def test_multimap_side_input_type_coercion(self):
+ with self.create_pipeline() as p:
+ main = p | 'main' >> beam.Create(['a', 'b'])
+ # The type of this side-input is forced to Any (overriding type
+ # inference). Without type coercion to Tuple[Any, Any], the usage of this
+ # side-input in AsMultiMap() below should fail.
+ side = (p | 'side' >> beam.Create([('a', 1), ('b', 2), ('a', 3)])
+ .with_output_types(typing.Any))
+ assert_that(
+ main | beam.Map(lambda k, d: (k, sorted(d[k])),
+ beam.pvalue.AsMultiMap(side)),
+ equal_to([('a', [1, 3]), ('b', [2])]))
+
def test_pardo_unfusable_side_inputs(self):
def cross_product(elem, sides):
for side in sides:
@@ -296,6 +311,36 @@
assert_that(p | beam.Create(inputs) | beam.ParDo(AddIndex()),
equal_to(expected))
+ @unittest.skip('TestStream not yet supported')
+ def test_teststream_pardo_timers(self):
+ timer_spec = userstate.TimerSpec('timer', userstate.TimeDomain.WATERMARK)
+
+ class TimerDoFn(beam.DoFn):
+ def process(self, element, timer=beam.DoFn.TimerParam(timer_spec)):
+ unused_key, ts = element
+ timer.set(ts)
+ timer.set(2 * ts)
+
+ @userstate.on_timer(timer_spec)
+ def process_timer(self):
+ yield 'fired'
+
+ ts = (TestStream()
+ .add_elements([('k1', 10)]) # Set timer for 20
+ .advance_watermark_to(100)
+ .add_elements([('k2', 100)]) # Set timer for 200
+ .advance_watermark_to(1000))
+
+ with self.create_pipeline() as p:
+ _ = (
+ p
+ | ts
+ | beam.ParDo(TimerDoFn())
+ | beam.Map(lambda x, ts=beam.DoFn.TimestampParam: (x, ts)))
+
+ #expected = [('fired', ts) for ts in (20, 200)]
+ #assert_that(actual, equal_to(expected))
+
def test_pardo_timers(self):
timer_spec = userstate.TimerSpec('timer', userstate.TimeDomain.WATERMARK)
@@ -430,8 +475,10 @@
assert isinstance(
restriction_tracker,
restriction_trackers.OffsetRestrictionTracker), restriction_tracker
- for k in range(*restriction_tracker.current_restriction()):
- yield element[k]
+ cur = restriction_tracker.start_position()
+ while restriction_tracker.try_claim(cur):
+ yield element[cur]
+ cur += 1
with self.create_pipeline() as p:
data = ['abc', 'defghijklmno', 'pqrstuv', 'wxyz']
@@ -454,14 +501,14 @@
assert isinstance(
restriction_tracker,
restriction_trackers.OffsetRestrictionTracker), restriction_tracker
- for k in range(*restriction_tracker.current_restriction()):
- if not restriction_tracker.try_claim(k):
- return
+ cur = restriction_tracker.start_position()
+ while restriction_tracker.try_claim(cur):
counter.inc()
- yield element[k]
- if k % 2 == 1:
+ yield element[cur]
+ if cur % 2 == 1:
restriction_tracker.defer_remainder()
return
+ cur += 1
with self.create_pipeline() as p:
data = ['abc', 'defghijklmno', 'pqrstuv', 'wxyz']
@@ -478,6 +525,34 @@
self.assertEqual(1, len(counters))
self.assertEqual(counters[0].committed, len(''.join(data)))
+ def _run_sdf_wrapper_pipeline(self, source, expected_value):
+ with self.create_pipeline() as p:
+ from apache_beam.options.pipeline_options import DebugOptions
+ experiments = (p._options.view_as(DebugOptions).experiments or [])
+
+ # Setup experiment option to enable using SDFBoundedSourceWrapper
+ if not 'use_sdf_bounded_source' in experiments:
+ experiments.append('use_sdf_bounded_source')
+
+ p._options.view_as(DebugOptions).experiments = experiments
+
+ actual = (
+ p | beam.io.Read(source)
+ )
+ assert_that(actual, equal_to(expected_value))
+
+ @mock.patch('apache_beam.io.iobase._SDFBoundedSourceWrapper.expand')
+ def test_sdf_wrapper_overrides_read(self, sdf_wrapper_mock_expand):
+ def _fake_wrapper_expand(pbegin):
+ return (pbegin
+ | beam.Create(['1']))
+
+ sdf_wrapper_mock_expand.side_effect = _fake_wrapper_expand
+ self._run_sdf_wrapper_pipeline(RangeSource(0, 4), ['1'])
+
+ def test_sdf_wrap_range_source(self):
+ self._run_sdf_wrapper_pipeline(RangeSource(0, 4), [0, 1, 2, 3])
+
def test_group_by_key(self):
with self.create_pipeline() as p:
res = (p
@@ -712,7 +787,7 @@
# These tests are kept in a separate group so that they are
-# not ran in he FnApiRunnerTestWithBundleRepeat which repeats
+# not ran in the FnApiRunnerTestWithBundleRepeat which repeats
# bundle processing. This breaks the byte sampling metrics as
# it makes the probability of sampling far too small
# upon repeating bundle processing due to unncessarily incrementing
@@ -1104,7 +1179,54 @@
runner=fn_api_runner.FnApiRunner(
default_environment=beam_runner_api_pb2.Environment(
urn=python_urns.EMBEDDED_PYTHON_GRPC,
- payload=b'2')))
+ payload=b'2,%d' % fn_api_runner.STATE_CACHE_SIZE)))
+
+
+class FnApiRunnerTestWithDisabledCaching(FnApiRunnerTest):
+
+ def create_pipeline(self):
+ return beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC,
+ # number of workers, state cache size
+ payload=b'2,0')))
+
+
+class FnApiRunnerTestWithMultiWorkers(FnApiRunnerTest):
+
+ def create_pipeline(self):
+ from apache_beam.options.pipeline_options import PipelineOptions
+ pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ p = beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(),
+ options=pipeline_options)
+ return p
+
+ def test_metrics(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+ def test_sdf_with_sdf_initiated_checkpointing(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+
+class FnApiRunnerTestWithGrpcAndMultiWorkers(FnApiRunnerTest):
+
+ def create_pipeline(self):
+ from apache_beam.options.pipeline_options import PipelineOptions
+ pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ p = beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC)),
+ options=pipeline_options)
+ return p
+
+ def test_metrics(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+ def test_sdf_with_sdf_initiated_checkpointing(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
class FnApiRunnerTestWithBundleRepeat(FnApiRunnerTest):
@@ -1117,6 +1239,25 @@
raise unittest.SkipTest("TODO: Avoid bundle finalizations on repeat.")
+class FnApiRunnerTestWithBundleRepeatAndMultiWorkers(FnApiRunnerTest):
+
+ def create_pipeline(self):
+ from apache_beam.options.pipeline_options import PipelineOptions
+ pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ return beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(bundle_repeat=3),
+ options=pipeline_options)
+
+ def test_register_finalizations(self):
+ raise unittest.SkipTest("TODO: Avoid bundle finalizations on repeat.")
+
+ def test_metrics(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+ def test_sdf_with_sdf_initiated_checkpointing(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+
class FnApiRunnerSplitTest(unittest.TestCase):
def create_pipeline(self):
@@ -1194,12 +1335,13 @@
element_counter = ElementCounter()
def split_manager(num_elements):
- element_counter.reset()
- breakpoint = element_counter.set_breakpoint(1)
- yield
- breakpoint.wait()
- yield 0
- breakpoint.clear()
+ if num_elements > 0:
+ element_counter.reset()
+ breakpoint = element_counter.set_breakpoint(1)
+ yield
+ breakpoint.wait()
+ yield 0
+ breakpoint.clear()
# Everything should be perfectly split.
elements = [2, 3]
@@ -1213,7 +1355,7 @@
is_first_bundle = [True] # emulate nonlocal for Python 2
def split_manager(num_elements):
- if is_first_bundle:
+ if is_first_bundle and num_elements > 0:
del is_first_bundle[:]
breakpoint = element_counter.set_breakpoint(1)
yield
@@ -1243,14 +1385,15 @@
element_counter = ElementCounter()
def split_manager(num_elements):
- element_counter.reset()
- wait_for = r.randrange(num_elements)
- breakpoint = element_counter.set_breakpoint(wait_for)
- yield
- breakpoint.wait()
- yield r.random()
- yield r.random()
- breakpoint.clear()
+ if num_elements > 0:
+ element_counter.reset()
+ wait_for = r.randrange(num_elements)
+ breakpoint = element_counter.set_breakpoint(wait_for)
+ yield
+ breakpoint.wait()
+ yield r.random()
+ yield r.random()
+ breakpoint.clear()
try:
elements = [r.randrange(5, 10) for _ in range(5)]
@@ -1265,18 +1408,17 @@
class EnumerateProvider(beam.transforms.core.RestrictionProvider):
def initial_restriction(self, element):
- return (0, element)
+ return restriction_trackers.OffsetRange(0, element)
def create_tracker(self, restriction):
- return restriction_trackers.OffsetRestrictionTracker(
- *restriction)
+ return restriction_trackers.OffsetRestrictionTracker(restriction)
def split(self, element, restriction):
# Don't do any initial splitting to simplify test.
return [restriction]
def restriction_size(self, element, restriction):
- return restriction[1] - restriction[0]
+ return restriction.size()
class EnumerateSdf(beam.DoFn):
def process(
@@ -1284,12 +1426,11 @@
element,
restriction_tracker=beam.DoFn.RestrictionParam(EnumerateProvider())):
to_emit = []
- for k in range(*restriction_tracker.current_restriction()):
- if restriction_tracker.try_claim(k):
- to_emit.append((element, k))
- element_counter.increment()
- else:
- break
+ cur = restriction_tracker.start_position()
+ while restriction_tracker.try_claim(cur):
+ to_emit.append((element, cur))
+ element_counter.increment()
+ cur += 1
# Emitting in batches for tighter testing.
yield to_emit
@@ -1410,19 +1551,68 @@
class ExpandStringsProvider(beam.transforms.core.RestrictionProvider):
"""A RestrictionProvider that used for sdf related tests."""
def initial_restriction(self, element):
- return (0, len(element))
+ return restriction_trackers.OffsetRange(0, len(element))
def create_tracker(self, restriction):
- return restriction_trackers.OffsetRestrictionTracker(
- restriction[0], restriction[1])
+ return restriction_trackers.OffsetRestrictionTracker(restriction)
def split(self, element, restriction):
- start, end = restriction
- middle = (end - start) // 2
- return [(start, middle), (middle, end)]
+ desired_bundle_size = restriction.size() // 2
+ return restriction.split(desired_bundle_size)
def restriction_size(self, element, restriction):
- return restriction[1] - restriction[0]
+ return restriction.size()
+
+
+class FnApiRunnerSplitTestWithMultiWorkers(FnApiRunnerSplitTest):
+
+ def create_pipeline(self):
+ from apache_beam.options.pipeline_options import PipelineOptions
+ pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+ p = beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC)),
+ options=pipeline_options)
+ return p
+
+ def test_checkpoint(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+ def test_split_half(self):
+ raise unittest.SkipTest("This test is for a single worker only.")
+
+
+class FnApiBasedLullLoggingTest(unittest.TestCase):
+ def create_pipeline(self):
+ return beam.Pipeline(
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC),
+ progress_request_frequency=0.5))
+
+ def test_lull_logging(self):
+
+ # TODO(BEAM-1251): Remove this test skip after dropping Py 2 support.
+ if sys.version_info < (3, 4):
+ self.skipTest('Log-based assertions are supported after Python 3.4')
+ try:
+ utils.check_compiled('apache_beam.runners.worker.opcounters')
+ except RuntimeError:
+ self.skipTest('Cython is not available')
+
+ with self.assertLogs(level='WARNING') as logs:
+ with self.create_pipeline() as p:
+ sdk_worker.DEFAULT_LOG_LULL_TIMEOUT_NS = 1000 * 1000 # Lull after 1 ms
+
+ _ = (p
+ | beam.Create([1])
+ | beam.Map(time.sleep))
+
+ self.assertRegexpMatches(
+ ''.join(logs.output),
+ '.*There has been a processing lull of over.*',
+ 'Unable to find a lull logged for this job.')
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py b/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
index ffbf9d5..4f3e2f9 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
@@ -45,9 +45,7 @@
COMBINE_URNS = frozenset([
common_urns.composites.COMBINE_PER_KEY.urn,
- common_urns.combine_components.COMBINE_PGBKCV.urn,
- common_urns.combine_components.COMBINE_MERGE_ACCUMULATORS.urn,
- common_urns.combine_components.COMBINE_EXTRACT_OUTPUTS.urn])
+])
PAR_DO_URNS = frozenset([
common_urns.primitives.PAR_DO.urn,
@@ -55,7 +53,8 @@
common_urns.sdf_components.SPLIT_RESTRICTION.urn,
common_urns.sdf_components.SPLIT_AND_SIZE_RESTRICTIONS.urn,
common_urns.sdf_components.PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS.urn,
- common_urns.sdf_components.PROCESS_ELEMENTS.urn])
+ common_urns.sdf_components.PROCESS_ELEMENTS.urn,
+])
IMPULSE_BUFFER = b'impulse'
@@ -308,13 +307,13 @@
@memoize_on_instance
def with_state_iterables(self, coder_id):
coder = self.components.coders[coder_id]
- if coder.spec.spec.urn == common_urns.coders.ITERABLE.urn:
+ if coder.spec.urn == common_urns.coders.ITERABLE.urn:
new_coder_id = unique_name(
self.components.coders, coder_id + '_state_backed')
new_coder = self.components.coders[new_coder_id]
new_coder.CopyFrom(coder)
- new_coder.spec.spec.urn = common_urns.coders.STATE_BACKED_ITERABLE.urn
- new_coder.spec.spec.payload = b'1'
+ new_coder.spec.urn = common_urns.coders.STATE_BACKED_ITERABLE.urn
+ new_coder.spec.payload = b'1'
new_coder.component_coder_ids[0] = self.with_state_iterables(
coder.component_coder_ids[0])
return new_coder_id
@@ -343,9 +342,9 @@
@memoize_on_instance
def length_prefixed_and_safe_coder(self, coder_id):
coder = self.components.coders[coder_id]
- if coder.spec.spec.urn == common_urns.coders.LENGTH_PREFIX.urn:
+ if coder.spec.urn == common_urns.coders.LENGTH_PREFIX.urn:
return coder_id, self.bytes_coder_id
- elif coder.spec.spec.urn in self._KNOWN_CODER_URNS:
+ elif coder.spec.urn in self._KNOWN_CODER_URNS:
new_component_ids = [
self.length_prefixed_coder(c) for c in coder.component_coder_ids]
if new_component_ids == coder.component_coder_ids:
@@ -373,9 +372,8 @@
self.components.coders, coder_id + '_length_prefixed')
self.components.coders[new_coder_id].CopyFrom(
beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.LENGTH_PREFIX.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.LENGTH_PREFIX.urn),
component_coder_ids=[coder_id]))
return new_coder_id, self.bytes_coder_id
@@ -608,25 +606,22 @@
accumulator_coder_id = combine_payload.accumulator_coder_id
key_accumulator_coder = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
component_coder_ids=[key_coder_id, accumulator_coder_id])
key_accumulator_coder_id = context.add_or_get_coder_id(
key_accumulator_coder)
accumulator_iter_coder = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.ITERABLE.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.ITERABLE.urn),
component_coder_ids=[accumulator_coder_id])
accumulator_iter_coder_id = context.add_or_get_coder_id(
accumulator_iter_coder)
key_accumulator_iter_coder = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
component_coder_ids=[key_coder_id, accumulator_iter_coder_id])
key_accumulator_iter_coder_id = context.add_or_get_coder_id(
key_accumulator_iter_coder)
@@ -768,20 +763,18 @@
main_input_id = transform.inputs[main_input_tag]
element_coder_id = context.components.pcollections[
main_input_id].coder_id
- # KV[element, restriction]
+ # Tuple[element, restriction]
paired_coder_id = context.add_or_get_coder_id(
beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
component_coder_ids=[element_coder_id,
pardo_payload.restriction_coder_id]))
- # KV[KV[element, restriction], double]
+ # Tuple[Tuple[element, restriction], double]
sized_coder_id = context.add_or_get_coder_id(
beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
component_coder_ids=[
paired_coder_id,
context.add_or_get_coder_id(
@@ -1020,9 +1013,6 @@
for output in transform.outputs.values():
producers_by_pcoll[output] = stage
- logging.debug('consumers\n%s', consumers_by_pcoll)
- logging.debug('producers\n%s', producers_by_pcoll)
-
# Now try to fuse away all pcollections.
for pcoll, producer in producers_by_pcoll.items():
write_pcoll = None
@@ -1180,15 +1170,14 @@
next(iter(transform.inputs.values()))]
# Create the appropriate coder for the timer PCollection.
key_coder_id = input_pcoll.coder_id
- if (pipeline_context.components.coders[key_coder_id].spec.spec.urn
+ if (pipeline_context.components.coders[key_coder_id].spec.urn
== common_urns.coders.KV.urn):
key_coder_id = pipeline_context.components.coders[
key_coder_id].component_coder_ids[0]
key_timer_coder_id = pipeline_context.add_or_get_coder_id(
beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.KV.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.KV.urn),
component_coder_ids=[key_coder_id, spec.timer_coder_id]))
# Inject the read and write pcollections.
timer_read_pcoll = unique_name(
@@ -1263,15 +1252,14 @@
"""
def windowed_coder_id(coder_id, window_coder_id):
proto = beam_runner_api_pb2.Coder(
- spec=beam_runner_api_pb2.SdkFunctionSpec(
- spec=beam_runner_api_pb2.FunctionSpec(
- urn=common_urns.coders.WINDOWED_VALUE.urn)),
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=common_urns.coders.WINDOWED_VALUE.urn),
component_coder_ids=[coder_id, window_coder_id])
return pipeline_context.add_or_get_coder_id(
proto, coder_id + '_windowed')
for pcoll in pipeline_context.components.pcollections.values():
- if (pipeline_context.components.coders[pcoll.coder_id].spec.spec.urn
+ if (pipeline_context.components.coders[pcoll.coder_id].spec.urn
!= common_urns.coders.WINDOWED_VALUE.urn):
new_coder_id = windowed_coder_id(
pcoll.coder_id,
@@ -1324,4 +1312,5 @@
def split_buffer_id(buffer_id):
+ """A buffer id is "kind:pcollection_id". Split into (kind, pcoll_id). """
return buffer_id.decode('utf-8').split(':', 1)
diff --git a/sdks/python/apache_beam/runners/portability/java_reference_runner_test.py b/sdks/python/apache_beam/runners/portability/java_reference_runner_test.py
deleted file mode 100644
index d6253f6..0000000
--- a/sdks/python/apache_beam/runners/portability/java_reference_runner_test.py
+++ /dev/null
@@ -1,166 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is an entry point for running validatesRunner tests with the Python
-# SDK and the Java Reference Runner. Executing this file starts up an instance
-# of the Java Reference Runner's job server before executing tests and teardown
-# the job server afterwards.
-from __future__ import absolute_import
-
-import argparse
-import logging
-import sys
-import unittest
-
-import apache_beam as beam
-from apache_beam.options.pipeline_options import DebugOptions
-from apache_beam.options.pipeline_options import PortableOptions
-from apache_beam.runners.portability import portable_runner
-from apache_beam.runners.portability import portable_runner_test
-from apache_beam.testing.util import assert_that
-from apache_beam.testing.util import equal_to
-
-if __name__ == '__main__':
- # Run as
- #
- # python -m apache_beam.runners.portability.java_reference_runner_test \
- # --job_server_jar=/path/to/job_server.jar \
- # --environment_type=docker \
- # [Test.test_method, ...]
-
- parser = argparse.ArgumentParser(add_help=True)
- parser.add_argument('--job_server_jar',
- help='Job server jar to submit jobs.')
- parser.add_argument('--environment_type', default='docker',
- help='Environment type. docker or process')
- parser.add_argument('--environment_config', help='Environment config.')
-
- known_args, args = parser.parse_known_args(sys.argv)
- sys.argv = args
-
- job_server_jar = known_args.job_server_jar
- environment_type = known_args.environment_type.lower()
- environment_config = (
- known_args.environment_config if known_args.environment_config else None)
-
- # This is defined here to only be run when we invoke this file explicitly.
- class JavaReferenceRunnerTest(portable_runner_test.PortableRunnerTest):
- _use_grpc = True
- _use_subprocesses = True
-
- @classmethod
- def _subprocess_command(cls, port):
- return [
- 'java',
- #'-Dorg.slf4j.simpleLogger.defaultLogLevel=info'
- '-jar', job_server_jar,
- '--port', str(port),
- ]
-
- @classmethod
- def get_runner(cls):
- return portable_runner.PortableRunner()
-
- def create_options(self):
- options = super(JavaReferenceRunnerTest, self).create_options()
- options.view_as(DebugOptions).experiments = ['beam_fn_api']
- options.view_as(PortableOptions).environment_type = (
- environment_type.upper())
- if environment_config:
- options.view_as(PortableOptions).environment_config = environment_config
- return options
-
- def test_assert_that(self):
- # We still want to make sure asserts fail, even if the message
- # isn't right (BEAM-6600).
- with self.assertRaises(Exception):
- with self.create_pipeline() as p:
- assert_that(p | beam.Create(['a', 'b']), equal_to(['a']))
-
- def test_pardo_side_inputs(self):
- # Skip until Reference Runner supports side unputs.
- raise unittest.SkipTest("BEAM-2928")
-
- def test_pardo_windowed_side_inputs(self):
- # Skip until Reference Runner supports side unputs.
- raise unittest.SkipTest("BEAM-2928")
-
- def test_flattened_side_input(self):
- # Skip until Reference Runner supports side unputs.
- raise unittest.SkipTest("BEAM-2928")
-
- def test_gbk_side_input(self):
- # Skip until Reference Runner supports side unputs.
- raise unittest.SkipTest("BEAM-2928")
-
- def test_multimap_side_input(self):
- # Skip until Reference Runner supports side unputs.
- raise unittest.SkipTest("BEAM-2928")
-
- def test_pardo_unfusable_side_inputs(self):
- # Skip until Reference Runner supports side unputs.
- raise unittest.SkipTest("BEAM-2928")
-
- def test_pardo_state_only(self):
- # Skip until Reference Runner supports state.
- raise unittest.SkipTest("BEAM-2917")
-
- def test_pardo_timers(self):
- # Skip until Reference Runner supports state.
- raise unittest.SkipTest("BEAM-2917")
-
- def test_pardo_state_timers(self):
- # Skip until Reference Runner supports state.
- raise unittest.SkipTest("BEAM-2917")
-
- def test_sdf(self):
- # Skip until Reference Runner supports SDF.
- raise unittest.SkipTest("BEAM-6651")
-
- # Can't read host files from within docker, read a "local" file there.
- def test_read(self):
- with self.create_pipeline() as p:
- lines = p | beam.io.ReadFromText('/etc/profile')
- assert_that(lines, lambda lines: len(lines) > 0)
-
- def test_large_elements(self):
- # Skip until Reference Runner supports large elements.
- raise unittest.SkipTest("BEAM-6622")
-
- def test_error_message_includes_stage(self):
- # Skip until Reference Runner provides message support.
- raise unittest.SkipTest("BEAM-6600")
-
- def test_error_traceback_includes_user_code(self):
- # Skip until Reference Runner provides message support.
- raise unittest.SkipTest("BEAM-6600")
-
- def test_metrics(self):
- # Skip until Reference Runner provides metrics support.
- raise unittest.SkipTest("BEAM-5452")
-
- def test_non_user_metrics(self):
- # Skip until Reference Runner provides metrics support.
- raise unittest.SkipTest("BEAM-5452")
-
- def test_progress_metrics(self):
- # Skip until Reference Runner provides metrics support.
- raise unittest.SkipTest("BEAM-5452")
-
- # Run the tests.
- logging.getLogger().setLevel(logging.INFO)
- unittest.main()
diff --git a/sdks/python/apache_beam/runners/portability/job_server.py b/sdks/python/apache_beam/runners/portability/job_server.py
index d8dd8a4..7cf8d43 100644
--- a/sdks/python/apache_beam/runners/portability/job_server.py
+++ b/sdks/python/apache_beam/runners/portability/job_server.py
@@ -18,20 +18,139 @@
from __future__ import absolute_import
import atexit
-import logging
import os
+import shutil
import signal
-import socket
+import subprocess
import sys
-import time
-from subprocess import Popen
-from subprocess import check_output
-from threading import Lock
+import tempfile
+import threading
+
+import grpc
+
+from apache_beam.portability.api import beam_job_api_pb2_grpc
+from apache_beam.runners.portability import local_job_service
+from apache_beam.utils import subprocess_server
+from apache_beam.version import __version__ as beam_version
-class DockerizedJobServer(object):
+class JobServer(object):
+ def start(self):
+ """Starts this JobServer, returning a grpc service to which to submit jobs.
+ """
+ raise NotImplementedError(type(self))
+
+ def stop(self):
+ """Stops this job server."""
+ raise NotImplementedError(type(self))
+
+
+class ExternalJobServer(JobServer):
+ def __init__(self, endpoint, timeout=None):
+ self._endpoint = endpoint
+ self._timeout = timeout
+
+ def start(self):
+ channel = grpc.insecure_channel(self._endpoint)
+ grpc.channel_ready_future(channel).result(timeout=self._timeout)
+ return beam_job_api_pb2_grpc.JobServiceStub(channel)
+
+ def stop(self):
+ pass
+
+
+class EmbeddedJobServer(JobServer):
+ def start(self):
+ return local_job_service.LocalJobServicer()
+
+ def stop(self):
+ pass
+
+
+class StopOnExitJobServer(JobServer):
+ """Wraps a JobServer such that its stop will automatically be called on exit.
"""
- Spins up the JobServer in a docker container for local execution
+ def __init__(self, job_server):
+ self._lock = threading.Lock()
+ self._job_server = job_server
+ self._started = False
+
+ def start(self):
+ with self._lock:
+ if not self._started:
+ self._endpoint = self._job_server.start()
+ self._started = True
+ atexit.register(self.stop)
+ signal.signal(signal.SIGINT, self.stop)
+ return self._endpoint
+
+ def stop(self):
+ with self._lock:
+ if self._started:
+ self._job_server.stop()
+ self._started = False
+
+
+class SubprocessJobServer(JobServer):
+ """An abstract base class for JobServers run as an external process."""
+ def __init__(self):
+ self._local_temp_root = None
+ self._server = None
+
+ def subprocess_cmd_and_endpoint(self):
+ raise NotImplementedError(type(self))
+
+ def start(self):
+ if self._server is None:
+ self._local_temp_root = tempfile.mkdtemp(prefix='beam-temp')
+ cmd, endpoint = self.subprocess_cmd_and_endpoint()
+ port = int(endpoint.split(':')[-1])
+ self._server = subprocess_server.SubprocessServer(
+ beam_job_api_pb2_grpc.JobServiceStub, cmd, port=port)
+ return self._server.start()
+
+ def stop(self):
+ if self._local_temp_root:
+ shutil.rmtree(self._local_temp_root)
+ self._local_temp_root = None
+ return self._server.stop()
+
+ def local_temp_dir(self, **kwargs):
+ return tempfile.mkdtemp(dir=self._local_temp_root, **kwargs)
+
+
+class JavaJarJobServer(SubprocessJobServer):
+
+ MAVEN_REPOSITORY = 'https://repo.maven.apache.org/maven2/org/apache/beam'
+ JAR_CACHE = os.path.expanduser("~/.apache_beam/cache")
+
+ def java_arguments(self, job_port, artifacts_dir):
+ raise NotImplementedError(type(self))
+
+ def path_to_jar(self):
+ raise NotImplementedError(type(self))
+
+ @staticmethod
+ def path_to_beam_jar(gradle_target):
+ return subprocess_server.JavaJarServer.path_to_beam_jar(gradle_target)
+
+ @staticmethod
+ def local_jar(url):
+ return subprocess_server.JavaJarServer.local_jar(url)
+
+ def subprocess_cmd_and_endpoint(self):
+ jar_path = self.local_jar(self.path_to_jar())
+ artifacts_dir = self.local_temp_dir(prefix='artifacts')
+ job_port, = subprocess_server.pick_port(None)
+ return (
+ ['java', '-jar', jar_path] + list(
+ self.java_arguments(job_port, artifacts_dir)),
+ 'localhost:%s' % job_port)
+
+
+class DockerizedJobServer(SubprocessJobServer):
+ """
+ Spins up the JobServer in a docker container for local execution.
"""
def __init__(self, job_host="localhost",
@@ -40,30 +159,29 @@
expansion_port=None,
harness_port_range=(8100, 8200),
max_connection_retries=5):
+ super(DockerizedJobServer, self).__init__()
self.job_host = job_host
self.job_port = job_port
self.expansion_port = expansion_port
self.artifact_port = artifact_port
self.harness_port_range = harness_port_range
self.max_connection_retries = max_connection_retries
- self.docker_process = None
- self.process_lock = Lock()
- def start(self):
+ def subprocess_cmd_and_endpoint(self):
# TODO This is hardcoded to Flink at the moment but should be changed
job_server_image_name = os.environ['USER'] + \
"-docker-apache.bintray.io/beam/flink-job-server:latest"
- docker_path = check_output(['which', 'docker']).strip()
+ docker_path = subprocess.check_output(
+ ['which', 'docker']).strip().decode('utf-8')
cmd = ["docker", "run",
# We mount the docker binary and socket to be able to spin up
# "sibling" containers for the SDK harness.
"-v", ':'.join([docker_path, "/bin/docker"]),
"-v", "/var/run/docker.sock:/var/run/docker.sock"]
- self.job_port, self.artifact_port, self.expansion_port = \
- DockerizedJobServer._pick_port(self.job_port,
- self.artifact_port,
- self.expansion_port)
+ self.job_port, self.artifact_port, self.expansion_port = (
+ subprocess_server.pick_port(
+ self.job_port, self.artifact_port, self.expansion_port))
args = ['--job-host', self.job_host,
'--job-port', str(self.job_port),
@@ -87,53 +205,5 @@
cmd.append("--network=host")
cmd.append(job_server_image_name)
- cmd += args
- logging.debug("Starting container with %s", cmd)
- try:
- self.docker_process = Popen(cmd)
- atexit.register(self.stop)
- signal.signal(signal.SIGINT, self.stop)
- except: # pylint:disable=bare-except
- logging.exception("Error bringing up container")
- self.stop()
-
- return "{}:{}".format(self.job_host, self.job_port)
-
- def stop(self):
- with self.process_lock:
- if not self.docker_process:
- return
- num_retries = 0
- while self.docker_process.poll() is None and \
- num_retries < self.max_connection_retries:
- logging.debug("Sending SIGINT to job_server container")
- self.docker_process.send_signal(signal.SIGINT)
- num_retries += 1
- time.sleep(1)
- if self.docker_process.poll is None:
- self.docker_process.kill()
-
- @staticmethod
- def _pick_port(*ports):
- """
- Returns a list of ports, same length as input ports list, but replaces
- all None or 0 ports with a random free port.
- """
- sockets = []
-
- def find_free_port(port):
- if port:
- return port
- else:
- s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
- sockets.append(s)
- s.bind(('localhost', 0))
- _, free_port = s.getsockname()
- return free_port
-
- ports = list(map(find_free_port, ports))
- # Close sockets only now to avoid the same port to be chosen twice
- for s in sockets:
- s.close()
- return ports
+ return cmd + args, '%s:%s' % (self.job_host, self.job_port)
diff --git a/sdks/python/apache_beam/runners/portability/local_job_service.py b/sdks/python/apache_beam/runners/portability/local_job_service.py
index 91ceff7..421c098 100644
--- a/sdks/python/apache_beam/runners/portability/local_job_service.py
+++ b/sdks/python/apache_beam/runners/portability/local_job_service.py
@@ -89,7 +89,7 @@
if os.path.exists(self._staging_dir) and self._cleanup_staging_dir:
shutil.rmtree(self._staging_dir, ignore_errors=True)
- def Prepare(self, request, context=None):
+ def Prepare(self, request, context=None, timeout=None):
# For now, just use the job name as the job id.
logging.debug('Got Prepare request.')
preparation_id = '%s-%s' % (request.job_name, uuid.uuid4())
@@ -121,22 +121,30 @@
artifact_staging_endpoint=self._artifact_staging_endpoint,
staging_session_token=preparation_id)
- def Run(self, request, context=None):
+ def Run(self, request, context=None, timeout=None):
job_id = request.preparation_id
logging.info("Runing job '%s'", job_id)
self._jobs[job_id].start()
return beam_job_api_pb2.RunJobResponse(job_id=job_id)
+ def GetJobs(self, request, context=None, timeout=None):
+ return beam_job_api_pb2.GetJobsResponse(
+ [job.to_runner_api(context) for job in self._jobs.values()])
+
def GetState(self, request, context=None):
return beam_job_api_pb2.GetJobStateResponse(
state=self._jobs[request.job_id].state)
- def Cancel(self, request, context=None):
+ def GetPipeline(self, request, context=None, timeout=None):
+ return beam_job_api_pb2.GetJobPipelineResponse(
+ pipeline=self._jobs[request.job_id]._pipeline_proto)
+
+ def Cancel(self, request, context=None, timeout=None):
self._jobs[request.job_id].cancel()
return beam_job_api_pb2.CancelJobRequest(
state=self._jobs[request.job_id].state)
- def GetStateStream(self, request, context=None):
+ def GetStateStream(self, request, context=None, timeout=None):
"""Yields state transitions since the stream started.
"""
if request.job_id not in self._jobs:
@@ -146,7 +154,7 @@
for state in job.get_state_stream():
yield beam_job_api_pb2.GetJobStateResponse(state=state)
- def GetMessageStream(self, request, context=None):
+ def GetMessageStream(self, request, context=None, timeout=None):
"""Yields messages since the stream started.
"""
if request.job_id not in self._jobs:
@@ -161,7 +169,7 @@
resp = beam_job_api_pb2.JobMessagesResponse(message_response=msg)
yield resp
- def DescribePipelineOptions(self, request, context=None):
+ def DescribePipelineOptions(self, request, context=None, timeout=None):
return beam_job_api_pb2.DescribePipelineOptionsResponse()
@@ -169,9 +177,10 @@
"""Manages a SDK worker implemented as a subprocess communicating over grpc.
"""
- def __init__(self, worker_command_line, control_address):
+ def __init__(self, worker_command_line, control_address, worker_id=None):
self._worker_command_line = worker_command_line
self._control_address = control_address
+ self._worker_id = worker_id
def run(self):
logging_server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
@@ -186,13 +195,20 @@
control_descriptor = text_format.MessageToString(
endpoints_pb2.ApiServiceDescriptor(url=self._control_address))
- p = subprocess.Popen(
- self._worker_command_line,
- shell=True,
- env=dict(
- os.environ,
- CONTROL_API_SERVICE_DESCRIPTOR=control_descriptor,
- LOGGING_API_SERVICE_DESCRIPTOR=logging_descriptor))
+ env_dict = dict(
+ os.environ,
+ CONTROL_API_SERVICE_DESCRIPTOR=control_descriptor,
+ LOGGING_API_SERVICE_DESCRIPTOR=logging_descriptor
+ )
+ # only add worker_id when it is set.
+ if self._worker_id:
+ env_dict['WORKER_ID'] = self._worker_id
+
+ with fn_api_runner.SUBPROCESS_LOCK:
+ p = subprocess.Popen(
+ self._worker_command_line,
+ shell=True,
+ env=env_dict)
try:
p.wait()
if p.returncode:
@@ -283,6 +299,13 @@
if isinstance(msg, int):
current_state = msg
+ def to_runner_api(self, context):
+ return beam_job_api_pb2.JobInfo(
+ job_id=self._job_id,
+ job_name=self._provision_info.job_name,
+ pipeline_options=self._pipeline_options,
+ state=self.state)
+
class BeamFnLoggingServicer(beam_fn_api_pb2_grpc.BeamFnLoggingServicer):
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner.py b/sdks/python/apache_beam/runners/portability/portable_runner.py
index 4759748..16c6eba 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner.py
@@ -17,39 +17,33 @@
from __future__ import absolute_import
-import atexit
import functools
import itertools
import json
import logging
-import os
-import subprocess
+import sys
import threading
import time
-from concurrent import futures
import grpc
+from apache_beam import version as beam_version
from apache_beam import metrics
from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.options.pipeline_options import PortableOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.portability import common_urns
-from apache_beam.portability.api import beam_fn_api_pb2
-from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.portability.api import beam_job_api_pb2
-from apache_beam.portability.api import beam_job_api_pb2_grpc
from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.portability.api import endpoints_pb2
from apache_beam.runners import runner
from apache_beam.runners.job import utils as job_utils
from apache_beam.runners.portability import fn_api_runner_transforms
-from apache_beam.runners.portability import local_job_service
+from apache_beam.runners.portability import job_server
from apache_beam.runners.portability import portable_stager
-from apache_beam.runners.portability.job_server import DockerizedJobServer
-from apache_beam.runners.worker import sdk_worker
from apache_beam.runners.worker import sdk_worker_main
+from apache_beam.runners.worker import worker_pool_main
__all__ = ['PortableRunner']
@@ -80,17 +74,23 @@
running and managing the job lies with the job service used.
"""
def __init__(self):
- self._job_endpoint = None
+ self._dockerized_job_server = None
@staticmethod
def default_docker_image():
- if 'USER' in os.environ:
- # Perhaps also test if this was built?
- logging.info('Using latest locally built Python SDK docker image.')
- return os.environ['USER'] + '-docker-apache.bintray.io/beam/python:latest'
- else:
- logging.warning('Could not find a Python SDK docker image.')
- return 'unknown'
+ sdk_version = beam_version.__version__
+ version_suffix = '.'.join([str(i) for i in sys.version_info[0:2]])
+ logging.warning('Make sure that locally built Python SDK docker image '
+ 'has Python %d.%d interpreter.' % (
+ sys.version_info[0], sys.version_info[1]))
+
+ image = ('apachebeam/python{version_suffix}_sdk:{tag}'.format(
+ version_suffix=version_suffix, tag=sdk_version))
+ logging.info(
+ 'Using Python SDK docker image: %s. If the image is not '
+ 'available at local, we will try to pull from hub.docker.com'
+ % (image))
+ return image
@staticmethod
def _create_environment(options):
@@ -129,11 +129,25 @@
env=(config.get('env') or '')
).SerializeToString())
elif environment_urn == common_urns.environments.EXTERNAL.urn:
+ def looks_like_json(environment_config):
+ import re
+ return re.match(r'\s*\{.*\}\s*$', environment_config)
+
+ if looks_like_json(portable_options.environment_config):
+ config = json.loads(portable_options.environment_config)
+ url = config.get('url')
+ if not url:
+ raise ValueError('External environment endpoint must be set.')
+ params = config.get('params')
+ else:
+ url = portable_options.environment_config
+ params = None
+
return beam_runner_api_pb2.Environment(
urn=common_urns.environments.EXTERNAL.urn,
payload=beam_runner_api_pb2.ExternalPayload(
- endpoint=endpoints_pb2.ApiServiceDescriptor(
- url=portable_options.environment_config)
+ endpoint=endpoints_pb2.ApiServiceDescriptor(url=url),
+ params=params
).SerializeToString())
else:
return beam_runner_api_pb2.Environment(
@@ -141,31 +155,34 @@
payload=(portable_options.environment_config.encode('ascii')
if portable_options.environment_config else None))
- def init_dockerized_job_server(self):
+ def default_job_server(self, portable_options):
# TODO Provide a way to specify a container Docker URL
# https://issues.apache.org/jira/browse/BEAM-6328
- docker = DockerizedJobServer()
- self._job_endpoint = docker.start()
+ if not self._dockerized_job_server:
+ self._dockerized_job_server = job_server.StopOnExitJobServer(
+ job_server.DockerizedJobServer())
+ return self._dockerized_job_server
+
+ def create_job_service(self, options):
+ job_endpoint = options.view_as(PortableOptions).job_endpoint
+ if job_endpoint:
+ if job_endpoint == 'embed':
+ server = job_server.EmbeddedJobServer()
+ else:
+ job_server_timeout = options.view_as(PortableOptions).job_server_timeout
+ server = job_server.ExternalJobServer(job_endpoint, job_server_timeout)
+ else:
+ server = self.default_job_server(options)
+ return server.start()
def run_pipeline(self, pipeline, options):
portable_options = options.view_as(PortableOptions)
- job_endpoint = portable_options.job_endpoint
# TODO: https://issues.apache.org/jira/browse/BEAM-5525
# portable runner specific default
if options.view_as(SetupOptions).sdk_location == 'default':
options.view_as(SetupOptions).sdk_location = 'container'
- if not job_endpoint:
- if not self._job_endpoint:
- self.init_dockerized_job_server()
- job_endpoint = self._job_endpoint
- job_service = None
- elif job_endpoint == 'embed':
- job_service = local_job_service.LocalJobServicer()
- else:
- job_service = None
-
# This is needed as we start a worker server if one is requested
# but none is provided.
if portable_options.environment_type == 'LOOPBACK':
@@ -173,10 +190,10 @@
DebugOptions).lookup_experiment(
'use_loopback_process_worker', False)
portable_options.environment_config, server = (
- BeamFnExternalWorkerPoolServicer.start(
+ worker_pool_main.BeamFnExternalWorkerPoolServicer.start(
sdk_worker_main._get_worker_count(options),
+ state_cache_size=sdk_worker_main._get_state_cache_size(options),
use_process=use_loopback_process_worker))
- globals()['x'] = server
cleanup_callbacks = [functools.partial(server.stop, 1)]
else:
cleanup_callbacks = []
@@ -239,12 +256,7 @@
known_runner_urns=flink_known_urns,
partial=True)
- if not job_service:
- channel = grpc.insecure_channel(job_endpoint)
- grpc.channel_ready_future(channel).result()
- job_service = beam_job_api_pb2_grpc.JobServiceStub(channel)
- else:
- channel = None
+ job_service = self.create_job_service(options)
# fetch runner options from job service
# retries in case the channel is not ready
@@ -254,10 +266,12 @@
try:
# This reports channel is READY but connections may fail
# Seems to be only an issue on Mac with port forwardings
- if channel:
- grpc.channel_ready_future(channel).result()
return job_service.DescribePipelineOptions(
- beam_job_api_pb2.DescribePipelineOptionsRequest())
+ beam_job_api_pb2.DescribePipelineOptionsRequest(),
+ timeout=portable_options.job_server_timeout)
+ except grpc.FutureTimeoutError:
+ # no retry for timeout errors
+ raise
except grpc._channel._Rendezvous as e:
num_retries += 1
if num_retries > max_retries:
@@ -297,7 +311,8 @@
prepare_response = job_service.Prepare(
beam_job_api_pb2.PrepareJobRequest(
job_name='job', pipeline=proto_pipeline,
- pipeline_options=job_utils.dict_to_struct(p_options)))
+ pipeline_options=job_utils.dict_to_struct(p_options)),
+ timeout=portable_options.job_server_timeout)
if prepare_response.artifact_staging_endpoint.url:
stager = portable_stager.PortableStager(
grpc.insecure_channel(prepare_response.artifact_staging_endpoint.url),
@@ -311,7 +326,8 @@
try:
state_stream = job_service.GetStateStream(
beam_job_api_pb2.GetJobStateRequest(
- job_id=prepare_response.preparation_id))
+ job_id=prepare_response.preparation_id),
+ timeout=portable_options.job_server_timeout)
# If there's an error, we don't always get it until we try to read.
# Fortunately, there's always an immediate current state published.
state_stream = itertools.chain(
@@ -319,12 +335,15 @@
state_stream)
message_stream = job_service.GetMessageStream(
beam_job_api_pb2.JobMessagesRequest(
- job_id=prepare_response.preparation_id))
+ job_id=prepare_response.preparation_id),
+ timeout=portable_options.job_server_timeout)
except Exception:
# TODO(BEAM-6442): Unify preparation_id and job_id for all runners.
state_stream = message_stream = None
- # Run the job and wait for a result.
+ # Run the job and wait for a result, we don't set a timeout here because
+ # it may take a long time for a job to complete and streaming
+ # jobs currently never return a response.
run_response = job_service.Run(
beam_job_api_pb2.RunJobRequest(
preparation_id=prepare_response.preparation_id,
@@ -343,7 +362,9 @@
class PortableMetrics(metrics.metric.MetricResults):
- def __init__(self):
+ def __init__(self, job_metrics_response):
+ # TODO(lgajowy): Convert portable metrics to MetricResults
+ # and allow querying them (BEAM-4775)
pass
def query(self, filter=None):
@@ -363,6 +384,7 @@
self._message_stream = message_stream
self._state_stream = state_stream
self._cleanup_callbacks = cleanup_callbacks
+ self._metrics = None
def cancel(self):
try:
@@ -388,7 +410,13 @@
return beam_job_api_pb2.JobState.Enum.Value(pipeline_state)
def metrics(self):
- return PortableMetrics()
+ if not self._metrics:
+
+ job_metrics_response = self._job_service.GetJobMetrics(
+ beam_job_api_pb2.GetJobMetricsRequest(job_id=self._job_id))
+
+ self._metrics = PortableMetrics(job_metrics_response)
+ return self._metrics
def _last_error_message(self):
# Filter only messages with the "message_response" and error messages.
@@ -448,51 +476,3 @@
self._cleanup_callbacks = ()
if has_exception:
raise
-
-
-class BeamFnExternalWorkerPoolServicer(
- beam_fn_api_pb2_grpc.BeamFnExternalWorkerPoolServicer):
-
- def __init__(self, worker_threads, use_process=False):
- self._worker_threads = worker_threads
- self._use_process = use_process
-
- @classmethod
- def start(cls, worker_threads=1, use_process=False):
- worker_server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
- worker_address = 'localhost:%s' % worker_server.add_insecure_port('[::]:0')
- beam_fn_api_pb2_grpc.add_BeamFnExternalWorkerPoolServicer_to_server(
- cls(worker_threads, use_process=use_process), worker_server)
- worker_server.start()
- return worker_address, worker_server
-
- def NotifyRunnerAvailable(self, start_worker_request, context):
- try:
- if self._use_process:
- command = ['python', '-c',
- 'from apache_beam.runners.worker.sdk_worker '
- 'import SdkHarness; '
- 'SdkHarness("%s",worker_count=%d,worker_id="%s").run()' % (
- start_worker_request.control_endpoint.url,
- self._worker_threads,
- start_worker_request.worker_id)]
- logging.warn("Starting worker with command %s" % (command))
- worker_process = subprocess.Popen(command, stdout=subprocess.PIPE)
-
- # Register to kill the subprocess on exit.
- atexit.register(worker_process.kill)
- else:
- worker = sdk_worker.SdkHarness(
- start_worker_request.control_endpoint.url,
- worker_count=self._worker_threads,
- worker_id=start_worker_request.worker_id)
- worker_thread = threading.Thread(
- name='run_worker_%s' % start_worker_request.worker_id,
- target=worker.run)
- worker_thread.daemon = True
- worker_thread.start()
-
- return beam_fn_api_pb2.NotifyRunnerAvailableResponse()
- except Exception as exn:
- return beam_fn_api_pb2.NotifyRunnerAvailableResponse(
- error=str(exn))
diff --git a/sdks/python/apache_beam/runners/portability/portable_runner_test.py b/sdks/python/apache_beam/runners/portability/portable_runner_test.py
index 977bbcd..80afea7 100644
--- a/sdks/python/apache_beam/runners/portability/portable_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/portable_runner_test.py
@@ -33,6 +33,7 @@
import apache_beam as beam
from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import DirectOptions
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import PortableOptions
from apache_beam.portability import common_urns
@@ -40,10 +41,12 @@
from apache_beam.portability.api import beam_job_api_pb2
from apache_beam.portability.api import beam_job_api_pb2_grpc
from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability.api import endpoints_pb2
from apache_beam.runners.portability import fn_api_runner_test
from apache_beam.runners.portability import portable_runner
from apache_beam.runners.portability.local_job_service import LocalJobServicer
from apache_beam.runners.portability.portable_runner import PortableRunner
+from apache_beam.runners.worker import worker_pool_main
from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
@@ -177,6 +180,8 @@
# Override the default environment type for testing.
options.view_as(PortableOptions).environment_type = (
python_urns.EMBEDDED_PYTHON)
+ # Enable caching (disabled by default)
+ options.view_as(DebugOptions).add_experiment('state_cache_size=100')
return options
def create_pipeline(self):
@@ -191,6 +196,7 @@
def create_options(self):
options = super(PortableRunnerOptimized, self).create_options()
options.view_as(DebugOptions).add_experiment('pre_optimize=all')
+ options.view_as(DebugOptions).add_experiment('state_cache_size=100')
return options
@@ -199,7 +205,8 @@
@classmethod
def setUpClass(cls):
cls._worker_address, cls._worker_server = (
- portable_runner.BeamFnExternalWorkerPoolServicer.start())
+ worker_pool_main.BeamFnExternalWorkerPoolServicer.start(
+ state_cache_size=100))
@classmethod
def tearDownClass(cls):
@@ -212,7 +219,6 @@
return options
-@unittest.skip("BEAM-3040")
class PortableRunnerTestWithSubprocesses(PortableRunnerTest):
_use_subprocesses = True
@@ -222,7 +228,9 @@
python_urns.SUBPROCESS_SDK)
options.view_as(PortableOptions).environment_config = (
b'%s -m apache_beam.runners.worker.sdk_worker_main' %
- sys.executable.encode('ascii'))
+ sys.executable.encode('ascii')).decode('utf-8')
+ # Enable caching (disabled by default)
+ options.view_as(DebugOptions).add_experiment('state_cache_size=100')
return options
@classmethod
@@ -234,6 +242,17 @@
]
+class PortableRunnerTestWithSubprocessesAndMultiWorkers(
+ PortableRunnerTestWithSubprocesses):
+ _use_subprocesses = True
+
+ def create_options(self):
+ options = super(PortableRunnerTestWithSubprocessesAndMultiWorkers, self) \
+ .create_options()
+ options.view_as(DirectOptions).direct_num_workers = 2
+ return options
+
+
class PortableRunnerInternalTest(unittest.TestCase):
def test__create_default_environment(self):
docker_image = PortableRunner.default_docker_image()
@@ -282,6 +301,62 @@
command='run.sh',
).SerializeToString()))
+ def test__create_external_environment(self):
+ self.assertEqual(
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': 'localhost:50000',
+ })), beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.EXTERNAL.urn,
+ payload=beam_runner_api_pb2.ExternalPayload(
+ endpoint=endpoints_pb2.ApiServiceDescriptor(
+ url='localhost:50000')
+ ).SerializeToString()))
+ raw_config = ' {"url":"localhost:50000", "params":{"test":"test"}} '
+ for env_config in (raw_config, raw_config.lstrip(), raw_config.strip()):
+ self.assertEqual(
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': env_config,
+ })), beam_runner_api_pb2.Environment(
+ urn=common_urns.environments.EXTERNAL.urn,
+ payload=beam_runner_api_pb2.ExternalPayload(
+ endpoint=endpoints_pb2.ApiServiceDescriptor(
+ url='localhost:50000'),
+ params={"test": "test"}
+ ).SerializeToString()))
+ with self.assertRaises(ValueError):
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': '{invalid}',
+ }))
+ with self.assertRaises(ValueError) as ctx:
+ PortableRunner._create_environment(PipelineOptions.from_dictionary({
+ 'environment_type': "EXTERNAL",
+ 'environment_config': '{"params":{"test":"test"}}',
+ }))
+ self.assertIn(
+ 'External environment endpoint must be set.', ctx.exception.args)
+
+
+def hasDockerImage():
+ image = PortableRunner.default_docker_image()
+ try:
+ check_image = subprocess.check_output("docker images -q %s" % image,
+ shell=True)
+ return check_image != ''
+ except Exception:
+ return False
+
+
+@unittest.skipIf(not hasDockerImage(), "docker not installed or "
+ "no docker image")
+class PortableRunnerTestWithLocalDocker(PortableRunnerTest):
+ def create_options(self):
+ options = super(PortableRunnerTestWithLocalDocker, self).create_options()
+ options.view_as(PortableOptions).job_endpoint = 'embed'
+ return options
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/runners/portability/spark_runner_test.py b/sdks/python/apache_beam/runners/portability/spark_runner_test.py
index 478e609..131714b 100644
--- a/sdks/python/apache_beam/runners/portability/spark_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/spark_runner_test.py
@@ -97,22 +97,6 @@
# Skip until Spark runner supports metrics.
raise unittest.SkipTest("BEAM-7219")
- def test_pardo_state_only(self):
- # Skip until Spark runner supports user state.
- raise unittest.SkipTest("BEAM-7044")
-
- def test_pardo_timers(self):
- # Skip until Spark runner supports timers.
- raise unittest.SkipTest("BEAM-7221")
-
- def test_pardo_state_timers(self):
- # Skip until Spark runner supports user state and timers.
- raise unittest.SkipTest("BEAM-7044, BEAM-7221")
-
- def test_windowed_pardo_state_timers(self):
- # Skip until Spark runner supports user state and timers.
- raise unittest.SkipTest("BEAM-7044, BEAM-7221")
-
def test_sdf(self):
# Skip until Spark runner supports SDF.
raise unittest.SkipTest("BEAM-7222")
@@ -121,6 +105,10 @@
# Skip until Spark runner supports SDF.
raise unittest.SkipTest("BEAM-7222")
+ def test_sdf_synthetic_source(self):
+ # Skip until Spark runner supports SDF.
+ raise unittest.SkipTest("BEAM-7222")
+
def test_external_transforms(self):
# Skip until Spark runner supports external transforms.
raise unittest.SkipTest("BEAM-7232")
diff --git a/sdks/python/apache_beam/runners/portability/stager.py b/sdks/python/apache_beam/runners/portability/stager.py
index 0448a25..fa982ff 100644
--- a/sdks/python/apache_beam/runners/portability/stager.py
+++ b/sdks/python/apache_beam/runners/portability/stager.py
@@ -606,7 +606,7 @@
]
if fetch_binary:
- logging.info('Downloading binary distribtution of the SDK from PyPi')
+ logging.info('Downloading binary distribution of the SDK from PyPi')
# Get a wheel distribution for the SDK from PyPI.
cmd_args.extend([
'--only-binary', ':all:', '--python-version', language_version_tag,
@@ -622,7 +622,7 @@
platform_tag))
]
else:
- logging.info('Downloading source distribtution of the SDK from PyPi')
+ logging.info('Downloading source distribution of the SDK from PyPi')
cmd_args.extend(['--no-binary', ':all:'])
expected_files = [
os.path.join(temp_dir, '%s-%s.zip' % (package_name, version)),
diff --git a/sdks/python/apache_beam/runners/runner.py b/sdks/python/apache_beam/runners/runner.py
index 7184562..e83fe238 100644
--- a/sdks/python/apache_beam/runners/runner.py
+++ b/sdks/python/apache_beam/runners/runner.py
@@ -19,6 +19,7 @@
from __future__ import absolute_import
+import importlib
import logging
import os
import shelve
@@ -29,45 +30,26 @@
__all__ = ['PipelineRunner', 'PipelineState', 'PipelineResult']
-def _get_runner_map(runner_names, module_path):
- """Create a map of runner name in lower case to full import path to the
- runner class.
- """
- return {runner_name.lower(): module_path + runner_name
- for runner_name in runner_names}
-
-
-_DIRECT_RUNNER_PATH = 'apache_beam.runners.direct.direct_runner.'
-_DATAFLOW_RUNNER_PATH = (
- 'apache_beam.runners.dataflow.dataflow_runner.')
-_TEST_RUNNER_PATH = 'apache_beam.runners.test.'
-_PYTHON_RPC_DIRECT_RUNNER = (
- 'apache_beam.runners.experimental.python_rpc_direct.'
- 'python_rpc_direct_runner.')
-_PORTABLE_RUNNER_PATH = ('apache_beam.runners.portability.portable_runner.')
-
-_KNOWN_PYTHON_RPC_DIRECT_RUNNER = ('PythonRPCDirectRunner',)
-_KNOWN_DIRECT_RUNNERS = ('DirectRunner', 'BundleBasedDirectRunner',
- 'SwitchingDirectRunner')
-_KNOWN_DATAFLOW_RUNNERS = ('DataflowRunner',)
-_KNOWN_TEST_RUNNERS = ('TestDataflowRunner', 'TestDirectRunner')
-_KNOWN_PORTABLE_RUNNERS = ('PortableRunner',)
-
-_RUNNER_MAP = {}
-_RUNNER_MAP.update(_get_runner_map(_KNOWN_DIRECT_RUNNERS,
- _DIRECT_RUNNER_PATH))
-_RUNNER_MAP.update(_get_runner_map(_KNOWN_DATAFLOW_RUNNERS,
- _DATAFLOW_RUNNER_PATH))
-_RUNNER_MAP.update(_get_runner_map(_KNOWN_PYTHON_RPC_DIRECT_RUNNER,
- _PYTHON_RPC_DIRECT_RUNNER))
-_RUNNER_MAP.update(_get_runner_map(_KNOWN_TEST_RUNNERS,
- _TEST_RUNNER_PATH))
-_RUNNER_MAP.update(_get_runner_map(_KNOWN_PORTABLE_RUNNERS,
- _PORTABLE_RUNNER_PATH))
-
_ALL_KNOWN_RUNNERS = (
- _KNOWN_DIRECT_RUNNERS + _KNOWN_DATAFLOW_RUNNERS + _KNOWN_TEST_RUNNERS +
- _KNOWN_PORTABLE_RUNNERS)
+ 'apache_beam.runners.dataflow.dataflow_runner.DataflowRunner',
+ 'apache_beam.runners.direct.direct_runner.BundleBasedDirectRunner',
+ 'apache_beam.runners.direct.direct_runner.DirectRunner',
+ 'apache_beam.runners.direct.direct_runner.SwitchingDirectRunner',
+ 'apache_beam.runners.portability.flink_runner.FlinkRunner',
+ 'apache_beam.runners.portability.portable_runner.PortableRunner',
+ 'apache_beam.runners.test.TestDirectRunner',
+ 'apache_beam.runners.test.TestDataflowRunner',
+)
+
+_KNOWN_RUNNER_NAMES = [path.split('.')[-1] for path in _ALL_KNOWN_RUNNERS]
+
+_RUNNER_MAP = {path.split('.')[-1].lower(): path
+ for path in _ALL_KNOWN_RUNNERS}
+
+# Allow this alias, but don't make public.
+_RUNNER_MAP['pythonrpcdirectrunner'] = (
+ 'apache_beam.runners.experimental'
+ '.python_rpc_direct.python_rpc_direct_runner.PythonRPCDirectRunner')
def create_runner(runner_name):
@@ -96,9 +78,9 @@
if '.' in runner_name:
module, runner = runner_name.rsplit('.', 1)
try:
- return getattr(__import__(module, {}, {}, [runner], 0), runner)()
+ return getattr(importlib.import_module(module), runner)()
except ImportError:
- if runner_name in _KNOWN_DATAFLOW_RUNNERS:
+ if 'dataflow' in runner_name.lower():
raise ImportError(
'Google Cloud Dataflow runner not available, '
'please install apache_beam[gcp]')
@@ -108,7 +90,7 @@
raise ValueError(
'Unexpected pipeline runner: %s. Valid values are %s '
'or the fully qualified name of a PipelineRunner subclass.' % (
- runner_name, ', '.join(_ALL_KNOWN_RUNNERS)))
+ runner_name, ', '.join(_KNOWN_RUNNER_NAMES)))
class PipelineRunner(object):
@@ -319,7 +301,7 @@
pipeline in. Currently, it represents the values of the dataflow
API JobState enum.
"""
- UNKNOWN = 'UNKNOWN' # not specified
+ UNKNOWN = 'UNKNOWN' # not specified by a runner, or unknown to a runner.
STARTING = 'STARTING' # not yet started
STOPPED = 'STOPPED' # paused or not yet started
RUNNING = 'RUNNING' # currently running
@@ -332,6 +314,8 @@
PENDING = 'PENDING' # the job has been created but is not yet running.
CANCELLING = 'CANCELLING' # job has been explicitly cancelled and is
# in the process of stopping
+ UNRECOGNIZED = 'UNRECOGNIZED' # the job state reported by a runner cannot be
+ # interpreted by the SDK.
@classmethod
def is_terminal(cls, state):
diff --git a/sdks/python/apache_beam/runners/sdf_common.py b/sdks/python/apache_beam/runners/sdf_common.py
index e057328..072d3dc 100644
--- a/sdks/python/apache_beam/runners/sdf_common.py
+++ b/sdks/python/apache_beam/runners/sdf_common.py
@@ -167,4 +167,4 @@
self.ptransform_side_inputs = ptransform_side_inputs
def expand(self, pcoll):
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
diff --git a/sdks/python/apache_beam/runners/worker/bundle_processor.py b/sdks/python/apache_beam/runners/worker/bundle_processor.py
index c35b646..30c6f3d 100644
--- a/sdks/python/apache_beam/runners/worker/bundle_processor.py
+++ b/sdks/python/apache_beam/runners/worker/bundle_processor.py
@@ -32,7 +32,7 @@
from builtins import object
from future.utils import itervalues
-from google import protobuf
+from google.protobuf import timestamp_pb2
import apache_beam as beam
from apache_beam import coders
@@ -60,12 +60,12 @@
# This module is experimental. No backwards-compatibility guarantees.
-DATA_INPUT_URN = 'urn:org.apache.beam:source:runner:0.1'
-DATA_OUTPUT_URN = 'urn:org.apache.beam:sink:runner:0.1'
-IDENTITY_DOFN_URN = 'urn:org.apache.beam:dofn:identity:0.1'
+DATA_INPUT_URN = 'beam:source:runner:0.1'
+DATA_OUTPUT_URN = 'beam:sink:runner:0.1'
+IDENTITY_DOFN_URN = 'beam:dofn:identity:0.1'
# TODO(vikasrk): Fix this once runner sends appropriate common_urns.
-OLD_DATAFLOW_RUNNER_HARNESS_PARDO_URN = 'urn:beam:dofn:javasdk:0.1'
-OLD_DATAFLOW_RUNNER_HARNESS_READ_URN = 'urn:org.apache.beam:source:java:0.1'
+OLD_DATAFLOW_RUNNER_HARNESS_PARDO_URN = 'beam:dofn:javasdk:0.1'
+OLD_DATAFLOW_RUNNER_HARNESS_READ_URN = 'beam:source:java:0.1'
URNS_NEEDING_PCOLLECTIONS = set([monitoring_infos.ELEMENT_COUNT_URN,
monitoring_infos.SAMPLED_BYTE_SIZE_URN])
@@ -74,14 +74,14 @@
"""Common baseclass for runner harness IO operations."""
def __init__(self, name_context, step_name, consumers, counter_factory,
- state_sampler, windowed_coder, target, data_channel):
+ state_sampler, windowed_coder, transform_id, data_channel):
super(RunnerIOOperation, self).__init__(
name_context, None, counter_factory, state_sampler)
self.windowed_coder = windowed_coder
self.windowed_coder_impl = windowed_coder.get_impl()
- # target represents the consumer for the bytes in the data plane for a
+ # transform_id represents the consumer for the bytes in the data plane for a
# DataInputOperation or a producer of these bytes for a DataOutputOperation.
- self.target = target
+ self.transform_id = transform_id
self.data_channel = data_channel
for _, consumer_ops in consumers.items():
for consumer in consumer_ops:
@@ -109,10 +109,10 @@
"""A source-like operation that gathers input from the runner."""
def __init__(self, operation_name, step_name, consumers, counter_factory,
- state_sampler, windowed_coder, input_target, data_channel):
+ state_sampler, windowed_coder, transform_id, data_channel):
super(DataInputOperation, self).__init__(
operation_name, step_name, consumers, counter_factory, state_sampler,
- windowed_coder, target=input_target, data_channel=data_channel)
+ windowed_coder, transform_id=transform_id, data_channel=data_channel)
# We must do this manually as we don't have a spec or spec.output_coders.
self.receivers = [
operations.ConsumerSet.create(
@@ -184,31 +184,34 @@
self.stop = stop_index
return self.stop - 1, None, None, self.stop
+ def progress_metrics(self):
+ with self.splitting_lock:
+ metrics = super(DataInputOperation, self).progress_metrics()
+ current_element_progress = self.receivers[0].current_element_progress()
+ if current_element_progress:
+ metrics.active_elements.fraction_remaining = (
+ current_element_progress.fraction_remaining)
+ return metrics
+
def finish(self):
with self.splitting_lock:
self.started = False
class _StateBackedIterable(object):
- def __init__(self, state_handler, state_key, coder_or_impl):
+ def __init__(self, state_handler, state_key, coder_or_impl,
+ is_cached=False):
self._state_handler = state_handler
self._state_key = state_key
if isinstance(coder_or_impl, coders.Coder):
self._coder_impl = coder_or_impl.get_impl()
else:
self._coder_impl = coder_or_impl
+ self._is_cached = is_cached
def __iter__(self):
- data, continuation_token = self._state_handler.blocking_get(self._state_key)
- while True:
- input_stream = coder_impl.create_InputStream(data)
- while input_stream.size() > 0:
- yield self._coder_impl.decode_from_stream(input_stream, True)
- if not continuation_token:
- break
- else:
- data, continuation_token = self._state_handler.blocking_get(
- self._state_key, continuation_token)
+ return self._state_handler.blocking_get(
+ self._state_key, self._coder_impl, is_cached=self._is_cached)
def __reduce__(self):
return list, (list(self),)
@@ -234,7 +237,7 @@
if target_window not in self._cache:
state_key = beam_fn_api_pb2.StateKey(
multimap_side_input=beam_fn_api_pb2.StateKey.MultimapSideInput(
- ptransform_id=self._transform_id,
+ transform_id=self._transform_id,
side_input_id=self._tag,
window=self._target_window_coder.encode(target_window),
key=b''))
@@ -283,7 +286,8 @@
self._cache = {}
-class CombiningValueRuntimeState(userstate.RuntimeState):
+class CombiningValueRuntimeState(userstate.CombiningValueRuntimeState):
+
def __init__(self, underlying_bag_state, combinefn):
self._combinefn = combinefn
self._underlying_bag_state = underlying_bag_state
@@ -337,8 +341,8 @@
coder_impl.FastPrimitivesCoderImpl.register_iterable_like_type(_ConcatIterable)
-# TODO(BEAM-5428): Implement cross-bundle state caching.
-class SynchronousBagRuntimeState(userstate.RuntimeState):
+class SynchronousBagRuntimeState(userstate.BagRuntimeState):
+
def __init__(self, state_handler, state_key, value_coder):
self._state_handler = state_handler
self._state_key = state_key
@@ -349,7 +353,8 @@
def read(self):
return _ConcatIterable(
[] if self._cleared else _StateBackedIterable(
- self._state_handler, self._state_key, self._value_coder),
+ self._state_handler, self._state_key, self._value_coder,
+ is_cached=True),
self._added_elements)
def add(self, value):
@@ -360,14 +365,76 @@
self._added_elements = []
def _commit(self):
+ to_await = None
if self._cleared:
- self._state_handler.blocking_clear(self._state_key)
+ to_await = self._state_handler.clear(self._state_key, is_cached=True)
if self._added_elements:
- value_coder_impl = self._value_coder.get_impl()
- out = coder_impl.create_OutputStream()
- for element in self._added_elements:
- value_coder_impl.encode_to_stream(element, out, True)
- self._state_handler.blocking_append(self._state_key, out.get())
+ to_await = self._state_handler.extend(
+ self._state_key,
+ self._value_coder.get_impl(),
+ self._added_elements,
+ is_cached=True)
+ if to_await:
+ # To commit, we need to wait on the last state request future to complete.
+ to_await.get()
+
+
+class SynchronousSetRuntimeState(userstate.SetRuntimeState):
+
+ def __init__(self, state_handler, state_key, value_coder):
+ self._state_handler = state_handler
+ self._state_key = state_key
+ self._value_coder = value_coder
+ self._cleared = False
+ self._added_elements = set()
+
+ def _compact_data(self, rewrite=True):
+ accumulator = set(_ConcatIterable(
+ set() if self._cleared else _StateBackedIterable(
+ self._state_handler, self._state_key, self._value_coder,
+ is_cached=True),
+ self._added_elements))
+
+ if rewrite and accumulator:
+ self._state_handler.clear(self._state_key, is_cached=True)
+ self._state_handler.extend(
+ self._state_key,
+ self._value_coder.get_impl(),
+ accumulator,
+ is_cached=True)
+
+ # Since everthing is already committed so we can safely reinitialize
+ # added_elements here.
+ self._added_elements = set()
+
+ return accumulator
+
+ def read(self):
+ return self._compact_data(rewrite=False)
+
+ def add(self, value):
+ if self._cleared:
+ # This is a good time explicitly clear.
+ self._state_handler.clear(self._state_key, is_cached=True)
+ self._cleared = False
+
+ self._added_elements.add(value)
+ if random.random() > 0.5:
+ self._compact_data()
+
+ def clear(self):
+ self._cleared = True
+ self._added_elements = set()
+
+ def _commit(self):
+ if self._cleared:
+ self._state_handler.clear(self._state_key, is_cached=True).get()
+ if self._added_elements:
+ self._state_handler.extend(
+ self._state_key,
+ self._value_coder.get_impl(),
+ self._added_elements,
+ is_cached=True).get()
class OutputTimer(object):
@@ -436,15 +503,27 @@
self._state_handler,
state_key=beam_fn_api_pb2.StateKey(
bag_user_state=beam_fn_api_pb2.StateKey.BagUserState(
- ptransform_id=self._transform_id,
+ transform_id=self._transform_id,
user_state_id=state_spec.name,
window=self._window_coder.encode(window),
- key=self._key_coder.encode(key))),
+ # State keys are expected in nested encoding format
+ key=self._key_coder.encode_nested(key))),
value_coder=state_spec.coder)
if isinstance(state_spec, userstate.BagStateSpec):
return bag_state
else:
return CombiningValueRuntimeState(bag_state, state_spec.combine_fn)
+ elif isinstance(state_spec, userstate.SetStateSpec):
+ return SynchronousSetRuntimeState(
+ self._state_handler,
+ state_key=beam_fn_api_pb2.StateKey(
+ bag_user_state=beam_fn_api_pb2.StateKey.BagUserState(
+ transform_id=self._transform_id,
+ user_state_id=state_spec.name,
+ window=self._window_coder.encode(window),
+ # State keys are expected in nested encoding format
+ key=self._key_coder.encode_nested(key))),
+ value_coder=state_spec.coder)
else:
raise NotImplementedError(state_spec)
@@ -475,9 +554,7 @@
class BundleProcessor(object):
- """A class for processing bundles of elements.
-
- """
+ """ A class for processing bundles of elements. """
def __init__(
self, process_bundle_descriptor, state_handler, data_channel_factory):
@@ -558,7 +635,7 @@
# TODO(robertwb): Is there a better way to pass the instruction id to
# the operation?
op.set_output_stream(op.data_channel.output_stream(
- instruction_id, op.target))
+ instruction_id, op.transform_id))
elif isinstance(op, DataInputOperation):
# We must wait until we receive "end of stream" for each of these ops.
expected_inputs.append(op)
@@ -574,19 +651,16 @@
# Inject inputs from data plane.
data_channels = collections.defaultdict(list)
- input_op_by_target = {}
+ input_op_by_transform_id = {}
for input_op in expected_inputs:
- data_channels[input_op.data_channel].append(input_op.target)
- # ignores input name
- input_op_by_target[
- input_op.target.primitive_transform_reference] = input_op
+ data_channels[input_op.data_channel].append(input_op.transform_id)
+ input_op_by_transform_id[input_op.transform_id] = input_op
- for data_channel, expected_targets in data_channels.items():
+ for data_channel, expected_transforms in data_channels.items():
for data in data_channel.input_elements(
- instruction_id, expected_targets):
- input_op_by_target[
- data.target.primitive_transform_reference
- ].process_encoded(data.data)
+ instruction_id, expected_transforms):
+ input_op_by_transform_id[
+ data.transform_id].process_encoded(data.data)
# Finish all operations.
for op in self.ops.values():
@@ -617,7 +691,7 @@
for op in self.ops.values():
if isinstance(op, DataInputOperation):
desired_split = bundle_split_request.desired_splits.get(
- op.target.primitive_transform_reference)
+ op.transform_id)
if desired_split:
split = op.try_split(desired_split.fraction_of_remainder,
desired_split.estimated_input_elements)
@@ -633,26 +707,25 @@
self.delayed_bundle_application(*element_residual))
split_response.channel_splits.extend([
beam_fn_api_pb2.ProcessBundleSplitResponse.ChannelSplit(
- ptransform_id=op.target.primitive_transform_reference,
- input_id=op.target.name,
+ transform_id=op.transform_id,
last_primary_element=primary_end,
first_residual_element=residual_start)])
return split_response
def delayed_bundle_application(self, op, deferred_remainder):
- ptransform_id, main_input_tag, main_input_coder, outputs = op.input_info
+ transform_id, main_input_tag, main_input_coder, outputs = op.input_info
# TODO(SDF): For non-root nodes, need main_input_coder + residual_coder.
element_and_restriction, watermark = deferred_remainder
if watermark:
- proto_watermark = protobuf.Timestamp()
+ proto_watermark = timestamp_pb2.Timestamp()
proto_watermark.FromMicroseconds(watermark.micros)
output_watermarks = {output: proto_watermark for output in outputs}
else:
output_watermarks = None
return beam_fn_api_pb2.DelayedBundleApplication(
application=beam_fn_api_pb2.BundleApplication(
- ptransform_id=ptransform_id,
+ transform_id=transform_id,
input_id=main_input_tag,
output_watermarks=output_watermarks,
element=main_input_coder.get_impl().encode_nested(
@@ -786,7 +859,7 @@
def create_operation(self, transform_id, consumers):
transform_proto = self.descriptor.transforms[transform_id]
if not transform_proto.unique_name:
- logging.warn("No unique name set for transform %s" % transform_id)
+ logging.debug("No unique name set for transform %s" % transform_id)
transform_proto.unique_name = transform_id
creator, parameter_type = self._known_urns[transform_proto.spec.urn]
payload = proto_utils.parse_Bytes(
@@ -797,12 +870,12 @@
if coder_id not in self.descriptor.coders:
raise KeyError("No such coder: %s" % coder_id)
coder_proto = self.descriptor.coders[coder_id]
- if coder_proto.spec.spec.urn:
+ if coder_proto.spec.urn:
return self.context.coders.get_by_id(coder_id)
else:
# No URN, assume cloud object encoding json bytes.
return operation_specs.get_coder_from_spec(
- json.loads(coder_proto.spec.spec.payload.decode('utf-8')))
+ json.loads(coder_proto.spec.payload.decode('utf-8')))
def get_windowed_coder(self, pcoll_id):
coder = self.get_coder(self.descriptor.pcollections[pcoll_id].coder_id)
@@ -868,9 +941,6 @@
output_consumers[:] = [TimerConsumer(tag, do_op)]
break
- target = beam_fn_api_pb2.Target(
- primitive_transform_reference=transform_id,
- name=only_element(list(transform_proto.outputs.keys())))
if grpc_port.coder_id:
output_coder = factory.get_coder(grpc_port.coder_id)
else:
@@ -886,16 +956,13 @@
factory.counter_factory,
factory.state_sampler,
output_coder,
- input_target=target,
+ transform_id=transform_id,
data_channel=factory.data_channel_factory.create_data_channel(grpc_port))
@BeamTransformFactory.register_urn(
DATA_OUTPUT_URN, beam_fn_api_pb2.RemoteGrpcPort)
def create(factory, transform_id, transform_proto, grpc_port, consumers):
- target = beam_fn_api_pb2.Target(
- primitive_transform_reference=transform_id,
- name=only_element(list(transform_proto.inputs.keys())))
if grpc_port.coder_id:
output_coder = factory.get_coder(grpc_port.coder_id)
else:
@@ -911,7 +978,7 @@
factory.counter_factory,
factory.state_sampler,
output_coder,
- target=target,
+ transform_id=transform_id,
data_channel=factory.data_channel_factory.create_data_channel(grpc_port))
@@ -1054,7 +1121,8 @@
for tag, si in pardo_proto.side_inputs.items()]
tagged_side_inputs.sort(
key=lambda tag_si: int(re.match('side([0-9]+)(-.*)?$',
- tag_si[0]).group(1)))
+ tag_si[0],
+ re.DOTALL).group(1)))
side_input_maps = [
StateBackedSideInputMap(
factory.state_handler,
@@ -1190,43 +1258,6 @@
@BeamTransformFactory.register_urn(
- common_urns.combine_components.COMBINE_PGBKCV.urn,
- beam_runner_api_pb2.CombinePayload)
-def create(factory, transform_id, transform_proto, payload, consumers):
- # TODO: Combine side inputs.
- serialized_combine_fn = pickler.dumps(
- (beam.CombineFn.from_runner_api(payload.combine_fn, factory.context),
- [], {}))
- return factory.augment_oldstyle_op(
- operations.PGBKCVOperation(
- transform_proto.unique_name,
- operation_specs.WorkerPartialGroupByKey(
- serialized_combine_fn,
- None,
- [factory.get_only_output_coder(transform_proto)]),
- factory.counter_factory,
- factory.state_sampler),
- transform_proto.unique_name,
- consumers)
-
-
-@BeamTransformFactory.register_urn(
- common_urns.combine_components.COMBINE_MERGE_ACCUMULATORS.urn,
- beam_runner_api_pb2.CombinePayload)
-def create(factory, transform_id, transform_proto, payload, consumers):
- return _create_combine_phase_operation(
- factory, transform_proto, payload, consumers, 'merge')
-
-
-@BeamTransformFactory.register_urn(
- common_urns.combine_components.COMBINE_EXTRACT_OUTPUTS.urn,
- beam_runner_api_pb2.CombinePayload)
-def create(factory, transform_id, transform_proto, payload, consumers):
- return _create_combine_phase_operation(
- factory, transform_proto, payload, consumers, 'extract')
-
-
-@BeamTransformFactory.register_urn(
common_urns.combine_components.COMBINE_PER_KEY_PRECOMBINE.urn,
beam_runner_api_pb2.CombinePayload)
def create(factory, transform_id, transform_proto, payload, consumers):
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py b/sdks/python/apache_beam/runners/worker/data_plane.py
index 276ca19..8324e6b 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
-"""Implementation of DataChannels for communicating across the data plane."""
+"""Implementation of ``DataChannel``s to communicate across the data plane."""
from __future__ import absolute_import
from __future__ import division
@@ -74,12 +74,13 @@
Read from this channel with the input_elements method::
- for elements_data in data_channel.input_elements(instruction_id, targets):
+ for elements_data in data_channel.input_elements(
+ instruction_id, transform_ids):
[process elements_data]
Write to this channel using the output_stream method::
- out1 = data_channel.output_stream(instruction_id, target1)
+ out1 = data_channel.output_stream(instruction_id, transform_id)
out1.write(...)
out1.close()
@@ -90,27 +91,27 @@
@abc.abstractmethod
def input_elements(
- self, instruction_id, expected_targets, abort_callback=None):
+ self, instruction_id, expected_transforms, abort_callback=None):
"""Returns an iterable of all Element.Data bundles for instruction_id.
This iterable terminates only once the full set of data has been recieved
- for each of the expected targets. It may block waiting for more data.
+ for each of the expected transforms. It may block waiting for more data.
Args:
instruction_id: which instruction the results must belong to
- expected_targets: which targets to wait on for completion
+ expected_transforms: which transforms to wait on for completion
abort_callback: a callback to invoke if blocking returning whether
to abort before consuming all the data
"""
raise NotImplementedError(type(self))
@abc.abstractmethod
- def output_stream(self, instruction_id, target):
- """Returns an output stream writing elements to target.
+ def output_stream(self, instruction_id, transform_id):
+ """Returns an output stream writing elements to transform_id.
Args:
instruction_id: which instruction this stream belongs to
- target: the target of the returned stream
+ transform_id: the transform_id of the returned stream
"""
raise NotImplementedError(type(self))
@@ -140,23 +141,23 @@
def inverse(self):
return self._inverse
- def input_elements(self, instruction_id, unused_expected_targets=None,
+ def input_elements(self, instruction_id, unused_expected_transforms=None,
abort_callback=None):
other_inputs = []
for data in self._inputs:
- if data.instruction_reference == instruction_id:
+ if data.instruction_id == instruction_id:
if data.data:
yield data
else:
other_inputs.append(data)
self._inputs = other_inputs
- def output_stream(self, instruction_id, target):
+ def output_stream(self, instruction_id, transform_id):
def add_to_inverse_output(data):
self._inverse._inputs.append( # pylint: disable=protected-access
beam_fn_api_pb2.Elements.Data(
- instruction_reference=instruction_id,
- target=target,
+ instruction_id=instruction_id,
+ transform_id=transform_id,
data=data))
return ClosableOutputStream(
add_to_inverse_output, flush_callback=add_to_inverse_output)
@@ -193,7 +194,7 @@
with self._receive_lock:
self._received.pop(instruction_id)
- def input_elements(self, instruction_id, expected_targets,
+ def input_elements(self, instruction_id, expected_transforms,
abort_callback=None):
"""
Generator to retrieve elements for an instruction_id
@@ -201,13 +202,13 @@
Args:
instruction_id(str): instruction_id for which data is read
- expected_targets(collection): expected targets
+ expected_transforms(collection): expected transforms
"""
received = self._receiving_queue(instruction_id)
- done_targets = []
+ done_transforms = []
abort_callback = abort_callback or (lambda: False)
try:
- while len(done_targets) < len(expected_targets):
+ while len(done_transforms) < len(expected_transforms):
try:
data = received.get(timeout=1)
except queue.Empty:
@@ -219,23 +220,23 @@
t, v, tb = self._exc_info
raise_(t, v, tb)
else:
- if not data.data and data.target in expected_targets:
- done_targets.append(data.target)
+ if not data.data and data.transform_id in expected_transforms:
+ done_transforms.append(data.transform_id)
else:
- assert data.target not in done_targets
+ assert data.transform_id not in done_transforms
yield data
finally:
# Instruction_ids are not reusable so Clean queue once we are done with
# an instruction_id
self._clean_receiving_queue(instruction_id)
- def output_stream(self, instruction_id, target):
+ def output_stream(self, instruction_id, transform_id):
def add_to_send_queue(data):
if data:
self._to_send.put(
beam_fn_api_pb2.Elements.Data(
- instruction_reference=instruction_id,
- target=target,
+ instruction_id=instruction_id,
+ transform_id=transform_id,
data=data))
def close_callback(data):
@@ -243,8 +244,8 @@
# End of stream marker.
self._to_send.put(
beam_fn_api_pb2.Elements.Data(
- instruction_reference=instruction_id,
- target=target,
+ instruction_id=instruction_id,
+ transform_id=transform_id,
data=b''))
return ClosableOutputStream(
close_callback, flush_callback=add_to_send_queue)
@@ -270,17 +271,17 @@
try:
for elements in elements_iterator:
for data in elements.data:
- self._receiving_queue(data.instruction_reference).put(data)
+ self._receiving_queue(data.instruction_id).put(data)
except: # pylint: disable=bare-except
if not self._closed:
- logging.exception('Failed to read inputs in the data plane')
+ logging.exception('Failed to read inputs in the data plane.')
self._exc_info = sys.exc_info()
raise
finally:
self._closed = True
self._reads_finished.set()
- def _start_reader(self, elements_iterator):
+ def set_inputs(self, elements_iterator):
reader = threading.Thread(
target=lambda: self._read_inputs(elements_iterator),
name='read_grpc_client_inputs')
@@ -293,16 +294,26 @@
def __init__(self, data_stub):
super(GrpcClientDataChannel, self).__init__()
- self._start_reader(data_stub.Data(self._write_outputs()))
+ self.set_inputs(data_stub.Data(self._write_outputs()))
-class GrpcServerDataChannel(
- beam_fn_api_pb2_grpc.BeamFnDataServicer, _GrpcDataChannel):
- """A DataChannel wrapping the server side of a BeamFnData connection."""
+class BeamFnDataServicer(beam_fn_api_pb2_grpc.BeamFnDataServicer):
+ """Implementation of BeamFnDataServicer for any number of clients"""
+
+ def __init__(self):
+ self._lock = threading.Lock()
+ self._connections_by_worker_id = collections.defaultdict(
+ _GrpcDataChannel)
+
+ def get_conn_by_worker_id(self, worker_id):
+ with self._lock:
+ return self._connections_by_worker_id[worker_id]
def Data(self, elements_iterator, context):
- self._start_reader(elements_iterator)
- for elements in self._write_outputs():
+ worker_id = dict(context.invocation_metadata()).get('worker_id')
+ data_conn = self.get_conn_by_worker_id(worker_id)
+ data_conn.set_inputs(elements_iterator)
+ for elements in data_conn._write_outputs():
yield elements
@@ -326,10 +337,11 @@
Caches the created channels by ``data descriptor url``.
"""
- def __init__(self, credentials=None):
+ def __init__(self, credentials=None, worker_id=None):
self._data_channel_cache = {}
self._lock = threading.Lock()
self._credentials = None
+ self._worker_id = worker_id
if credentials is not None:
logging.info('Using secure channel creds.')
self._credentials = credentials
@@ -339,7 +351,7 @@
if url not in self._data_channel_cache:
with self._lock:
if url not in self._data_channel_cache:
- logging.info('Creating channel for %s', url)
+ logging.info('Creating client data channel for %s', url)
# Options to have no limits (-1) on the size of the messages
# received or sent over the data plane. The actual buffer size
# is controlled in a layer above.
@@ -353,8 +365,8 @@
grpc_channel = GRPCChannelFactory.secure_channel(
url, self._credentials, options=channel_options)
# Add workerId to the grpc channel
- grpc_channel = grpc.intercept_channel(grpc_channel,
- WorkerIdInterceptor())
+ grpc_channel = grpc.intercept_channel(
+ grpc_channel, WorkerIdInterceptor(self._worker_id))
self._data_channel_cache[url] = GrpcClientDataChannel(
beam_fn_api_pb2_grpc.BeamFnDataStub(grpc_channel))
diff --git a/sdks/python/apache_beam/runners/worker/data_plane_test.py b/sdks/python/apache_beam/runners/worker/data_plane_test.py
index a2f85b2..d11390a 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane_test.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane_test.py
@@ -33,6 +33,7 @@
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.runners.worker import data_plane
+from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor
def timeout(timeout_secs):
@@ -61,16 +62,22 @@
@timeout(5)
def test_grpc_data_channel(self):
- data_channel_service = data_plane.GrpcServerDataChannel()
+ data_servicer = data_plane.BeamFnDataServicer()
+ worker_id = 'worker_0'
+ data_channel_service = \
+ data_servicer.get_conn_by_worker_id(worker_id)
server = grpc.server(futures.ThreadPoolExecutor(max_workers=2))
beam_fn_api_pb2_grpc.add_BeamFnDataServicer_to_server(
- data_channel_service, server)
+ data_servicer, server)
test_port = server.add_insecure_port('[::]:0')
server.start()
- data_channel_stub = beam_fn_api_pb2_grpc.BeamFnDataStub(
- grpc.insecure_channel('localhost:%s' % test_port))
+ grpc_channel = grpc.insecure_channel('localhost:%s' % test_port)
+ # Add workerId to the grpc channel
+ grpc_channel = grpc.intercept_channel(
+ grpc_channel, WorkerIdInterceptor(worker_id))
+ data_channel_stub = beam_fn_api_pb2_grpc.BeamFnDataStub(grpc_channel)
data_channel_client = data_plane.GrpcClientDataChannel(data_channel_stub)
try:
@@ -90,49 +97,41 @@
self._data_channel_test_one_direction(client, server)
def _data_channel_test_one_direction(self, from_channel, to_channel):
- def send(instruction_id, target, data):
- stream = from_channel.output_stream(instruction_id, target)
+ def send(instruction_id, transform_id, data):
+ stream = from_channel.output_stream(instruction_id, transform_id)
stream.write(data)
stream.close()
- target_1 = beam_fn_api_pb2.Target(
- primitive_transform_reference='1',
- name='out')
- target_2 = beam_fn_api_pb2.Target(
- primitive_transform_reference='2',
- name='out')
+ transform_1 = '1'
+ transform_2 = '2'
# Single write.
- send('0', target_1, b'abc')
+ send('0', transform_1, b'abc')
self.assertEqual(
- list(to_channel.input_elements('0', [target_1])),
+ list(to_channel.input_elements('0', [transform_1])),
[beam_fn_api_pb2.Elements.Data(
- instruction_reference='0',
- target=target_1,
+ instruction_id='0',
+ transform_id=transform_1,
data=b'abc')])
# Multiple interleaved writes to multiple instructions.
- target_2 = beam_fn_api_pb2.Target(
- primitive_transform_reference='2',
- name='out')
-
- send('1', target_1, b'abc')
- send('2', target_1, b'def')
+ send('1', transform_1, b'abc')
+ send('2', transform_1, b'def')
self.assertEqual(
- list(to_channel.input_elements('1', [target_1])),
+ list(to_channel.input_elements('1', [transform_1])),
[beam_fn_api_pb2.Elements.Data(
- instruction_reference='1',
- target=target_1,
+ instruction_id='1',
+ transform_id=transform_1,
data=b'abc')])
- send('2', target_2, b'ghi')
+ send('2', transform_2, b'ghi')
self.assertEqual(
- list(to_channel.input_elements('2', [target_1, target_2])),
+ list(to_channel.input_elements('2', [transform_1, transform_2])),
[beam_fn_api_pb2.Elements.Data(
- instruction_reference='2',
- target=target_1,
+ instruction_id='2',
+ transform_id=transform_1,
data=b'def'),
beam_fn_api_pb2.Elements.Data(
- instruction_reference='2',
- target=target_2,
+ instruction_id='2',
+ transform_id=transform_2,
data=b'ghi')])
diff --git a/sdks/python/apache_beam/runners/worker/log_handler.py b/sdks/python/apache_beam/runners/worker/log_handler.py
index 4eac50c..b38aaed 100644
--- a/sdks/python/apache_beam/runners/worker/log_handler.py
+++ b/sdks/python/apache_beam/runners/worker/log_handler.py
@@ -99,16 +99,22 @@
def close(self):
"""Flush out all existing log entries and unregister this handler."""
- self._alive = False
- # Acquiring the handler lock ensures ``emit`` is not run until the lock is
- # released.
- self.acquire()
- self._log_entry_queue.put(self._FINISHED, timeout=5)
- # wait on server to close.
- self._reader.join()
- self.release()
- # Unregister this handler.
- super(FnApiLogRecordHandler, self).close()
+ try:
+ self._alive = False
+ # Acquiring the handler lock ensures ``emit`` is not run until the lock is
+ # released.
+ self.acquire()
+ self._log_entry_queue.put(self._FINISHED, timeout=5)
+ # wait on server to close.
+ self._reader.join()
+ self.release()
+ # Unregister this handler.
+ super(FnApiLogRecordHandler, self).close()
+ except Exception:
+ # Log rather than raising exceptions, to avoid clobbering
+ # underlying errors that may have caused this to close
+ # prematurely.
+ logging.error("Error closing the logging channel.", exc_info=True)
def _write_log_entries(self):
done = False
diff --git a/sdks/python/apache_beam/runners/worker/operations.py b/sdks/python/apache_beam/runners/worker/operations.py
index 6a0ef72..28a2b4a 100644
--- a/sdks/python/apache_beam/runners/worker/operations.py
+++ b/sdks/python/apache_beam/runners/worker/operations.py
@@ -886,7 +886,8 @@
if windows is 0:
self.output(_globally_windowed_value.with_value((key, value)))
else:
- self.output(WindowedValue((key, value), windows[0].end, windows))
+ self.output(
+ WindowedValue((key, value), windows[0].max_timestamp(), windows))
class FlattenOperation(Operation):
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 0c52274..0efc791 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -37,20 +37,31 @@
from future.utils import raise_
from future.utils import with_metaclass
+from apache_beam.coders import coder_impl
from apache_beam.portability.api import beam_fn_api_pb2
from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.runners.worker import bundle_processor
from apache_beam.runners.worker import data_plane
from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.statecache import StateCache
from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor
+# This SDK harness will (by default), log a "lull" in processing if it sees no
+# transitions in over 5 minutes.
+# 5 minutes * 60 seconds * 1020 millis * 1000 micros * 1000 nanoseconds
+DEFAULT_LOG_LULL_TIMEOUT_NS = 5 * 60 * 1000 * 1000 * 1000
+
class SdkHarness(object):
REQUEST_METHOD_PREFIX = '_request_'
SCHEDULING_DELAY_THRESHOLD_SEC = 5*60 # 5 Minutes
def __init__(
- self, control_address, worker_count, credentials=None, worker_id=None,
+ self, control_address, worker_count,
+ credentials=None,
+ worker_id=None,
+ # Caching is disabled by default
+ state_cache_size=0,
profiler_factory=None):
self._alive = True
self._worker_count = worker_count
@@ -70,8 +81,9 @@
self._control_channel = grpc.intercept_channel(
self._control_channel, WorkerIdInterceptor(self._worker_id))
self._data_channel_factory = data_plane.GrpcClientDataChannelFactory(
- credentials)
- self._state_handler_factory = GrpcStateHandlerFactory(credentials)
+ credentials, self._worker_id)
+ self._state_handler_factory = GrpcStateHandlerFactory(state_cache_size,
+ credentials)
self._profiler_factory = profiler_factory
self._fns = {}
# BundleProcessor cache across all workers.
@@ -106,8 +118,8 @@
for _ in range(self._worker_count):
# SdkHarness manage function registration and share self._fns with all
# the workers. This is needed because function registration (register)
- # and exceution(process_bundle) are send over different request and we
- # do not really know which woker is going to process bundle
+ # and execution (process_bundle) are send over different request and we
+ # do not really know which worker is going to process bundle
# for a function till we get process_bundle request. Moreover
# same function is reused by different process bundle calls and
# potentially get executed by different worker. Hence we need a
@@ -124,7 +136,8 @@
yield response
self._alive = True
- monitoring_thread = threading.Thread(target=self._monitor_process_bundle)
+ monitoring_thread = threading.Thread(name='SdkHarness_monitor',
+ target=self._monitor_process_bundle)
monitoring_thread.daemon = True
monitoring_thread.start()
@@ -207,10 +220,10 @@
def _request_process_bundle_action(self, request):
def task():
- instruction_reference = getattr(
- request, request.WhichOneof('request')).instruction_reference
+ instruction_id = getattr(
+ request, request.WhichOneof('request')).instruction_id
# only process progress/split request when a bundle is in processing.
- if (instruction_reference in
+ if (instruction_id in
self._bundle_processor_cache.active_bundle_processors):
self._execute(
lambda: self.progress_worker.do_instruction(request), request)
@@ -218,9 +231,9 @@
self._execute(lambda: beam_fn_api_pb2.InstructionResponse(
instruction_id=request.instruction_id, error=(
'Process bundle request not yet scheduled for instruction {}' if
- instruction_reference in self._unscheduled_process_bundle else
+ instruction_id in self._unscheduled_process_bundle else
'Unknown process bundle instruction {}').format(
- instruction_reference)), request)
+ instruction_id)), request)
self._progress_thread_pool.submit(task)
@@ -330,9 +343,14 @@
class SdkWorker(object):
- def __init__(self, bundle_processor_cache, profiler_factory=None):
+ def __init__(self,
+ bundle_processor_cache,
+ profiler_factory=None,
+ log_lull_timeout_ns=None):
self.bundle_processor_cache = bundle_processor_cache
self.profiler_factory = profiler_factory
+ self.log_lull_timeout_ns = (log_lull_timeout_ns
+ or DEFAULT_LOG_LULL_TIMEOUT_NS)
def do_instruction(self, request):
request_type = request.WhichOneof('request')
@@ -359,10 +377,10 @@
def process_bundle(self, request, instruction_id):
bundle_processor = self.bundle_processor_cache.get(
- instruction_id, request.process_bundle_descriptor_reference)
+ instruction_id, request.process_bundle_descriptor_id)
try:
with bundle_processor.state_handler.process_instruction_id(
- instruction_id):
+ instruction_id, request.cache_tokens):
with self.maybe_profile(instruction_id):
delayed_applications, requests_finalization = (
bundle_processor.process_bundle(instruction_id))
@@ -384,7 +402,7 @@
def process_bundle_split(self, request, instruction_id):
processor = self.bundle_processor_cache.lookup(
- request.instruction_reference)
+ request.instruction_id)
if processor:
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
@@ -394,10 +412,35 @@
instruction_id=instruction_id,
error='Instruction not running: %s' % instruction_id)
+ def _log_lull_in_bundle_processor(self, processor):
+ state_sampler = processor.state_sampler
+ sampler_info = state_sampler.get_info()
+ if (sampler_info
+ and sampler_info.time_since_transition
+ and sampler_info.time_since_transition > self.log_lull_timeout_ns):
+ step_name = sampler_info.state_name.step_name
+ state_name = sampler_info.state_name.name
+ state_lull_log = (
+ 'There has been a processing lull of over %.2f seconds in state %s'
+ % (sampler_info.time_since_transition / 1e9, state_name))
+ step_name_log = (' in step %s ' % step_name) if step_name else ''
+
+ exec_thread = getattr(sampler_info, 'tracked_thread', None)
+ if exec_thread is not None:
+ thread_frame = sys._current_frames().get(exec_thread.ident) # pylint: disable=protected-access
+ stack_trace = '\n'.join(
+ traceback.format_stack(thread_frame)) if thread_frame else ''
+ else:
+ stack_trace = '-NOT AVAILABLE-'
+
+ logging.warning(
+ '%s%s. Traceback:\n%s', state_lull_log, step_name_log, stack_trace)
+
def process_bundle_progress(self, request, instruction_id):
# It is an error to get progress for a not-in-flight bundle.
- processor = self.bundle_processor_cache.lookup(
- request.instruction_reference)
+ processor = self.bundle_processor_cache.lookup(request.instruction_id)
+ if processor:
+ self._log_lull_in_bundle_processor(processor)
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
process_bundle_progress=beam_fn_api_pb2.ProcessBundleProgressResponse(
@@ -406,16 +449,16 @@
def finalize_bundle(self, request, instruction_id):
processor = self.bundle_processor_cache.lookup(
- request.instruction_reference)
+ request.instruction_id)
if processor:
try:
finalize_response = processor.finalize_bundle()
- self.bundle_processor_cache.release(request.instruction_reference)
+ self.bundle_processor_cache.release(request.instruction_id)
return beam_fn_api_pb2.InstructionResponse(
instruction_id=instruction_id,
finalize_bundle=finalize_response)
except:
- self.bundle_processor_cache.discard(request.instruction_reference)
+ self.bundle_processor_cache.discard(request.instruction_id)
raise
else:
return beam_fn_api_pb2.InstructionResponse(
@@ -458,11 +501,12 @@
Caches the created channels by ``state descriptor url``.
"""
- def __init__(self, credentials=None):
+ def __init__(self, state_cache_size, credentials=None):
self._state_handler_cache = {}
self._lock = threading.Lock()
self._throwing_state_handler = ThrowingStateHandler()
self._credentials = credentials
+ self._state_cache = StateCache(state_cache_size)
def create_state_handler(self, api_service_descriptor):
if not api_service_descriptor:
@@ -488,8 +532,10 @@
# Add workerId to the grpc channel
grpc_channel = grpc.intercept_channel(grpc_channel,
WorkerIdInterceptor())
- self._state_handler_cache[url] = GrpcStateHandler(
- beam_fn_api_pb2_grpc.BeamFnStateStub(grpc_channel))
+ self._state_handler_cache[url] = CachingMaterializingStateHandler(
+ self._state_cache,
+ GrpcStateHandler(
+ beam_fn_api_pb2_grpc.BeamFnStateStub(grpc_channel)))
return self._state_handler_cache[url]
def close(self):
@@ -497,28 +543,26 @@
for _, state_handler in self._state_handler_cache.items():
state_handler.done()
self._state_handler_cache.clear()
+ self._state_cache.evict_all()
class ThrowingStateHandler(object):
"""A state handler that errors on any requests."""
- def blocking_get(self, state_key, instruction_reference):
+ def blocking_get(self, state_key, coder):
raise RuntimeError(
'Unable to handle state requests for ProcessBundleDescriptor without '
- 'out state ApiServiceDescriptor for instruction %s and state key %s.'
- % (state_key, instruction_reference))
+ 'state ApiServiceDescriptor for state key %s.' % state_key)
- def blocking_append(self, state_key, data, instruction_reference):
+ def append(self, state_key, coder, elements):
raise RuntimeError(
'Unable to handle state requests for ProcessBundleDescriptor without '
- 'out state ApiServiceDescriptor for instruction %s and state key %s.'
- % (state_key, instruction_reference))
+ 'state ApiServiceDescriptor for state key %s.' % state_key)
- def blocking_clear(self, state_key, instruction_reference):
+ def clear(self, state_key):
raise RuntimeError(
'Unable to handle state requests for ProcessBundleDescriptor without '
- 'out state ApiServiceDescriptor for instruction %s and state key %s.'
- % (state_key, instruction_reference))
+ 'state ApiServiceDescriptor for state key %s.' % state_key)
class GrpcStateHandler(object):
@@ -526,7 +570,6 @@
_DONE = object()
def __init__(self, state_stub):
- self._lock = threading.Lock()
self._state_stub = state_stub
self._requests = queue.Queue()
self._responses_by_id = {}
@@ -561,7 +604,8 @@
def pull_responses():
try:
for response in responses:
- self._responses_by_id[response.id].set(response)
+ future = self._responses_by_id.pop(response.id)
+ future.set(response)
if self._done:
break
except: # pylint: disable=bare-except
@@ -576,7 +620,7 @@
self._done = True
self._requests.put(self._DONE)
- def blocking_get(self, state_key, continuation_token=None):
+ def get_raw(self, state_key, continuation_token=None):
response = self._blocking_request(
beam_fn_api_pb2.StateRequest(
state_key=state_key,
@@ -584,31 +628,34 @@
continuation_token=continuation_token)))
return response.get.data, response.get.continuation_token
- def blocking_append(self, state_key, data):
- self._blocking_request(
+ def append_raw(self, state_key, data):
+ return self._request(
beam_fn_api_pb2.StateRequest(
state_key=state_key,
append=beam_fn_api_pb2.StateAppendRequest(data=data)))
- def blocking_clear(self, state_key):
- self._blocking_request(
+ def clear(self, state_key):
+ return self._request(
beam_fn_api_pb2.StateRequest(
state_key=state_key,
clear=beam_fn_api_pb2.StateClearRequest()))
- def _blocking_request(self, request):
+ def _request(self, request):
request.id = self._next_id()
- request.instruction_reference = self._context.process_instruction_id
+ request.instruction_id = self._context.process_instruction_id
self._responses_by_id[request.id] = future = _Future()
self._requests.put(request)
- while not future.wait(timeout=1):
+ return future
+
+ def _blocking_request(self, request):
+ req_future = self._request(request)
+ while not req_future.wait(timeout=1):
if self._exc_info:
t, v, tb = self._exc_info
raise_(t, v, tb)
elif self._done:
raise RuntimeError()
- del self._responses_by_id[request.id]
- response = future.get()
+ response = req_future.get()
if response.error:
raise RuntimeError(response.error)
else:
@@ -619,6 +666,101 @@
return str(self._last_id)
+class CachingMaterializingStateHandler(object):
+ """ A State handler which retrieves and caches state. """
+
+ def __init__(self, global_state_cache, underlying_state):
+ self._underlying = underlying_state
+ self._state_cache = global_state_cache
+ self._context = threading.local()
+
+ @contextlib.contextmanager
+ def process_instruction_id(self, bundle_id, cache_tokens):
+ if getattr(self._context, 'cache_token', None) is not None:
+ raise RuntimeError(
+ 'Cache tokens already set to %s' % self._context.cache_token)
+ # TODO Also handle cache tokens for side input, if present:
+ # https://issues.apache.org/jira/browse/BEAM-8298
+ user_state_cache_token = None
+ for cache_token_struct in cache_tokens:
+ if cache_token_struct.HasField("user_state"):
+ # There should only be one user state token present
+ assert not user_state_cache_token
+ user_state_cache_token = cache_token_struct.token
+ try:
+ self._context.cache_token = user_state_cache_token
+ with self._underlying.process_instruction_id(bundle_id):
+ yield
+ finally:
+ self._context.cache_token = None
+
+ def blocking_get(self, state_key, coder, is_cached=False):
+ if not self._should_be_cached(is_cached):
+ # Cache disabled / no cache token. Can't do a lookup/store in the cache.
+ # Fall back to lazily materializing the state, one element at a time.
+ return self._materialize_iter(state_key, coder)
+ # Cache lookup
+ cache_state_key = self._convert_to_cache_key(state_key)
+ cached_value = self._state_cache.get(cache_state_key,
+ self._context.cache_token)
+ if cached_value is None:
+ # Cache miss, need to retrieve from the Runner
+ # TODO If caching is enabled, this materializes the entire state.
+ # Further size estimation or the use of the continuation token on the
+ # runner side could fall back to materializing one item at a time.
+ # https://jira.apache.org/jira/browse/BEAM-8297
+ materialized = cached_value = list(
+ self._materialize_iter(state_key, coder))
+ self._state_cache.put(
+ cache_state_key,
+ self._context.cache_token,
+ materialized)
+ return iter(cached_value)
+
+ def extend(self, state_key, coder, elements, is_cached=False):
+ if self._should_be_cached(is_cached):
+ # Update the cache
+ cache_key = self._convert_to_cache_key(state_key)
+ self._state_cache.extend(cache_key, self._context.cache_token, elements)
+ # Write to state handler
+ out = coder_impl.create_OutputStream()
+ for element in elements:
+ coder.encode_to_stream(element, out, True)
+ return self._underlying.append_raw(state_key, out.get())
+
+ def clear(self, state_key, is_cached=False):
+ if self._should_be_cached(is_cached):
+ cache_key = self._convert_to_cache_key(state_key)
+ self._state_cache.clear(cache_key, self._context.cache_token)
+ return self._underlying.clear(state_key)
+
+ def done(self):
+ self._underlying.done()
+
+ def _materialize_iter(self, state_key, coder):
+ """Materializes the state lazily, one element at a time.
+ :return A generator which returns the next element if advanced.
+ """
+ continuation_token = None
+ while True:
+ data, continuation_token = \
+ self._underlying.get_raw(state_key, continuation_token)
+ input_stream = coder_impl.create_InputStream(data)
+ while input_stream.size() > 0:
+ yield coder.decode_from_stream(input_stream, True)
+ if not continuation_token:
+ break
+
+ def _should_be_cached(self, request_is_cached):
+ return (self._state_cache.is_cache_enabled() and
+ request_is_cached and
+ self._context.cache_token)
+
+ @staticmethod
+ def _convert_to_cache_key(state_key):
+ return state_key.SerializeToString()
+
+
class _Future(object):
"""A simple future object to implement blocking requests.
"""
@@ -638,3 +780,11 @@
def set(self, value):
self._value = value
self._event.set()
+
+ @classmethod
+ def done(cls):
+ if not hasattr(cls, 'DONE'):
+ done_future = _Future()
+ done_future.set(None)
+ cls.DONE = done_future
+ return cls.DONE
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_main.py b/sdks/python/apache_beam/runners/worker/sdk_worker_main.py
index 3748bd4..c6cb8ed 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_main.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_main.py
@@ -31,9 +31,9 @@
from google.protobuf import text_format
from apache_beam.internal import pickler
-from apache_beam.options import pipeline_options
from apache_beam.options.pipeline_options import DebugOptions
from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import ProfilingOptions
from apache_beam.portability.api import endpoints_pb2
from apache_beam.runners.internal import names
from apache_beam.runners.worker.log_handler import FnApiLogRecordHandler
@@ -74,7 +74,7 @@
self.end_headers()
for line in StatusServer.get_thread_dump():
- self.wfile.write(line)
+ self.wfile.write(line.encode('utf-8'))
def log_message(self, f, *args):
"""Do not log any messages."""
@@ -91,21 +91,27 @@
def main(unused_argv):
"""Main entry point for SDK Fn Harness."""
if 'LOGGING_API_SERVICE_DESCRIPTOR' in os.environ:
- logging_service_descriptor = endpoints_pb2.ApiServiceDescriptor()
- text_format.Merge(os.environ['LOGGING_API_SERVICE_DESCRIPTOR'],
- logging_service_descriptor)
+ try:
+ logging_service_descriptor = endpoints_pb2.ApiServiceDescriptor()
+ text_format.Merge(os.environ['LOGGING_API_SERVICE_DESCRIPTOR'],
+ logging_service_descriptor)
- # Send all logs to the runner.
- fn_log_handler = FnApiLogRecordHandler(logging_service_descriptor)
- # TODO(BEAM-5468): This should be picked up from pipeline options.
- logging.getLogger().setLevel(logging.INFO)
- logging.getLogger().addHandler(fn_log_handler)
- logging.info('Logging handler created.')
+ # Send all logs to the runner.
+ fn_log_handler = FnApiLogRecordHandler(logging_service_descriptor)
+ # TODO(BEAM-5468): This should be picked up from pipeline options.
+ logging.getLogger().setLevel(logging.INFO)
+ logging.getLogger().addHandler(fn_log_handler)
+ logging.info('Logging handler created.')
+ except Exception:
+ logging.error("Failed to set up logging handler, continuing without.",
+ exc_info=True)
+ fn_log_handler = None
else:
fn_log_handler = None
# Start status HTTP server thread.
- thread = threading.Thread(target=StatusServer().start)
+ thread = threading.Thread(name='status_http_server',
+ target=StatusServer().start)
thread.daemon = True
thread.setName('status-server-demon')
thread.start()
@@ -122,6 +128,7 @@
semi_persistent_directory = None
logging.info('semi_persistent_directory: %s', semi_persistent_directory)
+ _worker_id = os.environ.get('WORKER_ID', None)
try:
_load_main_session(semi_persistent_directory)
@@ -141,8 +148,10 @@
SdkHarness(
control_address=service_descriptor.url,
worker_count=_get_worker_count(sdk_pipeline_options),
+ worker_id=_worker_id,
+ state_cache_size=_get_state_cache_size(sdk_pipeline_options),
profiler_factory=profiler.Profile.factory_from_options(
- sdk_pipeline_options.view_as(pipeline_options.ProfilingOptions))
+ sdk_pipeline_options.view_as(ProfilingOptions))
).run()
logging.info('Python sdk harness exiting.')
except: # pylint: disable=broad-except
@@ -181,7 +190,7 @@
future releases.
Returns:
- an int containing the worker_threads to use. Default is 1
+ an int containing the worker_threads to use. Default is 12
"""
experiments = pipeline_options.view_as(DebugOptions).experiments
@@ -197,6 +206,28 @@
return 12
+def _get_state_cache_size(pipeline_options):
+ """Defines the upper number of state items to cache.
+
+ Note: state_cache_size is an experimental flag and might not be available in
+ future releases.
+
+ Returns:
+ an int indicating the maximum number of items to cache.
+ Default is 0 (disabled)
+ """
+ experiments = pipeline_options.view_as(DebugOptions).experiments
+ experiments = experiments if experiments else []
+
+ for experiment in experiments:
+ # There should only be 1 match so returning from the loop
+ if re.match(r'state_cache_size=', experiment):
+ return int(
+ re.match(r'state_cache_size=(?P<state_cache_size>.*)',
+ experiment).group('state_cache_size'))
+ return 0
+
+
def _load_main_session(semi_persistent_directory):
"""Loads a pickled main session from the path specified."""
if semi_persistent_directory:
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py b/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
index e9b584a..cd33f7e 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py
@@ -122,7 +122,7 @@
Exception, sdk_worker_main._get_worker_count,
PipelineOptions.from_dictionary(json.loads(pipeline_options)))
else:
- self.assertEquals(
+ self.assertEqual(
sdk_worker_main._get_worker_count(
PipelineOptions.from_dictionary(json.loads(pipeline_options))),
expected)
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker_test.py b/sdks/python/apache_beam/runners/worker/sdk_worker_test.py
index 9b094b7..71263a8 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker_test.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker_test.py
@@ -100,7 +100,8 @@
server.start()
harness = sdk_worker.SdkHarness(
- "localhost:%s" % test_port, worker_count=worker_count)
+ "localhost:%s" % test_port, worker_count=worker_count,
+ state_cache_size=100)
harness.run()
for worker in harness.workers.queue:
diff --git a/sdks/python/apache_beam/runners/worker/statecache.py b/sdks/python/apache_beam/runners/worker/statecache.py
new file mode 100644
index 0000000..a4902c6
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statecache.py
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""A module for caching state reads/writes in Beam applications."""
+from __future__ import absolute_import
+
+import collections
+import logging
+from threading import Lock
+
+
+class StateCache(object):
+ """ Cache for Beam state access, scoped by state key and cache_token.
+
+ For a given state_key, caches a (cache_token, value) tuple and allows to
+ a) read from the cache (get),
+ if the currently stored cache_token matches the provided
+ a) write to the cache (put),
+ storing the new value alongside with a cache token
+ c) append to the cache (extend),
+ if the currently stored cache_token matches the provided
+
+ The operations on the cache are thread-safe for use by multiple workers.
+
+ :arg max_entries The maximum number of entries to store in the cache.
+ TODO Memory-based caching: https://issues.apache.org/jira/browse/BEAM-8297
+ """
+
+ def __init__(self, max_entries):
+ logging.info('Creating state cache with size %s', max_entries)
+ self._cache = self.LRUCache(max_entries, (None, None))
+ self._lock = Lock()
+
+ def get(self, state_key, cache_token):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ token, value = self._cache.get(state_key)
+ return value if token == cache_token else None
+
+ def put(self, state_key, cache_token, value):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ return self._cache.put(state_key, (cache_token, value))
+
+ def extend(self, state_key, cache_token, elements):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ token, value = self._cache.get(state_key)
+ if token in [cache_token, None]:
+ if value is None:
+ value = []
+ value.extend(elements)
+ self._cache.put(state_key, (cache_token, value))
+ else:
+ # Discard cached state if tokens do not match
+ self._cache.evict(state_key)
+
+ def clear(self, state_key, cache_token):
+ assert cache_token and self.is_cache_enabled()
+ with self._lock:
+ token, _ = self._cache.get(state_key)
+ if token in [cache_token, None]:
+ self._cache.put(state_key, (cache_token, []))
+ else:
+ # Discard cached state if tokens do not match
+ self._cache.evict(state_key)
+
+ def evict(self, state_key):
+ assert self.is_cache_enabled()
+ with self._lock:
+ self._cache.evict(state_key)
+
+ def evict_all(self):
+ with self._lock:
+ self._cache.evict_all()
+
+ def is_cache_enabled(self):
+ return self._cache._max_entries > 0
+
+ def __len__(self):
+ return len(self._cache)
+
+ class LRUCache(object):
+
+ def __init__(self, max_entries, default_entry):
+ self._max_entries = max_entries
+ self._default_entry = default_entry
+ self._cache = collections.OrderedDict()
+
+ def get(self, key):
+ value = self._cache.pop(key, self._default_entry)
+ if value != self._default_entry:
+ self._cache[key] = value
+ return value
+
+ def put(self, key, value):
+ self._cache[key] = value
+ while len(self._cache) > self._max_entries:
+ self._cache.popitem(last=False)
+
+ def evict(self, key):
+ self._cache.pop(key, self._default_entry)
+
+ def evict_all(self):
+ self._cache.clear()
+
+ def __len__(self):
+ return len(self._cache)
diff --git a/sdks/python/apache_beam/runners/worker/statecache_test.py b/sdks/python/apache_beam/runners/worker/statecache_test.py
new file mode 100644
index 0000000..8fedeaf
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statecache_test.py
@@ -0,0 +1,155 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for state caching."""
+from __future__ import absolute_import
+
+import logging
+import unittest
+
+from apache_beam.runners.worker.statecache import StateCache
+
+
+class StateCacheTest(unittest.TestCase):
+
+ def test_empty_cache_get(self):
+ cache = StateCache(5)
+ self.assertEqual(cache.get("key", 'cache_token'), None)
+ with self.assertRaises(Exception):
+ self.assertEqual(cache.get("key", None), None)
+
+ def test_put_get(self):
+ cache = StateCache(5)
+ cache.put("key", "cache_token", "value")
+ self.assertEqual(len(cache), 1)
+ self.assertEqual(cache.get("key", "cache_token"), "value")
+ self.assertEqual(cache.get("key", "cache_token2"), None)
+ with self.assertRaises(Exception):
+ self.assertEqual(cache.get("key", None), None)
+
+ def test_overwrite(self):
+ cache = StateCache(2)
+ cache.put("key", "cache_token", "value")
+ cache.put("key", "cache_token2", "value2")
+ self.assertEqual(len(cache), 1)
+ self.assertEqual(cache.get("key", "cache_token"), None)
+ self.assertEqual(cache.get("key", "cache_token2"), "value2")
+
+ def test_extend(self):
+ cache = StateCache(3)
+ cache.put("key", "cache_token", ['val'])
+ # test extend for existing key
+ cache.extend("key", "cache_token", ['yet', 'another', 'val'])
+ self.assertEqual(len(cache), 1)
+ self.assertEqual(cache.get("key", "cache_token"),
+ ['val', 'yet', 'another', 'val'])
+ # test extend without existing key
+ cache.extend("key2", "cache_token", ['another', 'val'])
+ self.assertEqual(len(cache), 2)
+ self.assertEqual(cache.get("key2", "cache_token"), ['another', 'val'])
+ # test eviction in case the cache token changes
+ cache.extend("key2", "new_token", ['new_value'])
+ self.assertEqual(cache.get("key2", "new_token"), None)
+ self.assertEqual(len(cache), 1)
+
+ def test_clear(self):
+ cache = StateCache(5)
+ cache.clear("new-key", "cache_token")
+ cache.put("key", "cache_token", ["value"])
+ self.assertEqual(len(cache), 2)
+ self.assertEqual(cache.get("new-key", "new_token"), None)
+ self.assertEqual(cache.get("key", "cache_token"), ['value'])
+ # test clear without existing key/token
+ cache.clear("non-existing", "token")
+ self.assertEqual(len(cache), 3)
+ self.assertEqual(cache.get("non-existing", "token"), [])
+ # test eviction in case the cache token changes
+ cache.clear("new-key", "wrong_token")
+ self.assertEqual(len(cache), 2)
+ self.assertEqual(cache.get("new-key", "cache_token"), None)
+ self.assertEqual(cache.get("new-key", "wrong_token"), None)
+
+ def test_max_size(self):
+ cache = StateCache(2)
+ cache.put("key", "cache_token", "value")
+ cache.put("key2", "cache_token", "value")
+ self.assertEqual(len(cache), 2)
+ cache.put("key2", "cache_token", "value")
+ self.assertEqual(len(cache), 2)
+ cache.put("key", "cache_token", "value")
+ self.assertEqual(len(cache), 2)
+
+ def test_evict_all(self):
+ cache = StateCache(5)
+ cache.put("key", "cache_token", "value")
+ cache.put("key2", "cache_token", "value2")
+ self.assertEqual(len(cache), 2)
+ cache.evict_all()
+ self.assertEqual(len(cache), 0)
+ self.assertEqual(cache.get("key", "cache_token"), None)
+ self.assertEqual(cache.get("key2", "cache_token"), None)
+
+ def test_lru(self):
+ cache = StateCache(5)
+ cache.put("key", "cache_token", "value")
+ cache.put("key2", "cache_token2", "value2")
+ cache.put("key3", "cache_token", "value0")
+ cache.put("key3", "cache_token", "value3")
+ cache.put("key4", "cache_token4", "value4")
+ cache.put("key5", "cache_token", "value0")
+ cache.put("key5", "cache_token", ["value5"])
+ self.assertEqual(len(cache), 5)
+ self.assertEqual(cache.get("key", "cache_token"), "value")
+ self.assertEqual(cache.get("key2", "cache_token2"), "value2")
+ self.assertEqual(cache.get("key3", "cache_token"), "value3")
+ self.assertEqual(cache.get("key4", "cache_token4"), "value4")
+ self.assertEqual(cache.get("key5", "cache_token"), ["value5"])
+ # insert another key to trigger cache eviction
+ cache.put("key6", "cache_token2", "value7")
+ self.assertEqual(len(cache), 5)
+ # least recently used key should be gone ("key")
+ self.assertEqual(cache.get("key", "cache_token"), None)
+ # trigger a read on "key2"
+ cache.get("key2", "cache_token")
+ # insert another key to trigger cache eviction
+ cache.put("key7", "cache_token", "value7")
+ self.assertEqual(len(cache), 5)
+ # least recently used key should be gone ("key3")
+ self.assertEqual(cache.get("key3", "cache_token"), None)
+ # trigger a put on "key2"
+ cache.put("key2", "cache_token", "put")
+ self.assertEqual(len(cache), 5)
+ # insert another key to trigger cache eviction
+ cache.put("key8", "cache_token", "value8")
+ self.assertEqual(len(cache), 5)
+ # least recently used key should be gone ("key4")
+ self.assertEqual(cache.get("key4", "cache_token"), None)
+ # make "key5" used by appending to it
+ cache.extend("key5", "cache_token", ["another"])
+ # least recently used key should be gone ("key6")
+ self.assertEqual(cache.get("key6", "cache_token"), None)
+
+ def test_is_cached_enabled(self):
+ cache = StateCache(1)
+ self.assertEqual(cache.is_cache_enabled(), True)
+ cache = StateCache(0)
+ self.assertEqual(cache.is_cache_enabled(), False)
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/runners/worker/worker_pool_main.py b/sdks/python/apache_beam/runners/worker/worker_pool_main.py
new file mode 100644
index 0000000..ef9e005
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/worker_pool_main.py
@@ -0,0 +1,191 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+Worker pool entry point.
+
+The worker pool exposes an RPC service that is used with EXTERNAL
+environment to start and stop the SDK workers.
+
+The worker pool uses child processes for parallelism; threads are
+subject to the GIL and not sufficient.
+
+This entry point is used by the Python SDK container in worker pool mode.
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import atexit
+import logging
+import subprocess
+import sys
+import threading
+import time
+from concurrent import futures
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker import sdk_worker
+
+
+class BeamFnExternalWorkerPoolServicer(
+ beam_fn_api_pb2_grpc.BeamFnExternalWorkerPoolServicer):
+
+ def __init__(self, worker_threads,
+ use_process=False,
+ container_executable=None,
+ state_cache_size=0):
+ self._worker_threads = worker_threads
+ self._use_process = use_process
+ self._container_executable = container_executable
+ self._state_cache_size = state_cache_size
+ self._worker_processes = {}
+
+ @classmethod
+ def start(cls, worker_threads=1, use_process=False, port=0,
+ state_cache_size=0, container_executable=None):
+ worker_server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
+ worker_address = 'localhost:%s' % worker_server.add_insecure_port(
+ '[::]:%s' % port)
+ worker_pool = cls(worker_threads,
+ use_process=use_process,
+ container_executable=container_executable,
+ state_cache_size=state_cache_size)
+ beam_fn_api_pb2_grpc.add_BeamFnExternalWorkerPoolServicer_to_server(
+ worker_pool,
+ worker_server)
+ worker_server.start()
+
+ # Register to kill the subprocesses on exit.
+ def kill_worker_processes():
+ for worker_process in worker_pool._worker_processes.values():
+ worker_process.kill()
+ atexit.register(kill_worker_processes)
+
+ return worker_address, worker_server
+
+ def StartWorker(self, start_worker_request, unused_context):
+ try:
+ if self._use_process:
+ command = ['python', '-c',
+ 'from apache_beam.runners.worker.sdk_worker '
+ 'import SdkHarness; '
+ 'SdkHarness('
+ '"%s",'
+ 'worker_count=%d,'
+ 'worker_id="%s",'
+ 'state_cache_size=%d'
+ ')'
+ '.run()' % (
+ start_worker_request.control_endpoint.url,
+ self._worker_threads,
+ start_worker_request.worker_id,
+ self._state_cache_size)]
+ if self._container_executable:
+ # command as per container spec
+ # the executable is responsible to handle concurrency
+ # for artifact retrieval and other side effects
+ command = [self._container_executable,
+ '--id=%s' % start_worker_request.worker_id,
+ '--logging_endpoint=%s'
+ % start_worker_request.logging_endpoint.url,
+ '--artifact_endpoint=%s'
+ % start_worker_request.artifact_endpoint.url,
+ '--provision_endpoint=%s'
+ % start_worker_request.provision_endpoint.url,
+ '--control_endpoint=%s'
+ % start_worker_request.control_endpoint.url,
+ ]
+
+ logging.warn("Starting worker with command %s" % command)
+ worker_process = subprocess.Popen(command, stdout=subprocess.PIPE,
+ close_fds=True)
+ self._worker_processes[start_worker_request.worker_id] = worker_process
+ else:
+ worker = sdk_worker.SdkHarness(
+ start_worker_request.control_endpoint.url,
+ worker_count=self._worker_threads,
+ worker_id=start_worker_request.worker_id,
+ state_cache_size=self._state_cache_size)
+ worker_thread = threading.Thread(
+ name='run_worker_%s' % start_worker_request.worker_id,
+ target=worker.run)
+ worker_thread.daemon = True
+ worker_thread.start()
+
+ return beam_fn_api_pb2.StartWorkerResponse()
+ except Exception as exn:
+ return beam_fn_api_pb2.StartWorkerResponse(error=str(exn))
+
+ def StopWorker(self, stop_worker_request, unused_context):
+ # applicable for process mode to ensure process cleanup
+ # thread based workers terminate automatically
+ worker_process = self._worker_processes.pop(stop_worker_request.worker_id,
+ None)
+ if worker_process:
+ def kill_worker_process():
+ try:
+ worker_process.kill()
+ except OSError:
+ # ignore already terminated process
+ return
+ logging.info("Stopping worker %s" % stop_worker_request.worker_id)
+ # communicate is necessary to avoid zombie process
+ # time box communicate (it has no timeout parameter in Py2)
+ threading.Timer(1, kill_worker_process).start()
+ worker_process.communicate()
+ return beam_fn_api_pb2.StopWorkerResponse()
+
+
+def main(argv=None):
+ """Entry point for worker pool service for external environments."""
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--threads_per_worker',
+ type=int,
+ default=argparse.SUPPRESS,
+ dest='worker_threads',
+ help='Number of threads per SDK worker.')
+ parser.add_argument('--container_executable',
+ type=str,
+ default=None,
+ help='Executable that implements the Beam SDK '
+ 'container contract.')
+ parser.add_argument('--service_port',
+ type=int,
+ required=True,
+ dest='port',
+ help='Bind port for the worker pool service.')
+
+ args, _ = parser.parse_known_args(argv)
+
+ address, server = (BeamFnExternalWorkerPoolServicer.start(use_process=True,
+ **vars(args)))
+ logging.getLogger().setLevel(logging.INFO)
+ logging.info('Started worker pool servicer at port: %s with executable: %s',
+ address, args.container_executable)
+ try:
+ while True:
+ time.sleep(60 * 60 * 24)
+ except KeyboardInterrupt:
+ server.stop(0)
+
+
+if __name__ == '__main__':
+ main(sys.argv)
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/__init__.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/__init__.py
new file mode 100644
index 0000000..cce3aca
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/__init__.py
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/preprocess.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/preprocess.py
new file mode 100644
index 0000000..7e85c91
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/preprocess.py
@@ -0,0 +1,258 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Preprocessor applying tf.transform to the chicago_taxi data."""
+from __future__ import absolute_import, division, print_function
+
+import argparse
+import os
+
+import tensorflow as tf
+import tensorflow_transform as transform
+import tensorflow_transform.beam as tft_beam
+from tensorflow_transform.coders import example_proto_coder
+from tensorflow_transform.tf_metadata import dataset_metadata, dataset_schema
+
+import apache_beam as beam
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.testing.load_tests.load_test_metrics_utils import (
+ MeasureTime, MetricsReader)
+from trainer import taxi
+
+
+def _fill_in_missing(x):
+ """Replace missing values in a SparseTensor.
+
+ Fills in missing values of `x` with '' or 0, and converts to a dense tensor.
+
+ Args:
+ x: A `SparseTensor` of rank 2. Its dense shape should have size at most 1
+ in the second dimension.
+
+ Returns:
+ A rank 1 tensor where missing values of `x` have been filled in.
+ """
+ default_value = '' if x.dtype == tf.string else 0
+ return tf.squeeze(
+ tf.sparse.to_dense(
+ tf.SparseTensor(x.indices, x.values, [x.dense_shape[0], 1]),
+ default_value),
+ axis=1)
+
+
+def transform_data(input_handle,
+ outfile_prefix,
+ working_dir,
+ schema_file,
+ transform_dir=None,
+ max_rows=None,
+ pipeline_args=None,
+ publish_to_bq=False,
+ project=None,
+ metrics_table=None,
+ metrics_dataset=None):
+ """The main tf.transform method which analyzes and transforms data.
+
+ Args:
+ input_handle: BigQuery table name to process specified as DATASET.TABLE or
+ path to csv file with input data.
+ outfile_prefix: Filename prefix for emitted transformed examples
+ working_dir: Directory in which transformed examples and transform function
+ will be emitted.
+ schema_file: An file path that contains a text-serialized TensorFlow
+ metadata schema of the input data.
+ transform_dir: Directory in which the transform output is located. If
+ provided, this will load the transform_fn from disk instead of computing
+ it over the data. Hint: this is useful for transforming eval data.
+ max_rows: Number of rows to query from BigQuery
+ pipeline_args: additional DataflowRunner or DirectRunner args passed to the
+ beam pipeline.
+ """
+
+ def preprocessing_fn(inputs):
+ """tf.transform's callback function for preprocessing inputs.
+
+ Args:
+ inputs: map from feature keys to raw not-yet-transformed features.
+
+ Returns:
+ Map from string feature key to transformed feature operations.
+ """
+ outputs = {}
+ for key in taxi.DENSE_FLOAT_FEATURE_KEYS:
+ # Preserve this feature as a dense float, setting nan's to the mean.
+ outputs[taxi.transformed_name(key)] = transform.scale_to_z_score(
+ _fill_in_missing(inputs[key]))
+
+ for key in taxi.VOCAB_FEATURE_KEYS:
+ # Build a vocabulary for this feature.
+ outputs[
+ taxi.transformed_name(key)] = transform.compute_and_apply_vocabulary(
+ _fill_in_missing(inputs[key]),
+ top_k=taxi.VOCAB_SIZE,
+ num_oov_buckets=taxi.OOV_SIZE)
+
+ for key in taxi.BUCKET_FEATURE_KEYS:
+ outputs[taxi.transformed_name(key)] = transform.bucketize(
+ _fill_in_missing(inputs[key]), taxi.FEATURE_BUCKET_COUNT)
+
+ for key in taxi.CATEGORICAL_FEATURE_KEYS:
+ outputs[taxi.transformed_name(key)] = _fill_in_missing(inputs[key])
+
+ # Was this passenger a big tipper?
+ taxi_fare = _fill_in_missing(inputs[taxi.FARE_KEY])
+ tips = _fill_in_missing(inputs[taxi.LABEL_KEY])
+ outputs[taxi.transformed_name(taxi.LABEL_KEY)] = tf.where(
+ tf.is_nan(taxi_fare),
+ tf.cast(tf.zeros_like(taxi_fare), tf.int64),
+ # Test if the tip was > 20% of the fare.
+ tf.cast(
+ tf.greater(tips, tf.multiply(taxi_fare, tf.constant(0.2))),
+ tf.int64))
+
+ return outputs
+ namespace = metrics_table
+ metrics_monitor = None
+ if publish_to_bq:
+ metrics_monitor = MetricsReader(
+ project_name=project,
+ bq_table=metrics_table,
+ bq_dataset=metrics_dataset,
+ filters=MetricsFilter().with_namespace(namespace)
+ )
+ schema = taxi.read_schema(schema_file)
+ raw_feature_spec = taxi.get_raw_feature_spec(schema)
+ raw_schema = dataset_schema.from_feature_spec(raw_feature_spec)
+ raw_data_metadata = dataset_metadata.DatasetMetadata(raw_schema)
+
+ pipeline = beam.Pipeline(argv=pipeline_args)
+ with tft_beam.Context(temp_dir=working_dir):
+ query = taxi.make_sql(input_handle, max_rows, for_eval=False)
+ raw_data = (
+ pipeline
+ | 'ReadBigQuery' >> beam.io.Read(
+ beam.io.BigQuerySource(query=query,
+ use_standard_sql=True))
+ | 'Measure time: start' >> beam.ParDo(MeasureTime(namespace)))
+ decode_transform = beam.Map(
+ taxi.clean_raw_data_dict, raw_feature_spec=raw_feature_spec)
+
+ if transform_dir is None:
+ decoded_data = raw_data | 'DecodeForAnalyze' >> decode_transform
+ transform_fn = (
+ (decoded_data, raw_data_metadata) |
+ ('Analyze' >> tft_beam.AnalyzeDataset(preprocessing_fn)))
+
+ _ = (transform_fn | ('WriteTransformFn' >>
+ tft_beam.WriteTransformFn(working_dir)))
+ else:
+ transform_fn = pipeline | tft_beam.ReadTransformFn(transform_dir)
+
+ # Shuffling the data before materialization will improve Training
+ # effectiveness downstream. Here we shuffle the raw_data (as opposed to
+ # decoded data) since it has a compact representation.
+ shuffled_data = raw_data | 'RandomizeData' >> beam.transforms.Reshuffle()
+
+ decoded_data = shuffled_data | 'DecodeForTransform' >> decode_transform
+ (transformed_data, transformed_metadata) = (
+ ((decoded_data, raw_data_metadata), transform_fn)
+ | 'Transform' >> tft_beam.TransformDataset())
+
+ coder = example_proto_coder.ExampleProtoCoder(transformed_metadata.schema)
+ _ = (
+ transformed_data
+ | 'SerializeExamples' >> beam.Map(coder.encode)
+ | 'Measure time: end' >> beam.ParDo(MeasureTime(namespace))
+ | 'WriteExamples' >> beam.io.WriteToTFRecord(
+ os.path.join(working_dir, outfile_prefix),
+ file_name_suffix='.gz')
+ )
+ result = pipeline.run()
+ result.wait_until_finish()
+ if metrics_monitor:
+ metrics_monitor.publish_metrics(result)
+
+
+def main():
+ tf.logging.set_verbosity(tf.logging.INFO)
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ '--input',
+ help=('Input BigQuery table to process specified as: '
+ 'DATASET.TABLE'))
+
+ parser.add_argument(
+ '--schema_file', help='File holding the schema for the input data')
+
+ parser.add_argument(
+ '--output_dir',
+ help=('Directory in which transformed examples and function '
+ 'will be emitted.'))
+
+ parser.add_argument(
+ '--outfile_prefix',
+ help='Filename prefix for emitted transformed examples')
+
+ parser.add_argument(
+ '--transform_dir',
+ required=False,
+ default=None,
+ help='Directory in which the transform output is located')
+
+ parser.add_argument(
+ '--max_rows',
+ help='Number of rows to query from BigQuery',
+ default=None,
+ type=int)
+ parser.add_argument(
+ '--publish_to_big_query',
+ help='Whether to publish to BQ',
+ default=None,
+ type=bool)
+
+ parser.add_argument(
+ '--metrics_dataset',
+ help='BQ dataset',
+ default=None,
+ type=str)
+
+ parser.add_argument(
+ '--metrics_table',
+ help='BQ table',
+ default=None,
+ type=str)
+
+ parser.add_argument(
+ '--metric_reporting_project',
+ help='BQ table project',
+ default=None,
+ type=str)
+
+ known_args, pipeline_args = parser.parse_known_args()
+ transform_data(
+ input_handle=known_args.input,
+ outfile_prefix=known_args.outfile_prefix,
+ working_dir=known_args.output_dir,
+ schema_file=known_args.schema_file,
+ transform_dir=known_args.transform_dir,
+ max_rows=known_args.max_rows,
+ pipeline_args=pipeline_args,
+ publish_to_bq=known_args.publish_to_big_query,
+ metrics_dataset=known_args.metrics_dataset,
+ metrics_table=known_args.metrics_table,
+ project=known_args.metric_reporting_project
+ )
+
+
+if __name__ == '__main__':
+ main()
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/process_tfma.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/process_tfma.py
new file mode 100644
index 0000000..e4c1869
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/process_tfma.py
@@ -0,0 +1,193 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Runs a batch job for performing Tensorflow Model Analysis."""
+
+from __future__ import absolute_import, division, print_function
+
+import argparse
+
+import tensorflow as tf
+import tensorflow_model_analysis as tfma
+from tensorflow_model_analysis.evaluators import evaluator
+
+import apache_beam as beam
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
+from apache_beam.testing.load_tests.load_test_metrics_utils import MetricsReader
+from trainer import taxi
+
+
+def process_tfma(schema_file,
+ big_query_table=None,
+ eval_model_dir=None,
+ max_eval_rows=None,
+ pipeline_args=None,
+ publish_to_bq=False,
+ project=None,
+ metrics_table=None,
+ metrics_dataset=None):
+ """Runs a batch job to evaluate the eval_model against the given input.
+
+ Args:
+ schema_file: A file containing a text-serialized Schema that describes the
+ eval data.
+ big_query_table: A BigQuery table name specified as DATASET.TABLE which
+ should be the input for evaluation. This can only be set if input_csv is
+ None.
+ eval_model_dir: A directory where the eval model is located.
+ max_eval_rows: Number of rows to query from BigQuery.
+ pipeline_args: additional DataflowRunner or DirectRunner args passed to
+ the beam pipeline.
+ publish_to_bq:
+ project:
+ metrics_dataset:
+ metrics_table:
+
+ Raises:
+ ValueError: if input_csv and big_query_table are not specified correctly.
+ """
+
+ if big_query_table is None:
+ raise ValueError(
+ '--big_query_table should be provided.')
+
+ slice_spec = [
+ tfma.slicer.SingleSliceSpec(),
+ tfma.slicer.SingleSliceSpec(columns=['trip_start_hour'])
+ ]
+ metrics_namespace = metrics_table
+
+ schema = taxi.read_schema(schema_file)
+
+ eval_shared_model = tfma.default_eval_shared_model(
+ eval_saved_model_path=eval_model_dir,
+ add_metrics_callbacks=[
+ tfma.post_export_metrics.calibration_plot_and_prediction_histogram(),
+ tfma.post_export_metrics.auc_plots()
+ ])
+
+ metrics_monitor = None
+ if publish_to_bq:
+ metrics_monitor = MetricsReader(
+ project_name=project,
+ bq_table=metrics_table,
+ bq_dataset=metrics_dataset,
+ filters=MetricsFilter().with_namespace(metrics_namespace)
+ )
+
+ pipeline = beam.Pipeline(argv=pipeline_args)
+
+ query = taxi.make_sql(big_query_table, max_eval_rows, for_eval=True)
+ raw_feature_spec = taxi.get_raw_feature_spec(schema)
+ raw_data = (
+ pipeline
+ | 'ReadBigQuery' >> beam.io.Read(
+ beam.io.BigQuerySource(query=query, use_standard_sql=True))
+ | 'Measure time: Start' >> beam.ParDo(MeasureTime(metrics_namespace))
+ | 'CleanData' >> beam.Map(lambda x: (
+ taxi.clean_raw_data_dict(x, raw_feature_spec))))
+
+ # Examples must be in clean tf-example format.
+ coder = taxi.make_proto_coder(schema)
+ # Prepare arguments for Extract, Evaluate and Write steps
+ extractors = tfma.default_extractors(
+ eval_shared_model=eval_shared_model,
+ slice_spec=slice_spec,
+ desired_batch_size=None,
+ materialize=False)
+
+ evaluators = tfma.default_evaluators(
+ eval_shared_model=eval_shared_model,
+ desired_batch_size=None,
+ num_bootstrap_samples=1)
+ _ = (
+ raw_data
+ | 'ToSerializedTFExample' >> beam.Map(coder.encode)
+ | 'Extract Results' >> tfma.InputsToExtracts()
+ | 'Extract and evaluate' >> tfma.ExtractAndEvaluate(
+ extractors=extractors,
+ evaluators=evaluators)
+ | 'Map Evaluations to PCollection' >> MapEvalToPCollection()
+ | 'Measure time: End' >> beam.ParDo(
+ MeasureTime(metrics_namespace))
+ )
+ result = pipeline.run()
+ result.wait_until_finish()
+ if metrics_monitor:
+ metrics_monitor.publish_metrics(result)
+
+
+@beam.ptransform_fn
+@beam.typehints.with_input_types(evaluator.Evaluation)
+@beam.typehints.with_output_types(beam.typehints.Any)
+def MapEvalToPCollection( # pylint: disable=invalid-name
+ evaluation):
+ return evaluation['metrics']
+
+
+def main():
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ parser = argparse.ArgumentParser()
+
+ parser.add_argument(
+ '--eval_model_dir',
+ help='Input path to the model which will be evaluated.')
+ parser.add_argument(
+ '--big_query_table',
+ help='BigQuery path to input examples which will be evaluated.')
+ parser.add_argument(
+ '--max_eval_rows',
+ help='Maximum number of rows to evaluate on.',
+ default=None,
+ type=int)
+ parser.add_argument(
+ '--schema_file', help='File holding the schema for the input data')
+ parser.add_argument(
+ '--publish_to_big_query',
+ help='Whether to publish to BQ',
+ default=None,
+ type=bool)
+ parser.add_argument(
+ '--metrics_dataset',
+ help='BQ dataset',
+ default=None,
+ type=str)
+ parser.add_argument(
+ '--metrics_table',
+ help='BQ table for storing metrics',
+ default=None,
+ type=str)
+ parser.add_argument(
+ '--metric_reporting_project',
+ help='BQ table project',
+ default=None,
+ type=str)
+
+ known_args, pipeline_args = parser.parse_known_args()
+
+ process_tfma(
+ big_query_table=known_args.big_query_table,
+ eval_model_dir=known_args.eval_model_dir,
+ max_eval_rows=known_args.max_eval_rows,
+ schema_file=known_args.schema_file,
+ pipeline_args=pipeline_args,
+ publish_to_bq=known_args.publish_to_big_query,
+ metrics_table=known_args.metrics_table,
+ metrics_dataset=known_args.metrics_dataset,
+ project=known_args.metric_reporting_project)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/requirements.txt b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/requirements.txt
new file mode 100644
index 0000000..11c445f
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/requirements.txt
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+jupyter>=1.0,<2
+notebook>=5.7.8,<5.8
+protobuf>=3.7.0,<3.8.0
+tensorflow>=1.13.1
+tensorflow-data-validation>=0.13.1,<0.14
+tensorflow-metadata>=0.13.0,<0.14
+tensorflow-model-analysis>=0.13.2,<0.14
+tensorflow-serving-api>=1.13.0,<1.14
+tensorflow-transform>=0.13.0,<0.14
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/run_chicago.sh b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/run_chicago.sh
new file mode 100755
index 0000000..40154ae
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/run_chicago.sh
@@ -0,0 +1,192 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This script builds a Docker container with the user specified requirements on top of
+# an existing Python worker Docker container (either one you build from source as
+# described in CONTAINERS.md or from a released Docker container).
+set -e
+echo Starting distributed TFDV stats computation and schema generation...
+
+if [[ -z "$1" ]]; then
+ echo "GCS bucket name required"
+ exit 1
+fi
+
+if [[ -z "$2" ]]; then
+ echo "Runner required"
+ exit 1
+fi
+
+if [[ -z "$3" ]]; then
+ echo "SDK location needed"
+ exit 1
+fi
+
+GCS_BUCKET=$1
+RUNNER=$2
+SDK_LOCATION=$3
+
+JOB_ID="chicago-taxi-tfdv-$(date +%Y%m%d-%H%M%S)"
+JOB_OUTPUT_PATH=${GCS_BUCKET}/${JOB_ID}/chicago_taxi_output
+TEMP_PATH=${GCS_BUCKET}/${JOB_ID}/tmp/
+GCP_PROJECT=$(gcloud config list --format 'value(core.project)' 2>/dev/null)
+MAX_ROWS=100000
+JOB_OUTPUT_PATH=${GCS_BUCKET}/${JOB_ID}/chicago_taxi_output
+TFT_OUTPUT_PATH=${JOB_OUTPUT_PATH}/tft_output
+EVAL_RESULT_DIR=${TFT_OUTPUT_PATH}/eval_result_dir
+
+
+# Variables needed for subsequent stages.
+TFDV_OUTPUT_PATH=${JOB_OUTPUT_PATH}/tfdv_output
+SCHEMA_PATH=${TFDV_OUTPUT_PATH}/schema.pbtxt
+
+echo Using GCP project: ${GCP_PROJECT}
+echo Job output path: ${JOB_OUTPUT_PATH}
+echo TFDV output path: ${TFDV_OUTPUT_PATH}
+
+
+# Analyze and validate
+# Compute stats and generate a schema based on the stats.
+
+python tfdv_analyze_and_validate.py \
+ --input bigquery-public-data.chicago_taxi_trips.taxi_trips \
+ --infer_schema \
+ --stats_path ${TFDV_OUTPUT_PATH}/train_stats.tfrecord \
+ --schema_path ${SCHEMA_PATH} \
+ --project ${GCP_PROJECT} \
+ --region us-central1 \
+ --temp_location ${TEMP_PATH} \
+ --experiments shuffle_mode=auto \
+ --job_name ${JOB_ID} \
+ --save_main_session \
+ --runner ${RUNNER} \
+ --max_rows=${MAX_ROWS} \
+ --publish_to_big_query=true \
+ --metrics_dataset='beam_performance' \
+ --metrics_table='tfdv_analyze' \
+ --metric_reporting_project ${GCP_PROJECT} \
+ --sdk_location=${SDK_LOCATION} \
+ --setup_file ./setup.py
+
+EVAL_JOB_ID=${JOB_ID}-eval
+
+# Compute stats for eval data and validate stats against the schema.
+python tfdv_analyze_and_validate.py \
+ --input bigquery-public-data.chicago_taxi_trips.taxi_trips \
+ --for_eval \
+ --schema_path ${SCHEMA_PATH} \
+ --validate_stats \
+ --stats_path ${TFDV_OUTPUT_PATH}/eval_stats.tfrecord \
+ --anomalies_path ${TFDV_OUTPUT_PATH}/anomalies.pbtxt \
+ --project ${GCP_PROJECT} \
+ --region us-central1 \
+ --temp_location ${TEMP_PATH} \
+ --experiments shuffle_mode=auto \
+ --job_name ${EVAL_JOB_ID} \
+ --save_main_session \
+ --runner ${RUNNER} \
+ --max_rows=${MAX_ROWS} \
+ --publish_to_big_query=true \
+ --metrics_dataset='beam_performance' \
+ --metrics_table='chicago_taxi_tfdv_validate' \
+ --sdk_location=${SDK_LOCATION} \
+ --metric_reporting_project ${GCP_PROJECT} \
+ --setup_file ./setup.py
+
+# End analyze and validate
+echo Preprocessing train data...
+
+python preprocess.py \
+ --output_dir ${TFT_OUTPUT_PATH} \
+ --outfile_prefix train_transformed \
+ --input bigquery-public-data.chicago_taxi_trips.taxi_trips \
+ --schema_file ${SCHEMA_PATH} \
+ --project ${GCP_PROJECT} \
+ --region us-central1 \
+ --temp_location ${TEMP_PATH} \
+ --experiments shuffle_mode=auto \
+ --job_name ${JOB_ID} \
+ --runner ${RUNNER} \
+ --max_rows ${MAX_ROWS} \
+ --publish_to_big_query=true \
+ --metrics_dataset='beam_performance' \
+ --metrics_table='chicago_taxi_preprocess' \
+ --sdk_location=${SDK_LOCATION} \
+ --metric_reporting_project ${GCP_PROJECT} \
+ --setup_file ./setup.py
+
+#Train ML engine
+TRAINER_JOB_ID="chicago_taxi_trainer_$(date +%Y%m%d_%H%M%S)"
+TRAIN_OUTPUT_PATH=${JOB_OUTPUT_PATH}/trainer_output
+WORKING_DIR=${TRAIN_OUTPUT_PATH}/working_dir
+
+MODEL_DIR=${TRAIN_OUTPUT_PATH}/model_dir
+# Inputs
+TRAIN_FILE=${TFT_OUTPUT_PATH}/train_transformed-*
+TF_VERSION=1.13
+#workaround for boto in virtualenv, required for the gsutil commands to work:
+export BOTO_CONFIG=/dev/null
+# Start clean, but don't fail if the path does not exist yet.
+gsutil rm ${TRAIN_OUTPUT_PATH} || true
+# Options
+TRAIN_STEPS=10000
+EVAL_STEPS=1000
+# Force a small eval so that the Estimator.train_and_eval() can be used to
+# save the model with its standard paths.
+EVAL_FILE=${TFT_OUTPUT_PATH}/train_transformed-*
+echo Training the model
+gcloud ml-engine jobs submit training ${TRAINER_JOB_ID} \
+ --stream-logs \
+ --job-dir ${MODEL_DIR} \
+ --runtime-version ${TF_VERSION} \
+ --module-name trainer.task \
+ --package-path trainer/ \
+ --region us-central1 \
+ -- \
+ --train-files ${TRAIN_FILE} \
+ --train-steps ${TRAIN_STEPS} \
+ --eval-files ${EVAL_FILE} \
+ --eval-steps ${EVAL_STEPS} \
+ --output-dir ${WORKING_DIR} \
+ --schema-file ${SCHEMA_PATH} \
+ --tf-transform-dir ${TFT_OUTPUT_PATH}
+
+# We evaluate with the last eval model written (hence tail -n1)
+EVAL_MODEL_DIR=${TRAIN_OUTPUT_PATH}/working_dir/eval_model_dir
+LAST_EVAL_MODEL_DIR=$(gsutil ls ${EVAL_MODEL_DIR} | tail -n1)
+
+echo Eval model dir: ${EVAL_MODEL_DIR}
+
+python process_tfma.py \
+ --big_query_table bigquery-public-data.chicago_taxi_trips.taxi_trips \
+ --schema_file ${SCHEMA_PATH} \
+ --eval_model_dir ${LAST_EVAL_MODEL_DIR} \
+ --project ${GCP_PROJECT} \
+ --region us-central1 \
+ --temp_location ${GCS_BUCKET}/${JOB_ID}/tmp/ \
+ --experiments shuffle_mode=auto \
+ --job_name ${JOB_ID} \
+ --save_main_session \
+ --runner ${RUNNER} \
+ --max_eval_rows=${MAX_ROWS} \
+ --publish_to_big_query=true \
+ --metrics_dataset='beam_performance' \
+ --metrics_table='chicago_taxi_process_tfma' \
+ --sdk_location=${SDK_LOCATION} \
+ --metric_reporting_project ${GCP_PROJECT} \
+ --setup_file ./setup.py
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/setup.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/setup.py
new file mode 100644
index 0000000..b96328f
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/setup.py
@@ -0,0 +1,41 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Setup dependencies for local and cloud deployment."""
+from __future__ import absolute_import
+
+import setuptools
+
+# LINT.IfChange
+TF_VERSION = '1.13.1'
+# LINT.ThenChange(train_mlengine.sh, start_model_server_mlengine.sh)
+
+if __name__ == '__main__':
+ setuptools.setup(
+ name='beam_chicago_taxi',
+ version='0.13.0',
+ packages=setuptools.find_packages(),
+ install_requires=[
+ 'jupyter>=1.0,<2',
+ 'notebook>=5.7.8,<5.8',
+ 'numpy>=1.14.5,<2',
+ 'protobuf>=3.7.0,<3.8.0',
+ 'tensorflow>=' + TF_VERSION,
+ 'tensorflow-data-validation>=0.13.1,<0.14',
+ 'tensorflow-metadata>=0.13.0,<0.14',
+ 'tensorflow-model-analysis>=0.13.2,<0.14',
+ 'tensorflow-serving-api>=1.13.0,<1.14',
+ 'tensorflow-transform>=0.13.0,<0.14',
+ ],
+ python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<4',
+ )
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/tfdv_analyze_and_validate.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/tfdv_analyze_and_validate.py
new file mode 100644
index 0000000..e5a1e10
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/tfdv_analyze_and_validate.py
@@ -0,0 +1,225 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Compute stats, infer schema, and validate stats for chicago taxi example."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+
+import numpy as np
+import tensorflow as tf
+from tensorflow.python.lib.io import file_io
+import tensorflow_data_validation as tfdv
+from tensorflow_metadata.proto.v0 import statistics_pb2
+
+import apache_beam as beam
+from apache_beam.metrics.metric import MetricsFilter
+from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
+from apache_beam.testing.load_tests.load_test_metrics_utils import MetricsReader
+
+from google.protobuf import text_format
+from trainer import taxi
+
+
+def infer_schema(stats_path, schema_path):
+ """Infers a schema from stats in stats_path.
+
+ Args:
+ stats_path: Location of the stats used to infer the schema.
+ schema_path: Location where the inferred schema is materialized.
+ """
+ print('Infering schema from statistics.')
+ schema = tfdv.infer_schema(
+ tfdv.load_statistics(stats_path), infer_feature_shape=False)
+ print(text_format.MessageToString(schema))
+
+ print('Writing schema to output path.')
+ file_io.write_string_to_file(schema_path, text_format.MessageToString(schema))
+
+
+def validate_stats(stats_path, schema_path, anomalies_path):
+ """Validates the statistics against the schema and materializes anomalies.
+
+ Args:
+ stats_path: Location of the stats used to infer the schema.
+ schema_path: Location of the schema to be used for validation.
+ anomalies_path: Location where the detected anomalies are materialized.
+ """
+ print('Validating schema against the computed statistics.')
+ schema = taxi.read_schema(schema_path)
+
+ stats = tfdv.load_statistics(stats_path)
+ anomalies = tfdv.validate_statistics(stats, schema)
+ print('Detected following anomalies:')
+ print(text_format.MessageToString(anomalies))
+
+ print('Writing anomalies to anomalies path.')
+ file_io.write_string_to_file(anomalies_path,
+ text_format.MessageToString(anomalies))
+
+
+def compute_stats(input_handle,
+ stats_path,
+ max_rows=None,
+ for_eval=False,
+ pipeline_args=None,
+ publish_to_bq=None,
+ metrics_dataset=None,
+ metrics_table=None,
+ project=None):
+ """Computes statistics on the input data.
+
+ Args:
+ input_handle: BigQuery table name to process specified as DATASET.TABLE or
+ path to csv file with input data.
+ stats_path: Directory in which stats are materialized.
+ max_rows: Number of rows to query from BigQuery
+ for_eval: Query for eval set rows from BigQuery
+ pipeline_args: additional DataflowRunner or DirectRunner args passed to the
+ beam pipeline.
+ """
+ namespace = metrics_table
+ pipeline = beam.Pipeline(argv=pipeline_args)
+ metrics_monitor = None
+ if publish_to_bq:
+ metrics_monitor = MetricsReader(
+ project_name=project,
+ bq_table=metrics_table,
+ bq_dataset=metrics_dataset,
+ filters=MetricsFilter().with_namespace(namespace),
+ )
+
+ query = taxi.make_sql(
+ table_name=input_handle, max_rows=max_rows, for_eval=for_eval)
+ raw_data = (
+ pipeline
+ | 'ReadBigQuery' >> beam.io.Read(
+ beam.io.BigQuerySource(query=query, use_standard_sql=True))
+ | 'Measure time: Start' >> beam.ParDo(MeasureTime(namespace))
+ | 'ConvertToTFDVInput' >> beam.Map(
+ lambda x: {key: np.asarray([x[key]])
+ for key in x if x[key] is not None}))
+
+ _ = (
+ raw_data
+ | 'GenerateStatistics' >> tfdv.GenerateStatistics()
+ | 'Measure time: End' >> beam.ParDo(MeasureTime(namespace))
+ | 'WriteStatsOutput' >> beam.io.WriteToTFRecord(
+ stats_path,
+ shard_name_template='',
+ coder=beam.coders.ProtoCoder(
+ statistics_pb2.DatasetFeatureStatisticsList)))
+ result = pipeline.run()
+ result.wait_until_finish()
+ if metrics_monitor:
+ metrics_monitor.publish_metrics(result)
+
+
+def main():
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ '--input',
+ help=('Input BigQuery table to process specified as: '
+ 'DATASET.TABLE or path to csv file with input data.'))
+
+ parser.add_argument(
+ '--stats_path',
+ help='Location for the computed stats to be materialized.')
+
+ parser.add_argument(
+ '--for_eval',
+ help='Query for eval set rows from BigQuery',
+ action='store_true')
+
+ parser.add_argument(
+ '--max_rows',
+ help='Number of rows to query from BigQuery',
+ default=None,
+ type=int)
+
+ parser.add_argument(
+ '--schema_path',
+ help='Location for the computed schema is located.',
+ default=None,
+ type=str)
+
+ parser.add_argument(
+ '--infer_schema',
+ help='If specified, also infers a schema based on the computed stats.',
+ action='store_true')
+
+ parser.add_argument(
+ '--validate_stats',
+ help='If specified, also validates the stats against the schema.',
+ action='store_true')
+
+ parser.add_argument(
+ '--anomalies_path',
+ help='Location for detected anomalies are materialized.',
+ default=None,
+ type=str)
+
+ parser.add_argument(
+ '--publish_to_big_query',
+ help='Whether to publish to BQ',
+ default=None,
+ type=bool)
+
+ parser.add_argument(
+ '--metrics_dataset',
+ help='BQ dataset',
+ default=None,
+ type=str)
+
+ parser.add_argument(
+ '--metrics_table',
+ help='BQ table',
+ default=None,
+ type=str)
+
+ parser.add_argument(
+ '--metric_reporting_project',
+ help='BQ table project',
+ default=None,
+ type=str)
+
+ known_args, pipeline_args = parser.parse_known_args()
+ compute_stats(
+ input_handle=known_args.input,
+ stats_path=known_args.stats_path,
+ max_rows=known_args.max_rows,
+ for_eval=known_args.for_eval,
+ pipeline_args=pipeline_args,
+ publish_to_bq=known_args.publish_to_big_query,
+ metrics_dataset=known_args.metrics_dataset,
+ metrics_table=known_args.metrics_table,
+ project=known_args.metric_reporting_project)
+ print('Stats computation done.')
+
+ if known_args.infer_schema:
+ infer_schema(
+ stats_path=known_args.stats_path, schema_path=known_args.schema_path)
+
+ if known_args.validate_stats:
+ validate_stats(
+ stats_path=known_args.stats_path,
+ schema_path=known_args.schema_path,
+ anomalies_path=known_args.anomalies_path)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/__init__.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/__init__.py
new file mode 100644
index 0000000..cce3aca
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/__init__.py
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/model.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/model.py
new file mode 100644
index 0000000..f04d0a8
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/model.py
@@ -0,0 +1,164 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Defines the model used to predict who will tip in the Chicago Taxi demo."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import tensorflow as tf
+
+import tensorflow_model_analysis as tfma
+from trainer import taxi
+
+
+def build_estimator(tf_transform_output, config, hidden_units=None):
+ """Build an estimator for predicting the tipping behavior of taxi riders.
+
+ Args:
+ tf_transform_output: A TFTransformOutput.
+ config: tf.contrib.learn.RunConfig defining the runtime environment for the
+ estimator (including model_dir).
+ hidden_units: [int], the layer sizes of the DNN (input layer first)
+
+ Returns:
+ Resulting DNNLinearCombinedClassifier.
+ """
+ transformed_feature_spec = (
+ tf_transform_output.transformed_feature_spec().copy())
+
+ transformed_feature_spec.pop(taxi.transformed_name(taxi.LABEL_KEY))
+
+ real_valued_columns = [
+ tf.feature_column.numeric_column(key, shape=())
+ for key in taxi.transformed_names(taxi.DENSE_FLOAT_FEATURE_KEYS)
+ ]
+ categorical_columns = [
+ tf.feature_column.categorical_column_with_identity(
+ key, num_buckets=taxi.VOCAB_SIZE + taxi.OOV_SIZE, default_value=0)
+ for key in taxi.transformed_names(taxi.VOCAB_FEATURE_KEYS)
+ ]
+ categorical_columns += [
+ tf.feature_column.categorical_column_with_identity(
+ key, num_buckets=taxi.FEATURE_BUCKET_COUNT, default_value=0)
+ for key in taxi.transformed_names(taxi.BUCKET_FEATURE_KEYS)
+ ]
+ categorical_columns += [
+ tf.feature_column.categorical_column_with_identity(
+ key, num_buckets=num_buckets, default_value=0)
+ for key, num_buckets in zip(
+ taxi.transformed_names(taxi.CATEGORICAL_FEATURE_KEYS), #
+ taxi.MAX_CATEGORICAL_FEATURE_VALUES)
+ ]
+ return tf.estimator.DNNLinearCombinedClassifier(
+ config=config,
+ linear_feature_columns=categorical_columns,
+ dnn_feature_columns=real_valued_columns,
+ dnn_hidden_units=hidden_units or [100, 70, 50, 25])
+
+
+def example_serving_receiver_fn(tf_transform_output, schema):
+ """Build the serving in inputs.
+
+ Args:
+ tf_transform_output: A TFTransformOutput.
+ schema: the schema of the input data.
+
+ Returns:
+ Tensorflow graph which parses examples, applying tf-transform to them.
+ """
+ raw_feature_spec = taxi.get_raw_feature_spec(schema)
+ raw_feature_spec.pop(taxi.LABEL_KEY)
+
+ raw_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(
+ raw_feature_spec, default_batch_size=None)
+ serving_input_receiver = raw_input_fn()
+
+ transformed_features = tf_transform_output.transform_raw_features(
+ serving_input_receiver.features)
+
+ return tf.estimator.export.ServingInputReceiver(
+ transformed_features, serving_input_receiver.receiver_tensors)
+
+
+def eval_input_receiver_fn(tf_transform_output, schema):
+ """Build everything needed for the tf-model-analysis to run the model.
+
+ Args:
+ tf_transform_output: A TFTransformOutput.
+ schema: the schema of the input data.
+
+ Returns:
+ EvalInputReceiver function, which contains:
+ - Tensorflow graph which parses raw untranformed features, applies the
+ tf-transform preprocessing operators.
+ - Set of raw, untransformed features.
+ - Label against which predictions will be compared.
+ """
+ # Notice that the inputs are raw features, not transformed features here.
+ raw_feature_spec = taxi.get_raw_feature_spec(schema)
+
+ serialized_tf_example = tf.placeholder(
+ dtype=tf.string, shape=[None], name='input_example_tensor')
+
+ # Add a parse_example operator to the tensorflow graph, which will parse
+ # raw, untransformed, tf examples.
+ features = tf.parse_example(serialized_tf_example, raw_feature_spec)
+
+ # Now that we have our raw examples, process them through the tf-transform
+ # function computed during the preprocessing step.
+ transformed_features = tf_transform_output.transform_raw_features(
+ features)
+
+ # The key name MUST be 'examples'.
+ receiver_tensors = {'examples': serialized_tf_example}
+
+ # NOTE: Model is driven by transformed features (since training works on the
+ # materialized output of TFT, but slicing will happen on raw features.
+ features.update(transformed_features)
+
+ return tfma.export.EvalInputReceiver(
+ features=features,
+ receiver_tensors=receiver_tensors,
+ labels=transformed_features[taxi.transformed_name(taxi.LABEL_KEY)])
+
+
+def _gzip_reader_fn():
+ """Small utility returning a record reader that can read gzip'ed files."""
+ return tf.TFRecordReader(
+ options=tf.python_io.TFRecordOptions(
+ compression_type=tf.python_io.TFRecordCompressionType.GZIP))
+
+
+def input_fn(filenames, tf_transform_output, batch_size=200):
+ """Generates features and labels for training or evaluation.
+
+ Args:
+ filenames: [str] list of CSV files to read data from.
+ tf_transform_output: A TFTransformOutput.
+ batch_size: int First dimension size of the Tensors returned by input_fn
+
+ Returns:
+ A (features, indices) tuple where features is a dictionary of
+ Tensors, and indices is a single Tensor of label indices.
+ """
+ transformed_feature_spec = (
+ tf_transform_output.transformed_feature_spec().copy())
+
+ transformed_features = tf.contrib.learn.io.read_batch_features(
+ filenames, batch_size, transformed_feature_spec, reader=_gzip_reader_fn)
+
+ # We pop the label because we do not want to use it as a feature while we're
+ # training.
+ return transformed_features, transformed_features.pop(
+ taxi.transformed_name(taxi.LABEL_KEY))
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/task.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/task.py
new file mode 100644
index 0000000..99dd49e
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/task.py
@@ -0,0 +1,189 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Trainer for the chicago_taxi demo."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import os
+
+import tensorflow as tf
+import tensorflow_model_analysis as tfma
+import tensorflow_transform as tft
+from trainer import model
+from trainer import taxi
+
+SERVING_MODEL_DIR = 'serving_model_dir'
+EVAL_MODEL_DIR = 'eval_model_dir'
+
+TRAIN_BATCH_SIZE = 40
+EVAL_BATCH_SIZE = 40
+
+# Number of nodes in the first layer of the DNN
+FIRST_DNN_LAYER_SIZE = 100
+NUM_DNN_LAYERS = 4
+DNN_DECAY_FACTOR = 0.7
+
+
+def train_and_maybe_evaluate(hparams):
+ """Run the training and evaluate using the high level API.
+
+ Args:
+ hparams: Holds hyperparameters used to train the model as name/value pairs.
+
+ Returns:
+ The estimator that was used for training (and maybe eval)
+ """
+ schema = taxi.read_schema(hparams.schema_file)
+ tf_transform_output = tft.TFTransformOutput(hparams.tf_transform_dir)
+
+ train_input = lambda: model.input_fn(
+ hparams.train_files,
+ tf_transform_output,
+ batch_size=TRAIN_BATCH_SIZE
+ )
+
+ eval_input = lambda: model.input_fn(
+ hparams.eval_files,
+ tf_transform_output,
+ batch_size=EVAL_BATCH_SIZE
+ )
+
+ train_spec = tf.estimator.TrainSpec(
+ train_input, max_steps=hparams.train_steps)
+
+ serving_receiver_fn = lambda: model.example_serving_receiver_fn(
+ tf_transform_output, schema)
+
+ exporter = tf.estimator.FinalExporter('chicago-taxi', serving_receiver_fn)
+ eval_spec = tf.estimator.EvalSpec(
+ eval_input,
+ steps=hparams.eval_steps,
+ exporters=[exporter],
+ name='chicago-taxi-eval')
+
+ run_config = tf.estimator.RunConfig(
+ save_checkpoints_steps=999, keep_checkpoint_max=1)
+
+ serving_model_dir = os.path.join(hparams.output_dir, SERVING_MODEL_DIR)
+ run_config = run_config.replace(model_dir=serving_model_dir)
+
+ estimator = model.build_estimator(
+ tf_transform_output,
+
+ # Construct layers sizes with exponetial decay
+ hidden_units=[
+ max(2, int(FIRST_DNN_LAYER_SIZE * DNN_DECAY_FACTOR**i))
+ for i in range(NUM_DNN_LAYERS)
+ ],
+ config=run_config)
+
+ tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
+
+ return estimator
+
+
+def run_experiment(hparams):
+ """Train the model then export it for tf.model_analysis evaluation.
+
+ Args:
+ hparams: Holds hyperparameters used to train the model as name/value pairs.
+ """
+ estimator = train_and_maybe_evaluate(hparams)
+
+ schema = taxi.read_schema(hparams.schema_file)
+ tf_transform_output = tft.TFTransformOutput(hparams.tf_transform_dir)
+
+ # Save a model for tfma eval
+ eval_model_dir = os.path.join(hparams.output_dir, EVAL_MODEL_DIR)
+
+ receiver_fn = lambda: model.eval_input_receiver_fn(
+ tf_transform_output, schema)
+
+ tfma.export.export_eval_savedmodel(
+ estimator=estimator,
+ export_dir_base=eval_model_dir,
+ eval_input_receiver_fn=receiver_fn)
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ # Input Arguments
+ parser.add_argument(
+ '--train-files',
+ help='GCS or local paths to training data',
+ nargs='+',
+ required=True)
+
+ parser.add_argument(
+ '--tf-transform-dir',
+ help='Tf-transform directory with model from preprocessing step',
+ required=True)
+
+ parser.add_argument(
+ '--output-dir',
+ help="""\
+ Directory under which which the serving model (under /serving_model_dir)\
+ and the tf-mode-analysis model (under /eval_model_dir) will be written\
+ """,
+ required=True)
+
+ parser.add_argument(
+ '--eval-files',
+ help='GCS or local paths to evaluation data',
+ nargs='+',
+ required=True)
+ # Training arguments
+ parser.add_argument(
+ '--job-dir',
+ help='GCS location to write checkpoints and export models',
+ required=True)
+
+ # Argument to turn on all logging
+ parser.add_argument(
+ '--verbosity',
+ choices=['DEBUG', 'ERROR', 'FATAL', 'INFO', 'WARN'],
+ default='INFO',
+ )
+ # Experiment arguments
+ parser.add_argument(
+ '--train-steps',
+ help='Count of steps to run the training job for',
+ required=True,
+ type=int)
+ parser.add_argument(
+ '--eval-steps',
+ help='Number of steps to run evalution for at each checkpoint',
+ default=100,
+ type=int)
+ parser.add_argument(
+ '--schema-file',
+ help='File holding the schema for the input data')
+
+ args = parser.parse_args()
+
+ # Set python level verbosity
+ tf.logging.set_verbosity(args.verbosity)
+ # Set C++ Graph Execution level verbosity
+ os.environ['TF_CPP_MIN_LOG_LEVEL'] = str(
+ tf.logging.__dict__[args.verbosity] / 10)
+
+ # Run the training job
+ hparams = tf.contrib.training.HParams(**args.__dict__)
+ run_experiment(hparams)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/taxi.py b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/taxi.py
new file mode 100644
index 0000000..5bf3191
--- /dev/null
+++ b/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/trainer/taxi.py
@@ -0,0 +1,186 @@
+# Copyright 2019 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Utility and schema methods for the chicago_taxi sample."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorflow_transform import coders as tft_coders
+from tensorflow_transform.tf_metadata import dataset_schema
+from tensorflow_transform.tf_metadata import schema_utils
+
+from google.protobuf import text_format
+from tensorflow.python.lib.io import file_io
+from tensorflow_metadata.proto.v0 import schema_pb2
+
+# Categorical features are assumed to each have a maximum value in the dataset.
+MAX_CATEGORICAL_FEATURE_VALUES = [24, 31, 12]
+
+CATEGORICAL_FEATURE_KEYS = [
+ 'trip_start_hour', 'trip_start_day', 'trip_start_month',
+ 'pickup_census_tract', 'dropoff_census_tract', 'pickup_community_area',
+ 'dropoff_community_area'
+]
+
+DENSE_FLOAT_FEATURE_KEYS = ['trip_miles', 'fare', 'trip_seconds']
+
+# Number of buckets used by tf.transform for encoding each feature.
+FEATURE_BUCKET_COUNT = 10
+
+BUCKET_FEATURE_KEYS = [
+ 'pickup_latitude', 'pickup_longitude', 'dropoff_latitude',
+ 'dropoff_longitude'
+]
+
+# Number of vocabulary terms used for encoding VOCAB_FEATURES by tf.transform
+VOCAB_SIZE = 1000
+
+# Count of out-of-vocab buckets in which unrecognized VOCAB_FEATURES are hashed.
+OOV_SIZE = 10
+
+VOCAB_FEATURE_KEYS = [
+ 'payment_type',
+ 'company',
+]
+
+LABEL_KEY = 'tips'
+FARE_KEY = 'fare'
+
+CSV_COLUMN_NAMES = [
+ 'pickup_community_area',
+ 'fare',
+ 'trip_start_month',
+ 'trip_start_hour',
+ 'trip_start_day',
+ 'trip_start_timestamp',
+ 'pickup_latitude',
+ 'pickup_longitude',
+ 'dropoff_latitude',
+ 'dropoff_longitude',
+ 'trip_miles',
+ 'pickup_census_tract',
+ 'dropoff_census_tract',
+ 'payment_type',
+ 'company',
+ 'trip_seconds',
+ 'dropoff_community_area',
+ 'tips',
+]
+
+
+def transformed_name(key):
+ return key + '_xf'
+
+
+def transformed_names(keys):
+ return [transformed_name(key) for key in keys]
+
+
+# Tf.Transform considers these features as "raw"
+def get_raw_feature_spec(schema):
+ return schema_utils.schema_as_feature_spec(schema).feature_spec
+
+
+def make_proto_coder(schema):
+ raw_feature_spec = get_raw_feature_spec(schema)
+ raw_schema = dataset_schema.from_feature_spec(raw_feature_spec)
+ return tft_coders.ExampleProtoCoder(raw_schema)
+
+
+def make_csv_coder(schema):
+ """Return a coder for tf.transform to read csv files."""
+ raw_feature_spec = get_raw_feature_spec(schema)
+ parsing_schema = dataset_schema.from_feature_spec(raw_feature_spec)
+ return tft_coders.CsvCoder(CSV_COLUMN_NAMES, parsing_schema)
+
+
+def clean_raw_data_dict(input_dict, raw_feature_spec):
+ """Clean raw data dict."""
+ output_dict = {}
+
+ for key in raw_feature_spec:
+ if key not in input_dict or not input_dict[key]:
+ output_dict[key] = []
+ else:
+ output_dict[key] = [input_dict[key]]
+ return output_dict
+
+
+def make_sql(table_name, max_rows=None, for_eval=False):
+ """Creates the sql command for pulling data from BigQuery.
+
+ Args:
+ table_name: BigQuery table name
+ max_rows: if set, limits the number of rows pulled from BigQuery
+ for_eval: True if this is for evaluation, false otherwise
+
+ Returns:
+ sql command as string
+ """
+ if for_eval:
+ # 1/3 of the dataset used for eval
+ where_clause = 'WHERE MOD(FARM_FINGERPRINT(unique_key), 3) = 0 ' \
+ 'AND pickup_latitude is not null AND pickup_longitude ' \
+ 'is not null AND dropoff_latitude is not null ' \
+ 'AND dropoff_longitude is not null'
+ else:
+ # 2/3 of the dataset used for training
+ where_clause = 'WHERE MOD(FARM_FINGERPRINT(unique_key), 3) > 0 ' \
+ 'AND pickup_latitude is not null AND pickup_longitude ' \
+ 'is not null AND dropoff_latitude is not null ' \
+ 'AND dropoff_longitude is not null'
+
+ limit_clause = ''
+ if max_rows:
+ limit_clause = 'LIMIT {max_rows}'.format(max_rows=max_rows)
+ return """
+ SELECT
+ CAST(pickup_community_area AS string) AS pickup_community_area,
+ CAST(dropoff_community_area AS string) AS dropoff_community_area,
+ CAST(pickup_census_tract AS string) AS pickup_census_tract,
+ CAST(dropoff_census_tract AS string) AS dropoff_census_tract,
+ fare,
+ EXTRACT(MONTH FROM trip_start_timestamp) AS trip_start_month,
+ EXTRACT(HOUR FROM trip_start_timestamp) AS trip_start_hour,
+ EXTRACT(DAYOFWEEK FROM trip_start_timestamp) AS trip_start_day,
+ UNIX_SECONDS(trip_start_timestamp) AS trip_start_timestamp,
+ pickup_latitude,
+ pickup_longitude,
+ dropoff_latitude,
+ dropoff_longitude,
+ trip_miles,
+ payment_type,
+ company,
+ trip_seconds,
+ tips
+ FROM `{table_name}`
+ {where_clause}
+ {limit_clause}
+""".format(
+ table_name=table_name, where_clause=where_clause, limit_clause=limit_clause)
+
+
+def read_schema(path):
+ """Reads a schema from the provided location.
+
+ Args:
+ path: The location of the file holding a serialized Schema proto.
+
+ Returns:
+ An instance of Schema or None if the input argument is None
+ """
+ result = schema_pb2.Schema()
+ contents = file_io.read_file_to_string(path)
+ text_format.Parse(contents, result)
+ return result
diff --git a/sdks/python/apache_beam/testing/datatype_inference.py b/sdks/python/apache_beam/testing/datatype_inference.py
new file mode 100644
index 0000000..b1a689c
--- /dev/null
+++ b/sdks/python/apache_beam/testing/datatype_inference.py
@@ -0,0 +1,144 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+
+import array
+import json
+from collections import OrderedDict
+
+import numpy as np
+from past.builtins import unicode
+
+from apache_beam.typehints import trivial_inference
+from apache_beam.typehints import typehints
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+try:
+ import pyarrow as pa
+except ImportError:
+ pa = None
+
+
+def infer_element_type(elements):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ Infer a Beam type for a list of elements.
+
+ Args:
+ elements (List[Any]): A list of elements for which the type should be
+ inferred.
+
+ Returns:
+ A Beam type encompassing all elements.
+ """
+ element_type = typehints.Union[[
+ trivial_inference.instance_to_type(e) for e in elements
+ ]]
+ return element_type
+
+
+def infer_typehints_schema(data):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ Infer Beam types for tabular data.
+
+ Args:
+ data (List[dict]): A list of dictionaries representing rows in a table.
+
+ Returns:
+ An OrderedDict mapping column names to Beam types.
+ """
+ column_data = OrderedDict()
+ for row in data:
+ for key, value in row.items():
+ column_data.setdefault(key, []).append(value)
+ column_types = OrderedDict([
+ (key, infer_element_type(values)) for key, values in column_data.items()
+ ])
+ return column_types
+
+
+def infer_avro_schema(data, use_fastavro=False):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ Infer avro schema for tabular data.
+
+ Args:
+ data (List[dict]): A list of dictionaries representing rows in a table.
+ use_fastavro (bool): A flag indicating whether the schema should be
+ constructed using fastavro.
+
+ Returns:
+ An avro schema object.
+ """
+ _typehint_to_avro_type = {
+ type(None): "null",
+ int: "int",
+ float: "double",
+ str: "string",
+ unicode: "string",
+ bytes: "bytes",
+ np.ndarray: "bytes",
+ array.array: "bytes",
+ }
+
+ def typehint_to_avro_type(value):
+ if isinstance(value, typehints.UnionConstraint):
+ return sorted(
+ typehint_to_avro_type(union_type) for union_type in value.union_types)
+ else:
+ return _typehint_to_avro_type[value]
+
+ column_types = infer_typehints_schema(data)
+ avro_fields = [{"name": str(key), "type": typehint_to_avro_type(value)}
+ for key, value in column_types.items()]
+ schema_dict = {
+ "namespace": "example.avro", "name": "User", "type": "record",
+ "fields": avro_fields
+ }
+ if use_fastavro:
+ from fastavro import parse_schema
+ return parse_schema(schema_dict)
+ else:
+ return Parse(json.dumps(schema_dict))
+
+
+def infer_pyarrow_schema(data):
+ """For internal use only; no backwards-compatibility guarantees.
+
+ Infer PyArrow schema for tabular data.
+
+ Args:
+ data (List[dict]): A list of dictionaries representing rows in a table.
+
+ Returns:
+ A PyArrow schema object.
+ """
+ column_data = OrderedDict()
+ for row in data:
+ for key, value in row.items():
+ column_data.setdefault(key, []).append(value)
+ column_types = OrderedDict([
+ (key, pa.array(value).type) for key, value in column_data.items()
+ ])
+ return pa.schema(list(column_types.items()))
diff --git a/sdks/python/apache_beam/testing/datatype_inference_test.py b/sdks/python/apache_beam/testing/datatype_inference_test.py
new file mode 100644
index 0000000..131eafb
--- /dev/null
+++ b/sdks/python/apache_beam/testing/datatype_inference_test.py
@@ -0,0 +1,207 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+
+import logging
+import unittest
+from collections import OrderedDict
+
+import numpy as np
+from parameterized import parameterized
+from past.builtins import unicode
+
+from apache_beam.testing import datatype_inference
+from apache_beam.typehints import typehints
+
+try:
+ import pyarrow as pa
+except ImportError:
+ pa = None
+
+TEST_DATA = [
+ {
+ "name": "empty",
+ "data": [],
+ "type_schema": OrderedDict([]),
+ "pyarrow_schema": pa.schema([]) if pa is not None else None,
+ "avro_schema": {
+ "namespace": "example.avro",
+ "name": "User",
+ "type": "record",
+ "fields": [],
+ },
+ },
+ {
+ "name":
+ "main",
+ "data": [
+ OrderedDict([
+ ("a", 1),
+ ("b", 0.12345),
+ ("c", u"Hello World!!"),
+ ("d", np.array([1, 2, 3])),
+ ("e", b"some bytes"),
+ ]),
+ OrderedDict([
+ ("a", -5),
+ ("b", 1234.567),
+ ("e", b"more bytes"),
+ ]),
+ OrderedDict([
+ ("a", 100000),
+ ("c", u"XoXoX"),
+ ("d", np.array([4, 5, 6])),
+ ("e", b""),
+ ]),
+ ],
+ "type_schema":
+ OrderedDict([
+ ("a", int),
+ ("b", float),
+ ("c", unicode),
+ ("d", np.ndarray),
+ ("e", bytes),
+ ]),
+ "pyarrow_schema":
+ pa.schema([
+ ("a", pa.int64()),
+ ("b", pa.float64()),
+ ("c", pa.string()),
+ ("d", pa.list_(pa.int64())),
+ ("e", pa.binary()),
+ ]) if pa is not None else None,
+ "avro_schema": {
+ "namespace":
+ "example.avro",
+ "name":
+ "User",
+ "type":
+ "record",
+ "fields": [
+ {"name": "a", "type": "int"},
+ {"name": "b", "type": "double"},
+ {"name": "c", "type": "string"},
+ {"name": "d", "type": "bytes"},
+ {"name": "e", "type": "bytes"},
+ ],
+ },
+ },
+]
+
+
+def nullify_data_and_schemas(test_data):
+ """Add a row with all columns set to None and adjust the schemas accordingly.
+ """
+
+ def nullify_avro_schema(schema):
+ """Add a 'null' type to every field."""
+ schema = schema.copy()
+ new_fields = []
+ for field in schema["fields"]:
+ if isinstance(field["type"], str):
+ new_fields.append(
+ {"name": field["name"], "type": sorted([field["type"], "null"])})
+ else:
+ new_fields.append(
+ {"name": field["name"], "type": sorted(field["type"] + ["null"])})
+ schema["fields"] = new_fields
+ return schema
+
+ def get_collumns_in_order(test_data):
+ """Get a list of columns while trying to maintain original order.
+
+ .. note::
+ Columns which do not apear until later rows are added to the end,
+ even if they preceed some columns which have already been added.
+ """
+ _seen = set()
+ columns = [
+ c for test_case in test_data for row in test_case["data"]
+ for c in row
+ if c not in _seen and not _seen.add(c)
+ ]
+ return columns
+
+ nullified_test_data = []
+ columns = get_collumns_in_order(test_data)
+ for test_case in test_data:
+ if not test_case["data"]:
+ continue
+ test_case = test_case.copy()
+ test_case["name"] = test_case["name"] + "_nullified"
+ test_case["data"] = test_case["data"] + [
+ OrderedDict([(c, None) for c in columns])
+ ]
+ test_case["type_schema"] = OrderedDict([
+ (k, typehints.Union[v, type(None)])
+ for k, v in test_case["type_schema"].items()
+ ])
+ test_case["avro_schema"] = nullify_avro_schema(test_case["avro_schema"])
+ nullified_test_data.append(test_case)
+ return nullified_test_data
+
+
+TEST_DATA += nullify_data_and_schemas(TEST_DATA)
+
+
+class DatatypeInferenceTest(unittest.TestCase):
+
+ @parameterized.expand([
+ (d["name"], d["data"], d["type_schema"]) for d in TEST_DATA
+ ])
+ def test_infer_typehints_schema(self, _, data, schema):
+ typehints_schema = datatype_inference.infer_typehints_schema(data)
+ self.assertEqual(typehints_schema, schema)
+
+ @parameterized.expand([
+ (d["name"], d["data"], d["pyarrow_schema"]) for d in TEST_DATA
+ ])
+ @unittest.skipIf(pa is None, "PyArrow is not installed")
+ def test_infer_pyarrow_schema(self, _, data, schema):
+ pyarrow_schema = datatype_inference.infer_pyarrow_schema(data)
+ self.assertEqual(pyarrow_schema, schema)
+
+ @parameterized.expand([
+ (d["name"], d["data"], d["avro_schema"]) for d in TEST_DATA
+ ])
+ def test_infer_avro_schema(self, _, data, schema):
+ schema = schema.copy() # Otherwise, it would be mutated by `.pop()`
+ avro_schema = datatype_inference.infer_avro_schema(data, use_fastavro=False)
+ avro_schema = avro_schema.to_json()
+ fields1 = avro_schema.pop("fields")
+ fields2 = schema.pop("fields")
+ self.assertDictEqual(avro_schema, schema)
+ for field1, field2 in zip(fields1, fields2):
+ self.assertDictEqual(field1, field2)
+
+ @parameterized.expand([
+ (d["name"], d["data"], d["avro_schema"]) for d in TEST_DATA
+ ])
+ def test_infer_fastavro_schema(self, _, data, schema):
+ from fastavro import parse_schema
+ schema = parse_schema(schema)
+ avro_schema = datatype_inference.infer_avro_schema(data, use_fastavro=True)
+ fields1 = avro_schema.pop("fields")
+ fields2 = schema.pop("fields")
+ self.assertDictEqual(avro_schema, schema)
+ for field1, field2 in zip(fields1, fields2):
+ self.assertDictEqual(field1, field2)
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/testing/extra_assertions.py b/sdks/python/apache_beam/testing/extra_assertions.py
new file mode 100644
index 0000000..53a9eeb
--- /dev/null
+++ b/sdks/python/apache_beam/testing/extra_assertions.py
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+
+import sys
+
+import numpy as np
+
+
+class ExtraAssertionsMixin(object):
+
+ if sys.version_info[0] < 3:
+
+ def assertCountEqual(self, first, second, msg=None):
+ """Assert that two containers have the same number of the same items in
+ any order.
+ """
+ return self.assertItemsEqual(first, second, msg=msg)
+
+ def assertUnhashableCountEqual(self, data1, data2):
+ """Assert that two containers have the same items, with special treatment
+ for numpy arrays.
+ """
+ try:
+ self.assertCountEqual(data1, data2)
+ except (TypeError, ValueError):
+ data1 = [self._to_hashable(d) for d in data1]
+ data2 = [self._to_hashable(d) for d in data2]
+ self.assertCountEqual(data1, data2)
+
+ def _to_hashable(self, element):
+ try:
+ hash(element)
+ return element
+ except TypeError:
+ pass
+
+ if isinstance(element, list):
+ return tuple(self._to_hashable(e) for e in element)
+
+ if isinstance(element, dict):
+ hashable_elements = []
+ for key, value in sorted(element.items(), key=lambda t: hash(t[0])):
+ hashable_elements.append((key, self._to_hashable(value)))
+ return tuple(hashable_elements)
+
+ if isinstance(element, np.ndarray):
+ return element.tobytes()
+
+ raise AssertionError("Encountered unhashable element: {}.".format(element))
diff --git a/sdks/python/apache_beam/testing/extra_assertions_test.py b/sdks/python/apache_beam/testing/extra_assertions_test.py
new file mode 100644
index 0000000..8948f4e
--- /dev/null
+++ b/sdks/python/apache_beam/testing/extra_assertions_test.py
@@ -0,0 +1,72 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+
+import logging
+import unittest
+
+import numpy as np
+
+from apache_beam.testing.extra_assertions import ExtraAssertionsMixin
+
+
+class ExtraAssertionsMixinTest(ExtraAssertionsMixin, unittest.TestCase):
+
+ def test_assert_array_count_equal_strings(self):
+ data1 = [u"±♠Ωℑ", u"hello", "world"]
+ data2 = ["hello", u"±♠Ωℑ", u"world"]
+ self.assertUnhashableCountEqual(data1, data2)
+
+ def test_assert_array_count_equal_mixed(self):
+ data1 = [
+ {'a': 1, 123: 1.234},
+ ['d', 1],
+ u"±♠Ωℑ",
+ np.zeros((3, 6)),
+ (1, 2, 3, 'b'),
+ 'def',
+ 100,
+ 'abc',
+ ('a', 'b', 'c'),
+ None,
+ ]
+ data2 = [
+ {123: 1.234, 'a': 1},
+ ('a', 'b', 'c'),
+ ['d', 1],
+ None,
+ 'abc',
+ 'def',
+ u"±♠Ωℑ",
+ 100,
+ (1, 2, 3, 'b'),
+ np.zeros((3, 6)),
+ ]
+ self.assertUnhashableCountEqual(data1, data2)
+ self.assertUnhashableCountEqual(data1 * 2, data2 * 2)
+
+ def test_assert_not_equal(self):
+ data1 = [{'a': 123, 'b': 321}, [1, 2, 3]]
+ data2 = [{'a': 123, 'c': 321}, [1, 2, 3]]
+ with self.assertRaises(AssertionError):
+ self.assertUnhashableCountEqual(data1, data2)
+
+
+if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
+ unittest.main()
diff --git a/sdks/python/apache_beam/testing/load_tests/build.gradle b/sdks/python/apache_beam/testing/load_tests/build.gradle
index 6fa1d18..8eba17f 100644
--- a/sdks/python/apache_beam/testing/load_tests/build.gradle
+++ b/sdks/python/apache_beam/testing/load_tests/build.gradle
@@ -20,6 +20,12 @@
apply plugin: org.apache.beam.gradle.BeamModulePlugin
applyPythonNature()
+dependencies {
+ distTarBall project(path: ":sdks:python", configuration: "distTarBall")
+}
+
+pythonVersion = '2.7'
+
description = "Apache Beam :: SDKs :: Python :: Load Tests"
def mainClassProperty = "loadTest.mainClass"
@@ -29,22 +35,19 @@
def runnerProperty = "runner"
-task run(type: Exec) {
- dependsOn 'setupVirtualenv'
- dependsOn 'installGcpTest'
-
+task run(type: Exec, dependsOn: installGcpTest) {
def loadTestArgs = project.findProperty(loadTestArgsProperty) ?: ""
def runnerArg = project.findProperty(runnerProperty) ?: ""
if (runnerArg == 'DataflowRunner' || runnerArg == 'TestDataflowRunner') {
- dependsOn 'sdist'
- loadTestArgs +=" --sdk_location=${project.buildDir}/apache-beam.tar.gz"
+ dependsOn ':sdks:python:sdist'
+ loadTestArgs +=" --sdk_location=${files(configurations.distTarBall.files).singleFile}"
}
environment "LOAD_TEST_ENABLED", 'true'
setWorkingDir "${project.rootDir}/sdks/python"
- commandLine 'sh', '-c', "${project.ext.envdir}/bin/python setup.py nosetests --test-pipeline-options=\"${parseOptions(loadTestArgs)}\" --tests ${mainClass}"
+ commandLine 'sh', '-c', "${project.ext.envdir}/bin/python setup.py nosetests --test-pipeline-options=\"${parseOptions(loadTestArgs)}\" --tests ${mainClass} --ignore-files \'.*py3\\d?\\.py\$\'"
ignoreExitValue true
doLast {
diff --git a/sdks/python/apache_beam/testing/load_tests/co_group_by_key_test.py b/sdks/python/apache_beam/testing/load_tests/co_group_by_key_test.py
index 6c050c3..57425a2 100644
--- a/sdks/python/apache_beam/testing/load_tests/co_group_by_key_test.py
+++ b/sdks/python/apache_beam/testing/load_tests/co_group_by_key_test.py
@@ -26,7 +26,9 @@
* metrics_table (optional) - name of BigQuery table where metrics
will be stored,
* input_options - options for Synthetic Sources,
-* co_input_options - options for Synthetic Sources.
+* co_input_options - options for Synthetic Sources,
+* iterations - number of reiterations over per-key-grouped values to perform
+(default: 1).
Example test run on DirectRunner:
@@ -36,6 +38,7 @@
--publish_to_big_query=true
--metrics_dataset=python_load_tests
--metrics_table=co_gbk
+ --iterations=1
--input_options='{
\"num_records\": 1000,
\"key_size\": 5,
@@ -59,6 +62,7 @@
--project=...
--metrics_dataset=python_load_tests
--metrics_table=co_gbk
+ --iterations=1
--input_options=\'
{"num_records": 1,
"key_size": 1,
@@ -90,6 +94,7 @@
--publish_to_big_query=true
--metrics_dataset=python_load_tests
--metrics_table=co_gbk
+ --iterations=1
--input_options='{
\"num_records\": 1000,
\"key_size\": 5,
@@ -115,6 +120,7 @@
--project=...
--metrics_dataset=python_load_tests
--metrics_table=co_gbk
+ --iterations=1
--temp_location=gs://...
--input_options=\'
{"num_records": 1,
@@ -162,16 +168,20 @@
super(CoGroupByKeyTest, self).setUp()
self.co_input_options = json.loads(
self.pipeline.get_option('co_input_options'))
+ self.iterations = self.get_option_or_default('iterations', 1)
- class _Ungroup(beam.DoFn):
- def process(self, element):
+ class _UngroupAndReiterate(beam.DoFn):
+ def process(self, element, iterations):
values = element[1]
inputs = values.get(INPUT_TAG)
co_inputs = values.get(CO_INPUT_TAG)
- for i in inputs:
- yield i
- for i in co_inputs:
- yield i
+ for i in range(iterations):
+ for value in inputs:
+ if i == iterations - 1:
+ yield value
+ for value in co_inputs:
+ if i == iterations - 1:
+ yield value
def testCoGroupByKey(self):
pc1 = (self.pipeline
@@ -194,8 +204,9 @@
)
# pylint: disable=expression-not-assigned
({INPUT_TAG: pc1, CO_INPUT_TAG: pc2}
- | 'CoGroupByKey: ' >> beam.CoGroupByKey()
- | 'Consume Joined Collections' >> beam.ParDo(self._Ungroup())
+ | 'CoGroupByKey ' >> beam.CoGroupByKey()
+ | 'Consume Joined Collections' >> beam.ParDo(self._UngroupAndReiterate(),
+ self.iterations)
| 'Measure time: End' >> beam.ParDo(MeasureTime(self.metrics_namespace))
)
diff --git a/sdks/python/apache_beam/testing/load_tests/combine_test.py b/sdks/python/apache_beam/testing/load_tests/combine_test.py
index 371f52e..9c2f2d0 100644
--- a/sdks/python/apache_beam/testing/load_tests/combine_test.py
+++ b/sdks/python/apache_beam/testing/load_tests/combine_test.py
@@ -18,6 +18,7 @@
This is Combine load test with Synthetic Source. Besides of the standard
input options there are additional options:
* fanout (optional) - number of GBK operations to run in parallel
+* top_count - an arguments passed to the Top combiner.
* project (optional) - the gcp project in case of saving
metrics in Big Query (in case of Dataflow Runner
it is required to specify project of runner),
@@ -37,6 +38,7 @@
--metrics_dataset=python_load_tests
--metrics_table=combine
--fanout=1
+ --top_count=1000
--input_options='{
\"num_records\": 300,
\"key_size\": 5,
@@ -62,7 +64,8 @@
"bundle_size_distribution_param": 1,
"force_initial_num_bundles": 1}\'
--runner=DirectRunner
- --fanout=1' \
+ --fanout=1
+ --top_count=1000' \
-PloadTest.mainClass=apache_beam.testing.load_tests.combine_test \
-Prunner=DirectRunner :sdks:python:apache_beam:testing:load-tests:run
@@ -72,6 +75,7 @@
--test-pipeline-options="
--runner=TestDataflowRunner
--fanout=1
+ --top_count=1000
--project=...
--staging_location=gs://...
--temp_location=gs://...
@@ -105,7 +109,8 @@
"bundle_size_distribution_param": 1,
"force_initial_num_bundles": 1}\'
--runner=TestDataflowRunner
- --fanout=1' \
+ --fanout=1
+ --top_count=1000' \
-PloadTest.mainClass=
apache_beam.testing.load_tests.combine_test \
-Prunner=
@@ -138,6 +143,11 @@
else:
self.fanout = int(self.fanout)
+ try:
+ self.top_count = int(self.pipeline.get_option('top_count'))
+ except (TypeError, ValueError):
+ self.fail('You should set \"--top_count\" option to use TOP combiners')
+
class _GetElement(beam.DoFn):
def process(self, element):
yield element
@@ -154,7 +164,7 @@
# pylint: disable=expression-not-assigned
(input
| 'Combine with Top %i' % branch >> beam.CombineGlobally(
- beam.combiners.TopCombineFn(1000))
+ beam.combiners.TopCombineFn(self.top_count))
| 'Consume %i' % branch >> beam.ParDo(self._GetElement())
| 'Measure time: End %i' % branch >> beam.ParDo(
MeasureTime(self.metrics_namespace))
diff --git a/sdks/python/apache_beam/testing/load_tests/load_test.py b/sdks/python/apache_beam/testing/load_tests/load_test.py
index a175888..cbe4d86 100644
--- a/sdks/python/apache_beam/testing/load_tests/load_test.py
+++ b/sdks/python/apache_beam/testing/load_tests/load_test.py
@@ -20,6 +20,7 @@
import logging
import unittest
+from apache_beam.metrics import MetricsFilter
from apache_beam.testing.load_tests.load_test_metrics_utils import MetricsReader
from apache_beam.testing.test_pipeline import TestPipeline
@@ -33,6 +34,8 @@
'numRecords': options.get('num_records'),
'keySizeBytes': options.get('key_size'),
'valueSizeBytes': options.get('value_size'),
+ 'hotKeyFraction': options.get('hot_key_fraction', 0),
+ 'numHotKeys': options.get('num_hot_keys', 0),
'bundleSizeDistribution': {
'type': options.get(
'bundle_size_distribution_type', 'const'
@@ -47,18 +50,20 @@
def setUp(self):
self.pipeline = TestPipeline()
self.input_options = json.loads(self.pipeline.get_option('input_options'))
+ self.project_id = self.pipeline.get_option('project')
self.publish_to_big_query = self.pipeline.get_option('publish_to_big_query')
+ self.metrics_dataset = self.pipeline.get_option('metrics_dataset')
self.metrics_namespace = self.pipeline.get_option('metrics_table')
- if not self.publish_to_big_query or self.publish_to_big_query != 'true':
+ if not self.are_metrics_collected():
logging.info('Metrics will not be collected')
self.metrics_monitor = None
else:
self.metrics_monitor = MetricsReader(
- project_name=self.pipeline.get_option('project'),
+ project_name=self.project_id,
bq_table=self.metrics_namespace,
- bq_dataset=self.pipeline.get_option('metrics_dataset'),
+ bq_dataset=self.metrics_dataset,
)
def tearDown(self):
@@ -68,6 +73,29 @@
if self.metrics_monitor:
self.metrics_monitor.publish_metrics(result)
+ def apply_filter(self, allowed):
+ """Prevents metrics from namespaces other than specified in the argument
+ from being published."""
+ if allowed:
+ self.metrics_monitor.filters = MetricsFilter().with_namespaces(allowed)
+
+ def get_option_or_default(self, opt_name, default=0):
+ """Returns a pipeline option or a default value if it was not provided.
+
+ The returned value is converted to an integer.
+ """
+ option = self.pipeline.get_option(opt_name)
+ try:
+ return int(option)
+ except TypeError:
+ return default
+ except ValueError as exc:
+ self.fail(str(exc))
+
+ def are_metrics_collected(self):
+ return self.publish_to_big_query == 'true' and None not in (
+ self.project_id, self.metrics_dataset, self.metrics_namespace)
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.DEBUG)
diff --git a/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py b/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
index 1d86178..ca3b3af 100644
--- a/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
+++ b/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py
@@ -25,8 +25,6 @@
Currently it is possible to have following metrics types:
* runtime
* total_bytes_count
-
-
"""
from __future__ import absolute_import
@@ -75,57 +73,181 @@
]
-def get_element_by_schema(schema_name, insert_list):
- for element in insert_list:
- if element['label'] == schema_name:
- return element['value']
+def parse_step(step_name):
+ """Replaces white spaces and removes 'Step:' label
+
+ Args:
+ step_name(str): step name passed in metric ParDo
+
+ Returns:
+ lower case step name without namespace and step label
+ """
+ return step_name.lower().replace(' ', '_').strip('step:_')
+
+
+def split_metrics_by_namespace_and_name(metrics, namespace, name):
+ """Splits metrics list namespace and name.
+
+ Args:
+ metrics: list of metrics from pipeline result
+ namespace(str): filter metrics by namespace
+ name(str): filter metrics by name
+
+ Returns:
+ two lists - one of metrics which are matching filters
+ and second of not matching
+ """
+ matching_metrics = []
+ not_matching_metrics = []
+ for dist in metrics:
+ if dist.key.metric.namespace == namespace\
+ and dist.key.metric.name == name:
+ matching_metrics.append(dist)
+ else:
+ not_matching_metrics.append(dist)
+ return matching_metrics, not_matching_metrics
+
+
+def get_generic_distributions(generic_dists, metric_id):
+ """Creates flatten list of distributions per its value type.
+ A generic distribution is the one which is not processed but saved in
+ the most raw version.
+
+ Args:
+ generic_dists: list of distributions to be saved
+ metric_id(uuid): id of the current test run
+
+ Returns:
+ list of dictionaries made from :class:`DistributionMetric`
+ """
+ return sum(
+ (get_all_distributions_by_type(dist, metric_id)
+ for dist in generic_dists),
+ []
+ )
+
+
+def get_all_distributions_by_type(dist, metric_id):
+ """Creates new list of objects with type of each distribution
+ metric value.
+
+ Args:
+ dist(object): DistributionMetric object to be parsed
+ metric_id(uuid): id of the current test run
+ Returns:
+ list of :class:`DistributionMetric` objects
+ """
+ submit_timestamp = time.time()
+ dist_types = ['mean', 'max', 'min', 'sum']
+ return [
+ get_distribution_dict(dist_type, submit_timestamp,
+ dist, metric_id)
+ for dist_type in dist_types
+ ]
+
+
+def get_distribution_dict(metric_type, submit_timestamp, dist, metric_id):
+ """Function creates :class:`DistributionMetric`
+
+ Args:
+ metric_type(str): type of value from distribution metric which will
+ be saved (ex. max, min, mean, sum)
+ submit_timestamp: timestamp when metric is saved
+ dist(object) distribution object from pipeline result
+ metric_id(uuid): id of the current test run
+
+ Returns:
+ dictionary prepared for saving according to schema
+ """
+ return DistributionMetric(dist, submit_timestamp, metric_id,
+ metric_type).as_dict()
class MetricsReader(object):
+ """
+ A :class:`MetricsReader` retrieves metrics from pipeline result,
+ prepares it for publishers and setup publishers.
+ """
publishers = []
- def __init__(self, project_name=None, bq_table=None, bq_dataset=None):
+ def __init__(self, project_name=None, bq_table=None, bq_dataset=None,
+ filters=None):
+ """Initializes :class:`MetricsReader` .
+
+ Args:
+ project_name (str): project with BigQuery where metrics will be saved
+ bq_table (str): BigQuery table where metrics will be saved
+ bq_dataset (str): BigQuery dataset where metrics will be saved
+ filters: MetricFilter to query only filtered metrics
+ """
+ self._namespace = bq_table
self.publishers.append(ConsoleMetricsPublisher())
check = project_name and bq_table and bq_dataset
if check:
bq_publisher = BigQueryMetricsPublisher(
project_name, bq_table, bq_dataset)
self.publishers.append(bq_publisher)
+ self.filters = filters
def publish_metrics(self, result):
- metrics = result.metrics().query()
+ metrics = result.metrics().query(self.filters)
+
+ # Metrics from pipeline result are stored in map with keys: 'gauges',
+ # 'distributions' and 'counters'.
+ # Under each key there is list of objects of each metric type. It is
+ # required to prepare metrics for publishing purposes. Expected is to have
+ # a list of dictionaries matching the schema.
insert_dicts = self._prepare_all_metrics(metrics)
if len(insert_dicts):
for publisher in self.publishers:
publisher.publish(insert_dicts)
def _prepare_all_metrics(self, metrics):
- submit_timestamp = time.time()
metric_id = uuid.uuid4().hex
- insert_rows = []
-
- for counter in metrics['counters']:
- counter_dict = CounterMetric(counter, submit_timestamp, metric_id)\
- .as_dict()
- insert_rows.append(counter_dict)
-
- dists = metrics['distributions']
- if len(dists) > 0:
- runtime = RuntimeMetric(dists, submit_timestamp, metric_id)\
- .as_dict()
- insert_rows.append(runtime)
-
+ insert_rows = self._get_counters(metrics['counters'], metric_id)
+ insert_rows += self._get_distributions(metrics['distributions'], metric_id)
return insert_rows
+ def _get_counters(self, counters, metric_id):
+ submit_timestamp = time.time()
+ return [
+ CounterMetric(counter, submit_timestamp, metric_id).as_dict()
+ for counter in counters
+ ]
+
+ def _get_distributions(self, distributions, metric_id):
+ rows = []
+ matching_namsespace, not_matching_namespace = \
+ split_metrics_by_namespace_and_name(distributions, self._namespace,
+ RUNTIME_METRIC)
+ runtime_metric = RuntimeMetric(matching_namsespace, metric_id)
+ rows.append(runtime_metric.as_dict())
+
+ rows += get_generic_distributions(not_matching_namespace, metric_id)
+ return rows
+
class Metric(object):
- value = None
- label = None
+ """Metric base class in ready-to-save format."""
- def __init__(self, submit_timestamp, metric_id):
+ def __init__(self, submit_timestamp, metric_id, value,
+ metric=None, label=None):
+ """Initializes :class:`Metric`
+
+ Args:
+ metric (object): object of metric result
+ submit_timestamp (float): date-time of saving metric to database
+ metric_id (uuid): unique id to identify test run
+ value: value of metric
+ label: custom metric name to be saved in database
+ """
self.submit_timestamp = submit_timestamp
self.metric_id = metric_id
+ self.label = label or metric.key.metric.namespace + \
+ '_' + parse_step(metric.key.step) + \
+ '_' + metric.key.metric.name
+ self.value = value
def as_dict(self):
return {SUBMIT_TIMESTAMP_LABEL: self.submit_timestamp,
@@ -136,17 +258,54 @@
class CounterMetric(Metric):
- def __init__(self, counter_dict, submit_timestamp, metric_id):
- super(CounterMetric, self).__init__(submit_timestamp, metric_id)
- self.value = counter_dict.committed
- self.label = str(counter_dict.key.metric.name)
+ """The Counter Metric in ready-to-publish format.
+
+ Args:
+ counter_metric (object): counter metric object from MetricResult
+ submit_timestamp (float): date-time of saving metric to database
+ metric_id (uuid): unique id to identify test run
+ """
+ def __init__(self, counter_metric, submit_timestamp, metric_id):
+ value = counter_metric.committed
+ super(CounterMetric, self).__init__(submit_timestamp, metric_id,
+ value, counter_metric)
+
+
+class DistributionMetric(Metric):
+ """The Distribution Metric in ready-to-publish format.
+
+ Args:
+ dist_metric (object): distribution metric object from MetricResult
+ submit_timestamp (float): date-time of saving metric to database
+ metric_id (uuid): unique id to identify test run
+ """
+ def __init__(self, dist_metric, submit_timestamp, metric_id, metric_type):
+ custom_label = dist_metric.key.metric.namespace + \
+ '_' + parse_step(dist_metric.key.step) + \
+ '_' + metric_type + \
+ '_' + dist_metric.key.metric.name
+ value = getattr(dist_metric.committed, metric_type)
+ super(DistributionMetric, self) \
+ .__init__(submit_timestamp, metric_id, value, dist_metric, custom_label)
class RuntimeMetric(Metric):
- def __init__(self, runtime_list, submit_timestamp, metric_id):
- super(RuntimeMetric, self).__init__(submit_timestamp, metric_id)
- self.value = self._prepare_runtime_metrics(runtime_list)
- self.label = RUNTIME_METRIC
+ """The Distribution Metric in ready-to-publish format.
+
+ Args:
+ runtime_list: list of distributions metrics from MetricResult
+ with runtime name
+ metric_id(uuid): unique id to identify test run
+ """
+ def __init__(self, runtime_list, metric_id):
+ value = self._prepare_runtime_metrics(runtime_list)
+ submit_timestamp = time.time()
+ # Label does not include step name, because it is one value calculated
+ # out of many steps
+ label = runtime_list[0].key.metric.namespace + \
+ '_' + RUNTIME_METRIC
+ super(RuntimeMetric, self).__init__(submit_timestamp, metric_id,
+ value, None, label)
def _prepare_runtime_metrics(self, distributions):
min_values = []
@@ -164,13 +323,15 @@
class ConsoleMetricsPublisher(object):
+ """A :class:`ConsoleMetricsPublisher` publishes collected metrics
+ to console output."""
def publish(self, results):
if len(results) > 0:
log = "Load test results for test: %s and timestamp: %s:" \
% (results[0][ID_LABEL], results[0][SUBMIT_TIMESTAMP_LABEL])
logging.info(log)
for result in results:
- log = "Metric: %s Value: %s" \
+ log = "Metric: %s Value: %d" \
% (result[METRICS_TYPE_LABEL], result[VALUE_LABEL])
logging.info(log)
else:
@@ -178,6 +339,8 @@
class BigQueryMetricsPublisher(object):
+ """A :class:`BigQueryMetricsPublisher` publishes collected metrics
+ to BigQuery output."""
def __init__(self, project_name, table, dataset):
self.bq = BigQueryClient(project_name, table, dataset)
@@ -193,6 +356,8 @@
class BigQueryClient(object):
+ """A :class:`BigQueryClient` publishes collected metrics to
+ BigQuery output."""
def __init__(self, project_name, table, dataset):
self._namespace = table
self._client = bigquery.Client(project=project_name)
@@ -235,7 +400,13 @@
class MeasureTime(beam.DoFn):
+ """A distribution metric prepared to be added to pipeline as ParDo
+ to measure runtime."""
def __init__(self, namespace):
+ """Initializes :class:`MeasureTime`.
+
+ namespace(str): namespace of metric
+ """
self.namespace = namespace
self.runtime = Metrics.distribution(self.namespace, RUNTIME_METRIC)
@@ -249,14 +420,34 @@
yield element
-def count_bytes(f):
- def repl(*args):
- namespace = args[2]
- counter = Metrics.counter(namespace, COUNTER_LABEL)
- element = args[1]
- _, value = element
- for i in range(len(value)):
- counter.inc(i)
- return f(*args)
+class MeasureBytes(beam.DoFn):
+ """Metric to measure how many bytes was observed in pipeline."""
+ LABEL = 'total_bytes'
- return repl
+ def __init__(self, namespace, extractor=None):
+ """Initializes :class:`MeasureBytes`.
+
+ Args:
+ namespace(str): metric namespace
+ extractor: function to extract elements to be count
+ """
+ self.namespace = namespace
+ self.counter = Metrics.counter(self.namespace, self.LABEL)
+ self.extractor = extractor if extractor else lambda x: (yield x)
+
+ def process(self, element, *args):
+ for value in self.extractor(element, *args):
+ self.counter.inc(len(value))
+ yield element
+
+
+class CountMessages(beam.DoFn):
+ LABEL = 'total_messages'
+
+ def __init__(self, namespace):
+ self.namespace = namespace
+ self.counter = Metrics.counter(self.namespace, self.LABEL)
+
+ def process(self, element):
+ self.counter.inc(1)
+ yield element
diff --git a/sdks/python/apache_beam/testing/load_tests/pardo_test.py b/sdks/python/apache_beam/testing/load_tests/pardo_test.py
index 873be2e..bc92a7b 100644
--- a/sdks/python/apache_beam/testing/load_tests/pardo_test.py
+++ b/sdks/python/apache_beam/testing/load_tests/pardo_test.py
@@ -17,25 +17,28 @@
"""
This is ParDo load test with Synthetic Source. Besides of the standard
input options there are additional options:
-* number_of_counter_operations - number of pardo operations
+* iterations - number of subsequent ParDo transforms to be performed,
+* number_of_counters - number of counter metrics to be created for one ParDo
+transform,
+* number_of_counter_operations - number of operations on counters to be
+performed in one ParDo,
* project (optional) - the gcp project in case of saving
metrics in Big Query (in case of Dataflow Runner
it is required to specify project of runner),
* publish_to_big_query - if metrics should be published in big query,
-* metrics_namespace (optional) - name of BigQuery dataset where metrics
+* metrics_dataset (optional) - name of BigQuery dataset where metrics
will be stored,
* metrics_table (optional) - name of BigQuery table where metrics
will be stored,
-* output (optional) - destination to save output, in case of no option
-output won't be written,
* input_options - options for Synthetic Sources.
Example test run on DirectRunner:
python setup.py nosetests \
--test-pipeline-options="
- --number_of_counter_operations=1000
- --output=gs://...
+ --iterations=1000
+ --number_of_counters=1
+ --number_of_counter_operations=1
--project=big-query-project
--publish_to_big_query=true
--metrics_dataset=python_load_tests
@@ -78,8 +81,9 @@
--staging_location=gs://...
--temp_location=gs://...
--sdk_location=./dist/apache-beam-x.x.x.dev0.tar.gz
- --output=gs://...
- --number_of_counter_operations=1000
+ --iterations=1000
+ --number_of_counters=1
+ --number_of_counter_operations=1
--publish_to_big_query=true
--metrics_dataset=python_load_tests
--metrics_table=pardo
@@ -120,6 +124,7 @@
import unittest
import apache_beam as beam
+from apache_beam.metrics import Metrics
from apache_beam.testing import synthetic_pipeline
from apache_beam.testing.load_tests.load_test import LoadTest
from apache_beam.testing.load_tests.load_test_metrics_utils import MeasureTime
@@ -132,22 +137,28 @@
@unittest.skipIf(not load_test_enabled, 'Enabled only for phrase triggering.')
class ParDoTest(LoadTest):
def setUp(self):
- self.output = self.pipeline.get_option('output')
- self.iterations = self.pipeline.get_option('number_of_counter_operations')
+ super(ParDoTest, self).setUp()
+ if self.are_metrics_collected():
+ self.apply_filter([self.metrics_namespace])
+ self.iterations = self.get_option_or_default('iterations')
+ self.number_of_counters = self.get_option_or_default('number_of_counters')
+ self.number_of_operations = self.get_option_or_default(
+ 'number_of_counter_operations')
def testParDo(self):
- class _GetElement(beam.DoFn):
- from apache_beam.testing.load_tests.load_test_metrics_utils import count_bytes
+ class CounterOperation(beam.DoFn):
+ def __init__(self, number_of_counters, number_of_operations):
+ self.number_of_operations = number_of_operations
+ self.counters = []
+ for i in range(number_of_counters):
+ self.counters.append(Metrics.counter('do-not-publish',
+ 'name-{}'.format(i)))
- @count_bytes
- def process(self, element, namespace, is_returning):
- if is_returning:
- yield element
-
- if not self.iterations:
- num_runs = 1
- else:
- num_runs = int(self.iterations)
+ def process(self, element):
+ for _ in range(self.number_of_operations):
+ for counter in self.counters:
+ counter.inc()
+ yield element
pc = (self.pipeline
| 'Read synthetic' >> beam.io.Read(
@@ -158,16 +169,11 @@
MeasureTime(self.metrics_namespace))
)
- for i in range(num_runs):
- is_returning = (i == (num_runs-1))
+ for i in range(self.iterations):
pc = (pc
| 'Step: %d' % i >> beam.ParDo(
- _GetElement(), self.metrics_namespace, is_returning)
- )
-
- if self.output:
- pc = (pc
- | "Write" >> beam.io.WriteToText(self.output)
+ CounterOperation(self.number_of_counters,
+ self.number_of_operations))
)
# pylint: disable=expression-not-assigned
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline.py b/sdks/python/apache_beam/testing/synthetic_pipeline.py
index 2cace10..50740ba 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline.py
@@ -46,6 +46,8 @@
from apache_beam.io import iobase
from apache_beam.io import range_trackers
from apache_beam.io import restriction_trackers
+from apache_beam.io.restriction_trackers import OffsetRange
+from apache_beam.io.restriction_trackers import OffsetRestrictionTracker
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.test_pipeline import TestPipeline
@@ -60,7 +62,7 @@
def parse_byte_size(s):
suffixes = 'BKMGTP'
if s[-1] in suffixes:
- return int(float(s[:-1]) * 1024**suffixes.index(s[-1]))
+ return int(float(s[:-1]) * 1024 ** suffixes.index(s[-1]))
return int(s)
@@ -76,10 +78,34 @@
return key[-1:] + key[:-1], value
+def initial_splitting_zipf(start_position, stop_position,
+ desired_num_bundles, distribution_parameter,
+ num_total_records=None):
+ """Split the given range (defined by start_position, stop_position) into
+ desired_num_bundles using zipf with the given distribution_parameter.
+ """
+ if not num_total_records:
+ num_total_records = stop_position - start_position
+ samples = np.random.zipf(distribution_parameter, desired_num_bundles)
+ total = sum(samples)
+ relative_bundle_sizes = [(float(sample) / total) for sample in samples]
+ bundle_ranges = []
+ start = start_position
+ index = 0
+ while start < stop_position:
+ if index == desired_num_bundles - 1:
+ bundle_ranges.append((start, stop_position))
+ break
+ stop = start + int(num_total_records * relative_bundle_sizes[index])
+ bundle_ranges.append((start, stop))
+ start = stop
+ index += 1
+ return bundle_ranges
+
+
class SyntheticStep(beam.DoFn):
"""A DoFn of which behavior can be controlled through prespecified parameters.
"""
-
def __init__(self, per_element_delay_sec=0, per_bundle_delay_sec=0,
output_records_per_input_record=1, output_filter_ratio=0):
if per_element_delay_sec and per_element_delay_sec < 1e-3:
@@ -116,6 +142,140 @@
yield element
+class NonLiquidShardingOffsetRangeTracker(OffsetRestrictionTracker):
+ """An OffsetRangeTracker that doesn't allow splitting. """
+
+ def try_split(self, split_offset):
+ pass # Don't split.
+
+ def checkpoint(self):
+ pass # Don't split.
+
+
+class SyntheticSDFStepRestrictionProvider(RestrictionProvider):
+ """A `RestrictionProvider` for SyntheticSDFStep.
+
+ An initial_restriction and split that operate on num_records and ignores
+ source description (element). Splits into initial_splitting_num_bundles.
+ Returns size_estimate_override as restriction size, if set. Otherwise uses
+ element size.
+
+ If initial_splitting_uneven_chunks, produces uneven chunks.
+
+ """
+
+ def __init__(self, num_records, initial_splitting_num_bundles,
+ initial_splitting_uneven_chunks, disable_liquid_sharding,
+ size_estimate_override):
+ self._num_records = num_records
+ self._initial_splitting_num_bundles = initial_splitting_num_bundles
+ self._initial_splitting_uneven_chunks = initial_splitting_uneven_chunks
+ self._disable_liquid_sharding = disable_liquid_sharding
+ self._size_estimate_override = size_estimate_override
+
+ def initial_restriction(self, element):
+ return OffsetRange(0, self._num_records)
+
+ def create_tracker(self, restriction):
+ if self._disable_liquid_sharding:
+ return NonLiquidShardingOffsetRangeTracker(restriction)
+ else:
+ return OffsetRestrictionTracker(restriction)
+
+ def split(self, element, restriction):
+ elems = restriction.size()
+ if (self._initial_splitting_uneven_chunks and
+ self._initial_splitting_num_bundles > 1 and elems > 1):
+ bundle_ranges = initial_splitting_zipf(
+ restriction.start, restriction.stop,
+ self._initial_splitting_num_bundles, 3.0)
+ for start, stop in bundle_ranges:
+ yield OffsetRange(start, stop)
+
+ else:
+ offsets_per_split = max(1, (elems // self._initial_splitting_num_bundles))
+ for split in restriction.split(offsets_per_split, offsets_per_split // 2):
+ yield split
+
+ def restriction_size(self, element, restriction):
+ if self._size_estimate_override is not None:
+ return self._size_estimate_override
+ element_size = len(element) if isinstance(element, str) else 1
+ return restriction.size() * element_size
+
+
+def get_synthetic_sdf_step(per_element_delay_sec=0,
+ per_bundle_delay_sec=0,
+ output_records_per_input_record=1,
+ output_filter_ratio=0,
+ initial_splitting_num_bundles=8,
+ initial_splitting_uneven_chunks=False,
+ disable_liquid_sharding=False,
+ size_estimate_override=None,):
+ """A function which returns a SyntheticSDFStep with given parameters. """
+
+ class SyntheticSDFStep(beam.DoFn):
+ """A SplittableDoFn of which behavior can be controlled through prespecified
+ parameters.
+ """
+
+ def __init__(self, per_element_delay_sec_arg, per_bundle_delay_sec_arg,
+ output_filter_ratio_arg, output_records_per_input_record_arg):
+ if per_element_delay_sec_arg:
+ per_element_delay_sec_arg = (
+ per_element_delay_sec_arg // output_records_per_input_record_arg)
+ if per_element_delay_sec_arg < 1e-3:
+ raise ValueError(
+ 'Per element sleep time must be at least 1e-3 after being '
+ 'divided among output elements.')
+ self._per_element_delay_sec = per_element_delay_sec_arg
+ self._per_bundle_delay_sec = per_bundle_delay_sec_arg
+ self._output_filter_ratio = output_filter_ratio_arg
+
+ def start_bundle(self):
+ self._start_time = time.time()
+
+ def finish_bundle(self):
+ # The target is for the enclosing stage to take as close to as possible
+ # the given number of seconds, so we only sleep enough to make up for
+ # overheads not incurred elsewhere.
+ to_sleep = self._per_bundle_delay_sec - (
+ time.time() - self._start_time)
+
+ # Ignoring sub-millisecond sleep times.
+ if to_sleep >= 1e-3:
+ time.sleep(to_sleep)
+
+ def process(self,
+ element,
+ restriction_tracker=beam.DoFn.RestrictionParam(
+ SyntheticSDFStepRestrictionProvider(
+ output_records_per_input_record,
+ initial_splitting_num_bundles,
+ initial_splitting_uneven_chunks,
+ disable_liquid_sharding,
+ size_estimate_override))):
+ filter_element = False
+ if self._output_filter_ratio > 0:
+ if np.random.random() < self._output_filter_ratio:
+ filter_element = True
+
+ current_restriction = restriction_tracker.current_restriction()
+ for cur in range(current_restriction.start, current_restriction.stop):
+ if not restriction_tracker.try_claim(cur):
+ return
+
+ if self._per_element_delay_sec:
+ time.sleep(self._per_element_delay_sec)
+
+ if not filter_element:
+ yield element
+ cur += 1
+
+ return SyntheticSDFStep(per_element_delay_sec, per_bundle_delay_sec,
+ output_filter_ratio, output_records_per_input_record)
+
+
class SyntheticSource(iobase.BoundedSource):
"""A custom source of a specified size.
"""
@@ -135,6 +295,9 @@
self._num_records = input_spec['numRecords']
self._key_size = maybe_parse_byte_size(input_spec.get('keySizeBytes', 1))
+ self._hot_key_fraction = input_spec.get('hotKeyFraction', 0)
+ self._num_hot_keys = input_spec.get('numHotKeys', 0)
+
self._value_size = maybe_parse_byte_size(
input_spec.get('valueSizeBytes', 1))
self._total_size = self.element_size * self._num_records
@@ -195,21 +358,9 @@
if self._initial_splitting == 'zipf':
desired_num_bundles = self._initial_splitting_num_bundles or math.ceil(
float(self.estimate_size()) / desired_bundle_size)
- samples = np.random.zipf(self._initial_splitting_distribution_parameter,
- desired_num_bundles)
- total = sum(samples)
- relative_bundle_sizes = [(float(sample) / total) for sample in samples]
- bundle_ranges = []
- start = start_position
- index = 0
- while start < stop_position:
- if index == desired_num_bundles - 1:
- bundle_ranges.append((start, stop_position))
- break
- stop = start + int(self._num_records * relative_bundle_sizes[index])
- bundle_ranges.append((start, stop))
- start = stop
- index += 1
+ bundle_ranges = initial_splitting_zipf(
+ start_position, stop_position, desired_num_bundles,
+ self._initial_splitting_distribution_parameter, self._num_records)
else:
if self._initial_splitting_num_bundles:
bundle_size_in_elements = max(1, int(
@@ -238,13 +389,25 @@
tracker = range_trackers.UnsplittableRangeTracker(tracker)
return tracker
+ def _gen_kv_pair(self, index):
+ r = np.random.RandomState(index)
+ rand = r.random_sample()
+
+ # Determines whether to generate hot key or not.
+ if rand < self._hot_key_fraction:
+ # Generate hot key.
+ # An integer is randomly selected from the range [0, numHotKeys-1]
+ # with equal probability.
+ r_hot = np.random.RandomState(index % self._num_hot_keys)
+ return r_hot.bytes(self._key_size), r.bytes(self._value_size)
+ else:
+ return r.bytes(self._key_size), r.bytes(self._value_size)
+
def read(self, range_tracker):
index = range_tracker.start_position()
while range_tracker.try_claim(index):
- r = np.random.RandomState(index)
-
time.sleep(self._sleep_per_input_record_sec)
- yield r.bytes(self._key_size), r.bytes(self._value_size)
+ yield self._gen_kv_pair(index)
index += 1
def default_output_coder(self):
@@ -262,7 +425,7 @@
{
'key_size': 1,
'value_size': 1,
- 'initial_splitting_num_bundles': 2,
+ 'initial_splitting_num_bundles': 8,
'initial_splitting_desired_bundle_size': 2,
'sleep_per_input_record_sec': 0,
'initial_splitting' : 'const'
@@ -272,15 +435,15 @@
"""
def initial_restriction(self, element):
- return (0, element['num_records'])
+ return OffsetRange(0, element['num_records'])
def create_tracker(self, restriction):
- return restriction_trackers.OffsetRestrictionTracker(
- restriction[0], restriction[1])
+ return restriction_trackers.OffsetRestrictionTracker(restriction)
def split(self, element, restriction):
bundle_ranges = []
- start_position, stop_position = restriction
+ start_position = restriction.start
+ stop_position = restriction.stop
element_size = element['key_size'] + element['value_size']
estimate_size = element_size * element['num_records']
if element['initial_splitting'] == 'zipf':
@@ -297,11 +460,11 @@
index = 0
while start < stop_position:
if index == desired_num_bundles - 1:
- bundle_ranges.append((start, stop_position))
+ bundle_ranges.append(OffsetRange(start, stop_position))
break
stop = start + int(
element['num_records'] * relative_bundle_sizes[index])
- bundle_ranges.append((start, stop))
+ bundle_ranges.append(OffsetRange(start, stop))
start = stop
index += 1
else:
@@ -317,12 +480,12 @@
for start in range(start_position, stop_position,
bundle_size_in_elements):
stop = min(start + bundle_size_in_elements, stop_position)
- bundle_ranges.append((start, stop))
+ bundle_ranges.append(OffsetRange(start, stop))
return bundle_ranges
def restriction_size(self, element, restriction):
return ((element['key_size'] + element['value_size'])
- * (restriction[1] - restriction[0]))
+ * restriction.size())
class SyntheticSDFAsSource(beam.DoFn):
@@ -334,7 +497,7 @@
{
'key_size': 1,
'value_size': 1,
- 'initial_splitting_num_bundles': 2,
+ 'initial_splitting_num_bundles': 8,
'initial_splitting_desired_bundle_size': 2,
'sleep_per_input_record_sec': 0,
'initial_splitting' : 'const'
@@ -360,12 +523,12 @@
element,
restriction_tracker=beam.DoFn.RestrictionParam(
SyntheticSDFSourceRestrictionProvider())):
- for k in range(*restriction_tracker.current_restriction()):
- if not restriction_tracker.try_claim(k):
- return
- r = np.random.RandomState(k)
+ cur = restriction_tracker.start_position()
+ while restriction_tracker.try_claim(cur):
+ r = np.random.RandomState(cur)
time.sleep(element['sleep_per_input_record_sec'])
yield r.bytes(element['key_size']), r.bytes(element['value_size'])
+ cur += 1
class ShuffleBarrier(beam.PTransform):
@@ -450,6 +613,13 @@
for each input element to a step.
(4) output_filter_ratio - the probability at which a step may filter out a
given element by not producing any output for that element.
+ (5) splittable - if the step should be splittable.
+ (6) initial_splitting_num_bundles - number of bundles initial split if step
+ is splittable.
+ (7) initial_splitting_uneven_chunks - if the bundles should be
+ unevenly-sized
+ (8) disable_liquid_sharding - if liquid sharding should be disabled
+ (9) size_estimate_override - the size estimate or None to use default
"""
all_steps = []
json_data = json.loads(json_str)
@@ -467,6 +637,21 @@
steps['output_filter_ratio'] = (
float(val['output_filter_ratio'])
if 'output_filter_ratio' in val else 0)
+ steps['splittable'] = (
+ bool(val['splittable'])
+ if 'splittable' in val else False)
+ steps['initial_splitting_num_bundles'] = (
+ int(val['initial_splitting_num_bundles'])
+ if 'initial_splitting_num_bundles' in val else 8)
+ steps['initial_splitting_uneven_chunks'] = (
+ bool(val['initial_splitting_uneven_chunks'])
+ if 'initial_splitting_uneven_chunks' in val else False)
+ steps['disable_liquid_sharding'] = (
+ bool(val['disable_liquid_sharding'])
+ if 'disable_liquid_sharding' in val else False)
+ steps['size_estimate_override'] = (
+ int(val['size_estimate_override'])
+ if 'size_estimate_override' in val else None)
all_steps.append(steps)
return all_steps
@@ -496,7 +681,9 @@
' Defaults to 0.'
'(3) An integer "output_records_per_input_record". Defaults to 1.'
'(4) A float "output_filter_ratio" in the range [0, 1] . '
- ' Defaults to 0.')
+ ' Defaults to 0.'
+ '(5) A bool "splittable" that defaults to false.'
+ '(6) An integer "initial_splitting_num_bundles". Defaults to 8.')
parser.add_argument(
'--input',
@@ -535,12 +722,12 @@
return parser.parse_known_args(args)
-def run(argv=None):
+def run(argv=None, save_main_session=True):
"""Runs the workflow."""
known_args, pipeline_args = parse_args(argv)
pipeline_options = PipelineOptions(pipeline_args)
- pipeline_options.view_as(SetupOptions).save_main_session = True
+ pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
input_info = known_args.input
@@ -595,14 +782,28 @@
new_pc_list = []
for pc_no, pc in enumerate(pc_list):
- new_pc = pc | 'SyntheticStep %d.%d' % (step_no, pc_no) >> beam.ParDo(
- SyntheticStep(
- per_element_delay_sec=steps['per_element_delay'],
- per_bundle_delay_sec=steps['per_bundle_delay'],
- output_records_per_input_record=
- steps['output_records_per_input_record'],
- output_filter_ratio=
- steps['output_filter_ratio']))
+ if steps['splittable']:
+ step = get_synthetic_sdf_step(
+ per_element_delay_sec=steps['per_element_delay'],
+ per_bundle_delay_sec=steps['per_bundle_delay'],
+ output_records_per_input_record=
+ steps['output_records_per_input_record'],
+ output_filter_ratio=steps['output_filter_ratio'],
+ initial_splitting_num_bundles=
+ steps['initial_splitting_num_bundles'],
+ initial_splitting_uneven_chunks=
+ steps['initial_splitting_uneven_chunks'],
+ disable_liquid_sharding=steps['disable_liquid_sharding'],
+ size_estimate_override=steps['size_estimate_override'])
+ else:
+ step = SyntheticStep(
+ per_element_delay_sec=steps['per_element_delay'],
+ per_bundle_delay_sec=steps['per_bundle_delay'],
+ output_records_per_input_record=
+ steps['output_records_per_input_record'],
+ output_filter_ratio=steps['output_filter_ratio'])
+ new_pc = pc | 'SyntheticStep %d.%d' % (
+ step_no, pc_no) >> beam.ParDo(step)
new_pc_list.append(new_pc)
pc_list = new_pc_list
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline_test.py b/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
index e786553..18d4bdd 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline_test.py
@@ -28,6 +28,7 @@
import apache_beam as beam
from apache_beam.io import source_test_utils
+from apache_beam.io.restriction_trackers import OffsetRange
from apache_beam.testing import synthetic_pipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
@@ -57,19 +58,110 @@
# pylint: disable=expression-not-assigned
- def testSyntheticStep(self):
- start = time.time()
+ def test_synthetic_step_multiplies_output_elements_count(self):
with beam.Pipeline() as p:
pcoll = p | beam.Create(list(range(10))) | beam.ParDo(
- synthetic_pipeline.SyntheticStep(0, 0.5, 10))
+ synthetic_pipeline.SyntheticStep(0, 0, 10))
assert_that(
pcoll | beam.combiners.Count.Globally(), equal_to([100]))
- elapsed = time.time() - start
- # TODO(chamikaramj): Fix the flaky time based bounds.
- self.assertTrue(0.5 <= elapsed <= 3, elapsed)
+ def test_minimal_runtime_with_synthetic_step_delay(self):
+ start = time.time()
+ with beam.Pipeline() as p:
+ p | beam.Create(list(range(10))) | beam.ParDo(
+ synthetic_pipeline.SyntheticStep(0, 0.5, 10))
- def testSyntheticSource(self):
+ elapsed = time.time() - start
+ self.assertGreaterEqual(elapsed, 0.5, elapsed)
+
+ def test_synthetic_sdf_step_multiplies_output_elements_count(self):
+ with beam.Pipeline() as p:
+ pcoll = p | beam.Create(list(range(10))) | beam.ParDo(
+ synthetic_pipeline.get_synthetic_sdf_step(0, 0, 10))
+ assert_that(
+ pcoll | beam.combiners.Count.Globally(), equal_to([100]))
+
+ def test_minimal_runtime_with_synthetic_sdf_step_bundle_delay(self):
+ start = time.time()
+ with beam.Pipeline() as p:
+ p | beam.Create(list(range(10))) | beam.ParDo(
+ synthetic_pipeline.get_synthetic_sdf_step(0, 0.5, 10))
+
+ elapsed = time.time() - start
+ self.assertGreaterEqual(elapsed, 0.5, elapsed)
+
+ def test_synthetic_step_split_provider(self):
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 5, 2, False, False, None)
+
+ self.assertEqual(
+ list(provider.split('ab', OffsetRange(2, 15))),
+ [OffsetRange(2, 8), OffsetRange(8, 15)])
+ self.assertEqual(
+ list(provider.split('ab', OffsetRange(0, 8))),
+ [OffsetRange(0, 4), OffsetRange(4, 8)])
+ self.assertEqual(
+ list(provider.split('ab', OffsetRange(0, 0))), [])
+ self.assertEqual(
+ list(provider.split('ab', OffsetRange(2, 3))), [OffsetRange(2, 3)])
+
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 10, 1, False, False, None)
+ self.assertEqual(list(provider.split('ab', OffsetRange(1, 10))),
+ [OffsetRange(1, 10)])
+ self.assertEqual(provider.restriction_size('ab', OffsetRange(1, 10)), 9 * 2)
+
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 10, 3, False, False, None)
+ self.assertEqual(list(provider.split('ab', OffsetRange(1, 10))),
+ [OffsetRange(1, 4), OffsetRange(4, 7), OffsetRange(7, 10)])
+ self.assertEqual(provider.initial_restriction('a'), OffsetRange(0, 10))
+
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 10, 3, False, False, 45)
+ self.assertEqual(provider.restriction_size('ab', OffsetRange(1, 3)), 45)
+
+ tracker = provider.create_tracker(OffsetRange(1, 6))
+ tracker.try_claim(1) # Claim to allow splitting.
+ self.assertEqual(tracker.try_split(.5),
+ (OffsetRange(1, 3), OffsetRange(3, 6)))
+
+ def verify_random_splits(self, provider, restriction, bundles):
+ ranges = list(provider.split('ab', restriction))
+
+ prior_stop = restriction.start
+ for r in ranges:
+ self.assertEqual(r.start, prior_stop)
+ prior_stop = r.stop
+ self.assertEqual(prior_stop, restriction.stop)
+ self.assertEqual(len(ranges), bundles)
+
+ def testSyntheticStepSplitProviderUnevenChunks(self):
+ bundles = 4
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 5, bundles, True, False, None)
+ self.verify_random_splits(provider, OffsetRange(4, 10), bundles)
+ self.verify_random_splits(provider, OffsetRange(4, 4), 0)
+ self.verify_random_splits(provider, OffsetRange(0, 1), 1)
+ self.verify_random_splits(provider, OffsetRange(0, bundles - 2), bundles)
+
+ def test_synthetic_step_split_provider_no_liquid_sharding(self):
+ # Verify Liquid Sharding Works
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 5, 5, True, False, None)
+ tracker = provider.create_tracker(OffsetRange(1, 6))
+ tracker.try_claim(2)
+ self.assertEqual(tracker.try_split(.5),
+ (OffsetRange(1, 4), OffsetRange(4, 6)))
+
+ # Verify No Liquid Sharding
+ provider = synthetic_pipeline.SyntheticSDFStepRestrictionProvider(
+ 5, 5, True, True, None)
+ tracker = provider.create_tracker(OffsetRange(1, 6))
+ tracker.try_claim(2)
+ self.assertEqual(tracker.try_split(3), None)
+
+ def test_synthetic_source(self):
def assert_size(element, expected_size):
assert len(element) == expected_size
with beam.Pipeline() as p:
@@ -83,27 +175,27 @@
assert_that(pcoll | beam.combiners.Count.Globally(),
equal_to([300]))
- def testSyntheticSourceSplitEven(self):
+ def test_synthetic_source_split_even(self):
source = synthetic_pipeline.SyntheticSource(
input_spec(1000, 1, 1, 'const', 0))
splits = source.split(100)
sources_info = [(split.source, split.start_position, split.stop_position)
for split in splits]
- self.assertEquals(20, len(sources_info))
+ self.assertEqual(20, len(sources_info))
source_test_utils.assert_sources_equal_reference_source(
(source, None, None), sources_info)
- def testSyntheticSourceSplitUneven(self):
+ def test_synthetic_source_split_uneven(self):
source = synthetic_pipeline.SyntheticSource(
input_spec(1000, 1, 1, 'zipf', 3, 10))
splits = source.split(100)
sources_info = [(split.source, split.start_position, split.stop_position)
for split in splits]
- self.assertEquals(10, len(sources_info))
+ self.assertEqual(10, len(sources_info))
source_test_utils.assert_sources_equal_reference_source(
(source, None, None), sources_info)
- def testSplitAtFraction(self):
+ def test_split_at_fraction(self):
source = synthetic_pipeline.SyntheticSource(input_spec(10, 1, 1))
source_test_utils.assert_split_at_fraction_exhaustive(source)
source_test_utils.assert_split_at_fraction_fails(source, 5, 0.3)
@@ -111,7 +203,12 @@
source, 1, 0.3)
def run_pipeline(self, barrier, writes_output=True):
- steps = [{'per_element_delay': 1}, {'per_element_delay': 1}]
+ steps = [{
+ 'per_element_delay': 1
+ }, {
+ 'per_element_delay': 1,
+ 'splittable': True
+ }]
args = ['--barrier=%s' % barrier, '--runner=DirectRunner',
'--steps=%s' % json.dumps(steps),
'--input=%s' % json.dumps(input_spec(10, 1, 1))]
@@ -119,7 +216,7 @@
output_location = tempfile.NamedTemporaryFile().name
args.append('--output=%s' % output_location)
- synthetic_pipeline.run(args)
+ synthetic_pipeline.run(args, save_main_session=False)
# Verify output
if writes_output:
@@ -130,22 +227,22 @@
self.assertEqual(10, len(read_output))
- def testPipelineShuffle(self):
+ def test_pipeline_shuffle(self):
self.run_pipeline('shuffle')
- def testPipelineSideInput(self):
+ def test_pipeline_side_input(self):
self.run_pipeline('side-input')
- def testPipelineExpandGBK(self):
+ def test_pipeline_expand_gbk(self):
self.run_pipeline('expand-gbk', False)
- def testPipelineExpandSideOutput(self):
+ def test_pipeline_expand_side_output(self):
self.run_pipeline('expand-second-output', False)
- def testPipelineMergeGBK(self):
+ def test_pipeline_merge_gbk(self):
self.run_pipeline('merge-gbk')
- def testPipelineMergeSideInput(self):
+ def test_pipeline_merge_side_input(self):
self.run_pipeline('merge-side-input')
diff --git a/sdks/python/apache_beam/testing/test_pipeline.py b/sdks/python/apache_beam/testing/test_pipeline.py
index f177e71..7a2d575 100644
--- a/sdks/python/apache_beam/testing/test_pipeline.py
+++ b/sdks/python/apache_beam/testing/test_pipeline.py
@@ -67,7 +67,8 @@
options=None,
argv=None,
is_integration_test=False,
- blocking=True):
+ blocking=True,
+ additional_pipeline_args=None):
"""Initialize a pipeline object for test.
Args:
@@ -88,6 +89,8 @@
test, :data:`False` otherwise.
blocking (bool): Run method will wait until pipeline execution is
completed.
+ additional_pipeline_args (List[str]): additional pipeline arguments to be
+ included when construction the pipeline options object.
Raises:
~exceptions.ValueError: if either the runner or options argument is not
@@ -95,7 +98,9 @@
"""
self.is_integration_test = is_integration_test
self.not_use_test_runner_api = False
- self.options_list = self._parse_test_option_args(argv)
+ additional_pipeline_args = additional_pipeline_args or []
+ self.options_list = (
+ self._parse_test_option_args(argv) + additional_pipeline_args)
self.blocking = blocking
if options is None:
options = PipelineOptions(self.options_list)
@@ -112,6 +117,9 @@
return result
+ def get_pipeline_options(self):
+ return self._options
+
def _parse_test_option_args(self, argv):
"""Parse value of command line argument: --test-pipeline-options to get
pipeline options.
diff --git a/sdks/python/apache_beam/testing/test_stream.py b/sdks/python/apache_beam/testing/test_stream.py
index dc01a99..02a8607 100644
--- a/sdks/python/apache_beam/testing/test_stream.py
+++ b/sdks/python/apache_beam/testing/test_stream.py
@@ -135,7 +135,7 @@
def expand(self, pbegin):
assert isinstance(pbegin, pvalue.PBegin)
self.pipeline = pbegin.pipeline
- return pvalue.PCollection(self.pipeline)
+ return pvalue.PCollection(self.pipeline, is_bounded=False)
def _infer_output_coder(self, input_type=None, input_coder=None):
return self.coder
@@ -194,7 +194,7 @@
return self
def advance_watermark_to_infinity(self):
- """Advance the watermark to the end of time."""
+ """Advance the watermark to the end of time, completing this TestStream."""
self.advance_watermark_to(timestamp.MAX_TIMESTAMP)
return self
diff --git a/sdks/python/apache_beam/testing/util.py b/sdks/python/apache_beam/testing/util.py
index 7e68540..6d77ee7 100644
--- a/sdks/python/apache_beam/testing/util.py
+++ b/sdks/python/apache_beam/testing/util.py
@@ -40,6 +40,8 @@
'assert_that',
'equal_to',
'is_empty',
+ 'is_not_empty',
+ 'matches_all',
# open_shards is internal and has no backwards compatibility guarantees.
'open_shards',
'TestWindowedValue',
@@ -119,15 +121,21 @@
# Try to compare actual and expected by sorting. This fails with a
# TypeError in Python 3 if different types are present in the same
- # collection.
+ # collection. It can also raise false negatives for types that don't have
+ # a deterministic sort order, like pyarrow Tables as of 0.14.1
try:
sorted_expected = sorted(expected)
sorted_actual = sorted(actual)
if sorted_expected != sorted_actual:
raise BeamAssertException(
'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
- # Fall back to slower method which works for different types on Python 3.
- except TypeError:
+ # Slower method, used in two cases:
+ # 1) If sorted expected != actual, use this method to verify the inequality.
+ # This ensures we don't raise any false negatives for types that don't
+ # have a deterministic sort order.
+ # 2) As a fallback if we encounter a TypeError in python 3. this method
+ # works on collections that have different types.
+ except (BeamAssertException, TypeError):
for element in actual:
try:
expected_list.remove(element)
@@ -141,6 +149,23 @@
return _equal
+def matches_all(expected):
+ """Matcher used by assert_that to check a set of matchers.
+
+ Args:
+ expected: A list of elements or hamcrest matchers to be used to match
+ the elements of a single PCollection.
+ """
+ def _matches(actual):
+ from hamcrest.core import assert_that as hamcrest_assert
+ from hamcrest.library.collection import contains_inanyorder
+ expected_list = list(expected)
+
+ hamcrest_assert(actual, contains_inanyorder(*expected_list))
+
+ return _matches
+
+
def is_empty():
def _empty(actual):
actual = list(actual)
@@ -150,6 +175,19 @@
return _empty
+def is_not_empty():
+ """
+ This is test method which makes sure that the pcol is not empty and it has
+ some data in it.
+ :return:
+ """
+ def _not_empty(actual):
+ actual = list(actual)
+ if not actual:
+ raise BeamAssertException('Failed assert: pcol is empty')
+ return _not_empty
+
+
def assert_that(actual, matcher, label='assert_that',
reify_windows=False, use_global_window=True):
"""A PTransform that checks a PCollection has an expected value.
diff --git a/sdks/python/apache_beam/testing/util_test.py b/sdks/python/apache_beam/testing/util_test.py
index 83b68e8..1fd1da6 100644
--- a/sdks/python/apache_beam/testing/util_test.py
+++ b/sdks/python/apache_beam/testing/util_test.py
@@ -23,10 +23,12 @@
from apache_beam import Create
from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import BeamAssertException
from apache_beam.testing.util import TestWindowedValue
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.testing.util import is_empty
+from apache_beam.testing.util import is_not_empty
from apache_beam.transforms.window import GlobalWindow
from apache_beam.transforms.window import IntervalWindow
from apache_beam.utils.timestamp import MIN_TIMESTAMP
@@ -99,6 +101,15 @@
with TestPipeline() as p:
assert_that(p | Create([1, 2, 3]), is_empty())
+ def test_assert_that_passes_is_not_empty(self):
+ with TestPipeline() as p:
+ assert_that(p | Create([1, 2, 3]), is_not_empty())
+
+ def test_assert_that_fails_on_empty_expected(self):
+ with self.assertRaises(BeamAssertException):
+ with TestPipeline() as p:
+ assert_that(p | Create([]), is_not_empty())
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/tools/coders_microbenchmark.py b/sdks/python/apache_beam/tools/coders_microbenchmark.py
index 29bdd0e..edaa3ea 100644
--- a/sdks/python/apache_beam/tools/coders_microbenchmark.py
+++ b/sdks/python/apache_beam/tools/coders_microbenchmark.py
@@ -179,7 +179,7 @@
random.seed(seed)
# TODO(BEAM-4441): Pick coders using type hints, for example:
- # tuple_coder = typecoders.registry.get_coder(typehints.Tuple[int, ...])
+ # tuple_coder = typecoders.registry.get_coder(typing.Tuple[int, ...])
benchmarks = [
coder_benchmark_factory(
coders.FastPrimitivesCoder(), small_int),
diff --git a/sdks/python/apache_beam/tools/microbenchmarks_test.py b/sdks/python/apache_beam/tools/microbenchmarks_test.py
index d306122..74949f6 100644
--- a/sdks/python/apache_beam/tools/microbenchmarks_test.py
+++ b/sdks/python/apache_beam/tools/microbenchmarks_test.py
@@ -21,7 +21,11 @@
import unittest
+from pkg_resources import DistributionNotFound
+from pkg_resources import get_distribution
+
from apache_beam.tools import coders_microbenchmark
+from apache_beam.tools import utils
class MicrobenchmarksTest(unittest.TestCase):
@@ -31,6 +35,22 @@
coders_microbenchmark.run_coder_benchmarks(
num_runs=1, input_size=10, seed=1, verbose=False)
+ def is_cython_installed(self):
+ try:
+ get_distribution('cython')
+ return True
+ except DistributionNotFound:
+ return False
+
+ def test_check_compiled(self):
+ if self.is_cython_installed():
+ utils.check_compiled('apache_beam.runners.worker.opcounters')
+ # Unfortunately, if cython is not installed, that doesn't mean we
+ # can rule out compiled modules (e.g. if Beam was installed from a wheel).
+ # Technically the other way around could be true as well, e.g. if
+ # Cython was installed after Beam, but this is rarer and more easily
+ # remedied.
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/tools/utils.py b/sdks/python/apache_beam/tools/utils.py
index 0d99b46..7c0211e 100644
--- a/sdks/python/apache_beam/tools/utils.py
+++ b/sdks/python/apache_beam/tools/utils.py
@@ -23,6 +23,7 @@
import collections
import gc
+import importlib
import os
import time
@@ -34,7 +35,7 @@
Args:
module: string, module name
"""
- check_module = __import__(module, globals(), locals())
+ check_module = importlib.import_module(module)
ext = os.path.splitext(check_module.__file__)[-1]
if ext in ('.py', '.pyc'):
raise RuntimeError(
diff --git a/sdks/python/apache_beam/transforms/__init__.py b/sdks/python/apache_beam/transforms/__init__.py
index 41cfcf6..ab35331 100644
--- a/sdks/python/apache_beam/transforms/__init__.py
+++ b/sdks/python/apache_beam/transforms/__init__.py
@@ -24,6 +24,7 @@
from apache_beam.transforms.core import *
from apache_beam.transforms.external import *
from apache_beam.transforms.ptransform import *
+from apache_beam.transforms.stats import *
from apache_beam.transforms.timeutil import TimeDomain
from apache_beam.transforms.util import *
diff --git a/sdks/python/apache_beam/transforms/combiners.py b/sdks/python/apache_beam/transforms/combiners.py
index e8345a1..49f15d3 100644
--- a/sdks/python/apache_beam/transforms/combiners.py
+++ b/sdks/python/apache_beam/transforms/combiners.py
@@ -27,24 +27,26 @@
import warnings
from builtins import object
from builtins import zip
+from typing import Any
+from typing import Dict
+from typing import Iterable
+from typing import List
+from typing import Tuple
+from typing import TypeVar
+from typing import Union
from past.builtins import long
+from apache_beam import typehints
from apache_beam.transforms import core
from apache_beam.transforms import cy_combiners
from apache_beam.transforms import ptransform
from apache_beam.transforms import window
from apache_beam.transforms.display import DisplayDataItem
-from apache_beam.typehints import KV
-from apache_beam.typehints import Any
-from apache_beam.typehints import Dict
-from apache_beam.typehints import Iterable
-from apache_beam.typehints import List
-from apache_beam.typehints import Tuple
-from apache_beam.typehints import TypeVariable
-from apache_beam.typehints import Union
from apache_beam.typehints import with_input_types
from apache_beam.typehints import with_output_types
+from apache_beam.utils.timestamp import Duration
+from apache_beam.utils.timestamp import Timestamp
__all__ = [
'Count',
@@ -53,12 +55,14 @@
'Top',
'ToDict',
'ToList',
+ 'Latest'
]
# Type variables
-T = TypeVariable('T')
-K = TypeVariable('K')
-V = TypeVariable('V')
+T = TypeVar('T')
+K = TypeVar('K')
+V = TypeVar('V')
+TimestampType = Union[int, long, float, Timestamp, Duration]
class Mean(object):
@@ -128,11 +132,13 @@
"""combiners.Count.PerElement counts how many times each element occurs."""
def expand(self, pcoll):
- paired_with_void_type = KV[pcoll.element_type, Any]
+ paired_with_void_type = typehints.Tuple[pcoll.element_type, Any]
+ output_type = typehints.KV[pcoll.element_type, int]
return (pcoll
| ('%s:PairWithVoid' % self.label >> core.Map(lambda x: (x, None))
.with_output_types(paired_with_void_type))
- | core.CombinePerKey(CountCombineFn()))
+ | core.CombinePerKey(CountCombineFn())
+ .with_output_types(output_type))
@with_input_types(Any)
@@ -312,7 +318,7 @@
"""Expands the transform.
Raises TypeCheckError: If the output type of the input PCollection is not
- compatible with KV[A, B].
+ compatible with Tuple[A, B].
Args:
pcoll: PCollection to process
@@ -350,7 +356,7 @@
@with_input_types(T)
-@with_output_types(KV[None, List[T]])
+@with_output_types(Tuple[None, List[T]])
class _TopPerBundle(core.DoFn):
def __init__(self, n, less_than, key):
self._n = n
@@ -385,7 +391,7 @@
(None, self._heap))
-@with_input_types(KV[None, Iterable[List[T]]])
+@with_input_types(Tuple[None, Iterable[List[T]]])
@with_output_types(List[T])
class _MergeTopPerBundle(core.DoFn):
def __init__(self, n, less_than, key):
@@ -858,3 +864,65 @@
def extract_only(self, accumulator):
return self.combine_fn.extract_output(accumulator)
+
+
+class Latest(object):
+ """Combiners for computing the latest element"""
+
+ @with_input_types(T)
+ @with_output_types(T)
+ class Globally(ptransform.PTransform):
+ """Compute the element with the latest timestamp from a
+ PCollection."""
+
+ @staticmethod
+ def add_timestamp(element, timestamp=core.DoFn.TimestampParam):
+ return [(element, timestamp)]
+
+ def expand(self, pcoll):
+ return (pcoll
+ | core.ParDo(self.add_timestamp)
+ .with_output_types(Tuple[T, TimestampType])
+ | core.CombineGlobally(LatestCombineFn()))
+
+ @with_input_types(Tuple[K, V])
+ @with_output_types(Tuple[K, V])
+ class PerKey(ptransform.PTransform):
+ """Compute elements with the latest timestamp for each key
+ from a keyed PCollection"""
+
+ @staticmethod
+ def add_timestamp(element, timestamp=core.DoFn.TimestampParam):
+ key, value = element
+ return [(key, (value, timestamp))]
+
+ def expand(self, pcoll):
+ return (pcoll
+ | core.ParDo(self.add_timestamp)
+ .with_output_types(Tuple[K, Tuple[T, TimestampType]])
+ | core.CombinePerKey(LatestCombineFn()))
+
+
+@with_input_types(Tuple[T, TimestampType])
+@with_output_types(T)
+class LatestCombineFn(core.CombineFn):
+ """CombineFn to get the element with the latest timestamp
+ from a PCollection."""
+
+ def create_accumulator(self):
+ return (None, window.MIN_TIMESTAMP)
+
+ def add_input(self, accumulator, element):
+ if accumulator[1] > element[1]:
+ return accumulator
+ else:
+ return element
+
+ def merge_accumulators(self, accumulators):
+ result = self.create_accumulator()
+ for accumulator in accumulators:
+ result = self.add_input(result, accumulator)
+ return result
+
+ def extract_output(self, accumulator):
+ return accumulator[0]
diff --git a/sdks/python/apache_beam/transforms/combiners_test.py b/sdks/python/apache_beam/transforms/combiners_test.py
index a73fbac..03d36e6 100644
--- a/sdks/python/apache_beam/transforms/combiners_test.py
+++ b/sdks/python/apache_beam/transforms/combiners_test.py
@@ -32,12 +32,14 @@
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.transforms import window
from apache_beam.transforms.core import CombineGlobally
from apache_beam.transforms.core import Create
from apache_beam.transforms.core import Map
from apache_beam.transforms.display import DisplayData
from apache_beam.transforms.display_test import DisplayDataItemMatcher
from apache_beam.transforms.ptransform import PTransform
+from apache_beam.typehints import TypeCheckError
class CombineTest(unittest.TestCase):
@@ -75,7 +77,7 @@
assert_that(result_bot, equal_to([[0, 1, 1, 1]]), label='assert:bot')
# Again for per-key combines.
- pcoll = pipeline | 'start-perkye' >> Create(
+ pcoll = pipeline | 'start-perkey' >> Create(
[('a', x) for x in [6, 3, 1, 1, 9, 1, 5, 2, 0, 6]])
result_key_top = pcoll | 'top-perkey' >> combine.Top.LargestPerKey(5)
result_key_bot = pcoll | 'bot-perkey' >> combine.Top.SmallestPerKey(4)
@@ -392,5 +394,92 @@
assert_that(result, equal_to([49.5]))
+class LatestTest(unittest.TestCase):
+
+ def test_globally(self):
+ l = [window.TimestampedValue(3, 100),
+ window.TimestampedValue(1, 200),
+ window.TimestampedValue(2, 300)]
+ with TestPipeline() as p:
+ # Map(lambda x: x) PTransform is added after Create here, because when
+ # a PCollection of TimestampedValues is created with Create PTransform,
+ # the timestamps are not assigned to it. Adding a Map forces the
+ # PCollection to go through a DoFn so that the PCollection consists of
+ # the elements with timestamps assigned to them instead of a PCollection
+ # of TimestampedValue(element, timestamp).
+ pc = p | Create(l) | Map(lambda x: x)
+ latest = pc | combine.Latest.Globally()
+ assert_that(latest, equal_to([2]))
+
+ def test_globally_empty(self):
+ l = []
+ with TestPipeline() as p:
+ pc = p | Create(l) | Map(lambda x: x)
+ latest = pc | combine.Latest.Globally()
+ assert_that(latest, equal_to([None]))
+
+ def test_per_key(self):
+ l = [window.TimestampedValue(('a', 1), 300),
+ window.TimestampedValue(('b', 3), 100),
+ window.TimestampedValue(('a', 2), 200)]
+ with TestPipeline() as p:
+ pc = p | Create(l) | Map(lambda x: x)
+ latest = pc | combine.Latest.PerKey()
+ assert_that(latest, equal_to([('a', 1), ('b', 3)]))
+
+ def test_per_key_empty(self):
+ l = []
+ with TestPipeline() as p:
+ pc = p | Create(l) | Map(lambda x: x)
+ latest = pc | combine.Latest.PerKey()
+ assert_that(latest, equal_to([]))
+
+
+class LatestCombineFnTest(unittest.TestCase):
+
+ def setUp(self):
+ self.fn = combine.LatestCombineFn()
+
+ def test_create_accumulator(self):
+ accumulator = self.fn.create_accumulator()
+ self.assertEqual(accumulator, (None, window.MIN_TIMESTAMP))
+
+ def test_add_input(self):
+ accumulator = self.fn.create_accumulator()
+ element = (1, 100)
+ new_accumulator = self.fn.add_input(accumulator, element)
+ self.assertEqual(new_accumulator, (1, 100))
+
+ def test_merge_accumulators(self):
+ accumulators = [(2, 400), (5, 100), (9, 200)]
+ merged_accumulator = self.fn.merge_accumulators(accumulators)
+ self.assertEqual(merged_accumulator, (2, 400))
+
+ def test_extract_output(self):
+ accumulator = (1, 100)
+ output = self.fn.extract_output(accumulator)
+ self.assertEqual(output, 1)
+
+ def test_with_input_types_decorator_violation(self):
+ l_int = [1, 2, 3]
+ l_dict = [{'a': 3}, {'g': 5}, {'r': 8}]
+ l_3_tuple = [(12, 31, 41), (12, 34, 34), (84, 92, 74)]
+
+ with self.assertRaises(TypeCheckError):
+ with TestPipeline() as p:
+ pc = p | Create(l_int)
+ _ = pc | beam.CombineGlobally(self.fn)
+
+ with self.assertRaises(TypeCheckError):
+ with TestPipeline() as p:
+ pc = p | Create(l_dict)
+ _ = pc | beam.CombineGlobally(self.fn)
+
+ with self.assertRaises(TypeCheckError):
+ with TestPipeline() as p:
+ pc = p | Create(l_3_tuple)
+ _ = pc | beam.CombineGlobally(self.fn)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/core.py b/sdks/python/apache_beam/transforms/core.py
index ca1fb46..a6e6669 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -20,10 +20,12 @@
from __future__ import absolute_import
import copy
+import inspect
import logging
import random
import re
import types
+import typing
from builtins import map
from builtins import object
from builtins import range
@@ -54,26 +56,30 @@
from apache_beam.transforms.window import TimestampedValue
from apache_beam.transforms.window import WindowedValue
from apache_beam.transforms.window import WindowFn
-from apache_beam.typehints import KV
-from apache_beam.typehints import Any
-from apache_beam.typehints import Iterable
-from apache_beam.typehints import Union
from apache_beam.typehints import trivial_inference
from apache_beam.typehints.decorators import TypeCheckError
from apache_beam.typehints.decorators import WithTypeHints
+from apache_beam.typehints.decorators import get_signature
from apache_beam.typehints.decorators import get_type_hints
-from apache_beam.typehints.decorators import getfullargspec
from apache_beam.typehints.trivial_inference import element_type
from apache_beam.typehints.typehints import is_consistent_with
from apache_beam.utils import urns
+try:
+ import funcsigs # Python 2 only.
+except ImportError:
+ funcsigs = None
+
+
__all__ = [
'DoFn',
'CombineFn',
'PartitionFn',
'ParDo',
'FlatMap',
+ 'FlatMapTuple',
'Map',
+ 'MapTuple',
'Filter',
'CombineGlobally',
'CombinePerKey',
@@ -89,9 +95,9 @@
]
# Type variables
-T = typehints.TypeVariable('T')
-K = typehints.TypeVariable('K')
-V = typehints.TypeVariable('V')
+T = typing.TypeVar('T')
+K = typing.TypeVar('K')
+V = typing.TypeVar('V')
class DoFnContext(object):
@@ -289,13 +295,43 @@
"""Return the function arguments based on the name provided. If they have
a _inspect_function attached to the class then use that otherwise default
to the modified version of python inspect library.
+
+ Returns:
+ Same as get_function_args_defaults.
"""
func_name = '_inspect_%s' % func
if hasattr(obj, func_name):
f = getattr(obj, func_name)
return f()
f = getattr(obj, func)
- return getfullargspec(f)
+ return get_function_args_defaults(f)
+
+
+def get_function_args_defaults(f):
+ """Returns the function arguments of a given function.
+
+ Returns:
+ (args: List[str], defaults: List[Any]). The first list names the
+ arguments of the method and the second one has the values of the default
+ arguments. This is similar to ``inspect.getfullargspec()``'s results, except
+ it doesn't include bound arguments and may follow function wrappers.
+ """
+ signature = get_signature(f)
+ # Fall back on funcsigs if inspect module doesn't have 'Parameter'; prefer
+ # inspect.Parameter over funcsigs.Parameter if both are available.
+ try:
+ parameter = inspect.Parameter
+ except AttributeError:
+ parameter = funcsigs.Parameter
+ # TODO(BEAM-5878) support kwonlyargs on Python 3.
+ _SUPPORTED_ARG_TYPES = [parameter.POSITIONAL_ONLY,
+ parameter.POSITIONAL_OR_KEYWORD]
+ args = [name for name, p in signature.parameters.items()
+ if p.kind in _SUPPORTED_ARG_TYPES]
+ defaults = [p.default for p in signature.parameters.values()
+ if p.kind in _SUPPORTED_ARG_TYPES and p.default is not p.empty]
+
+ return args, defaults
class RunnerAPIPTransformHolder(PTransform):
@@ -308,14 +344,34 @@
without a serialized Python `DoFn` object.
"""
- def __init__(self, proto):
+ def __init__(self, proto, context):
self._proto = proto
+ self._context = context
def proto(self):
"""Runner API payload for a `PTransform`"""
return self._proto
def to_runner_api(self, context, has_parts=False):
+ # TODO(BEAM-7850): no need to copy around Environment if it is a direct
+ # attribute of PTransform.
+ id_to_proto_map = self._context.environments.get_id_to_proto_map()
+ for env_id in id_to_proto_map:
+ if env_id not in context.environments:
+ context.environments.put_proto(env_id, id_to_proto_map[env_id])
+ else:
+ env1 = id_to_proto_map[env_id]
+ env2 = context.environments[env_id]
+ assert env1.urn == env2.proto.urn, (
+ 'Expected environments with the same ID to be equal but received '
+ 'environments with different URNs '
+ '%r and %r',
+ env1.urn, env2.proto.urn)
+ assert env1.payload == env2.proto.payload, (
+ 'Expected environments with the same ID to be equal but received '
+ 'environments with different payloads '
+ '%r and %r',
+ env1.payload, env2.proto.payload)
return self._proto
def get_restriction_coder(self):
@@ -420,19 +476,22 @@
SideInputParam = _DoFnParam('SideInputParam')
TimestampParam = _DoFnParam('TimestampParam')
WindowParam = _DoFnParam('WindowParam')
+ PaneInfoParam = _DoFnParam('PaneInfoParam')
WatermarkReporterParam = _DoFnParam('WatermarkReporterParam')
BundleFinalizerParam = _BundleFinalizerParam
-
- DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam,
- WindowParam, WatermarkReporterParam,
- BundleFinalizerParam]
+ KeyParam = _DoFnParam('KeyParam')
# Parameters to access state and timers. Not restricted to use only in the
# .process() method. Usage: DoFn.StateParam(state_spec),
- # DoFn.TimerParam(timer_spec).
+ # DoFn.TimerParam(timer_spec), DoFn.TimestampParam, DoFn.WindowParam,
+ # DoFn.KeyParam
StateParam = _StateDoFnParam
TimerParam = _TimerDoFnParam
+ DoFnProcessParams = [ElementParam, SideInputParam, TimestampParam,
+ WindowParam, WatermarkReporterParam, PaneInfoParam,
+ BundleFinalizerParam, KeyParam, StateParam, TimerParam]
+
RestrictionParam = _RestrictionDoFnParam
@staticmethod
@@ -459,6 +518,7 @@
of the parameter.
``DoFn.StateParam``: a ``userstate.RuntimeState`` object defined by the spec
of the parameter.
+ ``DoFn.KeyParam``: key associated with the element.
``DoFn.RestrictionParam``: an ``iobase.RestrictionTracker`` will be
provided here to allow treatment as a Splittable ``DoFn``. The restriction
tracker will be derived from the restriction provider in the parameter.
@@ -469,6 +529,9 @@
element: The element to be processed
*args: side inputs
**kwargs: other keyword arguments.
+
+ Returns:
+ An Iterable of output elements.
"""
raise NotImplementedError
@@ -516,10 +579,19 @@
def get_function_arguments(self, func):
return get_function_arguments(self, func)
+ def default_type_hints(self):
+ fn_type_hints = typehints.decorators.IOTypeHints.from_callable(self.process)
+ if fn_type_hints is not None:
+ try:
+ fn_type_hints.strip_iterable()
+ except ValueError as e:
+ raise ValueError('Return value not iterable: %s: %s' % (self, e))
+ return fn_type_hints
+
# TODO(sourabhbajaj): Do we want to remove the responsibility of these from
# the DoFn or maybe the runner
def infer_output_type(self, input_type):
- # TODO(robertwb): Side inputs types.
+ # TODO(BEAM-8247): Side inputs types.
# TODO(robertwb): Assert compatibility with input type hint?
return self._strip_output_annotations(
trivial_inference.infer_return_type(self.process, [input_type]))
@@ -530,7 +602,7 @@
# type inferencer understands.
if (type_hint in annotations
or trivial_inference.element_type(type_hint) in annotations):
- return Any
+ return typehints.Any
return type_hint
def _process_argspec_fn(self):
@@ -542,37 +614,19 @@
"""
return self.process
- def is_process_bounded(self):
- """Checks if an object is a bound method on an instance."""
- if not isinstance(self.process, types.MethodType):
- return False # Not a method
- if self.process.__self__ is None:
- return False # Method is not bound
- if issubclass(self.process.__self__.__class__, type) or \
- self.process.__self__.__class__ is type:
- return False # Method is a classmethod
- return True
-
urns.RunnerApiFn.register_pickle_urn(python_urns.PICKLED_DOFN)
def _fn_takes_side_inputs(fn):
try:
- argspec = getfullargspec(fn)
+ signature = get_signature(fn)
except TypeError:
# We can't tell; maybe it does.
return True
- is_bound = isinstance(fn, types.MethodType) and fn.__self__ is not None
- try:
- varkw = argspec.varkw
- kwonlyargs = argspec.kwonlyargs
- except AttributeError: # Python 2
- varkw = argspec.keywords
- kwonlyargs = []
-
- return (len(argspec.args) + len(kwonlyargs) > 1 + is_bound or
- argspec.varargs or varkw)
+ return (len(signature.parameters) > 1 or
+ any(p.kind == p.VAR_POSITIONAL or p.kind == p.VAR_KEYWORD
+ for p in signature.parameters.values()))
class CallableWrapperDoFn(DoFn):
@@ -584,7 +638,7 @@
them in transforms.
"""
- def __init__(self, fn):
+ def __init__(self, fn, fullargspec=None):
"""Initializes a CallableWrapperDoFn object wrapping a callable.
Args:
@@ -597,6 +651,7 @@
raise TypeError('Expected a callable object instead of: %r' % fn)
self._fn = fn
+ self._fullargspec = fullargspec
if isinstance(fn, (
types.BuiltinFunctionType, types.MethodType, types.FunctionType)):
self.process = fn
@@ -620,14 +675,21 @@
return 'CallableWrapperDoFn(%s)' % self._fn
def default_type_hints(self):
- type_hints = get_type_hints(self._fn)
+ fn_type_hints = typehints.decorators.IOTypeHints.from_callable(self._fn)
+ if fn_type_hints is not None:
+ try:
+ fn_type_hints.strip_iterable()
+ except ValueError as e:
+ raise ValueError('Return value not iterable: %s: %s' % (self._fn, e))
+ type_hints = get_type_hints(self._fn).with_defaults(fn_type_hints)
# If the fn was a DoFn annotated with a type-hint that hinted a return
# type compatible with Iterable[Any], then we strip off the outer
# container type due to the 'flatten' portion of FlatMap.
# TODO(robertwb): Should we require an iterable specification for FlatMap?
if type_hints.output_types:
args, kwargs = type_hints.output_types
- if len(args) == 1 and is_consistent_with(args[0], Iterable[Any]):
+ if len(args) == 1 and is_consistent_with(
+ args[0], typehints.Iterable[typehints.Any]):
type_hints = type_hints.copy()
type_hints.set_output_types(element_type(args[0]), **kwargs)
return type_hints
@@ -640,7 +702,10 @@
return getattr(self._fn, '_argspec_fn', self._fn)
def _inspect_process(self):
- return getfullargspec(self._process_argspec_fn())
+ if self._fullargspec:
+ return self._fullargspec
+ else:
+ return get_function_args_defaults(self._process_argspec_fn())
class CombineFn(WithTypeHints, HasDisplayData, urns.RunnerApiFn):
@@ -895,7 +960,8 @@
input_args, input_kwargs = tuple(input_kwargs.values()), {}
else:
raise TypeError('Combiner input type must be specified positionally.')
- if not is_consistent_with(input_args[0], Iterable[Any]):
+ if not is_consistent_with(
+ input_args[0], typehints.Iterable[typehints.Any]):
raise TypeCheckError(
'All functions for a Combine PTransform must accept a '
'single argument compatible with: Iterable[Any]. '
@@ -1102,7 +1168,7 @@
'key types. Consider adding an input type hint for this transform.',
key_coder, self)
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
def with_outputs(self, *tags, **main_kw):
"""Returns a tagged tuple allowing access to the outputs of a
@@ -1299,8 +1365,10 @@
# Proxy the type-hint information from the original function to this new
# wrapped function.
- get_type_hints(wrapper).input_types = get_type_hints(fn).input_types
- output_hint = get_type_hints(fn).simple_output_type(label)
+ type_hints = get_type_hints(fn).with_defaults(
+ typehints.decorators.IOTypeHints.from_callable(fn))
+ get_type_hints(wrapper).input_types = type_hints.input_types
+ output_hint = type_hints.simple_output_type(label)
if output_hint:
get_type_hints(wrapper).set_output_types(typehints.Iterable[output_hint])
# pylint: disable=protected-access
@@ -1312,12 +1380,155 @@
return pardo
+def MapTuple(fn, *args, **kwargs): # pylint: disable=invalid-name
+ r""":func:`MapTuple` is like :func:`Map` but expects tuple inputs and
+ flattens them into multiple input arguments.
+
+ beam.MapTuple(lambda a, b, ...: ...)
+
+ is equivalent to Python 2
+
+ beam.Map(lambda (a, b, ...), ...: ...)
+
+ In other words
+
+ beam.MapTuple(fn)
+
+ is equivalent to
+
+ beam.Map(lambda element, ...: fn(\*element, ...))
+
+ This can be useful when processing a PCollection of tuples
+ (e.g. key-value pairs).
+
+ Args:
+ fn (callable): a callable object.
+ *args: positional arguments passed to the transform callable.
+ **kwargs: keyword arguments passed to the transform callable.
+
+ Returns:
+ ~apache_beam.pvalue.PCollection:
+ A :class:`~apache_beam.pvalue.PCollection` containing the
+ :func:`MapTuple` outputs.
+
+ Raises:
+ ~exceptions.TypeError: If the **fn** passed as argument is not a callable.
+ Typical error is to pass a :class:`DoFn` instance which is supported only
+ for :class:`ParDo`.
+ """
+ if not callable(fn):
+ raise TypeError(
+ 'MapTuple can be used only with callable objects. '
+ 'Received %r instead.' % (fn))
+
+ label = 'MapTuple(%s)' % ptransform.label_from_callable(fn)
+
+ arg_names, defaults = get_function_args_defaults(fn)
+ num_defaults = len(defaults)
+ if num_defaults < len(args) + len(kwargs):
+ raise TypeError('Side inputs must have defaults for MapTuple.')
+
+ if defaults or args or kwargs:
+ wrapper = lambda x, *args, **kwargs: [fn(*(tuple(x) + args), **kwargs)]
+ else:
+ wrapper = lambda x: [fn(*x)]
+
+ # Proxy the type-hint information from the original function to this new
+ # wrapped function.
+ type_hints = get_type_hints(fn).with_defaults(
+ typehints.decorators.IOTypeHints.from_callable(fn))
+ get_type_hints(wrapper).input_types = type_hints.input_types
+ output_hint = type_hints.simple_output_type(label)
+ if output_hint:
+ get_type_hints(wrapper).set_output_types(typehints.Iterable[output_hint])
+
+ # Replace the first (args) component.
+ modified_arg_names = ['tuple_element'] + arg_names[-num_defaults:]
+ modified_argspec = (modified_arg_names, defaults)
+ pardo = ParDo(CallableWrapperDoFn(
+ wrapper, fullargspec=modified_argspec), *args, **kwargs)
+ pardo.label = label
+ return pardo
+
+
+def FlatMapTuple(fn, *args, **kwargs): # pylint: disable=invalid-name
+ r""":func:`FlatMapTuple` is like :func:`FlatMap` but expects tuple inputs and
+ flattens them into multiple input arguments.
+
+ beam.FlatMapTuple(lambda a, b, ...: ...)
+
+ is equivalent to Python 2
+
+ beam.FlatMap(lambda (a, b, ...), ...: ...)
+
+ In other words
+
+ beam.FlatMapTuple(fn)
+
+ is equivalent to
+
+ beam.FlatMap(lambda element, ...: fn(\*element, ...))
+
+ This can be useful when processing a PCollection of tuples
+ (e.g. key-value pairs).
+
+ Args:
+ fn (callable): a callable object.
+ *args: positional arguments passed to the transform callable.
+ **kwargs: keyword arguments passed to the transform callable.
+
+ Returns:
+ ~apache_beam.pvalue.PCollection:
+ A :class:`~apache_beam.pvalue.PCollection` containing the
+ :func:`FlatMapTuple` outputs.
+
+ Raises:
+ ~exceptions.TypeError: If the **fn** passed as argument is not a callable.
+ Typical error is to pass a :class:`DoFn` instance which is supported only
+ for :class:`ParDo`.
+ """
+ if not callable(fn):
+ raise TypeError(
+ 'FlatMapTuple can be used only with callable objects. '
+ 'Received %r instead.' % (fn))
+
+ label = 'FlatMapTuple(%s)' % ptransform.label_from_callable(fn)
+
+ arg_names, defaults = get_function_args_defaults(fn)
+ num_defaults = len(defaults)
+ if num_defaults < len(args) + len(kwargs):
+ raise TypeError('Side inputs must have defaults for FlatMapTuple.')
+
+ if defaults or args or kwargs:
+ wrapper = lambda x, *args, **kwargs: fn(*(tuple(x) + args), **kwargs)
+ else:
+ wrapper = lambda x: fn(*x)
+
+ # Proxy the type-hint information from the original function to this new
+ # wrapped function.
+ type_hints = get_type_hints(fn).with_defaults(
+ typehints.decorators.IOTypeHints.from_callable(fn))
+ get_type_hints(wrapper).input_types = type_hints.input_types
+ output_hint = type_hints.simple_output_type(label)
+ if output_hint:
+ get_type_hints(wrapper).set_output_types(output_hint)
+
+ # Replace the first (args) component.
+ modified_arg_names = ['tuple_element'] + arg_names[-num_defaults:]
+ modified_argspec = (modified_arg_names, defaults)
+ pardo = ParDo(CallableWrapperDoFn(
+ wrapper, fullargspec=modified_argspec), *args, **kwargs)
+ pardo.label = label
+ return pardo
+
+
def Filter(fn, *args, **kwargs): # pylint: disable=invalid-name
""":func:`Filter` is a :func:`FlatMap` with its callable filtering out
elements.
Args:
- fn (callable): a callable object.
+ fn (``Callable[..., bool]``): a callable object. First argument will be an
+ element.
*args: positional arguments passed to the transform callable.
**kwargs: keyword arguments passed to the transform callable.
@@ -1344,8 +1555,10 @@
wrapper.__name__ = fn.__name__
# Proxy the type-hint information from the function being wrapped, setting the
# output type to be the same as the input type.
- get_type_hints(wrapper).input_types = get_type_hints(fn).input_types
- output_hint = get_type_hints(fn).simple_output_type(label)
+ type_hints = get_type_hints(fn).with_defaults(
+ typehints.decorators.IOTypeHints.from_callable(fn))
+ get_type_hints(wrapper).input_types = type_hints.input_types
+ output_hint = type_hints.simple_output_type(label)
if (output_hint is None
and get_type_hints(wrapper).input_types
and get_type_hints(wrapper).input_types[0]):
@@ -1469,7 +1682,7 @@
combined = (pcoll
| 'KeyWithVoid' >> add_input_types(
Map(lambda v: (None, v)).with_output_types(
- KV[None, pcoll.element_type]))
+ typehints.KV[None, pcoll.element_type]))
| 'CombinePerKey' >> combine_per_key
| 'UnKey' >> Map(lambda k_v: k_v[1]))
@@ -1785,8 +1998,8 @@
| CombinePerKey(PostCombineFn()))
-@typehints.with_input_types(typehints.KV[K, V])
-@typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
+@typehints.with_input_types(typing.Tuple[K, V])
+@typehints.with_output_types(typing.Tuple[K, typing.Iterable[V]])
class GroupByKey(PTransform):
"""A group by key transform.
@@ -1812,7 +2025,8 @@
def infer_output_type(self, input_type):
key_type, value_type = trivial_inference.key_value_types(input_type)
- return Iterable[KV[key_type, typehints.WindowedValue[value_type]]]
+ return typehints.Iterable[
+ typehints.KV[key_type, typehints.WindowedValue[value_type]]]
def expand(self, pcoll):
# This code path is only used in the local direct runner. For Dataflow
@@ -1823,15 +2037,19 @@
# downstream to further PTransforms.
key_type, value_type = trivial_inference.key_value_types(input_type)
# Enforce the input to a GBK has a KV element type.
- pcoll.element_type = KV[key_type, value_type]
+ pcoll.element_type = typehints.KV[key_type, value_type]
typecoders.registry.verify_deterministic(
typecoders.registry.get_coder(key_type),
'GroupByKey operation "%s"' % self.label)
- reify_output_type = KV[key_type, typehints.WindowedValue[value_type]]
+ reify_output_type = typehints.KV[
+ key_type, typehints.WindowedValue[value_type]]
gbk_input_type = (
- KV[key_type, Iterable[typehints.WindowedValue[value_type]]])
- gbk_output_type = KV[key_type, Iterable[value_type]]
+ typehints.KV[
+ key_type,
+ typehints.Iterable[typehints.WindowedValue[value_type]]])
+ gbk_output_type = typehints.KV[
+ key_type, typehints.Iterable[value_type]]
# pylint: disable=bad-continuation
return (pcoll
@@ -1852,7 +2070,7 @@
def infer_output_type(self, input_type):
key_type, value_type = trivial_inference.key_value_types(input_type)
- return KV[key_type, Iterable[value_type]]
+ return typehints.KV[key_type, typehints.Iterable[value_type]]
def to_runner_api_parameter(self, unused_context):
return common_urns.primitives.GROUP_BY_KEY.urn, None
@@ -1865,21 +2083,21 @@
return True
-@typehints.with_input_types(typehints.KV[K, V])
-@typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
+@typehints.with_input_types(typing.Tuple[K, V])
+@typehints.with_output_types(typing.Tuple[K, typing.Iterable[V]])
class _GroupByKeyOnly(PTransform):
"""A group by key transform, ignoring windows."""
def infer_output_type(self, input_type):
key_type, value_type = trivial_inference.key_value_types(input_type)
- return KV[key_type, Iterable[value_type]]
+ return typehints.KV[key_type, typehints.Iterable[value_type]]
def expand(self, pcoll):
self._check_pcollection(pcoll)
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
-@typehints.with_input_types(typehints.KV[K, typehints.Iterable[V]])
-@typehints.with_output_types(typehints.KV[K, typehints.Iterable[V]])
+@typehints.with_input_types(typing.Tuple[K, typing.Iterable[V]])
+@typehints.with_output_types(typing.Tuple[K, typing.Iterable[V]])
class _GroupAlsoByWindow(ParDo):
"""The GroupAlsoByWindow transform."""
def __init__(self, windowing):
@@ -1889,7 +2107,7 @@
def expand(self, pcoll):
self._check_pcollection(pcoll)
- return pvalue.PCollection(pcoll.pipeline)
+ return pvalue.PCollection.from_(pcoll)
class _GroupAlsoByWindowDoFn(DoFn):
@@ -1903,7 +2121,8 @@
key_type, windowed_value_iter_type = trivial_inference.key_value_types(
input_type)
value_type = windowed_value_iter_type.inner_type.inner_type
- return Iterable[KV[key_type, Iterable[value_type]]]
+ return typehints.Iterable[
+ typehints.KV[key_type, typehints.Iterable[value_type]]]
def start_bundle(self):
# pylint: disable=wrong-import-order, wrong-import-position
@@ -2065,11 +2284,15 @@
new_windows = self.windowing.windowfn.assign(context)
yield WindowedValue(element, context.timestamp, new_windows)
- def __init__(self, windowfn, **kwargs):
+ def __init__(self,
+ windowfn,
+ trigger=None,
+ accumulation_mode=None,
+ timestamp_combiner=None):
"""Initializes a WindowInto transform.
Args:
- windowfn: Function to be used for windowing
+ windowfn (Windowing, WindowFn): Function to be used for windowing.
trigger: (optional) Trigger used for windowing, or None for default.
accumulation_mode: (optional) Accumulation mode used for windowing,
required for non-trivial triggers.
@@ -2080,19 +2303,14 @@
# Overlay windowing with kwargs.
windowing = windowfn
windowfn = windowing.windowfn
- # Use windowing to fill in defaults for kwargs.
- kwargs = dict(dict(
- trigger=windowing.triggerfn,
- accumulation_mode=windowing.accumulation_mode,
- timestamp_combiner=windowing.timestamp_combiner), **kwargs)
- # Use kwargs to simulate keyword-only arguments.
- triggerfn = kwargs.pop('trigger', None)
- accumulation_mode = kwargs.pop('accumulation_mode', None)
- timestamp_combiner = kwargs.pop('timestamp_combiner', None)
- if kwargs:
- raise ValueError('Unexpected keyword arguments: %s' % list(kwargs))
+
+ # Use windowing to fill in defaults for the extra arguments.
+ trigger = trigger or windowing.triggerfn
+ accumulation_mode = accumulation_mode or windowing.accumulation_mode
+ timestamp_combiner = timestamp_combiner or windowing.timestamp_combiner
+
self.windowing = Windowing(
- windowfn, triggerfn, accumulation_mode, timestamp_combiner)
+ windowfn, trigger, accumulation_mode, timestamp_combiner)
super(WindowInto, self).__init__(self.WindowIntoFn(self.windowing))
def get_windowing(self, unused_inputs):
@@ -2172,7 +2390,8 @@
def expand(self, pcolls):
for pcoll in pcolls:
self._check_pcollection(pcoll)
- result = pvalue.PCollection(self.pipeline)
+ is_bounded = all(pcoll.is_bounded for pcoll in pcolls)
+ result = pvalue.PCollection(self.pipeline, is_bounded=is_bounded)
result.element_type = typehints.Union[
tuple(pcoll.element_type for pcoll in pcolls)]
return result
@@ -2220,8 +2439,9 @@
def infer_output_type(self, unused_input_type):
if not self.values:
- return Any
- return Union[[trivial_inference.instance_to_type(v) for v in self.values]]
+ return typehints.Any
+ return typehints.Union[
+ [trivial_inference.instance_to_type(v) for v in self.values]]
def get_output_type(self):
return (self.get_type_hints().simple_output_type(self.label) or
diff --git a/sdks/python/apache_beam/transforms/dataflow_distribution_counter_test.py b/sdks/python/apache_beam/transforms/dataflow_distribution_counter_test.py
index 91a888a..bedad4b 100644
--- a/sdks/python/apache_beam/transforms/dataflow_distribution_counter_test.py
+++ b/sdks/python/apache_beam/transforms/dataflow_distribution_counter_test.py
@@ -29,7 +29,7 @@
def test_calculate_bucket_index_with_input_0(self):
counter = DataflowDistributionCounter()
index = counter.calculate_bucket_index(0)
- self.assertEquals(index, 0)
+ self.assertEqual(index, 0)
def test_calculate_bucket_index_within_max_long(self):
counter = DataflowDistributionCounter()
@@ -39,7 +39,7 @@
for multiplier in [1, 2, 5]:
value = multiplier * power_of_ten
actual_bucket = counter.calculate_bucket_index(value - 1)
- self.assertEquals(actual_bucket, bucket - 1)
+ self.assertEqual(actual_bucket, bucket - 1)
bucket += 1
power_of_ten *= 10
@@ -55,28 +55,28 @@
counter.add_input(element)
histogram = Mock(firstBucketOffset=None, bucketCounts=None)
counter.translate_to_histogram(histogram)
- self.assertEquals(counter.sum, expected_sum)
- self.assertEquals(counter.count, expected_count)
- self.assertEquals(counter.min, expected_min)
- self.assertEquals(counter.max, expected_max)
- self.assertEquals(histogram.firstBucketOffset, expected_first_bucket_index)
- self.assertEquals(histogram.bucketCounts, expected_buckets)
+ self.assertEqual(counter.sum, expected_sum)
+ self.assertEqual(counter.count, expected_count)
+ self.assertEqual(counter.min, expected_min)
+ self.assertEqual(counter.max, expected_max)
+ self.assertEqual(histogram.firstBucketOffset, expected_first_bucket_index)
+ self.assertEqual(histogram.bucketCounts, expected_buckets)
def test_translate_to_histogram_with_input_0(self):
counter = DataflowDistributionCounter()
counter.add_input(0)
histogram = Mock(firstBucketOffset=None, bucketCounts=None)
counter.translate_to_histogram(histogram)
- self.assertEquals(histogram.firstBucketOffset, 0)
- self.assertEquals(histogram.bucketCounts, [1])
+ self.assertEqual(histogram.firstBucketOffset, 0)
+ self.assertEqual(histogram.bucketCounts, [1])
def test_translate_to_histogram_with_max_input(self):
counter = DataflowDistributionCounter()
counter.add_input(INT64_MAX)
histogram = Mock(firstBucketOffset=None, bucketCounts=None)
counter.translate_to_histogram(histogram)
- self.assertEquals(histogram.firstBucketOffset, 57)
- self.assertEquals(histogram.bucketCounts, [1])
+ self.assertEqual(histogram.firstBucketOffset, 57)
+ self.assertEqual(histogram.bucketCounts, [1])
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/transforms/external.py b/sdks/python/apache_beam/transforms/external.py
index 9077ebe..75fe766 100644
--- a/sdks/python/apache_beam/transforms/external.py
+++ b/sdks/python/apache_beam/transforms/external.py
@@ -16,6 +16,8 @@
#
"""Defines Transform whose expansion is implemented elsewhere.
+
+No backward compatibility guarantees. Everything in this module is experimental.
"""
from __future__ import absolute_import
from __future__ import print_function
@@ -25,39 +27,238 @@
import threading
from apache_beam import pvalue
+from apache_beam.coders import registry
from apache_beam.portability import common_urns
from apache_beam.portability.api import beam_expansion_api_pb2
from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability.api.external_transforms_pb2 import ConfigValue
+from apache_beam.portability.api.external_transforms_pb2 import ExternalConfigurationPayload
from apache_beam.runners import pipeline_context
from apache_beam.transforms import ptransform
+from apache_beam.typehints.native_type_compatibility import convert_to_beam_type
+from apache_beam.typehints.trivial_inference import instance_to_type
+from apache_beam.typehints.typehints import Union
+from apache_beam.typehints.typehints import UnionConstraint
# Protect against environments where grpc is not available.
# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
try:
import grpc
from apache_beam.portability.api import beam_expansion_api_pb2_grpc
+ from apache_beam.utils import subprocess_server
except ImportError:
grpc = None
# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
+DEFAULT_EXPANSION_SERVICE = 'localhost:8097'
+
+
+def _is_optional_or_none(typehint):
+ return (type(None) in typehint.union_types
+ if isinstance(typehint, UnionConstraint) else typehint is type(None))
+
+
+def _strip_optional(typehint):
+ if not _is_optional_or_none(typehint):
+ return typehint
+ new_types = typehint.union_types.difference({type(None)})
+ if len(new_types) == 1:
+ return list(new_types)[0]
+ return Union[new_types]
+
+
+def iter_urns(coder, context=None):
+ yield coder.to_runner_api_parameter(context)[0]
+ for child in coder._get_component_coders():
+ for urn in iter_urns(child, context):
+ yield urn
+
+
+class PayloadBuilder(object):
+ """
+ Abstract base class for building payloads to pass to ExternalTransform.
+ """
+
+ @classmethod
+ def _config_value(cls, obj, typehint):
+ """
+ Helper to create a ConfigValue with an encoded value.
+ """
+ coder = registry.get_coder(typehint)
+ urns = list(iter_urns(coder))
+ if 'beam:coder:pickled_python:v1' in urns:
+ raise RuntimeError("Found non-portable coder for %s" % (typehint,))
+ return ConfigValue(
+ coder_urn=urns,
+ payload=coder.get_impl().encode_nested(obj))
+
+ def build(self):
+ """
+ :return: ExternalConfigurationPayload
+ """
+ raise NotImplementedError
+
+ def payload(self):
+ """
+ The serialized ExternalConfigurationPayload
+
+ :return: bytes
+ """
+ return self.build().SerializeToString()
+
+
+class SchemaBasedPayloadBuilder(PayloadBuilder):
+ """
+ Base class for building payloads based on a schema that provides
+ type information for each configuration value to encode.
+
+ Note that if the schema defines a type as Optional, the corresponding value
+ will be omitted from the encoded payload, and thus the native transform
+ will determine the default.
+ """
+
+ def __init__(self, values, schema):
+ """
+ :param values: mapping of config names to values
+ :param schema: mapping of config names to types
+ """
+ self._values = values
+ self._schema = schema
+
+ @classmethod
+ def _encode_config(cls, values, schema):
+ result = {}
+ for key, value in values.items():
+
+ try:
+ typehint = schema[key]
+ except KeyError:
+ raise RuntimeError("No typehint provided for key %r" % key)
+
+ typehint = convert_to_beam_type(typehint)
+
+ if value is None:
+ if not _is_optional_or_none(typehint):
+ raise RuntimeError("If value is None, typehint should be "
+ "optional. Got %r" % typehint)
+ # make it easy for user to filter None by default
+ continue
+ else:
+ # strip Optional from typehint so that pickled_python coder is not used
+ # for known types.
+ typehint = _strip_optional(typehint)
+ result[key] = cls._config_value(value, typehint)
+ return result
+
+ def build(self):
+ """
+ :return: ExternalConfigurationPayload
+ """
+ args = self._encode_config(self._values, self._schema)
+ return ExternalConfigurationPayload(configuration=args)
+
+
+class ImplicitSchemaPayloadBuilder(SchemaBasedPayloadBuilder):
+ """
+ Build a payload that generates a schema from the provided values.
+ """
+ def __init__(self, values):
+ schema = {key: instance_to_type(value) for key, value in values.items()}
+ super(ImplicitSchemaPayloadBuilder, self).__init__(values, schema)
+
+
+class NamedTupleBasedPayloadBuilder(SchemaBasedPayloadBuilder):
+ """
+ Build a payload based on a NamedTuple schema.
+ """
+ def __init__(self, tuple_instance):
+ """
+ :param tuple_instance: an instance of a typing.NamedTuple
+ """
+ super(NamedTupleBasedPayloadBuilder, self).__init__(
+ values=tuple_instance._asdict(), schema=tuple_instance._field_types)
+
+
+class AnnotationBasedPayloadBuilder(SchemaBasedPayloadBuilder):
+ """
+ Build a payload based on an external transform's type annotations.
+
+ Supported in python 3 only.
+ """
+ def __init__(self, transform, **values):
+ """
+ :param transform: a PTransform instance or class. type annotations will
+ be gathered from its __init__ method
+ :param values: values to encode
+ """
+ schema = {k: v for k, v in
+ transform.__init__.__annotations__.items()
+ if k in values}
+ super(AnnotationBasedPayloadBuilder, self).__init__(values, schema)
+
+
+class DataclassBasedPayloadBuilder(SchemaBasedPayloadBuilder):
+ """
+ Build a payload based on an external transform that uses dataclasses.
+
+ Supported in python 3 only.
+ """
+ def __init__(self, transform):
+ """
+ :param transform: a dataclass-decorated PTransform instance from which to
+ gather type annotations and values
+ """
+ import dataclasses
+ schema = {field.name: field.type for field in
+ dataclasses.fields(transform)}
+ super(DataclassBasedPayloadBuilder, self).__init__(
+ dataclasses.asdict(transform), schema)
+
class ExternalTransform(ptransform.PTransform):
+ """
+ External provides a cross-language transform via expansion services in
+ foreign SDKs.
+ Experimental; no backwards compatibility guarantees.
+ """
_namespace_counter = 0
_namespace = threading.local()
_EXPANDED_TRANSFORM_UNIQUE_NAME = 'root'
_IMPULSE_PREFIX = 'impulse'
- def __init__(self, urn, payload, endpoint):
- if grpc is None and isinstance(endpoint, str):
+ def __init__(self, urn, payload, expansion_service=None):
+ """Wrapper for an external transform with the given urn and payload.
+
+ :param urn: the unique beam identifier for this transform
+ :param payload: the payload, either as a byte string or a PayloadBuilder
+ :param expansion_service: an expansion service implementing the beam
+ ExpansionService protocol, either as an object with an Expand method
+ or an address (as a str) to a grpc server that provides this method.
+ """
+ expansion_service = expansion_service or DEFAULT_EXPANSION_SERVICE
+ if grpc is None and isinstance(expansion_service, str):
raise NotImplementedError('Grpc required for external transforms.')
- # TODO: Start an endpoint given an environment?
self._urn = urn
- self._payload = payload
- self._endpoint = endpoint
+ self._payload = (
+ payload.payload()
+ if isinstance(payload, PayloadBuilder)
+ else payload)
+ self._expansion_service = expansion_service
self._namespace = self._fresh_namespace()
+ def __post_init__(self, expansion_service):
+ """
+ This will only be invoked if ExternalTransform is used as a base class
+ for a class decorated with dataclasses.dataclass
+ """
+ ExternalTransform.__init__(
+ self,
+ self.URN,
+ DataclassBasedPayloadBuilder(self),
+ expansion_service)
+
def default_label(self):
return '%s(%s)' % (self.__class__.__name__, self._urn)
@@ -113,12 +314,12 @@
namespace=self._namespace,
transform=transform_proto)
- if isinstance(self._endpoint, str):
- with grpc.insecure_channel(self._endpoint) as channel:
+ if isinstance(self._expansion_service, str):
+ with grpc.insecure_channel(self._expansion_service) as channel:
response = beam_expansion_api_pb2_grpc.ExpansionServiceStub(
channel).Expand(request)
else:
- response = self._endpoint.Expand(request, None)
+ response = self._expansion_service.Expand(request, None)
if response.error:
raise RuntimeError(response.error)
@@ -217,6 +418,44 @@
for tag, pcoll in self._expanded_transform.outputs.items()})
+class JavaJarExpansionService(object):
+ """An expansion service based on an Java Jar file.
+
+ This can be passed into an ExternalTransform as the expansion_service
+ argument which will spawn a subprocess using this jar to expand the
+ transform.
+ """
+ def __init__(self, path_to_jar, extra_args=None):
+ if extra_args is None:
+ extra_args = ['{{PORT}}']
+ self._path_to_jar = path_to_jar
+ self._extra_args = extra_args
+
+ def Expand(self, request, context):
+ self._path_to_jar = subprocess_server.JavaJarServer.local_jar(
+ self._path_to_jar)
+ # Consider memoizing these servers (with some timeout).
+ with subprocess_server.JavaJarServer(
+ beam_expansion_api_pb2_grpc.ExpansionServiceStub,
+ self._path_to_jar,
+ self._extra_args) as service:
+ return service.Expand(request, context)
+
+
+class BeamJarExpansionService(JavaJarExpansionService):
+ """An expansion service based on an Beam Java Jar file.
+
+ Attempts to use a locally-build copy of the jar based on the gradle target,
+ if it exists, otherwise attempts to download it (with caching) from the
+ apache maven repository.
+ """
+ def __init__(self, gradle_target, extra_args=None, gradle_appendix=None):
+ path_to_jar = subprocess_server.JavaJarServer.path_to_beam_jar(
+ gradle_target,
+ gradle_appendix)
+ super(BeamJarExpansionService, self).__init__(path_to_jar, extra_args)
+
+
def memoize(func):
cache = {}
diff --git a/sdks/python/apache_beam/transforms/external_test.py b/sdks/python/apache_beam/transforms/external_test.py
index 5495ccc..ba315f9 100644
--- a/sdks/python/apache_beam/transforms/external_test.py
+++ b/sdks/python/apache_beam/transforms/external_test.py
@@ -15,28 +15,40 @@
# limitations under the License.
#
-"""Unit tests for the transform.util classes."""
+"""Unit tests for the transform.external classes."""
from __future__ import absolute_import
import argparse
+import logging
+import os
import subprocess
import sys
+import typing
import unittest
import grpc
from mock import patch
+from nose.plugins.attrib import attr
from past.builtins import unicode
import apache_beam as beam
from apache_beam import Pipeline
+from apache_beam.coders import FloatCoder
+from apache_beam.coders import IterableCoder
+from apache_beam.coders import StrUtf8Coder
+from apache_beam.coders import TupleCoder
+from apache_beam.coders import VarIntCoder
from apache_beam.options.pipeline_options import PipelineOptions
-from apache_beam.options.pipeline_options import SetupOptions
-from apache_beam.portability import python_urns
+from apache_beam.portability.api.external_transforms_pb2 import ConfigValue
+from apache_beam.portability.api.external_transforms_pb2 import ExternalConfigurationPayload
from apache_beam.runners.portability import expansion_service
from apache_beam.runners.portability.expansion_service_test import FibTransform
+from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.transforms.external import ImplicitSchemaPayloadBuilder
+from apache_beam.transforms.external import NamedTupleBasedPayloadBuilder
# Protect against environments where apitools library is not available.
# pylint: disable=wrong-import-order, wrong-import-position
@@ -47,6 +59,159 @@
# pylint: enable=wrong-import-order, wrong-import-position
+def get_payload(args):
+ return ExternalConfigurationPayload(configuration=args)
+
+
+class PayloadBase(object):
+ values = {
+ 'integer_example': 1,
+ 'string_example': u'thing',
+ 'list_of_strings': [u'foo', u'bar'],
+ 'optional_kv': (u'key', 1.1),
+ 'optional_integer': None,
+ }
+
+ bytes_values = {
+ 'integer_example': 1,
+ 'string_example': 'thing',
+ 'list_of_strings': ['foo', 'bar'],
+ 'optional_kv': ('key', 1.1),
+ 'optional_integer': None,
+ }
+
+ args = {
+ 'integer_example': ConfigValue(
+ coder_urn=['beam:coder:varint:v1'],
+ payload=VarIntCoder()
+ .get_impl().encode_nested(values['integer_example'])),
+ 'string_example': ConfigValue(
+ coder_urn=['beam:coder:string_utf8:v1'],
+ payload=StrUtf8Coder()
+ .get_impl().encode_nested(values['string_example'])),
+ 'list_of_strings': ConfigValue(
+ coder_urn=['beam:coder:iterable:v1',
+ 'beam:coder:string_utf8:v1'],
+ payload=IterableCoder(StrUtf8Coder())
+ .get_impl().encode_nested(values['list_of_strings'])),
+ 'optional_kv': ConfigValue(
+ coder_urn=['beam:coder:kv:v1',
+ 'beam:coder:string_utf8:v1',
+ 'beam:coder:double:v1'],
+ payload=TupleCoder([StrUtf8Coder(), FloatCoder()])
+ .get_impl().encode_nested(values['optional_kv'])),
+ }
+
+ def get_payload_from_typing_hints(self, values):
+ """Return ExternalConfigurationPayload based on python typing hints"""
+ raise NotImplementedError
+
+ def get_payload_from_beam_typehints(self, values):
+ """Return ExternalConfigurationPayload based on beam typehints"""
+ raise NotImplementedError
+
+ def test_typing_payload_builder(self):
+ result = self.get_payload_from_typing_hints(self.values)
+ expected = get_payload(self.args)
+ self.assertEqual(result, expected)
+
+ def test_typing_payload_builder_with_bytes(self):
+ """
+ string_utf8 coder will be used even if values are not unicode in python 2.x
+ """
+ result = self.get_payload_from_typing_hints(self.bytes_values)
+ expected = get_payload(self.args)
+ self.assertEqual(result, expected)
+
+ def test_typehints_payload_builder(self):
+ result = self.get_payload_from_beam_typehints(self.values)
+ expected = get_payload(self.args)
+ self.assertEqual(result, expected)
+
+ def test_typehints_payload_builder_with_bytes(self):
+ """
+ string_utf8 coder will be used even if values are not unicode in python 2.x
+ """
+ result = self.get_payload_from_beam_typehints(self.bytes_values)
+ expected = get_payload(self.args)
+ self.assertEqual(result, expected)
+
+ def test_optional_error(self):
+ """
+ value can only be None if typehint is Optional
+ """
+ with self.assertRaises(RuntimeError):
+ self.get_payload_from_typing_hints({k: None for k in self.values})
+
+
+class ExternalTuplePayloadTest(PayloadBase, unittest.TestCase):
+
+ def get_payload_from_typing_hints(self, values):
+ TestSchema = typing.NamedTuple(
+ 'TestSchema',
+ [
+ ('integer_example', int),
+ ('string_example', unicode),
+ ('list_of_strings', typing.List[unicode]),
+ ('optional_kv', typing.Optional[typing.Tuple[unicode, float]]),
+ ('optional_integer', typing.Optional[int]),
+ ]
+ )
+
+ builder = NamedTupleBasedPayloadBuilder(TestSchema(**values))
+ return builder.build()
+
+ def get_payload_from_beam_typehints(self, values):
+ raise unittest.SkipTest("Beam typehints cannot be used with "
+ "typing.NamedTuple")
+
+
+class ExternalImplicitPayloadTest(unittest.TestCase):
+ """
+ ImplicitSchemaPayloadBuilder works very differently than the other payload
+ builders
+ """
+ def test_implicit_payload_builder(self):
+ builder = ImplicitSchemaPayloadBuilder(PayloadBase.values)
+ result = builder.build()
+ expected = get_payload(PayloadBase.args)
+ self.assertEqual(result, expected)
+
+ def test_implicit_payload_builder_with_bytes(self):
+ values = PayloadBase.bytes_values
+ builder = ImplicitSchemaPayloadBuilder(values)
+ result = builder.build()
+ if sys.version_info[0] < 3:
+ # in python 2.x bytes coder will be inferred
+ args = {
+ 'integer_example': ConfigValue(
+ coder_urn=['beam:coder:varint:v1'],
+ payload=VarIntCoder()
+ .get_impl().encode_nested(values['integer_example'])),
+ 'string_example': ConfigValue(
+ coder_urn=['beam:coder:bytes:v1'],
+ payload=StrUtf8Coder()
+ .get_impl().encode_nested(values['string_example'])),
+ 'list_of_strings': ConfigValue(
+ coder_urn=['beam:coder:iterable:v1',
+ 'beam:coder:bytes:v1'],
+ payload=IterableCoder(StrUtf8Coder())
+ .get_impl().encode_nested(values['list_of_strings'])),
+ 'optional_kv': ConfigValue(
+ coder_urn=['beam:coder:kv:v1',
+ 'beam:coder:bytes:v1',
+ 'beam:coder:double:v1'],
+ payload=TupleCoder([StrUtf8Coder(), FloatCoder()])
+ .get_impl().encode_nested(values['optional_kv'])),
+ }
+ expected = get_payload(args)
+ self.assertEqual(result, expected)
+ else:
+ expected = get_payload(PayloadBase.args)
+ self.assertEqual(result, expected)
+
+
+@attr('UsesCrossLanguageTransforms')
class ExternalTransformTest(unittest.TestCase):
# This will be overwritten if set via a flag.
@@ -55,26 +220,33 @@
class _RunWithExpansion(object):
- def __init__(self, port, expansion_service_jar):
- self._port = port
- self._expansion_service_jar = expansion_service_jar
+ def __init__(self):
+ self._server = None
def __enter__(self):
- if not ExternalTransformTest.expansion_service_jar:
- raise unittest.SkipTest('No expansion service jar provided.')
+ if not (ExternalTransformTest.expansion_service_jar or
+ ExternalTransformTest.expansion_service_port):
+ raise unittest.SkipTest('No expansion service jar or port provided.')
+
+ ExternalTransformTest.expansion_service_port = (
+ ExternalTransformTest.expansion_service_port or 8091)
+
+ jar = ExternalTransformTest.expansion_service_jar
+ port = ExternalTransformTest.expansion_service_port
# Start the java server and wait for it to be ready.
- self._server = subprocess.Popen(
- ['java', '-jar', self._expansion_service_jar, str(self._port)])
+ if jar:
+ self._server = subprocess.Popen(['java', '-jar', jar, str(port)])
- port = ExternalTransformTest.expansion_service_port or 8091
address = 'localhost:%s' % str(port)
with grpc.insecure_channel(address) as channel:
grpc.channel_ready_future(channel).result()
def __exit__(self, type, value, traceback):
- self._server.kill()
+ if self._server:
+ self._server.kill()
+ self._server = None
def test_pipeline_generation(self):
pipeline = beam.Pipeline()
@@ -137,17 +309,13 @@
assert_that(p | FibTransform(6), equal_to([8]))
def test_java_expansion_portable_runner(self):
- pipeline_options = PipelineOptions(
- ['--runner=PortableRunner',
- '--experiments=beam_fn_api',
- '--environment_type=%s' % python_urns.EMBEDDED_PYTHON,
- '--job_endpoint=embed'])
+ ExternalTransformTest.expansion_service_port = os.environ.get(
+ 'EXPANSION_PORT')
+ if ExternalTransformTest.expansion_service_port:
+ ExternalTransformTest.expansion_service_port = int(
+ ExternalTransformTest.expansion_service_port)
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module level).
- pipeline_options.view_as(SetupOptions).save_main_session = True
-
- ExternalTransformTest.run_pipeline_with_portable_runner(pipeline_options)
+ ExternalTransformTest.run_pipeline_with_portable_runner(None)
@unittest.skipIf(apiclient is None, 'GCP dependencies are not installed')
def test_java_expansion_dataflow(self):
@@ -156,21 +324,17 @@
with patch.object(
apiclient.DataflowApplicationClient, 'create_job') as mock_create_job:
- port = ExternalTransformTest.expansion_service_port or 8091
- with self._RunWithExpansion(port, self.expansion_service_jar):
+ with self._RunWithExpansion():
pipeline_options = PipelineOptions(
['--runner=DataflowRunner',
'--project=dummyproject',
'--experiments=beam_fn_api',
'--temp_location=gs://dummybucket/'])
- # We use the save_main_session option because one or more DoFn's in this
- # workflow rely on global context (e.g., a module imported at module
- # level).
- pipeline_options.view_as(SetupOptions).save_main_session = True
-
# Run a simple count-filtered-letters pipeline.
- self.run_pipeline(pipeline_options, port, False)
+ self.run_pipeline(
+ pipeline_options, ExternalTransformTest.expansion_service_port,
+ False)
mock_args = mock_create_job.call_args_list
assert mock_args
@@ -181,33 +345,34 @@
@staticmethod
def run_pipeline_with_portable_runner(pipeline_options):
-
- port = ExternalTransformTest.expansion_service_port or 8091
-
- with ExternalTransformTest._RunWithExpansion(
- port, ExternalTransformTest.expansion_service_jar):
+ with ExternalTransformTest._RunWithExpansion():
# Run a simple count-filtered-letters pipeline.
- ExternalTransformTest.run_pipeline(pipeline_options, port, True)
+ ExternalTransformTest.run_pipeline(
+ pipeline_options, ExternalTransformTest.expansion_service_port, True)
@staticmethod
def run_pipeline(
- pipeline_options, expansion_service_port, wait_until_finish=True):
+ pipeline_options, expansion_service, wait_until_finish=True):
# The actual definitions of these transforms is in
# org.apache.beam.runners.core.construction.TestExpansionService.
- TEST_COUNT_URN = "pytest:beam:transforms:count"
- TEST_FILTER_URN = "pytest:beam:transforms:filter_less_than"
+ TEST_COUNT_URN = "beam:transforms:xlang:count"
+ TEST_FILTER_URN = "beam:transforms:xlang:filter_less_than_eq"
# Run a simple count-filtered-letters pipeline.
- p = beam.Pipeline(options=pipeline_options)
- address = 'localhost:%s' % str(expansion_service_port)
+ p = TestPipeline(options=pipeline_options)
+
+ if isinstance(expansion_service, int):
+ # Only the port was specified.
+ expansion_service = 'localhost:%s' % str(expansion_service)
+
res = (
p
| beam.Create(list('aaabccxyyzzz'))
| beam.Map(unicode)
# TODO(BEAM-6587): Use strings directly rather than ints.
| beam.Map(lambda x: int(ord(x)))
- | beam.ExternalTransform(TEST_FILTER_URN, b'middle', address)
- | beam.ExternalTransform(TEST_COUNT_URN, None, address)
+ | beam.ExternalTransform(TEST_FILTER_URN, b'middle', expansion_service)
+ | beam.ExternalTransform(TEST_COUNT_URN, None, expansion_service)
# # TODO(BEAM-6587): Remove when above is removed.
| beam.Map(lambda kv: (chr(kv[0]), kv[1]))
| beam.Map(lambda kv: '%s: %s' % kv))
@@ -220,9 +385,12 @@
if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.INFO)
parser = argparse.ArgumentParser()
parser.add_argument('--expansion_service_jar')
parser.add_argument('--expansion_service_port')
+ parser.add_argument('--expansion_service_target')
+ parser.add_argument('--expansion_service_target_appendix')
known_args, pipeline_args = parser.parse_known_args(sys.argv)
if known_args.expansion_service_jar:
@@ -232,6 +400,13 @@
known_args.expansion_service_port)
pipeline_options = PipelineOptions(pipeline_args)
ExternalTransformTest.run_pipeline_with_portable_runner(pipeline_options)
+ elif known_args.expansion_service_target:
+ pipeline_options = PipelineOptions(pipeline_args)
+ ExternalTransformTest.run_pipeline(
+ pipeline_options,
+ beam.transforms.external.BeamJarExpansionService(
+ known_args.expansion_service_target,
+ gradle_appendix=known_args.expansion_service_target_appendix))
else:
sys.argv = pipeline_args
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/external_test_py3.py b/sdks/python/apache_beam/transforms/external_test_py3.py
new file mode 100644
index 0000000..88fa870
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/external_test_py3.py
@@ -0,0 +1,93 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for the transform.external classes."""
+
+from __future__ import absolute_import
+
+import typing
+import unittest
+
+import apache_beam as beam
+from apache_beam import typehints
+from apache_beam.portability.api.external_transforms_pb2 import ExternalConfigurationPayload
+from apache_beam.transforms.external import AnnotationBasedPayloadBuilder
+from apache_beam.transforms.external_test import PayloadBase
+
+
+def get_payload(cls):
+ payload = ExternalConfigurationPayload()
+ payload.ParseFromString(cls._payload)
+ return payload
+
+
+class ExternalAnnotationPayloadTest(PayloadBase, unittest.TestCase):
+
+ def get_payload_from_typing_hints(self, values):
+ class AnnotatedTransform(beam.ExternalTransform):
+ URN = 'beam:external:fakeurn:v1'
+
+ def __init__(self,
+ integer_example: int,
+ string_example: str,
+ list_of_strings: typing.List[str],
+ optional_kv: typing.Optional[typing.Tuple[str, float]] = None,
+ optional_integer: typing.Optional[int] = None,
+ expansion_service=None):
+ super(AnnotatedTransform, self).__init__(
+ self.URN,
+ AnnotationBasedPayloadBuilder(
+ self,
+ integer_example=integer_example,
+ string_example=string_example,
+ list_of_strings=list_of_strings,
+ optional_kv=optional_kv,
+ optional_integer=optional_integer,
+ ),
+ expansion_service
+ )
+
+ return get_payload(AnnotatedTransform(**values))
+
+ def get_payload_from_beam_typehints(self, values):
+ class AnnotatedTransform(beam.ExternalTransform):
+ URN = 'beam:external:fakeurn:v1'
+
+ def __init__(self,
+ integer_example: int,
+ string_example: str,
+ list_of_strings: typehints.List[str],
+ optional_kv: typehints.Optional[typehints.KV[str, float]] = None,
+ optional_integer: typehints.Optional[int] = None,
+ expansion_service=None):
+ super(AnnotatedTransform, self).__init__(
+ self.URN,
+ AnnotationBasedPayloadBuilder(
+ self,
+ integer_example=integer_example,
+ string_example=string_example,
+ list_of_strings=list_of_strings,
+ optional_kv=optional_kv,
+ optional_integer=optional_integer,
+ ),
+ expansion_service
+ )
+
+ return get_payload(AnnotatedTransform(**values))
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/transforms/external_test_py37.py b/sdks/python/apache_beam/transforms/external_test_py37.py
new file mode 100644
index 0000000..ad1ff72
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/external_test_py37.py
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for the transform.external classes."""
+
+from __future__ import absolute_import
+
+import dataclasses
+import typing
+import unittest
+
+import apache_beam as beam
+from apache_beam import typehints
+from apache_beam.portability.api.external_transforms_pb2 import ExternalConfigurationPayload
+from apache_beam.transforms.external_test import PayloadBase
+
+
+def get_payload(cls):
+ payload = ExternalConfigurationPayload()
+ payload.ParseFromString(cls._payload)
+ return payload
+
+
+class ExternalDataclassesPayloadTest(PayloadBase, unittest.TestCase):
+
+ def get_payload_from_typing_hints(self, values):
+
+ @dataclasses.dataclass
+ class DataclassTransform(beam.ExternalTransform):
+ URN = 'beam:external:fakeurn:v1'
+
+ integer_example: int
+ string_example: str
+ list_of_strings: typing.List[str]
+ optional_kv: typing.Optional[typing.Tuple[str, float]] = None
+ optional_integer: typing.Optional[int] = None
+ expansion_service: dataclasses.InitVar[typing.Optional[str]] = None
+
+ return get_payload(DataclassTransform(**values))
+
+ def get_payload_from_beam_typehints(self, values):
+
+ @dataclasses.dataclass
+ class DataclassTransform(beam.ExternalTransform):
+ URN = 'beam:external:fakeurn:v1'
+
+ integer_example: int
+ string_example: str
+ list_of_strings: typehints.List[str]
+ optional_kv: typehints.Optional[typehints.KV[str, float]] = None
+ optional_integer: typehints.Optional[int] = None
+ expansion_service: dataclasses.InitVar[typehints.Optional[str]] = None
+
+ return get_payload(DataclassTransform(**values))
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/transforms/ptransform.py b/sdks/python/apache_beam/transforms/ptransform.py
index e472716..02a0ec3 100644
--- a/sdks/python/apache_beam/transforms/ptransform.py
+++ b/sdks/python/apache_beam/transforms/ptransform.py
@@ -58,11 +58,12 @@
from apache_beam.portability import python_urns
from apache_beam.transforms.display import DisplayDataItem
from apache_beam.transforms.display import HasDisplayData
+from apache_beam.typehints import native_type_compatibility
from apache_beam.typehints import typehints
from apache_beam.typehints.decorators import TypeCheckError
from apache_beam.typehints.decorators import WithTypeHints
+from apache_beam.typehints.decorators import get_signature
from apache_beam.typehints.decorators import getcallargs_forhints
-from apache_beam.typehints.decorators import getfullargspec
from apache_beam.typehints.trivial_inference import instance_to_type
from apache_beam.typehints.typehints import validate_composite_type_param
from apache_beam.utils import proto_utils
@@ -348,6 +349,8 @@
:class:`PTransform` object. This allows chaining type-hinting related
methods.
"""
+ input_type_hint = native_type_compatibility.convert_to_beam_type(
+ input_type_hint)
validate_composite_type_param(input_type_hint,
'Type hints for a PTransform')
return super(PTransform, self).with_input_types(input_type_hint)
@@ -369,6 +372,7 @@
:class:`PTransform` object. This allows chaining type-hinting related
methods.
"""
+ type_hint = native_type_compatibility.convert_to_beam_type(type_hint)
validate_composite_type_param(type_hint, 'Type hints for a PTransform')
return super(PTransform, self).with_output_types(type_hint)
@@ -383,7 +387,7 @@
def type_check_inputs_or_outputs(self, pvalueish, input_or_output):
hints = getattr(self.get_type_hints(), input_or_output + '_types')
- if not hints:
+ if hints is None or not any(hints):
return
arg_hints, kwarg_hints = hints
if arg_hints and kwarg_hints:
@@ -611,7 +615,7 @@
# For external transforms we cannot build a Python ParDo object so
# we build a holder transform instead.
from apache_beam.transforms.core import RunnerAPIPTransformHolder
- return RunnerAPIPTransformHolder(proto)
+ return RunnerAPIPTransformHolder(proto, context)
raise
def to_runner_api_parameter(self, unused_context):
@@ -736,6 +740,11 @@
"""
super(PTransformWithSideInputs, self).with_input_types(input_type_hint)
+ side_inputs_arg_hints = native_type_compatibility.convert_to_beam_types(
+ side_inputs_arg_hints)
+ side_input_kwarg_hints = native_type_compatibility.convert_to_beam_types(
+ side_input_kwarg_hints)
+
for si in side_inputs_arg_hints:
validate_composite_type_param(si, 'Type hints for a PTransform')
for si in side_input_kwarg_hints.values():
@@ -812,7 +821,7 @@
# TODO(BEAM-5878) Support keyword-only arguments.
try:
- if 'type_hints' in getfullargspec(self._fn).args:
+ if 'type_hints' in get_signature(self._fn).parameters:
args = (self.get_type_hints(),) + args
except TypeError:
# Might not be a function.
@@ -840,7 +849,7 @@
This wrapper provides an alternative, simpler way to define a PTransform.
The standard method is to subclass from PTransform and override the expand()
method. An equivalent effect can be obtained by defining a function that
- an input PCollection and additional optional arguments and returns a
+ accepts an input PCollection and additional optional arguments and returns a
resulting PCollection. For example::
@ptransform_fn
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py b/sdks/python/apache_beam/transforms/ptransform_test.py
index d863e2e..289954e 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -23,9 +23,9 @@
import collections
import operator
-import os
import re
import sys
+import typing
import unittest
from builtins import map
from builtins import range
@@ -844,7 +844,7 @@
self.check_label(beam.CombinePerKey(sum), r'CombinePerKey(sum)')
class MyDoFn(beam.DoFn):
- def process(self):
+ def process(self, unused_element):
pass
self.check_label(beam.ParDo(MyDoFn()), r'ParDo(MyDoFn)')
@@ -858,7 +858,7 @@
self.check_label('TestCPK' >> beam.CombinePerKey(sum), r'TestCPK')
class MyDoFn(beam.DoFn):
- def process(self):
+ def process(self, unused_element):
pass
self.check_label('TestParDo' >> beam.ParDo(MyDoFn()), r'TestParDo')
@@ -996,7 +996,7 @@
def test_pardo_does_not_type_check_using_type_hint_decorators(self):
@with_input_types(a=int)
- @with_output_types(typehints.List[str])
+ @with_output_types(typing.List[str])
def int_to_str(a):
return [str(a)]
@@ -1013,7 +1013,7 @@
def test_pardo_properly_type_checks_using_type_hint_decorators(self):
@with_input_types(a=str)
- @with_output_types(typehints.List[str])
+ @with_output_types(typing.List[str])
def to_all_upper_case(a):
return [a.upper()]
@@ -1163,28 +1163,29 @@
d = (self.p
| 'Str' >> beam.Create(['t', 'e', 's', 't']).with_output_types(str)
| ('Pair' >> beam.Map(lambda x: (x, ord(x)))
- .with_output_types(typehints.KV[str, str]))
+ .with_output_types(typing.Tuple[str, str]))
| _GroupByKeyOnly())
# Output type should correctly be deduced.
- # GBK-only should deduce that KV[A, B] is turned into KV[A, Iterable[B]].
- self.assertCompatible(typehints.KV[str, typehints.Iterable[str]],
+ # GBK-only should deduce that Tuple[A, B] is turned into
+ # Tuple[A, Iterable[B]].
+ self.assertCompatible(typing.Tuple[str, typing.Iterable[str]],
d.element_type)
def test_group_by_key_output_type_deduction(self):
d = (self.p
| 'Str' >> beam.Create(range(20)).with_output_types(int)
| ('PairNegative' >> beam.Map(lambda x: (x % 5, -x))
- .with_output_types(typehints.KV[int, int]))
+ .with_output_types(typing.Tuple[int, int]))
| beam.GroupByKey())
# Output type should correctly be deduced.
- # GBK should deduce that KV[A, B] is turned into KV[A, Iterable[B]].
- self.assertCompatible(typehints.KV[int, typehints.Iterable[int]],
+ # GBK should deduce that Tuple[A, B] is turned into Tuple[A, Iterable[B]].
+ self.assertCompatible(typing.Tuple[int, typing.Iterable[int]],
d.element_type)
def test_group_by_key_only_does_not_type_check(self):
- # GBK will be passed raw int's here instead of some form of KV[A, B].
+ # GBK will be passed raw int's here instead of some form of Tuple[A, B].
with self.assertRaises(typehints.TypeCheckError) as e:
(self.p
| beam.Create([1, 2, 3]).with_output_types(int)
@@ -1196,12 +1197,12 @@
e.exception.args[0])
def test_group_by_does_not_type_check(self):
- # Create is returning a List[int, str], rather than a KV[int, str] that is
- # aliased to Tuple[int, str].
+ # Create is returning a List[int, str], rather than a Tuple[int, str]
+ # that is aliased to Tuple[int, str].
with self.assertRaises(typehints.TypeCheckError) as e:
(self.p
| (beam.Create([[1], [2]])
- .with_output_types(typehints.Iterable[int]))
+ .with_output_types(typing.Iterable[int]))
| 'T' >> beam.GroupByKey())
self.assertEqual("Input type hint violation at T: "
@@ -1276,7 +1277,7 @@
self.p._options.view_as(TypeOptions).pipeline_type_check = False
self.p._options.view_as(TypeOptions).runtime_type_check = True
- @with_output_types(typehints.KV[int, str])
+ @with_output_types(typing.Tuple[int, str])
@with_input_types(x=str)
def group_with_upper_ord(x):
return (ord(x.upper()) % 5, x)
@@ -1298,11 +1299,11 @@
self.p._options.view_as(TypeOptions).pipeline_type_check = False
self.p._options.view_as(TypeOptions).runtime_type_check = True
- @with_output_types(typehints.KV[bool, int])
+ @with_output_types(typing.Tuple[bool, int])
@with_input_types(a=int)
def is_even_as_key(a):
# Simulate a programming error, should be: return (a % 2 == 0, a)
- # However this returns KV[int, int]
+ # However this returns Tuple[int, int]
return (a % 2, a)
(self.p
@@ -1327,7 +1328,7 @@
def test_pipeline_checking_satisfied_run_time_checking_satisfied(self):
self.p._options.view_as(TypeOptions).pipeline_type_check = False
- @with_output_types(typehints.KV[bool, int])
+ @with_output_types(typing.Tuple[bool, int])
@with_input_types(a=int)
def is_even_as_key(a):
# The programming error in the above test-case has now been fixed.
@@ -1371,7 +1372,7 @@
(self.p
| beam.Create([(1, 3.0), (2, 4.9), (3, 9.5)])
| ('Add' >> beam.FlatMap(lambda x_y: [x_y[0] + x_y[1]])
- .with_input_types(typehints.Tuple[int, int]).with_output_types(int))
+ .with_input_types(typing.Tuple[int, int]).with_output_types(int))
)
self.p.run()
@@ -1421,8 +1422,8 @@
(self.p
| beam.Create([(1, 3.0), (2, 4.9), (3, 9.5)])
| ('Swap' >> beam.FlatMap(lambda x_y1: [x_y1[0] + x_y1[1]])
- .with_input_types(typehints.Tuple[int, float])
- .with_output_types(typehints.Tuple[float, int]))
+ .with_input_types(typing.Tuple[int, float])
+ .with_output_types(typing.Tuple[float, int]))
)
self.p.run()
@@ -1472,13 +1473,9 @@
"Expected an instance of {}, "
"instead found 1.0, an instance of {}.".format(int, float))
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_combine_properly_pipeline_type_checks_using_decorator(self):
@with_output_types(int)
- @with_input_types(ints=typehints.Iterable[int])
+ @with_input_types(ints=typing.Iterable[int])
def sum_ints(ints):
return sum(ints)
@@ -1509,11 +1506,11 @@
def test_combine_pipeline_type_propagation_using_decorators(self):
@with_output_types(int)
- @with_input_types(ints=typehints.Iterable[int])
+ @with_input_types(ints=typing.Iterable[int])
def sum_ints(ints):
return sum(ints)
- @with_output_types(typehints.List[int])
+ @with_output_types(typing.List[int])
@with_input_types(n=int)
def range_from_zero(n):
return list(range(n+1))
@@ -1531,7 +1528,7 @@
self.p._options.view_as(TypeOptions).pipeline_type_check = False
@with_output_types(int)
- @with_input_types(ints=typehints.Iterable[int])
+ @with_input_types(ints=typing.Iterable[int])
def iter_mul(ints):
return reduce(operator.mul, ints, 1)
@@ -1548,7 +1545,7 @@
# Combine fn is returning the incorrect type
@with_output_types(int)
- @with_input_types(ints=typehints.Iterable[int])
+ @with_input_types(ints=typing.Iterable[int])
def iter_mul(ints):
return str(reduce(operator.mul, ints, 1))
@@ -1636,10 +1633,6 @@
'ParDo('
'SortJoin/CombinePerKey/')
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_mean_globally_pipeline_checking_satisfied(self):
d = (self.p
| 'C' >> beam.Create(range(5)).with_output_types(int)
@@ -1668,10 +1661,6 @@
self.assertEqual(expected_msg, e.exception.args[0])
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_mean_globally_runtime_checking_satisfied(self):
self.p._options.view_as(TypeOptions).runtime_type_check = True
@@ -1707,10 +1696,10 @@
d = (self.p
| beam.Create(range(5)).with_output_types(int)
| ('EvenGroup' >> beam.Map(lambda x: (not x % 2, x))
- .with_output_types(typehints.KV[bool, int]))
+ .with_output_types(typing.Tuple[bool, int]))
| 'EvenMean' >> combine.Mean.PerKey())
- self.assertCompatible(typehints.KV[bool, float], d.element_type)
+ self.assertCompatible(typing.Tuple[bool, float], d.element_type)
assert_that(d, equal_to([(False, 2.0), (True, 2.0)]))
self.p.run()
@@ -1719,7 +1708,7 @@
(self.p
| beam.Create(map(str, range(5))).with_output_types(str)
| ('UpperPair' >> beam.Map(lambda x: (x.upper(), x))
- .with_output_types(typehints.KV[str, str]))
+ .with_output_types(typing.Tuple[str, str]))
| 'EvenMean' >> combine.Mean.PerKey())
self.p.run()
@@ -1742,10 +1731,10 @@
d = (self.p
| beam.Create(range(5)).with_output_types(int)
| ('OddGroup' >> beam.Map(lambda x: (bool(x % 2), x))
- .with_output_types(typehints.KV[bool, int]))
+ .with_output_types(typing.Tuple[bool, int]))
| 'OddMean' >> combine.Mean.PerKey())
- self.assertCompatible(typehints.KV[bool, float], d.element_type)
+ self.assertCompatible(typing.Tuple[bool, float], d.element_type)
assert_that(d, equal_to([(False, 2.0), (True, 2.0)]))
self.p.run()
@@ -1757,7 +1746,7 @@
(self.p
| beam.Create(range(5)).with_output_types(int)
| ('OddGroup' >> beam.Map(lambda x: (x, str(bool(x % 2))))
- .with_output_types(typehints.KV[int, str]))
+ .with_output_types(typing.Tuple[int, str]))
| 'OddMean' >> combine.Mean.PerKey())
self.p.run()
@@ -1780,10 +1769,6 @@
self.assertStartswith(e.exception.args[0], expected_msg)
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_count_globally_pipeline_type_checking_satisfied(self):
d = (self.p
| 'P' >> beam.Create(range(5)).with_output_types(int)
@@ -1793,10 +1778,6 @@
assert_that(d, equal_to([5]))
self.p.run()
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_count_globally_runtime_type_checking_satisfied(self):
self.p._options.view_as(TypeOptions).runtime_type_check = True
@@ -1812,10 +1793,10 @@
d = (self.p
| beam.Create(range(5)).with_output_types(int)
| ('EvenGroup' >> beam.Map(lambda x: (not x % 2, x))
- .with_output_types(typehints.KV[bool, int]))
+ .with_output_types(typing.Tuple[bool, int]))
| 'CountInt' >> combine.Count.PerKey())
- self.assertCompatible(typehints.KV[bool, int], d.element_type)
+ self.assertCompatible(typing.Tuple[bool, int], d.element_type)
assert_that(d, equal_to([(False, 2), (True, 3)]))
self.p.run()
@@ -1837,10 +1818,10 @@
d = (self.p
| beam.Create(['t', 'e', 's', 't']).with_output_types(str)
| 'DupKey' >> beam.Map(lambda x: (x, x))
- .with_output_types(typehints.KV[str, str])
+ .with_output_types(typing.Tuple[str, str])
| 'CountDups' >> combine.Count.PerKey())
- self.assertCompatible(typehints.KV[str, int], d.element_type)
+ self.assertCompatible(typing.Tuple[str, int], d.element_type)
assert_that(d, equal_to([('e', 1), ('s', 1), ('t', 2)]))
self.p.run()
@@ -1849,7 +1830,7 @@
| beam.Create([1, 1, 2, 3]).with_output_types(int)
| 'CountElems' >> combine.Count.PerElement())
- self.assertCompatible(typehints.KV[int, int], d.element_type)
+ self.assertCompatible(typing.Tuple[int, int], d.element_type)
assert_that(d, equal_to([(1, 2), (2, 1), (3, 1)]))
self.p.run()
@@ -1874,7 +1855,7 @@
.with_output_types(bool)
| 'CountElems' >> combine.Count.PerElement())
- self.assertCompatible(typehints.KV[bool, int], d.element_type)
+ self.assertCompatible(typing.Tuple[bool, int], d.element_type)
assert_that(d, equal_to([(False, 1), (True, 4)]))
self.p.run()
@@ -1883,15 +1864,11 @@
| beam.Create(range(5, 11)).with_output_types(int)
| 'Top 3' >> combine.Top.Of(3))
- self.assertCompatible(typehints.Iterable[int],
+ self.assertCompatible(typing.Iterable[int],
d.element_type)
assert_that(d, equal_to([[10, 9, 8]]))
self.p.run()
- @unittest.skipIf(sys.version_info >= (3, 7, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.7. '
- 'See BEAM-6986')
def test_top_of_runtime_checking_satisfied(self):
self.p._options.view_as(TypeOptions).runtime_type_check = True
@@ -1899,7 +1876,7 @@
| beam.Create(list('testing')).with_output_types(str)
| 'AciiTop' >> combine.Top.Of(3))
- self.assertCompatible(typehints.Iterable[str], d.element_type)
+ self.assertCompatible(typing.Iterable[str], d.element_type)
assert_that(d, equal_to([['t', 't', 's']]))
self.p.run()
@@ -1920,10 +1897,10 @@
d = (self.p
| beam.Create(range(100)).with_output_types(int)
| ('GroupMod 3' >> beam.Map(lambda x: (x % 3, x))
- .with_output_types(typehints.KV[int, int]))
+ .with_output_types(typing.Tuple[int, int]))
| 'TopMod' >> combine.Top.PerKey(1))
- self.assertCompatible(typehints.Tuple[int, typehints.Iterable[int]],
+ self.assertCompatible(typing.Tuple[int, typing.Iterable[int]],
d.element_type)
assert_that(d, equal_to([(0, [99]), (1, [97]), (2, [98])]))
self.p.run()
@@ -1934,10 +1911,10 @@
d = (self.p
| beam.Create(range(21))
| ('GroupMod 3' >> beam.Map(lambda x: (x % 3, x))
- .with_output_types(typehints.KV[int, int]))
+ .with_output_types(typing.Tuple[int, int]))
| 'TopMod' >> combine.Top.PerKey(1))
- self.assertCompatible(typehints.KV[int, typehints.Iterable[int]],
+ self.assertCompatible(typing.Tuple[int, typing.Iterable[int]],
d.element_type)
assert_that(d, equal_to([(0, [18]), (1, [19]), (2, [20])]))
self.p.run()
@@ -1947,7 +1924,7 @@
| beam.Create([2, 2, 3, 3]).with_output_types(int)
| 'Sample' >> combine.Sample.FixedSizeGlobally(3))
- self.assertCompatible(typehints.Iterable[int], d.element_type)
+ self.assertCompatible(typing.Iterable[int], d.element_type)
def matcher(expected_len):
def match(actual):
@@ -1963,7 +1940,7 @@
| beam.Create([2, 2, 3, 3]).with_output_types(int)
| 'Sample' >> combine.Sample.FixedSizeGlobally(2))
- self.assertCompatible(typehints.Iterable[int], d.element_type)
+ self.assertCompatible(typing.Iterable[int], d.element_type)
def matcher(expected_len):
def match(actual):
@@ -1975,10 +1952,10 @@
def test_sample_per_key_pipeline_satisfied(self):
d = (self.p
| (beam.Create([(1, 2), (1, 2), (2, 3), (2, 3)])
- .with_output_types(typehints.KV[int, int]))
+ .with_output_types(typing.Tuple[int, int]))
| 'Sample' >> combine.Sample.FixedSizePerKey(2))
- self.assertCompatible(typehints.KV[int, typehints.Iterable[int]],
+ self.assertCompatible(typing.Tuple[int, typing.Iterable[int]],
d.element_type)
def matcher(expected_len):
@@ -1994,10 +1971,10 @@
d = (self.p
| (beam.Create([(1, 2), (1, 2), (2, 3), (2, 3)])
- .with_output_types(typehints.KV[int, int]))
+ .with_output_types(typing.Tuple[int, int]))
| 'Sample' >> combine.Sample.FixedSizePerKey(1))
- self.assertCompatible(typehints.KV[int, typehints.Iterable[int]],
+ self.assertCompatible(typing.Tuple[int, typing.Iterable[int]],
d.element_type)
def matcher(expected_len):
@@ -2013,7 +1990,7 @@
| beam.Create((1, 2, 3, 4)).with_output_types(int)
| combine.ToList())
- self.assertCompatible(typehints.List[int], d.element_type)
+ self.assertCompatible(typing.List[int], d.element_type)
def matcher(expected):
def match(actual):
@@ -2029,7 +2006,7 @@
| beam.Create(list('test')).with_output_types(str)
| combine.ToList())
- self.assertCompatible(typehints.List[str], d.element_type)
+ self.assertCompatible(typing.List[str], d.element_type)
def matcher(expected):
def match(actual):
@@ -2054,10 +2031,10 @@
def test_to_dict_pipeline_check_satisfied(self):
d = (self.p
| beam.Create(
- [(1, 2), (3, 4)]).with_output_types(typehints.Tuple[int, int])
+ [(1, 2), (3, 4)]).with_output_types(typing.Tuple[int, int])
| combine.ToDict())
- self.assertCompatible(typehints.Dict[int, int], d.element_type)
+ self.assertCompatible(typing.Dict[int, int], d.element_type)
assert_that(d, equal_to([{1: 2, 3: 4}]))
self.p.run()
@@ -2066,10 +2043,10 @@
d = (self.p
| (beam.Create([('1', 2), ('3', 4)])
- .with_output_types(typehints.Tuple[str, int]))
+ .with_output_types(typing.Tuple[str, int]))
| combine.ToDict())
- self.assertCompatible(typehints.Dict[str, int], d.element_type)
+ self.assertCompatible(typing.Dict[str, int], d.element_type)
assert_that(d, equal_to([{'1': 2, '3': 4}]))
self.p.run()
@@ -2089,10 +2066,6 @@
e.exception.args[0])
self.assertFalse(isinstance(e, typehints.TypeCheckError))
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_pardo_type_inference(self):
self.assertEqual(int,
beam.Filter(lambda x: False).infer_output_type(int))
@@ -2104,10 +2077,6 @@
typehints.Tuple[str, typehints.Iterable[int]],
_GroupByKeyOnly().infer_output_type(typehints.KV[str, int]))
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_pipeline_inference(self):
created = self.p | beam.Create(['a', 'b', 'c'])
mapped = created | 'pair with 1' >> beam.Map(lambda x: (x, 1))
@@ -2117,10 +2086,6 @@
self.assertEqual(typehints.KV[str, typehints.Iterable[int]],
grouped.element_type)
- @unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6.'
- 'See BEAM-6877')
def test_inferred_bad_kv_type(self):
with self.assertRaises(typehints.TypeCheckError) as e:
_ = (self.p
@@ -2155,5 +2120,46 @@
result.foo
+class TestPTransformFn(TypeHintTestCase):
+
+ def test_type_checking_fail(self):
+ @beam.ptransform_fn
+ def MyTransform(pcoll):
+ return pcoll | beam.ParDo(lambda x: [x]).with_output_types(str)
+
+ p = TestPipeline()
+ with self.assertRaisesRegexp(beam.typehints.TypeCheckError,
+ r'expected.*int.*got.*str'):
+ _ = (p
+ | beam.Create([1, 2])
+ | MyTransform().with_output_types(int))
+
+ def test_type_checking_success(self):
+ @beam.ptransform_fn
+ def MyTransform(pcoll):
+ return pcoll | beam.ParDo(lambda x: [x]).with_output_types(int)
+
+ p = TestPipeline()
+ _ = (p
+ | beam.Create([1, 2])
+ | MyTransform().with_output_types(int))
+ p.run()
+
+ def test_type_hints_arg(self):
+ # Tests passing type hints via the magic 'type_hints' argument name.
+ @beam.ptransform_fn
+ def MyTransform(pcoll, type_hints, test_arg):
+ self.assertEqual(test_arg, 'test')
+ return (pcoll
+ | beam.ParDo(lambda x: [x]).with_output_types(
+ type_hints.output_types[0][0]))
+
+ p = TestPipeline()
+ _ = (p
+ | beam.Create([1, 2])
+ | MyTransform('test').with_output_types(int))
+ p.run()
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/stats.py b/sdks/python/apache_beam/transforms/stats.py
new file mode 100644
index 0000000..5550e48
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/stats.py
@@ -0,0 +1,636 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""This module has all statistic related transforms."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import heapq
+import itertools
+import math
+import sys
+import typing
+from builtins import round
+
+from apache_beam import coders
+from apache_beam import typehints
+from apache_beam.transforms.core import *
+from apache_beam.transforms.display import DisplayDataItem
+from apache_beam.transforms.ptransform import PTransform
+
+__all__ = [
+ 'ApproximateQuantiles',
+ 'ApproximateUnique',
+]
+
+# Type variables
+T = typing.TypeVar('T')
+K = typing.TypeVar('K')
+V = typing.TypeVar('V')
+
+
+class ApproximateUnique(object):
+ """
+ Hashes input elements and uses those to extrapolate the size of the entire
+ set of hash values by assuming the rest of the hash values are as densely
+ distributed as the sample space.
+ """
+
+ _NO_VALUE_ERR_MSG = 'Either size or error should be set. Received {}.'
+ _MULTI_VALUE_ERR_MSG = 'Either size or error should be set. ' \
+ 'Received {size = %s, error = %s}.'
+ _INPUT_SIZE_ERR_MSG = 'ApproximateUnique needs a size >= 16 for an error ' \
+ '<= 0.50. In general, the estimation error is about ' \
+ '2 / sqrt(sample_size). Received {size = %s}.'
+ _INPUT_ERROR_ERR_MSG = 'ApproximateUnique needs an estimation error ' \
+ 'between 0.01 and 0.50. Received {error = %s}.'
+
+ @staticmethod
+ def parse_input_params(size=None, error=None):
+ """
+ Check if input params are valid and return sample size.
+
+ :param size: an int not smaller than 16, which we would use to estimate
+ number of unique values.
+ :param error: max estimation error, which is a float between 0.01 and 0.50.
+ If error is given, sample size will be calculated from error with
+ _get_sample_size_from_est_error function.
+ :return: sample size
+ :raises:
+ ValueError: If both size and error are given, or neither is given, or
+ values are out of range.
+ """
+
+ if None not in (size, error):
+ raise ValueError(ApproximateUnique._MULTI_VALUE_ERR_MSG % (size, error))
+ elif size is None and error is None:
+ raise ValueError(ApproximateUnique._NO_VALUE_ERR_MSG)
+ elif size is not None:
+ if not isinstance(size, int) or size < 16:
+ raise ValueError(ApproximateUnique._INPUT_SIZE_ERR_MSG % (size))
+ else:
+ return size
+ else:
+ if error < 0.01 or error > 0.5:
+ raise ValueError(ApproximateUnique._INPUT_ERROR_ERR_MSG % (error))
+ else:
+ return ApproximateUnique._get_sample_size_from_est_error(error)
+
+ @staticmethod
+ def _get_sample_size_from_est_error(est_err):
+ """
+ :return: sample size
+
+ Calculate sample size from estimation error
+ """
+ #math.ceil in python2.7 returns a float, while it returns an int in python3.
+ return int(math.ceil(4.0 / math.pow(est_err, 2.0)))
+
+ @typehints.with_input_types(T)
+ @typehints.with_output_types(int)
+ class Globally(PTransform):
+ """ Approximate.Globally approximate number of unique values"""
+
+ def __init__(self, size=None, error=None):
+ self._sample_size = ApproximateUnique.parse_input_params(size, error)
+
+ def expand(self, pcoll):
+ coder = coders.registry.get_coder(pcoll)
+ return pcoll \
+ | 'CountGlobalUniqueValues' \
+ >> (CombineGlobally(ApproximateUniqueCombineFn(self._sample_size,
+ coder)))
+
+ @typehints.with_input_types(typing.Tuple[K, V])
+ @typehints.with_output_types(typing.Tuple[K, int])
+ class PerKey(PTransform):
+ """ Approximate.PerKey approximate number of unique values per key"""
+
+ def __init__(self, size=None, error=None):
+ self._sample_size = ApproximateUnique.parse_input_params(size, error)
+
+ def expand(self, pcoll):
+ coder = coders.registry.get_coder(pcoll)
+ return pcoll \
+ | 'CountPerKeyUniqueValues' \
+ >> (CombinePerKey(ApproximateUniqueCombineFn(self._sample_size,
+ coder)))
+
+
+class _LargestUnique(object):
+ """
+ An object to keep samples and calculate sample hash space. It is an
+ accumulator of a combine function.
+ """
+ _HASH_SPACE_SIZE = 2.0 * sys.maxsize
+
+ def __init__(self, sample_size):
+ self._sample_size = sample_size
+ self._min_hash = sys.maxsize
+ self._sample_heap = []
+ self._sample_set = set()
+
+ def add(self, element):
+ """
+ :param an element from pcoll.
+ :return: boolean type whether the value is in the heap
+
+ Adds a value to the heap, returning whether the value is (large enough to
+ be) in the heap.
+ """
+ if len(self._sample_heap) >= self._sample_size and element < self._min_hash:
+ return False
+
+ if element not in self._sample_set:
+ self._sample_set.add(element)
+ heapq.heappush(self._sample_heap, element)
+
+ if len(self._sample_heap) > self._sample_size:
+ temp = heapq.heappop(self._sample_heap)
+ self._sample_set.remove(temp)
+ self._min_hash = self._sample_heap[0]
+ elif element < self._min_hash:
+ self._min_hash = element
+
+ return True
+
+ def get_estimate(self):
+ """
+ :return: estimation count of unique values
+
+ If heap size is smaller than sample size, just return heap size.
+ Otherwise, takes into account the possibility of hash collisions,
+ which become more likely than not for 2^32 distinct elements.
+ Note that log(1+x) ~ x for small x, so for sampleSize << maxHash
+ log(1 - sample_size/sample_space) / log(1 - 1/sample_space) ~ sample_size
+ and hence estimate ~ sample_size * hash_space / sample_space
+ as one would expect.
+
+ Given sample_size / sample_space = est / hash_space
+ est = sample_size * hash_space / sample_space
+
+ Given above sample_size approximate,
+ est = log1p(-sample_size/sample_space) / log1p(-1/sample_space)
+ * hash_space / sample_space
+ """
+
+ if len(self._sample_heap) < self._sample_size:
+ return len(self._sample_heap)
+ else:
+ sample_space_size = sys.maxsize - 1.0 * self._min_hash
+ est = (math.log1p(-self._sample_size / sample_space_size)
+ / math.log1p(-1 / sample_space_size)
+ * self._HASH_SPACE_SIZE
+ / sample_space_size)
+
+ return round(est)
+
+
+class ApproximateUniqueCombineFn(CombineFn):
+ """
+ ApproximateUniqueCombineFn computes an estimate of the number of
+ unique values that were combined.
+ """
+
+ def __init__(self, sample_size, coder):
+ self._sample_size = sample_size
+ self._coder = coder
+
+ def create_accumulator(self, *args, **kwargs):
+ return _LargestUnique(self._sample_size)
+
+ def add_input(self, accumulator, element, *args, **kwargs):
+ try:
+ accumulator.add(hash(self._coder.encode(element)))
+ return accumulator
+ except Exception as e:
+ raise RuntimeError("Runtime exception: %s", e)
+
+ # created an issue https://issues.apache.org/jira/browse/BEAM-7285 to speed up
+ # merge process.
+ def merge_accumulators(self, accumulators, *args, **kwargs):
+ merged_accumulator = self.create_accumulator()
+ for accumulator in accumulators:
+ for i in accumulator._sample_heap:
+ merged_accumulator.add(i)
+
+ return merged_accumulator
+
+ @staticmethod
+ def extract_output(accumulator):
+ return accumulator.get_estimate()
+
+ def display_data(self):
+ return {'sample_size': self._sample_size}
+
+
+class ApproximateQuantiles(object):
+ """
+ PTransfrom for getting the idea of data distribution using approximate N-tile
+ (e.g. quartiles, percentiles etc.) either globally or per-key.
+ """
+
+ @staticmethod
+ def _display_data(num_quantiles, key, reverse):
+ return {
+ 'num_quantiles': DisplayDataItem(num_quantiles, label="Quantile Count"),
+ 'key': DisplayDataItem(key.__name__ if hasattr(key, '__name__')
+ else key.__class__.__name__,
+ label='Record Comparer Key'),
+ 'reverse': DisplayDataItem(str(reverse), label='Is reversed')
+ }
+
+ @typehints.with_input_types(T)
+ @typehints.with_output_types(typing.List[T])
+ class Globally(PTransform):
+ """
+ PTransform takes PCollection and returns a list whose single value is
+ approximate N-tiles of the input collection globally.
+
+ Args:
+ num_quantiles: number of elements in the resulting quantiles values list.
+ key: (optional) Key is a mapping of elements to a comparable key, similar
+ to the key argument of Python's sorting methods.
+ reverse: (optional) whether to order things smallest to largest, rather
+ than largest to smallest
+ """
+
+ def __init__(self, num_quantiles, key=None, reverse=False):
+ self._num_quantiles = num_quantiles
+ self._key = key
+ self._reverse = reverse
+
+ def expand(self, pcoll):
+ return pcoll | CombineGlobally(ApproximateQuantilesCombineFn.create(
+ num_quantiles=self._num_quantiles, key=self._key,
+ reverse=self._reverse))
+
+ def display_data(self):
+ return ApproximateQuantiles._display_data(
+ num_quantiles=self._num_quantiles, key=self._key,
+ reverse=self._reverse)
+
+ @typehints.with_input_types(typing.Tuple[K, V])
+ @typehints.with_output_types(typing.Tuple[K, typing.List[V]])
+ class PerKey(PTransform):
+ """
+ PTransform takes PCollection of KV and returns a list based on each key
+ whose single value is list of approximate N-tiles of the input element of
+ the key.
+
+ Args:
+ num_quantiles: number of elements in the resulting quantiles values list.
+ key: (optional) Key is a mapping of elements to a comparable key, similar
+ to the key argument of Python's sorting methods.
+ reverse: (optional) whether to order things smallest to largest, rather
+ than largest to smallest
+ """
+
+ def __init__(self, num_quantiles, key=None, reverse=False):
+ self._num_quantiles = num_quantiles
+ self._key = key
+ self._reverse = reverse
+
+ def expand(self, pcoll):
+ return pcoll | CombinePerKey(ApproximateQuantilesCombineFn.create(
+ num_quantiles=self._num_quantiles, key=self._key,
+ reverse=self._reverse))
+
+ def display_data(self):
+ return ApproximateQuantiles._display_data(
+ num_quantiles=self._num_quantiles, key=self._key,
+ reverse=self._reverse)
+
+
+class _QuantileBuffer(object):
+ """A single buffer in the sense of the referenced algorithm.
+ (see http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.6513&rep=rep1
+ &type=pdf and ApproximateQuantilesCombineFn for further information)"""
+
+ def __init__(self, elements, level=0, weight=1):
+ self.elements = elements
+ self.level = level
+ self.weight = weight
+
+ def __lt__(self, other):
+ self.elements < other.elements
+
+ def sized_iterator(self):
+
+ class QuantileBufferIterator(object):
+ def __init__(self, elem, weight):
+ self._iter = iter(elem)
+ self.weight = weight
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ value = next(self._iter)
+ return (value, self.weight)
+
+ next = __next__ # For Python 2
+
+ return QuantileBufferIterator(self.elements, self.weight)
+
+
+class _QuantileState(object):
+ """
+ Compact summarization of a collection on which quantiles can be estimated.
+ """
+ min_val = None # Holds smallest item in the list
+ max_val = None # Holds largest item in the list
+
+ def __init__(self, buffer_size, num_buffers, unbuffered_elements, buffers):
+ self.buffer_size = buffer_size
+ self.num_buffers = num_buffers
+ self.buffers = buffers
+
+ # The algorithm requires that the manipulated buffers always be filled to
+ # capacity to perform the collapse operation. This operation can be extended
+ # to buffers of varying sizes by introducing the notion of fractional
+ # weights, but it's easier to simply combine the remainders from all shards
+ # into new, full buffers and then take them into account when computing the
+ # final output.
+ self.unbuffered_elements = unbuffered_elements
+
+ def is_empty(self):
+ """Check if the buffered & unbuffered elements are empty or not."""
+ return not self.unbuffered_elements and not self.buffers
+
+
+class ApproximateQuantilesCombineFn(CombineFn):
+ """
+ This combiner gives an idea of the distribution of a collection of values
+ using approximate N-tiles. The output of this combiner is the list of size of
+ the number of quantiles (num_quantiles), containing the input values of the
+ minimum value item of the list, the intermediate values (n-tiles) and the
+ maximum value item of the list, in the sort order provided via key (similar
+ to the key argument of Python's sorting methods).
+
+ If there are fewer values to combine than the number of quantile
+ (num_quantiles), then the resulting list will contain all the values being
+ combined, in sorted order.
+
+ If no `key` is provided, then the results are sorted in the natural order.
+
+ To evaluate the quantiles, we use the "New Algorithm" described here:
+
+ [MRL98] Manku, Rajagopalan & Lindsay, "Approximate Medians and other
+ Quantiles in One Pass and with Limited Memory", Proc. 1998 ACM SIGMOD,
+ Vol 27, No 2, p 426-435, June 1998.
+ http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.6513&rep=rep1
+ &type=pdf
+
+ The default error bound is (1 / N), though in practice the accuracy
+ tends to be much better.
+
+ Args:
+ num_quantiles: Number of quantiles to produce. It is the size of the final
+ output list, including the mininum and maximum value items.
+ buffer_size: The size of the buffers, corresponding to k in the referenced
+ paper.
+ num_buffers: The number of buffers, corresponding to b in the referenced
+ paper.
+ key: (optional) Key is a mapping of elements to a comparable key, similar
+ to the key argument of Python's sorting methods.
+ reverse: (optional) whether to order things smallest to largest, rather
+ than largest to smallest
+ """
+
+ # For alternating between biasing up and down in the above even weight
+ # collapse operation.
+ _offset_jitter = 0
+
+ # The cost (in time and space) to compute quantiles to a given accuracy is a
+ # function of the total number of elements in the data set. If an estimate is
+ # not known or specified, we use this as an upper bound. If this is too low,
+ # errors may exceed the requested tolerance; if too high, efficiency may be
+ # non-optimal. The impact is logarithmic with respect to this value, so this
+ # default should be fine for most uses.
+ _MAX_NUM_ELEMENTS = 1e9
+ _qs = None # Refers to the _QuantileState
+
+ def __init__(self, num_quantiles, buffer_size, num_buffers, key=None,
+ reverse=False):
+ if key:
+ self._comparator = lambda a, b: (key(a) < key(b)) - (key(a) > key(b)) \
+ if reverse else (key(a) > key(b)) - (key(a) < key(b))
+ else:
+ self._comparator = lambda a, b: (a < b) - (a > b) if reverse \
+ else (a > b) - (a < b)
+
+ self._num_quantiles = num_quantiles
+ self._buffer_size = buffer_size
+ self._num_buffers = num_buffers
+ self._key = key
+ self._reverse = reverse
+
+ @classmethod
+ def create(cls, num_quantiles, epsilon=None, max_num_elements=None, key=None,
+ reverse=False):
+ """
+ Creates an approximate quantiles combiner with the given key and desired
+ number of quantiles.
+
+ Args:
+ num_quantiles: Number of quantiles to produce. It is the size of the
+ final output list, including the mininum and maximum value items.
+ epsilon: (optional) The default error bound is `epsilon`, which holds as
+ long as the number of elements is less than `_MAX_NUM_ELEMENTS`.
+ Specifically, if one considers the input as a sorted list x_1, ...,
+ x_N, then the distance between each exact quantile x_c and its
+ approximation x_c' is bounded by `|c - c'| < epsilon * N`. Note that
+ these errors are worst-case scenarios. In practice the accuracy tends
+ to be much better.
+ max_num_elements: (optional) The cost (in time and space) to compute
+ quantiles to a given accuracy is a function of the total number of
+ elements in the data set.
+ key: (optional) Key is a mapping of elements to a comparable key, similar
+ to the key argument of Python's sorting methods.
+ reverse: (optional) whether to order things smallest to largest, rather
+ than largest to smallest
+ """
+ max_num_elements = max_num_elements or cls._MAX_NUM_ELEMENTS
+ if not epsilon:
+ epsilon = 1.0 / num_quantiles
+ b = 2
+ while (b - 2) * (1 << (b - 2)) < epsilon * max_num_elements:
+ b = b + 1
+ b = b - 1
+ k = max(2, math.ceil(max_num_elements / float(1 << (b - 1))))
+ return cls(num_quantiles=num_quantiles, buffer_size=k, num_buffers=b,
+ key=key, reverse=reverse)
+
+ def _add_unbuffered(self, qs, elem):
+ """
+ Add a new buffer to the unbuffered list, creating a new buffer and
+ collapsing if needed.
+ """
+ qs.unbuffered_elements.append(elem)
+ if len(qs.unbuffered_elements) == qs.buffer_size:
+ qs.unbuffered_elements.sort(key=self._key, reverse=self._reverse)
+ heapq.heappush(qs.buffers,
+ _QuantileBuffer(elements=qs.unbuffered_elements))
+ qs.unbuffered_elements = []
+ self._collapse_if_needed(qs)
+
+ def _offset(self, newWeight):
+ """
+ If the weight is even, we must round up or down. Alternate between these
+ two options to avoid a bias.
+ """
+ if newWeight % 2 == 1:
+ return (newWeight + 1) / 2
+ else:
+ self._offset_jitter = 2 - self._offset_jitter
+ return (newWeight + self._offset_jitter) / 2
+
+ def _collapse(self, buffers):
+ new_level = 0
+ new_weight = 0
+ for buffer_elem in buffers:
+ # As presented in the paper, there should always be at least two
+ # buffers of the same (minimal) level to collapse, but it is possible
+ # to violate this condition when combining buffers from independently
+ # computed shards. If they differ we take the max.
+ new_level = max([new_level, buffer_elem.level + 1])
+ new_weight = new_weight + buffer_elem.weight
+ new_elements = self._interpolate(buffers, self._buffer_size, new_weight,
+ self._offset(new_weight))
+ return _QuantileBuffer(new_elements, new_level, new_weight)
+
+ def _collapse_if_needed(self, qs):
+ while len(qs.buffers) > self._num_buffers:
+ toCollapse = []
+ toCollapse.append(heapq.heappop(qs.buffers))
+ toCollapse.append(heapq.heappop(qs.buffers))
+ minLevel = toCollapse[1].level
+
+ while len(qs.buffers) > 0 and qs.buffers[0].level == minLevel:
+ toCollapse.append(heapq.heappop(qs.buffers))
+
+ heapq.heappush(qs.buffers, self._collapse(toCollapse))
+
+ def _interpolate(self, i_buffers, count, step, offset):
+ """
+ Emulates taking the ordered union of all elements in buffers, repeated
+ according to their weight, and picking out the (k * step + offset)-th
+ elements of this list for `0 <= k < count`.
+ """
+
+ iterators = []
+ new_elements = []
+ compare_key = None
+ if self._key:
+ compare_key = lambda x: self._key(x[0])
+ for buffer_elem in i_buffers:
+ iterators.append(buffer_elem.sized_iterator())
+
+ # Python 3 `heapq.merge` support key comparison and returns an iterator and
+ # does not pull the data into memory all at once. Python 2 does not
+ # support comparison on its `heapq.merge` api, so we use the itertools
+ # which takes the `key` function for comparison and creates an iterator
+ # from it.
+ if sys.version_info[0] < 3:
+ sorted_elem = iter(
+ sorted(itertools.chain.from_iterable(iterators), key=compare_key,
+ reverse=self._reverse))
+ else:
+ sorted_elem = heapq.merge(*iterators, key=compare_key,
+ reverse=self._reverse)
+
+ weighted_element = next(sorted_elem)
+ current = weighted_element[1]
+ j = 0
+ while j < count:
+ target = j * step + offset
+ j = j + 1
+ try:
+ while current <= target:
+ weighted_element = next(sorted_elem)
+ current = current + weighted_element[1]
+ except StopIteration:
+ pass
+ new_elements.append(weighted_element[0])
+ return new_elements
+
+ def create_accumulator(self):
+ self._qs = _QuantileState(buffer_size=self._buffer_size,
+ num_buffers=self._num_buffers,
+ unbuffered_elements=[], buffers=[])
+ return self._qs
+
+ def add_input(self, quantile_state, element):
+ """
+ Add a new element to the collection being summarized by quantile state.
+ """
+ if quantile_state.is_empty():
+ quantile_state.min_val = quantile_state.max_val = element
+ elif self._comparator(element, quantile_state.min_val) < 0:
+ quantile_state.min_val = element
+ elif self._comparator(element, quantile_state.max_val) > 0:
+ quantile_state.max_val = element
+ self._add_unbuffered(quantile_state, elem=element)
+ return quantile_state
+
+ def merge_accumulators(self, accumulators):
+ """Merges all the accumulators (quantile state) as one."""
+ qs = self.create_accumulator()
+ for accumulator in accumulators:
+ if accumulator.is_empty():
+ continue
+ if not qs.min_val or self._comparator(accumulator.min_val,
+ qs.min_val) < 0:
+ qs.min_val = accumulator.min_val
+ if not qs.max_val or self._comparator(accumulator.max_val,
+ qs.max_val) > 0:
+ qs.max_val = accumulator.max_val
+
+ for unbuffered_element in accumulator.unbuffered_elements:
+ self._add_unbuffered(qs, unbuffered_element)
+
+ qs.buffers.extend(accumulator.buffers)
+ self._collapse_if_needed(qs)
+ return qs
+
+ def extract_output(self, accumulator):
+ """
+ Outputs num_quantiles elements consisting of the minimum, maximum and
+ num_quantiles - 2 evenly spaced intermediate elements. Returns the empty
+ list if no elements have been added.
+ """
+ if accumulator.is_empty():
+ return []
+
+ all_elems = accumulator.buffers
+ total_count = len(accumulator.unbuffered_elements)
+ for buffer_elem in all_elems:
+ total_count = total_count + accumulator.buffer_size * buffer_elem.weight
+
+ if accumulator.unbuffered_elements:
+ accumulator.unbuffered_elements.sort(key=self._key, reverse=self._reverse)
+ all_elems.append(_QuantileBuffer(accumulator.unbuffered_elements))
+
+ step = 1.0 * total_count / (self._num_quantiles - 1)
+ offset = (1.0 * total_count - 1) / (self._num_quantiles - 1)
+ quantiles = [accumulator.min_val]
+ quantiles.extend(
+ self._interpolate(all_elems, self._num_quantiles - 2, step, offset))
+ quantiles.append(accumulator.max_val)
+ return quantiles
diff --git a/sdks/python/apache_beam/transforms/stats_test.py b/sdks/python/apache_beam/transforms/stats_test.py
new file mode 100644
index 0000000..14027fd
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/stats_test.py
@@ -0,0 +1,624 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import division
+
+import math
+import random
+import unittest
+from builtins import range
+from collections import defaultdict
+
+import hamcrest as hc
+from parameterized import parameterized
+from tenacity import retry
+from tenacity import stop_after_attempt
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.display import DisplayData
+from apache_beam.transforms.display_test import DisplayDataItemMatcher
+from apache_beam.transforms.stats import ApproximateQuantilesCombineFn
+
+
+class ApproximateUniqueTest(unittest.TestCase):
+ """Unit tests for ApproximateUnique.Globally and ApproximateUnique.PerKey.
+ Hash() with Python3 is nondeterministic, so Approximation algorithm generates
+ different result each time and sometimes error rate is out of range, so add
+ retries for all tests who actually running approximation algorithm."""
+
+ def test_approximate_unique_global_by_invalid_size(self):
+ # test if the transformation throws an error as expected with an invalid
+ # small input size (< 16).
+ sample_size = 10
+ test_input = [random.randint(0, 1000) for _ in range(100)]
+
+ with self.assertRaises(ValueError) as e:
+ pipeline = TestPipeline()
+ _ = (pipeline
+ | 'create'
+ >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(size=sample_size))
+ pipeline.run()
+
+ expected_msg = beam.ApproximateUnique._INPUT_SIZE_ERR_MSG % (sample_size)
+
+ assert e.exception.args[0] == expected_msg
+
+ def test_approximate_unique_global_by_invalid_type_size(self):
+ # test if the transformation throws an error as expected with an invalid
+ # type of input size (not int).
+ sample_size = 100.0
+ test_input = [random.randint(0, 1000) for _ in range(100)]
+
+ with self.assertRaises(ValueError) as e:
+ pipeline = TestPipeline()
+ _ = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(size=sample_size))
+ pipeline.run()
+
+ expected_msg = beam.ApproximateUnique._INPUT_SIZE_ERR_MSG % (sample_size)
+
+ assert e.exception.args[0] == expected_msg
+
+ def test_approximate_unique_global_by_invalid_small_error(self):
+ # test if the transformation throws an error as expected with an invalid
+ # small input error (< 0.01).
+ est_err = 0.0
+ test_input = [random.randint(0, 1000) for _ in range(100)]
+
+ with self.assertRaises(ValueError) as e:
+ pipeline = TestPipeline()
+ _ = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(error=est_err))
+ pipeline.run()
+
+ expected_msg = beam.ApproximateUnique._INPUT_ERROR_ERR_MSG % (est_err)
+
+ assert e.exception.args[0] == expected_msg
+
+ def test_approximate_unique_global_by_invalid_big_error(self):
+ # test if the transformation throws an error as expected with an invalid
+ # big input error (> 0.50).
+ est_err = 0.6
+ test_input = [random.randint(0, 1000) for _ in range(100)]
+
+ with self.assertRaises(ValueError) as e:
+ pipeline = TestPipeline()
+ _ = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(error=est_err))
+ pipeline.run()
+
+ expected_msg = beam.ApproximateUnique._INPUT_ERROR_ERR_MSG % (est_err)
+
+ assert e.exception.args[0] == expected_msg
+
+ def test_approximate_unique_global_by_invalid_no_input(self):
+ # test if the transformation throws an error as expected with no input.
+ test_input = [random.randint(0, 1000) for _ in range(100)]
+
+ with self.assertRaises(ValueError) as e:
+ pipeline = TestPipeline()
+ _ = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally())
+ pipeline.run()
+
+ expected_msg = beam.ApproximateUnique._NO_VALUE_ERR_MSG
+ assert e.exception.args[0] == expected_msg
+
+ def test_approximate_unique_global_by_invalid_both_input(self):
+ # test if the transformation throws an error as expected with multi input.
+ test_input = [random.randint(0, 1000) for _ in range(100)]
+ est_err = 0.2
+ sample_size = 30
+
+ with self.assertRaises(ValueError) as e:
+ pipeline = TestPipeline()
+ _ = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(size=sample_size, error=est_err))
+ pipeline.run()
+
+ expected_msg = beam.ApproximateUnique._MULTI_VALUE_ERR_MSG % (
+ sample_size, est_err)
+
+ assert e.exception.args[0] == expected_msg
+
+ def test_get_sample_size_from_est_error(self):
+ # test if get correct sample size from input error.
+ assert beam.ApproximateUnique._get_sample_size_from_est_error(0.5) == 16
+ assert beam.ApproximateUnique._get_sample_size_from_est_error(0.4) == 25
+ assert beam.ApproximateUnique._get_sample_size_from_est_error(0.2) == 100
+ assert beam.ApproximateUnique._get_sample_size_from_est_error(0.1) == 400
+ assert beam.ApproximateUnique._get_sample_size_from_est_error(0.05) == 1600
+ assert beam.ApproximateUnique._get_sample_size_from_est_error(0.01) == 40000
+
+ @unittest.skip('Skip it because hash function is not good enough. '
+ 'TODO: BEAM-7654')
+ def test_approximate_unique_global_by_sample_size(self):
+ # test if estimation error with a given sample size is not greater than
+ # expected max error.
+ sample_size = 16
+ max_err = 2 / math.sqrt(sample_size)
+ test_input = [4, 34, 29, 46, 80, 66, 51, 81, 31, 9, 26, 36, 10, 41, 90, 35,
+ 33, 19, 88, 86, 28, 93, 38, 76, 15, 87, 12, 39, 84, 13, 32,
+ 49, 65, 100, 16, 27, 23, 30, 96, 54]
+
+ actual_count = len(set(test_input))
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(size=sample_size)
+ | 'compare'
+ >> beam.FlatMap(lambda x: [abs(x - actual_count) * 1.0
+ / actual_count <= max_err]))
+
+ assert_that(result, equal_to([True]),
+ label='assert:global_by_size')
+ pipeline.run()
+
+ @retry(reraise=True, stop=stop_after_attempt(5))
+ def test_approximate_unique_global_by_sample_size_with_duplicates(self):
+ # test if estimation error with a given sample size is not greater than
+ # expected max error with duplicated input.
+ sample_size = 30
+ max_err = 2 / math.sqrt(sample_size)
+ test_input = [10] * 50 + [20] * 50
+ actual_count = len(set(test_input))
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(size=sample_size)
+ | 'compare'
+ >> beam.FlatMap(lambda x: [abs(x - actual_count) * 1.0
+ / actual_count <= max_err]))
+
+ assert_that(result, equal_to([True]),
+ label='assert:global_by_size_with_duplicates')
+ pipeline.run()
+
+ @retry(reraise=True, stop=stop_after_attempt(5))
+ def test_approximate_unique_global_by_sample_size_with_small_population(self):
+ # test if estimation is exactly same to actual value when sample size is
+ # not smaller than population size (sample size > 100% of population).
+ sample_size = 31
+ test_input = [144, 160, 229, 923, 390, 756, 674, 769, 145, 888,
+ 809, 159, 222, 101, 943, 901, 876, 194, 232, 631,
+ 221, 829, 965, 729, 35, 33, 115, 894, 827, 364]
+ actual_count = len(set(test_input))
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(size=sample_size))
+
+ assert_that(result, equal_to([actual_count]),
+ label='assert:global_by_sample_size_with_small_population')
+ pipeline.run()
+
+ @unittest.skip('Skip because hash function is not good enough. '
+ 'TODO: BEAM-7654')
+ def test_approximate_unique_global_by_error(self):
+ # test if estimation error from input error is not greater than input error.
+ est_err = 0.3
+ test_input = [291, 371, 271, 126, 762, 391, 222, 565, 428, 786,
+ 801, 867, 337, 690, 261, 436, 311, 568, 946, 722,
+ 973, 386, 506, 546, 991, 450, 226, 889, 514, 693]
+ actual_count = len(set(test_input))
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(error=est_err)
+ | 'compare'
+ >> beam.FlatMap(lambda x: [abs(x - actual_count) * 1.0
+ / actual_count <= est_err]))
+
+ assert_that(result, equal_to([True]), label='assert:global_by_error')
+ pipeline.run()
+
+ @retry(reraise=True, stop=stop_after_attempt(5))
+ def test_approximate_unique_global_by_error_with_small_population(self):
+ # test if estimation error from input error of a small dataset is not
+ # greater than input error. Sample size is always not smaller than 16, so
+ # when population size is smaller than 16, estimation should be exactly
+ # same to actual value.
+ est_err = 0.01
+ test_input = [585, 104, 613, 503, 658, 640, 118, 492, 189, 798,
+ 756, 755, 839, 79, 393]
+ actual_count = len(set(test_input))
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(error=est_err))
+
+ assert_that(result, equal_to([actual_count]),
+ label='assert:global_by_error_with_small_population')
+ pipeline.run()
+
+ @retry(reraise=True, stop=stop_after_attempt(5))
+ def test_approximate_unique_perkey_by_size(self):
+ # test if est error per key from sample size is in a expected range.
+ sample_size = 20
+ max_err = 2 / math.sqrt(sample_size)
+ test_input = [(8, 73), (6, 724), (7, 70), (1, 576), (10, 120), (2, 662),
+ (7, 115), (3, 731), (6, 340), (6, 623), (1, 74), (9, 280),
+ (8, 298), (6, 440), (10, 243), (1, 125), (9, 754), (8, 833),
+ (9, 751), (4, 818), (6, 176), (9, 253), (2, 721), (8, 936),
+ (3, 691), (10, 685), (1, 69), (3, 155), (8, 86), (5, 693),
+ (2, 809), (4, 723), (8, 102), (9, 707), (8, 558), (4, 537),
+ (5, 371), (7, 432), (2, 51), (10, 397)]
+ actual_count_dict = defaultdict(set)
+ for (x, y) in test_input:
+ actual_count_dict[x].add(y)
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.PerKey(size=sample_size)
+ | 'compare'
+ >> beam.FlatMap(lambda x: [abs(x[1]
+ - len(actual_count_dict[x[0]]))
+ * 1.0 / len(actual_count_dict[x[0]])
+ <= max_err]))
+
+ assert_that(result, equal_to([True] * len(actual_count_dict)),
+ label='assert:perkey_by_size')
+ pipeline.run()
+
+ @retry(reraise=True, stop=stop_after_attempt(5))
+ def test_approximate_unique_perkey_by_error(self):
+ # test if estimation error per key from input err is in the expected range.
+ est_err = 0.01
+ test_input = [(9, 6), (5, 5), (6, 9), (2, 4), (8, 3), (9, 0), (6, 10),
+ (8, 8), (9, 7), (2, 0), (9, 2), (1, 3), (4, 0), (7, 6),
+ (10, 6), (4, 7), (5, 8), (7, 2), (7, 10), (5, 10)]
+ actual_count_dict = defaultdict(set)
+ for (x, y) in test_input:
+ actual_count_dict[x].add(y)
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.PerKey(error=est_err)
+ | 'compare'
+ >> beam.FlatMap(lambda x: [abs(x[1]
+ - len(actual_count_dict[x[0]]))
+ * 1.0 / len(actual_count_dict[x[0]])
+ <= est_err]))
+
+ assert_that(result, equal_to([True] * len(actual_count_dict)),
+ label='assert:perkey_by_error')
+ pipeline.run()
+
+ @retry(reraise=True, stop=stop_after_attempt(5))
+ def test_approximate_unique_globally_by_error_with_skewed_data(self):
+ # test if estimation error is within the expected range with skewed data.
+ est_err = 0.01
+ test_input = [19, 21, 32, 29, 5, 31, 52, 50, 59, 80, 7, 3, 34, 19, 13,
+ 6, 55, 1, 13, 90, 4, 18, 52, 33, 0, 77, 21, 26, 5, 18]
+ actual_count = len(set(test_input))
+
+ pipeline = TestPipeline()
+ result = (pipeline
+ | 'create' >> beam.Create(test_input)
+ | 'get_estimate'
+ >> beam.ApproximateUnique.Globally(error=est_err)
+ | 'compare'
+ >> beam.FlatMap(lambda x: [abs(x - actual_count) * 1.0
+ / actual_count <= est_err]))
+
+ assert_that(result, equal_to([True]),
+ label='assert:globally_by_error_with_skewed_data')
+ pipeline.run()
+
+
+class ApproximateQuantilesTest(unittest.TestCase):
+ _kv_data = [("a", 1), ("a", 2), ("a", 3), ("b", 1), ("b", 10), ("b", 10),
+ ("b", 100)]
+ _kv_str_data = [("a", "a"), ("a", "a"*2), ("a", "a"*3), ("b", "b"),
+ ("b", "b"*10), ("b", "b"*10), ("b", "b"*100)]
+
+ @staticmethod
+ def _quantiles_matcher(expected):
+ l = len(expected)
+
+ def assert_true(exp):
+ if not exp:
+ raise BeamAssertException('%s Failed assert True' % repr(exp))
+
+ def match(actual):
+ actual = actual[0]
+ for i in range(l):
+ if isinstance(expected[i], list):
+ assert_true(expected[i][0] <= actual[i] <= expected[i][1])
+ else:
+ equal_to([expected[i]])([actual[i]])
+
+ return match
+
+ @staticmethod
+ def _approx_quantile_generator(size, num_of_quantiles, absoluteError):
+ quantiles = [0]
+ k = 1
+ while k < num_of_quantiles - 1:
+ expected = (size - 1) * k / (num_of_quantiles - 1)
+ quantiles.append([expected - absoluteError, expected + absoluteError])
+ k = k + 1
+ quantiles.append(size - 1)
+ return quantiles
+
+ def test_quantiles_globaly(self):
+ with TestPipeline() as p:
+ pc = p | Create(list(range(101)))
+
+ quantiles = pc | 'Quantiles globally' >> \
+ beam.ApproximateQuantiles.Globally(5)
+ quantiles_reversed = pc | 'Quantiles globally reversed' >> \
+ beam.ApproximateQuantiles.Globally(5, reverse=True)
+
+ assert_that(quantiles, equal_to([[0, 25, 50, 75, 100]]),
+ label='checkQuantilesGlobally')
+ assert_that(quantiles_reversed, equal_to([[100, 75, 50, 25, 0]]),
+ label='checkReversedQuantiles')
+
+ def test_quantiles_per_key(self):
+ with TestPipeline() as p:
+ data = self._kv_data
+ pc = p | Create(data)
+
+ per_key = pc | 'Quantiles PerKey' >> beam.ApproximateQuantiles.PerKey(2)
+ per_key_reversed = (pc | 'Quantiles PerKey Reversed' >>
+ beam.ApproximateQuantiles.PerKey(2, reverse=True))
+
+ assert_that(per_key, equal_to([('a', [1, 3]), ('b', [1, 100])]),
+ label='checkQuantilePerKey')
+ assert_that(per_key_reversed, equal_to([('a', [3, 1]), ('b', [100, 1])]),
+ label='checkReversedQuantilesPerKey')
+
+ def test_quantiles_per_key_with_key_argument(self):
+ with TestPipeline() as p:
+ data = self._kv_str_data
+ pc = p | Create(data)
+
+ per_key = pc | 'Per Key' >> beam.ApproximateQuantiles.PerKey(2, key=len)
+ per_key_reversed = (pc | 'Per Key Reversed' >> beam.ApproximateQuantiles.
+ PerKey(2, key=len, reverse=True))
+
+ assert_that(per_key, equal_to([('a', ['a', 'a' * 3]),
+ ('b', ['b', 'b' * 100])]),
+ label='checkPerKey')
+ assert_that(per_key_reversed, equal_to([('a', ['a'*3, 'a']),
+ ('b', ['b'*100, 'b'])]),
+ label='checkPerKeyReversed')
+
+ def test_singleton(self):
+ with TestPipeline() as p:
+ data = [389]
+ pc = p | Create(data)
+ qunatiles = pc | beam.ApproximateQuantiles.Globally(5)
+ assert_that(qunatiles, equal_to([[389, 389, 389, 389, 389]]))
+
+ def test_uneven_quantiles(self):
+ with TestPipeline() as p:
+ pc = p | Create(list(range(5000)))
+ qunatiles = pc | beam.ApproximateQuantiles.Globally(37)
+ aprox_quantiles = self._approx_quantile_generator(size=5000,
+ num_of_quantiles=37,
+ absoluteError=20)
+ assert_that(qunatiles, self._quantiles_matcher(aprox_quantiles))
+
+ def test_large_quantiles(self):
+ with TestPipeline() as p:
+ pc = p | Create(list(range(10001)))
+ qunatiles = pc | beam.ApproximateQuantiles.Globally(50)
+ aprox_quantiles = self._approx_quantile_generator(size=10001,
+ num_of_quantiles=50,
+ absoluteError=20)
+ assert_that(qunatiles, self._quantiles_matcher(aprox_quantiles))
+
+ def test_random_quantiles(self):
+ with TestPipeline() as p:
+ data = list(range(101))
+ random.shuffle(data)
+ pc = p | Create(data)
+ quantiles = pc | beam.ApproximateQuantiles.Globally(5)
+ assert_that(quantiles, equal_to([[0, 25, 50, 75, 100]]))
+
+ def test_duplicates(self):
+ y = list(range(101))
+ data = []
+ for _ in range(10):
+ data.extend(y)
+
+ with TestPipeline() as p:
+ pc = p | Create(data)
+ quantiles = (pc | 'Quantiles Globally' >>
+ beam.ApproximateQuantiles.Globally(5))
+ quantiles_reversed = (pc | 'Quantiles Reversed' >>
+ beam.ApproximateQuantiles.Globally(5, reverse=True))
+
+ assert_that(quantiles, equal_to([[0, 25, 50, 75, 100]]),
+ label="checkQuantilesGlobally")
+ assert_that(quantiles_reversed, equal_to([[100, 75, 50, 25, 0]]),
+ label="checkQuantileReversed")
+
+ def test_lots_of_duplicates(self):
+ with TestPipeline() as p:
+ data = [1]
+ data.extend([2 for _ in range(299)])
+ data.extend([3 for _ in range(799)])
+ pc = p | Create(data)
+ quantiles = pc | beam.ApproximateQuantiles.Globally(5)
+ assert_that(quantiles, equal_to([[1, 2, 3, 3, 3]]))
+
+ def test_log_distribution(self):
+ with TestPipeline() as p:
+ data = [int(math.log(x)) for x in range(1, 1000)]
+ pc = p | Create(data)
+ quantiles = pc | beam.ApproximateQuantiles.Globally(5)
+ assert_that(quantiles, equal_to([[0, 5, 6, 6, 6]]))
+
+ def test_zipfian_distribution(self):
+ with TestPipeline() as p:
+ data = []
+ for i in range(1, 1000):
+ data.append(int(1000 / i))
+ pc = p | Create(data)
+ quantiles = pc | beam.ApproximateQuantiles.Globally(5)
+ assert_that(quantiles, equal_to([[1, 1, 2, 4, 1000]]))
+
+ def test_alternate_quantiles(self):
+ data = ["aa", "aaa", "aaaa", "b", "ccccc", "dddd", "zz"]
+ with TestPipeline() as p:
+ pc = p | Create(data)
+
+ globally = pc | 'Globally' >> beam.ApproximateQuantiles.Globally(3)
+ with_key = (pc | 'Globally with key' >>
+ beam.ApproximateQuantiles.Globally(3, key=len))
+ key_with_reversed = (pc | 'Globally with key and reversed' >>
+ beam.ApproximateQuantiles.Globally(
+ 3, key=len, reverse=True))
+
+ assert_that(globally, equal_to([["aa", "b", "zz"]]),
+ label='checkGlobally')
+ assert_that(with_key, equal_to([["b", "aaa", "ccccc"]]),
+ label='checkGloballyWithKey')
+ assert_that(key_with_reversed, equal_to([["ccccc", "aaa", "b"]]),
+ label='checkWithKeyAndReversed')
+
+ @staticmethod
+ def _display_data_matcher(instance):
+ expected_items = [
+ DisplayDataItemMatcher('num_quantiles', instance._num_quantiles),
+ DisplayDataItemMatcher('key', str(instance._key.__name__)),
+ DisplayDataItemMatcher('reverse', str(instance._reverse))
+ ]
+ return expected_items
+
+ def test_global_display_data(self):
+ transform = beam.ApproximateQuantiles.Globally(3, key=len, reverse=True)
+ data = DisplayData.create_from(transform)
+ expected_items = self._display_data_matcher(transform)
+ hc.assert_that(data.items, hc.contains_inanyorder(*expected_items))
+
+ def test_perkey_display_data(self):
+ transform = beam.ApproximateQuantiles.PerKey(3, key=len, reverse=True)
+ data = DisplayData.create_from(transform)
+ expected_items = self._display_data_matcher(transform)
+ hc.assert_that(data.items, hc.contains_inanyorder(*expected_items))
+
+
+def _build_quantilebuffer_test_data():
+ """
+ Test data taken from "Munro-Paterson Algorithm" reference values table of
+ "Approximate Medians and other Quantiles in One Pass and with Limited Memory"
+ paper. See ApproximateQuantilesCombineFn for paper reference.
+ """
+ epsilons = [0.1, 0.05, 0.01, 0.005, 0.001]
+ maxElementExponents = [5, 6, 7, 8, 9]
+ expectedNumBuffersValues = [
+ [11, 14, 17, 21, 24],
+ [11, 14, 17, 20, 23],
+ [9, 11, 14, 17, 21],
+ [8, 11, 14, 17, 20],
+ [6, 9, 11, 14, 17]
+ ]
+ expectedBufferSizeValues = [
+ [98, 123, 153, 96, 120],
+ [98, 123, 153, 191, 239],
+ [391, 977, 1221, 1526, 954],
+ [782, 977, 1221, 1526, 1908],
+ [3125, 3907, 9766, 12208, 15259]
+ ]
+ test_data = list()
+ i = 0
+ for epsilon in epsilons:
+ j = 0
+ for maxElementExponent in maxElementExponents:
+ test_data.append([
+ epsilon,
+ (10 ** maxElementExponent),
+ expectedNumBuffersValues[i][j],
+ expectedBufferSizeValues[i][j]
+ ])
+ j += 1
+ i += 1
+ return test_data
+
+
+class ApproximateQuantilesBufferTest(unittest.TestCase):
+ """ Approximate Quantiles Buffer Tests to ensure we are calculating the
+ optimal buffers."""
+
+ @parameterized.expand(_build_quantilebuffer_test_data)
+ def test_efficiency(self, epsilon, maxInputSize, expectedNumBuffers,
+ expectedBufferSize):
+ """
+ Verify the buffers are efficiently calculated according to the reference
+ table values.
+ """
+
+ combine_fn = ApproximateQuantilesCombineFn.create(
+ num_quantiles=10, max_num_elements=maxInputSize, epsilon=epsilon)
+ self.assertEqual(expectedNumBuffers, combine_fn._num_buffers,
+ "Number of buffers")
+ self.assertEqual(expectedBufferSize, combine_fn._buffer_size, "Buffer size")
+
+ @parameterized.expand(_build_quantilebuffer_test_data)
+ def test_correctness(self, epsilon, maxInputSize, *args):
+ """
+ Verify that buffers are correct according to the two constraint equations.
+ """
+ combine_fn = ApproximateQuantilesCombineFn.create(
+ num_quantiles=10, max_num_elements=maxInputSize, epsilon=epsilon)
+ b = combine_fn._num_buffers
+ k = combine_fn._buffer_size
+ n = maxInputSize
+ self.assertLessEqual((b - 2) * (1 << (b - 2)) + 0.5, (epsilon * n),
+ '(b-2)2^(b-2) + 1/2 <= eN')
+ self.assertGreaterEqual((k * 2) ** (b - 1), n, 'k2^(b-1) >= N')
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/transforms/trigger.py b/sdks/python/apache_beam/transforms/trigger.py
index b0d9a25..e1db856 100644
--- a/sdks/python/apache_beam/transforms/trigger.py
+++ b/sdks/python/apache_beam/transforms/trigger.py
@@ -67,8 +67,7 @@
class AccumulationMode(object):
- """Controls what to do with data when a trigger fires multiple times.
- """
+ """Controls what to do with data when a trigger fires multiple times."""
DISCARDING = beam_runner_api_pb2.AccumulationMode.DISCARDING
ACCUMULATING = beam_runner_api_pb2.AccumulationMode.ACCUMULATING
# TODO(robertwb): Provide retractions of previous outputs.
@@ -78,10 +77,7 @@
class _StateTag(with_metaclass(ABCMeta, object)):
"""An identifier used to store and retrieve typed, combinable state.
- The given tag must be unique for this stage. If CombineFn is None then
- all elements will be returned as a list, otherwise the given CombineFn
- will be applied (possibly incrementally and eagerly) when adding elements.
- """
+ The given tag must be unique for this step."""
def __init__(self, tag):
self.tag = tag
@@ -97,8 +93,21 @@
return _ValueStateTag(prefix + self.tag)
+class _SetStateTag(_StateTag):
+ """StateTag pointing to an element."""
+
+ def __repr__(self):
+ return 'SetStateTag({tag})'.format(tag=self.tag)
+
+ def with_prefix(self, prefix):
+ return _SetStateTag(prefix + self.tag)
+
+
class _CombiningValueStateTag(_StateTag):
- """StateTag pointing to an element, accumulated with a combiner."""
+ """StateTag pointing to an element, accumulated with a combiner.
+
+ The given tag must be unique for this step. The given CombineFn will be
+ applied (possibly incrementally and eagerly) when adding elements."""
# TODO(robertwb): Also store the coder (perhaps extracted from the combine_fn)
def __init__(self, tag, combine_fn):
@@ -297,6 +306,7 @@
"""
def __init__(self, delay=0):
+ """Initialize a processing time trigger with a delay in seconds."""
self.delay = delay
def __repr__(self):
@@ -327,12 +337,12 @@
proto.after_processing_time
.timestamp_transforms[0]
.delay
- .delay_millis))
+ .delay_millis) // 1000)
def to_runner_api(self, context):
delay_proto = beam_runner_api_pb2.TimestampTransform(
delay=beam_runner_api_pb2.TimestampTransform.Delay(
- delay_millis=self.delay))
+ delay_millis=self.delay * 1000))
return beam_runner_api_pb2.Trigger(
after_processing_time=beam_runner_api_pb2.Trigger.AfterProcessingTime(
timestamp_transforms=[delay_proto]))
@@ -865,6 +875,8 @@
original_tag.combine_fn.merge_accumulators(values))
elif isinstance(tag, _ListStateTag):
return [v for vs in values for v in vs]
+ elif isinstance(tag, _SetStateTag):
+ return {v for vs in values for v in vs}
elif isinstance(tag, _WatermarkHoldStateTag):
return tag.timestamp_combiner_impl.combine_all(values)
else:
@@ -1226,6 +1238,8 @@
self.state[window][tag.tag].append(value)
elif isinstance(tag, _ListStateTag):
self.state[window][tag.tag].append(value)
+ elif isinstance(tag, _SetStateTag):
+ self.state[window][tag.tag].append(value)
elif isinstance(tag, _WatermarkHoldStateTag):
self.state[window][tag.tag].append(value)
else:
@@ -1239,6 +1253,8 @@
return tag.combine_fn.apply(values)
elif isinstance(tag, _ListStateTag):
return values
+ elif isinstance(tag, _SetStateTag):
+ return values
elif isinstance(tag, _WatermarkHoldStateTag):
return tag.timestamp_combiner_impl.combine_all(values)
else:
diff --git a/sdks/python/apache_beam/transforms/trigger_test.py b/sdks/python/apache_beam/transforms/trigger_test.py
index aaea930..11ad465 100644
--- a/sdks/python/apache_beam/transforms/trigger_test.py
+++ b/sdks/python/apache_beam/transforms/trigger_test.py
@@ -617,7 +617,7 @@
if action != 'expect':
# Fail if we have output that was not expected in the transcript.
- self.assertEquals(
+ self.assertEqual(
[], output, msg='Unexpected output: %s before %s' % (output, line))
if action == 'input':
@@ -654,7 +654,7 @@
self.fail('Unknown action: ' + action)
# Fail if we have output that was not expected in the transcript.
- self.assertEquals([], output, msg='Unexpected output: %s' % output)
+ self.assertEqual([], output, msg='Unexpected output: %s' % output)
TRANSCRIPT_TEST_FILE = os.path.join(
diff --git a/sdks/python/apache_beam/transforms/userstate.py b/sdks/python/apache_beam/transforms/userstate.py
index aa4e866..4d7126e 100644
--- a/sdks/python/apache_beam/transforms/userstate.py
+++ b/sdks/python/apache_beam/transforms/userstate.py
@@ -22,7 +22,6 @@
from __future__ import absolute_import
-import itertools
import types
from builtins import object
@@ -60,6 +59,23 @@
element_coder_id=context.coders.get_id(self.coder)))
+class SetStateSpec(StateSpec):
+ """Specification for a user DoFn Set State cell"""
+
+ def __init__(self, name, coder):
+ if not isinstance(name, str):
+ raise TypeError("SetState name is not a string")
+ if not isinstance(coder, Coder):
+ raise TypeError("SetState coder is not of type Coder")
+ self.name = name
+ self.coder = coder
+
+ def to_runner_api(self, context):
+ return beam_runner_api_pb2.StateSpec(
+ set_spec=beam_runner_api_pb2.SetStateSpec(
+ element_coder_id=context.coders.get_id(self.coder)))
+
+
class CombiningValueStateSpec(StateSpec):
"""Specification for a user DoFn combining value state cell."""
@@ -251,108 +267,42 @@
class RuntimeState(object):
"""State interface object passed to user code."""
-
- def __init__(self, state_spec, state_tag, current_value_accessor):
- self._state_spec = state_spec
- self._state_tag = state_tag
- self._current_value_accessor = current_value_accessor
-
- @staticmethod
- def for_spec(state_spec, state_tag, current_value_accessor):
- if isinstance(state_spec, BagStateSpec):
- return BagRuntimeState(state_spec, state_tag, current_value_accessor)
- elif isinstance(state_spec, CombiningValueStateSpec):
- return CombiningValueRuntimeState(state_spec, state_tag,
- current_value_accessor)
- else:
- raise ValueError('Invalid state spec: %s' % state_spec)
-
- def _encode(self, value):
- return self._state_spec.coder.encode(value)
-
- def _decode(self, value):
- return self._state_spec.coder.decode(value)
-
def prefetch(self):
# The default implementation here does nothing.
pass
-# Sentinel designating an unread value.
-UNREAD_VALUE = object()
+class AccumulatingRuntimeState(RuntimeState):
+ def read(self):
+ raise NotImplementedError(type(self))
+
+ def add(self, value):
+ raise NotImplementedError(type(self))
+
+ def clear(self):
+ raise NotImplementedError(type(self))
-class BagRuntimeState(RuntimeState):
+class BagRuntimeState(AccumulatingRuntimeState):
"""Bag state interface object passed to user code."""
- def __init__(self, state_spec, state_tag, current_value_accessor):
- super(BagRuntimeState, self).__init__(
- state_spec, state_tag, current_value_accessor)
- self._cached_value = UNREAD_VALUE
- self._cleared = False
- self._new_values = []
- def read(self):
- if self._cached_value is UNREAD_VALUE:
- self._cached_value = self._current_value_accessor()
- if not self._cleared:
- encoded_values = itertools.chain(self._cached_value, self._new_values)
- else:
- encoded_values = self._new_values
- return (self._decode(v) for v in encoded_values)
-
- def add(self, value):
- self._new_values.append(self._encode(value))
-
- def clear(self):
- self._cleared = True
- self._cached_value = []
- self._new_values = []
+class SetRuntimeState(AccumulatingRuntimeState):
+ """Set state interface object passed to user code."""
-class CombiningValueRuntimeState(RuntimeState):
+class CombiningValueRuntimeState(AccumulatingRuntimeState):
"""Combining value state interface object passed to user code."""
- def __init__(self, state_spec, state_tag, current_value_accessor):
- super(CombiningValueRuntimeState, self).__init__(
- state_spec, state_tag, current_value_accessor)
- self._current_accumulator = UNREAD_VALUE
- self._modified = False
- self._combine_fn = state_spec.combine_fn
-
- def _read_initial_value(self):
- if self._current_accumulator is UNREAD_VALUE:
- existing_accumulators = list(
- self._decode(a) for a in self._current_value_accessor())
- if existing_accumulators:
- self._current_accumulator = self._combine_fn.merge_accumulators(
- existing_accumulators)
- else:
- self._current_accumulator = self._combine_fn.create_accumulator()
-
- def read(self):
- self._read_initial_value()
- return self._combine_fn.extract_output(self._current_accumulator)
-
- def add(self, value):
- self._read_initial_value()
- self._modified = True
- self._current_accumulator = self._combine_fn.add_input(
- self._current_accumulator, value)
-
- def clear(self):
- self._modified = True
- self._current_accumulator = self._combine_fn.create_accumulator()
-
class UserStateContext(object):
"""Wrapper allowing user state and timers to be accessed by a DoFnInvoker."""
def get_timer(self, timer_spec, key, window):
- raise NotImplementedError()
+ raise NotImplementedError(type(self))
def get_state(self, state_spec, key, window):
- raise NotImplementedError()
+ raise NotImplementedError(type(self))
def commit(self):
- raise NotImplementedError()
+ raise NotImplementedError(type(self))
diff --git a/sdks/python/apache_beam/transforms/userstate_test.py b/sdks/python/apache_beam/transforms/userstate_test.py
index 0a3e13c..8e55cee 100644
--- a/sdks/python/apache_beam/transforms/userstate_test.py
+++ b/sdks/python/apache_beam/transforms/userstate_test.py
@@ -27,17 +27,22 @@
from apache_beam.coders import IterableCoder
from apache_beam.coders import StrUtf8Coder
from apache_beam.coders import VarIntCoder
+from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.runners.common import DoFnSignature
from apache_beam.testing.test_pipeline import TestPipeline
from apache_beam.testing.test_stream import TestStream
+from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
+from apache_beam.transforms import trigger
from apache_beam.transforms import userstate
+from apache_beam.transforms import window
from apache_beam.transforms.combiners import ToListCombineFn
from apache_beam.transforms.combiners import TopCombineFn
from apache_beam.transforms.core import DoFn
from apache_beam.transforms.timeutil import TimeDomain
from apache_beam.transforms.userstate import BagStateSpec
from apache_beam.transforms.userstate import CombiningValueStateSpec
+from apache_beam.transforms.userstate import SetStateSpec
from apache_beam.transforms.userstate import TimerSpec
from apache_beam.transforms.userstate import get_dofn_specs
from apache_beam.transforms.userstate import is_stateful_dofn
@@ -63,6 +68,9 @@
@on_timer(EXPIRY_TIMER_1)
def on_expiry_1(self,
+ window=DoFn.WindowParam,
+ timestamp=DoFn.TimestampParam,
+ key=DoFn.KeyParam,
buffer=DoFn.StateParam(BUFFER_STATE_1),
timer_1=DoFn.TimerParam(EXPIRY_TIMER_1),
timer_2=DoFn.TimerParam(EXPIRY_TIMER_2),
@@ -108,7 +116,13 @@
CombiningValueStateSpec(123, VarIntCoder(), TopCombineFn(10))
with self.assertRaises(TypeError):
CombiningValueStateSpec('statename', VarIntCoder(), object())
- # BagStateSpec('bag', )
+ SetStateSpec('setstatename', VarIntCoder())
+
+ with self.assertRaises(TypeError):
+ SetStateSpec(123, VarIntCoder())
+ with self.assertRaises(TypeError):
+ SetStateSpec('setstatename', object())
+
# TODO: add more spec tests
with self.assertRaises(ValueError):
DoFn.TimerParam(BagStateSpec('elements', BytesCoder()))
@@ -409,6 +423,152 @@
['extra'],
StatefulDoFnOnDirectRunnerTest.all_records)
+ def test_simple_set_stateful_dofn(self):
+ class SimpleTestSetStatefulDoFn(DoFn):
+ BUFFER_STATE = SetStateSpec('buffer', VarIntCoder())
+ EXPIRY_TIMER = TimerSpec('expiry', TimeDomain.WATERMARK)
+
+ def process(self, element, buffer=DoFn.StateParam(BUFFER_STATE),
+ timer1=DoFn.TimerParam(EXPIRY_TIMER)):
+ unused_key, value = element
+ buffer.add(value)
+ timer1.set(20)
+
+ @on_timer(EXPIRY_TIMER)
+ def expiry_callback(self, buffer=DoFn.StateParam(BUFFER_STATE)):
+ yield sorted(buffer.read())
+
+ with TestPipeline() as p:
+ test_stream = (TestStream()
+ .advance_watermark_to(10)
+ .add_elements([1, 2, 3])
+ .add_elements([2])
+ .advance_watermark_to(24))
+ (p
+ | test_stream
+ | beam.Map(lambda x: ('mykey', x))
+ | beam.ParDo(SimpleTestSetStatefulDoFn())
+ | beam.ParDo(self.record_dofn()))
+
+ # Two firings should occur: once after element 3 since the timer should
+ # fire after the watermark passes time 20, and another time after element
+ # 4, since the timer issued at that point should fire immediately.
+ self.assertEqual(
+ [[1, 2, 3]],
+ StatefulDoFnOnDirectRunnerTest.all_records)
+
+ def test_clearing_set_state(self):
+ class SetStateClearingStatefulDoFn(beam.DoFn):
+
+ SET_STATE = SetStateSpec('buffer', StrUtf8Coder())
+ EMIT_TIMER = TimerSpec('emit_timer', TimeDomain.WATERMARK)
+ CLEAR_TIMER = TimerSpec('clear_timer', TimeDomain.WATERMARK)
+
+ def process(self,
+ element,
+ set_state=beam.DoFn.StateParam(SET_STATE),
+ emit_timer=beam.DoFn.TimerParam(EMIT_TIMER),
+ clear_timer=beam.DoFn.TimerParam(CLEAR_TIMER)):
+ value = element[1]
+ set_state.add(value)
+ clear_timer.set(100)
+ emit_timer.set(1000)
+
+ @on_timer(EMIT_TIMER)
+ def emit_values(self, set_state=beam.DoFn.StateParam(SET_STATE)):
+ for value in set_state.read():
+ yield value
+
+ @on_timer(CLEAR_TIMER)
+ def clear_values(self, set_state=beam.DoFn.StateParam(SET_STATE)):
+ set_state.clear()
+ set_state.add('different-value')
+
+ with TestPipeline() as p:
+ test_stream = (TestStream()
+ .advance_watermark_to(0)
+ .add_elements([('key1', 'value1')])
+ .advance_watermark_to(100))
+
+ _ = (p
+ | test_stream
+ | beam.ParDo(SetStateClearingStatefulDoFn())
+ | beam.ParDo(self.record_dofn()))
+
+ self.assertEqual(['different-value'],
+ StatefulDoFnOnDirectRunnerTest.all_records)
+
+ def test_stateful_set_state_portably(self):
+
+ class SetStatefulDoFn(beam.DoFn):
+
+ SET_STATE = SetStateSpec('buffer', VarIntCoder())
+
+ def process(self,
+ element,
+ set_state=beam.DoFn.StateParam(SET_STATE)):
+ _, value = element
+ aggregated_value = 0
+ set_state.add(value)
+ for saved_value in set_state.read():
+ aggregated_value += saved_value
+ yield aggregated_value
+
+ p = TestPipeline()
+ values = p | beam.Create([('key', 1),
+ ('key', 2),
+ ('key', 3),
+ ('key', 4),
+ ('key', 3)])
+ actual_values = (values
+ | beam.ParDo(SetStatefulDoFn()))
+
+ assert_that(actual_values, equal_to([1, 3, 6, 10, 10]))
+
+ result = p.run()
+ result.wait_until_finish()
+
+ def test_stateful_set_state_clean_portably(self):
+
+ class SetStateClearingStatefulDoFn(beam.DoFn):
+
+ SET_STATE = SetStateSpec('buffer', VarIntCoder())
+ EMIT_TIMER = TimerSpec('emit_timer', TimeDomain.WATERMARK)
+
+ def process(self,
+ element,
+ set_state=beam.DoFn.StateParam(SET_STATE),
+ emit_timer=beam.DoFn.TimerParam(EMIT_TIMER)):
+ _, value = element
+ set_state.add(value)
+
+ all_elements = [element for element in set_state.read()]
+
+ if len(all_elements) == 5:
+ set_state.clear()
+ set_state.add(100)
+ emit_timer.set(1)
+
+ @on_timer(EMIT_TIMER)
+ def emit_values(self, set_state=beam.DoFn.StateParam(SET_STATE)):
+ yield sorted(set_state.read())
+
+ p = TestPipeline()
+ values = p | beam.Create([('key', 1),
+ ('key', 2),
+ ('key', 3),
+ ('key', 4),
+ ('key', 5)])
+ actual_values = (values
+ | beam.Map(lambda t: window.TimestampedValue(t, 1))
+ | beam.WindowInto(window.FixedWindows(1))
+ | beam.ParDo(SetStateClearingStatefulDoFn()))
+
+ assert_that(actual_values, equal_to([[100]]))
+
+ result = p.run()
+ result.wait_until_finish()
+
def test_stateful_dofn_nonkeyed_input(self):
p = TestPipeline()
values = p | beam.Create([1, 2, 3])
@@ -557,16 +717,49 @@
[('timer1', 10), ('timer2', 20), ('timer3', 30)],
sorted(StatefulDoFnOnDirectRunnerTest.all_records))
+ def test_timer_output_timestamp_and_window(self):
+
+ class TimerEmittingStatefulDoFn(DoFn):
+ EMIT_TIMER_1 = TimerSpec('emit1', TimeDomain.WATERMARK)
+
+ def process(self, element, timer1=DoFn.TimerParam(EMIT_TIMER_1)):
+ timer1.set(10)
+
+ @on_timer(EMIT_TIMER_1)
+ def emit_callback_1(self,
+ window=DoFn.WindowParam,
+ ts=DoFn.TimestampParam,
+ key=DoFn.KeyParam):
+ yield ('timer1-{key}'.format(key=key),
+ int(ts), int(window.start), int(window.end))
+
+ pipeline_options = PipelineOptions()
+ with TestPipeline(options=pipeline_options) as p:
+ test_stream = (TestStream()
+ .advance_watermark_to(10)
+ .add_elements([1]))
+ (p
+ | test_stream
+ | beam.Map(lambda x: ('mykey', x))
+ | "window_into" >> beam.WindowInto(
+ window.FixedWindows(5),
+ accumulation_mode=trigger.AccumulationMode.DISCARDING)
+ | beam.ParDo(TimerEmittingStatefulDoFn())
+ | beam.ParDo(self.record_dofn()))
+
+ self.assertEqual(
+ [('timer1-mykey', 10, 10, 15)],
+ sorted(StatefulDoFnOnDirectRunnerTest.all_records))
+
def test_index_assignment(self):
class IndexAssigningStatefulDoFn(DoFn):
- INDEX_STATE = BagStateSpec('index', VarIntCoder())
+ INDEX_STATE = CombiningValueStateSpec('index', sum)
def process(self, element, state=DoFn.StateParam(INDEX_STATE)):
unused_key, value = element
- next_index, = list(state.read()) or [0]
- yield (value, next_index)
- state.clear()
- state.add(next_index + 1)
+ current_index = state.read()
+ yield (value, current_index)
+ state.add(1)
with TestPipeline() as p:
test_stream = (TestStream()
diff --git a/sdks/python/apache_beam/transforms/util.py b/sdks/python/apache_beam/transforms/util.py
index 64dce9d..bb7e522 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -24,17 +24,23 @@
import collections
import contextlib
import random
+import re
import time
+import typing
+import warnings
+from builtins import filter
from builtins import object
from builtins import range
from builtins import zip
from future.utils import itervalues
+from apache_beam import coders
from apache_beam import typehints
from apache_beam.metrics import Metrics
from apache_beam.portability import common_urns
from apache_beam.transforms import window
+from apache_beam.transforms.combiners import CountCombineFn
from apache_beam.transforms.core import CombinePerKey
from apache_beam.transforms.core import DoFn
from apache_beam.transforms.core import FlatMap
@@ -45,13 +51,19 @@
from apache_beam.transforms.core import Windowing
from apache_beam.transforms.ptransform import PTransform
from apache_beam.transforms.ptransform import ptransform_fn
+from apache_beam.transforms.timeutil import TimeDomain
from apache_beam.transforms.trigger import AccumulationMode
from apache_beam.transforms.trigger import AfterCount
+from apache_beam.transforms.userstate import BagStateSpec
+from apache_beam.transforms.userstate import CombiningValueStateSpec
+from apache_beam.transforms.userstate import TimerSpec
+from apache_beam.transforms.userstate import on_timer
from apache_beam.transforms.window import NonMergingWindowFn
from apache_beam.transforms.window import TimestampCombiner
from apache_beam.transforms.window import TimestampedValue
from apache_beam.utils import windowed_value
from apache_beam.utils.annotations import deprecated
+from apache_beam.utils.annotations import experimental
__all__ = [
'BatchElements',
@@ -59,14 +71,19 @@
'Distinct',
'Keys',
'KvSwap',
+ 'Regex',
+ 'Reify',
'RemoveDuplicates',
'Reshuffle',
+ 'ToString',
'Values',
+ 'WithKeys',
+ 'GroupIntoBatches'
]
-K = typehints.TypeVariable('K')
-V = typehints.TypeVariable('V')
-T = typehints.TypeVariable('T')
+K = typing.TypeVar('K')
+V = typing.TypeVar('V')
+T = typing.TypeVar('T')
class CoGroupByKey(PTransform):
@@ -84,16 +101,17 @@
'tag2': ... ,
... })
- For example, given:
+ For example, given::
{'tag1': pc1, 'tag2': pc2, 333: pc3}
- where:
+ where::
+
pc1 = [(k1, v1)]
pc2 = []
pc3 = [(k1, v31), (k1, v32), (k2, v33)]
- The output PCollection would be:
+ The output PCollection would be::
[(k1, {'tag1': [v1], 'tag2': [], 333: [v31, v32]}),
(k2, {'tag1': [], 'tag2': [], 333: [v33]})]
@@ -466,7 +484,7 @@
@typehints.with_input_types(T)
-@typehints.with_output_types(typehints.List[T])
+@typehints.with_output_types(typing.List[T])
class BatchElements(PTransform):
"""A Transform that batches elements for amortized processing.
@@ -559,8 +577,8 @@
return self._window_coder
-@typehints.with_input_types(typehints.KV[K, V])
-@typehints.with_output_types(typehints.KV[K, V])
+@typehints.with_input_types(typing.Tuple[K, V])
+@typehints.with_output_types(typing.Tuple[K, V])
class ReshufflePerKey(PTransform):
"""PTransform that returns a PCollection equivalent to its input,
but operationally provides some of the side effects of a GroupByKey,
@@ -576,7 +594,6 @@
# In this (common) case we can use a trivial trigger driver
# and avoid the (expensive) window param.
globally_windowed = window.GlobalWindows.windowed_value(None)
- window_fn = window.GlobalWindows()
MIN_TIMESTAMP = window.MIN_TIMESTAMP
def reify_timestamps(element, timestamp=DoFn.TimestampParam):
@@ -594,29 +611,24 @@
for (value, timestamp) in values]
else:
- # The linter is confused.
- # hash(1) is used to force "runtime" selection of _IdentityWindowFn
- # pylint: disable=abstract-class-instantiated
- cls = hash(1) and _IdentityWindowFn
- window_fn = cls(
- windowing_saved.windowfn.get_window_coder())
-
- def reify_timestamps(element, timestamp=DoFn.TimestampParam):
+ def reify_timestamps(element,
+ timestamp=DoFn.TimestampParam,
+ window=DoFn.WindowParam):
key, value = element
- return key, TimestampedValue(value, timestamp)
+ # Transport the window as part of the value and restore it later.
+ return key, windowed_value.WindowedValue(value, timestamp, [window])
- def restore_timestamps(element, window=DoFn.WindowParam):
- # Pass the current window since _IdentityWindowFn wouldn't know how
- # to generate it.
- key, values = element
- return [
- windowed_value.WindowedValue(
- (key, value.value), value.timestamp, [window])
- for value in values]
+ def restore_timestamps(element):
+ key, windowed_values = element
+ return [wv.with_value((key, wv.value)) for wv in windowed_values]
ungrouped = pcoll | Map(reify_timestamps)
+
+ # TODO(BEAM-8104) Using global window as one of the standard window.
+ # This is to mitigate the Dataflow Java Runner Harness limitation to
+ # accept only standard coders.
ungrouped._windowing = Windowing(
- window_fn,
+ window.GlobalWindows(),
triggerfn=AfterCount(1),
accumulation_mode=AccumulationMode.DISCARDING,
timestamp_combiner=TimestampCombiner.OUTPUT_AT_EARLIEST)
@@ -653,3 +665,416 @@
@PTransform.register_urn(common_urns.composites.RESHUFFLE.urn, None)
def from_runner_api_parameter(unused_parameter, unused_context):
return Reshuffle()
+
+
+@ptransform_fn
+def WithKeys(pcoll, k):
+ """PTransform that takes a PCollection, and either a constant key or a
+ callable, and returns a PCollection of (K, V), where each of the values in
+ the input PCollection has been paired with either the constant key or a key
+ computed from the value.
+ """
+ if callable(k):
+ return pcoll | Map(lambda v: (k(v), v))
+ return pcoll | Map(lambda v: (k, v))
+
+
+@experimental()
+@typehints.with_input_types(typing.Tuple[K, V])
+class GroupIntoBatches(PTransform):
+ """PTransform that batches the input into desired batch size. Elements are
+ buffered until they are equal to batch size provided in the argument at which
+ point they are output to the output Pcollection.
+
+ Windows are preserved (batches will contain elements from the same window)
+
+ GroupIntoBatches is experimental. Its use case will depend on the runner if
+ it has support of States and Timers.
+ """
+
+ def __init__(self, batch_size):
+ """Create a new GroupIntoBatches with batch size.
+
+ Arguments:
+ batch_size: (required) How many elements should be in a batch
+ """
+ warnings.warn('Use of GroupIntoBatches transform requires State/Timer '
+ 'support from the runner')
+ self.batch_size = batch_size
+
+ def expand(self, pcoll):
+ input_coder = coders.registry.get_coder(pcoll)
+ return pcoll | ParDo(_pardo_group_into_batches(
+ self.batch_size, input_coder))
+
+
+def _pardo_group_into_batches(batch_size, input_coder):
+ ELEMENT_STATE = BagStateSpec('values', input_coder)
+ COUNT_STATE = CombiningValueStateSpec('count', input_coder, CountCombineFn())
+ EXPIRY_TIMER = TimerSpec('expiry', TimeDomain.WATERMARK)
+
+ class _GroupIntoBatchesDoFn(DoFn):
+
+ def process(self, element,
+ window=DoFn.WindowParam,
+ element_state=DoFn.StateParam(ELEMENT_STATE),
+ count_state=DoFn.StateParam(COUNT_STATE),
+ expiry_timer=DoFn.TimerParam(EXPIRY_TIMER)):
+ # Allowed lateness not supported in Python SDK
+ # https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data
+ expiry_timer.set(window.end)
+ element_state.add(element)
+ count_state.add(1)
+ count = count_state.read()
+ if count >= batch_size:
+ batch = [element for element in element_state.read()]
+ yield batch
+ element_state.clear()
+ count_state.clear()
+
+ @on_timer(EXPIRY_TIMER)
+ def expiry(self, element_state=DoFn.StateParam(ELEMENT_STATE),
+ count_state=DoFn.StateParam(COUNT_STATE)):
+ batch = [element for element in element_state.read()]
+ if batch:
+ yield batch
+ element_state.clear()
+ count_state.clear()
+
+ return _GroupIntoBatchesDoFn()
+
+
+class ToString(object):
+ """
+ PTransform for converting a PCollection element, KV or PCollection Iterable
+ to string.
+ """
+
+ class Kvs(PTransform):
+ """
+ Transforms each element of the PCollection to a string on the key followed
+ by the specific delimiter and the value.
+ """
+
+ def __init__(self, delimiter=None):
+ self.delimiter = delimiter or ","
+
+ def expand(self, pcoll):
+ input_type = typing.Tuple[typing.Any, typing.Any]
+ output_type = str
+ return (pcoll | ('%s:KeyVaueToString' % self.label >> (Map(
+ lambda x: "{}{}{}".format(x[0], self.delimiter, x[1])))
+ .with_input_types(input_type)
+ .with_output_types(output_type)))
+
+ class Element(PTransform):
+ """
+ Transforms each element of the PCollection to a string.
+ """
+
+ def expand(self, pcoll):
+ input_type = T
+ output_type = str
+ return (pcoll | ('%s:ElementToString' % self.label >> (Map(
+ lambda x: str(x)))
+ .with_input_types(input_type)
+ .with_output_types(output_type)))
+
+ class Iterables(PTransform):
+ """
+ Transforms each item in the iterable of the input of PCollection to a
+ string. There is no trailing delimiter.
+ """
+
+ def __init__(self, delimiter=None):
+ self.delimiter = delimiter or ","
+
+ def expand(self, pcoll):
+ input_type = typing.Iterable[typing.Any]
+ output_type = str
+ return (pcoll | ('%s:IterablesToString' % self.label >> (
+ Map(lambda x: self.delimiter.join(str(_x) for _x in x)))
+ .with_input_types(input_type)
+ .with_output_types(output_type)))
+
+
+class Reify(object):
+ """PTransforms for converting between explicit and implicit form of various
+ Beam values."""
+
+ @typehints.with_input_types(T)
+ @typehints.with_output_types(T)
+ class Timestamp(PTransform):
+ """PTransform to wrap a value in a TimestampedValue with it's
+ associated timestamp."""
+
+ @staticmethod
+ def add_timestamp_info(element, timestamp=DoFn.TimestampParam):
+ yield TimestampedValue(element, timestamp)
+
+ def expand(self, pcoll):
+ return pcoll | ParDo(self.add_timestamp_info)
+
+ @typehints.with_input_types(T)
+ @typehints.with_output_types(T)
+ class Window(PTransform):
+ """PTransform to convert an element in a PCollection into a tuple of
+ (element, timestamp, window), wrapped in a TimestampedValue with it's
+ associated timestamp."""
+
+ @staticmethod
+ def add_window_info(element, timestamp=DoFn.TimestampParam,
+ window=DoFn.WindowParam):
+ yield TimestampedValue((element, timestamp, window), timestamp)
+
+ def expand(self, pcoll):
+ return pcoll | ParDo(self.add_window_info)
+
+ @typehints.with_input_types(typing.Tuple[K, V])
+ @typehints.with_output_types(typing.Tuple[K, V])
+ class TimestampInValue(PTransform):
+ """PTransform to wrap the Value in a KV pair in a TimestampedValue with
+ the element's associated timestamp."""
+
+ @staticmethod
+ def add_timestamp_info(element, timestamp=DoFn.TimestampParam):
+ key, value = element
+ yield (key, TimestampedValue(value, timestamp))
+
+ def expand(self, pcoll):
+ return pcoll | ParDo(self.add_timestamp_info)
+
+ @typehints.with_input_types(typing.Tuple[K, V])
+ @typehints.with_output_types(typing.Tuple[K, V])
+ class WindowInValue(PTransform):
+ """PTransform to convert the Value in a KV pair into a tuple of
+ (value, timestamp, window), with the whole element being wrapped inside a
+ TimestampedValue."""
+
+ @staticmethod
+ def add_window_info(element, timestamp=DoFn.TimestampParam,
+ window=DoFn.WindowParam):
+ key, value = element
+ yield TimestampedValue((key, (value, timestamp, window)), timestamp)
+
+ def expand(self, pcoll):
+ return pcoll | ParDo(self.add_window_info)
+
+
+class Regex(object):
+ """
+ PTransform to use Regular Expression to process the elements in a
+ PCollection.
+ """
+
+ ALL = "__regex_all_groups"
+
+ @staticmethod
+ def _regex_compile(regex):
+ """Return re.compile if the regex has a string value"""
+ if isinstance(regex, str):
+ regex = re.compile(regex)
+ return regex
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(str)
+ @ptransform_fn
+ def matches(pcoll, regex, group=0):
+ """
+ Returns the matches (group 0 by default) if zero or more characters at the
+ beginning of string match the regular expression. To match the entire
+ string, add "$" sign at the end of regex expression.
+
+ Group can be integer value or a string value.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ group: (optional) name/number of the group, it can be integer or a string
+ value. Defaults to 0, meaning the entire matched string will be
+ returned.
+ """
+ regex = Regex._regex_compile(regex)
+
+ def _process(element):
+ m = regex.match(element)
+ if m:
+ yield m.group(group)
+ return pcoll | FlatMap(_process)
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(typing.List[str])
+ @ptransform_fn
+ def all_matches(pcoll, regex):
+ """
+ Returns all matches (groups) if zero or more characters at the beginning
+ of string match the regular expression.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ """
+ regex = Regex._regex_compile(regex)
+
+ def _process(element):
+ m = regex.match(element)
+ if m:
+ yield [m.group(ix) for ix in range(m.lastindex + 1)]
+
+ return pcoll | FlatMap(_process)
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(typing.Tuple[str, str])
+ @ptransform_fn
+ def matches_kv(pcoll, regex, keyGroup, valueGroup=0):
+ """
+ Returns the KV pairs if the string matches the regular expression, deriving
+ the key & value from the specified group of the regular expression.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ keyGroup: The Regex group to use as the key. Can be int or str.
+ valueGroup: (optional) Regex group to use the value. Can be int or str.
+ The default value "0" returns entire matched string.
+ """
+ regex = Regex._regex_compile(regex)
+
+ def _process(element):
+ match = regex.match(element)
+ if match:
+ yield (match.group(keyGroup), match.group(valueGroup))
+ return pcoll | FlatMap(_process)
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(str)
+ @ptransform_fn
+ def find(pcoll, regex, group=0):
+ """
+ Returns the matches if a portion of the line matches the Regex. Returns
+ the entire group (group 0 by default). Group can be integer value or a
+ string value.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ group: (optional) name of the group, it can be integer or a string value.
+ """
+ regex = Regex._regex_compile(regex)
+
+ def _process(element):
+ r = regex.search(element)
+ if r:
+ yield r.group(group)
+ return pcoll | FlatMap(_process)
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(typing.Union[typing.List[str],
+ typing.Tuple[str, str]])
+ @ptransform_fn
+ def find_all(pcoll, regex, group=0, outputEmpty=True):
+ """
+ Returns the matches if a portion of the line matches the Regex. By default,
+ list of group 0 will return with empty items. To get all groups, pass the
+ `Regex.ALL` flag in the `group` parameter which returns all the groups in
+ the tuple format.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ group: (optional) name of the group, it can be integer or a string value.
+ outputEmpty: (optional) Should empty be output. True to output empties
+ and false if not.
+ """
+ regex = Regex._regex_compile(regex)
+
+ def _process(element):
+ matches = regex.finditer(element)
+ if group == Regex.ALL:
+ yield [(m.group(), m.groups()[0]) for m in matches if outputEmpty
+ or m.groups()[0]]
+ else:
+ yield [m.group(group) for m in matches if outputEmpty or m.group(group)]
+ return pcoll | FlatMap(_process)
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(typing.Tuple[str, str])
+ @ptransform_fn
+ def find_kv(pcoll, regex, keyGroup, valueGroup=0):
+ """
+ Returns the matches if a portion of the line matches the Regex. Returns the
+ specified groups as the key and value pair.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ keyGroup: The Regex group to use as the key. Can be int or str.
+ valueGroup: (optional) Regex group to use the value. Can be int or str.
+ The default value "0" returns entire matched string.
+ """
+ regex = Regex._regex_compile(regex)
+
+ def _process(element):
+ matches = regex.finditer(element)
+ if matches:
+ for match in matches:
+ yield (match.group(keyGroup), match.group(valueGroup))
+
+ return pcoll | FlatMap(_process)
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(str)
+ @ptransform_fn
+ def replace_all(pcoll, regex, replacement):
+ """
+ Returns the matches if a portion of the line matches the regex and
+ replaces all matches with the replacement string.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ replacement: the string to be substituted for each match.
+ """
+ regex = Regex._regex_compile(regex)
+ return pcoll | Map(lambda elem: regex.sub(replacement, elem))
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(str)
+ @ptransform_fn
+ def replace_first(pcoll, regex, replacement):
+ """
+ Returns the matches if a portion of the line matches the regex and replaces
+ the first match with the replacement string.
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ replacement: the string to be substituted for each match.
+ """
+ regex = Regex._regex_compile(regex)
+ return pcoll | Map(lambda elem: regex.sub(replacement, elem, 1))
+
+ @staticmethod
+ @typehints.with_input_types(str)
+ @typehints.with_output_types(typing.List[str])
+ @ptransform_fn
+ def split(pcoll, regex, outputEmpty=False):
+ """
+ Returns the list string which was splitted on the basis of regular
+ expression. It will not output empty items (by defaults).
+
+ Args:
+ regex: the regular expression string or (re.compile) pattern.
+ outputEmpty: (optional) Should empty be output. True to output empties
+ and false if not.
+ """
+ regex = Regex._regex_compile(regex)
+ outputEmpty = bool(outputEmpty)
+
+ def _process(element):
+ r = regex.split(element)
+ if r and not outputEmpty:
+ r = list(filter(None, r))
+ yield r
+
+ return pcoll | FlatMap(_process)
diff --git a/sdks/python/apache_beam/transforms/util_test.py b/sdks/python/apache_beam/transforms/util_test.py
index 31a44840..af2fc8c 100644
--- a/sdks/python/apache_beam/transforms/util_test.py
+++ b/sdks/python/apache_beam/transforms/util_test.py
@@ -20,24 +20,30 @@
from __future__ import absolute_import
from __future__ import division
+import itertools
import logging
+import math
import random
+import re
import time
import unittest
from builtins import object
from builtins import range
import apache_beam as beam
+from apache_beam import WindowInto
from apache_beam.coders import coders
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_stream import TestStream
from apache_beam.testing.util import TestWindowedValue
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import contains_in_any_order
from apache_beam.testing.util import equal_to
from apache_beam.transforms import util
from apache_beam.transforms import window
+from apache_beam.transforms.window import FixedWindows
from apache_beam.transforms.window import GlobalWindow
from apache_beam.transforms.window import GlobalWindows
from apache_beam.transforms.window import IntervalWindow
@@ -414,6 +420,484 @@
pipeline.run()
+class WithKeysTest(unittest.TestCase):
+
+ def setUp(self):
+ self.l = [1, 2, 3]
+
+ def test_constant_k(self):
+ with TestPipeline() as p:
+ pc = p | beam.Create(self.l)
+ with_keys = pc | util.WithKeys('k')
+ assert_that(with_keys, equal_to([('k', 1), ('k', 2), ('k', 3)], ))
+
+ def test_callable_k(self):
+ with TestPipeline() as p:
+ pc = p | beam.Create(self.l)
+ with_keys = pc | util.WithKeys(lambda x: x*x)
+ assert_that(with_keys, equal_to([(1, 1), (4, 2), (9, 3)]))
+
+
+class GroupIntoBatchesTest(unittest.TestCase):
+ NUM_ELEMENTS = 10
+ BATCH_SIZE = 5
+
+ @staticmethod
+ def _create_test_data():
+ scientists = [
+ "Einstein",
+ "Darwin",
+ "Copernicus",
+ "Pasteur",
+ "Curie",
+ "Faraday",
+ "Newton",
+ "Bohr",
+ "Galilei",
+ "Maxwell"
+ ]
+
+ data = []
+ for i in range(GroupIntoBatchesTest.NUM_ELEMENTS):
+ index = i % len(scientists)
+ data.append(("key", scientists[index]))
+ return data
+
+ def test_in_global_window(self):
+ pipeline = TestPipeline()
+ collection = pipeline \
+ | beam.Create(GroupIntoBatchesTest._create_test_data()) \
+ | util.GroupIntoBatches(GroupIntoBatchesTest.BATCH_SIZE)
+ num_batches = collection | beam.combiners.Count.Globally()
+ assert_that(num_batches,
+ equal_to([int(math.ceil(GroupIntoBatchesTest.NUM_ELEMENTS /
+ GroupIntoBatchesTest.BATCH_SIZE))]))
+ pipeline.run()
+
+ def test_in_streaming_mode(self):
+ timestamp_interval = 1
+ offset = itertools.count(0)
+ start_time = timestamp.Timestamp(0)
+ window_duration = 6
+ test_stream = (TestStream()
+ .advance_watermark_to(start_time)
+ .add_elements(
+ [TimestampedValue(x, next(offset) * timestamp_interval)
+ for x in GroupIntoBatchesTest._create_test_data()])
+ .advance_watermark_to(start_time + (window_duration - 1))
+ .advance_watermark_to(start_time + (window_duration + 1))
+ .advance_watermark_to(start_time +
+ GroupIntoBatchesTest.NUM_ELEMENTS)
+ .advance_watermark_to_infinity())
+ pipeline = TestPipeline()
+ # window duration is 6 and batch size is 5, so output batch size should be
+ # 5 (flush because of batchSize reached)
+ expected_0 = 5
+ # there is only one element left in the window so batch size should be 1
+ # (flush because of end of window reached)
+ expected_1 = 1
+ # collection is 10 elements, there is only 4 left, so batch size should be
+ # 4 (flush because end of collection reached)
+ expected_2 = 4
+
+ collection = pipeline | test_stream \
+ | WindowInto(FixedWindows(window_duration)) \
+ | util.GroupIntoBatches(GroupIntoBatchesTest.BATCH_SIZE)
+ num_elements_in_batches = collection | beam.Map(len)
+
+ result = pipeline.run()
+ result.wait_until_finish()
+ assert_that(num_elements_in_batches,
+ equal_to([expected_0, expected_1, expected_2]))
+
+
+class ToStringTest(unittest.TestCase):
+
+ def test_tostring_elements(self):
+
+ with TestPipeline() as p:
+ result = (p | beam.Create([1, 1, 2, 3]) | util.ToString.Element())
+ assert_that(result, equal_to(["1", "1", "2", "3"]))
+
+ def test_tostring_iterables(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create([("one", "two", "three"),
+ ("four", "five", "six")])
+ | util.ToString.Iterables())
+ assert_that(result, equal_to(["one,two,three", "four,five,six"]))
+
+ def test_tostring_iterables_with_delimeter(self):
+ with TestPipeline() as p:
+ data = [("one", "two", "three"), ("four", "five", "six")]
+ result = (p | beam.Create(data) | util.ToString.Iterables("\t"))
+ assert_that(result, equal_to(["one\ttwo\tthree", "four\tfive\tsix"]))
+
+ def test_tostring_kvs(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create([("one", 1), ("two", 2)]) | util.ToString.Kvs())
+ assert_that(result, equal_to(["one,1", "two,2"]))
+
+ def test_tostring_kvs_delimeter(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create([("one", 1), ("two", 2)]) |
+ util.ToString.Kvs("\t"))
+ assert_that(result, equal_to(["one\t1", "two\t2"]))
+
+
+class ReifyTest(unittest.TestCase):
+
+ def test_timestamp(self):
+ l = [TimestampedValue('a', 100),
+ TimestampedValue('b', 200),
+ TimestampedValue('c', 300)]
+ expected = [TestWindowedValue('a', 100, [GlobalWindow()]),
+ TestWindowedValue('b', 200, [GlobalWindow()]),
+ TestWindowedValue('c', 300, [GlobalWindow()])]
+ with TestPipeline() as p:
+ # Map(lambda x: x) PTransform is added after Create here, because when
+ # a PCollection of TimestampedValues is created with Create PTransform,
+ # the timestamps are not assigned to it. Adding a Map forces the
+ # PCollection to go through a DoFn so that the PCollection consists of
+ # the elements with timestamps assigned to them instead of a PCollection
+ # of TimestampedValue(element, timestamp).
+ pc = p | beam.Create(l) | beam.Map(lambda x: x)
+ reified_pc = pc | util.Reify.Timestamp()
+ assert_that(reified_pc, equal_to(expected), reify_windows=True)
+
+ def test_window(self):
+ l = [GlobalWindows.windowed_value('a', 100),
+ GlobalWindows.windowed_value('b', 200),
+ GlobalWindows.windowed_value('c', 300)]
+ expected = [TestWindowedValue(('a', 100, GlobalWindow()), 100,
+ [GlobalWindow()]),
+ TestWindowedValue(('b', 200, GlobalWindow()), 200,
+ [GlobalWindow()]),
+ TestWindowedValue(('c', 300, GlobalWindow()), 300,
+ [GlobalWindow()])]
+ with TestPipeline() as p:
+ pc = p | beam.Create(l)
+ # Map(lambda x: x) PTransform is added after Create here, because when
+ # a PCollection of WindowedValues is created with Create PTransform,
+ # the windows are not assigned to it. Adding a Map forces the
+ # PCollection to go through a DoFn so that the PCollection consists of
+ # the elements with timestamps assigned to them instead of a PCollection
+ # of WindowedValue(element, timestamp, window).
+ pc = pc | beam.Map(lambda x: x)
+ reified_pc = pc | util.Reify.Window()
+ assert_that(reified_pc, equal_to(expected), reify_windows=True)
+
+ def test_timestamp_in_value(self):
+ l = [TimestampedValue(('a', 1), 100),
+ TimestampedValue(('b', 2), 200),
+ TimestampedValue(('c', 3), 300)]
+ expected = [TestWindowedValue(('a', TimestampedValue(1, 100)), 100,
+ [GlobalWindow()]),
+ TestWindowedValue(('b', TimestampedValue(2, 200)), 200,
+ [GlobalWindow()]),
+ TestWindowedValue(('c', TimestampedValue(3, 300)), 300,
+ [GlobalWindow()])]
+ with TestPipeline() as p:
+ pc = p | beam.Create(l) | beam.Map(lambda x: x)
+ reified_pc = pc | util.Reify.TimestampInValue()
+ assert_that(reified_pc, equal_to(expected), reify_windows=True)
+
+ def test_window_in_value(self):
+ l = [GlobalWindows.windowed_value(('a', 1), 100),
+ GlobalWindows.windowed_value(('b', 2), 200),
+ GlobalWindows.windowed_value(('c', 3), 300)]
+ expected = [TestWindowedValue(('a', (1, 100, GlobalWindow())), 100,
+ [GlobalWindow()]),
+ TestWindowedValue(('b', (2, 200, GlobalWindow())), 200,
+ [GlobalWindow()]),
+ TestWindowedValue(('c', (3, 300, GlobalWindow())), 300,
+ [GlobalWindow()])]
+ with TestPipeline() as p:
+ # Map(lambda x: x) hack is used for the same reason here.
+ # Also, this makes the typehint on Reify.WindowInValue work.
+ pc = p | beam.Create(l) | beam.Map(lambda x: x)
+ reified_pc = pc | util.Reify.WindowInValue()
+ assert_that(reified_pc, equal_to(expected), reify_windows=True)
+
+
+class RegexTest(unittest.TestCase):
+
+ def test_find(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["aj", "xj", "yj", "zj"])
+ | util.Regex.find("[xyz]"))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ def test_find_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("[xyz]")
+ result = (p | beam.Create(["aj", "xj", "yj", "zj"]) | util.Regex.find(rc))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ def test_find_group(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["aj", "xj", "yj", "zj"])
+ | util.Regex.find("([xyz])j", group=1))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ def test_find_empty(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "b", "c", "d"])
+ | util.Regex.find("[xyz]"))
+ assert_that(result, equal_to([]))
+
+ def test_find_group_name(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["aj", "xj", "yj", "zj"])
+ | util.Regex.find("(?P<namedgroup>[xyz])j", group="namedgroup"))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ def test_find_group_name_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("(?P<namedgroup>[xyz])j")
+ result = (p | beam.Create(["aj", "xj", "yj", "zj"]) | util.Regex.find(
+ rc, group="namedgroup"))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ def test_find_all_groups(self):
+ data = ["abb ax abbb", "abc qwerty abcabcd xyz"]
+ with TestPipeline() as p:
+ pcol = (p | beam.Create(data))
+
+ assert_that(pcol | 'with default values' >> util.Regex.find_all('a(b*)'),
+ equal_to([['abb', 'a', 'abbb'], ['ab', 'ab', 'ab']]),
+ label='CheckWithDefaultValues')
+
+ assert_that(pcol | 'group 1' >> util.Regex.find_all('a(b*)', 1),
+ equal_to([['b', 'b', 'b'], ['bb', '', 'bbb']]),
+ label='CheckWithGroup1')
+
+ assert_that(pcol | 'group 1 non empty' >> util.Regex.find_all(
+ 'a(b*)', 1, outputEmpty=False),
+ equal_to([['b', 'b', 'b'], ['bb', 'bbb']]),
+ label='CheckGroup1NonEmpty')
+
+ assert_that(pcol | 'named group' >> util.Regex.find_all(
+ 'a(?P<namedgroup>b*)', 'namedgroup'),
+ equal_to([['b', 'b', 'b'], ['bb', '', 'bbb']]),
+ label='CheckNamedGroup')
+
+ assert_that(pcol | 'all groups' >> util.Regex.find_all(
+ 'a(?P<namedgroup>b*)', util.Regex.ALL),
+ equal_to([[('ab', 'b'), ('ab', 'b'), ('ab', 'b')],
+ [('abb', 'bb'), ('a', ''), ('abbb', 'bbb')]]),
+ label='CheckAllGroups')
+
+ assert_that(pcol | 'all non empty groups' >> util.Regex.find_all(
+ 'a(b*)', util.Regex.ALL, outputEmpty=False),
+ equal_to([[('ab', 'b'), ('ab', 'b'), ('ab', 'b')],
+ [('abb', 'bb'), ('abbb', 'bbb')]]),
+ label='CheckAllNonEmptyGroups')
+
+ def test_find_kv(self):
+ with TestPipeline() as p:
+ pcol = (p | beam.Create(['a b c d']))
+ assert_that(pcol | 'key 1' >> util.Regex.find_kv(
+ 'a (b) (c)', 1,), equal_to([('b', 'a b c')]), label='CheckKey1')
+
+ assert_that(pcol | 'key 1 group 1' >> util.Regex.find_kv(
+ 'a (b) (c)', 1, 2), equal_to([('b', 'c')]), label='CheckKey1Group1')
+
+ def test_find_kv_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("a (b) (c)")
+ result = (p | beam.Create(["a b c"]) | util.Regex.find_kv(rc, 1, 2))
+ assert_that(result, equal_to([("b", "c")]))
+
+ def test_find_kv_none(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["x y z"])
+ | util.Regex.find_kv("a (b) (c)", 1, 2))
+ assert_that(result, equal_to([]))
+
+ def test_match(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "x", "y", "z"])
+ | util.Regex.matches("[xyz]"))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "ax", "yby", "zzc"])
+ | util.Regex.matches("[xyz]"))
+ assert_that(result, equal_to(["y", "z"]))
+
+ def test_match_entire_line(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "x", "y", "ay", "zz"])
+ | util.Regex.matches("[xyz]$"))
+ assert_that(result, equal_to(["x", "y"]))
+
+ def test_match_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("[xyz]")
+ result = (p | beam.Create(["a", "x", "y", "z"]) | util.Regex.matches(rc))
+ assert_that(result, equal_to(["x", "y", "z"]))
+
+ def test_match_none(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "b", "c", "d"])
+ | util.Regex.matches("[xyz]"))
+ assert_that(result, equal_to([]))
+
+ def test_match_group(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "x xxx", "x yyy", "x zzz"])
+ | util.Regex.matches("x ([xyz]*)", 1))
+ assert_that(result, equal_to(("xxx", "yyy", "zzz")))
+
+ def test_match_group_name(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "x xxx", "x yyy", "x zzz"])
+ | util.Regex.matches("x (?P<namedgroup>[xyz]*)", 'namedgroup'))
+ assert_that(result, equal_to(("xxx", "yyy", "zzz")))
+
+ def test_match_group_name_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("x (?P<namedgroup>[xyz]*)")
+ result = (p | beam.Create(["a", "x xxx", "x yyy", "x zzz"])
+ | util.Regex.matches(rc, 'namedgroup'))
+ assert_that(result, equal_to(("xxx", "yyy", "zzz")))
+
+ def test_match_group_empty(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a", "b", "c", "d"])
+ | util.Regex.matches("x (?P<namedgroup>[xyz]*)", 'namedgroup'))
+ assert_that(result, equal_to([]))
+
+ def test_all_matched(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a x", "x x", "y y", "z z"])
+ | util.Regex.all_matches("([xyz]) ([xyz])"))
+ expected_result = [["x x", "x", "x"], ["y y", "y", "y"],
+ ["z z", "z", "z"]]
+ assert_that(result, equal_to(expected_result))
+
+ def test_all_matched_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("([xyz]) ([xyz])")
+ result = (p | beam.Create(["a x", "x x", "y y", "z z"])
+ | util.Regex.all_matches(rc))
+ expected_result = [["x x", "x", "x"], ["y y", "y", "y"],
+ ["z z", "z", "z"]]
+ assert_that(result, equal_to(expected_result))
+
+ def test_match_group_kv(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a b c"])
+ | util.Regex.matches_kv("a (b) (c)", 1, 2))
+ assert_that(result, equal_to([("b", "c")]))
+
+ def test_match_group_kv_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("a (b) (c)")
+ pcol = (p | beam.Create(["a b c"]))
+ assert_that(pcol | 'key 1' >> util.Regex.matches_kv(
+ rc, 1), equal_to([("b", "a b c")]), label="CheckKey1")
+
+ assert_that(pcol | 'key 1 group 2' >> util.Regex.matches_kv(
+ rc, 1, 2), equal_to([("b", "c")]), label="CheckKey1Group2")
+
+ def test_match_group_kv_none(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["x y z"])
+ | util.Regex.matches_kv("a (b) (c)", 1, 2))
+ assert_that(result, equal_to([]))
+
+ def test_match_kv_group_names(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["a b c"]) | util.Regex.matches_kv(
+ "a (?P<keyname>b) (?P<valuename>c)", 'keyname', 'valuename'))
+ assert_that(result, equal_to([("b", "c")]))
+
+ def test_match_kv_group_names_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("a (?P<keyname>b) (?P<valuename>c)")
+ result = (p | beam.Create(["a b c"]) | util.Regex.matches_kv(
+ rc, 'keyname', 'valuename'))
+ assert_that(result, equal_to([("b", "c")]))
+
+ def test_match_kv_group_name_none(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["x y z"]) | util.Regex.matches_kv(
+ "a (?P<keyname>b) (?P<valuename>c)", 'keyname', 'valuename'))
+ assert_that(result, equal_to([]))
+
+ def test_replace_all(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["xj", "yj", "zj"]) | util.Regex.replace_all(
+ "[xyz]", "new"))
+ assert_that(result, equal_to(["newj", "newj", "newj"]))
+
+ def test_replace_all_mixed(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["abc", "xj", "yj", "zj", "def"])
+ | util.Regex.replace_all("[xyz]", 'new'))
+ assert_that(result, equal_to(["abc", "newj", "newj", "newj", "def"]))
+
+ def test_replace_all_mixed_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("[xyz]")
+ result = (p | beam.Create(
+ ["abc", "xj", "yj", "zj", "def"]) | util.Regex.replace_all(rc, 'new'))
+ assert_that(result, equal_to(["abc", "newj", "newj", "newj", "def"]))
+
+ def test_replace_first(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["xjx", "yjy", "zjz"])
+ | util.Regex.replace_first("[xyz]", 'new'))
+ assert_that(result, equal_to(["newjx", "newjy", "newjz"]))
+
+ def test_replace_first_mixed(self):
+ with TestPipeline() as p:
+ result = (p | beam.Create(["abc", "xjx", "yjy", "zjz", "def"])
+ | util.Regex.replace_first("[xyz]", 'new'))
+ assert_that(result, equal_to(["abc", "newjx", "newjy", "newjz", "def"]))
+
+ def test_replace_first_mixed_pattern(self):
+ with TestPipeline() as p:
+ rc = re.compile("[xyz]")
+ result = (p | beam.Create(["abc", "xjx", "yjy", "zjz", "def"])
+ | util.Regex.replace_first(rc, 'new'))
+ assert_that(result, equal_to(["abc", "newjx", "newjy", "newjz", "def"]))
+
+ def test_split(self):
+ with TestPipeline() as p:
+ data = ["The quick brown fox jumps over the lazy dog"]
+ result = (p | beam.Create(data) | util.Regex.split("\\W+"))
+ expected_result = [["The", "quick", "brown", "fox", "jumps", "over",
+ "the", "lazy", "dog"]]
+ assert_that(result, equal_to(expected_result))
+
+ def test_split_pattern(self):
+ with TestPipeline() as p:
+ data = ["The quick brown fox jumps over the lazy dog"]
+ rc = re.compile("\\W+")
+ result = (p | beam.Create(data) | util.Regex.split(rc))
+ expected_result = [["The", "quick", "brown", "fox", "jumps", "over",
+ "the", "lazy", "dog"]]
+ assert_that(result, equal_to(expected_result))
+
+ def test_split_with_empty(self):
+ with TestPipeline() as p:
+ data = ["The quick brown fox jumps over the lazy dog"]
+ result = (p | beam.Create(data) | util.Regex.split("\\s", True))
+ expected_result = [['The', '', 'quick', '', '', 'brown', 'fox', 'jumps',
+ 'over', '', '', '', 'the', 'lazy', 'dog']]
+ assert_that(result, equal_to(expected_result))
+
+ def test_split_without_empty(self):
+ with TestPipeline() as p:
+ data = ["The quick brown fox jumps over the lazy dog"]
+ result = (p | beam.Create(data) | util.Regex.split("\\s", False))
+ expected_result = [["The", "quick", "brown", "fox", "jumps", "over",
+ "the", "lazy", "dog"]]
+ assert_that(result, equal_to(expected_result))
+
+
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
unittest.main()
diff --git a/sdks/python/apache_beam/transforms/window.py b/sdks/python/apache_beam/transforms/window.py
index 1990532..e477303 100644
--- a/sdks/python/apache_beam/transforms/window.py
+++ b/sdks/python/apache_beam/transforms/window.py
@@ -361,12 +361,22 @@
Attributes:
size: Size of the window as seconds.
- offset: Offset of this window as seconds since Unix epoch. Windows start at
- t=N * size + offset where t=0 is the epoch. The offset must be a value
- in range [0, size). If it is not it will be normalized to this range.
+ offset: Offset of this window as seconds. Windows start at
+ t=N * size + offset where t=0 is the UNIX epoch. The offset must be a
+ value in range [0, size). If it is not it will be normalized to this
+ range.
"""
def __init__(self, size, offset=0):
+ """Initialize a ``FixedWindows`` function for a given size and offset.
+
+ Args:
+ size (int): Size of the window in seconds.
+ offset(int): Offset of this window as seconds. Windows start at
+ t=N * size + offset where t=0 is the UNIX epoch. The offset must be a
+ value in range [0, size). If it is not it will be normalized to this
+ range.
+ """
if size <= 0:
raise ValueError('The size parameter must be strictly positive.')
self.size = Duration.of(size)
diff --git a/sdks/python/apache_beam/typehints/decorators.py b/sdks/python/apache_beam/typehints/decorators.py
index 3c27e00..4c485fe 100644
--- a/sdks/python/apache_beam/typehints/decorators.py
+++ b/sdks/python/apache_beam/typehints/decorators.py
@@ -61,8 +61,8 @@
def int_to_str(a):
return str(a)
-Type-hinting a function with arguments that unpack tuples are also supported. As
-an example, such a function would be defined as::
+Type-hinting a function with arguments that unpack tuples are also supported
+(in Python 2 only). As an example, such a function would be defined as::
def foo((a, b)):
...
@@ -86,6 +86,8 @@
from __future__ import absolute_import
import inspect
+import logging
+import sys
import types
from builtins import next
from builtins import object
@@ -98,6 +100,12 @@
from apache_beam.typehints.typehints import check_constraint
from apache_beam.typehints.typehints import validate_composite_type_param
+try:
+ import funcsigs # Python 2 only.
+except ImportError:
+ funcsigs = None
+
+
__all__ = [
'with_input_types',
'with_output_types',
@@ -105,13 +113,17 @@
'TypeCheckError',
]
-
# This is missing in the builtin types module. str.upper is arbitrary, any
# method on a C-implemented type will do.
# pylint: disable=invalid-name
_MethodDescriptorType = type(str.upper)
# pylint: enable=invalid-name
+_ANY_VAR_POSITIONAL = typehints.Tuple[typehints.Any, ...]
+_ANY_VAR_KEYWORD = typehints.Dict[typehints.Any, typehints.Any]
+# TODO(BEAM-8280): Remove this when from_callable is ready to be enabled.
+_enable_from_callable = False
+
try:
_original_getfullargspec = inspect.getfullargspec
_use_full_argspec = True
@@ -121,6 +133,8 @@
def getfullargspec(func):
+ # Python 3: Use get_signature instead.
+ assert sys.version_info < (3,), 'This method should not be used in Python 3'
try:
return _original_getfullargspec(func)
except TypeError:
@@ -148,11 +162,60 @@
raise
+def get_signature(func):
+ """Like inspect.signature(), but supports Py2 as well.
+
+ This module uses inspect.signature instead of getfullargspec since in the
+ latter: 'the "self" parameter is always reported, even for bound methods'
+ https://github.com/python/cpython/blob/44f91c388a6f4da9ed3300df32ca290b8aa104ea/Lib/inspect.py#L1103
+ """
+ # Fall back on funcsigs if inspect module doesn't have 'signature'; prefer
+ # inspect.signature over funcsigs.signature if both are available.
+ if hasattr(inspect, 'signature'):
+ inspect_ = inspect
+ else:
+ inspect_ = funcsigs
+
+ try:
+ signature = inspect_.signature(func)
+ except ValueError:
+ # Fall back on a catch-all signature.
+ params = [
+ inspect_.Parameter('_', inspect_.Parameter.POSITIONAL_OR_KEYWORD),
+ inspect_.Parameter('__unknown__varargs',
+ inspect_.Parameter.VAR_POSITIONAL),
+ inspect_.Parameter('__unknown__keywords',
+ inspect_.Parameter.VAR_KEYWORD)]
+
+ signature = inspect_.Signature(params)
+
+ # This is a specialization to hint the first argument of certain builtins,
+ # such as str.strip.
+ if isinstance(func, _MethodDescriptorType):
+ params = list(signature.parameters.values())
+ if params[0].annotation == params[0].empty:
+ params[0] = params[0].replace(annotation=func.__objclass__)
+ signature = signature.replace(parameters=params)
+
+ # This is a specialization to hint the return value of type callables.
+ if (signature.return_annotation == signature.empty and
+ isinstance(func, type)):
+ signature = signature.replace(return_annotation=typehints.normalize(func))
+
+ return signature
+
+
class IOTypeHints(object):
"""Encapsulates all type hint information about a Dataflow construct.
This should primarily be used via the WithTypeHints mixin class, though
may also be attached to other objects (such as Python functions).
+
+ Attributes:
+ input_types: (tuple, dict) List of typing types, and an optional dictionary.
+ May be None. The list and dict correspond to args and kwargs.
+ output_types: (tuple, dict) List of typing types, and an optional dictionary
+ (unused). Only the first element of the list is used. May be None.
"""
__slots__ = ('input_types', 'output_types')
@@ -160,6 +223,53 @@
self.input_types = input_types
self.output_types = output_types
+ @staticmethod
+ def from_callable(fn):
+ """Construct an IOTypeHints object from a callable's signature.
+
+ Supports Python 3 annotations. For partial annotations, sets unknown types
+ to Any, _ANY_VAR_POSITIONAL, or _ANY_VAR_KEYWORD.
+
+ Returns:
+ A new IOTypeHints or None if no annotations found.
+ """
+ if not _enable_from_callable:
+ return None
+ signature = get_signature(fn)
+ if (all(param.annotation == param.empty
+ for param in signature.parameters.values())
+ and signature.return_annotation == signature.empty):
+ return None
+ input_args = []
+ input_kwargs = {}
+ for param in signature.parameters.values():
+ if param.annotation == param.empty:
+ if param.kind == param.VAR_POSITIONAL:
+ input_args.append(_ANY_VAR_POSITIONAL)
+ elif param.kind == param.VAR_KEYWORD:
+ input_kwargs[param.name] = _ANY_VAR_KEYWORD
+ elif param.kind == param.KEYWORD_ONLY:
+ input_kwargs[param.name] = typehints.Any
+ else:
+ input_args.append(typehints.Any)
+ else:
+ if param.kind in [param.KEYWORD_ONLY, param.VAR_KEYWORD]:
+ input_kwargs[param.name] = param.annotation
+ else:
+ assert param.kind in [param.POSITIONAL_ONLY,
+ param.POSITIONAL_OR_KEYWORD,
+ param.VAR_POSITIONAL], \
+ 'Unsupported Parameter kind: %s' % param.kind
+ input_args.append(param.annotation)
+ output_args = []
+ if signature.return_annotation != signature.empty:
+ output_args.append(signature.return_annotation)
+ else:
+ output_args.append(typehints.Any)
+
+ return IOTypeHints(input_types=(tuple(input_args), input_kwargs),
+ output_types=(tuple(output_args), {}))
+
def set_input_types(self, *args, **kwargs):
self.input_types = args, kwargs
@@ -170,22 +280,55 @@
if self.output_types:
args, kwargs = self.output_types
if len(args) != 1 or kwargs:
- raise TypeError('Expected simple output type hint for %s' % context)
+ raise TypeError(
+ 'Expected single output type hint for %s but got: %s' % (
+ context, self.output_types))
return args[0]
+ def has_simple_output_type(self):
+ """Whether there's a single positional output type."""
+ return (self.output_types and len(self.output_types[0]) == 1 and
+ not self.output_types[1])
+
+ def strip_iterable(self):
+ """Removes outer Iterable (or equivalent) from output type.
+
+ Only affects instances with simple output types, otherwise is a no-op.
+
+ Example: Generator[Tuple(int, int)] becomes Tuple(int, int)
+
+ Raises:
+ ValueError if output type is simple and not iterable.
+ """
+ if not self.has_simple_output_type():
+ return
+ yielded_type = typehints.get_yielded_type(self.output_types[0][0])
+ self.output_types = ((yielded_type,), {})
+
def copy(self):
return IOTypeHints(self.input_types, self.output_types)
def with_defaults(self, hints):
if not hints:
return self
- elif not self:
- return hints
- return IOTypeHints(self.input_types or hints.input_types,
- self.output_types or hints.output_types)
+ if self._has_input_types():
+ input_types = self.input_types
+ else:
+ input_types = hints.input_types
+ if self._has_output_types():
+ output_types = self.output_types
+ else:
+ output_types = hints.output_types
+ return IOTypeHints(input_types, output_types)
+
+ def _has_input_types(self):
+ return self.input_types is not None and any(self.input_types)
+
+ def _has_output_types(self):
+ return self.output_types is not None and any(self.output_types)
def __bool__(self):
- return bool(self.input_types or self.output_types)
+ return self._has_input_types() or self._has_output_types()
def __repr__(self):
return 'IOTypeHints[inputs=%s, outputs=%s]' % (
@@ -208,6 +351,13 @@
return self._type_hints
def get_type_hints(self):
+ """Gets and/or initializes type hints for this object.
+
+ If type hints have not been set, attempts to initialize type hints in this
+ order:
+ - Using self.default_type_hints().
+ - Using self.__class__ type hints.
+ """
return (self._get_or_create_type_hints()
.with_defaults(self.default_type_hints())
.with_defaults(get_type_hints(self.__class__)))
@@ -216,10 +366,14 @@
return None
def with_input_types(self, *arg_hints, **kwarg_hints):
+ arg_hints = native_type_compatibility.convert_to_beam_types(arg_hints)
+ kwarg_hints = native_type_compatibility.convert_to_beam_types(kwarg_hints)
self._get_or_create_type_hints().set_input_types(*arg_hints, **kwarg_hints)
return self
def with_output_types(self, *arg_hints, **kwarg_hints):
+ arg_hints = native_type_compatibility.convert_to_beam_types(arg_hints)
+ kwarg_hints = native_type_compatibility.convert_to_beam_types(kwarg_hints)
self._get_or_create_type_hints().set_output_types(*arg_hints, **kwarg_hints)
return self
@@ -244,7 +398,7 @@
def _unpack_positional_arg_hints(arg, hint):
"""Unpacks the given hint according to the nested structure of arg.
- For example, if arg is [[a, b], c] and hint is Tuple[Any, int], than
+ For example, if arg is [[a, b], c] and hint is Tuple[Any, int], then
this function would return ((Any, Any), int) so it can be used in conjunction
with inspect.getcallargs.
"""
@@ -261,8 +415,20 @@
def getcallargs_forhints(func, *typeargs, **typekwargs):
- """Like inspect.getcallargs, but understands that Tuple[] and an Any unpack.
+ """Like inspect.getcallargs, with support for declaring default args as Any.
+
+ In Python 2, understands that Tuple[] and an Any unpack.
+
+ Returns:
+ (Dict[str, Any]) A dictionary from arguments names to values.
"""
+ if sys.version_info < (3,):
+ return getcallargs_forhints_impl_py2(func, typeargs, typekwargs)
+ else:
+ return getcallargs_forhints_impl_py3(func, typeargs, typekwargs)
+
+
+def getcallargs_forhints_impl_py2(func, typeargs, typekwargs):
argspec = getfullargspec(func)
# Turn Tuple[x, y] into (x, y) so getcallargs can do the proper unpacking.
packed_typeargs = [_unpack_positional_arg_hints(arg, hint)
@@ -272,21 +438,14 @@
# Monkeypatch inspect.getfullargspec to allow passing non-function objects.
# getfullargspec (getargspec on Python 2) are used by inspect.getcallargs.
# TODO(BEAM-5490): Reimplement getcallargs and stop relying on monkeypatch.
- if _use_full_argspec:
- inspect.getfullargspec = getfullargspec
- else: # Python 2
- inspect.getargspec = getfullargspec
-
+ inspect.getargspec = getfullargspec
try:
callargs = inspect.getcallargs(func, *packed_typeargs, **typekwargs)
except TypeError as e:
raise TypeCheckError(e)
finally:
# Revert monkey-patch.
- if _use_full_argspec:
- inspect.getfullargspec = _original_getfullargspec
- else:
- inspect.getargspec = _original_getfullargspec
+ inspect.getargspec = _original_getfullargspec
if argspec.defaults:
# Declare any default arguments to be Any.
@@ -297,27 +456,121 @@
callargs[var] = typehints.Any
# Patch up varargs and keywords
if argspec.varargs:
+ # TODO(BEAM-8122): This will always assign _ANY_VAR_POSITIONAL. Should be
+ # "callargs.get(...) or _ANY_VAR_POSITIONAL".
callargs[argspec.varargs] = typekwargs.get(
- argspec.varargs, typehints.Tuple[typehints.Any, ...])
- if _use_full_argspec:
- varkw = argspec.varkw
- else: # Python 2
- varkw = argspec.keywords
+ argspec.varargs, _ANY_VAR_POSITIONAL)
+ varkw = argspec.keywords
if varkw:
# TODO(robertwb): Consider taking the union of key and value types.
- callargs[varkw] = typekwargs.get(
- varkw, typehints.Dict[typehints.Any, typehints.Any])
+ callargs[varkw] = typekwargs.get(varkw, _ANY_VAR_KEYWORD)
# TODO(BEAM-5878) Support kwonlyargs.
return callargs
+def _normalize_var_positional_hint(hint):
+ """Converts a var_positional hint into Tuple[Union[<types>], ...] form.
+
+ Args:
+ hint: (tuple) Should be either a tuple of one or more types, or a single
+ Tuple[<type>, ...].
+
+ Raises:
+ TypeCheckError if hint does not have the right form.
+ """
+ if not hint or type(hint) != tuple:
+ raise TypeCheckError('Unexpected VAR_POSITIONAL value: %s' % hint)
+
+ if len(hint) == 1 and isinstance(hint[0], typehints.TupleSequenceConstraint):
+ # Example: tuple(Tuple[Any, ...]) -> Tuple[Any, ...]
+ return hint[0]
+ else:
+ # Example: tuple(int, str) -> Tuple[Union[int, str], ...]
+ return typehints.Tuple[typehints.Union[hint], ...]
+
+
+def _normalize_var_keyword_hint(hint, arg_name):
+ """Converts a var_keyword hint into Dict[<key type>, <value type>] form.
+
+ Args:
+ hint: (dict) Should either contain a pair (arg_name,
+ Dict[<key type>, <value type>]), or one or more possible types for the
+ value.
+ arg_name: (str) The keyword receiving this hint.
+
+ Raises:
+ TypeCheckError if hint does not have the right form.
+ """
+ if not hint or type(hint) != dict:
+ raise TypeCheckError('Unexpected VAR_KEYWORD value: %s' % hint)
+ keys = list(hint.keys())
+ values = list(hint.values())
+ if (len(values) == 1 and
+ keys[0] == arg_name and
+ isinstance(values[0], typehints.DictConstraint)):
+ # Example: dict(kwargs=Dict[str, Any]) -> Dict[str, Any]
+ return values[0]
+ else:
+ # Example: dict(k1=str, k2=int) -> Dict[str, Union[str,int]]
+ return typehints.Dict[str, typehints.Union[values]]
+
+
+def getcallargs_forhints_impl_py3(func, type_args, type_kwargs):
+ """Bind type_args and type_kwargs to func.
+
+ Works like inspect.getcallargs, with some modifications to support type hint
+ checks.
+ For unbound args, will use annotations and fall back to Any (or variants of
+ Any).
+
+ Returns:
+ A mapping from parameter name to argument.
+ """
+ try:
+ signature = get_signature(func)
+ except ValueError as e:
+ logging.warning('Could not get signature for function: %s: %s', func, e)
+ return {}
+ try:
+ bindings = signature.bind(*type_args, **type_kwargs)
+ except TypeError as e:
+ # Might be raised due to too few or too many arguments.
+ raise TypeCheckError(e)
+ bound_args = bindings.arguments
+ for param in signature.parameters.values():
+ if param.name in bound_args:
+ # Bound: unpack/convert variadic arguments.
+ if param.kind == param.VAR_POSITIONAL:
+ bound_args[param.name] = _normalize_var_positional_hint(
+ bound_args[param.name])
+ elif param.kind == param.VAR_KEYWORD:
+ bound_args[param.name] = _normalize_var_keyword_hint(
+ bound_args[param.name], param.name)
+ else:
+ # Unbound: must have a default or be variadic.
+ if param.annotation != param.empty:
+ bound_args[param.name] = param.annotation
+ elif param.kind == param.VAR_POSITIONAL:
+ bound_args[param.name] = _ANY_VAR_POSITIONAL
+ elif param.kind == param.VAR_KEYWORD:
+ bound_args[param.name] = _ANY_VAR_KEYWORD
+ elif param.default is not param.empty:
+ # Declare unbound parameters with defaults to be Any.
+ bound_args[param.name] = typehints.Any
+ else:
+ # This case should be caught by signature.bind() above.
+ raise ValueError('Unexpected unbound parameter: %s' % param.name)
+
+ return dict(bound_args)
+
+
def get_type_hints(fn):
"""Gets the type hint associated with an arbitrary object fn.
- Always returns a valid IOTypeHints object, creating one if necissary.
+ Always returns a valid IOTypeHints object, creating one if necessary.
"""
# pylint: disable=protected-access
if not hasattr(fn, '_type_hints'):
@@ -327,7 +580,8 @@
# Can't add arbitrary attributes to this object,
# but might have some restrictions anyways...
hints = IOTypeHints()
- if isinstance(fn, _MethodDescriptorType):
+ # Python 3.7 introduces annotations for _MethodDescriptorTypes.
+ if isinstance(fn, _MethodDescriptorType) and sys.version_info < (3, 7):
hints.set_input_types(fn.__objclass__)
return hints
return fn._type_hints
diff --git a/sdks/python/apache_beam/typehints/decorators_test.py b/sdks/python/apache_beam/typehints/decorators_test.py
new file mode 100644
index 0000000..645fae6
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/decorators_test.py
@@ -0,0 +1,130 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for decorators module."""
+
+from __future__ import absolute_import
+
+import sys
+import unittest
+
+from apache_beam.typehints import Any
+from apache_beam.typehints import List
+from apache_beam.typehints import WithTypeHints
+from apache_beam.typehints import decorators
+
+decorators._enable_from_callable = True
+
+
+class IOTypeHintsTest(unittest.TestCase):
+
+ def test_get_signature(self):
+ # Basic coverage only to make sure function works in Py2 and Py3.
+ def fn(a, b=1, *c, **d):
+ return a, b, c, d
+ s = decorators.get_signature(fn)
+ self.assertListEqual(list(s.parameters), ['a', 'b', 'c', 'd'])
+
+ def test_get_signature_builtin(self):
+ # Tests a builtin function for 3.7+ and fallback result for older versions.
+ s = decorators.get_signature(list)
+ if sys.version_info < (3, 7):
+ self.assertListEqual(list(s.parameters),
+ ['_', '__unknown__varargs', '__unknown__keywords'])
+ else:
+ self.assertListEqual(list(s.parameters),
+ ['iterable'])
+ self.assertEqual(s.return_annotation, List[Any])
+
+ def test_from_callable_without_annotations(self):
+ # Python 2 doesn't support annotations. See decorators_test_py3.py for that.
+ def fn(a, b=None, *args, **kwargs):
+ return a, b, args, kwargs
+ th = decorators.IOTypeHints.from_callable(fn)
+ self.assertIsNone(th)
+
+ def test_from_callable_builtin(self):
+ th = decorators.IOTypeHints.from_callable(len)
+ self.assertIsNone(th)
+
+ def test_from_callable_method_descriptor(self):
+ # from_callable() injects an annotation in this special type of builtin.
+ th = decorators.IOTypeHints.from_callable(str.strip)
+ if sys.version_info >= (3, 7):
+ self.assertEqual(th.input_types, ((str, Any), {}))
+ else:
+ self.assertEqual(th.input_types,
+ ((str, decorators._ANY_VAR_POSITIONAL),
+ {'__unknown__keywords': decorators._ANY_VAR_KEYWORD}))
+ self.assertEqual(th.output_types, ((Any,), {}))
+
+
+class WithTypeHintsTest(unittest.TestCase):
+ def test_get_type_hints_no_settings(self):
+ class Base(WithTypeHints):
+ pass
+
+ th = Base().get_type_hints()
+ self.assertEqual(th.input_types, None)
+ self.assertEqual(th.output_types, None)
+
+ def test_get_type_hints_class_decorators(self):
+ @decorators.with_input_types(int, str)
+ @decorators.with_output_types(int)
+ class Base(WithTypeHints):
+ pass
+
+ th = Base().get_type_hints()
+ self.assertEqual(th.input_types, ((int, str), {}))
+ self.assertEqual(th.output_types, ((int, ), {}))
+
+ def test_get_type_hints_class_defaults(self):
+ class Base(WithTypeHints):
+ def default_type_hints(self):
+ return decorators.IOTypeHints(
+ input_types=((int, str), {}),
+ output_types=((int, ), {}))
+
+ th = Base().get_type_hints()
+ self.assertEqual(th.input_types, ((int, str), {}))
+ self.assertEqual(th.output_types, ((int, ), {}))
+
+ def test_get_type_hints_precedence_defaults_over_decorators(self):
+ @decorators.with_input_types(int)
+ @decorators.with_output_types(str)
+ class Base(WithTypeHints):
+ def default_type_hints(self):
+ return decorators.IOTypeHints(
+ input_types=((float, ), {}))
+
+ th = Base().get_type_hints()
+ self.assertEqual(th.input_types, ((float, ), {}))
+ self.assertEqual(th.output_types, ((str, ), {}))
+
+ def test_get_type_hints_precedence_instance_over_defaults(self):
+ class Base(WithTypeHints):
+ def default_type_hints(self):
+ return decorators.IOTypeHints(
+ input_types=((float, ), {}), output_types=((str, ), {}))
+
+ th = Base().with_input_types(int).get_type_hints()
+ self.assertEqual(th.input_types, ((int, ), {}))
+ self.assertEqual(th.output_types, ((str, ), {}))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/typehints/decorators_test_py3.py b/sdks/python/apache_beam/typehints/decorators_test_py3.py
new file mode 100644
index 0000000..7659a72
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/decorators_test_py3.py
@@ -0,0 +1,112 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for decorators module with Python 3 syntax not supported by 2.7."""
+
+from __future__ import absolute_import
+
+import unittest
+
+from apache_beam.typehints import Any
+from apache_beam.typehints import Dict
+from apache_beam.typehints import List
+from apache_beam.typehints import Tuple
+from apache_beam.typehints import TypeVariable
+from apache_beam.typehints import decorators
+
+decorators._enable_from_callable = True
+T = TypeVariable('T')
+
+
+class IOTypeHintsTest(unittest.TestCase):
+
+ def test_from_callable(self):
+ def fn(a: int, b: str = None, *args: Tuple[T], foo: List[int],
+ **kwargs: Dict[str, str]) -> Tuple:
+ return a, b, args, foo, kwargs
+ th = decorators.IOTypeHints.from_callable(fn)
+ self.assertEqual(th.input_types, (
+ (int, str, Tuple[T]), {'foo': List[int], 'kwargs': Dict[str, str]}))
+ self.assertEqual(th.output_types, ((Tuple,), {}))
+
+ def test_from_callable_partial_annotations(self):
+ def fn(a: int, b=None, *args, foo: List[int], **kwargs):
+ return a, b, args, foo, kwargs
+ th = decorators.IOTypeHints.from_callable(fn)
+ self.assertEqual(th.input_types,
+ ((int, Any, Tuple[Any, ...]),
+ {'foo': List[int], 'kwargs': Dict[Any, Any]}))
+ self.assertEqual(th.output_types, ((Any,), {}))
+
+ def test_from_callable_class(self):
+ class Class(object):
+ def __init__(self, unused_arg: int):
+ pass
+
+ th = decorators.IOTypeHints.from_callable(Class)
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((Class,), {}))
+
+ def test_from_callable_method(self):
+ class Class(object):
+ def method(self, arg: T = None) -> None:
+ pass
+
+ th = decorators.IOTypeHints.from_callable(Class.method)
+ self.assertEqual(th.input_types, ((Any, T), {}))
+ self.assertEqual(th.output_types, ((None,), {}))
+
+ th = decorators.IOTypeHints.from_callable(Class().method)
+ self.assertEqual(th.input_types, ((T,), {}))
+ self.assertEqual(th.output_types, ((None,), {}))
+
+ def test_getcallargs_forhints(self):
+ def fn(a: int, b: str = None, *args: Tuple[T], foo: List[int],
+ **kwargs: Dict[str, str]) -> Tuple:
+ return a, b, args, foo, kwargs
+ callargs = decorators.getcallargs_forhints(fn, float, foo=List[str])
+ self.assertDictEqual(callargs,
+ {'a': float,
+ 'b': str,
+ 'args': Tuple[T],
+ 'foo': List[str],
+ 'kwargs': Dict[str, str]})
+
+ def test_getcallargs_forhints_default_arg(self):
+ # Default args are not necessarily types, so they should be ignored.
+ def fn(a=List[int], b=None, *args, foo=(), **kwargs) -> Tuple:
+ return a, b, args, foo, kwargs
+ callargs = decorators.getcallargs_forhints(fn)
+ self.assertDictEqual(callargs,
+ {'a': Any,
+ 'b': Any,
+ 'args': Tuple[Any, ...],
+ 'foo': Any,
+ 'kwargs': Dict[Any, Any]})
+
+ def test_getcallargs_forhints_missing_arg(self):
+ def fn(a, b=None, *args, foo, **kwargs):
+ return a, b, args, foo, kwargs
+
+ with self.assertRaisesRegexp(decorators.TypeCheckError, "missing.*'a'"):
+ decorators.getcallargs_forhints(fn, foo=List[int])
+ with self.assertRaisesRegexp(decorators.TypeCheckError, "missing.*'foo'"):
+ decorators.getcallargs_forhints(fn, 5)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/typehints/native_type_compatibility.py b/sdks/python/apache_beam/typehints/native_type_compatibility.py
index d08049a..43cdedc 100644
--- a/sdks/python/apache_beam/typehints/native_type_compatibility.py
+++ b/sdks/python/apache_beam/typehints/native_type_compatibility.py
@@ -37,6 +37,8 @@
def _get_compatible_args(typ):
+ if isinstance(typ, typing.TypeVar):
+ return (typ.__name__,)
# On Python versions < 3.5.3, the Tuple and Union type from typing do
# not have an __args__ attribute, but a __tuple_params__, and a
# __union_params__ argument respectively.
@@ -86,6 +88,11 @@
try:
return issubclass(derived, parent)
except TypeError:
+ if hasattr(derived, '__origin__'):
+ try:
+ return issubclass(derived.__origin__, parent)
+ except TypeError:
+ pass
return False
@@ -94,16 +101,48 @@
def _match_same_type(match_against):
- # For Union types. They can't be compared with isinstance either, so we
- # have to compare their types directly.
+ # For types that can't be compared with isinstance or _safe_issubclass.
return lambda user_type: type(user_type) == type(match_against)
+def _match_is_exactly_iterable(user_type):
+ # Avoid unintentionally catching all subtypes (e.g. strings and mappings).
+ if sys.version_info < (3, 7):
+ expected_origin = typing.Iterable
+ else:
+ expected_origin = collections.abc.Iterable
+ return getattr(user_type, '__origin__', None) is expected_origin
+
+
def _match_is_named_tuple(user_type):
return (_safe_issubclass(user_type, typing.Tuple) and
hasattr(user_type, '_field_types'))
+def _match_is_union(user_type):
+ # For non-subscripted unions (Python 2.7.14+ with typing 3.64)
+ if user_type is typing.Union:
+ return True
+
+ try: # Python 3.5.2
+ if isinstance(user_type, typing.UnionMeta):
+ return True
+ except AttributeError:
+ pass
+
+ try: # Python 3.5.4+, or Python 2.7.14+ with typing 3.64
+ return user_type.__origin__ is typing.Union
+ except AttributeError:
+ pass
+
+ return False
+
+
+# Mapping from typing.TypeVar/typehints.TypeVariable ids to an object of the
+# other type. Bidirectional mapping preserves typing.TypeVar instances.
+_type_var_cache = {}
+
+
def convert_to_beam_type(typ):
"""Convert a given typing type to a Beam type.
@@ -117,6 +156,20 @@
Raises:
~exceptions.ValueError: The type was malformed.
"""
+ if isinstance(typ, typing.TypeVar):
+ # This is a special case, as it's not parameterized by types.
+ # Also, identity must be preserved through conversion (i.e. the same
+ # TypeVar instance must get converted into the same TypeVariable instance).
+ # A global cache should be OK as the number of distinct type variables
+ # is generally small.
+ if id(typ) not in _type_var_cache:
+ new_type_variable = typehints.TypeVariable(typ.__name__)
+ _type_var_cache[id(typ)] = new_type_variable
+ _type_var_cache[id(new_type_variable)] = typ
+ return _type_var_cache[id(typ)]
+ elif getattr(typ, '__module__', None) != 'typing':
+ # Only translate types from the typing module.
+ return typ
type_map = [
_TypeMapEntry(
@@ -128,6 +181,10 @@
arity=2,
beam_type=typehints.Dict),
_TypeMapEntry(
+ match=_match_is_exactly_iterable,
+ arity=1,
+ beam_type=typehints.Iterable),
+ _TypeMapEntry(
match=_match_issubclass(typing.List),
arity=1,
beam_type=typehints.List),
@@ -144,10 +201,15 @@
match=_match_issubclass(typing.Tuple),
arity=-1,
beam_type=typehints.Tuple),
+ _TypeMapEntry(match=_match_is_union, arity=-1, beam_type=typehints.Union),
_TypeMapEntry(
- match=_match_same_type(typing.Union),
- arity=-1,
- beam_type=typehints.Union)
+ match=_match_issubclass(typing.Generator),
+ arity=3,
+ beam_type=typehints.Generator),
+ _TypeMapEntry(
+ match=_match_issubclass(typing.Iterator),
+ arity=1,
+ beam_type=typehints.Iterator),
]
# Find the first matching entry.
@@ -189,3 +251,73 @@
return {k: convert_to_beam_type(v) for k, v in args.items()}
else:
return [convert_to_beam_type(v) for v in args]
+
+
+def convert_to_typing_type(typ):
+ """Converts a given Beam type to a typing type.
+
+ This is the reverse of convert_to_beam_type.
+
+ Args:
+ typ: If a typehints.TypeConstraint, the type to convert. Otherwise, typ
+ will be unchanged.
+
+ Returns:
+ Converted version of typ, or unchanged.
+
+ Raises:
+ ~exceptions.ValueError: The type was malformed or could not be converted.
+ """
+ if isinstance(typ, typehints.TypeVariable):
+ # This is a special case, as it's not parameterized by types.
+ # Also, identity must be preserved through conversion (i.e. the same
+ # TypeVariable instance must get converted into the same TypeVar instance).
+ # A global cache should be OK as the number of distinct type variables
+ # is generally small.
+ if id(typ) not in _type_var_cache:
+ new_type_variable = typing.TypeVar(typ.name)
+ _type_var_cache[id(typ)] = new_type_variable
+ _type_var_cache[id(new_type_variable)] = typ
+ return _type_var_cache[id(typ)]
+ elif not getattr(typ, '__module__', None).endswith('typehints'):
+ # Only translate types from the typehints module.
+ return typ
+
+ if isinstance(typ, typehints.AnyTypeConstraint):
+ return typing.Any
+ if isinstance(typ, typehints.DictConstraint):
+ return typing.Dict[convert_to_typing_type(typ.key_type),
+ convert_to_typing_type(typ.value_type)]
+ if isinstance(typ, typehints.ListConstraint):
+ return typing.List[convert_to_typing_type(typ.inner_type)]
+ if isinstance(typ, typehints.IterableTypeConstraint):
+ return typing.Iterable[convert_to_typing_type(typ.inner_type)]
+ if isinstance(typ, typehints.UnionConstraint):
+ return typing.Union[tuple(convert_to_typing_types(typ.union_types))]
+ if isinstance(typ, typehints.SetTypeConstraint):
+ return typing.Set[convert_to_typing_type(typ.inner_type)]
+ if isinstance(typ, typehints.TupleConstraint):
+ return typing.Tuple[tuple(convert_to_typing_types(typ.tuple_types))]
+ if isinstance(typ, typehints.TupleSequenceConstraint):
+ return typing.Tuple[convert_to_typing_type(typ.inner_type), ...]
+ if isinstance(typ, typehints.IteratorTypeConstraint):
+ return typing.Iterator[convert_to_typing_type(typ.yielded_type)]
+
+ raise ValueError('Failed to convert Beam type: %s' % typ)
+
+
+def convert_to_typing_types(args):
+ """Convert the given list or dictionary of args to typing types.
+
+ Args:
+ args: Either an iterable of types, or a dictionary where the values are
+ types.
+
+ Returns:
+ If given an iterable, a list of converted types. If given a dictionary,
+ a dictionary with the same keys, and values which have been converted.
+ """
+ if isinstance(args, dict):
+ return {k: convert_to_typing_type(v) for k, v in args.items()}
+ else:
+ return [convert_to_typing_type(v) for v in args]
diff --git a/sdks/python/apache_beam/typehints/native_type_compatibility_test.py b/sdks/python/apache_beam/typehints/native_type_compatibility_test.py
index 692a801..bca9d50 100644
--- a/sdks/python/apache_beam/typehints/native_type_compatibility_test.py
+++ b/sdks/python/apache_beam/typehints/native_type_compatibility_test.py
@@ -19,13 +19,15 @@
from __future__ import absolute_import
-import os
import sys
import typing
import unittest
-from apache_beam.typehints import native_type_compatibility
from apache_beam.typehints import typehints
+from apache_beam.typehints.native_type_compatibility import convert_to_beam_type
+from apache_beam.typehints.native_type_compatibility import convert_to_beam_types
+from apache_beam.typehints.native_type_compatibility import convert_to_typing_type
+from apache_beam.typehints.native_type_compatibility import convert_to_typing_types
_TestNamedTuple = typing.NamedTuple('_TestNamedTuple',
[('age', int), ('name', bytes)])
@@ -37,10 +39,6 @@
pass
-@unittest.skipIf(sys.version_info >= (3, 7, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.7. '
- 'See BEAM-6985')
class NativeTypeCompatibilityTest(unittest.TestCase):
def test_convert_to_beam_type(self):
@@ -52,13 +50,14 @@
('simple dict', typing.Dict[bytes, int],
typehints.Dict[bytes, int]),
('simple list', typing.List[int], typehints.List[int]),
+ ('simple iterable', typing.Iterable[int], typehints.Iterable[int]),
('simple optional', typing.Optional[int], typehints.Optional[int]),
('simple set', typing.Set[float], typehints.Set[float]),
('simple unary tuple', typing.Tuple[bytes],
typehints.Tuple[bytes]),
('simple union', typing.Union[int, bytes, float],
typehints.Union[int, bytes, float]),
- ('namedtuple', _TestNamedTuple, typehints.Any),
+ ('namedtuple', _TestNamedTuple, _TestNamedTuple),
('test class', _TestClass, _TestClass),
('test class in list', typing.List[_TestClass],
typehints.List[_TestClass]),
@@ -66,23 +65,42 @@
bytes, typing.Union[int, bytes, float]]]],
typehints.Tuple[bytes, typehints.List[typehints.Tuple[
bytes, typehints.Union[int, bytes, float]]]]),
+ # TODO(BEAM-7713): This case seems to fail on Py3.5.2 but not 3.5.4.
+ ('arbitrary-length tuple', typing.Tuple[int, ...],
+ typehints.Tuple[int, ...])
+ if sys.version_info >= (3, 5, 4) else None,
('flat alias', _TestFlatAlias, typehints.Tuple[bytes, float]),
('nested alias', _TestNestedAlias,
typehints.List[typehints.Tuple[bytes, float]]),
('complex dict',
typing.Dict[bytes, typing.List[typing.Tuple[bytes, _TestClass]]],
typehints.Dict[bytes, typehints.List[typehints.Tuple[
- bytes, _TestClass]]])
+ bytes, _TestClass]]]),
+ ('type var', typing.TypeVar('T'), typehints.TypeVariable('T')),
+ ('nested type var',
+ typing.Tuple[typing.TypeVar('K'), typing.TypeVar('V')],
+ typehints.Tuple[typehints.TypeVariable('K'),
+ typehints.TypeVariable('V')]),
+ ('iterator', typing.Iterator[typing.Any],
+ typehints.Iterator[typehints.Any]),
]
for test_case in test_cases:
+ if test_case is None:
+ continue
# Unlike typing types, Beam types are guaranteed to compare equal.
description = test_case[0]
typing_type = test_case[1]
- beam_type = test_case[2]
- self.assertEqual(
- native_type_compatibility.convert_to_beam_type(typing_type),
- beam_type, description)
+ expected_beam_type = test_case[2]
+ converted_beam_type = convert_to_beam_type(typing_type)
+ self.assertEqual(converted_beam_type, expected_beam_type, description)
+ converted_typing_type = convert_to_typing_type(converted_beam_type)
+ self.assertEqual(converted_typing_type, typing_type, description)
+
+ def test_generator_converted_to_iterator(self):
+ self.assertEqual(
+ typehints.Iterator[int],
+ convert_to_beam_type(typing.Generator[int, None, None]))
def test_convert_nested_to_beam_type(self):
self.assertEqual(
@@ -99,9 +117,10 @@
beam_types = [bytes, typehints.List[bytes],
typehints.List[typehints.Tuple[bytes, int]],
typehints.Union[int, typehints.List[int]]]
- self.assertEqual(
- native_type_compatibility.convert_to_beam_types(typing_types),
- beam_types)
+ converted_beam_types = convert_to_beam_types(typing_types)
+ self.assertEqual(converted_beam_types, beam_types)
+ converted_typing_types = convert_to_typing_types(converted_beam_types)
+ self.assertEqual(converted_typing_types, typing_types)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/typehints/opcodes.py b/sdks/python/apache_beam/typehints/opcodes.py
index 8bdf16b..6aa5de2 100644
--- a/sdks/python/apache_beam/typehints/opcodes.py
+++ b/sdks/python/apache_beam/typehints/opcodes.py
@@ -29,6 +29,7 @@
from __future__ import absolute_import
import inspect
+import logging
import sys
import types
from functools import reduce
@@ -155,14 +156,13 @@
if base in (str, unicode):
out = base
elif (isinstance(index, Const) and isinstance(index.value, int)
- and isinstance(base, typehints.TupleHint.TupleConstraint)):
- const_index = index.value
- if -len(base.tuple_types) < const_index < len(base.tuple_types):
- out = base.tuple_types[const_index]
- else:
+ and isinstance(base, typehints.IndexableTypeConstraint)):
+ try:
+ out = base._constraint_for_index(index.value)
+ except IndexError:
out = element_type(base)
- elif index == slice:
- out = typehints.List[element_type(base)]
+ elif index == slice and isinstance(base, typehints.IndexableTypeConstraint):
+ out = base
else:
out = element_type(base)
state.stack.append(out)
@@ -208,6 +208,14 @@
new_element_type]]
+def map_add(state, arg):
+ new_key_type = Const.unwrap(state.stack.pop())
+ new_value_type = Const.unwrap(state.stack.pop())
+ state.stack[-arg] = Dict[
+ Union[state.stack[-arg].key_type, new_key_type],
+ Union[state.stack[-arg].value_type, new_value_type]]
+
+
load_locals = push_value(Dict[str, Any])
# return_value
@@ -272,7 +280,9 @@
state.stack[-arg:] = [List[reduce(union, state.stack[-arg:], Union[()])]]
-build_map = push_value(Dict[Any, Any])
+# A Dict[Union[], Union[]] is the type of an empty dict.
+def build_map(state, unused_arg):
+ state.stack.append(Dict[Union[()], Union[()]])
def load_attr(state, arg):
@@ -296,6 +306,20 @@
state.stack.append(Any)
+def load_method(state, arg):
+ """Like load_attr. Replaces TOS object with method and TOS."""
+ o = state.stack.pop()
+ name = state.get_name(arg)
+ if isinstance(o, Const):
+ method = Const(getattr(o.value, name))
+ elif isinstance(o, typehints.AnyTypeConstraint):
+ method = typehints.Any
+ else:
+ method = Const(BoundMethod(getattr(o, name), o))
+
+ state.stack.append(method)
+
+
def compare_op(state, unused_arg):
# Could really be anything...
state.stack[-2:] = [bool]
@@ -334,8 +358,8 @@
state.vars[arg] = Any # really an error
-def load_closure(state, unused_arg):
- state.stack.append(Any) # really a Cell
+def load_closure(state, arg):
+ state.stack.append(state.get_closure(arg))
def load_deref(state, arg):
@@ -343,13 +367,6 @@
# raise_varargs
-def call_function(state, arg, has_var=False, has_kw=False):
- # TODO(robertwb): Recognize builtins and dataflow objects
- # (especially special return values).
- pop_count = (arg & 0xFF) + 2 * (arg >> 8) + 1 + has_var + has_kw
- state.stack[-pop_count:] = [Any]
-
-
def make_function(state, arg):
"""Creates a function with the arguments at the top of the stack.
"""
@@ -358,11 +375,36 @@
if sys.version_info[0] == 2:
func_code = state.stack[-1].value
func = types.FunctionType(func_code, globals)
- else:
+ # argc is the number of default parameters. Ignored here.
+ pop_count = 1 + arg
+ else: # Python 3.x
func_name = state.stack[-1].value
func_code = state.stack[-2].value
- func = types.FunctionType(func_code, globals, name=func_name)
- state.stack.append(Const(func))
+ pop_count = 2
+ closure = None
+ if sys.version_info[:2] == (3, 5):
+ # https://docs.python.org/3.5/library/dis.html#opcode-MAKE_FUNCTION
+ num_default_pos_args = (arg & 0xff)
+ num_default_kwonly_args = ((arg >> 8) & 0xff)
+ num_annotations = ((arg >> 16) & 0x7fff)
+ pop_count += (num_default_pos_args + 2 * num_default_kwonly_args +
+ num_annotations + num_annotations > 0)
+ elif sys.version_info >= (3, 6):
+ # arg contains flags, with corresponding stack values if positive.
+ # https://docs.python.org/3.6/library/dis.html#opcode-MAKE_FUNCTION
+ pop_count += bin(arg).count('1')
+ if arg & 0x08:
+ # Convert types in Tuple constraint to a tuple of CPython cells.
+ # https://stackoverflow.com/a/44670295
+ closure = tuple(
+ (lambda _: lambda: _)(t).__closure__[0]
+ for t in state.stack[-3].tuple_types)
+
+ func = types.FunctionType(func_code, globals, name=func_name,
+ closure=closure)
+
+ assert pop_count <= len(state.stack)
+ state.stack[-pop_count:] = [Const(func)]
def make_closure(state, arg):
@@ -373,13 +415,36 @@
state.stack[-arg:] = [slice] # a slice object
-def call_function_var(state, arg):
- call_function(state, arg, has_var=True)
+def _unpack_lists(state, arg):
+ """Extract inner types of Lists and Tuples.
+
+ Pops arg count items from the stack, concatenates their inner types into 1
+ list, and returns that list.
+ Example: if stack[-arg:] == [[i1, i2], [i3]], the output is [i1, i2, i3]
+ """
+ types = []
+ for i in range(arg, 0, -1):
+ type_constraint = state.stack[-i]
+ if isinstance(type_constraint, typehints.IndexableTypeConstraint):
+ types.extend(type_constraint._inner_types())
+ else:
+ logging.debug('Unhandled type_constraint: %r', type_constraint)
+ types.append(typehints.Any)
+ state.stack[-arg:] = []
+ return types
-def call_function_kw(state, arg):
- call_function(state, arg, has_kw=True)
+def build_list_unpack(state, arg):
+ """Joins arg count iterables from the stack into a single list."""
+ state.stack.append(List[Union[_unpack_lists(state, arg)]])
-def call_function_var_wk(state, arg):
- call_function(state, arg, has_var=True, has_kw=True)
+def build_tuple_unpack(state, arg):
+ """Joins arg count iterables from the stack into a single tuple."""
+ state.stack.append(Tuple[_unpack_lists(state, arg)])
+
+
+def build_tuple_unpack_with_call(state, arg):
+ """Same as build_tuple_unpack, with an extra fn argument at the bottom of the
+ stack, which remains untouched."""
+ build_tuple_unpack(state, arg)
diff --git a/sdks/python/apache_beam/typehints/trivial_inference.py b/sdks/python/apache_beam/typehints/trivial_inference.py
index 9380d7d..c67cb7b 100644
--- a/sdks/python/apache_beam/typehints/trivial_inference.py
+++ b/sdks/python/apache_beam/typehints/trivial_inference.py
@@ -27,6 +27,7 @@
import inspect
import pprint
import sys
+import traceback
import types
from builtins import object
from builtins import zip
@@ -141,11 +142,20 @@
def const_type(self, i):
return Const(self.co.co_consts[i])
+ def get_closure(self, i):
+ num_cellvars = len(self.co.co_cellvars)
+ if i < num_cellvars:
+ return self.vars[i]
+ else:
+ return self.f.__closure__[i - num_cellvars].cell_contents
+
def closure_type(self, i):
- ncellvars = len(self.co.co_cellvars)
- if i < ncellvars:
- return Any
- return Const(self.f.__closure__[i - ncellvars].cell_contents)
+ """Returns a TypeConstraint or Const."""
+ val = self.get_closure(i)
+ if isinstance(val, typehints.TypeConstraint):
+ return val
+ else:
+ return Const(val)
def get_global(self, i):
name = self.get_name(i)
@@ -192,6 +202,20 @@
return typehints.Union[a, b]
+def finalize_hints(type_hint):
+ """Sets type hint for empty data structures to Any."""
+ def visitor(tc, unused_arg):
+ if isinstance(tc, typehints.DictConstraint):
+ empty_union = typehints.Union[()]
+ if tc.key_type == empty_union:
+ tc.key_type = Any
+ if tc.value_type == empty_union:
+ tc.value_type = Any
+
+ if isinstance(type_hint, typehints.TypeConstraint):
+ type_hint.visit(visitor, None)
+
+
def element_type(hint):
"""Returns the element type of a composite type.
"""
@@ -277,6 +301,8 @@
else:
return Any
except TypeInferenceError:
+ if debug:
+ traceback.print_exc()
return Any
except Exception:
if debug:
@@ -305,6 +331,7 @@
if debug:
print()
print(f, id(f), input_types)
+ dis.dis(f)
from . import opcodes
simple_ops = dict((k.upper(), v) for k, v in opcodes.__dict__.items())
@@ -312,7 +339,7 @@
code = co.co_code
end = len(code)
pc = 0
- extended_arg = 0
+ extended_arg = 0 # Python 2 only.
free = None
yields = set()
@@ -324,28 +351,44 @@
states = collections.defaultdict(lambda: None)
jumps = collections.defaultdict(int)
+ # In Python 3, use dis library functions to disassemble bytecode and handle
+ # EXTENDED_ARGs.
+ is_py3 = sys.version_info[0] == 3
+ if is_py3:
+ ofs_table = {} # offset -> instruction
+ for instruction in dis.get_instructions(f):
+ ofs_table[instruction.offset] = instruction
+
+ # Python 2 - 3.5: 1 byte opcode + optional 2 byte arg (1 or 3 bytes).
+ # Python 3.6+: 1 byte opcode + 1 byte arg (2 bytes, arg may be ignored).
+ if sys.version_info >= (3, 6):
+ inst_size = 2
+ opt_arg_size = 0
+ else:
+ inst_size = 1
+ opt_arg_size = 2
+
last_pc = -1
- while pc < end:
+ while pc < end: # pylint: disable=too-many-nested-blocks
start = pc
- if sys.version_info[0] == 2:
- op = ord(code[pc])
+ if is_py3:
+ instruction = ofs_table[pc]
+ op = instruction.opcode
else:
- op = code[pc]
+ op = ord(code[pc])
if debug:
print('-->' if pc == last_pc else ' ', end=' ')
print(repr(pc).rjust(4), end=' ')
print(dis.opname[op].ljust(20), end=' ')
- pc += 1
+ pc += inst_size
if op >= dis.HAVE_ARGUMENT:
- if sys.version_info[0] == 2:
- arg = ord(code[pc]) + ord(code[pc + 1]) * 256 + extended_arg
- elif sys.version_info[0] == 3 and sys.version_info[1] < 6:
- arg = code[pc] + code[pc + 1] * 256 + extended_arg
+ if is_py3:
+ arg = instruction.arg
else:
- pass # TODO(luke-zhu): Python 3.6 bytecode to wordcode changes
+ arg = ord(code[pc]) + ord(code[pc + 1]) * 256 + extended_arg
extended_arg = 0
- pc += 2
+ pc += opt_arg_size
if op == dis.EXTENDED_ARG:
extended_arg = arg * 65536
if debug:
@@ -365,7 +408,7 @@
free = co.co_cellvars + co.co_freevars
print('(' + free[arg] + ')', end=' ')
- # Acutally emulate the op.
+ # Actually emulate the op.
if state is None and states[start] is None:
# No control reaches here (yet).
if debug:
@@ -376,41 +419,80 @@
opname = dis.opname[op]
jmp = jmp_state = None
if opname.startswith('CALL_FUNCTION'):
- # Each keyword takes up two arguments on the stack (name and value).
- standard_args = (arg & 0xFF) + 2 * (arg >> 8)
- var_args = 'VAR' in opname
- kw_args = 'KW' in opname
- pop_count = standard_args + var_args + kw_args + 1
- if depth <= 0:
- return_type = Any
- elif arg >> 8:
- # TODO(robertwb): Handle this case.
- return_type = Any
- elif isinstance(state.stack[-pop_count], Const):
- # TODO(robertwb): Handle this better.
- if var_args or kw_args:
- state.stack[-1] = Any
- state.stack[-var_args - kw_args] = Any
+ if sys.version_info < (3, 6):
+ # Each keyword takes up two arguments on the stack (name and value).
+ standard_args = (arg & 0xFF) + 2 * (arg >> 8)
+ var_args = 'VAR' in opname
+ kw_args = 'KW' in opname
+ pop_count = standard_args + var_args + kw_args + 1
+ if depth <= 0:
+ return_type = Any
+ elif arg >> 8:
+ # TODO(robertwb): Handle this case.
+ return_type = Any
+ elif isinstance(state.stack[-pop_count], Const):
+ # TODO(robertwb): Handle this better.
+ if var_args or kw_args:
+ state.stack[-1] = Any
+ state.stack[-var_args - kw_args] = Any
+ return_type = infer_return_type(state.stack[-pop_count].value,
+ state.stack[1 - pop_count:],
+ debug=debug,
+ depth=depth - 1)
+ else:
+ return_type = Any
+ state.stack[-pop_count:] = [return_type]
+ else: # Python 3.6+
+ if opname == 'CALL_FUNCTION':
+ pop_count = arg + 1
+ if depth <= 0:
+ return_type = Any
+ else:
+ return_type = infer_return_type(state.stack[-pop_count].value,
+ state.stack[1 - pop_count:],
+ debug=debug,
+ depth=depth - 1)
+ elif opname == 'CALL_FUNCTION_KW':
+ # TODO(udim): Handle keyword arguments. Requires passing them by name
+ # to infer_return_type.
+ pop_count = arg + 2
+ return_type = Any
+ elif opname == 'CALL_FUNCTION_EX':
+ # stack[-has_kwargs]: Map of keyword args.
+ # stack[-1 - has_kwargs]: Iterable of positional args.
+ # stack[-2 - has_kwargs]: Function to call.
+ has_kwargs = arg & 1 # type: int
+ pop_count = has_kwargs + 2
+ if has_kwargs:
+ # TODO(udim): Unimplemented. Requires same functionality as a
+ # CALL_FUNCTION_KW implementation.
+ return_type = Any
+ else:
+ args = state.stack[-1]
+ _callable = state.stack[-2]
+ if isinstance(args, typehints.ListConstraint):
+ # Case where there's a single var_arg argument.
+ args = [args]
+ elif isinstance(args, typehints.TupleConstraint):
+ args = list(args._inner_types())
+ return_type = infer_return_type(_callable.value,
+ args,
+ debug=debug,
+ depth=depth - 1)
+ else:
+ raise TypeInferenceError('unable to handle %s' % opname)
+ state.stack[-pop_count:] = [return_type]
+ elif opname == 'CALL_METHOD':
+ pop_count = 1 + arg
+ # LOAD_METHOD will return a non-Const (Any) if loading from an Any.
+ if isinstance(state.stack[-pop_count], Const) and depth > 0:
return_type = infer_return_type(state.stack[-pop_count].value,
state.stack[1 - pop_count:],
debug=debug,
depth=depth - 1)
else:
- return_type = Any
+ return_type = typehints.Any
state.stack[-pop_count:] = [return_type]
- elif (opname == 'BINARY_SUBSCR'
- and isinstance(state.stack[1], Const)
- and isinstance(state.stack[0], typehints.IndexableTypeConstraint)):
- if debug:
- print("Executing special case binary subscript")
- idx = state.stack.pop()
- src = state.stack.pop()
- try:
- state.stack.append(src._constraint_for_index(idx.value))
- except Exception as e:
- if debug:
- print("Exception {0} during special case indexing".format(e))
- state.stack.append(Any)
elif opname in simple_ops:
if debug:
print("Executing simple op " + opname)
@@ -445,7 +527,7 @@
raise TypeInferenceError('unable to handle %s' % opname)
if jmp is not None:
- # TODO(robertwb): Is this guerenteed to converge?
+ # TODO(robertwb): Is this guaranteed to converge?
new_state = states[jmp] | jmp_state
if jmp < pc and new_state != states[jmp] and jumps[pc] < 5:
jumps[pc] += 1
@@ -461,6 +543,7 @@
result = typehints.Iterable[reduce(union, Const.unwrap_all(yields))]
else:
result = reduce(union, Const.unwrap_all(returns))
+ finalize_hints(result)
if debug:
print(f, id(f), input_types, '->', result)
diff --git a/sdks/python/apache_beam/typehints/trivial_inference_test.py b/sdks/python/apache_beam/typehints/trivial_inference_test.py
index b59da72..ff0949b 100644
--- a/sdks/python/apache_beam/typehints/trivial_inference_test.py
+++ b/sdks/python/apache_beam/typehints/trivial_inference_test.py
@@ -19,7 +19,6 @@
from __future__ import absolute_import
-import os
import sys
import unittest
@@ -29,14 +28,12 @@
global_int = 1
-@unittest.skipIf(sys.version_info >= (3, 6, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.6. '
- 'See BEAM-6877')
class TrivialInferenceTest(unittest.TestCase):
- def assertReturnType(self, expected, f, inputs=()):
- self.assertEquals(expected, trivial_inference.infer_return_type(f, inputs))
+ def assertReturnType(self, expected, f, inputs=(), depth=5):
+ self.assertEqual(
+ expected,
+ trivial_inference.infer_return_type(f, inputs, debug=True, depth=depth))
def testIdentity(self):
self.assertReturnType(int, lambda x: x, [int])
@@ -63,8 +60,15 @@
typehints.Tuple[int, str], reverse, [typehints.Tuple[str, float, int]])
self.assertReturnType(
typehints.Tuple[int, int], reverse, [typehints.List[int]])
+
+ def testGetItemSlice(self):
self.assertReturnType(
typehints.List[int], lambda v: v[::-1], [typehints.List[int]])
+ self.assertReturnType(
+ typehints.Tuple[int], lambda v: v[::-1], [typehints.Tuple[int]])
+ self.assertReturnType(str, lambda v: v[::-1], [str])
+ self.assertReturnType(typehints.Any, lambda v: v[::-1], [typehints.Any])
+ self.assertReturnType(typehints.Any, lambda v: v[::-1], [object])
def testUnpack(self):
def reverse(a_b):
@@ -112,10 +116,6 @@
lambda xs: [x for x in xs],
[typehints.Tuple[int, ...]])
- @unittest.skipIf(sys.version_info[0] == 3 and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3. '
- 'See BEAM-6877')
def testTupleListComprehension(self):
self.assertReturnType(
typehints.List[int],
@@ -125,9 +125,13 @@
typehints.List[typehints.Union[int, float]],
lambda xs: [x for x in xs],
[typehints.Tuple[int, float]])
- # TODO(luke-zhu): This test fails in Python 3
+ if sys.version_info[:2] == (3, 5):
+ # A better result requires implementing the MAKE_CLOSURE opcode.
+ expected = typehints.Any
+ else:
+ expected = typehints.List[typehints.Tuple[str, int]]
self.assertReturnType(
- typehints.List[typehints.Tuple[str, int]],
+ expected,
lambda kvs: [(kvs[0], v) for v in kvs[1]],
[typehints.Tuple[str, typehints.Iterable[int]]])
self.assertReturnType(
@@ -206,12 +210,71 @@
typehints.Dict[typehints.Any, typehints.Any], lambda: {})
def testDictComprehension(self):
- # Just ensure it doesn't crash.
fields = []
+ if sys.version_info >= (3, 6):
+ expected_type = typehints.Dict[typehints.Any, typehints.Any]
+ else:
+ # For Python 2, just ensure it doesn't crash.
+ expected_type = typehints.Any
self.assertReturnType(
- typehints.Any,
+ expected_type,
lambda row: {f: row[f] for f in fields}, [typehints.Any])
+ def testDictComprehensionSimple(self):
+ self.assertReturnType(
+ typehints.Dict[str, int],
+ lambda _list: {'a': 1 for _ in _list}, [])
+
+ def testDepthFunction(self):
+ def f(i):
+ return i
+ self.assertReturnType(typehints.Any, lambda i: f(i), [int], depth=0)
+ self.assertReturnType(int, lambda i: f(i), [int], depth=1)
+
+ def testDepthMethod(self):
+ class A(object):
+ def m(self, x):
+ return x
+
+ self.assertReturnType(typehints.Any, lambda: A().m(3), depth=0)
+ self.assertReturnType(int, lambda: A().m(3), depth=1)
+
+ self.assertReturnType(typehints.Any, lambda: A.m(A(), 3.0), depth=0)
+ self.assertReturnType(float, lambda: A.m(A(), 3.0), depth=1)
+
+ def testBuildTupleUnpackWithCall(self):
+ # Lambda uses BUILD_TUPLE_UNPACK_WITH_CALL opcode in Python 3.6, 3.7.
+ def fn(x1, x2, *unused_args):
+ return x1, x2
+
+ self.assertReturnType(typehints.Tuple[str, float],
+ lambda x1, x2, _list: fn(x1, x2, *_list),
+ [str, float, typehints.List[int]])
+ # No *args
+ self.assertReturnType(typehints.Tuple[str, typehints.List[int]],
+ lambda x1, x2, _list: fn(x1, x2, *_list),
+ [str, typehints.List[int]])
+
+ @unittest.skipIf(sys.version_info < (3, 6), 'CALL_FUNCTION_EX is new in 3.6')
+ def testCallFunctionEx(self):
+ # Test when fn arguments are built using BUiLD_LIST.
+ def fn(*args):
+ return args
+
+ self.assertReturnType(typehints.List[typehints.Union[str, float]],
+ lambda x1, x2: fn(*[x1, x2]),
+ [str, float])
+
+ @unittest.skipIf(sys.version_info < (3, 6), 'CALL_FUNCTION_EX is new in 3.6')
+ def testCallFunctionExKwargs(self):
+ def fn(x1, x2, **unused_kwargs):
+ return x1, x2
+
+ # Keyword args are currently unsupported for CALL_FUNCTION_EX.
+ self.assertReturnType(typehints.Any,
+ lambda x1, x2, _dict: fn(x1, x2, **_dict),
+ [str, float, typehints.List[int]])
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/typehints/trivial_inference_test_py3.py b/sdks/python/apache_beam/typehints/trivial_inference_test_py3.py
new file mode 100644
index 0000000..291e52e
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/trivial_inference_test_py3.py
@@ -0,0 +1,50 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Tests for apache_beam.typehints.trivial_inference that use Python 3 syntax.
+"""
+
+from __future__ import absolute_import
+
+import unittest
+
+from apache_beam.typehints import trivial_inference
+from apache_beam.typehints import typehints
+
+
+class TrivialInferenceTest(unittest.TestCase):
+
+ def assertReturnType(self, expected, f, inputs=(), depth=5):
+ self.assertEqual(
+ expected,
+ trivial_inference.infer_return_type(f, inputs, debug=True, depth=depth))
+
+ def testBuildListUnpack(self):
+ # Lambda uses BUILD_LIST_UNPACK opcode in Python 3.
+ self.assertReturnType(typehints.List[int],
+ lambda _list: [*_list, *_list, *_list],
+ [typehints.List[int]])
+
+ def testBuildTupleUnpack(self):
+ # Lambda uses BUILD_TUPLE_UNPACK opcode in Python 3.
+ self.assertReturnType(typehints.Tuple[int, str, str],
+ lambda _list1, _list2: (*_list1, *_list2, *_list2),
+ [typehints.List[int], typehints.List[str]])
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/typehints/typecheck.py b/sdks/python/apache_beam/typehints/typecheck.py
index b69abae..a92aafb 100644
--- a/sdks/python/apache_beam/typehints/typecheck.py
+++ b/sdks/python/apache_beam/typehints/typecheck.py
@@ -71,9 +71,6 @@
def finish_bundle(self, *args, **kwargs):
return self.wrapper(self.dofn.finish_bundle, args, kwargs)
- def is_process_bounded(self):
- return self.dofn.is_process_bounded()
-
class OutputCheckWrapperDoFn(AbstractDoFnWrapper):
"""A DoFn that verifies against common errors in the output type."""
diff --git a/sdks/python/apache_beam/typehints/typed_pipeline_test.py b/sdks/python/apache_beam/typehints/typed_pipeline_test.py
index 1d94b18..f6da7a0 100644
--- a/sdks/python/apache_beam/typehints/typed_pipeline_test.py
+++ b/sdks/python/apache_beam/typehints/typed_pipeline_test.py
@@ -19,7 +19,6 @@
from __future__ import absolute_import
-import os
import sys
import typing
import unittest
@@ -32,12 +31,15 @@
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.typehints import WithTypeHints
-from apache_beam.typehints.decorators import getfullargspec
+from apache_beam.typehints import decorators
+from apache_beam.typehints.decorators import get_signature
# These test often construct a pipeline as value | PTransform to test side
# effects (e.g. errors).
# pylint: disable=expression-not-assigned
+decorators._enable_from_callable = True
+
class MainInputTest(unittest.TestCase):
@@ -48,10 +50,6 @@
with self.assertRaises(typehints.TypeCheckError):
[1, 2, 3] | beam.Map(repeat, 3)
- @unittest.skipIf(sys.version_info >= (3, 7, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.7. '
- 'See BEAM-6988')
def test_non_function(self):
result = ['a', 'bb', 'c'] | beam.Map(str.upper)
self.assertEqual(['A', 'BB', 'C'], sorted(result))
@@ -65,11 +63,16 @@
result = ['1', '10', '100'] | beam.Map(int, 16)
self.assertEqual([1, 16, 256], sorted(result))
+ @unittest.skipIf(
+ sys.version_info.major >= 3 and sys.version_info < (3, 7, 0),
+ 'Function signatures for builtins are not available in Python 3 before '
+ 'version 3.7.')
+ def test_non_function_fails(self):
with self.assertRaises(typehints.TypeCheckError):
[1, 2, 3] | beam.Map(str.upper)
def test_loose_bounds(self):
- @typehints.with_input_types(typehints.Union[int, float])
+ @typehints.with_input_types(typing.Union[int, float])
@typehints.with_output_types(str)
def format_number(x):
return '%g' % x
@@ -86,12 +89,26 @@
result = [1, 2, 3] | beam.ParDo(MyDoFn())
self.assertEqual(['1', '2', '3'], sorted(result))
- with self.assertRaises(typehints.TypeCheckError):
+ with self.assertRaisesRegexp(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
['a', 'b', 'c'] | beam.ParDo(MyDoFn())
- with self.assertRaises(typehints.TypeCheckError):
+ with self.assertRaisesRegexp(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
[1, 2, 3] | (beam.ParDo(MyDoFn()) | 'again' >> beam.ParDo(MyDoFn()))
+ @unittest.skip('BEAM-7981: Iterable in output type should not be removed.')
+ def test_typed_callable_iterable_output(self):
+ # TODO(BEAM-7981): 2.7 and 3.x both erroneously strip the Iterable, but the
+ # test only fails in 3.x.
+ @typehints.with_input_types(int)
+ @typehints.with_output_types(typehints.Iterable[str])
+ def do_fn(element):
+ return [[str(element)] * 2]
+
+ result = [1, 2] | beam.ParDo(do_fn)
+ self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
+
def test_typed_dofn_instance(self):
class MyDoFn(beam.DoFn):
def process(self, element):
@@ -112,13 +129,9 @@
def filter_fn(data):
return data % 2
- self.assertEquals([1, 3], [1, 2, 3] | beam.Filter(filter_fn))
+ self.assertEqual([1, 3], [1, 2, 3] | beam.Filter(filter_fn))
-@unittest.skipIf(sys.version_info >= (3, 7, 0) and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3.7. '
- 'See BEAM-6987')
class NativeTypesTest(unittest.TestCase):
def test_good_main_input(self):
@@ -171,8 +184,10 @@
['a', 'bb', 'c'] | beam.Map(repeat, times='z')
with self.assertRaises(typehints.TypeCheckError):
['a', 'bb', 'c'] | beam.Map(repeat, 3, 4)
- if not getfullargspec(repeat).defaults:
- with self.assertRaises(typehints.TypeCheckError):
+ if all(param.default == param.empty
+ for param in get_signature(repeat).parameters.values()):
+ with self.assertRaisesRegexp(typehints.TypeCheckError,
+ r'(takes exactly|missing a required)'):
['a', 'bb', 'c'] | beam.Map(repeat)
def test_basic_side_input_hint(self):
@@ -197,7 +212,7 @@
@typehints.with_input_types(str)
def repeat(s, times=3):
return s * times
- # No type checking on dfault arg.
+ # No type checking on default arg.
self._run_repeat_test_good(repeat)
@OptionsContext(pipeline_type_check=True)
@@ -209,9 +224,52 @@
result = ['a', 'bb', 'c'] | beam.Map(repeat, 3)
self.assertEqual(['aaa', 'bbbbbb', 'ccc'], sorted(result))
- # TODO(robertwb): Support partially defined varargs.
- # with self.assertRaises(typehints.TypeCheckError):
- # ['a', 'bb', 'c'] | beam.Map(repeat, 'z')
+ if sys.version_info >= (3,):
+ with self.assertRaisesRegexp(
+ typehints.TypeCheckError,
+ r'requires Tuple\[int, ...\] but got Tuple\[str, ...\]'):
+ ['a', 'bb', 'c'] | beam.Map(repeat, 'z')
+
+ def test_var_positional_only_side_input_hint(self):
+ # Test that a lambda that accepts only a VAR_POSITIONAL can accept
+ # side-inputs.
+ # TODO(BEAM-8247): There's a bug with trivial_inference inferring the output
+ # type when side-inputs are used (their type hints are not passed). Remove
+ # with_output_types(...) when this bug is fixed.
+ result = (['a', 'b', 'c']
+ | beam.Map(lambda *args: args, 5).with_input_types(int, str)
+ .with_output_types(typehints.Tuple[str, int]))
+ self.assertEqual([('a', 5), ('b', 5), ('c', 5)], sorted(result))
+
+ # Type hint order doesn't matter for VAR_POSITIONAL.
+ result = (['a', 'b', 'c']
+ | beam.Map(lambda *args: args, 5).with_input_types(int, str)
+ .with_output_types(typehints.Tuple[str, int]))
+ self.assertEqual([('a', 5), ('b', 5), ('c', 5)], sorted(result))
+
+ if sys.version_info >= (3,):
+ with self.assertRaisesRegexp(
+ typehints.TypeCheckError,
+ r'requires Tuple\[Union\[int, str\], ...\] but got '
+ r'Tuple\[Union\[float, int\], ...\]'):
+ _ = [1.2] | beam.Map(lambda *_: 'a', 5).with_input_types(int, str)
+
+ def test_var_keyword_side_input_hint(self):
+ # Test that a lambda that accepts a VAR_KEYWORD can accept
+ # side-inputs.
+ result = (['a', 'b', 'c']
+ | beam.Map(lambda e, **kwargs: (e, kwargs), kw=5)
+ .with_input_types(str, ignored=int))
+ self.assertEqual([('a', {'kw': 5}), ('b', {'kw': 5}), ('c', {'kw': 5})],
+ sorted(result))
+
+ if sys.version_info >= (3,):
+ with self.assertRaisesRegexp(
+ typehints.TypeCheckError,
+ r'requires Dict\[str, str\] but got Dict\[str, int\]'):
+ _ = (['a', 'b', 'c']
+ | beam.Map(lambda e, **_: 'a', kw=5)
+ .with_input_types(str, ignored=str))
def test_deferred_side_inputs(self):
@typehints.with_input_types(str, int)
@@ -228,7 +286,7 @@
main_input | 'bis' >> beam.Map(repeat, pvalue.AsSingleton(bad_side_input))
def test_deferred_side_input_iterable(self):
- @typehints.with_input_types(str, typehints.Iterable[str])
+ @typehints.with_input_types(str, typing.Iterable[str])
def concat(glue, items):
return glue.join(sorted(items))
with TestPipeline() as p:
@@ -294,5 +352,35 @@
self.test_input | self.CustomTransform().with_output_types(int)
+class AnnotationsTest(unittest.TestCase):
+
+ def test_pardo_wrapper_builtin_method(self):
+ th = beam.ParDo(str.strip).get_type_hints()
+ if sys.version_info < (3, 7):
+ self.assertEqual(th.input_types, ((str,), {}))
+ else:
+ # Python 3.7+ has annotations for CPython builtins
+ # (_MethodDescriptorType).
+ self.assertEqual(th.input_types, ((str, typehints.Any), {}))
+ self.assertEqual(th.output_types, ((typehints.Any,), {}))
+
+ def test_pardo_wrapper_builtin_type(self):
+ th = beam.ParDo(list).get_type_hints()
+ if sys.version_info < (3, 7):
+ self.assertEqual(th.input_types, (
+ (typehints.Any, typehints.decorators._ANY_VAR_POSITIONAL),
+ {'__unknown__keywords': typehints.decorators._ANY_VAR_KEYWORD}))
+ else:
+ # Python 3.7+ supports signatures for builtins like 'list'.
+ self.assertEqual(th.input_types, ((typehints.Any, ), {}))
+
+ self.assertEqual(th.output_types, ((typehints.Any,), {}))
+
+ def test_pardo_wrapper_builtin_func(self):
+ th = beam.ParDo(len).get_type_hints()
+ self.assertIsNone(th.input_types)
+ self.assertIsNone(th.output_types)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py b/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py
new file mode 100644
index 0000000..1c0c66c
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py
@@ -0,0 +1,226 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for type-hint objects and decorators - Python 3 syntax specific.
+"""
+
+from __future__ import absolute_import
+
+import unittest
+
+import apache_beam as beam
+from apache_beam import typehints
+from apache_beam.typehints import decorators
+
+decorators._enable_from_callable = True
+
+
+class MainInputTest(unittest.TestCase):
+
+ def test_typed_dofn_method(self):
+ # process annotations are recognized and take precedence over decorators.
+ @typehints.with_input_types(typehints.Tuple[int, int])
+ @typehints.with_output_types(int)
+ class MyDoFn(beam.DoFn):
+ def process(self, element: int) -> typehints.Tuple[str]:
+ return tuple(str(element))
+
+ result = [1, 2, 3] | beam.ParDo(MyDoFn())
+ self.assertEqual(['1', '2', '3'], sorted(result))
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
+ ['a', 'b', 'c'] | beam.ParDo(MyDoFn())
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
+ [1, 2, 3] | (beam.ParDo(MyDoFn()) | 'again' >> beam.ParDo(MyDoFn()))
+
+ def test_typed_dofn_instance(self):
+ # Type hints applied to DoFn instance take precedence over decorators and
+ # process annotations.
+ @typehints.with_input_types(typehints.Tuple[int, int])
+ @typehints.with_output_types(int)
+ class MyDoFn(beam.DoFn):
+ def process(self, element: typehints.Tuple[int, int]) -> \
+ typehints.List[int]:
+ return [str(element)]
+ my_do_fn = MyDoFn().with_input_types(int).with_output_types(str)
+
+ result = [1, 2, 3] | beam.ParDo(my_do_fn)
+ self.assertEqual(['1', '2', '3'], sorted(result))
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
+ ['a', 'b', 'c'] | beam.ParDo(my_do_fn)
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
+ [1, 2, 3] | (beam.ParDo(my_do_fn) | 'again' >> beam.ParDo(my_do_fn))
+
+ def test_typed_callable_instance(self):
+ # Type hints applied to ParDo instance take precedence over callable
+ # decorators and annotations.
+ @typehints.with_input_types(typehints.Tuple[int, int])
+ @typehints.with_output_types(int)
+ def do_fn(element: typehints.Tuple[int, int]) -> typehints.Generator[int]:
+ yield str(element)
+ pardo = beam.ParDo(do_fn).with_input_types(int).with_output_types(str)
+
+ result = [1, 2, 3] | pardo
+ self.assertEqual(['1', '2', '3'], sorted(result))
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
+ ['a', 'b', 'c'] | pardo
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*int.*got.*str'):
+ [1, 2, 3] | (pardo | 'again' >> pardo)
+
+ @unittest.skip('BEAM-7981: Iterable in output type should not be removed.')
+ def test_typed_callable_iterable_output(self):
+ # TODO(BEAM-7981): Both Iterables get stripped in
+ # CallableWrapperDoFn.default_type_hints, but only one should.
+ def do_fn(element: int) -> typehints.Iterable[typehints.Iterable[str]]:
+ return [[str(element)] * 2]
+
+ result = [1, 2] | beam.ParDo(do_fn)
+ self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
+
+ def test_typed_dofn_method_not_iterable(self):
+ class MyDoFn(beam.DoFn):
+ def process(self, element: int) -> str:
+ return str(element)
+
+ with self.assertRaisesRegex(ValueError, r'str.*is not iterable'):
+ [1, 2, 3] | beam.ParDo(MyDoFn())
+
+ def test_typed_callable_not_iterable(self):
+ def do_fn(element: typehints.Tuple[int, int]) -> int:
+ return element[0]
+ with self.assertRaisesRegex(ValueError, r'int.*is not iterable'):
+ [1, 2, 3] | beam.ParDo(do_fn)
+
+ def test_typed_dofn_kwonly(self):
+ class MyDoFn(beam.DoFn):
+ # TODO(BEAM-5878): A kwonly argument like
+ # timestamp=beam.DoFn.TimestampParam would not work here.
+ def process(self, element: int, *, side_input: str) -> \
+ typehints.Generator[typehints.Optional[int]]:
+ yield str(element) if side_input else None
+ my_do_fn = MyDoFn()
+
+ result = [1, 2, 3] | beam.ParDo(my_do_fn, side_input='abc')
+ self.assertEqual(['1', '2', '3'], sorted(result))
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*str.*got.*int.*side_input'):
+ [1, 2, 3] | beam.ParDo(my_do_fn, side_input=1)
+
+ def test_type_dofn_var_kwargs(self):
+ class MyDoFn(beam.DoFn):
+ def process(self, element: int, **side_inputs: typehints.Dict[str, str]) \
+ -> typehints.Generator[typehints.Optional[int]]:
+ yield str(element) if side_inputs else None
+ my_do_fn = MyDoFn()
+
+ result = [1, 2, 3] | beam.ParDo(my_do_fn, foo='abc', bar='def')
+ self.assertEqual(['1', '2', '3'], sorted(result))
+
+ with self.assertRaisesRegex(typehints.TypeCheckError,
+ r'requires.*str.*got.*int.*side_inputs'):
+ [1, 2, 3] | beam.ParDo(my_do_fn, a=1)
+
+
+class AnnotationsTest(unittest.TestCase):
+
+ def test_pardo_dofn(self):
+ class MyDoFn(beam.DoFn):
+ def process(self, element: int) -> typehints.Generator[str]:
+ yield str(element)
+
+ th = beam.ParDo(MyDoFn()).get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((str,), {}))
+
+ def test_pardo_dofn_not_iterable(self):
+ class MyDoFn(beam.DoFn):
+ def process(self, element: int) -> str:
+ return str(element)
+
+ with self.assertRaisesRegexp(ValueError, r'Return value not iterable'):
+ _ = beam.ParDo(MyDoFn()).get_type_hints()
+
+ def test_pardo_wrapper(self):
+ def do_fn(element: int) -> typehints.Iterable[str]:
+ return [str(element)]
+
+ th = beam.ParDo(do_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((str,), {}))
+
+ def test_pardo_wrapper_not_iterable(self):
+ def do_fn(element: int) -> str:
+ return str(element)
+
+ with self.assertRaisesRegexp(ValueError, r'Return value not iterable'):
+ _ = beam.ParDo(do_fn).get_type_hints()
+
+ def test_flat_map_wrapper(self):
+ def map_fn(element: int) -> typehints.Iterable[int]:
+ return [element, element + 1]
+
+ th = beam.FlatMap(map_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((int,), {}))
+
+ def test_flat_map_tuple_wrapper(self):
+ def tuple_map_fn(a: str, b: str, c: str) -> typehints.Iterable[str]:
+ return [a, b, c]
+
+ th = beam.FlatMapTuple(tuple_map_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((str, str, str), {}))
+ self.assertEqual(th.output_types, ((str,), {}))
+
+ def test_map_wrapper(self):
+ def map_fn(unused_element: int) -> int:
+ return 1
+
+ th = beam.Map(map_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((int,), {}))
+
+ def test_map_tuple(self):
+ def tuple_map_fn(a: str, b: str, c: str) -> str:
+ return a + b + c
+
+ th = beam.MapTuple(tuple_map_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((str, str, str), {}))
+ self.assertEqual(th.output_types, ((str,), {}))
+
+ def test_filter_wrapper(self):
+ def filter_fn(element: int) -> bool:
+ return bool(element % 2)
+
+ th = beam.Filter(filter_fn).get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((bool,), {}))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/typehints/typehints.py b/sdks/python/apache_beam/typehints/typehints.py
index 4510d4a..4a9c739 100644
--- a/sdks/python/apache_beam/typehints/typehints.py
+++ b/sdks/python/apache_beam/typehints/typehints.py
@@ -67,6 +67,7 @@
import collections
import copy
+import logging
import sys
import types
from builtins import next
@@ -525,6 +526,7 @@
# Flatten nested Union's and duplicated repeated type hints.
params = set()
+ dict_union = None
for t in type_params:
validate_composite_type_param(
t, error_msg_prefix='All parameters to a Union hint'
@@ -532,9 +534,18 @@
if isinstance(t, self.UnionConstraint):
params |= t.union_types
+ elif isinstance(t, DictConstraint):
+ if dict_union is None:
+ dict_union = t
+ else:
+ dict_union.key_type = Union[dict_union.key_type, t.key_type]
+ dict_union.value_type = Union[dict_union.value_type, t.value_type]
else:
params.add(t)
+ if dict_union is not None:
+ params.add(dict_union)
+
if Any in params:
return Any
elif len(params) == 1:
@@ -797,7 +808,7 @@
def _consistent_with_check_(self, sub):
return (isinstance(sub, self.__class__)
and is_consistent_with(sub.key_type, self.key_type)
- and is_consistent_with(sub.key_type, self.key_type))
+ and is_consistent_with(sub.value_type, self.value_type))
def _raise_hint_exception_or_inner_exception(self, is_key,
incorrect_instance,
@@ -976,6 +987,13 @@
def __repr__(self):
return 'Iterator[%s]' % _unified_repr(self.yielded_type)
+ def __eq__(self, other):
+ return (type(self) == type(other)
+ and self.yielded_type == other.yielded_type)
+
+ def __hash__(self):
+ return hash(self.yielded_type)
+
def _inner_types(self):
yield self.yielded_type
@@ -1058,7 +1076,22 @@
class GeneratorHint(IteratorHint):
- pass
+ """A Generator type hint.
+
+ Subscriptor is in the form [yield_type, send_type, return_type], however
+ only yield_type is supported. The 2 others are expected to be None.
+ """
+
+ def __getitem__(self, type_params):
+ if isinstance(type_params, tuple) and len(type_params) == 3:
+ yield_type, send_type, return_type = type_params
+ if send_type is not None:
+ logging.warning('Ignoring send_type hint: %s' % send_type)
+ if send_type is not None:
+ logging.warning('Ignoring return_type hint: %s' % return_type)
+ else:
+ yield_type = type_params
+ return self.IteratorTypeConstraint(yield_type)
# Create the actual instances for all defined type-hints above.
@@ -1110,9 +1143,9 @@
def is_consistent_with(sub, base):
- """Returns whether the type a is consistent with b.
+ """Checks whether sub a is consistent with base.
- This is accordig to the terminology of PEP 483/484. This relationship is
+ This is according to the terminology of PEP 483/484. This relationship is
neither symmetric nor transitive, but a good mnemonic to keep in mind is that
is_consistent_with(a, b) is roughly equivalent to the issubclass(a, b)
relation, but also handles the special Any type as well as type
@@ -1135,11 +1168,42 @@
return issubclass(sub, base)
-def coerce_to_kv_type(element_type, label=None):
+def get_yielded_type(type_hint):
+ """Obtains the type of elements yielded by an iterable.
+
+ Note that "iterable" here means: can be iterated over in a for loop.
+
+ Args:
+ type_hint: (TypeConstraint) The iterable in question. Must be normalize()-d.
+
+ Returns:
+ Yielded type of the iterable.
+
+ Raises:
+ ValueError if not iterable.
+ """
+ if isinstance(type_hint, AnyTypeConstraint):
+ return type_hint
+ if is_consistent_with(type_hint, Iterator[Any]):
+ return type_hint.yielded_type
+ if is_consistent_with(type_hint, Tuple[Any, ...]):
+ return Union[type_hint.tuple_types]
+ if is_consistent_with(type_hint, Iterable[Any]):
+ return type_hint.inner_type
+ raise ValueError('%s is not iterable' % type_hint)
+
+
+def coerce_to_kv_type(element_type, label=None, side_input_producer=None):
"""Attempts to coerce element_type to a compatible kv type.
Raises an error on failure.
"""
+ if side_input_producer:
+ consumer = 'side-input of %r (producer: %r)' % (label,
+ side_input_producer)
+ else:
+ consumer = '%r' % label
+
# If element_type is not specified, then treat it as `Any`.
if not element_type:
return KV[Any, Any]
@@ -1148,8 +1212,8 @@
return element_type
else:
raise ValueError(
- "Tuple input to %r must be have two components. "
- "Found %s." % (label, element_type))
+ "Tuple input to %s must have two components. "
+ "Found %s." % (consumer, element_type))
elif isinstance(element_type, AnyTypeConstraint):
# `Any` type needs to be replaced with a KV[Any, Any] to
# satisfy the KV form.
@@ -1163,5 +1227,5 @@
else:
# TODO: Possibly handle other valid types.
raise ValueError(
- "Input to %r must be compatible with KV[Any, Any]. "
- "Found %s." % (label, element_type))
+ "Input to %s must be compatible with KV[Any, Any]. "
+ "Found %s." % (consumer, element_type))
diff --git a/sdks/python/apache_beam/typehints/typehints_test.py b/sdks/python/apache_beam/typehints/typehints_test.py
index f002cd0..0f1fe61 100644
--- a/sdks/python/apache_beam/typehints/typehints_test.py
+++ b/sdks/python/apache_beam/typehints/typehints_test.py
@@ -20,25 +20,27 @@
from __future__ import absolute_import
import functools
-import inspect
+import sys
import unittest
from builtins import next
from builtins import range
import apache_beam.typehints.typehints as typehints
from apache_beam.typehints import Any
+from apache_beam.typehints import Dict
from apache_beam.typehints import Tuple
from apache_beam.typehints import TypeCheckError
from apache_beam.typehints import Union
+from apache_beam.typehints import native_type_compatibility
from apache_beam.typehints import with_input_types
from apache_beam.typehints import with_output_types
from apache_beam.typehints.decorators import GeneratorWrapper
from apache_beam.typehints.decorators import _check_instance_type
from apache_beam.typehints.decorators import _interleave_type_check
from apache_beam.typehints.decorators import _positional_arg_hints
+from apache_beam.typehints.decorators import get_signature
from apache_beam.typehints.decorators import get_type_hints
from apache_beam.typehints.decorators import getcallargs_forhints
-from apache_beam.typehints.decorators import getfullargspec
from apache_beam.typehints.typehints import is_consistent_with
@@ -58,7 +60,7 @@
if hints.input_types: # pylint: disable=too-many-nested-blocks
input_hints = getcallargs_forhints(
f, *hints.input_types[0], **hints.input_types[1])
- inputs = inspect.getcallargs(f, *args, **kwargs)
+ inputs = get_signature(f).bind(*args, **kwargs).arguments
for var, hint in input_hints.items():
value = inputs[var]
new_value = check_or_interleave(hint, value, var)
@@ -67,7 +69,7 @@
kwargs[var] = new_value
else:
args = list(args)
- for ix, pvar in enumerate(getfullargspec(f).args):
+ for ix, pvar in enumerate(get_signature(f).parameters):
if pvar == var:
args[ix] = new_value
break
@@ -97,11 +99,13 @@
class TypeHintTestCase(unittest.TestCase):
def assertCompatible(self, base, sub): # pylint: disable=invalid-name
+ base, sub = native_type_compatibility.convert_to_beam_types([base, sub])
self.assertTrue(
is_consistent_with(sub, base),
'%s is not consistent with %s' % (sub, base))
def assertNotCompatible(self, base, sub): # pylint: disable=invalid-name
+ base, sub = native_type_compatibility.convert_to_beam_type([base, sub])
self.assertFalse(
is_consistent_with(sub, base),
'%s is consistent with %s' % (sub, base))
@@ -227,6 +231,17 @@
"instead.",
e.exception.args[0])
+ def test_dict_union(self):
+ hint = Union[typehints.Dict[Any, int],
+ typehints.Dict[Union[()], Union[()]]]
+ self.assertEqual(typehints.Dict[Any, int], hint)
+
+ def test_empty_union(self):
+ self.assertEqual(typehints.Union[()],
+ typehints.Union[typehints.Union[()], typehints.Union[()]])
+ self.assertEqual(int,
+ typehints.Union[typehints.Union[()], int])
+
class OptionalHintTestCase(TypeHintTestCase):
@@ -522,11 +537,13 @@
hint1 = typehints.Dict[int, str]
hint2 = typehints.Dict[bool, int]
hint3 = typehints.Dict[int, typehints.List[typehints.Tuple[str, str, str]]]
+ hint4 = typehints.Dict[int, int]
self.assertCompatible(hint1, hint1)
self.assertCompatible(hint3, hint3)
self.assertNotCompatible(hint3, 4)
- self.assertNotCompatible(hint2, hint1)
+ self.assertNotCompatible(hint2, hint1) # Key incompatibility.
+ self.assertNotCompatible(hint1, hint4) # Value incompatibility.
def test_repr(self):
hint3 = typehints.Dict[int, typehints.List[typehints.Tuple[str, str, str]]]
@@ -738,6 +755,11 @@
self.assertCompatible(typehints.Iterator[int], typehints.Iterator[int])
self.assertNotCompatible(typehints.Iterator[str], typehints.Iterator[float])
+ def test_conversion(self):
+ self.assertCompatible(typehints.Iterator[int], typehints.Generator[int])
+ self.assertCompatible(typehints.Iterator[int],
+ typehints.Generator[int, None, None])
+
def test_generator_return_hint_invalid_yield_type(self):
@check_type_hints
@with_output_types(typehints.Iterator[int])
@@ -878,10 +900,50 @@
self.assertEqual(3, add(1, 2))
-class ReturnsDecoratorTestCase(TypeHintTestCase):
+class InputDecoratorTestCase(TypeHintTestCase):
+ def test_valid_hint(self):
+ @with_input_types(int, int)
+ def unused_add(a, b):
+ return a + b
+
+ @with_input_types(int, b=int)
+ def unused_add2(a, b):
+ return a + b
+
+ @with_input_types(a=int, b=int)
+ def unused_add3(a, b):
+ return a + b
+
+ def test_invalid_kw_hint(self):
+ with self.assertRaisesRegexp(TypeError, r'\[1, 2\]'):
+ @with_input_types(a=[1, 2])
+ def unused_foo(a):
+ pass
+
+ def test_invalid_pos_hint(self):
+ with self.assertRaisesRegexp(TypeError, r'\[1, 2\]'):
+ @with_input_types([1, 2])
+ def unused_foo(a):
+ pass
+
+
+class OutputDecoratorTestCase(TypeHintTestCase):
+
+ def test_valid_hint(self):
+ @with_output_types(int)
+ def unused_foo():
+ return 5
+
+ @with_output_types(None)
+ def unused_foo():
+ return 5
+
+ @with_output_types(Tuple[int, str])
+ def unused_foo():
+ return 5, 'bar'
def test_no_kwargs_accepted(self):
- with self.assertRaises(ValueError):
+ with self.assertRaisesRegexp(ValueError, r'must be positional'):
@with_output_types(m=int)
def unused_foo():
return 5
@@ -918,12 +980,14 @@
e.exception.args[0])
def test_type_check_simple_type(self):
+ @check_type_hints
@with_output_types(str)
def upper(a):
return a.upper()
self.assertEqual('TEST', upper('test'))
def test_type_check_composite_type(self):
+ @check_type_hints
@with_output_types(typehints.List[typehints.Tuple[int, int]])
def bar():
return [(i, i+1) for i in range(5)]
@@ -931,6 +995,7 @@
self.assertEqual([(0, 1), (1, 2), (2, 3), (3, 4), (4, 5)], bar())
def test_any_return_type_hint(self):
+ @check_type_hints
@with_output_types(typehints.Any)
def bar():
return 'foo'
@@ -1048,24 +1113,130 @@
self.assertFalse(is_consistent_with(Union[str, int], str))
def test_positional_arg_hints(self):
- self.assertEquals(typehints.Any, _positional_arg_hints('x', {}))
- self.assertEquals(int, _positional_arg_hints('x', {'x': int}))
- self.assertEquals(typehints.Tuple[int, typehints.Any],
- _positional_arg_hints(['x', 'y'], {'x': int}))
+ self.assertEqual(typehints.Any, _positional_arg_hints('x', {}))
+ self.assertEqual(int, _positional_arg_hints('x', {'x': int}))
+ self.assertEqual(typehints.Tuple[int, typehints.Any],
+ _positional_arg_hints(['x', 'y'], {'x': int}))
+
+ @staticmethod
+ def relax_for_py2(tuple_hint):
+ if sys.version_info >= (3,):
+ return tuple_hint
+ else:
+ return Tuple[Any, ...]
def test_getcallargs_forhints(self):
def func(a, b_c, *d):
- b, c = b_c # pylint: disable=unused-variable
- return None
- self.assertEquals(
+ return a, b_c, d
+
+ self.assertEqual(
{'a': Any, 'b_c': Any, 'd': Tuple[Any, ...]},
getcallargs_forhints(func, *[Any, Any]))
- self.assertEquals(
- {'a': Any, 'b_c': Any, 'd': Tuple[Any, ...]},
- getcallargs_forhints(func, *[Any, Any, Any, int]))
- self.assertEquals(
+ self.assertEqual(
+ {'a': Any, 'b_c': Any,
+ 'd': self.relax_for_py2(Tuple[Union[int, str], ...])},
+ getcallargs_forhints(func, *[Any, Any, str, int]))
+ self.assertEqual(
{'a': int, 'b_c': Tuple[str, Any], 'd': Tuple[Any, ...]},
getcallargs_forhints(func, *[int, Tuple[str, Any]]))
+ self.assertEqual(
+ {'a': Any, 'b_c': Any, 'd': self.relax_for_py2(Tuple[str, ...])},
+ getcallargs_forhints(func, *[Any, Any, Tuple[str, ...]]))
+ self.assertEqual(
+ {'a': Any, 'b_c': Any,
+ 'd': self.relax_for_py2(Tuple[Union[Tuple[str, ...], int], ...])},
+ getcallargs_forhints(func, *[Any, Any, Tuple[str, ...], int]))
+
+ @unittest.skipIf(sys.version_info < (3,),
+ 'kwargs not supported in Py2 version of this function')
+ def test_getcallargs_forhints_varkw(self):
+ def func(a, b_c, *d, **e):
+ return a, b_c, d, e
+
+ self.assertEqual(
+ {'a': Any, 'b_c': Any, 'd': Tuple[Any, ...],
+ 'e': Dict[str, Union[str, int]]},
+ getcallargs_forhints(func, *[Any, Any], **{'kw1': str, 'kw2': int}))
+ self.assertEqual(
+ {'a': Any, 'b_c': Any, 'd': Tuple[Any, ...],
+ 'e': Dict[str, Union[str, int]]},
+ getcallargs_forhints(func, *[Any, Any], e=Dict[str, Union[int, str]]))
+ self.assertEqual(
+ {'a': Any, 'b_c': Any, 'd': Tuple[Any, ...],
+ 'e': Dict[str, Dict[str, Union[str, int]]]},
+ # keyword is not 'e', thus the Dict is considered a value hint.
+ getcallargs_forhints(func, *[Any, Any], kw1=Dict[str, Union[int, str]]))
+
+ def test_getcallargs_forhints_builtins(self):
+ if sys.version_info < (3, 7):
+ # Signatures for builtins are not supported in 3.5 and 3.6.
+ self.assertEqual(
+ {'_': str,
+ '__unknown__varargs': Tuple[Any, ...],
+ '__unknown__keywords': typehints.Dict[Any, Any]},
+ getcallargs_forhints(str.upper, str))
+ self.assertEqual(
+ {'_': str,
+ '__unknown__varargs': self.relax_for_py2(Tuple[str, ...]),
+ '__unknown__keywords': typehints.Dict[Any, Any]},
+ getcallargs_forhints(str.strip, str, str))
+ self.assertEqual(
+ {'_': str,
+ '__unknown__varargs':
+ self.relax_for_py2(Tuple[typehints.List[int], ...]),
+ '__unknown__keywords': typehints.Dict[Any, Any]},
+ getcallargs_forhints(str.join, str, typehints.List[int]))
+ else:
+ self.assertEqual(
+ {'self': str},
+ getcallargs_forhints(str.upper, str))
+ # str.strip has an optional second argument.
+ self.assertEqual({'self': str, 'chars': Any},
+ getcallargs_forhints(str.strip, str))
+ self.assertEqual(
+ {'self': str, 'iterable': typehints.List[int]},
+ getcallargs_forhints(str.join, str, typehints.List[int]))
+
+
+class TestGetYieldedType(unittest.TestCase):
+ def test_iterables(self):
+ self.assertEqual(int, typehints.get_yielded_type(typehints.Iterable[int]))
+ self.assertEqual(int, typehints.get_yielded_type(typehints.Iterator[int]))
+ self.assertEqual(int, typehints.get_yielded_type(typehints.Generator[int]))
+ self.assertEqual(int, typehints.get_yielded_type(typehints.List[int]))
+ self.assertEqual(typehints.Union[int, str],
+ typehints.get_yielded_type(typehints.Tuple[int, str]))
+ self.assertEqual(int, typehints.get_yielded_type(typehints.Set[int]))
+
+ def test_not_iterable(self):
+ with self.assertRaisesRegexp(ValueError, r'not iterable'):
+ typehints.get_yielded_type(int)
+
+
+class TestCoerceToKvType(TypeHintTestCase):
+ def test_coercion_success(self):
+ cases = [
+ ((Any, ), typehints.KV[Any, Any]),
+ ((typehints.KV[Any, Any],), typehints.KV[Any, Any]),
+ ((typehints.Tuple[str, int],), typehints.KV[str, int]),
+ ]
+ for args, expected in cases:
+ self.assertEqual(typehints.coerce_to_kv_type(*args), expected)
+ self.assertCompatible(args[0], expected)
+
+ def test_coercion_fail(self):
+ cases = [
+ ((str, 'label', 'producer'), r'producer.*compatible'),
+ ((Tuple[str],), r'two components'),
+ # It seems that the only Unions that may be successfully coerced are not
+ # Unions but Any (e.g. Union[Any, Tuple[Any, Any]] is Any).
+ ((Union[str, int],), r'compatible'),
+ ((Union,), r'compatible'),
+ ((typehints.List[Any],), r'compatible'),
+ ]
+ for args, regex in cases:
+ with self.assertRaisesRegexp(ValueError, regex):
+ typehints.coerce_to_kv_type(*args)
if __name__ == '__main__':
diff --git a/sdks/python/apache_beam/typehints/typehints_test_py3.py b/sdks/python/apache_beam/typehints/typehints_test_py3.py
new file mode 100644
index 0000000..01df57c
--- /dev/null
+++ b/sdks/python/apache_beam/typehints/typehints_test_py3.py
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for the type-hint objects and decorators with Python 3 syntax not
+supported by 2.7."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import unittest
+
+from apache_beam.transforms.core import DoFn
+from apache_beam.typehints import KV
+from apache_beam.typehints import Iterable
+from apache_beam.typehints import decorators
+
+decorators._enable_from_callable = True
+
+
+class TestParDoAnnotations(unittest.TestCase):
+ def test_with_side_input(self):
+ class MyDoFn(DoFn):
+ def process(self, element: float, side_input: str) -> \
+ Iterable[KV[str, float]]:
+ pass
+ th = MyDoFn().get_type_hints()
+ self.assertEqual(th.input_types, ((float, str), {}))
+ self.assertEqual(th.output_types, ((KV[str, float],), {}))
+
+ def test_pep484_annotations(self):
+ class MyDoFn(DoFn):
+ def process(self, element: int) -> Iterable[str]:
+ pass
+
+ print(MyDoFn().get_type_hints())
+ th = MyDoFn().get_type_hints()
+ self.assertEqual(th.input_types, ((int,), {}))
+ self.assertEqual(th.output_types, ((str,), {}))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/sdks/python/apache_beam/utils/retry.py b/sdks/python/apache_beam/utils/retry.py
index 6a57762..59d8dec 100644
--- a/sdks/python/apache_beam/utils/retry.py
+++ b/sdks/python/apache_beam/utils/retry.py
@@ -104,6 +104,15 @@
return not isinstance(exception, PermanentException)
+# TODO(BEAM-6202): Dataflow returns 404 for job ids that actually exist.
+# Retry on those errors.
+def retry_on_server_errors_and_notfound_filter(exception):
+ if HttpError is not None and isinstance(exception, HttpError):
+ if exception.status_code == 404: # 404 Not Found
+ return True
+ return retry_on_server_errors_filter(exception)
+
+
def retry_on_server_errors_and_timeout_filter(exception):
if HttpError is not None and isinstance(exception, HttpError):
if exception.status_code == 408: # 408 Request Timeout
diff --git a/sdks/python/apache_beam/utils/subprocess_server.py b/sdks/python/apache_beam/utils/subprocess_server.py
new file mode 100644
index 0000000..737e220
--- /dev/null
+++ b/sdks/python/apache_beam/utils/subprocess_server.py
@@ -0,0 +1,227 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import logging
+import os
+import shutil
+import signal
+import socket
+import subprocess
+import tempfile
+import threading
+import time
+
+import grpc
+from future.moves.urllib.error import URLError
+from future.moves.urllib.request import urlopen
+
+from apache_beam.version import __version__ as beam_version
+
+
+class SubprocessServer(object):
+ """An abstract base class for running GRPC Servers as an external process.
+
+ This class acts as a context which will start up a server, provides a stub
+ to connect to it, and then shuts the server down. For example::
+
+ with SubprocessServer(GrpcStubClass, [executable, arg, ...]) as stub:
+ stub.CallService(...)
+ """
+ def __init__(self, stub_class, cmd, port=None):
+ """Creates the server object.
+
+ :param stub_class: the auto-generated GRPC client stub class used for
+ connecting to the GRPC service
+ :param cmd: command (including arguments) for starting up the server,
+ suitable for passing to `subprocess.POpen`.
+ :param port: (optional) the port at which the subprocess will serve its
+ service. If not given, one will be randomly chosen and the special
+ string "{{PORT}}" will be substituted in the command line arguments
+ with the chosen port.
+ """
+ self._process_lock = threading.RLock()
+ self._process = None
+ self._stub_class = stub_class
+ self._cmd = [str(arg) for arg in cmd]
+ self._port = port
+
+ def __enter__(self):
+ return self.start()
+
+ def __exit__(self, *unused_args):
+ self.stop()
+
+ def start(self):
+ with self._process_lock:
+ if self._process:
+ self.stop()
+ if self._port:
+ port = self._port
+ cmd = self._cmd
+ else:
+ port, = pick_port(None)
+ cmd = [arg.replace('{{PORT}}', str(port)) for arg in self._cmd]
+ endpoint = 'localhost:%s' % port
+ logging.warn("Starting service with %s", str(cmd).replace("',", "'"))
+ try:
+ self._process = subprocess.Popen(cmd)
+ wait_secs = .1
+ channel = grpc.insecure_channel(endpoint)
+ channel_ready = grpc.channel_ready_future(channel)
+ while True:
+ if self._process.poll() is not None:
+ logging.error("Starting job service with %s", cmd)
+ raise RuntimeError(
+ 'Service failed to start up with error %s' %
+ self._process.poll())
+ try:
+ channel_ready.result(timeout=wait_secs)
+ break
+ except (grpc.FutureTimeoutError, grpc._channel._Rendezvous):
+ wait_secs *= 1.2
+ logging.log(logging.WARNING if wait_secs > 1 else logging.DEBUG,
+ 'Waiting for grpc channel to be ready at %s.',
+ endpoint)
+ return self._stub_class(channel)
+ except: # pylint: disable=bare-except
+ logging.exception("Error bringing up service")
+ self.stop()
+ raise
+
+ def stop(self):
+ with self._process_lock:
+ if not self._process:
+ return
+ for _ in range(5):
+ if self._process.poll() is not None:
+ break
+ logging.debug("Sending SIGINT to job_server")
+ self._process.send_signal(signal.SIGINT)
+ time.sleep(1)
+ if self._process.poll() is None:
+ self._process.kill()
+ self._process = None
+
+ def local_temp_dir(self, **kwargs):
+ return tempfile.mkdtemp(dir=self._local_temp_root, **kwargs)
+
+
+class JavaJarServer(SubprocessServer):
+
+ APACHE_REPOSITORY = 'https://repo.maven.apache.org/maven2'
+ BEAM_GROUP_ID = 'org.apache.beam'
+ JAR_CACHE = os.path.expanduser("~/.apache_beam/cache/jars")
+
+ def __init__(self, stub_class, path_to_jar, java_arguments):
+ super(JavaJarServer, self).__init__(
+ stub_class, ['java', '-jar', path_to_jar] + list(java_arguments))
+
+ @classmethod
+ def jar_name(cls, artifact_id, version, classifier=None, appendix=None):
+ return '-'.join(filter(
+ None, [artifact_id, appendix, version, classifier])) + '.jar'
+
+ @classmethod
+ def path_to_maven_jar(
+ cls,
+ artifact_id,
+ group_id,
+ version,
+ repository=APACHE_REPOSITORY,
+ classifier=None):
+ return '/'.join([
+ repository,
+ group_id.replace('.', '/'),
+ artifact_id,
+ version,
+ cls.jar_name(artifact_id, version, classifier)])
+
+ @classmethod
+ def path_to_beam_jar(cls, gradle_target, appendix=None):
+ gradle_package = gradle_target.strip(':')[:gradle_target.rindex(':')]
+ artifact_id = 'beam-' + gradle_package.replace(':', '-')
+ project_root = os.path.sep.join(
+ os.path.abspath(__file__).split(os.path.sep)[:-5])
+ local_path = os.path.join(
+ project_root,
+ gradle_package.replace(':', os.path.sep),
+ 'build',
+ 'libs',
+ cls.jar_name(
+ artifact_id,
+ beam_version.replace('.dev', ''),
+ classifier='SNAPSHOT',
+ appendix=appendix))
+ if os.path.exists(local_path):
+ logging.info('Using pre-built snapshot at %s', local_path)
+ return local_path
+ elif '.dev' in beam_version:
+ # TODO: Attempt to use nightly snapshots?
+ raise RuntimeError(
+ 'Please build the server with \n cd %s; ./gradlew %s' % (
+ os.path.abspath(project_root), gradle_target))
+ else:
+ return cls.path_to_maven_jar(
+ artifact_id, cls.BEAM_GROUP_ID, beam_version, cls.APACHE_REPOSITORY)
+
+ @classmethod
+ def local_jar(cls, url):
+ # TODO: Verify checksum?
+ if os.path.exists(url):
+ return url
+ else:
+ logging.warning('Downloading job server jar from %s' % url)
+ cached_jar = os.path.join(cls.JAR_CACHE, os.path.basename(url))
+ if not os.path.exists(cached_jar):
+ if not os.path.exists(cls.JAR_CACHE):
+ os.makedirs(cls.JAR_CACHE)
+ # TODO: Clean up this cache according to some policy.
+ try:
+ url_read = urlopen(url)
+ with open(cached_jar + '.tmp', 'wb') as jar_write:
+ shutil.copyfileobj(url_read, jar_write, length=1 << 20)
+ os.rename(cached_jar + '.tmp', cached_jar)
+ except URLError as e:
+ raise RuntimeError(
+ 'Unable to fetch remote job server jar at %s: %s' % (url, e))
+ return cached_jar
+
+
+def pick_port(*ports):
+ """
+ Returns a list of ports, same length as input ports list, but replaces
+ all None or 0 ports with a random free port.
+ """
+ sockets = []
+
+ def find_free_port(port):
+ if port:
+ return port
+ else:
+ s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ sockets.append(s)
+ s.bind(('localhost', 0))
+ _, free_port = s.getsockname()
+ return free_port
+
+ ports = list(map(find_free_port, ports))
+ # Close sockets only now to avoid the same port to be chosen twice
+ for s in sockets:
+ s.close()
+ return ports
diff --git a/sdks/python/apache_beam/utils/timestamp.py b/sdks/python/apache_beam/utils/timestamp.py
index 2ed775c..9bccdfd 100644
--- a/sdks/python/apache_beam/utils/timestamp.py
+++ b/sdks/python/apache_beam/utils/timestamp.py
@@ -25,9 +25,9 @@
import datetime
import functools
-import re
from builtins import object
+import dateutil.parser
import pytz
from past.builtins import long
@@ -75,9 +75,6 @@
return seconds
return Timestamp(seconds)
- RFC_3339_RE = re.compile(
- r'^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.(\d+))?Z$')
-
@staticmethod
def _epoch_datetime_utc():
return datetime.datetime.fromtimestamp(0, pytz.utc)
@@ -98,20 +95,18 @@
def from_rfc3339(cls, rfc3339):
"""Create a ``Timestamp`` instance from an RFC 3339 compliant string.
+ .. note::
+ All timezones are implicitly converted to UTC.
+
Args:
rfc3339: String in RFC 3339 form.
"""
- dt_args = []
- match = cls.RFC_3339_RE.match(rfc3339)
- if match is None:
- raise ValueError('Could not parse RFC 3339 string: %s' % rfc3339)
- for s in match.groups():
- if s is not None:
- dt_args.append(int(s))
- else:
- dt_args.append(0)
- dt_args += (pytz.utc, )
- dt = datetime.datetime(*dt_args)
+ try:
+ dt = dateutil.parser.isoparse(rfc3339).astimezone(pytz.UTC)
+ except ValueError as e:
+ raise ValueError(
+ "Could not parse RFC 3339 string '{}' due to error: '{}'.".format(
+ rfc3339, e))
return cls.from_utc_datetime(dt)
def predecessor(self):
@@ -151,7 +146,10 @@
def __eq__(self, other):
# Allow comparisons between Duration and Timestamp values.
if not isinstance(other, Duration):
- other = Timestamp.of(other)
+ try:
+ other = Timestamp.of(other)
+ except TypeError:
+ return NotImplemented
return self.micros == other.micros
def __ne__(self, other):
diff --git a/sdks/python/apache_beam/utils/timestamp_test.py b/sdks/python/apache_beam/utils/timestamp_test.py
index 8296dc6..a2cbc5f 100644
--- a/sdks/python/apache_beam/utils/timestamp_test.py
+++ b/sdks/python/apache_beam/utils/timestamp_test.py
@@ -65,6 +65,15 @@
self.assertEqual(rfc3339_str,
Timestamp.from_rfc3339(rfc3339_str).to_rfc3339())
+ def test_from_rfc3339_with_timezone(self):
+ test_cases = [
+ (1458328979.123456, '2016-03-18T23:22:59.123456+04:00'),
+ (1458357779.123456, '2016-03-18T23:22:59.123456-04:00'),
+ ]
+ for seconds_float, rfc3339_str in test_cases:
+ self.assertEqual(Timestamp(seconds_float),
+ Timestamp.from_rfc3339(rfc3339_str))
+
def test_from_rfc3339_failure(self):
with self.assertRaisesRegexp(ValueError, 'parse'):
Timestamp.from_rfc3339('not rfc3339')
diff --git a/sdks/python/apache_beam/utils/windowed_value.py b/sdks/python/apache_beam/utils/windowed_value.py
index 8239abf..5570c45 100644
--- a/sdks/python/apache_beam/utils/windowed_value.py
+++ b/sdks/python/apache_beam/utils/windowed_value.py
@@ -46,7 +46,13 @@
class PaneInfo(object):
- """Describes the trigger firing information for a given WindowedValue."""
+ """Describes the trigger firing information for a given WindowedValue.
+
+ "Panes" represent individual firings on a single window. ``PaneInfo``s are
+ passed downstream after trigger firings. They contain information about
+ whether it's an early/on time/late firing, if it's the last or first firing
+ from a window, and the index of the firing.
+ """
def __init__(self, is_first, is_last, timing, index, nonspeculative_index):
self._is_first = is_first
@@ -202,10 +208,10 @@
return not self == other
def __hash__(self):
- return (hash(self.value) +
- 3 * self.timestamp_micros +
- 7 * hash(self.windows) +
- 11 * hash(self.pane_info))
+ return ((hash(self.value) & 0xFFFFFFFFFFFFFFF) +
+ 3 * (self.timestamp_micros & 0xFFFFFFFFFFFFFF) +
+ 7 * (hash(self.windows) & 0xFFFFFFFFFFFFF) +
+ 11 * (hash(self.pane_info) & 0xFFFFFFFFFFFFF))
def with_value(self, new_value):
"""Creates a new WindowedValue with the same timestamps and windows as this.
diff --git a/sdks/python/apache_beam/version.py b/sdks/python/apache_beam/version.py
index d4053e6..1365114 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.14.0.dev'
+__version__ = '2.17.0.dev'
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 7bd3984..b8c82a4 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -16,374 +16,80 @@
* limitations under the License.
*/
-import org.apache.tools.ant.taskdefs.condition.Os
-
plugins { id 'org.apache.beam.module' }
applyPythonNature()
-enablePythonPerformanceTest()
/*************************************************************************************************/
// Basic build and Python environment setup/cleanup
-task buildPython(dependsOn: 'setupVirtualenv') {
+task buildPython {
+ dependsOn 'setupVirtualenv'
+
doLast {
- println 'Building Python Dependencies'
+ logger.info('Building Python Dependencies')
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && python setup.py build --build-base ${project.buildDir}"
+ args '-c', ". ${envdir}/bin/activate && python setup.py build --build-base ${buildDir}"
}
}
}
build.dependsOn buildPython
+// Create a Python source distribution tarball.
+def tarball = "apache-beam.tar.gz"
+task sdist {
+ dependsOn setupVirtualenv
-/*************************************************************************************************/
-// Unit tests for Python 2
-// See Python 3 tests in test-suites/tox
+ doLast {
+ // Build artifact
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && python setup.py -q sdist --formats zip,gztar --dist-dir ${buildDir}"
+ }
-task lint {}
-check.dependsOn lint
+ def collection = fileTree(buildDir){ include '**/*.tar.gz' exclude '**/apache-beam.tar.gz', 'srcs/**'}
-toxTask "lintPy27", "py27-lint"
-lint.dependsOn lintPy27
-
-toxTask "lintPy27_3", "py27-lint3"
-lint.dependsOn lintPy27_3
-
-toxTask "testPy2Gcp", "py27-gcp"
-test.dependsOn testPy2Gcp
-
-toxTask "testPython2", "py27"
-test.dependsOn testPython2
-
-toxTask "testPy2Cython", "py27-cython"
-test.dependsOn testPy2Cython
-// Ensure that testPy2Cython runs exclusively to other tests. This line is not
-// actually required, since gradle doesn't do parallel execution within a
-// project.
-testPy2Cython.mustRunAfter testPython2, testPy2Gcp
-
-toxTask "docs", "docs"
-assemble.dependsOn docs
-
-toxTask "cover", "cover"
-
-task preCommitPy2() {
- dependsOn "docs"
- dependsOn "testPy2Cython"
- dependsOn "testPython2"
- dependsOn "testPy2Gcp"
- dependsOn "lint"
+ // we need a fixed name for the artifact
+ copy { from collection.singleFile; into buildDir; rename { tarball } }
+ logger.info('Create distribution tar file {} in {}', tarball, buildDir)
+ }
+ inputs.files pythonSdkDeps
+ outputs.file "${buildDir}/${tarball}"
}
-task portablePreCommit() {
- dependsOn ':runners:flink:1.5:job-server-container:docker'
- dependsOn ':sdks:python:container:docker'
- dependsOn portableWordCountTask('portableWordCountBatch', false)
- dependsOn portableWordCountTask('portableWordCountStreaming', true)
+artifacts {
+ distTarBall file: file("${buildDir}/${tarball}"), builtBy: sdist
}
/*************************************************************************************************/
-// E2E integration testing and validates runner testing
-
-// Basic test options for ITs running on Jenkins.
-def basicTestOpts = [
- "--nocapture", // print stdout instantly
- "--processes=8", // run tests in parallel
- "--process-timeout=4500", // timeout of whole command execution
-]
-
-task directRunnerIT(dependsOn: 'installGcpTest') {
- // Run IT tests with TestDirectRunner in batch.
- doLast {
- def tests = [
- "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
- "apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
- "apache_beam.io.gcp.big_query_query_to_table_it_test:BigQueryQueryToTableIT",
- "apache_beam.io.gcp.bigquery_io_read_it_test",
- "apache_beam.io.gcp.datastore.v1new.datastore_write_it_test",
- ]
- def batchTestOpts = basicTestOpts + ["--tests=${tests.join(',')}"]
- def argMap = ["runner": "TestDirectRunner",
- "test_opts": batchTestOpts]
- def batchCmdArgs = project.mapToArgString(argMap)
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ./scripts/run_integration_test.sh $batchCmdArgs"
- }
- }
-
- // Run IT tests with TestDirectRunner in streaming.
- doLast {
- def tests = [
- "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
- "apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
- ]
- def streamingTestOpts = basicTestOpts + ["--tests=${tests.join(',')}"]
- def argMap = ["runner": "TestDirectRunner",
- "streaming": "true",
- "test_opts": streamingTestOpts]
- def streamingCmdArgs = project.mapToArgString(argMap)
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ./scripts/run_integration_test.sh $streamingCmdArgs"
- }
- }
-}
-
-// Before running this, you need to:
-//
-// 1. Build the SDK container:
-//
-// ./gradlew -p sdks/python/container docker
-//
-// 2. Either a) or b)
-// a) If you want the Job Server to run in a Docker container:
-//
-// ./gradlew :runners:flink:1.5:job-server-container:docker
-//
-// b) Otherwise, start a local JobService, for example, the Portable Flink runner
-// (in a separate shell since it continues to run):
-//
-// ./gradlew :runners:flink:1.5:job-server:runShadow
-//
-// Then you can run this example:
-//
-// Docker (2a):
-//
-// ./gradlew :sdks:python:portableWordCount
-//
-// Local JobService (2b):
-//
-// ./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099
-//
-task portableWordCount {
- dependsOn portableWordCountTask('portableWordCountExample', project.hasProperty("streaming"))
-}
-
-def portableWordCountTask(name, streaming) {
- tasks.create(name) {
- dependsOn = ['installGcpTest']
- mustRunAfter = [':runners:flink:1.5:job-server-container:docker', ':sdks:python:container:docker']
- doLast {
- // TODO: Figure out GCS credentials and use real GCS input and output.
- def options = [
- "--input=/etc/profile",
- "--output=/tmp/py-wordcount-direct",
- "--runner=PortableRunner",
- "--experiments=worker_threads=100",
- "--parallelism=2",
- "--shutdown_sources_on_final_watermark",
- ]
- if (streaming)
- options += ["--streaming"]
- else
- // workaround for local file output in docker container
- options += ["--environment_cache_millis=10000"]
- if (project.hasProperty("jobEndpoint"))
- options += ["--job_endpoint=${project.property('jobEndpoint')}"]
- if (project.hasProperty("environmentType")) {
- options += ["--environment_type=${project.property('environmentType')}"]
- }
- if (project.hasProperty("environmentConfig")) {
- options += ["--environment_config=${project.property('environmentConfig')}"]
- }
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && python -m apache_beam.examples.wordcount ${options.join(' ')}"
- // TODO: Check that the output file is generated and runs.
- }
- }
- }
-}
-
-// Run PostCommit integration tests on default runner (TestDataflowRunner)
-task postCommitIT(dependsOn: ['installGcpTest', 'sdist']) {
- dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
-
- def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
-
- doLast {
- def testOpts = basicTestOpts + ["--attr=IT"]
- def cmdArgs = project.mapToArgString(["test_opts": testOpts,
- "worker_jar": dataflowWorkerJar])
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ./scripts/run_integration_test.sh $cmdArgs"
- }
- }
-}
-
-task validatesRunnerBatchTests(dependsOn: ['installGcpTest', 'sdist']) {
- dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
-
- def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
-
- doLast {
- def testOpts = basicTestOpts + ["--attr=ValidatesRunner"]
- def cmdArgs = project.mapToArgString(["test_opts": testOpts,
- "worker_jar": dataflowWorkerJar])
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ./scripts/run_integration_test.sh $cmdArgs"
- }
- }
-}
-
-task validatesRunnerStreamingTests(dependsOn: ['installGcpTest', 'sdist']) {
- dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
-
- def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
-
- doLast {
- // TODO(BEAM-3544,BEAM-5025): Disable tests with 'sickbay-streaming' tag.
- def testOpts = basicTestOpts + ["--attr=ValidatesRunner,!sickbay-streaming"]
- def argMap = ["test_opts": testOpts,
- "streaming": "true",
- "worker_jar": dataflowWorkerJar]
- def cmdArgs = project.mapToArgString(argMap)
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ./scripts/run_integration_test.sh $cmdArgs"
- }
- }
-}
-
-task hdfsIntegrationTest(dependsOn: 'installGcpTest') {
- doLast {
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh"
- }
- }
-}
-
-task sparkValidatesRunner() {
- dependsOn 'createProcessWorker'
- dependsOn 'setupVirtualenv'
- dependsOn ':beam-runners-spark-job-server:shadowJar'
- doLast {
- def environment_config = "'{\"command\": \"${project(":beam-sdks-python:").buildDir.absolutePath}/sdk_worker.sh\"}'"
- def argMap = [
- "environment_type" : "PROCESS",
- "spark_job_server_jar": project(":beam-runners-spark-job-server:").shadowJar.archivePath,
- "environment_config": environment_config,
- ]
- def argString = project.mapToArgString(argMap)
-
- // Optionally specify test function names separated by space e.g.:
- // ./gradlew :beam-sdks-python:sparkValidatesRunner -Ptests="test_external_transforms test_read"
- // Otherwise run all test functions under SparkRunnerTest
- def tests = project.hasProperty('tests') ?
- project.property('tests').split().collect{ "SparkRunnerTest.$it" }.join(' ') : ''
-
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e .[test] && python -m apache_beam.runners.portability.spark_runner_test $tests $argString"
- }
- }
-}
-
-class CompatibilityMatrixConfig {
- // Execute batch or streaming pipelines.
- boolean streaming = false
- // Execute on Docker or Process based environment.
- SDK_WORKER_TYPE workerType = SDK_WORKER_TYPE.DOCKER
-
- enum SDK_WORKER_TYPE {
- DOCKER, PROCESS, LOOPBACK
- }
-
- // Whether to pre-optimize the pipeline with the Python optimizer.
- boolean preOptimize = false
-}
-
-def flinkCompatibilityMatrix = {
- def config = it ? it as CompatibilityMatrixConfig : new CompatibilityMatrixConfig()
- def workerType = config.workerType.name()
- def streaming = config.streaming
- def environment_config = config.workerType == CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS ? "--environment_config='{\"command\": \"${project(":sdks:python").buildDir.absolutePath}/sdk_worker.sh\"}'" : ""
- def name = "flinkCompatibilityMatrix${streaming ? 'Streaming' : 'Batch'}${config.preOptimize ? 'PreOptimize' : ''}${workerType}"
- def extra_experiments = []
- if (config.preOptimize)
- extra_experiments.add('pre_optimize=all')
- tasks.create(name: name) {
- dependsOn 'setupVirtualenv'
- dependsOn ':runners:flink:1.5:job-server:shadowJar'
- if (workerType.toLowerCase() == 'docker')
- dependsOn ':sdks:python:container:docker'
- else if (workerType.toLowerCase() == 'process')
- dependsOn 'createProcessWorker'
- doLast {
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e .[test] && python -m apache_beam.runners.portability.flink_runner_test --flink_job_server_jar=${project(":runners:flink:1.5:job-server:").shadowJar.archivePath} --environment_type=${workerType} ${environment_config} ${streaming ? '--streaming' : ''} ${extra_experiments ? '--extra_experiments=' + extra_experiments.join(',') : ''}"
- }
- }
- }
-}
-
-task flinkCompatibilityMatrixDocker() {
- dependsOn flinkCompatibilityMatrix(streaming: false)
- dependsOn flinkCompatibilityMatrix(streaming: true)
-}
-
-task flinkCompatibilityMatrixProcess() {
- dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
- dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
-}
-
-task flinkCompatibilityMatrixLoopback() {
- dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
- dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
- dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK, preOptimize: true)
-}
-
-task flinkValidatesRunner() {
- dependsOn 'flinkCompatibilityMatrixLoopback'
-}
-
-// Run Python ValidatesRunner tests using the Java ReferenceRunner as a job server and Docker as
-// the SDK environment.
-task javaReferenceRunnerValidatesRunner() {
- dependsOn 'setupVirtualenv'
- dependsOn ':runners:reference:job-server:shadowJar'
- dependsOn ':sdks:python:container:docker'
- doLast {
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e .[test] && python -m apache_beam.runners.portability.java_reference_runner_test --job_server_jar=${project(":runners:reference:job-server:").shadowJar.archivePath} --environment_type=DOCKER"
- }
- }
-}
-
-task postCommit() {
- dependsOn "crossLanguageTests"
- dependsOn "directRunnerIT"
- dependsOn "hdfsIntegrationTest"
- dependsOn "postCommitIT"
-}
-
-
-/*************************************************************************************************/
-// Other build and analysis tasks
+// Non-testing builds and analysis tasks
// Snapshot of dependency requirements defined in setup.py.
// Results will be stored in files under Gradle build directory.
-task depSnapshot(dependsOn: 'installGcpTest') {
+task depSnapshot {
+ dependsOn 'installGcpTest'
+
doLast {
- println 'Snapshoting full dependencies requirements with versions info to requirements.txt.'
+ def outputDir = file(buildDir)
+ if (!outputDir.exists()) {
+ outputDir.mkdirs()
+ }
+ def requirementsFile = "${outputDir}/requirements.txt"
+
+ logger.info('Snapshoting full dependencies requirements with versions info to build/requirements.txt.')
exec {
// Remove useless item "pkg-resources" from file which is introduced by a bug in Ubuntu.
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip freeze --local --all | grep -v \"pkg-resources\" > ${project.buildDir}/requirements.txt"
+ args '-c', ". ${envdir}/bin/activate && pip freeze --local --all | grep -v \"pkg-resources\" > ${requirementsFile}"
}
}
}
-task dependencyUpdates(dependsOn: ':dependencyUpdates') {
+task dependencyUpdates {
+ dependsOn ':dependencyUpdates'
+
doLast {
exec {
executable 'sh'
@@ -396,81 +102,3 @@
dependsOn 'sdist'
dependsOn 'depSnapshot'
}
-
-project.task('createProcessWorker') {
- dependsOn ':sdks:python:container:build'
- dependsOn 'setupVirtualenv'
- def sdkWorkerFile = file("${project.buildDir}/sdk_worker.sh")
- def osType = 'linux'
- if (Os.isFamily(Os.FAMILY_MAC))
- osType = 'darwin'
- def workerScript = "${project(":sdks:python:container:").buildDir.absolutePath}/target/launcher/${osType}_amd64/boot"
- def sdkWorkerFileCode = "sh -c \"pip=`which pip` . ${project.ext.envdir}/bin/activate && ${workerScript} \$* \""
- outputs.file sdkWorkerFile
- doLast {
- sdkWorkerFile.write sdkWorkerFileCode
- exec {
- commandLine('sh', '-c', ". ${project.ext.envdir}/bin/activate && cd ${project.projectDir} && python setup.py install ")
- }
- exec {
- commandLine('chmod', '+x', sdkWorkerFile)
- }
- }
-}
-
-project.task('crossLanguagePythonJavaFlink') {
- dependsOn 'setupVirtualenv'
- dependsOn ':runners:flink:1.5:job-server-container:docker'
- dependsOn ':sdks:python:container:docker'
- dependsOn ':sdks:java:container:docker'
- dependsOn ':runners:core-construction-java:buildTestExpansionServiceJar'
-
- doLast {
- def testServiceExpansionJar = project(":runners:core-construction-java:").buildTestExpansionServiceJar.archivePath
- def options = [
- "--runner=PortableRunner",
- "--experiments=worker_threads=100",
- "--parallelism=2",
- "--shutdown_sources_on_final_watermark",
- "--environment_cache_millis=10000",
- "--expansion_service_port=8096",
- "--expansion_service_jar=${testServiceExpansionJar}",
- ]
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e .[test] && python -m apache_beam.transforms.external_test ${options.join(' ')}"
- }
- }
-}
-
-project.task('crossLanguagePortableWordCount') {
- dependsOn 'setupVirtualenv'
- dependsOn ':runners:flink:1.5:job-server-container:docker'
- dependsOn ':sdks:python:container:docker'
- dependsOn ':sdks:java:container:docker'
- dependsOn ':runners:core-construction-java:buildTestExpansionServiceJar'
-
- doLast {
- def testServiceExpansionJar = project(":runners:core-construction-java:").buildTestExpansionServiceJar.archivePath
- def options = [
- "--input=/etc/profile",
- "--output=/tmp/py-wordcount-portable",
- "--runner=PortableRunner",
- "--experiments=worker_threads=100",
- "--parallelism=2",
- "--shutdown_sources_on_final_watermark",
- "--environment_cache_millis=10000",
- "--expansion_service_jar=${testServiceExpansionJar}",
- ]
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && pip install -e .[test] && python -m apache_beam.examples.wordcount_xlang ${options.join(' ')}"
- // TODO: Check that the output file is generated and runs.
- }
- }
-}
-
-project.task('crossLanguageTests') {
- dependsOn "crossLanguagePythonJavaFlink"
- dependsOn "crossLanguagePortableWordCount"
-}
diff --git a/sdks/python/container/Dockerfile b/sdks/python/container/Dockerfile
index 4203b74..66f233f 100644
--- a/sdks/python/container/Dockerfile
+++ b/sdks/python/container/Dockerfile
@@ -16,7 +16,8 @@
# limitations under the License.
###############################################################################
-FROM python:2-stretch
+ARG py_version
+FROM python:"${py_version}"-stretch
MAINTAINER "Apache Beam <dev@beam.apache.org>"
# Install native bindings required for dependencies.
diff --git a/sdks/python/container/base_image_requirements.txt b/sdks/python/container/base_image_requirements.txt
index 1f3e7d6..8579592 100644
--- a/sdks/python/container/base_image_requirements.txt
+++ b/sdks/python/container/base_image_requirements.txt
@@ -25,49 +25,50 @@
avro==1.8.2;python_version<="2.7"
avro-python3==1.8.2;python_version>="3.4"
-fastavro==0.21.4
+fastavro==0.21.24
crcmod==1.7
dill==0.2.9
-future==0.16.0
+future==0.17.1
futures==3.2.0;python_version<"3.0"
-grpcio==1.15.0
-hdfs==2.1.0
-httplib2==0.9.2
+grpcio==1.22.0
+hdfs==2.5.6
+httplib2==0.12.0
mock==2.0.0
oauth2client==3.0.0
-protobuf==3.6.1
-pyarrow==0.11.1
-pydot==1.2.4
-pyparsing==2.3.1
-pytz==2018.4
+protobuf==3.9.0
+pyarrow==0.14.0
+pydot==1.4.1
+pytz==2019.1
pyvcf==0.6.8;python_version<"3.0"
-pyyaml==3.12
-typing==3.6.1
+pyyaml==3.13
+typing==3.6.6
# Setup packages
nose==1.3.7
# GCP extra features
-google-apitools==0.5.26
-googledatastore==7.0.1;python_version<"3.0"
-google-cloud-pubsub==0.39.0
-google-cloud-bigquery==1.6.0
+google-apitools==0.5.28
+googledatastore==7.0.2;python_version<"3.0"
+google-cloud-pubsub==0.39.1
+google-cloud-bigquery==1.17.0
proto-google-cloud-datastore-v1==0.90.4
-google-cloud-bigtable==0.31.1
-google-cloud-core==0.28.1
+google-cloud-bigtable==0.32.1
+google-cloud-core==1.0.2
+google-cloud-datastore==1.7.4
# Optional packages
-cython==0.28.1
-python-snappy==0.5.3
+cython==0.29.10
+python-snappy==0.5.4
# These are additional packages likely to be used by customers.
-numpy==1.15.2
-scipy==1.1.0
-pandas==0.23.4
-protorpc==0.11.1
+numpy==1.16.4
+scipy==1.2.2
+pandas==0.24.2
+protorpc==0.12.0
python-gflags==3.0.6
-setuptools<=39.1.0 # requirement for Tensorflow.
-tensorflow==1.11.0
+
+tensorflow==1.14.0
+pymongo==3.8.0
# Packages needed for testing.
tenacity>=5.0.2
diff --git a/sdks/python/container/boot.go b/sdks/python/container/boot.go
index 321e7e5..71da7b9 100644
--- a/sdks/python/container/boot.go
+++ b/sdks/python/container/boot.go
@@ -25,6 +25,7 @@
"os"
"path/filepath"
"strings"
+ "time"
"github.com/apache/beam/sdks/go/pkg/beam/artifact"
pbjob "github.com/apache/beam/sdks/go/pkg/beam/model/jobmanagement_v1"
@@ -33,6 +34,7 @@
"github.com/apache/beam/sdks/go/pkg/beam/util/execx"
"github.com/apache/beam/sdks/go/pkg/beam/util/grpcx"
"github.com/golang/protobuf/proto"
+ "github.com/nightlyone/lockfile"
)
var (
@@ -40,6 +42,7 @@
// Contract: https://s.apache.org/beam-fn-api-container-contract.
+ workerPool = flag.Bool("worker_pool", false, "Run as worker pool (optional).")
id = flag.String("id", "", "Local identifier (required).")
loggingEndpoint = flag.String("logging_endpoint", "", "Logging endpoint (required).")
artifactEndpoint = flag.String("artifact_endpoint", "", "Artifact endpoint (required).")
@@ -55,10 +58,25 @@
requirementsFile = "requirements.txt"
sdkSrcFile = "dataflow_python_sdk.tar"
extraPackagesFile = "extra_packages.txt"
+ workerPoolIdEnv = "BEAM_PYTHON_WORKER_POOL_ID"
)
func main() {
flag.Parse()
+
+ if *workerPool == true {
+ workerPoolId := fmt.Sprintf("%d", os.Getpid())
+ os.Setenv(workerPoolIdEnv, workerPoolId)
+ args := []string{
+ "-m",
+ "apache_beam.runners.worker.worker_pool_main",
+ "--service_port=50000",
+ "--container_executable=/opt/apache/beam/boot",
+ }
+ log.Printf("Starting worker pool %v: python %v", workerPoolId, strings.Join(args, " "))
+ log.Fatalf("Python SDK worker pool exited: %v", execx.Execute("python", args...))
+ }
+
if *id == "" {
log.Fatal("No id provided.")
}
@@ -91,18 +109,30 @@
}
// (2) Retrieve and install the staged packages.
+ //
+ // Guard from concurrent artifact retrieval and installation,
+ // when called by child processes in a worker pool.
- dir := filepath.Join(*semiPersistDir, "staged")
+ materializeArtifactsFunc := func() {
+ dir := filepath.Join(*semiPersistDir, "staged")
- files, err := artifact.Materialize(ctx, *artifactEndpoint, info.GetRetrievalToken(), dir)
- if err != nil {
- log.Fatalf("Failed to retrieve staged files: %v", err)
+ files, err := artifact.Materialize(ctx, *artifactEndpoint, info.GetRetrievalToken(), dir)
+ if err != nil {
+ log.Fatalf("Failed to retrieve staged files: %v", err)
+ }
+
+ // TODO(herohde): the packages to install should be specified explicitly. It
+ // would also be possible to install the SDK in the Dockerfile.
+ if setupErr := installSetupPackages(files, dir); setupErr != nil {
+ log.Fatalf("Failed to install required packages: %v", setupErr)
+ }
}
- // TODO(herohde): the packages to install should be specified explicitly. It
- // would also be possible to install the SDK in the Dockerfile.
- if setupErr := installSetupPackages(files, dir); setupErr != nil {
- log.Fatalf("Failed to install required packages: %v", setupErr)
+ workerPoolId := os.Getenv(workerPoolIdEnv)
+ if workerPoolId != "" {
+ multiProcessExactlyOnce(materializeArtifactsFunc, "beam.install.complete."+workerPoolId)
+ } else {
+ materializeArtifactsFunc()
}
// (3) Invoke python
@@ -162,3 +192,44 @@
}
return ret
}
+
+// Call the given function exactly once across multiple worker processes.
+// The need for multiple processes is specific to the Python SDK due to the GIL.
+// Should another SDK require it, this could be separated out as shared utility.
+func multiProcessExactlyOnce(actionFunc func(), completeFileName string) {
+ installCompleteFile := filepath.Join(os.TempDir(), completeFileName)
+
+ // skip if install already complete, no need to lock
+ _, err := os.Stat(installCompleteFile)
+ if err == nil {
+ return
+ }
+
+ lock, err := lockfile.New(filepath.Join(os.TempDir(), completeFileName+".lck"))
+ if err != nil {
+ log.Fatalf("Cannot init artifact retrieval lock: %v", err)
+ }
+
+ for err = lock.TryLock(); err != nil; err = lock.TryLock() {
+ if _, ok := err.(lockfile.TemporaryError); ok {
+ time.Sleep(5 * time.Second)
+ log.Printf("Worker %v waiting for artifact retrieval lock: %v", *id, lock)
+ } else {
+ log.Fatalf("Worker %v could not obtain artifact retrieval lock: %v", *id, err)
+ }
+ }
+ defer lock.Unlock()
+
+ // skip if install already complete
+ _, err = os.Stat(installCompleteFile)
+ if err == nil {
+ return
+ }
+
+ // do the real work
+ actionFunc()
+
+ // mark install complete
+ os.OpenFile(installCompleteFile, os.O_RDONLY|os.O_CREATE, 0666)
+
+}
diff --git a/sdks/python/container/build.gradle b/sdks/python/container/build.gradle
index a6650e9..2f7662c 100644
--- a/sdks/python/container/build.gradle
+++ b/sdks/python/container/build.gradle
@@ -18,7 +18,6 @@
plugins { id 'org.apache.beam.module' }
applyGoNature()
-applyDockerNature()
description = "Apache Beam :: SDKs :: Python :: Container"
@@ -39,16 +38,7 @@
build name: './github.com/apache/beam/sdks/go', dir: project(':sdks:go').projectDir
test name: './github.com/apache/beam/sdks/go', dir: project(':sdks:go').projectDir
}
- sdkSourceTarball project(path: ":sdks:python", configuration: "distConfig")
-}
-
-task copyDockerfileDependencies(type: Copy, dependsOn: goBuild) {
- from configurations.sdkSourceTarball
- from file("./base_image_requirements.txt")
- into "build/target"
- if(configurations.sdkSourceTarball.isEmpty()) {
- throw new StopExecutionException('sdk source tarball is empty');
- }
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
}
golang {
@@ -60,15 +50,13 @@
}
}
-docker {
- name containerImageName(name: "python")
- files "./build"
+task buildAll {
+ dependsOn ':sdks:python:container:py2:docker'
+ dependsOn ':sdks:python:container:py35:docker'
+ dependsOn ':sdks:python:container:py36:docker'
+ dependsOn ':sdks:python:container:py37:docker'
}
artifacts {
sdkHarnessLauncher file: file('./build/target/launcher'), builtBy: goBuild
}
-
-// Ensure that making the docker image builds any required artifacts
-dockerPrepare.dependsOn goBuild
-dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/sdks/python/container/extra_requirements/Dockerfile b/sdks/python/container/extra_requirements/Dockerfile
deleted file mode 100644
index bbc39a6..0000000
--- a/sdks/python/container/extra_requirements/Dockerfile
+++ /dev/null
@@ -1,27 +0,0 @@
-###############################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-###############################################################################
-
-ARG BASE_PYTHON_IMAGE
-FROM $BASE_PYTHON_IMAGE
-MAINTAINER "Apache Beam <dev@beam.apache.org>"
-
-COPY requirements.txt /tmp/user_requirements.txt
-RUN pip install -r /tmp/user_requirements.txt && \
- # Remove pip cache.
- rm -rf /root/.cache/pip
-
diff --git a/sdks/python/container/py2/build.gradle b/sdks/python/container/py2/build.gradle
new file mode 100644
index 0000000..64f39f0
--- /dev/null
+++ b/sdks/python/container/py2/build.gradle
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'base'
+ id 'org.apache.beam.module'
+}
+applyDockerNature()
+
+description = "Apache Beam :: SDKs :: Python :: Container :: Python 2 Container"
+
+configurations {
+ sdkSourceTarball
+ sdkHarnessLauncher
+}
+
+dependencies {
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
+ sdkHarnessLauncher project(path: ":sdks:python:container", configuration: "sdkHarnessLauncher")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.sdkSourceTarball
+ from file("../base_image_requirements.txt")
+ into "build/target"
+ if(configurations.sdkSourceTarball.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+task copyLauncherDependencies(type: Copy) {
+ from configurations.sdkHarnessLauncher
+ into "build/target/launcher"
+ if(configurations.sdkHarnessLauncher.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+docker {
+ name containerImageName(
+ name: "python2.7_sdk",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] : "apachebeam",
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project['python_sdk_version'])
+ files "../Dockerfile", "./build"
+ buildArgs(['py_version': "2.7"])
+}
+
+dockerPrepare.dependsOn copyLauncherDependencies
+dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/sdks/python/container/py3/Dockerfile b/sdks/python/container/py3/Dockerfile
deleted file mode 100644
index 5fa63fe..0000000
--- a/sdks/python/container/py3/Dockerfile
+++ /dev/null
@@ -1,53 +0,0 @@
-###############################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-###############################################################################
-
-FROM python:3.5-stretch
-MAINTAINER "Apache Beam <dev@beam.apache.org>"
-
-# Install native bindings required for dependencies.
-RUN apt-get update && \
- apt-get install -y \
- # These packages are needed for "pip install python-snappy" below.
- libsnappy-dev \
- # This package is needed for "pip install pyyaml" below to have c bindings.
- libyaml-dev \
- && \
- rm -rf /var/lib/apt/lists/*
-
-# Install packages required by the Python SDK and common dependencies of the user code.
-
-# SDK dependencies not listed in base_image_requirements.txt will be installed when we install SDK
-# in the next RUN statement.
-
-COPY target/base_image_requirements.txt /tmp/base_image_requirements.txt
-RUN \
- pip install -r /tmp/base_image_requirements.txt && \
- # Check that the fast implementation of protobuf is used.
- python -c "from google.protobuf.internal import api_implementation; assert api_implementation._default_implementation_type == 'cpp'; print ('Verified fast protobuf used.')" && \
- # Remove pip cache.
- rm -rf /root/.cache/pip
-
-
-COPY target/apache-beam.tar.gz /opt/apache/beam/tars/
-RUN pip install /opt/apache/beam/tars/apache-beam.tar.gz[gcp] && \
- # Remove pip cache.
- rm -rf /root/.cache/pip
-
-ADD target/linux_amd64/boot /opt/apache/beam/
-
-ENTRYPOINT ["/opt/apache/beam/boot"]
diff --git a/sdks/python/container/py3/build.gradle b/sdks/python/container/py3/build.gradle
deleted file mode 100644
index 05710bc..0000000
--- a/sdks/python/container/py3/build.gradle
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-plugins {
- id 'base'
- id 'org.apache.beam.module'
-}
-applyDockerNature()
-
-description = "Apache Beam :: SDKs :: Python :: Container :: Python 3 Container"
-
-configurations {
- sdkSourceTarball
- sdkHarnessLauncher
-}
-
-dependencies {
- sdkSourceTarball project(path: ":sdks:python", configuration: "distConfig")
- sdkHarnessLauncher project(path: ":sdks:python:container", configuration: "sdkHarnessLauncher")
-}
-
-task copyDockerfileDependencies(type: Copy) {
- from configurations.sdkSourceTarball
- from configurations.sdkHarnessLauncher
- from file("../base_image_requirements.txt")
- into "build/target"
- if(configurations.sdkSourceTarball.isEmpty() || configurations.sdkHarnessLauncher.isEmpty()) {
- throw new StopExecutionException();
- }
-}
-
-docker {
- name containerImageName(name: "python3")
- files "./build"
-}
-
-dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/sdks/python/container/py35/build.gradle b/sdks/python/container/py35/build.gradle
new file mode 100644
index 0000000..024847b
--- /dev/null
+++ b/sdks/python/container/py35/build.gradle
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'base'
+ id 'org.apache.beam.module'
+}
+applyDockerNature()
+
+description = "Apache Beam :: SDKs :: Python :: Container :: Python 35 Container"
+
+configurations {
+ sdkSourceTarball
+ sdkHarnessLauncher
+}
+
+dependencies {
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
+ sdkHarnessLauncher project(path: ":sdks:python:container", configuration: "sdkHarnessLauncher")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.sdkSourceTarball
+ from file("../base_image_requirements.txt")
+ into "build/target"
+ if(configurations.sdkSourceTarball.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+task copyLauncherDependencies(type: Copy) {
+ from configurations.sdkHarnessLauncher
+ into "build/target/launcher"
+ if(configurations.sdkHarnessLauncher.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+docker {
+ name containerImageName(
+ name: "python3.5_sdk",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] : "apachebeam",
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project['python_sdk_version'])
+ files "../Dockerfile", "./build"
+ buildArgs(['py_version': "3.5"])
+}
+
+dockerPrepare.dependsOn copyLauncherDependencies
+dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/sdks/python/container/py36/build.gradle b/sdks/python/container/py36/build.gradle
new file mode 100644
index 0000000..f81f6ec
--- /dev/null
+++ b/sdks/python/container/py36/build.gradle
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'base'
+ id 'org.apache.beam.module'
+}
+applyDockerNature()
+
+description = "Apache Beam :: SDKs :: Python :: Container :: Python 36 Container"
+
+configurations {
+ sdkSourceTarball
+ sdkHarnessLauncher
+}
+
+dependencies {
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
+ sdkHarnessLauncher project(path: ":sdks:python:container", configuration: "sdkHarnessLauncher")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.sdkSourceTarball
+ from file("../base_image_requirements.txt")
+ into "build/target"
+ if(configurations.sdkSourceTarball.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+task copyLauncherDependencies(type: Copy) {
+ from configurations.sdkHarnessLauncher
+ into "build/target/launcher"
+ if(configurations.sdkHarnessLauncher.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+docker {
+ name containerImageName(
+ name: "python3.6_sdk",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] : "apachebeam",
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project['python_sdk_version'])
+ files "../Dockerfile", "./build"
+ buildArgs(['py_version': "3.6"])
+}
+
+dockerPrepare.dependsOn copyLauncherDependencies
+dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/sdks/python/container/py37/build.gradle b/sdks/python/container/py37/build.gradle
new file mode 100644
index 0000000..a7f10c4
--- /dev/null
+++ b/sdks/python/container/py37/build.gradle
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+ id 'base'
+ id 'org.apache.beam.module'
+}
+applyDockerNature()
+
+description = "Apache Beam :: SDKs :: Python :: Container :: Python 37 Container"
+
+configurations {
+ sdkSourceTarball
+ sdkHarnessLauncher
+}
+
+dependencies {
+ sdkSourceTarball project(path: ":sdks:python", configuration: "distTarBall")
+ sdkHarnessLauncher project(path: ":sdks:python:container", configuration: "sdkHarnessLauncher")
+}
+
+task copyDockerfileDependencies(type: Copy) {
+ from configurations.sdkSourceTarball
+ from file("../base_image_requirements.txt")
+ into "build/target"
+ if(configurations.sdkSourceTarball.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+task copyLauncherDependencies(type: Copy) {
+ from configurations.sdkHarnessLauncher
+ into "build/target/launcher"
+ if(configurations.sdkHarnessLauncher.isEmpty()) {
+ throw new StopExecutionException();
+ }
+}
+
+docker {
+ name containerImageName(
+ name: "python3.7_sdk",
+ root: project.rootProject.hasProperty(["docker-repository-root"]) ?
+ project.rootProject["docker-repository-root"] : "apachebeam",
+ tag: project.rootProject.hasProperty(["docker-tag"]) ?
+ project.rootProject["docker-tag"] : project['python_sdk_version'])
+ files "../Dockerfile", "./build"
+ buildArgs(['py_version': "3.7"])
+}
+
+
+dockerPrepare.dependsOn copyLauncherDependencies
+dockerPrepare.dependsOn copyDockerfileDependencies
diff --git a/sdks/python/container/run_validatescontainer.sh b/sdks/python/container/run_validatescontainer.sh
index 639f4dc..216d617 100755
--- a/sdks/python/container/run_validatescontainer.sh
+++ b/sdks/python/container/run_validatescontainer.sh
@@ -25,14 +25,16 @@
#
# Execute from the root of the repository:
# test Python2 container: ./sdks/python/container/run_validatescontainer.sh python2
-# test Python3 container: ./sdks/python/container/run_validatescontainer.sh python3
+# test Python3 container: ./sdks/python/container/run_validatescontainer.sh python35
+# test Python3 container: ./sdks/python/container/run_validatescontainer.sh python36
+# test Python3 container: ./sdks/python/container/run_validatescontainer.sh python37
echo "This script must be executed in the root of beam project. Please set LOCAL_PATH, GCS_LOCATION, and PROJECT as desired."
if [[ $# != 1 ]]; then
printf "Usage: \n$> ./sdks/python/container/run_validatescontainer.sh <python_version>"
printf "\n\tpython_version: [required] Python version used for container build and run tests."
- printf " Use 'python2' for Python2, 'python3' for Python3."
+ printf " Use 'python2' for Python2, 'python35' for Python3.5, python36 for Python3.6, python37 for Python3.7."
exit 1
fi
@@ -47,19 +49,30 @@
# Other variables branched by Python version.
if [[ $1 == "python2" ]]; then
- IMAGE_NAME="python" # Use this to create CONTAINER_IMAGE variable.
- CONTAINER_PROJECT="sdks:python:container" # Use this to build container by Gradle.
+ IMAGE_NAME="python2.7_sdk" # Use this to create CONTAINER_IMAGE variable.
+ CONTAINER_PROJECT="sdks:python:container:py2" # Use this to build container by Gradle.
GRADLE_PY3_FLAG="" # Use this in Gradle command.
PY_INTERPRETER="python" # Use this in virtualenv command.
-elif [[ $1 == "python3" ]]; then
- IMAGE_NAME="python3" # Use this to create CONTAINER_IMAGE variable.
- CONTAINER_PROJECT="sdks:python:container:py3" # Use this to build container by Gradle.
+elif [[ $1 == "python35" ]]; then
+ IMAGE_NAME="python3.5_sdk" # Use this to create CONTAINER_IMAGE variable.
+ CONTAINER_PROJECT="sdks:python:container:py35" # Use this to build container by Gradle.
GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
PY_INTERPRETER="python3.5" # Use this in virtualenv command.
+elif [[ $1 == "python36" ]]; then
+ IMAGE_NAME="python3.6_sdk" # Use this to create CONTAINER_IMAGE variable.
+ CONTAINER_PROJECT="sdks:python:container:py36" # Use this to build container by Gradle.
+ GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
+ PY_INTERPRETER="python3.6" # Use this in virtualenv command.
+elif [[ $1 == "python37" ]]; then
+ IMAGE_NAME="python3.7_sdk" # Use this to create CONTAINER_IMAGE variable.
+ CONTAINER_PROJECT="sdks:python:container:py37" # Use this to build container by Gradle.
+ GRADLE_PY3_FLAG="-Ppython3" # Use this in Gradle command.
+ PY_INTERPRETER="python3.7" # Use this in virtualenv command.
else
- echo "Must set Python version with 'python2' or 'python3' from commandline."
+ echo "Must set Python version with one of 'python2', 'python35', 'python36' and 'python37' from commandline."
exit 1
fi
+XUNIT_FILE="nosetests-$IMAGE_NAME.xml"
# Verify in the root of the repository
test -d sdks/python/container
@@ -110,6 +123,9 @@
--nologcapture \
--processes=1 \
--process-timeout=900 \
+ --with-xunitmp \
+ --xunitmp-file=$XUNIT_FILE \
+ --ignore-files '.*py3\d?\.py$' \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
diff --git a/sdks/python/gen_protos.py b/sdks/python/gen_protos.py
index 2053ad3..4416959 100644
--- a/sdks/python/gen_protos.py
+++ b/sdks/python/gen_protos.py
@@ -49,13 +49,16 @@
]
-def generate_proto_files(force=False):
+def generate_proto_files(force=False, log=None):
try:
import grpc_tools # pylint: disable=unused-variable
except ImportError:
warnings.warn('Installing grpcio-tools is recommended for development.')
+ if log is None:
+ log = logging.getLogger(__name__)
+
py_sdk_root = os.path.dirname(os.path.abspath(__file__))
common = os.path.join(py_sdk_root, '..', 'common')
proto_dirs = [os.path.join(py_sdk_root, path) for path in BEAM_PROTO_PATHS]
@@ -67,7 +70,7 @@
if out_files and not proto_files and not force:
# We have out_files but no protos; assume they're up to date.
# This is actually the common case (e.g. installation from an sdist).
- logging.info('No proto files; using existing generated files.')
+ log.info('No proto files; using existing generated files.')
return
elif not out_files and not proto_files:
@@ -78,11 +81,27 @@
raise RuntimeError(
'No proto files found in %s.' % proto_dirs)
- # Regenerate iff the proto files or this file are newer.
- elif force or not out_files or len(out_files) < len(proto_files) or (
+ if force:
+ regenerate = 'forced'
+ elif not out_files:
+ regenerate = 'no output files'
+ elif len(out_files) < len(proto_files):
+ regenerate = 'not enough output files'
+ elif (
min(os.path.getmtime(path) for path in out_files)
<= max(os.path.getmtime(path)
for path in proto_files + [os.path.realpath(__file__)])):
+ regenerate = 'output files are out-of-date'
+ elif len(out_files) > len(proto_files):
+ regenerate = 'output files without corresponding .proto files'
+ # too many output files: probably due to switching between git branches.
+ # remove them so they don't trigger constant regeneration.
+ for out_file in out_files:
+ os.remove(out_file)
+ else:
+ regenerate = None
+
+ if regenerate:
try:
from grpc_tools import protoc
except ImportError:
@@ -103,7 +122,8 @@
if p.exitcode:
raise ValueError("Proto generation failed (see log for details).")
else:
- logging.info('Regenerating out-of-date Python proto definitions.')
+
+ log.info('Regenerating Python proto definitions (%s).' % regenerate)
builtin_protos = pkg_resources.resource_filename('grpc_tools', '_proto')
args = (
[sys.executable] + # expecting to be called from command line
@@ -119,21 +139,20 @@
'Protoc returned non-zero status (see logs for details): '
'%s' % ret_code)
- # copy resource files
- for path in MODEL_RESOURCES:
- shutil.copy2(os.path.join(py_sdk_root, path), out_dir)
+ # copy resource files
+ for path in MODEL_RESOURCES:
+ shutil.copy2(os.path.join(py_sdk_root, path), out_dir)
- ret_code = subprocess.call(["pip", "install", "future==0.16.0"])
- if ret_code:
- raise RuntimeError(
- 'Error installing future during proto generation')
+ ret_code = subprocess.call(["pip", "install", "future==0.16.0"])
+ if ret_code:
+ raise RuntimeError(
+ 'Error installing future during proto generation')
- ret_code = subprocess.call(
- ["futurize", "--both-stages", "--write", "--verbose", "--no-diff",
- out_dir])
- if ret_code:
- raise RuntimeError(
- 'Error applying futurize to generated protobuf python files.')
+ ret_code = subprocess.call(
+ ["futurize", "--both-stages", "--write", "--no-diff", out_dir])
+ if ret_code:
+ raise RuntimeError(
+ 'Error applying futurize to generated protobuf python files.')
# Though wheels are available for grpcio-tools, setup_requires uses
diff --git a/sdks/python/scripts/add_requirements.sh b/sdks/python/scripts/add_requirements.sh
deleted file mode 100755
index af2a878..0000000
--- a/sdks/python/scripts/add_requirements.sh
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This script builds a Docker container with the user specified requirements on top of
-# an existing Python worker Docker container (either one you build from source as
-# described in CONTAINERS.md or from a released Docker container).
-
-# Quit on any errors
-set -e
-
-echo "To add requirements you will need a requirements.txt (you can specify with
- the env variable USER_REQUIREMENTS) and somewhere to push the resulting docker
- image (e.g bintrary, GCP container registry)."
-
-# Be really verbose about each command we are running
-set -x
-
-
-USER_REQUIREMENTS=${USER_REQUIREMENTS:-requirements.txt}
-SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
-BASE_PYTHON_IMAGE=${BASE_PYTHON_IMAGE:-"$(whoami)-docker-apache.bintray.io/beam/python"}
-NEW_PYTHON_IMAGE=${NEW_PYTHON_IMAGE:-"${BASE_PYTHON_IMAGE}-with-requirements"}
-DOCKER_FILE="${SCRIPT_DIR}/../container/extra_requirements/Dockerfile"
-TEMP_DIR=$(mktemp -d -t "boo-loves-beam-XXXXXXXXXXXXXXX")
-cp $DOCKER_FILE $TEMP_DIR
-cp $USER_REQUIREMENTS $TEMP_DIR/requirements.txt
-pushd $TEMP_DIR
-docker build . -t $NEW_PYTHON_IMAGE --build-arg BASE_PYTHON_IMAGE=$BASE_PYTHON_IMAGE
-popd
-rm -rf $TEMP_DIR
diff --git a/sdks/python/scripts/generate_pydoc.sh b/sdks/python/scripts/generate_pydoc.sh
index 7564b49..eab8aad 100755
--- a/sdks/python/scripts/generate_pydoc.sh
+++ b/sdks/python/scripts/generate_pydoc.sh
@@ -66,6 +66,7 @@
apache_beam/runners/dataflow/internal/
apache_beam/runners/portability/
apache_beam/runners/worker/
+ apache_beam/testing/benchmarks/chicago_taxi/
apache_beam/tools/map_fn_microbenchmark.*
apache_beam/transforms/cy_combiners.*
apache_beam/transforms/cy_dataflow_distribution_counter.*
@@ -75,6 +76,8 @@
*_pb2.py
*_test.py
*_test_common.py
+ # TODO(BEAM-7847): Remove this once doc generation can parse Py3 syntax.
+ *_py3*.py
)
python $(type -p sphinx-apidoc) -fMeT -o target/docs/source apache_beam \
@@ -117,7 +120,7 @@
intersphinx_mapping = {
'python': ('https://docs.python.org/2', None),
'hamcrest': ('https://pyhamcrest.readthedocs.io/en/stable/', None),
- 'google-cloud': ('https://googleapis.github.io/google-cloud-python/latest/', None),
+ 'google-cloud-datastore': ('https://googleapis.dev/python/datastore/latest/', None),
}
# Since private classes are skipped by sphinx, if there is any cross reference
diff --git a/sdks/python/scripts/run_dependency_check.sh b/sdks/python/scripts/run_dependency_check.sh
index 6fc8766..d000139 100755
--- a/sdks/python/scripts/run_dependency_check.sh
+++ b/sdks/python/scripts/run_dependency_check.sh
@@ -21,7 +21,8 @@
set -v
# Virtualenv for the rest of the script to run setup
-/usr/bin/virtualenv sdks/python
+rm -rf sdks/
+virtualenv sdks/python
. sdks/python/bin/activate
pip install -e .[gcp,test,docs]
diff --git a/sdks/python/scripts/run_expansion_services.sh b/sdks/python/scripts/run_expansion_services.sh
new file mode 100755
index 0000000..0cf18c6
--- /dev/null
+++ b/sdks/python/scripts/run_expansion_services.sh
@@ -0,0 +1,136 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+read -r -d '' USAGE <<END
+Usage: run_expansion_services.sh (start|stop) [options]
+Options:
+ --group_id [unique id for stop services later]
+ --java_expansion_service_jar [path to expansion service jar]
+ --python_virtualenv_dir [path to virtualenv root dir]
+ --python_expansion_service_module [name of expansion service module]
+ --java_port [number]
+ --python_port [number]
+END
+
+while [[ $# -gt 0 ]]; do
+ key="$1"
+ case $key in
+ --group_id)
+ GROUP_ID="$2"
+ shift
+ shift
+ ;;
+ --java_expansion_service_jar)
+ JAVA_EXPANSION_SERVICE_JAR="$2"
+ shift
+ shift
+ ;;
+ --python_virtualenv_dir)
+ PYTHON_VIRTUALENV_DIR="$2"
+ shift
+ shift
+ ;;
+ --python_expansion_service_module)
+ PYTHON_EXPANSION_SERVICE_MODULE="$2"
+ shift
+ shift
+ ;;
+ --java_port)
+ JAVA_PORT="$2"
+ shift
+ shift
+ ;;
+ --python_port)
+ PYTHON_PORT="$2"
+ shift
+ shift
+ ;;
+ start)
+ STARTSTOP="$1"
+ shift
+ ;;
+ stop)
+ STARTSTOP="$1"
+ shift
+ ;;
+ *)
+ echo "Unknown option: $1"
+ echo "$USAGE"
+ exit 1
+ ;;
+ esac
+done
+
+FILE_BASE="beam-expansion-service"
+if [ -v GROUP_ID ]; then
+ FILE_BASE="$FILE_BASE-$GROUP_ID"
+fi
+
+TEMP_DIR=/tmp
+pid=$TEMP_DIR/$FILE_BASE.pid
+lock=$TEMP_DIR/$FILE_BASE.lock
+
+command -v flock >/dev/null 2>&1
+if [[ $? -eq 0 ]]; then
+ exec 200>$lock
+ if ! flock -n 200; then
+ echo "script already running."
+ exit 0
+ fi
+fi
+
+case $STARTSTOP in
+ start)
+ if [ -f "$pid" ]; then
+ echo "services already running."
+ exit 0
+ fi
+
+ echo "Launching Java expansion service @ $JAVA_PORT"
+ java -jar $JAVA_EXPANSION_SERVICE_JAR $JAVA_PORT >$TEMP_DIR/$FILE_BASE-java.log 2>&1 </dev/null &
+ mypid=$!
+ if kill -0 $mypid >/dev/null 2>&1; then
+ echo $mypid >> $pid
+ else
+ echo "Can't start Java expansion service."
+ fi
+
+ echo "Launching Python expansion service @ $PYTHON_PORT"
+ sh -c ". $PYTHON_VIRTUALENV_DIR/bin/activate && python -m $PYTHON_EXPANSION_SERVICE_MODULE -p $PYTHON_PORT" >$TEMP_DIR/$FILE_BASE-python.log 2>&1 </dev/null &
+ mypid=$!
+ if kill -0 $mypid >/dev/null 2>&1; then
+ echo $mypid >> $pid
+ else
+ echo "Can't start Python expansion service."
+ fi
+ ;;
+ stop)
+ if [ -f "$pid" ]; then
+ while read stop_pid; do
+ if kill -0 $stop_pid >/dev/null 2>&1; then
+ echo "Stopping expansion service pid: $stop_pid."
+ kill $stop_pid
+ else
+ echo "Skipping invalid pid: $stop_pid."
+ fi
+ done < $pid
+ rm $pid
+ fi
+ ;;
+esac
+flock -u 200
diff --git a/sdks/python/scripts/run_integration_test.sh b/sdks/python/scripts/run_integration_test.sh
index 6e7b3ca..1133147 100755
--- a/sdks/python/scripts/run_integration_test.sh
+++ b/sdks/python/scripts/run_integration_test.sh
@@ -18,10 +18,10 @@
###########################################################################
#
-# This script is useful to run single or a set of Python integration tests
-# manually or through Gradle. Note, this script doesn't setup python
-# environment which is required before running tests. Use Gradle task
-# `:sdks:python:integrationTests` to do both together.
+# This script is used in Gradle to run single or a set of Python integration tests
+# locally or on Jenkins. Note, this script doesn't setup python environment which is
+# required for integration test. In order to do so, run Gradle tasks defined in
+# :sdks:python:test-suites instead.
#
# In order to run test with customer options, use following commandline flags:
#
@@ -48,6 +48,9 @@
# during execution. Commonly used options like `--attr`,
# `--tests`, `--nologcapture`. More can be found in
# https://nose.readthedocs.io/en/latest/man.html#options
+# suite -> Namespace for this run of tests. Required if running
+# under Jenkins. Used to differentiate runs of the same
+# tests with different interpreters/dependencies/etc.
#
# Example usages:
# - Run full set of PostCommit tests with default pipeline options:
@@ -72,6 +75,7 @@
STREAMING=false
WORKER_JAR=""
KMS_KEY_NAME="projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test"
+SUITE=""
# Default test (nose) options.
# Run WordCountIT.test_wordcount_it by default if no test options are
@@ -142,6 +146,11 @@
shift # past argument
shift # past value
;;
+ --suite)
+ SUITE="$2"
+ shift # past argument
+ shift # past value
+ ;;
*) # unknown option
echo "Unknown option: $1"
exit 1
@@ -149,6 +158,12 @@
esac
done
+if [[ "$JENKINS_HOME" != "" && "$SUITE" == "" ]]; then
+ echo "Argument --suite is required in a Jenkins environment."
+ exit 1
+fi
+XUNIT_FILE="nosetests-$SUITE.xml"
+
set -o errexit
@@ -171,12 +186,11 @@
if [[ -z $PIPELINE_OPTS ]]; then
- # Create a tarball if not exists
+ # Get tar ball path
if [[ $(find ${SDK_LOCATION} 2> /dev/null) ]]; then
- SDK_LOCATION=$(find ${SDK_LOCATION})
+ SDK_LOCATION=$(find ${SDK_LOCATION} | tail -n1)
else
- python setup.py -q sdist
- SDK_LOCATION=$(ls dist/apache-beam-*.tar.gz | tail -n1)
+ echo "[WARNING] Could not find SDK tarball in SDK_LOCATION: $SDK_LOCATION."
fi
# Install test dependencies for ValidatesRunner tests.
@@ -227,6 +241,10 @@
echo ">>> RUNNING integration tests with pipeline options: $PIPELINE_OPTS"
echo ">>> test options: $TEST_OPTS"
+# TODO(BEAM-3713): Pass $SUITE once migrated to pytest. xunitmp doesn't support
+# suite names.
python setup.py nosetests \
--test-pipeline-options="$PIPELINE_OPTS" \
+ --with-xunitmp --xunitmp-file=$XUNIT_FILE \
+ --ignore-files '.*py3\d?\.py$' \
$TEST_OPTS
diff --git a/sdks/python/scripts/run_mini_py3lint.sh b/sdks/python/scripts/run_mini_py3lint.sh
index 27ca3ce..0bd7e0e 100755
--- a/sdks/python/scripts/run_mini_py3lint.sh
+++ b/sdks/python/scripts/run_mini_py3lint.sh
@@ -38,6 +38,24 @@
MODULE=apache_beam
+PYTHON_MINOR=$(python -c 'import sys; print(sys.version_info[1])')
+if [[ "${PYTHON_MINOR}" == 5 ]]; then
+ EXCLUDED_PY3_FILES=$(find ${MODULE} | grep 'py3[6-9]\.py$')
+ echo -e "Excluding Py3 files:\n${EXCLUDED_PY3_FILES}"
+else
+ EXCLUDED_PY3_FILES=""
+fi
+
+FILES_TO_IGNORE=""
+for file in ${EXCLUDED_PY3_FILES}; do
+ if test -z "$FILES_TO_IGNORE"
+ then FILES_TO_IGNORE="$(basename $file)"
+ else FILES_TO_IGNORE="$FILES_TO_IGNORE, $(basename $file)"
+ fi
+done
+
+echo -e "Skipping lint for files:\n${FILES_TO_IGNORE}"
+
usage(){ echo "Usage: $0 [MODULE|--help] # The default MODULE is $MODULE"; }
if test $# -gt 0; then
@@ -48,4 +66,5 @@
fi
echo "Running flake8 for module $MODULE:"
-flake8 $MODULE --count --select=E9,F821,F822,F823 --show-source --statistics
+flake8 $MODULE --count --select=E9,F821,F822,F823 --show-source --statistics \
+ --exclude="${FILES_TO_IGNORE}"
diff --git a/sdks/python/scripts/run_pylint.sh b/sdks/python/scripts/run_pylint.sh
index 2814b4f..27de9a3 100755
--- a/sdks/python/scripts/run_pylint.sh
+++ b/sdks/python/scripts/run_pylint.sh
@@ -60,24 +60,34 @@
apache_beam/portability/api/*pb2*.py
)
+PYTHON_MAJOR=$(python -c 'import sys; print(sys.version_info[0])')
+if [[ "${PYTHON_MAJOR}" == 2 ]]; then
+ EXCLUDED_PY3_FILES=$(find ${MODULE} | grep 'py3[0-9]*\.py$')
+ echo -e "Excluding Py3 files:\n${EXCLUDED_PY3_FILES}"
+else
+ EXCLUDED_PY3_FILES=""
+fi
+
FILES_TO_IGNORE=""
-for file in "${EXCLUDED_GENERATED_FILES[@]}"; do
+for file in "${EXCLUDED_GENERATED_FILES[@]}" ${EXCLUDED_PY3_FILES}; do
if test -z "$FILES_TO_IGNORE"
then FILES_TO_IGNORE="$(basename $file)"
else FILES_TO_IGNORE="$FILES_TO_IGNORE, $(basename $file)"
fi
done
-echo "Skipping lint for generated files: $FILES_TO_IGNORE"
-echo "Running pylint for module $MODULE:"
+echo -e "Skipping lint for files:\n${FILES_TO_IGNORE}"
+echo -e "Linting modules:\n${MODULE}"
+
+echo "Running pylint..."
pylint -j8 ${MODULE} --ignore-patterns="$FILES_TO_IGNORE"
-echo "Running pycodestyle for module $MODULE:"
+echo "Running pycodestyle..."
pycodestyle ${MODULE} --exclude="$FILES_TO_IGNORE"
-echo "Running flake8 for module $MODULE:"
-# TODO(BEAM-3959): Add F821 (undefined names) as soon as that test passes
-flake8 ${MODULE} --count --select=E9,F822,F823 --show-source --statistics
+echo "Running flake8..."
+flake8 ${MODULE} --count --select=E9,F821,F822,F823 --show-source --statistics \
+ --exclude="${FILES_TO_IGNORE}"
-echo "Running isort for module $MODULE:"
+echo "Running isort..."
# Skip files where isort is behaving weirdly
ISORT_EXCLUDED=(
"apiclient.py"
@@ -89,6 +99,11 @@
"fast_coders_test.py"
"slow_coders_test.py"
"vcfio.py"
+ "tfdv_analyze_and_validate.py"
+ "preprocess.py"
+ "model.py"
+ "taxi.py"
+ "process_tfma.py"
)
SKIP_PARAM=""
for file in "${ISORT_EXCLUDED[@]}"; do
@@ -100,8 +115,13 @@
isort ${MODULE} -p apache_beam --line-width 120 --check-only --order-by-type \
--combine-star --force-single-line-imports --diff --recursive ${SKIP_PARAM}
-echo "Checking unittest.main for module ${MODULE}:"
-TESTS_MISSING_MAIN=$(find ${MODULE} | grep '\.py$' | xargs grep -l '^import unittest$' | xargs grep -L unittest.main)
+echo "Checking unittest.main..."
+TESTS_MISSING_MAIN=$(
+ find ${MODULE} \
+ | grep '\.py$' \
+ | xargs grep -l '^import unittest$' \
+ | xargs grep -L unittest.main \
+ || true)
if [ -n "${TESTS_MISSING_MAIN}" ]; then
echo -e "\nThe following files are missing a call to unittest.main():"
for FILE in ${TESTS_MISSING_MAIN}; do
diff --git a/sdks/python/setup.cfg b/sdks/python/setup.cfg
index 69a5187..361fe14 100644
--- a/sdks/python/setup.cfg
+++ b/sdks/python/setup.cfg
@@ -51,3 +51,6 @@
[coverage:xml]
output = target/site/cobertura/coverage.xml
+
+[isort]
+known_standard_library = dataclasses
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 7915971..3e42485 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -24,6 +24,7 @@
import platform
import sys
import warnings
+from distutils import log
from distutils.version import StrictVersion
# Pylint and isort disagree here.
@@ -34,7 +35,6 @@
from setuptools.command.build_py import build_py
from setuptools.command.develop import develop
from setuptools.command.egg_info import egg_info
-from setuptools.command.sdist import sdist
from setuptools.command.test import test
@@ -105,34 +105,44 @@
'avro>=1.8.1,<2.0.0; python_version < "3.0"',
'avro-python3>=1.8.1,<2.0.0; python_version >= "3.0"',
'crcmod>=1.7,<2.0',
- 'dill>=0.2.9,<0.2.10',
+ # Dill doesn't guarantee comatibility between releases within minor version.
+ 'dill>=0.3.0,<0.3.1',
'fastavro>=0.21.4,<0.22',
+ 'funcsigs>=1.0.2,<2; python_version < "3.0"',
'future>=0.16.0,<1.0.0',
'futures>=3.2.0,<4.0.0; python_version < "3.0"',
- 'grpcio>=1.8,<2',
+ 'grpcio>=1.12.1,<2',
'hdfs>=2.1.0,<3.0.0',
'httplib2>=0.8,<=0.12.0',
'mock>=1.0.1,<3.0.0',
+ 'pymongo>=3.8.0,<4.0.0',
'oauth2client>=2.0.1,<4',
- # grpcio 1.8.1 and above requires protobuf 3.5.0.post1.
'protobuf>=3.5.0.post1,<4',
# [BEAM-6287] pyarrow is not supported on Windows for Python 2
- ('pyarrow>=0.11.1,<0.14.0; python_version >= "3.0" or '
+ ('pyarrow>=0.11.1,<0.15.0; python_version >= "3.0" or '
'platform_system != "Windows"'),
- 'pydot>=1.2.0,<1.3',
+ 'pydot>=1.2.0,<2',
+ 'python-dateutil>=2.8.0,<3',
'pytz>=2018.3',
# [BEAM-5628] Beam VCF IO is not supported in Python 3.
'pyvcf>=0.6.8,<0.7.0; python_version < "3.0"',
- 'pyyaml>=3.12,<4.0.0',
'typing>=3.6.0,<3.7.0; python_version < "3.5.0"',
]
+# [BEAM-8181] pyarrow cannot be installed on 32-bit Windows platforms.
+if sys.platform == 'win32' and sys.maxsize <= 2**32:
+ REQUIRED_PACKAGES = [
+ p for p in REQUIRED_PACKAGES if not p.startswith('pyarrow')
+ ]
+
REQUIRED_TEST_PACKAGES = [
'nose>=1.3.7',
+ 'nose_xunitmp>=0.4.1',
'numpy>=1.14.3,<2',
- 'pandas>=0.23.4,<0.24',
+ 'pandas>=0.23.4,<0.25',
'parameterized>=0.6.0,<0.7.0',
'pyhamcrest>=1.9,<2.0',
+ 'pyyaml>=3.12,<6.0.0',
'tenacity>=5.0.2,<6.0',
]
@@ -143,12 +153,12 @@
'proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4; python_version < "3.0"',
# [BEAM-4543] googledatastore is not supported in Python 3.
'googledatastore>=7.0.1,<7.1; python_version < "3.0"',
- 'google-cloud-datastore>=1.7.1,<2.0.0',
- 'google-cloud-pubsub>=0.39.0,<0.40.0',
+ 'google-cloud-datastore>=1.7.1,<1.8.0',
+ 'google-cloud-pubsub>=0.39.0,<1.1.0',
# GCP packages required by tests
- 'google-cloud-bigquery>=1.6.0,<1.7.0',
- 'google-cloud-core>=0.28.1,<0.30.0',
- 'google-cloud-bigtable>=0.31.1,<0.33.0',
+ 'google-cloud-bigquery>=1.6.0,<1.18.0',
+ 'google-cloud-core>=0.28.1,<2',
+ 'google-cloud-bigtable>=0.31.1,<1.1.0',
]
@@ -161,7 +171,7 @@
class cmd(original_cmd, object):
def run(self):
- gen_protos.generate_proto_files()
+ gen_protos.generate_proto_files(log=log)
super(cmd, self).run()
return cmd
except ImportError:
@@ -171,10 +181,10 @@
python_requires = '>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'
-if sys.version_info[0] == 3:
+if sys.version_info[0] == 2:
warnings.warn(
- 'Python 3 support for the Apache Beam SDK is not yet fully supported. '
- 'You may encounter buggy behavior or missing features.')
+ 'You are using Apache Beam with Python 2. '
+ 'New releases of Apache Beam will soon support Python 3 only.')
setuptools.setup(
name=PACKAGE_NAME,
@@ -218,6 +228,8 @@
'Operating System :: POSIX :: Linux',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.5',
+ 'Programming Language :: Python :: 3.6',
+ 'Programming Language :: Python :: 3.7',
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Python Modules',
],
@@ -231,7 +243,6 @@
'build_py': generate_protos_first(build_py),
'develop': generate_protos_first(develop),
'egg_info': generate_protos_first(egg_info),
- 'sdist': generate_protos_first(sdist),
'test': generate_protos_first(test),
},
)
diff --git a/sdks/python/test-suites/dataflow/build.gradle b/sdks/python/test-suites/dataflow/build.gradle
deleted file mode 100644
index fe21d90..0000000
--- a/sdks/python/test-suites/dataflow/build.gradle
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-plugins { id 'org.apache.beam.module' }
-applyPythonNature()
-
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
-
-task preCommitIT(dependsOn: ['sdist', 'installGcpTest']) {
- doLast {
- // Basic integration tests to run in PreCommit
- def precommitTests = [
- "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
- "apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
- ]
- def testOpts = [
- "--tests=${precommitTests.join(',')}",
- "--nocapture", // Print stdout instantly
- "--processes=2", // Number of tests running in parallel
- "--process-timeout=1800", // Timeout of whole command execution
- ]
- def cmdArgs = project.mapToArgString([
- "test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz"
- ])
-
- exec {
- executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
- }
- }
-}
diff --git a/sdks/python/test-suites/dataflow/py2/build.gradle b/sdks/python/test-suites/dataflow/py2/build.gradle
new file mode 100644
index 0000000..1f28959
--- /dev/null
+++ b/sdks/python/test-suites/dataflow/py2/build.gradle
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyPythonNature()
+enablePythonPerformanceTest()
+
+dependencies {
+ distTarBall project(path: ":sdks:python", configuration: "distTarBall")
+}
+
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
+
+// Basic test options for ITs running on Jenkins.
+def basicTestOpts = [
+ "--nocapture", // print stdout instantly
+ "--processes=8", // run tests in parallel
+ "--process-timeout=4500", // timeout of whole command execution
+]
+
+task preCommitIT {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ // Basic integration tests to run in PreCommit
+ def precommitTests = [
+ "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+ "apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+ ]
+ def testOpts = [
+ "--tests=${precommitTests.join(',')}",
+ "--nocapture", // Print stdout instantly
+ "--processes=2", // Number of tests running in parallel
+ "--process-timeout=1800", // Timeout of whole command execution
+ ]
+ def cmdArgs = mapToArgString([
+ "test_opts": testOpts,
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "preCommitIT-df"
+ ])
+
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+// Run PostCommit integration tests on default runner (TestDataflowRunner)
+task postCommitIT {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ def testOpts = basicTestOpts + ["--attr=IT"]
+ def cmdArgs = mapToArgString([
+ "test_opts": testOpts,
+ "worker_jar": dataflowWorkerJar,
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "suite": "postCommitIT-df"
+ ])
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task validatesRunnerBatchTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ def testOpts = basicTestOpts + ["--attr=ValidatesRunner"]
+ def cmdArgs = mapToArgString([
+ "test_opts": testOpts,
+ "worker_jar": dataflowWorkerJar,
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "suite": "validatesRunnerBatchTests-df"
+ ])
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task validatesRunnerStreamingTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ // TODO(BEAM-3544,BEAM-5025): Disable tests with 'sickbay-streaming' tag.
+ def testOpts = basicTestOpts + ["--attr=ValidatesRunner,!sickbay-streaming"]
+ def argMap = ["test_opts": testOpts,
+ "streaming": "true",
+ "worker_jar": dataflowWorkerJar,
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "suite": "validatesRunnerStreamingTests-df"]
+ def cmdArgs = mapToArgString(argMap)
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task dataflowChicagoTaxiExample {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+
+ def gcsRoot = findProperty('gcsRoot')
+ def runner = findProperty('runner')
+ def cliArgs = "${gcsRoot} ${runner} ${files(configurations.distTarBall.files).singleFile}"
+
+ doLast {
+ exec {
+ workingDir "$rootProject.projectDir/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/"
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && pip install -r requirements.txt"
+ }
+ exec {
+ workingDir "$rootProject.projectDir/sdks/python/apache_beam/testing/benchmarks/chicago_taxi/"
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ./run_chicago.sh ${cliArgs}"
+ }
+ }
+}
diff --git a/sdks/python/test-suites/dataflow/py35/build.gradle b/sdks/python/test-suites/dataflow/py35/build.gradle
index c5b21d2..08ba22d 100644
--- a/sdks/python/test-suites/dataflow/py35/build.gradle
+++ b/sdks/python/test-suites/dataflow/py35/build.gradle
@@ -20,10 +20,15 @@
applyPythonNature()
enablePythonPerformanceTest()
+dependencies {
+ distTarBall project(path: ":sdks:python", configuration: "distTarBall")
+}
+
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.5'
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
+
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
// Basic test options for ITs running on Jenkins.
def basicTestOpts = [
@@ -32,41 +37,70 @@
"--process-timeout=4500", // timeout of whole command execution
]
-task postCommitIT(dependsOn: ['sdist', 'installGcpTest']) {
- dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+task postCommitIT {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ':runners:google-cloud-dataflow-java:worker:shadowJar'
def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
doLast {
def testOpts = basicTestOpts + ["--attr=IT"]
- def cmdArgs = project.mapToArgString([
+ def cmdArgs = mapToArgString([
"test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz",
- "worker_jar": dataflowWorkerJar
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "postCommitIT-df-py35"
])
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
-task validatesRunnerBatchTests(dependsOn: ['installGcpTest', 'sdist']) {
+task validatesRunnerBatchTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
doLast {
def testOpts = basicTestOpts + ["--attr=ValidatesRunner"]
- def cmdArgs = project.mapToArgString([
+ def cmdArgs = mapToArgString([
"test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz",
- "worker_jar": dataflowWorkerJar
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "validatesRunnerBatchTests-df-py35"
])
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task validatesRunnerStreamingTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ // TODO(BEAM-3544,BEAM-5025): Disable tests with 'sickbay-streaming' tag.
+ def testOpts = basicTestOpts + ["--attr=ValidatesRunner,!sickbay-streaming"]
+ def argMap = ["test_opts": testOpts,
+ "streaming": "true",
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "validatesRunnerStreamingTests-df-py35"]
+ def cmdArgs = mapToArgString(argMap)
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
diff --git a/sdks/python/test-suites/dataflow/py36/build.gradle b/sdks/python/test-suites/dataflow/py36/build.gradle
index 4900d1c..3205318 100644
--- a/sdks/python/test-suites/dataflow/py36/build.gradle
+++ b/sdks/python/test-suites/dataflow/py36/build.gradle
@@ -18,11 +18,17 @@
apply plugin: org.apache.beam.gradle.BeamModulePlugin
applyPythonNature()
+enablePythonPerformanceTest()
+
+dependencies {
+ distTarBall project(path: ":sdks:python", configuration: "distTarBall")
+}
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.6'
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
+
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
// Basic test options for ITs running on Jenkins.
def basicTestOpts = [
@@ -31,7 +37,9 @@
"--process-timeout=4500", // timeout of whole command execution
]
-task postCommitIT(dependsOn: ['sdist', 'installGcpTest']) {
+task postCommitIT {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
@@ -39,33 +47,60 @@
doLast {
def testOpts = basicTestOpts + ["--attr=IT"]
- def cmdArgs = project.mapToArgString([
+ def cmdArgs = mapToArgString([
"test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz",
- "worker_jar": dataflowWorkerJar
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "postCommitIT-df-py36"
])
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
-task validatesRunnerBatchTests(dependsOn: ['installGcpTest', 'sdist']) {
+task validatesRunnerBatchTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
doLast {
def testOpts = basicTestOpts + ["--attr=ValidatesRunner"]
- def cmdArgs = project.mapToArgString([
+ def cmdArgs = mapToArgString([
"test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz",
- "worker_jar": dataflowWorkerJar
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "validatesRunnerBatchTests-df-py36"
])
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task validatesRunnerStreamingTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ // TODO(BEAM-3544,BEAM-5025): Disable tests with 'sickbay-streaming' tag.
+ def testOpts = basicTestOpts + ["--attr=ValidatesRunner,!sickbay-streaming"]
+ def argMap = ["test_opts": testOpts,
+ "streaming": "true",
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "validatesRunnerStreamingTests-df-py36"]
+ def cmdArgs = mapToArgString(argMap)
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
diff --git a/sdks/python/test-suites/dataflow/py37/build.gradle b/sdks/python/test-suites/dataflow/py37/build.gradle
index 4d7aefc..844c60a 100644
--- a/sdks/python/test-suites/dataflow/py37/build.gradle
+++ b/sdks/python/test-suites/dataflow/py37/build.gradle
@@ -18,11 +18,17 @@
apply plugin: org.apache.beam.gradle.BeamModulePlugin
applyPythonNature()
+enablePythonPerformanceTest()
+
+dependencies {
+ distTarBall project(path: ":sdks:python", configuration: "distTarBall")
+}
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.7'
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
+
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
// Basic test options for ITs running on Jenkins.
def basicTestOpts = [
@@ -31,7 +37,42 @@
"--process-timeout=4500", // timeout of whole command execution
]
-task postCommitIT(dependsOn: ['sdist', 'installGcpTest']) {
+task preCommitIT {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ // Basic integration tests to run in PreCommit
+ def precommitTests = [
+ "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+ "apache_beam.examples.streaming_wordcount_it_test:StreamingWordCountIT.test_streaming_wordcount_it",
+ ]
+ def testOpts = [
+ "--tests=${precommitTests.join(',')}",
+ "--nocapture", // Print stdout instantly
+ "--processes=2", // Number of tests running in parallel
+ "--process-timeout=1800", // Timeout of whole command execution
+ ]
+ def cmdArgs = mapToArgString([
+ "test_opts": testOpts,
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "preCommitIT-df-py37"
+ ])
+
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task postCommitIT {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
@@ -39,33 +80,60 @@
doLast {
def testOpts = basicTestOpts + ["--attr=IT"]
- def cmdArgs = project.mapToArgString([
+ def cmdArgs = mapToArgString([
"test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz",
- "worker_jar": dataflowWorkerJar
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "postCommitIT-df-py37"
])
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
-task validatesRunnerBatchTests(dependsOn: ['installGcpTest', 'sdist']) {
+task validatesRunnerBatchTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
doLast {
def testOpts = basicTestOpts + ["--attr=ValidatesRunner"]
- def cmdArgs = project.mapToArgString([
+ def cmdArgs = mapToArgString([
"test_opts": testOpts,
- "sdk_location": "${project.buildDir}/apache-beam.tar.gz",
- "worker_jar": dataflowWorkerJar
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "validatesRunnerBatchTests-df-py37"
])
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
+ }
+ }
+}
+
+task validatesRunnerStreamingTests {
+ dependsOn 'installGcpTest'
+ dependsOn ':sdks:python:sdist'
+ dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
+
+ def dataflowWorkerJar = project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
+
+ doLast {
+ // TODO(BEAM-3544,BEAM-5025): Disable tests with 'sickbay-streaming' tag.
+ def testOpts = basicTestOpts + ["--attr=ValidatesRunner,!sickbay-streaming"]
+ def argMap = ["test_opts": testOpts,
+ "streaming": "true",
+ "sdk_location": files(configurations.distTarBall.files).singleFile,
+ "worker_jar": dataflowWorkerJar,
+ "suite": "validatesRunnerStreamingTests-df-py37"]
+ def cmdArgs = mapToArgString(argMap)
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
diff --git a/sdks/python/test-suites/direct/py2/build.gradle b/sdks/python/test-suites/direct/py2/build.gradle
new file mode 100644
index 0000000..e422b74
--- /dev/null
+++ b/sdks/python/test-suites/direct/py2/build.gradle
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyPythonNature()
+
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
+
+// Basic test options for ITs running on Jenkins.
+def basicTestOpts = [
+ "--nocapture", // print stdout instantly
+ "--processes=8", // run tests in parallel
+ "--process-timeout=4500", // timeout of whole command execution
+]
+
+task directRunnerIT {
+ dependsOn 'installGcpTest'
+
+ // Run IT tests with TestDirectRunner in batch.
+ doLast {
+ def tests = [
+ "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+ "apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
+ "apache_beam.io.gcp.big_query_query_to_table_it_test:BigQueryQueryToTableIT",
+ "apache_beam.io.gcp.bigquery_io_read_it_test",
+ "apache_beam.io.gcp.datastore.v1new.datastore_write_it_test",
+ ]
+ def batchTestOpts = basicTestOpts + ["--tests=${tests.join(',')}"]
+ def argMap = ["runner": "TestDirectRunner",
+ "test_opts": batchTestOpts,
+ "suite": "directRunnerIT-batch"]
+ def batchCmdArgs = mapToArgString(argMap)
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $batchCmdArgs"
+ }
+ }
+
+ // Run IT tests with TestDirectRunner in streaming.
+ doLast {
+ def tests = [
+ "apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it",
+ "apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
+ "apache_beam.io.gcp.bigquery_test:BigQueryStreamingInsertTransformIntegrationTests\
+.test_multiple_destinations_transform",
+ "apache_beam.io.gcp.bigquery_test:PubSubBigQueryIT",
+ "apache_beam.io.gcp.bigquery_file_loads_test:BigQueryFileLoadsIT.test_bqfl_streaming",
+ ]
+ def streamingTestOpts = basicTestOpts + ["--tests=${tests.join(',')}"]
+ def argMap = ["runner": "TestDirectRunner",
+ "streaming": "true",
+ "test_opts": streamingTestOpts,
+ "suite": "directRunnerIT-streaming"]
+ def streamingCmdArgs = mapToArgString(argMap)
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $streamingCmdArgs"
+ }
+ }
+}
+
+task hdfsIntegrationTest {
+ dependsOn 'installGcpTest'
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${rootDir}/sdks/python/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh python:2"
+ }
+ }
+}
+
+task mongodbioIT {
+ dependsOn 'setupVirtualenv'
+
+ Random r = new Random()
+ def port = r.nextInt(1000) + 27017
+ def containerName = "mongoioit" + port
+
+ def options = [
+ "--mongo_uri=mongodb://localhost:" + port
+ ]
+
+ // Pull the latest mongodb docker image and run
+ doFirst {
+ exec {
+ executable 'sh'
+ args '-c', "docker pull mongo && docker run --name ${containerName} -p ${port}:27017 -d mongo:latest"
+ }
+ }
+
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && pip install -e ${rootDir}/sdks/python/[test] && python -m apache_beam.io.mongodbio_it_test ${options.join(' ')}"
+ }
+ exec {
+ executable 'sh'
+ args '-c', "docker stop ${containerName} && docker rm ${containerName}"
+ }
+ }
+}
+
+
diff --git a/sdks/python/test-suites/direct/py35/build.gradle b/sdks/python/test-suites/direct/py35/build.gradle
index 56b77a3..c8b672d 100644
--- a/sdks/python/test-suites/direct/py35/build.gradle
+++ b/sdks/python/test-suites/direct/py35/build.gradle
@@ -22,9 +22,11 @@
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.5'
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
-task postCommitIT(dependsOn: 'installGcpTest') {
+task postCommitIT {
+ dependsOn 'installGcpTest'
+
// Run IT tests with TestDirectRunner in batch in Python 3.
doLast {
def batchTests = [
@@ -32,6 +34,8 @@
"apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
"apache_beam.io.gcp.big_query_query_to_table_it_test:BigQueryQueryToTableIT",
"apache_beam.io.gcp.bigquery_io_read_it_test",
+ "apache_beam.io.gcp.bigquery_read_it_test",
+ "apache_beam.io.gcp.bigquery_write_it_test",
"apache_beam.io.gcp.datastore.v1new.datastore_write_it_test",
]
def testOpts = [
@@ -41,11 +45,43 @@
"--process-timeout=4500", // timeout of whole command execution
]
def argMap = ["runner": "TestDirectRunner",
- "test_opts": testOpts]
- def batchCmdArgs = project.mapToArgString(argMap)
+ "test_opts": testOpts,
+ "suite": "postCommitIT-direct-py35"]
+ def batchCmdArgs = mapToArgString(argMap)
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $batchCmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $batchCmdArgs"
+ }
+ }
+}
+
+task mongodbioIT {
+ dependsOn 'setupVirtualenv'
+
+ Random r = new Random()
+ def port = r.nextInt(1000) + 27017
+ def containerName = "mongoioit" + port
+
+ def options = [
+ "--mongo_uri=mongodb://localhost:" + port
+ ]
+
+ // Pull the latest mongodb docker image and run
+ doFirst {
+ exec {
+ executable 'sh'
+ args '-c', "docker pull mongo && docker run --name ${containerName} -p ${port}:27017 -d mongo:latest"
+ }
+ }
+
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && pip install -e ${rootDir}/sdks/python/[test] && python -m apache_beam.io.mongodbio_it_test ${options.join(' ')}"
+ }
+ exec {
+ executable 'sh'
+ args '-c', "docker stop ${containerName} && docker rm ${containerName}"
}
}
}
diff --git a/sdks/python/test-suites/direct/py36/build.gradle b/sdks/python/test-suites/direct/py36/build.gradle
index a2c4311..b1bfaa2 100644
--- a/sdks/python/test-suites/direct/py36/build.gradle
+++ b/sdks/python/test-suites/direct/py36/build.gradle
@@ -22,9 +22,11 @@
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.6'
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
+def runScriptsDir = "${rootDir}/sdks/python/scripts"
-task postCommitIT(dependsOn: 'installGcpTest') {
+task postCommitIT {
+ dependsOn 'installGcpTest'
+
// Run IT tests with TestDirectRunner in batch in Python 3.
doLast {
def batchTests = [
@@ -32,6 +34,8 @@
"apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
"apache_beam.io.gcp.big_query_query_to_table_it_test:BigQueryQueryToTableIT",
"apache_beam.io.gcp.bigquery_io_read_it_test",
+ "apache_beam.io.gcp.bigquery_read_it_test",
+ "apache_beam.io.gcp.bigquery_write_it_test",
"apache_beam.io.gcp.datastore.v1new.datastore_write_it_test",
]
def testOpts = [
@@ -41,11 +45,12 @@
"--process-timeout=4500", // timeout of whole command execution
]
def argMap = ["runner": "TestDirectRunner",
- "test_opts": testOpts]
- def batchCmdArgs = project.mapToArgString(argMap)
+ "test_opts": testOpts,
+ "suite": "postCommitIT-direct-py36"]
+ def batchCmdArgs = mapToArgString(argMap)
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $batchCmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $batchCmdArgs"
}
}
}
diff --git a/sdks/python/test-suites/direct/py37/build.gradle b/sdks/python/test-suites/direct/py37/build.gradle
index 8d78627..c5c30fb 100644
--- a/sdks/python/test-suites/direct/py37/build.gradle
+++ b/sdks/python/test-suites/direct/py37/build.gradle
@@ -22,9 +22,11 @@
// Required to setup a Python 3 virtualenv.
pythonVersion = '3.7'
-def runScriptsDir = "${project.rootDir}/sdks/python/scripts"
+def pythonDir = "${rootDir}/sdks/python"
-task postCommitIT(dependsOn: 'installGcpTest') {
+task postCommitIT {
+ dependsOn 'installGcpTest'
+
// Run IT tests with TestDirectRunner in batch in Python 3.
doLast {
def batchTests = [
@@ -32,6 +34,8 @@
"apache_beam.io.gcp.pubsub_integration_test:PubSubIntegrationTest",
"apache_beam.io.gcp.big_query_query_to_table_it_test:BigQueryQueryToTableIT",
"apache_beam.io.gcp.bigquery_io_read_it_test",
+ "apache_beam.io.gcp.bigquery_read_it_test",
+ "apache_beam.io.gcp.bigquery_write_it_test",
"apache_beam.io.gcp.datastore.v1new.datastore_write_it_test",
]
def testOpts = [
@@ -41,11 +45,22 @@
"--process-timeout=4500", // timeout of whole command execution
]
def argMap = ["runner": "TestDirectRunner",
- "test_opts": testOpts]
- def batchCmdArgs = project.mapToArgString(argMap)
+ "test_opts": testOpts,
+ "suite": "postCommitIT-direct-py37"]
+ def batchCmdArgs = mapToArgString(argMap)
exec {
executable 'sh'
- args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $batchCmdArgs"
+ args '-c', ". ${envdir}/bin/activate && ${pythonDir}/scripts/run_integration_test.sh $batchCmdArgs"
+ }
+ }
+}
+
+task hdfsIntegrationTest {
+ dependsOn 'installGcpTest'
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && ${pythonDir}/apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh python:3.7"
}
}
}
diff --git a/sdks/python/test-suites/portable/common.gradle b/sdks/python/test-suites/portable/common.gradle
new file mode 100644
index 0000000..690c09d
--- /dev/null
+++ b/sdks/python/test-suites/portable/common.gradle
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def pythonRootDir = "${rootDir}/sdks/python"
+def pythonContainerSuffix = project.ext.pythonVersion == '2.7' ? '2' : project.ext.pythonVersion.replace('.', '')
+def pythonContainerTask = ":sdks:python:container:py${pythonContainerSuffix}:docker"
+
+class CompatibilityMatrixConfig {
+ // Execute batch or streaming pipelines.
+ boolean streaming = false
+ // Execute on Docker or Process based environment.
+ SDK_WORKER_TYPE workerType = SDK_WORKER_TYPE.DOCKER
+
+ enum SDK_WORKER_TYPE {
+ DOCKER, PROCESS, LOOPBACK
+ }
+
+ // Whether to pre-optimize the pipeline with the Python optimizer.
+ boolean preOptimize = false
+}
+
+def flinkCompatibilityMatrix = {
+ def config = it ? it as CompatibilityMatrixConfig : new CompatibilityMatrixConfig()
+ def workerType = config.workerType.name()
+ def streaming = config.streaming
+ def environment_config = config.workerType == CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS ? "--environment_config='{\"command\": \"${buildDir.absolutePath}/sdk_worker.sh\"}'" : ""
+ def name = "flinkCompatibilityMatrix${streaming ? 'Streaming' : 'Batch'}${config.preOptimize ? 'PreOptimize' : ''}${workerType}"
+ def extra_experiments = []
+ if (config.preOptimize)
+ extra_experiments.add('pre_optimize=all')
+ tasks.create(name: name) {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.8:job-server:shadowJar'
+ if (workerType.toLowerCase() == 'docker')
+ dependsOn pythonContainerTask
+ else if (workerType.toLowerCase() == 'process')
+ dependsOn 'createProcessWorker'
+ doLast {
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.runners.portability.flink_runner_test --flink_job_server_jar=${project(":runners:flink:1.8:job-server:").shadowJar.archivePath} --environment_type=${workerType} ${environment_config} ${streaming ? '--streaming' : ''} ${extra_experiments ? '--extra_experiments=' + extra_experiments.join(',') : ''}"
+ }
+ }
+ }
+}
+
+task flinkCompatibilityMatrixDocker() {
+ dependsOn flinkCompatibilityMatrix(streaming: false)
+ dependsOn flinkCompatibilityMatrix(streaming: true)
+}
+
+task flinkCompatibilityMatrixProcess() {
+ dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
+ dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.PROCESS)
+}
+
+task flinkCompatibilityMatrixLoopback() {
+ dependsOn flinkCompatibilityMatrix(streaming: false, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
+ dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK)
+ dependsOn flinkCompatibilityMatrix(streaming: true, workerType: CompatibilityMatrixConfig.SDK_WORKER_TYPE.LOOPBACK, preOptimize: true)
+}
+
+task flinkValidatesRunner() {
+ dependsOn 'flinkCompatibilityMatrixLoopback'
+}
diff --git a/sdks/python/test-suites/portable/py2/build.gradle b/sdks/python/test-suites/portable/py2/build.gradle
new file mode 100644
index 0000000..5ceac52
--- /dev/null
+++ b/sdks/python/test-suites/portable/py2/build.gradle
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.tools.ant.taskdefs.condition.Os
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyPythonNature()
+
+def pythonRootDir = "${rootDir}/sdks/python"
+
+/*************************************************************************************************/
+
+addPortableWordCountTasks()
+
+task preCommitPy2() {
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ dependsOn ':sdks:python:container:py2:docker'
+ dependsOn portableWordCountBatch
+ dependsOn portableWordCountStreaming
+}
+
+// TODO: Move the rest of this file into ../common.gradle.
+
+// Before running this, you need to:
+//
+// 1. Build the SDK container:
+//
+// ./gradlew -p sdks/python/container buildAll
+//
+// 2. Either a) or b)
+// a) If you want the Job Server to run in a Docker container:
+//
+// ./gradlew :runners:flink:1.8:job-server-container:docker
+//
+// b) Otherwise, start a local JobService, for example, the Portable Flink runner
+// (in a separate shell since it continues to run):
+//
+// ./gradlew :runners:flink:1.8:job-server:runShadow
+//
+// Then you can run this example:
+//
+// Docker (2a):
+//
+// ./gradlew :sdks:python:test-suites:portable:py2:portableWordCount
+//
+// Local JobService (2b):
+//
+// ./gradlew :sdks:python:test-suites:portable:py2:portableWordCount -PjobEndpoint=localhost:8099
+//
+task portableWordCount {
+ dependsOn project.hasProperty("streaming") ? portableWordCountStreaming : portableWordCountBatch
+}
+
+/*************************************************************************************************/
+
+task crossLanguagePythonJavaDirect {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':sdks:java:container:docker'
+ dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+
+ doLast {
+ def options = [
+ "--expansion_service_target=sdks:java:testing:expansion-service:buildTestExpansionServiceJar",
+ "--expansion_service_target_appendix=testExpansionService",
+ ]
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.transforms.external_test ${options.join(' ')}"
+ }
+ }
+}
+
+task crossLanguagePythonJavaFlink {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ dependsOn ':sdks:python:container:py2:docker'
+ dependsOn ':sdks:java:container:docker'
+ dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+
+ doLast {
+ def testServiceExpansionJar = project(":sdks:java:testing:expansion-service:").buildTestExpansionServiceJar.archivePath
+ def options = [
+ "--runner=PortableRunner",
+ "--experiments=worker_threads=100",
+ "--parallelism=2",
+ "--shutdown_sources_on_final_watermark",
+ "--environment_cache_millis=10000",
+ "--expansion_service_port=8096",
+ "--expansion_service_jar=${testServiceExpansionJar}",
+ ]
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.transforms.external_test ${options.join(' ')}"
+ }
+ }
+}
+
+task crossLanguagePortableWordCount {
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ dependsOn ':sdks:python:container:py2:docker'
+ dependsOn ':sdks:java:container:docker'
+ dependsOn ':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+
+ doLast {
+ def testServiceExpansionJar = project(":sdks:java:testing:expansion-service:").buildTestExpansionServiceJar.archivePath
+ def options = [
+ "--input=/etc/profile",
+ "--output=/tmp/py-wordcount-portable",
+ "--runner=PortableRunner",
+ "--experiments=worker_threads=100",
+ "--parallelism=2",
+ "--shutdown_sources_on_final_watermark",
+ "--environment_cache_millis=10000",
+ "--expansion_service_jar=${testServiceExpansionJar}",
+ ]
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.examples.wordcount_xlang ${options.join(' ')}"
+ // TODO: Check that the output file is generated and runs.
+ }
+ }
+}
+
+task crossLanguageTests {
+ dependsOn "crossLanguagePythonJavaFlink"
+ dependsOn "crossLanguagePortableWordCount"
+}
+
+/*************************************************************************************************/
+
+task createProcessWorker {
+ dependsOn ':sdks:python:container:build'
+ dependsOn 'setupVirtualenv'
+ def sdkWorkerFile = file("${buildDir}/sdk_worker.sh")
+ def osType = 'linux'
+ if (Os.isFamily(Os.FAMILY_MAC))
+ osType = 'darwin'
+ def workerScript = "${project(":sdks:python:container:").buildDir.absolutePath}/target/launcher/${osType}_amd64/boot"
+ def sdkWorkerFileCode = "sh -c \"pip=`which pip` . ${envdir}/bin/activate && ${workerScript} \$* \""
+ outputs.file sdkWorkerFile
+ doLast {
+ sdkWorkerFile.write sdkWorkerFileCode
+ exec {
+ commandLine('sh', '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test]")
+ }
+ exec {
+ commandLine('chmod', '+x', sdkWorkerFile)
+ }
+ }
+}
+
+task sparkValidatesRunner() {
+ dependsOn 'createProcessWorker'
+ dependsOn 'setupVirtualenv'
+ dependsOn ':runners:spark:job-server:shadowJar'
+ doLast {
+ def environment_config = "'{\"command\": \"${buildDir.absolutePath}/sdk_worker.sh\"}'"
+ def argMap = [
+ "environment_type" : "PROCESS",
+ "spark_job_server_jar": project(":runners:spark:job-server:").shadowJar.archivePath,
+ "environment_config": environment_config,
+ ]
+ def argString = mapToArgString(argMap)
+
+ // Optionally specify test function names separated by space e.g.:
+ // ./gradlew :sdks:python:test-suites:portable:py2:sparkValidatesRunner -Ptests="test_external_transforms test_read"
+ // Otherwise run all test functions under SparkRunnerTest
+ def tests = project.hasProperty('tests') ?
+ project.property('tests').split().collect{ "SparkRunnerTest.$it" }.join(' ') : ''
+
+ exec {
+ executable 'sh'
+ args '-c', ". ${envdir}/bin/activate && cd ${pythonRootDir} && pip install -e .[test] && python -m apache_beam.runners.portability.spark_runner_test $tests $argString"
+ }
+ }
+}
+
+apply from: "../common.gradle"
diff --git a/sdks/python/test-suites/portable/py35/build.gradle b/sdks/python/test-suites/portable/py35/build.gradle
new file mode 100644
index 0000000..b0d670c
--- /dev/null
+++ b/sdks/python/test-suites/portable/py35/build.gradle
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyPythonNature()
+// Required to setup a Python 3.5 virtualenv.
+pythonVersion = '3.5'
+apply from: "../common.gradle"
+
+addPortableWordCountTasks()
+
+task preCommitPy35() {
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ dependsOn ':sdks:python:container:py35:docker'
+ dependsOn portableWordCountBatch
+ dependsOn portableWordCountStreaming
+}
diff --git a/sdks/python/test-suites/portable/py36/build.gradle b/sdks/python/test-suites/portable/py36/build.gradle
new file mode 100644
index 0000000..70fbdce
--- /dev/null
+++ b/sdks/python/test-suites/portable/py36/build.gradle
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyPythonNature()
+// Required to setup a Python 3.6 virtualenv.
+pythonVersion = '3.6'
+apply from: "../common.gradle"
+
+addPortableWordCountTasks()
+
+task preCommitPy36() {
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ dependsOn ':sdks:python:container:py36:docker'
+ dependsOn portableWordCountBatch
+ dependsOn portableWordCountStreaming
+}
diff --git a/sdks/python/test-suites/portable/py37/build.gradle b/sdks/python/test-suites/portable/py37/build.gradle
new file mode 100644
index 0000000..fa2ead2
--- /dev/null
+++ b/sdks/python/test-suites/portable/py37/build.gradle
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyPythonNature()
+// Required to setup a Python 3.7 virtualenv.
+pythonVersion = '3.7'
+apply from: "../common.gradle"
+
+addPortableWordCountTasks()
+
+task preCommitPy37() {
+ dependsOn ':runners:flink:1.8:job-server-container:docker'
+ dependsOn ':sdks:python:container:py37:docker'
+ dependsOn portableWordCountBatch
+ dependsOn portableWordCountStreaming
+}
diff --git a/sdks/python/test-suites/tox/py2/build.gradle b/sdks/python/test-suites/tox/py2/build.gradle
new file mode 100644
index 0000000..867c33e
--- /dev/null
+++ b/sdks/python/test-suites/tox/py2/build.gradle
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Unit tests for Python 2
+ */
+
+plugins { id 'org.apache.beam.module' }
+applyPythonNature()
+
+task lint {}
+check.dependsOn lint
+
+toxTask "lintPy27", "py27-lint"
+lint.dependsOn lintPy27
+
+toxTask "lintPy27_3", "py27-lint3"
+lint.dependsOn lintPy27_3
+
+toxTask "testPy2Gcp", "py27-gcp"
+test.dependsOn testPy2Gcp
+
+toxTask "testPython2", "py27"
+test.dependsOn testPython2
+
+toxTask "testPy2Cython", "py27-cython"
+test.dependsOn testPy2Cython
+// Ensure that testPy2Cython runs exclusively to other tests. This line is not
+// actually required, since gradle doesn't do parallel execution within a
+// project.
+testPy2Cython.mustRunAfter testPython2, testPy2Gcp
+
+toxTask "docs", "docs"
+assemble.dependsOn docs
+
+toxTask "cover", "cover"
+
+task preCommitPy2() {
+ dependsOn "docs"
+ dependsOn "testPy2Cython"
+ dependsOn "testPython2"
+ dependsOn "testPy2Gcp"
+}
diff --git a/sdks/python/test-suites/tox/py35/build.gradle b/sdks/python/test-suites/tox/py35/build.gradle
index ca3d37c..62cf4bd 100644
--- a/sdks/python/test-suites/tox/py35/build.gradle
+++ b/sdks/python/test-suites/tox/py35/build.gradle
@@ -49,5 +49,4 @@
dependsOn "testPython35"
dependsOn "testPy35Gcp"
dependsOn "testPy35Cython"
- dependsOn "lint"
}
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 2a221b0..f3ee356 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -31,7 +31,7 @@
extras = test
# Don't warn that these commands aren't installed.
whitelist_externals =
- find
+ false
time
deps =
cython: cython==0.28.1
@@ -50,32 +50,33 @@
{toxinidir}/scripts/run_tox_cleanup.sh
commands_post =
{toxinidir}/scripts/run_tox_cleanup.sh
+commands = false {envname} is misconfigured
[testenv:py27]
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3\d?\.py$' {posargs}
[testenv:py35]
setenv =
RUN_SKIPPED_PY3_TESTS=0
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[6-9]\.py$' {posargs}
[testenv:py36]
setenv =
RUN_SKIPPED_PY3_TESTS=0
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[7-9]\.py$' {posargs}
[testenv:py37]
setenv =
RUN_SKIPPED_PY3_TESTS=0
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
[testenv:py27-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
@@ -85,7 +86,7 @@
platform = linux2
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3\d?\.py$' {posargs}
[testenv:py35-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
@@ -97,7 +98,7 @@
RUN_SKIPPED_PY3_TESTS=0
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[5-9]\.py$' {posargs}
[testenv:py36-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
@@ -109,7 +110,7 @@
RUN_SKIPPED_PY3_TESTS=0
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[7-9]\.py$' {posargs}
[testenv:py37-cython]
# cython tests are only expected to work in linux (2.x and 3.x)
@@ -121,33 +122,40 @@
RUN_SKIPPED_PY3_TESTS=0
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
[testenv:py27-gcp]
extras = test,gcp
commands =
python apache_beam/examples/complete/autocomplete_test.py
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3\d?\.py$' {posargs}
# Old and new Datastore client unit tests cannot be run in the same process
# due to conflicting protobuf modules.
# TODO(BEAM-4543): Remove these separate nosetests invocations once the
# googledatastore dependency is removed.
- python setup.py nosetests --tests apache_beam.io.gcp.datastore.v1
- python setup.py nosetests --tests apache_beam.io.gcp.datastore.v1new
+ python setup.py nosetests {posargs} --tests apache_beam.io.gcp.datastore.v1
+ python setup.py nosetests {posargs} --tests apache_beam.io.gcp.datastore.v1new
[testenv:py35-gcp]
setenv =
RUN_SKIPPED_PY3_TESTS=0
extras = test,gcp
commands =
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[6-9]\.py$' {posargs}
+
+[testenv:py36-gcp]
+setenv =
+ RUN_SKIPPED_PY3_TESTS=0
+extras = test,gcp
+commands =
+ python setup.py nosetests --ignore-files '.*py3[7-9]\.py$' {posargs}
[testenv:py37-gcp]
setenv =
RUN_SKIPPED_PY3_TESTS=0
extras = test,gcp
commands =
- python setup.py nosetests
+ python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
[testenv:py27-lint]
deps =
diff --git a/settings.gradle b/settings.gradle
index d67f31b..7a03955 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -31,14 +31,6 @@
include ":runners:direct-java"
include ":runners:extensions-java:metrics"
/* Begin Flink Runner related settings */
-// Flink 1.5 (with Scala 2.11 suffix)
-include ":runners:flink:1.5"
-include ":runners:flink:1.5:job-server"
-include ":runners:flink:1.5:job-server-container"
-// Flink 1.6
-include ":runners:flink:1.6"
-include ":runners:flink:1.6:job-server"
-include ":runners:flink:1.6:job-server-container"
// Flink 1.7
include ":runners:flink:1.7"
include ":runners:flink:1.7:job-server"
@@ -53,10 +45,9 @@
include ":runners:google-cloud-dataflow-java:examples"
include ":runners:google-cloud-dataflow-java:examples-streaming"
include ":runners:java-fn-execution"
-include ":runners:jet-experimental"
+include ":runners:jet"
include ":runners:local-java"
include ":runners:reference:java"
-include ":runners:reference:job-server"
include ":runners:spark"
include ":runners:spark:job-server"
include ":runners:samza"
@@ -82,9 +73,12 @@
include ":sdks:java:extensions:sql:shell"
include ":sdks:java:extensions:sql:hcatalog"
include ":sdks:java:extensions:sql:datacatalog"
+include ":sdks:java:extensions:sql:zetasql"
+include ":sdks:java:extensions:zetasketch"
include ":sdks:java:fn-execution"
include ":sdks:java:harness"
include ":sdks:java:io:amazon-web-services"
+include ":sdks:java:io:amazon-web-services2"
include ":sdks:java:io:amqp"
include ":sdks:java:io:cassandra"
include ":sdks:java:io:clickhouse"
@@ -95,6 +89,7 @@
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-6"
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common"
include ":sdks:java:io:file-based-io-tests"
+include ':sdks:java:io:bigquery-io-perf-tests'
include ":sdks:java:io:google-cloud-platform"
include ":sdks:java:io:hadoop-common"
include ":sdks:java:io:hadoop-file-system"
@@ -117,27 +112,39 @@
include ":sdks:java:io:synthetic"
include ":sdks:java:javadoc"
include ":sdks:java:testing:load-tests"
+include ":sdks:java:testing:test-utils"
include ":sdks:java:maven-archetypes:examples"
include ":sdks:java:maven-archetypes:starter"
include ":sdks:java:testing:nexmark"
+include ":sdks:java:testing:expansion-service"
include ":sdks:python"
include ":sdks:python:apache_beam:testing:load_tests"
include ":sdks:python:container"
-include ":sdks:python:container:py3"
-include ":sdks:python:test-suites:dataflow"
+include ":sdks:python:container:py2"
+include ":sdks:python:container:py35"
+include ":sdks:python:container:py36"
+include ":sdks:python:container:py37"
+include ":sdks:python:test-suites:dataflow:py2"
include ":sdks:python:test-suites:dataflow:py35"
include ":sdks:python:test-suites:dataflow:py36"
include ":sdks:python:test-suites:dataflow:py37"
+include ":sdks:python:test-suites:direct:py2"
include ":sdks:python:test-suites:direct:py35"
include ":sdks:python:test-suites:direct:py36"
include ":sdks:python:test-suites:direct:py37"
+include ":sdks:python:test-suites:portable:py2"
+include ":sdks:python:test-suites:portable:py35"
+include ":sdks:python:test-suites:portable:py36"
+include ":sdks:python:test-suites:portable:py37"
+include ":sdks:python:test-suites:tox:py2"
include ":sdks:python:test-suites:tox:py35"
include ":sdks:python:test-suites:tox:py36"
include ":sdks:python:test-suites:tox:py37"
-include ":vendor:grpc-1_13_1"
-include ":sdks:java:testing:test-utils"
+include ":vendor:grpc-1_21_0"
+include ":vendor:bytebuddy-1_9_3"
+include ":vendor:calcite-1_20_0"
+include ":vendor:guava-26_0-jre"
include ":vendor:sdks-java-extensions-protobuf"
-include ":vendor:guava-20_0"
include ":website"
include ":runners:google-cloud-dataflow-java:worker:legacy-worker"
include ":runners:google-cloud-dataflow-java:worker"
diff --git a/vendor/bytebuddy-1_9_3/build.gradle b/vendor/bytebuddy-1_9_3/build.gradle
new file mode 100644
index 0000000..73bf327
--- /dev/null
+++ b/vendor/bytebuddy-1_9_3/build.gradle
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.vendor-java' }
+
+description = "Apache Beam :: Vendored Dependencies :: ByteBuddy :: 1.9.3"
+
+group = "org.apache.beam"
+version = "0.1"
+
+vendorJava(
+ dependencies: ["net.bytebuddy:byte-buddy:1.9.3"],
+ relocations: [
+ "net.bytebuddy": "org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy"
+ ],
+ groupId: group,
+ artifactId: "beam-vendor-bytebuddy-1_9_3",
+ version: version
+)
diff --git a/vendor/calcite-1_20_0/build.gradle b/vendor/calcite-1_20_0/build.gradle
new file mode 100644
index 0000000..09d9fa6
--- /dev/null
+++ b/vendor/calcite-1_20_0/build.gradle
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.vendor-java' }
+
+description = "Apache Beam :: Vendored Dependencies :: Calcite 1.20.0"
+
+group = "org.apache.beam"
+version = "0.1"
+
+def calcite_version = "1.20.0"
+def avatica_version = "1.15.0"
+def prefix = "org.apache.beam.vendor.calcite.v1_20_0"
+
+List<String> packagesToRelocate = [
+ "com.esri",
+ "com.google.common",
+ "com.google.thirdparty",
+ "com.google.protobuf",
+ "com.fasterxml",
+ "com.jayway",
+ "com.yahoo",
+ "org.apache.calcite",
+ "org.apache.commons",
+ "org.apache.http",
+ "org.codehaus",
+ "org.pentaho",
+ "org.yaml"
+]
+
+vendorJava(
+ dependencies: [
+ "org.apache.calcite:calcite-core:$calcite_version",
+ "org.apache.calcite:calcite-linq4j:$calcite_version",
+ "org.apache.calcite.avatica:avatica-core:$avatica_version",
+ library.java.protobuf_java,
+ library.java.slf4j_api
+ ],
+ relocations: packagesToRelocate.collectEntries {
+ [ (it): "${prefix}.${it}" ]
+ },
+ exclusions: [
+ "org/slf4j/**"
+ ],
+ groupId: group,
+ artifactId: "beam-vendor-calcite-1_20_0",
+ version: version,
+)
diff --git a/vendor/grpc-1_13_1/build.gradle b/vendor/grpc-1_13_1/build.gradle
deleted file mode 100644
index 9d119c9..0000000
--- a/vendor/grpc-1_13_1/build.gradle
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.beam.gradle.GrpcVendoring
-
-plugins { id 'org.apache.beam.vendor-java' }
-
-description = "Apache Beam :: Vendored Dependencies :: gRPC :: 1.13.1"
-
-group = "org.apache.beam"
-version = "0.2"
-
-vendorJava(
- dependencies: GrpcVendoring.dependencies(),
- runtimeDependencies: GrpcVendoring.runtimeDependencies(),
- relocations: GrpcVendoring.relocations(),
- exclusions: GrpcVendoring.exclusions(),
- artifactId: "beam-vendor-grpc-1_13_1",
- groupId: group,
- version: version,
-)
diff --git a/vendor/grpc-1_21_0/build.gradle b/vendor/grpc-1_21_0/build.gradle
new file mode 100644
index 0000000..16ded33
--- /dev/null
+++ b/vendor/grpc-1_21_0/build.gradle
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.beam.gradle.GrpcVendoring
+
+plugins { id 'org.apache.beam.vendor-java' }
+
+description = "Apache Beam :: Vendored Dependencies :: gRPC :: 1.21.0"
+
+group = "org.apache.beam"
+version = "0.1"
+
+vendorJava(
+ dependencies: GrpcVendoring.dependencies(),
+ runtimeDependencies: GrpcVendoring.runtimeDependencies(),
+ relocations: GrpcVendoring.relocations(),
+ exclusions: GrpcVendoring.exclusions(),
+ artifactId: "beam-vendor-grpc-1_21_0",
+ groupId: group,
+ version: version,
+)
diff --git a/vendor/guava-20_0/build.gradle b/vendor/guava-20_0/build.gradle
deleted file mode 100644
index 419e656..0000000
--- a/vendor/guava-20_0/build.gradle
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-plugins { id 'org.apache.beam.vendor-java' }
-
-description = "Apache Beam :: Vendored Dependencies :: Guava 20"
-
-group = "org.apache.beam"
-version = "0.1"
-
-vendorJava(
- dependencies: ["com.google.guava:guava:20.0"],
- relocations: [
- "com.google.common": "org.apache.beam.vendor.guava.v20_0.com.google.common",
- "com.google.thirdparty": "org.apache.beam.vendor.guava.v20_0.com.google.thirdparty",
- ],
- groupId: group,
- artifactId: "beam-vendor-guava-20_0",
- version: version,
-)
diff --git a/vendor/guava-26_0-jre/build.gradle b/vendor/guava-26_0-jre/build.gradle
new file mode 100644
index 0000000..66f1527
--- /dev/null
+++ b/vendor/guava-26_0-jre/build.gradle
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins { id 'org.apache.beam.vendor-java' }
+
+description = "Apache Beam :: Vendored Dependencies :: Guava 26.0-jre"
+
+group = "org.apache.beam"
+version = "0.1"
+
+vendorJava(
+ dependencies: ["com.google.guava:guava:26.0-jre"],
+ relocations: [
+ "com.google.common": "org.apache.beam.vendor.guava.v26_0_jre.com.google.common",
+ "com.google.thirdparty": "org.apache.beam.vendor.guava.v26_0_jre.com.google.thirdparty",
+ ],
+ exclusions: [
+ "com/google/errorprone/**",
+ "com/google/j2objc/annotations/**",
+ "javax/annotation/**",
+ "org/checkerframework/**",
+ "org/codehaus/mojo/animal_sniffer/**",
+ ],
+ groupId: group,
+ artifactId: "beam-vendor-guava-26_0-jre",
+ version: version,
+)
diff --git a/vendor/sdks-java-extensions-protobuf/build.gradle b/vendor/sdks-java-extensions-protobuf/build.gradle
index d083537..e3f0c94 100644
--- a/vendor/sdks-java-extensions-protobuf/build.gradle
+++ b/vendor/sdks-java-extensions-protobuf/build.gradle
@@ -18,11 +18,12 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
- exportJavadoc: false,
- shadowClosure: DEFAULT_SHADOW_CLOSURE << {
+ automaticModuleName: 'org.apache.beam.vendor.sdks.java.extensions.protobuf',
+ exportJavadoc: false,
+ shadowClosure: {
dependencies {
- include(dependency('com.google.guava:guava:20.0'))
- include(dependency('com.google.protobuf:protobuf-java:3.5.1'))
+ include(dependency('com.google.guava:guava:26.0-jre'))
+ include(dependency('com.google.protobuf:protobuf-java:3.7.1'))
}
// We specifically relocate beam-sdks-extensions-protobuf under a vendored namespace
// but also vendor guava and protobuf to the same vendored namespace as the model/*
@@ -31,11 +32,11 @@
relocate "org.apache.beam.sdk.extensions.protobuf", "org.apache.beam.vendor.sdk.v2.sdk.extensions.protobuf"
// guava uses the com.google.common and com.google.thirdparty package namespaces
- relocate "com.google.common", "org.apache.beam.vendor.grpc.v1p13p1.com.google.common"
- relocate "com.google.thirdparty", "org.apache.beam.vendor.grpc.v1p13p1.com.google.thirdparty"
+ relocate "com.google.common", "org.apache.beam.vendor.grpc.v1p21p0.com.google.common"
+ relocate "com.google.thirdparty", "org.apache.beam.vendor.grpc.v1p21p0.com.google.thirdparty"
- relocate "com.google.protobuf", "org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf"
- }
+ relocate "com.google.protobuf", "org.apache.beam.vendor.grpc.v1p21p0.com.google.protobuf"
+ }
)
description = "Apache Beam :: Vendored Dependencies :: SDKs :: Java :: Extensions :: Protobuf"
@@ -53,7 +54,7 @@
}
dependencies {
- compile 'com.google.guava:guava:20.0'
- compile 'com.google.protobuf:protobuf-java:3.5.1'
+ compile 'com.google.guava:guava:26.0-jre'
+ compile 'com.google.protobuf:protobuf-java:3.7.1'
shadow project(path: ":sdks:java:core", configuration: "shadow")
}
diff --git a/website/_config.yml b/website/_config.yml
index 70db165..19eb6d1 100644
--- a/website/_config.yml
+++ b/website/_config.yml
@@ -62,7 +62,7 @@
toc_levels: 2..6
# The most recent release of Beam.
-release_latest: 2.12.0
+release_latest: 2.16.0
# Plugins are configured in the Gemfile.
diff --git a/website/build.gradle b/website/build.gradle
index 9beac9c..ebb1583 100644
--- a/website/build.gradle
+++ b/website/build.gradle
@@ -199,16 +199,18 @@
def author = System.env.ghprbPullAuthorLogin
if (author == null) {
// If the author is not defined, it's most probably running locally.
- // Try to infer the author from the remote URL
- for (remote in grgit.remote.list()) {
- if (remote.getName() == 'origin') {
- // remote.url = 'git@github.com:author/beam.git'
- author = remote.url.split(':')[1].split('/')[0]
- break
+ if (grgit != null) {
+ // If on git, try to infer the author from the remote URL
+ for (remote in grgit.remote.list()) {
+ if (remote.getName() == 'origin') {
+ // remote.url = 'git@github.com:author/beam.git'
+ author = remote.url.split(':')[1].split('/')[0]
+ break
+ }
}
}
}
- // Jenkins stores the branch in botho of the following environment variables.
+ // Jenkins stores the branch in both of the following environment variables.
def branch = System.env.ghprbSourceBranch ?: System.env.GIT_BRANCH
if (branch == null && grgit) {
branch = grgit.branch.current().getName()
diff --git a/website/notebooks/docs.yaml b/website/notebooks/docs.yaml
new file mode 100644
index 0000000..c0a8e5a
--- /dev/null
+++ b/website/notebooks/docs.yaml
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Python transform catalog
+documentation/transforms/python/elementwise/filter:
+ title: Filter - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/flatmap:
+ title: FlatMap - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/keys:
+ title: Keys - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/kvswap:
+ title: KvSwap - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/map:
+ title: Map - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/pardo:
+ title: ParDo - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/partition:
+ title: Partition - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/regex:
+ title: Regex - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+# documentation/transforms/python/elementwise/reify:
+# title: Reify - element-wise transform
+# languages: py
+# imports:
+# 1: [setup.md]
+
+documentation/transforms/python/elementwise/tostring:
+ title: ToString - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+documentation/transforms/python/elementwise/values:
+ title: Values - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
+
+# documentation/transforms/python/elementwise/withkeys:
+# title: WithKeys - element-wise transform
+# languages: py
+# imports:
+# 1: [setup.md]
+
+documentation/transforms/python/elementwise/withtimestamps:
+ title: WithTimestamps - element-wise transform
+ languages: py
+ imports:
+ 1: [setup.md]
diff --git a/website/notebooks/generate.py b/website/notebooks/generate.py
new file mode 100644
index 0000000..6af6355
--- /dev/null
+++ b/website/notebooks/generate.py
@@ -0,0 +1,128 @@
+#!/usr/bin/env python
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This script generates notebooks for all the files defined in `docs.yaml`.
+# It has to be run manually if the docs or code snippets are updated.
+#
+# To run, you have to install `md2ipynb`.
+# pip install -U md2ipynb
+#
+# Then it can be run without any arguments.
+# python website/notebooks/generate.py
+#
+# This creates the output notebooks in the `examples/notebooks` directory.
+# You have to commit the generated notebooks after generating them.
+
+import logging
+import md2ipynb
+import nbformat
+import os
+import sys
+import yaml
+
+docs_logo_url = 'https://beam.apache.org/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png'
+
+
+def run(docs, variables=None,
+ inputs_dir='.', outputs_dir='.', imports_dir='.', include_dir='.'):
+
+ errors = []
+ for basename, doc in docs.items():
+ languages=doc.get('languages', 'py java go').split()
+ for lang in languages:
+ # Read the imports defined in the docs.yaml.
+ imports = {
+ i: [os.path.join(imports_dir, path) for path in imports]
+ for i, imports in doc.get('imports', {}).items()
+ }
+
+ # Make sure the first import in section 0 is the license.md.
+ if 0 not in imports:
+ imports[0] = []
+ imports[0].insert(0, os.path.join(imports_dir, 'license.md'))
+
+ # Create a new notebook from the Markdown file contents.
+ input_file = basename + '.md'
+ ipynb_file = '/'.join([outputs_dir, '{}-{}.ipynb'.format(basename, lang)])
+ try:
+ notebook = md2ipynb.new_notebook(
+ input_file=os.path.join(inputs_dir, input_file),
+ variables=variables,
+ imports=imports,
+ notebook_title=doc.get('title', os.path.basename(basename).replace('-', ' ')),
+ keep_classes=['language-' + lang, 'shell-sh'],
+ filter_classes='notebook-skip',
+ docs_url='https://beam.apache.org/' + basename.replace('-', ''),
+ docs_logo_url=docs_logo_url,
+ github_ipynb_url='https://github.com/apache/beam/blob/master/' + ipynb_file,
+ include_dir=include_dir,
+ )
+ logging.info('{} succeeded'.format(input_file))
+
+ # Write the notebook to file.
+ output_dir = os.path.dirname(ipynb_file)
+ if not os.path.exists(output_dir):
+ os.makedirs(output_dir)
+ with open(ipynb_file, 'w') as f:
+ nbformat.write(notebook, f)
+ except Exception as e:
+ logging.error('{} failed: {}'.format(input_file, e))
+ errors.append((input_file, e))
+
+ if errors:
+ import traceback
+ sys.stdout.flush()
+ sys.stderr.flush()
+ print('')
+ print('=' * 60)
+ print(' Errors')
+ for input_file, e in errors:
+ print('')
+ print(input_file)
+ print('-' * len(input_file))
+ traceback.print_exception(type(e), e, e.__traceback__)
+
+ print('')
+ print('{} files processed ({} succeeded, {} failed)'.format(
+ len(docs), len(docs) - len(errors), len(errors)))
+
+
+if __name__ == '__main__':
+ logging.basicConfig(level=logging.INFO)
+
+ script_dir = os.path.dirname(os.path.realpath(__file__))
+ root_dir = os.path.realpath(os.path.join(script_dir, '..', '..'))
+
+ docs_file = os.path.join(script_dir, 'docs.yaml')
+ with open(docs_file) as f:
+ docs = yaml.load(f.read())
+
+ variables_file = os.path.join(root_dir, 'website', '_config.yml')
+ with open(variables_file) as f:
+ variables = {'site': yaml.load(f.read())}
+ variables['site']['baseurl'] = variables['site']['url']
+
+ run(
+ docs=docs,
+ variables=variables,
+ inputs_dir=os.path.join(root_dir, 'website', 'src'),
+ outputs_dir=os.path.join(root_dir, 'examples', 'notebooks'),
+ imports_dir=os.path.join(script_dir, 'imports'),
+ include_dir=os.path.join(root_dir, 'website', 'src', '_includes'),
+ )
diff --git a/website/notebooks/imports/license.md b/website/notebooks/imports/license.md
new file mode 100644
index 0000000..a05bcd8
--- /dev/null
+++ b/website/notebooks/imports/license.md
@@ -0,0 +1,19 @@
+```
+#@title Licensed under the Apache License, Version 2.0 (the "License")
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+```
diff --git a/website/notebooks/imports/setup.md b/website/notebooks/imports/setup.md
new file mode 100644
index 0000000..5d1a628
--- /dev/null
+++ b/website/notebooks/imports/setup.md
@@ -0,0 +1,30 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+## Setup
+
+To run a code cell, you can click the **Run cell** button at the top left of the cell,
+or select it and press **`Shift+Enter`**.
+Try modifying a code cell and re-running it to see what happens.
+
+> To learn more about Colab, see
+> [Welcome to Colaboratory!](https://colab.sandbox.google.com/notebooks/welcome.ipynb).
+
+{:.language-py}
+First, let's install the `apache-beam` module.
+
+{:.language-py}
+```sh
+pip install --quiet -U apache-beam
+```
diff --git a/website/src/.htaccess b/website/src/.htaccess
index 700f289..20d4586 100644
--- a/website/src/.htaccess
+++ b/website/src/.htaccess
@@ -21,4 +21,4 @@
# The following redirect maintains the previously supported URLs.
RedirectMatch permanent "/documentation/sdks/(javadoc|pydoc)(.*)" "https://beam.apache.org/releases/$1$2"
# Keep this updated to point to the current release.
-RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.12.0$2"
+RedirectMatch "/releases/([^/]+)/current(.*)" "https://beam.apache.org/releases/$1/2.16.0$2"
diff --git a/website/src/_data/authors.yml b/website/src/_data/authors.yml
index 16c6f1a..6830fe1 100644
--- a/website/src/_data/authors.yml
+++ b/website/src/_data/authors.yml
@@ -22,6 +22,12 @@
altay:
name: Ahmet Altay
email: altay@apache.org
+angoenka:
+ name: Ankur Goenka
+ email: goenka@apache.org
+anton:
+ name: Anton Kedin
+ email: anton@apache.org
ccy:
name: Charles Chen
email: ccy@apache.org
@@ -44,6 +50,10 @@
name: Harshit Dwivedi
email: harshithdwivedi@gmail.com
twitter: harshithdwivedi
+henryken:
+ name: Henry Suryawirawan
+ email: henry.ken@gmail.com
+ twitter: henry_ken
iemejia:
name: Ismaël Mejía
email: iemejia@apache.org
@@ -67,6 +77,10 @@
name: Leonid Kuligin
email: kuligin@google.com
twitter: lkulighin
+markliu:
+ name: Mark Liu
+ email: markliu@apache.org
+ twitter:
robertwb:
name: Robert Bradshaw
email: robertwb@apache.org
@@ -109,3 +123,11 @@
name: Matthias Baetens
email: baetensmatthias@gmail.com
twitter: matthiasbaetens
+rez:
+ name: Reza Rokni
+ email: rez@google.com
+ twitter: rarokni
+ttanay:
+ name: Tanay Tummalapalli
+ email: ttanay100@gmail.com
+ twitter: ttanay100
diff --git a/website/src/_data/capability-matrix.yml b/website/src/_data/capability-matrix.yml
index 007de40..fb3eaa0 100644
--- a/website/src/_data/capability-matrix.yml
+++ b/website/src/_data/capability-matrix.yml
@@ -34,6 +34,8 @@
name: Apache Samza
- class: nemo
name: Apache Nemo
+ - class: jet
+ name: Hazelcast Jet
categories:
- description: What is being computed?
@@ -89,6 +91,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: GroupByKey
values:
- class: model
@@ -135,6 +141,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Flatten
values:
- class: model
@@ -181,6 +191,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Combine
values:
- class: model
@@ -227,6 +241,10 @@
l1: 'Yes'
l2: fully supported
l3: 'Batch mode uses pre-aggregation'
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: 'Batch mode uses pre-aggregation'
- name: Composite Transforms
values:
- class: model
@@ -273,6 +291,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Partially'
+ l2: supported via inlining
+ l3: ''
- name: Side Inputs
values:
- class: model
@@ -319,6 +341,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Partially'
+ l2: with restrictions
+ l3: Supported only when the side input source is bounded and windowing uses global window
- name: Source API
values:
- class: model
@@ -365,6 +391,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Splittable DoFn (SDF)
values:
- class: model
@@ -411,6 +441,10 @@
l1: 'No'
l2: not implemented
l3: ''
+ - class: jet
+ l1: 'No'
+ l2: not implemented
+ l3: ''
- name: Metrics
values:
- class: model
@@ -419,8 +453,8 @@
l3: Allow transforms to gather simple metrics across bundles in a <tt>PTransform</tt>. Provide a mechanism to obtain both committed and attempted metrics. Semantically similar to using an additional output, but support partial results as the transform executes, and support both committed and attempted values. Will likely want to augment <tt>Metrics</tt> to be more useful for processing unbounded data by making them windowed.
- class: dataflow
l1: 'Partially'
- l2: In batch mode, Dataflow supports committed and attempted Counters and Distributions.
- l3: Gauge metrics are not supported in batch mode. Metrics are not yet supported at all in streaming mode, but this support is coming soon ([BEAM-2059](https://issues.apache.org/jira/browse/BEAM-2059)).
+ l2: ''
+ l3: Gauge metrics are not supported. All other metric types are supported.
- class: flink
l1: 'Partially'
l2: All metrics types are supported.
@@ -457,6 +491,10 @@
l1: 'No'
l2: not implemented
l3: ''
+ - class: jet
+ l1: 'Partially'
+ l2: All metrics types supported, both in batching and streaming mode.
+ l3: Doesn't differentiate between committed and attempted values.
- name: Stateful Processing
values:
- class: model
@@ -503,6 +541,10 @@
l1: 'No'
l2: not implemented
l3: ''
+ - class: jet
+ l1: 'Partially'
+ l2: non-merging windows
+ l3: ''
- description: Where in event time?
anchor: where
color-b: '37d'
@@ -556,6 +598,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- name: Fixed windows
values:
- class: model
@@ -602,6 +648,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- name: Sliding windows
values:
- class: model
@@ -648,6 +698,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- name: Session windows
values:
- class: model
@@ -694,6 +748,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- name: Custom windows
values:
- class: model
@@ -740,6 +798,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- name: Custom merging windows
values:
- class: model
@@ -786,6 +848,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- name: Timestamp control
values:
- class: model
@@ -832,6 +898,10 @@
l1: 'Yes'
l2: supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: supported
+ l3: ''
- description: When in processing time?
anchor: when
@@ -887,6 +957,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Event-time triggers
values:
@@ -934,6 +1008,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Processing-time triggers
values:
@@ -981,6 +1059,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Count triggers
values:
@@ -1028,6 +1110,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: '[Meta]data driven triggers'
values:
@@ -1076,6 +1162,10 @@
l1: 'No'
l2: pending model support
l3: ''
+ - class: jet
+ l1: 'No'
+ l2: pending model support
+ l3: ''
- name: Composite triggers
values:
@@ -1123,6 +1213,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Allowed lateness
values:
@@ -1170,6 +1264,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Timers
values:
@@ -1217,6 +1315,10 @@
l1: 'No'
l2: not implemented
l3: ''
+ - class: jet
+ l1: 'Partially'
+ l2: non-merging windows
+ l3: ''
- description: How do refinements relate?
anchor: how
@@ -1272,6 +1374,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: Accumulating
values:
@@ -1319,6 +1425,10 @@
l1: 'Yes'
l2: fully supported
l3: ''
+ - class: jet
+ l1: 'Yes'
+ l2: fully supported
+ l3: ''
- name: 'Accumulating & Retracting'
values:
@@ -1367,3 +1477,110 @@
l1: 'No'
l2: pending model support
l3: ''
+ - class: jet
+ l1: 'No'
+ l2: pending model support
+ l3: ''
+ - description: Additional common features not yet part of the Beam model
+ anchor: misc
+ color-b: 'aaa'
+ color-y: 'bbb'
+ color-p: 'ccc'
+ color-n: 'ddd'
+ rows:
+ - name: Drain
+ values:
+ - class: model
+ l1: 'Partially'
+ l2:
+ l3: APIs and semantics for draining a pipeline are under discussion. This would cause incomplete aggregations to be emitted regardless of trigger and tagged with metadata indicating it is incompleted.
+ - class: dataflow
+ l1: 'Partially'
+ l2:
+ l3: Dataflow has a native drain operation, but it does not work in the presence of event time timer loops. Final implemention pending model support.
+ - class: flink
+ l1: 'Partially'
+ l2:
+ l3: Flink supports taking a "savepoint" of the pipeline and shutting the pipeline down after its completion.
+ - class: spark
+ l1:
+ l2:
+ l3:
+ - class: apex
+ l1:
+ l2:
+ l3:
+ - class: gearpump
+ l1:
+ l2:
+ l3:
+ - class: mapreduce
+ l1:
+ l2:
+ l3:
+ - class: jstorm
+ l1:
+ l2:
+ l3:
+ - class: ibmstreams
+ l1:
+ l2:
+ l3:
+ - class: samza
+ l1:
+ l2:
+ l3:
+ - class: nemo
+ l1:
+ l2:
+ l3:
+ - name: Checkpoint
+ values:
+ - class: model
+ l1: 'Partially'
+ l2:
+ l3: APIs and semantics for saving a pipeline checkpoint are under discussion. This would be a runner-specific materialization of the pipeline state required to resume or duplicate the pipeline.
+ - class: dataflow
+ l1: 'No'
+ l2:
+ l3:
+ - class: flink
+ l1: 'Partially'
+ l2:
+ l3: Flink has a native savepoint capability.
+ - class: spark
+ l1: 'Partially'
+ l2:
+ l3: Spark has a native savepoint capability.
+ - class: apex
+ l1:
+ l2:
+ l3:
+ - class: gearpump
+ l1:
+ l2:
+ l3:
+ - class: mapreduce
+ l1:
+ l2:
+ l3:
+ - class: jstorm
+ l1:
+ l2:
+ l3:
+ - class: ibmstreams
+ l1:
+ l2:
+ l3:
+ - class: samza
+ l1: 'Partially'
+ l2:
+ l3: Samza has a native checkpoint capability.
+ - class: nemo
+ l1:
+ l2:
+ l3:
+ - class: jet
+ l1:
+ l2:
+ l3:
diff --git a/website/src/_includes/button-pydoc.md b/website/src/_includes/button-pydoc.md
new file mode 100644
index 0000000..c0135aa
--- /dev/null
+++ b/website/src/_includes/button-pydoc.md
@@ -0,0 +1,23 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+{% capture button_url %}https://beam.apache.org/releases/pydoc/current/{{ include.path }}.html#{{ include.path }}.{{ include.class }}{% endcapture %}
+
+{% include button.md
+ url=button_url
+ logo="https://beam.apache.org/images/logos/sdks/python.png"
+ text="Pydoc"
+%}
+
+<br><br><br>
diff --git a/website/src/_includes/button.md b/website/src/_includes/button.md
new file mode 100644
index 0000000..de7a414
--- /dev/null
+++ b/website/src/_includes/button.md
@@ -0,0 +1,21 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+{% if include.attrib %}
+{:{{ include.attrib }}}{% endif %}
+<table align="left" style="margin-right:1em">
+ <td>
+ <a class="button" target="_blank" href="{{ include.url }}">{% if include.logo %}<img src="{{ include.logo }}" width="32px" height="32px" alt="{{ include.text }}" /> {% endif %}{{ include.text }}</a>
+ </td>
+</table>
diff --git a/website/src/_includes/buttons-code-snippet.md b/website/src/_includes/buttons-code-snippet.md
new file mode 100644
index 0000000..54ac914
--- /dev/null
+++ b/website/src/_includes/buttons-code-snippet.md
@@ -0,0 +1,43 @@
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+{% capture colab_logo %}https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png{% endcapture %}
+{% capture github_logo %}https://www.tensorflow.org/images/GitHub-Mark-32px.png{% endcapture %}
+
+{% capture notebook_url %}https://colab.research.google.com/github/{{ site.branch_repo }}/{{ include.notebook }}{% endcapture %}
+{% capture notebook_java %}{{ notebook_url }}-java.ipynb{% endcapture %}
+{% capture notebook_py %}{{ notebook_url }}-py.ipynb{% endcapture %}
+{% capture notebook_go %}{{ notebook_url }}-go.ipynb{% endcapture %}
+
+{% capture code_url %}https://github.com/{{ site.branch_repo }}{% endcapture %}
+{% capture code_java %}{{ code_url }}/{{ include.java }}{% endcapture %}
+{% capture code_py %}{{ code_url }}/{{ include.py }}{% endcapture %}
+{% capture code_go %}{{ code_url }}/{{ include.go }}{% endcapture %}
+
+{% if include.java %}
+{% if include.notebook %}{% include button.md url=notebook_java logo=colab_logo text="Run code now" attrib=".language-java .notebook-skip" %}{% endif %}
+{% include button.md url=code_java logo=github_logo text="View source code" attrib=".language-java" %}
+{% endif %}
+
+{% if include.py %}
+{% if include.notebook %}{% include button.md url=notebook_py logo=colab_logo text="Run code now" attrib=".language-py .notebook-skip" %}{% endif %}
+{% include button.md url=code_py logo=github_logo text="View source code" attrib=".language-py" %}
+{% endif %}
+
+{% if include.go %}
+{% if include.notebook %}{% include button.md url=notebook_go logo=colab_logo text="Run code now" attrib=".language-go .notebook-skip" %}{% endif %}
+{% include button.md url=code_go logo=github_logo text="View source code" attrib=".language-go" %}
+{% endif %}
+
+<br><br><br>
diff --git a/website/src/_includes/capability-matrix.md b/website/src/_includes/capability-matrix.md
index 6ebfcaa..369de31 100644
--- a/website/src/_includes/capability-matrix.md
+++ b/website/src/_includes/capability-matrix.md
@@ -15,7 +15,14 @@
<table class='{{ cap-style }}'>
{% for category in cap-data.categories %}
<tr class='{{ cap-style }}' id='cap-{{ cap-view }}-{{ category.anchor }}'>
- <th class='{{ cap-style }} color-metadata format-category' colspan='5' style='color:#{{ category.color-b }}'>{% if cap-view != 'blog' %}<div class='cap-toggle' onclick='ToggleTables({{ cap-toggle-details }}, "cap-{{ cap-other-view }}-{{ category.anchor }}")'>({% if cap-toggle-details == 1 %}expand{% else %}collapse{% endif %} details)</div>{% endif %}{{ category.description }}</th>
+ <th class='{{ cap-style }} color-metadata format-category' colspan='8' style='color:#{{ category.color-b }}'>
+ {% if cap-view != 'blog' %}
+ <div class='cap-toggle' onclick='ToggleTables({{ cap-toggle-details }}, "cap-{{ cap-other-view }}-{{ category.anchor }}")'>
+ ({% if cap-toggle-details == 1 %}click to expand{% else %}click to collapse{% endif %} details)
+ </div>
+ {% endif %}
+ {{ category.description }}
+ </th>
</tr>
<tr class='{{ cap-style }}'>
<th class='{{ cap-style }} color-capability'></th>
diff --git a/website/src/_includes/head.html b/website/src/_includes/head.html
index b8fa280..2fd0083 100644
--- a/website/src/_includes/head.html
+++ b/website/src/_includes/head.html
@@ -18,7 +18,7 @@
<meta name="description" content="{% if page.excerpt %}{{ page.excerpt | strip_html | strip_newlines | truncate: 160 }}{% else %}{{ site.description }}{% endif %}">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
<link rel="stylesheet" href="{{ "/css/site.css" | prepend: site.baseurl }}">
- <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"></script>
+ <script src="https://code.jquery.com/jquery-2.2.4.min.js"></script>
<script src="{{ "/js/bootstrap.min.js" | prepend: site.baseurl }}"></script>
<script src="{{ "/js/language-switch.js" | prepend: site.baseurl }}"></script>
<script src="{{ "/js/fix-menu.js" | prepend: site.baseurl }}"></script>
diff --git a/website/src/_includes/header.html b/website/src/_includes/header.html
index 6c197f9..3f83ab6 100644
--- a/website/src/_includes/header.html
+++ b/website/src/_includes/header.html
@@ -52,6 +52,22 @@
<li><a href="{{ site.baseurl }}/blog">Blog</a></li>
</ul>
<ul class="nav navbar-nav navbar-right">
+ <li>
+ <div style="width: 300px;">
+ <script>
+ (function() {
+ var cx = '012923275103528129024:4emlchv9wzi';
+ var gcse = document.createElement('script');
+ gcse.type = 'text/javascript';
+ gcse.async = true;
+ gcse.src = 'https://cse.google.com/cse.js?cx=' + cx;
+ var s = document.getElementsByTagName('script')[0];
+ s.parentNode.insertBefore(gcse, s);
+ })();
+ </script>
+ <gcse:search></gcse:search>
+ </div>
+ </li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false"><img src="https://www.apache.org/foundation/press/kit/feather_small.png" alt="Apache Logo" style="height:20px;"><span class="caret"></span></a>
<ul class="dropdown-menu dropdown-menu-right">
diff --git a/website/src/_includes/section-menu/documentation.html b/website/src/_includes/section-menu/documentation.html
index 4b0b785..529c264e 100644
--- a/website/src/_includes/section-menu/documentation.html
+++ b/website/src/_includes/section-menu/documentation.html
@@ -12,7 +12,6 @@
<li><span class="section-nav-list-main-title">Documentation</span></li>
<li><a href="{{ site.baseurl }}/documentation">Using the Documentation</a></li>
-<li><a href="{{ site.baseurl }}/documentation/execution-model">Beam Execution Model</a></li>
<li>
<span class="section-nav-list-title">Pipeline development lifecycle</span>
@@ -112,15 +111,166 @@
<li><a href="{{ site.baseurl }}/documentation/programming-guide/#composite-triggers">Composite triggers</a></li>
</ul>
</li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Metrics</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/programming-guide/#metrics">Metrics basics</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/programming-guide/#types-of-metrics">Types of metrics</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/programming-guide/#querying-metrics">Querying metrics</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/programming-guide/#using-metrics">Using metrics in pipeline</a></li>
+ </ul>
+ </li>
</ul>
</li>
+
<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">Learning Resources</span>
+ <span class="section-nav-list-title">Transform catalog</span>
+
+ <ul class="section-nav-list">
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Python</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/overview/">Overview</a></li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Element-wise</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/filter/">Filter</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap/">FlatMap</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/keys/">Keys</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap/">KvSwap</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/map/">Map</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/pardo/">ParDo</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/partition/">Partition</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/regex/">Regex</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/reify/">Reify</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/tostring/">ToString</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/values/">Values</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/withtimestamps/">WithTimestamps</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Aggregation</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey/">CoGroupByKey</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/combineglobally/">CombineGlobally</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/count/">Count</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/distinct/">Distinct</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/groupbykey/">GroupByKey</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/mean/">Mean</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/sample/">Sample</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/top/">Top</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Other</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/other/create/">Create</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/other/flatten/">Flatten</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/other/reshuffle/">Reshuffle</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/python/other/windowinto/">WindowInto</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Java</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/overview/">Overview</a></li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Element-wise</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/filter/">Filter</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/flatmapelements/">FlatMapElements</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/keys/">Keys</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/kvswap/">KvSwap</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/mapelements/">MapElements</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/pardo/">ParDo</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/partition/">Partition</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/regex/">Regex</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/reify/">Reify</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/values/">Values</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/tostring/">ToString</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/withkeys/">WithKeys</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/withtimestamps/">WithTimestamps</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Aggregation</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/approximatequantiles/">ApproximateQuantiles</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/approximateunique/">ApproximateUnique</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/cogroupbykey/">CoGroupByKey</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/combine/">Combine</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/combinewithcontext/">CombineWithContext</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/count/">Count</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/distinct/">Distinct</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey/">GroupByKey</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupintobatches/">GroupIntoBatches</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/latest/">Latest</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/max/">Max</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/mean/">Mean</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/min/">Min</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/sample/">Sample</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/sum/">Sum</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/top/">Top</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Other</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/other/create/">Create</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/other/flatten/">Flatten</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/other/passert/">PAssert</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/other/view/">View</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/transforms/java/other/window/">Window</a></li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+</li>
+
+<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Common pipeline patterns</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/patterns/overview/">Overview</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/file-processing/">File processing</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/side-inputs/">Side inputs</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/pipeline-options/">Pipeline options</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/custom-io/">Custom I/O</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/patterns/custom-windows/">Custom windows</a></li>
+ </ul>
+</li>
+
+<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Runtime systems</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/runtime/model/">Execution model</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/runtime/environments/">Runtime environments</a></li>
+ </ul>
+</li>
+
+<li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Learning resources</span>
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#getting-started">Getting Started</a></li>
<li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#articles">Articles</a></li>
<li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#interactive-labs">Interactive Labs</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#beam-katas">Beam Katas</a></li>
<li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#code-examples">Code Examples</a></li>
<li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#api-reference">API Reference</a></li>
<li><a href="{{ site.baseurl }}/documentation/resources/learning-resources/#feedback-and-suggestions">Feedback and Suggestions</a></li>
diff --git a/website/src/_includes/section-menu/runners.html b/website/src/_includes/section-menu/runners.html
index dd77b7b..3444e0d 100644
--- a/website/src/_includes/section-menu/runners.html
+++ b/website/src/_includes/section-menu/runners.html
@@ -20,3 +20,4 @@
<li><a href="{{ site.baseurl }}/documentation/runners/samza/">Apache Samza</a></li>
<li><a href="{{ site.baseurl }}/documentation/runners/spark/">Apache Spark</a></li>
<li><a href="{{ site.baseurl }}/documentation/runners/dataflow/">Google Cloud Dataflow</a></li>
+<li><a href="{{ site.baseurl }}/documentation/runners/jet/">Hazelcast Jet</a></li>
diff --git a/website/src/_includes/section-menu/sdks.html b/website/src/_includes/section-menu/sdks.html
index f15f0ac..15d97a9 100644
--- a/website/src/_includes/section-menu/sdks.html
+++ b/website/src/_includes/section-menu/sdks.html
@@ -60,19 +60,49 @@
<li><a href="{{ site.baseurl }}/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
<li><a href="{{ site.baseurl }}/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
- <span class="section-nav-list-title">SQL Reference</span>
+ <span class="section-nav-list-title">Apache Calcite dialect</span>
<ul class="section-nav-list">
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/data-types/">Data types</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/lexical/">Lexical structure</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/select/">SELECT</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/windowing-and-triggering/">Windowing & Triggering</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/joins/">Joins</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/scalar-functions/">Scalar functions</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/aggregate-functions/">Aggregate functions</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/user-defined-functions/">User-defined functions</a></li>
- <li><a href="{{ site.baseurl }}/documentation/dsls/sql/set/">SET Pipeline Options</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/calcite/overview/">Calcite support overview</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/calcite/query-syntax/">Query syntax</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/calcite/lexical/">Lexical structure</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/calcite/data-types/">Data types</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/calcite/scalar-functions/">Scalar functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/calcite/aggregate-functions/">Aggregate functions</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">ZetaSQL dialect</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/overview/">ZetaSQL support overview</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/syntax/">Function call rules</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conversion-rules/">Conversion rules</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/query-syntax/">Query syntax</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/lexical/">Lexical structure</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/data-types/">Data types</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/operators/">Operators</a></li>
+
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Scalar functions</span>
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/string-functions/">String functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/math-functions/">Mathematical functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conditional-expressions/">Conditional expressions</a></li>
+ </ul>
+ </li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions/">Aggregate functions</a></li>
+ </ul>
+ </li>
+ <li class="section-nav-item--collapsible">
+ <span class="section-nav-list-title">Beam SQL extensions</span>
+
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/extensions/create-external-table/">CREATE EXTERNAL TABLE</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/extensions/windowing-and-triggering/">Windowing & triggering</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/extensions/joins/">Joins</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/extensions/user-defined-functions/">User-defined functions</a></li>
+ <li><a href="{{ site.baseurl }}/documentation/dsls/sql/extensions/set/">SET pipeline options</a></li>
</ul>
</li>
</ul>
diff --git a/website/src/_posts/2017-02-13-stateful-processing.md b/website/src/_posts/2017-02-13-stateful-processing.md
index f51b307..a778dde 100644
--- a/website/src/_posts/2017-02-13-stateful-processing.md
+++ b/website/src/_posts/2017-02-13-stateful-processing.md
@@ -28,6 +28,9 @@
Beam: how it works, how it fits in with the other features of the Beam model,
what you might use it for, and what it looks like in code.
+**Note: This post has been updated in May of 2019, to include Python
+snippets!**
+
<!--more-->
> **Warning: new features ahead!**: This is a very new aspect of the Beam
@@ -278,7 +281,7 @@
// A state cell holding a single Integer per key+window
@StateId("index")
- private final StateSpec<ValueState<Integer>> indexSpec =
+ private final StateSpec<ValueState<Integer>> indexSpec =
StateSpecs.value(VarIntCoder.of());
@ProcessElement
@@ -293,8 +296,14 @@
```
```py
-# State and timers are not yet supported in Beam's Python SDK.
-# Watch this space!
+class IndexAssigningStatefulDoFn(DoFn):
+ INDEX_STATE = CombiningStateSpec('index', sum)
+
+ def process(self, element, index=DoFn.StateParam(INDEX_STATE)):
+ unused_key, value = element
+ current_index = index.read()
+ yield (value, current_index)
+ index.add(1)
```
Let's dissect this:
@@ -371,8 +380,20 @@
```
```py
-# State and timers are not yet supported in Beam's Python SDK.
-# Watch this space!
+class ModelFromEventsFn(apache_beam.core.CombineFn):
+
+ def create_accumulator(self):
+ # Create a new empty model
+ return Model()
+
+ def add_input(self, model, input):
+ return model.update(input)
+
+ def merge_accumulators(self, accumulators):
+ # Custom merging logic
+
+ def extract_output(self, model):
+ return model
```
Now you have a way to compute the model of a particular user for a window as
@@ -407,8 +428,24 @@
```
```py
-# State and timers are not yet supported in Beam's Python SDK.
-# Watch this space!
+# Events is a collection of (user, event) pairs.
+events = (p | ReadFromEventSource() | beam.WindowInto(....))
+
+user_models = beam.pvalue.AsDict(
+ events
+ | beam.core.CombinePerKey(ModelFromEventsFn()))
+
+def event_prediction(user_event, models):
+ user = user_event[0]
+ event = user_event[1]
+
+ # Retrieve the model calculated for this user
+ model = models[user]
+
+ return (user, model.prediction(event))
+
+# Predictions is a collection of (user, prediction) pairs.
+predictions = events | beam.Map(event_prediction, user_models)
```
In this pipeline, there is just one model emitted by the `Combine.perKey(...)`
@@ -441,8 +478,15 @@
```
```py
-# State and timers are not yet supported in Beam's Python SDK.
-# Watch this space!
+events = ...
+
+user_models = beam.pvalue.AsDict(
+ events
+ | beam.WindowInto(GlobalWindows(),
+ trigger=trigger.AfterAll(
+ trigger.AfterCount(1),
+ trigger.AfterProcessingTime(1)))
+ | beam.CombinePerKey(ModelFromEventsFn()))
```
This is often a pretty nice tradeoff between latency and cost: If a huge flood
@@ -499,8 +543,31 @@
```
```py
-# State and timers are not yet supported in Beam's Python SDK.
-# Watch this space!
+class ModelStatefulFn(beam.DoFn):
+
+ PREVIOUS_PREDICTION = BagStateSpec('previous_pred_state', PredictionCoder())
+ MODEL_STATE = CombiningValueStateSpec('model_state',
+ ModelCoder(),
+ ModelFromEventsFn())
+
+ def process(self,
+ user_event,
+ previous_pred_state=beam.DoFn.StateParam(PREVIOUS_PREDICTION),
+ model_state=beam.DoFn.StateParam(MODEL_STATE)):
+ user = user_event[0]
+ event = user_event[1]
+ model = model_state.read()
+ previous_prediction = previous_pred_state.read()
+
+ new_prediction = model.prediction(event)
+ model_state.add(event)
+
+ if (previous_prediction is None
+ or self.should_output_prediction(
+ previous_prediction, new_prediction)):
+ previous_pred_state.clear()
+ previous_pred_state.add(new_prediction)
+ yield (user, new_prediction)
```
Let's walk through it,
@@ -510,20 +577,21 @@
the prediction output previously.
- Access to the two state cells by annotation in the `@ProcessElement` method
is as before.
- - You read the current model via `modelState.read()`. Because state is also
- per-key-and-window, this is a model just for the UserId of the Event
+ - You read the current model via `modelState.read()`.
+ per-key-and-window, this is a model just for the UserId of the Event
currently being processed.
- You derive a new prediction `model.prediction(event)` and compare it against
- the last one you output, accessed via `previousPredicationState.read()`.
+ the last one you output, accessed via
+ `previousPredicationState.read()`.
- You then update the model `model.update()` and write it via
- `modelState.write(...)`. It is perfectly fine to mutate the value you pulled
- out of state as long as you also remember to write the mutated value, in the
- same way you are encouraged to mutate `CombineFn` accumulators.
+ `modelState.write(...)`. It is perfectly fine to mutate the value
+ you pulled out of state as long as you also remember to write the mutated
+ value, in the same way you are encouraged to mutate `CombineFn` accumulators.
- If the prediction has changed a significant amount since the last time you
- output, you emit it via `context.output(...)` and save the prediction using
- `previousPredictionState.write(...)`. Here the decision is relative to the
- prior prediction output, not the last one computed - realistically you might
- have some complex conditions here.
+ output, you emit it via `context.output(...)` and
+ save the prediction using `previousPredictionState.write(...)`.
+ Here the decision is relative to the prior prediction output, not the last
+ one computed - realistically you might have some complex conditions here.
Most of the above is just talking through Java! But before you go out and
convert all of your pipelines to use stateful processing, I want to go over
diff --git a/website/src/_posts/2018-01-09-beam-a-look-back.md b/website/src/_posts/2018-01-09-beam-a-look-back.md
index 90c16b0..5c58bc9 100644
--- a/website/src/_posts/2018-01-09-beam-a-look-back.md
+++ b/website/src/_posts/2018-01-09-beam-a-look-back.md
@@ -64,7 +64,7 @@
In addition to runners, Beam added new IO connectors, some notable ones being
the Cassandra, MQTT, AMQP, HBase/HCatalog, JDBC, Solr, Tika, Redis, and
-ElasticSearch connectors. Beam’s IO connectors make it possible to read from or
+Elasticsearch connectors. Beam’s IO connectors make it possible to read from or
write to data sources/sinks even when they are not natively supported by the
underlying execution engine. Beam also provides fully pluggable filesystem
support, allowing us to support and extend our coverage to HDFS, S3, Azure
diff --git a/website/src/_posts/2019-03-18-beam-summit-site.md b/website/src/_posts/2019-03-18-beam-summit-site.md
new file mode 100644
index 0000000..8b6dcfa
--- /dev/null
+++ b/website/src/_posts/2019-03-18-beam-summit-site.md
@@ -0,0 +1,38 @@
+---
+layout: post
+title: "Announcing Beam Summit Site"
+date: 2019-03-18 00:00:01 -0800
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - aizhamal
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are thrilled to announce the launch of our new website dedicated to Beam Summits!
+
+The [beamsummit.org](https://beamsummit.org) site provides all the information you need towards the upcoming Beam Summits in Europe, Asia and North America in 2019. You will find information about the conference theme, the speakers and sessions, the abstract submission timeline and the registration process, the conference venues and hosting cities - and much more that you will find useful until and during the Beam Summits 2019.
+
+We are working to make the website easy to use, so that anyone who is organizing a Beam event can rely on it. You can find the [code for it in Github](https://github.com/matthiasa4/hoverboard).
+
+The pages will be updated on a regular basis, but we also love hearing thoughts from our community! Let us know if you have any questions, comments or suggestions, and help us improve! Also, if you are thinking of organizing a Beam event, please feel free to reach out for support, and to use the code in GitHub as well.
+
+We sincerely hope that you like the new Beam Summit website and will find it useful for accessing information. Enjoy browsing around!
+
+See you in Berlin!
+
+#beamsummit2019.
+
diff --git a/website/src/_posts/2019-05-01-adding-data-sources-to-sql.md b/website/src/_posts/2019-05-01-adding-data-sources-to-sql.md
new file mode 100644
index 0000000..f8d0f4c
--- /dev/null
+++ b/website/src/_posts/2019-05-01-adding-data-sources-to-sql.md
@@ -0,0 +1,202 @@
+---
+layout: post
+title: "Adding new Data Sources to Beam SQL CLI"
+date: 2019-06-04 00:00:01 -0800
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - pabloem
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+A new, exciting feature that came to Apache Beam is the ability to use
+SQL in your pipelines. This is done using Beam's
+[`SqlTransform`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/SqlTransform.html)
+in Java pipelines.
+
+Beam also has a fancy new SQL command line that you can use to query your
+data interactively, be it Batch or Streaming. If you haven't tried it, check out
+[http://bit.ly/ExploreBeamSQL](http://bit.ly/ExploreBeamSQL).
+
+A nice feature of the SQL CLI is that you can use `CREATE EXTERNAL TABLE`
+commands to *add* data sources to be accessed in the CLI. Currently, the CLI
+supports creating tables from BigQuery, PubSub, Kafka, and text files. In this
+post, we explore how to add new data sources, so that you will be able to
+consume data from other Beam sources.
+
+<!--more-->
+
+The table provider we will be implementing in this post will be generating a
+continuous unbounded stream of integers. It will be based on the
+[`GenerateSequence` PTransform](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/GenerateSequence.html)
+from the Beam SDK. In the end will be able to define and use the sequence generator
+in SQL like this:
+
+```
+CREATE EXTERNAL TABLE -- all tables in Beam are external, they are not persisted
+ sequenceTable -- table alias that will be used in queries
+ (
+ sequence BIGINT, -- sequence number
+ event_timestamp TIMESTAMP -- timestamp of the generated event
+ )
+TYPE sequence -- type identifies the table provider
+TBLPROPERTIES '{ elementsPerSecond : 12 }' -- optional rate at which events are generated
+```
+
+And we'll be able to use it in queries like so:
+
+```
+SELECT sequence FROM sequenceTable;
+```
+
+Let's dive in!
+
+### Implementing a `TableProvider`
+
+Beam's `SqlTransform` works by relying on `TableProvider`s, which it uses when
+one uses a `CREATE EXTERNAL TABLE` statement. If you are looking to add a new
+data source to the Beam SQL CLI, then you will want to add a `TableProvider` to
+do it. In this post, I will show what steps are necessary to create a new table
+provider for the
+[`GenerateSequence` transform](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/GenerateSequence.html) available in the Java SDK.
+
+The `TableProvider` classes are under
+[`sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/`](https://github.com/apache/beam/tree/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider). If you look in there, you can find providers, and their implementations, for all available data sources. So, you just need to add the one you want, along with an implementation of `BaseBeamTable`.
+
+### The GenerateSequenceTableProvider
+
+Our table provider looks like this:
+
+```java
+@AutoService(TableProvider.class)
+public class GenerateSequenceTableProvider extends InMemoryMetaTableProvider {
+
+ @Override
+ public String getTableType() {
+ return "sequence";
+ }
+
+ @Override
+ public BeamSqlTable buildBeamSqlTable(Table table) {
+ return new GenerateSequenceTable(table);
+ }
+}
+```
+
+All it does is give a type to the table - and it implements the
+`buildBeamSqlTable` method, which simply returns a `BeamSqlTable` defined by
+our `GenerateSequenceTable` implementation.
+
+### The GenerateSequenceTable
+
+We want a table implementation that supports streaming properly, so we will
+allow users to define the number of elements to be emitted per second. We will
+define a simple table that emits sequential integers in a streaming fashion.
+This looks like so:
+
+```java
+class GenerateSequenceTable extends BaseBeamTable implements Serializable {
+ public static final Schema TABLE_SCHEMA =
+ Schema.of(Field.of("sequence", FieldType.INT64), Field.of("event_time", FieldType.DATETIME));
+
+ Integer elementsPerSecond = 5;
+
+ GenerateSequenceTable(Table table) {
+ super(TABLE_SCHEMA);
+ if (table.getProperties().containsKey("elementsPerSecond")) {
+ elementsPerSecond = table.getProperties().getInteger("elementsPerSecond");
+ }
+ }
+
+ @Override
+ public PCollection.IsBounded isBounded() {
+ return IsBounded.UNBOUNDED;
+ }
+
+ @Override
+ public PCollection<Row> buildIOReader(PBegin begin) {
+ return begin
+ .apply(GenerateSequence.from(0).withRate(elementsPerSecond, Duration.standardSeconds(1)))
+ .apply(
+ MapElements.into(TypeDescriptor.of(Row.class))
+ .via(elm -> Row.withSchema(TABLE_SCHEMA).addValues(elm, Instant.now()).build()))
+ .setRowSchema(getSchema());
+ }
+
+ @Override
+ public POutput buildIOWriter(PCollection<Row> input) {
+ throw new UnsupportedOperationException("buildIOWriter unsupported!");
+ }
+}
+```
+
+
+
+## The real fun
+
+Now that we have implemented the two basic classes (a `BaseBeamTable`, and a
+`TableProvider`), we can start playing with them. After building the
+[SQL CLI](https://beam.apache.org/documentation/dsls/sql/shell/), we
+can now perform selections on the table:
+
+```
+0: BeamSQL> CREATE EXTERNAL TABLE input_seq (
+. . . . . > sequence BIGINT COMMENT 'this is the primary key',
+. . . . . > event_time TIMESTAMP COMMENT 'this is the element timestamp'
+. . . . . > )
+. . . . . > TYPE 'sequence';
+No rows affected (0.005 seconds)
+```
+
+And let's select a few rows:
+
+```
+0: BeamSQL> SELECT * FROM input_seq LIMIT 5;
++---------------------+------------+
+| sequence | event_time |
++---------------------+------------+
+| 0 | 2019-05-21 00:36:33 |
+| 1 | 2019-05-21 00:36:33 |
+| 2 | 2019-05-21 00:36:33 |
+| 3 | 2019-05-21 00:36:33 |
+| 4 | 2019-05-21 00:36:33 |
++---------------------+------------+
+5 rows selected (1.138 seconds)
+```
+
+Now let's try something more interesting. Such as grouping. This will also let
+us make sure that we're providing the timestamp for each row properly:
+
+```
+0: BeamSQL> SELECT
+. . . . . > COUNT(sequence) as elements,
+. . . . . > TUMBLE_START(event_time, INTERVAL '2' SECOND) as window_start
+. . . . . > FROM input_seq
+. . . . . > GROUP BY TUMBLE(event_time, INTERVAL '2' SECOND) LIMIT 5;
++---------------------+--------------+
+| elements | window_start |
++---------------------+--------------+
+| 6 | 2019-06-05 00:39:24 |
+| 10 | 2019-06-05 00:39:26 |
+| 10 | 2019-06-05 00:39:28 |
+| 10 | 2019-06-05 00:39:30 |
+| 10 | 2019-06-05 00:39:32 |
++---------------------+--------------+
+5 rows selected (10.142 seconds)
+```
+
+And voilà! We can start playing with some interesting streaming queries to our
+sequence generator.
diff --git a/website/src/_posts/2019-05-22-beam-2.13.0.md b/website/src/_posts/2019-05-22-beam-2.13.0.md
new file mode 100644
index 0000000..c81aa2d
--- /dev/null
+++ b/website/src/_posts/2019-05-22-beam-2.13.0.md
@@ -0,0 +1,77 @@
+---
+layout: post
+title: "Apache Beam 2.13.0"
+date: 2019-06-07 00:00:01 -0800
+# Date above corrected but keep the old URL:
+permalink: /blog/2019/05/22/beam-2.13.0.html
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - goenka
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.13.0 release of Beam. This release includes both improvements and new functionality.
+See the [download page]({{ site.baseurl }}/get-started/downloads/#2130-2019-05-21) for this release.<!--more-->
+For more information on changes in 2.13.0, check out the
+[detailed release notes](https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345166).
+
+## Highlights
+
+### I/Os
+
+* Support reading query results with the BigQuery storage API.
+* Support KafkaIO to be configured externally for use with other SDKs.
+* BigQuery IO now supports BYTES datatype on Python 3.
+* Avro IO support enabled on Python 3.
+* For Python 3 pipelines, the default Avro library used by Beam AvroIO and Dataflow workers was switched from avro-python3 to fastavro.
+
+
+### New Features / Improvements
+
+* Flink 1.8 support added.
+* Support to run word count on Portable Spark runner.
+* ElementCount metrics in FnApi Dataflow Runner.
+* Support to create BinaryCombineFn from lambdas.
+
+
+### Breaking Changes
+* When writing BYTES Datatype into Bigquery with Beam Bigquery IO on Python DirectRunner, users need to base64-encode bytes values before passing them to Bigquery IO. Accordingly, when reading bytes data from BigQuery, the IO will also return base64-encoded bytes. This change only affects Bigquery IO on Python DirectRunner. New DirectRunner behavior is consistent with treatment of Bytes by Beam Java Bigquery IO, and Python Dataflow Runner.
+
+
+### Bugfixes
+
+* Various bug fixes and performance improvements.
+
+## List of Contributors
+
+According to git shortlog, the following people contributed to the 2.13.0 release. Thank you to all contributors!
+
+Aaron Li, Ahmet Altay, Aizhamal Nurmamat kyzy, Alex Amato, Alexey Romanenko,
+Andrew Pilloud, Ankur Goenka, Anton Kedin, apstndb, Boyuan Zhang, Brian Hulette,
+Brian Quinlan, Chamikara Jayalath, Cyrus Maden, Daniel Chen, Daniel Oliveira,
+David Cavazos, David Moravek, David Yan, EdgarLGB, Etienne Chauchot, frederik2,
+Gleb Kanterov, Harshit Dwivedi, Harsh Vardhan, Heejong Lee, Hennadiy Leontyev,
+Henri-Mayeul de Benque, Ismaël Mejía, Jae-woo Kim, Jamie Kirkpatrick, Jan Lukavský,
+Jason Kuster, Jean-Baptiste Onofré, JohnZZGithub, Jozef Vilcek, Juta, Kenneth Jung,
+Kenneth Knowles, Kyle Weaver, Łukasz Gajowy, Luke Cwik, Mark Liu, Mathieu Blanchard,
+Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia, Mike Kaplinskiy,
+Mike Pedersen, Mikhail Gryzykhin, Mikhail-Ivanov, Niklas Hansson, pabloem,
+Pablo Estrada, Pranay Nanda, Reuven Lax, Richard Moorhead, Robbe Sneyders,
+Robert Bradshaw, Robert Burke, Roman van der Krogt, rosetn, Rui Wang, Ryan Yuan,
+Sam Whittle, sudhan499, Sylwester Kardziejonek, Ted, Thomas Weise, Tim Robertson,
+ttanay, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Xinyu Liu, Yifan Zou,
+yoshiki.obata, Yueyang Qiu
diff --git a/website/src/_posts/2019-05-30-beam-kata-release.md b/website/src/_posts/2019-05-30-beam-kata-release.md
new file mode 100644
index 0000000..89f5a54
--- /dev/null
+++ b/website/src/_posts/2019-05-30-beam-kata-release.md
@@ -0,0 +1,59 @@
+---
+layout: post
+title: "Apache Beam Katas"
+date: 2019-05-30 00:00:01 -0800
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - henryken
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+We are happy to announce
+[Apache Beam Katas](https://github.com/apache/beam/tree/master/learning/katas), a set of
+interactive Beam coding exercises (i.e. [code katas](http://codekata.com/)) that can help you in
+learning Apache Beam concepts and programming model hands-on.
+
+<!--more-->
+
+Beam Katas objective is to provide a series of structured hands-on learning experiences for learners
+to understand about Apache Beam and its SDKs by solving exercises with gradually increasing
+complexity. It is built based on
+[JetBrains Educational Products](https://www.jetbrains.com/education/). Beam Katas is available for
+both Java and Python SDKs. Currently we have about 20 lessons that cover Apache Beam fundamentals,
+such as core transforms, common transforms, and simple use case (word count), with more katas to
+be added in the coming future.
+
+
+To start with the courses, you can simply download
+[IntelliJ Edu](https://www.jetbrains.com/education/download/#section=idea) or
+[PyCharm Edu](https://www.jetbrains.com/education/download/#section=pycharm-edu) and then browse
+the integrated Stepik courses from the menu. Search for “Beam Katas” and once the course is loaded
+on the IDE, you’re good to go.
+
+We have plans to add more katas covering more topics including some of the intermediate and
+advanced ones in the coming future, such as windowing, streaming, and use case patterns. We would
+also like to welcome you to [contribute](https://github.com/apache/beam) by building and adding more katas that you think would be
+useful for people to learn more about Apache Beam, and eventually become Beam Masters!
+
+<br/>
+
+<img src="{{ "/images/blog/beam-kata/beam-kata-intellij-edu-1.png" | prepend: site.baseurl }}" alt="Beam Kata - IntelliJ Edu" width="363" height="350">
+<img src="{{ "/images/blog/beam-kata/beam-kata-intellij-edu-2.png" | prepend: site.baseurl }}" alt="Beam Kata - IntelliJ Edu" width="455" height="350">
+
+<img src="{{ "/images/blog/beam-kata/beam-kata-pycharm-edu-1.png" | prepend: site.baseurl }}" alt="Beam Kata - PyCharm Edu" width="363" height="350">
+<img src="{{ "/images/blog/beam-kata/beam-kata-pycharm-edu-2.png" | prepend: site.baseurl }}" alt="Beam Kata - PyCharm Edu" width="459" height="350">
diff --git a/website/src/_posts/2019-06-11-looping-timers.md b/website/src/_posts/2019-06-11-looping-timers.md
new file mode 100644
index 0000000..0ba8b52
--- /dev/null
+++ b/website/src/_posts/2019-06-11-looping-timers.md
@@ -0,0 +1,349 @@
+---
+layout: post
+title: "Looping timers in Apache Beam"
+date: 2019-06-11 00:00:01 -0800
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - rez
+ - klk
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+Apache Beam’s primitives let you build expressive data pipelines, suitable for a
+variety of use cases. One specific use case is the analysis of time series data
+in which continuous sequences across window boundaries are important. A few fun
+challenges arise as you tackle this type of data and in this blog we will
+explore one of those in more detail and make use of the Timer API
+([blog post]({{ site.baseurl }}/blog/2017/08/28/timely-processing.html))
+using the "looping timer" pattern.
+
+<!--more-->
+
+With Beam in streaming mode, you can take streams of data and build analytical
+transforms to produce results on the data. But for time series data, the absence
+of data is useful information. So how can we produce results in the absence of
+data?
+
+Let's use a more concrete example to illustrate the requirement. Imagine you
+have a simple pipeline that sums the number of events coming from an IoT device
+every minute. We would like to produce the value 0 when no data has been seen
+within a specific time interval. So why can this get tricky? Well it is easy to
+build a simple pipeline that counts events as they arrive, but when there is no
+event, there is nothing to count!
+
+Let's build a simple pipeline to work with:
+
+```
+ // We will start our timer at 1 sec from the fixed upper boundary of our
+ // minute window
+ Instant now = Instant.parse("2000-01-01T00:00:59Z");
+
+ // ----- Create some dummy data
+
+ // Create 3 elements, incrementing by 1 minute and leaving a time gap between
+ // element 2 and element 3
+ TimestampedValue<KV<String, Integer>> time_1 =
+ TimestampedValue.of(KV.of("Key_A", 1), now);
+
+ TimestampedValue<KV<String, Integer>> time_2 =
+ TimestampedValue.of(KV.of("Key_A", 2),
+ now.plus(Duration.standardMinutes(1)));
+
+ // No Value for start time + 2 mins
+ TimestampedValue<KV<String, Integer>> time_3 =
+ TimestampedValue.of(KV.of("Key_A", 3),
+ now.plus(Duration.standardMinutes(3)));
+
+ // Create pipeline
+ PipelineOptions options = PipelineOptionsFactory.fromArgs(args).withValidation()
+ .as(PipelineOptions.class);
+
+ Pipeline p = Pipeline.create(options);
+
+ // Apply a fixed window of duration 1 min and Sum the results
+ p.apply(Create.timestamped(time_1, time_2, time_3))
+ .apply(
+ Window.<KV<String,Integer>>into(
+FixedWindows.<Integer>of(Duration.standardMinutes(1))))
+ .apply(Sum.integersPerKey())
+ .apply(ParDo.of(new DoFn<KV<String, Integer>, KV<String, Integer>>() {
+
+ @ProcessElement public void process(ProcessContext c) {
+ LOG.info("Value is {} timestamp is {}", c.element(), c.timestamp());
+ }
+ }));
+
+ p.run();
+```
+
+Running that pipeline will result in the following output:
+
+```
+INFO LoopingTimer - Value is KV{Key_A, 1} timestamp is 2000-01-01T00:00:59.999Z
+INFO LoopingTimer - Value is KV{Key_A, 3} timestamp is 2000-01-01T00:03:59.999Z
+INFO LoopingTimer - Value is KV{Key_A, 2} timestamp is 2000-01-01T00:01:59.999Z
+```
+
+> Note: The lack of order in the output should be expected, however the
+> key-window tuple is correctly computed.
+
+
+As expected, we see output in each of the interval windows which had a data
+point with a timestamp between the minimum and maximum value of the window.
+There was a data point at timestamps 00:00:59, 00:01:59 and 00:03:59, which
+fell into the following interval windows.
+
+* [00:00:00, 00:00:59.999)
+* [00:01:00, 00:01:59.999)
+* [00:03:00, 00:03:59.999)
+
+But as there was no data between 00:02:00 and 00:02:59, no value is produced
+for interval window [00:02:00,00:02:59.999).
+
+How can we get Beam to output values for that missing window? First, let’s walk
+through some options that do not make use of the Timer API.
+
+
+## Option 1: External heartbeat
+
+We can use an external system to emit a value for each time interval and inject
+it into the stream of data that Beam consumes. This simple option moves any
+complexity out of the Beam pipeline. But using an external system means we need
+to monitor this system and perform other maintenance tasks in tandem with the
+Beam pipeline.
+
+
+## Option 2: Use a generated source in the Beam pipeline
+
+We can use a generating source to emit the value using this code snippet:
+
+```
+pipeline.apply(GenerateSequence.
+ from(0).withRate(1,Duration.standardSeconds(1L)))
+```
+
+We can then:
+
+1. Use a DoFn to convert the value to zero.
+2. Flatten this value with the real source.
+3. Produce a PCollection which has ticks in every time interval.
+
+This is also a simple way of producing a value in each time interval.
+
+
+## Option 1 & 2 The problem with multiple keys
+
+Both options 1 and 2 work well for the case where there the pipeline processes a
+single key. Let’s now deal with the case where instead of 1 IoT device, there
+are 1000s or 100,000s of these devices, each with a unique key. To make option 1
+or option 2 work in this scenario, we need to carry out an extra step: creating
+a FanOut DoFn. Each tick needs to be distributed to all the potential keys, so
+we need to create a FanOut DoFn that takes the dummy value and generates a
+key-value pair for every available key.
+
+For example, let's assume we have 3 keys for 3 IoT devices, {key1,key2,key3}.
+Using the method we outlined in Option 2 when we get the first element from
+GenerateSequence, we need to create a loop in the DoFn that generates 3
+key-value pairs. These pairs become the heartbeat value for each of the IoT
+devices.
+
+And things get a lot more fun when we need to deal with lots of IoT devices,
+with a list of keys that are dynamically changing. We would need to add a
+transform that does a Distinct operation and feed the data produced as a
+side-input into the FanOut DoFn.
+
+
+## Option 3: Implementing a heartbeat using Beam timers
+
+So how do timers help? Well let's have a look at a new transform:
+
+Edit: Looping Timer State changed from Boolean to Long to allow for min value check.
+
+```java
+public static class LoopingStatefulTimer extends DoFn<KV<String, Integer>, KV<String, Integer>> {
+
+ Instant stopTimerTime;
+
+ LoopingStatefulTimer(Instant stopTime){
+ this.stopTimerTime = stopTime;
+ }
+
+ @StateId("loopingTimerTime")
+ private final StateSpec<ValueState<Long>> loopingTimerTime =
+ StateSpecs.value(BigEndianLongCoder.of());
+
+ @StateId("key")
+ private final StateSpec<ValueState<String>> key =
+ StateSpecs.value(StringUtf8Coder.of());
+
+ @TimerId("loopingTimer")
+ private final TimerSpec loopingTimer =
+ TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+ @ProcessElement public void process(ProcessContext c, @StateId("key") ValueState<String> key,
+ @StateId("loopingTimerTime") ValueState<Long> loopingTimerTime,
+ @TimerId("loopingTimer") Timer loopingTimer) {
+
+ // If the timer has been set already, or if the value is smaller than
+ // the current element + window duration, do not set
+ Long currentTimerValue = loopingTimerTime.read();
+ Instant nextTimerTimeBasedOnCurrentElement = c.timestamp().plus(Duration.standardMinutes(1));
+
+ if (currentTimerValue == null || currentTimerValue >
+ nextTimerTimeBasedOnCurrentElement.getMillis()) {
+ loopingTimer.set(nextTimerTimeBasedOnCurrentElement);
+ loopingTimerTime.write(nextTimerTimeBasedOnCurrentElement.getMillis());
+ }
+
+ // We need this value so that we can output a value for the correct key in OnTimer
+ if (key.read() == null) {
+ key.write(c.element().getKey());
+ }
+
+ c.output(c.element());
+ }
+
+ @OnTimer("loopingTimer")
+ public void onTimer(
+ OnTimerContext c,
+ @StateId("key") ValueState<String> key,
+ @TimerId("loopingTimer") Timer loopingTimer) {
+
+ LOG.info("Timer @ {} fired", c.timestamp());
+ c.output(KV.of(key.read(), 0));
+
+ // If we do not put in a “time to live” value, then the timer would loop forever
+ Instant nextTimer = c.timestamp().plus(Duration.standardMinutes(1));
+ if (nextTimer.isBefore(stopTimerTime)) {
+ loopingTimer.set(nextTimer);
+ } else {
+ LOG.info(
+ "Timer not being set as exceeded Stop Timer value {} ",
+ stopTimerTime);
+ }
+ }
+ }
+```
+
+There are two data values that the state API needs to keep:
+
+1. A boolean `timeRunning` value used to avoid resetting the timer if it’s
+ already running.
+2. A "*key*" state object value that allows us to store the key that we are
+ working with. This information will be needed in the `OnTimer` event later.
+
+We also have a Timer with the ID `**loopingTimer**` that acts as our per
+interval alarm clock. Note that the timer is an *event timer*. It fires based on
+the watermark, not on the passage of time as the pipeline runs.
+
+Next, let's unpack what's happening in the @ProcessElement block:
+
+The first element to come to this block will:
+
+1. Set the state of the `timerRunner` to True.
+2. Write the value of the key from the key-value pair into the key StateValue.
+3. The code sets the value of the timer to fire one minute after the elements
+ timestamp. Note that the maximum value allowed for this timestamp is
+ XX:XX:59.999. This places the maximum alarm value at the upper boundary of
+ the next time interval.
+4. Finally, we output the data from the `@ProcessElement` block using
+ `c.output`.
+
+In the @OnTimer block, the following occurs:
+
+1. The code emits a value with the key pulled from our key StateValue and a
+ value of 0. The timestamp of the event corresponds to the event time of the
+ timer firing.
+2. We set a new timer for one minute from now, unless we are past the
+ `stopTimerTime` value. Your use case will normally have more complex stopping
+ conditions, but we use a simple condition here to allow us to keep the
+ illustrated code simple. The topic of stopping conditions is discussed in
+ more detail later.
+
+And that's it, let's add our transform back into the pipeline:
+
+```java
+ // Apply a fixed window of duration 1 min and Sum the results
+ p.apply(Create.timestamped(time_1, time_2, time_3)).apply(
+ Window.<KV<String, Integer>>into(FixedWindows.<Integer>of(Duration.standardMinutes(1))))
+ // We use a combiner to reduce the number of calls in keyed state
+ // from all elements to 1 per FixedWindow
+ .apply(Sum.integersPerKey())
+ .apply(Window.into(new GlobalWindows()))
+ .apply(ParDo.of(new LoopingStatefulTimer(Instant.parse("2000-01-01T00:04:00Z"))))
+ .apply(Window.into(FixedWindows.of(Duration.standardMinutes(1))))
+ .apply(Sum.integersPerKey())
+ .apply(ParDo.of(new DoFn<KV<String, Integer>, KV<String, Integer>>() {
+
+ @ProcessElement public void process(ProcessContext c) {
+
+ LOG.info("Value is {} timestamp is {}", c.element(), c.timestamp());
+
+ }
+ }));
+```
+
+1. In the first part of the pipeline we create FixedWindows and reduce the value
+ per key down to a single Sum.
+2. Next we re-window the output into a GlobalWindow. Since state and timers are
+ per window, they must be set within the window boundary. We want the looping
+ timer to span all the fixed windows, so we set it up in the global window.
+3. We then add our LoopingStatefulTimer DoFn.
+4. Finally, we reapply the FixedWindows and Sum our values.
+
+This pipeline ensures that a value of zero exists for each interval window, even
+if the Source of the pipeline emitted a value in the minimum and maximum
+boundaries of the interval window. This means that we can mark the absence of
+data.
+
+You might question why we use two reducers with multiple `Sum.integersPerKey`.
+Why not just use one? Functionally, using one would also produce the correct
+result. However, putting two `Sum.integersPerKey` gives us a nice performance
+advantage. It reduces the number of elements from many to just one per time
+interval. This can reduce the number of reads of the State API during the
+`@ProcessElement` calls.
+
+Here is the logging output of running our modified pipeline:
+
+```
+INFO LoopingTimer - Timer @ 2000-01-01T00:01:59.999Z fired
+INFO LoopingTimer - Timer @ 2000-01-01T00:02:59.999Z fired
+INFO LoopingTimer - Timer @ 2000-01-01T00:03:59.999Z fired
+INFO LoopingTimer - Timer not being set as exceeded Stop Timer value 2000-01-01T00:04:00.000Z
+INFO LoopingTimer - Value is KV{Key_A, 1} timestamp is 2000-01-01T00:00:59.999Z
+INFO LoopingTimer - Value is KV{Key_A, 0} timestamp is 2000-01-01T00:02:59.999Z
+INFO LoopingTimer - Value is KV{Key_A, 2} timestamp is 2000-01-01T00:01:59.999Z
+INFO LoopingTimer - Value is KV{Key_A, 3} timestamp is 2000-01-01T00:03:59.999Z
+```
+
+Yay! We now have output from the time interval [00:01:00, 00:01:59.999), even
+though the source dataset has no elements in that interval.
+
+In this blog, we covered one of the fun areas around time series use cases and
+worked through several options, including an advanced use case of the Timer API.
+Happy looping everyone!
+
+**Note:** Looping timers is an interesting new use case for the Timer API and
+runners will need to add support for it with all of their more advanced
+feature sets. You can experiment with this pattern today using the
+DirectRunner. For other runners, please look out for their release notes on
+support for dealing with this use case in production.
+
+([Capability Matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/))
+
+
+Runner specific notes:
+Google Cloud Dataflow Runners Drain feature does not support looping timers (Link to matrix)
diff --git a/website/src/_posts/2019-07-31-beam-2.14.0.md b/website/src/_posts/2019-07-31-beam-2.14.0.md
new file mode 100644
index 0000000..310e55e
--- /dev/null
+++ b/website/src/_posts/2019-07-31-beam-2.14.0.md
@@ -0,0 +1,106 @@
+---
+layout: post
+title: "Apache Beam 2.14.0"
+date: 2019-07-31 00:00:01 -0800
+# If date above changes, still keep the old URL:
+permalink: /blog/2019/07/31/beam-2.14.0.html
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - anton
+ - altay
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.14.0 release of Beam. This release includes both improvements and new functionality.
+See the [download page]({{ site.baseurl }}/get-started/downloads/#2140-2019-08-01) for this release.<!--more-->
+For more information on changes in 2.14.0, check out the
+[detailed release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345431).
+
+## Highlights
+
+ * Python 3 support is extended to Python 3.6 and 3.7; in addition to various other Python 3 [improvements](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16890504).
+ * Spark portable runner (batch) now [available](https://lists.apache.org/thread.html/c43678fc24c9a1dc9f48c51c51950aedcb9bc0fd3b633df16c3d595a@%3Cuser.beam.apache.org%3E) for Java, Python, Go.
+ * Added new runner: Hazelcast Jet Runner. ([BEAM-7305](https://issues.apache.org/jira/browse/BEAM-7305))
+
+### I/Os
+
+* Schema support added to BigQuery reads. (Java) ([BEAM-6673](https://issues.apache.org/jira/browse/BEAM-6673))
+* Schema support added to JDBC source. (Java) ([BEAM-6674](https://issues.apache.org/jira/browse/BEAM-6674))
+* BigQuery support for `bytes` is fixed. (Python 3) ([BEAM-6769](https://issues.apache.org/jira/browse/BEAM-6769))
+* Added DynamoDB IO. (Java) ([BEAM-7043](https://issues.apache.org/jira/browse/BEAM-7043))
+* Added support unbounded reads with HCatalogIO (Java) ([BEAM-7450](https://issues.apache.org/jira/browse/BEAM-7450))
+* Added BoundedSource wrapper for SDF. (Python) ([BEAM-7443](https://issues.apache.org/jira/browse/BEAM-7443))
+* Added support for INCRBY/DECRBY operations in RedisIO. ([BEAM-7286](https://issues.apache.org/jira/browse/BEAM-7286))
+* Added Support for ValueProvider defined GCS Location for WriteToBigQuery with File Loads. (Java) (([BEAM-7603](https://issues.apache.org/jira/browse/BEAM-7603)))
+
+
+### New Features / Improvements
+
+* Python SDK add support for DoFn `setup` and `teardown` methods. ([BEAM-562](https://issues.apache.org/jira/browse/BEAM-562))
+* Python SDK adds new transforms: [ApproximateUnique](https://issues.apache.org/jira/browse/BEAM-6693), [Latest](https://issues.apache.org/jira/browse/BEAM-6695), [Reify](https://issues.apache.org/jira/browse/BEAM-7019), [ToString](https://issues.apache.org/jira/browse/BEAM-7021), [WithKeys](https://issues.apache.org/jira/browse/BEAM-7023).
+* Added hook for user-defined JVM initialization in workers. ([BEAM-6872](https://issues.apache.org/jira/browse/BEAM-6872))
+* Added support for SQL Row Estimation for BigQueryTable. ([BEAM-7513](https://issues.apache.org/jira/browse/BEAM-7513))
+* Auto sharding of streaming sinks in FlinkRunner. ([BEAM-5865](https://issues.apache.org/jira/browse/BEAM-5865))
+* Removed the Hadoop dependency from the external sorter. ([BEAM-7268](https://issues.apache.org/jira/browse/BEAM-7268))
+* Added option to expire portable SDK worker environments. ([BEAM-7348](https://issues.apache.org/jira/browse/BEAM-7348))
+* Beam does not relocate Guava anymore and depends only on its own vendored version of Guava. ([BEAM-6620](https://issues.apache.org/jira/browse/BEAM-6620))
+
+
+### Breaking Changes
+* Deprecated set/getClientConfiguration in Jdbc IO. ([BEAM-7263](https://issues.apache.org/jira/browse/BEAM-7263))
+
+
+### Bugfixes
+
+* Fixed reading of concatenated compressed files. (Python) ([BEAM-6952](https://issues.apache.org/jira/browse/BEAM-6952))
+* Fixed re-scaling issues on Flink >= 1.6 versions. ([BEAM-7144](https://issues.apache.org/jira/browse/BEAM-7144))
+* Fixed SQL EXCEPT DISTINCT behavior. ([BEAM-7194](https://issues.apache.org/jira/browse/BEAM-7194))
+* Fixed OOM issues with bounded Reads for Flink Runner. ([BEAM-7442](https://issues.apache.org/jira/browse/BEAM-7442))
+* Fixed HdfsFileSystem to correctly match directories. ([BEAM-7561](https://issues.apache.org/jira/browse/BEAM-7561))
+* Upgraded Spark runner to use spark version 2.4.3. ([BEAM-7265](https://issues.apache.org/jira/browse/BEAM-7265))
+* Upgraded Jackson to version 2.9.9. ([BEAM-7465](https://issues.apache.org/jira/browse/BEAM-7465))
+* Various other bug fixes and performance improvements.
+
+
+### Known Issues
+
+* Do **NOT** use Python MongoDB source in this release. Python MongoDB source [added](https://issues.apache.org/jira/browse/BEAM-5148) in this release has a known issue that can result in data loss. See ([BEAM-7866](https://issues.apache.org/jira/browse/BEAM-7866)) for details.
+
+
+## List of Contributors
+
+According to git shortlog, the following people contributed to the 2.14.0 release. Thank you to all contributors!
+
+Ahmet Altay, Aizhamal Nurmamat kyzy, Ajo Thomas, Alex Amato, Alexey Romanenko,
+Alexey Strokach, Alex Van Boxel, Alireza Samadian, Andrew Pilloud,
+Ankit Jhalaria, Ankur Goenka, Anton Kedin, Aryan Naraghi, Bartok Jozsef,
+Bora Kaplan, Boyuan Zhang, Brian Hulette, Cam Mach, Chamikara Jayalath,
+Charith Ellawala, Charles Chen, Colm O hEigeartaigh, Cyrus Maden,
+Daniel Mills, Daniel Oliveira, David Cavazos, David Moravek, David Yan,
+Daniel Lescohier, Elwin Arens, Etienne Chauchot, Fábio Franco Uechi,
+Finch Keung, Frederik Bode, Gregory Kovelman, Graham Polley, Hai Lu, Hannah Jiang,
+Harshit Dwivedi, Harsh Vardhan, Heejong Lee, Henry Suryawirawan,
+Ismaël Mejía, Jan Lukavský, Jean-Baptiste Onofré, Jozef Vilcek, Juta, Kai Jiang,
+Kamil Wu, Kasia Kucharczyk, Kenneth Knowles, Kyle Weaver, Lara Schmidt,
+Łukasz Gajowy, Luke Cwik, Manu Zhang, Mark Liu, Matthias Baetens,
+Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia,
+Mikhail Gryzykhin, Ming Liang, Neville Li, Pablo Estrada, Paul Suganthan,
+Peter Backx, Rakesh Kumar, Rasmi Elasmar, Reuven Lax, Reza Rokni, Robbe Sneyders,
+Robert Bradshaw, Robert Burke, Rose Nguyen, Rui Wang, Ruoyun Huang,
+Shoaib Zafar, Slava Chernyak, Steve Niemitz, Tanay Tummalapalli, Thomas Weise,
+Tim Robertson, Tim van der Lippe, Udi Meiri, Valentyn Tymofieiev, Varun Dhussa,
+Viktor Gerdin, Yichi Zhang, Yifan Mai, Yifan Zou, Yueyang Qiu.
diff --git a/website/src/_posts/2019-08-22-beam-2.15.0.md b/website/src/_posts/2019-08-22-beam-2.15.0.md
new file mode 100644
index 0000000..4346dcf
--- /dev/null
+++ b/website/src/_posts/2019-08-22-beam-2.15.0.md
@@ -0,0 +1,87 @@
+---
+layout: post
+title: "Apache Beam 2.15.0"
+date: 2019-08-22 00:00:01 -0800
+# Date above corrected but keep the old URL:
+permalink: /blog/2019/08/22/beam-2.15.0.html
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - yifanzou
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.15.0 release of Beam. This release includes both improvements and new functionality.
+See the [download page]({{ site.baseurl }}/get-started/downloads/#2150-2019-08-22) for this release.<!--more-->
+For more information on changes in 2.15.0, check out the
+[detailed release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345489).
+
+## Highlights
+
+ * Vendored Guava was upgraded to version 26.0.
+ * Support multi-process execution on the FnApiRunner for Python. ([BEAM-3645](https://issues.apache.org/jira/browse/BEAM-3645))
+
+
+### I/Os
+
+* Add AvroIO.sink for IndexedRecord (FileIO compatible). ([BEAM-6480](https://issues.apache.org/jira/browse/BEAM-6480))
+* Add support for writing to BigQuery clustered tables. ([BEAM-5191](https://issues.apache.org/jira/browse/BEAM-5191))
+
+### New Features / Improvements
+
+* Support ParquetTable in SQL. ([BEAM-7728](https://issues.apache.org/jira/browse/BEAM-7728))
+* Add hot key detection to Dataflow Runner. ([BEAM-7820](https://issues.apache.org/jira/browse/BEAM-7820))
+* Support schemas in the JDBC sink. ([BEAM-6675](https://issues.apache.org/jira/browse/BEAM-6675))
+* Report GCS throttling time to Dataflow autoscaler for better autoscaling. ([BEAM-7667](https://issues.apache.org/jira/browse/BEAM-7667))
+* Support transform_name_mapping option in Python SDK for `--update` use. ([BEAM-7761](https://issues.apache.org/jira/browse/BEAM-7761))
+* Dependency: Upgrade Jackson databind to version 2.9.9.3 ([BEAM-7880](https://issues.apache.org/jira/browse/BEAM-7880))
+
+### Bugfixes
+
+* Various bug fixes and performance improvements.
+
+
+### Known Issues
+
+* [BEAM-7616](https://issues.apache.org/jira/browse/BEAM-7616) urlopen calls may get stuck. (Regression from 2.14.0)
+* [BEAM-8111](https://issues.apache.org/jira/browse/BEAM-8111) SchemaCoder fails on Dataflow, preventing the use of SqlTransform and schema-aware transforms. (Regression from 2.14.0)
+
+
+### Breaking Changes
+* `--region` flag will be a required flag in the future for Dataflow. A warning is added to warn for this future change. ([BEAM-7833](https://issues.apache.org/jira/browse/BEAM-7833))
+
+
+
+## List of Contributors
+
+ According to git shortlog, the following people contributed to the 2.15.0 release. Thank you to all contributors!
+
+ Ahmet Altay, Alexey Romanenko, Alex Goos, Alireza Samadian, Andrew Pilloud, Ankur Goenka,
+Anton Kedin, Aryan Naraghi, Bartok Jozsef, bmv126, B M VISHWAS, Boyuan Zhang,
+Brian Hulette, brucearctor, Cade Markegard, Cam Mach, Chad Dombrova,
+Chaim Turkel, Chamikara Jayalath, Charith Ellawala, Claire McGinty, Craig Chambers,
+Daniel Oliveira, David Cavazos, David Moravek, Dominic Mitchell, Dustin Rhodes,
+Etienne Chauchot, Filipe Regadas, Gleb Kanterov, Gunnar Schulze, Hannah Jiang,
+Heejong Lee, Henry Suryawirawan, Ismaël Mejía, Ivo Galic, Jan Lukavský,
+Jawad, Juta, Juta Staes, Kai Jiang, Kamil Wasilewski, Kasia Kucharczyk,
+Kenneth Jung, Kenneth Knowles, Kyle Weaver, Lily Li, Logan HAUSPIE, lostluck,
+Łukasz Gajowy, Luke Cwik, Mark Liu, Matt Helm, Maximilian Michels,
+Michael Luckey, Mikhail Gryzykhin, Neville Li, Nicholas Rucci, pabloem,
+Pablo Estrada, Paul King, Paul Suganthan, Raheel Khan, Rakesh Kumar,
+Reza Rokni, Robert Bradshaw, Robert Burke, rosetn, Rui Wang, Ryan Skraba, RyanSkraba,
+Sahith Nallapareddy, Sam Rohde, Sam Whittle, Steve Niemitz, Tanay Tummalapalli, Thomas Weise,
+Tianyang Hu, ttanay, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Wout Scheepers,
+yanzhi, Yekut, Yichi Zhang, Yifan Zou, yoshiki.obata, Yueyang Qiu, Yunqing Zhou
diff --git a/website/src/_posts/2019-09-04-gsoc-19.md b/website/src/_posts/2019-09-04-gsoc-19.md
new file mode 100644
index 0000000..8fa16c6
--- /dev/null
+++ b/website/src/_posts/2019-09-04-gsoc-19.md
@@ -0,0 +1,93 @@
+---
+layout: post
+title: "Google Summer of Code '19"
+date: 2019-09-04 00:00:01 -0800
+permalink: /blog/2019/09/04/gsoc-19.html
+excerpt_separator: <!--more-->
+categories: blog gsoc
+authors:
+- ttanay
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+
+Google Summer of Code was an amazing learning experience for me.
+I contributed to open source, learned about Apache Beam’s internals and worked with the best engineers in the world.
+
+<!--more-->
+
+## Motivation
+Two of my friends had participated in GSoC in 2018. I was intrigued by their experience.
+The idea of working on open-source software that could potentially be used by developers across the world, while being mentored by the best people in a field was exciting!
+So, I decided to give Google Summer of Code a shot this year.
+
+## What is Google Summer of Code?
+[Google Summer of Code](https://summerofcode.withgoogle.com/) is a global program hosted by Google focused on introducing students to open source software development.
+Students work on a 3 month programming project with an open source organization during their break from university.
+
+## Why Apache Beam?
+While interning at [Atlan](https://atlan.com/), I discovered the field of Data Engineering. I found the challenges and the discussions of the engineers there interesting. While researching for my internship project, I came across the Streaming Systems book. It introduced me to the unified model of Apache Beam for Batch and Streaming Systems, which I was fascinated by.
+I wanted to explore Data Engineering, so for GSoC, I wanted to work on a project in that field. Towards the end of my internship, I started contributing to Apache Airflow(very cool project) and Apache Beam, hoping one of them would participate in GSoC. I got lucky!
+
+[Also, Spotify’s Discover Weekly uses Apache Beam!](https://youtu.be/U2eWLb-LD44)
+
+## Preparation
+I had already read the [Streaming Systems book](http://streamingsystems.net/). So, I had an idea of the concepts that Beam is built on, but had never actually used Beam.
+Before actually submitting a proposal, I went through a bunch of resources to make sure I had a concrete understanding of Beam.
+I read the [Streaming 101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101) and [Streaming 102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102) blogs by Tyler Akidau. They are the perfect introduction to Beam’s unified model for Batch and Streaming.
+In addition, I watched all Beam talks on YouTube. You can find them on the [Beam Website](https://beam.apache.org/documentation/resources/videos-and-podcasts/).
+Beam has really good documentation. The [Programming Guide](https://beam.apache.org/documentation/programming-guide/) lays out all of Beam’s concepts really well. [Beam’s execution model](https://beam.apache.org/documentation/runtime/model) is also documented well and is a must-read to understand how Beam processes data.
+[waitingforcode.com](https://www.waitingforcode.com/apache-beam) also has good blog posts about Beam concepts.
+To get a better sense of the Beam codebase, I played around with it and worked on some PRs to understand Beam better and got familiar with the test suite and workflows.
+
+## GSoC Journey
+GSoC has 2 phases. The first is the Community Bonding period in which students get familiar with the project and the community. The other is the actual Coding Period in which students work on their projects. Since the Coding Period has three evaluations spaced out by a month, I divided my project into three parts focusing on the implementation, tests, and documentation or improvements.
+
+### Project
+My project([BEAM-6611](https://issues.apache.org/jira/browse/BEAM-6611)) added support for File Loads method of inserting data into BigQuery for streaming pipelines. It builds on PR - [#7655](https://github.com/apache/beam/pull/7655) for [BEAM-6553](https://issues.apache.org/jira/browse/BEAM-6553) that added support in the Python SDK for writing to BigQuery using File Loads method for Batch pipelines. Streaming pipelines with non-default Windowing, Triggering and Accumulation mode can write data to BigQuery using file loads method. In case of failure, the pipeline will fail atomically. This means that each record will be loaded into BigQuery at-most-once.
+You can find my proposal [here](https://docs.google.com/document/d/15Peyd3Z_wu5rvGWw8lMLpZuTyyreM_JOAEFFWvF97YY/edit?usp=sharing).
+
+### Community Bonding
+When GSoC started, my semester end exams had not yet finished. As a result, I couldn’t get much done. I worked on three PTransforms for the Python SDK - Latest, WithKeys and Reify.
+
+### Coding Period I
+In this period, I wrote some Integration Tests for the BigQuery sink using Streaming Inserts in streaming mode. I worked on a failing integration test for my project. I also finished the implementation of my project. But, one PostCommit test didn’t pass. I realized that the matcher for the Integration Test that queried BigQuery for the results was intended to be used in Batch mode. So, I wrote a version of the matcher to work in streaming mode.
+
+### Coding Period II
+Even after I had added the matcher for streaming mode, the PostComit tests did not pass. A test was being run even though it was not specified. I isolated the failure to a [limitation](https://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess/multiprocess.html#other-differences-in-test-running) of the multiprocess plugin for [nose(a Python test framework)](https://nose.readthedocs.io/en/latest/) due to which it found more tests than had been specified. It took me a while to figure this out. In this period, changes for my project got merged.
+I also worked on small issues related to testing.
+
+This period was marked by a few exciting events:
+ - Ending up in the top #100 contributors to apache/beam.
+ - My first ever PR Review on an open source project.
+
+
+
+### Coding Period III
+This was the final coding period before the program ended. Since my project was merged earlier than expected, my mentor suggested another issue([BEAM-7742](https://issues.apache.org/jira/browse/BEAM-7742)) in the same area - BigQueryIO, that I found interesting. So, I worked on partitioning written files in BigQuery to ensure that all load jobs triggered adhere to the load job size limitations specified for BigQuery.
+While working on my project, I was using a pipeline that uses PubSub as a source and BigQuery as a sink to validate my changes. My mentor suggested we add them to the Beam test suite as it would be the ultimate test for BigQueryIO. I also worked on adding this test to Beam.
+
+You can find the list of PRs I worked on [here](https://github.com/apache/beam/pulls?utf8=%E2%9C%93&q=is%3Apr+author%3Attanay).
+
+## Conclusion
+GSoC has been a lesson in discipline and goal-setting for me. Deciding what I wanted to work on and how much I wanted to get done each week was an important lesson.
+I had never worked remotely, so this was a new experience. Although I struggled with it initially, I appreciate the flexibility that it comes with.
+I also had a lot of fun learning about Apache Beam’s internals, and other tools in the same ecosystem.
+This was also the first time I had written code with a test-first approach.
+
+I thank my mentor - Pablo Estrada, Apache Beam, The Apache Software Foundation and Google Summer of Code for this opportunity. I am also grateful to my mentor for helping me with everything I needed and more, and the Apache Beam community for being supportive and encouraging.
+
+With the right effort, perseverance, conviction, and a plan, anything is possible. Anything.
diff --git a/website/src/_posts/2019-10-07-beam-2.16.0.md b/website/src/_posts/2019-10-07-beam-2.16.0.md
new file mode 100644
index 0000000..41d36c4
--- /dev/null
+++ b/website/src/_posts/2019-10-07-beam-2.16.0.md
@@ -0,0 +1,102 @@
+---
+layout: post
+title: "Apache Beam 2.16.0"
+date: 2019-10-07 00:00:01 -0800
+# Date above corrected but keep the old URL:
+permalink: /blog/2019/10/07/beam-2.16.0.html
+excerpt_separator: <!--more-->
+categories: blog
+authors:
+ - markliu
+
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+We are happy to present the new 2.16.0 release of Beam. This release includes both improvements and new functionality.
+See the [download page]({{ site.baseurl }}/get-started/downloads/#2160-2019-10-07) for this release.<!--more-->
+For more information on changes in 2.16.0, check out the
+[detailed release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345494).
+
+## Highlights
+
+ * Customizable Docker container images released and supported by Beam portable runners on Python 2.7, 3.5, 3.6, 3.7. ([BEAM-7907](https://issues.apache.org/jira/browse/BEAM-7907))
+ * Integration improvements for Python Streaming on Dataflow including service features like autoscaling, drain, update, streaming engine and counter updates.
+
+
+### New Features / Improvements
+
+* A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation. ([BEAM-7013](https://issues.apache.org/jira/browse/BEAM-7013))
+* Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow. ([BEAM-7045](https://issues.apache.org/jira/browse/BEAM-7045))
+* Add SetState in Python sdk. ([BEAM-7741](https://issues.apache.org/jira/browse/BEAM-7741))
+* Add hot key detection to Dataflow Runner. ([BEAM-7820](https://issues.apache.org/jira/browse/BEAM-7820))
+* Add ability to get the list of submitted jobs from gRPC JobService. ([BEAM-7927](https://issues.apache.org/jira/browse/BEAM-7927))
+* Portable Flink pipelines can now be bundled into executable jars. ([BEAM-7966](https://issues.apache.org/jira/browse/BEAM-7966), [BEAM-7967](https://issues.apache.org/jira/browse/BEAM-7967))
+* SQL join selection should be done in planner, not in expansion to PTransform. ([BEAM-6114](https://issues.apache.org/jira/browse/BEAM-6114))
+* A Python Sink for BigQuery with File Loads in Streaming. ([BEAM-6611](https://issues.apache.org/jira/browse/BEAM-6611))
+* Python BigQuery sink should be able to handle 15TB load job quota. ([BEAM-7588](https://issues.apache.org/jira/browse/BEAM-7588))
+* Spark portable runner: reuse SDK harness. ([BEAM-7600](https://issues.apache.org/jira/browse/BEAM-7600))
+* BigQuery File Loads to work well with load job size limits. ([BEAM-7742](https://issues.apache.org/jira/browse/BEAM-7742))
+* External environment with containerized worker pool. ([BEAM-7980](https://issues.apache.org/jira/browse/BEAM-7980))
+* Use OffsetRange as restriction for OffsetRestrictionTracker. ([BEAM-8014](https://issues.apache.org/jira/browse/BEAM-8014))
+* Get logs for SDK worker Docker containers. ([BEAM-8015](https://issues.apache.org/jira/browse/BEAM-8015))
+* PCollection boundedness is tracked and propagated in python sdk. ([BEAM-8088](https://issues.apache.org/jira/browse/BEAM-8088))
+
+
+### Dependency Changes
+
+* Upgrade "com.amazonaws:amazon-kinesis-producer" to version 0.13.1. ([BEAM-7894](https://issues.apache.org/jira/browse/BEAM-7894))
+* Upgrade to joda time 2.10.3 to get updated TZDB. ([BEAM-8161](https://issues.apache.org/jira/browse/BEAM-8161))
+* Upgrade Jackson to version 2.9.10. ([BEAM-8299](https://issues.apache.org/jira/browse/BEAM-8299))
+* Upgrade grpcio minimum required version to 1.12.1. ([BEAM-7986](https://issues.apache.org/jira/browse/BEAM-7986))
+* Upgrade funcsigs minimum required version to 1.0.2 in Python2. ([BEAM-7060](https://issues.apache.org/jira/browse/BEAM-7060))
+* Upgrade google-cloud-pubsub maximum required version to 1.0.0. ([BEAM-5539](https://issues.apache.org/jira/browse/BEAM-5539))
+* Upgrade google-cloud-bigtable maximum required version to 1.0.0. ([BEAM-5539](https://issues.apache.org/jira/browse/BEAM-5539))
+* Upgrade dill version to 0.3.0. ([BEAM-8324](https://issues.apache.org/jira/browse/BEAM-8324))
+
+
+### Bugfixes
+
+* Various bug fixes and performance improvements.
+
+
+### Known Issues
+
+* Given that Python 2 will reach EOL on Jan 1 2020, Python 2 users of Beam will now receive a warning that new releases of Apache Beam will soon support Python 3 only.
+* Filesystems not properly registered using FileIO.write in FlinkRunner. ([BEAM-8303](https://issues.apache.org/jira/browse/BEAM-8303))
+* Performance regression in Java DirectRunner in streaming mode. ([BEAM-8363](https://issues.apache.org/jira/browse/BEAM-8363))
+
+
+## List of Contributors
+
+ According to git shortlog, the following people contributed to the 2.16.0 release. Thank you to all contributors!
+
+Ahmet Altay, Alex Van Boxel, Alexey Romanenko, Alexey Strokach, Alireza Samadian,
+Andre-Philippe Paquet, Andrew Pilloud, Ankur Goenka, Anton Kedin, Aryan Naraghi,
+B M VISHWAS, Bartok Jozsef, Bill Neubauer, Boyuan Zhang, Brian Hulette, Bruno Volpato,
+Chad Dombrova, Chamikara Jayalath, Charith Ellawala, Charles Chen, Claire McGinty,
+Cyrus Maden, Daniel Oliveira, Dante, David Cavazos, David Moravek, David Yan,
+Dominic Mitchell, Elias Djurfeldt, Enrico Canzonieri, Etienne Chauchot, Gleb Kanterov,
+Hai Lu, Hannah Jiang, Heejong Lee, Ian Lance Taylor, Ismaël Mejía, Jack Whelpton,
+James Wen, Jan Lukavský, Jean-Baptiste Onofré, Jofre, Kai Jiang, Kamil Wasilewski,
+Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, Kirill Kozlov, Kohki YAMAGIWA,
+Kyle Weaver, Kyle Winkelman, Ludovic Post, Luis Enrique Ortíz Ramirez, Luke Cwik,
+Mark Liu, Maximilian Michels, Michal Walenia, Mike Kaplinskiy, Mikhail Gryzykhin,
+NING KANG, Oliver Henlich, Pablo Estrada, Rakesh Kumar, Renat Nasyrov, Reuven Lax,
+Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Skraba, Sahith Nallapareddy,
+Salman Raza, Sam Rohde, Saul Chavez, Shoaib, Shoaib Zafar, Slava Chernyak, Tanay Tummalapalli,
+Thinh Ha, Thomas Weise, Tianzi Cai, Tim van der Lippe, Tomer Zeltzer, Tudor Marian,
+Udi Meiri, Valentyn Tymofieiev, Yichi Zhang, Yifan Zou, Yueyang Qiu, gxercavins,
+jesusrv1103, lostluck, matt-darwin, mrociorg, ostrokach, parahul, rahul8383, rosetn,
+sunjincheng121, the1plummie, ttanay, tvalentyn, venn001, yoshiki.obata, Łukasz Gajowy
diff --git a/website/src/_sass/_global.sass b/website/src/_sass/_global.sass
index 8bdfd68..db0deb2 100644
--- a/website/src/_sass/_global.sass
+++ b/website/src/_sass/_global.sass
@@ -68,4 +68,4 @@
.container-main-content
padding: 0 20px
- position: relative
+ position: relative
\ No newline at end of file
diff --git a/website/src/_sass/capability-matrix.scss b/website/src/_sass/capability-matrix.scss
index 2e71f4a..6bd629a 100644
--- a/website/src/_sass/capability-matrix.scss
+++ b/website/src/_sass/capability-matrix.scss
@@ -136,6 +136,6 @@
text-align:center;
cursor:pointer;
position:absolute;
- font-size:12px;
+ font-size:16px;
font-weight:normal;
}
diff --git a/website/src/community/in-person.md b/website/src/community/in-person.md
index 84df7b0..47f4741 100644
--- a/website/src/community/in-person.md
+++ b/website/src/community/in-person.md
@@ -28,7 +28,7 @@
## Meetups
-We occassionally meetup in various locations around the globe. Active or to-be-started meetups include:
+We occasionally meet up in various locations around the globe. Active or to-be-started meetups include:
| Meetup City | Name |
| ----------------- | ---------------|
@@ -37,11 +37,12 @@
| San Francisco | [Bay Area Apache Beam](https://www.meetup.com/San-Francisco-Apache-Beam/) |
| Los Angeles | [Los Angeles Apache Beam](https://www.meetup.com/Los-Angeles-Apache-Beam/) |
| Washington DC | [Washington DC Apache Beam Meetup](https://www.meetup.com/DC-Apache-Beam/) |
-| New York City | [New York Apache Beam](https://www.meetup.com/New-York-Apache-Beam/) |:
+| New York City | [New York Apache Beam](https://www.meetup.com/New-York-Apache-Beam/) |
+| Paris | [Paris Apache Beam](https://www.meetup.com/Paris-Apache-Beam-Meetup/) |:
{:.table}
-The above are the meetups that are already known to the community (please add if you are organizing one!). For Meetups that are tagged with 'Apache Beam', see the [list](https://www.meetup.com/topics/apache-beam/).
+The above are the meetups that are already known to the community (please add if you are organizing one!). For Meetups that are tagged with 'Apache Beam', see the [list](https://www.meetup.com/topics/apache-beam/).
## Conference Talks
-You can find a list [here](https://docs.google.com/spreadsheets/d/1CloF63FOKSPM6YIuu8eExjhX6xrIiOp5j4zPbSg3Apo/)
\ No newline at end of file
+You can find a list [here](https://docs.google.com/spreadsheets/d/1CloF63FOKSPM6YIuu8eExjhX6xrIiOp5j4zPbSg3Apo/)
diff --git a/website/src/contribute/design-documents.md b/website/src/contribute/design-documents.md
index 18ba7ec..c4fbb89 100644
--- a/website/src/contribute/design-documents.md
+++ b/website/src/contribute/design-documents.md
@@ -3,6 +3,7 @@
title: 'Beam Design Documents'
section_menu: section-menu/contribute.html
permalink: /contribute/design-documents/
+redirect_to: https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
@@ -17,145 +18,3 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
-# Design Documents
-This is a collection of documents that may or may not be up to date.
-
-## Documents by category
-### Project Incubation (2016)
-- Technical Vision [[doc](https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPgXNQA1oD-rGooWOSwAqh8/edit)], [[slides](https://docs.google.com/presentation/d/1E9seGPB_VXtY_KZP4HngDPTbsu5RVZFFaTlwEYa88Zw)]
-- Repository Structure [[doc](https://docs.google.com/document/d/1mTeZED33Famq25XedbKeDlGIJRvtzCXjSfwH9NKQYUE)]
-- Flink runner: Current status and development roadmap [[doc](https://docs.google.com/document/d/1QM_X70VvxWksAQ5C114MoAKb1d9Vzl2dLxEZM4WYogo)]
-- Spark Runner Technical Vision [[doc](https://docs.google.com/document/d/1y4qlQinjjrusGWlgq-mYmbxRW2z7-_X5Xax-GG0YsC0)]
-- PPMC deep dive [[slides](https://docs.google.com/presentation/d/1uTb7dx4-Y2OM_B0_3XF_whwAL2FlDTTuq2QzP9sJ4Mg)]
-
-### Beam Model
-- Checkpoints [[doc](https://s.apache.org/FIWQ)]
-- A New DoFn [[doc](https://s.apache.org/a-new-dofn)], [[slides](https://s.apache.org/presenting-a-new-dofn)]
-- Proposed Splittable DoFn API changes [[doc](https://docs.google.com/document/d/1BGc8pM1GOvZhwR9SARSVte-20XEoBUxrGJ5gTWXdv3c)]
-- Splittable DoFn (Obsoletes Source API) [[doc](http://s.apache.org/splittable-do-fn)]
- - Reimplementing Beam API classes on top of Splittable DoFn on top of Source API [[doc](https://s.apache.org/sdf-via-source)]
- - New TextIO features based on SDF [[doc](http://s.apache.org/textio-sdf)]
- - Watch transform [[doc](http://s.apache.org/beam-watch-transform)]
- - Bundles w/ SplittableDoFns [[doc](https://s.apache.org/beam-bundles-backlog-splitting)]
-- State and Timers for DoFn [[doc](https://s.apache.org/beam-state)]
-- ContextFn [[doc](http://s.apache.org/context-fn)]
-- Static Display Data [[doc](https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM)]
-- Lateness (and Panes) in Apache Beam [[doc](https://s.apache.org/beam-lateness)]
-- Triggers in Apache Beam [[doc](https://s.apache.org/beam-triggers)]
-- Triggering is for sinks [[doc](https://s.apache.org/beam-sink-triggers)] (not implemented)
-- Pipeline Drain [[doc](https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8)]
-- Pipelines Considered Harmful [[doc](https://s.apache.org/no-beam-pipeline)]
-- Side-Channel Inputs [[doc](https://docs.google.com/document/d/1e_-MenoW2cQ-6-EGVVqfOR-B9FovVXqXyUm4-ZwlgKA)]
-- Dynamic Pipeline Options [[doc](https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o)]
-- SDK Support for Reading Dynamic PipelineOptions [[doc](https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8)]
-- Fine-grained Resource Configuration in Beam [[doc](https://docs.google.com/document/d/1N0y64dbzmukLLEy6M9CygdI_H88pIS3NtcOAkL5-oVw)]
-- External Join with KV Stores [[doc](https://docs.google.com/document/d/1B-XnUwXh64lbswRieckU0BxtygSV58hysqZbpZmk03A)]
-- Error Reporting Callback (WIP) [[doc](https://docs.google.com/document/d/1o2VXwCL97k3G-1BR9RSKNc6XtJTIA6SEKPMne91S67Y)]
-- Snapshotting and Updating Beam Pipelines [[doc](https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY)]
-- Requiring PTransform to set a coder on its resulting collections [[mail](https://lists.apache.org/thread.html/1dde0b5a93c2983cbab5f68ce7c74580102f5bb2baaa816585d7eabb@%3Cdev.beam.apache.org%3E)]
-- Support of @RequiresStableInput annotation [[doc](https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM)], [[mail](https://lists.apache.org/thread.html/ae3c838df060e47148439d1dad818d5e927b2a25ff00cc4153221dff@%3Cdev.beam.apache.org%3E)]
-- [PROPOSAL] @onwindowexpiration [[mail](https://lists.apache.org/thread.html/1dab7f17c97378e665928b11116cbd887dc7be93390ab26c593ee49a@%3Cdev.beam.apache.org%3E)]
-- AutoValue Coding and Row Support [[doc](https://docs.google.com/document/d/1ucoik4WzUDfilqIz3I1AuMHc1J8DE6iv7gaUCDI42BI)]
-
-### IO / Filesystem
-- IOChannelFactory Redesign [[doc](https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M)]
-- Configurable BeamFileSystem [[doc](https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8)]
-- New API for writing files in Beam [[doc](http://s.apache.org/fileio-write)]
-- Dynamic file-based sinks [[doc](https://docs.google.com/document/d/1Bd9mJO1YC8vOoFObJFupVURBMCl7jWt6hOgw6ClwxE4)]
-- Event Time and Watermarks in KafkaIO [[doc](https://docs.google.com/document/d/1DyWcLJpALRoUfvYUbiPCDVikYb_Xz2X7Co2aDUVVd4I)]
-- Exactly-once Kafka sink [[doc](https://lists.apache.org/thread.html/fb394e576e6e858205307b033c5a5c6cc3923a17606814a54036c570@%3Cdev.beam.apache.org%3E)]
-
-### Metrics
-- Get Metrics API: Metric Extraction via proto RPC API. [[doc](https://s.apache.org/get-metrics-api)]
-- Metrics API [[doc](http://s.apache.org/beam-metrics-api)]
-- I/O Metrics [[doc](https://s.apache.org/standard-io-metrics)]
-- Metrics extraction independent from runners / execution engines [[doc](https://s.apache.org/runner_independent_metrics_extraction)]
-- Watermark Metrics [[doc](https://docs.google.com/document/d/1ykjjG97DjVQP73jGbotGRbtK38hGvFbokNEOuNO4DAo)]
-- Support Dropwizard Metrics in Beam [[doc](https://docs.google.com/document/d/1-35iyCIJ9P4EQONlakgXBFRGUYoOLanq2Uf2sw5EjJw)]
-
-### Runners
-- Runner Authoring Guide [[doc](https://s.apache.org/beam-runner-guide)] (obsoletes [[doc](http://s.apache.org/beam-runner-api)] and [[doc](https://s.apache.org/beam-runner-1-pager)])
-- Composite PInputs, POutputs, and the Runner API [[doc](https://s.apache.org/beam-runner-composites)]
-- Side Input Architecture for Apache Beam [[doc](https://s.apache.org/beam-side-inputs-1-pager)]
-- Runner supported features plugin [[doc](https://s.apache.org/k79W)]
-- Structured streaming Spark Runner [[doc](https://s.apache.org/spark-structured-streaming-runner)]
-
-### SQL / Schema
-- Streams and Tables [[doc](https://s.apache.org/beam-streams-tables)]
-- Streaming SQL [[doc](http://s.apache.org/streaming-sql-spec)]
-- Schema-Aware PCollections [[doc](https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc)]
-- Pubsub to Beam SQL [[doc](https://docs.google.com/document/d/1554kJD33ovkBDvSNjasHu90L_EZOS26ZHr4ao1muS-A)]
-- Apache Beam Proposal: design of DSL SQL interface [[doc](https://docs.google.com/document/d/1uWXL_yF3UUO5GfCxbL6kWsmC8xCWfICU3RwiQKsk7Mk)]
-- Calcite/Beam SQL Windowing [[doc](https://docs.google.com/document/d/1yuG_fAnbAKEq3qz2jdf8qxyEIZ3xJAbCF1bbd_Y9Ia8)]
-- Reject Unsupported Windowing Strategies in JOIN [[doc](https://docs.google.com/document/d/1Me0orPfH6vEFjfsTGcZ5ELWg-sw4st1ZvXqYyr7Pexc)]
-- Beam DSL_SQL branch API review [[doc](https://s.apache.org/beam-sql-dsl-api-review)]
-- Complex Types Support for Beam SQL DDL [[mail](https://lists.apache.org/thread.html/c494e521cb6865b1ae19a68e8e653afc562df7744e8d08087249cbe0@%3Cdev.beam.apache.org%3E)]
-- [SQL] Reject unsupported inputs to Joins [[mail](https://lists.apache.org/thread.html/e7a442fa9cf6b76a5b435493170508f6c42fb9ccef9bcef434424f79@%3Cdev.beam.apache.org%3E)]
-- Integrating runners & IO [[doc](https://docs.google.com/document/d/1ZFVlnldrIYhUgOfxIT2JcmTFFSWTl4HwAnQsnwiNL1g)]
-- Beam SQL Pipeline Options [[doc](https://docs.google.com/document/d/1UTsSBuruJRfGnVOS9eXbQI6NauCD4WnSAPgA_Y0zjdk)]
-- Unbounded limit [[doc](https://docs.google.com/document/d/13zeTewHH9nfwhSlcE4x77WQwr1U2Z4sTiNRjOXUj2aw)]
-- Portable Beam Schemas [[doc](https://s.apache.org/beam-schemas)]
-
-### Portability
-- Fn API
- - Apache Beam Fn API Overview [[doc](https://s.apache.org/beam-fn-api)]
- - Processing a Bundle [[doc](https://s.apache.org/beam-fn-api-processing-a-bundle)]
- - Progress [[doc](https://s.apache.org/beam-fn-api-progress-reporting)]
- - Graphical view of progress [[doc](https://docs.google.com/document/d/1Dx18qBTvFWNqwLeecemOpKfleKzFyeV3Qwh71SHATvY)]
- - Fn State API and Bundle Processing [[doc](https://s.apache.org/beam-fn-state-api-and-bundle-processing)]
- - Checkpointing and splitting of Beam bundles over the Fn API, with application to SDF [[doc](https://s.apache.org/beam-breaking-fusion)]
- - How to send and receive data [[doc](https://s.apache.org/beam-fn-api-send-and-receive-data)]
- - Defining and adding SDK Metrics [[doc](https://s.apache.org/beam-fn-api-metrics)]
- - SDK harness container contract [[doc](https://s.apache.org/beam-fn-api-container-contract)]
- - Structure and Lifting of Combines [[doc](https://s.apache.org/beam-runner-api-combine-model)]
-- Cross-language Beam Pipelines [[doc](https://s.apache.org/beam-mixed-language-pipelines)]
-- SDK X with Runner Y using Runner API [[doc](https://s.apache.org/beam-job-api)]
-- Flink Portable Runner Overview [[doc](https://s.apache.org/portable-flink-runner-overview)]
-- Launching portable pipeline on Flink Runner [[doc](https://docs.google.com/document/d/1xOaEEJrMmiSHprd-WiYABegfT129qqF-idUBINjxz8s)]
-- Portability support [[table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI)]
-- Portability Prototype [[doc](https://s.apache.org/beam-portability-team-doc)]
-- Portable Artifact Staging [[doc](https://docs.google.com/document/d/12zNk3O2nhTB8Zmxw5U78qXrvlk5r42X8tqF248IDlpI)]
-- Portable Beam on Flink [[doc](https://s.apache.org/portable-beam-on-flink)]
-- Portability API: How to Checkpoint and Split Bundles [[doc](https://s.apache.org/beam-checkpoint-and-split-bundles)]
-- Portability API: How to Finalize Bundles [[doc](https://s.apache.org/beam-finalizing-bundles)]
-- Side Input in Universal Reference Runner [[doc](https://docs.google.com/document/d/13N0OJ7QJm81wcgu13pi9GuN29UUxN2iIFn_H8lKpDks)]
-- Spark Portable Runner Overview [[doc](https://docs.google.com/document/d/1j8GERTiHUuc6CzzCXZHc38rBn41uWfATBh2-5JN8hro)]
-- Cross-Language Pipelines & Legacy IO [[doc](https://s.apache.org/beam-cross-language-io)]
-
-### Build / Testing
-- More Expressive PAsserts [[doc](https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4)]
-- Mergebot design document [[doc](https://docs.google.com/document/d/18iFnW6egjqd_ADXCTQcuAkkz3J96LHdV5DlYUhXHf0M)]
-- Performance tests for commonly used file-based I/O PTransforms [[doc](https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE)]
-- Performance tests results analysis and basic regression detection [[doc](https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA)]
-- Eventual PAssert [[doc](https://docs.google.com/document/d/1X_3KH_6QyfOSnh5kNK-fHlkEDrwPVpA2RnRggMMxhUk)]
-- Testing I/O Transforms in Apache Beam [[doc](https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I)]
-- Reproducible Environment for Jenkins Tests By Using Container [[doc](https://docs.google.com/document/d/1U7FeVMiHiBP-pFm4ULotqG1QqZY0fi7g9ZwTmeIgvvM)]
-- Keeping precommit times fast [[doc](https://docs.google.com/document/d/1udtvggmS2LTMmdwjEtZCcUQy6aQAiYTI3OrTP8CLfJM/edit?usp=sharing)]
-- Increase Beam post-commit tests stability [[doc](https://docs.google.com/document/d/1sczGwnCvdHiboVajGVdnZL0rfnr7ViXXAebBAf_uQME)]
-- Beam-Site Automation Reliability [[doc](https://s.apache.org/beam-site-automation)]
-- Managing outdated dependencies [[doc](https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA)]
-- Automation For Beam Dependency Check [[doc](https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA)]
-- Test performance of core Apache Beam operations [[doc](https://s.apache.org/load-test-basic-operations)]
-- Add static code analysis quality gates to Beam [[doc](https://docs.google.com/document/d/1YbV18mrHujmiLBtadS1WzCVeiI3Lo7W6awWJDA4A98o)]
-
-### Python
-- Beam Python User State and Timer APIs [[doc](https://s.apache.org/beam-python-user-state-and-timers)]
-- Python Kafka connector [[doc](https://docs.google.com/document/d/1ogRS-e-HYYTHsXi_l2zDUUOnvfzEbub3BFkPrYIOawU)]
-- Python 3 support [[doc](https://s.apache.org/beam-python-3)]
-- Splittable DoFn for Python SDK [[doc](http://s.apache.org/splittable-do-fn-python-sdk)]
-- Parquet IO for Python SDK [[doc](https://docs.google.com/document/d/1-FT6zmjYhYFWXL8aDM5mNeiUnZdKnnB021zTo4S-0Wg)]
-- Building Python Wheels [[doc](https://docs.google.com/document/d/1MRVFs48e6g7wORshr2UpuOVD_yTSJTbmR65_j8XbGek)]
-
-### Go
-- Apache Beam Go SDK design [[doc](https://s.apache.org/beam-go-sdk-design-rfc)]
-- Go SDK Vanity Import Path [[doc](https://s.apache.org/go-beam-vanity-import)]
-- Go SDK Integration Tests [[doc](https://docs.google.com/document/d/1jy6EE7D4RjgfNV0FhD3rMsT1YKhnUfcHRZMAlC6ygXw)]
-
-## Other
-- Euphoria - High-Level Java 8 DSL [[doc](https://s.apache.org/beam-euphoria)]
-- Apache Beam Code Review Guide [[doc](https://docs.google.com/document/d/1ZgAsSqEX9CaiTycrcR-tdc3X7MWlyT-F32jfMl89kDQ)]
-
-Some of documents are available on this [google drive](https://drive.google.com/corp/drive/folders/0B-IhJZh9Ab52OFBVZHpsNjc4eXc)
-
-To add new design document it is recommended to use this [design document template](https://docs.google.com/document/d/1kVePqjt2daZd0bQHGUwghlcLbhvrny7VpflAzk9sjUg)
diff --git a/website/src/contribute/index.md b/website/src/contribute/index.md
index 1d99e9e..fa8a18f 100644
--- a/website/src/contribute/index.md
+++ b/website/src/contribute/index.md
@@ -95,9 +95,9 @@
the [dev@ mailing list]({{ site.baseurl }}/community/contact-us)
to introduce yourself and to be added as a contributor in the Beam issue tracker including your
ASF Jira Username. For example [this welcome email](
- https://lists.apache.org/thread.html/e6018c2aaf7dc7895091434295e5b0fafe192b975e3e3761fcf0cda7@%3Cdev.beam.apache.org%3E)
+ https://lists.apache.org/thread.html/e6018c2aaf7dc7895091434295e5b0fafe192b975e3e3761fcf0cda7@%3Cdev.beam.apache.org%3E).
1. If your change is large or it is your first change, it is a good idea to
- [discuss it on the dev@ mailing list]({{ site.baseurl }}/community/contact-us/)
+ [discuss it on the dev@ mailing list]({{ site.baseurl }}/community/contact-us/).
1. For large changes create a design doc
([template](https://s.apache.org/beam-design-doc-template),
[examples](https://s.apache.org/beam-design-docs)) and email it to the [dev@ mailing list]({{ site.baseurl }}/community/contact-us).
diff --git a/website/src/contribute/ptransform-style-guide.md b/website/src/contribute/ptransform-style-guide.md
index dfdf07e..4f4d253 100644
--- a/website/src/contribute/ptransform-style-guide.md
+++ b/website/src/contribute/ptransform-style-guide.md
@@ -408,7 +408,7 @@
The container class must have a private constructor, so it can't be instantiated directly.
-Document common stuff at `BlahIO` level, and each factory method individually.
+Document common stuff at `FooIO` level, and each factory method individually.
```java
/** Transforms for clustering data. */
@@ -425,7 +425,7 @@
public static class Hierarchically extends PTransform<...> { ... }
}
-public lass FooIO {
+public class FooIO {
// Force use of static factory methods.
private FooIO() {}
diff --git a/website/src/contribute/release-guide.md b/website/src/contribute/release-guide.md
index 49c2246..03fb6cc 100644
--- a/website/src/contribute/release-guide.md
+++ b/website/src/contribute/release-guide.md
@@ -33,7 +33,7 @@
Please remember that publishing software has legal consequences. This guide complements the foundation-wide [Product Release Policy](http://www.apache.org/dev/release.html) and [Release Distribution Policy](http://www.apache.org/dev/release-distribution).
-## Overview
+### Overview
![Alt text]({{ "/images/release-guide-1.png" | prepend: site.baseurl }} "Release Process"){:width="100%"}
@@ -48,9 +48,8 @@
1. Finalize the release
1. Promote the release
-**********
-## Decide to release
+### Decide to release
Deciding to release and selecting a Release Manager is the first step of the release process. This is a consensus-based decision of the entire community.
@@ -65,7 +64,7 @@
**********
-## Prepare for the release
+## 1. Prepare for the release
Before your first release, you should perform one-time configuration steps. This will set up your security keys for signing the release and access to various release repositories.
@@ -74,8 +73,20 @@
__NOTE__: If you are using [GitHub two-factor authentication](https://help.github.com/articles/securing-your-account-with-two-factor-authentication-2fa/) and haven't configure HTTPS access,
please follow [the guide](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) to configure command line access.
+
+### Accounts
+
+Please have these credentials ready at hand, you will likely need to enter them multiple times:
+
+* GPG pass phrase (see the next section);
+* Apache ID and Password;
+* GitHub ID and Password.
+* DockerHub ID and Password. (You should be a member of maintainer team; email at dev@ if you are not.)
+
+
### One-time setup instructions
+
#### GPG Key
You need to have a GPG key to sign the release artifacts. Please be aware of the ASF-wide [release signing guidelines](https://www.apache.org/dev/release-signing.html). If you don’t have a GPG key associated with your Apache account, please create one according to the guidelines.
@@ -98,6 +109,9 @@
**NOTES**: Only PMC can write into [release repo](https://dist.apache.org/repos/dist/release/beam/).
1. Start GPG agents.
+__NOTE__: When generating the key, please make sure you choose the key type as __RSA and RSA (default)__ and key size as __4096 bit__.
+
+
##### Run all commands manually
* Get more entropy for creating a GPG key
@@ -146,6 +160,7 @@
1. Choose `User Token` from the dropdown, then click `Access User Token`. Copy a snippet of the Maven XML configuration block.
1. Insert this snippet twice into your global Maven `settings.xml` file, typically `${HOME}/.m2/settings.xml`. The end result should look like this, where `TOKEN_NAME` and `TOKEN_PASSWORD` are your secret tokens:
+ <!-- make sure you have the root `settings node: -->
<settings>
<servers>
<server>
@@ -160,11 +175,15 @@
</server>
</servers>
</settings>
+__NOTE__: make sure the XML you end up with matches the structure above.
#### Submit your GPG public key into MIT PGP Public Key Server
In order to make yourself have right permission to stage java artifacts in Apache Nexus staging repository,
please submit your GPG public key into [MIT PGP Public Key Server](http://pgp.mit.edu:11371/).
+If MIT doesn't work for you (it probably won't, it's slow, returns 502 a lot, Nexus might error out not being able to find the keys),
+use a keyserver at `ubuntu.com` instead: http://keyserver.ubuntu.com/.
+
#### Website development setup
Updating the Beam website requires submitting PRs to both the main `apache/beam`
@@ -190,6 +209,32 @@
Release manager needs to have an account with PyPI. If you need one, [register at PyPI](https://pypi.python.org/account/register/). You also need to be a maintainer (or an owner) of the [apache-beam](https://pypi.python.org/pypi/apache-beam) package in order to push a new release. Ask on the mailing list for assistance.
+#### Login to DockerHub
+Run following command manually. It will ask you to input your DockerHub ID and password if
+authorization info cannot be found from ~/.docker/config.json file.
+```
+docker login docker.io
+```
+After successful login, authorization info will be stored at ~/.docker/config.json file. For example,
+```
+"https://index.docker.io/v1/": {
+ "auth": "aGFubmFoamlhbmc6cmtkdGpmZ2hrMTIxMw=="
+}
+```
+Release managers should have push permission; please ask for help at dev@.
+```
+From: Release Manager
+To: dev@beam.apache.org
+Subject: DockerHub Push Permission
+
+Hi DockerHub Admins
+
+I need push permission to proceed with release, can you please add me to maintainer team?
+My docker hub ID is: xxx
+
+Thanks,
+Release Manager
+```
### Create a new version in JIRA
When contributors resolve an issue in JIRA, they are tagging it with a release that will contain their changes. With the release currently underway, new issues should be resolved against a subsequent future release. Therefore, you should create a release item for this subsequent release, as follows:
@@ -200,7 +245,11 @@
1. Add a new release. Choose the next minor version number after the version currently underway, select the release cut date (today’s date) as the `Start Date`, and choose `Add`.
1. At the end of the release, go to the same page and mark the recently released version as released. Use the `...` menu and choose `Release`.
-### Create a release branch in apache/beam repository
+
+**********
+
+
+## 2. Create a release branch in apache/beam repository
Attention: Only committer has permission to create release branch in apache/beam.
@@ -214,14 +263,14 @@
* Usage
```
# Cut a release branch
- ./beam/release/src/main/scripts/cut_release_branch.sh
- --release= ${RELEASE_VERSION}
+ ./beam/release/src/main/scripts/cut_release_branch.sh \
+ --release=${RELEASE_VERSION} \
--next_release=${NEXT_VERSION}
# Show help page
./beam/release/src/main/scripts/cut_release_branch.sh -h
```
-* Tasks included
+* The script will:
1. Create release-${RELEASE_VERSION} branch locally.
1. Change and commit dev versoin number in master branch:
@@ -231,9 +280,10 @@
1. Change and commit version number in release branch:
[version.py](https://github.com/apache/beam/blob/release-2.6.0/sdks/python/apache_beam/version.py#L21),
- [build.gradle](https://github.com/apache/beam/blob/release-2.6.0/runners/google-cloud-dataflow-java/build.gradle#L39)
+ [build.gradle](https://github.com/apache/beam/blob/release-2.6.0/runners/google-cloud-dataflow-java/build.gradle#L39),
+ [gradle.properties](https://github.com/apache/beam/blob/release-2.16.0/gradle.properties#L27)
-#### Run all steps manually
+#### (Alternative) Run all steps manually
* Checkout working branch
Check out the version of the codebase from which you start the release. For a new minor or major release, this may be `HEAD` of the `master` branch. To build a hotfix/incremental release, instead of the `master` branch, use the release tag of the release being patched. (Please make sure your cloned repository is up-to-date before starting.)
@@ -283,6 +333,7 @@
DEV=${RELEASE}.dev
sed -i -e "s/${DEV}/${RELEASE}/g" sdks/python/apache_beam/version.py
+ sed -i -e "s/${DEV}/${RELEASE}/g" gradle.properties
sed -i -e "s/'beam-master-.*'/'beam-${RELEASE}'/g" runners/google-cloud-dataflow-java/build.gradle
@@ -302,46 +353,67 @@
./beam/release/src/main/scripts/start_snapshot_build.sh
-* Tasks included
+* The script will:
1. Install [hub](https://github.com/github/hub) with your agreement.
1. Touch an empty txt file and commit changes into ```${your remote beam repo}/snapshot_build```
1. Use hub to create a PR against apache:master, which triggers a Jenkins job to build snapshot.
-* Tasks you need to do manually
+* Tasks you need to do manually to __verify the SNAPSHOT build__
1. Check whether the Jenkins job gets triggered. If not, please comment ```Run Gradle Publish``` into the generated PR.
1. After verifying build succeeded, you need to close PR manually.
-#### Do all operations manually
+#### (Alternative) Do all operations manually
* Find one PR against apache:master in beam.
* Comment ```Run Gradle Publish``` in this pull request to trigger build.
* Verify that build succeeds.
-### Verify release branch
+**********
+
+## 3. Verify release branch
+
+After the release branch is cut you need to make sure it builds and has no significant issues that would block the creation of the release candidate.
There are 2 ways to perform this verification, either running automation script(recommended), or running all commands manually.
#### Run automation script (verify_release_build.sh)
* Script: [verify_release_build.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh)
* Usage
+ 1. Create a personal access token from your Github account. See instruction [here](https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line).
+ It'll be used by the script for accessing Github API.
+ 1. Update required configurations listed in `RELEASE_BUILD_CONFIGS` in [script.config](https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config)
+ 1. Then run
+ ```
+ cd beam/release/src/main/scripts && ./verify_release_build.sh
+ ```
+ 1. Trigger `beam_Release_Gradle_Build` and all PostCommit Jenkins jobs from PR (which is created by previous step).
+ To do so, only add one trigger phrase per comment. See `JOB_TRIGGER_PHRASES` in [verify_release_build.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L43)
+ for full list of phrases.
- ```
- ./beam/release/src/main/scripts/verify_release_build.sh
- ```
+* Tasks included in the script
+ 1. Installs ```hub``` with your agreement and setup local git repo;
+ 1. Create a test PR against release branch;
-* Tasks included
- 1. Install ```pip```, ```virtualenv```, ```cython``` and ```/usr/bin/time``` with your agreements.
- 2. Run ```gradle release build``` against release branch.
+Jenkins job `beam_Release_Gradle_Build` basically run `./gradlew build -PisRelease`.
+This only verifies that everything builds with unit tests passing.
-* Tasks you need to do manually
+#### Verify the build succeeds
+
+* Tasks you need to do manually to __verify the build succeed__:
1. Check the build result.
2. If build failed, scan log will contain all failures.
3. You should stabilize the release branch until release build succeeded.
- 4. The script will output a set of Jenkins phrases to enter in the created PR
+ 4. The script will output a set of Jenkins phrases to enter in the created PR.
+
+There are some projects that don't produce the artifacts, e.g. `beam-test-tools`, you may be able to
+ignore failures there.
-#### Run all commands manually
+To triage the failures and narrow things down you may want to look at `settings.gradle` and run the build only for the
+projects you're interested at the moment, e.g. `./gradlew :runners:java-fn-execution`.
+
+#### (Alternative) Run release build manually (locally)
* Pre-installation for python build
1. Install pip
@@ -365,12 +437,6 @@
sudo apt-get install python3.6-dev
sudo apt-get install python3.7-dev
```
- 1. Make sure your ```time``` alias to ```/usr/bin/time```, if not:
-
- ```
- sudo apt-get install time
- alias time='/usr/bin/time'
- ```
* Run gradle release build
@@ -391,6 +457,10 @@
./gradlew build -PisRelease --no-parallel --scan --stacktrace --continue
```
+ To speed things up locally you might want to omit `--no-parallel`. You can also omit `--continue`
+ if you want build fails after the first error instead of continuing, it may be easier and faster
+ to find environment issues this way without having to wait until the full build completes.
+
#### Create release-blocking issues in JIRA
@@ -409,29 +479,17 @@
* Description: Description of failure
-### Update and Verify Javadoc
+**********
-The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`.
-The file `sdks/java/javadoc/build.gradle` contains a list of modules to include
-in and exclude, plus a list of offline URLs that populate links from Beam's
-Javadoc to the Javadoc for other modules that Beam depends on.
-
-* Confirm that new modules added since the last release have been added to the
- inclusion list as appropriate.
-
-* Confirm that the excluded package list is up to date.
-
-* Verify the version numbers for offline links match the versions used by Beam. If
- the version number has changed, download a new version of the corresponding
- `<module>-docs/package-list` file.
-
-### Triage release-blocking issues in JIRA
+## 4. Triage release-blocking issues in JIRA
There could be outstanding release-blocking issues, which should be triaged before proceeding to build a release candidate. We track them by assigning a specific `Fix version` field even before the issue resolved.
The list of release-blocking issues is available at the [version status page](https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel). Triage each unresolved issue with one of the following resolutions:
+The release manager should triage what does and does not block a release. An issue should not block the release if the problem exists in the current released version or is a bug in new functionality that does not exist in the current released version. It should be a blocker if the bug is a regression between the currently released version and the release in progress and has no easy workaround.
+
For all JIRA issues:
* If the issue has been resolved and JIRA was not updated, resolve it accordingly.
@@ -465,58 +523,69 @@
Adjust any of the above properties to the improve clarity and presentation of the Release Notes.
-### Checklist to proceed to the next step
-* Release Manager’s GPG key is published to `dist.apache.org`
-* Release Manager’s GPG key is configured in `git` configuration
-* Release Manager has `org.apache.beam` listed under `Staging Profiles` in Nexus
-* Release Manager’s Nexus User Token is configured in `settings.xml`
-* JIRA release item for the subsequent release has been created
-* All test failures from branch verification have associated JIRA issues
-* There are no release blocking JIRA issues
-* Release Notes in JIRA have been audited and adjusted
-* Combined javadoc has the appropriate contents.
-* Release branch has been created
-* There are no open pull requests to release branch
-* Originating branch has the version information updated to the new version
-* Nightly snapshot is in progress (do revisit it continually)
+### Review cherry-picks
+
+Check if there are outstanding cherry-picks into the release branch, [e.g. for `2.14.0`](https://github.com/apache/beam/pulls?utf8=%E2%9C%93&q=is%3Apr+base%3Arelease-2.14.0).
+Make sure they have blocker JIRAs attached and are OK to get into the release by checking with community if needed.
+
**********
-## Build a release candidate
+
+## 5. Build a release candidate
+
+### Checklist before proceeding
+
+* Release Manager’s GPG key is published to `dist.apache.org`;
+* Release Manager’s GPG key is configured in `git` configuration;
+* Release Manager has `org.apache.beam` listed under `Staging Profiles` in Nexus;
+* Release Manager’s Nexus User Token is configured in `settings.xml`;
+* JIRA release item for the subsequent release has been created;
+* All test failures from branch verification have associated JIRA issues;
+* There are no release blocking JIRA issues;
+* Release Notes in JIRA have been audited and adjusted;
+* Combined javadoc has the appropriate contents;
+* Release branch has been created;
+* There are no open pull requests to release branch;
+* Originating branch has the version information updated to the new version;
+* Nightly snapshot is in progress (do revisit it continually);
The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized.
For this step, we recommend you using automation script to create a RC, but you still can perform all steps manually if you want.
-### Run build_release_candidate.sh to create RC
+
+### Run build_release_candidate.sh to create a release candidate
+
* Script: [build_release_candidate.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh)
* Usage
./beam/release/src/main/scripts/build_release_candidate.sh
-* Tasks included
+* The script will:
1. Run gradle release to create rc tag and push source release into github repo.
1. Run gradle publish to push java artifacts into Maven staging repo.
- __NOTE__: In order to public staging artifacts, you need to goto the staging repo to close the staging repository on Apache Nexus.
+ __NOTE__: In order to public staging artifacts, you need to goto the [staging repo](https://repository.apache.org/#stagingRepositories) to close the staging repository on Apache Nexus.
When prompted for a description, enter “Apache Beam, version X, release candidate Y”.
1. Stage source release into dist.apache.org dev [repo](https://dist.apache.org/repos/dist/dev/beam/).
1. Stage,sign and hash python binaries into dist.apache.ord dev repo python dir
+ 1. Stage SDK docker images to [https://hub.docker.com/u/apachebeam](https://hub.docker.com/u/apachebeam).
1. Create a PR to update beam and beam-site, changes includes:
* Copy python doc into beam-site
* Copy java doc into beam-site
* Update release version into [_config.yml](https://github.com/apache/beam/blob/master/website/_config.yml).
-* Tasks you need to do manually
+#### Tasks you need to do manually
1. Add new release into `website/src/get-started/downloads.md`.
1. Update last release download links in `website/src/get-started/downloads.md`.
1. Update `website/src/.htaccess` to redirect to the new version.
1. Build and stage python wheels.
-### Run all steps manually
+### (Alternative) Run all steps manually
#### Build and stage Java artifacts with Gradle
@@ -598,7 +667,81 @@
Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).
-#### Build the Pydoc API reference
+#### Stage SDK images on hub.docker.com
+* Build Python images and push to DockerHub.
+
+```
+./gradlew :sdks:python:container:buildAll -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+
+PYTHON_VER=("python2.7" "python3.5" "python3.6" "python3.7")
+for ver in "${PYTHON_VER[@]}"; do
+ docker push apachebeam/${ver}_sdk:${RELEASE}_rc{RC_NUM} &
+done
+```
+
+* Build Java images and push to DockerHub.
+
+```
+./gradlew :sdks:java:container:dockerPush -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+```
+
+* Build Go images and push to DockerHub.
+
+```
+./gradlew :sdks:go:container:dockerPush -Pdocker-tag=${RELEASE}_rc{RC_NUM}
+```
+
+Clean up images from local
+
+```
+for ver in "${PYTHON_VER[@]}"; do
+ docker rmi -f apachebeam/${ver}_sdk:${RELEASE}_rc{RC_NUM}
+done
+docker rmi -f apachebeam/java_sdk:${RELEASE}_rc{RC_NUM}
+docker rmi -f apachebeam/go_sdk:${RELEASE}_rc{RC_NUM}
+```
+
+How to find images:
+1. Visit [https://hub.docker.com/u/apachebeam](https://hub.docker.com/u/apachebeam)
+2. Visit each repository and navigate to *tags* tab.
+3. Verify images are pushed with tags: ${RELEASE}_rc{RC_NUM}
+
+### Build and stage python wheels
+
+There is a wrapper repo [beam-wheels](https://github.com/apache/beam-wheels) to help build python wheels.
+
+If you are interested in how it works, please refer to the [structure section](https://github.com/apache/beam-wheels#structure).
+
+Please follow the [user guide](https://github.com/apache/beam-wheels#user-guide) to build python wheels.
+
+Once all python wheels have been staged [dist.apache.org](https://dist.apache.org/repos/dist/dev/beam/),
+please run [./sign_hash_python_wheels.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/sign_hash_python_wheels.sh) to sign and hash python wheels.
+
+
+**********
+
+
+## 6. Prepare documents
+
+### Update and Verify Javadoc
+
+The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`.
+
+The file `sdks/java/javadoc/build.gradle` contains a list of modules to include
+in and exclude, plus a list of offline URLs that populate links from Beam's
+Javadoc to the Javadoc for other modules that Beam depends on.
+
+* Confirm that new modules added since the last release have been added to the
+ inclusion list as appropriate.
+
+* Confirm that the excluded package list is up to date.
+
+* Verify the version numbers for offline links match the versions used by Beam. If
+ the version number has changed, download a new version of the corresponding
+ `<module>-docs/package-list` file.
+
+
+### Build the Pydoc API reference
Make sure you have ```tox``` installed:
@@ -611,7 +754,7 @@
```
By default the Pydoc is generated in `sdks/python/target/docs/_build`. Let `${PYDOC_ROOT}` be the absolute path to `_build`.
-#### Propose pull requests for website updates
+### Propose pull requests for website updates
Beam publishes API reference manuals for each release on the website. For Java
and Python SDKs, that’s Javadoc and PyDoc, respectively. The final step of
@@ -647,28 +790,74 @@
[/website/src/.htaccess](https://github.com/apache/beam/blob/master/website/src/.htaccess)
to point to the new release. See file history for examples.
-#### Build and stage python wheels
-There is a wrapper repo [beam-wheels](https://github.com/apache/beam-wheels) to help build python wheels.
+### Blog post
-If you are interested in how it works, please refer to the [structure section](https://github.com/apache/beam-wheels#structure).
+Write a blog post similar to https://beam.apache.org/blog/2019/08/22/beam-2.15.0.html
-Please follow the [user guide](https://github.com/apache/beam-wheels#user-guide) to build python wheels.
+__Tip__: Use git log to find contributors to the releases. (e.g: `git log --pretty='%aN' ^v2.10.0 v2.11.0 | sort | uniq`).
+Make sure to clean it up, as there may be duplicate or incorrect user names.
-Once all python wheels have been staged [dist.apache.org](https://dist.apache.org/repos/dist/dev/beam/),
-please run [./sign_hash_python_wheels.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/sign_hash_python_wheels.sh) to sign and hash python wheels.
+__NOTE__: Make sure to include any breaking changes, even to `@Experimental` features,
+all major features and bug fixes, and all known issues.
-### Write the Beam blog post and create a pull request
+Template:
-Major or otherwise important releases should have a blog post. Write one if needed for this particular release. Minor releases that don’t introduce new major functionality don’t necessarily need to be blogged.
+```
+ We are happy to present the new {$RELEASE_VERSION} release of Beam. This release includes both improvements and new functionality.
+ See the [download page]({{ site.baseurl }}/get-started/downloads/{$DOWNLOAD_ANCHOR}) for this release.<!--more-->
+ For more information on changes in {$RELEASE_VERSION}, check out the
+ [detailed release notes]({$JIRA_RELEASE_NOTES}).
-*Tip:* Use git log to find contributors to the releases. (e.g: `git log --pretty='%aN' ^v2.10.0 v2.11.0 | sort | uniq`).
+ ## Highlights
+
+ * New highly anticipated feature X added to Python SDK ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+ * New highly anticipated feature Y added to JavaSDK ([BEAM-Y](https://issues.apache.org/jira/browse/BEAM-Y)).
+
+ {$TOPICS e.g.:}
+ ### I/Os
+ * Support for X source added (Java) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+ {$TOPICS}
+
+ ### New Features / Improvements
+
+ * X feature added (Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+ * Y feature added (Java) [BEAM-Y](https://issues.apache.org/jira/browse/BEAM-Y).
+
+ ### Breaking Changes
+
+ * X behavior was changed ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+ * Y behavior was changed ([BEAM-Y](https://issues.apache.org/jira/browse/BEAM-X)).
+
+ ### Deprecations
+
+ * X behavior is deprecated and will be removed in X versions ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+
+ ### Bugfixes
+
+ * Fixed X (Python) ([BEAM-Y](https://issues.apache.org/jira/browse/BEAM-X)).
+ * Fixed Y (Java) ([BEAM-Y](https://issues.apache.org/jira/browse/BEAM-Y)).
+
+ ### Known Issues
+
+ * {$KNOWN_ISSUE_1}
+ * {$KNOWN_ISSUE_2}
+
+
+ ## List of Contributors
+
+ According to git shortlog, the following people contributed to the 2.XX.0 release. Thank you to all contributors!
+
+ ${CONTRIBUTORS}
+ ```
+
#### Checklist to proceed to the next step
1. Maven artifacts deployed to the staging repository of [repository.apache.org](https://repository.apache.org/content/repositories/)
1. Source distribution deployed to the dev repository of [dist.apache.org](https://dist.apache.org/repos/dist/dev/beam/)
1. Website pull request proposed to list the [release]({{ site.baseurl }}/get-started/downloads/), publish the [Java API reference manual](https://beam.apache.org/releases/javadoc/), and publish the [Python API reference manual](https://beam.apache.org/releases/pydoc/).
+1. Docker images are published to [DockerHub](https://hub.docker.com/u/apachebeam) with tags: {RELEASE}_rc{RC_NUM}.
You can (optionally) also do additional verification by:
1. Check that Python zip file contains the `README.md`, `NOTICE`, and `LICENSE` files.
@@ -676,10 +865,17 @@
1. Check signatures (e.g. `gpg --verify apache-beam-1.2.3-python.zip.asc apache-beam-1.2.3-python.zip`)
1. `grep` for legal headers in each file.
1. Run all jenkins suites and include links to passing tests in the voting email. (Select "Run with parameters")
+1. Pull docker images to make sure they are pullable.
+```
+docker pull {image_name}
+docker pull apachebeam/python3.5_sdk:2.16.0_rc1
+```
+
**********
-## Vote on the release candidate
+
+## 7. Vote and validate release candidate
Once you have built and individually reviewed the release candidate, please share it for the community-wide review. Please review foundation-wide [voting guidelines](http://www.apache.org/foundation/voting.html) for more information.
@@ -704,6 +900,7 @@
* Java artifacts were built with Maven MAVEN_VERSION and OpenJDK/Oracle JDK JDK_VERSION.
* Python artifacts are deployed along with the source release to the dist.apache.org [2].
* Validation sheet with a tab for 1.2.3 release to help with validation [9].
+ * Docker images puhlished to Docker Hub [10].
The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.
@@ -719,7 +916,8 @@
[7] https://github.com/apache/beam-site/pull/...
[8] https://github.com/apache/beam/pull/...
[9] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=...
-
+ [10] https://hub.docker.com/u/apachebeam
+
If there are any issues found in the release candidate, reply on the vote thread to cancel the vote. There’s no need to wait 72 hours. Proceed to the `Fix Issues` step below and address the problem. However, some issues don’t require cancellation. For example, if an issue is found in the website pull request, just correct it on the spot and the vote can continue as-is.
If there are no issues, reply on the vote thread to close the voting. Then, tally the votes in a separate email. Here’s an email template; please adjust as you see fit.
@@ -749,13 +947,17 @@
* Script: [run_rc_validation.sh](https://github.com/apache/beam/blob/master/release/src/main/scripts/run_rc_validation.sh)
* Usage
-
- ./beam/release/src/main/scripts/run_rc_validation.sh
+ 1. First update required configurations listed in `RC_VALIDATE_CONFIGS` in
+ [script.config](https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config)
+ 1. Then run
+ ```
+ cd beam/release/src/main/scripts && ./run_rc_validation.sh
+ ```
* Tasks included
1. Run Java quickstart with Direct Runner, Apex local runner, Flink local runner, Spark local runner and Dataflow runner.
1. Run Java Mobile Games(UserScore, HourlyTeamScore, Leaderboard) with Dataflow runner.
- 1. Create a PR against apache:master to trigger python validation job, including
+ 1. Create a PR to trigger python validation job, including
* Python quickstart in batch and streaming mode with direct runner and Dataflow runner.
* Python Mobile Games(UserScore, HourlyTeamScore) with direct runner and Dataflow runner.
1. Run Python Streaming MobileGames, includes
@@ -791,7 +993,7 @@
```
Flink Local Runner
```
- ./gradlew :runners:flink:1.5:runQuickstartJavaFlinkLocal \
+ ./gradlew :runners:flink:1.8:runQuickstartJavaFlinkLocal \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY} \
-Pver=${RELEASE_VERSION}
```
@@ -805,7 +1007,7 @@
```
./gradlew :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY} \
- -Pver= ${RELEASE_VERSION}\
+ -Pver=${RELEASE_VERSION} \
-PgcpProject=${YOUR_GCP_PROJECT} \
-PgcsBucket=${YOUR_GCP_BUCKET}
```
@@ -833,7 +1035,7 @@
```
./gradlew :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY} \
- -Pver= ${RELEASE_VERSION}\
+ -Pver=${RELEASE_VERSION} \
-PgcpProject=${YOUR_GCP_PROJECT} \
-PgcsBucket=${YOUR_GCP_BUCKET} \
-PbqDataset=${YOUR_DATASET} -PpubsubTopic=${YOUR_PROJECT_PUBSUB_TOPIC}
@@ -983,14 +1185,9 @@
* Goto your BigQuery console and check whether your ${USER}_test has game_stats_teams and game_stats_sessions table.
* bq head -n 10 ${USER}_test.game_stats_teams
* bq head -n 10 ${USER}_test.game_stats_sessions
-
-### Checklist to proceed to the finalization step
-1. Community votes to release the proposed candidate, with at least three approving PMC votes
-**********
-
-## Fix any issues
+### Fix any issues
Any issues identified during the community review and vote should be fixed in this step. Additionally, any JIRA issues created from the initial branch verification should be fixed.
@@ -1002,23 +1199,28 @@
1. Issues identified during vote have been resolved, with fixes committed to the release branch.
2. All issues tagged with `Fix-Version` for the current release should be closed.
+3. Community votes to release the proposed candidate, with at least three approving PMC votes
+
**********
-## Finalize the release
+
+## 8. Finalize the release
Once the release candidate has been reviewed and approved by the community, the release should be finalized. This involves the final deployment of the release candidate to the release repositories, merging of the website changes, etc.
### Deploy artifacts to Maven Central Repository
Use the Apache Nexus repository to release the staged binary artifacts to the Maven Central repository. In the `Staging Repositories` section, find the relevant release candidate `orgapachebeam-XXX` entry and click `Release`. Drop all other release candidates that are not being released.
+__NOTE__: If you are using [GitHub two-factor authentication](https://help.github.com/articles/securing-your-account-with-two-factor-authentication-2fa/) and haven't configure HTTPS access,
+please follow [the guide](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) to configure command line access.
### Deploy Python artifacts to PyPI
-1. Create a new release and upload the Python zip file for the new release using the [PyPI UI] (https://pypi.python.org/pypi/apache-beam)
-1. Alternatively, use the command line tool to upload the new release `twine upload apache-beam-${RELEASE}.zip`
-
-Note: It is important to rename `apache-beam-${RELEASE}-python.zip` to `apache-beam-${RELEASE}.zip` before uploading, because PyPI expects a filename in the `<package-name>-<package-version>` format.
+1. Download everything from https://dist.apache.org/repos/dist/dev/beam/2.14.0/python/ ;
+2. Keep only things that you see in https://pypi.org/project/apache-beam/#files , e.g. `.zip`, `.whl`,
+ delete the `.asc`, `.sha512`;
+3. Upload the new release `twine upload *` from the directory with the `.zip` and `.whl` files;
#### Deploy source release to dist.apache.org
@@ -1026,6 +1228,23 @@
Move last release artifacts from `dist.apache.org` to `archive.apache.org` using Subversion. Then update download address for last release version, [example PR](https://github.com/apache/beam-site/pull/478).
+__NOTE__: Only PMC members have permissions to do it, ping [dev@](mailto:dev@beam.apache.org) for assitance;
+
+Make sure the download address for last release version is upldaed, [example PR](https://github.com/apache/beam-site/pull/478).
+
+### Deploy SDK docker images to DockerHub
+TODO(hannahjiang): change link to master branch after #9560 is merged.
+
+* Script: [publish_docker_images.sh](https://github.com/Hannah-Jiang/beam/blob/release_script_for_containers/release/src/main/scripts/publish_docker_images.sh)
+* Usage
+```
+./beam/release/src/main/scripts/publish_docker_images.sh
+```
+Verify that:
+* Images are published at [DockerHub](https://hub.docker.com/u/apachebeam) with tags {RELEASE} and *latest*.
+* Images with *latest* tag are pointing to current release by confirming
+ 1. Digest of the image with *latest* tag is the same as the one with {RELEASE} tag.
+
### Git tag
Create and push a new signed tag for the released version by copying the tag for the final release candidate, as follows:
@@ -1042,10 +1261,14 @@
In JIRA, inside [version management](https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions), hover over the current release and a settings menu will appear. Click `Release`, and select today’s date.
+__NOTE__: Only PMC members have permissions to do it, ping [dev@](mailto:dev@beam.apache.org) for assitance;
+
### Recordkeeping with ASF
Use reporter.apache.org to seed the information about the release into future project reports.
+__NOTE__: Only PMC members have permissions to do it, ping [dev@](mailto:dev@beam.apache.org) for assitance;
+
### Checklist to proceed to the next step
* Maven artifacts released and indexed in the [Maven Central Repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22)
@@ -1056,9 +1279,11 @@
* Release version finalized in JIRA. (Note: Not all committers have administrator access to JIRA. If you end up getting permissions errors ask on the mailing list for assistance.)
* Release version is listed at reporter.apache.org
+
**********
-## Promote the release
+
+## 9. Promote the release
Once the release has been finalized, the last step of the process is to promote the release within the project and beyond.
@@ -1069,6 +1294,8 @@
Announce on the release on the user@ mailing list, listing major improvements and contributions.
Announce the release on the announce@apache.org mailing list.
+__NOTE__: This can only be done from `@apache.org` email address.
+
### Social media
diff --git a/website/src/contribute/runner-guide.md b/website/src/contribute/runner-guide.md
index bd70899..c0f6d57 100644
--- a/website/src/contribute/runner-guide.md
+++ b/website/src/contribute/runner-guide.md
@@ -214,7 +214,7 @@
The `run(Pipeline)` method should be asynchronous and results in a
PipelineResult which generally will be a job descriptor for your data
-processing engine, provides methods for checking its status, canceling it, and
+processing engine, providing methods for checking its status, canceling it, and
waiting for it to terminate.
## Implementing the Beam Primitives
@@ -228,7 +228,7 @@
The primitives are designed for the benefit of pipeline authors, not runner
authors. Each represents a different conceptual mode of operation (external IO,
element-wise, grouping, windowing, union) rather than a specific implementation
-decision. The same primitive may require very different implementation based
+decision. The same primitive may require a very different implementation based
on how the user instantiates it. For example, a `ParDo` that uses state or
timers may require key partitioning, a `GroupByKey` with speculative triggering
may require a more costly or complex implementation, and `Read` is completely
@@ -297,7 +297,7 @@
remains for simplicity for users)
* _ProcessElement_ / _OnTimer_ - called for each element and timer activation
* _FinishBundle_ - essentially "flush"; required to be called before
- considering elements actually processed
+ considering elements as actually processed
* _Teardown_ - release resources that were used across bundles; calling this
can be best effort due to failures
@@ -350,7 +350,7 @@
A side input is a global view of a window of a `PCollection`. This distinguishes
it from the main input, which is processed one element at a time. The SDK/user
prepares a `PCollection` adequately, the runner materializes it, and then the
-runner feeds it to the `DoFn`. See the
+runner feeds it to the `DoFn`.
What you will need to implement is to inspect the materialization requested for
the side input, and prepare it appropriately, and corresponding interactions
@@ -396,7 +396,7 @@
_Main design document: [https://s.apache.org/beam-state](https://s.apache.org/beam-state)_
-When `ParDo` includes state and timers, its execution on your runner is usually
+When a `ParDo` includes state and timers, its execution on your runner is usually
very different. See the full details beyond those covered here.
State and timers are partitioned per key and window. You may need or want to
@@ -416,7 +416,7 @@
_Main design document: [https://s.apache.org/splittable-do-fn](https://s.apache.org/splittable-do-fn)_
Splittable `DoFn` is a generalization and combination of `ParDo` and `Read`. It
-is per-element processing where each element the capabilities of being "split"
+is per-element processing where each element has the capability of being "split"
in the same ways as a `BoundedSource` or `UnboundedSource`. This enables better
performance for use cases such as a `PCollection` of names of large files where
you want to read each of them. Previously they would have to be static data in
@@ -459,7 +459,7 @@
#### Implementing via GroupByKeyOnly + GroupAlsoByWindow
The Java codebase includes support code for a particularly common way of
-implement the full `GroupByKey` operation: first group the keys, and then group
+implementing the full `GroupByKey` operation: first group the keys, and then group
by window. For merging windows, this is essentially required, since merging is
per key.
@@ -506,7 +506,7 @@
The window primitive applies a `WindowFn` UDF to place each input element into
one or more windows of its output PCollection. Note that the primitive also
generally configures other aspects of the windowing strategy for a `PCollection`,
-but the fully constructed graph that your runner receive will already have a
+but the fully constructed graph that your runner receives will already have a
complete windowing strategy for each `PCollection`.
To implement this primitive, you need to invoke the provided WindowFn on each
@@ -543,14 +543,14 @@
it like a stream. The capabilities are:
* `split(int)` - your runner should call this to get the desired parallelism
- * `createReader(...)` - call this to start reading elements; it is an enhanced iterator that also vends:
+ * `createReader(...)` - call this to start reading elements; it is an enhanced iterator that also provides:
* watermark (for this source) which you should propagate downstream
- timestamps, which you should associate with elements read
+ * timestamps, which you should associate with elements read
* record identifiers, so you can dedup downstream if needed
* progress indication of its backlog
* checkpointing
* `requiresDeduping` - this indicates that there is some chance that the source
- may emit dupes; your runner should do its best to dedupe based on the
+ may emit duplicates; your runner should do its best to dedupe based on the
identifier attached to emitted records
An unbounded source has a custom type of checkpoints and an associated coder for serializing them.
@@ -562,7 +562,7 @@
* `split(int)` - your runner should call this to get desired initial parallelism (but you can often steal work later)
* `getEstimatedSizeBytes(...)` - self explanatory
- * `createReader(...)` - call this to start reading elements; it is an enhanced iterator, with also:
+ * `createReader(...)` - call this to start reading elements; it is an enhanced iterator that also provides:
* timestamps to associate with each element read
* `splitAtFraction` for dynamic splitting to enable work stealing, and other
methods to support it - see the [Beam blog post on dynamic work
@@ -669,7 +669,7 @@
}
```
-Enable these tests in other languages is unexplored.
+Enabling these tests in other languages is unexplored.
## Integrating your runner nicely with SDKs
@@ -863,12 +863,12 @@
A `FunctionSpec` includes a URN identifying the function as well as an arbitrary
fixed parameter. For example the (hypothetical) "max" CombineFn might have the
-URN `urn:beam:combinefn:max:0.1` and a parameter that indicates by what
+URN `beam:combinefn:max:0.1` and a parameter that indicates by what
comparison to take the max.
For most UDFs in a pipeline constructed using a particular language's SDK, the
URN will indicate that the SDK must interpret it, for example
-`urn:beam:dofn:javasdk:0.1` or `urn:beam:dofn:pythonsdk:0.1`. The parameter
+`beam:dofn:javasdk:0.1` or `beam:dofn:pythonsdk:0.1`. The parameter
will contain serialized code, such as a Java-serialized `DoFn` or a Python
pickled `DoFn`.
diff --git a/website/src/documentation/dsls/sql/aggregate-functions.md b/website/src/documentation/dsls/sql/aggregate-functions.md
deleted file mode 100644
index 12a33eb..0000000
--- a/website/src/documentation/dsls/sql/aggregate-functions.md
+++ /dev/null
@@ -1,34 +0,0 @@
----
-layout: section
-title: "Beam SQL: Aggregate functions"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/aggregate-functions/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL: Aggregate functions
-
-Beam SQL has implemented the following built-in functions See also [Calcite
-SQL's operators and functions
-reference](http://calcite.apache.org/docs/reference.html#operators-and-functions)
-
-| Operator syntax | Description |
-| ---- | ---- |
-| COUNT(*) | Returns the number of input rows |
-| AVG(numeric) | Returns the average (arithmetic mean) of numeric across all input values |
-| SUM(numeric) | Returns the sum of numeric across all input values |
-| MAX(value) | Returns the maximum value of value across all input values |
-| MIN(value) | Returns the minimum value of value across all input values |
-{:.table}
diff --git a/website/src/documentation/dsls/sql/calcite/aggregate-functions.md b/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
new file mode 100644
index 0000000..1416ad9
--- /dev/null
+++ b/website/src/documentation/dsls/sql/calcite/aggregate-functions.md
@@ -0,0 +1,33 @@
+---
+layout: section
+title: "Beam Calcite SQL aggregate functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/calcite/aggregate-functions/
+redirect_from: /documentation/dsls/sql/aggregate-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam Calcite SQL aggregate functions
+
+This page documents Apache Calcite aggregate functions supported by Beam Calcite SQL.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| COUNT(*) | Returns the number of input rows |
+| AVG(numeric) | Returns the average (arithmetic mean) of numeric across all input values |
+| SUM(numeric) | Returns the sum of numeric across all input values |
+| MAX(value) | Returns the maximum value of value across all input values |
+| MIN(value) | Returns the minimum value of value across all input values |
+{:.table}
diff --git a/website/src/documentation/dsls/sql/calcite/data-types.md b/website/src/documentation/dsls/sql/calcite/data-types.md
new file mode 100644
index 0000000..26040b1
--- /dev/null
+++ b/website/src/documentation/dsls/sql/calcite/data-types.md
@@ -0,0 +1,45 @@
+---
+layout: section
+title: "Beam Calcite SQL data types"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/calcite/data-types/
+redirect_from: /documentation/dsls/sql/data-types/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam Calcite SQL data types
+
+Beam SQL supports standard SQL scalar data types as well as extensions
+including arrays, maps, and nested rows. This page documents supported
+[Apache Calcite data types](http://calcite.apache.org/docs/reference.html#data-types) supported by Beam Calcite SQL.
+
+In Java, these types are mapped to Java types large enough to hold the
+full range of values.
+
+| SQL Type | Description | Java class |
+| --------- | ------------ | ---------- |
+| TINYINT | 1 byte signed integer in range -128 to 127 | java.lang.Byte |
+| SMALLINT | 2 byte signed integer in range -32768 to 32767 | java.lang.Short |
+| INTEGER | 4 byte signed integer in range -2147483648 to 2147483647 | java.lang.Integer |
+| BIGINT | 8 byte signed integer in range -9223372036854775808 to 9223372036854775807 | java.lang.Long |
+| FLOAT | 4 byte floating point | java.lang.Float |
+| DOUBLE | 8 byte floating point | java.lang.Double |
+| DECIMAL | Arbitrary precision decimal value | java.math.BigDecimal |
+| VARCHAR | Arbitrary length string | java.lang.String |
+| TIMESTAMP | Millisecond precision timestamp | org.joda.ReadableInstant |
+| ARRAY<type> | Ordered list of values | java.util.List |
+| MAP<type, type> | Finite unordered map | java.util.Map |
+| ROW<fields> | Nested row | org.apache.beam.sdk.values.Row |
+{:.table}
diff --git a/website/src/documentation/dsls/sql/calcite/lexical-structure.md b/website/src/documentation/dsls/sql/calcite/lexical-structure.md
new file mode 100644
index 0000000..e9dd95b
--- /dev/null
+++ b/website/src/documentation/dsls/sql/calcite/lexical-structure.md
@@ -0,0 +1,1049 @@
+---
+layout: section
+title: "Beam Calcite SQL lexical structure"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/calcite/lexical/
+redirect_from: /documentation/dsls/sql/lexical/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam Calcite SQL lexical structure
+
+A Beam Calcite SQL statements are comprised of a series of tokens. Tokens include
+*identifiers,* *quoted identifiers, literals*, *keywords*, *operators*,
+and *special characters*. Tokens can be separated by whitespace (space,
+backspace, tab, newline) or comments.
+
+Identifiers
+-----------
+
+Identifiers are names that are associated with columns, tables, and
+other database objects.
+
+Identifiers must begin with a letter or an underscore. Subsequent
+characters can be letters, numbers, or underscores. Quoted identifiers
+are identifiers enclosed by backtick (`` ` ``) characters and can contain
+any character, such as spaces or symbols. However, quoted identifiers
+cannot be empty. [Reserved Keywords](#reserved-keywords) can only be used
+as identifiers if enclosed by backticks.
+
+Syntax (defined here as a regular expression):
+
+`[A-Za-z_][A-Za-z_0-9]*`
+
+Examples:
+
+```
+Customers5
+_dataField1
+ADGROUP
+```
+
+Invalid examples:
+
+```
+5Customers
+_dataField!
+GROUP
+```
+
+`5Customers` begins with a number, not a letter or underscore.
+`_dataField!` contains the special character "!" which is not a letter,
+number, or underscore. `GROUP` is a reserved keyword, and therefore
+cannot be used as an identifier without being enclosed by backtick
+characters.
+
+Both identifiers and quoted identifiers are case insensitive, with some
+nuances. See [Case Sensitivity](#case-sensitivity) for further details.
+
+Quoted identifiers have the same escape sequences as string literals,
+defined below.
+
+Literals
+--------
+
+A literal represents a constant value of a built-in data type. Some, but
+not all, data types can be expressed as literals.
+
+### String Literals
+
+Both string and bytes literals must be *quoted* with single
+(`'`) quotation mark.
+
+**Quoted literals:**
+
+<table>
+<thead>
+<tr>
+<th>Literal</th>
+<th>Examples</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Quoted string</td>
+<td><ul><li><code>'it''s'</code></li><li><code>'Title: "Boy"'</code></li></ul></td>
+<td>Quoted strings enclosed by single (<code>'</code>) quotes can contain unescaped double (<code>"</code>) quotes. <br />Two quotation marks (<code>''</code>) is the escape sequence.<br />Quoted strings can contain newlines.</td>
+</tr>
+</tbody>
+</table>
+
+### Integer Literals
+
+Integer literals are either a sequence of decimal digits (0 through 9).
+Integers can be prefixed by "`+`" or "`-`" to represent positive and
+negative values, respectively.
+
+Examples:
+
+```
+123
+-123
+```
+
+An integer literal is interpreted as an `BIGINT`.
+
+### Floating Point Literals
+
+Syntax options:
+
+```
+[+-]DIGITS.[DIGITS][e[+-]DIGITS]
+[DIGITS].DIGITS[e[+-]DIGITS]
+DIGITSe[+-]DIGITS
+```
+
+`DIGITS` represents one or more decimal numbers (0 through 9) and `e`
+represents the exponent marker (e or E).
+
+Examples:
+
+```
+123.456e-67
+.1E4
+58.
+4e2
+```
+
+Numeric literals that contain either a decimal point or an exponent
+marker are presumed to be type double.
+
+Implicit coercion of floating point literals to float type is possible
+if the value is within the valid float range.
+
+There is no literal representation of NaN or infinity.
+
+### Array Literals
+
+Array literals are a comma-separated lists of elements enclosed in
+square brackets prefixed with the `ARRAY` keyword.
+
+Examples:
+
+```
+ARRAY[1, 2, 3]
+ARRAY['x', 'y', 'xy']
+```
+
+### Struct Literals
+
+Syntax:
+
+```
+(elem[, elem...])
+```
+
+where `elem` is an element in the struct. `elem` must be a literal data
+type, not an expression or column name.
+
+The output type is an anonymous struct type (structs are not named
+types) with anonymous fields with types matching the types of the input
+expressions.
+
+<table>
+<thead>
+<tr>
+<th>Example</th>
+<th>Output Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>(1, 2, 3)</code></td>
+<td><code>STRUCT<BIGINT,BIGINT,BIGINT></code></td>
+</tr>
+<tr>
+<td><code>(1, 'abc')</code></td>
+<td><code>STRUCT<BIGINT,STRING></code></td>
+</tr>
+</tbody>
+</table>
+
+### Date Literals
+
+Syntax:
+
+```
+DATE 'YYYY-M[M]-D[D]'
+```
+
+Date literals contain the `DATE` keyword followed by a string literal
+that conforms to the canonical date format, enclosed in single quotation
+marks. Date literals support a range between the years 1 and 9999,
+inclusive. Dates outside of this range are invalid.
+
+For example, the following date literal represents September 27, 2014:
+
+```
+DATE '2014-09-27'
+```
+
+String literals in canonical date format also implicitly coerce to DATE
+type when used where a DATE-type expression is expected. For example, in
+the query
+
+```
+SELECT * FROM foo WHERE date_col = "2014-09-27"
+```
+
+the string literal `"2014-09-27"` will be coerced to a date literal.
+
+### Time Literals
+
+Syntax:
+
+```
+TIME '[H]H:[M]M:[S]S[.DDDDDD]]'
+```
+
+TIME literals contain the `TIME` keyword and a string literal that
+conforms to the canonical time format, enclosed in single quotation
+marks.
+
+For example, the following time represents 12:30 p.m.:
+
+```
+TIME '12:30:00.45'
+```
+
+### Timestamp literals
+
+Syntax:
+
+```
+TIMESTAMP 'YYYY-[M]M-[D]D [[H]H:[M]M:[S]S[.DDDDDD]]'
+```
+
+Timestamp literals contain the `TIMESTAMP` keyword and a string literal
+that conforms to the canonical timestamp format, enclosed in single
+quotation marks.
+
+Timestamp literals support a range between the years 1 and 9999,
+inclusive. Timestamps outside of this range are invalid.
+
+For example, the following timestamp represents 12:30 p.m. on September
+27, 2014:
+
+```
+TIMESTAMP '2014-09-27 12:30:00.45'
+```
+
+Case Sensitivity
+----------------
+
+Beam SQL follows these rules for case sensitivity:
+
+<table>
+<thead>
+<tr>
+<th>Category</th>
+<th>Case Sensitive?</th>
+<th>Notes</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Keywords</td>
+<td>No</td>
+<td></td>
+</tr>
+<tr>
+<td>Function names</td>
+<td>No</td>
+<td></td>
+</tr>
+<tr>
+<td>Table names</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>Column names</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>String values</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>String comparisons</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>Aliases within a query</td>
+<td>No</td>
+<td></td>
+</tr>
+<tr>
+<td>Regular expression matching</td>
+<td>See Notes</td>
+<td>Regular expression matching is case sensitive by default, unless the expression itself specifies that it should be case insensitive.</td>
+</tr>
+<tr>
+<td><code>LIKE</code> matching</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+</tbody>
+</table>
+
+Reserved Keywords
+-----------------
+
+Keywords are a group of tokens that have special meaning in the Beam SQL
+language, and have the following characteristics:
+
+- Keywords cannot be used as identifiers unless enclosed by backtick
+ (\`) characters.
+- Keywords are case insensitive.
+
+Beam SQL has the following reserved keywords.
+
+<table style="table-layout: fixed; width: 110%">
+<tbody>
+<tr>
+<td>
+A<br />
+ABS<br />
+ABSOLUTE<br />
+ACTION<br />
+ADA<br />
+ADD<br />
+ADMIN<br />
+AFTER<br />
+ALL<br />
+ALLOCATE<br />
+ALLOW<br />
+ALTER<br />
+ALWAYS<br />
+AND<br />
+ANY<br />
+APPLY<br />
+ARE<br />
+ARRAY<br />
+ARRAY_MAX_CARDINALITY<br />
+AS<br />
+ASC<br />
+ASENSITIVE<br />
+ASSERTION<br />
+ASSIGNMENT<br />
+ASYMMETRIC<br />
+AT<br />
+ATOMIC<br />
+ATTRIBUTE<br />
+ATTRIBUTES<br />
+AUTHORIZATION<br />
+AVG<br />
+BEFORE<br />
+BEGIN<br />
+BEGIN_FRAME<br />
+BEGIN_PARTITION<br />
+BERNOULLI<br />
+BETWEEN<br />
+BIGINT<br />
+BINARY<br />
+BIT<br />
+BLOB<br />
+BOOLEAN<br />
+BOTH<br />
+BREADTH<br />
+BY<br />
+C<br />
+CALL<br />
+CALLED<br />
+CARDINALITY<br />
+CASCADE<br />
+CASCADED<br />
+CASE<br />
+CAST<br />
+CATALOG<br />
+CATALOG_NAME<br />
+CEIL<br />
+CEILING<br />
+CENTURY<br />
+CHAIN<br />
+CHAR<br />
+CHAR_LENGTH<br />
+CHARACTER<br />
+CHARACTER_LENGTH<br />
+CHARACTER_SET_CATALOG<br />
+CHARACTER_SET_NAME<br />
+CHARACTER_SET_SCHEMA<br />
+CHARACTERISTICS<br />
+CHARACTERS<br />
+CHECK<br />
+CLASSIFIER<br />
+CLASS_ORIGIN<br />
+CLOB<br />
+CLOSE<br />
+COALESCE<br />
+COBOL<br />
+COLLATE<br />
+COLLATION<br />
+COLLATION_CATALOG<br />
+COLLATION_NAME<br />
+COLLATION_SCHEMA<br />
+COLLECT<br />
+COLUMN<br />
+COLUMN_NAME<br />
+COMMAND_FUNCTION<br />
+COMMAND_FUNCTION_CODE<br />
+COMMENT<br />
+COMMIT<br />
+COMMITTED<br />
+CONDITION<br />
+CONDITION_NUMBER<br />
+CONNECT<br />
+CONNECTION<br />
+CONNECTION_NAME<br />
+CONSTRAINT<br />
+CONSTRAINT_CATALOG<br />
+CONSTRAINT_NAME<br />
+CONSTRAINT_SCHEMA<br />
+CONSTRAINTS<br />
+CONSTRUCTOR<br />
+CONTAINS<br />
+CONTINUE<br />
+CONVERT<br />
+CORR<br />
+CORRESPONDING<br />
+COUNT<br />
+COVAR_POP<br />
+COVAR_SAMP<br />
+CREATE<br />
+CROSS<br />
+CUBE<br />
+CUME_DIST<br />
+CURRENT<br />
+CURRENT_CATALOG<br />
+CURRENT_DATE<br />
+CURRENT_DEFAULT_TRANSFORM_GROUP<br />
+CURRENT_PATH<br />
+CURRENT_ROLE<br />
+CURRENT_ROW<br />
+CURRENT_SCHEMA<br />
+CURRENT_TIME<br />
+CURRENT_TIMESTAMP<br />
+CURRENT_TRANSFORM_GROUP_FOR_TYPE<br />
+CURRENT_USER<br />
+CURSOR<br />
+CURSOR_NAME<br />
+CYCLE<br />
+DATA<br />
+DATABASE<br />
+DATE<br />
+DATETIME_INTERVAL_CODE<br />
+DATETIME_INTERVAL_PRECISION<br />
+DAY<br />
+DEALLOCATE<br />
+DEC<br />
+DECADE<br />
+DECIMAL<br />
+DECLARE<br />
+DEFAULT<br />
+DEFAULTS<br />
+DEFERRABLE<br />
+DEFERRED<br />
+DEFINE<br />
+DEFINED<br />
+DEFINER<br />
+DEGREE<br />
+DELETE<br />
+DENSE_RANK<br />
+DEPTH<br />
+DEREF<br />
+DERIVED<br />
+DESC<br />
+DESCRIBE<br />
+DESCRIPTION<br />
+DESCRIPTOR<br />
+DETERMINISTIC<br />
+DIAGNOSTICS<br />
+DISALLOW<br />
+DISCONNECT<br />
+DISPATCH<br />
+DISTINCT<br />
+DOMAIN<br />
+DOUBLE<br />
+DOW<br />
+DOY<br />
+DROP<br />
+DYNAMIC<br />
+DYNAMIC_FUNCTION<br />
+DYNAMIC_FUNCTION_CODE<br />
+EACH<br />
+ELEMENT<br />
+ELSE<br />
+EMPTY<br />
+END<br />
+END-EXEC<br />
+END_FRAME<br />
+END_PARTITION<br />
+EPOCH<br />
+EQUALS<br />
+ESCAPE<br />
+EVERY<br />
+EXCEPT<br />
+EXCEPTION<br />
+EXCLUDE<br />
+EXCLUDING<br />
+EXEC<br />
+EXECUTE<br />
+EXISTS<br />
+EXP<br />
+EXPLAIN<br />
+EXTEND<br />
+EXTERNAL<br />
+EXTRACT<br />
+FALSE<br />
+FETCH<br />
+FILTER<br />
+FINAL<br />
+FIRST<br />
+FIRST_VALUE<br />
+FLOAT<br />
+FLOOR<br />
+FOLLOWING<br />
+FOR<br />
+FOREIGN<br />
+FORTRAN<br />
+FOUND<br />
+FRAC_SECOND<br />
+FRAME_ROW<br />
+FREE<br />
+FROM<br />
+FULL<br />
+FUNCTION<br />
+FUSION<br />
+G<br />
+GENERAL<br />
+GENERATED<br />
+GEOMETRY<br />
+GET<br />
+GLOBAL<br />
+GO<br />
+GOTO<br />
+GRANT<br />
+GRANTED<br />
+GROUP<br />
+GROUPING<br />
+GROUPS<br />
+HAVING<br />
+HIERARCHY<br />
+HOLD<br />
+HOUR<br />
+IDENTITY<br />
+IF<br />
+IMMEDIATE<br />
+IMMEDIATELY<br />
+IMPLEMENTATION<br />
+IMPORT<br />
+IN<br />
+INCLUDING<br />
+INCREMENT<br />
+INDICATOR<br />
+INITIAL<br />
+INITIALLY<br />
+INNER<br />
+INOUT<br />
+INPUT<br />
+INSENSITIVE<br />
+INSERT<br />
+INSTANCE<br />
+INSTANTIABLE<br />
+INT<br />
+INTEGER<br />
+INTERSECT<br />
+INTERSECTION<br />
+INTERVAL<br />
+INTO<br />
+INVOKER<br />
+IS<br />
+ISOLATION<br />
+JAVA<br />
+JOIN<br />
+JSON<br />
+K<br />
+KEY<br />
+KEY_MEMBER<br />
+KEY_TYPE<br />
+LABEL<br />
+LAG<br />
+LANGUAGE<br />
+LARGE<br />
+LAST<br />
+LAST_VALUE<br />
+LATERAL<br />
+LEAD<br />
+LEADING<br />
+LEFT<br />
+LENGTH<br />
+LEVEL<br />
+LIBRARY<br />
+LIKE<br />
+LIKE_REGEX<br />
+LIMIT<br />
+LN<br />
+LOCAL<br />
+LOCALTIME<br />
+LOCALTIMESTAMP<br />
+LOCATION<br />
+LOCATOR<br />
+LOWER<br />
+M<br />
+MAP<br />
+MATCH<br />
+MATCHED<br />
+MATCHES<br />
+MATCH_NUMBER<br />
+MATCH_RECOGNIZE<br />
+MAX<br />
+MAXVALUE<br />
+MEASURES<br />
+MEMBER<br />
+MERGE<br />
+MESSAGE_LENGTH<br />
+MESSAGE_OCTET_LENGTH<br />
+MESSAGE_TEXT<br />
+METHOD<br />
+MICROSECOND<br />
+MILLENNIUM<br />
+MIN<br />
+MINUTE<br />
+MINVALUE<br />
+MOD<br />
+MODIFIES<br />
+MODULE<br />
+MONTH<br />
+MORE<br />
+MULTISET<br />
+MUMPS<br />
+NAME<br />
+NAMES<br />
+NATIONAL<br />
+NATURAL<br />
+NCHAR<br />
+NCLOB<br />
+NESTING<br />
+</td>
+<td>
+NEW<br />
+NEXT<br />
+NO<br />
+NONE<br />
+NORMALIZE<br />
+NORMALIZED<br />
+NOT<br />
+NTH_VALUE<br />
+NTILE<br />
+NULL<br />
+NULLABLE<br />
+NULLIF<br />
+NULLS<br />
+NUMBER<br />
+NUMERIC<br />
+OBJECT<br />
+OCCURRENCES_REGEX<br />
+OCTET_LENGTH<br />
+OCTETS<br />
+OF<br />
+OFFSET<br />
+OLD<br />
+OMIT<br />
+ON<br />
+ONE<br />
+ONLY<br />
+OPEN<br />
+OPTION<br />
+OPTIONS<br />
+OR<br />
+ORDER<br />
+ORDERING<br />
+ORDINALITY<br />
+OTHERS<br />
+OUT<br />
+OUTER<br />
+OUTPUT<br />
+OVER<br />
+OVERLAPS<br />
+OVERLAY<br />
+OVERRIDING<br />
+PAD<br />
+PARAMETER<br />
+PARAMETER_MODE<br />
+PARAMETER_NAME<br />
+PARAMETER_ORDINAL_POSITION<br />
+PARAMETER_SPECIFIC_CATALOG<br />
+PARAMETER_SPECIFIC_NAME<br />
+PARAMETER_SPECIFIC_SCHEMA<br />
+PARTIAL<br />
+PARTITION<br />
+PASCAL<br />
+PASSTHROUGH<br />
+PAST<br />
+PATH<br />
+PATTERN<br />
+PER<br />
+PERCENT<br />
+PERCENTILE_CONT<br />
+PERCENTILE_DISC<br />
+PERCENT_RANK<br />
+PERIOD<br />
+PERMUTE<br />
+PLACING<br />
+PLAN<br />
+PLI<br />
+PORTION<br />
+POSITION<br />
+POSITION_REGEX<br />
+POWER<br />
+PRECEDES<br />
+PRECEDING<br />
+PRECISION<br />
+PREPARE<br />
+PRESERVE<br />
+PREV<br />
+PRIMARY<br />
+PRIOR<br />
+PRIVILEGES<br />
+PROCEDURE<br />
+PUBLIC<br />
+QUARTER<br />
+RANGE<br />
+RANK<br />
+READ<br />
+READS<br />
+REAL<br />
+RECURSIVE<br />
+REF<br />
+REFERENCES<br />
+REFERENCING<br />
+REGR_AVGX<br />
+REGR_AVGY<br />
+REGR_COUNT<br />
+REGR_INTERCEPT<br />
+REGR_R2<br />
+REGR_SLOPE<br />
+REGR_SXX<br />
+REGR_SXY<br />
+REGR_SYY<br />
+RELATIVE<br />
+RELEASE<br />
+REPEATABLE<br />
+REPLACE<br />
+RESET<br />
+RESTART<br />
+RESTRICT<br />
+RESULT<br />
+RETURN<br />
+RETURNED_CARDINALITY<br />
+RETURNED_LENGTH<br />
+RETURNED_OCTET_LENGTH<br />
+RETURNED_SQLSTATE<br />
+RETURNS<br />
+REVOKE<br />
+RIGHT<br />
+ROLE<br />
+ROLLBACK<br />
+ROLLUP<br />
+ROUTINE<br />
+ROUTINE_CATALOG<br />
+ROUTINE_NAME<br />
+ROUTINE_SCHEMA<br />
+ROW<br />
+ROW_COUNT<br />
+ROW_NUMBER<br />
+ROWS<br />
+RUNNING<br />
+SAVEPOINT<br />
+SCALE<br />
+SCHEMA<br />
+SCHEMA_NAME<br />
+SCOPE<br />
+SCOPE_CATALOGS<br />
+SCOPE_NAME<br />
+SCOPE_SCHEMA<br />
+SCROLL<br />
+SEARCH<br />
+SECOND<br />
+SECTION<br />
+SECURITY<br />
+SEEK<br />
+SELECT<br />
+SELF<br />
+SENSITIVE<br />
+SEQUENCE<br />
+SERIALIZABLE<br />
+SERVER<br />
+SERVER_NAME<br />
+SESSION<br />
+SESSION_USER<br />
+SET<br />
+SETS<br />
+MINUS<br />
+SHOW<br />
+SIMILAR<br />
+SIMPLE<br />
+SIZE<br />
+SKIP<br />
+SMALLINT<br />
+SOME<br />
+SOURCE<br />
+SPACE<br />
+SPECIFIC<br />
+SPECIFIC_NAME<br />
+SPECIFICTYPE<br />
+SQL<br />
+SQLEXCEPTION<br />
+SQLSTATE<br />
+SQLWARNING<br />
+SQL_BIGINT<br />
+SQL_BINARY<br />
+SQL_BIT<br />
+SQL_BLOB<br />
+SQL_BOOLEAN<br />
+SQL_CHAR<br />
+SQL_CLOB<br />
+SQL_DATE<br />
+SQL_DECIMAL<br />
+SQL_DOUBLE<br />
+SQL_FLOAT<br />
+SQL_INTEGER<br />
+SQL_INTERVAL_DAY<br />
+SQL_INTERVAL_DAY_TO_HOUR<br />
+SQL_INTERVAL_DAY_TO_MINUTE<br />
+SQL_INTERVAL_DAY_TO_SECOND<br />
+SQL_INTERVAL_HOUR<br />
+SQL_INTERVAL_HOUR_TO_MINUTE<br />
+SQL_INTERVAL_HOUR_TO_SECOND<br />
+SQL_INTERVAL_MINUTE<br />
+SQL_INTERVAL_MINUTE_TO_SECOND<br />
+SQL_INTERVAL_MONTH<br />
+SQL_INTERVAL_SECOND<br />
+SQL_INTERVAL_YEAR<br />
+SQL_INTERVAL_YEAR_TO_MONTH<br />
+SQL_LONGVARBINARY<br />
+SQL_LONGVARCHAR<br />
+SQL_LONGVARNCHAR<br />
+SQL_NCHAR<br />
+SQL_NCLOB<br />
+SQL_NUMERIC<br />
+SQL_NVARCHAR<br />
+SQL_REAL<br />
+SQL_SMALLINT<br />
+SQL_TIME<br />
+SQL_TIMESTAMP<br />
+SQL_TINYINT<br />
+SQL_TSI_DAY<br />
+SQL_TSI_FRAC_SECOND<br />
+SQL_TSI_HOUR<br />
+SQL_TSI_MICROSECOND<br />
+SQL_TSI_MINUTE<br />
+SQL_TSI_MONTH<br />
+SQL_TSI_QUARTER<br />
+SQL_TSI_SECOND<br />
+SQL_TSI_WEEK<br />
+SQL_TSI_YEAR<br />
+SQL_VARBINARY<br />
+SQL_VARCHAR<br />
+SQRT<br />
+START<br />
+STATE<br />
+STATEMENT<br />
+STATIC<br />
+STDDEV_POP<br />
+STDDEV_SAMP<br />
+STREAM<br />
+STRUCTURE<br />
+STYLE<br />
+SUBCLASS_ORIGIN<br />
+SUBMULTISET<br />
+SUBSET<br />
+SUBSTITUTE<br />
+SUBSTRING<br />
+SUBSTRING_REGEX<br />
+SUCCEEDS<br />
+SUM<br />
+SYMMETRIC<br />
+SYSTEM<br />
+SYSTEM_TIME<br />
+SYSTEM_USER<br />
+TABLE<br />
+TABLE_NAME<br />
+TABLESAMPLE<br />
+TBLPROPERTIES<br />
+TEMPORARY<br />
+THEN<br />
+TIES<br />
+TIME<br />
+TIMESTAMP<br />
+TIMESTAMPADD<br />
+TIMESTAMPDIFF<br />
+TIMEZONE_HOUR<br />
+TIMEZONE_MINUTE<br />
+TINYINT<br />
+TO<br />
+TOP_LEVEL_COUNT<br />
+TRAILING<br />
+TRANSACTION<br />
+TRANSACTIONS_ACTIVE<br />
+TRANSACTIONS_COMMITTED<br />
+TRANSACTIONS_ROLLED_BACK<br />
+TRANSFORM<br />
+TRANSFORMS<br />
+TRANSLATE<br />
+TRANSLATE_REGEX<br />
+TRANSLATION<br />
+TREAT<br />
+TRIGGER<br />
+TRIGGER_CATALOG<br />
+TRIGGER_NAME<br />
+TRIGGER_SCHEMA<br />
+TRIM<br />
+TRIM_ARRAY<br />
+TRUE<br />
+TRUNCATE<br />
+TYPE<br />
+UESCAPE<br />
+UNBOUNDED<br />
+UNCOMMITTED<br />
+UNDER<br />
+UNION<br />
+UNIQUE<br />
+UNKNOWN<br />
+UNNAMED<br />
+UNNEST<br />
+UPDATE<br />
+UPPER<br />
+UPSERT<br />
+USAGE<br />
+USER<br />
+USER_DEFINED_TYPE_CATALOG<br />
+USER_DEFINED_TYPE_CODE<br />
+USER_DEFINED_TYPE_NAME<br />
+USER_DEFINED_TYPE_SCHEMA<br />
+USING<br />
+VALUE<br />
+VALUES<br />
+VALUE_OF<br />
+VAR_POP<br />
+VAR_SAMP<br />
+VARBINARY<br />
+VARCHAR<br />
+VARYING<br />
+VERSION<br />
+VERSIONING<br />
+VIEW<br />
+WEEK<br />
+WHEN<br />
+WHENEVER<br />
+WHERE<br />
+WIDTH_BUCKET<br />
+WINDOW<br />
+WITH<br />
+WITHIN<br />
+WITHOUT<br />
+WORK<br />
+WRAPPER<br />
+WRITE<br />
+XML<br />
+YEAR<br />
+ZONE<br />
+</td>
+</tr>
+</tbody>
+</table>
+
+Terminating Semicolons
+----------------------
+
+Statements can optionally use a terminating semicolon (`;`) in the
+context of a query string submitted through an Application Programming
+Interface (API). Some interactive tools require statements to have a
+terminating semicolon. In a request containing multiple statements,
+statements must be separated by semicolons, but the semicolon is
+optional for the final statement.
+
+Comments
+--------
+
+Comments are sequences of characters that are ignored by the parser.
+Beam SQL supports the following types of comments.
+
+### Single line comments
+
+Single line comments are supported by prepending `--` before the comment.
+
+**Examples**
+
+`SELECT x FROM T; --x is a field and T is a table`
+
+Comment includes all characters from the '`--`' sequence to the end of
+the line. You can optionally add a space after the '`--`'.
+
+### Multiline comments
+
+Multiline comments are supported by enclosing the comment using
+`/* <comment> */`.
+
+**Example:**
+
+```
+SELECT x FROM T /* x is a field and T is a table */
+WHERE x = 3;
+```
+
+**Invalid example:**
+
+```
+SELECT x FROM T /* comment starts here
+ /* comment ends on this line */
+ this line is not considered a comment */
+WHERE x = 3;
+```
+
+Comment includes all characters, including newlines, enclosed by the
+first occurrence of '`/*`' and the first subsequent occurrence of
+'`*/`'. Nested comments are not supported. The second example contains a
+nested comment that renders the query invalid.
+
+> Portions of this page are modifications based on work created and
+> [shared by Google](https://developers.google.com/terms/site-policies)
+> and used according to terms described in the [Creative Commons 3.0
+> Attribution License](http://creativecommons.org/licenses/by/3.0/).
diff --git a/website/src/documentation/dsls/sql/calcite/overview.md b/website/src/documentation/dsls/sql/calcite/overview.md
new file mode 100644
index 0000000..8498802
--- /dev/null
+++ b/website/src/documentation/dsls/sql/calcite/overview.md
@@ -0,0 +1,81 @@
+---
+layout: section
+title: "Beam Calcite SQL overview"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/calcite/overview/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Beam Calcite SQL overview
+
+[Apache Calcite](http://calcite.apache.org) is a widespread SQL dialect used in
+big data processing with some streaming enhancements. Beam Calcite SQL is the default Beam SQL dialect.
+
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects, including Beam Calcite SQL.
+
+## Query syntax
+Query statements scan one or more tables or expressions and return the computed result rows. For more information about query statements in Beam Calcite SQL, see the [Query syntax]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/query-syntax) reference.
+
+## Lexical structure
+A Beam SQL statement comprises a series of tokens. For more information about tokens in Beam Calcite SQL, see the [Lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/lexical) reference.
+
+## Data types
+Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. For more information about scalar data in Beam Calcite SQL, see the [Data types]({{ site.baseurl }}/documentation/dsls/sql/calcite/data-types) reference.
+
+## Functions and operators
+The following table summarizes the Apache Calcite functions and operators supported by Beam Calcite SQL.
+
+<table class="table-bordered table-striped">
+ <tr><th>Operators and functions</th><th>Beam SQL support status</th></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#operator-precedence">Operator precedence</a></td><td>Yes</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#comparison-operators">Comparison operators</a></td><td class="style1">See Beam SQL <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/scalar-functions/#comparison-functions-and-operators">scalar functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#logical-operators">Logical operators</a></td><td>See Beam SQL <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/scalar-functions/#logical-functions-and-operators">scalar functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#arithmetic-operators-and-functions">Arithmetic operators and functions</a></td><td>See Beam SQL <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/scalar-functions/#arithmetic-expressions">scalar functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#character-string-operators-and-functions">Character string operators and functions</a></td><td>See Beam SQL <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/scalar-functions/#string-functions">scalar functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#binary-string-operators-and-functions">Binary string operators and functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#datetime-functions">Date/time functions</a></td><td>See Beam SQL <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/scalar-functions/#date-functions">scalar functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#system-functions">System functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#conditional-functions-and-operators">Conditional functions and operators</a></td><td>See Beam SQL <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/scalar-functions/#conditional-functions">scalar functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#type-conversion">Type conversion</a></td><td>Yes</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#value-constructors">Value constructors</a></td><td>No, except array</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#collection-functions">Collection functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#period-predicates">Period predicates</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#jdbc-function-escape">JDBC function escape</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#aggregate-functions">Aggregate functions</a></td>
+<td>See Beam SQL extension <a href="{{ site.baseurl
+}}/documentation/dsls/sql/calcite/aggregate-functions/">aggregate functions</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#window-functions">Window functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#grouping-functions">Grouping functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#grouped-window-functions">Grouped window functions</a></td><td>See Beam SQL extension <a href="{{ site.baseurl
+}}/documentation/dsls/sql/windowing-and-triggering/">windowing and triggering</a></td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#grouped-auxiliary-functions">Grouped auxiliary functions</a></td><td>Yes, except SESSION_END</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#spatial-functions">Spatial functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#geometry-creation-functions-3d">Geometry creation functions (3D)</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#geometry-predicates">Geometry predicates</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#json-functions">JSON functions</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#user-defined-functions">User-defined functions</a></td>
+<td>See Beam SQL extension <a href="{{ site.baseurl
+}}/documentation/dsls/sql/user-defined-functions/">user-defined functions</a>. You cannot call functions with <a href="http://calcite.apache.org/docs/reference.html#calling-functions-with-named-and-optional-parameters">named and optional parameters</a>.</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#match_recognize">MATCH_RECOGNIZE</a></td><td>No</td></tr>
+<tr><td><a href="http://calcite.apache.org/docs/reference.html#ddl-extensions">DDL Extensions</a></td><td>See Beam SQL extension <a href="{{ site.baseurl
+}}/documentation/dsls/sql/create-external-table/">CREATE EXTERNAL TABLE</a></td></tr>
+</table>
diff --git a/website/src/documentation/dsls/sql/calcite/query-syntax.md b/website/src/documentation/dsls/sql/calcite/query-syntax.md
new file mode 100644
index 0000000..e55d562
--- /dev/null
+++ b/website/src/documentation/dsls/sql/calcite/query-syntax.md
@@ -0,0 +1,716 @@
+---
+layout: section
+title: "Beam Calcite SQL query syntax"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/calcite/query-syntax/
+redirect_from: /documentation/dsls/sql/statements/select/
+ /documentation/dsls/sql/select/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam Calcite SQL query syntax
+
+Query statements scan one or more tables or expressions and return the computed
+result rows.
+
+Generally, the semantics of queries is standard. See the following
+sections to learn about extensions for supporting Beam's unified
+batch/streaming model:
+
+ - [Joins]({{ site.baseurl}}/documentation/dsls/sql/extensions/joins)
+ - [Windowing & Triggering]({{ site.baseurl}}/documentation/dsls/sql/windowing-and-triggering/)
+
+The main functionality of Beam SQL is the `SELECT` statement. This is how you
+query and join data. The operations supported are a subset of
+[Apache Calcite SQL](http://calcite.apache.org/docs/reference.html#grammar).
+
+## SQL Syntax
+
+ query_statement:
+ [ WITH with_query_name AS ( query_expr ) [, ...] ]
+ query_expr
+
+ query_expr:
+ { select | ( query_expr ) | query_expr set_op query_expr }
+ [ LIMIT count [ OFFSET skip_rows ] ]
+
+ select:
+ SELECT [{ ALL | DISTINCT }]
+ { [ expression. ]* [ EXCEPT ( column_name [, ...] ) ]
+ [ REPLACE ( expression [ AS ] column_name [, ...] ) ]
+ | expression [ [ AS ] alias ] } [, ...]
+ [ FROM from_item [, ...] ]
+ [ WHERE bool_expression ]
+ [ GROUP BY { expression [, ...] | ROLLUP ( expression [, ...] ) } ]
+ [ HAVING bool_expression ]
+
+ set_op:
+ UNION { ALL | DISTINCT } | INTERSECT DISTINCT | EXCEPT DISTINCT
+
+ from_item: {
+ table_name [ [ AS ] alias ] |
+ join |
+ ( query_expr ) [ [ AS ] alias ]
+ with_query_name [ [ AS ] alias ]
+ }
+
+ join:
+ from_item [ join_type ] JOIN from_item
+ [ { ON bool_expression | USING ( join_column [, ...] ) } ]
+
+ join_type:
+ { INNER | CROSS | FULL [OUTER] | LEFT [OUTER] | RIGHT [OUTER] }
+
+Notation:
+
+- Square brackets "\[ \]" indicate optional clauses.
+- Parentheses "( )" indicate literal parentheses.
+- The vertical bar "|" indicates a logical OR.
+- Curly braces "{ }" enclose a set of options.
+- A comma followed by an ellipsis within square brackets "\[, ... \]"
+ indicates that the preceding item can repeat in a comma-separated list.
+
+## SELECT list
+
+Syntax:
+
+ SELECT [{ ALL | DISTINCT }]
+ { [ expression. ]*
+ | expression [ [ AS ] alias ] } [, ...]
+
+The `SELECT` list defines the columns that the query will return. Expressions in
+the `SELECT` list can refer to columns in any of the `from_item`s in its
+corresponding `FROM` clause.
+
+Each item in the `SELECT` list is one of:
+
+- \*
+- `expression`
+- `expression.*`
+
+### SELECT \*
+
+`SELECT *`, often referred to as *select star*, produces one output column for
+each column that is visible after executing the full query.
+
+```
+SELECT * FROM (SELECT 'apple' AS fruit, 'carrot' AS vegetable);
+
++-------+-----------+
+| fruit | vegetable |
++-------+-----------+
+| apple | carrot |
++-------+-----------+
+```
+
+### SELECT `expression`
+
+Items in a `SELECT` list can be expressions. These expressions evaluate to a
+single value and produce one output column, with an optional explicit `alias`.
+
+If the expression does not have an explicit alias, it receives an implicit alias
+according to the rules for [implicit aliases](#implicit-aliases), if possible.
+Otherwise, the column is anonymous and you cannot refer to it by name elsewhere
+in the query.
+
+### SELECT `expression.*` {#select-expression_1}
+
+An item in a `SELECT` list can also take the form of `expression.*`. This
+produces one output column for each column or top-level field of `expression`.
+The expression must be a table alias.
+
+The following query produces one output column for each column in the table
+`groceries`, aliased as `g`.
+
+```
+WITH groceries AS
+ (SELECT 'milk' AS dairy,
+ 'eggs' AS protein,
+ 'bread' AS grain)
+SELECT g.*
+FROM groceries AS g;
+
++-------+---------+-------+
+| dairy | protein | grain |
++-------+---------+-------+
+| milk | eggs | bread |
++-------+---------+-------+
+```
+
+### SELECT modifiers
+
+You can modify the results returned from a `SELECT` query, as follows.
+
+#### SELECT DISTINCT
+
+A `SELECT DISTINCT` statement discards duplicate rows and returns only the
+remaining rows. `SELECT DISTINCT` cannot return columns of the following types:
+
+- STRUCT
+- ARRAY
+
+#### SELECT ALL
+
+A `SELECT ALL` statement returns all rows, including duplicate rows. `SELECT
+ALL` is the default behavior of `SELECT`.
+
+### Aliases
+
+See [Aliases](#aliases_2) for information on syntax and visibility for
+`SELECT` list aliases.
+
+## FROM clause
+
+The `FROM` clause indicates the table or tables from which to retrieve rows, and
+specifies how to join those rows together to produce a single stream of rows for
+processing in the rest of the query.
+
+### Syntax
+
+ from_item: {
+ table_name [ [ AS ] alias ] |
+ join |
+ ( query_expr ) [ [ AS ] alias ] |
+ with_query_name [ [ AS ] alias ]
+ }
+
+#### table\_name
+
+The name (optionally qualified) of an existing table.
+
+ SELECT * FROM Roster;
+ SELECT * FROM beam.Roster;
+
+#### join
+
+See [JOIN Types](#join-types) below and [Joins]({{ site.baseurl}}/documentation/dsls/sql/extensions/joins).
+
+#### select {#select_1}
+
+`( select ) [ [ AS ] alias ]` is a table [subquery](#subqueries).
+
+#### with\_query\_name
+
+The query names in a `WITH` clause (see [WITH Clause](#with-clause)) act like
+names of temporary tables that you can reference anywhere in the `FROM` clause.
+In the example below, `subQ1` and `subQ2` are `with_query_names`.
+
+Example:
+
+ WITH
+ subQ1 AS (SELECT * FROM Roster WHERE SchoolID = 52),
+ subQ2 AS (SELECT SchoolID FROM subQ1)
+ SELECT DISTINCT * FROM subQ2;
+
+The `WITH` clause hides any permanent tables with the same name for the duration
+of the query, unless you qualify the table name, e.g. `beam.Roster`.
+
+### Subqueries
+
+A subquery is a query that appears inside another statement, and is written
+inside parentheses. These are also referred to as "sub-SELECTs" or "nested
+SELECTs". The full `SELECT` syntax is valid in subqueries.
+
+There are two types of subquery:
+
+- Expression Subqueries
+ which you can use in a query wherever expressions are valid. Expression
+ subqueries return a single value.
+- Table subqueries, which you can use only in a `FROM` clause. The outer query
+ treats the result of the subquery as a table.
+
+Note that there must be parentheses around both types of subqueries.
+
+Example:
+
+```
+SELECT AVG ( PointsScored )
+FROM
+( SELECT PointsScored
+ FROM Stats
+ WHERE SchoolID = 77 )
+```
+
+Optionally, a table subquery can have an alias.
+
+Example:
+
+```
+SELECT r.LastName
+FROM
+( SELECT * FROM Roster) AS r;
+```
+
+### Aliases {#aliases_1}
+
+See [Aliases](#aliases_2) for information on syntax and visibility for
+`FROM` clause aliases.
+
+## JOIN types
+
+Also see [Joins]({{ site.baseurl}}/documentation/dsls/sql/extensions/joins).
+
+### Syntax {#syntax_1}
+
+ join:
+ from_item [ join_type ] JOIN from_item
+ [ ON bool_expression | USING ( join_column [, ...] ) ]
+
+ join_type:
+ { INNER | CROSS | FULL [OUTER] | LEFT [OUTER] | RIGHT [OUTER] }
+
+The `JOIN` clause merges two `from_item`s so that the `SELECT` clause can query
+them as one source. The `join_type` and `ON` or `USING` clause (a "join
+condition") specify how to combine and discard rows from the two `from_item`s to
+form a single source.
+
+All `JOIN` clauses require a `join_type`.
+
+A `JOIN` clause requires a join condition unless one of the following conditions
+is true:
+
+- `join_type` is `CROSS`.
+- One or both of the `from_item`s is not a table, e.g. an `array_path` or
+ `field_path`.
+
+### \[INNER\] JOIN
+
+An `INNER JOIN`, or simply `JOIN`, effectively calculates the Cartesian product
+of the two `from_item`s and discards all rows that do not meet the join
+condition. "Effectively" means that it is possible to implement an `INNER JOIN`
+without actually calculating the Cartesian product.
+
+### CROSS JOIN
+
+`CROSS JOIN` is generally not yet supported.
+
+### FULL \[OUTER\] JOIN
+
+A `FULL OUTER JOIN` (or simply `FULL JOIN`) returns all fields for all rows in
+both `from_item`s that meet the join condition.
+
+`FULL` indicates that *all rows* from both `from_item`s are returned, even if
+they do not meet the join condition. For streaming jobs, all rows that are
+not late according to default trigger and belonging to the same window
+if there's non-global window applied.
+
+`OUTER` indicates that if a given row from one `from_item` does not join to any
+row in the other `from_item`, the row will return with NULLs for all columns
+from the other `from_item`.
+
+Also see [Joins]({{ site.baseurl}}/documentation/dsls/sql/extensions/joins).
+
+### LEFT \[OUTER\] JOIN
+
+The result of a `LEFT OUTER JOIN` (or simply `LEFT JOIN`) for two `from_item`s
+always retains all rows of the left `from_item` in the `JOIN` clause, even if no
+rows in the right `from_item` satisfy the join predicate.
+
+`LEFT` indicates that all rows from the *left* `from_item` are returned; if a
+given row from the left `from_item` does not join to any row in the *right*
+`from_item`, the row will return with NULLs for all columns from the right
+`from_item`. Rows from the right `from_item` that do not join to any row in the
+left `from_item` are discarded.
+
+### RIGHT \[OUTER\] JOIN
+
+The result of a `RIGHT OUTER JOIN` (or simply `RIGHT JOIN`) is similar and
+symmetric to that of `LEFT OUTER JOIN`.
+
+### ON clause
+
+The `ON` clause contains a `bool_expression`. A combined row (the result of
+joining two rows) meets the join condition if `bool_expression` returns TRUE.
+
+Example:
+
+```
+SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;
+```
+
+### USING clause
+
+The `USING` clause requires a `column_list` of one or more columns which occur
+in both input tables. It performs an equality comparison on that column, and the
+rows meet the join condition if the equality comparison returns TRUE.
+
+In most cases, a statement with the `USING` keyword is equivalent to using the
+`ON` keyword. For example, the statement:
+
+```
+SELECT FirstName
+FROM Roster INNER JOIN PlayerStats
+USING (LastName);
+```
+
+is equivalent to:
+
+```
+SELECT FirstName
+FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;
+```
+
+The results from queries with `USING` do differ from queries that use `ON` when
+you use `SELECT *`. To illustrate this, consider the query:
+
+```
+SELECT * FROM Roster INNER JOIN PlayerStats
+USING (LastName);
+```
+
+This statement returns the rows from `Roster` and `PlayerStats` where
+`Roster.LastName` is the same as `PlayerStats.LastName`. The results include a
+single `LastName` column.
+
+By contrast, consider the following query:
+
+```
+SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;
+```
+
+This statement returns the rows from `Roster` and `PlayerStats` where
+`Roster.LastName` is the same as `PlayerStats.LastName`. The results include two
+`LastName` columns; one from `Roster` and one from `PlayerStats`.
+
+### Sequences of JOINs
+
+The `FROM` clause can contain multiple `JOIN` clauses in sequence.
+
+Example:
+
+```
+SELECT * FROM a LEFT JOIN b ON TRUE LEFT JOIN c ON TRUE;
+```
+
+where `a`, `b`, and `c` are any `from_item`s. JOINs are bound from left to
+right, but you can insert parentheses to group them in a different order.
+
+## WHERE clause
+
+### Syntax {#syntax_2}
+
+```
+WHERE bool_expression
+```
+
+The `WHERE` clause filters out rows by evaluating each row against
+`bool_expression`, and discards all rows that do not return TRUE (that is, rows
+that return FALSE or NULL).
+
+Example:
+
+```
+SELECT * FROM Roster
+WHERE SchoolID = 52;
+```
+
+The `bool_expression` can contain multiple sub-conditions.
+
+Example:
+
+```
+SELECT * FROM Roster
+WHERE LastName LIKE 'Mc%' OR LastName LIKE 'Mac%';
+```
+
+You cannot reference column aliases from the `SELECT` list in the `WHERE`
+clause.
+
+Expressions in an `INNER JOIN` have an equivalent expression in the `WHERE`
+clause. For example, a query using `INNER` `JOIN` and `ON` has an equivalent
+expression using `CROSS JOIN` and `WHERE`.
+
+Example - this query:
+
+```
+SELECT * FROM Roster INNER JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;
+```
+
+is equivalent to:
+
+```
+SELECT * FROM Roster CROSS JOIN TeamMascot
+WHERE Roster.SchoolID = TeamMascot.SchoolID;
+```
+
+## GROUP BY clause
+
+Also see [Windowing & Triggering]({{ site.baseurl}}/documentation/dsls/sql/windowing-and-triggering/)
+
+### Syntax {#syntax_3}
+
+ GROUP BY { expression [, ...] | ROLLUP ( expression [, ...] ) }
+
+The `GROUP BY` clause groups together rows in a table with non-distinct values
+for the `expression` in the `GROUP BY` clause. For multiple rows in the source
+table with non-distinct values for `expression`, the `GROUP BY` clause produces
+a single combined row. `GROUP BY` is commonly used when aggregate functions are
+present in the `SELECT` list, or to eliminate redundancy in the output.
+
+Example:
+
+```
+SELECT SUM(PointsScored), LastName
+FROM PlayerStats
+GROUP BY LastName;
+```
+
+## HAVING clause
+
+### Syntax {#syntax_4}
+
+```
+HAVING bool_expression
+```
+
+The `HAVING` clause is similar to the `WHERE` clause: it filters out rows that
+do not return TRUE when they are evaluated against the `bool_expression`.
+
+As with the `WHERE` clause, the `bool_expression` can be any expression that
+returns a boolean, and can contain multiple sub-conditions.
+
+The `HAVING` clause differs from the `WHERE` clause in that:
+
+- The `HAVING` clause requires `GROUP BY` or aggregation to be present in the
+ query.
+- The `HAVING` clause occurs after `GROUP BY` and aggregation.
+ This means that the `HAVING` clause is evaluated once for every
+ aggregated row in the result set. This differs from the `WHERE` clause,
+ which is evaluated before `GROUP BY` and aggregation.
+
+The `HAVING` clause can reference columns available via the `FROM` clause, as
+well as `SELECT` list aliases. Expressions referenced in the `HAVING` clause
+must either appear in the `GROUP BY` clause or they must be the result of an
+aggregate function:
+
+```
+SELECT LastName
+FROM Roster
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;
+```
+
+## Set operators
+
+### Syntax {#syntax_6}
+
+ UNION { ALL | DISTINCT } | INTERSECT DISTINCT | EXCEPT DISTINCT
+
+Set operators combine results from two or more input queries into a single
+result set. You must specify `ALL` or `DISTINCT`; if you specify `ALL`, then all
+rows are retained. If `DISTINCT` is specified, duplicate rows are discarded.
+
+If a given row R appears exactly m times in the first input query and n times in
+the second input query (m >= 0, n >= 0):
+
+- For `UNION ALL`, R appears exactly m + n times in the result.
+- For `UNION DISTINCT`, the `DISTINCT` is computed after the `UNION` is
+ computed, so R appears exactly one time.
+- For `INTERSECT DISTINCT`, the `DISTINCT` is computed after the result above
+ is computed.
+- For `EXCEPT DISTINCT`, row R appears once in the output if m > 0 and
+ n = 0.
+- If there are more than two input queries, the above operations generalize
+ and the output is the same as if the inputs were combined incrementally from
+ left to right.
+
+The following rules apply:
+
+- For set operations other than `UNION ALL`, all column types must support
+ equality comparison.
+- The input queries on each side of the operator must return the same number
+ of columns.
+- The operators pair the columns returned by each input query according to the
+ columns' positions in their respective `SELECT` lists. That is, the first
+ column in the first input query is paired with the first column in the
+ second input query.
+- The result set always uses the column names from the first input query.
+- The result set always uses the supertypes of input types in corresponding
+ columns, so paired columns must also have either the same data type or a
+ common supertype.
+- You must use parentheses to separate different set operations; for this
+ purpose, set operations such as `UNION ALL` and `UNION DISTINCT` are
+ different. If the statement only repeats the same set operation, parentheses
+ are not necessary.
+
+Examples:
+
+```
+query1 UNION ALL (query2 UNION DISTINCT query3)
+query1 UNION ALL query2 UNION ALL query3
+```
+
+Invalid:
+
+ query1 UNION ALL query2 UNION DISTINCT query3
+ query1 UNION ALL query2 INTERSECT ALL query3; // INVALID.
+
+### UNION
+
+The `UNION` operator combines the result sets of two or more input queries by
+pairing columns from the result set of each query and vertically concatenating
+them.
+
+### INTERSECT
+
+The `INTERSECT` operator returns rows that are found in the result sets of both
+the left and right input queries. Unlike `EXCEPT`, the positioning of the input
+queries (to the left vs. right of the `INTERSECT` operator) does not matter.
+
+### EXCEPT
+
+The `EXCEPT` operator returns rows from the left input query that are not
+present in the right input query.
+
+## LIMIT clause and OFFSET clause
+
+### Syntax {#syntax_7}
+
+```
+LIMIT count [ OFFSET skip_rows ]
+```
+
+`LIMIT` specifies a non-negative `count` of type INTEGER, and no more than `count`
+rows will be returned. `LIMIT` `0` returns 0 rows. If there is a set operation,
+`LIMIT` is applied after the set operation is evaluated.
+
+`OFFSET` specifies a non-negative `skip_rows` of type INTEGER, and only rows from
+that offset in the table will be considered.
+
+These clauses accept only literal or parameter values.
+
+The rows that are returned by `LIMIT` and `OFFSET` is unspecified.
+
+## WITH clause
+
+The `WITH` clause contains one or more named subqueries which execute every time
+a subsequent `SELECT` statement references them. Any clause or subquery can
+reference subqueries you define in the `WITH` clause. This includes any `SELECT`
+statements on either side of a set operator, such as `UNION`.
+
+Example:
+
+```
+WITH subQ1 AS (SELECT SchoolID FROM Roster),
+ subQ2 AS (SELECT OpponentID FROM PlayerStats)
+SELECT * FROM subQ1
+UNION ALL
+SELECT * FROM subQ2;
+```
+
+## Aliases {#aliases_2}
+
+An alias is a temporary name given to a table, column, or expression present in
+a query. You can introduce explicit aliases in the `SELECT` list or `FROM`
+clause, or Beam will infer an implicit alias for some expressions.
+Expressions with neither an explicit nor implicit alias are anonymous and the
+query cannot reference them by name.
+
+### Explicit alias syntax
+
+You can introduce explicit aliases in either the `FROM` clause or the `SELECT`
+list.
+
+In a `FROM` clause, you can introduce explicit aliases for any item, including
+tables, arrays, subqueries, and `UNNEST` clauses, using `[AS] alias`. The `AS`
+keyword is optional.
+
+Example:
+
+```
+SELECT s.FirstName, s2.SongName
+FROM Singers AS s JOIN Songs AS s2 ON s.SingerID = s2.SingerID;
+```
+
+You can introduce explicit aliases for any expression in the `SELECT` list using
+`[AS] alias`. The `AS` keyword is optional.
+
+Example:
+
+```
+SELECT s.FirstName AS name, LOWER(s.FirstName) AS lname
+FROM Singers s;
+```
+
+### Explicit alias visibility
+
+After you introduce an explicit alias in a query, there are restrictions on
+where else in the query you can reference that alias. These restrictions on
+alias visibility are the result of Beam's name scoping rules.
+
+#### FROM clause aliases
+
+Beam processes aliases in a `FROM` clause from left to right, and aliases
+are visible only to subsequent `JOIN` clauses.
+
+### Ambiguous aliases
+
+Beam provides an error if a name is ambiguous, meaning it can resolve to
+more than one unique object.
+
+Examples:
+
+This query contains column names that conflict between tables, since both
+`Singers` and `Songs` have a column named `SingerID`:
+
+```
+SELECT SingerID
+FROM Singers, Songs;
+```
+
+### Implicit aliases
+
+In the `SELECT` list, if there is an expression that does not have an explicit
+alias, Beam assigns an implicit alias according to the following rules.
+There can be multiple columns with the same alias in the `SELECT` list.
+
+- For identifiers, the alias is the identifier. For example, `SELECT abc`
+ implies `AS abc`.
+- For path expressions, the alias is the last identifier in the path. For
+ example, `SELECT abc.def.ghi` implies `AS ghi`.
+- For field access using the "dot" member field access operator, the alias is
+ the field name. For example, `SELECT (struct_function()).fname` implies `AS
+ fname`.
+
+In all other cases, there is no implicit alias, so the column is anonymous and
+cannot be referenced by name. The data from that column will still be returned
+and the displayed query results may have a generated label for that column, but
+the label cannot be used like an alias.
+
+In a `FROM` clause, `from_item`s are not required to have an alias. The
+following rules apply:
+
+If there is an expression that does not have an explicit alias, Beam assigns
+an implicit alias in these cases:
+
+- For identifiers, the alias is the identifier. For example, `FROM abc`
+ implies `AS abc`.
+- For path expressions, the alias is the last identifier in the path. For
+ example, `FROM abc.def.ghi` implies `AS ghi`
+
+Table subqueries do not have implicit aliases.
+
+`FROM UNNEST(x)` does not have an implicit alias.
+
+> Portions of this page are modifications based on
+> [work](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax)
+> created and
+> [shared by Google](https://developers.google.com/terms/site-policies)
+> and used according to terms described in the [Creative Commons 3.0
+> Attribution License](http://creativecommons.org/licenses/by/3.0/).
diff --git a/website/src/documentation/dsls/sql/calcite/scalar-functions.md b/website/src/documentation/dsls/sql/calcite/scalar-functions.md
new file mode 100644
index 0000000..616e9c2
--- /dev/null
+++ b/website/src/documentation/dsls/sql/calcite/scalar-functions.md
@@ -0,0 +1,133 @@
+---
+layout: section
+title: "Beam Calcite SQL scalar functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/calcite/scalar-functions/
+redirect_from: /documentation/dsls/sql/scalar-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam Calcite SQL scalar functions
+
+This page documents the Apache Calcite functions supported by Beam Calcite SQL.
+
+## Comparison functions and operators
+
+| Operator syntax | Description |
+| ---- | ---- |
+| value1 = value2 | Equals |
+| value1 <> value2 | Not equal |
+| value1 > value2 | Greater than |
+| value1 >= value2 | Greater than or equal |
+| value1 < value2 | Less than |
+| value1 <= value2 | Less than or equal |
+| value IS NULL | Whether value is null |
+| value IS NOT NULL | Whether value is not null |
+{:.table}
+
+## Logical functions and operators
+
+| Operator syntax | Description |
+| ---- | ---- |
+| boolean1 OR boolean2 | Whether boolean1 is TRUE or boolean2 is TRUE |
+| boolean1 AND boolean2 | Whether boolean1 and boolean2 are both TRUE |
+| NOT boolean | Whether boolean is not TRUE; returns UNKNOWN if boolean is UNKNOWN |
+{:.table}
+
+## Arithmetic expressions
+
+| Operator syntax | Description|
+| ---- | ---- |
+| numeric1 + numeric2 | Returns numeric1 plus numeric2|
+| numeric1 - numeric2 | Returns numeric1 minus numeric2|
+| numeric1 * numeric2 | Returns numeric1 multiplied by numeric2|
+| numeric1 / numeric2 | Returns numeric1 divided by numeric2|
+| MOD(numeric, numeric) | Returns the remainder (modulus) of numeric1 divided by numeric2. The result is negative only if numeric1 is negative|
+{:.table}
+
+## Math functions
+
+| Operator syntax | Description |
+| ---- | ---- |
+| ABS(numeric) | Returns the absolute value of numeric |
+| SQRT(numeric) | Returns the square root of numeric |
+| LN(numeric) | Returns the natural logarithm (base e) of numeric |
+| LOG10(numeric) | Returns the base 10 logarithm of numeric |
+| EXP(numeric) | Returns e raised to the power of numeric |
+| ACOS(numeric) | Returns the arc cosine of numeric |
+| ASIN(numeric) | Returns the arc sine of numeric |
+| ATAN(numeric) | Returns the arc tangent of numeric |
+| COT(numeric) | Returns the cotangent of numeric |
+| DEGREES(numeric) | Converts numeric from radians to degrees |
+| RADIANS(numeric) | Converts numeric from degrees to radians |
+| SIGN(numeric) | Returns the signum of numeric |
+| SIN(numeric) | Returns the sine of numeric |
+| TAN(numeric) | Returns the tangent of numeric |
+| ROUND(numeric1, numeric2) | Rounds numeric1 to numeric2 places right to the decimal point |
+{:.table}
+
+## Date functions
+
+| Operator syntax | Description |
+| ---- | ---- |
+| LOCALTIME | Returns the current date and time in the session time zone in a value of datatype TIME |
+| LOCALTIME(precision) | Returns the current date and time in the session time zone in a value of datatype TIME, with precision digits of precision |
+| LOCALTIMESTAMP | Returns the current date and time in the session time zone in a value of datatype TIMESTAMP |
+| LOCALTIMESTAMP(precision) | Returns the current date and time in the session time zone in a value of datatype TIMESTAMP, with precision digits of precision |
+| CURRENT_TIME | Returns the current time in the session time zone, in a value of datatype TIMESTAMP WITH TIME ZONE |
+| CURRENT_DATE | Returns the current date in the session time zone, in a value of datatype DATE |
+| CURRENT_TIMESTAMP | Returns the current date and time in the session time zone, in a value of datatype TIMESTAMP WITH TIME ZONE |
+| EXTRACT(timeUnit FROM datetime) | Extracts and returns the value of a specified datetime field from a datetime value expression |
+| FLOOR(datetime TO timeUnit) | Rounds datetime down to timeUnit |
+| CEIL(datetime TO timeUnit) | Rounds datetime up to timeUnit |
+| YEAR(date) | Equivalent to EXTRACT(YEAR FROM date). Returns an integer. |
+| QUARTER(date) | Equivalent to EXTRACT(QUARTER FROM date). Returns an integer between 1 and 4. |
+| MONTH(date) | Equivalent to EXTRACT(MONTH FROM date). Returns an integer between 1 and 12. |
+| WEEK(date) | Equivalent to EXTRACT(WEEK FROM date). Returns an integer between 1 and 53. |
+| DAYOFYEAR(date) | Equivalent to EXTRACT(DOY FROM date). Returns an integer between 1 and 366. |
+| DAYOFMONTH(date) | Equivalent to EXTRACT(DAY FROM date). Returns an integer between 1 and 31. |
+| DAYOFWEEK(date) | Equivalent to EXTRACT(DOW FROM date). Returns an integer between 1 and 7. |
+| HOUR(date) | Equivalent to EXTRACT(HOUR FROM date). Returns an integer between 0 and 23. |
+| MINUTE(date) | Equivalent to EXTRACT(MINUTE FROM date). Returns an integer between 0 and 59. |
+| SECOND(date) | Equivalent to EXTRACT(SECOND FROM date). Returns an integer between 0 and 59. |
+{:.table}
+
+## String functions
+
+| Operator syntax | Description |
+| ---- | ---- |
+| string \|\| string | Concatenates two character strings |
+| CHAR_LENGTH(string) | Returns the number of characters in a character string |
+| CHARACTER_LENGTH(string) | As CHAR_LENGTH(string) |
+| UPPER(string) | Returns a character string converted to upper case |
+| LOWER(string) | Returns a character string converted to lower case |
+| POSITION(string1 IN string2) | Returns the position of the first occurrence of string1 in string2 |
+| POSITION(string1 IN string2 FROM integer) | Returns the position of the first occurrence of string1 in string2 starting at a given point (not standard SQL) |
+| TRIM( { BOTH \| LEADING \| TRAILING } string1 FROM string2) | Removes the longest string containing only the characters in string1 from the start/end/both ends of string1 |
+| OVERLAY(string1 PLACING string2 FROM integer [ FOR integer2 ]) | Replaces a substring of string1 with string2 |
+| SUBSTRING(string FROM integer) | Returns a substring of a character string starting at a given point |
+| SUBSTRING(string FROM integer FOR integer) | Returns a substring of a character string starting at a given point with a given length |
+| INITCAP(string) | Returns string with the first letter of each word converter to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters. |
+{:.table}
+
+## Conditional functions
+
+| Operator syntax | Description |
+| ---- | ---- |
+| CASE value <br>WHEN value1 [, value11 ]* THEN result1 <br>[ WHEN valueN [, valueN1 ]* THEN resultN ]* <br>[ ELSE resultZ ] <br>END | Simple case |
+| CASE <br>WHEN condition1 THEN result1 <br>[ WHEN conditionN THEN resultN ]* <br>[ ELSE resultZ ] <br>END | Searched case |
+| NULLIF(value, value) | Returns NULL if the values are the same. For example, NULLIF(5, 5) returns NULL; NULLIF(5, 0) returns 5. |
+| COALESCE(value, value [, value ]*) | Provides a value if the first value is null. For example, COALESCE(NULL, 5) returns 5. |
+{:.table}
diff --git a/website/src/documentation/dsls/sql/create-external-table.md b/website/src/documentation/dsls/sql/create-external-table.md
deleted file mode 100644
index f332880..0000000
--- a/website/src/documentation/dsls/sql/create-external-table.md
+++ /dev/null
@@ -1,364 +0,0 @@
----
-layout: section
-title: "Beam SQL: CREATE EXTERNAL TABLE Statement"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/create-external-table/
-redirect_from:
- - /documentation/dsls/sql/statements/create-table/
- - /documentation/dsls/sql/create-table/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# CREATE EXTERNAL TABLE
-
-Beam SQL's `CREATE EXTERNAL TABLE` statement registers a virtual table that maps to an
-[external storage system]({{ site.baseurl }}/documentation/io/built-in/).
-For some storage systems, `CREATE EXTERNAL TABLE` does not create a physical table until
-a write occurs. After the physical table exists, you can access the table with
-the `SELECT`, `JOIN`, and `INSERT INTO` statements.
-
-The `CREATE EXTERNAL TABLE` statement includes a schema and extended clauses.
-
-## Syntax
-
-```
-CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE type
-[LOCATION location]
-[TBLPROPERTIES tblProperties]
-
-simpleType: TINYINT | SMALLINT | INTEGER | BIGINT | FLOAT | DOUBLE | DECIMAL | BOOLEAN | DATE | TIME | TIMESTAMP | CHAR | VARCHAR
-
-fieldType: simpleType | MAP<simpleType, fieldType> | ARRAY<fieldType> | ROW<tableElement [, tableElement ]*>
-
-tableElement: columnName fieldType [ NOT NULL ]
-```
-
-* `IF NOT EXISTS`: Optional. If the table is already registered, Beam SQL
- ignores the statement instead of returning an error.
-* `tableName`: The case sensitive name of the table to create and register,
- specified as an
- [Identifier]({{ site.baseurl }}/documentation/dsls/sql/lexical/#identifiers).
- The table name does not need to match the name in the underlying data
- storage system.
-* `tableElement`: `columnName` `fieldType` `[ NOT NULL ]`
- * `columnName`: The case sensitive name of the column, specified as a
- backtick_quoted_expression.
- * `fieldType`: The field's type, specified as one of the following types:
- * `simpleType`: `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`,
- `DOUBLE`, `DECIMAL`, `BOOLEAN`, `DATE`, `TIME`, `TIMESTAMP`, `CHAR`,
- `VARCHAR`
- * `MAP<simpleType, fieldType>`
- * `ARRAY<fieldType>`
- * `ROW<tableElement [, tableElement ]*>`
- * `NOT NULL`: Optional. Indicates that the column is not nullable.
-* `type`: The I/O transform that backs the virtual table, specified as an
- [Identifier]({{ site.baseurl }}/documentation/dsls/sql/lexical/#identifiers)
- with one of the following values:
- * `bigquery`
- * `pubsub`
- * `kafka`
- * `text`
-* `location`: The I/O specific location of the underlying table, specified as
- a [String
- Literal]({{ site.baseurl }}/documentation/dsls/sql/lexical/#string-literals).
- See the I/O specific sections for `location` format requirements.
-* `tblProperties`: The I/O specific quoted key value JSON object with extra
- configuration, specified as a [String
- Literal]({{ site.baseurl }}/documentation/dsls/sql/lexical/#string-literals).
- See the I/O specific sections for `tblProperties` format requirements.
-
-## BigQuery
-
-### Syntax
-
-```
-CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE bigquery
-LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
-```
-
-* `LOCATION:`Location of the table in the BigQuery CLI format.
- * `PROJECT_ID`: ID of the Google Cloud Project
- * `DATASET`: BigQuery Dataset ID
- * `TABLE`: BigQuery Table ID within the Dataset
-
-### Read Mode
-
-Beam SQL supports reading columns with simple types (`simpleType`) and arrays of simple
-types (`ARRAY<simpleType>`).
-
-### Write Mode
-
-if the table does not exist, Beam creates the table specified in location when
-the first record is written. If the table does exist, the specified columns must
-match the existing table.
-
-### Schema
-
-Schema-related errors will cause the pipeline to crash. The Map type is not
-supported. Beam SQL types map to [BigQuery Standard SQL
-types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types)
-as follows:
-
-<table>
- <tr>
- <td>Beam SQL Type
- </td>
- <td>BigQuery Standard SQL Type
- </td>
- </tr>
- <tr>
- <td>TINYINT, SMALLINT, INTEGER, BIGINT
- </td>
- <td>INT64
- </td>
- </tr>
- <tr>
- <td>FLOAT, DOUBLE, DECIMAL
- </td>
- <td>FLOAT64
- </td>
- </tr>
- <tr>
- <td>BOOLEAN
- </td>
- <td>BOOL
- </td>
- </tr>
- <tr>
- <td>DATE
- </td>
- <td>DATE
- </td>
- </tr>
- <tr>
- <td>TIME
- </td>
- <td>TIME
- </td>
- </tr>
- <tr>
- <td>TIMESTAMP
- </td>
- <td>TIMESTAMP
- </td>
- </tr>
- <tr>
- <td>CHAR, VARCHAR
- </td>
- <td>STRING
- </td>
- </tr>
- <tr>
- <td>MAP
- </td>
- <td>(not supported)
- </td>
- </tr>
- <tr>
- <td>ARRAY
- </td>
- <td>ARRAY
- </td>
- </tr>
- <tr>
- <td>ROW
- </td>
- <td>STRUCT
- </td>
- </tr>
-</table>
-
-### Example
-
-```
-CREATE EXTERNAL TABLE users (id INTEGER, username VARCHAR)
-TYPE bigquery
-LOCATION 'testing-integration:apache.users'
-```
-
-## Pub/Sub
-
-### Syntax
-
-```
-CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName
- (
- event_timestamp TIMESTAMP,
- attributes MAP<VARCHAR, VARCHAR>,
- payload ROW<tableElement [, tableElement ]*>
- )
-TYPE pubsub
-LOCATION 'projects/[PROJECT]/topics/[TOPIC]'
-TBLPROPERTIES '{"timestampAttributeKey": "key", "deadLetterQueue": "projects/[PROJECT]/topics/[TOPIC]"}'
-```
-
-* `event_timestamp`: The event timestamp associated with the Pub/Sub message
- by PubsubIO. It can be one of the following:
- * Message publish time, which is provided by Pub/Sub. This is the default
- value if no extra configuration is provided.
- * A timestamp specified in one of the user-provided message attributes.
- The attribute key is configured by the `timestampAttributeKey` field of
- the `tblProperties` blob. The value of the attribute should conform to
- the [requirements of
- PubsubIO](https://beam.apache.org/releases/javadoc/2.4.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String-),
- which is either millis since Unix epoch or [RFC 339
- ](https://www.ietf.org/rfc/rfc3339.txt)date string.
-* `attributes`: The user-provided attributes map from the Pub/Sub message;
-* `payload`: The schema of the JSON payload of the Pub/Sub message. No other
- payload formats are currently supported by Beam SQL. If a record can't be
- unmarshalled, the record is written to the topic specified in the
- `deadLeaderQueue` field of the `tblProperties` blob. If no dead-letter queue
- is specified in this case, an exception is thrown and the pipeline will
- crash.
-* `LOCATION`:
- * `PROJECT`: ID of the Google Cloud Project
- * `TOPIC`: The Pub/Sub topic name. A subscription will be created
- automatically, but the subscription is not cleaned up automatically.
- Specifying an existing subscription is not supported.
-* `TBLPROPERTIES`:
- * `timestampAttributeKey`: Optional. The key which contains the event
- timestamp associated with the Pub/Sub message. If not specified, the
- message publish timestamp is used as an event timestamp for
- windowing/watermarking.
- * `deadLetterQueue`: The topic into which messages are written if the
- payload was not parsed. If not specified, an exception is thrown for
- parsing failures.
-
-### Read Mode
-
-PubsubIO is currently limited to read access only.
-
-### Write Mode
-
-Not supported. PubSubIO is currently limited to read access only in Beam SQL.
-
-### Schema
-
-Pub/Sub messages have metadata associated with them, and you can reference this
-metadata in your queries. For each message, Pub/Sub exposes its publish time and
-a map of user-provided attributes in addition to the payload (unstructured in
-the general case). This information must be preserved and accessible from the
-SQL statements. Currently, this means that PubsubIO tables require you to
-declare a special set of columns, as shown below.
-
-### Supported Payload
-
-* JSON Objects
- * Beam only supports querying messages with payload containing JSON
- objects. Beam attempts to parse JSON to match the schema of the
- `payload` field.
-
-### Example
-
-```
-CREATE EXTERNAL TABLE locations (event_timestamp TIMESTAMP, attributes MAP<VARCHAR, VARCHAR>, payload ROW<id INTEGER, location VARCHAR>)
-TYPE pubsub
-LOCATION 'projects/testing-integration/topics/user-location'
-```
-
-## Kafka
-
-KafkaIO is experimental in Beam SQL.
-
-### Syntax
-
-```
-CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE kafka
-LOCATION 'kafka://localhost:2181/brokers'
-TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", "topic2"]}'
-```
-
-* `LOCATION`: The Kafka topic URL.
-* `TBLPROPERTIES`:
- * `bootstrap.servers`: Optional. Allows you to specify the bootstrap
- server.
- * `topics`: Optional. Allows you to specify specific topics.
-
-### Read Mode
-
-Read Mode supports reading from a topic.
-
-### Write Mode
-
-Write Mode supports writing to a topic.
-
-### Supported Payload
-
-* CSV
- * Beam parses the messages, attempting to parse fields according to the
- types specified in the schema.
-
-### Schema
-
-Only simple types are supported.
-
-## Text
-
-TextIO is experimental in Beam SQL. Read Mode and Write Mode do not currently
-access the same underlying data.
-
-### Syntax
-
-```
-CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
-TYPE text
-LOCATION '/home/admin/orders'
-TBLPROPERTIES '{"format: "Excel"}'
-```
-
-* `LOCATION`: The path to the file for Read Mode. The prefix for Write Mode.
-* `TBLPROPERTIES`:
- * `format`: Optional. Allows you to specify the CSV Format, which controls
- the field delimeter, quote character, record separator, and other properties.
- See the following table:
-
-
-| Value for `format` | Field delimiter | Quote | Record separator | Ignore empty lines? | Allow missing column names? |
-|--------------------|-----------------|-------|------------------|---------------------|-----------------------------|
-| `default` | `,` | `"` | `\r\n` | Yes | No |
-| `rfc4180` | `,` | `"` | `\r\n` | No | No |
-| `excel` | `,` | `"` | `\r\n` | No | Yes |
-| `tdf` | `\t` | `"` | `\r\n` | Yes | No |
-| `mysql` | `\t` | none | `\n` | No | No |
-{:.table-bordered}
-
-### Read Mode
-
-Read Mode supports reading from a file.
-
-### Write Mode
-
-Write Mode supports writing to a set of files. TextIO creates file on writes.
-
-### Supported Payload
-
-* CSV
- * Beam parses the messages, attempting to parse fields according to the
- types specified in the schema using org.apache.commons.csv.
-
-### Schema
-
-Only simple types are supported.
-
-### Example
-
-```
-CREATE EXTERNAL TABLE orders (id INTEGER, price INTEGER)
-TYPE text
-LOCATION '/home/admin/orders'
-```
diff --git a/website/src/documentation/dsls/sql/data-types.md b/website/src/documentation/dsls/sql/data-types.md
deleted file mode 100644
index 67ad56d..0000000
--- a/website/src/documentation/dsls/sql/data-types.md
+++ /dev/null
@@ -1,46 +0,0 @@
----
-layout: section
-title: "Beam SQL: Data Types"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/data-types/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL Data Types
-
-Beam SQL supports standard SQL scalar data types as well as extensions
-including arrays, maps, and nested rows.
-
-In Beam Java, these types are mapped to Java types large enough to hold the
-full range of values.
-
-| SQL Type | Description | Java class |
-| --------- | ------------ | ---------- |
-| TINYINT | 1 byte signed integer in range -128 to 127 | java.lang.Byte |
-| SMALLINT | 2 byte signed integer in range -32768 to 32767 | java.lang.Short |
-| INTEGER | 4 byte signed integer in range -2147483648 to 2147483647 | java.lang.Integer |
-| BIGINT | 8 byte signed integer in range -9223372036854775808 to 9223372036854775807 | java.lang.Long |
-| FLOAT | 4 byte floating point | java.lang.Float |
-| DOUBLE | 8 byte floating point | java.lang.Double |
-| DECIMAL | Arbitrary precision decimal value | java.math.BigDecimal |
-| VARCHAR | Arbitrary length string | java.lang.String |
-| TIMESTAMP | Millisecond precision timestamp | org.joda.ReadableInstant |
-| ARRAY<type> | Ordered list of values | java.util.List |
-| MAP<type, type> | Finite unordered map | java.util.Map |
-| ROW<fields> | Nested row | org.apache.beam.sdk.values.Row |
-{:.table}
-
-See also the [documentation for Calcite SQL's data
-types](http://calcite.apache.org/docs/reference.html#data-types)
diff --git a/website/src/documentation/dsls/sql/extensions/create-external-table.md b/website/src/documentation/dsls/sql/extensions/create-external-table.md
new file mode 100644
index 0000000..81d7dae
--- /dev/null
+++ b/website/src/documentation/dsls/sql/extensions/create-external-table.md
@@ -0,0 +1,365 @@
+---
+layout: section
+title: "Beam SQL extension: CREATE EXTERNAL TABLE Statement"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/extensions/create-external-table/
+redirect_from:
+ - /documentation/dsls/sql/create-external-table/
+ - /documentation/dsls/sql/statements/create-table/
+ - /documentation/dsls/sql/create-table/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam SQL extensions: CREATE EXTERNAL TABLE
+
+Beam SQL's `CREATE EXTERNAL TABLE` statement registers a virtual table that maps to an
+[external storage system]({{ site.baseurl }}/documentation/io/built-in/).
+For some storage systems, `CREATE EXTERNAL TABLE` does not create a physical table until
+a write occurs. After the physical table exists, you can access the table with
+the `SELECT`, `JOIN`, and `INSERT INTO` statements.
+
+The `CREATE EXTERNAL TABLE` statement includes a schema and extended clauses.
+
+## Syntax
+
+```
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+TYPE type
+[LOCATION location]
+[TBLPROPERTIES tblProperties]
+
+simpleType: TINYINT | SMALLINT | INTEGER | BIGINT | FLOAT | DOUBLE | DECIMAL | BOOLEAN | DATE | TIME | TIMESTAMP | CHAR | VARCHAR
+
+fieldType: simpleType | MAP<simpleType, fieldType> | ARRAY<fieldType> | ROW<tableElement [, tableElement ]*>
+
+tableElement: columnName fieldType [ NOT NULL ]
+```
+
+* `IF NOT EXISTS`: Optional. If the table is already registered, Beam SQL
+ ignores the statement instead of returning an error.
+* `tableName`: The case sensitive name of the table to create and register,
+ specified as an
+ [Identifier]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical#identifiers).
+ The table name does not need to match the name in the underlying data
+ storage system.
+* `tableElement`: `columnName` `fieldType` `[ NOT NULL ]`
+ * `columnName`: The case sensitive name of the column, specified as a
+ backtick_quoted_expression.
+ * `fieldType`: The field's type, specified as one of the following types:
+ * `simpleType`: `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `FLOAT`,
+ `DOUBLE`, `DECIMAL`, `BOOLEAN`, `DATE`, `TIME`, `TIMESTAMP`, `CHAR`,
+ `VARCHAR`
+ * `MAP<simpleType, fieldType>`
+ * `ARRAY<fieldType>`
+ * `ROW<tableElement [, tableElement ]*>`
+ * `NOT NULL`: Optional. Indicates that the column is not nullable.
+* `type`: The I/O transform that backs the virtual table, specified as an
+ [Identifier]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical/#identifiers)
+ with one of the following values:
+ * `bigquery`
+ * `pubsub`
+ * `kafka`
+ * `text`
+* `location`: The I/O specific location of the underlying table, specified as
+ a [String
+ Literal]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical/#string-literals).
+ See the I/O specific sections for `location` format requirements.
+* `tblProperties`: The I/O specific quoted key value JSON object with extra
+ configuration, specified as a [String
+ Literal]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical/#string-literals).
+ See the I/O specific sections for `tblProperties` format requirements.
+
+## BigQuery
+
+### Syntax
+
+```
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+TYPE bigquery
+LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
+```
+
+* `LOCATION:`Location of the table in the BigQuery CLI format.
+ * `PROJECT_ID`: ID of the Google Cloud Project
+ * `DATASET`: BigQuery Dataset ID
+ * `TABLE`: BigQuery Table ID within the Dataset
+
+### Read Mode
+
+Beam SQL supports reading columns with simple types (`simpleType`) and arrays of simple
+types (`ARRAY<simpleType>`).
+
+### Write Mode
+
+if the table does not exist, Beam creates the table specified in location when
+the first record is written. If the table does exist, the specified columns must
+match the existing table.
+
+### Schema
+
+Schema-related errors will cause the pipeline to crash. The Map type is not
+supported. Beam SQL types map to [BigQuery Standard SQL
+types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types)
+as follows:
+
+<table>
+ <tr>
+ <td>Beam SQL Type
+ </td>
+ <td>BigQuery Standard SQL Type
+ </td>
+ </tr>
+ <tr>
+ <td>TINYINT, SMALLINT, INTEGER, BIGINT
+ </td>
+ <td>INT64
+ </td>
+ </tr>
+ <tr>
+ <td>FLOAT, DOUBLE, DECIMAL
+ </td>
+ <td>FLOAT64
+ </td>
+ </tr>
+ <tr>
+ <td>BOOLEAN
+ </td>
+ <td>BOOL
+ </td>
+ </tr>
+ <tr>
+ <td>DATE
+ </td>
+ <td>DATE
+ </td>
+ </tr>
+ <tr>
+ <td>TIME
+ </td>
+ <td>TIME
+ </td>
+ </tr>
+ <tr>
+ <td>TIMESTAMP
+ </td>
+ <td>TIMESTAMP
+ </td>
+ </tr>
+ <tr>
+ <td>CHAR, VARCHAR
+ </td>
+ <td>STRING
+ </td>
+ </tr>
+ <tr>
+ <td>MAP
+ </td>
+ <td>(not supported)
+ </td>
+ </tr>
+ <tr>
+ <td>ARRAY
+ </td>
+ <td>ARRAY
+ </td>
+ </tr>
+ <tr>
+ <td>ROW
+ </td>
+ <td>STRUCT
+ </td>
+ </tr>
+</table>
+
+### Example
+
+```
+CREATE EXTERNAL TABLE users (id INTEGER, username VARCHAR)
+TYPE bigquery
+LOCATION 'testing-integration:apache.users'
+```
+
+## Pub/Sub
+
+### Syntax
+
+```
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName
+ (
+ event_timestamp TIMESTAMP,
+ attributes MAP<VARCHAR, VARCHAR>,
+ payload ROW<tableElement [, tableElement ]*>
+ )
+TYPE pubsub
+LOCATION 'projects/[PROJECT]/topics/[TOPIC]'
+TBLPROPERTIES '{"timestampAttributeKey": "key", "deadLetterQueue": "projects/[PROJECT]/topics/[TOPIC]"}'
+```
+
+* `event_timestamp`: The event timestamp associated with the Pub/Sub message
+ by PubsubIO. It can be one of the following:
+ * Message publish time, which is provided by Pub/Sub. This is the default
+ value if no extra configuration is provided.
+ * A timestamp specified in one of the user-provided message attributes.
+ The attribute key is configured by the `timestampAttributeKey` field of
+ the `tblProperties` blob. The value of the attribute should conform to
+ the [requirements of
+ PubsubIO](https://beam.apache.org/releases/javadoc/2.4.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.Read.html#withTimestampAttribute-java.lang.String-),
+ which is either millis since Unix epoch or [RFC 339
+ ](https://www.ietf.org/rfc/rfc3339.txt)date string.
+* `attributes`: The user-provided attributes map from the Pub/Sub message;
+* `payload`: The schema of the JSON payload of the Pub/Sub message. No other
+ payload formats are currently supported by Beam SQL. If a record can't be
+ unmarshalled, the record is written to the topic specified in the
+ `deadLeaderQueue` field of the `tblProperties` blob. If no dead-letter queue
+ is specified in this case, an exception is thrown and the pipeline will
+ crash.
+* `LOCATION`:
+ * `PROJECT`: ID of the Google Cloud Project
+ * `TOPIC`: The Pub/Sub topic name. A subscription will be created
+ automatically, but the subscription is not cleaned up automatically.
+ Specifying an existing subscription is not supported.
+* `TBLPROPERTIES`:
+ * `timestampAttributeKey`: Optional. The key which contains the event
+ timestamp associated with the Pub/Sub message. If not specified, the
+ message publish timestamp is used as an event timestamp for
+ windowing/watermarking.
+ * `deadLetterQueue`: The topic into which messages are written if the
+ payload was not parsed. If not specified, an exception is thrown for
+ parsing failures.
+
+### Read Mode
+
+PubsubIO is currently limited to read access only.
+
+### Write Mode
+
+Not supported. PubSubIO is currently limited to read access only in Beam SQL.
+
+### Schema
+
+Pub/Sub messages have metadata associated with them, and you can reference this
+metadata in your queries. For each message, Pub/Sub exposes its publish time and
+a map of user-provided attributes in addition to the payload (unstructured in
+the general case). This information must be preserved and accessible from the
+SQL statements. Currently, this means that PubsubIO tables require you to
+declare a special set of columns, as shown below.
+
+### Supported Payload
+
+* JSON Objects
+ * Beam only supports querying messages with payload containing JSON
+ objects. Beam attempts to parse JSON to match the schema of the
+ `payload` field.
+
+### Example
+
+```
+CREATE EXTERNAL TABLE locations (event_timestamp TIMESTAMP, attributes MAP<VARCHAR, VARCHAR>, payload ROW<id INTEGER, location VARCHAR>)
+TYPE pubsub
+LOCATION 'projects/testing-integration/topics/user-location'
+```
+
+## Kafka
+
+KafkaIO is experimental in Beam SQL.
+
+### Syntax
+
+```
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+TYPE kafka
+LOCATION 'kafka://localhost:2181/brokers'
+TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", "topic2"]}'
+```
+
+* `LOCATION`: The Kafka topic URL.
+* `TBLPROPERTIES`:
+ * `bootstrap.servers`: Optional. Allows you to specify the bootstrap
+ server.
+ * `topics`: Optional. Allows you to specify specific topics.
+
+### Read Mode
+
+Read Mode supports reading from a topic.
+
+### Write Mode
+
+Write Mode supports writing to a topic.
+
+### Supported Payload
+
+* CSV
+ * Beam parses the messages, attempting to parse fields according to the
+ types specified in the schema.
+
+### Schema
+
+Only simple types are supported.
+
+## Text
+
+TextIO is experimental in Beam SQL. Read Mode and Write Mode do not currently
+access the same underlying data.
+
+### Syntax
+
+```
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+TYPE text
+LOCATION '/home/admin/orders'
+TBLPROPERTIES '{"format: "Excel"}'
+```
+
+* `LOCATION`: The path to the file for Read Mode. The prefix for Write Mode.
+* `TBLPROPERTIES`:
+ * `format`: Optional. Allows you to specify the CSV Format, which controls
+ the field delimeter, quote character, record separator, and other properties.
+ See the following table:
+
+
+| Value for `format` | Field delimiter | Quote | Record separator | Ignore empty lines? | Allow missing column names? |
+|--------------------|-----------------|-------|------------------|---------------------|-----------------------------|
+| `default` | `,` | `"` | `\r\n` | Yes | No |
+| `rfc4180` | `,` | `"` | `\r\n` | No | No |
+| `excel` | `,` | `"` | `\r\n` | No | Yes |
+| `tdf` | `\t` | `"` | `\r\n` | Yes | No |
+| `mysql` | `\t` | none | `\n` | No | No |
+{:.table-bordered}
+
+### Read Mode
+
+Read Mode supports reading from a file.
+
+### Write Mode
+
+Write Mode supports writing to a set of files. TextIO creates file on writes.
+
+### Supported Payload
+
+* CSV
+ * Beam parses the messages, attempting to parse fields according to the
+ types specified in the schema using org.apache.commons.csv.
+
+### Schema
+
+Only simple types are supported.
+
+### Example
+
+```
+CREATE EXTERNAL TABLE orders (id INTEGER, price INTEGER)
+TYPE text
+LOCATION '/home/admin/orders'
+```
diff --git a/website/src/documentation/dsls/sql/extensions/joins.md b/website/src/documentation/dsls/sql/extensions/joins.md
new file mode 100644
index 0000000..c5d0b1f
--- /dev/null
+++ b/website/src/documentation/dsls/sql/extensions/joins.md
@@ -0,0 +1,73 @@
+---
+layout: section
+title: "Beam SQL extensions: Joins"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/extensions/joins/
+redirect-from: /documentation/dsls/sql/joins/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam SQL extensions: Joins
+
+Supported `JOIN` types in Beam SQL:
+* `INNER`, `LEFT OUTER`, `RIGHT OUTER`
+* Only equijoins (where join condition is an equality check) are supported
+
+Unsupported `JOIN` types in Beam SQL:
+* `CROSS JOIN` is not supported (full cartesian product with no `ON` clause)
+* `FULL OUTER JOIN` is not supported (combination of `LEFT OUTER` and `RIGHT OUTER` joins)
+
+The scenarios of join can be categorized into 3 cases:
+
+1. Bounded input `JOIN` bounded input
+2. Unbounded input `JOIN` unbounded input
+3. Unbounded input `JOIN` bounded input
+
+## Bounded JOIN Bounded {#join-bounded-bounded}
+
+Standard join implementation is used. All elements from one input are matched
+with all elements from another input. Due to the fact that both inputs are
+bounded, no windowing or triggering is involved.
+
+## Unbounded JOIN Unbounded {#join-unbounded-unbounded}
+
+Standard join implementation is used. All elements from one input are matched
+with all elements from another input.
+
+**Windowing and Triggering**
+
+The following properties must be satisfied when joining unbounded inputs:
+
+ - Inputs must have compatible windows, otherwise `IllegalArgumentException`
+ will be thrown.
+ - Triggers on each input should only fire once per window. Currently this
+ means that the only supported trigger in this case is `DefaultTrigger` with
+ zero allowed lateness. Using any other trigger will result in
+ `UnsupportedOperationException` thrown.
+
+This means that inputs are joined per-window. That is, when the trigger fires
+(only once), then join is performed on all elements in the current window in
+both inputs. This allows to reason about what kind of output is going to be
+produced.
+
+**Note:** similarly to `GroupByKeys` `JOIN` will update triggers using
+`Trigger.continuationTrigger()`. Other aspects of the inputs' windowing
+strategies remain unchanged.
+
+## Unbounded JOIN Bounded {#join-unbounded-bounded}
+
+For this type of `JOIN` bounded input is treated as a side-input by the
+implementation. This means that window/trigger is inherented from upstreams.
+
diff --git a/website/src/documentation/dsls/sql/extensions/set.md b/website/src/documentation/dsls/sql/extensions/set.md
new file mode 100644
index 0000000..f0f03ac
--- /dev/null
+++ b/website/src/documentation/dsls/sql/extensions/set.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Beam SQL extensions: SET and RESET Statement"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/extensions/set/
+redirect_from: /documentation/dsls/sql/set/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam SQL extensions: SET and RESET Pipeline Options
+
+Beam SQL's `SET` and `RESET` statements allow the user to [configure Pipeline
+Options]({{ site.baseurl }}/documentation/programming-guide/#configuring-pipeline-options)
+via the SQL shell. These are the same Pipeline Options passed to other Beam
+applications on the command line in the `--<option>=<value>` format.
+
+## Syntax
+
+```
+SET option = value
+```
+
+The SET command sets a Pipeline Option.
+
+* `option`: The case sensitive name of the Pipeline Option, specified as an
+ [Identifier]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical/#identifiers).
+* `value`: The case sensitive value of the Pipeline Option, specified as an
+ [Identifier]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical/#identifiers).
+ For flag options that have no value on the command line, use `true`.
+
+```
+RESET option
+```
+
+The RESET command resets a Pipeline Option to its default value.
+
+* `option`: The case sensitive name of the Pipeline Option, specified as an
+ [Identifier]({{ site.baseurl }}/documentation/dsls/sql/calcite/lexical#identifiers).
+
+## Common Options
+
+* ```SET project = `my_gcp_project` ```: Sets the default GCP project
+ to`my_gcp_project`.
+* `SET runner = DataflowRunner`: Sets the pipeline to run on Dataflow.
diff --git a/website/src/documentation/dsls/sql/extensions/user-defined-functions.md b/website/src/documentation/dsls/sql/extensions/user-defined-functions.md
new file mode 100644
index 0000000..e7fe561
--- /dev/null
+++ b/website/src/documentation/dsls/sql/extensions/user-defined-functions.md
@@ -0,0 +1,128 @@
+---
+layout: section
+title: "Beam SQL extensions: User-defined functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/extensions/user-defined-functions/
+redirect_from: /documentation/dsls/sql/user-defined-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam SQL extensions: User-defined functions
+
+If Beam SQL does not have a scalar function or aggregate function to meet your
+needs, they can be authored in Java and invoked in your SQL query. These
+are commonly called UDF (for scalar functions) and UDAFs (for aggregate functions).
+
+## Create and specify a User Defined Function (UDF)
+
+A UDF can be the following:
+- Any Java method that takes zero or more scalar fields and
+ returns one scalar value.
+- A `SerializableFunction`.
+
+Below is an example of UDF and how to use it in DSL:
+
+```java
+/**
+ * A example UDF for test.
+ */
+public static class CubicInteger implements BeamSqlUdf {
+ public static Integer eval(Integer input){
+ return input * input * input;
+ }
+}
+
+/**
+ * Another example UDF with {@link SerializableFunction}.
+ */
+public static class CubicIntegerFn implements SerializableFunction<Integer, Integer> {
+ @Override
+ public Integer apply(Integer input) {
+ return input * input * input;
+ }
+}
+
+// Define a SQL query which calls the above UDFs
+String sql =
+ "SELECT f_int, cubic1(f_int), cubic2(f_int)"
+ + "FROM PCOLLECTION "
+ + "WHERE f_int = 2";
+
+// Create and apply the PTransform representing the query.
+// Register the UDFs used in the query by calling '.registerUdf()' with
+// either a class which implements BeamSqlUdf or with
+// an instance of the SerializableFunction;
+PCollection<Row> result =
+ input.apply(
+ "udfExample",
+ SqlTransform
+ .query(sql)
+ .registerUdf("cubic1", CubicInteger.class)
+ .registerUdf("cubic2", new CubicIntegerFn())
+```
+
+## Create and specify a User Defined Aggregate Function (UDAF)
+
+Beam SQL can accept a `CombineFn` as UDAF. Registration is similar to the UDF
+example above:
+
+```java
+/**
+ * UDAF(CombineFn) for test, which returns the sum of square.
+ */
+public static class SquareSum extends CombineFn<Integer, Integer, Integer> {
+ @Override
+ public Integer createAccumulator() {
+ return 0;
+ }
+
+ @Override
+ public Integer addInput(Integer accumulator, Integer input) {
+ return accumulator + input * input;
+ }
+
+ @Override
+ public Integer mergeAccumulators(Iterable<Integer> accumulators) {
+ int v = 0;
+ Iterator<Integer> ite = accumulators.iterator();
+ while (ite.hasNext()) {
+ v += ite.next();
+ }
+ return v;
+ }
+
+ @Override
+ public Integer extractOutput(Integer accumulator) {
+ return accumulator;
+ }
+}
+
+// Define a SQL query which calls the above UDAF
+String sql =
+ "SELECT f_int1, squaresum(f_int2) "
+ + "FROM PCOLLECTION "
+ + "GROUP BY f_int2";
+
+// Create and apply the PTransform representing the query.
+// Register the UDAFs used in the query by calling '.registerUdaf()' by
+// providing it an instance of the CombineFn
+PCollection<Row> result =
+ input.apply(
+ "udafExample",
+ SqlTransform
+ .query(sql)
+ .registerUdaf("squaresum", new SquareSum()));
+```
+
diff --git a/website/src/documentation/dsls/sql/extensions/windowing-and-triggering.md b/website/src/documentation/dsls/sql/extensions/windowing-and-triggering.md
new file mode 100644
index 0000000..0c772aa
--- /dev/null
+++ b/website/src/documentation/dsls/sql/extensions/windowing-and-triggering.md
@@ -0,0 +1,67 @@
+---
+layout: section
+title: "Beam DSLs: SQL"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/extensions/windowing-and-triggering/
+redirect_from: /documentation/dsls/sql/windowing-and-triggering/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam SQL extensions: Windowing and triggering
+
+You can use Beam's windowing semantics in two ways:
+
+ - you can configure windowing on your input `PCollections` before passing them
+ to a `BeamSql` transform
+ - you can use windowing extensions in your windowing query, which will override
+ the windowing of your input `PCollections`
+
+Triggering can only be used by setting it on your input `PCollections`; there
+are no SQL extensions for specifying triggering.
+
+This section covers the use of SQL extensions to directly apply windowing.
+
+Beam SQL supports windowing functions specified in `GROUP BY` clause.
+`TIMESTAMP` field is required in this case. It is used as event timestamp for
+rows.
+
+Supported windowing functions:
+* `TUMBLE`, or fixed windows. Example of how define a fixed window with duration of 1 hour:
+```
+ SELECT f_int, COUNT(*)
+ FROM PCOLLECTION
+ GROUP BY
+ f_int,
+ TUMBLE(f_timestamp, INTERVAL '1' HOUR)
+```
+* `HOP`, or sliding windows. Example of how to define a sliding windows for every 30 minutes with 1 hour duration:
+```
+ SELECT f_int, COUNT(*)
+ FROM PCOLLECTION
+ GROUP BY
+ f_int,
+ HOP(f_timestamp, INTERVAL '30' MINUTE, INTERVAL '1' HOUR)
+```
+* `SESSION`, session windows. Example of how to define a session window with 5 minutes gap duration:
+```
+ SELECT f_int, COUNT(*)
+ FROM PCOLLECTION
+ GROUP BY
+ f_int,
+ SESSION(f_timestamp, INTERVAL '5' MINUTE)
+```
+
+**Note:** When no windowing function is specified in the query, then windowing strategy of the input `PCollections` is unchanged by the SQL query. If windowing function is specified in the query, then the windowing function of the `PCollection` is updated accordingly, but trigger stays unchanged.
+
diff --git a/website/src/documentation/dsls/sql/joins.md b/website/src/documentation/dsls/sql/joins.md
deleted file mode 100644
index ec68a24..0000000
--- a/website/src/documentation/dsls/sql/joins.md
+++ /dev/null
@@ -1,76 +0,0 @@
----
-layout: section
-title: "Beam SQL: Joins"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/joins/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL: Joins
-
-Supported `JOIN` types in Beam SQL:
-* `INNER`, `LEFT OUTER`, `RIGHT OUTER`
-* Only equijoins (where join condition is an equality check) are supported
-
-Unsupported `JOIN` types in Beam SQL:
-* `CROSS JOIN` is not supported (full cartesian product with no `ON` clause)
-* `FULL OUTER JOIN` is not supported (combination of `LEFT OUTER` and `RIGHT OUTER` joins)
-
-The scenarios of join can be categorized into 3 cases:
-
-1. Bounded input `JOIN` bounded input
-2. Unbounded input `JOIN` unbounded input
-3. Unbounded input `JOIN` bounded input
-
-## Bounded JOIN Bounded {#join-bounded-bounded}
-
-Standard join implementation is used. All elements from one input are matched
-with all elements from another input. Due to the fact that both inputs are
-bounded, no windowing or triggering is involved.
-
-## Unbounded JOIN Unbounded {#join-unbounded-unbounded}
-
-Standard join implementation is used. All elements from one input are matched
-with all elements from another input.
-
-**Windowing and Triggering**
-
-The following properties must be satisfied when joining unbounded inputs:
-
- - Inputs must have compatible windows, otherwise `IllegalArgumentException`
- will be thrown.
- - Triggers on each input should only fire once per window. Currently this
- means that the only supported trigger in this case is `DefaultTrigger` with
- zero allowed lateness. Using any other trigger will result in
- `UnsupportedOperationException` thrown.
-
-This means that inputs are joined per-window. That is, when the trigger fires
-(only once), then join is performed on all elements in the current window in
-both inputs. This allows to reason about what kind of output is going to be
-produced.
-
-**Note:** similarly to `GroupByKeys` `JOIN` will update triggers using
-`Trigger.continuationTrigger()`. Other aspects of the inputs' windowing
-strategies remain unchanged.
-
-## Unbounded JOIN Bounded {#join-unbounded-bounded}
-
-For this type of `JOIN` bounded input is treated as a side-input by the
-implementation.
-
-This means that
-
- - window/trigger is inherented from upstreams, which should be consistent
-
diff --git a/website/src/documentation/dsls/sql/lexical-structure.md b/website/src/documentation/dsls/sql/lexical-structure.md
deleted file mode 100644
index 423871c..0000000
--- a/website/src/documentation/dsls/sql/lexical-structure.md
+++ /dev/null
@@ -1,1048 +0,0 @@
----
-layout: section
-title: "Beam SQL: Lexical Structure"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/lexical/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL Lexical Structure
-
-A Beam SQL statement comprises a series of tokens. Tokens include
-*identifiers,* *quoted identifiers, literals*, *keywords*, *operators*,
-and *special characters*. Tokens can be separated by whitespace (space,
-backspace, tab, newline) or comments.
-
-Identifiers
------------
-
-Identifiers are names that are associated with columns, tables, and
-other database objects.
-
-Identifiers must begin with a letter or an underscore. Subsequent
-characters can be letters, numbers, or underscores. Quoted identifiers
-are identifiers enclosed by backtick (`` ` ``) characters and can contain
-any character, such as spaces or symbols. However, quoted identifiers
-cannot be empty. [Reserved Keywords](#reserved-keywords) can only be used
-as identifiers if enclosed by backticks.
-
-Syntax (defined here as a regular expression):
-
-`[A-Za-z_][A-Za-z_0-9]*`
-
-Examples:
-
-```
-Customers5
-_dataField1
-ADGROUP
-```
-
-Invalid examples:
-
-```
-5Customers
-_dataField!
-GROUP
-```
-
-`5Customers` begins with a number, not a letter or underscore.
-`_dataField!` contains the special character "!" which is not a letter,
-number, or underscore. `GROUP` is a reserved keyword, and therefore
-cannot be used as an identifier without being enclosed by backtick
-characters.
-
-Both identifiers and quoted identifiers are case insensitive, with some
-nuances. See [Case Sensitivity](#case-sensitivity) for further details.
-
-Quoted identifiers have the same escape sequences as string literals,
-defined below.
-
-Literals
---------
-
-A literal represents a constant value of a built-in data type. Some, but
-not all, data types can be expressed as literals.
-
-### String Literals
-
-Both string and bytes literals must be *quoted* with single
-(`'`) quotation mark.
-
-**Quoted literals:**
-
-<table>
-<thead>
-<tr>
-<th>Literal</th>
-<th>Examples</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Quoted string</td>
-<td><ul><li><code>'it''s'</code></li><li><code>'Title: "Boy"'</code></li></ul></td>
-<td>Quoted strings enclosed by single (<code>'</code>) quotes can contain unescaped double (<code>"</code>) quotes. <br />Two quotation marks (<code>''</code>) is the escape sequence.<br />Quoted strings can contain newlines.</td>
-</tr>
-</tbody>
-</table>
-
-### Integer Literals
-
-Integer literals are either a sequence of decimal digits (0 through 9).
-Integers can be prefixed by "`+`" or "`-`" to represent positive and
-negative values, respectively.
-
-Examples:
-
-```
-123
--123
-```
-
-An integer literal is interpreted as an `BIGINT`.
-
-### Floating Point Literals
-
-Syntax options:
-
-```
-[+-]DIGITS.[DIGITS][e[+-]DIGITS]
-[DIGITS].DIGITS[e[+-]DIGITS]
-DIGITSe[+-]DIGITS
-```
-
-`DIGITS` represents one or more decimal numbers (0 through 9) and `e`
-represents the exponent marker (e or E).
-
-Examples:
-
-```
-123.456e-67
-.1E4
-58.
-4e2
-```
-
-Numeric literals that contain either a decimal point or an exponent
-marker are presumed to be type double.
-
-Implicit coercion of floating point literals to float type is possible
-if the value is within the valid float range.
-
-There is no literal representation of NaN or infinity.
-
-### Array Literals
-
-Array literals are a comma-separated lists of elements enclosed in
-square brackets prefixed with the `ARRAY` keyword.
-
-Examples:
-
-```
-ARRAY[1, 2, 3]
-ARRAY['x', 'y', 'xy']
-```
-
-### Struct Literals
-
-Syntax:
-
-```
-(elem[, elem...])
-```
-
-where `elem` is an element in the struct. `elem` must be a literal data
-type, not an expression or column name.
-
-The output type is an anonymous struct type (structs are not named
-types) with anonymous fields with types matching the types of the input
-expressions.
-
-<table>
-<thead>
-<tr>
-<th>Example</th>
-<th>Output Type</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td><code>(1, 2, 3)</code></td>
-<td><code>STRUCT<BIGINT,BIGINT,BIGINT></code></td>
-</tr>
-<tr>
-<td><code>(1, 'abc')</code></td>
-<td><code>STRUCT<BIGINT,STRING></code></td>
-</tr>
-</tbody>
-</table>
-
-### Date Literals
-
-Syntax:
-
-```
-DATE 'YYYY-M[M]-D[D]'
-```
-
-Date literals contain the `DATE` keyword followed by a string literal
-that conforms to the canonical date format, enclosed in single quotation
-marks. Date literals support a range between the years 1 and 9999,
-inclusive. Dates outside of this range are invalid.
-
-For example, the following date literal represents September 27, 2014:
-
-```
-DATE '2014-09-27'
-```
-
-String literals in canonical date format also implicitly coerce to DATE
-type when used where a DATE-type expression is expected. For example, in
-the query
-
-```
-SELECT * FROM foo WHERE date_col = "2014-09-27"
-```
-
-the string literal `"2014-09-27"` will be coerced to a date literal.
-
-### Time Literals
-
-Syntax:
-
-```
-TIME '[H]H:[M]M:[S]S[.DDDDDD]]'
-```
-
-TIME literals contain the `TIME` keyword and a string literal that
-conforms to the canonical time format, enclosed in single quotation
-marks.
-
-For example, the following time represents 12:30 p.m.:
-
-```
-TIME '12:30:00.45'
-```
-
-### Timestamp literals
-
-Syntax:
-
-```
-TIMESTAMP 'YYYY-[M]M-[D]D [[H]H:[M]M:[S]S[.DDDDDD]]'
-```
-
-Timestamp literals contain the `TIMESTAMP` keyword and a string literal
-that conforms to the canonical timestamp format, enclosed in single
-quotation marks.
-
-Timestamp literals support a range between the years 1 and 9999,
-inclusive. Timestamps outside of this range are invalid.
-
-For example, the following timestamp represents 12:30 p.m. on September
-27, 2014:
-
-```
-TIMESTAMP '2014-09-27 12:30:00.45'
-```
-
-Case Sensitivity
-----------------
-
-Beam SQL follows these rules for case sensitivity:
-
-<table>
-<thead>
-<tr>
-<th>Category</th>
-<th>Case Sensitive?</th>
-<th>Notes</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Keywords</td>
-<td>No</td>
-<td></td>
-</tr>
-<tr>
-<td>Function names</td>
-<td>No</td>
-<td></td>
-</tr>
-<tr>
-<td>Table names</td>
-<td>Yes</td>
-<td></td>
-</tr>
-<tr>
-<td>Column names</td>
-<td>Yes</td>
-<td></td>
-</tr>
-<tr>
-<td>String values</td>
-<td>Yes</td>
-<td></td>
-</tr>
-<tr>
-<td>String comparisons</td>
-<td>Yes</td>
-<td></td>
-</tr>
-<tr>
-<td>Aliases within a query</td>
-<td>No</td>
-<td></td>
-</tr>
-<tr>
-<td>Regular expression matching</td>
-<td>See Notes</td>
-<td>Regular expression matching is case sensitive by default, unless the expression itself specifies that it should be case insensitive.</td>
-</tr>
-<tr>
-<td><code>LIKE</code> matching</td>
-<td>Yes</td>
-<td> </td>
-</tr>
-</tbody>
-</table>
-
-Reserved Keywords
------------------
-
-Keywords are a group of tokens that have special meaning in the Beam SQL
-language, and have the following characteristics:
-
-- Keywords cannot be used as identifiers unless enclosed by backtick
- (\`) characters.
-- Keywords are case insensitive.
-
-Beam SQL has the following reserved keywords.
-
-<table style="table-layout: fixed; width: 110%">
-<tbody>
-<tr>
-<td>
-A<br />
-ABS<br />
-ABSOLUTE<br />
-ACTION<br />
-ADA<br />
-ADD<br />
-ADMIN<br />
-AFTER<br />
-ALL<br />
-ALLOCATE<br />
-ALLOW<br />
-ALTER<br />
-ALWAYS<br />
-AND<br />
-ANY<br />
-APPLY<br />
-ARE<br />
-ARRAY<br />
-ARRAY_MAX_CARDINALITY<br />
-AS<br />
-ASC<br />
-ASENSITIVE<br />
-ASSERTION<br />
-ASSIGNMENT<br />
-ASYMMETRIC<br />
-AT<br />
-ATOMIC<br />
-ATTRIBUTE<br />
-ATTRIBUTES<br />
-AUTHORIZATION<br />
-AVG<br />
-BEFORE<br />
-BEGIN<br />
-BEGIN_FRAME<br />
-BEGIN_PARTITION<br />
-BERNOULLI<br />
-BETWEEN<br />
-BIGINT<br />
-BINARY<br />
-BIT<br />
-BLOB<br />
-BOOLEAN<br />
-BOTH<br />
-BREADTH<br />
-BY<br />
-C<br />
-CALL<br />
-CALLED<br />
-CARDINALITY<br />
-CASCADE<br />
-CASCADED<br />
-CASE<br />
-CAST<br />
-CATALOG<br />
-CATALOG_NAME<br />
-CEIL<br />
-CEILING<br />
-CENTURY<br />
-CHAIN<br />
-CHAR<br />
-CHAR_LENGTH<br />
-CHARACTER<br />
-CHARACTER_LENGTH<br />
-CHARACTER_SET_CATALOG<br />
-CHARACTER_SET_NAME<br />
-CHARACTER_SET_SCHEMA<br />
-CHARACTERISTICS<br />
-CHARACTERS<br />
-CHECK<br />
-CLASSIFIER<br />
-CLASS_ORIGIN<br />
-CLOB<br />
-CLOSE<br />
-COALESCE<br />
-COBOL<br />
-COLLATE<br />
-COLLATION<br />
-COLLATION_CATALOG<br />
-COLLATION_NAME<br />
-COLLATION_SCHEMA<br />
-COLLECT<br />
-COLUMN<br />
-COLUMN_NAME<br />
-COMMAND_FUNCTION<br />
-COMMAND_FUNCTION_CODE<br />
-COMMENT<br />
-COMMIT<br />
-COMMITTED<br />
-CONDITION<br />
-CONDITION_NUMBER<br />
-CONNECT<br />
-CONNECTION<br />
-CONNECTION_NAME<br />
-CONSTRAINT<br />
-CONSTRAINT_CATALOG<br />
-CONSTRAINT_NAME<br />
-CONSTRAINT_SCHEMA<br />
-CONSTRAINTS<br />
-CONSTRUCTOR<br />
-CONTAINS<br />
-CONTINUE<br />
-CONVERT<br />
-CORR<br />
-CORRESPONDING<br />
-COUNT<br />
-COVAR_POP<br />
-COVAR_SAMP<br />
-CREATE<br />
-CROSS<br />
-CUBE<br />
-CUME_DIST<br />
-CURRENT<br />
-CURRENT_CATALOG<br />
-CURRENT_DATE<br />
-CURRENT_DEFAULT_TRANSFORM_GROUP<br />
-CURRENT_PATH<br />
-CURRENT_ROLE<br />
-CURRENT_ROW<br />
-CURRENT_SCHEMA<br />
-CURRENT_TIME<br />
-CURRENT_TIMESTAMP<br />
-CURRENT_TRANSFORM_GROUP_FOR_TYPE<br />
-CURRENT_USER<br />
-CURSOR<br />
-CURSOR_NAME<br />
-CYCLE<br />
-DATA<br />
-DATABASE<br />
-DATE<br />
-DATETIME_INTERVAL_CODE<br />
-DATETIME_INTERVAL_PRECISION<br />
-DAY<br />
-DEALLOCATE<br />
-DEC<br />
-DECADE<br />
-DECIMAL<br />
-DECLARE<br />
-DEFAULT<br />
-DEFAULTS<br />
-DEFERRABLE<br />
-DEFERRED<br />
-DEFINE<br />
-DEFINED<br />
-DEFINER<br />
-DEGREE<br />
-DELETE<br />
-DENSE_RANK<br />
-DEPTH<br />
-DEREF<br />
-DERIVED<br />
-DESC<br />
-DESCRIBE<br />
-DESCRIPTION<br />
-DESCRIPTOR<br />
-DETERMINISTIC<br />
-DIAGNOSTICS<br />
-DISALLOW<br />
-DISCONNECT<br />
-DISPATCH<br />
-DISTINCT<br />
-DOMAIN<br />
-DOUBLE<br />
-DOW<br />
-DOY<br />
-DROP<br />
-DYNAMIC<br />
-DYNAMIC_FUNCTION<br />
-DYNAMIC_FUNCTION_CODE<br />
-EACH<br />
-ELEMENT<br />
-ELSE<br />
-EMPTY<br />
-END<br />
-END-EXEC<br />
-END_FRAME<br />
-END_PARTITION<br />
-EPOCH<br />
-EQUALS<br />
-ESCAPE<br />
-EVERY<br />
-EXCEPT<br />
-EXCEPTION<br />
-EXCLUDE<br />
-EXCLUDING<br />
-EXEC<br />
-EXECUTE<br />
-EXISTS<br />
-EXP<br />
-EXPLAIN<br />
-EXTEND<br />
-EXTERNAL<br />
-EXTRACT<br />
-FALSE<br />
-FETCH<br />
-FILTER<br />
-FINAL<br />
-FIRST<br />
-FIRST_VALUE<br />
-FLOAT<br />
-FLOOR<br />
-FOLLOWING<br />
-FOR<br />
-FOREIGN<br />
-FORTRAN<br />
-FOUND<br />
-FRAC_SECOND<br />
-FRAME_ROW<br />
-FREE<br />
-FROM<br />
-FULL<br />
-FUNCTION<br />
-FUSION<br />
-G<br />
-GENERAL<br />
-GENERATED<br />
-GEOMETRY<br />
-GET<br />
-GLOBAL<br />
-GO<br />
-GOTO<br />
-GRANT<br />
-GRANTED<br />
-GROUP<br />
-GROUPING<br />
-GROUPS<br />
-HAVING<br />
-HIERARCHY<br />
-HOLD<br />
-HOUR<br />
-IDENTITY<br />
-IF<br />
-IMMEDIATE<br />
-IMMEDIATELY<br />
-IMPLEMENTATION<br />
-IMPORT<br />
-IN<br />
-INCLUDING<br />
-INCREMENT<br />
-INDICATOR<br />
-INITIAL<br />
-INITIALLY<br />
-INNER<br />
-INOUT<br />
-INPUT<br />
-INSENSITIVE<br />
-INSERT<br />
-INSTANCE<br />
-INSTANTIABLE<br />
-INT<br />
-INTEGER<br />
-INTERSECT<br />
-INTERSECTION<br />
-INTERVAL<br />
-INTO<br />
-INVOKER<br />
-IS<br />
-ISOLATION<br />
-JAVA<br />
-JOIN<br />
-JSON<br />
-K<br />
-KEY<br />
-KEY_MEMBER<br />
-KEY_TYPE<br />
-LABEL<br />
-LAG<br />
-LANGUAGE<br />
-LARGE<br />
-LAST<br />
-LAST_VALUE<br />
-LATERAL<br />
-LEAD<br />
-LEADING<br />
-LEFT<br />
-LENGTH<br />
-LEVEL<br />
-LIBRARY<br />
-LIKE<br />
-LIKE_REGEX<br />
-LIMIT<br />
-LN<br />
-LOCAL<br />
-LOCALTIME<br />
-LOCALTIMESTAMP<br />
-LOCATION<br />
-LOCATOR<br />
-LOWER<br />
-M<br />
-MAP<br />
-MATCH<br />
-MATCHED<br />
-MATCHES<br />
-MATCH_NUMBER<br />
-MATCH_RECOGNIZE<br />
-MAX<br />
-MAXVALUE<br />
-MEASURES<br />
-MEMBER<br />
-MERGE<br />
-MESSAGE_LENGTH<br />
-MESSAGE_OCTET_LENGTH<br />
-MESSAGE_TEXT<br />
-METHOD<br />
-MICROSECOND<br />
-MILLENNIUM<br />
-MIN<br />
-MINUTE<br />
-MINVALUE<br />
-MOD<br />
-MODIFIES<br />
-MODULE<br />
-MONTH<br />
-MORE<br />
-MULTISET<br />
-MUMPS<br />
-NAME<br />
-NAMES<br />
-NATIONAL<br />
-NATURAL<br />
-NCHAR<br />
-NCLOB<br />
-NESTING<br />
-</td>
-<td>
-NEW<br />
-NEXT<br />
-NO<br />
-NONE<br />
-NORMALIZE<br />
-NORMALIZED<br />
-NOT<br />
-NTH_VALUE<br />
-NTILE<br />
-NULL<br />
-NULLABLE<br />
-NULLIF<br />
-NULLS<br />
-NUMBER<br />
-NUMERIC<br />
-OBJECT<br />
-OCCURRENCES_REGEX<br />
-OCTET_LENGTH<br />
-OCTETS<br />
-OF<br />
-OFFSET<br />
-OLD<br />
-OMIT<br />
-ON<br />
-ONE<br />
-ONLY<br />
-OPEN<br />
-OPTION<br />
-OPTIONS<br />
-OR<br />
-ORDER<br />
-ORDERING<br />
-ORDINALITY<br />
-OTHERS<br />
-OUT<br />
-OUTER<br />
-OUTPUT<br />
-OVER<br />
-OVERLAPS<br />
-OVERLAY<br />
-OVERRIDING<br />
-PAD<br />
-PARAMETER<br />
-PARAMETER_MODE<br />
-PARAMETER_NAME<br />
-PARAMETER_ORDINAL_POSITION<br />
-PARAMETER_SPECIFIC_CATALOG<br />
-PARAMETER_SPECIFIC_NAME<br />
-PARAMETER_SPECIFIC_SCHEMA<br />
-PARTIAL<br />
-PARTITION<br />
-PASCAL<br />
-PASSTHROUGH<br />
-PAST<br />
-PATH<br />
-PATTERN<br />
-PER<br />
-PERCENT<br />
-PERCENTILE_CONT<br />
-PERCENTILE_DISC<br />
-PERCENT_RANK<br />
-PERIOD<br />
-PERMUTE<br />
-PLACING<br />
-PLAN<br />
-PLI<br />
-PORTION<br />
-POSITION<br />
-POSITION_REGEX<br />
-POWER<br />
-PRECEDES<br />
-PRECEDING<br />
-PRECISION<br />
-PREPARE<br />
-PRESERVE<br />
-PREV<br />
-PRIMARY<br />
-PRIOR<br />
-PRIVILEGES<br />
-PROCEDURE<br />
-PUBLIC<br />
-QUARTER<br />
-RANGE<br />
-RANK<br />
-READ<br />
-READS<br />
-REAL<br />
-RECURSIVE<br />
-REF<br />
-REFERENCES<br />
-REFERENCING<br />
-REGR_AVGX<br />
-REGR_AVGY<br />
-REGR_COUNT<br />
-REGR_INTERCEPT<br />
-REGR_R2<br />
-REGR_SLOPE<br />
-REGR_SXX<br />
-REGR_SXY<br />
-REGR_SYY<br />
-RELATIVE<br />
-RELEASE<br />
-REPEATABLE<br />
-REPLACE<br />
-RESET<br />
-RESTART<br />
-RESTRICT<br />
-RESULT<br />
-RETURN<br />
-RETURNED_CARDINALITY<br />
-RETURNED_LENGTH<br />
-RETURNED_OCTET_LENGTH<br />
-RETURNED_SQLSTATE<br />
-RETURNS<br />
-REVOKE<br />
-RIGHT<br />
-ROLE<br />
-ROLLBACK<br />
-ROLLUP<br />
-ROUTINE<br />
-ROUTINE_CATALOG<br />
-ROUTINE_NAME<br />
-ROUTINE_SCHEMA<br />
-ROW<br />
-ROW_COUNT<br />
-ROW_NUMBER<br />
-ROWS<br />
-RUNNING<br />
-SAVEPOINT<br />
-SCALE<br />
-SCHEMA<br />
-SCHEMA_NAME<br />
-SCOPE<br />
-SCOPE_CATALOGS<br />
-SCOPE_NAME<br />
-SCOPE_SCHEMA<br />
-SCROLL<br />
-SEARCH<br />
-SECOND<br />
-SECTION<br />
-SECURITY<br />
-SEEK<br />
-SELECT<br />
-SELF<br />
-SENSITIVE<br />
-SEQUENCE<br />
-SERIALIZABLE<br />
-SERVER<br />
-SERVER_NAME<br />
-SESSION<br />
-SESSION_USER<br />
-SET<br />
-SETS<br />
-MINUS<br />
-SHOW<br />
-SIMILAR<br />
-SIMPLE<br />
-SIZE<br />
-SKIP<br />
-SMALLINT<br />
-SOME<br />
-SOURCE<br />
-SPACE<br />
-SPECIFIC<br />
-SPECIFIC_NAME<br />
-SPECIFICTYPE<br />
-SQL<br />
-SQLEXCEPTION<br />
-SQLSTATE<br />
-SQLWARNING<br />
-SQL_BIGINT<br />
-SQL_BINARY<br />
-SQL_BIT<br />
-SQL_BLOB<br />
-SQL_BOOLEAN<br />
-SQL_CHAR<br />
-SQL_CLOB<br />
-SQL_DATE<br />
-SQL_DECIMAL<br />
-SQL_DOUBLE<br />
-SQL_FLOAT<br />
-SQL_INTEGER<br />
-SQL_INTERVAL_DAY<br />
-SQL_INTERVAL_DAY_TO_HOUR<br />
-SQL_INTERVAL_DAY_TO_MINUTE<br />
-SQL_INTERVAL_DAY_TO_SECOND<br />
-SQL_INTERVAL_HOUR<br />
-SQL_INTERVAL_HOUR_TO_MINUTE<br />
-SQL_INTERVAL_HOUR_TO_SECOND<br />
-SQL_INTERVAL_MINUTE<br />
-SQL_INTERVAL_MINUTE_TO_SECOND<br />
-SQL_INTERVAL_MONTH<br />
-SQL_INTERVAL_SECOND<br />
-SQL_INTERVAL_YEAR<br />
-SQL_INTERVAL_YEAR_TO_MONTH<br />
-SQL_LONGVARBINARY<br />
-SQL_LONGVARCHAR<br />
-SQL_LONGVARNCHAR<br />
-SQL_NCHAR<br />
-SQL_NCLOB<br />
-SQL_NUMERIC<br />
-SQL_NVARCHAR<br />
-SQL_REAL<br />
-SQL_SMALLINT<br />
-SQL_TIME<br />
-SQL_TIMESTAMP<br />
-SQL_TINYINT<br />
-SQL_TSI_DAY<br />
-SQL_TSI_FRAC_SECOND<br />
-SQL_TSI_HOUR<br />
-SQL_TSI_MICROSECOND<br />
-SQL_TSI_MINUTE<br />
-SQL_TSI_MONTH<br />
-SQL_TSI_QUARTER<br />
-SQL_TSI_SECOND<br />
-SQL_TSI_WEEK<br />
-SQL_TSI_YEAR<br />
-SQL_VARBINARY<br />
-SQL_VARCHAR<br />
-SQRT<br />
-START<br />
-STATE<br />
-STATEMENT<br />
-STATIC<br />
-STDDEV_POP<br />
-STDDEV_SAMP<br />
-STREAM<br />
-STRUCTURE<br />
-STYLE<br />
-SUBCLASS_ORIGIN<br />
-SUBMULTISET<br />
-SUBSET<br />
-SUBSTITUTE<br />
-SUBSTRING<br />
-SUBSTRING_REGEX<br />
-SUCCEEDS<br />
-SUM<br />
-SYMMETRIC<br />
-SYSTEM<br />
-SYSTEM_TIME<br />
-SYSTEM_USER<br />
-TABLE<br />
-TABLE_NAME<br />
-TABLESAMPLE<br />
-TBLPROPERTIES<br />
-TEMPORARY<br />
-THEN<br />
-TIES<br />
-TIME<br />
-TIMESTAMP<br />
-TIMESTAMPADD<br />
-TIMESTAMPDIFF<br />
-TIMEZONE_HOUR<br />
-TIMEZONE_MINUTE<br />
-TINYINT<br />
-TO<br />
-TOP_LEVEL_COUNT<br />
-TRAILING<br />
-TRANSACTION<br />
-TRANSACTIONS_ACTIVE<br />
-TRANSACTIONS_COMMITTED<br />
-TRANSACTIONS_ROLLED_BACK<br />
-TRANSFORM<br />
-TRANSFORMS<br />
-TRANSLATE<br />
-TRANSLATE_REGEX<br />
-TRANSLATION<br />
-TREAT<br />
-TRIGGER<br />
-TRIGGER_CATALOG<br />
-TRIGGER_NAME<br />
-TRIGGER_SCHEMA<br />
-TRIM<br />
-TRIM_ARRAY<br />
-TRUE<br />
-TRUNCATE<br />
-TYPE<br />
-UESCAPE<br />
-UNBOUNDED<br />
-UNCOMMITTED<br />
-UNDER<br />
-UNION<br />
-UNIQUE<br />
-UNKNOWN<br />
-UNNAMED<br />
-UNNEST<br />
-UPDATE<br />
-UPPER<br />
-UPSERT<br />
-USAGE<br />
-USER<br />
-USER_DEFINED_TYPE_CATALOG<br />
-USER_DEFINED_TYPE_CODE<br />
-USER_DEFINED_TYPE_NAME<br />
-USER_DEFINED_TYPE_SCHEMA<br />
-USING<br />
-VALUE<br />
-VALUES<br />
-VALUE_OF<br />
-VAR_POP<br />
-VAR_SAMP<br />
-VARBINARY<br />
-VARCHAR<br />
-VARYING<br />
-VERSION<br />
-VERSIONING<br />
-VIEW<br />
-WEEK<br />
-WHEN<br />
-WHENEVER<br />
-WHERE<br />
-WIDTH_BUCKET<br />
-WINDOW<br />
-WITH<br />
-WITHIN<br />
-WITHOUT<br />
-WORK<br />
-WRAPPER<br />
-WRITE<br />
-XML<br />
-YEAR<br />
-ZONE<br />
-</td>
-</tr>
-</tbody>
-</table>
-
-Terminating Semicolons
-----------------------
-
-Statements can optionally use a terminating semicolon (`;`) in the
-context of a query string submitted through an Application Programming
-Interface (API). Some interactive tools require statements to have a
-terminating semicolon. In a request containing multiple statements,
-statements must be separated by semicolons, but the semicolon is
-optional for the final statement.
-
-Comments
---------
-
-Comments are sequences of characters that are ignored by the parser.
-Beam SQL supports the following types of comments.
-
-### Single line comments
-
-Single line comments are supported by prepending `--` before the comment.
-
-**Examples**
-
-`SELECT x FROM T; --x is a field and T is a table`
-
-Comment includes all characters from the '`--`' sequence to the end of
-the line. You can optionally add a space after the '`--`'.
-
-### Multiline comments
-
-Multiline comments are supported by enclosing the comment using
-`/* <comment> */`.
-
-**Example:**
-
-```
-SELECT x FROM T /* x is a field and T is a table */
-WHERE x = 3;
-```
-
-**Invalid example:**
-
-```
-SELECT x FROM T /* comment starts here
- /* comment ends on this line */
- this line is not considered a comment */
-WHERE x = 3;
-```
-
-Comment includes all characters, including newlines, enclosed by the
-first occurrence of '`/*`' and the first subsequent occurrence of
-'`*/`'. Nested comments are not supported. The second example contains a
-nested comment that renders the query invalid.
-
-> Portions of this page are modifications based on work created and
-> [shared by Google](https://developers.google.com/terms/site-policies)
-> and used according to terms described in the [Creative Commons 3.0
-> Attribution License](http://creativecommons.org/licenses/by/3.0/).
diff --git a/website/src/documentation/dsls/sql/overview.md b/website/src/documentation/dsls/sql/overview.md
index 6be9e43..0405d50 100644
--- a/website/src/documentation/dsls/sql/overview.md
+++ b/website/src/documentation/dsls/sql/overview.md
@@ -18,26 +18,51 @@
limitations under the License.
-->
-# Beam SQL: Overview
+# Beam SQL overview
Beam SQL allows a Beam user (currently only available in Beam Java) to query
bounded and unbounded `PCollections` with SQL statements. Your SQL query
is translated to a `PTransform`, an encapsulated segment of a Beam pipeline.
You can freely mix SQL `PTransforms` and other `PTransforms` in your pipeline.
-There are three main things you will need to know to use SQL in your pipeline:
+Beam SQL includes the following dialects:
- - [Apache Calcite](http://calcite.apache.org): a widespread SQL dialect used in
- big data processing with some streaming enhancements. Calcite provides the
- basic dialect underlying Beam SQL. We have added additional extensions to
- make it easy to leverage Beam's unified batch/streaming model and support
- for complex data types.
- - [SqlTransform](https://beam.apache.org/releases/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html):
- the interface for creating `PTransforms` from SQL queries.
+- [Beam Calcite SQL](http://calcite.apache.org)
+- [Beam ZetaSQL](https://github.com/google/zetasql)
+
+Beam Calcite SQL is a variant of Apache Calcite, a dialect widespread in
+big data processing. Beam Calcite SQL is the default Beam SQL dialect. Beam ZetaSQL is more compatible with BigQuery, so it's especially useful in pipelines that [write to or read from BigQuery tables](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html).
+
+To change dialects, pass [the dialect's full package name](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/package-summary.html) to the [`setPlannerName`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/sql/impl/BeamSqlPipelineOptions.html#setPlannerName-java.lang.String-) method in the [`PipelineOptions`](https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/options/PipelineOptions.html) interface.
+
+There are two additional concepts you need to know to use SQL in your pipeline:
+
+ - [SqlTransform](https://beam.apache.org/releases/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/extensions/sql/SqlTransform.html): the interface for creating `PTransforms` from SQL queries.
- [Row](https://beam.apache.org/releases/javadoc/{{ site.release_latest }}/index.html?org/apache/beam/sdk/values/Row.html):
the type of elements that Beam SQL operates on. A `PCollection<Row>` plays the role of a table.
+## Walkthrough
The [SQL pipeline walkthrough]({{ site.baseurl
-}}/documentation/dsls/sql/walkthrough) works through how you use
-these.
+}}/documentation/dsls/sql/walkthrough) works through how to use Beam SQL with example code.
+## Shell
+The Beam SQL shell allows you to write pipelines as SQL queries without using the Java SDK.
+The [Shell page]({{ site.baseurl
+}}/documentation/dsls/sql/shell) describes how to work with the interactive Beam SQL shell.
+
+## Apache Calcite dialect
+The [Beam Calcite SQL overview]({{ site.baseurl
+}}/documentation/dsls/sql/calcite/overview) summarizes Apache Calcite operators,
+functions, syntax, and data types supported by Beam Calcite SQL.
+
+## ZetaSQL dialect
+For more information on the ZetaSQL features in Beam SQL, see the [Beam ZetaSQL dialect reference]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/overview).
+
+To switch to Beam ZetaSQL, configure the [pipeline options](https://beam.apache.org/releases/javadoc/2.15.0/org/apache/beam/sdk/options/PipelineOptions.html) as follows:
+```
+setPlannerName("org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner")
+```
+
+## Beam SQL extensions
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects.
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/scalar-functions.md b/website/src/documentation/dsls/sql/scalar-functions.md
deleted file mode 100644
index 161fb91..0000000
--- a/website/src/documentation/dsls/sql/scalar-functions.md
+++ /dev/null
@@ -1,134 +0,0 @@
----
-layout: section
-title: "Beam SQL: Scalar functions"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/scalar-functions/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL: Scalar functions
-
-Beam SQL has implemented the following built-in functions See also [Calcite
-SQL's operators and functions
-reference](http://calcite.apache.org/docs/reference.html#operators-and-functions)
-
-## Comparison functions and operators
-
-| Operator syntax | Description |
-| ---- | ---- |
-| value1 = value2 | Equals |
-| value1 <> value2 | Not equal |
-| value1 > value2 | Greater than |
-| value1 >= value2 | Greater than or equal |
-| value1 < value2 | Less than |
-| value1 <= value2 | Less than or equal |
-| value IS NULL | Whether value is null |
-| value IS NOT NULL | Whether value is not null |
-{:.table}
-
-## Logical functions and operators
-
-| Operator syntax | Description |
-| ---- | ---- |
-| boolean1 OR boolean2 | Whether boolean1 is TRUE or boolean2 is TRUE |
-| boolean1 AND boolean2 | Whether boolean1 and boolean2 are both TRUE |
-| NOT boolean | Whether boolean is not TRUE; returns UNKNOWN if boolean is UNKNOWN |
-{:.table}
-
-## Arithmetic expressions
-
-| Operator syntax | Description|
-| ---- | ---- |
-| numeric1 + numeric2 | Returns numeric1 plus numeric2|
-| numeric1 - numeric2 | Returns numeric1 minus numeric2|
-| numeric1 * numeric2 | Returns numeric1 multiplied by numeric2|
-| numeric1 / numeric2 | Returns numeric1 divided by numeric2|
-| MOD(numeric, numeric) | Returns the remainder (modulus) of numeric1 divided by numeric2. The result is negative only if numeric1 is negative|
-{:.table}
-
-## Math functions
-
-| Operator syntax | Description |
-| ---- | ---- |
-| ABS(numeric) | Returns the absolute value of numeric |
-| SQRT(numeric) | Returns the square root of numeric |
-| LN(numeric) | Returns the natural logarithm (base e) of numeric |
-| LOG10(numeric) | Returns the base 10 logarithm of numeric |
-| EXP(numeric) | Returns e raised to the power of numeric |
-| ACOS(numeric) | Returns the arc cosine of numeric |
-| ASIN(numeric) | Returns the arc sine of numeric |
-| ATAN(numeric) | Returns the arc tangent of numeric |
-| COT(numeric) | Returns the cotangent of numeric |
-| DEGREES(numeric) | Converts numeric from radians to degrees |
-| RADIANS(numeric) | Converts numeric from degrees to radians |
-| SIGN(numeric) | Returns the signum of numeric |
-| SIN(numeric) | Returns the sine of numeric |
-| TAN(numeric) | Returns the tangent of numeric |
-| ROUND(numeric1, numeric2) | Rounds numeric1 to numeric2 places right to the decimal point |
-{:.table}
-
-## Date functions
-
-| Operator syntax | Description |
-| ---- | ---- |
-| LOCALTIME | Returns the current date and time in the session time zone in a value of datatype TIME |
-| LOCALTIME(precision) | Returns the current date and time in the session time zone in a value of datatype TIME, with precision digits of precision |
-| LOCALTIMESTAMP | Returns the current date and time in the session time zone in a value of datatype TIMESTAMP |
-| LOCALTIMESTAMP(precision) | Returns the current date and time in the session time zone in a value of datatype TIMESTAMP, with precision digits of precision |
-| CURRENT_TIME | Returns the current time in the session time zone, in a value of datatype TIMESTAMP WITH TIME ZONE |
-| CURRENT_DATE | Returns the current date in the session time zone, in a value of datatype DATE |
-| CURRENT_TIMESTAMP | Returns the current date and time in the session time zone, in a value of datatype TIMESTAMP WITH TIME ZONE |
-| EXTRACT(timeUnit FROM datetime) | Extracts and returns the value of a specified datetime field from a datetime value expression |
-| FLOOR(datetime TO timeUnit) | Rounds datetime down to timeUnit |
-| CEIL(datetime TO timeUnit) | Rounds datetime up to timeUnit |
-| YEAR(date) | Equivalent to EXTRACT(YEAR FROM date). Returns an integer. |
-| QUARTER(date) | Equivalent to EXTRACT(QUARTER FROM date). Returns an integer between 1 and 4. |
-| MONTH(date) | Equivalent to EXTRACT(MONTH FROM date). Returns an integer between 1 and 12. |
-| WEEK(date) | Equivalent to EXTRACT(WEEK FROM date). Returns an integer between 1 and 53. |
-| DAYOFYEAR(date) | Equivalent to EXTRACT(DOY FROM date). Returns an integer between 1 and 366. |
-| DAYOFMONTH(date) | Equivalent to EXTRACT(DAY FROM date). Returns an integer between 1 and 31. |
-| DAYOFWEEK(date) | Equivalent to EXTRACT(DOW FROM date). Returns an integer between 1 and 7. |
-| HOUR(date) | Equivalent to EXTRACT(HOUR FROM date). Returns an integer between 0 and 23. |
-| MINUTE(date) | Equivalent to EXTRACT(MINUTE FROM date). Returns an integer between 0 and 59. |
-| SECOND(date) | Equivalent to EXTRACT(SECOND FROM date). Returns an integer between 0 and 59. |
-{:.table}
-
-## String functions
-
-| Operator syntax | Description |
-| ---- | ---- |
-| string \|\| string | Concatenates two character strings |
-| CHAR_LENGTH(string) | Returns the number of characters in a character string |
-| CHARACTER_LENGTH(string) | As CHAR_LENGTH(string) |
-| UPPER(string) | Returns a character string converted to upper case |
-| LOWER(string) | Returns a character string converted to lower case |
-| POSITION(string1 IN string2) | Returns the position of the first occurrence of string1 in string2 |
-| POSITION(string1 IN string2 FROM integer) | Returns the position of the first occurrence of string1 in string2 starting at a given point (not standard SQL) |
-| TRIM( { BOTH \| LEADING \| TRAILING } string1 FROM string2) | Removes the longest string containing only the characters in string1 from the start/end/both ends of string1 |
-| OVERLAY(string1 PLACING string2 FROM integer [ FOR integer2 ]) | Replaces a substring of string1 with string2 |
-| SUBSTRING(string FROM integer) | Returns a substring of a character string starting at a given point |
-| SUBSTRING(string FROM integer FOR integer) | Returns a substring of a character string starting at a given point with a given length |
-| INITCAP(string) | Returns string with the first letter of each word converter to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters. |
-{:.table}
-
-## Conditional functions
-
-| Operator syntax | Description |
-| ---- | ---- |
-| CASE value <br>WHEN value1 [, value11 ]* THEN result1 <br>[ WHEN valueN [, valueN1 ]* THEN resultN ]* <br>[ ELSE resultZ ] <br>END | Simple case |
-| CASE <br>WHEN condition1 THEN result1 <br>[ WHEN conditionN THEN resultN ]* <br>[ ELSE resultZ ] <br>END | Searched case |
-| NULLIF(value, value) | Returns NULL if the values are the same. For example, NULLIF(5, 5) returns NULL; NULLIF(5, 0) returns 5. |
-| COALESCE(value, value [, value ]*) | Provides a value if the first value is null. For example, COALESCE(NULL, 5) returns 5. |
-{:.table}
diff --git a/website/src/documentation/dsls/sql/select.md b/website/src/documentation/dsls/sql/select.md
deleted file mode 100644
index f3a135f..0000000
--- a/website/src/documentation/dsls/sql/select.md
+++ /dev/null
@@ -1,715 +0,0 @@
----
-layout: section
-title: "Beam SQL: SELECT Statement"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/select/
-redirect_from: /documentation/dsls/sql/statements/select/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# SELECT
-
-The main functionality of Beam SQL is the `SELECT` statement. This is how you
-query and join data. The operations supported are a subset of
-[Apache Calcite SQL](http://calcite.apache.org/docs/reference.html#grammar).
-
-Generally, the semantics of queries is standard. Please see the following
-sections to learn about extensions for supporting Beam's unified
-batch/streaming model:
-
- - [Joins]({{ site.baseurl}}/documentation/dsls/sql/joins)
- - [Windowing & Triggering]({{ site.baseurl}}/documentation/dsls/sql/windowing-and-triggering/)
-
-Query statements scan one or more tables or expressions and return the computed
-result rows. This topic describes the syntax for SQL queries in Beam.
-
-## SQL Syntax
-
- query_statement:
- [ WITH with_query_name AS ( query_expr ) [, ...] ]
- query_expr
-
- query_expr:
- { select | ( query_expr ) | query_expr set_op query_expr }
- [ LIMIT count [ OFFSET skip_rows ] ]
-
- select:
- SELECT [{ ALL | DISTINCT }]
- { [ expression. ]* [ EXCEPT ( column_name [, ...] ) ]
- [ REPLACE ( expression [ AS ] column_name [, ...] ) ]
- | expression [ [ AS ] alias ] } [, ...]
- [ FROM from_item [, ...] ]
- [ WHERE bool_expression ]
- [ GROUP BY { expression [, ...] | ROLLUP ( expression [, ...] ) } ]
- [ HAVING bool_expression ]
-
- set_op:
- UNION { ALL | DISTINCT } | INTERSECT DISTINCT | EXCEPT DISTINCT
-
- from_item: {
- table_name [ [ AS ] alias ] |
- join |
- ( query_expr ) [ [ AS ] alias ]
- with_query_name [ [ AS ] alias ]
- }
-
- join:
- from_item [ join_type ] JOIN from_item
- [ { ON bool_expression | USING ( join_column [, ...] ) } ]
-
- join_type:
- { INNER | CROSS | FULL [OUTER] | LEFT [OUTER] | RIGHT [OUTER] }
-
-Notation:
-
-- Square brackets "\[ \]" indicate optional clauses.
-- Parentheses "( )" indicate literal parentheses.
-- The vertical bar "|" indicates a logical OR.
-- Curly braces "{ }" enclose a set of options.
-- A comma followed by an ellipsis within square brackets "\[, ... \]"
- indicates that the preceding item can repeat in a comma-separated list.
-
-## SELECT list
-
-Syntax:
-
- SELECT [{ ALL | DISTINCT }]
- { [ expression. ]*
- | expression [ [ AS ] alias ] } [, ...]
-
-The `SELECT` list defines the columns that the query will return. Expressions in
-the `SELECT` list can refer to columns in any of the `from_item`s in its
-corresponding `FROM` clause.
-
-Each item in the `SELECT` list is one of:
-
-- \*
-- `expression`
-- `expression.*`
-
-### SELECT \*
-
-`SELECT *`, often referred to as *select star*, produces one output column for
-each column that is visible after executing the full query.
-
-```
-SELECT * FROM (SELECT 'apple' AS fruit, 'carrot' AS vegetable);
-
-+-------+-----------+
-| fruit | vegetable |
-+-------+-----------+
-| apple | carrot |
-+-------+-----------+
-```
-
-### SELECT `expression`
-
-Items in a `SELECT` list can be expressions. These expressions evaluate to a
-single value and produce one output column, with an optional explicit `alias`.
-
-If the expression does not have an explicit alias, it receives an implicit alias
-according to the rules for [implicit aliases](#implicit-aliases), if possible.
-Otherwise, the column is anonymous and you cannot refer to it by name elsewhere
-in the query.
-
-### SELECT `expression.*` {#select-expression_1}
-
-An item in a `SELECT` list can also take the form of `expression.*`. This
-produces one output column for each column or top-level field of `expression`.
-The expression must be a table alias.
-
-The following query produces one output column for each column in the table
-`groceries`, aliased as `g`.
-
-```
-WITH groceries AS
- (SELECT 'milk' AS dairy,
- 'eggs' AS protein,
- 'bread' AS grain)
-SELECT g.*
-FROM groceries AS g;
-
-+-------+---------+-------+
-| dairy | protein | grain |
-+-------+---------+-------+
-| milk | eggs | bread |
-+-------+---------+-------+
-```
-
-### SELECT modifiers
-
-You can modify the results returned from a `SELECT` query, as follows.
-
-#### SELECT DISTINCT
-
-A `SELECT DISTINCT` statement discards duplicate rows and returns only the
-remaining rows. `SELECT DISTINCT` cannot return columns of the following types:
-
-- STRUCT
-- ARRAY
-
-#### SELECT ALL
-
-A `SELECT ALL` statement returns all rows, including duplicate rows. `SELECT
-ALL` is the default behavior of `SELECT`.
-
-### Aliases
-
-See [Aliases](#aliases_2) for information on syntax and visibility for
-`SELECT` list aliases.
-
-## FROM clause
-
-The `FROM` clause indicates the table or tables from which to retrieve rows, and
-specifies how to join those rows together to produce a single stream of rows for
-processing in the rest of the query.
-
-### Syntax
-
- from_item: {
- table_name [ [ AS ] alias ] |
- join |
- ( query_expr ) [ [ AS ] alias ] |
- with_query_name [ [ AS ] alias ]
- }
-
-#### table\_name
-
-The name (optionally qualified) of an existing table.
-
- SELECT * FROM Roster;
- SELECT * FROM beam.Roster;
-
-#### join
-
-See [JOIN Types](#join-types) below and [Joins]({{ site.baseurl}}/documentation/dsls/sql/joins).
-
-#### select {#select_1}
-
-`( select ) [ [ AS ] alias ]` is a table [subquery](#subqueries).
-
-#### with\_query\_name
-
-The query names in a `WITH` clause (see [WITH Clause](#with-clause)) act like
-names of temporary tables that you can reference anywhere in the `FROM` clause.
-In the example below, `subQ1` and `subQ2` are `with_query_names`.
-
-Example:
-
- WITH
- subQ1 AS (SELECT * FROM Roster WHERE SchoolID = 52),
- subQ2 AS (SELECT SchoolID FROM subQ1)
- SELECT DISTINCT * FROM subQ2;
-
-The `WITH` clause hides any permanent tables with the same name for the duration
-of the query, unless you qualify the table name, e.g. `beam.Roster`.
-
-### Subqueries
-
-A subquery is a query that appears inside another statement, and is written
-inside parentheses. These are also referred to as "sub-SELECTs" or "nested
-SELECTs". The full `SELECT` syntax is valid in subqueries.
-
-There are two types of subquery:
-
-- Expression Subqueries
- which you can use in a query wherever expressions are valid. Expression
- subqueries return a single value.
-- Table subqueries, which you can use only in a `FROM` clause. The outer query
- treats the result of the subquery as a table.
-
-Note that there must be parentheses around both types of subqueries.
-
-Example:
-
-```
-SELECT AVG ( PointsScored )
-FROM
-( SELECT PointsScored
- FROM Stats
- WHERE SchoolID = 77 )
-```
-
-Optionally, a table subquery can have an alias.
-
-Example:
-
-```
-SELECT r.LastName
-FROM
-( SELECT * FROM Roster) AS r;
-```
-
-### Aliases {#aliases_1}
-
-See [Aliases](#aliases_2) for information on syntax and visibility for
-`FROM` clause aliases.
-
-## JOIN types
-
-Also see [Joins]({{ site.baseurl}}/documentation/dsls/sql/joins).
-
-### Syntax {#syntax_1}
-
- join:
- from_item [ join_type ] JOIN from_item
- [ ON bool_expression | USING ( join_column [, ...] ) ]
-
- join_type:
- { INNER | CROSS | FULL [OUTER] | LEFT [OUTER] | RIGHT [OUTER] }
-
-The `JOIN` clause merges two `from_item`s so that the `SELECT` clause can query
-them as one source. The `join_type` and `ON` or `USING` clause (a "join
-condition") specify how to combine and discard rows from the two `from_item`s to
-form a single source.
-
-All `JOIN` clauses require a `join_type`.
-
-A `JOIN` clause requires a join condition unless one of the following conditions
-is true:
-
-- `join_type` is `CROSS`.
-- One or both of the `from_item`s is not a table, e.g. an `array_path` or
- `field_path`.
-
-### \[INNER\] JOIN
-
-An `INNER JOIN`, or simply `JOIN`, effectively calculates the Cartesian product
-of the two `from_item`s and discards all rows that do not meet the join
-condition. "Effectively" means that it is possible to implement an `INNER JOIN`
-without actually calculating the Cartesian product.
-
-### CROSS JOIN
-
-`CROSS JOIN` is generally not yet supported.
-
-### FULL \[OUTER\] JOIN
-
-A `FULL OUTER JOIN` (or simply `FULL JOIN`) returns all fields for all rows in
-both `from_item`s that meet the join condition.
-
-`FULL` indicates that *all rows* from both `from_item`s are returned, even if
-they do not meet the join condition. For streaming jobs, all rows that are
-not late according to default trigger and belonging to the same window
-if there's non-global window applied.
-
-`OUTER` indicates that if a given row from one `from_item` does not join to any
-row in the other `from_item`, the row will return with NULLs for all columns
-from the other `from_item`.
-
-Also see [Joins]({{ site.baseurl}}/documentation/dsls/sql/joins).
-
-### LEFT \[OUTER\] JOIN
-
-The result of a `LEFT OUTER JOIN` (or simply `LEFT JOIN`) for two `from_item`s
-always retains all rows of the left `from_item` in the `JOIN` clause, even if no
-rows in the right `from_item` satisfy the join predicate.
-
-`LEFT` indicates that all rows from the *left* `from_item` are returned; if a
-given row from the left `from_item` does not join to any row in the *right*
-`from_item`, the row will return with NULLs for all columns from the right
-`from_item`. Rows from the right `from_item` that do not join to any row in the
-left `from_item` are discarded.
-
-### RIGHT \[OUTER\] JOIN
-
-The result of a `RIGHT OUTER JOIN` (or simply `RIGHT JOIN`) is similar and
-symmetric to that of `LEFT OUTER JOIN`.
-
-### ON clause
-
-The `ON` clause contains a `bool_expression`. A combined row (the result of
-joining two rows) meets the join condition if `bool_expression` returns TRUE.
-
-Example:
-
-```
-SELECT * FROM Roster INNER JOIN PlayerStats
-ON Roster.LastName = PlayerStats.LastName;
-```
-
-### USING clause
-
-The `USING` clause requires a `column_list` of one or more columns which occur
-in both input tables. It performs an equality comparison on that column, and the
-rows meet the join condition if the equality comparison returns TRUE.
-
-In most cases, a statement with the `USING` keyword is equivalent to using the
-`ON` keyword. For example, the statement:
-
-```
-SELECT FirstName
-FROM Roster INNER JOIN PlayerStats
-USING (LastName);
-```
-
-is equivalent to:
-
-```
-SELECT FirstName
-FROM Roster INNER JOIN PlayerStats
-ON Roster.LastName = PlayerStats.LastName;
-```
-
-The results from queries with `USING` do differ from queries that use `ON` when
-you use `SELECT *`. To illustrate this, consider the query:
-
-```
-SELECT * FROM Roster INNER JOIN PlayerStats
-USING (LastName);
-```
-
-This statement returns the rows from `Roster` and `PlayerStats` where
-`Roster.LastName` is the same as `PlayerStats.LastName`. The results include a
-single `LastName` column.
-
-By contrast, consider the following query:
-
-```
-SELECT * FROM Roster INNER JOIN PlayerStats
-ON Roster.LastName = PlayerStats.LastName;
-```
-
-This statement returns the rows from `Roster` and `PlayerStats` where
-`Roster.LastName` is the same as `PlayerStats.LastName`. The results include two
-`LastName` columns; one from `Roster` and one from `PlayerStats`.
-
-### Sequences of JOINs
-
-The `FROM` clause can contain multiple `JOIN` clauses in sequence.
-
-Example:
-
-```
-SELECT * FROM a LEFT JOIN b ON TRUE LEFT JOIN c ON TRUE;
-```
-
-where `a`, `b`, and `c` are any `from_item`s. JOINs are bound from left to
-right, but you can insert parentheses to group them in a different order.
-
-## WHERE clause
-
-### Syntax {#syntax_2}
-
-```
-WHERE bool_expression
-```
-
-The `WHERE` clause filters out rows by evaluating each row against
-`bool_expression`, and discards all rows that do not return TRUE (that is, rows
-that return FALSE or NULL).
-
-Example:
-
-```
-SELECT * FROM Roster
-WHERE SchoolID = 52;
-```
-
-The `bool_expression` can contain multiple sub-conditions.
-
-Example:
-
-```
-SELECT * FROM Roster
-WHERE LastName LIKE 'Mc%' OR LastName LIKE 'Mac%';
-```
-
-You cannot reference column aliases from the `SELECT` list in the `WHERE`
-clause.
-
-Expressions in an `INNER JOIN` have an equivalent expression in the `WHERE`
-clause. For example, a query using `INNER` `JOIN` and `ON` has an equivalent
-expression using `CROSS JOIN` and `WHERE`.
-
-Example - this query:
-
-```
-SELECT * FROM Roster INNER JOIN TeamMascot
-ON Roster.SchoolID = TeamMascot.SchoolID;
-```
-
-is equivalent to:
-
-```
-SELECT * FROM Roster CROSS JOIN TeamMascot
-WHERE Roster.SchoolID = TeamMascot.SchoolID;
-```
-
-## GROUP BY clause
-
-Also see [Windowing & Triggering]({{ site.baseurl}}/documentation/dsls/sql/windowing-and-triggering/)
-
-### Syntax {#syntax_3}
-
- GROUP BY { expression [, ...] | ROLLUP ( expression [, ...] ) }
-
-The `GROUP BY` clause groups together rows in a table with non-distinct values
-for the `expression` in the `GROUP BY` clause. For multiple rows in the source
-table with non-distinct values for `expression`, the `GROUP BY` clause produces
-a single combined row. `GROUP BY` is commonly used when aggregate functions are
-present in the `SELECT` list, or to eliminate redundancy in the output.
-
-Example:
-
-```
-SELECT SUM(PointsScored), LastName
-FROM PlayerStats
-GROUP BY LastName;
-```
-
-## HAVING clause
-
-### Syntax {#syntax_4}
-
-```
-HAVING bool_expression
-```
-
-The `HAVING` clause is similar to the `WHERE` clause: it filters out rows that
-do not return TRUE when they are evaluated against the `bool_expression`.
-
-As with the `WHERE` clause, the `bool_expression` can be any expression that
-returns a boolean, and can contain multiple sub-conditions.
-
-The `HAVING` clause differs from the `WHERE` clause in that:
-
-- The `HAVING` clause requires `GROUP BY` or aggregation to be present in the
- query.
-- The `HAVING` clause occurs after `GROUP BY` and aggregation.
- This means that the `HAVING` clause is evaluated once for every
- aggregated row in the result set. This differs from the `WHERE` clause,
- which is evaluated before `GROUP BY` and aggregation.
-
-The `HAVING` clause can reference columns available via the `FROM` clause, as
-well as `SELECT` list aliases. Expressions referenced in the `HAVING` clause
-must either appear in the `GROUP BY` clause or they must be the result of an
-aggregate function:
-
-```
-SELECT LastName
-FROM Roster
-GROUP BY LastName
-HAVING SUM(PointsScored) > 15;
-```
-
-## Set operators
-
-### Syntax {#syntax_6}
-
- UNION { ALL | DISTINCT } | INTERSECT DISTINCT | EXCEPT DISTINCT
-
-Set operators combine results from two or more input queries into a single
-result set. You must specify `ALL` or `DISTINCT`; if you specify `ALL`, then all
-rows are retained. If `DISTINCT` is specified, duplicate rows are discarded.
-
-If a given row R appears exactly m times in the first input query and n times in
-the second input query (m >= 0, n >= 0):
-
-- For `UNION ALL`, R appears exactly m + n times in the result.
-- For `UNION DISTINCT`, the `DISTINCT` is computed after the `UNION` is
- computed, so R appears exactly one time.
-- For `INTERSECT DISTINCT`, the `DISTINCT` is computed after the result above
- is computed.
-- For `EXCEPT DISTINCT`, row R appears once in the output if m > 0 and
- n = 0.
-- If there are more than two input queries, the above operations generalize
- and the output is the same as if the inputs were combined incrementally from
- left to right.
-
-The following rules apply:
-
-- For set operations other than `UNION ALL`, all column types must support
- equality comparison.
-- The input queries on each side of the operator must return the same number
- of columns.
-- The operators pair the columns returned by each input query according to the
- columns' positions in their respective `SELECT` lists. That is, the first
- column in the first input query is paired with the first column in the
- second input query.
-- The result set always uses the column names from the first input query.
-- The result set always uses the supertypes of input types in corresponding
- columns, so paired columns must also have either the same data type or a
- common supertype.
-- You must use parentheses to separate different set operations; for this
- purpose, set operations such as `UNION ALL` and `UNION DISTINCT` are
- different. If the statement only repeats the same set operation, parentheses
- are not necessary.
-
-Examples:
-
-```
-query1 UNION ALL (query2 UNION DISTINCT query3)
-query1 UNION ALL query2 UNION ALL query3
-```
-
-Invalid:
-
- query1 UNION ALL query2 UNION DISTINCT query3
- query1 UNION ALL query2 INTERSECT ALL query3; // INVALID.
-
-### UNION
-
-The `UNION` operator combines the result sets of two or more input queries by
-pairing columns from the result set of each query and vertically concatenating
-them.
-
-### INTERSECT
-
-The `INTERSECT` operator returns rows that are found in the result sets of both
-the left and right input queries. Unlike `EXCEPT`, the positioning of the input
-queries (to the left vs. right of the `INTERSECT` operator) does not matter.
-
-### EXCEPT
-
-The `EXCEPT` operator returns rows from the left input query that are not
-present in the right input query.
-
-## LIMIT clause and OFFSET clause
-
-### Syntax {#syntax_7}
-
-```
-LIMIT count [ OFFSET skip_rows ]
-```
-
-`LIMIT` specifies a non-negative `count` of type INTEGER, and no more than `count`
-rows will be returned. `LIMIT` `0` returns 0 rows. If there is a set operation,
-`LIMIT` is applied after the set operation is evaluated.
-
-`OFFSET` specifies a non-negative `skip_rows` of type INTEGER, and only rows from
-that offset in the table will be considered.
-
-These clauses accept only literal or parameter values.
-
-The rows that are returned by `LIMIT` and `OFFSET` is unspecified.
-
-## WITH clause
-
-The `WITH` clause contains one or more named subqueries which execute every time
-a subsequent `SELECT` statement references them. Any clause or subquery can
-reference subqueries you define in the `WITH` clause. This includes any `SELECT`
-statements on either side of a set operator, such as `UNION`.
-
-Example:
-
-```
-WITH subQ1 AS (SELECT SchoolID FROM Roster),
- subQ2 AS (SELECT OpponentID FROM PlayerStats)
-SELECT * FROM subQ1
-UNION ALL
-SELECT * FROM subQ2;
-```
-
-## Aliases {#aliases_2}
-
-An alias is a temporary name given to a table, column, or expression present in
-a query. You can introduce explicit aliases in the `SELECT` list or `FROM`
-clause, or Beam will infer an implicit alias for some expressions.
-Expressions with neither an explicit nor implicit alias are anonymous and the
-query cannot reference them by name.
-
-### Explicit alias syntax
-
-You can introduce explicit aliases in either the `FROM` clause or the `SELECT`
-list.
-
-In a `FROM` clause, you can introduce explicit aliases for any item, including
-tables, arrays, subqueries, and `UNNEST` clauses, using `[AS] alias`. The `AS`
-keyword is optional.
-
-Example:
-
-```
-SELECT s.FirstName, s2.SongName
-FROM Singers AS s JOIN Songs AS s2 ON s.SingerID = s2.SingerID;
-```
-
-You can introduce explicit aliases for any expression in the `SELECT` list using
-`[AS] alias`. The `AS` keyword is optional.
-
-Example:
-
-```
-SELECT s.FirstName AS name, LOWER(s.FirstName) AS lname
-FROM Singers s;
-```
-
-### Explicit alias visibility
-
-After you introduce an explicit alias in a query, there are restrictions on
-where else in the query you can reference that alias. These restrictions on
-alias visibility are the result of Beam's name scoping rules.
-
-#### FROM clause aliases
-
-Beam processes aliases in a `FROM` clause from left to right, and aliases
-are visible only to subsequent `JOIN` clauses.
-
-### Ambiguous aliases
-
-Beam provides an error if a name is ambiguous, meaning it can resolve to
-more than one unique object.
-
-Examples:
-
-This query contains column names that conflict between tables, since both
-`Singers` and `Songs` have a column named `SingerID`:
-
-```
-SELECT SingerID
-FROM Singers, Songs;
-```
-
-### Implicit aliases
-
-In the `SELECT` list, if there is an expression that does not have an explicit
-alias, Beam assigns an implicit alias according to the following rules.
-There can be multiple columns with the same alias in the `SELECT` list.
-
-- For identifiers, the alias is the identifier. For example, `SELECT abc`
- implies `AS abc`.
-- For path expressions, the alias is the last identifier in the path. For
- example, `SELECT abc.def.ghi` implies `AS ghi`.
-- For field access using the "dot" member field access operator, the alias is
- the field name. For example, `SELECT (struct_function()).fname` implies `AS
- fname`.
-
-In all other cases, there is no implicit alias, so the column is anonymous and
-cannot be referenced by name. The data from that column will still be returned
-and the displayed query results may have a generated label for that column, but
-the label cannot be used like an alias.
-
-In a `FROM` clause, `from_item`s are not required to have an alias. The
-following rules apply:
-
-If there is an expression that does not have an explicit alias, Beam assigns
-an implicit alias in these cases:
-
-- For identifiers, the alias is the identifier. For example, `FROM abc`
- implies `AS abc`.
-- For path expressions, the alias is the last identifier in the path. For
- example, `FROM abc.def.ghi` implies `AS ghi`
-
-Table subqueries do not have implicit aliases.
-
-`FROM UNNEST(x)` does not have an implicit alias.
-
-> Portions of this page are modifications based on
-> [work](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax)
-> created and
-> [shared by Google](https://developers.google.com/terms/site-policies)
-> and used according to terms described in the [Creative Commons 3.0
-> Attribution License](http://creativecommons.org/licenses/by/3.0/).
diff --git a/website/src/documentation/dsls/sql/set.md b/website/src/documentation/dsls/sql/set.md
deleted file mode 100644
index c2be505..0000000
--- a/website/src/documentation/dsls/sql/set.md
+++ /dev/null
@@ -1,55 +0,0 @@
----
-layout: section
-title: "Beam SQL: SET and RESET Statement"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/set/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# SET and RESET Pipeline Options
-
-Beam SQL's `SET` and `RESET` statements allow the user to [configure Pipeline
-Options]({{ site.baseurl }}/documentation/programming-guide/#configuring-pipeline-options)
-via the SQL shell. These are the same Pipeline Options passed to other Beam
-applications on the command line in the `--<option>=<value>` format.
-
-## Syntax
-
-```
-SET option = value
-```
-
-The SET command sets a Pipeline Option.
-
-* `option`: The case sensitive name of the Pipeline Option, specified as an
- [Identifier]({{ site.baseurl }}/documentation/dsls/sql/lexical/#identifiers).
-* `value`: The case sensitive value of the Pipeline Option, specified as an
- [Identifier]({{ site.baseurl }}/documentation/dsls/sql/lexical/#identifiers).
- For flag options that have no value on the command line, use `true`.
-
-```
-RESET option
-```
-
-The RESET command resets a Pipeline Option to its default value.
-
-* `option`: The case sensitive name of the Pipeline Option, specified as an
- [Identifier]({{ site.baseurl }}/documentation/dsls/sql/lexical/#identifiers).
-
-## Common Options
-
-* ```SET project = `my_gcp_project` ```: Sets the default GCP project
- to`my_gcp_project`.
-* `SET runner = DataflowRunner`: Sets the pipeline to run on Dataflow.
diff --git a/website/src/documentation/dsls/sql/shell.md b/website/src/documentation/dsls/sql/shell.md
index dae4939..1317575 100644
--- a/website/src/documentation/dsls/sql/shell.md
+++ b/website/src/documentation/dsls/sql/shell.md
@@ -31,9 +31,9 @@
To use Beam SQL shell, you must first clone the [Beam SDK repository](https://github.com/apache/beam). Then, from the root of the repository clone, execute the following commands to run the shell:
```
-./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
-./sdks/java/extensions/sql/shell/build/install/beam-sdks-java-extensions-sql-shell/bin/beam-sdks-java-extensions-sql-shell
+./sdks/java/extensions/sql/shell/build/install/shell/bin/shell
```
After you run the commands, the SQL shell starts and you can type queries:
@@ -88,7 +88,7 @@
+--------+
```
-_For more information about `SELECT` syntax, see the [SELECT reference page]({{ site.baseurl }}/documentation/dsls/sql/select/)._
+_For more information about `SELECT` syntax, see the [Query syntax page]({{ site.baseurl }}/documentation/dsls/sql/calcite/query-syntax/)._
To write data to the CSV file, use the `INSERT INTO … SELECT ...` statement:
@@ -119,7 +119,7 @@
1. Make sure the SQL shell includes the desired runner. Add the corresponding project id to the `-Pbeam.sql.shell.bundled` parameter of the Gradle invocation ([source code](https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/shell/build.gradle), [project ids](https://github.com/apache/beam/blob/master/settings.gradle)). For example, use the following command to include Flink runner and KafkaIO:
```
- ./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+ ./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
```
_Note: You can bundle multiple runners (using a comma-separated list) or other additional components in the same manner. For example, you can add support for more I/Os._
@@ -145,7 +145,7 @@
You can also build your own standalone package for SQL shell using `distZip` or `distTar` tasks. For example:
```
-./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' distZip
+./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' distZip
ls ./sdks/java/extensions/sql/shell/build/distributions/
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.tar beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.zip
diff --git a/website/src/documentation/dsls/sql/user-defined-functions.md b/website/src/documentation/dsls/sql/user-defined-functions.md
deleted file mode 100644
index 53021fd..0000000
--- a/website/src/documentation/dsls/sql/user-defined-functions.md
+++ /dev/null
@@ -1,124 +0,0 @@
----
-layout: section
-title: "Beam SQL: User-defined functions"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/user-defined-functions/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL: User-defined functions
-
-If Beam SQL does not have a scalar function or aggregate function to meet your
-needs, they can be authored in Java and invoked in your SQL query. These
-are commonly called UDF (for scalar functions) and UDAFs (for aggregate functions).
-
-## Create and specify User Defined Function (UDF)
-
-A UDF can be 1) any Java method that takes zero or more scalar fields and
-return one scalar value, or 2) a `SerializableFunction`. Below is an example of
-UDF and how to use it in DSL:
-
-```java
-/**
- * A example UDF for test.
- */
-public static class CubicInteger implements BeamSqlUdf {
- public static Integer eval(Integer input){
- return input * input * input;
- }
-}
-
-/**
- * Another example UDF with {@link SerializableFunction}.
- */
-public static class CubicIntegerFn implements SerializableFunction<Integer, Integer> {
- @Override
- public Integer apply(Integer input) {
- return input * input * input;
- }
-}
-
-// Define a SQL query which calls the above UDFs
-String sql =
- "SELECT f_int, cubic1(f_int), cubic2(f_int)"
- + "FROM PCOLLECTION "
- + "WHERE f_int = 2";
-
-// Create and apply the PTransform representing the query.
-// Register the UDFs used in the query by calling '.registerUdf()' with
-// either a class which implements BeamSqlUdf or with
-// an instance of the SerializableFunction;
-PCollection<Row> result =
- input.apply(
- "udfExample",
- SqlTransform
- .query(sql)
- .registerUdf("cubic1", CubicInteger.class)
- .registerUdf("cubic2", new CubicIntegerFn())
-```
-
-## Create and specify User Defined Aggregate Function (UDAF)
-
-Beam SQL can accept a `CombineFn` as UDAF. Registration is similar to the UDF
-example above:
-
-```java
-/**
- * UDAF(CombineFn) for test, which returns the sum of square.
- */
-public static class SquareSum extends CombineFn<Integer, Integer, Integer> {
- @Override
- public Integer createAccumulator() {
- return 0;
- }
-
- @Override
- public Integer addInput(Integer accumulator, Integer input) {
- return accumulator + input * input;
- }
-
- @Override
- public Integer mergeAccumulators(Iterable<Integer> accumulators) {
- int v = 0;
- Iterator<Integer> ite = accumulators.iterator();
- while (ite.hasNext()) {
- v += ite.next();
- }
- return v;
- }
-
- @Override
- public Integer extractOutput(Integer accumulator) {
- return accumulator;
- }
-}
-
-// Define a SQL query which calls the above UDAF
-String sql =
- "SELECT f_int1, squaresum(f_int2) "
- + "FROM PCOLLECTION "
- + "GROUP BY f_int2";
-
-// Create and apply the PTransform representing the query.
-// Register the UDAFs used in the query by calling '.registerUdaf()' by
-// providing it an instance of the CombineFn
-PCollection<Row> result =
- input.apply(
- "udafExample",
- SqlTransform
- .query(sql)
- .registerUdaf("squaresum", new SquareSum()));
-```
-
diff --git a/website/src/documentation/dsls/sql/windowing-and-triggering.md b/website/src/documentation/dsls/sql/windowing-and-triggering.md
deleted file mode 100644
index d893247..0000000
--- a/website/src/documentation/dsls/sql/windowing-and-triggering.md
+++ /dev/null
@@ -1,66 +0,0 @@
----
-layout: section
-title: "Beam DSLs: SQL"
-section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/windowing-and-triggering/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Beam SQL: Windowing and triggering
-
-You can use Beam's windowing semantics in two ways:
-
- - you can configure windowing on your input `PCollections` before passing them
- to a `BeamSql` transform
- - you can use windowing extensions in your windowing query, which will override
- the windowing of your input `PCollections`
-
-Triggering can only be used by setting it on your input `PCollections`; there
-are no SQL extensions for specifying triggering.
-
-This section covers the use of SQL extensions to directly apply windowing.
-
-Beam SQL supports windowing functions specified in `GROUP BY` clause.
-`TIMESTAMP` field is required in this case. It is used as event timestamp for
-rows.
-
-Supported windowing functions:
-* `TUMBLE`, or fixed windows. Example of how define a fixed window with duration of 1 hour:
-```
- SELECT f_int, COUNT(*)
- FROM PCOLLECTION
- GROUP BY
- f_int,
- TUMBLE(f_timestamp, INTERVAL '1' HOUR)
-```
-* `HOP`, or sliding windows. Example of how to define a sliding windows for every 30 minutes with 1 hour duration:
-```
- SELECT f_int, COUNT(*)
- FROM PCOLLECTION
- GROUP BY
- f_int,
- HOP(f_timestamp, INTERVAL '30' MINUTE, INTERVAL '1' HOUR)
-```
-* `SESSION`, session windows. Example of how to define a session window with 5 minutes gap duration:
-```
- SELECT f_int, COUNT(*)
- FROM PCOLLECTION
- GROUP BY
- f_int,
- SESSION(f_timestamp, INTERVAL '5' MINUTE)
-```
-
-**Note:** when no windowing function is specified in the query, then windowing strategy of the input `PCollections` is unchanged by the SQL query. If windowing function is specified in the query, then the windowing function of the `PCollection` is updated accordingly, but trigger stays unchanged.
-
diff --git a/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md b/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
new file mode 100644
index 0000000..c708af2
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
@@ -0,0 +1,210 @@
+---
+layout: section
+title: "Beam ZetaSQL aggregate functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/aggregate-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL aggregate functions
+
+This page documents the ZetaSQL aggregate functions supported by Beam ZetaSQL.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| [COUNT(*)](#count) | Returns the number of input rows |
+| [AVG(FLOAT64)](#avg) | Returns the average of non-`NULL` input values |
+| [SUM(numeric)](#sum) | Returns the sum of non-`NULL` values |
+| [MAX(value)](#max) | Returns the maximum non-`NULL` value |
+| [MIN(value)](#min) | Returns the minimum non-`NULL` value |
+{:.table}
+
+## AVG
+
+```
+AVG(expression)
+```
+
+**Description**
+
+Returns the average of non-`NULL` input values.
+
+**Supported Argument Types**
+
+FLOAT64. Note that, for floating point input types, the return result
+is non-deterministic, which means you might receive a different result each time
+you use this function.
+
+**Returned Data Types**
+
++ FLOAT64
+
+
+**Examples**
+
+```
+SELECT AVG(x) as avg
+FROM UNNEST([0, 2, NULL, 4, 4, 5]) as x;
+
++-----+
+| avg |
++-----+
+| 3 |
++-----+
+
+```
+
+## COUNT
+
+1. `COUNT(*)`
+
+2. `COUNT(expression)`
+
+**Description**
+
+1. Returns the number of rows in the input.
+2. Returns the number of rows with `expression` evaluated to any value other
+ than `NULL`.
+
+**Supported Argument Types**
+
+`expression` can be any data type.
+
+**Return Data Types**
+
+INT64
+
+**Examples**
+
+```
+SELECT COUNT(*) AS count_star, COUNT(x) AS count_x
+FROM UNNEST([1, 4, NULL, 4, 5]) AS x;
+
++------------+---------+
+| count_star | count_x |
++------------+---------+
+| 5 | 4 |
++------------+---------+
+
+
+```
+
+## MAX
+```
+MAX(expression)
+```
+
+**Description**
+
+Returns the maximum value of non-`NULL` expressions. Returns `NULL` if there
+are zero input rows or `expression` evaluates to `NULL` for all rows.
+
+**Supported Argument Types**
+
+Any data type except:
++ `ARRAY`
++ `STRUCT`
+
+**Return Data Types**
+
+Same as the data type used as the input values.
+
+**Examples**
+
+```
+SELECT MAX(x) AS max
+FROM UNNEST([8, NULL, 37, 4, NULL, 55]) AS x;
+
++-----+
+| max |
++-----+
+| 55 |
++-----+
+
+
+```
+
+## MIN
+```
+MIN(expression)
+```
+
+**Description**
+
+Returns the minimum value of non-`NULL` expressions. Returns `NULL` if there
+are zero input rows or `expression` evaluates to `NULL` for all rows.
+
+**Supported Argument Types**
+
+Any data type except:
++ `ARRAY`
++ `STRUCT`
+
+**Return Data Types**
+
+Same as the data type used as the input values.
+
+**Examples**
+
+```
+SELECT MIN(x) AS min
+FROM UNNEST([8, NULL, 37, 4, NULL, 55]) AS x;
+
++-----+
+| min |
++-----+
+| 4 |
++-----+
+
+
+```
+
+## SUM
+```
+SUM(expression)
+```
+
+**Description**
+
+Returns the sum of non-null values.
+
+If the expression is a floating point value, the sum is non-deterministic, which means you might receive a different result each time you use this function.
+
+**Supported Argument Types**
+
+Any supported numeric data types.
+
+**Return Data Types**
+
++ Returns INT64 if the input is an integer.
++ Returns FLOAT64 if the input is a floating point
+value.
+
+Returns `NULL` if the input contains only `NULL`s.
+
+**Examples**
+
+```
+SELECT SUM(x) AS sum
+FROM UNNEST([1, 2, 3, 4, 5, 4, 3, 2, 1]) AS x;
+
++-----+
+| sum |
++-----+
+| 25 |
++-----+
+
+
+```
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md b/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
new file mode 100644
index 0000000..845a335
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
@@ -0,0 +1,116 @@
+---
+layout: section
+title: "Beam ZetaSQL conditional expressions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/conditional-expressions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL conditional expressions
+
+This page documents the ZetaSQL scalar functions supported by Beam ZetaSQL.
+
+<table>
+<thead>
+<tr>
+<th>Syntax</th>
+<th>Input Data Types</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+
+<tr>
+ <td><pre>CASE expr
+ WHEN value THEN result
+ [WHEN ...]
+ [ELSE else_result]
+ END</pre></td>
+<td><code>expr</code> and <code>value</code>: Any type</td>
+<td><code>result</code> and <code>else_result</code>: Supertype of input
+types.</td>
+<td>Compares <code>expr</code> to value of each successive <code>WHEN</code>
+clause and returns the first result where this comparison returns true. The
+remaining <code>WHEN</code> clauses and <code>else_result</code> are not
+evaluated. If the
+<code>expr = value</code> comparison returns false or <code>NULL</code> for
+all <code>WHEN</code> clauses, returns
+<code>else_result</code> if present; if not present, returns <code>NULL</code>.
+<code>expr</code> and <code>value</code> expressions
+must be implicitly coercible to a common supertype; equality comparisons are
+done on coerced values. <code>result</code> and <code>else_result</code>
+expressions must be coercible to a common supertype.</td>
+</tr>
+
+
+<tr>
+ <td><pre>CASE
+ WHEN cond1 THEN result
+ [WHEN cond2...]
+ [ELSE else_result]
+ END</pre></td>
+<td><code>cond</code>: BOOL</td>
+<td><code>result</code> and <code>else_result</code>: Supertype of input
+types.</td>
+<td>Evaluates condition <code>cond</code> of each successive <code>WHEN</code>
+clause and returns the first result where the condition is true; any remaining
+<code>WHEN</code> clauses and <code>else_result</code> are not evaluated. If all
+conditions are false or <code>NULL</code>, returns
+<code>else_result</code> if present; if not present, returns
+<code>NULL</code>. <code>result</code> and <code>else_result</code>
+expressions must be implicitly coercible to a common supertype. </td>
+</tr>
+
+<tr>
+<td><a id="coalesce"></a>COALESCE(expr1, ..., exprN)</td>
+<td>Any type</td>
+<td>Supertype of input types</td>
+<td>Returns the value of the first non-null expression. The remaining
+expressions are not evaluated. All input expressions must be implicitly
+coercible to a common supertype.</td>
+</tr>
+<tr>
+<td><a id="if"></a>IF(cond, true_result, else_result)</td>
+<td><code>cond</code>: BOOL</td>
+<td><code>true_result</code> and <code>else_result</code>: Any type.</td>
+<td>If <code>cond</code> is true, returns <code>true_result</code>, else returns
+<code>else_result</code>. <code>else_result</code> is not evaluated if
+<code>cond</code> is true. <code>true_result</code> is not evaluated if
+<code>cond</code> is false or <code>NULL</code>. <code>true_result</code> and
+<code>else_result</code> must be coercible to a common supertype.</td>
+</tr>
+<tr>
+<td><a id="ifnull"></a>IFNULL(expr, null_result)</td>
+<td>Any type</td>
+<td>Any type or supertype of input types.</td>
+<td>If <code>expr</code> is <code>NULL</code>, return <code>null_result</code>. Otherwise,
+return <code>expr</code>. If <code>expr</code> is not <code>NULL</code>,
+<code>null_result</code> is not evaluated. <code>expr</code> and
+<code>null_result</code> must be implicitly coercible to a common
+supertype. Synonym for <code>COALESCE(expr, null_result)</code>.</td>
+</tr>
+<tr>
+<td><a id="nullif"></a>NULLIF(expression, expression_to_match)</td>
+<td>Any type T or subtype of T</td>
+<td>Any type T or subtype of T</td>
+<td>Returns <code>NULL</code> if <code>expression = expression_to_match</code>
+is true, otherwise returns <code>expression</code>. <code>expression</code> and
+<code>expression_to_match</code> must be implicitly coercible to a common
+supertype; equality comparison is done on coerced values.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/conversion-rules.md b/website/src/documentation/dsls/sql/zetasql/conversion-rules.md
new file mode 100644
index 0000000..3a8df1d
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/conversion-rules.md
@@ -0,0 +1,193 @@
+---
+layout: section
+title: "Beam ZetaSQL conversion rules"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/conversion-rules/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL conversion rules
+
+Conversion includes, but is not limited to, casting and coercion:
+
++ Casting is explicit conversion and uses the `CAST()` function.
++ Coercion is implicit conversion, which Beam SQL performs
+ automatically under the conditions described below.
+
+
+The table below summarizes all possible `CAST`s and coercions. "Coercion To" applies to all *expressions* of a given data type (e.g. a column).
+
+<table>
+<thead>
+<tr>
+<th>From Type</th>
+<th>CAST to</th>
+<th>Coercion To</th>
+</tr>
+</thead>
+<tbody>
+
+
+<tr>
+<td>INT64</td>
+<td><span>INT64</span><br /><span>FLOAT64</span><br /><span>STRING</span><br /></td>
+<td><span>FLOAT64</span><br /></td>
+</tr>
+
+<tr>
+<td>FLOAT64</td>
+<td><span>FLOAT64</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>BOOL</td>
+<td><span>BOOL</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>STRING</td>
+<td><span>INT64</span><br /><span>STRING</span><br /><span>BYTES</span><br /><span>TIMESTAMP</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>BYTES</td>
+<td><span>BYTES</span><br /><span>STRING</span><br /></td>
+<td> </td>
+</tr>
+
+<tr>
+<td>TIMESTAMP</td>
+<td><span>STRING</span><br /><span>TIMESTAMP</span><br /></td>
+<td> </td>
+</tr>
+
+
+<tr>
+<td>ARRAY</td>
+<td>ARRAY</td>
+<td> </td>
+</tr>
+
+
+
+<tr>
+<td>STRUCT</td>
+<td>STRUCT</td>
+<td> </td>
+</tr>
+
+
+</tbody>
+</table>
+{:.table}
+
+## Casting
+
+Syntax:
+
+```
+CAST(expr AS typename)
+```
+
+Cast syntax is used in a query to indicate that the result type of an
+expression should be converted to some other type.
+
+Example:
+
+```
+CAST(x=1 AS STRING)
+```
+
+This results in `"true"` if `x` is `1`, `"false"` for any other non-`NULL`
+value, and `NULL` if `x` is `NULL`.
+
+Casts between supported types that do not successfully map from the original
+value to the target domain produce runtime errors. For example, casting
+BYTES to STRING where the
+byte sequence is not valid UTF-8 results in a runtime error.
+
+
+
+When casting an expression `x` of the following types, these rules apply:
+
+<table>
+<tr>
+<th>From</th>
+<th>To</th>
+<th>Rule(s) when casting <code>x</code></th>
+</tr>
+<tr>
+<td>INT64</td>
+<td>FLOAT64</td>
+<td>Returns a close but potentially not exact
+FLOAT64
+value.</td>
+</tr>
+<tr>
+<td>FLOAT64</td>
+<td>STRING</td>
+<td>Returns an approximate string representation.<br />
+</td>
+</tr>
+<tr>
+<td>STRING</td>
+<td>BYTES</td>
+<td>STRINGs are cast to BYTES using UTF-8 encoding. For example, the STRING "©",
+when cast to BYTES, would become a 2-byte sequence with the hex values C2 and
+A9.</td>
+</tr>
+
+<tr>
+<td>BYTES</td>
+<td>STRING</td>
+<td>Returns <code>x</code> interpreted as a UTF-8 STRING.<br />
+For example, the BYTES literal
+<code>b'\xc2\xa9'</code>, when cast to STRING, is interpreted as UTF-8 and
+becomes the unicode character "©".<br />
+An error occurs if <code>x</code> is not valid UTF-8.</td>
+</tr>
+
+<tr>
+<td>ARRAY</td>
+<td>ARRAY</td>
+<td>Must be the exact same ARRAY type.</td>
+</tr>
+
+<tr>
+<td>STRUCT</td>
+<td>STRUCT</td>
+<td>Allowed if the following conditions are met:<br />
+<ol>
+<li>The two STRUCTs have the same number of fields.</li>
+<li>The original STRUCT field types can be explicitly cast to the corresponding
+target STRUCT field types (as defined by field order, not field name).</li>
+</ol>
+</td>
+</tr>
+
+</table>
+{:.table}
+
+
+## Coercion
+
+Beam SQL coerces the result type of an expression to another type if
+needed to match function signatures. For example, if function func() is defined to take a single argument of type INT64 and an expression is used as an argument that has a result type of FLOAT64, then the result of the expression will be coerced to INT64 type before func() is computed.
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/data-types.md b/website/src/documentation/dsls/sql/zetasql/data-types.md
new file mode 100644
index 0000000..ccd756e6f
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/data-types.md
@@ -0,0 +1,457 @@
+---
+layout: section
+title: "Beam ZetaSQL data types"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/data-types/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL lexical structure
+
+<p>
+Beam ZetaSQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. This page documents the ZetaSQL data types supported in Beam ZetaSQL.
+</p>
+
+<h2 id="data-type-properties">Data type properties</h2>
+
+<p>The following table contains data type properties and the data types that
+each property applies to:</p>
+
+<table>
+<thead>
+<tr>
+<th>Property</th>
+<th>Description</th>
+<th>Applies To</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Nullable</td>
+<td nowrap=""><code>NULL</code> is a valid value.</td>
+<td>
+All data types, with the following exceptions:
+<ul>
+<li>ARRAYs cannot be <code>NULL</code>.</li>
+<li><code>NULL ARRAY</code> elements cannot persist to a table.</li>
+<li>Queries cannot handle <code>NULL ARRAY</code> elements.</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>Orderable</td>
+<td nowrap="">Can be used in an <code>ORDER BY</code> clause.</td>
+<td>All data types except for:
+<ul>
+<li>ARRAY</li>
+<li>STRUCT</li>
+</ul></td>
+</tr>
+<tr>
+<td>Groupable</td>
+<td nowrap="">Can generally appear in an expression following<br>
+<code>GROUP BY</code>, <code>DISTINCT</code>, or <code>PARTITION BY</code>.<br>
+ However, <code>PARTITION BY</code> expressions cannot include<br>
+ the floating point types <code>FLOAT</code> and <code>DOUBLE</code>.</br></br></br></td>
+<td>All data types except for:<ul>
+<li>ARRAY</li>
+<li>STRUCT</li>
+<li>FLOAT64</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>Comparable</td>
+<td>Values of the same type can be compared to each other.</td>
+<td>All data types, with the following exceptions:
+
+ARRAY comparisons are not supported.
+
+<br/><br/>
+Equality comparisons for STRUCTs are supported field by field, in field order.
+Field names are ignored. Less than and greater than comparisons are not
+supported.
+
+<br/><br/>
+<br/><br/>
+All types that support comparisons
+can be used in a <code>JOIN</code> condition. See
+<a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/query-syntax#join_types">JOIN
+Types</a> for an explanation of join conditions.
+</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h2 id="numeric-types">Numeric types</h2>
+
+<p>Numeric types include integer types and floating point types.</p>
+
+<h3 id="integer-type">Integer type</h3>
+
+<p>Integers are numeric values that do not have fractional components.</p>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Storage Size</th>
+<th>Range</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>INT64</code></td>
+<td>8 bytes</td>
+<td>-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h3 id="floating-point-type">Floating point type</h3>
+
+<p>Floating point values are approximate numeric values with fractional components.</p>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Storage Size</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>FLOAT64</code></td>
+<td>8 bytes</td>
+<td>Double precision (approximate) decimal values.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<h2 id="boolean-type">Boolean type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>BOOL</code></td>
+<td>Boolean values are represented by the keywords <code>TRUE</code> and
+<code>FALSE</code> (case insensitive).</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h2 id="string-type">String type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRING</code></td>
+<td>Variable-length character (Unicode) data.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>Input STRING values must be UTF-8 encoded and output STRING values will be UTF-8
+encoded. Alternate encodings like CESU-8 and Modified UTF-8 are not treated as
+valid UTF-8.</p>
+<p>All functions and operators that act on STRING values operate on Unicode
+characters rather than bytes. For example, when functions like <code>SUBSTR</code> and <code>LENGTH</code>
+are applied to STRING input, the functions count Unicode characters, not bytes. Comparisons are
+defined on Unicode characters. Comparisons for less than and <code>ORDER BY</code> compare
+character by character, and lower unicode code points are considered lower
+characters.</p>
+
+<h2 id="bytes-type">Bytes type</h2>
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>BYTES</code></td>
+<td>Variable-length binary data.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p>STRING and BYTES are separate types that cannot be used interchangeably. Casts between STRING and BYTES enforce
+that the bytes are encoded using UTF-8.</p>
+
+<h2 id="timestamp-type">Timestamp type</h2>
+
+Caution: {{product_name_short}} SQL has millisecond `TIMESTAMP` precision. If a
+`TIMESTAMP` field has sub-millisecond precision, {{product_name_short}} SQL
+throws an `IllegalArgumentException`.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+<th>Range</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>TIMESTAMP</code></td>
+<td>Represents an absolute point in time, with
+ millisecond
+precision.</td>
+<td>0001-01-01 00:00:00 to 9999-12-31 23:59:59.999 UTC.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>A timestamp represents an absolute point in time, independent of any time zone
+or convention such as Daylight Savings Time.</p>
+
+<h3 id="canonical-format_2">Canonical format</h3>
+<pre class="codehilite"><code>YYYY-[M]M-[D]D[( |T)[H]H:[M]M:[S]S[.DDD]][time zone]</code></pre>
+<ul>
+<li><code>YYYY</code>: Four-digit year</li>
+<li><code>[M]M</code>: One or two digit month</li>
+<li><code>[D]D</code>: One or two digit day</li>
+<li><code>( |T)</code>: A space or a <code>T</code> separator</li>
+<li><code>[H]H</code>: One or two digit hour (valid values from 00 to 23)</li>
+<li><code>[M]M</code>: One or two digit minutes (valid values from 00 to 59)</li>
+<li><code>[S]S</code>: One or two digit seconds (valid values from 00 to 59)</li>
+<li><code>[.DDD]</code>: Up to three fractional digits (i.e. up to millisecond precision)</li>
+<li><code>[time zone]</code>: String representing the time zone. See the <a href="#time-zones">time zones</a>
+ section for details.</li>
+</ul>
+<p>Time zones are used when parsing timestamps or formatting timestamps for display.
+The timestamp value itself does not store a specific time zone. A
+string-formatted timestamp may include a time zone. When a time zone is not
+explicitly specified, the default time zone, UTC, is used.</p>
+<h3 id="time-zones">Time zones</h3>
+<p>Time zones are represented by strings in one of these two canonical formats:</p>
+<ul>
+<li>Offset from Coordinated Universal Time (UTC), or the letter <code>Z</code> for UTC</li>
+<li>Time zone name from the <a href="http://www.iana.org/time-zones">tz database</a></li>
+</ul>
+<h4 id="offset-from-coordinated-universal-time-utc">Offset from Coordinated Universal Time (UTC)</h4>
+<h5 id="offset-format">Offset Format</h5>
+<pre class="codehilite"><code>(+|-)H[H][:M[M]]
+Z</code></pre>
+<h5 id="examples">Examples</h5>
+<pre class="codehilite"><code>-08:00
+-8:15
++3:00
++07:30
+-7
+Z</code></pre>
+<p>When using this format, no space is allowed between the time zone and the rest
+of the timestamp.</p>
+<pre class="codehilite"><code>2014-09-27 12:30:00.45-8:00
+2014-09-27T12:30:00.45Z</code></pre>
+<h4 id="time-zone-name">Time zone name</h4>
+<p>Time zone names are from the <a href="http://www.iana.org/time-zones">tz database</a>. For a
+less comprehensive but simpler reference, see the
+<a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">List of tz database time zones</a>
+on Wikipedia.</p>
+<h5 id="format">Format</h5>
+<pre class="codehilite"><code>continent/[region/]city</code></pre>
+<h5 id="examples_1">Examples</h5>
+<pre class="codehilite"><code>America/Los_Angeles
+America/Argentina/Buenos_Aires</code></pre>
+<p>When using a time zone name, a space is required between the name and the rest
+of the timestamp:</p>
+<pre class="codehilite"><code>2014-09-27 12:30:00.45 America/Los_Angeles</code></pre>
+<p>Note that not all time zone names are interchangeable even if they do happen to
+report the same time during a given part of the year. For example,
+<code>America/Los_Angeles</code> reports the same time as <code>UTC-7:00</code> during Daylight
+Savings Time, but reports the same time as <code>UTC-8:00</code> outside of Daylight
+Savings Time.</p>
+<p>If a time zone is not specified, the default time zone value is used.</p>
+<h4 id="leap-seconds">Leap seconds</h4>
+<p>A timestamp is simply an offset from 1970-01-01 00:00:00 UTC, assuming there are
+exactly 60 seconds per minute. Leap seconds are not represented as part of a
+stored timestamp.</p>
+<p>If your input contains values that use ":60" in the seconds field to represent a
+leap second, that leap second is not preserved when converting to a timestamp
+value. Instead that value is interpreted as a timestamp with ":00" in the
+seconds field of the following minute.</p>
+<p>Leap seconds do not affect timestamp computations. All timestamp computations
+are done using Unix-style timestamps, which do not reflect leap seconds. Leap
+seconds are only observable through functions that measure real-world time. In
+these functions, it is possible for a timestamp second to be skipped or repeated
+when there is a leap second.</p>
+<h2 id="array-type">Array type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>ARRAY</code></td>
+<td>Ordered list of zero or more elements of any non-ARRAY type.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>An ARRAY is an ordered list of zero or more elements of non-ARRAY values.
+ARRAYs of ARRAYs are not allowed. Queries that would produce an ARRAY of
+ARRAYs will return an error. Instead a STRUCT must be inserted between the
+ARRAYs using the <code>SELECT AS STRUCT</code> construct.</p>
+<p>An empty ARRAY and a <code>NULL</code> ARRAY are two distinct values. ARRAYs can contain
+<code>NULL</code> elements.</p>
+<h3 id="declaring-an-array-type">Declaring an ARRAY type</h3>
+<p>ARRAY types are declared using the angle brackets (<code><</code> and <code>></code>). The type
+of the elements of an ARRAY can be arbitrarily complex with the exception that
+an ARRAY cannot directly contain another ARRAY.</p>
+<h4 id="format_1">Format</h4>
+<pre class="codehilite"><code>ARRAY<T></code></pre>
+<h4 id="examples_2">Examples</h4>
+<table>
+<thead>
+<tr>
+<th>Type Declaration</th>
+<th>Meaning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>
+ARRAY<INT64>
+</code>
+</td>
+<td>Simple ARRAY of 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<STRUCT<INT64, INT64>>
+</code>
+</td>
+<td>An ARRAY of STRUCTs, each of which contains two 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<ARRAY<INT64>>
+</code><br/>
+(not supported)
+</td>
+<td>This is an <strong>invalid</strong> type declaration which is included here
+just in case you came looking for how to create a multi-level ARRAY. ARRAYs
+cannot contain ARRAYs directly. Instead see the next example.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+ARRAY<STRUCT<ARRAY<INT64>>>
+</code>
+</td>
+<td>An ARRAY of ARRAYS of 64-bit integers. Notice that there is a STRUCT between
+the two ARRAYs because ARRAYs cannot hold other ARRAYs directly.</td>
+</tr>
+</tbody></table>
+{:.table}
+<h2 id="struct-type">Struct type</h2>
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRUCT</code></td>
+<td>Container of ordered fields each with a type (required) and field name
+(optional).</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h3 id="declaring-a-struct-type">Declaring a STRUCT type</h3>
+<p>STRUCT types are declared using the angle brackets (<code><</code> and <code>></code>). The type of
+the elements of a STRUCT can be arbitrarily complex.</p>
+<h4 id="format_2">Format</h4>
+<pre class="codehilite"><code>STRUCT<T></code></pre>
+<h4 id="examples_3">Examples</h4>
+<table>
+<thead>
+<tr>
+<th>Type Declaration</th>
+<th>Meaning</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>
+STRUCT<INT64>
+</code>
+</td>
+<td>Simple STRUCT with a single unnamed 64-bit integer field.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+STRUCT<x STRUCT<y INT64, z INT64>>
+</code>
+</td>
+<td>A STRUCT with a nested STRUCT named <code>x</code> inside it. The STRUCT
+<code>x</code> has two fields, <code>y</code> and <code>z</code>, both of which
+are 64-bit integers.</td>
+</tr>
+<tr>
+<td nowrap="">
+<code>
+STRUCT<inner_array ARRAY<INT64>>
+</code>
+</td>
+<td>A STRUCT containing an ARRAY named <code>inner_array</code> that holds
+64-bit integer elements.</td>
+</tr>
+</tbody></table>
+{:.table}
+
+<h3 id="limited-comparisons-for-struct">Limited comparisons for STRUCT</h3>
+<p>STRUCTs can be directly compared using equality operators:</p>
+<ul>
+<li>Equal (<code>=</code>)</li>
+<li>Not Equal (<code>!=</code> or <code><></code>)</li>
+<li>[<code>NOT</code>] <code>IN</code></li>
+</ul>
+<p>Notice, though, that these direct equality comparisons compare the fields of
+the STRUCT pairwise in ordinal order ignoring any field names. If instead you
+want to compare identically named fields of a STRUCT, you can compare the
+individual fields directly.</p>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/lexical.md b/website/src/documentation/dsls/sql/zetasql/lexical.md
new file mode 100644
index 0000000..0117140
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/lexical.md
@@ -0,0 +1,573 @@
+---
+layout: section
+title: "Beam ZetaSQL lexical structure"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/lexical/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL lexical structure
+
+<p>Beam ZetaSQL statements are comprised of a series of tokens. Tokens include
+<em>identifiers,</em> <em>quoted identifiers, literals</em>, <em>keywords</em>, <em>operators</em>, and
+<em>special characters</em>. Tokens can be separated by whitespace (space, backspace,
+tab, newline) or comments.</p>
+
+<p><a id="identifiers"></a></p>
+
+<h2>Identifiers</h2>
+
+<p>Identifiers are names that are associated with columns, tables, and other
+database objects.</p>
+
+<p>There are two ways to specify an identifier: unquoted or quoted:</p>
+
+<ul>
+ <li>Unquoted identifiers must begin with a letter or an underscore.
+ Subsequent characters can be letters, numbers, or underscores.</li>
+ <li>Quoted identifiers are enclosed by backtick (`) characters and can
+ contain any character, such as spaces or symbols. However, quoted identifiers
+ cannot be empty. <a href="#reserved_keywords">Reserved Keywords</a> can only be used as
+ identifiers if enclosed by backticks.</li>
+</ul>
+
+Syntax (presented as a grammar with regular expressions, ignoring whitespace):
+
+<pre>
+<span class="var">identifier</span>: { quoted_identifier | unquoted_identifier }
+<span class="var">unquoted_identifier</span>: <code>[A-Za-z_][A-Za-z_0-9]*</code>
+<span class="var">quoted_identifier</span>: <code>\`[^\\\`\r\n]</code> <span class="var">any_escape*</span> <code>\`</code>
+<span class="var">any_escape</span>: <code>\\(. | \n | \r | \r\n)</code>
+</pre>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>Customers5
+_dataField1
+ADGROUP</code></pre>
+
+<p>Invalid examples:</p>
+
+<pre class="codehilite"><code>5Customers
+_dataField!
+GROUP</code></pre>
+
+<p><code>5Customers</code> begins with a number, not a letter or underscore. <code>_dataField!</code>
+contains the special character "!" which is not a letter, number, or underscore.
+<code>GROUP</code> is a reserved keyword, and therefore cannot be used as an identifier
+without being enclosed by backtick characters.</p>
+
+<p>Both identifiers and quoted identifiers are case insensitive, with some
+nuances. See <a href="#case_sensitivity">Case Sensitivity</a> for further details.</p>
+
+<p>Quoted identifiers have the same escape sequences as string literals,
+defined below.</p>
+
+<p><a id="literals"></a></p>
+
+<h2>Literals</h2>
+
+<p>A literal represents a constant value of a built-in data type. Some, but not
+all, data types can be expressed as literals.</p>
+
+<p><a id="string_and_bytes_literals"></a></p>
+
+<h3 id="string-and-bytes-literals">String and Bytes Literals</h3>
+
+<p>Both string and bytes literals must be <em>quoted</em>, either with single (<code>'</code>) or
+double (<code>"</code>) quotation marks, or <em>triple-quoted</em> with groups of three single
+(<code>'''</code>) or three double (<code>"""</code>) quotation marks.</p>
+
+<p><strong>Quoted literals:</strong></p>
+
+<table>
+<thead>
+<tr>
+<th>Literal</th>
+<th>Examples</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Quoted string</td>
+<td><ul><li><code>"abc"</code></li><li><code>"it's"</code></li><li><code>'it\'s'</code></li><li><code>'Title: "Boy"'</code></li></ul></td>
+<td>Quoted strings enclosed by single (<code>'</code>) quotes can contain unescaped double (<code>"</code>) quotes, and vice versa. <br>Backslashes (<code>\</code>) introduce escape sequences. See Escape Sequences table below.<br>Quoted strings cannot contain newlines, even when preceded by a backslash (<code>\</code>).</br></br></td>
+</tr>
+<tr>
+<td>Triple-quoted string</td>
+<td><ul><li><code>"""abc"""</code></li><li><code>'''it's'''</code></li><li><code>'''Title:"Boy"'''</code></li><li><code>'''two<br>lines'''</br></code></li><li><code>'''why\?'''</code></li></ul></td>
+<td>Embedded newlines and quotes are allowed without escaping - see fourth example.<br>Backslashes (<code>\</code>) introduce escape sequences. See Escape Sequences table below.<br>A trailing unescaped backslash (<code>\</code>) at the end of a line is not allowed.<br>Three unescaped quotes in a row which match the starting quotes will end the string.</br></br></br></td>
+</tr>
+<tr>
+<td>Raw string</td>
+<td><ul><li><code>R"abc+"</code></li><li> <code>r'''abc+'''</code></li><li> <code>R"""abc+"""</code></li><li><code>r'f\(abc,(.*),def\)'</code></li></ul></td>
+<td>Quoted or triple-quoted literals that have the raw string literal prefix (<code>r</code> or <code>R</code>) are interpreted as raw/regex strings.<br>Backslash characters (<code>\</code>) do not act as escape characters. If a backslash followed by another character occurs inside the string literal, both characters are preserved.<br>A raw string cannot end with an odd number of backslashes.<br>Raw strings are useful for constructing regular expressions.</br></br></br></td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p>Prefix characters (<code>r</code>, <code>R</code>, <code>b</code>, <code>B)</code> are optional for quoted or triple-quoted strings, and indicate that the string is a raw/regex string or a byte sequence, respectively. For
+example, <code>b'abc'</code> and <code>b'''abc'''</code> are both interpreted as type bytes. Prefix characters are case insensitive.</p>
+
+<p><strong>Quoted literals with prefixes:</strong></p>
+
+<p>The table below lists all valid escape sequences for representing non-alphanumeric characters in string literals.
+Any sequence not in this table produces an error.</p>
+
+<table>
+<thead>
+<tr>
+<th>Escape Sequence</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>\a</code></td>
+<td>Bell</td>
+</tr>
+<tr>
+<td><code>\b</code></td>
+<td>Backspace</td>
+</tr>
+<tr>
+<td><code>\f</code></td>
+<td>Formfeed</td>
+</tr>
+<tr>
+<td><code>\n</code></td>
+<td>Newline</td>
+</tr>
+<tr>
+<td><code>\r</code></td>
+<td>Carriage Return</td>
+</tr>
+<tr>
+<td><code>\t</code></td>
+<td>Tab</td>
+</tr>
+<tr>
+<td><code>\v</code></td>
+<td>Vertical Tab</td>
+</tr>
+<tr>
+<td><code>\\</code></td>
+<td>Backslash (<code>\</code>)</td>
+</tr>
+<tr>
+<td><code>\?</code></td>
+<td>Question Mark (<code>?</code>)</td>
+</tr>
+<tr>
+<td><code>\"</code></td>
+<td>Double Quote (<code>"</code>)</td>
+</tr>
+<tr>
+<td><code>\'</code></td>
+<td>Single Quote (<code>'</code>)</td>
+</tr>
+<tr>
+<td><code>\`</code></td>
+<td>Backtick (<code>`</code>)</td>
+</tr>
+<tr>
+<td><code>\ooo</code></td>
+<td>Octal escape, with exactly three digits (in the range 0-7). Decodes to a single Unicode character (in string literals).</td>
+</tr>
+<tr>
+<td><code>\xhh</code> or <code>\Xhh</code></td>
+<td>Hex escape, with exactly two hex digits (0-9 or A-F or a-f). Decodes to a single Unicode character (in string literals). Examples:<ul style="list-style-type:none"><li><code>'\x41'</code> == <code>'A'</code></li><li><code>'\x41B'</code> is <code>'AB'</code></li><li><code>'\x4'</code> is an error</li></ul></td>
+</tr>
+<tr>
+<td><code>\uhhhh</code></td>
+<td>Unicode escape, with lowercase 'u' and exactly four hex digits. Valid only in string literals or identifiers.<br/>Note that the range D800-DFFF is not allowed, as these are surrogate unicode values.</td>
+</tr>
+<tr>
+<td><code>\Uhhhhhhhh</code></td>
+<td>Unicode escape, with uppercase 'U' and exactly eight hex digits. Valid only in string literals or identifiers.<br/>Note that the range D800-DFFF is not allowed, as these are surrogate unicode values. Also, values greater than 10FFFF are not allowed.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="integer_literals"></a></p>
+
+<h3 id="integer-literals">Integer Literals</h3>
+
+<p>Integer literals are either a sequence of decimal digits (0 through
+9) or a hexadecimal value that is prefixed with "<code>0x</code>". Integers can be prefixed
+by "<code>+</code>" or "<code>-</code>" to represent positive and negative values, respectively.</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>123
+0xABC
+-123</code></pre>
+
+<p>An integer literal is interpreted as an <code>INT64</code>.</p>
+
+<p><a id="floating_point_literals"></a></p>
+
+<h3 id="floating-point-literals">Floating Point Literals</h3>
+
+<p>Syntax options:</p>
+
+<pre class="codehilite"><code>[+-]DIGITS.[DIGITS][e[+-]DIGITS]
+[DIGITS].DIGITS[e[+-]DIGITS]
+DIGITSe[+-]DIGITS</code></pre>
+
+<p><code>DIGITS</code> represents one or more decimal numbers (0 through 9) and <code>e</code> represents the exponent marker (e or E).</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>123.456e-67
+.1E4
+58.
+4e2</code></pre>
+
+<p>Numeric literals that contain
+either a decimal point or an exponent marker are presumed to be type double.</p>
+
+<p>Implicit coercion of floating point literals to float type is possible if the
+value is within the valid float range.</p>
+
+<p>There is no literal
+representation of NaN or infinity.</p>
+
+<p><a id="array_literals"></a></p>
+
+<h3 id="array-literals">Array Literals</h3>
+
+<p>Array literals are a comma-separated lists of elements
+enclosed in square brackets. The <code>ARRAY</code> keyword is optional, and an explicit
+element type T is also optional.</p>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>[1, 2, 3]
+['x', 'y', 'xy']
+ARRAY[1, 2, 3]
+ARRAY<string>['x', 'y', 'xy']
+</code></pre>
+
+<h3 id="timestamp-literals">Timestamp literals</h3>
+
+<p>Syntax:</p>
+
+<pre class="codehilite"><code>TIMESTAMP 'YYYY-[M]M-[D]D [[H]H:[M]M:[S]S[.DDDDDD]] [timezone]'</code></pre>
+
+<p>Timestamp literals contain the <code>TIMESTAMP</code> keyword and a string literal that
+conforms to the canonical timestamp format, enclosed in single quotation marks.</p>
+
+<p>Timestamp literals support a range between the years 1 and 9999, inclusive.
+Timestamps outside of this range are invalid.</p>
+
+<p>A timestamp literal can include a numerical suffix to indicate the timezone:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00.45-08'</code></pre>
+
+<p>If this suffix is absent, the default timezone, UTC, is used.</p>
+
+<p>For example, the following timestamp represents 12:30 p.m. on September 27,
+2014, using the timezone, UTC:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00.45'</code></pre>
+
+<p>For more information on timezones, see <a href="#timezone">Timezone</a>.</p>
+
+<p>String literals with the canonical timestamp format, including those with
+timezone names, implicitly coerce to a timestamp literal when used where a
+timestamp expression is expected. For example, in the following query, the
+string literal <code>"2014-09-27 12:30:00.45 America/Los_Angeles"</code> is coerced
+to a timestamp literal.</p>
+
+<pre class="codehilite"><code>SELECT * FROM foo
+WHERE timestamp_col = "2014-09-27 12:30:00.45 America/Los_Angeles"</code></pre>
+
+<h4 id="timezone">Timezone</h4>
+
+<p>Since timestamp literals must be mapped to a specific point in time, a timezone
+is necessary to correctly interpret a literal. If a timezone is not specified
+as part of the literal itself, then the default timezone value, which is set by
+the Beam SQL implementation, is used.</p>
+
+<p>Timezones are represented by strings in the following canonical format, which
+represents the offset from Coordinated Universal Time (UTC).</p>
+
+<p>Format:</p>
+
+<pre class="codehilite"><code>(+|-)H[H][:M[M]]</code></pre>
+
+<p>Examples:</p>
+
+<pre class="codehilite"><code>'-08:00'
+'-8:15'
+'+3:00'
+'+07:30'
+'-7'</code></pre>
+
+<p>Timezones can also be expressed using string timezone names from the
+<a href="http://www.iana.org/time-zones">tz database</a>. For a less comprehensive but
+simpler reference, see the
+<a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">List of tz database timezones</a>
+on Wikipedia. Canonical timezone names have the format
+<code><continent/[region/]city></code>, such as <code>America/Los_Angeles</code>.</p>
+
+<p>Note that not all timezone names are interchangeable even if they do happen to
+report the same time during a given part of the year. For example, <code>America/Los_Angeles</code> reports the same time as <code>UTC-7:00</code> during Daylight Savings Time, but reports the same time as <code>UTC-8:00</code> outside of Daylight Savings Time.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>TIMESTAMP '2014-09-27 12:30:00 America/Los_Angeles'
+TIMESTAMP '2014-09-27 12:30:00 America/Argentina/Buenos_Aires'</code></pre>
+
+<p><a id="case_sensitivity"></a></p>
+
+<h2 id="case-sensitivity">Case Sensitivity</h2>
+
+<p>Beam SQL follows these rules for case sensitivity:</p>
+
+<table>
+<thead>
+<tr>
+<th>Category</th>
+<th>Case Sensitive?</th>
+<th>Notes</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Keywords</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Function names</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Table names</td>
+<td>See Notes</td>
+<td>Table names are usually case insensitive, but may be case sensitive when querying a database that uses case sensitive table names.</td>
+</tr>
+<tr>
+<td>Column names</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>String values</td>
+<td>Yes</td>
+<td></td>
+</tr>
+<tr>
+<td>String comparisons</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+<tr>
+<td>Aliases within a query</td>
+<td>No</td>
+<td> </td>
+</tr>
+<tr>
+<td>Regular expression matching</td>
+<td>See Notes</td>
+<td>Regular expression matching is case sensitive by default, unless the expression itself specifies that it should be case insensitive.</td>
+</tr>
+<tr>
+<td><code>LIKE</code> matching</td>
+<td>Yes</td>
+<td> </td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="reserved_keywords"></a></p>
+
+<h2 id="reserved-keywords">Reserved Keywords</h2>
+
+<p>Keywords are a group of tokens that have special meaning in the Beam SQL
+language, and have the following characteristics:</p>
+
+<ul>
+<li>Keywords cannot be used as identifiers unless enclosed by backtick (`) characters.</li>
+<li>Keywords are case insensitive.</li>
+</ul>
+
+<p>Beam SQL has the following reserved keywords.</p>
+
+<table style="table-layout: fixed; width: 110%">
+<tbody>
+<tr>
+<td>
+ALL<br/>
+AND<br/>
+ANY<br/>
+ARRAY<br/>
+AS<br/>
+ASC<br/>
+ASSERT_ROWS_MODIFIED<br/>
+AT<br/>
+BETWEEN<br/>
+BY<br/>
+CASE<br/>
+CAST<br/>
+COLLATE<br/>
+CONTAINS<br/>
+CREATE<br/>
+CROSS<br/>
+CUBE<br/>
+CURRENT<br/>
+DEFAULT<br/>
+DEFINE<br/>
+DESC<br/>
+DISTINCT<br/>
+ELSE<br/>
+END<br/>
+</td>
+<td>
+ENUM<br/>
+ESCAPE<br/>
+EXCEPT<br/>
+EXCLUDE<br/>
+EXISTS<br/>
+EXTRACT<br/>
+FALSE<br/>
+FETCH<br/>
+FOLLOWING<br/>
+FOR<br/>
+FROM<br/>
+FULL<br/>
+GROUP<br/>
+GROUPING<br/>
+GROUPS<br/>
+HASH<br/>
+HAVING<br/>
+IF<br/>
+IGNORE<br/>
+IN<br/>
+INNER<br/>
+INTERSECT<br/>
+INTERVAL<br/>
+INTO<br/>
+</td>
+<td>
+IS<br/>
+JOIN<br/>
+LATERAL<br/>
+LEFT<br/>
+LIKE<br/>
+LIMIT<br/>
+LOOKUP<br/>
+MERGE<br/>
+NATURAL<br/>
+NEW<br/>
+NO<br/>
+NOT<br/>
+NULL<br/>
+NULLS<br/>
+OF<br/>
+ON<br/>
+OR<br/>
+ORDER<br/>
+OUTER<br/>
+OVER<br/>
+PARTITION<br/>
+PRECEDING<br/>
+PROTO<br/>
+RANGE<br/>
+</td>
+<td>
+RECURSIVE<br/>
+RESPECT<br/>
+RIGHT<br/>
+ROLLUP<br/>
+ROWS<br/>
+SELECT<br/>
+SET<br/>
+SOME<br/>
+STRUCT<br/>
+TABLESAMPLE<br/>
+THEN<br/>
+TO<br/>
+TREAT<br/>
+TRUE<br/>
+UNBOUNDED<br/>
+UNION<br/>
+UNNEST<br/>
+USING<br/>
+WHEN<br/>
+WHERE<br/>
+WINDOW<br/>
+WITH<br/>
+WITHIN<br/>
+</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+<p><a id="terminating_semicolons"></a></p>
+
+<h2 id="terminating-semicolons">Terminating Semicolons</h2>
+
+<p>Statements can optionally use a terminating semicolon (<code>;</code>) in the context of a query string submitted through an Application Programming Interface (API). Some interactive tools require statements to have a terminating semicolon.
+In a request containing multiple statements, statements must be separated by semicolons, but the semicolon is optional for the final statement.</p>
+
+<h2 id="comments">Comments</h2>
+
+<p>Comments are sequences of characters that are ignored by the parser. Beam SQL
+supports the following types of comments.</p>
+
+<h3 id="single-line-comments">Single line comments</h3>
+
+<p>Single line comments are supported by prepending <code>#</code> or <code>--</code> before the
+comment.</p>
+
+<p><strong>Examples</strong></p>
+
+<p><code>SELECT x FROM T; # x is a field and T is a table</code></p>
+
+<p>Comment includes all characters from the '#' character to the end of the line.</p>
+
+<p><code>SELECT x FROM T; --x is a field and T is a table</code></p>
+
+<p>Comment includes all characters from the '<code>--</code>' sequence to the end of the line. You can optionally add a space after the '<code>--</code>'.</p>
+
+<h3 id="multiline-comments">Multiline comments</h3>
+
+<p>Multiline comments are supported by enclosing the comment using <code>/* <comment> */</code>.</p>
+
+<p><strong>Example:</strong></p>
+
+<pre class="codehilite"><code>SELECT x FROM T /* x is a field and T is a table */
+WHERE x = 3;</code></pre>
+
+<p><strong>Invalid example:</strong></p>
+
+<pre class="codehilite"><code>SELECT x FROM T /* comment starts here
+ /* comment ends on this line */
+ this line is not considered a comment */
+WHERE x = 3;</code></pre>
+
+<p>Comment includes all characters, including newlines, enclosed by the first
+occurrence of '<code>/*</code>' and the first subsequent occurrence of '<code>*/</code>'. Nested
+comments are not supported. The second example contains a nested comment that
+renders the query invalid.</p>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/math-functions.md b/website/src/documentation/dsls/sql/zetasql/math-functions.md
new file mode 100644
index 0000000..81019bb
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/math-functions.md
@@ -0,0 +1,132 @@
+---
+layout: section
+title: "Beam ZetaSQL mathematical functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/math-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL mathematical functions
+
+This page documents ZetaSQL scalar functions supported by Beam ZetaSQL.
+
+All mathematical functions return `NULL` if any of the input parameters is `NULL`.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| MOD(X, Y) | Returns the remainder of the division of X by Y |
+| CEIL(X) | Returns the smallest integral value (with FLOAT64 type) that is not less than X |
+| CEILING(X) | Synonym of CEIL(X) |
+| FLOOR(X) | Returns the largest integral value (with FLOAT64 type) that is not greater than X |
+{:.table}
+
+## MOD
+
+```
+MOD(X, Y)
+```
+
+**Description**
+
+Modulo function: returns the remainder of the division of X by Y. Returned value
+has the same sign as X.
+
+## CEIL
+
+```
+CEIL(X)
+```
+
+**Description**
+
+Returns the smallest integral value (with FLOAT64
+type) that is not less than X.
+
+## CEILING
+
+```
+CEILING(X)
+```
+
+**Description**
+
+Synonym of CEIL(X)
+
+## FLOOR
+
+```
+FLOOR(X)
+```
+
+**Description**
+
+Returns the largest integral value (with FLOAT64
+type) that is not greater than X.
+
+### Example rounding function behavior
+Example behavior of Cloud Dataflow SQL rounding functions:
+
+<table>
+<thead>
+<tr>
+<th>Input "X"</th>
+<th>CEIL(X)</th>
+<th>FLOOR(X)</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>2.0</td>
+<td>2.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.3</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.8</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>2.5</td>
+<td>3.0</td>
+<td>2.0</td>
+</tr>
+<tr>
+<td>-2.3</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>-2.8</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>-2.5</td>
+<td>-2.0</td>
+<td>-3.0</td>
+</tr>
+<tr>
+<td>0</td>
+<td>0</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/operators.md b/website/src/documentation/dsls/sql/zetasql/operators.md
new file mode 100644
index 0000000..971a51c
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/operators.md
@@ -0,0 +1,597 @@
+---
+layout: section
+title: "Beam ZetaSQL operators"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/operators/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL operators
+
+Operators are represented by special characters or keywords; they do not use
+function call syntax. An operator manipulates any number of data inputs, also
+called operands, and returns a result.
+
+Common conventions:
+
++ Unless otherwise specified, all operators return `NULL` when one of the
+ operands is `NULL`.
+
+The following table lists all supported operators from highest to
+lowest precedence. Precedence determines the order in which operators will be evaluated within a statement.
+
+<table>
+ <thead>
+ <tr>
+ <th>Order of Precedence</th>
+ <th>Operator</th>
+ <th>Input Data Types</th>
+ <th>Name</th>
+ <th>Operator Arity</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>1</td>
+ <td>.</td>
+ <td><span> STRUCT</span><br></td>
+ <td>Member field access operator</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[ ]</td>
+ <td>ARRAY</td>
+ <td>Array position. Must be used with OFFSET or ORDINAL&mdash.</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>2</td>
+ <td>-</td>
+ <td>All numeric types</td>
+ <td>Unary minus</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>3</td>
+ <td>*</td>
+ <td>All numeric types</td>
+ <td>Multiplication</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>/</td>
+ <td>All numeric types</td>
+ <td>Division</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>4</td>
+ <td>+</td>
+ <td>All numeric types</td>
+ <td>Addition</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>-</td>
+ <td>All numeric types</td>
+ <td>Subtraction</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>5 (Comparison Operators)</td>
+ <td>=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Equal</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td><</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Less than</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>></td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Greater than</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td><=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Less than or equal to</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>>=</td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Greater than or equal to</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>!=, <></td>
+ <td>Any comparable type. See
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types">Data Types</a> for
+ a complete list.</td>
+ <td>Not equal</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] LIKE</td>
+ <td>STRING and byte</td>
+ <td>Value does [not] match the pattern specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] BETWEEN</td>
+ <td>Any comparable types. See Data Types for list.</td>
+ <td>Value is [not] within the range specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>[NOT] IN</td>
+ <td>Any comparable types. See Data Types for list.</td>
+ <td>Value is [not] in the set of values specified</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] <code>NULL</code></td>
+ <td>All</td>
+ <td>Value is [not] <code>NULL</code></td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] TRUE</td>
+ <td>BOOL</td>
+ <td>Value is [not] TRUE.</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td> </td>
+ <td>IS [NOT] FALSE</td>
+ <td>BOOL</td>
+ <td>Value is [not] FALSE.</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>6</td>
+ <td>NOT</td>
+ <td>BOOL</td>
+ <td>Logical NOT</td>
+ <td>Unary</td>
+ </tr>
+ <tr>
+ <td>7</td>
+ <td>AND</td>
+ <td>BOOL</td>
+ <td>Logical AND</td>
+ <td>Binary</td>
+ </tr>
+ <tr>
+ <td>8</td>
+ <td>OR</td>
+ <td>BOOL</td>
+ <td>Logical OR</td>
+ <td>Binary</td>
+ </tr>
+ </tbody>
+</table>
+{:.table}
+
+Operators with the same precedence are left associative. This means that those
+operators are grouped together starting from the left and moving right. For
+example, the expression:
+
+`x AND y AND z`
+
+is interpreted as
+
+`( ( x AND y ) AND z )`
+
+The expression:
+
+```
+x * y / z
+```
+
+is interpreted as:
+
+```
+( ( x * y ) / z )
+```
+
+All comparison operators have the same priority and are grouped using left
+associativity. However, comparison operators are not associative. As a result,
+it is recommended that you use parentheses to improve readability and ensure
+expressions are resolved as desired. For example:
+
+`(x < y) IS FALSE`
+
+is recommended over:
+
+`x < y IS FALSE`
+
+## Element access operators
+
+<table>
+<thead>
+<tr>
+<th>Operator</th>
+<th>Syntax</th>
+<th>Input Data Types</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>.</td>
+<td>expression.fieldname1...</td>
+<td><span> STRUCT</span><br></td>
+<td>Type T stored in fieldname1</td>
+<td>Dot operator. Can be used to access nested fields,
+e.g.expression.fieldname1.fieldname2...</td>
+</tr>
+<tr>
+<td>[ ]</td>
+<td>array_expression [position_keyword (int_expression ) ]</td>
+<td>See ARRAY Functions.</td>
+<td>Type T stored in ARRAY</td>
+<td>position_keyword is either OFFSET or ORDINAL.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+## Arithmetic operators
+
+All arithmetic operators accept input of numeric type T, and the result type
+has type T unless otherwise indicated in the description below:
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Addition</td>
+<td>X + Y</td>
+</tr>
+<tr>
+<td>Subtraction</td>
+<td>X - Y</td>
+</tr>
+<tr>
+<td>Multiplication</td>
+<td>X * Y</td>
+</tr>
+<tr>
+<td>Division</td>
+<td>X / Y</td>
+</tr>
+<tr>
+<td>Unary Minus</td>
+<td>- X</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+Result types for Addition and Multiplication:
+
+<table>
+<thead>
+<tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>INT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Subtraction:
+
+<table>
+<thead>
+<tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>INT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Division:
+
+<table>
+<thead>
+ <tr><th> </th><th>INT64</th><th>FLOAT64</th></tr>
+</thead>
+<tbody><tr><td>INT64</td><td>FLOAT64</td><td>FLOAT64</td></tr><tr><td>FLOAT64</td><td>FLOAT64</td><td>FLOAT64</td></tr></tbody>
+</table>
+{:.table}
+
+Result types for Unary Minus:
+
+<table>
+<thead>
+<tr>
+<th>Input Data Type</th>
+<th>Result Data Type</th>
+</tr>
+</thead>
+<tbody>
+
+<tr>
+<td>INT64</td>
+<td>INT64</td>
+</tr>
+
+<tr>
+<td>FLOAT64</td>
+<td>FLOAT64</td>
+</tr>
+
+</tbody>
+</table>
+{:.table}
+
+## Logical operators
+
+All logical operators allow only BOOL input.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Logical NOT</td>
+<td nowrap>NOT X</td>
+<td>Returns FALSE if input is TRUE. Returns TRUE if input is FALSE. Returns <code>NULL</code>
+otherwise.</td>
+</tr>
+<tr>
+<td>Logical AND</td>
+<td nowrap>X AND Y</td>
+<td>Returns FALSE if at least one input is FALSE. Returns TRUE if both X and Y
+are TRUE. Returns <code>NULL</code> otherwise.</td>
+</tr>
+<tr>
+<td>Logical OR</td>
+<td nowrap>X OR Y</td>
+<td>Returns FALSE if both X and Y are FALSE. Returns TRUE if at least one input
+is TRUE. Returns <code>NULL</code> otherwise.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+## Comparison operators
+
+Comparisons always return BOOL. Comparisons generally
+require both operands to be of the same type. If operands are of different
+types, and if Cloud Dataflow SQL can convert the values of those types to a
+common type without loss of precision, Cloud Dataflow SQL will generally coerce
+them to that common type for the comparison; Cloud Dataflow SQL will generally
+[coerce literals to the type of non-literals]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/conversion-rules/#coercion), where
+present. Comparable data types are defined in
+[Data Types]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types).
+
+
+
+STRUCTs support only 4 comparison operators: equal
+(=), not equal (!= and <>), and IN.
+
+The following rules apply when comparing these data types:
+
++ FLOAT64
+ : All comparisons with NaN return FALSE,
+ except for `!=` and `<>`, which return TRUE.
++ BOOL: FALSE is less than TRUE.
++ STRING: Strings are
+ compared codepoint-by-codepoint, which means that canonically equivalent
+ strings are only guaranteed to compare as equal if
+ they have been normalized first.
++ `NULL`: The convention holds here: any operation with a `NULL` input returns
+ `NULL`.
+
+<table>
+<thead>
+<tr>
+<th>Name</th>
+<th>Syntax</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Less Than</td>
+<td>X < Y</td>
+<td>Returns TRUE if X is less than Y.</td>
+</tr>
+<tr>
+<td>Less Than or Equal To</td>
+<td>X <= Y</td>
+<td>Returns TRUE if X is less than or equal to Y.</td>
+</tr>
+<tr>
+<td>Greater Than</td>
+<td>X > Y</td>
+<td>Returns TRUE if X is greater than Y.</td>
+</tr>
+<tr>
+<td>Greater Than or Equal To</td>
+<td>X >= Y</td>
+<td>Returns TRUE if X is greater than or equal to Y.</td>
+</tr>
+<tr>
+<td>Equal</td>
+<td>X = Y</td>
+<td>Returns TRUE if X is equal to Y.</td>
+</tr>
+<tr>
+<td>Not Equal</td>
+<td>X != Y<br>X <> Y</td>
+<td>Returns TRUE if X is not equal to Y.</td>
+</tr>
+<tr>
+<td>BETWEEN</td>
+<td>X [NOT] BETWEEN Y AND Z</td>
+<td>Returns TRUE if X is [not] within the range specified. The result of "X
+BETWEEN Y AND Z" is equivalent to "Y <= X AND X <= Z" but X is evaluated
+only once in the former.</td>
+</tr>
+<tr>
+<td>LIKE</td>
+<td>X [NOT] LIKE Y</td>
+<td>Checks if the STRING in the first operand X
+matches a pattern specified by the second operand Y. Expressions can contain
+these characters:
+<ul>
+<li>A percent sign "%" matches any number of characters or bytes</li>
+<li>An underscore "_" matches a single character or byte</li>
+<li>You can escape "\", "_", or "%" using two backslashes. For example, <code>
+"\\%"</code>. If you are using raw strings, only a single backslash is
+required. For example, <code>r"\%"</code>.</li>
+</ul>
+</td>
+</tr>
+<tr>
+<td>IN</td>
+<td>Multiple - see below</td>
+<td>Returns FALSE if the right operand is empty. Returns <code>NULL</code> if the left
+operand is <code>NULL</code>. Returns TRUE or <code>NULL</code>, never FALSE, if the right operand
+contains <code>NULL</code>. Arguments on either side of IN are general expressions. Neither
+operand is required to be a literal, although using a literal on the right is
+most common. X is evaluated only once.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+When testing values that have a STRUCT data type for
+equality, it's possible that one or more fields are `NULL`. In such cases:
+
++ If all non-NULL field values are equal, the comparison returns NULL.
++ If any non-NULL field values are not equal, the comparison returns false.
+
+The following table demonstrates how STRUCT data
+types are compared when they have fields that are `NULL` valued.
+
+<table>
+<thead>
+<tr>
+<th>Struct1</th>
+<th>Struct2</th>
+<th>Struct1 = Struct2</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>NULL</code></td>
+</tr>
+<tr>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>STRUCT(2, NULL)</code></td>
+<td><code>FALSE</code></td>
+</tr>
+<tr>
+<td><code>STRUCT(1,2)</code></td>
+<td><code>STRUCT(1, NULL)</code></td>
+<td><code>NULL</code></td>
+</tr>
+</tbody>
+</table>
+{:.table}
+
+
+
+## IS operators
+
+IS operators return TRUE or FALSE for the condition they are testing. They never
+return `NULL`, even for `NULL` inputs. If NOT is present, the output BOOL value
+is inverted.
+
+<table>
+<thead>
+<tr>
+<th>Function Syntax</th>
+<th>Input Data Type</th>
+<th>Result Data Type</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+ <td><pre>X IS [NOT] NULL</pre></td>
+<td>Any value type</td>
+<td>BOOL</td>
+<td>Returns TRUE if the operand X evaluates to <code>NULL</code>, and returns FALSE
+otherwise.</td>
+</tr>
+<tr>
+ <td><pre>X IS [NOT] TRUE</pre></td>
+<td>BOOL</td>
+<td>BOOL</td>
+<td>Returns TRUE if the BOOL operand evaluates to TRUE. Returns FALSE
+otherwise.</td>
+</tr>
+<tr>
+ <td><pre>X IS [NOT] FALSE</pre></td>
+<td>BOOL</td>
+<td>BOOL</td>
+<td>Returns TRUE if the BOOL operand evaluates to FALSE. Returns FALSE
+otherwise.</td>
+</tr>
+</tbody>
+</table>
+{:.table}
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/overview.md b/website/src/documentation/dsls/sql/zetasql/overview.md
new file mode 100644
index 0000000..1edaa99
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/overview.md
@@ -0,0 +1,67 @@
+---
+layout: section
+title: "Beam ZetaSQL overview"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/overview/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Beam ZetaSQL overview
+Beam SQL supports a varient of the [ZetaSQL](https://github.com/google/zetasql) language. ZetaSQL is similar to the language in BigQuery's SQL framework. This Beam SQL dialect is especially useful in pipelines that [write to or read from BigQuery tables](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.html).
+
+Beam SQL has additional extensions leveraging Beam’s unified batch/streaming model and processing complex data types. You can use these extensions with all Beam SQL dialects, including Beam ZetaSQL.
+
+## Query syntax
+Query statements scan tables or expressions and return the computed result rows. For more information about query statements in Beam ZetaSQL, see the [Query syntax]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/query-syntax) reference and [Function call rules]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/syntax).
+
+## Lexical structure
+A Beam SQL statement comprises a series of tokens. For more information about tokens in Beam ZetaSQL, see the [Lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical) reference.
+
+## Data types
+Beam SQL supports standard SQL scalar data types as well as extensions including arrays, maps, and nested rows. For more information about scalar data in Beam ZetaSQL, see the [Data types]({{ site.baseurl }}/documentation/dsls/sql/zetasql/data-types) reference.
+
+## Functions and operators
+The following table summarizes the [ZetaSQL functions and operators](https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md) supported by Beam ZetaSQL.
+<table class="table-bordered table-striped">
+ <tr><th>Operators and functions</th><th>Beam ZetaSQL support</th></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/conversion_rules.md">Type conversion</a></td><td>Yes</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/aggregate_functions.md">Aggregate functions</a></td><td>See Beam SQL <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions">aggregate functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/statistical_aggregate_functions.md">Statistical aggregate functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/approximate_aggregate_functions.md">Approximate aggregate functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/hll_functions.md">HyperLogLog++ functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/functions-and-operators.md#kll16-quantile-functions">KLL16 quantile functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/numbering_functions.md">Numbering functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/bit_functions.md">Bit functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/mathematical_functions.md">Mathematical functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/math-functions">mathematical functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/navigation_functions.md">Navigation functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/aggregate_analytic_functions.md">Aggregate analytic functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/aggregate-functions">aggregate functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/hash_functions.md">Hash functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/string_functions.md">String functions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/string-functions">string functions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/json_functions.md">JSON functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/array_functions.md">Array functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/date_functions.md">Date functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/datetime_functions.md">DateTime functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/time_functions.md">Time functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/timestamp_functions.md">Timestamp functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/protocol-buffers.md">Protocol buffer functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/security_functions.md">Security functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/net_functions.md">Net functions</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/operators.md">Operator precedence</a></td><td>Yes</td></tr>
+ <tr><td><a href="">Conditional expressions</a></td><td>See <a href="{{ site.baseurl }}/documentation/dsls/sql/zetasql/conditional-expressions">conditional expressions</a></td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/expression_subqueries.md">Expression subqueries</a></td><td>No</td></tr>
+ <tr><td><a href="https://github.com/google/zetasql/blob/master/docs/debugging_functions.md">Debugging functions</a></td><td>No</td></tr>
+</table>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/query-syntax.md b/website/src/documentation/dsls/sql/zetasql/query-syntax.md
new file mode 100644
index 0000000..7a82e38
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/query-syntax.md
@@ -0,0 +1,1215 @@
+---
+layout: section
+title: "Beam ZetaSQL query syntax"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/query-syntax/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL query syntax
+
+<p>Query statements scan one or more tables, streams, or expressions and return
+ the computed result rows.</p>
+
+<h2 id="sql-syntax">SQL Syntax</h2>
+
+<pre>
+<span class="var">query_statement</span>:
+ <span class="var">query_expr</span>
+
+<span class="var">query_expr</span>:
+ [ <a href="#with-clause">WITH</a> <span class="var"><a href="#with_query_name">with_query_name</a></span> AS ( <span class="var">query_expr</span> ) [, ...] ]
+ { <span class="var">select</span> | ( <span class="var">query_expr</span> ) | <span class="var">query_expr</span> <span class="var">set_op</span> <span class="var">query_expr</span> }
+ [ [ <a href="#order-by-clause">ORDER</a> BY <span class="var">expression</span> [{ ASC | DESC }] [, ...] ] <a href="#limit-clause-and-offset-clause">LIMIT</a> <span class="var">count</span> [ OFFSET <span class="var">skip_rows</span> ] ]
+
+<span class="var">select</span>:
+ <a href="#select-list">SELECT</a> [ ALL | DISTINCT ] { * | <span class="var">expression</span> [ [ AS ] <span class="var">alias</span> ] } [, ...]
+ [ <a href="#from-clause">FROM</a> <span class="var">from_item</span> ]
+ [ <a href="#where-clause">WHERE</a> <span class="var">bool_expression</span> ]
+ [ <a href="#group-by-clause">GROUP</a> BY <span class="var">expression</span> [, ...] ]
+ [ <a href="#having-clause">HAVING</a> <span class="var">bool_expression</span> ]
+
+<span class="var">set_op</span>:
+ <a href="#union">UNION</a> { ALL | DISTINCT } | <a href="#intersect">INTERSECT</a> { ALL | DISTINCT } | <a href="#except">EXCEPT</a> { ALL | DISTINCT }
+
+<span class="var">from_item</span>: {
+ <span class="var">table_name</span> [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var">join</span> |
+ ( <span class="var">query_expr</span> ) [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var"><a href="#with_query_name">with_query_name</a></span> [ [ AS ] <span class="var">alias</span> ]
+}
+<span class="var">table_name</span>:
+ <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical#identifiers"><span class="var">identifier</span></a> [ . <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical#identifiers"><span class="var">identifier</span></a> ...]
+
+<span class="var">join</span>:
+ <span class="var">from_item</span> [ <span class="var">join_type</span> ] <a href="#join-types">JOIN</a> <span class="var">from_item</span>
+ <a href="#on-clause">ON</a> <span class="var">bool_expression</span>
+
+<span class="var">join_type</span>:
+ { <a href="#inner-join">INNER</a> | <a href="#full-outer-join">FULL [OUTER]</a> | <a href="#left-outer-join">LEFT [OUTER]</a> | <a href="#right-outer-join">RIGHT [OUTER]</a> }
+
+</pre>
+
+<p>Notation:</p>
+
+<ul>
+<li>Square brackets "[ ]" indicate optional clauses.</li>
+<li>Parentheses "( )" indicate literal parentheses.</li>
+<li>The vertical bar "|" indicates a logical OR.</li>
+<li>Curly braces "{ }" enclose a set of options.</li>
+<li>A comma followed by an ellipsis within square brackets "[, ... ]" indicates that
+ the preceding item can repeat in a comma-separated list.</li>
+</ul>
+
+<h2 id="select-list">SELECT list</h2>
+
+<p>Syntax:</p>
+
+<pre>
+SELECT [ ALL ]
+ { * | <span class="var">expression</span> [ [ AS ] <span class="var">alias</span> ] } [, ...]
+</pre>
+
+<p>The <code>SELECT</code> list defines the columns that the query will return. Expressions in
+the <code>SELECT</code> list can refer to columns in any of the <code>from_item</code>s in its
+corresponding <code>FROM</code> clause.</p>
+
+<p>Each item in the <code>SELECT</code> list is one of:</p>
+
+<ul>
+<li>*</li>
+<li><code>expression</code></li>
+</ul>
+
+<h3 id="select">SELECT *</h3>
+
+<p><code>SELECT *</code>, often referred to as <em>select star</em>, produces one output column for
+each column that is visible after executing the full query.</p>
+
+<pre class="codehilite"><code>SELECT * FROM (SELECT "apple" AS fruit, "carrot" AS vegetable);
+
++-------+-----------+
+| fruit | vegetable |
++-------+-----------+
+| apple | carrot |
++-------+-----------+</code></pre>
+
+<h3 id="select-expression">SELECT <code>expression</code></h3>
+<p><strong>Caution:</strong> In the top-level
+ <code>SELECT</code>, you must either use an explicitly selected column name,
+ or if you are using an expression, you must use an explicit alias.</p>
+
+<p>Items in a <code>SELECT</code> list can be expressions. These expressions evaluate to a
+single value and produce one output column, with an optional explicit <code>alias</code>.</p>
+
+<p>If the expression does not have an explicit alias, it receives an implicit alias
+according to the rules for implicit aliases, if possible.
+Otherwise, the column is anonymous and you cannot refer to it by name elsewhere
+in the query.</p>
+
+<h3 id="select-modifiers">SELECT modifiers</h3>
+
+<p>You can modify the results returned from a <code>SELECT</code> query, as follows.</p>
+
+<h4 id="select-all">SELECT ALL</h4>
+
+<p>A <code>SELECT ALL</code> statement returns all rows, including duplicate rows.
+<code>SELECT ALL</code> is the default behavior of <code>SELECT</code>.</p>
+
+<h3 id="aliases">Aliases</h3>
+
+<p>See <a href="#using_aliases">Aliases</a> for information on syntax and visibility for
+<code>SELECT</code> list aliases.</p>
+
+<h2 id="from-clause">FROM clause</h2>
+
+<p>The <code>FROM</code> clause indicates the tables or streams from which to retrieve rows, and
+specifies how to join those rows together to produce a single stream of
+rows for processing in the rest of the query.</p>
+
+<h3 id="syntax">Syntax</h3>
+
+<pre>
+<span class="var">from_item</span>: {
+ <span class="var">table_name</span> [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var">join</span> |
+ ( <span class="var">query_expr</span> ) [ [ AS ] <span class="var">alias</span> ] |
+ <span class="var"><a href="#with_query_name">with_query_name</a></span> [ [ AS ] <span class="var">alias</span> ]
+}
+</pre>
+
+<h4 id="table_name">table_name</h4>
+
+The fully-qualified SQL name of a data source queryable by Beam
+ SQL, specified by a dot-separated list of identifiers using
+ [Standard SQL lexical structure]({{ site.baseurl
+}}/documentation/dsls/sql/zetasql/lexical). You
+ must use backticks to enclose identifiers that contain characters which
+ are not letters, numbers, or underscores.
+
+<pre>
+SELECT * FROM bigquery.table.`my-project`.baseball.roster;
+SELECT * FROM pubsub.topic.`my-project`.incoming_events;
+</pre>
+
+<h4 id="join">join</h4>
+
+<p>See <a href="#join_types">JOIN Types</a> below.</p>
+
+<h4 id="select_1">select</h4>
+
+<p><code>( select ) [ [ AS ] alias ]</code> is a table <a href="#subqueries">subquery</a>.</p>
+
+<h4 id="with_query_name">with_query_name</h4>
+
+<p>The query names in a <code>WITH</code> clause (see <a href="#with_clause">WITH Clause</a>) act like names of temporary tables that you
+can reference anywhere in the <code>FROM</code> clause. In the example below,
+<code>subQ1</code> and <code>subQ2</code> are <code>with_query_names</code>.</p>
+
+<p>Example:</p>
+
+<pre>
+WITH
+ subQ1 AS (SELECT * FROM Roster WHERE SchoolID = 52),
+ subQ2 AS (SELECT SchoolID FROM subQ1)
+SELECT DISTINCT * FROM subQ2;
+</pre>
+
+<p>The <code>WITH</code> clause hides any permanent tables with the same name
+for the duration of the query, unless you qualify the table name, e.g.
+
+ <code>db.Roster</code>.
+
+</p>
+
+<p><a id="subqueries"></a></p>
+
+<h3>Subqueries</h3>
+
+<p>A subquery is a query that appears inside another statement, and is written
+inside parentheses. These are also referred to as "sub-SELECTs" or
+"nested SELECTs". The full <code>SELECT</code> syntax is valid in
+subqueries.</p>
+
+<p>There are two types of subquery:</p>
+
+<ul>
+<li>Expression subqueries,
+ which you can use in a query wherever expressions are valid. Expression
+ subqueries return a single value.</li>
+<li>Table subqueries, which you can use only in a <code>FROM</code> clause. The outer
+query treats the result of the subquery as a table.</li>
+</ul>
+
+<p>Note that there must be parentheses around both types of subqueries.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT AVG ( PointsScored )
+FROM
+( SELECT PointsScored
+ FROM Stats
+ WHERE SchoolID = 77 )</code></pre>
+
+<p>Optionally, a table subquery can have an alias.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT r.LastName
+FROM
+( SELECT * FROM Roster) AS r;</code></pre>
+
+<h3 id="aliases_1">Aliases</h3>
+
+<p>See <a href="#using_aliases">Aliases</a> for information on syntax and visibility for
+<code>FROM</code> clause aliases.</p>
+
+<p><a id="join_types"></a></p>
+
+<h2 id="join-types">JOIN types</h2>
+
+<h3 id="syntax_1">Syntax</h3>
+
+<pre>
+<span class="var">join</span>:
+ <span class="var">from_item</span> [ <span class="var">join_type</span> ] JOIN <span class="var">from_item</span>
+ <a href="#on-clause">ON</a> <span class="var">bool_expression</span>
+
+<span class="var">join_type</span>:
+ { <a href="#inner-join">INNER</a> | <a href="#full-outer-join">FULL [OUTER]</a> | <a href="#left-outer-join">LEFT [OUTER]</a> | <a href="#right-outer-join">RIGHT [OUTER]</a> }
+</pre>
+
+<p>The <code>JOIN</code> clause merges two <code>from_item</code>s so that the <code>SELECT</code> clause can
+query them as one source. The <code>join_type</code> and <code>ON</code> clause (a
+"join condition") specify how to combine and discard rows from the two
+<code>from_item</code>s to form a single source.</p>
+
+<p>All <code>JOIN</code> clauses require a <code>join_type</code>.</p>
+
+<aside class="caution"> <strong>Caution:</strong> <p>Currently, only equi-join
+is supported. Joins must use the following form:</p>
+
+<pre>
+<span class="var">join condition</span>: conjunction_clause [AND ...]
+<span class="var">conjunction_clause</span>: { column | field_access } = { column | field_access }
+</pre>
+</aside>
+
+<h3 id="inner-join">[INNER] JOIN</h3>
+
+<p>An <code>INNER JOIN</code>, or simply <code>JOIN</code>, effectively calculates the Cartesian product
+of the two <code>from_item</code>s and discards all rows that do not meet the join
+condition. "Effectively" means that it is possible to implement an <code>INNER JOIN</code>
+without actually calculating the Cartesian product.</p>
+
+<h3 id="full-outer-join">FULL [OUTER] JOIN</h3>
+
+<p>A <code>FULL OUTER JOIN</code> (or simply <code>FULL JOIN</code>) returns all fields for all rows in
+both <code>from_item</code>s that meet the join condition.</p>
+
+<p><code>FULL</code> indicates that <em>all rows</em> from both <code>from_item</code>s are
+returned, even if they do not meet the join condition.</p>
+
+<p><code>OUTER</code> indicates that if a given row from one <code>from_item</code> does not
+join to any row in the other <code>from_item</code>, the row will return with NULLs
+for all columns from the other <code>from_item</code>.</p>
+
+<h3 id="left-outer-join">LEFT [OUTER] JOIN</h3>
+
+<p>The result of a <code>LEFT OUTER JOIN</code> (or simply <code>LEFT JOIN</code>) for two
+<code>from_item</code>s always retains all rows of the left <code>from_item</code> in the
+<code>JOIN</code> clause, even if no rows in the right <code>from_item</code> satisfy the join
+predicate.</p>
+
+<p><code>LEFT</code> indicates that all rows from the <em>left</em> <code>from_item</code> are
+returned; if a given row from the left <code>from_item</code> does not join to any row
+in the <em>right</em> <code>from_item</code>, the row will return with NULLs for all
+columns from the right <code>from_item</code>. Rows from the right <code>from_item</code> that
+do not join to any row in the left <code>from_item</code> are discarded.</p>
+
+<h3 id="right-outer-join">RIGHT [OUTER] JOIN</h3>
+
+<p>The result of a <code>RIGHT OUTER JOIN</code> (or simply <code>RIGHT JOIN</code>) is similar and
+symmetric to that of <code>LEFT OUTER JOIN</code>.</p>
+
+<p><a id="on_clause"></a></p>
+
+<h3 id="on-clause">ON clause</h3>
+
+<p>The <code>ON</code> clause contains a <code>bool_expression</code>. A combined row (the result of
+joining two rows) meets the join condition if <code>bool_expression</code> returns
+TRUE.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster INNER JOIN PlayerStats
+ON Roster.LastName = PlayerStats.LastName;</code></pre>
+
+
+<p><a id="sequences_of_joins"></a></p>
+
+<h3 id="sequences-of-joins">Sequences of JOINs</h3>
+
+<p>The <code>FROM</code> clause can contain multiple <code>JOIN</code> clauses in sequence.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM a LEFT JOIN b ON TRUE LEFT JOIN c ON TRUE;</code></pre>
+
+<p>where <code>a</code>, <code>b</code>, and <code>c</code> are any <code>from_item</code>s. JOINs are bound from left to
+right, but you can insert parentheses to group them in a different order.</p>
+
+<p><a id="where_clause"></a></p>
+
+<h2 id="where-clause">WHERE clause</h2>
+
+<h3 id="syntax_2">Syntax</h3>
+
+<pre class="codehilite"><code>WHERE bool_expression</code></pre>
+
+<p>The <code>WHERE</code> clause filters out rows by evaluating each row against
+<code>bool_expression</code>, and discards all rows that do not return TRUE (that is,
+rows that return FALSE or NULL).</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster
+WHERE SchoolID = 52;</code></pre>
+
+<p>The <code>bool_expression</code> can contain multiple sub-conditions.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT * FROM Roster
+WHERE STARTS_WITH(LastName, "Mc") OR STARTS_WITH(LastName, "Mac");</code></pre>
+
+<p>You cannot reference column aliases from the <code>SELECT</code> list in the <code>WHERE</code>
+clause.</p>
+
+<p><a id="group_by_clause"></a></p>
+
+<h2 id="group-by-clause">GROUP BY clause</h2>
+
+<h3 id="syntax_3">Syntax</h3>
+
+<pre>
+GROUP BY <span class="var">expression</span> [, ...]
+</pre>
+
+<p>The <code>GROUP BY</code> clause groups together rows in a table with non-distinct values
+for the <code>expression</code> in the <code>GROUP BY</code> clause. For multiple rows in the
+source table with non-distinct values for <code>expression</code>, the
+<code>GROUP BY</code> clause produces a single combined row. <code>GROUP BY</code> is commonly used
+when aggregate functions are present in the <code>SELECT</code> list, or to eliminate
+redundancy in the output. The data type of <code>expression</code> must be <a href="{{ site.baseurl
+}}/documentation/dsls/sql/zetasql/data-types#data-type-properties">groupable</a>.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName
+FROM PlayerStats
+GROUP BY LastName;</code></pre>
+
+<p>The <code>GROUP BY</code> clause can refer to expression names in the <code>SELECT</code> list. The
+<code>GROUP BY</code> clause also allows ordinal references to expressions in the <code>SELECT</code>
+list using integer values. <code>1</code> refers to the first expression in the
+<code>SELECT</code> list, <code>2</code> the second, and so forth. The expression list can combine
+ordinals and expression names.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName, FirstName
+FROM PlayerStats
+GROUP BY LastName, FirstName;</code></pre>
+
+<p>The query above is equivalent to:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName, FirstName
+FROM PlayerStats
+GROUP BY 2, FirstName;</code></pre>
+
+<p><code>GROUP BY</code> clauses may also refer to aliases. If a query contains aliases in
+the <code>SELECT</code> clause, those aliases override names in the corresponding <code>FROM</code>
+clause.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SUM(PointsScored), LastName as last_name
+FROM PlayerStats
+GROUP BY last_name;</code></pre>
+
+<p><a id="having_clause"></a></p>
+
+<h2 id="having-clause">HAVING clause</h2>
+
+<h3 id="syntax_4">Syntax</h3>
+
+<pre class="codehilite"><code>HAVING bool_expression</code></pre>
+
+<p>The <code>HAVING</code> clause is similar to the <code>WHERE</code> clause: it filters out rows that
+do not return TRUE when they are evaluated against the <code>bool_expression</code>.</p>
+
+<p>As with the <code>WHERE</code> clause, the <code>bool_expression</code> can be any expression
+that returns a boolean, and can contain multiple sub-conditions.</p>
+
+<p>The <code>HAVING</code> clause differs from the <code>WHERE</code> clause in that:</p>
+
+<ul>
+<li>The <code>HAVING</code> clause requires <code>GROUP BY</code> or aggregation to be present in the
+ query.</li>
+<li>The <code>HAVING</code> clause occurs after <code>GROUP BY</code> and aggregation, and before
+ <code>ORDER BY</code>. This means that the <code>HAVING</code> clause is evaluated once for every
+ aggregated row in the result set. This differs from the <code>WHERE</code> clause,
+ which is evaluated before <code>GROUP BY</code> and aggregation.</li>
+</ul>
+
+<p>The <code>HAVING</code> clause can reference columns available via the <code>FROM</code> clause, as
+well as <code>SELECT</code> list aliases. Expressions referenced in the <code>HAVING</code> clause
+must either appear in the <code>GROUP BY</code> clause or they must be the result of an
+aggregate function:</p>
+
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<p>If a query contains aliases in the <code>SELECT</code> clause, those aliases override names
+in a <code>FROM</code> clause.</p>
+
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored) AS ps
+FROM Roster
+GROUP BY LastName
+HAVING ps > 0;</code></pre>
+
+<p><a id="mandatory_aggregation"></a></p>
+
+<h3 id="mandatory-aggregation">Mandatory aggregation</h3>
+
+<p>Aggregation does not have to be present in the <code>HAVING</code> clause itself, but
+aggregation must be present in at least one of the following forms:</p>
+
+<h4 id="aggregation-function-in-the-select-list">Aggregation function in the <code>SELECT</code> list.</h4>
+
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored) AS total
+FROM PlayerStats
+GROUP BY LastName
+HAVING total > 15;</code></pre>
+
+<h4 id="aggregation-function-in-the-having-clause">Aggregation function in the 'HAVING' clause.</h4>
+
+<pre class="codehilite"><code>SELECT LastName
+FROM PlayerStats
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<h4 id="aggregation-in-both-the-select-list-and-having-clause">Aggregation in both the <code>SELECT</code> list and <code>HAVING</code> clause.</h4>
+
+<p>When aggregation functions are present in both the <code>SELECT</code> list and <code>HAVING</code>
+clause, the aggregation functions and the columns they reference do not need
+to be the same. In the example below, the two aggregation functions,
+<code>COUNT()</code> and <code>SUM()</code>, are different and also use different columns.</p>
+
+<pre class="codehilite"><code>SELECT LastName, COUNT(*)
+FROM PlayerStats
+GROUP BY LastName
+HAVING SUM(PointsScored) > 15;</code></pre>
+
+<p><a id="limit-clause_and_offset_clause"></a></p>
+
+<h2 id="limit-clause-and-offset-clause">LIMIT clause and OFFSET clause</h2>
+
+<h3 id="syntax_6">Syntax</h3>
+
+<pre class="codehilite"><code>[ ORDER BY expression [{ASC | DESC}] [,...] ] LIMIT count [ OFFSET skip_rows ]</code></pre>
+
+<p>The <code>ORDER BY</code> clause specifies a column or expression as the sort criterion for
+the result set. If an ORDER BY clause is not present, the order of the results
+of a query is not defined. The default sort direction is <code>ASC</code>, which sorts the
+results in ascending order of <code>expression</code> values. <code>DESC</code> sorts the results in
+descending order. Column aliases from a <code>FROM</code> clause or <code>SELECT</code> list are
+allowed. If a query contains aliases in the <code>SELECT</code> clause, those aliases
+override names in the corresponding <code>FROM</code> clause.</p>
+
+<p>It is possible to order by multiple columns.</p>
+
+<p>The following rules apply when ordering values:</p>
+
+<ul>
+<li>NULLs: In the context of the <code>ORDER BY</code> clause, NULLs are the minimum
+ possible value; that is, NULLs appear first in <code>ASC</code> sorts and last in <code>DESC</code>
+ sorts.</li>
+</ul>
+
+<p><code>LIMIT</code> specifies a non-negative <code>count</code> of type INT64,
+and no more than <code>count</code> rows will be returned. <code>LIMIT</code> <code>0</code> returns 0 rows. If
+there is a set
+operation, <code>LIMIT</code> is applied after the
+set operation
+is evaluated.</p>
+
+<p><code>OFFSET</code> specifies a non-negative <code>skip_rows</code> of type
+INT64, and only rows from
+that offset in the table will be considered.</p>
+
+<p>These clauses accept only literal or parameter values.</p>
+
+<p>The rows that are returned by <code>LIMIT</code> and <code>OFFSET</code> is unspecified unless these
+operators are used after <code>ORDER BY</code>.</p>
+
+<p><a id="with_clause"></a></p>
+
+<h2 id="with-clause">WITH clause</h2>
+
+<p>The <code>WITH</code> clause binds the results of one or more named subqueries to temporary
+table names. Each introduced table name is visible in subsequent <code>SELECT</code>
+expressions within the same query expression. This includes the following kinds
+of <code>SELECT</code> expressions:</p>
+
+<ul>
+<li>Any <code>SELECT</code> expressions in subsequent <code>WITH</code> bindings</li>
+<li>Top level <code>SELECT</code> expressions in the query expression on both sides of a set
+ operator such as <code>UNION</code></li>
+<li><code>SELECT</code> expressions inside subqueries within the same query expression</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>WITH subQ1 AS (SELECT SchoolID FROM Roster),
+ subQ2 AS (SELECT OpponentID FROM PlayerStats)
+SELECT * FROM subQ1
+UNION ALL
+SELECT * FROM subQ2;</code></pre>
+
+<p>The following are scoping rules for <code>WITH</code> clauses:</p>
+
+<ul>
+<li>Aliases are scoped so that the aliases introduced in a <code>WITH</code> clause are
+ visible only in the later subqueries in the same <code>WITH</code> clause, and in the
+ query under the <code>WITH</code> clause.</li>
+<li>Aliases introduced in the same <code>WITH</code> clause must be unique, but the same
+ alias can be used in multiple <code>WITH</code> clauses in the same query. The local
+ alias overrides any outer aliases anywhere that the local alias is visible.</li>
+</ul>
+
+<p>Beam SQL does not support <code>WITH RECURSIVE</code>.</p>
+
+<p><a name="using_aliases"></a></p>
+
+<h2 id="aliases_2">Aliases</h2>
+
+<p>An alias is a temporary name given to a table, column, or expression present in
+a query. You can introduce explicit aliases in the <code>SELECT</code> list or <code>FROM</code>
+clause.</p>
+
+<p><a id="explicit_alias_syntax"></a></p>
+
+<h3 id="explicit-alias-syntax">Explicit alias syntax</h3>
+
+<p>You can introduce explicit aliases in either the <code>FROM</code> clause or the <code>SELECT</code>
+list.</p>
+
+<p>In a <code>FROM</code> clause, you can introduce explicit aliases for any item, including
+tables, arrays and subqueries, using <code>[AS] alias</code>. The <code>AS</code>
+keyword is optional.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT s.FirstName, s2.SongName
+FROM Singers AS s, (SELECT * FROM Songs) AS s2;</code></pre>
+
+<p>You can introduce explicit aliases for any expression in the <code>SELECT</code> list using
+<code>[AS] alias</code>. The <code>AS</code> keyword is optional.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT s.FirstName AS name, LOWER(s.FirstName) AS lname
+FROM Singers s;</code></pre>
+
+<p><a id="alias_visibility"></a></p>
+
+<h3 id="explicit-alias-visibility">Explicit alias visibility</h3>
+
+<p>After you introduce an explicit alias in a query, there are restrictions on
+where else in the query you can reference that alias. These restrictions on
+alias visibility are the result of Beam SQL's name scoping rules.</p>
+
+<p><a id="from_clause_aliases"></a></p>
+
+<h4 id="from-clause-aliases">FROM clause aliases</h4>
+
+<p>Beam SQL processes aliases in a <code>FROM</code> clause from left to right,
+and aliases are visible only to subsequent path expressions in a <code>FROM</code>
+clause.</p>
+
+<p>Example:</p>
+
+<p>Assume the <code>Singers</code> table had a <code>Concerts</code> column of <code>ARRAY</code> type.</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM Singers AS s, s.Concerts;</code></pre>
+
+<p>Invalid:</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM s.Concerts, Singers AS s; // INVALID.</code></pre>
+
+<p><code>FROM</code> clause aliases are <strong>not</strong> visible to subqueries in the same <code>FROM</code>
+clause. Subqueries in a <code>FROM</code> clause cannot contain correlated references to
+other tables in the same <code>FROM</code> clause.</p>
+
+<p>Invalid:</p>
+
+<pre class="codehilite"><code>SELECT FirstName
+FROM Singers AS s, (SELECT (2020 - ReleaseDate) FROM s) // INVALID.</code></pre>
+
+<p>You can use any column name from a table in the <code>FROM</code> as an alias anywhere in
+the query, with or without qualification with the table name.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT FirstName, s.ReleaseDate
+FROM Singers s WHERE ReleaseDate = 1975;</code></pre>
+
+<p><a id="select-list_aliases"></a></p>
+
+<h4 id="select-list-aliases">SELECT list aliases</h4>
+
+<p>Aliases in the <code>SELECT</code> list are <strong>visible only</strong> to the following clauses:</p>
+
+<ul>
+<li><code>GROUP BY</code> clause</li>
+<li><code>ORDER BY</code> clause</li>
+<li><code>HAVING</code> clause</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT LastName AS last, SingerID
+FROM Singers
+ORDER BY last;</code></pre>
+
+<p><a id="aliases_clauses"></a></p>
+
+<h3 id="explicit-aliases-in-group-by-order-by-and-having-clauses">Explicit aliases in GROUP BY, ORDER BY, and HAVING clauses</h3>
+
+<p>These three clauses, <code>GROUP BY</code>, <code>ORDER BY</code>, and <code>HAVING</code>, can refer to only the
+following values:</p>
+
+<ul>
+<li>Tables in the <code>FROM</code> clause and any of their columns.</li>
+<li>Aliases from the <code>SELECT</code> list.</li>
+</ul>
+
+<p><code>GROUP BY</code> and <code>ORDER BY</code> can also refer to a third group:</p>
+
+<ul>
+<li>Integer literals, which refer to items in the <code>SELECT</code> list. The integer <code>1</code>
+ refers to the first item in the <code>SELECT</code> list, <code>2</code> refers to the second item,
+ etc.</li>
+</ul>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT SingerID AS sid, COUNT(Songid) AS s2id
+FROM Songs
+GROUP BY 1
+ORDER BY 2 DESC LIMIT 10;</code></pre>
+
+<p>The query above is equivalent to:</p>
+
+<pre class="codehilite"><code>SELECT SingerID AS sid, COUNT(Songid) AS s2id
+FROM Songs
+GROUP BY sid
+ORDER BY s2id DESC LIMIT 10;</code></pre>
+
+<p><a id="ambiguous_aliases"></a></p>
+
+<h3 id="ambiguous-aliases">Ambiguous aliases</h3>
+
+<p>Beam SQL provides an error if a name is ambiguous, meaning it can
+resolve to more than one unique object.</p>
+
+<p>Examples:</p>
+
+<p>This query contains column names that conflict between tables, since both
+<code>Singers</code> and <code>Songs</code> have a column named <code>SingerID</code>:</p>
+
+<pre class="codehilite"><code>SELECT SingerID
+FROM Singers, Songs;</code></pre>
+
+<p>This query contains aliases that are ambiguous in the <code>GROUP BY</code> clause because
+they are duplicated in the <code>SELECT</code> list:</p>
+
+<pre class="codehilite"><code>SELECT FirstName AS name, LastName AS name,
+FROM Singers
+GROUP BY name;</code></pre>
+
+<p>Ambiguity between a <code>FROM</code> clause column name and a <code>SELECT</code> list alias in
+<code>GROUP BY</code>:</p>
+
+<pre class="codehilite"><code>SELECT UPPER(LastName) AS LastName
+FROM Singers
+GROUP BY LastName;</code></pre>
+
+<p>The query above is ambiguous and will produce an error because <code>LastName</code> in the
+<code>GROUP BY</code> clause could refer to the original column <code>LastName</code> in <code>Singers</code>, or
+it could refer to the alias <code>AS LastName</code>, whose value is <code>UPPER(LastName)</code>.</p>
+
+<p>The same rules for ambiguity apply to path expressions. Consider the following
+query where <code>table</code> has columns <code>x</code> and <code>y</code>, and column <code>z</code> is of type STRUCT
+and has fields <code>v</code>, <code>w</code>, and <code>x</code>.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT x, z AS T
+FROM table T
+GROUP BY T.x;</code></pre>
+
+<p>The alias <code>T</code> is ambiguous and will produce an error because <code>T.x</code> in the <code>GROUP
+BY</code> clause could refer to either <code>table.x</code> or <code>table.z.x</code>.</p>
+
+<p>A name is <strong>not</strong> ambiguous in <code>GROUP BY</code>, <code>ORDER BY</code> or <code>HAVING</code> if it is both
+a column name and a <code>SELECT</code> list alias, as long as the name resolves to the
+same underlying object.</p>
+
+<p>Example:</p>
+
+<pre class="codehilite"><code>SELECT LastName, BirthYear AS BirthYear
+FROM Singers
+GROUP BY BirthYear;</code></pre>
+
+<p>The alias <code>BirthYear</code> is not ambiguous because it resolves to the same
+underlying column, <code>Singers.BirthYear</code>.</p>
+
+<p><a id="appendix_a_examples_with_sample_data"></a></p>
+<h2 id="appendix-a-examples-with-sample-data">Appendix A: examples with sample data</h2>
+<p><a id="sample_tables"></a></p>
+<h3 id="sample-tables">Sample tables</h3>
+<p>The following three tables contain sample data about athletes, their schools,
+and the points they score during the season. These tables will be used to
+illustrate the behavior of different query clauses.</p>
+<p>Table Roster:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>SchoolID</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The Roster table includes a list of player names (LastName) and the unique ID
+assigned to their school (SchoolID).</p>
+<p>Table PlayerStats:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>OpponentID</th>
+<th>PointsScored</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>51</td>
+<td>3</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>77</td>
+<td>0</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>77</td>
+<td>1</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>52</td>
+<td>4</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>50</td>
+<td>13</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The PlayerStats table includes a list of player names (LastName) and the unique
+ID assigned to the opponent they played in a given game (OpponentID) and the
+number of points scored by the athlete in that game (PointsScored).</p>
+<p>Table TeamMascot:</p>
+<table>
+<thead>
+<tr>
+<th>SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>The TeamMascot table includes a list of unique school IDs (SchoolID) and the
+mascot for that school (Mascot).</p>
+<p><a id="join_types_examples"></a></p>
+<h3 id="join-types_1">JOIN types</h3>
+<p>1) [INNER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>2) FULL [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster FULL JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+<td>NULL</td>
+<td>NULL</td>
+</tr>
+<tr>
+<td>NULL</td>
+<td>NULL</td>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>3) LEFT [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster LEFT JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Eisenhower</td>
+<td>77</td>
+<td>NULL</td>
+<td>NULL</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>4) RIGHT [OUTER] JOIN</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT * FROM Roster RIGHT JOIN TeamMascot
+ON Roster.SchoolID = TeamMascot.SchoolID;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>Roster.SchoolId</th>
+<th>TeamMascot.SchoolId</th>
+<th>Mascot</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>50</td>
+<td>50</td>
+<td>Jaguars</td>
+</tr>
+<tr>
+<td>Davis</td>
+<td>51</td>
+<td>51</td>
+<td>Knights</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>52</td>
+<td>52</td>
+<td>Lakers</td>
+</tr>
+<tr>
+<td>NULL</td>
+<td>NULL</td>
+<td>53</td>
+<td>Mustangs</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<h3 id="group-by-clause_1">GROUP BY clause</h3>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT LastName, SUM(PointsScored)
+FROM PlayerStats
+GROUP BY LastName;</code></pre>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+<th>SUM</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+<td>7</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>13</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="set_operators"></a></p>
+<h3 id="set-operators">Set operators</h3>
+<p><a id="union"></a></p>
+<h4 id="union_1">UNION</h4>
+<p>The <code>UNION</code> operator combines the result sets of two or more <code>SELECT</code> statements
+by pairing columns from the result set of each <code>SELECT</code> statement and vertically
+concatenating them.</p>
+<p>Example:</p>
+<pre class="codehilite"><code>SELECT Mascot AS X, SchoolID AS Y
+FROM TeamMascot
+UNION ALL
+SELECT LastName, PointsScored
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>X</th>
+<th>Y</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Jaguars</td>
+<td>50</td>
+</tr>
+<tr>
+<td>Knights</td>
+<td>51</td>
+</tr>
+<tr>
+<td>Lakers</td>
+<td>52</td>
+</tr>
+<tr>
+<td>Mustangs</td>
+<td>53</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>3</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>0</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+<td>1</td>
+</tr>
+<tr>
+<td>Adams</td>
+<td>4</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+<td>13</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="intersect"></a></p>
+<h4 id="intersect_1">INTERSECT</h4>
+<p>This query returns the last names that are present in both Roster and
+PlayerStats.</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+INTERSECT ALL
+SELECT LastName
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Adams</td>
+</tr>
+<tr>
+<td>Coolidge</td>
+</tr>
+<tr>
+<td>Buchanan</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p><a id="except"></a></p>
+<h4 id="except_1">EXCEPT</h4>
+<p>The query below returns last names in Roster that are <strong>not </strong>present in
+PlayerStats.</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM Roster
+EXCEPT DISTINCT
+SELECT LastName
+FROM PlayerStats;</code></pre>
+<p>Results:</p>
+<table>
+<thead>
+<tr>
+<th>LastName</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>Eisenhower</td>
+</tr>
+<tr>
+<td>Davis</td>
+</tr>
+</tbody>
+</table>
+{:.table}
+<p>Reversing the order of the <code>SELECT</code> statements will return last names in
+PlayerStats that are <strong>not</strong> present in Roster:</p>
+<pre class="codehilite"><code>SELECT LastName
+FROM PlayerStats
+EXCEPT DISTINCT
+SELECT LastName
+FROM Roster;</code></pre>
+<p>Results:</p>
+<pre class="codehilite"><code>(empty)</code></pre>
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/string-functions.md b/website/src/documentation/dsls/sql/zetasql/string-functions.md
new file mode 100644
index 0000000..0d3cb70
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/string-functions.md
@@ -0,0 +1,657 @@
+---
+layout: section
+title: "Beam ZetaSQL string functions"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/string-functions/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL string functions
+
+This page documents the ZetaSQL string functions supported by Beam ZetaSQL.
+
+These string functions work on STRING data. STRING values must be well-formed UTF-8. All string comparisons are done byte-by-byte, without regard to Unicode
+canonical equivalence.
+
+| Operator syntax | Description |
+| ---- | ---- |
+| [CHAR_LENGTH(value)](#char_length) | Returns the length of the string in characters |
+| [CHARACTER_LENGTH(value)](#character_length) | Synonym for CHAR_LENGTH |
+| [CONCAT(value1[, ...])](#concat) | Concatenates up to five values into a single result |
+| [ENDS_WITH(value1, value2)](#ends_with) | Returns TRUE if the second value is a suffix of the first |
+| [LTRIM(value1[, value2])](#ltrim) | Identical to TRIM, but only removes leading characters. |
+| [REPLACE(original_value, from_value, to_value)](#replace) | Replaces all occurrences of `from_value` with `to_value` in `original_value` |
+| [REVERSE(value)](#reverse) | Returns the reverse of the input string |
+| [RTRIM(value1[, value2])](#rtrim) | Identical to TRIM, but only removes trailing characters |
+| [STARTS_WITH(value1, value2)](#starts_with) | Returns TRUE if the second value is a prefix of the first. |
+| [SUBSTR(value, position[, length])](#substr) | Returns a substring of the supplied value |
+| [TRIM(value1[, value2])](#trim) | Removes all leading and trailing characters that match `value2` |
+{:.table}
+
+## CHAR_LENGTH
+
+```
+CHAR_LENGTH(value)
+```
+
+**Description**
+
+Returns the length of the STRING in characters.
+
+**Return type**
+
+INT64
+
+
+**Examples**
+
+```
+
+Table example:
+
++----------------+
+| characters |
++----------------+
+| абвгд |
++----------------+
+
+SELECT
+ characters,
+ CHAR_LENGTH(characters) AS char_length_example
+FROM example;
+
++------------+---------------------+
+| characters | char_length_example |
++------------+---------------------+
+| абвгд | 5 |
++------------+---------------------+
+
+```
+## CHARACTER_LENGTH
+```
+CHARACTER_LENGTH(value)
+```
+
+**Description**
+
+Synonym for [CHAR_LENGTH](#char_length).
+
+**Return type**
+
+INT64
+
+
+**Examples**
+
+```
+
+Table example:
+
++----------------+
+| characters |
++----------------+
+| абвгд |
++----------------+
+
+SELECT
+ characters,
+ CHARACTER_LENGTH(characters) AS char_length_example
+FROM example;
+
++------------+---------------------+
+| characters | char_length_example |
++------------+---------------------+
+| абвгд | 5 |
++------------+---------------------+
+
+```
+
+
+## CONCAT
+```
+CONCAT(value1[, ...])
+```
+
+**Description**
+
+Concatenates up to five values into a single
+result.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table Employees:
+
++-------------+-----------+
+| first_name | last_name |
++-------------+-----------+
+| John | Doe |
+| Jane | Smith |
+| Joe | Jackson |
++-------------+-----------+
+
+SELECT
+ CONCAT(first_name, " ", last_name)
+ AS full_name
+FROM Employees;
+
++---------------------+
+| full_name |
++---------------------+
+| John Doe |
+| Jane Smith |
+| Joe Jackson |
++---------------------+
+```
+
+## ENDS_WITH
+```
+ENDS_WITH(value1, value2)
+```
+
+**Description**
+
+Takes two values. Returns TRUE if the second value is a
+suffix of the first.
+
+**Return type**
+
+BOOL
+
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ ENDS_WITH(item, "e") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| True |
+| False |
+| True |
++---------+
+
+```
+
+## LTRIM
+```
+LTRIM(value1[, value2])
+```
+
+**Description**
+
+Identical to [TRIM](#trim), but only removes leading characters.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ CONCAT("#", LTRIM(item), "#") as example
+FROM items;
+
++-------------+
+| example |
++-------------+
+| #apple # |
+| #banana # |
+| #orange # |
++-------------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ LTRIM(item, "*") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| apple*** |
+| banana*** |
+| orange*** |
++-----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| xxxapplexxx |
+| yyybananayyy |
+| zzzorangezzz |
+| xyzpearzyz |
++----------------+
+
+SELECT
+ LTRIM(item, "xyz") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| applexxx |
+| bananayyy |
+| orangezzz |
+| pearxyz |
++-----------+
+```
+
+## REPLACE
+```
+REPLACE(original_value, from_value, to_value)
+```
+
+**Description**
+
+Replaces all occurrences of `from_value` with `to_value` in `original_value`.
+If `from_value` is empty, no replacement is made.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```sql
+
++--------------------+
+| dessert |
++--------------------+
+| apple pie |
+| blackberry pie |
+| cherry pie |
++--------------------+
+
+SELECT
+ REPLACE (dessert, "pie", "cobbler") as example
+FROM desserts;
+
++--------------------+
+| example |
++--------------------+
+| apple cobbler |
+| blackberry cobbler |
+| cherry cobbler |
++--------------------+
+```
+
+## REVERSE
+
+```
+REVERSE(value)
+```
+
+**Description**
+
+Returns the reverse of the input STRING.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+WITH example AS (
+ SELECT "foo" AS sample_string UNION ALL
+ SELECT "абвгд" AS sample_string
+)
+SELECT
+ sample_string,
+ REVERSE(sample_string) AS reverse_string
+FROM example;
+
++---------------+----------------+
+| sample_string | reverse_string |
++---------------+----------------+
+| foo | oof |
+| абвгд | дгвба |
++---------------+----------------+
+```
+
+## RTRIM
+```
+RTRIM(value1[, value2])
+```
+
+**Description**
+
+Identical to [TRIM](#trim), but only removes trailing characters.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ RTRIM(item, "*") as example
+FROM items;
+
++-----------+
+| example |
++-----------+
+| ***apple |
+| ***banana |
+| ***orange |
++-----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| applexxx |
+| bananayyy |
+| orangezzz |
+| pearxyz |
++----------------+
+
+SELECT
+ RTRIM(item, "xyz") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
+| pear |
++---------+
+```
+
+## STARTS_WITH
+```
+STARTS_WITH(value1, value2)
+```
+
+**Description**
+
+Takes two values. Returns TRUE if the second value is a prefix
+of the first.
+
+**Return type**
+
+BOOL
+
+**Examples**
+
+```
+
+SELECT
+ STARTS_WITH(item, "b") as example
+FROM (
+ SELECT "foo" as item
+ UNION ALL SELECT "bar" as item
+ UNION ALL SELECT "baz" as item) AS items;
+
+
++---------+
+| example |
++---------+
+| False |
+| True |
+| True |
++---------+
+```
+## SUBSTR
+```
+SUBSTR(value, position[, length])
+```
+
+**Description**
+
+Returns a substring of the supplied value. The
+`position` argument is an integer specifying the starting position of the
+substring, with position = 1 indicating the first character or byte. The
+`length` argument is the maximum number of characters for STRING arguments.
+
+If `position` is negative, the function counts from the end of `value`,
+with -1 indicating the last character.
+
+If `position` is a position off the left end of the
+STRING (`position` = 0 or
+`position` < `-LENGTH(value)`), the function starts
+from position = 1. If `length` exceeds the length of
+`value`, returns fewer than `length` characters.
+
+If `length` is less than 0, the function returns an error.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, 2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| pple |
+| anana |
+| range |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, 2, 2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| pp |
+| an |
+| ra |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ SUBSTR(item, -2) as example
+FROM items;
+
++---------+
+| example |
++---------+
+| le |
+| na |
+| ge |
++---------+
+```
+## TRIM
+```
+TRIM(value1[, value2])
+```
+
+**Description**
+
+Removes all leading and trailing characters that match `value2`. If `value2` is
+not specified, all leading and trailing whitespace characters (as defined by the
+Unicode standard) are removed.
+
+If `value2` contains more than one character, the function removes all leading
+or trailing characters contained in `value2`.
+
+**Return type**
+
+STRING
+
+**Examples**
+
+```
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| apple |
+| banana |
+| orange |
++----------------+
+
+SELECT
+ CONCAT("#", TRIM(item), "#") as example
+FROM items;
+
++----------+
+| example |
++----------+
+| #apple# |
+| #banana# |
+| #orange# |
++----------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| ***apple*** |
+| ***banana*** |
+| ***orange*** |
++----------------+
+
+SELECT
+ TRIM(item, "*") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
++---------+
+
+
+Table items:
+
++----------------+
+| item |
++----------------+
+| xxxapplexxx |
+| yyybananayyy |
+| zzzorangezzz |
+| xyzpearxyz |
++----------------+
+
+SELECT
+ TRIM(item, "xyz") as example
+FROM items;
+
++---------+
+| example |
++---------+
+| apple |
+| banana |
+| orange |
+| pear |
++---------+
+```
\ No newline at end of file
diff --git a/website/src/documentation/dsls/sql/zetasql/syntax.md b/website/src/documentation/dsls/sql/zetasql/syntax.md
new file mode 100644
index 0000000..5a0dec6
--- /dev/null
+++ b/website/src/documentation/dsls/sql/zetasql/syntax.md
@@ -0,0 +1,34 @@
+---
+layout: section
+title: "Beam ZetaSQL function call rules"
+section_menu: section-menu/sdks.html
+permalink: /documentation/dsls/sql/zetasql/syntax/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Beam ZetaSQL function call rules
+
+The following rules apply to all functions unless explicitly indicated otherwise in the function description:
+
++ For functions that accept numeric types, if one operand is a floating point
+ operand and the other operand is another numeric type, both operands are
+ converted to FLOAT64 before the function is
+ evaluated.
++ If an operand is `NULL`, the result is `NULL`, with the exception of the
+ IS operator.
+
++ For functions that are time zone sensitive (as indicated in the function
+ description), the default time zone, UTC, is used if a time
+ zone is not specified.
\ No newline at end of file
diff --git a/website/src/documentation/execution-model.md b/website/src/documentation/execution-model.md
deleted file mode 100644
index 0e191e7..0000000
--- a/website/src/documentation/execution-model.md
+++ /dev/null
@@ -1,210 +0,0 @@
----
-layout: section
-title: "Beam Execution Model"
-section_menu: section-menu/documentation.html
-permalink: /documentation/execution-model/
----
-<!--
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Apache Beam Execution Model
-
-The Beam model allows runners to execute your pipeline in different ways. You
-may observe various effects as a result of the runner’s choices. This page
-describes these effects so you can better understand how Beam pipelines execute.
-
-## Processing of elements
-
-The serialization and communication of elements between machines is one of the
-most expensive operations in a distributed execution of your pipeline. Avoiding
-this serialization may require re-processing elements after failures or may
-limit the distribution of output to other machines.
-
-### Serialization and communication
-
-The runner might serialize elements between machines for communication purposes
-and for other reasons such as persistence.
-
-A runner may decide to transfer elements between transforms in a variety of
-ways, such as:
-
-1. Routing elements to a worker for processing as part of a grouping operation.
- This may involve serializing elements and grouping or sorting them by their
- key.
-1. Redistributing elements between workers to adjust parallelism. This may
- involve serializing elements and communicating them to other workers.
-1. Using the elements in a side input to a `ParDo`. This may require
- serializing the elements and broadcasting them to all the workers executing
- the `ParDo`.
-1. Passing elements between transforms that are running on the same worker.
- This may allow the runner to avoid serializing elements; instead, the runner
- can just pass the elements in memory.
-
-Some situations where the runner may serialize and persist elements are:
-
-1. When used as part of a stateful `DoFn`, the runner may persist values to some
- state mechanism.
-1. When committing the results of processing, the runner may persist the outputs
- as a checkpoint.
-
-### Bundling and persistence
-
-Beam pipelines often focus on "[embarassingly parallel](https://en.wikipedia.org/wiki/embarrassingly_parallel)"
-problems. Because of this, the APIs emphasize processing elements in parallel,
-which makes it difficult to express actions like "assign a sequence number to
-each element in a PCollection". This is intentional as such algorithms are much
-more likely to suffer from scalability problems.
-
-Processing all elements in parallel also has some drawbacks. Specifically, it
-makes it impossible to batch any operations, such as writing elements to a sink
-or checkpointing progress during processing.
-
-Instead of processing all elements simultaneously, the elements in a
-`PCollection` are processed in _bundles_. The division of the collection into
-bundles is arbitrary and selected by the runner. This allows the runner to
-choose an appropriate middle-ground between persisting results after every
-element, and having to retry everything if there is a failure. For example, a
-streaming runner may prefer to process and commit small bundles, and a batch
-runner may prefer to process larger bundles.
-
-## Failures and parallelism within and between transforms {#parallelism}
-
-In this section, we discuss how elements in the input collection are processed
-in parallel, and how transforms are retried when failures occur.
-
-### Data-parallelism within one transform {#data-parallelism}
-
-When executing a single `ParDo`, a runner might divide an example input
-collection of nine elements into two bundles as shown in figure 1.
-
-![Bundle A contains five elements. Bundle B contains four elements.](
- {{ "/images/execution_model_bundling.svg" | prepend: site.baseurl }})
-
-*Figure 1: A runner divides an input collection into two bundles.*
-
-When the `ParDo` executes, workers may process the two bundles in parallel as
-shown in figure 2.
-
-![Two workers process the two bundles in parallel. Worker one processes bundle
- A. Worker two processes bundle B.](
- {{ "/images/execution_model_bundling_gantt.svg" | prepend: site.baseurl }})
-
-*Figure 2: Two workers process the two bundles in parallel.*
-
-Since elements cannot be split, the maximum parallelism for a transform depends
-on the number of elements in the collection. In figure 3, the input collection
-has nine elements, so the maximum parallelism is nine.
-
-![Nine workers process a nine element input collection in parallel.](
- {{ "/images/execution_model_bundling_gantt_max.svg" | prepend: site.baseurl }})
-
-*Figure 3: Nine workers process a nine element input collection in parallel.*
-
-Note: Splittable ParDo allows splitting the processing of a single input across
-multiple bundles. This feature is a work in progress.
-
-### Dependent-parallelism between transforms {#dependent-parallellism}
-
-`ParDo` transforms that are in sequence may be _dependently parallel_ if the
-runner chooses to execute the consuming transform on the producing transform's
-output elements without altering the bundling. In figure 4, `ParDo1` and
-`ParDo2` are _dependently parallel_ if the output of `ParDo1` for a given
-element must be processed on the same worker.
-
-![ParDo1 processes an input collection that contains bundles A and B. ParDo2 then
- processes the output collection from ParDo1, which contains bundles C and D.](
- {{ "/images/execution_model_bundling_multi.svg" | prepend: site.baseurl }})
-
-*Figure 4: Two transforms in sequence and their corresponding input collections.*
-
-Figure 5 shows how these dependently parallel transforms might execute. The
-first worker executes `ParDo1` on the elements in bundle A (which results in
-bundle C), and then executes `ParDo2` on the elements in bundle C. Similarly,
-the second worker executes `ParDo1` on the elements in bundle B (which results
-in bundle D), and then executes `ParDo2` on the elements in bundle D.
-
-![Worker one executes ParDo1 on bundle A and Pardo2 on bundle C. Worker two
- executes ParDo1 on bundle B and ParDo2 on bundle D.](
- {{ "/images/execution_model_bundling_multi_gantt.svg" | prepend: site.baseurl }})
-
-*Figure 5: Two workers execute dependently parallel ParDo transforms.*
-
-Executing transforms this way allows a runner to avoid redistributing elements
-between workers, which saves on communication costs. However, the maximum parallelism
-now depends on the maximum parallelism of the first of the dependently parallel
-steps.
-
-### Failures within one transform
-
-If processing of an element within a bundle fails, the entire bundle fails. The
-elements in the bundle must be retried (otherwise the entire pipeline fails),
-although they do not need to be retried with the same bundling.
-
-For this example, we will use the `ParDo` from figure 1 that has an input
-collection with nine elements and is divided into two bundles.
-
-In figure 6, the first worker successfully processes all five elements in bundle
-A. The second worker processes the four elements in bundle B: the first two
-elements were successfully processed, the third element’s processing failed, and
-there is one element still awaiting processing.
-
-We see that the runner retries all elements in bundle B and the processing
-completes successfully the second time. Note that the retry does not necessarily
-happen on the same worker as the original processing attempt, as shown in the
-figure.
-
-![Worker two fails to process an element in bundle B. Worker one finishes
- processing bundle A and then successfully retries to execute bundle B.](
- {{ "/images/execution_model_failure_retry.svg" | prepend: site.baseurl }})
-
-*Figure 6: The processing of an element within bundle B fails, and another worker
-retries the entire bundle.*
-
-Because we encountered a failure while processing an element in the input
-bundle, we had to reprocess _all_ of the elements in the input bundle. This means
-the runner must throw away the entire output bundle since all of the results it
-contains will be recomputed.
-
-Note that if the failed transform is a `ParDo`, then the `DoFn` instance is torn
-down and abandoned.
-
-### Coupled failure: Failures between transforms {#coupled-failure}
-
-If a failure to process an element in `ParDo2` causes `ParDo1` to re-execute,
-these two steps are said to be _co-failing_.
-
-For this example, we will use the two `ParDo`s from figure 4.
-
-In figure 7, worker two successfully executes `ParDo1` on all elements in bundle
-B. However, the worker fails to process an element in bundle D, so `ParDo2`
-fails (shown as the red X). As a result, the runner must discard and recompute
-the output of `ParDo2`. Because the runner was executing `ParDo1` and `ParDo2`
-together, the output bundle from `ParDo1` must also be thrown away, and all
-elements in the input bundle must be retried. These two `ParDo`s are co-failing.
-
-![Worker two fails to process en element in bundle D, so all elements in both
- bundle B and bundle D must be retried.](
- {{ "/images/execution_model_bundling_coupled_failure.svg" | prepend: site.baseurl }})
-
-*Figure 7: Processing of an element within bundle D fails, so all elements in
-the input bundle are retried.*
-
-Note that the retry does not necessarily have the same processing time as the
-original attempt, as shown in the diagram.
-
-All `DoFns` that experience coupled failures are terminated and must be torn
-down since they aren’t following the normal `DoFn` lifecycle .
-
-Executing transforms this way allows a runner to avoid persisting elements
-between transforms, saving on persistence costs.
diff --git a/website/src/documentation/index.md b/website/src/documentation/index.md
index 3587378..5000ea5 100644
--- a/website/src/documentation/index.md
+++ b/website/src/documentation/index.md
@@ -29,8 +29,8 @@
Learn about the Beam Programming Model and the concepts common to all Beam SDKs and Runners.
-* The [Programming Guide]({{ site.baseurl }}/documentation/programming-guide/) introduces all the key Beam concepts.
-* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/execution-model/) to better understand how pipelines execute.
+* Read the [Programming Guide]({{ site.baseurl }}/documentation/programming-guide/), which introduces all the key Beam concepts.
+* Learn about Beam's [execution model]({{ site.baseurl }}/documentation/runtime/model) to better understand how pipelines execute.
* Visit [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources) for some of our favorite articles and talks about Beam.
## Pipeline Fundamentals
@@ -61,9 +61,10 @@
* [GearpumpRunner]({{ site.baseurl }}/documentation/runners/gearpump/): Runs on [Apache Gearpump (incubating)](http://gearpump.apache.org).
* [SamzaRunner]({{ site.baseurl }}/documentation/runners/samza/): Runs on [Apache Samza](http://samza.apache.org).
* [NemoRunner]({{ site.baseurl }}/documentation/runners/nemo/): Runs on [Apache Nemo](http://nemo.apache.org).
+* [JetRunner]({{ site.baseurl }}/documentation/runners/jet/): Runs on [Hazelcast Jet](https://jet.hazelcast.org/).
### Choosing a Runner
Beam is designed to enable pipelines to be portable across different runners. However, given every runner has different capabilities, they also have different abilities to implement the core concepts in the Beam model. The [Capability Matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/) provides a detailed comparison of runner functionality.
-Once you have chosen which runner to use, see that runner's page for more information about any initial runner-specific setup as well as any required or optional `PipelineOptions` for configuring it's execution. You may also want to refer back to the Quickstart for [Java]({{ site.baseurl }}/get-started/quickstart-java), [Python]({{ site.baseurl }}/get-started/quickstart-py) or [Go]({{ site.baseurl }}/get-started/quickstart-go) for instructions on executing the sample WordCount pipeline.
+Once you have chosen which runner to use, see that runner's page for more information about any initial runner-specific setup as well as any required or optional `PipelineOptions` for configuring its execution. You may also want to refer back to the Quickstart for [Java]({{ site.baseurl }}/get-started/quickstart-java), [Python]({{ site.baseurl }}/get-started/quickstart-py) or [Go]({{ site.baseurl }}/get-started/quickstart-go) for instructions on executing the sample WordCount pipeline.
diff --git a/website/src/documentation/io/built-in-google-bigquery.md b/website/src/documentation/io/built-in-google-bigquery.md
index 5f1bab9..aedc73d 100644
--- a/website/src/documentation/io/built-in-google-bigquery.md
+++ b/website/src/documentation/io/built-in-google-bigquery.md
@@ -163,6 +163,43 @@
disposition](#create-disposition) of `CREATE_NEVER`. [Creating a table
schema](#creating-a-table-schema) covers schemas in more detail.
+### Data types
+
+BigQuery supports the following data types: STRING, BYTES, INTEGER, FLOAT,
+NUMERIC, BOOLEAN, TIMESTAMP, DATE, TIME, DATETIME and GEOGRAPHY.
+All possible values are described at [https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types).
+BigQueryIO allows you to use all of these data types. The following example
+shows the correct format for data types used when reading from and writing to
+BigQuery:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:BigQueryDataTypes
+%}```
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:model_bigqueryio_data_types
+%}```
+
+<!-- Java specific -->
+
+{:.language-java}
+As of Beam 2.7.0, the NUMERIC data type is supported. This data type supports
+high-precision decimal numbers (precision of 38 digits, scale of 9 digits).
+The GEOGRAPHY data type works with Well-Known Text (See [https://en.wikipedia.org/wiki/Well-known_text](https://en.wikipedia.org/wiki/Well-known_text)
+format for reading and writing to BigQuery.
+BigQuery IO requires values of BYTES datatype to be encoded using base64
+encoding when writing to BigQuery. When bytes are read from BigQuery they are
+returned as base64-encoded strings.
+
+<!-- Python specific -->
+
+{:.language-py}
+As of Beam 2.7.0, the NUMERIC data type is supported. This data type supports
+high-precision decimal numbers (precision of 38 digits, scale of 9 digits).
+The GEOGRAPHY data type works with Well-Known Text (See [https://en.wikipedia.org/wiki/Well-known_text](https://en.wikipedia.org/wiki/Well-known_text)
+format for reading and writing to BigQuery.
+BigQuery IO requires values of BYTES datatype to be encoded using base64
+encoding when writing to BigQuery. When bytes are read from BigQuery they are
+returned as base64-encoded bytes.
## Reading from BigQuery
@@ -278,10 +315,10 @@
Beam's support for the BigQuery Storage API has the following limitations:
* The SDK for Python does not support the BigQuery Storage API.
-* You must read from a table. Reading with a query string is not currently
- supported.
* Dynamic work re-balancing is not currently supported. As a result, reads might
be less efficient in the presence of stragglers.
+* SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
+ can only read from a table.
Because this is currently a Beam experimental feature, export based reads are
recommended for production jobs.
@@ -302,7 +339,7 @@
you must also specify a [TableReadOptions](https://googleapis.github.io/google-cloud-java/google-api-grpc/apidocs/index.html?com/google/cloud/bigquery/storage/v1beta1/ReadOptions.TableReadOptions.html)
proto using the [withReadOptions](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.TypedRead.html#withReadOptions-com.google.cloud.bigquery.storage.v1beta1.ReadOptions.TableReadOptions-) method.
-The following code snippet is from the [BigQueryTornadoes
+The following code snippet reads from a table. This example is from the [BigQueryTornadoes
example](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/cookbook/BigQueryTornadoes.java).
When the example's read method option is set to `DIRECT_READ`, the pipeline uses
the BigQuery Storage API and column projection to read public samples of weather
@@ -326,6 +363,14 @@
# The SDK for Python does not support the BigQuery Storage API.
```
+The following code snippet reads with a query string.
+
+```java
+// Snippet not yet available (BEAM-7034).
+```
+```py
+# The SDK for Python does not support the BigQuery Storage API.
+```
## Writing to BigQuery
diff --git a/website/src/documentation/io/built-in-hadoop.md b/website/src/documentation/io/built-in-hadoop.md
index fd330ec..09fcd7f 100644
--- a/website/src/documentation/io/built-in-hadoop.md
+++ b/website/src/documentation/io/built-in-hadoop.md
@@ -186,13 +186,13 @@
To read data from Elasticsearch, use `EsInputFormat`, which needs following properties to be set:
```java
-Configuration elasticSearchConf = new Configuration();
-elasticSearchConf.set("es.nodes", ElasticsearchHostIp);
-elasticSearchConf.set("es.port", "9200");
-elasticSearchConf.set("es.resource", "ElasticIndexName/ElasticTypeName");
-elasticSearchConf.setClass("key.class", org.apache.hadoop.io.Text Text.class, Object.class);
-elasticSearchConf.setClass("value.class", org.elasticsearch.hadoop.mr.LinkedMapWritable LinkedMapWritable.class, Object.class);
-elasticSearchConf.setClass("mapreduce.job.inputformat.class", org.elasticsearch.hadoop.mr.EsInputFormat EsInputFormat.class, InputFormat.class);
+Configuration elasticsearchConf = new Configuration();
+elasticsearchConf.set("es.nodes", ElasticsearchHostIp);
+elasticsearchConf.set("es.port", "9200");
+elasticsearchConf.set("es.resource", "ElasticIndexName/ElasticTypeName");
+elasticsearchConf.setClass("key.class", org.apache.hadoop.io.Text Text.class, Object.class);
+elasticsearchConf.setClass("value.class", org.elasticsearch.hadoop.mr.LinkedMapWritable LinkedMapWritable.class, Object.class);
+elasticsearchConf.setClass("mapreduce.job.inputformat.class", org.elasticsearch.hadoop.mr.EsInputFormat EsInputFormat.class, InputFormat.class);
```
```py
@@ -203,7 +203,7 @@
```java
PCollection<KV<Text, LinkedMapWritable>> elasticData = p.apply("read",
- HadoopFormatIO.<Text, LinkedMapWritable>read().withConfiguration(elasticSearchConf));
+ HadoopFormatIO.<Text, LinkedMapWritable>read().withConfiguration(elasticsearchConf));
```
```py
diff --git a/website/src/documentation/io/built-in-hcatalog.md b/website/src/documentation/io/built-in-hcatalog.md
index e28890c..55aaf71 100644
--- a/website/src/documentation/io/built-in-hcatalog.md
+++ b/website/src/documentation/io/built-in-hcatalog.md
@@ -1,6 +1,6 @@
---
layout: section
-title: "Apache HCatalog InputFormat IO"
+title: "Apache HCatalog I/O connector"
section_menu: section-menu/documentation.html
permalink: /documentation/io/built-in/hcatalog/
---
@@ -67,7 +67,7 @@
### Using older versions of HCatalog (1.x)
-`HCatalogIO` is build for Apache HCatalog versions 2 and up and will not work out of the box for older versions of HCatalog.
+`HCatalogIO` is built for Apache HCatalog versions 2 and up and will not work out of the box for older versions of HCatalog.
The following illustrates a workaround to work with Hive 1.1.
Include the following Hive 1.2 jars in the über jar you build.
diff --git a/website/src/documentation/io/built-in-parquet.md b/website/src/documentation/io/built-in-parquet.md
new file mode 100644
index 0000000..4cac171
--- /dev/null
+++ b/website/src/documentation/io/built-in-parquet.md
@@ -0,0 +1,148 @@
+---
+layout: section
+title: "Apache Parquet I/O connector"
+section_menu: section-menu/documentation.html
+permalink: /documentation/io/built-in/parquet/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+[Built-in I/O Transforms]({{site.baseurl}}/documentation/io/built-in/)
+
+# Apache Parquet I/O connector
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+The Beam SDKs include built-in transforms that can read data from and write data
+to [Apache Parquet](https://parquet.apache.org) files.
+
+## Before you start
+
+<!-- Java specific -->
+
+{:.language-java}
+To use ParquetIO, add the Maven artifact dependency to your `pom.xml` file.
+
+```java
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-parquet</artifactId>
+ <version>{{ site.release_latest }}</version>
+</dependency>
+```
+
+{:.language-java}
+Additional resources:
+
+{:.language-java}
+* [ParquetIO source code](https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java)
+* [ParquetIO Javadoc](https://beam.apache.org/releases/javadoc/{{ site.release_latest }}/org/apache/beam/sdk/io/parquet/ParquetIO.html)
+
+<!-- Python specific -->
+
+{:.language-py}
+ParquetIO comes preinstalled with the Apache Beam python sdk..
+
+{:.language-py}
+Additional resources:
+
+{:.language-py}
+* [ParquetIO source code](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py)
+* [ParquetIO Pydoc](https://beam.apache.org/releases/pydoc/{{ site.release_latest }}/apache_beam.io.parquetio.html)
+
+{:.language-java}
+#### Using ParquetIO with Spark before 2.4
+
+{:.language-java}
+`ParquetIO` depends on an API introduced in Apache Parquet 1.10.0. **Spark 2.4.x is compatible and no additional steps are necessary**. Older versions of Spark will not work out of the box since a pre-installed version of Parquet libraries will take precedence during execution. The following workaround should be applied.
+
+{:.language-java}
+> **Note**: The following technique allows you to execute your pipeline with `ParquetIO` correctly.
+> The Parquet files that are consumed or generated by this Beam connector should remain interoperable with the other tools on your cluster.
+
+{:.language-java}
+Include the Parquet artifact normally and ensure that it brings in the correct version of Parquet as a transitive dependency.
+
+{:.language-java}
+```
+<dependency>
+ <groupId>org.apache.beam</groupId>
+ <artifactId>beam-sdks-java-io-parquet</artifactId>
+ <version>${beam.version}</version>
+</dependency>
+```
+
+{:.language-java}
+Relocate the following packages:
+
+{:.language-java}
+```
+<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <createDependencyReducedPom>false</createDependencyReducedPom>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <shadedArtifactAttached>true</shadedArtifactAttached>
+ <shadedClassifierName>shaded</shadedClassifierName>
+ <relocations>
+ <relocation>
+ <pattern>org.apache.parquet</pattern>
+ <shadedPattern>shaded.org.apache.parquet</shadedPattern>
+ </relocation>
+ <!-- Some packages are shaded already, and on the original spark classpath. Shade them more. -->
+ <relocation>
+ <pattern>shaded.parquet</pattern>
+ <shadedPattern>reshaded.parquet</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.avro</pattern>
+ <shadedPattern>shaded.org.apache.avro</shadedPattern>
+ </relocation>
+ </relocations>
+ <transformers>
+ <transformer
+ implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+</plugin>
+```
+
+{:.language-java}
+This technique has been tested to work on Spark 2.2.3, Spark 2.3.3 and Spark 2.4.3 (although it is optional for Spark 2.4+).
diff --git a/website/src/documentation/io/built-in.md b/website/src/documentation/io/built-in.md
index b612712..a11a2e8 100644
--- a/website/src/documentation/io/built-in.md
+++ b/website/src/documentation/io/built-in.md
@@ -42,9 +42,7 @@
<p><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java">TFRecordIO</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java">XmlIO</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java">TikaIO</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java">ParquetIO</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java">RabbitMqIO</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java">SqsIO</a></p>
+ <p><a href="{{site.baseurl}}/documentation/io/built-in/parquet">ParquetIO</a></p>
</td>
<td>
<p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis">Amazon Kinesis</a></p>
@@ -53,6 +51,8 @@
<p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub">Google Cloud Pub/Sub</a></p>
<p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/jms">JMS</a></p>
<p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/mqtt">MQTT</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java">RabbitMqIO</a></p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java">SqsIO</a></p>
</td>
<td>
<p><a href="https://github.com/apache/beam/tree/master/sdks/java/io/cassandra">Apache Cassandra</a></p>
@@ -76,18 +76,18 @@
<td>
<p>Beam Python supports Apache HDFS, Google Cloud Storage, and local filesystems.</p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">avroio</a></p>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py">parquetio</a></p>
+ <p><a href="{{site.baseurl}}/documentation/io/built-in/parquet">parquetio.py</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">textio</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/tfrecordio.py">tfrecordio</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/vcfio.py">vcfio</a></p>
</td>
<td>
- <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/pubsub.py">Google Cloud Pub/Sub</a></p>
</td>
<td>
<p><a href="{{site.baseurl}}/documentation/io/built-in/google-bigquery/">Google BigQuery</a></p>
<p><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/io/gcp/datastore">Google Cloud Datastore</a></p>
<p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigtableio.py">Google Cloud Bigtable</a> (Write)</p>
+ <p><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/mongodbio.py">MongoDB</a></p>
</td>
</tr>
<tr>
diff --git a/website/src/documentation/io/testing.md b/website/src/documentation/io/testing.md
index b00d6df..1e945ae 100644
--- a/website/src/documentation/io/testing.md
+++ b/website/src/documentation/io/testing.md
@@ -141,117 +141,20 @@
However, when working locally, there is no requirement to use Kubernetes. All of the test infrastructure allows you to pass in connection info, so developers can use their preferred hosting infrastructure for local development.
-### Running integration tests {#running-integration-tests}
+### Running integration tests on your machine {#running-integration-tests-on-your-machine}
-The high level steps for running an integration test are:
+You can always run the IO integration tests on your own machine. The high level steps for running an integration test are:
1. Set up the data store corresponding to the test being run.
1. Run the test, passing it connection info from the just created data store.
1. Clean up the data store.
-Since setting up data stores and running the tests involves a number of steps, and we wish to time these tests when running performance benchmarks, we use PerfKit Benchmarker to manage the process end to end. With a single command, you can go from an empty Kubernetes cluster to a running integration test.
-However, **PerfKit Benchmarker is not required for running integration tests**. Therefore, we have listed the steps for both using PerfKit Benchmarker, and manually running the tests below.
-
-
-#### Using PerfKit Benchmarker {#using-perfkit-benchmarker}
-
-Prerequisites:
-1. [Install PerfKit Benchmarker](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker)
-1. Have a running Kubernetes cluster you can connect to locally using kubectl. We recommend using Google Kubernetes Engine - it's proven working for all the use cases we tested.
-
-You won’t need to invoke PerfKit Benchmarker directly. Run `./gradlew performanceTest` task in project's root directory, passing kubernetes scripts of your choice (located in .test_infra/kubernetes directory). It will setup PerfKitBenchmarker for you.
-
-Example run with the [Direct]({{ site.baseurl }}/documentation/runners/direct/) runner:
-```
-./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml" -DintegrationTestRunner=direct
-```
-
-
-Example run with the [Google Cloud Dataflow]({{ site.baseurl }}/documentation/runners/dataflow/) runner:
-```
-./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=1000", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/jdbc/ -DintegrationTest=org.apache.beam.sdk.io.jdbc.JdbcIOIT -DkubernetesScripts="/Users/me/beam/.test-infra/kubernetes/postgres/postgres-service-for-local-dev.yml" -DbeamITOptions="/Users/me/beam/.test-infra/kubernetes/postgres/pkb-config-local.yml" -DintegrationTestRunner=dataflow
-```
-
-Example run with the HDFS filesystem and Cloud Dataflow runner:
-
-```
-./gradlew performanceTest -DpkbLocation="/Users/me/PerfKitBenchmarker/pkb.py" -DintegrationTestPipelineOptions='["--numberOfRecords=100000", "--project=GOOGLE_CLOUD_PROJECT", "--tempRoot=GOOGLE_STORAGE_BUCKET"]' -DitModule=sdks/java/io/file-based-io-tests/ -DintegrationTest=org.apache.beam.sdk.io.text.TextIOIT -DkubernetesScripts=".test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster.yml,.test-infra/kubernetes/hadoop/LargeITCluster/hdfs-multi-datanode-cluster-for-local-dev.yml" -DbeamITOptions=".test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml" -DintegrationTestRunner=dataflow -DbeamExtraProperties='[filesystem=hdfs]'
-```
-
-NOTE: When using Direct runner along with HDFS cluster, please set `export HADOOP_USER_NAME=root` before runnning `performanceTest` task.
-
-Parameter descriptions:
-
-
-<table class="table">
- <thead>
- <tr>
- <td>
- <strong>Option</strong>
- </td>
- <td>
- <strong>Function</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>-DpkbLocation
- </td>
- <td>Path to PerfKit Benchmarker project.
- </td>
- </tr>
- <tr>
- <td>-DintegrationTestPipelineOptions
- </td>
- <td>Passes pipeline options directly to the test being run. Note that some pipeline options may be runner specific (like "--project" or "--tempRoot").
- </td>
- </tr>
- <tr>
- <td>-DitModule
- </td>
- <td>Specifies the project submodule of the I/O to test.
- </td>
- </tr>
- <tr>
- <td>-DintegrationTest
- </td>
- <td>Specifies the test to be run (fully qualified reference to class/test method).
- </td>
- </tr>
- <tr>
- <td>-DkubernetesScripts
- </td>
- <td>Paths to scripts with necessary kubernetes infrastructure.
- </td>
- </tr>
- <tr>
- <td>-DbeamITOptions
- </td>
- <td>Path to file with Benchmark configuration (static and dynamic pipeline options. See below for description).
- </td>
- </tr>
- <tr>
- <td>-DintegrationTestRunner
- </td>
- <td>Runner to be used for running the test. Currently possible options are: direct, dataflow.
- </td>
- </tr>
- <tr>
- <td>-DbeamExtraProperties
- </td>
- <td>Any other "extra properties" to be passed to Gradle, eg. "'[filesystem=hdfs]'".
- </td>
- </tr>
- </tbody>
-</table>
-
-#### Without PerfKit Benchmarker {#without-perfkit-benchmarker}
+#### Data store setup/cleanup {#datastore-setup-cleanup}
If you're using Kubernetes scripts to host data stores, make sure you can connect to your cluster locally using kubectl. If you have your own data stores already setup, you just need to execute step 3 from below list.
1. Set up the data store corresponding to the test you wish to run. You can find Kubernetes scripts for all currently supported data stores in [.test-infra/kubernetes](https://github.com/apache/beam/tree/master/.test-infra/kubernetes).
- 1. In some cases, there is a setup script (*.sh). In other cases, you can just run ``kubectl create -f [scriptname]`` to create the data store.
+ 1. In some cases, there is a dedicated setup script (*.sh). In other cases, you can just run ``kubectl create -f [scriptname]`` to create the data store. You can also let [kubernetes.sh](https://github.com/apache/beam/blob/master/.test-infra/kubernetes/kubernetes.sh) script perform some standard steps for you.
1. Convention dictates there will be:
1. A yml script for the data store itself, plus a `NodePort` service. The `NodePort` service opens a port to the data store for anyone who connects to the Kubernetes cluster's machines from within same subnetwork. Such scripts are typically useful when running the scripts on Minikube Kubernetes Engine.
1. A separate script, with LoadBalancer service. Such service will expose an _external ip_ for the datastore. Such scripts are needed when external access is required (eg. on Jenkins).
@@ -266,10 +169,9 @@
1. JDBC: `kubectl delete -f .test-infra/kubernetes/postgres/postgres.yml`
1. Elasticsearch: `bash .test-infra/kubernetes/elasticsearch/teardown.sh`
-##### integrationTest Task {#integration-test-task}
+#### Running a particular test {#running-a-test}
-Since `performanceTest` task involved running PerfkitBenchmarker, we can't use it to run the tests manually. For such purposes a more "low-level" task called `integrationTest` was introduced.
-
+`integrationTest` is a dedicated gradle task for running IO integration tests.
Example usage on Cloud Dataflow runner:
@@ -335,9 +237,11 @@
</tbody>
</table>
-#### Running Integration Tests on Pull Requests {#running-on-pull-requests}
+### Running Integration Tests on Pull Requests {#running-integration-tests-on-pull-requests}
-Thanks to [ghprb](https://github.com/janinko/ghprb) plugin it is possible to run Jenkins jobs when specific phrase is typed in a Github Pull Request's comment. Integration tests that have Jenkins job defined can be triggered this way. You can run integration tests using these phrases:
+Most of the IO integration tests have dedicated Jenkins jobs that run periodically to collect metrics and avoid regressions. Thanks to [ghprb](https://github.com/janinko/ghprb) plugin it is also possible to trigger these jobs on demand once a specific phrase is typed in a Github Pull Request's comment. This way tou can check if your contribution to a certain IO is an improvement or if it makes things worse (hopefully not!).
+
+To run IO Integration Tests type the following comments in your Pull Request:
<table class="table">
<thead>
@@ -437,7 +341,7 @@
### Performance testing dashboard {#performance-testing-dashboard}
-We measure the performance of IOITs by gathering test execution times from Jenkins jobs that run periodically. The consequent results are stored in a database (BigQuery), therefore we can display them in a form of plots.
+As mentioned before, we measure the performance of IOITs by gathering test execution times from Jenkins jobs that run periodically. The consequent results are stored in a database (BigQuery), therefore we can display them in a form of plots.
The dashboard gathering all the results is available here: [Performance Testing Dashboard](https://s.apache.org/io-test-dashboards)
@@ -446,9 +350,9 @@
There are three components necessary to implement an integration test:
* **Test code**: the code that does the actual testing: interacting with the I/O transform, reading and writing data, and verifying the data.
* **Kubernetes scripts**: a Kubernetes script that sets up the data store that will be used by the test code.
-* **Integrate with PerfKit Benchmarker**: this allows users to easily invoke PerfKit Benchmarker, creating the Kubernetes resources and running the test code.
+* **Jenkins jobs**: a Jenkins Job DSL script that performs all necessary steps for setting up the data sources, running and cleaning up after the test.
-These three pieces are discussed in detail below.
+These two pieces are discussed in detail below.
#### Test Code {#test-code}
@@ -492,255 +396,16 @@
1. Official Docker images, because they have security fixes and guaranteed maintenance.
1. Non-official Docker images, or images from other providers that have good maintainers (e.g. [quay.io](http://quay.io/)).
+#### Jenkins jobs {#jenkins-jobs}
-#### Integrate with PerfKit Benchmarker {#integrate-with-perfkit-benchmarker}
+You can find examples of existing IOIT jenkins job definitions in [.test-infra/jenkins](https://github.com/apache/beam/tree/master/.test-infra/jenkins) directory. Look for files caled job_PerformanceTest_*.groovy. The most prominent examples are:
+* [JDBC](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy) IOIT job
+* [MongoDB](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy) IOIT job
+* [File-based](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy) IOIT jobs
+
+Notice that there is a utility class helpful in creating the jobs easily without forgetting important steps or repeating code. See [Kubernetes.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/Kubernetes.groovy) for more details.
-To allow developers to easily invoke your I/O integration test, you should create a PerfKit Benchmarker benchmark configuration file for the data store. Each pipeline option needed by the integration test should have a configuration entry. This is to be passed to perfkit via "beamITOptions" option in "performanceTest" task (described above). The goal is that a checked in config has defaults such that other developers can run the test without changing the configuration.
-
-
-#### Defining the benchmark configuration file {#defining-the-benchmark-configuration-file}
-
-The benchmark configuration file is a yaml file that defines the set of pipeline options for a specific data store. Some of these pipeline options are **static** - they are known ahead of time, before the data store is created (e.g. username/password). Others options are **dynamic** - they are only known once the data store is created (or after we query the Kubernetes cluster for current status).
-
-All known cases of dynamic pipeline options are for extracting the IP address that the test needs to connect to. For I/O integration tests, we must allow users to specify:
-
-
-
-* The type of the IP address to get (load balancer/node address)
-* The pipeline option to pass that IP address to
-* How to find the Kubernetes resource with that value (ie. what load balancer service name? what node selector?)
-
-The style of dynamic pipeline options used here should support a variety of other types of values derived from Kubernetes, but we do not have specific examples.
-
-The dynamic pipeline options are:
-
-
-<table class="table">
- <thead>
- <tr>
- <td>
- <strong>Type name</strong>
- </td>
- <td>
- <strong>Meaning</strong>
- </td>
- <td>
- <strong>Selector field name</strong>
- </td>
- <td>
- <strong>Selector field value</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>NodePortIp
- </td>
- <td>We will be using the IP address of a k8s NodePort service, the value will be an IP address of a Pod
- </td>
- <td>podLabel
- </td>
- <td>A kubernetes label selector for a pod whose IP address can be used to connect to
- </td>
- </tr>
- <tr>
- <td>LoadBalancerIp
- </td>
- <td>We will be using the IP address of a k8s LoadBalancer, the value will be an IP address of the load balancer
- </td>
- <td>serviceName
- </td>
- <td>The name of the LoadBalancer kubernetes service.
- </td>
- </tr>
- </tbody>
-</table>
-
-#### Benchmark configuration files: full example configuration file {#benchmark-configuration-files-full-example-configuration-file}
-
-A configuration file will look like this:
-```
-static_pipeline_options:
- -postgresUser: postgres
- -postgresPassword: postgres
-dynamic_pipeline_options:
- - paramName: PostgresIp
- type: NodePortIp
- podLabel: app=postgres
-```
-
-
-and may contain the following elements:
-
-
-<table class="table">
- <thead>
- <tr>
- <td><strong>Configuration element</strong>
- </td>
- <td><strong>Description and how to change when adding a new test</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>static_pipeline_options
- </td>
- <td>The set of preconfigured pipeline options.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options
- </td>
- <td>The set of pipeline options that PerfKit Benchmarker will determine at runtime.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options.name
- </td>
- <td>The name of the parameter to be passed to gradle's invocation of the I/O integration test.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options.type
- </td>
- <td>The method of determining the value of the pipeline options.
- </td>
- </tr>
- <tr>
- <td>dynamic_pipeline_options - other attributes
- </td>
- <td>These vary depending on the type of the dynamic pipeline option - see the table of dynamic pipeline options for a description.
- </td>
- </tr>
- </tbody>
-</table>
-
-
-
-#### Customizing PerfKit Benchmarker behaviour {#customizing-perf-kit-benchmarker-behaviour}
-
-In most cases, to run the _performanceTest_ task it is sufficient to pass the properties described above, which makes it easy to use. However, users can customize Perfkit Benchmarker's behavior even more by pasing some extra Gradle properties:
-
-
-<table class="table">
- <thead>
- <tr>
- <td><strong>PerfKit Benchmarker Parameter</strong>
- </td>
- <td><strong>Corresponding Gradle property</strong>
- </td>
- <td><strong>Default value</strong>
- </td>
- <td><strong>Description</strong>
- </td>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>dpb_log_level
- </td>
- <td>-DlogLevel
- </td>
- <td>INFO
- </td>
- <td>Data Processing Backend's log level.
- </td>
- </tr>
- <tr>
- <td>gradle_binary
- </td>
- <td>-DgradleBinary
- </td>
- <td>./gradlew
- </td>
- <td>Path to gradle binary.
- </td>
- </tr>
- <tr>
- <td>official
- </td>
- <td>-Dofficial
- </td>
- <td>false
- </td>
- <td>If true, the benchmark results are marked as "official" and can be displayed on PerfKitExplorer dashboards.
- </td>
- </tr>
- <tr>
- <td>benchmarks
- </td>
- <td>-Dbenchmarks
- </td>
- <td>beam_integration_benchmark
- </td>
- <td>Defines the PerfKit Benchmarker benchmark to run. This is same for all I/O integration tests.
- </td>
- </tr>
- <tr>
- <td>beam_prebuilt
- </td>
- <td>-DbeamPrebuilt
- </td>
- <td>true
- </td>
- <td>If false, PerfKit Benchmarker runs the build task before running the tests.
- </td>
- </tr>
- <tr>
- <td>beam_sdk
- </td>
- <td>-DbeamSdk
- </td>
- <td>java
- </td>
- <td>Beam's sdk to be used by PerfKit Benchmarker.
- </td>
- </tr>
- <tr>
- <td>beam_timeout
- </td>
- <td>-DitTimeout
- </td>
- <td>1200
- </td>
- <td>Timeout (in seconds) after which PerfKit Benchmarker will stop executing the benchmark (and will fail).
- </td>
- </tr>
- <tr>
- <td>kubeconfig
- </td>
- <td>-Dkubeconfig
- </td>
- <td>~/.kube/config
- </td>
- <td>Path to kubernetes configuration file.
- </td>
- </tr>
- <tr>
- <td>kubectl
- </td>
- <td>-Dkubectl
- </td>
- <td>kubectl
- </td>
- <td>Path to kubernetes executable.
- </td>
- </tr>
- <tr>
- <td>beam_extra_properties
- </td>
- <td>-DbeamExtraProperties
- </td>
- <td>(empty string)
- </td>
- <td>Any additional properties to be appended to benchmark execution command.
- </td>
- </tr>
- </tbody>
-</table>
-
-#### Small Scale and Large Scale Integration Tests {#small-scale-and-large-scale-integration-tests}
+### Small Scale and Large Scale Integration Tests {#small-scale-and-large-scale-integration-tests}
Apache Beam expects that it can run integration tests in multiple configurations:
* Small scale
diff --git a/website/src/documentation/patterns/custom-io.md b/website/src/documentation/patterns/custom-io.md
new file mode 100644
index 0000000..d816010
--- /dev/null
+++ b/website/src/documentation/patterns/custom-io.md
@@ -0,0 +1,42 @@
+---
+layout: section
+title: "Custom I/O patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/custom-io/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Custom I/O patterns
+
+This page describes common patterns in pipelines with [custom I/O connectors]({{ site.baseurl }}/documentation/io/developing-io-overview/). Custom I/O connectors connect pipelines to databases that aren't supported by Beam's [built-in I/O transforms]({{ site.baseurl }}/documentation/io/built-in/).
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+## Choosing between built-in and custom connectors
+
+[Built-in I/O connectors]({{ site.baseurl }}/documentation/io/built-in/) are tested and hardened, so use them whenever possible. Only use custom I/O connectors when:
+
+* No built-in options exist
+* Your pipeline pulls in a small subset of source data
+
+For instance, use a custom I/O connector to enrich pipeline elements with a small subset of source data. If you’re processing a sales order and adding information to each purchase, you can use a custom I/O connector to pull the small subset of data into your pipeline (instead of processing the entire source).
+
+Beam distributes work across many threads, so custom I/O connectors can increase your data source’s load average. You can reduce the load with the <span class="language-java">[start](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.StartBundle.html)</span><span class="language-py">[start](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.start_bundle)</span> and <span class="language-java">[finish](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.FinishBundle.html)</span><span class="language-py">[finish](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.finish_bundle)</span> bundle annotations.
\ No newline at end of file
diff --git a/website/src/documentation/patterns/custom-windows.md b/website/src/documentation/patterns/custom-windows.md
new file mode 100644
index 0000000..c3ef84a
--- /dev/null
+++ b/website/src/documentation/patterns/custom-windows.md
@@ -0,0 +1,114 @@
+---
+layout: section
+title: "Custom window patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/custom-windows/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Custom window patterns
+The samples on this page demonstrate common custom window patterns. You can create custom windows with [`WindowFn` functions]({{ site.baseurl }}/documentation/programming-guide/#provided-windowing-functions). For more information, see the [programming guide section on windowing]({{ site.baseurl }}/documentation/programming-guide/#windowing).
+
+**Note**: Custom merging windows isn't supported in Python (with fnapi).
+
+## Using data to dynamically set session window gaps
+
+You can modify the [`assignWindows`](https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/windowing/SlidingWindows.html) function to use data-driven gaps, then window incoming data into sessions.
+
+Access the `assignWindows` function through `WindowFn.AssignContext.element()`. The original, fixed-duration `assignWindows` function is:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow1
+%}
+```
+
+### Creating data-driven gaps
+To create data-driven gaps, add the following snippets to the `assignWindows` function:
+- A default value for when the custom gap is not present in the data
+- A way to set the attribute from the main pipeline as a method of the custom windows
+
+For example, the following function assigns each element to a window between the timestamp and `gapDuration`:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow3
+%}
+```
+
+Then, set the `gapDuration` field in a windowing function:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow2
+%}
+```
+
+### Windowing messages into sessions
+After creating data-driven gaps, you can window incoming data into the new, custom sessions.
+
+First, set the session length to the gap duration:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow4
+%}
+```
+
+Lastly, window data into sessions in your pipeline:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:CustomSessionWindow6
+%}
+```
+
+### Example data and windows
+The following test data tallies two users' scores with and without the `gap` attribute:
+
+```
+.apply("Create data", Create.timestamped(
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"12\",\"gap\":\"5\"}", new Instant()),
+ TimestampedValue.of("{\"user\":\"user-2\",\"score\":\"4\"}", new Instant()),
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"-3\",\"gap\":\"5\"}", new Instant().plus(2000)),
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"2\",\"gap\":\"5\"}", new Instant().plus(9000)),
+ TimestampedValue.of("{\"user\":\"user-1\",\"score\":\"7\",\"gap\":\"5\"}", new Instant().plus(12000)),
+ TimestampedValue.of("{\"user\":\"user-2\",\"score\":\"10\"}", new Instant().plus(12000)))
+ .withCoder(StringUtf8Coder.of()))
+```
+
+The diagram below visualizes the test data:
+
+![Two sets of data and the standard and dynamic sessions with which the data is windowed.]( {{ "/images/standard-vs-dynamic-sessions.png" | prepend: site.baseurl }})
+
+#### Standard sessions
+
+Standard sessions use the following windows and scores:
+```
+user=user-2, score=4, window=[2019-05-26T13:28:49.122Z..2019-05-26T13:28:59.122Z)
+user=user-1, score=18, window=[2019-05-26T13:28:48.582Z..2019-05-26T13:29:12.774Z)
+user=user-2, score=10, window=[2019-05-26T13:29:03.367Z..2019-05-26T13:29:13.367Z)
+```
+
+User #1 sees two events separated by 12 seconds. With standard sessions, the gap defaults to 10 seconds; both scores are in different sessions, so the scores aren't added.
+
+User #2 sees four events, seperated by two, seven, and three seconds, respectively. Since none of the gaps are greater than the default, the four events are in the same standard session and added together (18 points).
+
+#### Dynamic sessions
+The dynamic sessions specify a five-second gap, so they use the following windows and scores:
+
+```
+user=user-2, score=4, window=[2019-05-26T14:30:22.969Z..2019-05-26T14:30:32.969Z)
+user=user-1, score=9, window=[2019-05-26T14:30:22.429Z..2019-05-26T14:30:30.553Z)
+user=user-1, score=9, window=[2019-05-26T14:30:33.276Z..2019-05-26T14:30:41.849Z)
+user=user-2, score=10, window=[2019-05-26T14:30:37.357Z..2019-05-26T14:30:47.357Z)
+```
+
+With dynamic sessions, User #2 gets different scores. The third messages arrives seven seconds after the second message, so it's grouped into a different session. The large, 18-point session is split into two 9-point sessions.
\ No newline at end of file
diff --git a/website/src/documentation/patterns/file-processing.md b/website/src/documentation/patterns/file-processing.md
new file mode 100644
index 0000000..592a58b
--- /dev/null
+++ b/website/src/documentation/patterns/file-processing.md
@@ -0,0 +1,107 @@
+---
+layout: section
+title: "File processing patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/file-processing/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# File processing patterns
+
+This page describes common file processing tasks. For more information on file-based I/O, see [Pipeline I/O]({{ site.baseurl }}/documentation/programming-guide/#pipeline-io) and [File-based input and output data]({{ site.baseurl }}/documentation/programming-guide/#file-based-data).
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+## Processing files as they arrive
+
+This section shows you how to process files as they arrive in your file system or object store (like Google Cloud Storage). You can continuously read files or trigger stream and processing pipelines when a file arrives.
+
+### Continuous read mode
+
+{:.language-java}
+You can use `FileIO` or `TextIO` to continuously read the source for new files.
+
+{:.language-java}
+Use the [`FileIO`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html) class to continuously watch a single file pattern. The following example matches a file pattern repeatedly every 30 seconds, continuously returns new matched files as an unbounded `PCollection<Metadata>`, and stops if no new files appear for one hour:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternProcessNewFilesSnip1
+%}
+```
+
+{:.language-java}
+The [`TextIO`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html) class `watchForNewFiles` property streams new file matches.
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternProcessNewFilesSnip2
+%}
+```
+
+{:.language-java}
+Some runners may retain file lists during updates, but file lists don’t persist when you restart a pipeline. You can save file lists by:
+
+{:.language-java}
+* Storing processed filenames in an external file and deduplicating the lists at the next transform
+* Adding timestamps to filenames, writing a glob pattern to pull in only new files, and matching the pattern when the pipeline restarts
+
+{:.language-py}
+The continuous-read option is not available for Python.
+
+### Stream processing triggered from external source
+
+A streaming pipeline can process data from an unbounded source. For example, to trigger stream processing with Google Cloud Pub/Sub:
+
+1. Use an external process to detect when new files arrive.
+1. Send a Google Cloud Pub/Sub message with a URI to the file.
+1. Access the URI from a `DoFn` that follows the Google Cloud Pub/Sub source.
+1. Process the file.
+
+### Batch processing triggered from external source
+
+To start or schedule a batch pipeline job when a file arrives, write the triggering event in the source file itself. This has the most latency because the pipeline must initialize before processing. It’s best suited for low-frequency, large, file-size updates.
+
+## Accessing filenames
+
+{:.language-java}
+Use the `FileIO` class to read filenames in a pipeline job. `FileIO` returns a `PCollection<ReadableFile>` object, and the `ReadableFile` instance contains the filename.
+
+{:.language-java}
+To access filenames:
+
+{:.language-java}
+1. Create a `ReadableFile` instance with `FileIO`. `FileIO` returns a `PCollection<ReadableFile>` object. The `ReadableFile` class contains the filename.
+1. Call the `readFullyAsUTF8String()` method to read the file into memory and return the filename as a `String` object. If memory is limited, you can use utility classes like [`FileSystems`](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html) to work directly with the file.
+
+{:.language-py}
+To read filenames in a pipeline job:
+
+{:.language-py}
+1. Collect the list of file URIs. You can use the [`FileSystems`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.filesystems.html?highlight=filesystems#module-apache_beam.io.filesystems) module to get a list of files that match a glob pattern.
+1. Pass the file URIs to a `PCollection`.
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:FileProcessPatternAccessMetadataSnip1
+%}
+```
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:FileProcessPatternAccessMetadataSnip1
+%}
+```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/overview.md b/website/src/documentation/patterns/overview.md
new file mode 100644
index 0000000..8ecca0b
--- /dev/null
+++ b/website/src/documentation/patterns/overview.md
@@ -0,0 +1,48 @@
+---
+layout: section
+title: "Overview"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/overview/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Common pipeline patterns
+
+Pipeline patterns demonstrate common Beam use cases. Pipeline patterns are based on real-world Beam deployments. Each pattern has a description, examples, and a solution or psuedocode.
+
+**File processing patterns** - Patterns for reading from and writing to files
+* [Processing files as they arrive]({{ site.baseurl }}/documentation/patterns/file-processing/#processing-files-as-they-arrive)
+* [Accessing filenames]({{ site.baseurl }}/documentation/patterns/file-processing/#accessing-filenames)
+
+**Side input patterns** - Patterns for processing supplementary data
+* [Slowly updating global window side inputs]({{ site.baseurl }}/documentation/patterns/side-inputs/#slowly-updating-global-window-side-inputs)
+
+**Pipeline option patterns** - Patterns for configuring pipelines
+* [Retroactively logging runtime parameters]({{ site.baseurl }}/documentation/patterns/pipeline-options/#retroactively-logging-runtime-parameters)
+
+**Custom I/O patterns** - Patterns for pipeline I/O
+* [Choosing between built-in and custom connectors]({{ site.baseurl }}/documentation/patterns/custom-io/#choosing-between-built-in-and-custom-connectors)
+
+**Custom window patterns** - Patterns for windowing functions
+* [Using data to dynamically set session window gaps]({{ site.baseurl }}/documentation/patterns/custom-windows/#using-data-to-dynamically-set-session-window-gaps)
+
+## Contributing a pattern
+
+To contribute a new pipeline pattern, create an issue with the [`pipeline-patterns` label](https://issues.apache.org/jira/browse/BEAM-7449?jql=labels%20%3D%20pipeline-patterns) and add details to the issue description. See [Get started contributing]({{ site.baseurl }}/contribute/) for more information.
+
+## What's next
+
+* Try an [end-to-end example]({{ site.baseurl }}/get-started/try-apache-beam/)
+* Execute your pipeline on a [runner]({{ site.baseurl }}/documentation/runners/capability-matrix/)
diff --git a/website/src/documentation/patterns/pipeline-options.md b/website/src/documentation/patterns/pipeline-options.md
new file mode 100644
index 0000000..84d1bf5
--- /dev/null
+++ b/website/src/documentation/patterns/pipeline-options.md
@@ -0,0 +1,47 @@
+---
+layout: section
+title: "Pipeline option patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/pipeline-options/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Pipeline option patterns
+
+The samples on this page show you common pipeline configurations. For more information about pipeline configuration options, see [Creating a pipeline]({{ site.baseurl }}/documentation/programming-guide/#creating-a-pipeline) and [Configuring pipeline options]({{ site.baseurl }}/documentation/programming-guide/#configuring-pipeline-options).
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java" class="active">Java SDK</li>
+ <li data-type="language-py">Python SDK</li>
+ </ul>
+</nav>
+
+## Retroactively logging runtime parameters
+
+Use the `ValueProvider` interface to access runtime parameters after completing a pipeline job.
+
+You can use the `ValueProvider` interface to pass runtime parameters to your pipeline, but you can only log the parameters from within the the Beam DAG. A solution is to add a pipeline [branch]({{ site.baseurl }}/documentation/programming-guide/#applying-transforms) with a `DoFn` that processes a placeholder value and then logs the runtime parameters:
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:AccessingValueProviderInfoAfterRunSnip1
+%}
+```
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:AccessingValueProviderInfoAfterRunSnip1
+%}
+```
\ No newline at end of file
diff --git a/website/src/documentation/patterns/side-inputs.md b/website/src/documentation/patterns/side-inputs.md
new file mode 100644
index 0000000..854c276
--- /dev/null
+++ b/website/src/documentation/patterns/side-inputs.md
@@ -0,0 +1,48 @@
+---
+layout: section
+title: "Side input patterns"
+section_menu: section-menu/documentation.html
+permalink: /documentation/patterns/side-inputs/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Side input patterns
+
+The samples on this page show you common Beam side input patterns. A side input is an additional input that your `DoFn` can access each time it processes an element in the input `PCollection`. For more information, see the [programming guide section on side inputs]({{ site.baseurl }}/documentation/programming-guide/#side-inputs).
+
+## Slowly updating global window side inputs
+
+You can retrieve side inputs from global windows to use them in a pipeline job with non-global windows, like a `FixedWindow`.
+
+To slowly update global window side inputs in pipelines with non-global windows:
+
+1. Write a `DoFn` that periodically pulls data from a bounded source into a global window.
+
+ a. Use the `GenerateSequence` source transform to periodically emit a value.
+
+ b. Instantiate a data-driven trigger that activates on each element and pulls data from a bounded source.
+
+ c. Fire the trigger to pass the data into the global window.
+
+1. Create the side input for downstream transforms. The side input should fit into memory.
+
+The global window side input triggers on processing time, so the main pipeline nondeterministically matches the side input to elements in event time.
+
+For instance, the following code sample uses a `Map` to create a `DoFn`. The `Map` becomes a `View.asSingleton` side input that’s rebuilt on each counter tick. The side input updates every 5 seconds in order to demonstrate the workflow. In a real-world scenario, the side input would typically update every few hours or once per day.
+
+```java
+{% github_sample /apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/snippets/Snippets.java tag:SideInputPatternSlowUpdateGlobalWindowSnip1
+%}
+```
\ No newline at end of file
diff --git a/website/src/documentation/programming-guide.md b/website/src/documentation/programming-guide.md
index ea72a22..0c8a2c1 100644
--- a/website/src/documentation/programming-guide.md
+++ b/website/src/documentation/programming-guide.md
@@ -39,6 +39,9 @@
</ul>
</nav>
+{:.language-py}
+The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)). For best results, use Beam with Python 3.
+
## 1. Overview {#overview}
To use Beam, you need to first create a driver program using the classes in one
@@ -188,13 +191,17 @@
You can add your own custom options in addition to the standard
`PipelineOptions`. To add your own options, define an interface with getter and
-setter methods for each option, as in the following example:
+setter methods for each option, as in the following example for
+adding `input` and `output` custom options:
```java
public interface MyOptions extends PipelineOptions {
- String getMyCustomOption();
- void setMyCustomOption(String myCustomOption);
- }
+ String getInput();
+ void setInput(String input);
+
+ String getOutput();
+ void setOutput(String output);
+}
```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipeline_options_define_custom
@@ -214,11 +221,16 @@
```java
public interface MyOptions extends PipelineOptions {
- @Description("My custom command line argument.")
- @Default.String("DEFAULT")
- String getMyCustomOption();
- void setMyCustomOption(String myCustomOption);
- }
+ @Description("Input for the pipeline")
+ @Default.String("gs://my-bucket/input")
+ String getInput();
+ void setInput(String input);
+
+ @Description("Output for the pipeline")
+ @Default.String("gs://my-bucket/input")
+ String getOutput();
+ void setOutput(String output);
+}
```
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipeline_options_define_custom_with_help_and_default
@@ -226,8 +238,8 @@
```
```go
var (
- input = flag.String("input", "gs://my-bucket/input", "File(s) to read.")
- output = flag.String("output", "gs://my-bucket/output", "Output file.")
+ input = flag.String("input", "gs://my-bucket/input", "Input for the pipeline")
+ output = flag.String("output", "gs://my-bucket/output", "Output for the pipeline")
)
```
@@ -251,7 +263,7 @@
.as(MyOptions.class);
```
-Now your pipeline can accept `--myCustomOption=value` as a command-line argument.
+Now your pipeline can accept `--input=value` and `--output=value` as command-line arguments.
## 3. PCollections {#pcollections}
@@ -302,7 +314,7 @@
// Create the PCollection 'lines' by applying a 'Read' transform.
PCollection<String> lines = p.apply(
- "ReadMyFile", TextIO.read().from("protocol://path/to/some/inputData.txt"));
+ "ReadMyFile", TextIO.read().from("gs://some/inputData.txt"));
}
```
```py
@@ -310,7 +322,7 @@
%}
```
```go
-lines := textio.Read(s, "protocol://path/to/some/inputData.txt")
+lines := textio.Read(s, "gs://some/inputData.txt")
```
See the [section on I/O](#pipeline-io) to learn more about how to read from the
@@ -514,12 +526,14 @@
a branching pipeline, like so:
```java
-[Output PCollection 1] = [Input PCollection].apply([Transform 1])
-[Output PCollection 2] = [Input PCollection].apply([Transform 2])
+[PCollection of database table rows] = [Database Table Reader].apply([Read Transform])
+[PCollection of 'A' names] = [PCollection of database table rows].apply([Transform A])
+[PCollection of 'B' names] = [PCollection of database table rows].apply([Transform B])
```
```py
-[Output PCollection 1] = [Input PCollection] | [Transform 1]
-[Output PCollection 2] = [Input PCollection] | [Transform 2]
+[PCollection of database table rows] = [Database Table Reader] | [Read Transform]
+[PCollection of 'A' names] = [PCollection of database table rows] | [Transform A]
+[PCollection of 'B' names] = [PCollection of database table rows] | [Transform B]
```
The resulting workflow graph from the branching pipeline above looks like this.
@@ -1079,12 +1093,6 @@
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:combine_custom_average_define
%}```
-If you are combining a `PCollection` of key-value pairs, [per-key
-combining](#combining-values-in-a-keyed-pcollection) is often enough. If
-you need the combining strategy to change based on the key (for example, MIN for
-some users and MAX for other users), you can define a `KeyedCombineFn` to access
-the key within the combining strategy.
-
##### 4.2.4.3. Combining a PCollection into a single value {#combining-pcollection}
Use the global combine to transform all of the elements in a given `PCollection`
@@ -1360,7 +1368,7 @@
to a `ParDo` transform in the form of side inputs. A side input is an additional
input that your `DoFn` can access each time it processes an element in the input
`PCollection`. When you specify a side input, you create a view of some other
-data that can be read from within the `ParDo` transform's `DoFn` while procesing
+data that can be read from within the `ParDo` transform's `DoFn` while processing
each element.
Side inputs are useful if your `ParDo` needs to inject additional data when
@@ -1560,23 +1568,40 @@
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_pardo_with_undeclared_outputs
%}```
-{:.language-java}
#### 4.5.3. Accessing additional parameters in your DoFn {#other-dofn-parameters}
{:.language-java}
In addition to the element and the `OutputReceiver`, Beam will populate other parameters to your DoFn's `@ProcessElement` method.
Any combination of these parameters can be added to your process method in any order.
+{:.language-py}
+In addition to the element, Beam will populate other parameters to your DoFn's `process` method.
+Any combination of these parameters can be added to your process method in any order.
+
{:.language-java}
**Timestamp:**
To access the timestamp of an input element, add a parameter annotated with `@Timestamp` of type `Instant`. For example:
+{:.language-py}
+**Timestamp:**
+To access the timestamp of an input element, add a keyword parameter default to `DoFn.TimestampParam`. For example:
+
```java
.of(new DoFn<String, String>() {
public void processElement(@Element String word, @Timestamp Instant timestamp) {
}})
```
+```py
+import apache_beam as beam
+
+class ProcessRecord(beam.DoFn):
+
+ def process(self, element, timestamp=beam.DoFn.TimestampParam):
+ # access timestamp of element.
+ pass
+
+```
{:.language-java}
**Window:**
@@ -1586,23 +1611,57 @@
`@ProcessElement` method will be invoked multiple time for the element, once for each window. For example, when fixed windows
are being used, the window is of type `IntervalWindow`.
+{:.language-py}
+**Window:**
+To access the window an input element falls into, add a keyword parameter default to `DoFn.WindowParam`.
+If an element falls in multiple windows (for example, this will happen when using `SlidingWindows`), then the
+`process` method will be invoked multiple time for the element, once for each window.
+
```java
.of(new DoFn<String, String>() {
public void processElement(@Element String word, IntervalWindow window) {
}})
```
+```py
+import apache_beam as beam
+
+class ProcessRecord(beam.DoFn):
+
+ def process(self, element, window=beam.DoFn.WindowParam):
+ # access window e.g window.end.micros
+ pass
+
+```
+
{:.language-java}
**PaneInfo:**
When triggers are used, Beam provides a `PaneInfo` object that contains information about the current firing. Using `PaneInfo`
you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.
+{:.language-py}
+**PaneInfo:**
+When triggers are used, Beam provides a `DoFn.PaneInfoParam` object that contains information about the current firing. Using `DoFn.PaneInfoParam`
+you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.
+This feature implementation in python sdk is not fully completed, see more at [BEAM-3759](https://issues.apache.org/jira/browse/BEAM-3759).
+
```java
.of(new DoFn<String, String>() {
public void processElement(@Element String word, PaneInfo paneInfo) {
}})
```
+```py
+import apache_beam as beam
+
+class ProcessRecord(beam.DoFn):
+
+ def process(self, element, pane_info=beam.DoFn.PaneInfoParam):
+ # access pane info e.g pane_info.is_first, pane_info.is_last, pane_info.timing
+ pass
+
+```
+
{:.language-java}
**PipelineOptions:**
The `PipelineOptions` for the current pipeline can always be accessed in a process method by adding it as a parameter:
@@ -1613,12 +1672,75 @@
```
{:.language-java}
-`@OnTimer` methods can also access many of these parameters. Timestamp, window, `PipelineOptions`, `OutputReceiver`, and
+`@OnTimer` methods can also access many of these parameters. Timestamp, Window, key, `PipelineOptions`, `OutputReceiver`, and
`MultiOutputReceiver` parameters can all be accessed in an `@OnTimer` method. In addition, an `@OnTimer` method can take
a parameter of type `TimeDomain` which tells whether the timer is based on event time or processing time.
Timers are explained in more detail in the
[Timely (and Stateful) Processing with Apache Beam]({{ site.baseurl }}/blog/2017/08/28/timely-processing.html) blog post.
+{:.language-py}
+**Timer and State:**
+In addition to aforementioned parameters, user defined Timer and State parameters can be used in a Stateful DoFn.
+Timers and States are explained in more detail in the
+[Timely (and Stateful) Processing with Apache Beam]({{ site.baseurl }}/blog/2017/08/28/timely-processing.html) blog post.
+
+```py
+
+class StatefulDoFn(beam.DoFn):
+ """An example stateful DoFn with state and timer"""
+
+ BUFFER_STATE_1 = BagStateSpec('buffer1', beam.BytesCoder())
+ BUFFER_STATE_2 = BagStateSpec('buffer2', beam.VarIntCoder())
+ WATERMARK_TIMER = TimerSpec('watermark_timer', TimeDomain.WATERMARK)
+
+ def process(self,
+ element,
+ timestamp=beam.DoFn.TimestampParam,
+ window=beam.DoFn.WindowParam,
+ buffer_1=beam.DoFn.StateParam(BUFFER_STATE_1),
+ buffer_2=beam.DoFn.StateParam(BUFFER_STATE_2),
+ watermark_timer=beam.DoFn.TimerParam(WATERMARK_TIMER)):
+
+ # Do you processing here
+ key, value = element
+ # Read all the data from buffer1
+ all_values_in_buffer_1 = [x for x in buffer_1.read()]
+
+ if StatefulDoFn._is_clear_buffer_1_required(all_values_in_buffer_1):
+ # clear the buffer data if required conditions are met.
+ buffer_1.clear()
+
+ # add the value to buffer 2
+ buffer_2.add(value)
+
+ if StatefulDoFn._all_condition_met():
+ # Clear the timer if certain condition met and you don't want to trigger
+ # the callback method.
+ watermark_timer.clear()
+
+ yield element
+
+ @on_timer(WATERMARK_TIMER)
+ def on_expiry_1(self,
+ timestamp=beam.DoFn.TimestampParam,
+ window=beam.DoFn.WindowParam,
+ key=beam.DoFn.KeyParam,
+ buffer_1=beam.DoFn.StateParam(BUFFER_STATE_1),
+ buffer_2=beam.DoFn.StateParam(BUFFER_STATE_2)):
+ # Window and key parameters are really useful especially for debugging issues.
+ yield 'expired1'
+
+ @staticmethod
+ def _all_condition_met():
+ # some logic
+ return True
+
+ @staticmethod
+ def _is_clear_buffer_1_required(buffer_1_data):
+ # Some business logic
+ return True
+
+```
### 4.6. Composite transforms {#composite-transforms}
Transforms can have a nested structure, where a complex transform performs
@@ -2261,14 +2383,14 @@
The simplest form of windowing is using **fixed time windows**: given a
timestamped `PCollection` which might be continuously updating, each window
-might capture (for example) all elements with timestamps that fall into a five
-minute interval.
+might capture (for example) all elements with timestamps that fall into a 30
+second interval.
A fixed time window represents a consistent duration, non overlapping time
-interval in the data stream. Consider windows with a five-minute duration: all
+interval in the data stream. Consider windows with a 30 second duration: all
of the elements in your unbounded `PCollection` with timestamp values from
-0:00:00 up to (but not including) 0:05:00 belong to the first window, elements
-with timestamp values from 0:05:00 up to (but not including) 0:10:00 belong to
+0:00:00 up to (but not including) 0:00:30 belong to the first window, elements
+with timestamp values from 0:00:30 up to (but not including) 0:01:00 belong to
the second window, and so on.
![Diagram of fixed time windows, 30s in duration]({{ "/images/fixed-time-windows.png" | prepend: site.baseurl }} "Fixed time windows, 30s in duration")
@@ -2279,15 +2401,15 @@
A **sliding time window** also represents time intervals in the data stream;
however, sliding time windows can overlap. For example, each window might
-capture five minutes worth of data, but a new window starts every ten seconds.
+capture 60 seconds worth of data, but a new window starts every 30 seconds.
The frequency with which sliding windows begin is called the _period_.
-Therefore, our example would have a window _duration_ of five minutes and a
-_period_ of ten seconds.
+Therefore, our example would have a window _duration_ of 60 seconds and a
+_period_ of 30 seconds.
Because multiple windows overlap, most elements in a data set will belong to
more than one window. This kind of windowing is useful for taking running
averages of data; using sliding time windows, you can compute a running average
-of the past five minutes' worth of data, updated every ten seconds, in our
+of the past 60 seconds' worth of data, updated every 30 seconds, in our
example.
![Diagram of sliding time windows, with 1 minute window duration and 30s window period]({{ "/images/sliding-time-windows.png" | prepend: site.baseurl }} "Sliding time windows, with 1 minute window duration and 30s window period")
@@ -2436,7 +2558,7 @@
watermark that estimates the lag time. In practice, your `PCollection`'s data
source determines the watermark, and watermarks can be more precise or complex.
-Beam's default windowing configuration tries to determines when all data has
+Beam's default windowing configuration tries to determine when all data has
arrived (based on the type of data source) and then advances the watermark past
the end of the window. This default configuration does _not_ allow late data.
[Triggers](#triggers) allow you to modify and refine the windowing strategy for
@@ -2837,3 +2959,143 @@
```py
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_other_composite_triggers
%}```
+
+## 9. Metrics {#metrics}
+In the Beam model, metrics provide some insight into the current state of a user pipeline,
+potentially while the pipeline is running. There could be different reasons for that, for instance:
+* Check the number of errors encountered while running a specific step in the pipeline;
+* Monitor the number of RPCs made to backend service;
+* Retrieve an accurate count of the number of elements that have been processed;
+* ...and so on.
+
+### 9.1 The main concepts of Beam metrics
+* **Named**. Each metric has a name which consists of a namespace and an actual name. The
+ namespace can be used to differentiate between multiple metrics with the same name and also
+ allows querying for all metrics within a specific namespace.
+* **Scoped**. Each metric is reported against a specific step in the pipeline, indicating what
+ code was running when the metric was incremented.
+* **Dynamically Created**. Metrics may be created during runtime without pre-declaring them, in
+ much the same way a logger could be created. This makes it easier to produce metrics in utility
+ code and have them usefully reported.
+* **Degrade Gracefully**. If a runner doesn’t support some part of reporting metrics, the
+ fallback behavior is to drop the metric updates rather than failing the pipeline. If a runner
+ doesn’t support some part of querying metrics, the runner will not return the associated data.
+
+Reported metrics are implicitly scoped to the transform within the pipeline that reported them.
+This allows reporting the same metric name in multiple places and identifying the value each
+transform reported, as well as aggregating the metric across the entire pipeline.
+
+> **Note:** It is runner-dependent whether metrics are accessible during pipeline execution or only
+after jobs have completed.
+
+### 9.2 Types of metrics {#types-of-metrics}
+There are three types of metrics that are supported for the moment: `Counter`, `Distribution` and
+`Gauge`.
+
+**Counter**: A metric that reports a single long value and can be incremented or decremented.
+
+```java
+Counter counter = Metrics.counter( "namespace", "counter1");
+
+@ProcessElement
+public void processElement(ProcessContext context) {
+ // count the elements
+ counter.inc();
+ ...
+}
+```
+
+**Distribution**: A metric that reports information about the distribution of reported values.
+
+```java
+Distribution distribution = Metrics.distribution( "namespace", "distribution1");
+
+@ProcessElement
+public void processElement(ProcessContext context) {
+ Integer element = context.element();
+ // create a distribution (histogram) of the values
+ distribution.update(element);
+ ...
+}
+```
+
+**Gauge**: A metric that reports the latest value out of reported values. Since metrics are
+collected from many workers the value may not be the absolute last, but one of the latest values.
+
+```java
+Gauge gauge = Metrics.gauge( "namespace", "gauge1");
+
+@ProcessElement
+public void processElement(ProcessContext context) {
+ Integer element = context.element();
+ // create a gauge (latest value received) of the values
+ gauge.set(element);
+ ...
+}
+```
+
+### 9.3 Querying metrics {#querying-metrics}
+`PipelineResult` has a method `metrics()` which returns a `MetricResults` object that allows
+accessing metrics. The main method available in `MetricResults` allows querying for all metrics
+matching a given filter.
+
+```java
+public interface PipelineResult {
+ MetricResults metrics();
+}
+
+public abstract class MetricResults {
+ public abstract MetricQueryResults queryMetrics(@Nullable MetricsFilter filter);
+}
+
+public interface MetricQueryResults {
+ Iterable<MetricResult<Long>> getCounters();
+ Iterable<MetricResult<DistributionResult>> getDistributions();
+ Iterable<MetricResult<GaugeResult>> getGauges();
+}
+
+public interface MetricResult<T> {
+ MetricName getName();
+ String getStep();
+ T getCommitted();
+ T getAttempted();
+}
+```
+
+### 9.4 Using metrics in pipeline {#using-metrics}
+Below, there is a simple example of how to use a `Counter` metric in a user pipeline.
+
+```java
+// creating a pipeline with custom metrics DoFn
+pipeline
+ .apply(...)
+ .apply(ParDo.of(new MyMetricsDoFn()));
+
+pipelineResult = pipeline.run().waitUntilFinish(...);
+
+// request the metric called "counter1" in namespace called "namespace"
+MetricQueryResults metrics =
+ pipelineResult
+ .metrics()
+ .queryMetrics(
+ MetricsFilter.builder()
+ .addNameFilter(MetricNameFilter.named("namespace", "counter1"))
+ .build());
+
+// print the metric value - there should be only one line because there is only one metric
+// called "counter1" in the namespace called "namespace"
+for (MetricResult<Long> counter: metrics.getCounters()) {
+ System.out.println(counter.getName() + ":" + counter.getAttempted());
+}
+
+public class MyMetricsDoFn extends DoFn<Integer, Integer> {
+ private final Counter counter = Metrics.counter( "namespace", "counter1");
+
+ @ProcessElement
+ public void processElement(ProcessContext context) {
+ // count the elements
+ counter.inc();
+ context.output(context.element());
+ }
+}
+```
diff --git a/website/src/documentation/resources/learning-resources.md b/website/src/documentation/resources/learning-resources.md
index 47959e3..bf917cf 100644
--- a/website/src/documentation/resources/learning-resources.md
+++ b/website/src/documentation/resources/learning-resources.md
@@ -47,7 +47,7 @@
* **[Programming Guide](https://beam.apache.org/documentation/programming-guide/)** - The Programming Guide contains more in-depth information on most topics in the Apache Beam SDK. These include descriptions on how everything works as well as code snippets to see how to use every part. This can be used as a reference guidebook.
* **[The world beyond batch: Streaming 101](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101)** - Covers some basic background information, terminology, time domains, batch processing, and streaming.
* **[The world beyond batch: Streaming 102](https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102)** - Tour of the unified batch and streaming programming model in Beam, alongside with an example to explain many of the concepts.
-* **[Apache Beam Execution Model](https://beam.apache.org/documentation/execution-model/)** - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.
+* **[Apache Beam Execution Model](https://beam.apache.org/documentation/runtime/model)** - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.
### Common Patterns
@@ -95,6 +95,33 @@
* **[NDVI from Landsat Images](https://qwiklabs.com/focuses/1849?locale=en&parent=catalog)** (45m) - Process Landsat satellite data in a distributed environment to compute the [Normalized Difference Vegetation Index](https://en.wikipedia.org/wiki/Normalized_difference_vegetation_index) (NDVI).
* **[Simulate historic flights](https://qwiklabs.com/focuses/1159?locale=en&parent=catalog)** (60m) - Simulate real-time historic internal flights in the United States and store the resulting simulated data in BigQuery.
+## Beam Katas {#beam-katas}
+
+Beam Katas are interactive Beam coding exercises (i.e. [code katas](http://codekata.com/))
+that can help you to learn Apache Beam concepts and programming model hands-on.
+Built based on [JetBrains Educational Products](https://www.jetbrains.com/education/), Beam Katas
+objective is to provide a series of structured hands-on learning experiences for learners
+to understand about Apache Beam and its SDKs by solving exercises with gradually increasing
+complexity. Beam Katas are available for both Java and Python SDKs.
+
+### Java
+
+* Download [IntelliJ Edu](https://www.jetbrains.com/education/download/#section=idea)
+* Upon opening the IDE, expand the "Learn and Teach" menu, then select "Browse Courses"
+* Search for "Beam Katas - Java"
+* Expand the "Advanced Settings" and modify the "Location" and "Jdk" appropriately
+* Click "Join"
+* [Learn more](https://www.jetbrains.com/help/education/learner-start-guide.html?section=Introduction%20to%20Java#explore_course) about how to use the Education product
+
+### Python
+
+* Download [PyCharm Edu](https://www.jetbrains.com/education/download/#section=pycharm-edu)
+* Upon opening the IDE, expand the "Learn and Teach" menu, then select "Browse Courses"
+* Search for "Beam Katas - Python"
+* Expand the "Advanced Settings" and modify the "Location" and "Interpreter" appropriately
+* Click "Join"
+* [Learn more](https://www.jetbrains.com/help/education/learner-start-guide.html?section=Introduction%20to%20Python#explore_course) about how to use the Education product
+
## Code Examples {#code-examples}
### Java
diff --git a/website/src/documentation/resources/videos-and-podcasts.md b/website/src/documentation/resources/videos-and-podcasts.md
index a10ea0f..90ecc69 100644
--- a/website/src/documentation/resources/videos-and-podcasts.md
+++ b/website/src/documentation/resources/videos-and-podcasts.md
@@ -108,6 +108,26 @@
<iframe width="560" height="315" src="https://www.youtube.com/embed/E1k0B9LN46M" frameborder="0" allowfullscreen></iframe>
+### Nexmark Evaluating Big Data systems with Apache Beam
+
+ApacheCon, Miami, 2017
+
+Presented by Etienne Chauchot and Ismaël Mejia, *Apache Beam PMC members*
+
+<iframe src="//www.slideshare.net/slideshow/embed_code/key/auWXjEK7GTkiUK" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/EtienneChauchot/nexmark-with-beam" title="Nexmark with Beam" target="_blank">Nexmark with Beam</a> </strong> by <strong><a href="https://www.slideshare.net/EtienneChauchot" target="_blank">Etienne Chauchot</a></strong> </div>
+<!--<audio controls class="wp-audio-shortcode" id="audio-3597-1_html5"src="https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3?_=1"><source type="audio/mpeg" src="https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3?_=1"><a href="https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3">https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3</a></audio>-->
+<a href="https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3">Play audio podcast</a>
+
+### Universal metrics with Apache Beam
+
+ApacheCon, Montreal, 2018
+
+Presented by Etienne Chauchot, *Apache Beam PMC member*
+
+<iframe src="//www.slideshare.net/slideshow/embed_code/key/kKJRzR8HxkxLsR" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/EtienneChauchot/universal-metrics-with-apache-beam" title="Universal metrics with Apache Beam" target="_blank">Universal metrics with Apache Beam</a> </strong> by <strong><a href="https://www.slideshare.net/EtienneChauchot" target="_blank">Etienne Chauchot</a></strong> </div>
+<!--<audio controls class="wp-audio-shortcode" id="audio-3597-1_html5"src="https://feathercastapache.files.wordpress.com/2018/09/03-universal-metrics-with-beam-etienne-chauchot.mp3?_=1"><source type="audio/mpeg" src="https://feathercastapache.files.wordpress.com/2018/09/03-universal-metrics-with-beam-etienne-chauchot.mp3?_=1"><a href="https://feathercastapache.files.wordpress.com/2018/09/03-universal-metrics-with-beam-etienne-chauchot.mp3">https://feathercastapache.files.wordpress.com/2018/09/03-universal-metrics-with-beam-etienne-chauchot.mp3</a></audio> -->
+<a href="https://feathercastapache.files.wordpress.com/2018/09/03-universal-metrics-with-beam-etienne-chauchot.mp3">Play audio podcast</a>
+
## Next Steps
* Take a self-paced tour through our [Learning Resources]({{ site.baseurl }}/documentation/resources/learning-resources).
\ No newline at end of file
diff --git a/website/src/documentation/runners/direct.md b/website/src/documentation/runners/direct.md
index f61619f..2e763bf 100644
--- a/website/src/documentation/runners/direct.md
+++ b/website/src/documentation/runners/direct.md
@@ -82,4 +82,75 @@
If your pipeline uses an unbounded data source or sink, you must set the `streaming` option to `true`.
+### Execution Mode
+Python [FnApiRunner](https://beam.apache.org/contribute/runner-guide/#the-fn-api) supports multi-threading and multi-processing mode.
+
+#### Setting parallelism
+
+Number of threads or subprocesses is defined by setting the `direct_num_workers` option. There are several ways to set this option.
+
+* Passing through CLI when executing a pipeline.
+```
+python wordcount.py --input xx --output xx --direct_num_workers 2
+```
+
+* Setting with `PipelineOptions`.
+```
+from apache_beam.options.pipeline_options import PipelineOptions
+pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
+```
+
+* Adding to existing `PipelineOptions`.
+```
+from apache_beam.options.pipeline_options import DirectOptions
+pipeline_options = PipelineOptions(xxx)
+pipeline_options.view_as(DirectOptions).direct_num_workers = 2
+```
+
+#### Running with multi-threading mode
+
+```
+import argparse
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.portability import fn_api_runner
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability import python_urns
+
+parser = argparse.ArgumentParser()
+parser.add_argument(...)
+known_args, pipeline_args = parser.parse_known_args(argv)
+pipeline_options = PipelineOptions(pipeline_args)
+
+p = beam.Pipeline(options=pipeline_options,
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.EMBEDDED_PYTHON_GRPC)))
+```
+
+#### Running with multi-processing mode
+
+```
+import argparse
+import sys
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.runners.portability import fn_api_runner
+from apache_beam.portability.api import beam_runner_api_pb2
+from apache_beam.portability import python_urns
+
+parser = argparse.ArgumentParser()
+parser.add_argument(...)
+known_args, pipeline_args = parser.parse_known_args(argv)
+pipeline_options = PipelineOptions(pipeline_args)
+
+p = beam.Pipeline(options=pipeline_options,
+ runner=fn_api_runner.FnApiRunner(
+ default_environment=beam_runner_api_pb2.Environment(
+ urn=python_urns.SUBPROCESS_SDK,
+ payload=b'%s -m apache_beam.runners.worker.sdk_worker_main'
+ % sys.executable.encode('ascii'))))
+```
diff --git a/website/src/documentation/runners/flink.md b/website/src/documentation/runners/flink.md
index 6d28146..515d8e2 100644
--- a/website/src/documentation/runners/flink.md
+++ b/website/src/documentation/runners/flink.md
@@ -39,8 +39,8 @@
It is important to understand that the Flink Runner comes in two flavors:
-1. A *legacy Runner* which supports only Java (and other JVM-based languages)
-2. A *portable Runner* which supports Java/Python/Go
+1. The original *classic Runner* which supports only Java (and other JVM-based languages)
+2. The newer *portable Runner* which supports Java/Python/Go
You may ask why there are two Runners?
@@ -49,8 +49,8 @@
architecture of the Runners had to be changed significantly to support executing
pipelines written in other languages.
-If your applications only use Java, then you should currently go with the legacy
-Runner. Eventually, the portable Runner will replace the legacy Runner because
+If your applications only use Java, then you should currently go with the classic
+Runner. Eventually, the portable Runner will replace the classic Runner because
it contains the generalized framework for executing Java, Python, Go, and more
languages in the future.
@@ -59,14 +59,14 @@
portability, please visit the [Portability page]({{site.baseurl
}}/roadmap/portability/).
-Consequently, this guide is split into two parts to document the legacy and
+Consequently, this guide is split into two parts to document the classic and
the portable functionality of the Flink Runner. Please use the switcher below to
select the appropriate Runner:
<nav class="language-switcher">
<strong>Adapt for:</strong>
<ul>
- <li data-type="language-java">Legacy (Java)</li>
+ <li data-type="language-java">Classic (Java)</li>
<li data-type="language-py">Portable (Java/Python/Go)</li>
</ul>
</nav>
@@ -103,12 +103,33 @@
<th>Artifact Id</th>
</tr>
<tr>
- <td>>=2.13.0</td>
+ <td rowspan="2">2.17.0</td>
<td>1.8.x</td>
<td>beam-runners-flink-1.8</td>
</tr>
<tr>
- <td rowspan="3">>=2.10.0</td>
+ <td>1.7.x</td>
+ <td>beam-runners-flink-1.7</td>
+</tr>
+<tr>
+ <td rowspan="4">2.13.0 - 2.16.0</td>
+ <td>1.8.x</td>
+ <td>beam-runners-flink-1.8</td>
+</tr>
+<tr>
+ <td>1.7.x</td>
+ <td>beam-runners-flink-1.7</td>
+</tr>
+<tr>
+ <td>1.6.x</td>
+ <td>beam-runners-flink-1.6</td>
+</tr>
+<tr>
+ <td>1.5.x</td>
+ <td>beam-runners-flink_2.11</td>
+</tr>
+<tr>
+ <td rowspan="3">2.10.0 - 2.16.0</td>
<td>1.7.x</td>
<td>beam-runners-flink-1.7</td>
</tr>
@@ -218,7 +239,7 @@
</span>
```java
-$ bin/flink -c org.apache.beam.examples.WordCount /path/to/your.jar
+$ bin/flink run -c org.apache.beam.examples.WordCount /path/to/your.jar
--runner=FlinkRunner --other-parameters
```
@@ -247,33 +268,33 @@
As of now you will need a copy of Apache Beam's source code. You can
download it on the [Downloads page]({{ site.baseurl
}}/get-started/downloads/). In the future there will be pre-built Docker images
-available.
+available. To run a pipeline on an embedded Flink cluster:
</span>
-<span class="language-py">1. *Only required once:* Build the SDK harness container: `./gradlew :sdks:python:container:docker`
-</span>
-
-<span class="language-py">2. Start the JobService endpoint: `./gradlew :runners:flink:1.5:job-server:runShadow`
+<span class="language-py">1. Start the JobService endpoint: `./gradlew :runners:flink:1.8:job-server:runShadow`
</span>
<span class="language-py">
The JobService is the central instance where you submit your Beam pipeline to.
-The JobService will create a Flink job for the pipeline and execute the job
-job. To execute the job on a Flink cluster, the Beam JobService needs to be
+The JobService will create a Flink job for the pipeline and execute the job.
+To execute the job on a Flink cluster, the Beam JobService needs to be
provided with the Flink JobManager address.
</span>
-<span class="language-py">3. Submit the Python pipeline to the above endpoint by using the `PortableRunner` and `job_endpoint` set to `localhost:8099` (this is the default address of the JobService). For example:
+<span class="language-py">2. Submit the Python pipeline to the above endpoint by using the `PortableRunner`, `job_endpoint` set to `localhost:8099` (this is the default address of the JobService), and `environment_type` set to `LOOPBACK`. For example:
</span>
```py
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
-options = PipelineOptions(["--runner=PortableRunner", "--job_endpoint=localhost:8099"])
-p = beam.Pipeline(options)
-..
-p.run()
+options = PipelineOptions([
+ "--runner=PortableRunner",
+ "--job_endpoint=localhost:8099",
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
```
<span class="language-py">
@@ -283,12 +304,32 @@
<span class="language-py">1. Start a Flink cluster which exposes the Rest interface on `localhost:8081` by default.
</span>
-<span class="language-py">2. Start JobService with Flink Rest endpoint: `./gradlew :runners:flink:1.5:job-server:runShadow -PflinkMasterUrl=localhost:8081`.
+<span class="language-py">2. Start JobService with Flink Rest endpoint: `./gradlew :runners:flink:1.8:job-server:runShadow -PflinkMasterUrl=localhost:8081`.
</span>
<span class="language-py">3. Submit the pipeline as above.
+Note however that `environment_type=LOOPBACK` is only intended for local testing.
+See [here]({{ site.baseurl }}/roadmap/portability/#sdk-harness-config) for details.
</span>
+<span class="language-py">As of Beam 2.15.0, steps 2 and 3 can be automated in Python by using the `FlinkRunner`,
+plus the optional `flink_version` and `flink_master_url` options if required, i.e.
+</span>
+
+```py
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+
+options = PipelineOptions([
+ "--runner=FlinkRunner",
+ "--flink_version=1.8",
+ "--flink_master_url=localhost:8081",
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
+```
+
## Additional information and caveats
### Monitoring your job
@@ -588,7 +629,7 @@
The [Beam Capability Matrix]({{ site.baseurl
}}/documentation/runners/capability-matrix/) documents the
-capabilities of the legacy Flink Runner.
+capabilities of the classic Flink Runner.
The [Portable Capability
Matrix](https://s.apache.org/apache-beam-portability-support-table) documents
diff --git a/website/src/documentation/runners/jet.md b/website/src/documentation/runners/jet.md
new file mode 100644
index 0000000..6b07e7f
--- /dev/null
+++ b/website/src/documentation/runners/jet.md
@@ -0,0 +1,164 @@
+---
+layout: section
+title: "Hazelcast Jet Runner"
+section_menu: section-menu/runners.html
+permalink: /documentation/runners/jet/
+redirect_from: /learn/runners/jet/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+## Overview
+
+The Hazelcast Jet Runner can be used to execute Beam pipelines using [Hazelcast
+Jet](https://jet.hazelcast.org/).
+
+The Jet Runner and Jet are suitable for large scale continuous jobs and provide:
+* Support for both batch (bounded) and streaming (unbounded) data sets
+* A runtime that supports very high throughput and low event latency at the same time
+* Natural back-pressure in streaming programs
+* Distributed massively parallel data processing engine with in memory storage
+
+It's important to note that the Jet Runner is currently in an *EXPERIMENTAL* state and can not make use of many of
+the capabilities present in Jet:
+* Jet has full Fault Tolerance support, the Jet Runner does not; if a job fails it must be restarted
+* Internal performance of Jet is extremely high.
+The Runner can't match it as of now because Beam pipeline optimization/surgery has not been fully implemented.
+
+The [Beam Capability Matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/) documents the
+supported capabilities of the Jet Runner.
+
+## Running WordCount with the Hazelcast Jet Runner
+
+### Generating the Beam examples project ##
+Just follow the instruction from the [Java Quickstart page]({{ site.baseurl }}/get-started/quickstart-java/#get-the-wordcount-code)
+
+### Running WordCount on a Local Jet Cluster ##
+Issue following command in the Beam examples project to start new Jet cluster and run the WordCount example on it.
+```
+ $ mvn package exec:java \
+ -DskipTests \
+ -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="\
+ --runner=JetRunner \
+ --jetLocalMode=3 \
+ --inputFile=pom.xml \
+ --output=counts" \
+ -Pjet-runner
+```
+
+### Running WordCount on a Remote Jet Cluster ##
+Download latest stable Hazelcast Jet code from [Hazelcast Website](https://jet.hazelcast.org/download/) and
+start Jet cluster.
+The simplest way is to start Jet cluster member using the `jet-start` script that comes with Jet distribution.
+The members use the [auto discovery feature](https://docs.hazelcast.org/docs/3.12/manual/html-single/index.html#setting-up-clusters)
+to form a cluster. Let's start up a cluster formed by two members:
+
+```
+ $ cd hazelcast-jet
+ $ bin/jet-start.sh &
+ $ bin/jet-start.sh &
+```
+
+Check the cluster is up and running:
+```
+ $ ./jet.sh cluster
+```
+
+You should see something like:
+```
+State: ACTIVE
+Version: 3.0
+Size: 2
+
+ADDRESS UUID
+[192.168.0.117]:5701 76bea7ba-f032-4c25-ad04-bdef6782f481
+[192.168.0.117]:5702 03ecfaa2-be16-41b6-b5cf-eea584d7fb86
+```
+
+Download [Jet Management Center](https://docs.hazelcast.org/docs/jet-management-center/3.0/manual/)
+from the same location and use it to monitor your cluster and later executions.
+
+Change directory to the Beam Examples project and issue following command to submit and execute your
+Pipeline on the remote Jet cluster.
+Make sure to distribute the input file (file with the words to be counted) to all machines where the
+cluster runs. The word count job won't be able to read the data otherwise.
+
+```
+ $ mvn package exec:java \
+ -DskipTests \
+ -Dexec.mainClass=org.apache.beam.examples.WordCount \
+ -Dexec.args="\
+ --runner=JetRunner \
+ --jetServers=192.168.0.117:5701,192.168.0.117:5702 \
+ --codeJarPathname=target/word-count-beam-bundled-0.1.jar \
+ --inputFile=<INPUT_FILE_AVAILABLE_ON_ALL_CLUSTER_MEMBERS> \
+ --output=/tmp/counts" \
+ -Pjet-runner
+```
+
+## Pipeline Options for the Jet Runner
+
+<table class="table table-bordered">
+<tr>
+ <th>Field</th>
+ <th>Description</th>
+ <th>Default Value</th>
+</tr>
+<tr>
+ <td><code>runner</code></td>
+ <td>The pipeline runner to use. This option allows you to determine the pipeline runner at runtime.</td>
+ <td>Set to <code>JetRunner</code> to run using Jet.</td>
+</tr>
+<tr>
+ <td><code>jetGroupName</code></td>
+ <td>The name of the Hazelcast Group to join, in essence an ID of the Jet Cluster that will be
+ used by the Runner. With groups it is possible to create multiple clusters where each cluster has its own
+ group and doesn't interfere with other clusters.</td>
+ <td><code>jet</code></td>
+</tr>
+<tr>
+ <td><code>jetServers</code></td>
+ <td>List of the addresses of Jet Cluster members, needed when the Runner doesn't start its own Jet Cluster,
+ but makes use of an external, independently started one. Takes the form of a comma separated list of ip/hostname-port pairs,
+ like this: <code>192.168.0.117:5701,192.168.0.117:5702</code></td>
+ <td><code>127.0.0.1:5701</code></td>
+</tr>
+<tr>
+ <td><code>codeJarPathname</code></td>
+ <td>Also a property needed only when using external Jet Clusters, specifies the location of a fat jar
+ containing all the code that needs to run on the cluster (so at least the pipeline and the runner code). The value
+ is any string that is acceptad by <code>new java.io.File()</code> as a parameter.</td>
+ <td>Has no default value.</td>
+</tr>
+<tr>
+ <td><code>jetLocalMode</code></td>
+ <td>The number of Jet Cluster members that should be started locally by the Runner. If it's <code>0</code>
+ then the Runner will be using an external cluster. If greater, then the Runner will be using a cluster started by itself.</td>
+ <td><code>0</code></td>
+</tr>
+<tr>
+ <td><code>jetDefaultParallelism</code></td>
+ <td>Local parallelism of Jet members, the number of processors of each vertex of the DAG that will be created on each
+ Jet Cluster member.</td>
+ <td><code>2</code></td>
+</tr>
+<tr>
+ <td><code>jetProcessorsCooperative</code></td>
+ <td>Boolean flag specifying if Jet Processors for DoFns are allowed to be cooperative (ie. use green threads instead of
+ dedicated OS ones). If set to true than all such Processors will be cooperative, except when they have no outputs
+ (so they are assumed to be syncs).</td>
+ <td><code>false</code></td>
+</tr>
+</table>
diff --git a/website/src/documentation/runners/spark.md b/website/src/documentation/runners/spark.md
index 49fa7ee..fa48df6 100644
--- a/website/src/documentation/runners/spark.md
+++ b/website/src/documentation/runners/spark.md
@@ -35,11 +35,41 @@
_**Note:**_ _support for the Beam Model in streaming is currently experimental, follow-up in the [mailing list]({{ site.baseurl }}/get-started/support/) for status updates._
+## Portability
+
+The Spark runner comes in two flavors:
+
+1. A *legacy Runner* which supports only Java (and other JVM-based languages)
+2. A *portable Runner* which supports Java, Python, and Go
+
+Beam and its Runners originally only supported JVM-based languages
+(e.g. Java/Scala/Kotlin). Python and Go SDKs were added later on. The
+architecture of the Runners had to be changed significantly to support executing
+pipelines written in other languages.
+
+If your applications only use Java, then you should currently go with the legacy
+Runner. If you want to run Python or Go pipelines with Beam on Spark, you need to use
+the portable Runner. For more information on portability, please visit the
+[Portability page]({{site.baseurl }}/roadmap/portability/).
+
+This guide is split into two parts to document the legacy and
+the portable functionality of the Spark Runner. Please use the switcher below to
+select the appropriate Runner:
+
+<nav class="language-switcher">
+ <strong>Adapt for:</strong>
+ <ul>
+ <li data-type="language-java">Legacy (Java)</li>
+ <li data-type="language-py">Portable (Java/Python/Go)</li>
+ </ul>
+</nav>
+
## Spark Runner prerequisites and setup
The Spark runner currently supports Spark's 2.x branch, and more specifically any version greater than 2.2.0.
-You can add a dependency on the latest version of the Spark runner by adding to your pom.xml the following:
+<span class="language-java">You can add a dependency on the latest version of the Spark runner by adding to your pom.xml the following:</span>
+
```java
<dependency>
<groupId>org.apache.beam</groupId>
@@ -50,7 +80,8 @@
### Deploying Spark with your application
-In some cases, such as running in local mode/Standalone, your (self-contained) application would be required to pack Spark by explicitly adding the following dependencies in your pom.xml:
+<span class="language-java">In some cases, such as running in local mode/Standalone, your (self-contained) application would be required to pack Spark by explicitly adding the following dependencies in your pom.xml:</span>
+
```java
<dependency>
<groupId>org.apache.spark</groupId>
@@ -65,7 +96,8 @@
</dependency>
```
-And shading the application jar using the maven shade plugin:
+<span class="language-java">And shading the application jar using the maven shade plugin:</span>
+
```java
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -102,14 +134,60 @@
</plugin>
```
-After running <code>mvn package</code>, run <code>ls target</code> and you should see (assuming your artifactId is `beam-examples` and the version is `1.0.0`):
-```
+<span class="language-java">After running <code>mvn package</code>, run <code>ls target</code> and you should see (assuming your artifactId is `beam-examples` and the version is `1.0.0`):</span>
+
+<code class="language-java">
beam-examples-1.0.0-shaded.jar
+</code>
+
+<span class="language-java">To run against a Standalone cluster simply run:</span>
+
+<code class="language-java">
+spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT target/beam-examples-1.0.0-shaded.jar --runner=SparkRunner
+</code>
+
+<span class="language-py">
+You will need Docker to be installed in your execution environment. To develop
+Apache Beam with Python you have to install the Apache Beam Python SDK: `pip
+install apache_beam`. Please refer to the [Python documentation]({{ site.baseurl }}/documentation/sdks/python/)
+on how to create a Python pipeline.
+</span>
+
+```python
+pip install apache_beam
```
-To run against a Standalone cluster simply run:
-```
-spark-submit --class com.beam.examples.BeamPipeline --master spark://HOST:PORT target/beam-examples-1.0.0-shaded.jar --runner=SparkRunner
+<span class="language-py">
+As of now you will need a copy of Apache Beam's source code. You can
+download it on the [Downloads page]({{ site.baseurl
+}}/get-started/downloads/). In the future there will be pre-built Docker images
+available.
+</span>
+
+<span class="language-py">1. Start the JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`
+</span>
+
+<span class="language-py">
+The JobService is the central instance where you submit your Beam pipeline.
+The JobService will create a Spark job for the pipeline and execute the
+job. To execute the job on a Spark cluster, the Beam JobService needs to be
+provided with the Spark master address.
+</span>
+
+<span class="language-py">2. Submit the Python pipeline to the above endpoint by using the `PortableRunner`, `job_endpoint` set to `localhost:8099` (this is the default address of the JobService), and `environment_type` set to `LOOPBACK`. For example:
+</span>
+
+```py
+import apache_beam as beam
+from apache_beam.options.pipeline_options import PipelineOptions
+
+options = PipelineOptions([
+ "--runner=PortableRunner",
+ "--job_endpoint=localhost:8099",
+ "--environment_type=LOOPBACK"
+])
+with beam.Pipeline(options) as p:
+ ...
```
### Running on a pre-deployed Spark cluster
@@ -117,11 +195,27 @@
Deploying your Beam pipeline on a cluster that already has a Spark deployment (Spark classes are available in container classpath) does not require any additional dependencies.
For more details on the different deployment modes see: [Standalone](http://spark.apache.org/docs/latest/spark-standalone.html), [YARN](http://spark.apache.org/docs/latest/running-on-yarn.html), or [Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html).
+<span class="language-py">1. Start a Spark cluster which exposes the master on port 7077 by default.
+</span>
+
+<span class="language-py">2. Start JobService that will connect with the Spark master: `./gradlew :runners:spark:job-server:runShadow -PsparkMasterUrl=spark://localhost:7077`.
+</span>
+
+<span class="language-py">3. Submit the pipeline as above.
+Note however that `environment_type=LOOPBACK` is only intended for local testing.
+See [here]({{ site.baseurl }}/roadmap/portability/#sdk-harness-config) for details.
+</span>
+
+<span class="language-py">
+(Note that, depending on your cluster setup, you may need to change the `environment_type` option.
+See [here]({{ site.baseurl }}/roadmap/portability/#sdk-harness-config) for details.)
+</span>
+
## Pipeline options for the Spark Runner
When executing your pipeline with the Spark Runner, you should consider the following pipeline options.
-<table class="table table-bordered">
+<table class="language-java table table-bordered">
<tr>
<th>Field</th>
<th>Description</th>
@@ -159,6 +253,24 @@
</tr>
</table>
+<table class="language-py table table-bordered">
+<tr>
+ <th>Field</th>
+ <th>Description</th>
+ <th>Value</th>
+</tr>
+<tr>
+ <td><code>--runner</code></td>
+ <td>The pipeline runner to use. This option allows you to determine the pipeline runner at runtime.</td>
+ <td>Set to <code>PortableRunner</code> to run using Spark.</td>
+</tr>
+<tr>
+ <td><code>--job_endpoint</code></td>
+ <td>Job service endpoint to use. Should be in the form hostname:port, e.g. localhost:3000</td>
+ <td>Set to match your job service endpoint (localhost:8099 by default)</td>
+</tr>
+</table>
+
## Additional notes
### Using spark-submit
@@ -172,14 +284,23 @@
You can monitor a running Spark job using the Spark [Web Interfaces](http://spark.apache.org/docs/latest/monitoring.html#web-interfaces). By default, this is available at port `4040` on the driver node. If you run Spark on your local machine that would be `http://localhost:4040`.
Spark also has a history server to [view after the fact](http://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact).
+<span class="language-java">
Metrics are also available via [REST API](http://spark.apache.org/docs/latest/monitoring.html#rest-api).
Spark provides a [metrics system](http://spark.apache.org/docs/latest/monitoring.html#metrics) that allows reporting Spark metrics to a variety of Sinks. The Spark runner reports user-defined Beam Aggregators using this same metrics system and currently supports <code>GraphiteSink</code> and <code>CSVSink</code>, and providing support for additional Sinks supported by Spark is easy and straight-forward.
+</span>
+<span class="language-py">Spark metrics are not yet supported on the portable runner.</span>
### Streaming Execution
+<span class="language-java">
If your pipeline uses an <code>UnboundedSource</code> the Spark Runner will automatically set streaming mode. Forcing streaming mode is mostly used for testing and is not recommended.
+</span>
+<span class="language-py">Streaming is not yet supported on the Spark portable runner.</span>
### Using a provided SparkContext and StreamingListeners
+<span class="language-java">
If you would like to execute your Spark job with a provided <code>SparkContext</code>, such as when using the [spark-jobserver](https://github.com/spark-jobserver/spark-jobserver), or use <code>StreamingListeners</code>, you can't use <code>SparkPipelineOptions</code> (the context or a listener cannot be passed as a command-line argument anyway).
Instead, you should use <code>SparkContextOptions</code> which can only be used programmatically and is not a common <code>PipelineOptions</code> implementation.
+</span>
+<span class="language-py">Provided SparkContext and StreamingListeners are not supported on the Spark portable runner.</span>
diff --git a/website/src/documentation/runtime/environments.md b/website/src/documentation/runtime/environments.md
new file mode 100644
index 0000000..22a96dd
--- /dev/null
+++ b/website/src/documentation/runtime/environments.md
@@ -0,0 +1,183 @@
+---
+layout: section
+title: "Runtime environments"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/environments/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Runtime environments
+
+The Beam SDK runtime environment is isolated from other runtime systems because the SDK runtime environment is [containerized](https://s.apache.org/beam-fn-api-container-contract) with [Docker](https://www.docker.com/). This means that any execution engine can run the Beam SDK.
+
+This page describes how to customize, build, and push Beam SDK container images.
+
+Before you begin, install [Docker](https://www.docker.com/) on your workstation.
+
+## Customizing container images
+
+You can add extra dependencies to container images so that you don't have to supply the dependencies to execution engines.
+
+To customize a container image, either:
+* [Write a new](#writing-new-dockerfiles) [Dockerfile](https://docs.docker.com/engine/reference/builder/) on top of the original.
+* [Modify](#modifying-dockerfiles) the [original Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile) and reimage the container.
+
+It's often easier to write a new Dockerfile. However, by modifying the original Dockerfile, you can customize anything (including the base OS).
+
+### Writing new Dockerfiles on top of the original {#writing-new-dockerfiles}
+
+1. Pull a [prebuilt SDK container image](https://hub.docker.com/u/apachebeam) for your [target](https://docs.docker.com/docker-hub/repos/#searching-for-repositories) language and version. The following example pulls the latest Python SDK:
+```
+docker pull apachebeam/python3.7_sdk
+```
+2. [Write a new Dockerfile](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) that [designates](https://docs.docker.com/engine/reference/builder/#from) the original as its [parent](https://docs.docker.com/glossary/?term=parent%20image).
+3. [Build](#building-container-images) a child image.
+
+### Modifying the original Dockerfile {#modifying-dockerfiles}
+
+1. Clone the `beam` repository:
+```
+git clone https://github.com/apache/beam.git
+```
+2. Customize the [Dockerfile](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile). If you're adding dependencies from [PyPI](https://pypi.org/), use [`base_image_requirements.txt`](https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt) instead.
+3. [Reimage](#building-container-images) the container.
+
+### Testing customized images
+
+To test a customized image locally, run a pipeline with PortableRunner and set the `--environment_config` flag to the image path:
+
+{:.runner-direct}
+
+```
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output /path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=embed \
+--environment_config=path/to/container/image
+```
+
+{:.runner-flink-local}
+
+```
+# Start a Flink job server on localhost:8099
+./gradlew :runners:flink:1.5:job-server:runShadow
+
+# Run a pipeline on the Flink job server
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output=/path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=localhost:8099 \
+--environment_config=path/to/container/image
+```
+
+{:.runner-spark-local}
+
+```
+# Start a Spark job server on localhost:8099
+./gradlew :runners:spark:job-server:runShadow
+
+# Run a pipeline on the Spark job server
+python -m apache_beam.examples.wordcount \
+--input=/path/to/inputfile \
+--output=path/to/write/counts \
+--runner=PortableRunner \
+--job_endpoint=localhost:8099 \
+--environment_config=path/to/container/image
+```
+
+To test a customized image on the Google Cloud Dataflow runner, use the `DataflowRunner` option and the `worker_harness_container_image` flag:
+
+```
+python -m apache_beam.examples.wordcount \
+--input=path/to/inputfile \
+--output=/path/to/write/counts \
+--runner=DataflowRunner \
+--project={gcp_project_id} \
+--temp_location={gcs_location} \ \
+--experiment=beam_fn_api \
+--sdk_location=[…]/beam/sdks/python/container/py{version}/build/target/apache-beam.tar.gz \
+--worker_harness_container_image=path/to/container/image
+
+# The sdk_location option accepts four Python version variables: 2, 35, 36, and 37
+```
+
+## Building container images
+
+To build Beam SDK container images:
+
+1. Navigate to the local copy of your [customized container image](#customizing-container-images).
+2. Run Gradle with the `docker` target. If you're [building a child image](#writing-new-dockerfiles), set the optional `--file` flag to the new Dockerfile. If you're [building an image from an original Dockerfile](#modifying-dockerfiles), ignore the `--file` flag and use a default repository:
+
+```
+# The default repository of each SDK
+./gradlew [--file=path/to/new/Dockerfile] :sdks:java:container:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:go:container:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py2:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py35:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py36:docker
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container:py37:docker
+
+# Shortcut for building all four Python SDKs
+./gradlew [--file=path/to/new/Dockerfile] :sdks:python:container buildAll
+```
+
+To examine the containers that you built, run `docker images` from anywhere in the command line. If you successfully built all of the container images, the command prints a table like the following:
+```
+REPOSITORY TAG IMAGE ID CREATED SIZE
+apachebeam/java_sdk latest 16ca619d489e 2 weeks ago 550MB
+apachebeam/python2.7_sdk latest b6fb40539c29 2 weeks ago 1.78GB
+apachebeam/python3.5_sdk latest bae309000d09 2 weeks ago 1.85GB
+apachebeam/python3.6_sdk latest 42faad307d1a 2 weeks ago 1.86GB
+apachebeam/python3.7_sdk latest 18267df54139 2 weeks ago 1.86GB
+apachebeam/go_sdk latest 30cf602e9763 2 weeks ago 124MB
+```
+
+### Overriding default Docker targets
+
+The default [tag](https://docs.docker.com/engine/reference/commandline/tag/) is `latest` and the default repositories are in the Docker Hub `apachebeam` namespace. The `docker` command-line tool implicitly [pushes container images](#pushing-container-images) to this location.
+
+To tag a local image, set the `docker-tag` option when building the container. The following command tags a Python SDK image with a date.
+```
+./gradlew :sdks:python:container:py2:docker -Pdocker-tag=2019-10-04
+```
+
+To change the repository, set the `docker-repository-root` option to a new location. The following command sets the `docker-repository-root` to a Bintray repository named `apache`.
+```
+./gradlew :sdks:python:container:py2:docker -Pdocker-repository-root=$USER-docker-apache.bintray.io/beam/python
+```
+
+## Pushing container images
+
+After [building a container image](#building-container-images), you can store it in a remote Docker repository.
+
+The following steps push a Python SDK image to the [`docker-root-repository` value](#overriding-default-docker-targets).
+
+1. Sign in to your Docker registry:
+```
+docker login
+```
+2. Navigate to the local copy of your container image and upload it to the remote repository:
+```
+docker push apachebeam/python2.7_sdk
+```
+
+To download the image again, run `docker pull`:
+```
+docker pull apachebeam/python2.7_sdk
+```
+
+> **Note**: After pushing a container image, the remote image ID and digest match the local image ID and digest.
\ No newline at end of file
diff --git a/website/src/documentation/runtime/model.md b/website/src/documentation/runtime/model.md
new file mode 100644
index 0000000..4affaca
--- /dev/null
+++ b/website/src/documentation/runtime/model.md
@@ -0,0 +1,212 @@
+---
+layout: section
+title: "Execution model"
+section_menu: section-menu/documentation.html
+permalink: /documentation/runtime/model/
+redirect_from:
+ - /documentation/execution-model/
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Execution model
+
+The Beam model allows runners to execute your pipeline in different ways. You
+may observe various effects as a result of the runner’s choices. This page
+describes these effects so you can better understand how Beam pipelines execute.
+
+## Processing of elements
+
+The serialization and communication of elements between machines is one of the
+most expensive operations in a distributed execution of your pipeline. Avoiding
+this serialization may require re-processing elements after failures or may
+limit the distribution of output to other machines.
+
+### Serialization and communication
+
+The runner might serialize elements between machines for communication purposes
+and for other reasons such as persistence.
+
+A runner may decide to transfer elements between transforms in a variety of
+ways, such as:
+
+* Routing elements to a worker for processing as part of a grouping operation.
+ This may involve serializing elements and grouping or sorting them by their
+ key.
+* Redistributing elements between workers to adjust parallelism. This may
+ involve serializing elements and communicating them to other workers.
+* Using the elements in a side input to a `ParDo`. This may require
+ serializing the elements and broadcasting them to all the workers executing
+ the `ParDo`.
+* Passing elements between transforms that are running on the same worker.
+ This may allow the runner to avoid serializing elements; instead, the runner
+ can just pass the elements in memory.
+
+Some situations where the runner may serialize and persist elements are:
+
+1. When used as part of a stateful `DoFn`, the runner may persist values to some
+ state mechanism.
+1. When committing the results of processing, the runner may persist the outputs
+ as a checkpoint.
+
+### Bundling and persistence
+
+Beam pipelines often focus on "[embarassingly parallel](https://en.wikipedia.org/wiki/embarrassingly_parallel)"
+problems. Because of this, the APIs emphasize processing elements in parallel,
+which makes it difficult to express actions like "assign a sequence number to
+each element in a PCollection". This is intentional as such algorithms are much
+more likely to suffer from scalability problems.
+
+Processing all elements in parallel also has some drawbacks. Specifically, it
+makes it impossible to batch any operations, such as writing elements to a sink
+or checkpointing progress during processing.
+
+Instead of processing all elements simultaneously, the elements in a
+`PCollection` are processed in _bundles_. The division of the collection into
+bundles is arbitrary and selected by the runner. This allows the runner to
+choose an appropriate middle-ground between persisting results after every
+element, and having to retry everything if there is a failure. For example, a
+streaming runner may prefer to process and commit small bundles, and a batch
+runner may prefer to process larger bundles.
+
+## Failures and parallelism within and between transforms {#parallelism}
+
+In this section, we discuss how elements in the input collection are processed
+in parallel, and how transforms are retried when failures occur.
+
+### Data-parallelism within one transform {#data-parallelism}
+
+When executing a single `ParDo`, a runner might divide an example input
+collection of nine elements into two bundles as shown in figure 1.
+
+![Bundle A contains five elements. Bundle B contains four elements.](
+ {{ "/images/execution_model_bundling.svg" | prepend: site.baseurl }})
+
+*Figure 1: A runner divides an input collection into two bundles.*
+
+When the `ParDo` executes, workers may process the two bundles in parallel as
+shown in figure 2.
+
+![Two workers process the two bundles in parallel. Worker one processes bundle
+ A. Worker two processes bundle B.](
+ {{ "/images/execution_model_bundling_gantt.svg" | prepend: site.baseurl }})
+
+*Figure 2: Two workers process the two bundles in parallel.*
+
+Since elements cannot be split, the maximum parallelism for a transform depends
+on the number of elements in the collection. In figure 3, the input collection
+has nine elements, so the maximum parallelism is nine.
+
+![Nine workers process a nine element input collection in parallel.](
+ {{ "/images/execution_model_bundling_gantt_max.svg" | prepend: site.baseurl }})
+
+*Figure 3: Nine workers process a nine element input collection in parallel.*
+
+Note: Splittable ParDo allows splitting the processing of a single input across
+multiple bundles. This feature is a work in progress.
+
+### Dependent-parallelism between transforms {#dependent-parallellism}
+
+`ParDo` transforms that are in sequence may be _dependently parallel_ if the
+runner chooses to execute the consuming transform on the producing transform's
+output elements without altering the bundling. In figure 4, `ParDo1` and
+`ParDo2` are _dependently parallel_ if the output of `ParDo1` for a given
+element must be processed on the same worker.
+
+![ParDo1 processes an input collection that contains bundles A and B. ParDo2 then
+ processes the output collection from ParDo1, which contains bundles C and D.](
+ {{ "/images/execution_model_bundling_multi.svg" | prepend: site.baseurl }})
+
+*Figure 4: Two transforms in sequence and their corresponding input collections.*
+
+Figure 5 shows how these dependently parallel transforms might execute. The
+first worker executes `ParDo1` on the elements in bundle A (which results in
+bundle C), and then executes `ParDo2` on the elements in bundle C. Similarly,
+the second worker executes `ParDo1` on the elements in bundle B (which results
+in bundle D), and then executes `ParDo2` on the elements in bundle D.
+
+![Worker one executes ParDo1 on bundle A and Pardo2 on bundle C. Worker two
+ executes ParDo1 on bundle B and ParDo2 on bundle D.](
+ {{ "/images/execution_model_bundling_multi_gantt.svg" | prepend: site.baseurl }})
+
+*Figure 5: Two workers execute dependently parallel ParDo transforms.*
+
+Executing transforms this way allows a runner to avoid redistributing elements
+between workers, which saves on communication costs. However, the maximum parallelism
+now depends on the maximum parallelism of the first of the dependently parallel
+steps.
+
+### Failures within one transform
+
+If processing of an element within a bundle fails, the entire bundle fails. The
+elements in the bundle must be retried (otherwise the entire pipeline fails),
+although they do not need to be retried with the same bundling.
+
+For this example, we will use the `ParDo` from figure 1 that has an input
+collection with nine elements and is divided into two bundles.
+
+In figure 6, the first worker successfully processes all five elements in bundle
+A. The second worker processes the four elements in bundle B: the first two
+elements were successfully processed, the third element’s processing failed, and
+there is one element still awaiting processing.
+
+We see that the runner retries all elements in bundle B and the processing
+completes successfully the second time. Note that the retry does not necessarily
+happen on the same worker as the original processing attempt, as shown in the
+figure.
+
+![Worker two fails to process an element in bundle B. Worker one finishes
+ processing bundle A and then successfully retries to execute bundle B.](
+ {{ "/images/execution_model_failure_retry.svg" | prepend: site.baseurl }})
+
+*Figure 6: The processing of an element within bundle B fails, and another worker
+retries the entire bundle.*
+
+Because we encountered a failure while processing an element in the input
+bundle, we had to reprocess _all_ of the elements in the input bundle. This means
+the runner must throw away the entire output bundle since all of the results it
+contains will be recomputed.
+
+Note that if the failed transform is a `ParDo`, then the `DoFn` instance is torn
+down and abandoned.
+
+### Coupled failure: Failures between transforms {#coupled-failure}
+
+If a failure to process an element in `ParDo2` causes `ParDo1` to re-execute,
+these two steps are said to be _co-failing_.
+
+For this example, we will use the two `ParDo`s from figure 4.
+
+In figure 7, worker two successfully executes `ParDo1` on all elements in bundle
+B. However, the worker fails to process an element in bundle D, so `ParDo2`
+fails (shown as the red X). As a result, the runner must discard and recompute
+the output of `ParDo2`. Because the runner was executing `ParDo1` and `ParDo2`
+together, the output bundle from `ParDo1` must also be thrown away, and all
+elements in the input bundle must be retried. These two `ParDo`s are co-failing.
+
+![Worker two fails to process en element in bundle D, so all elements in both
+ bundle B and bundle D must be retried.](
+ {{ "/images/execution_model_bundling_coupled_failure.svg" | prepend: site.baseurl }})
+
+*Figure 7: Processing of an element within bundle D fails, so all elements in
+the input bundle are retried.*
+
+Note that the retry does not necessarily have the same processing time as the
+original attempt, as shown in the diagram.
+
+All `DoFns` that experience coupled failures are terminated and must be torn
+down since they aren’t following the normal `DoFn` lifecycle .
+
+Executing transforms this way allows a runner to avoid persisting elements
+between transforms, saving on persistence costs.
\ No newline at end of file
diff --git a/website/src/documentation/sdks/euphoria.md b/website/src/documentation/sdks/euphoria.md
index 3819c13..81e015e 100644
--- a/website/src/documentation/sdks/euphoria.md
+++ b/website/src/documentation/sdks/euphoria.md
@@ -528,7 +528,7 @@
```
## Translation
-Euphoria API is build on top of Beam Java SDK. The API is transparently translated into Beam's `PTransforms` in background.
+Euphoria API is built on top of Beam Java SDK. The API is transparently translated into Beam's `PTransforms` in background.
The fact that Euphoria API is translated to Beam Java SDK give us option to fine tune the translation itself. Translation of an `Operator` is realized through implementations of `OperatorTranslator`.
Euphoria uses `TranslationProvider` to decide which translator should be used. User of Euphoria API can supply its own `OperatorTranslator` through `TranslationProvider` by extending `EuphoriaOptions`.
diff --git a/website/src/documentation/sdks/nexmark.md b/website/src/documentation/sdks/nexmark.md
index d319a00..d5230da 100644
--- a/website/src/documentation/sdks/nexmark.md
+++ b/website/src/documentation/sdks/nexmark.md
@@ -149,7 +149,7 @@
-P nexmark.runner
The Gradle project name of the runner, such as ":runners:direct-java" or
- ":runners:flink:1.5. The project names can be found in the root
+ ":runners:flink:1.8. The project names can be found in the root
`settings.gradle`.
Test data is deterministically synthesized on demand. The test
@@ -557,7 +557,7 @@
Batch Mode:
./gradlew :sdks:java:testing:nexmark:run \
- -Pnexmark.runner=":runners:flink:1.5" \
+ -Pnexmark.runner=":runners:flink:1.8" \
-Pnexmark.args="
--runner=FlinkRunner
--suite=SMOKE
@@ -570,7 +570,7 @@
Streaming Mode:
./gradlew :sdks:java:testing:nexmark:run \
- -Pnexmark.runner=":runners:flink:1.5" \
+ -Pnexmark.runner=":runners:flink:1.8" \
-Pnexmark.args="
--runner=FlinkRunner
--suite=SMOKE
@@ -618,7 +618,7 @@
Launch:
./gradlew :sdks:java:testing:nexmark:run \
- -Pnexmark.runner=":beam-runners-google-cloud-dataflow" \
+ -Pnexmark.runner=":runners:google-cloud-dataflow-java" \
-Pnexmark.args="
--runner=DataflowRunner
--suite=SMOKE
@@ -630,17 +630,14 @@
--zone=${ZONE}
--workerMachineType=n1-highmem-8
--stagingLocation=${STAGING_LOCATION}
- --streaming=true
--sourceType=PUBSUB
--pubSubMode=PUBLISH_ONLY
--pubsubTopic=${PUBSUB_TOPIC}
--resourceNameMode=VERBATIM
--manageResources=false
- --monitorJobs=false
--numEventGenerators=64
--numWorkers=16
--maxNumWorkers=16
- --suite=SMOKE
--firstEventRate=100000
--nextEventRate=100000
--ratePeriodSec=3600
diff --git a/website/src/documentation/sdks/python-dependencies.md b/website/src/documentation/sdks/python-dependencies.md
index 53440fe..70da2bd 100644
--- a/website/src/documentation/sdks/python-dependencies.md
+++ b/website/src/documentation/sdks/python-dependencies.md
@@ -29,6 +29,165 @@
<p>To see the compile and runtime dependencies for your Beam SDK version, expand
the relevant section below.</p>
+<details><summary markdown="span"><b>2.16.0</b></summary>
+
+<p>Beam SDK for Python 2.16.0 has the following compile and
+ runtime dependencies.</p>
+
+<table class="table-bordered table-striped">
+ <tr><th>Package</th><th>Version</th></tr>
+ <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr>
+ <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr>
+ <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr>
+ <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr>
+ <tr><td>dill</td><td>>=0.3.0,<0.3.1</td></tr>
+ <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr>
+ <tr><td>funcsigs</td><td>>=1.0.2,<2; python_version < "3.0"</td></tr>
+ <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr>
+ <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr>
+ <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr>
+ <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.18.0</td></tr>
+ <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<1.1.0</td></tr>
+ <tr><td>google-cloud-core</td><td>>=0.28.1,<2</td></tr>
+ <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr>
+ <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<1.1.0</td></tr>
+ <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr>
+ <tr><td>grpcio</td><td>>=1.12.1,<2</td></tr>
+ <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr>
+ <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr>
+ <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr>
+ <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr>
+ <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr>
+ <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr>
+ <tr><td>pyarrow</td><td>>=0.11.1,<0.15.0; python_version >= "3.0" or platform_system != "Windows"</td></tr>
+ <tr><td>pydot</td><td>>=1.2.0,<2</td></tr>
+ <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr>
+ <tr><td>python-dateutil</td><td>>=2.8.0,<3</td></tr>
+ <tr><td>pytz</td><td>>=2018.3</td></tr>
+ <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr>
+ <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr>
+ <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr>
+</table>
+
+</details>
+
+<details><summary markdown="span"><b>2.15.0</b></summary>
+
+<p>Beam SDK for Python 2.15.0 has the following compile and
+ runtime dependencies.</p>
+<table class="table-bordered table-striped">
+ <tr><th>Package</th><th>Version</th></tr>
+ <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr>
+ <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr>
+ <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr>
+ <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr>
+ <tr><td>dill</td><td>>=0.2.9,<0.2.10</td></tr>
+ <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr>
+ <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr>
+ <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr>
+ <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr>
+ <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.18.0</td></tr>
+ <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<0.33.0</td></tr>
+ <tr><td>google-cloud-core</td><td>>=0.28.1,<2</td></tr>
+ <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr>
+ <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<0.40.0</td></tr>
+ <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr>
+ <tr><td>grpcio</td><td>>=1.8,<2</td></tr>
+ <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr>
+ <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr>
+ <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr>
+ <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr>
+ <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr>
+ <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr>
+ <tr><td>pyarrow</td><td>>=0.11.1,<0.15.0; python_version >= "3.0" or platform_system != "Windows"</td></tr>
+ <tr><td>pydot</td><td>>=1.2.0,<2</td></tr>
+ <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr>
+ <tr><td>pytz</td><td>>=2018.3</td></tr>
+ <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr>
+ <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr>
+ <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr>
+</table>
+
+</details>
+
+<details><summary markdown="span"><b>2.14.0</b></summary>
+
+<p>Beam SDK for Python 2.14.0 has the following compile and
+ runtime dependencies.</p>
+<table class="table-bordered table-striped">
+ <tr><th>Package</th><th>Version</th></tr>
+ <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr>
+ <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr>
+ <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr>
+ <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr>
+ <tr><td>dill</td><td>>=0.2.9,<0.2.10</td></tr>
+ <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr>
+ <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr>
+ <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr>
+ <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr>
+ <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.7.0</td></tr>
+ <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<0.33.0</td></tr>
+ <tr><td>google-cloud-core</td><td>>=0.28.1,<0.30.0</td></tr>
+ <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr>
+ <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<0.40.0</td></tr>
+ <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr>
+ <tr><td>grpcio</td><td>>=1.8,<2</td></tr>
+ <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr>
+ <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr>
+ <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr>
+ <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr>
+ <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr>
+ <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr>
+ <tr><td>pyarrow</td><td>>=0.11.1,<0.15.0; python_version >= "3.0" or platform_system != "Windows"</td></tr>
+ <tr><td>pydot</td><td>>=1.2.0,<1.3</td></tr>
+ <tr><td>pymongo</td><td>>=3.8.0,<4.0.0</td></tr>
+ <tr><td>pytz</td><td>>=2018.3</td></tr>
+ <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr>
+ <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr>
+ <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr>
+</table>
+
+</details>
+
+<details><summary markdown="span"><b>2.13.0</b></summary>
+
+<p>Beam SDK for Python 2.13.0 has the following compile and
+ runtime dependencies.</p>
+
+<table class="table-bordered table-striped">
+ <tr><th>Package</th><th>Version</th></tr>
+ <tr><td>avro-python3</td><td>>=1.8.1,<2.0.0; python_version >= "3.0"</td></tr>
+ <tr><td>avro</td><td>>=1.8.1,<2.0.0; python_version < "3.0"</td></tr>
+ <tr><td>cachetools</td><td>>=3.1.0,<4</td></tr>
+ <tr><td>crcmod</td><td>>=1.7,<2.0</td></tr>
+ <tr><td>dill</td><td>>=0.2.9,<0.2.10</td></tr>
+ <tr><td>fastavro</td><td>>=0.21.4,<0.22</td></tr>
+ <tr><td>future</td><td>>=0.16.0,<1.0.0</td></tr>
+ <tr><td>futures</td><td>>=3.2.0,<4.0.0; python_version < "3.0"</td></tr>
+ <tr><td>google-apitools</td><td>>=0.5.28,<0.5.29</td></tr>
+ <tr><td>google-cloud-bigquery</td><td>>=1.6.0,<1.7.0</td></tr>
+ <tr><td>google-cloud-bigtable</td><td>>=0.31.1,<0.33.0</td></tr>
+ <tr><td>google-cloud-core</td><td>>=0.28.1,<0.30.0</td></tr>
+ <tr><td>google-cloud-datastore</td><td>>=1.7.1,<1.8.0</td></tr>
+ <tr><td>google-cloud-pubsub</td><td>>=0.39.0,<0.40.0</td></tr>
+ <tr><td>googledatastore</td><td>>=7.0.1,<7.1; python_version < "3.0"</td></tr>
+ <tr><td>grpcio</td><td>>=1.8,<2</td></tr>
+ <tr><td>hdfs</td><td>>=2.1.0,<3.0.0</td></tr>
+ <tr><td>httplib2</td><td>>=0.8,<=0.12.0</td></tr>
+ <tr><td>mock</td><td>>=1.0.1,<3.0.0</td></tr>
+ <tr><td>oauth2client</td><td>>=2.0.1,<4</td></tr>
+ <tr><td>proto-google-cloud-datastore-v1</td><td>>=0.90.0,<=0.90.4; python_version < "3.0"</td></tr>
+ <tr><td>protobuf</td><td>>=3.5.0.post1,<4</td></tr>
+ <tr><td>pyarrow</td><td>>=0.11.1,<0.14.0; python_version >= "3.0" or platform_system != "Windows"</td></tr>
+ <tr><td>pydot</td><td>>=1.2.0,<1.3</td></tr>
+ <tr><td>pytz</td><td>>=2018.3</td></tr>
+ <tr><td>pyvcf</td><td>>=0.6.8,<0.7.0; python_version < "3.0"</td></tr>
+ <tr><td>pyyaml</td><td>>=3.12,<4.0.0</td></tr>
+ <tr><td>typing</td><td>>=3.6.0,<3.7.0; python_version < "3.5.0"</td></tr>
+</table>
+
+</details>
+
<details><summary markdown="span"><b>2.12.0</b></summary>
<p>Beam SDK for Python 2.12.0 has the following compile and
diff --git a/website/src/documentation/sdks/python-pipeline-dependencies.md b/website/src/documentation/sdks/python-pipeline-dependencies.md
index 327e00b..46d2fc4 100644
--- a/website/src/documentation/sdks/python-pipeline-dependencies.md
+++ b/website/src/documentation/sdks/python-pipeline-dependencies.md
@@ -23,7 +23,7 @@
When you run your pipeline locally, the packages that your pipeline depends on are available because they are installed on your local machine. However, when you want to run your pipeline remotely, you must make sure these dependencies are available on the remote machines. This tutorial shows you how to make your dependencies available to the remote workers. Each section below refers to a different source that your package may have been installed from.
-**Note:** Remote workers used for pipeline execution typically have a standard Python 2.7 distribution installation. If your code relies only on standard Python packages, then you probably don't need to do anything on this page.
+**Note:** Remote workers used for pipeline execution typically have a standard Python distribution installation in a Debian-based container image. If your code relies only on standard Python packages, then you probably don't need to do anything on this page.
## PyPI Dependencies {#pypi-dependencies}
diff --git a/website/src/documentation/sdks/python-streaming.md b/website/src/documentation/sdks/python-streaming.md
index 6340fc4..ea08da9 100644
--- a/website/src/documentation/sdks/python-streaming.md
+++ b/website/src/documentation/sdks/python-streaming.md
@@ -20,7 +20,7 @@
# Python Streaming Pipelines
-Python streaming pipeline execution is experimentally available (with some
+Python streaming pipeline execution became available (with some
[limitations](#unsupported-features)) starting with Beam SDK version 2.5.0.
@@ -183,18 +183,13 @@
- Custom source API
- Splittable `DoFn` API
- Handling of late data
-- User-defined custom `WindowFn`
+- User-defined custom merging `WindowFn` (with fnapi)
### DataflowRunner specific features
Additionally, `DataflowRunner` does not currently support the following Cloud
Dataflow specific features with Python streaming execution.
-- Streaming autoscaling
-- Updating existing pipelines
- Cloud Dataflow Templates
-- Some monitoring features, such as msec counters, display data, metrics, and
- element counts for transforms. However, logging, watermarks, and element
- counts for sources are supported.
diff --git a/website/src/documentation/transforms/java/aggregation/approximatequantiles.md b/website/src/documentation/transforms/java/aggregation/approximatequantiles.md
new file mode 100644
index 0000000..561cca0
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/approximatequantiles.md
@@ -0,0 +1,43 @@
+---
+layout: section
+title: "ApproximateQuantiles"
+permalink: /documentation/transforms/java/aggregation/approximatequantiles/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# ApproximateQuantiles
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/ApproximateQuantiles.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Takes a comparison function and the desired number of quantiles *n*, either
+globally or per-key. Using an approximation algorithm, it returns the
+minimum value, *n-2* intermediate values, and the maximum value.
+
+## Examples
+**Example**: to compute the quartiles of a `PCollection` of integers, we
+would use `ApproximateQuantiles.globally(5)`. This will produce a list
+containing 5 values: the minimum value, Quartile 1 value, Quartile 2
+value, Quartile 3 value, and the maximum value.
+
+## Related transforms
+* [ApproximateUnique]({{ site.baseurl }}/documentation/transforms/java/aggregation/approximateunique)
+ estimates the number of distinct elements or distinct values in key-value pairs
+* [Combine]({{ site.baseurl }}/documentation/transforms/java/aggregation/combine)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/approximateunique.md b/website/src/documentation/transforms/java/aggregation/approximateunique.md
new file mode 100644
index 0000000..9b3e6d0
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/approximateunique.md
@@ -0,0 +1,40 @@
+---
+layout: section
+title: "ApproximateUnique"
+permalink: /documentation/transforms/java/aggregation/approximateunique/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# ApproximateUnique
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/ApproximateUnique.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Transforms for estimating the number of distinct elements in a collection
+or the number of distinct values associated with each key in a collection
+of key-value pairs.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [Count]({{ site.baseurl }}/documentation/transforms/java/aggregation/count)
+ counts the number of elements within each aggregation.
+* [Distinct]({{ site.baseurl }}/documentation/transforms/java/aggregation/distinct)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/cogroupbykey.md b/website/src/documentation/transforms/java/aggregation/cogroupbykey.md
new file mode 100644
index 0000000..a546058
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/cogroupbykey.md
@@ -0,0 +1,73 @@
+---
+layout: section
+title: "CoGroupByKey"
+permalink: /documentation/transforms/java/aggregation/cogroupbykey/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# CoGroupByKey
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/join/CoGroupByKey.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Aggregates all input elements by their key and allows downstream processing
+to consume all values associated with the key. While `GroupByKey` performs
+this operation over a single input collection and thus a single type of
+input values, `CoGroupByKey` operates over multiple input collections. As
+a result, the result for each key is a tuple of the values associated with
+that key in each input collection.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#cogroupbykey).
+
+## Examples
+**Example**: Say you have two different files with user data; one file has
+names and email addresses and the other file has names and phone numbers.
+
+You can join those two data sets, using the username as a common key and the
+other data as the associated values. After the join, you have one data set
+that contains all of the information (email addresses and phone numbers)
+associated with each name.
+
+```java
+PCollection<KV<UID, Integer>> pt1 = /* ... */;
+PCollection<KV<UID, String>> pt2 = /* ... */;
+
+final TupleTag<Integer> t1 = new TupleTag<>();
+final TupleTag<String> t2 = new TupleTag<>();
+PCollection<KV<UID, CoGBKResult>> result =
+ KeyedPCollectionTuple.of(t1, pt1).and(t2, pt2)
+ .apply(CoGroupByKey.create());
+result.apply(ParDo.of(new DoFn<KV<K, CoGbkResult>, /* some result */>() {
+ @ProcessElement
+ public void processElement(ProcessContext c) {
+ KV<K, CoGbkResult> e = c.element();
+ CoGbkResult result = e.getValue();
+ // Retrieve all integers associated with this key from pt1
+ Iterable<Integer> allIntegers = result.getAll(t1);
+ // Retrieve the string associated with this key from pt2.
+ // Note: This will fail if multiple values had the same key in pt2.
+ String string = e.getOnly(t2);
+ ...
+}));
+```
+
+## Related transforms
+* [GroupByKey]({{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey)
+ takes one input collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/combine.md b/website/src/documentation/transforms/java/aggregation/combine.md
new file mode 100644
index 0000000..3bca34b
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/combine.md
@@ -0,0 +1,82 @@
+---
+layout: section
+title: "Combine"
+permalink: /documentation/transforms/java/aggregation/combine/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Combine
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Combine.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+A user-defined `CombineFn` may be applied to combine all elements in a
+`PCollection` (global combine) or to combine all elements associated
+with each key.
+
+While the result is similar to applying a `GroupByKey` followed by
+aggregating values in each `Iterable`, there is an impact
+on the code you must write as well as the performance of the pipeline.
+Writing a `ParDo` that counts the number of elements in each value
+would be very straightforward. However, as described in the execution
+model, it would also require all values associated with each key to be
+processed by a single worker. This introduces a lot of communication overhead.
+Using a `CombineFn` requires the code be structured as an associative and
+commumative operation. But, it allows the use of partial sums to be precomputed.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#combine).
+
+## Examples
+**Example 1**: Global combine
+Use the global combine to combine all of the elements in a given `PCollection`
+into a single value, represented in your pipeline as a new `PCollection` containing
+one element. The following example code shows how to apply the Beam-provided
+sum combine function to produce a single sum value for a `PCollection` of integers.
+
+```java
+// Sum.SumIntegerFn() combines the elements in the input PCollection. The resulting PCollection, called sum,
+// contains one value: the sum of all the elements in the input PCollection.
+PCollection<Integer> pc = ...;
+PCollection<Integer> sum = pc.apply(
+ Combine.globally(new Sum.SumIntegerFn()));
+```
+
+**Example 2**: Keyed combine
+Use a keyed combine to to combine all of the values associated with each key
+into a single output value for each key. As with the global combine, the
+function passed to a keyed combine must be associative and commutative.
+
+```java
+// PCollection is grouped by key and the Double values associated with each key are combined into a Double.
+PCollection<KV<String, Double>> salesRecords = ...;
+PCollection<KV<String, Double>> totalSalesPerPerson =
+ salesRecords.apply(Combine.<String, Double, Double>perKey(
+ new Sum.SumDoubleFn()));
+// The combined value is of a different type than the original collection of values per key. PCollection has
+// keys of type String and values of type Integer, and the combined value is a Double.
+PCollection<KV<String, Integer>> playerAccuracy = ...;
+PCollection<KV<String, Double>> avgAccuracyPerPlayer =
+ playerAccuracy.apply(Combine.<String, Integer, Double>perKey(
+ new MeanInts())));
+```
+
+## Related transforms
+* [CombineWithContext]({{ site.baseurl }}/documentation/transforms/java/aggregation/combinewithcontext)
+* [GroupByKey]({{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/combinewithcontext.md b/website/src/documentation/transforms/java/aggregation/combinewithcontext.md
new file mode 100644
index 0000000..20f29dd
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/combinewithcontext.md
@@ -0,0 +1,37 @@
+---
+layout: section
+title: "CombineWithContext"
+permalink: /documentation/transforms/java/aggregation/combinewithcontext/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# CombineWithContext
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/curent/index.html?org/apache/beam/sdk/transforms/CombineWithContext.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+A class of transforms that contains combine functions that have access to `PipelineOptions` and side inputs through `CombineWithContext.Context`.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [Combine]({{ site.baseurl }}/documentation/transforms/java/aggregation/combine)
+ for combining all values associated with a key to a single result
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/count.md b/website/src/documentation/transforms/java/aggregation/count.md
new file mode 100644
index 0000000..17b9ff7
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/count.md
@@ -0,0 +1,50 @@
+---
+layout: section
+title: "Count"
+permalink: /documentation/transforms/java/aggregation/count/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Count
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Count.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Counts the number of elements within each aggregation. The `Count`
+transform has three varieties:
+
+* `Count.globally()` counts the number of elements in the entire
+ `PCollection`. The result is a collection with a single element.
+* `Count.perKey()` counts how many elements are associated with each
+ key. It ignores the values. The resulting collection has one
+ output for every key in the input collection.
+* `Count.perElement()` counts how many times each element appears
+ in the input collection. The output collection is a key-value
+ pair, containing each unique element and the number of times it
+ appeared in the original collection.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [ApproximateUnique]({{ site.baseurl }}/documentation/transforms/java/aggregation/approximateunique)
+ estimates the number of distinct elements or distinct values in key-value pairs
+* [Sum]({{ site.baseurl }}/documentation/transforms/java/aggregation/sum) computes
+ the sum of elements in a collection
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/distinct.md b/website/src/documentation/transforms/java/aggregation/distinct.md
new file mode 100644
index 0000000..08a3bcd
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/distinct.md
@@ -0,0 +1,43 @@
+---
+layout: section
+title: "Distinct"
+permalink: /documentation/transforms/java/aggregation/distinct/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Distinct
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Distinct.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Produces a collection containing distinct elements of the input collection.
+
+On some data sets, it might be more efficient to compute an approximate
+answer using `ApproximateUnique`, which also allows for determining distinct
+values for each key.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [Count]({{ site.baseurl }}/documentation/transforms/java/aggregation/count)
+ counts the number of elements within each aggregation.
+* [ApproximateUnique]({{ site.baseurl }}/documentation/transforms/java/aggregation/approximateunique)
+ estimates the number of distinct elements in a collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/groupbykey.md b/website/src/documentation/transforms/java/aggregation/groupbykey.md
new file mode 100644
index 0000000..f73cb4f
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/groupbykey.md
@@ -0,0 +1,50 @@
+---
+layout: section
+title: "GroupByKey"
+permalink: /documentation/transforms/java/aggregation/groupbykey/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# GroupByKey
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/GroupByKey.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Takes a keyed collection of elements and produces a collection where
+each element consists of a key and an `Iterable` of all values
+associated with that key.
+
+The results can be combined with windowing to subdivide each key
+based on time or triggering to produce partial aggregations. Either
+windowing or triggering is necessary when processing unbounded collections.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#groupbykey).
+
+## Examples
+**Example 1**: (a, 1), (b, 2), (a, 3) will result into (a, [1, 3]), (b, [2]).
+
+**Example 2**: Given a collection of customer orders keyed by postal code,
+you could use `GroupByKey` to get the collection of all orders in each postal code.
+
+## Related transforms
+* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/java/aggregation/cogroupbykey)
+ for multiple input collections
+* [Combine]({{ site.baseurl }}/documentation/transforms/java/aggregation/combine)
+ for combining all values associated with a key to a single result
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/groupintobatches.md b/website/src/documentation/transforms/java/aggregation/groupintobatches.md
new file mode 100644
index 0000000..8874648
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/groupintobatches.md
@@ -0,0 +1,42 @@
+---
+layout: section
+title: "GroupIntoBatches"
+permalink: /documentation/transforms/java/aggregation/groupintobatches/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# GroupIntoBatches
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/GroupIntoBatches.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Batches inputs to a desired batch size.
+
+Batches contain only elements of a single key. Elements are buffered until
+`batchSize` number of elements buffered. Then, these elements are output
+to the output collection.
+
+Batches contain elements from the same window, so windows are preserved. Batches might contain elements from more than one bundle.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [GroupByKey]({{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey) takes one input collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/latest.md b/website/src/documentation/transforms/java/aggregation/latest.md
new file mode 100644
index 0000000..a2ed7a5
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/latest.md
@@ -0,0 +1,52 @@
+---
+layout: section
+title: "Latest"
+permalink: /documentation/transforms/java/aggregation/latest/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Latest
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Latest.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+A transform and `Combine.CombineFn` for computing the latest element in a collection.
+
+* `Latest.globally()` takes a collection of values and produces the collection
+ containing the single value with the latest implicit timestamp.
+* `Latest.perKey()` takes a collection of key value pairs, and returns the
+ latest value for each key, according to the implicit timestamp.
+
+For elements with the same timestamp, the output element is arbitrarily selected.
+
+## Examples
+**Example**: compute the latest value for each session
+```java
+ PCollection input = ...;
+ PCollection sessioned = input
+ .apply(Window.into(Sessions.withGapDuration(Duration.standardMinutes(5)));
+ PCollection latestValues = sessioned.apply(Latest.globally());
+```
+
+## Related transforms
+* [Reify]({{ site.baseurl }}/documentation/transforms/java/elementwise/reify)
+ converts between explicit and implicit form of various Beam values
+* [WithTimestamps]({{ site.baseurl }}/documentation/transforms/java/elementwise/withtimestamps)
+ assigns timestamps to all the elements of a collection
diff --git a/website/src/documentation/transforms/java/aggregation/max.md b/website/src/documentation/transforms/java/aggregation/max.md
new file mode 100644
index 0000000..941647f
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/max.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Max"
+permalink: /documentation/transforms/java/aggregation/max/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Max
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Max.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Provides a variety of different transforms for computing the maximum
+values in a collection, either globally or for each key.
+
+## Examples
+**Example 1**: get the maximum of a `PCollection` of `Doubles`.
+
+```java
+PCollection<Double> input = ...;
+PCollection<Double> max = input.apply(Max.doublesGlobally());
+```
+
+**Example 2**: calculate the maximum of the `Integers` associated
+with each unique key (which is of type `String`).
+
+```java
+PCollection<KV<String, Integer>> input = ...;
+PCollection<KV<String, Integer>> maxPerKey = input
+ .apply(Max.integersPerKey());
+```
+
+## Related transforms
+* [Min]({{ site.baseurl }}/documentation/transforms/java/aggregation/min)
+ for computing minimum values in a collection
+* [Mean]({{ site.baseurl }}/documentation/transforms/java/aggregation/mean)
+ for computing the arithmetic mean of the elements in a collection
+* [Combine]({{ site.baseurl }}/documentation/transforms/java/aggregation/combine)
+ for combining all values associated with a key to a single result
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/mean.md b/website/src/documentation/transforms/java/aggregation/mean.md
new file mode 100644
index 0000000..36b487e
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/mean.md
@@ -0,0 +1,58 @@
+---
+layout: section
+title: "Mean"
+permalink: /documentation/transforms/java/aggregation/mean/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Mean
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Mean.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Transforms for computing the arithmetic mean of the elements in a collection,
+or the mean of the values associated with each key in a collection of key-value pairs.
+
+* `Mean.globally()` returns a transform that then returns a collection whose contents is the mean of the input collection's elements. If there are no elements in the input collection, it returns 0.
+* `Mean.perKey()` returns a transform that returns a collection that contains an output element mapping each distinct key in the input collection to the mean of the values associated with that key in the input collection.
+
+## Examples
+**Example 1**: get the mean of a `PCollection` of `Longs`.
+
+```java
+PCollection<Double> input = ...;
+PCollection<Double> mean = input.apply(Mean.globally());
+```
+
+**Example 2**: calculate the mean of the `Integers` associated with each unique key (which is of type `String`).
+
+```java
+PCollection<KV<String, Integer>> input = ...;
+PCollection<KV<String, Integer>> meanPerKey =
+ input.apply(Mean.perKey());
+```
+
+## Related transforms
+* [Max]({{ site.baseurl }}/documentation/transforms/java/aggregation/max)
+ for computing maximum values in a collection
+* [Min]({{ site.baseurl }}/documentation/transforms/java/aggregation/min)
+ for computing maximum values in a collection
+* [Combine]({{ site.baseurl }}/documentation/transforms/java/aggregation/combine)
+ for combining all values associated with a key to a single result
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/min.md b/website/src/documentation/transforms/java/aggregation/min.md
new file mode 100644
index 0000000..538b02f
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/min.md
@@ -0,0 +1,42 @@
+---
+layout: section
+title: "Min"
+permalink: /documentation/transforms/java/aggregation/min/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Min
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Min.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Provides a variety of different transforms for computing the minimum
+values in a collection, either globally or for each key.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [Max]({{ site.baseurl }}/documentation/transforms/java/aggregation/max)
+ for computing maximum values in a collection
+* [Mean]({{ site.baseurl }}/documentation/transforms/java/aggregation/mean)
+ for computing the arithmetic mean of the elements in a collection
+* [Combine]({{ site.baseurl }}/documentation/transforms/java/aggregation/combine)
+ for combining all values associated with a key to a single result
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/sample.md b/website/src/documentation/transforms/java/aggregation/sample.md
new file mode 100644
index 0000000..6f7e181
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/sample.md
@@ -0,0 +1,40 @@
+---
+layout: section
+title: "Sample"
+permalink: /documentation/transforms/java/aggregation/sample/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Sample
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Sample.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Transforms for taking samples of the elements in a collection, or
+samples of the values associated with each key in a collection of key-value pairs.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [Top]({{ site.baseurl }}/documentation/transforms/java/aggregation/top)
+ finds the largest (or smallest) set of elements in a collection
+* [Latest]({{ site.baseurl }}/documentation/transforms/java/aggregation/latest)
+ computes the latest element in a collection
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/sum.md b/website/src/documentation/transforms/java/aggregation/sum.md
new file mode 100644
index 0000000..ee1f282
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/sum.md
@@ -0,0 +1,51 @@
+---
+layout: section
+title: "Sum"
+permalink: /documentation/transforms/java/aggregation/sum/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Sum
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Sum.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Transforms for computing the sum of the elements in a collection, or the sum of the
+values associated with each key in a collection of key-value pairs.
+
+## Examples
+**Example 1**: get the sum of a `PCollection` of `Doubles`.
+
+```java
+PCollection<Double> input = ...;
+PCollection<Double> sum = input.apply(Sum.doublesGlobally());
+```
+
+Example 2: calculate the sum of the `Integers` associated with each unique key (which is of type `String`).
+
+```java
+PCollection<KV<String, Integer>> input = ...;
+PCollection<KV<String, Integer>> sumPerKey = input
+ .apply(Sum.integersPerKey());
+```
+
+## Related transforms
+* [Count]({{ site.baseurl }}/documentation/transforms/java/aggregation/count)
+ counts the number of elements within each aggregation
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/aggregation/top.md b/website/src/documentation/transforms/java/aggregation/top.md
new file mode 100644
index 0000000..7f060f1
--- /dev/null
+++ b/website/src/documentation/transforms/java/aggregation/top.md
@@ -0,0 +1,39 @@
+---
+layout: section
+title: "Top"
+permalink: /documentation/transforms/java/aggregation/top/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Top
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Top.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Transforms for finding the largest (or smallest) set of elements in
+a collection, or the largest (or smallest) set of values associated
+with each key in a collection of key-value pairs.
+
+## Examples
+See [BEAM-7703](https://issues.apache.org/jira/browse/BEAM-7703) for updates.
+
+## Related transforms
+* [Sample]({{ site.baseurl }}/documentation/transforms/java/aggregation/sample)
+ takes samples of collection
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/filter.md b/website/src/documentation/transforms/java/element-wise/filter.md
new file mode 100644
index 0000000..975fa09
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/filter.md
@@ -0,0 +1,62 @@
+---
+layout: section
+title: "Filter"
+permalink: /documentation/transforms/java/elementwise/filter/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Filter
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Filter.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Given a predicate, filter out all elements that don't satisfy that predicate.
+May also be used to filter based on an inequality with a given value based
+on the natural ordering of the element.
+
+## Examples
+**Example 1**: Filtering with a predicate
+
+```java
+PCollection<String> allStrings = Create.of("Hello", "world", "hi");
+PCollection<String> longStrings = allStrings
+ .apply(Filter.by(new SerializableFunction<String, Boolean>() {
+ @Override
+ public Boolean apply(String input) {
+ return input.length() > 3;
+ }
+ }));
+```
+The result is a `PCollection` containing "Hello" and "world".
+
+**Example 2**: Filtering with an inequality
+
+```java
+PCollection<Long> numbers = Create.of(1L, 2L, 3L, 4L, 5L);
+PCollection<Long> bigNumbers = numbers.apply(Filter.greaterThan(3));
+PCollection<Long> smallNumbers = numbers.apply(Filter.lessThanEq(3));
+```
+Other variants include `Filter.greaterThanEq`, `Filter.lessThan` and `Filter.equal`.
+
+## Related transforms
+* [FlatMapElements]({{ site.baseurl }}/documentation/transforms/java/elementwise/flatmapelements) behaves the same as `Map`, but for
+ each input it might produce zero or more outputs.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/java/elementwise/pardo) is the most general element-wise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/flatmapelements.md b/website/src/documentation/transforms/java/element-wise/flatmapelements.md
new file mode 100644
index 0000000..62fe93061
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/flatmapelements.md
@@ -0,0 +1,40 @@
+---
+layout: section
+title: "FlatMapElements"
+permalink: /documentation/transforms/java/elementwise/flatmapelements/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# FlatMapElements
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/FlatMapElements.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Applies a simple 1-to-many mapping function over each element in the
+collection. The many elements are flattened into the resulting collection.
+
+## Examples
+See [BEAM-7702](https://issues.apache.org/jira/browse/BEAM-7702) for updates.
+
+## Related transforms
+* [Filter]({{ site.baseurl }}/documentation/transforms/java/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/java/elementwise/pardo) is the most general element-wise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/keys.md b/website/src/documentation/transforms/java/element-wise/keys.md
new file mode 100644
index 0000000..b0d0738
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/keys.md
@@ -0,0 +1,43 @@
+---
+layout: section
+title: "Keys"
+permalink: /documentation/transforms/java/elementwise/keys/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Keys
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Keys.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Takes a collection of key-value pairs, and returns the key of each element.
+
+## Examples
+**Example**
+
+```java
+PCollection<KV<String, Integer>> keyValuePairs = /* ... */;
+PCollection<String> keys = keyValuePairs.apply(Keys.create());
+```
+
+## Related transforms
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/java/elementwise/kvswap) swaps key-value pair values.
+* [Values]({{ site.baseurl }}/documentation/transforms/java/elementwise/values) for extracting the value of each element.
+* [WithKeys]({{ site.baseurl }}/documentation/transforms/java/elementwise/withkeys) for adding a key to each element.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/kvswap.md b/website/src/documentation/transforms/java/element-wise/kvswap.md
new file mode 100644
index 0000000..96a423c
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/kvswap.md
@@ -0,0 +1,43 @@
+---
+layout: section
+title: "KvSwap"
+permalink: /documentation/transforms/java/elementwise/kvswap/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# KvSwap
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/KvSwap.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Takes a collection of key-value pairs and returns a collection of key-value pairs which has each key and value swapped.
+
+## Examples
+**Example**:
+
+```java
+PCollection<KV<String, Integer>> strIntPairs = /* ... */;
+PCollection<KV<Integer, String>> intStrPairs = strIntPairs.apply(KvSwap.create());
+```
+
+## Related transforms
+* [Keys]({{ site.baseurl }}/documentation/transforms/java/elementwise/keys) for extracting the key of each component.
+* [Values]({{ site.baseurl }}/documentation/transforms/java/elementwise/values) for extracting the value of each element.
+* [WithKeys]({{ site.baseurl }}/documentation/transforms/java/elementwise/withkeys) for adding a key to each element.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/mapelements.md b/website/src/documentation/transforms/java/element-wise/mapelements.md
new file mode 100644
index 0000000..e916e2e
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/mapelements.md
@@ -0,0 +1,63 @@
+---
+layout: section
+title: "MapElements"
+permalink: /documentation/transforms/java/elementwise/mapelements/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# MapElements
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/MapElements.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Applies a simple 1-to-1 mapping function over each element in the collection.
+
+## Examples
+**Example 1**: providing the mapping function using a `SimpleFunction`
+
+```java
+PCollection<String> lines = Create.of("Hello World", "Beam is fun");
+PCollection<Integer> lineLengths = lines.apply(MapElements.via(
+ new SimpleFunction<String, Integer>() {
+ @Override
+ public Integer apply(String line) {
+ return line.length();
+ }
+ });
+```
+
+**Example 2**: providing the mapping function using a `SerializableFunction`,
+which allows the use of Java 8 lambdas. Due to type erasure, you need
+to provide a hint indicating the desired return type.
+
+```java
+PCollection<String> lines = Create.of("Hello World", "Beam is fun");
+PCollection<Integer> lineLengths = lines.apply(MapElements
+ .into(TypeDescriptors.integers())
+ .via((String line) -> line.length()));
+```
+
+## Related transforms
+* [FlatMapElements]({{ site.baseurl }}/documentation/transforms/java/elementwise/flatmapelements) behaves the same as `Map`, but for
+ each input it may produce zero or more outputs.
+* [Filter]({{ site.baseurl }}/documentation/transforms/java/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/java/elementwise/pardo) is the most general element-wise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/pardo.md b/website/src/documentation/transforms/java/element-wise/pardo.md
new file mode 100644
index 0000000..3a7b979
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/pardo.md
@@ -0,0 +1,152 @@
+---
+layout: section
+title: "ParDo"
+permalink: /documentation/transforms/java/elementwise/pardo/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# ParDo
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/ParDo.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+A transform for generic parallel processing. A `ParDo` transform considers each
+element in the input `PCollection`, performs some processing function
+(your user code) on that element, and emits zero or more elements to
+an output PCollection.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#pardo).
+
+## Examples
+**Example 1**: Passing side inputs
+
+```java
+ // Pass side inputs to your ParDo transform by invoking .withSideInputs.
+ // Inside your DoFn, access the side input by using the method DoFn.ProcessContext.sideInput.
+
+ // The input PCollection to ParDo.
+ PCollection<String> words = ...;
+
+ // A PCollection of word lengths that we'll combine into a single value.
+ PCollection<Integer> wordLengths = ...; // Singleton PCollection
+
+ // Create a singleton PCollectionView from wordLengths using Combine.globally and View.asSingleton.
+ final PCollectionView<Integer> maxWordLengthCutOffView =
+ wordLengths.apply(Combine.globally(new Max.MaxIntFn()).asSingletonView());
+
+
+ // Apply a ParDo that takes maxWordLengthCutOffView as a side input.
+ PCollection<String> wordsBelowCutOff =
+ words.apply(ParDo
+ .of(new DoFn<String, String>() {
+ public void processElement(ProcessContext c) {
+ String word = c.element();
+ // In our DoFn, access the side input.
+ int lengthCutOff = c.sideInput(maxWordLengthCutOffView);
+ if (word.length() <= lengthCutOff) {
+ c.output(word);
+ }
+ }
+ }).withSideInputs(maxWordLengthCutOffView)
+ );
+```
+
+**Example 2**: Emitting to multiple outputs in your `DoFn`
+
+```java
+// To emit elements to multiple output PCollections, create a TupleTag object to identify each collection
+// that your ParDo produces. For example, if your ParDo produces three output PCollections (the main output
+// and two additional outputs), you must create three TupleTags. The following example code shows how to
+// create TupleTags for a ParDo with three output PCollections.
+
+ // Input PCollection to our ParDo.
+ PCollection<String> words = ...;
+
+ // The ParDo will filter words whose length is below a cutoff and add them to
+ // the main ouput PCollection<String>.
+ // If a word is above the cutoff, the ParDo will add the word length to an
+ // output PCollection<Integer>.
+ // If a word starts with the string "MARKER", the ParDo will add that word to an
+ // output PCollection<String>.
+ final int wordLengthCutOff = 10;
+
+ // Create three TupleTags, one for each output PCollection.
+ // Output that contains words below the length cutoff.
+ final TupleTag<String> wordsBelowCutOffTag =
+ new TupleTag<String>(){};
+ // Output that contains word lengths.
+ final TupleTag<Integer> wordLengthsAboveCutOffTag =
+ new TupleTag<Integer>(){};
+ // Output that contains "MARKER" words.
+ final TupleTag<String> markedWordsTag =
+ new TupleTag<String>(){};
+
+// Passing Output Tags to ParDo:
+// After you specify the TupleTags for each of your ParDo outputs, pass the tags to your ParDo by invoking
+// .withOutputTags. You pass the tag for the main output first, and then the tags for any additional outputs
+// in a TupleTagList. Building on our previous example, we pass the three TupleTags for our three output
+// PCollections to our ParDo. Note that all of the outputs (including the main output PCollection) are
+// bundled into the returned PCollectionTuple.
+
+ PCollectionTuple results =
+ words.apply(ParDo
+ .of(new DoFn<String, String>() {
+ // DoFn continues here.
+ ...
+ })
+ // Specify the tag for the main output.
+ .withOutputTags(wordsBelowCutOffTag,
+ // Specify the tags for the two additional outputs as a TupleTagList.
+ TupleTagList.of(wordLengthsAboveCutOffTag)
+ .and(markedWordsTag)));
+```
+
+**Example 3**: Tags for multiple outputs
+
+```java
+// Inside your ParDo's DoFn, you can emit an element to a specific output PCollection by passing in the
+// appropriate TupleTag when you call ProcessContext.output.
+// After your ParDo, extract the resulting output PCollections from the returned PCollectionTuple.
+// Based on the previous example, this shows the DoFn emitting to the main output and two additional outputs.
+
+ .of(new DoFn<String, String>() {
+ public void processElement(ProcessContext c) {
+ String word = c.element();
+ if (word.length() <= wordLengthCutOff) {
+ // Emit short word to the main output.
+ // In this example, it is the output with tag wordsBelowCutOffTag.
+ c.output(word);
+ } else {
+ // Emit long word length to the output with tag wordLengthsAboveCutOffTag.
+ c.output(wordLengthsAboveCutOffTag, word.length());
+ }
+ if (word.startsWith("MARKER")) {
+ // Emit word to the output with tag markedWordsTag.
+ c.output(markedWordsTag, word);
+ }
+ }}));
+```
+
+
+## Related transforms
+* [MapElements]({{ site.baseurl }}/documentation/transforms/java/elementwise/mapelements)
+ applies a simple 1-to-1 mapping function over each element in the collection.
+* [Filter]({{ site.baseurl }}/documentation/transforms/java/elementwise/filter)
+ is useful if the function is just deciding whether to output an element or not.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/partition.md b/website/src/documentation/transforms/java/element-wise/partition.md
new file mode 100644
index 0000000..583f631
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/partition.md
@@ -0,0 +1,62 @@
+---
+layout: section
+title: "Partition"
+permalink: /documentation/transforms/java/elementwise/partition/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Partition
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Partition.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Separates elements in a collection into multiple output collections. The partitioning function contains the logic that determines how to separate the elements of the input collection into each resulting partition output collection.
+
+The number of partitions must be determined at graph construction time. You cannot determine the number of partitions in mid-pipeline.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#partition).
+
+## Examples
+**Example**: dividing a `PCollection` into percentile groups
+
+```java
+// Provide an int value with the desired number of result partitions, and a PartitionFn that represents the
+// partitioning function. In this example, we define the PartitionFn in-line. Returns a PCollectionList
+// containing each of the resulting partitions as individual PCollection objects.
+PCollection<Student> students = ...;
+// Split students up into 10 partitions, by percentile:
+PCollectionList<Student> studentsByPercentile =
+ students.apply(Partition.of(10, new PartitionFn<Student>() {
+ public int partitionFor(Student student, int numPartitions) {
+ return student.getPercentile() // 0..99
+ * numPartitions / 100;
+ }}));
+
+// You can extract each partition from the PCollectionList using the get method, as follows:
+PCollection<Student> fortiethPercentile = studentsByPercentile.get(4);
+```
+
+## Related transforms
+* [Filter]({{ site.baseurl }}/documentation/transforms/java/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/java/elementwise/pardo) is the most general element-wise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/java/aggregation/cogroupbykey)
+ performs a per-key equijoin.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/regex.md b/website/src/documentation/transforms/java/element-wise/regex.md
new file mode 100644
index 0000000..5fff595
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/regex.md
@@ -0,0 +1,36 @@
+---
+layout: section
+title: "Regex"
+permalink: /documentation/transforms/java/elementwise/regex/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Regex
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Regex.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Provides a variety of functionality based on regular expressions.
+
+## Examples
+See [BEAM-7702](https://issues.apache.org/jira/browse/BEAM-7702) for updates.
+
+## Related transforms
+* [MapElements]({{ site.baseurl }}/documentation/transforms/java/elementwise/mapelements)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/reify.md b/website/src/documentation/transforms/java/element-wise/reify.md
new file mode 100644
index 0000000..84c3241
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/reify.md
@@ -0,0 +1,39 @@
+---
+layout: section
+title: "Reify"
+permalink: /documentation/transforms/java/elementwise/reify/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Reify
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Reify.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Transforms for converting between explicit and implicit form of various Beam values.
+
+## Examples
+See [BEAM-7702](https://issues.apache.org/jira/browse/BEAM-7702) for updates.
+
+## Related transforms
+* [WithTimestamps]({{ site.baseurl }}/documentation/transforms/java/elementwise/withtimestamps)
+ assigns timestamps to all the elements of a collection
+* [Window]({{ site.baseurl }}/documentation/transforms/java/other/window/) divides up or
+ groups the elements of a collection into finite windows
diff --git a/website/src/documentation/transforms/java/element-wise/tostring.md b/website/src/documentation/transforms/java/element-wise/tostring.md
new file mode 100644
index 0000000..dde6ac0
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/tostring.md
@@ -0,0 +1,37 @@
+---
+layout: section
+title: "ToString"
+permalink: /documentation/transforms/java/elementwise/tostring/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# ToString
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/ToString.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+A variety of utility transforms for invoking the `toString()` method
+on every element in the input collection.
+
+## Examples
+See [BEAM-7702](https://issues.apache.org/jira/browse/BEAM-7702) for updates.
+
+## Related transforms
+* [MapElements]({{ site.baseurl }}/documentation/transforms/java/elementwise/mapelements)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/values.md b/website/src/documentation/transforms/java/element-wise/values.md
new file mode 100644
index 0000000..b810db4
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/values.md
@@ -0,0 +1,44 @@
+---
+layout: section
+title: "Values"
+permalink: /documentation/transforms/java/elementwise/values/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Values
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Values.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+The `Values` transform takes a collection of key-value pairs, and
+returns the value of each element.
+
+## Examples
+**Example**
+
+```java
+PCollection<KV<String, Integer>> keyValuePairs = /* ... */;
+PCollection<Integer> values = keyValuePairs.apply(Values.create());
+```
+
+## Related transforms
+* [Keys]({{ site.baseurl }}/documentation/transforms/java/elementwise/keys) for extracting the key of each component.
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/java/elementwise/kvswap) swaps key-value pair values.
+* [WithKeys]({{ site.baseurl }}/documentation/transforms/java/elementwise/withkeys) for adding a key to each element.
diff --git a/website/src/documentation/transforms/java/element-wise/withkeys.md b/website/src/documentation/transforms/java/element-wise/withkeys.md
new file mode 100644
index 0000000..6c6df43
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/withkeys.md
@@ -0,0 +1,55 @@
+---
+layout: section
+title: "WithKeys"
+permalink: /documentation/transforms/java/elementwise/withkeys/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# WithKeys
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/WithKeys.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Takes a `PCollection<V>` and produces a `PCollection<KV<K, V>>` by associating
+each input element with a key.
+
+There are two versions of `WithKeys`, depending on how the key should be determined:
+
+* `WithKeys.of(SerializableFunction<V, K> fn)` takes a function to
+ compute the key from each value.
+* `WithKeys.of(K key)` associates each value with the specified key.
+
+## Examples
+**Example**
+```java
+PCollection<String> words = Create.of("Hello", "World", "Beam", "is", "fun");
+PCollection<KV<Integer, String>> lengthAndWord =
+ words.apply(WithKeys.of(new SerialiazableFunction<String, Integer>() {
+ @Override
+ public Integer apply(String s) {
+ return s.length();
+ }
+ });
+```
+
+## Related transforms
+* [Keys]({{ site.baseurl }}/documentation/transforms/java/elementwise/keys) for extracting the key of each component.
+* [Values]({{ site.baseurl }}/documentation/transforms/java/elementwise/values) for extracting the value of each element.
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/java/elementwise/kvswap) swaps key-value pair values.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/element-wise/withtimestamps.md b/website/src/documentation/transforms/java/element-wise/withtimestamps.md
new file mode 100644
index 0000000..a2f7e41
--- /dev/null
+++ b/website/src/documentation/transforms/java/element-wise/withtimestamps.md
@@ -0,0 +1,36 @@
+---
+layout: section
+title: "WithTimestamps"
+permalink: /documentation/transforms/java/elementwise/withtimestamps/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# WithTimestamps
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/WithTimestamps.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Assigns timestamps to all the elements of a collection.
+
+## Examples
+See [BEAM-7702](https://issues.apache.org/jira/browse/BEAM-7702) for updates.
+
+## Related transforms
+* [Reify]({{ site.baseurl }}/documentation/transforms/java/elementwise/reify)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/index.md b/website/src/documentation/transforms/java/index.md
new file mode 100644
index 0000000..b36e305
--- /dev/null
+++ b/website/src/documentation/transforms/java/index.md
@@ -0,0 +1,81 @@
+---
+layout: section
+title: "Java transform catalog overview"
+permalink: /documentation/transforms/java/overview/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Java transform catalog overview
+
+## Element-wise
+
+<table class="table-bordered table-striped">
+ <tr><th>Transform</th><th>Description</th></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/filter">Filter</a></td><td>Given a predicate, filter out all elements that don't satisfy the predicate.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/flatmapelements">FlatMapElements</a></td><td>Applies a function that returns a collection to every element in the input and
+ outputs all resulting elements.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/keys">Keys</a></td><td>Extracts the key from each element in a collection of key-value pairs.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/kvswap">KvSwap</a></td><td>Swaps the key and value of each element in a collection of key-value pairs.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/mapelements">MapElements</a></td><td>Applies a function to every element in the input and outputs the result.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/pardo">ParDo</a></td><td>The most-general mechanism for applying a user-defined <code>DoFn</code> to every element
+ in the input collection.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/partition">Partition</a></td><td>Routes each input element to a specific output collection based on some partition
+ function.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/regex">Regex</a></td><td>Filters input string elements based on a regex. May also transform them based on the matching groups.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/reify">Reify</a></td><td>Transforms for converting between explicit and implicit form of various Beam values.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/tostring">ToString</a></td><td>Transforms every element in an input collection to a string.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/withkeys">WithKeys</a></td><td>Produces a collection containing each element from the input collection converted to a key-value pair, with a key selected by applying a function to the input element.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/withtimestamps">WithTimestamps</a></td><td>Applies a function to determine a timestamp to each element in the output collection,
+ and updates the implicit timestamp associated with each input. Note that it is only safe to adjust timestamps forwards.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/elementwise/values">Values</a></td><td>Extracts the value from each element in a collection of key-value pairs.</td></tr>
+</table>
+
+
+
+## Aggregation
+<table class="table-bordered table-striped">
+ <tr><th>Transform</th><th>Description</th></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/approximatequantiles">ApproximateQuantiles</a></td><td>Uses an approximation algorithm to estimate the data distribution within each aggregation using a specified number of quantiles.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/approximateunique">ApproximateUnique</a></td><td>Uses an approximation algorithm to estimate the number of unique elements within each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/cogroupbykey/">CoGroupByKey</a></td><td>Similar to <code>GroupByKey</code>, but groups values associated with each key into a batch of a given size</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/combine">Combine</a></td><td>Transforms to combine elements according to a provided <code>CombineFn</code>.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/combinewithcontext">CombineWithContext</a></td><td>An extended version of Combine which allows accessing side-inputs and other context.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/count">Count</a></td><td>Counts the number of elements within each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/distinct">Distinct</a></td><td>Produces a collection containing distinct elements from the input collection.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupbykey">GroupByKey</a></td><td>Takes a keyed collection of elements and produces a collection where each element
+ consists of a key and all values associated with that key.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/groupintobatches">GroupIntoBatches</a></td><td>Batches values associated with keys into <code>Iterable</code> batches of some size. Each batch contains elements associated with a specific key.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/latest">Latest</a></td><td>Selects the latest element within each aggregation according to the implicit timestamp.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/max">Max</a></td><td>Outputs the maximum element within each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/mean">Mean</a></td><td>Computes the average within each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/min">Min</a></td><td>Outputs the minimum element within each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/sample">Sample</a></td><td>Randomly select some number of elements from each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/sum">Sum</a></td><td>Compute the sum of elements in each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/aggregation/top">Top</a></td><td>Compute the largest element(s) in each aggregation.</td></tr>
+</table>
+
+
+## Other
+<table class="table-bordered table-striped">
+ <tr><th>Transform</th><th>Description</th></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/other/create">Create</a></td><td>Creates a collection from an in-memory list.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/other/flatten">Flatten</a></td><td>Given multiple input collections, produces a single output collection containing
+ all elements from all of the input collections.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/other/passert">PAssert</a></td><td>A transform to assert the contents of a <code>PCollection</code> used as part of testing a pipeline either locally or with a runner.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/other/view">View</a></td><td>Operations for turning a collection into view that may be used as a side-input to a <code>ParDo</code>.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/java/other/window">Window</a></td><td>Logically divides up or groups the elements of a collection into finite
+ windows according to a provided <code>WindowFn</code>.</td></tr>
+</table>
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/other/create.md b/website/src/documentation/transforms/java/other/create.md
new file mode 100644
index 0000000..d81607b
--- /dev/null
+++ b/website/src/documentation/transforms/java/other/create.md
@@ -0,0 +1,40 @@
+---
+layout: section
+title: "Create"
+permalink: /documentation/transforms/java/other/create/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Create
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Create.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Creates a collection containing a specified set of elements. This is useful
+for testing, as well as creating an initial input to process in parallel.
+For example, a single element to execute a one-time `ParDo` or a list of filenames to be read.
+
+
+## Examples
+
+See [BEAM-7704](https://issues.apache.org/jira/browse/BEAM-7704) for updates.
+
+## Related transforms
+N/A
diff --git a/website/src/documentation/transforms/java/other/flatten.md b/website/src/documentation/transforms/java/other/flatten.md
new file mode 100644
index 0000000..c8c22aa
--- /dev/null
+++ b/website/src/documentation/transforms/java/other/flatten.md
@@ -0,0 +1,67 @@
+---
+layout: section
+title: "Flatten"
+permalink: /documentation/transforms/java/other/flatten/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Flatten
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Flatten.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Merges multiple `PCollection` objects into a single logical `PCollection`.
+
+By default, the coder for the output `PCollection` is the same as the coder
+for the first `PCollection` in the input `PCollectionList`. However, the
+input `PCollection` objects can each use different coders, as long as
+they all contain the same data type in your chosen language.
+
+When using `Flatten` to merge `PCollection` objects that have a windowing
+strategy applied, all of the `PCollection` objects you want to merge must
+use a compatible windowing strategy and window sizing. For example, all
+the collections you're merging must all use (hypothetically) identical
+5-minute fixed windows or 4-minute sliding windows starting every 30 seconds.
+
+If your pipeline attempts to use `Flatten` to merge `PCollection` objects
+with incompatible windows, Beam generates an `IllegalStateException` error
+when your pipeline is constructed
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#flatten).
+
+## Examples
+**Example**: Apply a `Flatten` transform to merge multiple `PCollection` objects
+
+```java
+// Flatten takes a PCollectionList of PCollection objects of a given type.
+// Returns a single PCollection that contains all of the elements in the PCollection objects in that list.
+PCollection<String> pc1 = Create.of("Hello");
+PCollection<String> pc2 = Create.of("World", "Beam");
+PCollection<String> pc3 = Create.of("Is", "Fun");
+PCollectionList<String> collections = PCollectionList.of(pc1).and(pc2).and(pc3);
+
+PCollection<String> merged = collections.apply(Flatten.<String>pCollections());
+```
+The resulting collection now has all the elements: "Hello", "World",
+"Beam", "Is", and "Fun".
+
+## Related transforms
+* [ParDo]({{ site.baseurl }}/documentation/transforms/java/elementwise/pardo)
+* [Partition]({{ site.baseurl }}/documentation/transforms/java/elementwise/partition)
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/other/passert.md b/website/src/documentation/transforms/java/other/passert.md
new file mode 100644
index 0000000..85d4201
--- /dev/null
+++ b/website/src/documentation/transforms/java/other/passert.md
@@ -0,0 +1,61 @@
+---
+layout: section
+title: "PAssert"
+permalink: /documentation/transforms/java/other/passert/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# PAssert
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/PAssert.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+`PAssert` is a class included in the Beam Java SDK that is an
+assertion on the contents of a `PCollection`. You can use `PAssert` to verify
+that a `PCollection` contains a specific set of expected elements.
+
+## Examples
+For a given `PCollection`, you can use `PAssert` to verify the contents as follows:
+```java
+PCollection<String> output = ...;
+
+// Check whether a PCollection contains some elements in any order.
+PAssert.that(output)
+.containsInAnyOrder(
+ "elem1",
+ "elem3",
+ "elem2");
+```
+
+Any code that uses `PAssert` must link in `JUnit` and `Hamcrest`.
+If you're using Maven, you can link in `Hamcrest` by adding the
+following dependency to your project's pom.xml file:
+
+```java
+<dependency>
+ <groupId>org.hamcrest</groupId>
+ <artifactId>hamcrest-all</artifactId>
+ <version>1.3</version>
+ <scope>test</scope>
+</dependency>
+```
+
+## Related transforms
+* TestStream
\ No newline at end of file
diff --git a/website/src/documentation/transforms/java/other/view.md b/website/src/documentation/transforms/java/other/view.md
new file mode 100644
index 0000000..fe30b99
--- /dev/null
+++ b/website/src/documentation/transforms/java/other/view.md
@@ -0,0 +1,37 @@
+---
+layout: section
+title: "View"
+permalink: /documentation/transforms/java/other/view/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# View
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/2.13.0/index.html?org/apache/beam/sdk/transforms/View.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Operations for turning a collection into view that may be used as a side-input to a `ParDo`.
+
+## Examples
+See [BEAM-7704](https://issues.apache.org/jira/browse/BEAM-7704) for updates.
+
+## Related transforms
+* [ParDo]({{ site.baseurl }}/documentation/transforms/java/elementwise/pardo)
+* [CombineWithContext]({{ site.baseurl }}/documentation/transforms/java/aggregation/combinewithcontext)
diff --git a/website/src/documentation/transforms/java/other/window.md b/website/src/documentation/transforms/java/other/window.md
new file mode 100644
index 0000000..038f18c
--- /dev/null
+++ b/website/src/documentation/transforms/java/other/window.md
@@ -0,0 +1,40 @@
+---
+layout: section
+title: "Window"
+permalink: /documentation/transforms/java/other/window/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Window
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/windowing/Window.html">
+ <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="20px"
+ alt="Javadoc" />
+ Javadoc
+ </a>
+</table>
+<br>
+Logically divides up or groups the elements of a collection into finite
+windows according to a function.
+
+## Examples
+See [BEAM-7704](https://issues.apache.org/jira/browse/BEAM-7704) for updates.
+
+## Related transforms
+* [Reify]({{ site.baseurl }}/documentation/transforms/java/elementwise/reify)
+ converts between explicit and implicit form of various Beam values.
+* [WithTimestamps]({{ site.baseurl }}/documentation/transforms/java/elementwise/withtimestamps)
+ applies a function to determine a timestamp to each element in the output collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/approximatequantiles.md b/website/src/documentation/transforms/python/aggregation/approximatequantiles.md
new file mode 100644
index 0000000..4fb577e
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/approximatequantiles.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "ApproximateQuantiles"
+permalink: /documentation/transforms/python/aggregation/approximatequantiles/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# ApproximateQuantiles
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/approximateunique.md b/website/src/documentation/transforms/python/aggregation/approximateunique.md
new file mode 100644
index 0000000..d1a5f85
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/approximateunique.md
@@ -0,0 +1,25 @@
+---
+layout: section
+title: "ApproximateUnique"
+permalink: /documentation/transforms/python/aggregation/approximateunique/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# ApproximateUnique
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/cogroupbykey.md b/website/src/documentation/transforms/python/aggregation/cogroupbykey.md
new file mode 100644
index 0000000..c1020c9
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/cogroupbykey.md
@@ -0,0 +1,45 @@
+---
+layout: section
+title: "CoGroupByKey"
+permalink: /documentation/transforms/python/aggregation/cogroupbykey/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# CoGroupByKey
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Aggregates all input elements by their key and allows downstream processing
+to consume all values associated with the key. While `GroupByKey` performs
+this operation over a single input collection and thus a single type of input
+values, `CoGroupByKey` operates over multiple input collections. As a result,
+the result for each key is a tuple of the values associated with that key in
+each input collection.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#cogroupbykey).
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [CombineGlobally]({{ site.baseurl }}/documentation/transforms/python/aggregation/combineglobally) to combine elements.
+* [GroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/groupbykey) takes one input collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/combineglobally.md b/website/src/documentation/transforms/python/aggregation/combineglobally.md
new file mode 100644
index 0000000..1451f87
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/combineglobally.md
@@ -0,0 +1,43 @@
+---
+layout: section
+title: "CombineGlobally"
+permalink: /documentation/transforms/python/aggregation/combineglobally/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# CombineGlobally
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+
+Combines all elements in a collection.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#combine).
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+You can use the following combiner transforms:
+* [Mean]({{ site.baseurl }}/documentation/transforms/python/aggregation/mean)
+* [Count]({{ site.baseurl }}/documentation/transforms/python/aggregation/count)
+* [Top]({{ site.baseurl }}/documentation/transforms/python/aggregation/top)
+* [Sample]({{ site.baseurl }}/documentation/transforms/python/aggregation/sample)
diff --git a/website/src/documentation/transforms/python/aggregation/combinewithcontext.md b/website/src/documentation/transforms/python/aggregation/combinewithcontext.md
new file mode 100644
index 0000000..4e23c47
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/combinewithcontext.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "CombineWithContext"
+permalink: /documentation/transforms/python/aggregation/combinewithcontext/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# CombineWithContext
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/count.md b/website/src/documentation/transforms/python/aggregation/count.md
new file mode 100644
index 0000000..3ea9a23
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/count.md
@@ -0,0 +1,36 @@
+---
+layout: section
+title: "Count"
+permalink: /documentation/transforms/python/aggregation/count/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Count
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/2.12.0/apache_beam.transforms.combiners.html?#apache_beam.transforms.combiners.Count">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Counts the number of elements within each aggregation.
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+N/A
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/distinct.md b/website/src/documentation/transforms/python/aggregation/distinct.md
new file mode 100644
index 0000000..ccd0fb6
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/distinct.md
@@ -0,0 +1,37 @@
+---
+layout: section
+title: "Distinct"
+permalink: /documentation/transforms/python/aggregation/distinct/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Distinct
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Distinct">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Produces a collection containing distinct elements of the input collection.
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [Count]({{ site.baseurl }}/documentation/transforms/python/aggregation/count) counts the number of elements within each aggregation.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/groupbykey.md b/website/src/documentation/transforms/python/aggregation/groupbykey.md
new file mode 100644
index 0000000..17fe94d
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/groupbykey.md
@@ -0,0 +1,41 @@
+---
+layout: section
+title: "GroupByKey"
+permalink: /documentation/transforms/python/aggregation/groupbykey/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# GroupByKey
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.GroupByKey">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Takes a keyed collection of elements and produces a collection
+where each element consists of a key and all values associated with that key.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#groupbykey).
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [CombineGlobally]({{ site.baseurl }}/documentation/transforms/python/aggregation/combineglobally) for combining all values associated with a key to a single result.
+* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey) for multiple input collections.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/groupintobatches.md b/website/src/documentation/transforms/python/aggregation/groupintobatches.md
new file mode 100644
index 0000000..93f02a3
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/groupintobatches.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "GroupIntoBatches"
+permalink: /documentation/transforms/python/aggregation/groupintobatches/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# GroupIntoBatches
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/latest.md b/website/src/documentation/transforms/python/aggregation/latest.md
new file mode 100644
index 0000000..38dc93b
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/latest.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "Latest"
+permalink: /documentation/transforms/python/aggregation/latest/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Latest
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [Sample]({{ site.baseurl }}/documentation/transforms/python/aggregation/sample) to combine elements. takes samples of the elements in a collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/max.md b/website/src/documentation/transforms/python/aggregation/max.md
new file mode 100644
index 0000000..0754546
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/max.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "Max"
+permalink: /documentation/transforms/python/aggregation/max/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Max
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/mean.md b/website/src/documentation/transforms/python/aggregation/mean.md
new file mode 100644
index 0000000..9145c7d
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/mean.md
@@ -0,0 +1,39 @@
+---
+layout: section
+title: "Mean"
+permalink: /documentation/transforms/python/aggregation/mean/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Mean
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html?highlight=mean#apache_beam.transforms.combiners.Mean">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Transforms for computing the arithmetic mean of the elements in a collection,
+or the mean of the values associated with each key in a collection of
+key-value pairs.
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [CombineGlobally]({{ site.baseurl }}/documentation/transforms/python/aggregation/combineglobally) to combine elements.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/min.md b/website/src/documentation/transforms/python/aggregation/min.md
new file mode 100644
index 0000000..51579cf
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/min.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "Min"
+permalink: /documentation/transforms/python/aggregation/min/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Min
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/sample.md b/website/src/documentation/transforms/python/aggregation/sample.md
new file mode 100644
index 0000000..313ce88
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/sample.md
@@ -0,0 +1,38 @@
+---
+layout: section
+title: "Sample"
+permalink: /documentation/transforms/python/aggregation/sample/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Sample
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/2.12.0/apache_beam.transforms.combiners.html?#apache_beam.transforms.combiners.Sample">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Transforms for taking samples of the elements in a collection, or
+samples of the values associated with each key in a collection of
+key-value pairs.
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [Top]({{ site.baseurl }}/documentation/transforms/python/aggregation/top) finds the largest or smallest element.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/sum.md b/website/src/documentation/transforms/python/aggregation/sum.md
new file mode 100644
index 0000000..c3a12d8
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/sum.md
@@ -0,0 +1,26 @@
+---
+layout: section
+title: "Sum"
+permalink: /documentation/transforms/python/aggregation/sum/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Sum
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/aggregation/top.md b/website/src/documentation/transforms/python/aggregation/top.md
new file mode 100644
index 0000000..2eb5a3b
--- /dev/null
+++ b/website/src/documentation/transforms/python/aggregation/top.md
@@ -0,0 +1,38 @@
+---
+layout: section
+title: "Top"
+permalink: /documentation/transforms/python/aggregation/top/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Top
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/2.12.0/apache_beam.transforms.combiners.html?#apache_beam.transforms.combiners.Top">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Transforms for finding the largest (or smallest) set of elements in
+a collection, or the largest (or smallest) set of values associated
+with each key in a collection of key-value pairs.
+
+## Examples
+See [BEAM-7390](https://issues.apache.org/jira/browse/BEAM-7390) for updates.
+
+## Related transforms
+* [Sample]({{ site.baseurl }}/documentation/transforms/python/aggregation/sample) to combine elements. takes samples of the elements in a collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/elementwise/filter.md b/website/src/documentation/transforms/python/elementwise/filter.md
new file mode 100644
index 0000000..b879988
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/filter.md
@@ -0,0 +1,176 @@
+---
+layout: section
+title: "Filter"
+permalink: /documentation/transforms/python/elementwise/filter/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Filter
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Filter" %}
+
+Given a predicate, filter out all elements that don't satisfy that predicate.
+May also be used to filter based on an inequality with a given value based
+on the comparison ordering of the element.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Filter` in multiple ways to filter out produce by their duration value.
+
+`Filter` accepts a function that keeps elements that return `True`, and filters out the remaining elements.
+
+### Example 1: Filtering with a function
+
+We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 2: Filtering with a lambda function
+
+We can also use lambda functions to simplify **Example 1**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 3: Filtering with multiple arguments
+
+You can pass functions with multiple arguments to `Filter`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `has_duration` takes `plant` and `duration` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 4: Filtering with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `'perennial'` as a singleton.
+We then use that value to filter out perennials.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+### Example 5: Filtering with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:valid_plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 6: Filtering with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py tag:filter_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Filter`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py tag:perennials %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it might produce zero or more outputs.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Filter" %}
diff --git a/website/src/documentation/transforms/python/elementwise/flatmap.md b/website/src/documentation/transforms/python/elementwise/flatmap.md
new file mode 100644
index 0000000..fc5f2fc
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/flatmap.md
@@ -0,0 +1,240 @@
+---
+layout: section
+title: "FlatMap"
+permalink: /documentation/transforms/python/elementwise/flatmap/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# FlatMap
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="FlatMap" %}
+
+Applies a simple 1-to-many mapping function over each element in the collection.
+The many elements are flattened into the resulting collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `FlatMap` in multiple ways to yield zero or more elements per each input element into the resulting `PCollection`.
+
+`FlatMap` accepts a function that returns an `iterable`,
+where each of the output `iterable`'s elements is an element of the resulting `PCollection`.
+
+### Example 1: FlatMap with a predefined function
+
+We use the function `str.split` which takes a single `str` element and outputs a `list` of `str`s.
+This pipeline splits the input element using whitespaces, creating a list of zero or more elements.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_simple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 2: FlatMap with a function
+
+We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 3: FlatMap with a lambda function
+
+For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
+Each input element is already an `iterable`, where each element is what we want in the resulting `PCollection`.
+We use a lambda function that returns the same input element it received.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 4: FlatMap with a generator
+
+For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
+We use a generator to iterate over the input list and yield each of the elements.
+Each yielded result in the generator is an element in the resulting `PCollection`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_generator %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 5: FlatMapTuple for key-value pairs
+
+If your `PCollection` consists of `(key, value)` pairs,
+you can use `FlatMapTuple` to unpack them into different function arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_tuple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMapTuple`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 6: FlatMap with multiple arguments
+
+You can pass functions with multiple arguments to `FlatMap`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `split_words` takes `text` and `delimiter` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 7: FlatMap with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `','` as a singleton.
+We then use that value as the delimiter for the `str.split` method.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+### Example 8: FlatMap with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:valid_plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 9: FlatMap with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py tag:flatmap_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `FlatMap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py tag:valid_plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap"
+%}
+
+## Related transforms
+
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="FlatMap" %}
diff --git a/website/src/documentation/transforms/python/elementwise/keys.md b/website/src/documentation/transforms/python/elementwise/keys.md
new file mode 100644
index 0000000..4473ee9
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/keys.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Keys"
+permalink: /documentation/transforms/python/elementwise/keys/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Keys
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Keys" %}
+
+Takes a collection of key-value pairs and returns the key of each element.
+
+## Example
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `Keys` to extract the keys and discard the values.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py tag:keys %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Keys`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py tag:icons %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/keys"
+%}
+
+## Related transforms
+
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
+* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Keys" %}
diff --git a/website/src/documentation/transforms/python/elementwise/kvswap.md b/website/src/documentation/transforms/python/elementwise/kvswap.md
new file mode 100644
index 0000000..2268100
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/kvswap.md
@@ -0,0 +1,57 @@
+---
+layout: section
+title: "KvSwap"
+permalink: /documentation/transforms/python/elementwise/kvswap/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Kvswap
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="KvSwap" %}
+
+Takes a collection of key-value pairs and returns a collection of key-value pairs
+which has each key and value swapped.
+
+## Examples
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `KvSwap` to swap the keys and values.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py tag:kvswap %}```
+
+{:.notebook-skip}
+Output `PCollection` after `KvSwap`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/kvswap"
+%}
+
+## Related transforms
+
+* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
+* [Values]({{ site.baseurl }}/documentation/transforms/python/elementwise/values) for extracting the value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="KvSwap" %}
diff --git a/website/src/documentation/transforms/python/elementwise/map.md b/website/src/documentation/transforms/python/elementwise/map.md
new file mode 100644
index 0000000..2ad0e33
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/map.md
@@ -0,0 +1,216 @@
+---
+layout: section
+title: "Map"
+permalink: /documentation/transforms/python/elementwise/map/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Map
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Map" %}
+
+Applies a simple 1-to-1 mapping function over each element in the collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Map` in multiple ways to transform every element in the `PCollection`.
+
+`Map` accepts a function that returns a single element for every input element in the `PCollection`.
+
+### Example 1: Map with a predefined function
+
+We use the function `str.strip` which takes a single `str` element and outputs a `str`.
+It strips the input element's whitespaces, including newlines and tabs.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_simple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 2: Map with a function
+
+We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\n'` characters from each element.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_function %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 3: Map with a lambda function
+
+We can also use lambda functions to simplify **Example 2**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 4: Map with multiple arguments
+
+You can pass functions with multiple arguments to `Map`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In this example, `strip` takes `text` and `chars` as arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 5: MapTuple for key-value pairs
+
+If your `PCollection` consists of `(key, value)` pairs,
+you can use `MapTuple` to unpack them into different function arguments.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_tuple %}```
+
+{:.notebook-skip}
+Output `PCollection` after `MapTuple`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 6: Map with side inputs as singletons
+
+If the `PCollection` has a single value, such as the average from another computation,
+passing the `PCollection` as a *singleton* accesses that value.
+
+In this example, we pass a `PCollection` the value `'# \n'` as a singleton.
+We then use that value as the characters for the `str.strip` method.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_singleton %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+### Example 7: Map with side inputs as iterators
+
+If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
+This accesses elements lazily as they are needed,
+so it is possible to iterate over large `PCollection`s that won't fit into memory.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_iter %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
+> but this requires that all the elements fit into memory.
+
+### Example 8: Map with side inputs as dictionaries
+
+If a `PCollection` is small enough to fit into memory, then that `PCollection` can be passed as a *dictionary*.
+Each element must be a `(key, value)` pair.
+Note that all the elements of the `PCollection` must fit into memory for this.
+If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py tag:map_side_inputs_dict %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Map`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py tag:plant_details %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it may produce zero or more outputs.
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Map" %}
diff --git a/website/src/documentation/transforms/python/elementwise/pardo.md b/website/src/documentation/transforms/python/elementwise/pardo.md
new file mode 100644
index 0000000..40315a8
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/pardo.md
@@ -0,0 +1,167 @@
+---
+layout: section
+title: "ParDo"
+permalink: /documentation/transforms/python/elementwise/pardo/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# ParDo
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="ParDo" %}
+
+A transform for generic parallel processing.
+A `ParDo` transform considers each element in the input `PCollection`,
+performs some processing function (your user code) on that element,
+and emits zero or more elements to an output `PCollection`.
+
+See more information in the
+[Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#pardo).
+
+## Examples
+
+In the following examples, we explore how to create custom `DoFn`s and access
+the timestamp and windowing information.
+
+### Example 1: ParDo with a simple DoFn
+
+The following example defines a simple `DoFn` class called `SplitWords`
+which stores the `delimiter` as an object field.
+The `process` method is called once per element,
+and it can yield zero or more output elements.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ParDo`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
+
+### Example 2: ParDo with timestamp and window information
+
+In this example, we add new parameters to the `process` method to bind parameter values at runtime.
+
+* [`beam.DoFn.TimestampParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.TimestampParam)
+ binds the timestamp information as an
+ [`apache_beam.utils.timestamp.Timestamp`](https://beam.apache.org/releases/pydoc/current/apache_beam.utils.timestamp.html#apache_beam.utils.timestamp.Timestamp)
+ object.
+* [`beam.DoFn.WindowParam`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.WindowParam)
+ binds the window information as the appropriate
+ [`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)
+ object.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn_params %}```
+
+{:.notebook-skip}
+`stdout` output:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:dofn_params %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
+
+### Example 3: ParDo with DoFn methods
+
+A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)
+can be customized with a number of methods that can help create more complex behaviors.
+You can customize what a worker does when it starts and shuts down with `setup` and `teardown`.
+You can also customize what to do when a
+[*bundle of elements*](https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence)
+starts and finishes with `start_bundle` and `finish_bundle`.
+
+* [`DoFn.setup()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.setup):
+ Called *once per `DoFn` instance* when the `DoFn` instance is initialized.
+ `setup` need not to be cached, so it could be called more than once per worker.
+ This is a good place to connect to database instances, open network connections or other resources.
+
+* [`DoFn.start_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.start_bundle):
+ Called *once per bundle of elements* before calling `process` on the first element of the bundle.
+ This is a good place to start keeping track of the bundle elements.
+
+* [**`DoFn.process(element, *args, **kwargs)`**](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.process):
+ Called *once per element*, can *yield zero or more elements*.
+ Additional `*args` or `**kwargs` can be passed through
+ [`beam.ParDo()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.ParDo).
+ **[required]**
+
+* [`DoFn.finish_bundle()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.finish_bundle):
+ Called *once per bundle of elements* after calling `process` after the last element of the bundle,
+ can *yield zero or more elements*. This is a good place to do batch calls on a bundle of elements,
+ such as running a database query.
+
+ For example, you can initialize a batch in `start_bundle`,
+ add elements to the batch in `process` instead of yielding them,
+ then running a batch query on those elements on `finish_bundle`, and yielding all the results.
+
+ Note that yielded elements from `finish_bundle` must be of the type
+ [`apache_beam.utils.windowed_value.WindowedValue`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/windowed_value.py).
+ You need to provide a timestamp as a unix timestamp, which you can get from the last processed element.
+ You also need to provide a window, which you can get from the last processed element like in the example below.
+
+* [`DoFn.teardown()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.teardown):
+ Called *once (as a best effort) per `DoFn` instance* when the `DoFn` instance is shutting down.
+ This is a good place to close database instances, close network connections or other resources.
+
+ Note that `teardown` is called as a *best effort* and is *not guaranteed*.
+ For example, if the worker crashes, `teardown` might not be called.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py tag:pardo_dofn_methods %}```
+
+{:.notebook-skip}
+`stdout` output:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py tag:results %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo"
+%}
+
+> *Known issues:*
+>
+> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)
+> `DoFn.setup()` doesn't run for streaming jobs running in the `DirectRunner`.
+> * [[BEAM-7340]](https://issues.apache.org/jira/browse/BEAM-7340)
+> `DoFn.teardown()` metrics are lost.
+
+## Related transforms
+
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) behaves the same, but produces exactly one output for each input.
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`,
+ but for each input it may produce zero or more outputs.
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="ParDo" %}
diff --git a/website/src/documentation/transforms/python/elementwise/partition.md b/website/src/documentation/transforms/python/elementwise/partition.md
new file mode 100644
index 0000000..00949c3
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/partition.md
@@ -0,0 +1,144 @@
+---
+layout: section
+title: "Partition"
+permalink: /documentation/transforms/python/elementwise/partition/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Partition
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Partition" %}
+
+Separates elements in a collection into multiple output
+collections. The partitioning function contains the logic that determines how
+to separate the elements of the input collection into each resulting
+partition output collection.
+
+The number of partitions must be determined at graph construction time.
+You cannot determine the number of partitions in mid-pipeline
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#partition).
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of produce with their icon, name, and duration.
+Then, we apply `Partition` in multiple ways to split the `PCollection` into multiple `PCollections`.
+
+`Partition` accepts a function that receives the number of partitions,
+and returns the index of the desired partition for the element.
+The number of partitions passed must be a positive integer,
+and it must return an integer in the range `0` to `num_partitions-1`.
+
+### Example 1: Partition with a function
+
+In the following example, we have a known list of durations.
+We partition the `PCollection` into one `PCollection` for every duration type.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_function %}```
+
+{:.notebook-skip}
+Output `PCollection`s:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:partitions %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
+
+### Example 2: Partition with a lambda function
+
+We can also use lambda functions to simplify **Example 1**.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_lambda %}```
+
+{:.notebook-skip}
+Output `PCollection`s:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:partitions %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
+
+### Example 3: Partition with multiple arguments
+
+You can pass functions with multiple arguments to `Partition`.
+They are passed as additional positional arguments or keyword arguments to the function.
+
+In machine learning, it is a common task to split data into
+[training and a testing datasets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).
+Typically, 80% of the data is used for training a model and 20% is used for testing.
+
+In this example, we split a `PCollection` dataset into training and testing datasets.
+We define `split_dataset`, which takes the `plant` element, `num_partitions`,
+and an additional argument `ratio`.
+The `ratio` is a list of numbers which represents the ratio of how many items will go into each partition.
+`num_partitions` is used by `Partitions` as a positional argument,
+while `plant` and `ratio` are passed to `split_dataset`.
+
+If we want an 80%/20% split, we can specify a ratio of `[8, 2]`, which means that for every 10 elements,
+8 go into the first partition and 2 go into the second.
+In order to determine which partition to send each element, we have different buckets.
+For our case `[8, 2]` has **10** buckets,
+where the first 8 buckets represent the first partition and the last 2 buckets represent the second partition.
+
+First, we check that the ratio list's length corresponds to the `num_partitions` we pass.
+We then get a bucket index for each element, in the range from 0 to 9 (`num_buckets-1`).
+We could do `hash(element) % len(ratio)`, but instead we sum all the ASCII characters of the
+JSON representation to make it deterministic.
+Finally, we loop through all the elements in the ratio and have a running total to
+identify the partition index to which that bucket corresponds.
+
+This `split_dataset` function is generic enough to support any number of partitions by any ratio.
+You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py tag:partition_multiple_arguments %}```
+
+{:.notebook-skip}
+Output `PCollection`s:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py tag:train_test %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition"
+%}
+
+## Related transforms
+
+* [Filter]({{ site.baseurl }}/documentation/transforms/python/elementwise/filter) is useful if the function is just
+ deciding whether to output an element or not.
+* [ParDo]({{ site.baseurl }}/documentation/transforms/python/elementwise/pardo) is the most general elementwise mapping
+ operation, and includes other abilities such as multiple output collections and side-inputs.
+* [CoGroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey)
+performs a per-key equijoin.
+
+{% include button-pydoc.md path="apache_beam.transforms.core" class="Partition" %}
diff --git a/website/src/documentation/transforms/python/elementwise/regex.md b/website/src/documentation/transforms/python/elementwise/regex.md
new file mode 100644
index 0000000..e8c1df9
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/regex.md
@@ -0,0 +1,299 @@
+---
+layout: section
+title: "Regex"
+permalink: /documentation/transforms/python/elementwise/regex/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Regex
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Regex" %}
+
+Filters input string elements based on a regex. May also transform them based on the matching groups.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` of text strings.
+Then, we use the `Regex` transform to search, replace, and split through the text elements using
+[regular expressions](https://docs.python.org/3/library/re.html).
+
+You can use tools to help you create and test your regular expressions, such as
+[regex101](https://regex101.com/).
+Make sure to specify the Python flavor at the left side bar.
+
+Lets look at the
+[regular expression `(?P<icon>[^\s,]+), *(\w+), *(\w+)`](https://regex101.com/r/Z7hTTj/3)
+for example.
+It matches anything that is not a whitespace `\s` (`[ \t\n\r\f\v]`) or comma `,`
+until a comma is found and stores that in the named group `icon`,
+this can match even `utf-8` strings.
+Then it matches any number of whitespaces, followed by at least one word character
+`\w` (`[a-zA-Z0-9_]`), which is stored in the second group for the *name*.
+It does the same with the third group for the *duration*.
+
+> *Note:* To avoid unexpected string escaping in your regular expressions,
+> it is recommended to use
+> [raw strings](https://docs.python.org/3/reference/lexical_analysis.html?highlight=raw#string-and-bytes-literals)
+> such as `r'raw-string'` instead of `'escaped-string'`.
+
+### Example 1: Regex match
+
+`Regex.matches` keeps only the elements that match the regular expression,
+returning the matched group.
+The argument `group` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.matches` starts to match the regular expression at the beginning of the string.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+[`Regex.find(regex)`](#example-4-regex-find).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_matches %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.matches`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 2: Regex match with all groups
+
+`Regex.all_matches` keeps only the elements that match the regular expression,
+returning *all groups* as a list.
+The groups are returned in the order encountered in the regular expression,
+including `group 0` (the entire match) as the first group.
+
+`Regex.all_matches` starts to match the regular expression at the beginning of the string.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_all_matches %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.all_matches`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_all_matches %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 3: Regex match into key-value pairs
+
+`Regex.matches_kv` keeps only the elements that match the regular expression,
+returning a key-value pair using the specified groups.
+The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.
+The argument `valueGroup` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.matches_kv` starts to match the regular expression at the beginning of the string.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+To start matching at any point instead of the beginning of the string, use
+[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_matches_kv %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.matches_kv`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches_kv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 4: Regex find
+
+`Regex.find` keeps only the elements that match the regular expression,
+returning the matched group.
+The argument `group` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.find` matches the first occurrence of the regular expression in the string.
+To start matching at the beginning, add `'^'` at the beginning of the regular expression.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+If you need to match from the start only, consider using
+[`Regex.matches(regex)`](#example-1-regex-match).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.find`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_matches %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 5: Regex find all
+
+`Regex.find_all` returns a list of all the matches of the regular expression,
+returning the matched group.
+The argument `group` is set to `0` by default, but can be set to a group number like `3`, to a named group like `'icon'`, or to `Regex.ALL` to return all groups.
+The argument `outputEmpty` is set to `True` by default, but can be set to `False` to skip elements where no matches were found.
+
+`Regex.find_all` matches the regular expression anywhere it is found in the string.
+To start matching at the beginning, add `'^'` at the start of the regular expression.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+If you need to match all groups from the start only, consider using
+[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find_all %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.find_all`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_find_all %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 6: Regex find as key-value pairs
+
+`Regex.find_kv` returns a list of all the matches of the regular expression,
+returning a key-value pair using the specified groups.
+The argument `keyGroup` is set to a group number like `3`, or to a named group like `'icon'`.
+The argument `valueGroup` is set to `0` (the entire match) by default,
+but can be set to a group number like `3`, or to a named group like `'icon'`.
+
+`Regex.find_kv` matches the first occurrence of the regular expression in the string.
+To start matching at the beginning, add `'^'` at the beginning of the regular expression.
+To match until the end of the string, add `'$'` at the end of the regular expression.
+
+If you need to match as key-value pairs from the start only, consider using
+[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs).
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_find_kv %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.find_kv`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_find_kv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 7: Regex replace all
+
+`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.
+You can also use
+[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
+on the `replacement`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_replace_all %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.replace_all`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_replace_all %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 8: Regex replace first
+
+`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.
+You can also use
+[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
+on the `replacement`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_replace_first %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.replace_first`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_replace_first %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+### Example 9: Regex split
+
+`Regex.split` returns the list of strings that were delimited by the specified regular expression.
+The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py tag:regex_split %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Regex.split`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py tag:plants_split %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex"
+%}
+
+## Related transforms
+
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
+ each input it may produce zero or more outputs.
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Regex" %}
diff --git a/website/src/documentation/transforms/python/elementwise/reify.md b/website/src/documentation/transforms/python/elementwise/reify.md
new file mode 100644
index 0000000..2c7e842
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/reify.md
@@ -0,0 +1,28 @@
+---
+layout: section
+title: "Reify"
+permalink: /documentation/transforms/python/elementwise/reify/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Reify
+Transforms for converting between explicit and implicit form of various Beam values.
+
+## Examples
+See [BEAM-7389](https://issues.apache.org/jira/browse/BEAM-7389) for updates.
+
+## Related transforms
+* [WithTimestamps]({{ site.baseurl }}/documentation/transforms/python/elementwise/withtimestamps) assigns timestamps to all the elements of a collection.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/elementwise/tostring.md b/website/src/documentation/transforms/python/elementwise/tostring.md
new file mode 100644
index 0000000..c935b52
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/tostring.md
@@ -0,0 +1,104 @@
+---
+layout: section
+title: "ToString"
+permalink: /documentation/transforms/python/elementwise/tostring/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# ToString
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="ToString" %}
+
+Transforms every element in an input collection to a string.
+
+## Examples
+
+Any non-string element can be converted to a string using standard Python functions and methods.
+Many I/O transforms, such as
+[`textio.WriteToText`](https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html#apache_beam.io.textio.WriteToText),
+expect their input elements to be strings.
+
+### Example 1: Key-value pairs to string
+
+The following example converts a `(key, value)` pair into a string delimited by `','`.
+You can specify a different delimiter using the `delimiter` argument.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_kvs %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+### Example 2: Elements to string
+
+The following example converts a dictionary into a string.
+The string output will be equivalent to `str(element)`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_element %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plant_lists %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+### Example 3: Iterables to string
+
+The following example converts an iterable, in this case a list of strings,
+into a string delimited by `','`.
+You can specify a different delimiter using the `delimiter` argument.
+The string output will be equivalent to `iterable.join(delimiter)`.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py tag:tostring_iterables %}```
+
+{:.notebook-skip}
+Output `PCollection` after `ToString`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py tag:plants_csv %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring"
+%}
+
+## Related transforms
+
+* [Map]({{ site.baseurl }}/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="ToString" %}
diff --git a/website/src/documentation/transforms/python/elementwise/values.md b/website/src/documentation/transforms/python/elementwise/values.md
new file mode 100644
index 0000000..ae79578
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/values.md
@@ -0,0 +1,56 @@
+---
+layout: section
+title: "Values"
+permalink: /documentation/transforms/python/elementwise/values/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Values
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Values" %}
+
+Takes a collection of key-value pairs, and returns the value of each element.
+
+## Example
+
+In the following example, we create a pipeline with a `PCollection` of key-value pairs.
+Then, we apply `Values` to extract the values and discard the keys.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py tag:values %}```
+
+{:.notebook-skip}
+Output `PCollection` after `Values`:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py tag:plants %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/values"
+%}
+
+## Related transforms
+
+* [Keys]({{ site.baseurl }}/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
+* [KvSwap]({{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
+
+{% include button-pydoc.md path="apache_beam.transforms.util" class="Values" %}
diff --git a/website/src/documentation/transforms/python/elementwise/withkeys.md b/website/src/documentation/transforms/python/elementwise/withkeys.md
new file mode 100644
index 0000000..f9244e1
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/withkeys.md
@@ -0,0 +1,27 @@
+---
+layout: section
+title: "WithKeys"
+permalink: /documentation/transforms/python/elementwise/withkeys/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# WithKeys
+Not available.
+
+## Examples
+See [BEAM-7389](https://issues.apache.org/jira/browse/BEAM-7389) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/elementwise/withtimestamps.md b/website/src/documentation/transforms/python/elementwise/withtimestamps.md
new file mode 100644
index 0000000..ea64c4e
--- /dev/null
+++ b/website/src/documentation/transforms/python/elementwise/withtimestamps.md
@@ -0,0 +1,120 @@
+---
+layout: section
+title: "WithTimestamps"
+permalink: /documentation/transforms/python/elementwise/withtimestamps/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# WithTimestamps
+
+<script type="text/javascript">
+localStorage.setItem('language', 'language-py')
+</script>
+
+Assigns timestamps to all the elements of a collection.
+
+## Examples
+
+In the following examples, we create a pipeline with a `PCollection` and attach a timestamp value to each of its elements.
+When windowing and late data play an important role in streaming pipelines, timestamps are especially useful.
+
+### Example 1: Timestamp by event time
+
+The elements themselves often already contain a timestamp field.
+`beam.window.TimestampedValue` takes a value and a
+[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)
+in the form of seconds.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_event_time %}```
+
+{:.notebook-skip}
+Output `PCollection` after getting the timestamps:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_timestamps %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
+
+To convert from a
+[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)
+to `unix_time` you can use
+[`time.mktime`](https://docs.python.org/3/library/time.html#time.mktime).
+For more information on time formatting options, see
+[`time.strftime`](https://docs.python.org/3/library/time.html#time.strftime).
+
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:time_tuple2unix_time %}```
+
+To convert from a
+[`datetime.datetime`](https://docs.python.org/3/library/datetime.html#datetime.datetime)
+to `unix_time` you can use convert it to a `time.struct_time` first with
+[`datetime.timetuple`](https://docs.python.org/3/library/datetime.html#datetime.datetime.timetuple).
+
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:datetime2unix_time %}```
+
+### Example 2: Timestamp by logical clock
+
+If each element has a chronological number, these numbers can be used as a
+[logical clock](https://en.wikipedia.org/wiki/Logical_clock).
+These numbers have to be converted to a *"seconds"* equivalent, which can be especially important depending on your windowing and late data rules.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_logical_clock %}```
+
+{:.notebook-skip}
+Output `PCollection` after getting the timestamps:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_events %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
+
+### Example 3: Timestamp by processing time
+
+If the elements do not have any time data available, you can also use the current processing time for each element.
+Note that this grabs the local time of the *worker* that is processing each element.
+Workers might have time deltas, so using this method is not a reliable way to do precise ordering.
+
+By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline.
+
+```py
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py tag:withtimestamps_processing_time %}```
+
+{:.notebook-skip}
+Output `PCollection` after getting the timestamps:
+
+{:.notebook-skip}
+```
+{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py tag:plant_processing_times %}```
+
+{% include buttons-code-snippet.md
+ py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps"
+%}
+
+## Related transforms
+
+* [Reify]({{ site.baseurl }}/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values.
diff --git a/website/src/documentation/transforms/python/index.md b/website/src/documentation/transforms/python/index.md
new file mode 100644
index 0000000..dad96b0
--- /dev/null
+++ b/website/src/documentation/transforms/python/index.md
@@ -0,0 +1,86 @@
+---
+layout: section
+title: "Python transform catalog overview"
+permalink: /documentation/transforms/python/overview/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Python transform catalog overview
+
+## Element-wise
+
+<table class="table-bordered table-striped">
+ <tr><th>Transform</th><th>Description</th></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/filter">Filter</a></td><td>Given a predicate, filter out all elements that don't satisfy the predicate.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap">FlatMap</a></td><td>Applies a function that returns a collection to every element in the input and
+ outputs all resulting elements.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/keys">Keys</a></td><td>Extracts the key from each element in a collection of key-value pairs.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/kvswap">KvSwap</a></td><td>Swaps the key and value of each element in a collection of key-value pairs.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/map">Map</a></td><td>Applies a function to every element in the input and outputs the result.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/pardo">ParDo</a></td><td>The most-general mechanism for applying a user-defined <code>DoFn</code> to every element
+ in the input collection.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/partition">Partition</a></td><td>Routes each input element to a specific output collection based on some partition
+ function.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/regex">Regex</a></td><td>Filters input string elements based on a regex. May also transform them based on the matching groups.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/reify">Reify</a></td><td>Transforms for converting between explicit and implicit form of various Beam values.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/tostring">ToString</a></td><td>Transforms every element in an input collection a string.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/withtimestamps">WithTimestamps</a></td><td>Applies a function to determine a timestamp to each element in the output collection,
+ and updates the implicit timestamp associated with each input. Note that it is only
+ safe to adjust timestamps forwards.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/elementwise/values">Values</a></td><td>Extracts the value from each element in a collection of key-value pairs.</td></tr>
+</table>
+
+
+
+## Aggregation
+<table class="table-bordered table-striped">
+ <tr><th>Transform</th><th>Description</th></tr>
+ <tr><td>ApproximateQuantiles</td><td>Not available. See <a href="https://issues.apache.org/jira/browse/BEAM-6694">BEAM-6694</a> for updates.</td></tr>
+ <tr><td>ApproximateUnique</td><td>Not available. See <a href="https://issues.apache.org/jira/browse/BEAM-6693">BEAM-6693</a> for updates.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/cogroupbykey">CoGroupByKey</a></td><td>Takes several keyed collections of elements and produces a collection where each element
+ consists of a key and all values associated with that key.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/combineglobally">CombineGlobally</a></td><td>Transforms to combine elements.</td></tr>
+ <tr><td>CombineWithContext</td><td>Not available.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/count">Count</a></td><td>Counts the number of elements within each aggregation.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/distinct">Distinct</a></td><td>Produces a collection containing distinct elements from the input collection.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/groupbykey">GroupByKey</a></td><td>Takes a keyed collection of elements and produces a collection where each element
+ consists of a key and all values associated with that key.</td></tr>
+ <tr><td>GroupIntoBatches</td><td>Not available. See <a href="https://issues.apache.org/jira/browse/BEAM-6696">BEAM-6696</a> for updates.</td></tr>
+ <tr><td>Latest</td><td>Not available. See <a href="https://issues.apache.org/jira/browse/BEAM-6695">BEAM-6695</a> for updates.</td></tr>
+ <tr><td>Max</td><td>Not available.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/mean">Mean</a></td><td>Computes the average within each aggregation.</td></tr>
+ <tr><td>Min</td><td>Not available.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/sample">Sample</a></td><td>Randomly select some number of elements from each aggregation.</td></tr>
+ <tr><td>Sum</td><td>Not available.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/aggregation/top">Top</a></td><td>Compute the largest element(s) in each aggregation.</td></tr>
+</table>
+
+
+## Other
+<table class="table-bordered table-striped">
+ <tr><th>Transform</th><th>Description</th></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/other/create">Create</a></td><td>Creates a collection from an in-memory list.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/other/flatten">Flatten</a></td><td>Given multiple input collections, produces a single output collection containing
+ all elements from all of the input collections.
+</td></tr>
+ <tr><td>PAssert</td><td>Not available.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/other/reshuffle">Reshuffle</a></td><td>Given an input collection, redistributes the elements between workers. This is
+ most useful for adjusting parallelism or preventing coupled failures.</td></tr>
+ <tr><td>View</td><td>Not available.</td></tr>
+ <tr><td><a href="{{ site.baseurl }}/documentation/transforms/python/other/windowinto">WindowInto</a></td><td>Logically divides up or groups the elements of a collection into finite
+ windows according to a function.</td></tr>
+</table>
+
diff --git a/website/src/documentation/transforms/python/other/create.md b/website/src/documentation/transforms/python/other/create.md
new file mode 100644
index 0000000..d6ce766
--- /dev/null
+++ b/website/src/documentation/transforms/python/other/create.md
@@ -0,0 +1,38 @@
+---
+layout: section
+title: "Create"
+permalink: /documentation/transforms/python/other/create/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Create
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Create">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Creates a collection containing a specified set of elements. This is
+useful for testing, as well as creating an initial input to process
+in parallel. For example, a single element to execute a one-time
+`ParDo` or a list of filenames to be read.
+
+## Examples
+See [BEAM-7391](https://issues.apache.org/jira/browse/BEAM-7391) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/other/flatten.md b/website/src/documentation/transforms/python/other/flatten.md
new file mode 100644
index 0000000..37ecfb9
--- /dev/null
+++ b/website/src/documentation/transforms/python/other/flatten.md
@@ -0,0 +1,43 @@
+---
+layout: section
+title: "Flatten"
+permalink: /documentation/transforms/python/other/flatten/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Flatten
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=flatten#apache_beam.transforms.core.Flatten">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+Merges multiple `PCollection` objects into a single logical
+`PCollection`. A transform for `PCollection` objects
+that store the same data type.
+
+See more information in the [Beam Programming Guide]({{ site.baseurl }}/documentation/programming-guide/#flatten).
+
+## Examples
+See [BEAM-7391](https://issues.apache.org/jira/browse/BEAM-7391) for updates.
+
+## Related transforms
+* [FlatMap]({{ site.baseurl }}/documentation/transforms/python/elementwise/flatmap) applies a simple 1-to-many mapping
+ function over each element in the collection. This transform might produce zero
+ or more outputs.
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/other/passert.md b/website/src/documentation/transforms/python/other/passert.md
new file mode 100644
index 0000000..9ec6a98
--- /dev/null
+++ b/website/src/documentation/transforms/python/other/passert.md
@@ -0,0 +1,25 @@
+---
+layout: section
+title: "PAssert"
+permalink: /documentation/transforms/python/other/passert/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# PAssert
+
+## Examples
+See [BEAM-7391](https://issues.apache.org/jira/browse/BEAM-7391) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/other/reshuffle.md b/website/src/documentation/transforms/python/other/reshuffle.md
new file mode 100644
index 0000000..f1b636a
--- /dev/null
+++ b/website/src/documentation/transforms/python/other/reshuffle.md
@@ -0,0 +1,41 @@
+---
+layout: section
+title: "Reshuffle"
+permalink: /documentation/transforms/python/other/reshuffle/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Reshuffle
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html?highlight=reshuffle#apache_beam.transforms.util.Reshuffle">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px"
+ alt="Pydoc" />
+ Pydoc
+ </a>
+</table>
+<br>
+ Adds a temporary random key to each element in a collection, reshuffles
+ these keys, and removes the temporary key. This redistributes the
+ elements between workers and returns a collection equivalent to its
+ input collection. This is most useful for adjusting parallelism or
+ preventing coupled failures.
+
+## Examples
+See [BEAM-7391](https://issues.apache.org/jira/browse/BEAM-7391) for updates.
+
+## Related transforms
+N/A
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/other/view.md b/website/src/documentation/transforms/python/other/view.md
new file mode 100644
index 0000000..67475be
--- /dev/null
+++ b/website/src/documentation/transforms/python/other/view.md
@@ -0,0 +1,25 @@
+---
+layout: section
+title: "View"
+permalink: /documentation/transforms/python/other/view/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# View
+
+## Examples
+See [BEAM-7391](https://issues.apache.org/jira/browse/BEAM-7391) for updates.
+
+## Related transforms
\ No newline at end of file
diff --git a/website/src/documentation/transforms/python/other/windowinto.md b/website/src/documentation/transforms/python/other/windowinto.md
new file mode 100644
index 0000000..dd0367e
--- /dev/null
+++ b/website/src/documentation/transforms/python/other/windowinto.md
@@ -0,0 +1,41 @@
+---
+layout: section
+title: "WindowInto"
+permalink: /documentation/transforms/python/other/windowinto/
+section_menu: section-menu/documentation.html
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# WindowInto
+<table align="left">
+ <a target="_blank" class="button"
+ href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html?highlight=window#module-apache_beam.transforms.window">
+ <img src="https://beam.apache.org/images/logos/sdks/python.png" width="20px" height="20px" alt="Pydoc">
+ Pydoc
+ </a>
+</table>
+<br>
+Logically divides up or groups the elements of a collection into finite
+windows according to a function.
+
+## Examples
+See [BEAM-7391](https://issues.apache.org/jira/browse/BEAM-7391) for updates.
+
+## Related transforms
+* [GroupByKey]({{ site.baseurl }}/documentation/transforms/python/aggregation/groupbykey)
+ produces a collection where each element consists of a key and all values associated
+ with that key.
+* [Timestamp]({{ site.baseurl }}/documentation/transforms/python/elementwise/withtimestamps)
+ applies a function to determine a timestamp to each element in the output collection.
\ No newline at end of file
diff --git a/website/src/get-started/beam-overview.md b/website/src/get-started/beam-overview.md
index 6d70ab84..6521d17 100644
--- a/website/src/get-started/beam-overview.md
+++ b/website/src/get-started/beam-overview.md
@@ -51,6 +51,7 @@
* Apache Samza ![Apache Samza logo]({{ "/images/logos/runners/samza.png" | prepend: site.baseurl }}){:height="20px" width="50"}
* Apache Spark ![Apache Spark logo]({{ "/images/logos/runners/spark.png" | prepend: site.baseurl }})
* Google Cloud Dataflow ![Google Cloud Dataflow logo]({{ "/images/logos/runners/dataflow.png" | prepend: site.baseurl }})
+* Hazelcast Jet ![Hazelcast Jet logo]({{ "/images/logos/runners/jet.png" | prepend: site.baseurl }})
**Note:** You can always execute your pipeline locally for testing and debugging purposes.
diff --git a/website/src/get-started/downloads.md b/website/src/get-started/downloads.md
index e9744c2..ab19e5b 100644
--- a/website/src/get-started/downloads.md
+++ b/website/src/get-started/downloads.md
@@ -90,24 +90,52 @@
## Releases
+## 2.16.0 (2019-10-07)
+Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.16.0/apache-beam-2.16.0-source-release.zip).
+[SHA-512](https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.sha512).
+[signature](https://www.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.asc).
+
+[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345494).
+
+## 2.15.0 (2019-08-22)
+Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.15.0/apache-beam-2.15.0-source-release.zip).
+[SHA-512](https://www.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip.sha512).
+[signature](https://www.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip.asc).
+
+[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345489).
+
+### 2.14.0 (2019-08-01)
+Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.14.0/apache-beam-2.14.0-source-release.zip).
+[SHA-512](https://www.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.sha512).
+[signature](https://www.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.asc).
+
+[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345431).
+
+### 2.13.0 (2019-05-21)
+Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.13.0/apache-beam-2.13.0-source-release.zip).
+[SHA-512](https://www.apache.org/dist/beam/2.13.0/apache-beam-2.13.0-source-release.zip.sha512).
+[signature](https://www.apache.org/dist/beam/2.13.0/apache-beam-2.13.0-source-release.zip.asc).
+
+[Release notes](https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12345166).
+
### 2.12.0 (2019-04-25)
-Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.12.0/apache-beam-2.12.0-source-release.zip).
-[SHA-512](https://www.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip.sha512).
-[signature](https://www.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip.asc).
+Official [source code download](http://archive.apache.org/dyn/closer.cgi/beam/2.12.0/apache-beam-2.12.0-source-release.zip).
+[SHA-512](https://archive.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip.sha512).
+[signature](https://archive.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip.asc).
[Release notes](https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12344944).
### 2.11.0 (2019-02-26)
-Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.11.0/apache-beam-2.11.0-source-release.zip).
-[SHA-512](https://www.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip.sha512).
-[signature](https://www.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip.asc).
+Official [source code download](http://archive.apache.org/dyn/closer.cgi/beam/2.11.0/apache-beam-2.11.0-source-release.zip).
+[SHA-512](https://archive.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip.sha512).
+[signature](https://archive.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip.asc).
[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12344775).
### 2.10.0 (2019-02-01)
-Official [source code download](http://www.apache.org/dyn/closer.cgi/beam/2.10.0/apache-beam-2.10.0-source-release.zip).
-[SHA-512](https://www.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip.sha512).
-[signature](https://www.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip.asc).
+Official [source code download](http://archive.apache.org/dyn/closer.cgi/beam/2.10.0/apache-beam-2.10.0-source-release.zip).
+[SHA-512](https://archive.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip.sha512).
+[signature](https://archive.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip.asc).
[Release notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12344540).
diff --git a/website/src/get-started/quickstart-go.md b/website/src/get-started/quickstart-go.md
index 1151fa4..f661728 100644
--- a/website/src/get-started/quickstart-go.md
+++ b/website/src/get-started/quickstart-go.md
@@ -71,7 +71,7 @@
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
--staging_location gs://<your-gcs-bucket>/binaries/ \
- --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
+ --worker_harness_container_image=apachebeam/go_sdk:latest
```
{:.runner-nemo}
diff --git a/website/src/get-started/quickstart-java.md b/website/src/get-started/quickstart-java.md
index 7b74720..5a4525b 100644
--- a/website/src/get-started/quickstart-java.md
+++ b/website/src/get-started/quickstart-java.md
@@ -116,7 +116,7 @@
## Run WordCount
-A single Beam pipeline can run on multiple Beam [runners]({{ site.baseurl }}/documentation#runners), including the [ApexRunner]({{ site.baseurl }}/documentation/runners/apex), [FlinkRunner]({{ site.baseurl }}/documentation/runners/flink), [SparkRunner]({{ site.baseurl }}/documentation/runners/spark), [NemoRunner]({{ site.baseurl }}/documentation/runners/nemo), or [DataflowRunner]({{ site.baseurl }}/documentation/runners/dataflow). The [DirectRunner]({{ site.baseurl }}/documentation/runners/direct) is a common runner for getting started, as it runs locally on your machine and requires no specific setup.
+A single Beam pipeline can run on multiple Beam [runners]({{ site.baseurl }}/documentation#runners), including the [ApexRunner]({{ site.baseurl }}/documentation/runners/apex), [FlinkRunner]({{ site.baseurl }}/documentation/runners/flink), [SparkRunner]({{ site.baseurl }}/documentation/runners/spark), [NemoRunner]({{ site.baseurl }}/documentation/runners/nemo), [JetRunner]({{ site.baseurl }}/documentation/runners/jet), or [DataflowRunner]({{ site.baseurl }}/documentation/runners/dataflow). The [DirectRunner]({{ site.baseurl }}/documentation/runners/direct) is a common runner for getting started, as it runs locally on your machine and requires no specific setup.
After you've chosen which runner you'd like to use:
@@ -185,6 +185,13 @@
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -Pjet-runner
+$ java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts
+```
+
For Windows PowerShell:
{:.runner-direct}
@@ -244,6 +251,13 @@
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+PS> mvn package -P jet-runner
+PS> java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount `
+ --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/pom.xml --output=counts
+```
+
## Inspect the results
Once the pipeline has completed, you can view the output. You'll notice that there may be multiple output files prefixed by `count`. The exact number of these files is decided by the runner, giving it the flexibility to do efficient, distributed execution.
@@ -288,6 +302,11 @@
$ ls counts*
```
+{:.runner-jet}
+```
+$ ls counts*
+```
+
When you look into the contents of the file, you'll see that they contain unique words and the number of occurrences of each word. The order of elements within the file may differ because the Beam model does not generally guarantee ordering, again to allow runners to optimize for efficiency.
{:.runner-direct}
@@ -395,6 +414,22 @@
...
```
+{:.runner-jet}
+```
+$ more counts*
+FlinkRunner: 1
+cleanupDaemonThreads: 2
+sdks: 4
+unit: 1
+Apache: 3
+IO: 2
+copyright: 1
+governing: 1
+overrides: 1
+YARN: 1
+...
+```
+
## Next Steps
* Learn more about the [Beam SDK for Java]({{ site.baseurl }}/documentation/sdks/java/)
diff --git a/website/src/get-started/quickstart-py.md b/website/src/get-started/quickstart-py.md
index 95740d5..19471ab 100644
--- a/website/src/get-started/quickstart-py.md
+++ b/website/src/get-started/quickstart-py.md
@@ -27,11 +27,13 @@
* TOC
{:toc}
+The Python SDK supports Python 2.7, 3.5, 3.6, and 3.7. New Python SDK releases will stop supporting Python 2.7 in 2020 ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)). For best results, use Beam with Python 3.
+
## Set up your environment
### Check your Python version
-The Beam SDK for Python requires Python version 2.7.x. Check that you have version 2.7.x by running:
+The Beam SDK requires Python 2 users to use Python 2.7 and Python 3 users to use Python 3.5 or higher. Check your version by running:
```
python --version
@@ -176,17 +178,20 @@
{:.runner-flink-local}
```
-This runner is not yet available for the Python SDK.
+Currently, running wordcount.py on Flink requires a full download of the Beam source code.
+See https://beam.apache.org/roadmap/portability/#python-on-flink for more information.
```
{:.runner-flink-cluster}
```
-This runner is not yet available for the Python SDK.
+Currently, running wordcount.py on Flink requires a full download of the Beam source code.
+See https://beam.apache.org/documentation/runners/flink/ for more information.
```
{:.runner-spark}
```
-This runner is not yet available for the Python SDK.
+Currently, running wordcount.py on Spark requires a full download of the Beam source code.
+See https://beam.apache.org/roadmap/portability/#python-on-spark for more information.
```
{:.runner-dataflow}
diff --git a/website/src/get-started/wordcount-example.md b/website/src/get-started/wordcount-example.md
index 910a626..1019710 100644
--- a/website/src/get-started/wordcount-example.md
+++ b/website/src/get-started/wordcount-example.md
@@ -390,6 +390,12 @@
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts
+```
+
To view the full code in Java, see
**[WordCount](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java).**
@@ -407,17 +413,20 @@
{:.runner-flink-local}
```
-This runner is not yet available for the Python SDK.
+Currently, running wordcount.py on Flink requires a full download of the Beam source code.
+See https://beam.apache.org/roadmap/portability/#python-on-flink for more information.
```
{:.runner-flink-cluster}
```
-This runner is not yet available for the Python SDK.
+Currently, running wordcount.py on Flink requires a full download of the Beam source code.
+See https://beam.apache.org/documentation/runners/flink/ for more information.
```
{:.runner-spark}
```
-This runner is not yet available for the Python SDK.
+Currently, running wordcount.py on Spark requires a full download of the Beam source code.
+See https://beam.apache.org/roadmap/portability/#python-on-spark for more information.
```
{:.runner-dataflow}
@@ -441,6 +450,11 @@
This runner is not yet available for the Python SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Python SDK.
+```
+
To view the full code in Python, see
**[wordcount.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py).**
@@ -483,7 +497,7 @@
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
--staging_location gs://<your-gcs-bucket>/binaries/ \
- --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
+ --worker_harness_container_image=apachebeam/go_sdk:latest
```
{:.runner-samza-local}
@@ -496,6 +510,11 @@
This runner is not yet available for the Go SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Go SDK.
+```
+
To view the full code in Go, see
**[wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/wordcount/wordcount.go).**
@@ -736,6 +755,12 @@
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.DebuggingWordCount \
+ --runner=JetRunner --jetLocalMode=3 --output=counts
+```
+
To view the full code in Java, see
[DebuggingWordCount](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java).
@@ -787,6 +812,11 @@
This runner is not yet available for the Python SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Python SDK.
+```
+
To view the full code in Python, see
**[wordcount_debugging.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_debugging.py).**
@@ -842,6 +872,11 @@
This runner is not yet available for the Go SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Go SDK.
+```
+
To view the full code in Go, see
**[debugging_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/debugging_wordcount/debugging_wordcount.go).**
@@ -1085,6 +1120,12 @@
--runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts
```
+{:.runner-jet}
+```
+$ mvn package -P jet-runner && java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WindowedWordCount \
+ --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts
+```
+
To view the full code in Java, see
**[WindowedWordCount](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java).**
@@ -1140,6 +1181,11 @@
This runner is not yet available for the Python SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Python SDK.
+```
+
To view the full code in Python, see
**[windowed_wordcount.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/windowed_wordcount.py).**
@@ -1195,6 +1241,11 @@
This runner is not yet available for the Go SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Go SDK.
+```
+
To view the full code in Go, see
**[windowed_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/windowed_wordcount/windowed_wordcount.go).**
@@ -1235,7 +1286,15 @@
```
```py
-# This feature is not yet available in the Beam SDK for Python.
+def main(arvg=None):
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--input-file',
+ dest='input_file',
+ default='/Users/home/words-example.txt')
+ known_args, pipeline_args = parser.parse_known_args(argv)
+ pipeline_options = PipelineOptions(pipeline_args)
+ p = beam.Pipeline(options=pipeline_options)
+ lines = p | 'read' >> ReadFromText(known_args.input_file)
```
```go
@@ -1267,7 +1326,7 @@
```
```py
-# This feature is not yet available in the Beam SDK for Python.
+beam.Map(AddTimestampFn(timestamp_seconds))
```
```go
@@ -1308,7 +1367,16 @@
```
```py
-# This feature is not yet available in the Beam SDK for Python.
+class AddTimestampFn(beam.DoFn):
+
+ def __init__(self, min_timestamp, max_timestamp):
+ self.min_timestamp = min_timestamp
+ self.max_timestamp = max_timestamp
+
+ def process(self, element):
+ return window.TimestampedValue(
+ element,
+ random.randint(self.min_timestamp, self.max_timestamp))
```
```go
@@ -1344,7 +1412,7 @@
```
```py
-# This feature is not yet available in the Beam SDK for Python.
+windowed_words = input | beam.WindowInto(window.FixedWindows(60 * window_size_minutes))
```
```go
@@ -1362,7 +1430,7 @@
```
```py
-# This feature is not yet available in the Beam SDK for Python.
+word_counts = windowed_words | CountWords()
```
```go
@@ -1440,6 +1508,11 @@
This runner is not yet available for the Python SDK.
```
+{:.runner-jet}
+```
+This runner is not yet available for the Python SDK.
+```
+
To view the full code in Python, see
**[streaming_wordcount.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/streaming_wordcount.py).**
@@ -1499,4 +1572,4 @@
* Dive in to some of our favorite [Videos and Podcasts]({{ site.baseurl }}/documentation/resources/videos-and-podcasts).
* Join the Beam [users@]({{ site.baseurl }}/community/contact-us) mailing list.
-Please don't hesitate to [reach out]({{ site.baseurl }}/community/contact-us) if you encounter any issues!
\ No newline at end of file
+Please don't hesitate to [reach out]({{ site.baseurl }}/community/contact-us) if you encounter any issues!
diff --git a/website/src/images/blog/beam-kata/beam-kata-intellij-edu-1.png b/website/src/images/blog/beam-kata/beam-kata-intellij-edu-1.png
new file mode 100644
index 0000000..98a3a80
--- /dev/null
+++ b/website/src/images/blog/beam-kata/beam-kata-intellij-edu-1.png
Binary files differ
diff --git a/website/src/images/blog/beam-kata/beam-kata-intellij-edu-2.png b/website/src/images/blog/beam-kata/beam-kata-intellij-edu-2.png
new file mode 100644
index 0000000..1631c14
--- /dev/null
+++ b/website/src/images/blog/beam-kata/beam-kata-intellij-edu-2.png
Binary files differ
diff --git a/website/src/images/blog/beam-kata/beam-kata-pycharm-edu-1.png b/website/src/images/blog/beam-kata/beam-kata-pycharm-edu-1.png
new file mode 100644
index 0000000..cba9f04
--- /dev/null
+++ b/website/src/images/blog/beam-kata/beam-kata-pycharm-edu-1.png
Binary files differ
diff --git a/website/src/images/blog/beam-kata/beam-kata-pycharm-edu-2.png b/website/src/images/blog/beam-kata/beam-kata-pycharm-edu-2.png
new file mode 100644
index 0000000..87f96ac
--- /dev/null
+++ b/website/src/images/blog/beam-kata/beam-kata-pycharm-edu-2.png
Binary files differ
diff --git a/website/src/images/logos/runners/jet.png b/website/src/images/logos/runners/jet.png
new file mode 100644
index 0000000..1eb2738
--- /dev/null
+++ b/website/src/images/logos/runners/jet.png
Binary files differ
diff --git a/website/src/images/standard-vs-dynamic-sessions.png b/website/src/images/standard-vs-dynamic-sessions.png
new file mode 100644
index 0000000..832a181
--- /dev/null
+++ b/website/src/images/standard-vs-dynamic-sessions.png
Binary files differ
diff --git a/website/src/js/section-nav.js b/website/src/js/section-nav.js
index 11e1c30..ad8e523 100644
--- a/website/src/js/section-nav.js
+++ b/website/src/js/section-nav.js
@@ -95,8 +95,10 @@
var _self = this;
var sectionNavEl = $("." + idSectionNav);
var sectionNavHeight = $(sectionNavEl).height();
+ var mainContent = $(".container-main-content");
- $(".container-main-content").css({"min-height": sectionNavHeight});
+ mainContent.css({"min-height": sectionNavHeight});
+ sectionNavEl.css({"max-height": mainContent.css("height")});
$(window).resize(function () {
if ($(window).width() > CONST.DESKTOP_BREAKPOINT) {
@@ -166,4 +168,4 @@
"classNameNavActiveItem": ".section-nav a.active"
}
).init();
-});
+});
\ No newline at end of file
diff --git a/website/src/roadmap/portability.md b/website/src/roadmap/portability.md
index bb22c0f..4143357 100644
--- a/website/src/roadmap/portability.md
+++ b/website/src/roadmap/portability.md
@@ -45,7 +45,7 @@
The portability API consists of a set of smaller contracts that
isolate SDKs and runners for job submission, management and
-execution. These contracts use protobufs and gRPC for broad language
+execution. These contracts use protobufs and [gRPC](https://grpc.io) for broad language
support.
* **Job submission and management**: The _Runner API_ defines a
@@ -144,25 +144,44 @@
MVP, and FeatureCompletness nearly done (missing SDF, timers) for
SDKs, Python ULR, and shared java runners library.
-Flink is the first runner to fully leverage this, with focus moving to
-Performance.
+Currently, the Flink and Spark runners support portable pipeline execution.
See the
[Portability support table](https://s.apache.org/apache-beam-portability-support-table)
for details.
-### Running Python wordcount on Flink or Spark {#python-on-flink}
+Prerequisites: [Docker](https://docs.docker.com/compose/install/), [Python](https://docs.python-guide.org/starting/install3/linux/), [Java 8](https://openjdk.java.net/install/)
-Currently, the Flink and Spark runners support portable pipeline execution.
-To run a basic Python wordcount (in batch mode) with embedded Flink or Spark:
+### Running Python wordcount on Flink {#python-on-flink}
-1. Run once to build the SDK harness container: `./gradlew :sdks:python:container:docker`
-2. Choose one:
- * Start the Flink portable JobService endpoint: `./gradlew :runners:flink:1.5:job-server:runShadow`
- * Or start the Spark portable JobService endpoint: `./gradlew :runners:spark:job-server:runShadow`
-3. Submit the wordcount pipeline to above endpoint: `./gradlew :sdks:python:portableWordCount -PjobEndpoint=localhost:8099 -PenvironmentType=LOOPBACK`
-
-To run the pipeline in streaming mode (currently only supported on Flink): `./gradlew :beam-sdks-python:portableWordCount -PjobEndpoint=localhost:8099 -Pstreaming`
-
+The Beam Flink runner can run Python pipelines in batch and streaming modes.
Please see the [Flink Runner page]({{ site.baseurl }}/documentation/runners/flink/) for more information on
how to run portable pipelines on top of Flink.
+### Running Python wordcount on Spark {#python-on-spark}
+
+The Beam Spark runner can run Python pipelines in batch mode.
+Please see the [Spark Runner page]({{ site.baseurl }}/documentation/runners/spark/) for more information on
+how to run portable pipelines on top of Spark.
+
+Python streaming mode is not yet supported on Spark.
+
+## SDK Harness Configuration {#sdk-harness-config}
+
+The Beam Python SDK allows configuration of the SDK harness to accommodate varying cluster setups.
+
+- `environment_type` determines where user code will be executed.
+ - `LOOPBACK`: User code is executed within the same process that submitted the pipeline. This
+ option is useful for local testing. However, it is not suitable for a production environment,
+ as it requires a connection between the original Python process and the worker nodes, and
+ performs work on the machine the job originated from, not the worker nodes.
+ - `PROCESS`: User code is executed by processes that are automatically started by the runner on
+ each worker node.
+ - `DOCKER` (default): User code is executed within a container started on each worker node.
+ This requires docker to be installed on worker nodes. For more information, see
+ [here]({{ site.baseurl }}/documentation/runtime/environments/).
+- `environment_config` configures the environment depending on the value of `environment_type`.
+ - When `environment_type=DOCKER`: URL for the Docker container image.
+ - When `environment_type=PROCESS`: JSON of the form `{"os": "<OS>", "arch": "<ARCHITECTURE>",
+ "command": "<process to execute>", "env":{"<Environment variables 1>": "<ENV_VAL>"} }`. All
+ fields in the JSON are optional except `command`.
+- `sdk_worker_parallelism` sets the number of SDK workers that will run on each worker node.
\ No newline at end of file
diff --git a/website/src/roadmap/python-sdk.md b/website/src/roadmap/python-sdk.md
index c1d1fa3..f9e6b24 100644
--- a/website/src/roadmap/python-sdk.md
+++ b/website/src/roadmap/python-sdk.md
@@ -22,14 +22,16 @@
## Python 3 Support
-Apache Beam 2.11.0 is the first release that offers partial support for Python 3. As of 2.11.0, only Python 3.5 on Direct and Dataflow runners has been sufficiently tested, and Python 3 support remains an active work in progress. Current goal is to extend Beam codebase compatibility with Python 3.6, 3.7, address [known issues](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16789854&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16789854) and increase test coverage.
+Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We continue to [improve](https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050) the experience for Python 3 users and phase out Python 2 support ([BEAM-8371](https://issues.apache.org/jira/browse/BEAM-8371)):
- - [Proposal](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE)
- [Kanban Board](https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&view=detail)
- [Python 3 Conversion Quick Start Guide](https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA)
- [Tracking Issue](https://issues.apache.org/jira/browse/BEAM-1251)
+ - [Original Proposal](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE)
-Contributions are welcome! If you are interested to help, you can select an unassigned issue in the Kanban board and assign it to yourself. Comment on the issue if you cannot assign it yourself.
-When submitting a new PR, please tag [@aaltay](https://github.com/aaltay), [@fredo838](https://github.com/fredo838), [@Juta](https://github.com/Juta), and [@tvalentyn](https://github.com/tvalentyn).
+Contributions and feedback are welcome!
+If you are interested to help, you can select an unassigned issue in the Kanban board and assign it to yourself. Comment on the issue if you cannot assign it yourself. When submitting a new PR, please tag [@aaltay](https://github.com/aaltay), and [@tvalentyn](https://github.com/tvalentyn).
+
+To report a Python3-related issue, it is best to create a subtask to [BEAM-1251](https://issues.apache.org/jira/browse/BEAM-1251) , and cc: [~altay] and [~tvalentyn] in JIRA comment. You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate.
